LOS ANGELES (Reuters) - L.A. Reid, "The X Factor" judge, says he is leaving the TV talent show next season after two years on the panel.

Reid, 56, chairman and chief executive of Epic Records, told "Access Hollywood," the television program and website, he has decided to leave the Fox reality singing show to return to the record label full time.

"I have decided that I will not return to 'The X Factor' next year," Reid told "Access Hollywood" late Thursday. "I have to go back and I have a company to run that I've kind of neglected, and it saddens me a little bit, but only a little bit."

He added that the show was "a nice break, it was a nice departure from what I've done for the past 20 years, but now I gotta go back to work."

Fox declined to comment on Reid's departure on Friday.

Reid joined "The X Factor" when Cowell introduced the show in the United States in September 2011. Reid sat alongside Paula Abdul, former Pussycat Dolls singer Nicole Scherzinger and Cowell.

Cowell fired Abdul and Scherzinger after a disappointing first season and brought in pop stars Britney Spears and Demi Lovato.

But "The X Factor" audiences have dropped this year to an average 9.7 million from about 12.5 million an episode in 2011.

The show broadcasts a two-part finale next week with the winner earning a $5 million prize and record contract.

Epic Records, a unit of Sony Music Entertainment, which commands a roster of artists including Avril Lavigne, will sign the winners of "The X Factor."