Celebrity website TMZ reported that producers dropped the 27-year-old from Newcastle, in northern England, because they feared US audiences would have trouble with her accent.
Ex-Pussycat Doll Nicole Scherzinger, who has appeared on the UK show, was tipped as a likely replacement.
Cole was due to appear alongside former "American Idol" judge Paula Abdul, Antonio LA Reid and "The X Factor" creator Simon Cowell, when the show starts in September.
She was not appearing in "The X Factor" in Britain this autumn because of her American role.
Cole's UK publicist has declined to comment on the story but the Sun reported that she was quitting the programme because she was missing friends and family in Britain.
A source on the show told the paper: "It hasn't worked out. She's homesick and wants to return to the UK. She's had enough. Whether she ends up back on UK X Factor is open to speculation."
