"We are the least racist show you could possibly find in the world," the 54-year-old judge told EntertainmentWise at the Britain's Got Talent press launch.
"So not only is [O'Dowd] ignorant, it's actually insulting to hear him say that and it’s insulting to our audience.”
X Factor winners: Where are they now?
X Factor winners: Where are they now?
1/10 2013: Sam Bailey
The clear winner from the outset, Sam Bailey went on to beat teen singers Luke Friend and Nicholas McDonald. Bailey's cover of Demi Lovato's 'Skyscraper' went straight to number one on 22 December 2013, becoming the first X Factor winner in three years to gain the Christmas number one spot. Her debut album is due to be released on 24 March. She is also currently pregnant with her third child.
2/10 2012: James Arthur
He secured the biggest-selling X Factor winner's song ever with 'Impossible', but last year's victor James Arthur has been at the centre of controversy recently. Arthur got into a Twitter spat with fellow former contestant Lucy Spraggan when he used a homophobic lyric in a rap. The singer apologised on-air for 'abusing [his] position as an X Factor winner' after a performance on the results show a few weeks ago. Arthur's new single 'Recovery' is currently number 19 in the Official UK Singles Chart.
3/10 2011: Little Mix
Girl band Little Mix are one of The X Factor's success stories. Formerly known as Rhythmix, the group released an autobiography in August 2012 and their debut album came out that November, reaching number three. Little Mix's Perrie Edwards is engaged to One Direction's Zayn Malik which, you know, helps with the the whole fame thing. The band toured earlier this year and have enjoyed US success too. Their latest album Salute was released on 15 November.
4/10 2010: Matt Cardle
Singer-songwriter Matt Cardle performed 'Unfaithful' with Rihanna during the 2010 X Factor finals. He famously criticised Syco for being 'very pop' when he, apparently, wasn't. Shortly afterwards, he left Cowell's label. Cardle returned to the charts when his first album 'The Fire' made number eight and was well-received by critics. He recorded a duet with Spice Girl Melanie C, 'Loving You', in the summer and will be touring new album 'Porcelain' in April 2014.
5/10 2009: Joe McElderry
Rage Against The Machine's 'Killing In The Name' famously beat Joe McElderry to the Christmas number one in 2009, following an internet campaign to do battle with Cowell's chart machine. McElderry's luck failed to pick up and he soon found himself winning yet another talent show, Pop Star to Opera Star. Recently, he's enjoyed some success in the classical world and appeared in several West End shows. He is signed with Universal Music's Decca label.
6/10 2008: Alexandra Burke
Alexandra Burke's cover of Leonard Cohen's 'Hallelujah' sold more than a million copies in the UK, a success she followed with five top ten singles and a hit album. Burke has been a brand ambassador for a women's deodorant and signed a modelling contract in 2010. She joined The X Factor as a guest judge during the final of the eight series. The 'Bad Boys' singer has another album out next year but it will be released on RCA records after her recent split from Simon Cowell's label.
7/10 2007: Leon Jackson
'When You Believe' became the fastest-selling single of 2007 when winner Leon Jackson released it before Christmas, but he was dropped in 2009 following poor sales. His third single 'Creative' only reached number 94 in the charts and half his tour dates were cancelled. At present, Jackson is reportedly writing a second album that he will release independently.
8/10 2006: Leona Lewis
She's The X Factor's darling and one of the only acts still with Simon Cowell's label. Leona Lewis enjoyed huge international success after winning the 2006 show and releasing her first single 'A Moment Like This'. Recently, she has struggled, with sales of her latest album 'Glassheart' down steeply from those of her debut in 2007.
9/10 2005: Shayne Ward
After his victory on the second show, Ward achieved the fourth fastest-selling UK single ever with 'That's My Goal'. Several other hits and three albums followed and at one stage, he was reportedly considered to replace Jason Orange in Take That. Ward was dropped six years later to make room for other upcoming stars on Syco's label and has since turned his hand to performing in London musical Rock of Ages with Justin Lee Collins. The Mancunian singer appeared in Dancing on Ice in January 2013 but was eliminated in the fifth round.
10/10 2004: Steve Brookstein
The first winner of The X Factor, Steve Brookstein scored a number one single with 'Against All Odds' but was dumped by Sony BMG just eight months afterwards for being a one-hit wonder. Since his brief success evaporated, Brookstein has performed on cruise ships and was spotted in 2010 performing in a small Cornish pub for £2.50 a ticket. Brookstein no longer appears in publicity for The X Factor and as a result, he has repeatedly hit out at both the show and Simon Cowell. He is, however, still singing, and has a new album 'Soul Journey' out in February.
Simon Woolley, director of Operation Black Vote which promotes racial justice in public life, supported O'Dowd's comments.
"I wouldn't describe voters as racist but there's something more subtle going on which makes it harder for contestants with a 'darker' physical complexion to win public support," he said. "Generally the 'blacker' the artist is, the harder it is for the public to warm to them."
But producers of The X Factor have rejected claims of racism, arguing 40 per cent of the 2013 finalists were black or mixed race.
A spokesperson said: "The X Factor has a long-standing record of celebrating a diverse range of artists, including two former winners, Leona Lewis and Alexandra Burke, and numerous others who have reached the final stages of the competition, including this year's semi-finalists Rough Copy."
Last year's contestant Hannah Barrett described being subjected to "horrible racist comments" during her time on the show, telling Reveal magazine that most of the insults come from "black people who just have lighter skin than me."