A recording of him making the allegations was played at the singer’s drugs trial last week, in which he was heard telling ‘Fake Sheikh’ Mazher Mahmood that he knew people that the X Factor judge has slept with.
The trial has since been thrown out of court and Cowell’s spokesperson released a statement denying the claims.
Varey says he was “very drunk” when he made the allegations.
“I can no longer recall all of the details of that evening but I have listened to the recording made by Mahmood,” he said. “I was clearly very drunk.
"At some point and utterly out of the blue Mahmood asked me if Simon Cowell was gay. I said he was.
“This is not true. I do not know why I said it, but assume it is because I was so drunk and felt that I was giving Mahmood the type of information he was seeking.
"I am very sorry for this."
On 22 July, Cowell’s representatives released a statement addressing the claims.
“Simon was referred to during the trial without forewarning, resulting in widespread media coverage of untrue claims regarding his private life,” read the statement.
“In 2014 the question of whether someone is or is not gay is antiquated. (As it happens he isn't, though if he were, he would simply have said so)."
"However, the issue was the false suggestion made by Mr Varey that Simon - who is renowned for his honesty and candidness - had thus not been truthful in the public arena and this is what we have been obliged to clarify.”
Contostavlos’s trial was thrown out on 21 July after the judge believed there had been “serious misconduct” by Sun on Sunday undercover reporter Mahmood, and that there were “strong grounds” to believe that Mahmood had “lied” at a pre-trial hearing.
Contostavlos was accused of helping supply cocaine, which she firmly denied. All charges against her have been dropped.