James Mullord, who used to represent the Babyshambles frontman, said he had been accused of betraying Moss by her inner circle, but insisted he had nothing to do with the mobile-phone footage.
Moss lost several lucrative fashion contracts and went into rehabilitation after the Daily Mirror published pictures in which she appeared to be taking the class-A drug. It soon became clear that the video must have originated from one of the 14 people present at a recording session in west London attended by both Moss and Doherty.
Mr Mullord told the London Evening Standard he had received text messages and phone calls accusing him of "double-crossing" Doherty. Shortly after the photographs were published, the rock star's ex-manager left for France and then Scotland, but he vehemently denied he was the source of the film, which has since been broadcast by Sky. He claimed the messages were sent to him while he was in Paris by Miranda Davis, the wife of Doherty's producer Mick Jones, formerly of the Clash.
One of the messages said: "I would never lower myself to the likes of you! Looser! [sic]" A second text read: "Don't think you will enjoy Paris." A third said: "You should be ashamed of yourself you even double crossed mick! And pete! Cant help but think u r capable of anything..." A fourth, sent in response to a denial from Mr Mullord, stated: "... he who doth protest too much... i rest my case!"
Mr Mullord said: "I would never have done this to Pete or Kate. I have done nothing but look after that guy for two years. I considered Pete the closest person on the planet other than my mother. I helped get him out of prison. I helped get him over drugs. It wasn't me.
"This has totally ruined my career working with Pete. I don't know what will happen."
Insiders say someone within the Babyshambles entourage was paid £50,000 up front by the Mirror to obtain evidence that Moss has taken cocaine, plus a further £50,000 when they came up with the goods.
Mr Mullord said: "I have earned quite well over the past two years, but I don't have anything like £100,000 in the bank and I know if I had this video it was worth a million. I wouldn't have done it. I spent a couple of days in England after the story broke trying to get to the bottom of it, so I was a bit surprised when I was in Paris to find myself being accused of it."
He added that he had since turned down an offer of £200,000 from a tabloid newspaper to sell stories about Doherty and his on-off girlfriend, Moss.
Defending the supermodel, Mr Mullord said he had refused to describe her as an unfit mother. "Kate is very good with her daughter and would never take drugs in her presence. I think she has been unfairly treated. She has been painted as a woman with a drug problem. Kate ... is not the kind of person who will have drugs on her, but she will occasionally take drugs."
A spokesman for Babyshambles, who also represents Mr Jones, said: "As far as I'm aware James had nothing to do with it." The spokesman said he believed Mr Jones and Ms Davis were on holiday. Asked whether she believed Mr Mullord had sold the film, Ms Davis told the Standard: "I don't know anything about it."