Is Britain's most radical police chief being forced out by a gay witch-hunt?

Controversial commander prepared to launch public attack on the Met unless he is returned to his job in Lambeth

It was a typically indiscreet – and provocative – response by Britain's most controversial police officer.

On learning that Scotland Yard intended to move him to a dull backroom job and abolish his high-profile post as Commander of the London borough of Lambeth, Brian Paddick defiantly declared that it was "Lambeth or bust".

But with almost no chance of the 44-year-old getting his way, it looks certain that Commander Paddick and the Metropolitan Police are heading for an explosive confrontation.

Britain highest-ranking openly gay officer is not expected to go quietly An e-mail sent by Mr Paddick to his supporters suggests he is prepared to drag the Met through the courts if necessary.

In the next few weeks, unless a deal can be struck between Mr Paddick and his superiors, the Commander is likely to risk ending his police career by publicly attacking the Met. Book and television deals are already in the offing.

Mr Paddick and his supporters – who include the London Mayor Ken Livingstone – believe he is the victim of a homophobic press and a police service that still finds it hard to stomach a homosexual cop with power. His critics whisper that the forward-thinking, charismatic officer has become increasingly arrogant, outspoken – and even unhinged.

Ever since the Oxford University graduate sprang to national attention with his liberal approach to cannabis, he has become the bogeyman of the right-wing press and been treated with suspicion by some of his colleagues. Shortly after embarking on the "softly, softly" drugs trial, he became front-page news with the publication of a former boyfriend's "kiss and tell" story of cannabis smoking and nights of passion. Mr Paddick was moved to a desk job to await the outcome of a formal police inquiry.

This week, as Mr Paddick was finally told by the Crown Prosecution Service that he had been cleared of any criminal wrongdoing, he was also informed that he would not be returning to his role as Commander of Lambeth. Instead, he would take charge of a computer programme that tells officers how to solve crimes. A job, he insists, to which he is unsuited. He is now considering leaving the police, possibly on grounds of ill health – or suing.

Born in Balham, south London, he rapidly rose through the ranks to become a sergeant in Brixton at 22, later a CID manager at Notting Hill, and chief superintendent in charge of the borough of Merton at 40. He was married between the ages of 24 and 29.

His big break came in December 2000. Impressed by his ability to communicate with minorities, his ideas and honesty – he told a police promotion board that his most difficult decision had been to tell his wife he was gay – Sir John Stevens, the Met Commissioner, promoted him to the rank of Commander and put him in charge of the multiracial borough of Lambeth, considered one of the toughest jobs in policing.

Mr Paddick came to the attention of the press in July 2001, when he set up a pilot scheme in Brixton under which anyone caught with a small amount of cannabis was released without charge. The trial was to allow officers to concentrate on emergency calls, but it was branded by some as a "soft" approach to drugs.

He was soon back in the news when he told a committee of MPs that it was a "waste of resources" to arrest people with small quantities of cocaine and ecstasy. For expressing those views he was reprimanded by Sir John Stevens.

After that furore, Sir John warned Mr Paddick that he should keep a lower profile. But last February it was discovered that Mr Paddick had posted comments on the radical internet chatroom Urban75.com, under the tag name of "Brian: The Commander" saying that he found the concept of anarchism "attractive".

The remarks caused a furore and led to a second carpeting by the Commissioner. His liberal views quickly attracted the attention of the press, most notably the Daily Mail and the Mail on Sunday, which considered him a dangerous maverick who was soft on drugs. And it was with some inevitability that two Sunday tabloid newspapers published the revelations of one of Mr Paddick's old boyfriends.

James Renolleau, 36, claimed his lover of five years had smoked cannabis on 100 occasions – a claim that Mr Paddick denies – and allowed him to smoke the drug in the couple's Westminster flat, which he admits. It was also claimed that Mr Paddick ignored Met rules and failed to tell his superiors that Mr Renolleau was on bail facing criminal charges. The newspapers alleged that Mr Paddick had cruised gay saunas and had sex in public, matters he denies.

Following these allegations Mr Paddick was moved to a job with human resources while a police investigation examined the allegations. It was during his forced exile that relations between Mr Paddick and several of his fellow police chiefs began to sour.

Scotland Yard became increasingly uneasy about the Lambeth drugs experiment, and following evidence that dealers were moving into the area and unsubstantiated claims that children were being hooked on the drug, they ended it in August. Along with other forces they now intend to operate a tougher "three strikes and you're arrested" policy for cannabis possession.

Incensed by what he believed was a whispering campaign aimed at ruining his reputation, Mr Paddick fired off a series of complaints, including a solicitor's letter to the Commissioner, claiming officers were trying to smear his name by describing him as "unstable" and "unpredictable" and repeating false allegations of wrongdoing.

The Commander learned yesterday that he would not face any criminal action. The Crown Prosecution Service announced that there was "insufficient evidence" to prosecute the Commander for possession of cannabis. It also said that it would not be in the public interest to prosecute him for allowing cannabis to be smoked in his home.

The Police Authority meet next Monday to decided whether disciplinary action is necessary. They are expected to admonish him at worse.

With the threat of legal action lifted, Mr Paddick passionately believes that he should have his old job back. In an e-mail to supporters this week, he writes: "The tabloid newspapers should not be allowed to dictate to the people of Lambeth or the Commissioner who they have as the Borough Commander for Lambeth... Nothing less than being restored to my post as Borough Commander for Lambeth will do."

He also complains that he had been given a job that he was unsuited to do, along with the Met's most senior Asian and black officers.

Later today Mr Paddick is due to meet Sir John Stevens to discuss his future. It promises to be an interesting encounter.

Paddick's e-mail to his supporters

My feelings at this moment are as follows:

1. The job in Lambeth is the one I have wanted to do for the past 20 years. I believe that everything I have done to date, all the experiences both in my professional and private life, have been preparing me to take on this role. I am not prepared to throw away the chance to return to this role if there is any possibility of doing so.

2. I believe that I am the best person for the job bar none.

3. There are questions of principle at stake. The tabloid newspapers should not be allowed to dictate to the people of Lambeth or the Commissioner who they have as the Borough Commander for Lambeth. No one should be permanently damaged as a result of breaches of confidentiality by a long-term partner. No one should be permanently damaged as a result of lies told by a vexatious former partner corrupted by chequebook journalism.

4. Senior officers in the Met have tried to smear my reputation. They are now trying to silence me and place me in a role of which I have no knowledge or experience. A major task in the new role is already 18 months behind schedule and the Met is likely to be subject to public criticism by the Home Office. My line managers and I are likely to be saddled with that criticism which would damage our professional reputations.

Nothing less than being restored to my post as Borough Commander for Lambeth will do. I am taking advice from employment lawyers.

Lambeth or bust!

Brian