Eight years ago, I left my civil service job as director of the UK Anti-Drug Co-Ordination Unit. I went partly because I was sick of having to implement policies that I knew, and my political masters knew, were unsupported by evidence. Yesterday, after a surreal flurry of media requests referring to a blog I wrote that questioned the wisdom of the UK's drug policies, I found myself in the thick of the debate again, and I was sorry to discover that the terms hadn't changed a bit.

I was being interviewed on the BBC World Service, and after I tried to explain why I believe that drugs should be decriminalised, the person representing the other side of the argument pointed out that drugs are terrible, that they destroy lives. Now, I am a deeply boring, undruggy person myself, and I think the world would be a better place without drugs. But I think that we must live in the world as it is, and not as we want it to be. And so my answer was, yes, I know that drugs are terrible. I'm not saying that drugs should be decriminalised because it would be fun if we could all get stoned with impunity. I'm saying that we've tried minimising harm through a draconian legal policy. It is now clear that enforcement and supply-side interventions are largely pointless. They haven't worked. There is evidence that this works.

Unfortunately, evidence is still not a major component in our policy. Take cannabis. When I was in the Anti-Drug Unit, the moves towards making it a class C drug began, and I hoped that our position on drugs was finally moving in a rational direction. But then Gordon Brown ignored his scientific advisers to make it a class B again. It was a decision that pandered to the instincts of the tabloids, and it made no sense whatsoever.

There is no doubt at all that the benefits to society of the fall in crime as a result of legalisation would be dramatic. The argument always put forward against this is that there would be a commensurate increase in drug use as a result of legalisation. This, it seems to me, is a bogus point: tobacco is a legal drug, whose use is declining, and precisely because it is legal, its users are far more amenable to Government control, education programmes and taxation than they would be otherwise. Studies suggest that the market is already almost saturated, and anyone who wishes to purchase the drug of their choice anywhere in the UK can already do so. The idea that many people are holding back solely because of a law which they know is already unenforceable is ridiculous.

Ultimately, people will make choices which harm themselves, whether they involve diet, smoking, drinking, lack of exercise, sexual activity or pursuit of extreme sports. In all these instances, the Government rightly takes the line that if these activities are to be pursued, society will ensure that those who pursue them have access to accurate information about the risks; can access assistance to change their harmful habits should they so wish; are protected by a legal standards regime; are taxed accordingly; and – crucially – do not harm other people. Only in the field of drugs does the Government take a different line.

The case is overwhelming. But I fear that policy will not catch up with the facts any time soon. It would take a mature society to accept that some individuals may hurt, or even kill themselves, as a result of a policy change, even if the evidence suggested that fewer people died or were harmed as a result. It would take a brave government to face down the tabloid fury in the face of anecdotes about middle-class children who bought drugs legally and came to grief, and this is not a brave government.

I think what was truly depressing about my time in the civil service was that the professionals I met from every sector held the same view: the illegality of drugs causes far more problems for society and the individual than it solves. Yet publicly, all those people were forced to repeat the mantra that the Government would be "tough on drugs", even though they all knew that the policy was causing harm.

I recall a conversation I had with a Number 10 policy advisor about a series of announcements in which we were to emphasise the shift of resources to treatment and highlight successes in prevention and education. She asked me whether we couldn't arrange for "a drugs bust in Brighton" at the same time, or "a boat speeding down the Thames to catch smugglers". For that advisor, what worked mattered considerably less than what would play well in the right-wing press. The tragedy of our drugs policy is that it is dictated by tabloid irrationality, and not by evidence.