Michael Brown: David Davis is one of my best friends, but his sole character flaw has got the better of him

In 1986 I foolishly made a speech in the House of Commons when I pledged to my constituents that "so long as I am the Member of Parliament for Brigg and Cleethorpes, no nuclear waste will be disposed of anywhere in my constituency". The Thatcher government had named my area as a potential disposal site, with the consequent collapse in house prices for thousands of my electors. My opponents immediately twigged that if I was unable to change my own party's policy I would have to resign and fight a by-election as an independent Conservative. Thankfully, the government blinked first, abandoned the proposal and I did not have to deliver on my principled, rash, courageous – but wholly self-indulgent – commitment.

David Davis is one of my closest friends and certainly the Member of Parliament I know best. He was my constituency neighbour between 1987 and 1997. Whenever I have been in trouble, he has always been there for me and would bail me out of any hole I got myself into. He is bold, brave, courageous – and wrong – in this incredible decision to put principle before career.

He did not tell me of his intentions yesterday (I last spoke to him last Saturday) probably for fear that I might be successful in dissuading him from his decisions to resign from both the Shadow Cabinet and from Parliament. I would have been incredulous – still am – at such a prospect. There is absolutely no doubt that Mr Davis has done much to successfully re-position the Conservative Party, alongside the Liberal Democrats, as a true defender of our ancient rights and civil liberties. He has been deeply moved by the arguments presented to him by Liberty and has spent months persuading David Cameron and his Shadow Cabinet colleagues of the principles and fundamental nature of habeas corpus.

Under the previous leader, Michael Howard, it was Mr Davis who stiffened the sinews of the Tory party to oppose the introduction of identity cards. And he has seen off three Labour home secretaries and a Home Office minister of state. He has been universally regarded as the best shadow home secretary for years. All this has been gratuitously sacrificed on an issue that, while important, is hardly going to be understood by the electors of Haltemprice and Howden. And there will be many who are bound to question his motives, leading some to conclude that this was a merely a convoluted way of resigning from the Cameron front bench because of policy differences.

I have always regarded Mr Davis as ambitious but principled. However, on this occasion, he has allowed his one character flaw, his ego, to get the better of him. This is the moment when, after his outstanding tenure as shadow home secretary for the past five years, he should have been preparing for the prospect of actually becoming home secretary in 23 months' time. But whether or not he wins his by-election, he will certainly not be re-entering the Shadow Cabinet on his return to Parliament next month. And by the time of the general election in 2010, I cannot see the new Conservative prime minister, David Cameron, picking up the phone to Dominic Grieve to tell him that he will be replaced by Mr Davis. But Mr Davis must know all this. At a stroke, he has removed himself from frontline politics in the knowledge that he will be nothing more than a highly principled senior backbencher.

Whether Mr Davis has already seen tensions ahead in his relationship with Mr Cameron may yet turn out to be the sub-plot of this truly bizarre decision. There has always been a certain nervousness among the Cameroons about the Labour charge of the Tories appearing to be "soft on terrorism". Perhaps Mr Davis is anticipating the risk that, after the Lords rejects the Government's 42-day detention proposals, Mr Cameron will put him under pressure to allow the Government to secure the passage of the bill, with the Tories standing on the sidelines. Perhaps, also, Mr Cameron was taken aback by Mr Davis's commitment – on behalf of the Tory front bench – that the measure would be repealed by an incoming Tory government. Although Mr Grieve has so far re-stated the commitment, it may rank low down the order of priorities of a new Tory prime minister.

Somehow, this extraordinary decision may have more to do with the possibly testy relationship between Mr Davis and Mr Cameron. But it will provide an unnecessary distraction from the Tory task of maintaining the summer heat on Labour. For the next five weeks, the attention of Westminster will focus as much on the internal affairs of the Tories as on the problems facing the Government.

For all Mr Cameron's warm words of support for Mr Davis yesterday, there will surely be anger in Tory circles that today's headlines, instead of being dominated by Labour turmoil following the large Labour backbench rebellion, will be dominated by Mr Davis. This was a moment when Labour, already in the doldrums, should have no Tory distractions to let Mr Brown off the hook.

I simply don't believe that ordinary voters will understand what Mr Davis is doing. Of course he is right to draw attention to the whole raft of anti-liberal and anti- citizen measures introduced by successive Labour home secretaries. But if it is true that polls suggest a large majority of the public are in favour of 42-day detention, it is reasonable to suppose that the same proportion of voters in Mr Davis's constituency similarly support the proposal. The Liberal Democrats have decided not to contest the seat. Labour, sensibly, are doing the same. But there will inevitably be the rag-bag of Monster Raving Loony, Miss Whiplash, anti-Europe and other assorted independent candidates that will detract from Mr Davis's campaign.

And though Mr Davis may set the initial terms on which the by-election is called, there is no guarantee that voters will allow him off the hook over Tory policies on tax, Europe and public spending. They will be as concerned about the price of petrol, energy bills and mortgages. And then what? So he wins – after 21 years as their MP, it would be shocking if he did not. He returns to Parliament and takes his seat on the backbenches – for the rest of his political career. And well before the time of the next general election, I suspect that Mr Cameron may quickly heave a huge sigh of relief that he can ignore his former leadership rival. Meanwhile, Britain will have been denied a principled and decent home secretary through a brave, courageous act of self-indulgence by Mr Davis.