Is Coulson the most dangerous man in Britain? (As the NoW might say)

Some senior Labour figures think so. As the privacy controversy spirals, Andy Coulson is truly in the thick of it. But where will this tale of modern Britain end?

Until last week, few people would have recognised a picture of Andy Coulson. Tell them he is a 42-year-old former journalist from Basildon in Essex who works for the Government and they would be none the wiser, and that is just the way he wants it. The greater the lustre of his boss, and the lower his own profile, the better. He is one of the most powerful people in the country, the man David Cameron goes to first for advice on the media.

Alastair Campbell used to say that when the PR person becomes the story, then it is time to go. Last week, when The New York Times published a detailed investigation into illegal activities at the News of the World, Andy Coulson became the story even before most people had heard of him, and he now represents the most toxic threat to the credibility of the Government. It is a remarkable turnaround, and one that casts light on the fulcrum of public life, but which the public rarely gets to see.

Cameron was grateful to appoint Andy Coulson, a former editor of the News of the World, as his head of communications in May 2007. He had been searching for some months for someone to perform the job Campbell had performed for Tony Blair. He sounded out several journalists, and offered the job to at least three of them. Then William Hague, whose judgement Cameron trusted, suggested Coulson might be just the man. He was tough, understood the rougher end of the press and available.

The problem was that Coulson was soiled goods. Under his editorship, two people working for the News of the World had gone to prison for illegally listening in to mobile phone messages. Evidence was produced in court that the practice was not restricted to hacking into the phones of Princes William and Harry. The court accepted that five other celebrities had also been victims.

The NoW said that the paper's royal correspondent Clive Goodman and private investigator Glenn Mulcaire had gone out on a limb. Nobody on the paper had sanctioned their behaviour, it said, and no other executives had authorised such illegal methods. Coulson, a notably hands-on editor, resigned, ostensibly doing the decent thing and saying he had to accept responsibility for what had happened on his watch. In his statement, though, he was careful not to deny that he had known about it. (That was left to the off-the-record briefers.)

In fact, nobody at News International fought to keep Coulson, even if there was a feeling that the quicker he went, the sooner he could be rehabilitated and come back into the fold. On the contrary, he had embarrassed them, and he had to pay the price. He was no happier than were Goodman and Mulcaire – who sued for wrongful dismissal and later signed confidentiality agreements – about carrying the can for something that many red-top journalists regard as an accepted trick of the trade.

Coulson was not without friends, and plenty of people had reason not to see him out of work for too long. The obvious plus for Cameron was that he could offer an invaluable link with Rupert Murdoch's News International, owners of The Sun, The Times and the News of the World. Murdoch has a record for backing election-winners. His benediction almost makes victory self-fulfilling. While Cameron was seeking to convince sceptical parts of his party that he was more than a jumped-up media creation, the endorsement of the man whose papers had backed Margaret Thatcher and Blair would be invaluable. Yet Murdoch thought Cameron a lightweight. What had he ever run? Did he have the steel of a real leader? Cameron needed a bridgehead in Wapping.

With the encouragement of the PR man Matthew Freud (Rupert Murdoch's son-in-law, handily) and George Osborne, Cameron agreed to take on Coulson. Whether Cameron asked Coulson how much he had known about the illegality at the NoW under his watch is unclear, although he seems to have accepted the assurances of Hague, Osborne and Freud that there were no skeletons in his cupboard. Colleagues at the time said "the pluses outweighed the minuses".

And for three years, Coulson offered Cameron exactly what he wanted: guidance as to what would "play" with the press, a feel for what "real" people (ie, those outside Cameron's narrow, gilded set) wanted, and access to the nation's most powerful media baron. As Gordon Brown stumbled and fell out of favour, the Murdoch empire rebalanced, as modern jargon has it, towards the Tories. Rebekah Brooks, Coulson's close confidante and predecessor at the helm of the News of the World and now chief executive of News International, came to appreciate the Cameron magic. James Murdoch, her boss, was also won over, with the grudging and uncertain endorsement of his father, Rupert. The new Camerlot that was going to run Britain had been born, and Coulson was a crucial part. Not bad for a man some thought should have gone to prison a couple of years earlier.

But the phone hacking issue refused to go away. When the police took away Glenn Mulcaire's files, they took away a hot potato that continues to glow in the offices on New Scotland Yard. A lot of people had had their phones hacked. Gordon Taylor, chief executive of the Professional Footballers' Association, was one. He was the person the players would ring up for advice if they had been caught doing what overpaid, testosterone-charged young men tend to do. Who better to be the subject of some illegal "finger-fishing", as some called it? His voicemails would be a treasure trove of gossip. He took a legal case for breach of privacy (his own, interestingly). The News of the World agreed to settle, forking out a total of approaching £700,000 in damages and costs.

More recently, another of the five in the original court case, Max Clifford, brought a case. After Mr Justice Vos ordered the News of the World to reveal all the relevant documents, including, crucially, details of precisely who Mulcaire had had dealings with on the paper (assuming it was not just Goodman), a deal was struck. Clifford was paid £1m and dropped the case. A third person among the five, the sports agent Sky Andrew, has a case pending and has so far rejected attempts to settle.

There are an estimated 20 other cases pending against the paper, most involving well-known public figures. A handful of lawyers approached the police, asking if their clients were among those who featured in the Mulcaire files, and many had this confirmed. But what of the police? The apparently industrial extent of illegality surely calls for more than two hapless individuals going to prison for a few months. The police were asked about this last year by the Commons Select Committee on Culture, Media and Sport. Assistant Commissioner John Yates explained that the presence of such names and/or phone numbers (sometimes with security PINs) on those files is not in itself proof that a phone was hacked. When the case against Goodman and Mulcaire was brought, he explained, they went after those cases that had the greatest chance of success. In some cases the victims – often embarrassed by the stories the News of the World had run – preferred not to give evidence. In any event, the impression had been created that the extent of the crime had not been matched by the number of collars felt.

At which point, the conspiracy theorists went into overdrive. Why were the police "covering up" for the Murdoch empire, they asked. Why was Andy Hayman, the former officer in charge of the Yard investigation who said police had found "only a handful" of victims, now writing a column for News International? The Commons Media Committee wanted to know how much collusion there was between the Met and the News of the World. Hadn't the NoW's celebrated "Fake Sheikh", responsible for last week's sting against three Pakistani Test cricketers, been responsible for a number of stories that led to police prosecutions and imprisonment? This, it was said, was evidence of an unhealthily cosy relationship, further evidence that Rupert Murdoch rules the world.

The truth is rarely so simple, surely, and doesn't seem to be in this case. Cock-up has played a major hand, though. Sources in the Met agree the job could have been more thorough, that many whose names came up in Mulcaire's files should have been told that they may well have been victims of hacking. They should also have followed up on some loose ends in the court case when other NoW journalists were mentioned. (A "Neville" came up, incriminatingly, but despite the paper's senior reporter being called Neville Thurlbeck, he was not approached.) But while there is indeed a degree of co-operation between the police and the NoW – and other newspapers, for that matter – any suggestion of impropriety is strongly denied.

"We did the Goodman/Mulcaire case; we made an example of them, but there was a limit to what more we could do that would have a strong probability of getting a conviction," said one senior officer. "To be quite honest, after that our priorities lay elsewhere." There continues to be concern among lawyers that the police, in part for data protection reasons, are withholding evidence against NoW, but the police say – because of the difficulty of securing further convictions – they feel the matter has run its course. "We just want it to go away," said one source some weeks ago.

It may not. Last week's NYT article alleged that the police had failed to pass on full details of their evidence to the Crown Prosecution Service, adding further weight to the charge that they had withheld evidence. Such claims seem likely to be examined by the Home Secretary, Theresa May, who has the option of asking the Independent Police Complaints Commission to investigate the Met's handling of the saga.

But what of Coulson? Last week, Sean Hoare became the first of his former colleagues to go on the record in denouncing Coulson, a charge Downing Street ascribes to personal bitterness. Yesterday came reports that Coulson has been asked by the defence's lawyers to appear at the trial of the former MSP Tommy Sheridan in his trial for perjury. The NoW had made allegations about Sheridan's sex life, for which Sheridan won damages.

Elsewhere, lawyers are plodding through police files as quickly as the police allow. More and more are lining up to bring cases against the NoW, and before long one seems likely to end up in court. Today, The Independent on Sunday reveals that Peter Mandelson's number was found on Mulcaire's files, and the list of big names is far from exhausted. The Commons Media Committee was told by senior executives on the News of the World – including Coulson, for the first time – that they knew nothing about the phone hacking, a claim the committee found "inconceivable". With that claim of ignorance, Coulson has raised the stakes.

David Cameron said everyone was entitled to "a second chance", but that was before Coulson told the committee he knew nothing about the phone hacking. It is possible he didn't, and that the circumstantial evidence and his former colleague Sean Hoare are misleading us. Asked about it by Cameron about 18 months ago, he said "categorically" there was nothing in it, that he knew nothing about any phone hacking.

So what can Cameron do? Coulson is the story, and is a distraction, even if he is as pure as the driven snow. The weight of circumstantial – and, increasingly, legal evidence – is too strong for this to blow over. Cameron's instinct, if he continues to believe in Coulson's innocence, is to circle the wagons and back his man, for personal as well as professional reasons. So should he ask him to step aside until this is all cleared up? For his friends at Wapping, even that would be an admission of guilt, which could have major and diverse knock-on consequences for his former colleagues.

But if Cameron comes to doubt Coulson's innocence all that could be irrelevant. He is equipped with a considerable temper, as those who have let him down can confirm, and he does not respond well to being shown to have been in the wrong. If he joins those who think it "inconceivable" that Coulson didn't know, even Coulson's worried friends in Wapping are unlikely to be able to save him.

Indeed, the spotlight that has only recently been pointed at Coulson could once again swivel towards them. If it was to emerge that Coulson knew, then did the executive chairman of News International, Les Hinton, the managing editor, Stuart Kuttner, and Rebekah Wade, each of whom also told the Commons they knew nothing?

Even if Coulson is able to tough it out, presumably that won't mean the legal claims will go away. So far, the NoW has paid out more than £1.5m in settlements to unhappy victims of its phone hacking. If there is nothing to hide, maybe they will let one of these cases go the distance.

What would settle the matter once and for all is if Glenn Mulcaire, the private investigator at the centre of it all, were to break his confidentiality agreement and tell his story (although paying a convicted criminal would be a breach of Press Complaints Commission rules). Mulcaire, a former professional footballer, lives, jobless, in south London with his wife and five children, seemingly mortified by his past. He has had approaches to tell all, but they have come to nothing. In any other story of this type, the press would be falling over itself to offer him money and he'd be fighting off the bidders. But because so few papers have been following the story, his market value has remained low. Who knows? That may yet change.

Celebrity trail: Footballers, politicians, models – all were on the list of targets

Targets of the phone-hacking scandal have one thing in common. They were all at the centre of a story the News of the World was working on or else they stood in the way of one. The Old Bailey heard during the trial of the royal editor, Clive Goodman, and private investigator Glenn Mulcaire that the first attack came in early 2005 but compelling evidence suggests the practice had been going on for at least two years before then.

Other targets of Mulcaire named during the trial included Simon Hughes MP, the current deputy leader of the Lib Dems. The supermodel Elle Macpherson, the subject of numerous tabloid tales was another. The other two were both men who acted for countless celebrities. Max Clifford, the PR guru, was one. The other was the football agent Sky Andrew, whose clients included Sol Campbell and Jermain Defoe. Other football faces targeted included Sven-Goran Eriksson, the former England coach, and the striker turned pundit Alan Shearer.

A police raid on Mulcaire's home found a list of a further 50 people for whom he had obtained mobile phone details, but for which police said there was no evidence they had been hacked. These include then serving cabinet ministers including Tessa Jowell, at the time culture secretary. One report suggested the Scotland Yard commissioner Sir Ian Blair was also on the list although police later denied this. Brian Paddick, once the UK's most senior openly gay policeman, was one who did feature.

Other lists featured Gwyneth Paltrow and George Michael. The late reality TV star Jade Goody and the radio presenter Vanessa Feltz also attracted Mulcaire's interest. Politicians were sprinkled liberally through his work, ranging from John Prescott to backbenchers such as Lembit Opik and Chris Bryant.

Andy McCorkell

Timeline...

August 2005 Private investigator Glenn Mulcaire, in the pay of the News of the World, hacks into the phone messages of Gordon Taylor, chief executive of the Professional Footballers' Association.

November 2005 NoW royal editor, Clive Goodman, commissions Mulcaire to hack into phone messages of staff at St James's Palace. Stories appear about Prince William, leading to suspicions that telephone calls are being intercepted.

December 2005 Scotland Yard is alerted.

November 2006 Goodman admits conspiracy to intercept calls "without lawful authority". Mulcaire pleads guilty to the same offence and to further charges of intercepting phone calls. NoW editor Andy Coulson says: "I have put in place measures to ensure that they will not be repeated."

January 2007 Goodman sentenced to four months prison and Mulcaire six months. Coulson resigns.

July 2007 David Cameron makes Coulson the Conservative Party's director of communications. Gordon Taylor sues NoW for involvement in illegal interception of messages on his phone.

July 2009 Three inquiries launched by the Director of Public Prosecutions, the Press Complaints Commission and a parliamentary select committee after newspaper revelations that News International, owners of the NoW, paid more than £1m to settle legal cases threatening to reveal evidence of criminal methods used to get stories.

February 2010 Culture select committee criticises the "collective amnesia" of News International chiefs.

September 2010 Newspaper revelations that phone hacking is more widespread than News International has acknowledged.