They may not run the country but they run the people who do

'An insidious bunch of backbiters' is how one insider sees Labour's corps of advisers. What do they do for their money – and is their influence really so malign?

The spin-doctor leaned over the lunch table and whispered to the expectant journalist opposite. "We're going to put a thermo-nuclear device under him." Three months later, the minister was out of a job. And the mythical special adviser – the 30-something New Labour storm-trooper, cynically manufacturing the news, riding roughshod over the impartial civil service, bullying journalists and newspaper editors, briefing against his master's rivals, loyal to no one but the cabinet minister he served – was born.

The Jo Moore affair has done little to shake this perception. Instead, it has played to the stereotype. But does the network of special advisers that operates up and down Whitehall really look like its popular image? Of course, they – and the ministers who rely so heavily on them – see it differently.

They paint themselves as hardworking, policy as well as press advisers – and a crucial part of the system, a necessary evil that prevents the civil servants from having to dirty their hands with politics. Through their weekly meetings, held at Downing Street on Friday lunchtime and chaired by a key member of the No 10 policy unit, they claim to ensure the Blairite dream of "joined-up government" is kept alive. And by presenting things professionally – putting a political veneer on government policy – the public gets to know what it needs to know "in context".

Whichever view you take, special advisers – political appointees paid for by the taxpayer – are here to stay. And they are not of themselves anything new. The Tories, notwithstanding their regular attacks on the shady backroom boys of Blairland, introduced the system in the 1970s and were willing exponents of it when in power.

Admittedly, there are more of them now: 65 in total, compared with 38 under the Tories. But many, like the "young turks" of the Labour spin system who ended up on the green benches in 2001, became MPs.

The catch is that people who don't have power and want it don't like people who do, especially when they operate behind the scenes, are not politically accountable and can be paid anything from £34,000 to, in exceptional circumstances, £125,000 for doing whatever it is they do.

Following a string of controversies surrounding spin-doctors in Tony Blair's 1997 government, steps were taken to shine a light at the murky world of special advisers. A code of conduct was drawn up, detailing for the first time the job of a special adviser and the limitations on their work. It sounded pretty innocent, if not on occasion a little dull.

Reviewing policy papers, checking facts and research findings, preparing "speculative" policy papers to "generate long-term policy thinking", liaising with the party and outside interest groups, speech-writing, providing expert advice, attending party functions and representing the views of their minister to the media. There it was: what everyone wanted to know. Inevitably, it was not enough.

The bland document was no match for the tales of backbiting, the refusal of government departments to issue information beyond the identities of their special advisers and the seemingly vast influence exerted by these unelected men and women, derided by John Prescott in 1999 as Mr Blair's "faceless wonders".

The power of the advisers closest to Tony Blair is immense. Alastair Campbell, the tough-talking ex-Mirror journalist and Mr Blair's former press secretary, and Anji Hunter, the Prime Minister's old student friend, a skilled networker who has guarded access to her boss since he entered Downing Street, were both sitting at the negotiating table on the PM's recent trip to Pakistan. It was an indication of just how important they had become.

Also on the payroll at No 10 is Jonathan Powell, Mr Blair's chief of staff. Mr Powell is the silky smooth brother of Margaret Thatcher's former aide Sir Charles Powell who came to Mr Blair's notice after serving in the British Embassy in Washington. Hilary Coffman, who is rare in surviving both the Neil Kinnock and John Smith eras, is a senior figure in the press office. Others, including Geoff Norris, Roger Liddle – an economics expert – Andrew Adonis and Liz Lloyd, are policy advisers. Mr Campbell's partner Fiona Millar handles Cherie Blair.

The Downing Street network reaches out to its Whitehall subsidiaries in the weekly special advisers meeting. But in the intervening period, the departmental special advisers – often in offices right next door to their cabinet minister – are servants to only one master.

This collection of former think-tank employees, ex-party press officers, former Labour-friendly journalists and high-fliers in Labour student politics – many of whom have known each other for years – owe their survival to the survival of their minister. They know their career may only last as long as his or hers does.

One New Labour insider said: "You get special advisers who are undoubtedly experts or you will get people who are just political hacks who will do your bidding. A cabinet minister who is highly ambitious will often hire people who they perceive as having good contacts press-wise.

"They are thinking about their long-term future when they appoint a special adviser. They use them to keep up their contacts in the party, in the unions, make sure they have the right profile. A lot of them are thinking about their own career. A special adviser is there to ensure that future."

When working properly, the trust built up between special advisers and their ministers provides a "short cut" to the ministers themselves which can be helpful both to journalists and the smarter civil servants. It becomes a problem only when political tensions – either rivalry between ministers or mistrust between the civil service and political appointees – are allowed to surface.

One civil service press officer said: "They're fine as long as they know where the line is. When they cross it, as they sometimes do, that's when the problems start." Another source of hostility is elected MPs who resent the power vested in these unelected apparatchiks.

A former minister said in the wake of the Jo Moore incident: "This is long overdue. There's a lot of discontent that we cannot destroy somebody's career who is a special adviser but 60 ministerial careers have been destroyed since 1997 without anyone having done anything wrong. Ministers are regarded as less important and more disposable than special advisers."

Another said some of the special advisers had just become too big for their boots. "It's time they learned a bit of humility." A third was less charitable: "I think they should get rid of the lot of them. They are an insidious bunch of backbiters. The democratic process would lose nothing by losing them."

The truth is that in a government obsessed with image, the only problem with special advisers occurs when they become the story. That's what happened to Gordon Brown's former spin-doctor Charlie Whelan. And it could yet do for Jo Moore. For the rest of them, it may be time to retire to the shadows.