Andreas Whittam Smith: We all pay the price of ministers denigrating the Civil Service

Finding that the Government has been paying some of its senior employees through private companies is like discovering brown leaves in a summer garden. The plants are unhealthy. This excrescence is in the same category as giving bonuses to civil servants and making heavy use of expensive consultants. The three examples have this in common: they are a way of dealing with a demoralised civil service. Provide financial perks; plug gaps. Yet the explanation for this unfortunate state of affairs is the behaviour of ministers themselves. They have systematically denigrated the civil service over a period of 30 years.

Before then, under what Professor Anthony King in his book The British Constitution called the "old constitution", government ministers "mostly governed Britain in collaboration with their officials, seldom in competition with them, never (or almost never) in opposition to them". This partnership, or "companionable embrace" as it was called, could cope with the fact that many senior ministers who were given charge of huge departments lacked the personal qualities appropriate for their responsibilities. This deficiency has, if anything, got worse in recent years.

Jacqui Smith, the former Home Secretary, was charmingly frank in describing her own unreadiness. In a magazine interview, she confessed that she had "never run a major organisation" before becoming Home Secretary in 2007. Nor had any of her immediate predecessors: John Reid, Charles Clarke, David Blunkett, Jack Straw, Michael Howard, Kenneth Clarke and Kenneth Baker.

A sustained denigration of the capacity and ethos of the traditional civil service began when Margaret Thatcher became Prime Minister. She had formed her views when she was Secretary of State for Education in the Ted Heath government that took office in 1970. The Permanent Secretary, Sir William Pile, later recounted how he had dealt with the new minister. It wasn't, as he later realised, the best way of going about things: "We followed our old traditional course of speaking up for what the Department had always done, or what we thought the Department should do, as opposed to what ministers were going to tell us to do... I regarded this as a necessary, professional job to be done up to the point when the minister said to you that we're going to do the opposite."

This didn't impress the new minister. As Sir William recalled: "Within 10 minutes of her arrival she had uncovered two things to us. One is what I would call an innate wariness of the civil service, quite possibly even distrust. And secondly a page from an exercise book with 18 things she wanted done that day."

Civil servants, Mrs Thatcher thought, were not interested in argument. Eager to keep the ship of state on an even keel, they were essentially uncreative people.

Sir John Hoskyns, the first head of Thatcher's policy unit, argued that civil servants could only function "by cultivating a passionless detachment, as if the processes they were engaged in were happening in a faraway country which they service only on a retainer basis". How could you have a radical government, he asked, without radically minded officials?

Christopher Foster, in his book British Government in Crisis (2005), writes that Mrs Thatcher narrowed the scope within which civil servants could challenge ministers, but they were still seen as partners. Mr Blair "treated them as subordinates and excluded them from central policy-making unless ready to be politicised".

Professor King argued that this change was established under Thatcher, confirmed under Major and more or less set in concrete by New Labour: "Can-do man was in and wait-a-minute man was out." If ministers started to march off in the wrong direction, nobody any longer dared say so.

Bringing special advisers into government departments also undermined morale. They are not civil servants but often transfers from the research departments of the political parties or from think-tanks. They answer directly to ministers. They are purely political appointments and leave en bloc when there is a change in government.

Harold Wilson appointed 10 or a dozen special advisers. By the time of his successor, James Callaghan, the number had risen to 30. During Mrs Thatcher's period, the number fluctuated between half a dozen and 30. John Major encouraged their appointment and by the time he left office, the number had grown to 40.

Under Tony Blair there was a vast increase. Sir Christopher Foster commented that after 1997 special advisers replaced civil servants as ministers' alter egos.

In No 10 itself, all but two civil servants were exiled to offices distant from the Prime Minister while special advisers were given privileged access.

Of course when Mrs Thatcher first entered Downing Street, the civil service was far from perfect. Yet the remedies were always in ministers' hands. Finally civil servants will do whatever ministers ask them to do, providing it is lawful. You only have to ask, and ask again, and again until you get what you want. All the rest is flailing around and leads to the sort of mess that this week's report into the UK Border Agency reveals.