Steve Hilton has finally left the building. He left Downing Street before the vast majority of the population had the slightest idea who he was. But his departure is important, because it marks a change in the Prime Minister. Hilton's parting shot was a memo saying that welfare spending should be cut more deeply and that civil servants are so inefficient that more than half should be sacked. As Danny Alexander, the Liberal Democrat Chief Secretary to the Treasury, said last week: "Most people would bring in a cake on their last day."

It was a typical farewell: impatient, drastic and thoroughly impractical. But for a long time David Cameron felt that he needed someone who was not in awe of officialdom and customary procedure. Hilton was one of his closest friends and advisers from their time in Conservative Central Office in 1992. That was the formative year for them both: the year in which they won an election against expectations and then lost to the money markets over the ERM, the precursor of the euro.

In the long wilderness years that followed, Hilton saw Cameron as the candidate who could answer the deadly threat posed to the Tory party by Tony Blair. It was Hilton who branded Cameron as a new kind of liberal Conservative. In opposition, he was a creative force behind detoxification. In government, his role changed. It was his job to annoy civil servants by asking that most searching question: "Why?" When he was told, for example, that the Government had no choice but to enact the EU agency worker directive last October, he commissioned his own advice from a friendly Eurosceptic lawyer (who said the Government had no choice).

Some civil servants found him infuriating. Others at least pretended to find him refreshing. One of the most senior says Hilton could "sometimes lose his temper" and was "sometimes out of order", but insisted he was, "overwhelmingly, a valuable non-party-political force".

It was not just officials whose patience he tried, though. In their revised biography of the Prime Minister, James Hanning and Francis Elliott quote a friend as saying: "David's less indulgent of him than he used to be." They say Cameron increasingly preferred the "cold calculation" of George Osborne's advice to Hilton's more "idealistic enthusiasms". They suggest that the cooling was mutual, reporting that Hilton is disappointed not just with the civil service but with Cameron's reluctance to take it on.

It is a buddy-movie break-up to touch the hardest of hearts. Yet it also makes a difference to the character of the Government, because it coincides with a mid-term shift in how Cameron is seen by the voters. The Prime Minister used to be more popular than the Conservative Party. He seemed to play the part of unifying national leader well. But our opinion poll today is the first from ComRes that puts his personal rating below Ed Miliband's.

This is nothing to do with the falling out with Hilton, of course, but it marks a big setback for the Hilton mission to present Cameron as a different kind of Tory. The most important immediate cause was last month's Budget: the cut in the top rate of income tax put Cameron on the side of the rich; and the multiple pile-up that followed put him on the side of incompetence. A damaging mix, summed up in the phrase "out of touch", now applied by two-thirds of voters to their government.

Incidentally, let us note that the importance of Cameron's privileged background in all this is almost nil. He is not seen as being out of touch because he went to Eton but because of his policies. When this newspaper commissioned focus groups last year to investigate class, we found that people regarded Cameron and Miliband as equally "posh". What has changed is that Cameron was seen as competent, quite sympathetic, confident and strong. Now he isn't, and some of those qualities have passed, by the optical illusion of contrast, to Miliband.

The Budget was a historic error, from which all of Cameron's problems flow. It was a betrayal of Hiltonism. In opposition, Hilton would never have contemplated a tax cut for the rich that would so offend the AB liberal demographic group.

Yet Hilton himself was complicit in the turn away from his own policy of Tory decontamination. Once in government and charged with progress-chasing, he allowed his frustrations with Europe and the Liberal Democrats to override the need to appeal to centrist voters. By railing against Europe and offending the Lib Dems by trying to curtail maternity rights and to make it easier to sack workers, he seemed to be reverting to a William Hague 2001 strategy.

The failure of George Osborne's overclaimed deficit-reduction plan has not helped. Last week brought new evidence of the collapse of Hiltonism. Cameron has allowed Ed Miliband to re-forge the coalition of the Blairites and Brownites. Peter Mandelson co-authored an article on the economy with Ed Balls, and Andrew Adonis returned to the fold to review Labour's industrial policy.

Hilton's departure marks the end of Cameron Mark I and the beginning of Mark II. I don't know if Hilton was, in the end, the right person to do it, but it seems to me that Hiltonism – defined as persuading the voters that the Tories are not the party which looks after its own – is the key to the next election.

twitter.com/JohnRentoul; independent.co.uk/johnrentoul