It is tempting to ask whether the departure from 10 Downing Street of someone of whom few have heard will matter. Steve Hilton is mainly known for having come up with the phrase Big Society and for not wearing shoes. No one elected him and he hardly even has a job title. He is just a Friend of Dave and now he is off to Stanford University in California for a year, where he will "focus on innovation in government, public services and communities around the world".

Yet you do not need much history to know about the importance of unelected advisers. Constitutionally, elected politicians take decisions and are answerable to Parliament and to the voters for them. But advisers, and especially the Prime Minister's advisers, can be more powerful than Cabinet ministers. Thus Alan Walters, Margaret Thatcher's economic adviser, forced the resignation of Nigel Lawson, because Thatcher preferred Walters' advice to her chancellor's. And Tony Blair wanted Alastair Campbell to stay, because he knew that Gordon Brown was more afraid of Campbell than of any mere minister.

Such examples do not negate the principle of ministerial responsibility. They simply reflect the power in the British system of the prime minister, who remains responsible for the decisions he or she takes, regardless of who gave the advice. But they help to explain why we should take the comings and goings of prime ministerial advisers seriously. And Steve Hilton is Cameron's oldest and closest adviser.

I first met him in the 1992 election campaign, when he was the link person between Conservative Central Office and its advertising agency, Saatchi & Saatchi. He told me that I ought to meet a Tory researcher called David Cameron, because he was brilliant and would be prime minister one day. When he did become prime minister 18 years later, Hilton was by his side. More than anyone apart from Cameron himself, more even than George Osborne, Hilton was the brains behind his elevation to No 10. Cameron was the main author of the plan to "decontaminate" the Conservative brand, but Hilton was the one who came up with the ideas – green, socially liberal, silicon savvy.

After the election, Hilton changed from being a restless seeker after messages, slogans and big ideas to being a restless progress-chaser. That was when he started to clash with cabinet ministers, notably Vince Cable, the Business Secretary. Last year, he took legal advice on European employment law, bypassing all civil service procedure. He wanted to resist giving more rights to agency workers, to let small companies hire and fire at will, and to curb maternity pay. He had an uncomfortable relationship with Osborne, who had a Treasury view of any mad schemes that looked as if they might cost a lot of money, and a realpolitik view of mad schemes that looked as if they might cost a lot of votes.

In this second category were the NHS reforms, which Hilton strongly supported. That was a Battle for Cameron's Ear that Hilton won: last summer's "pause" in the health Bill was a way of saving it rather than ditching it. The Bill will become law shortly; the Welfare Reform Bill was passed last week; and academies and free schools have reached a critical mass: so Downing Street says that now is a good moment for Hilton to spend more time with his wife and young children in Palo Alto.

Well, that is one story. But Hilton's restlessness and his closeness to Cameron mean that his departure will have consequences. That may be why the Prime Minister devoted his speech to the Tory spring conference yesterday to explaining why Hilton's departure would make no difference. Those are not quite the words that he used. The words that he used were, "I didn't come into politics to play it safe", and a lot about "tough and bold actions". He did not mention Hilton by name, but the message was: I have not lost my reforming zeal. (If Hilton worked in the White House, his secret service code name would be Reforming Zeal.)

Thus the Prime Minister tried to pre-empt a story about his premiership that Hilton's departure reinforces, which is that he leads a wishy-washy government that does not believe in very much, meandering to find the path of least resistance between public opinion, the Liberal Democrats and Europe.

This might be known as the Hilton Paradox. As the author of the Big Society, the director of the hug-a-husky photocall and the model of dress-down Downing Street, Hilton is often identified with Cameron's centrist tendencies. But Hilton's greatest frustrations in No 10 have been with trying to do right-wing things. He is a believer in Californian free enterprise and a Eurosceptic. He wanted Cameron to defy the European Court of Human Rights, and talks in private about Britain leaving the EU.

Equally, his departure might seem to confirm Cameron's loss of interest in green policies. Certainly, the promise of the "greenest government ever" seems to have been forgotten the moment it was uttered in the first flush of coalition. But Hilton's departure might have the opposite effect, because since the election Hilton has told friends that he is not convinced about man-made global warming after all.

So Hilton's absence (to contemplate his unlikely ambition to be Mayor of London) will change the centre of government. It will "take the wackiness out of it", according to one of Hilton's friends, ending the "mad quest for new ideas".

But your view of whether or not that is a good thing does not fit easily in a left-right scale. Blairite Eurosceptics who want interesting stories to put in newspapers will regret his departure. But if you are a believer in the virtues of compromise and want a steady liberal conservatism, then you should wish Hilton well in the land of Google and sunshine.

twitter.com/JohnRentoul; facebook.com/john.rentoul