Mary Ann Sieghart: The Tories will need more blue-skies thinking to win the next election

Years ago, back when Boris still needed a surname, a young Steve Hilton hoped to run for Mayor of London. But the Tories were nowhere near ready to choose a shaven-headed, tieless iconoclast as their candidate. Later, Hilton tried to be selected as an MP, but although he looked like many a metropolitan media professional, he was far too modern for the Conservative local associations. They preferred the throwback of a good-looking, public-school banker in pin-stripes.

So, now that the Prime Minister's chief adviser is leaving Downing Street for California, where does that leave his mission to modernise the Conservative Party?

Just as it's hard to recall last summer's heat in a sleety March, so those "let-sunshine-win-the-day" speeches by the pre-credit crunch David Cameron are now a fading memory. But they were Hilton's doing. So was the replacement of the royal blue Tory logo by an informal green squiggle representing a tree. It was meant to remind us not just of the grand old English oak, but also the Tories' new passion for the environment.

Now the skies are grey, we all feel poor and miserable and, as one minister put it to me yesterday: "Greenery has been mugged by economic reality". Some fear the whole programme of Tory modernisation has been mugged by the new austerity and that the departure of its prime mover will see it fall lifeless to the ground. For there are many in the Conservative party who believe that modernisation is an unaffordable luxury when the main task is to reduce the deficit.

They couldn't be more wrong. And it's not as if they are ideological opponents of Steve Hilton. If anything, Hilton sits on the right of the Tory party. But unlike them, he has always understood that, whatever the Conservatives said, they wouldn't be heard if they seemed out of touch and out-of-date. That meant suing for peace with the Sixties – accepting gays, women's rights, multicultural Britain and the green agenda – before the party could hope to be elected.

Hilton, with George Osborne, sold Cameron on this idea. The Prime Minister's whole leadership pitch was based on bringing the Tory party up-to-date. But while most people believe Cameron is sincere, they are reluctant to credit the party with having changed enough. Which explains why they failed to win an overall majority.

At a recent presentation to Tory MPs, the party's head of strategy, Andrew Cooper, said that about 10 per cent of the electorate agree with Cameron on the deficit, think that he is a strong and fair Prime Minister, but don't like his party and aren't inclined to vote for it.

These are the crucial voters who will determine whether or not the Conservatives win a majority at the next election.

They are more likely to be young, female, northern, working in the public sector or a member of an ethnic minority. These voters still see the Tories as callous, out of touch, the party of the rich and obsessed with issues that don't matter to them, like foxhunting. Modernisation – or detoxification – still has a long way to go.

Hilton hasn't always been right in his prescriptions. His obsession with deregulation led him to suggest abolishing maternity rights, which was hardly going to bring women flocking to the party. But at least he understands how we live in 21st-century Britain and in that respect he is still a rarity in his party.

For instance, Hilton has been behind the push to open up government data. Now that sounds dry and irrelevant until you understand its potential. The local crime maps put online by the police last year crashed after receiving 18 million hits in an hour. Already developers are taking raw government data and turning it into useful apps.

But Hilton's impatience with the time it takes to get anything done in government was always likely to boil over, given his mercurial personality. As one big fan of his says: "He's brilliant, absolutely brilliant. Almost impossible to work with, but absolutely brilliant."

If even his supporters find him impossible, you can imagine the eruptions with civil servants.

Many advisers struggle to make the move from opposition to government and Hilton found it particularly hard. In opposition, he and Cameron were like equals, and – because leaders of the opposition can only say things, not do them – Hilton merely had to put words in Cameron's mouth to accomplish his mission. Once in No 10, things were different. Cameron was now surrounded by a private office, sinuous and experienced civil servants and quickly assumed the aura of office. The two were no longer equals and Hilton found the obstacles in the way of doing, rather than just saying them, immensely frustrating. A colleague recalls him walking, shoeless, into a vast Downing Street meeting packed with officials: "This was one man against the system and it was about to swallow him up."

Some advisers manage to be both radical and practical at the same time: Matthew Taylor and Geoff Mulgan spring to mind from the Blair era. But Cameron likes to have balancing personalities around him. So Hilton suggests something radical and the Cabinet Secretary, Jeremy Heywood, will explain why it can't be done. Or Hilton comes up with something romantic and Osborne says it is politically impossible. Now there will be no radical romantic to counterbalance the consolidators.

If his absence is only for a year, that may not matter so much. Ministers are currently obsessed with delivery and implementation rather than blue-sky ideas. In the office of Oliver Letwin, Cabinet Office minister, is a vast whiteboard, with each department and its priorities scribbled in different colours. From time to time, one gets wiped off and another added.

Meticulous follow-through is not best suited perhaps to a thinker like Hilton. Whether he will want to return, though, is another matter. Coming back to an old job is never quite the same. But Cameron will badly need Hilton to inject some passion, energy and modernity into the next manifesto. For without him, Labour's "same old Tories" line will be powerful indeed.