Cameron's policy guru will not return to No 10

It started with 'hug a husky' and the Big Society. Now the PM has had enough of his blue-sky thinker

Steve Hilton, the blue-sky thinker who virtually invented David Cameron's brand of modern Conservatism, will not return to Downing Street after his sabbatical in California, The Independent on Sunday has learned.

Relations between the Prime Minister and his policy guru are understood to have soured in recent weeks, as Mr Hilton became increasingly frustrated with the slow pace of change in Whitehall and began to doubt Mr Cameron's commitment to radical reform.

Friends of Mr Hilton have ruled out a return. "He won't be back," said one Whitehall insider. "There's already not much trace of him left in Downing Street."

The loss of Mr Hilton – the brains behind the infamous "hug a husky" photo opportunity, the party's tree logo and the Big Society – will be keenly felt by Mr Cameron, whose modernisation of the Tory party was spearheaded by his oldest friend in politics. But sources close to the Prime Minister say Mr Cameron had "had enough" of the wild ideas generated by his policy guru in the run-up to his departure, including slashing the welfare bill by £25bn, cutting the civil service by 90 per cent and giving small businesses the power to "fire at will" to get rid of under performing staff.

Famed for padding around the corridors of power in his socks, Mr Hilton announced in March that he was leaving for an unpaid sabbatical at Stanford University in the US, where his wife, Rachel Whetstone, works for Google. It was assumed he would return in time for the 2015 general election. But another source said: "He absolutely won't be coming back. Cameron is totally furious. He has completely had enough."

Danny Alexander, the Liberal Democrat Chief Secretary to the Treasury, vented his frustration when he told a journalists' lunch: "Our friend who's departed to California has made suggestions in his parting memo. As someone said, most people would bring in a cake on their last day."

However, allies of Mr Hilton dispute that he deliberately caused problems just before jetting off. "The idea that in the last few weeks he went mad is bizarre," said one source familiar with the Hilton working style. "He was putting forward preposterous ideas from day one."

Plans are now being drawn up to put policy formation on a more "formal basis", which could include drafting in high-flying Tory backbenchers including Nick de Bois, Liz Truss and Kwasi Kwarteng into a more politicised policy unit. Having delayed shaking up his ministerial team for two years, the Prime Minister fears many of the ambitious 2010 intake would feel aggrieved if they were overlooked. Government sources admit they hope the Jubilee weekend will kick-start a feel-good period that will last through the Olympics.

After another chaotic week of U-turns, the Treasury has insisted there will be no more changes, but Tory MPs are preparing to push for Mr Osborne to abandon plans to withdraw child benefit from high earners. "It is an issue that is going to come back to haunt George," said one.