Andrew Grice: The hurdles for Mr Brown get higher by the day

The last time I went to see Tony Blair in Downing Street, the first thing he said was: "They tell me you have gone over to Gordon." I was taken aback. Gone over? To dinner? To his office? No, of course, to Gordon's gang, a defector from Tony's tribe.

I mumbled something about being a journalist, not a player, and wanting to talk to all sides in all parties. Then we had a good chat as he chomped his way through a giant bacon sandwich.

Mr Blair's revealing opening remark came back to mind this week when Charles Clarke, in his latest volcanic eruption against Gordon Brown, complained that Brown allies used "Blairite" and "Blairism" as a term of abuse in Labour's internal debate. It was rather overlooked because of his call for Mr Brown to stand down if he can't revive Labour's prospects soon – and for the Cabinet to push him out unless he jumps first (which he won't).

Mr Clarke, the former home secretary, argued in the New Statesman that it was time for Labour to move on from the Blair-Brown faultline which has run through the party since Mr Blair became its leader in 1994. Proving his point, Tony Woodley, joint leader of the biggest trade union Unite, popped up on Radio 4 yesterday and used "Blairite" as an insult.

If there is a Labour leadership election – and it is still an "if"– no one will rush to be the "Blairite" candidate. David Miliband's instincts were always to the left of Mr Blair's, but he may struggle to win such a contest if his opponents label him as the Blairite.

It's a bit odd that Labour folk think like this, given that Mr Blair won three general elections. It's also surprising that Mr Blair is still such a presence in Labour's debate, as he left No 10 and the Commons 16 months ago. He has tried not to stalk the Downing Street corridors like a ghost, in the way Margaret Thatcher did not allow John Major to escape her shadow. Whatever his private dismay at Mr Brown's performance, he has not uttered a word of criticism.

In a leaked email, Mr Blair warned that his successor had played into the Tories' hands because he "dissed" Labour's record since 1997 and "junked the TB policy agenda but had nothing to put in its place".

Interestingly, this view is not confined to Blairites. It is shared by some of Mr Brown's closest allies, who believe their man's biggest mistake is his failure to set out a mission statement for his government. "Gordon has wasted too much time in the past year looking backwards – at the Blair period and his own record at the Treasury," one told me. "He has gotto look forward now."

To say that time is short is an understatement. Mr Brown needed a good start to his much-trumpeted fightback this week. Great expectations about a non-existent "economic plan" were allowed to build, so that his measures on housing and fuel poverty were doomed to be dubbed a "damp squib" in the same media shorthand.

It has been an inauspicious start. Alistair Darling's off-message gloom about the economy. Jacqui Smith's leaked warning that crime will rise in the downturn. Mr Clarke's outburst. And a cack-handed decision to announce the housing market measures on the day when the Organisation for Economic Co-operation and Development was to publish its forecasts. Its prediction that Britain is entering a recession stole the headlines.

"Gordon has got to show leadership and grip," one cabinet loyalist said. Other members of the Cabinet appear to be keeping their heads down. I sense that some are sitting on their hands rather than rushing to help him make a success of his "one last chance" to turn things round.

"We have not given Gordon much support in the past year," one cabinet member admitted. Others concede they have become virtual technocrats, swamped by the pressures of running a department, but blame an ineffectual Downing Street machine for the lack of clear political marching orders and coherent line of attack on David Cameron's Tories.

Allies are urging Mr Brown to remember his own mantra that Labour is "best when it's boldest" rather than fret about the reaction of a largely hostile media, or comparisons with his predecessor. It is finally time for him to be his own man, they say.

But the hurdles for Mr Brown get higher by the day. His friends are praying that he will throw off his self-imposed shackles and explain his Government's mission. After a dodgy start to the Great Relaunch, they admit an awful lot is now riding on his address to the Labour conference on 23 September. It is no exaggeration to say he needs to make the speech of his life.

Interesting? Click here to explore further