Andrew Grice: Darkening mood in the Downing Street bunker

"He is saddened but resilient," is how one close ally sums up Gordon Brown's mood. "He's been around a long time; he's pretty tough. He'll just carry on. He's not going to walk away."

A cabinet minister said Mr Brown was "much more resilient than John Major". It was a revealing comparison. Mr Brown hates being compared to the former Tory prime minister. Labour figures insist the party is not divided like the Conservative Party in the Major years, and point out that Labour enjoys a healthy Commons majority.

Related articles

• Neal Lawson: Labour requires a different leadership
• Labour's civil war intensifies as Balls lashes out at critics
• Jeremy Warner's Outlook: Can Darling survive a reshuffle?
• Minister rules out free care for elderly
• Leading article: Dignity and decent care, from cradle to the grave
• Steve Richards: Gordon Brown has found problems will only erupt by remaining silent about tax policies

But events at Westminster are eerily familiar. The governing party appears to be heading for a disastrous election defeat; a resurgent opposition has an attractive young leader; the Prime Minister is seen as weak and dithering and his authority is openly challenged by some of his own MPs and ministers. Above all, he is not in control of events but buffeted around. Repeated attempts to regain the initiative by announcing new policies are drowned out by what Sir John used to bemoan as "noises off".

The noises off for Mr Brown are getting louder. This was supposed to be the week when Labour moved on from its 1 May local election rout by announcing a draft Queen's Speech tomorrow, setting out the measures for the next parliamentary session, such as help for first-time buyers and a right for more mothers to request flexible working.

Mr Brown was due to kickstart this campaign yesterday with a speech on social care for the elderly and disabled. Yet when his ministers took to the airwaves to trumpet a review of care policy, they had to fend off questions about Mr Brown's future, character and behaviour while Tony Blair was in Downing Street.

This was due largely to a weekend media frenzy over books by three key figures from the Blair era – Cherie Blair, John Prescott and Lord Levy, Mr Blair's chief fundraiser. The combination of books to sell and scores to settle was toxic. Some in the Downing Street bunker hoped that getting all three books at once might clear the decks for the policy blitz. They had no such luck. Frank Field, the independent-minded Labour MP who fell out with Mr Brown while he was welfare reform minister, told the BBC's World Service he would be "very surprised" if he led Labour into the election and that his abolition of the 10p income tax rate could push him out this year.

The tax row is probably what has hurt a bruised Mr Brown most. Although irritated by what he calls "tittle tattle" in books, he is deeply wounded by the allegation that his final Budget as Chancellor clobbered the low-paid. "It's a personal tragedy for Gordon," said one cabinet minister. "He feels he has done more than anyone to put money in the pockets of the poor, yet he is going down as someone who took money away from them. It's a travesty."

Yet there is also criticism in Labour circles that Mr Brown has been "in denial" about the impact of his tax shake-up and the burdens on ordinary families from the rising cost of living.

The new team installed by Mr Brown to beef up the Downing Street operation, headed by his strategy chief Stephen Carter, has advised him to show people he is "on your side".

Some advisers are urging Mr Brown to do less and do it better. But there is little sign of him easing up. Some ministers receive emails from him in the early hours, unsure whether he has not been to bed or has got up early. No wonder he looks tired.

Mr Brown's critics say he must make real progress by the party's annual conference in September. That deadline could be shortened if Labour loses next week's by-election in Crewe and Nantwich.

The sense of panic has not spread to the Cabinet. Ministers discount talk of the men in grey suits, led by Jack Straw, asking Brown to stand down for the sake of the party. "It's up to Gordon," said one Cabinet member.

Time is running short. "If we don't stabilise this very quickly, we are fucked," said another minister. As he is one of Mr Brown's closest friends, you can imagine what his enemies are saying.

More from Andrew Grice