Gordon Brown appealed today for discipline in the Labour Party amid fresh criticism of his leadership following Labour's humiliating by-election defeat in Norwich North.
The Prime Minister used a newspaper interview to insist there was work to be done over the summer as MPs and ministers headed off for their summer break.
He told the Sunday Mirror: "We've got to show that we are a disciplined party getting on with the work of government.
"I think people are very clear that we've got a task ahead. We've got work to do to prepare for the autumn."
The premier said he was determined to spend plenty of time with his children during his break from Westminster.
But he also stressed that he would remain focused on the country, currently facing the challenges of swine flu and recession.
"My attention is focused on what I can do for the country. I will not be diverted," he said.
His comments come amid renewed concerns about his leadership among Labour backbenchers, who said he had until the party conference season in September to demonstrate he was up to the job.
The Tories overturned a Labour majority of more than 5,000 in Norwich North on Thursday to win by 7,348 votes.
A repeat of the 16.4% swing in a general election would install David Cameron in Number 10 with a majority of more than 200 seats in the Commons.
Senior Labour backbencher Barry Sheerman said Labour needed to get its "act together" and stop blaming its difficulties on the MPs' expenses scandal.
"It's partly a question of leadership, it's partly a question of ideas," he said.
Mr Sheerman accused Mr Brown of failing to connect with voters, adding: "I'm saying he's got the summer to recognise this isn't about members' expenses, it's about something much more fundamental.
"People don't want to know about what we've done, they want to know what we're going to do and whether we've got the leadership and the forward-vision thinking to do it."
Former Home Secretary Charles Clarke blamed the dramatic swing away from Labour on Mr Brown's response to the expenses crisis.
The by-election was forced by the resignation of Labour's Ian Gibson as the MP for Norwich North after he was told by a party disciplinary panel he would not be allowed to stand at the next election over revelations about his expenses claims.
Mr Clarke, MP for the neighbouring seat of Norwich South, said the Prime Minister had not been fair in his treatment of MPs.
Writing in the Independent, Mr Clarke said: "This incompetent and unjust style has deeply damaged democratic politics.
"Moreover the appalling result in Norwich illustrates the important political side-effect that Labour, as the governing party, has been injured worst of all.
"The main reason for the Norwich result was that voters there were quite clear that it was for them, not the Labour leadership, to decide whether or not Ian Gibson remained their MP."
Another Labour backbencher, Kate Hoey, said the Prime Minister needed to look at how he led the Labour Party.
"By-elections are always unique but there is no doubt about it that this is a bad result," she told the BBC.
"The Prime Minister, I hope, will be looking at how he's looking to lead the party and to talk to the party, and a lot of party members feel that they are not listened to."
Ms Hoey played down the prospect of a leadership election, saying there was nobody vying for the position at present, but warned that the party conference could become a focus for discontent.
Tony Lloyd, chairman of the Parliamentary Labour Party, accepted that morale was low among MPs, but also dismissed any suggestion of a bid to oust the Prime Minister.
"There will be no leadership challenge to Gordon Brown," he said.
The Prime Minister is taking a month off from Westminster, but is not going abroad. Instead he will spend the time at home in Scotland and in the Lake District.
The workaholic raced back from Dorset just a few hours into his summer holiday to deal with the foot and mouth crisis in 2007.
And he told journalists on Wednesday that this year he would only be spending "a few days" on holiday before "getting on with the job".
But, in his Sunday Mirror interview today, he made clear he wanted to dedicate some time to wife Sarah and their two sons, five-year-old John and Fraser, three.
"If you are away for a week you notice how your children change and you have got to re-win their interest," he said.
"It's important you understand you have got to spend time with your children."