It was all Angela Merkel's fault. That is, the photo of a jubilant David Cameron holding his arms high during a G8 summit at Camp David as Chelsea beat Bayern Munich on penalties.

The timing was bad, coming on the day an updated biography of Mr Cameron disclosed his capacity for "chillaxing" at Chequers with a karaoke machine, a game of snooker, a tennis machine called "the Clegger" and three or four glasses of wine over Sunday lunch.

The G8 photo reinforced an unwanted – and unfair – image of a Prime Minister a bit too lazy for the nation's good in hard economic times, who enjoys power for its own sake. The picture looked staged for the British media, but wasn't. The snapper was a German in a Bayern shirt, hoping to grab Mrs Merkel at his team's moment of triumph. The German Chancellor is more of a footie fan than Mr Cameron. The G8 leaders had dipped into the Champions League final earlier but it was Mrs Merkel who dragged Mr Cameron and Barack Obama back to the TV set when the game went to penalties.

The photo fuelled the gossip among disgruntled Conservative backbenches about the man they call "DVD Dave" because of his love for watching TV box sets with his wife Samantha. It's not that they begrudge him some downtime. It's just, they say, that they don't know what he stands for and they want him to stand up more strongly to those pesky Liberal Democrats. In recent weeks, I have been surprised how strongly feelings are running against Mr Cameron among his own backbench troops. Correction: many do not regard themselves as "his".

Their latest bugbear is his refusal to force the Liberal Democrats to swallow the proposals by Adrian Beecroft, the venture capitalist and Tory donor, to fire workers "at will" –with some compensation but without the right to take employers to a tribunal. Many Tories see that as a missed opportunity to secure growth.

Worryingly for Mr Cameron, criticism is not confined to the usual right-wing suspects who have never forgiven him for not winning an overall majority in 2010 nor come to terms with the Coalition. Plotting against the PM has moved up a gear since the Tories did badly in this month's local elections. I am told that only two MPs have written to the 1922 Committee chairman asking for a vote of no confidence in Mr Cameron. So there is no chance of Nadine Dorries, the MP who branded Mr Cameron and George Osborne as "two arrogant posh boys", gathering the 46 signatures needed to force such a vote.

However, Tory MPs are already plotting about what they would do if Mr Cameron again fails to win a majority in 2015. Many feel they were bounced into accepting the Coalition two years ago and would demand a much bigger say on the Government's programme next time. More threatening is talk of an immediate post-election attempt to oust Mr Cameron – even if he remained PM by putting together another coalition.

It sounds mad, but perfectly sane Tories are talking about it. Who do they see as his successor? Boris Johnson, the only Tory to win a major election since John Major won the 1992 general election. Although he insists he will remain Mayor of London until 2016, close allies believe he will find a Commons seat shortly before the 2015 election.

Mr Cameron's Tory critics moan about an absence of ideology but should not be totally surprised.

In 2005, he identified himself with "practical conservatism". Today he must perform a daily balancing act. For starters, he must juggle the coalition inside his party. One No 10 insider says: "He has got to decide whether the Conservatives are the party of Nick Boles [a prominent moderniser] or Peter Bone [a right-wing, hardline Eurosceptic]."

At the same time, he must keep Nick Clegg and Co on board. Then there's the small matter of public opinion, when any government would be suffering mid-term blues – even without a recession. Andrew Cooper, the No 10 strategy director and polling number-cruncher, has found that, after the economy, the issues which most concern voters are immigration and welfare – and Labour is now doing surprisingly well on these two fronts even though its policies are weaker than the Tories'.

If Mr Cameron addresses this by veering right, Mr Clegg would sniff a gap in the market. He certainly needs one: after losing a big chunk of support to Labour which will not come back, the Liberal Democrats will need to appeal to "soft" Tory voters in 2015.

As Francis Elliott and James Hanning conclude in the biography, Cameron – Practically a Conservative: "David Cameron is still trusted by many to 'do the right thing', even if he cannot in advance tell them what it might be. But if he wants to win an outright majority, or even a second term of coalition, he must provide a more reassuring guide than pragmatism informed by decency."