Mary Ann Sieghart: Trouble is, David shares Ed's faults

I have a suspicion things would be only marginally different had the Labour leader's brother been in charge of the party

The five-second pause was as telling as the answer itself. I had asked a senior Labour politician yesterday whether he could imagine Ed Miliband as Prime Minister. After a silence that was agonising for us both, he conceded: "It's a struggle." He is not alone – it's a struggle for voters too. A YouGov poll for the Sunday Times finds that two-thirds of people think Miliband isn't providing an effective opposition to David Cameron and 41 per cent of his own supporters think the party made a mistake in choosing him.

So it's not enough for the people around the Labour leader to dismiss criticism of him as tittle-tattle or disloyalty. There is clearly a problem out there, well beyond the borders of SW1. As one Labour MP put it to me: "The really big concern is the people you meet down the supermarket who aren't political, whose opening line is, 'You've not chosen the right leader.' That's a very bad place to be."

He and his colleagues are torn between appreciating Miliband's decency and despairing at his inability to make a mark. "I'm so depressed about it all," said one veteran yesterday. "We're just completely incapable of challenging Cameron. Ed's like a frightened rabbit. We're useless, absolutely useless."

That's a bit strong – after all, Labour is still five points ahead in the polls, well up on its disastrous general election result. But given that the Government has a net satisfaction rating of minus 24 points, and the Liberal Democrats' support has shrunk to single figures, you would expect Miliband to be doing a lot better. As one former Cabinet minister says, "The current success of the party is despite him, not because of him, and that's not sustainable at an election when there's a huge focus on the leader."

What's more, the only concrete test of public opinion under Miliband's leadership so far was pretty disappointing. Last month, he managed to stack up votes in Labour's heartlands in the north of England and Wales, but lost disastrously in Scotland. And in eastern and southern England, Labour experienced a drought as bad as the one now hitting East Anglia. Labour had only one victory, in Gravesham. It failed to win target councils in Dover, Thanet or Thurrock. In Brighton and Hove, the Tories lost eight seats, but the Greens gained 10.

Labour has to win back the south to win an election. But the prospects are receding fast. Seats that Labour lost marginally in 2005 had become safe Tory redoubts by 2010 – like Hemel Hempstead, which the Conservatives won with a majority of less than 500 in 2005, but where Labour now lies third, more than 13,000 votes behind the Conservative MP.

The trouble is that Miliband wasn't elected as the sort of moderate, modernising leader who might appeal to southern voters. He won on a left-wing ticket, even if his head now tells him he can only win a general election from the centre. As a result, his party is all over the place. Does it agree with Ken Clarke's liberal sentencing proposals or not? It depends which day of the week it is. Does it support the Tories' continuation of the education reforms Labour itself put in place? Not if you ask the shadow education secretary, Andy Burnham. And what's its position on health? Who can tell?

The other problem is that some of the best brains in the party have not just slunk into the kitchen, but stumbled off into the night. Blairites such as James Purnell, Alan Milburn and John Reid have left Parliament altogether. Several senior supporters of David Miliband who were offered the job of Ed's chief of staff have turned it down. It's not that the party is dangerously divided, but that one third of it has largely gone absent.

Another third, while ostensibly loyal, is desperate for Miliband to fail. Ed Balls's machine is hard at work on his behalf, trying to soften his thuggish image and, to some extent, succeeding. Balls has been buttering up his colleagues, speaking in their constituencies and trying to substitute charm for his more usual disdain. The plan is that, if Miliband does tank, more likely after the next election than before, Balls can blame the Blairites for disloyalty and seize power.

So it suits him for David Miliband to be talked up as a potential replacement now. He knows that the Labour Party never replaces duff leaders. If Michael Foot, Neil Kinnock and Gordon Brown could survive, Ed Miliband certainly can. He is better than any of them.

And, because we have no counterfactual, we can't tell how a party led by David would now be doing. I have a suspicion that things would be only marginally different. Take this pronouncement: "It is wrong to play games with the welfare state; that is why we cut in half the level of benefit fraud we inherited from the Tories; but it is also wrong to play fast and loose with other people's money on the trading floors of the City. Responsibility must run from top to bottom or else it does not run at all." And this: "We are a party that stands for an ethic of responsibility and solidarity from top to bottom ... People should not leave their duty to each other at the door of the boardroom or the entrance to their local benefits office." One came from the leadership speech David never made; the other forms part of an address Ed will make today.

David would have been better at taking responsibility for the deficit. He would have been tougher on crime and unafraid to outflank the Tories on the right. He would have been more decisive. But there are serious faults that the brothers share.

It's not just that both are seen as rather nerdy and unworldly. Worse, there is a deep-seated uncertainty at their core. David could never quite bring himself to challenge Gordon Brown, even when all the ground had been prepared by his allies. Ed could never quite bring himself to propose toJustine until the political pressure made his vacillation untenable.

And, while Ed is being rightly criticised for his woolliness over policy (70 per cent of voters aren't clear what he stands for), David can be frustratingly opaque too. In Downing Street meetings, when he worked for Tony Blair, he would come out with a string of abstract nouns like "empowerment", but when the Prime Minister pressed him on what he actually meant, he could never put it in concrete policy terms. Were he now leader, that would be a real problem.

In yesterday's poll, 41 per cent of voters thought David would be a better leader than Ed, with only 6 per cent disagreeing. They are right that David would be better, but perhaps not as dramatically as they suppose. For they are not comparing like with like. David is still untarnished; he hasn't had tosuffer any of the opprobrium that comes with being leader.

Had David won instead, there would have been many more banana photos and geeky insults. By now, wistful MPs would be saying that Ed spoke human, unlike his big brother. Labour would still have fantasised about the brother over the water whichever one was in charge. In fact, the differences between the two are nothing like as great as the romantics would have us think.

m.sieghart@independent.co.uk