Andrew Grice: Members fear Clegg could drive away the left that gave him power

"He must be the first party leader to invite people not to vote for him." That was how one bemused senior Liberal Democrat reacted to Nick Clegg's declaration that his party had no future as "a receptacle for left-wing dissatisfaction with Labour".

His remarks, in interviews with The Independent last Saturday and The Observer the next day, caused rumblings at fringe meetings at his party's Liverpool conference and much late-night grumbling in the bars. "We shouldn't be saying 'good riddance' to a section of the electorate," one MP said. "He should remember some Tories came over to us because of the Iraq war too."

Clegg allies insist he was not giving up his party's centre-left credentials because it has hooked up with the Conservatives. They say he was merely recognising that the "Iraq factor" would diminish over time; that some centre-left voters who backed the Liberal Democrats in May would never forgive him for entering the Coalition; and that Labour might tack left in the luxury of opposition, as it appears to be doing over spending cuts.

"The big challenge is how we broaden our base and replace these voters," one Clegg adviser admitted. The hope is that by showing they are no longer a wasted vote and can act responsibly in government, the Liberal Democrats can appeal to a swathe of centre ground and floating voters who have not taken them seriously in the past.

The only trouble is that David Cameron is already fishing in the same pool. It is easy to work out his pitch for the 2015 election: "I told you I was a liberal Conservative and now I have proved it." Mr Clegg's message is harder to envisage. "We need Cameron to lurch to the right and bring Michael Howard and John Redwood back into the Cabinet," quipped one Clegg ally. It won't happen, of course.

The fear among some Liberal Democrats is that Mr Clegg is senselessly giving up on a large number of voters. Andrew Cooper, director of the Populus polling company, told a fringe meeting the Liberal Democrats could not afford to be so dismissive of left-fringe protest voters because there are so many of them – almost 40 per cent of those who backed the party in this year's election.

He believes some of this group, who regard themselves as "non-Conservatives", could be wooed back by the Liberal Democrats if Labour is forced to spell out where it would cut or if the Government's strategy is seen to work.

Mr Clegg's declaration also threatens to put him out of step with his own party. A YouGov survey of 560 Liberal Democrat members found that, on a left-right scale, 65 per cent identified themselves as being left of centre. On a scale of minus 100 (very left-wing) to plus 100 (very right-wing), the members' average score was minus 32. Mr Clegg is seen by members as more centrist, with a score of minus 7, while Simon Hughes, the deputy leader, is rated closer to the average at minus 42.

Mr Clegg made a powerful case for the decision to join the Coalition and the majority of his party now accepts it. This week's conference showed that the party has grown up.

He will have a much harder time at next year's conference – especially if the public rejects the alternative vote in the referendum next May and his party takes a hammering at the elections to the Scottish Parliament, Welsh Assembly and English councils on the same day. In a year's time, Liberal Democrat activists will be able to measure Mr Clegg's clout – notably on whether he can blunt the impact of the cuts on the most vulnerable.

Don't expect the Liberal Democrats to walk out of the Coalition. They have nowhere else to go and they don't want an election before 2015. But it isn't all good news for Mr Clegg. His party's anger could well be directed at him next year. He may need all the public support he can muster to fend off his internal critics.