Michael Brown: Watch out for the 'others' - they could stop David Cameron winning a general election

In the past few weeks, unease among the grass-roots has resulted in some defections to Ukip

It was probably no more than coincidence that David Cameron's positive embrace, this week, of Iain Duncan Smith's publication emphasising the benefits of marriage as a way of countering "breakdown Britain" came only the day before the latest Populus poll shows the Tory advance stalling. Yesterday's poll shows the Tories down 2 points, on 34 per cent, giving them only a single percentage point lead over Labour. And some on the right suggest that this is due to a drift away from the Tories by more traditional voters uneasy at Mr Cameron's pitch to the centre ground. They appeared to be reassured, however, by the apparently more traditional fare foreshadowed by Mr Duncan Smith's social justice commission.

Certainly Mr Cameron's embrace of the commission contrasts with the cooler reception given to Lord Forsyth's similarly traditional tax proposals two months ago. On that occasion, Mr Cameron left it to his shadow Chancellor, George Osborne, to distance the front bench from calls for tax cuts. By contrast, Mr Cameron seemed happy this week to be identified personally with a message more subliminally Thatcherite than that usually offered by his Notting Hill outriders. Does this suggest that Mr Cameron is anxious to throw a bone or two in the direction of the Tory right for fear of losing his "core" vote?

Certainly there is some worrying evidence that his drive to the centre in order to convert the followers of Polly Toynbee upset hardcore Tories. Now that Ms Toynbee has spurned the Tories' billing and cooing as soon as the IDS document was published, I doubt that there will be any instruction to remove the yellowing photos of Churchill and Thatcher from the Conservative Club in Cleethorpes in favour of the arguably more attractive photo that adorns Ms Toynbee's byline.

Mr Cameron will brush aside the latest poll figures on the basis that his party is still ahead and that the general trend of the past 12 months supports the view that the Tories are back in contention as a serious alternative to Labour. But something funny is going on. In 1992 it was claimed to be "The Sun wot won it" for the Tories. Next time it may well be the "others wot lose it" for the Tories. The most revealing, but little-noticed statistic in the latest poll is the rise to 14 per cent in the support for "other" parties. "Other" does not include the Lib Dems, currently on 19 per cent.

This growth in voter support for "others" has taken place by stealth, almost unnoticed by Westminster politicians, but it has the potential to prevent Mr Cameron entering Downing Street. In the 1992 general election, the three main parties accounted for a total of 94 per cent of all votes cast in the United Kingdom. By the 2005 general election this figure had fallen to 89 per cent. "Other" (which excludes the Welsh and Scottish Nationalists) support increased from 3 per cent to 8 per cent.

This trend looks set to continue with the latest poll suggesting that the Greens and Ukip are the principal beneficiaries. It is the latter that have the greatest impact on Tory fortunes with the ConservativeHome.com website recently suggesting that one third of Tory members are dissatisfied with Mr Cameron's leadership. On its own, this figure need not worry Mr Cameron - after all, a similar number of members voted against him, for his rival David Davis, in the leadership ballot last year. But 43 per cent of Tory members said that Ukip was the political party closest to their views. Tory members are, of course, not necessarily the same thing as Tory voters.

While the Referendum Party added to the Tory woes in 1997, it did not stand between the Labour victory and the Tory defeat. In 2005, however, Ukip could already claim to have helped Mr Blair to a Commons majority of 67 instead of what might have been half that figure. In 17 constituencies where the Labour majority over the Tories was less than 1,000, the Ukip vote was greater. Imagine how long Mr Blair might have survived if the overall Labour majority in the Commons had been 33.

In the past few weeks, unease among the grass-roots has resulted in some defections to Ukip, including the man who ran William Hague's local party when he was leader. Another defector, Mark Hudson, who was the Sevenoaks constituency chairman and on the Tory candidates' list, quit for Ukip because he claimed that Mr Cameron is leading the Tories away from their core values. He told the World at One yesterday that "it's not that I have left the Conservative Party, but that the Conservative Party has left me."

Mr Hudson is concerned about the "hug the hoodie" line one week contrasting sharply with a recent newspaper article by the Tory leader calling for tough action on crime. Bewailing the lack of difference between "New Labour and Blue Labour", he suggested that fellow travellers had the option of staying at home or voting for fringe parties such as Ukip.

So far the mumblings have been confined to party activists concerned as much about party autonomy and the selection of candidates from Mr Cameron's "A list" as about policy direction. Troubled MPs have, by contrast, been muted. The Tory leader is certainly delivering on his promise to select more women candidates, with the latest figures showing that over one third of candidates for Tory-held and winnable seats are women. By his own standard, set at the beginning of his leadership, this is a success story, although he acknowledged that a similar drive for a more diverse ethnic mix of candidates has proved more difficult.

Balancing the desire of local parties to maintain their autonomy with the leader's pledge to change the face of the party has caused tensions. Last week, the 16-strong executive committee of the winnable Tynemouth constituency selected a woman of outstanding quality, Wendy Horton, from a short list drawn up by the wider party membership. But had the members had the final say, Michael Macintyre, a local councillor, would probably have been chosen. The problem of motivating a volunteer army to work for a candidate not of their choice suggests that some may take their ball home. Last year, no Ukip candidate stood in Tynemouth. But it is almost certain that Ukip will select a candidate at the next election, when just a few hundred votes could deny Ms Horton victory.

Next year, Angela Merkel's presidency of the EU and her desire to return to the European Constitution will give Ukip the oxygen of publicity, forcing the Tories to confront matters continental. Sensibly, Mr Cameron has steered clear of "banging on" about Europe, but he needs to watch the right side of his back as growing evidence emerges that while Ukip may have no prospect of winning even one Commons seat, they could certainly withhold the keys of Downing Street from him. Mr Cameron may have no option but to continue cracking his modernising whip. But volunteers can vote as they please.

mrbrown@talktalk.net