David Cameron is basking in the Portuguese sun, and he has earned his holiday. But he has some serious thinking to do.

When he became the unlikely leader of the Conservative Party in 2005 he set about modernising his election-losing party, seeking a move to the centre ground. Tony Blair had changed the game, having, for example, appointed openly gay ministers and brought in civil partnerships. Confronted with a Britain more at ease with itself, Cameron – against his own instincts – had to speak the language of Blair. He asked us all to believe this move came from the heart, but, having known Cameron for more than 20 years, I saw a side of him I had never seen.

When the Tories needed the Lib Dems to form a government, Cameron had an opportunity to cement the change, to show his party had changed for ever. But we've been disappointed.

Setting aside some highly questionable appointments, last week we saw the most sinister errors of judgement of his premiership in the form of ads in the poorest taste imaginable. What on earth must the Lib Dems be thinking of their coalition partners, given the revelation that the "Go home or face arrest" campaign was orchestrated in private? Neither the Deputy Prime Minister Nick Clegg, nor the Lib Dem Home Office minister Jeremy Browne, knew it was coming. Worse still, the Human Rights Commission has announced an investigation into police tactics, seemingly Home Office-sanctioned, for "racially profiling" and targeting "ethnic" citizens and illegally demanding to see their papers. So who did authorise this tactic?

It could have turned out differently. The Tory party's broadly liberal tradition is just as strong as the more voluble right. What could have been perceived as a formidable, philosophically coherent, modernising government is drowning in the vulgarity of opportunistic spin-doctors. I would love to have seen what a real coalition policy unit, led by David Laws, Michael Gove and David Willetts, and licensed to stride beyond the hallowed text of the Coalition Agreement, could have spawned across Whitehall. Instead it is being torn to shreds by the rigid thinking of small-minded backbenchers. As an aide to three Tory leaders, it saddens me to say this.

I have never called anyone racist. It is an often used and discredited cry. But this time, I believe the Conservative Party has shown it is racist. It will do anything, right or wrong, to bolster its poll ratings. Cameron's hard work around a modernity agenda ran into the sand. His party has never really embraced change, but he gave up the fight too soon. In short, he talks like a liberal but still allows his party to behave like "the same old Tories". Who would vote for that?

Last week's developments are no coincidence. The general election campaign, under the supervision of Lynton Crosby, has started two years early, and you can bet race and immigration will be a big part of Tory strategy. A decade ago, Theresa May, supposedly a moderniser, was brave enough to tell the Tories they were seen as "the nasty party". Yet now she presides over this foul talk. Et tu, Theresa?