Of all the tricky topics for Ed Miliband, immigration comes second only to the economy.

The Labour leader's speech on the subject yesterday was therefore a key moment, an opportunity both to ditch a troublesome legacy of the party's past and to put his own stamp on its future. As a piece of political positioning, it was passable enough. As a statement of policy, it could hardly have been more wrong.

First, the politics. Mr Miliband's starting point was the admission that the Labour government grossly underestimated how many people would look for work in Britain when 10 new countries joined the EU in 2004. The decision to waive transitional immigration rules was, accordingly, a mistake – one that would not be repeated. Mr Miliband hopes such statements will draw a line under the charge that, enthralled by globalisation, Labour abandoned swathes of its traditional supporters. He aims to prove he is listening to real people's concerns, and to close the gap epitomised by the spectacle of Gordon Brown caught describing a voter as "bigoted" for asking about immigration in the run-up to the general election.

It is a difficult manoeuvre to pull off. The danger is that the Labour leader leaves himself open to charges of xenophobia; indeed, his pronouncements met with immediate dismay from many more left-wing supporters. But with support trickling away in areas with fast-growing immigrant populations, Mr Miliband judges that the gains outweigh the risks. And his suggestion that Labour's failure was, at least in part, due to its overlooking the class element – the fact that low-wage workers were those most affected by immigration – is very much a "dog whistle" to the party base, for all his claims to the contrary.

But yesterday's speech was also part of Mr Miliband's efforts to set a coherent new direction, hence the link to so-called "responsible capitalism". It is here that the substance of the arguments come in. And here that the problems really start.

Superficially at least, the Labour leader paints a compelling picture. The wave of immigration mistakenly unleashed in 2004 has, he says, driven down wages and put pressure on scarce resources such as housing; it has strained social ties and exacerbated the private sector's reckless short-termism. But while it is certainly true that British workers have suffered in terms of jobs not won, it is far from certain that wages have been forced down.

That there are migrant workers sleeping 10 to a room and working painfully long hours for illegally low wages is, of course, an undesirable development. Minimum-wage legislation should indeed be more rigorously enforced. But such practices are only a fraction of the whole. And what Mr Miliband failed to mention is that the vast majority of workers from, say, Poland are hired not because they are cheap, but because they are better qualified, more motivated and more reliable.

Labour's response to the problem is, therefore, entirely back to front. If British workers are struggling to compete for jobs, the answer is not, as Mr Miliband is suggesting, to use the law to tilt the balance artificially back in their favour. Rather, it is to ensure they are better equipped. That means paying more attention to education, training and working culture. It means making our workforce better, not trying to lock out the competition.

For generations, the UK has thrived on, and benefited from, the efforts of immigrants. Now is no different. Mr Miliband's proposals may fit with his rhetoric about callous capitalism, they may work well in differentiating him from the Coalition, and they may appeal to the fears of a section of the electorate. But they are not in the best, long-term interests of Britain.