But "they" are the British, who have taken advantage of the European Union to head for warmer climes or better jobs in vast numbers over the past 25 years. More Britons left home to live elsewhere in Europe in the last two decades than any other EU nationality, according to figures published by the Council of Europe.
The figures show that if there is one nation with an interest in removing border controls and easing the freedom of movement in Europe, it is us. Though after the performance of British football hooligans in Dublin last week, it might be fair to ask whether other countries want to let us in.
There are, according to EU figures, about 350,000 Britons living in Europe outside Britain, half of whom went there in the 1970s and 1980s. They form the largest European expatriate colonies in Spain, Greece, Ireland and Portugal, the second largest in the Netherlands and the third largest in Italy. Even taking into account the size of populations, it seems that only Ireland saw a larger exodus to Europe in the 1970s and 1980s.
According to figures from the Council of Europe, the number of Britons in Spain increased by 600 per cent from 1971 to 1992, up to 90,000. They are almost certainly the largest single expatriate national group resident in Spain. There are twice as many Britons in Spain as Germans, so all that stuff about poolside loungers doesn't stand up to much scrutiny. We get there first because we live there.
The number of Britons in Portugal went up from 4,000 to 9,000 over the same period, comprising the largest national group after those from former Portuguese colonies. Greece has 21,000 British residents, up from 3,000 in 1971.
These movements partly reflect people retiring to the sun. But there is also increasing evidence of Britons seeking job opportunities in Europe's wealthier nations. There are about 100,000 British subjects in Germany, for instance, and that does not just reflect the continuing military presence. More Britons arrived there in the 1970s and 1980s than any other nationality apart from the Poles, Turks and Yugoslavs. There are 50,000 in France, an increase of 25,000 since we entered Europe.
The EU member states with the largest expatriate populations elsewhere in Europe are Spain, Greece, Italy and Portugal. But as those countries have become wealthier in the past 20 years, many of their citizens have returned home.
The argument that the rest of the world is desperate to get into Britain does not seem to hold water, as far as the EU is concerned at least. There are about 38,000 French in Britain and 42,000 Germans, for instance, so we evidently do not have quite the same attraction for them as they do for us. But then we are the sixth poorest of the 15 EU countries, with Spain catching up fast. Welfare benefits are far from being the most generous in Europe.
The scare story retailed by Mr Wardle was that there are vast numbers of immigrants in the rest of Europe desperate to come to Britain and claim welfare benefits: the figure of 15 million was quoted. In fact, there are about that number of people in Europe living outside their country of origin, but 5.5 million of them are from the other EU countries, who can go wherever they want, even with passport controls.
Of the remaining 9.5 million, 1.2 million are already legally resident in Britain. That leaves about 8.3 million people resident in other countries. Many of those in Germany are ethnic Germans, who seem unlikely to leave for a non-German speaking country. And the non-EU populations of most other states are largely made up of those from their former colonies.
Turning immigration into Britain into a scare story is easy, but fundamentally belied by the figures. By European standards, Britain has a relatively low foreign population. Though we have the second largest in absolute terms (about 2 million,of whom one third are Irish), in percentage terms we lag behind Luxembourg, Switzerland, Austria, Belgium, the Netherlands, Sweden, Denmark, France, Germany and Norway. We are way behind Germany, which has about 6 million foreign residents.
Nor is there any evidence of a huge influx of asylum seekers to Britain, despite the Home Office's decision to clamp down last week. Most of those who come legally to Britain - about 85 per cent - are joining their families. We saw a decline in asylum applications in 1993, for instance, and never experienced the huge numbers that scared Germany during the early 1990s.
There was a huge problem of illegal migration into Europe through clandestine means at the beginning of the decade. The evidence seems to be that this is declining. Last week, for instance, Romania - one of the main sources of legal and illegal immigration into the EU in the early 1990s - said the number of people trying to leave had fallen from 144,000 in 1990 to 22,500 last year.
The widely rumoured exodus of Russians into Europe - figures as high as 2 million were mooted - has not happened. The Bosnian crisis led to a huge refugee outflow, but that has also peaked.
By portraying EU policy as aimed at removing defences against immigration, Mr Wardle was also being deeply misleading. The problems which Germany experienced earlier in the 1990s have led to a Europe-wide clampdown on asylum and migration. The main focus of EU efforts in the past five years has been to put up the barriers, not to lower the drawbridge.