There are 23 fewer jobs for British workers for every 100 migrants from outside the EU, the Government's immigration advisers said today.
An increase of 100 foreign-born working-age migrants in the UK was linked to a reduction of 23 Britons in employment between 1995 and 2010, the Migration Advisory Committee (Mac) said.
It comes after a report by the National Institute of Economic and Social Research (Niesr) said the number of immigrants coming to the UK had little or no impact on the number of unemployed.
Average wages remain the same, the Mac said, but it added: "Migrants are found to increase wages at the top of the UK wage distribution and to lower wages at the bottom of the distribution."
The impact and displacement of British workers also does not last forever, the Mac report found.
"Those migrants who have been in the UK for over five years are not associated with displacement of British-born workers," it said.
Between 1995 and 2010, the total working-age migrant employment rose by 2.1 million and currently displaces 160,000 British-born workers, it said.
But the report added that EU migration had "little or no impact on the native employment rate".
Professor David Metcalf, chairman of the Mac, said: "Assessing the impacts of migration is not a simple decision and our conclusions will require careful consideration by the Government.
"However, our research suggests that non-EEA migration is associated with some displacement of British workers.
"Financial impacts of migration are also complicated but considering overall GDP does not present a true picture.
"Instead, the impact of migration on the economic well-being of the resident population should be the focus."
He went on: "Impact assessments must also consider wider effects such as the effects of skills transfer from migrants and their impacts on public finances, employability of UK workers, housing and transport.
"Although difficult to measure, these will ensure we can better understand the effects of migration."
Asked if there would be 160,000 extra jobs for British workers if there had been no immigration from outside the EU, Prof Metcalf said: "Yes. That would be a reasonable way of putting it."
While he said it was difficult to identify the occupations which would be most affected, he highlighted jobs in information and communications technology, and in hospitality and retail - where a large number of foreign students are employed part-time - as sectors which could see the most impact.
Health and care services have also employed large numbers of migrants, he said, but this was mainly at a time of shortage of UK workers so British jobs were unlikely to have been displaced.
Asked why EU migrants had no impact on British jobs, Prof Metcalf said: "It may well be that the EU migrants are disproportionately less-skilled and it may be that the labour market can adjust. It may well be that that's tempered the impact."
The Mac was asked to look at the impact of immigration from outside the EU and how that information was used in official impact assessments of the Government's migration policies.
Prof Metcalf said the current system, which uses GDP to look at the impact on both UK residents and migrants, "can't be the right way of thinking about this".
It would be better to consider the impact on the economic well-being of the resident population alone, he said.
Any assessment of the economic and social impacts of immigration - and of specific immigration policies - critically depends on whose interests are taken into consideration, the Migration Observatory at the University of Oxford said.
Dr Scott Blinder, a senior researcher at the centre, said: "This report highlights the need to decide and articulate more clearly whose needs Government is prioritising when developing immigration policy.
"Trade-offs need to be confronted head on. Without more debate and clarity about whose interests policy is trying to maximise, we cannot hope to reach more agreement about the costs and benefits of specific policies."
Immigration Minister Damian Green said: "This Government is working to reduce net migration from the hundreds of thousands a year we saw under the last government, to the tens of thousands we saw in the 1990s.
"Controlled immigration can bring benefits to the UK, but uncontrolled immigration can put pressure on public services, on infrastructure and on community relations.
"This report makes clear that it can also put pressure on the local labour market.
"We thank the Mac for its work and will now consider the report more fully as we work to regain control over our immigration system."
Matt Cavanagh, associate director at the Institute for Public Policy Research, said the Mac report would be "seized on as 'gotcha' moment confirming the popular view that immigrants take jobs off British people".
But he warned that the link only relates to non-EU migrants and only in the short term.
"The Mac's overall finding, covering EU and non-EU migrants, and those who stay over five years, is that the 2.1 million additional foreign workers in 2010 compared with 1995 are 'associated with' 160,000 fewer resident workers: a ratio of one in 13 rather than one in four," he said.
"Britain clearly has a major problem with unemployment, including a problem with youth unemployment that predated the financial crisis.
"But its causes are too complex to be reduced to blaming immigration, just as the effects of immigration on the labour market are too complex to be reduced to endlessly repeated headlines about 'foreigners taking all the jobs'."
He went on: "Many people - including coalition ministers - are strongly wedded to an intuitive picture in which the two are locked together in a zero-sum game: migrants compete for jobs with residents, therefore more migrants must mean fewer jobs for those already here.
"Economists have been more sceptical."
Sir Andrew Green, chairman of the campaign group Migration Watch UK, said: "This is a thoroughly professional report.
"The committee have had the courage to say straight out that immigration can add to unemployment, especially during a recession.
"They are also right to draw attention to impacts that are harder to quantify - such as housing and congestion."