Fitch has downgraded the UK's credit rating to AA from AA+ with a negative outlook, hinting that further downgrades might follow.
It cited reduced immigration as one of the reasons for the UK's weaker economy.
"Medium-term growth will also likely be weaker due to less favourable terms for exports to the EU, lower immigration and a reduction in foreign direct investment. An adjustment in the value of sterling and changes in the business environment could also affect growth," Fitch said.
Research has long shown that the UK will be worse off without its immigrant workers.
The National Institute of Economic and Social Research has said that reducing immigration by two-thirds will see the UK economy shrink 9 per cent by 2065.
The UK Government said in the run up to the vote that immigrants from the European Economic Area, including people from Liechtenstein, Iceland, Norway and Switzerland, contribute more than £3bn to the UK economy in taxes and income, a figure that dwarfs the £500m they take in benefits.
Leave campaigners, carried to victory on a mandate to reduce immigration, have so far yet to spell out how they many migrants they plan to let in after Brexit.
But since the vote, immigrants have been the target of a growing number of hate crimes in the UK. Polish families have reported having signs telling them to leave the EU posted through their letter boxes in Huntingdon in Cambridgeshire and verbal abuse directed at immigrants on streets.
A YouGov survey in March cited "immigrants willing to work for low wages" as one of the top three economic problems for leave voters, alongside EU rules and regulations and the last Labour government.
Peter Kellner, president of YouGov, said that the YouGov research down the years showed that immigrant workers were often blamed for the UK's economic problems.
"Most voters who strongly oppose the EU are also want to pull up the drawbridge between Britain and the rest of the world – not just by stopping the flow of immigrants but also, for example, ending overseas aid," Kellner said.
"Some leading advocates of Brexit, such as David Davis and Douglas Carswell, preach an alternative internationalism to EU membership, rather than a retreat into isolationism. But few voters want to leave the EU in order to pursue a more open and generous relationship with the rest of the world," he added.
The Fitch downgrade is the first sign that "pulling up the drawbridge" is going to hurt the UK.
Downgrades are important because they mean that the UK may find it more expensive to borrow money in international financial markets, because lenders can be less sure of the UK's ability to pay the money back.
S&P, another of thee three top credit rating agencies, has also downgraded the UK from AAA, saying the decision to leave the EU could lead to "a deterioration of the UK's economic performance, including its large financial services sector".
Moody's, the third of the big three, downgraded the UK's credit outlook from stable to negative the day after the UK voted to leave.
Join our new commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies