More than 5,000 foreign criminals who should have been deported remain in the UK, including almost 4,000 who are free on the streets, figures showed today.
The UK Border Agency (UKBA) must reduce the number of its decisions which are overturned, mainly because of human rights claims, and end the cycle of appeals which bump up the cost to the taxpayer but still fail to see the offender deported, inspectors said.
But the warning comes after Home Secretary Theresa May said she wants to change the rules which prevent foreign inmates from being deported once they have served their prison sentence because it would breach their human rights.
The real problem lay with the way the British courts interpreted the law, she said, adding that the right to a family life was not "absolute" and could not be allowed to "drive a coach and horses through our immigration system".
John Vine, the independent chief inspector of the UKBA, said the number of foreign criminals who are not deported or cannot be deported at the end of their sentence, was increasing.
"More must be done to actively manage these cases - they represent a growing cost to the taxpayer and cannot be ignored," he said.
Inspectors found a total of 3,775 former foreign national prisoners who should have been deported had been released from custody and were living in the community.
More than 1,600 others remained in detention, having completed their prison sentence.
A further 12 are missing after either being released directly from court or referred incorrectly, he said.
The report found "significant disparity" between the UKBA's and the courts' interpretation of whether a foreign prisoner should be able to remain in the UK on human rights grounds.
A total of 425 cases had been overturned, "the overwhelming majority on human rights grounds", compared with just 151 cases in which the offenders had been granted permission to stay in the first instance.
In all, a third of the appeals against deportation lodged by foreign prisoners in the 12 months to February were successful, the figures showed.
Foreign prisoners were also being held for longer, up to 190 days in January from 143 days in February 2010, and one in four who were held after the end of their jail term had been detained for longer than a year.
Mr Vine said there was "genuine fear and reluctance to release, given the potential implications of a foreign national prisoner committing a further offence, but no evidence that a detailed assessment of the risk of reoffending had taken place in each case".
While 1,102 foreign offenders were released on bail by the courts between February 2010 and January, only 109 were released from detention.
The figures also showed that between 2007 and 2010, 20,360 foreign national prisoners were deported from the UK, 5,235 last year alone.
Half of those deported last year, more than 2,500, left under the "cost-effective" facilitated returns scheme, the report said.
Mr Vine said: "The agency can still improve the way it handles foreign national prisoners.
"A significant number of appeals continue to be allowed against decisions to deport, in most cases because deportation would breach the UK's obligation to the individual under the Human Rights Act.
"The agency must work to reduce the number of decisions overturned on appeal and take full account of the courts' decisions in deciding whether deportation action is appropriate or whether it would breach a person's rights under Article 8.
"If not, there will continue to be a cycle of appeals, at considerable cost to the taxpayer even though the outcome will eventually be that the foreign national prisoner is entitled to remain in the UK."
Immigration Minister Damian Green said: "This Government is tackling the problems of the past - we are starting the deportation process earlier and removing foreign criminals quicker than ever.
"Where a foreign criminal poses a risk to the public, we believe they should stay in detention.
"Foreign criminals in the community awaiting deportation will be subject to stringent reporting restrictions while every effort is made to remove them from the country."
He added: "For too long Article 8 has been used to place the family rights of foreign criminals above the rights of the British public, which is why we will change the immigration rules to ensure a better balance."