Normunds Belskis, from Latvia's interior ministry, said his government had proof the would-be refugees reachedLatvia from Russia and Belarus, and said they should be returned there. He denied UNHCR's claims that the asylum-seekers were living in inhumane conditions, said they had received plenty of food, water and medical attention and insisted they were "far better off than they would be if we put them in a military barracks".
The 104 would-be refugees, mostly Iraqis, have spent the past week in a railway siding near Karsava, close to Latvia's border with Russia. In the preceding week they were shunted several times between Latvia, Russia and Lithuania, none of which would accept them.
The UNCHR has condemned all parties in the dispute, accusing them of treating those on board the train "like cattle".
Two UN representatives who visited the group said the situation was "grave". With no proper heating and not enough blankets, many were suffering from cold. Insufficient food and unclean water, supplied by a fire brigade, had led to widespread diarrhoea, they reported.
In addition to 85 Iraqis, the group includes 13 Afghans, four Palestinians and two Iranians. More than half are children. According to the UNHCR, most are genuine asylum-seekers.
Neither Latvia nor Lithuania, both of which regained their independence from the Soviet Union in 1991, have legislation that covers asylum procedures. They claim they are too poor to cope with refugees.
"We do not have reception centres to house such people, nor the money to build them," Mr Belskis said.
The Baltic states fear that if they allow any refugees to stay, they will be swamped with thousands more, using the Baltic states as stepping stones to the West. The UNHCR has offered technical help to Latvia, to provide the asylum-seekers with better interim accommodation.