Many white working-class communities believe their views on immigration are being ignored, while those coming into the country are given preferential access to housing and benefits.
The research by the Department for Communities and Local Government found that white working-class communities felt they had been "betrayed" and abandoned by the establishment, which no longer had their concerns at heart.
The Communities Secretary, Hazel Blears, said that politicians had to start engaging with the disenchantment among poorer, white sections of society in order to combat growing "myths" over the treatment of immigrants.
Her department's report suggested, she said, that the resentment, unfairness and disempowerment perceived by the group together with the absence of an "open and honest discussion" about immigration had created fertile ground for the far-right to exploit.
Its conclusions, based on interviews with people living on council estates in Birmingham, Milton Keynes, Thetford, Runcorn and Widnes, found that while there was a high amount of disquiet over benefits given to immigrants, few of those asked had experienced regular contact with people from ethnic minorities.
It said that hostility towards them was worst in deprived, working class areas because "people who have the least are more likely to be afraid of things being taken away from them".
Ms Blears warned that branding the views "racist" risked alienating the communities affected even further, and called on politicians to respond to their "real and perceived sense of unfairness".
"White working class people living on estates sometimes just don't feel anyone is listening or speaking up for them. Whilst they might not be experiencing the direct impact of migration, their fear of it is acute," she said.
"Changes in communities can generate unease and uncertainty. These changes need to be explained and questions need to be answered or the myths that currently surround the treatment of ethnic minorities 'jumping the queue' will become increasingly hard to shift."
Conservative community cohesion and social action spokeswoman, Baroness Warsi, said the report proved that New Labour had "completely lost touch with their so-called roots".
"The danger for the rest of us is that this has now created a ticking time bomb of racial and class prejudice.
"Amongst other things this has also demonstrated the dangers of Labour's past use of identity politics for electoral purposes."