He said many black people were fed up with racism and would welcome the chance of 'conditional return'.
This should be backed with compensation and job training for them and investment in the infrastructure of the Third World to 'reverse the brain drain' to prosperous northern countries.
'I'm talking about a conditional return, a return of people with a number of conditions attached,' he told a news conference at Westminster. 'I do not believe the option I'm talking about - conditional return - is necessarily an option for most black people. But some black people will want to take up that particular option.
'If they wish to do so, I think it should be possible.'
He emphasised: 'The question of whether people can return home or not has been a taboo subject for years. But people are talking about it.
'Some people have already gone home and others are making plans to go back home. Other people have come to me to say that they would like to go back to their countries of origin if the conditions were right and if they were able to do so and live in reasonable peace and harmony for the rest of their lives.'
But Mr Grant, a campaigner for black people's rights, had a cold reception from fellow black Labour MPs and from anti-racism groups. There is a fear that any suggestion of 'repatriation' - a word Mr Grant did not use - would play into the hands of far-right groups such as the British National Party, which recently won its first council seat in a by-election in Millwall.
Paul Boateng, MP for Brent South and Labour's front bench spokesman on legal affairs, said: 'Repatriation, whether voluntary or otherwise, has no place in the promotion of good race relations. Such a policy would be totally misconceived. Those who advocate it are profoundly mistaken.'
Diane Abbott, Labour MP for Hackney North and Stoke Newington, said: 'I think it's unfortunate because it opens the door to the whole debate on forcible repatriation.
'What we want is a better life for black people living here.'
The Anti-Racist Alliance issued a statement saying it believed Mr Grant's views were 'dangerous, ill-advised and pandering to the worst manifestations of racism'.