The comments, by Paddy Hill, have brought a storm of Unionist protest in Northern Ireland, with the Rev Ian Paisley denouncing him as having 'murder in his heart'.
The organisers of the Bloody Sunday commemoration rally in Londonderry, where Mr Hill was speaking, have dissociated themselves from his speech.
The RUC is trying to obtain a transcript of his remarks, following which a dossier may be sent to the Director of Public Prosecutions.
Mr Hill was released in March 1991, after spending 16 years in prison, when the convictions of the Birmingham Six were quashed.
Mr Hill declared: 'If the British come here to shoot our men and women, the only way is for Irish people to shoot them back. If it takes another 800 years we'll get them out of the country. We'll come out on top because this is our country, not theirs.
'We don't want to leave future generations the legacy of British rule. There is only one way to send them back - that's in boxes - because that's what they do to us.'
Mr Hill also criticised the Irish government, saying: 'They are ashamed of being Irish and they are filthy - Brits in disguise.'
The Secretary of State for Northern Ireland, Sir Patrick Mayhew, has given an assurance that the jailed army intelligence agent Brian Nelson will serve his full sentence.
Nelson, who operated as an army agent inside a loyalist paramilitary organisation, was given a 10-year sentence for a series of terrorist offences.
His transfer to an English prison last year brought claims that his early release was planned. Sir Patrick, however, has told Labour's spokesman on Northern Ireland, Kevin McNamara, that he will not be treated differently from other prisoners.Reuse content