Patterson, in London to train Donovan 'Razor' Ruddock for his contest against Lennox Lewis at Earls Court on Saturday, goes back to a boyhood spent in Brooklyn, New York, when the only prospect held out to him was a life of vagrancy and thieving, and probably incarceration in a state penitentiary.
He speaks of taking up residence on a narrow ledge in a subway tunnel, being small enough at the time to clamber through the fanlights of the grocery stores he frequently set out to steal from, and seldom showing up at school.
It is when Patterson gets around to his great sporting achievements, a gold medal at 165lb when only 17 in the 1952 Helsinki Olympics, winning the professional heavyweight world championship and then becoming the first to regain it, that you discover him to be a man of quite remarkable humility.
When Patterson stood on the Olympic rostrum, and when he gained the greatest prize in boxing four years afterwards by knocking out Archie Moore in five rounds, it did not come to him that he was superior to other men. 'It was enough to think myself equal,' he tells Morgan.
Patterson's appealing philosophy, that which he continues to propound in comfortable middle- age without a trace of subservience, is especially relevant in a week that has seen storm clouds gathering again over sport in South Africa.
Even people who never miss an opportunity to declare themselves ignorant of athletic pursuits are probably aware of the threat that was entered against the South African rugby tour of England when critical support for it was withdrawn by the African National Congress in concert with a statement issued by the politically powerful National Olympic Sports Congress.
Reprieved yesterday when the ANC softened its attitude, calling upon anti-apartheid groups in this country to abort plans for disruptive demonstrations, the tour, which includes a match against England at Twickenham on 14 November, will now go ahead.
Let us admit that the problem of creating a more democratic society in South Africa is immense and maddeningly complex, and hard to simplify. However, a view long held here, and one that appears to have made me persona non grata in the republic and provoked some quite bitter responses, is that South African rugby deserves all the calumny it gets.
Leaving aside the conveniently fatuous notion that sport and politics should be kept apart, there is still plenty of evidence to suggest that in the numb Afrikaner mind, rugby remains a blazing symbol of white supremacy.
If there was a hint of mischievous premeditation in the timing of the NOSC's original statement this week, when objections were lodged on the basis that the South African Rugby Union has reneged on an agreement to develop the game in the townships, its big mistake was prevarication.
When, in defiance, the white anthem, Die Stem, was sung with chilling fervour before the Test against New Zealand in Pretoria two months ago that announced the return of South African rugby from isolation, the ANC should have immediately put a stop to the following week's game against Australia.
As far as thought can reach, and because of the hypocrisy that prevails in many quarters of rugby, it remains a tough ideological game for the ANC to win.
Recently, in conversation with a retired internationalist of considerable repute, indeed a liberally minded man, it was put to me that there are faults on both sides, and more attention should be paid to the fact that rugby does not appeal to the vast majority of blacks in South Africa.
If there is a reason for this beyond the conclusion that the game has been maintained as a white preserve, it is beyond me.
Presumably, the tour goes ahead on the basis that it would be impractical and counter-productive to call it off at such a late stage. But we have not heard the last of this, and nor should we unless South Africa is fully prepared to implement its proclaimed commitment to a more democratic policy.
We are talking here about equality, the holy grail Floyd Patterson quested for and found in the professional ring.Reuse content