Steve Bunce on Boxing: Ali was brutal in the ring – and could be horribly cruel outside it

Muhammad Ali batters Ernie Terrell in Houston in 1967 (Getty Images)

Muhammad Ali is 70 today and his life has reached a stage where golden moments are all that really matter. However, it is still important to remember the other bits. It was not all glory for Ali because the death threats were real, the CIA had numerous files, many of his fights struggled to sell and the KKK made him an honorary member.

The tales of hardship have been told and retold, often made more dramatic as the cruel years have denied the man the right to correct inaccuracies. There still remain parts of the myth and the magic that are not excusable with a comic wink. I also think that he would stop laughing at the memory of being embraced by the KKK after making a series of racist comments. Sure, it was fun in the re-telling but it never made his great rival Joe Frazier laugh; Frazier was dirt-poor and really Southern and grew up hearing first-hand accounts of lynchings.

"No intelligent black man and black woman in his or her right mind wants white boys and white girls coming to their homes to marry their black sons and daughters," Ali said, much to the delight of the Klan elders.

I often wonder how we would deal with Bobby Moore if he had offered something similar in support of Enoch Powell's "Rivers of Blood" speech, just a year or so after winning the World Cup as England's captain.

The covert monitoring by both the CIA and the FBI increased after his 1966 assertion that he had: "No quarrel with the Vietcong." The fallout was spectacular, with planned fights being scrapped as cinemas withdrew their facilities for live screenings, and there was a mass call for a boycott of his future fights. This all happened about 18 months before he lost his right to fight and led to a series of four quick fights on the road to escape the hate and intensity of his critics. Sure, he loved London but he hated not fighting and not earning even more!

In March 1966 he met the iron-jawed local George Chuvalo, who had accepted the fight at short notice when Ernie Terrell was forced to withdraw, in Canada at the start of six hectic months. Terrell had agreed a deal where his purse increased with each and every ticket sold in cinemas; after Ali's Vietcong comment the fight was simply not viable because cinema owners withdrew their support.

"It was a difficult time, a hard time to sell Muhammad to the American people," said Bob Arum, who was part of a company called Main Bout Inc, which was created in early 1966 to market ancillary rights to Ali's fights. When Ali stepped in the ring against Chuvalo it was only his third fight since winning the title two years earlier and the reality is that it meant very little; he was hated by most of America.

So much is written and remembered from the Sixties about the two Sonny Liston fights, then Ali's open conversion to the Nation of Islam and then his refusal to be inducted. His name change, a gradual process from Cassius Clay via Cassius X to Muhammad Ali, was ignored on both sides of the Atlantic by most of the media until – and even after – the first Joe Frazier fight, which is astounding and embarrassing.

The sweet irony is that the neglected Ali from the Sixties was a much better fighter than the far more famous man from the Fight of the Century (1971), the Rumble in the Jungle (1974) and the Thrilla in Manila (1975). Floyd Patterson and Chuvalo fought both versions and their testimony is reinforced by Ali's own assertions. However, he was nasty in the ring.

"Take the Terrell Ali and the Foreman Ali and the younger one would win," Ali said.

The Terrell fight did eventually happen, his penultimate fight before being banished, at the Astrodome in Houston. It was not a pretty sight. Ali brutalised Terrell, taunted him and inflicted illegal and potentially lasting damage to Terrell's eyes by rubbing his face on the ropes. The veterans' veteran of Fleet Street, George Whiting of the Evening Standard, a rare British presence at ringside, wrote with disgust about Ali's antics, which included spitting at the feet of his bruised, bloody and beaten opponent. I interviewed Terrell 30 years later and he was still not smiling.

There was one more massacre, when Ali thankfully seemed to take pity on a weary Zora Folley, before he lost his licence and many of his friends. "We missed his peak, we were robbed of his best years," insists Angelo Dundee, his trainer and passenger. It was during the exile, which lasted from June 1967 until October 1970, that Ali was helped in a variety of ways by Frazier. It was a friendship that never lasted and I remember Frazier telling me about the moment Ali switched and became his tormentor, a savage role that continued for too long and was laughed at by too many.

They were in Frazier's gym in Philadelphia, no great entourages, just a few people including a photographer and they were required to get in the ring for a publicity shot. "We got in that ring and the insults started," Frazier said.

"I'd never heard him talk that way before and told him to 'quit'. That was it." Ali was at Frazier's funeral two months ago, the loss clear on a face that stopped showing emotion a long, long time ago. I would like to think that during the service Ali ran through decades of cruelty to Frazier in his mind; it is about time that one or two other parts of Ali's life were viewed without the benefit of invention and rose-tinted glasses.

The Ali who defeated Terrell was unbeatable, the greatest fighter ever, but he was also ruthless and unpleasant – I have no problem with that, but please don't tell me that all the other stuff was done and said with a twinkle in his eye.