Asked by the representative of the Smethwick Telephone and Warley Courier what should be done about it, he replied: 'I would not wait for the fascist elements in Smethwick to erect gas ovens.'
Nine days later, on Sunday, 21 February, 1965, the charismatic spokesman for Black Power was hit by 16 bullets as he rose to address a meeting at the Audubon Ballroom on New York's West 166th Street.
His visit to Smethwick does not feature in Spike Lee's film, Malcolm X, which opens in London on Friday. Birmingham, Alabama, had a much more powerful influence on the main character than Birmingham, West Midlands.
Smethwick lies just to the north-west of Birmingham. Its residents were always fiercely independent of their much bigger neighbour, only to be swallowed up by the newly created Metropolitan Borough of Sandwell 18 years ago. Ten years before that, in the mid-Sixties, theirs was a small town on the edge of the Black Country, which derives its name from its sooty industrial heritage rather than anything else.
In parts of London, Birmingham, Manchester and other big cities there was considerable antagonism by the host community towards those termed 'coloured immigrants' from the Caribbean and the Indian sub-continent. But it was little Smethwick that had leapt to national notoriety because of the general election of 1964.
The sitting Labour MP was Patrick Gordon Walker (born Worthing, educated Wellington and Christ Church and Commonwealth Secretary in the Attlee government). He lost to Peter Griffiths, a Tory populist (born Smethwick, educated at a local grammar school). The swing from Labour to the Conservatives was 7.2 per cent, compared with a national swing towards Labour of 3.5 per cent.
No one was in any doubt about the issue that had so confounded the national trend. This was the constituency where posters appeared carrying the message: 'If you want a nigger for a neighbour, vote Labour.' Mr Griffiths and his supporters have always claimed thay they were the work of far-right extremists who attached themselves to his campaign.
But during the municipal elections two years earlier, Mr Gordon Walker used the columns of the Smethwick Telephone to accuse the Conservatives of organising gangs of children to chant the same loathsome slogan.
Mr Griffiths wrote in reply: 'We can't stop children reflecting the views of their parents. The people of Smethwick certainly don't want integration.'
Today Mr Griffiths is Conservative MP for Portsmouth North. He has denounced racism on more than one occasion. In fact, the disclaimers began soon after his arrival in the House of Commons. Harold Wilson, the Prime Minister, described the new member for Smethwick as a 'parliamentary leper', to outraged cries from the benches opposite.
Malcolm X's arrival in Smethwick the following year caused less of an uproar. Most of the townsfolk were blissfully unaware of his existence in their midst until they read about it in the following Friday's Telephone.
He had flown first from New York to Paris, where he was scheduled to speak to a Congress of African students. But he was refused entry to France as 'an undesirable person'. Britain allowed him in to address the London School of Economics and the Birmingham University Students' Union.
For the BBC's Tonight programme it was an ideal opportunity to put the X factor into a feature on a new magazine called Magnet, which was targeted at the black population.
For Avtar Singh Joahl, now general secretary of the Indian Workers' Association (GB), it was a chance to meet one of his political heroes. 'We felt we were linking up the struggle of black people in this country with those in the States,' he recalls.
'Malcolm came out very clearly on questions of power and exploitation. He would be considered an extremist to those who were practising racism, but to black and Asian people his visit was an act of solidarity.'
Significantly, Malcolm X posed for cameramen by a For Sale notice on one of the terraced houses in Marshall Street. A group of white residents, led by one Alice Groves, had successfully petitioned the Tory council to purchase houses that came on the market in their neighbourhood compulsorily and let them to white families only.
This master plan for racial mingling did not run to buying houses in predominantly white areas and letting them exclusively to black and Asian families. Eventually it was vetoed by the Ministry of Housing.
There were other areas of blatant discrimination on racial grounds. When Mr Joahl took Malcolm X for a pint of bitter, they went to one of the pubs that did not operate a colour bar. 'It was full of Indians,' he recalled, 'and they all wanted to shake his hand. They were happy he was there.'
In those days, Asians and Afro-Caribbeans were refused drinks at pubs and clubs elsewhere in the country. But Smethwick had found itself once again catapulted into the national headlines when a colour bar was found to be operating at the Labour Club.
In his 1965 book Immigration and Race in British Politics, Paul Foot pinpoints 'anti-immigrant sentiments' in the local Labour Party as a key issue in Smethwick. It undermined, he wrote, any determined and principled response to the right-wing extremists who had infiltrated the Conservative Association and been given plenty of space to spread their propaganda through the local newspaper.
What had made the town such a magnet for immigrant workers was the concentration of foundries offering work. 'They accepted jobs the host community wouldn't do,' says Ron Davis, former Sandwell Council leader who campaigned for Mr Gordon Walker in 1964. 'The average life expectancy of a man in the moulding plant was 34.'
Mr Joahl worked as a moulder's mate for less than half his pay. 'The moulders were all white. The mates were nearly all Asian. We did most of the work, but they were getting pounds 16 and we were coming out with pounds 7 10s ( pounds 7.50).'
As a staunch trade unionist he campaigned not just about wages but also discrimination in the workplace. A company called Midland Motor Cylinders had one lavatory for Asian and black workers and another for Europeans.
Many of the Asians who started work in those long-deceased moulding plants now run shops and businesses of their own. Sikh licensees run some of the pubs that once operated colour bars.
The Red Cow on Smethwick High Street is a good example. On the lounge bar wall is a picture of the pub's multi-racial football team, a blend of Asian, white and Afro-Caribbean. In the street outside, a black policeman was supervising the reloading of a builder's van that was causing an obstruction.
Round the corner from Marshall Street, Jim Neenan, a retired grocer, told me: 'I'm one of the few white people in my street, but it doesn't bother me. My Indian neighbour came round to see me when I came out of hospital. He even bought me a Christmas present.
'Not like the white family that lived there before. They went off to Weston after 25 years and never said a word.'
Past experience forced Sandwell Council to take staunch measures to combat racism. An equal opportunities recruitment policy is edging towards the 14 per cent that will ensure that staffing in the Council House reflects the ethnic balance of the borough.
A police community relations committee was working before such bodies were recommended by the Scarman report of the early Eighties. One outcome has been the provision of a phone line specifically for racial attacks.
They still go on, and by no means all are reported. Smethwick, where the ethnic population is around 40 per cent, has the social problems that you might expect in an area of considerable deprivation. But the apocalyptic warnings of the Sixties have not come to pass. The fears and the suspicions have subsided.
Malcolm X did not live long enough to know that the biggest concentration of ovens in Smethwick is in the gas showrooms on Bearwood Road.