Sajid Javid, the housing secretary, made the remark in the House of Commons chamber, where parliamentary privilege protects him against a lawsuit.
Momentum has threatened the cabinet minister with legal action if he repeats the comment outside Parliament.
Responding to a question about abuse of councillors, Mr Javid said: “I am very concerned about the intimidation of councillors, which is, of course, wrong at every level. Decent Opposition members will recognise the intimidation that there has been, especially in London, of Labour leaders.
“Just yesterday, there were reports of a meeting of the hard left neo-fascist Momentum group, which was trying to remove Wandsworth councillors.”
Laura Parker, the organisation’s national coordinator, told The Independent the comments were "offensive and absurd".
She said: “Fascism has been responsible for the oppression, imprisonment and murder of millions of people, and throwing around the term so carelessly is deeply disrespectful to those who have made great sacrifices fighting fascism both here and abroad.
“If Sajid Javid stands by his remarks, he should repeat them outside parliament and without parliamentary privilege. Otherwise, he should apologise to our 38,000 members."
A Momentum source told The Independent the group would be instructing lawyers if Mr Javid repeats the claim outside Parliament.
MPs also demanded that Mr Javid withdraw the statement and apologise.
Labour backbencher John Mann, a vocal critic of Mr Corbyn, said: "Understating is a big problem. So is making things up. Javed [sic] should quickly retract and apologise.
He added: "In most areas Momentum are a bunch of veteran, well-meaning people obsessed with minutes, matters arising, bureaucracy and incapable of talking to ordinary voters."
Shadow Cabinet Office minister Jon Trickett called Mr Javid's remarks "shocking and disgraceful", while Labour MP Chris Williamson said they were "a menace and a disgrace". Both called on the Housing Secretary to apologise.
Another Labour MP, Alex Sobel, said: "I heard this in the Chamber and I was appalled. Using parliamentary privilege to call Labour Party members, including in my own constituency party, many of whom have opposed real fascists, is a stain on the Tories and Sajid should apologise."
Former shadow cabinet minister Clive Lewis called Mr Javid ` "muppet" and challenged him to repeat his claim outside the House of Commons chamber.
Green Party leader Caroline Lucas also joined calls for Mr Javid to apologise, saying his comments were "absurd and disgraceful".
The Independent has contacted Mr Javid for comment but none had arrived at the time of publication.
Join our commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies