A group calling themselves “Yellow Vests UK” have been returning to Westminster almost daily to confront and heckle pro-EU demonstrators who gather at College Green.
On Monday, they chanted "Soubry is a Nazi" as the MP did television interviews before following her along the street shouting abuse.
Harry Todd, national organiser for Leave Means Leave, told The Independent: "Their tactics are completely and utterly unacceptable. I condemn their behaviour in the strongest terms.
"They are people who are angry and they don't know how to express it so they are lashing out. They genuinely feel betrayed. I understand their frustration and I understand their anger, but their tactics are wrong and it doesn't help anyone."
He said he had himself been shouted by Ms Soubry's abusers after he refused to let them use Leave Means Leave's placards.
However, other Brexit supporters downplayed the harassment. One woman demonstrating in College Green on Tuesday, who gave her name only as Carol, said: "What you're seeing is public anger boiling over. Yes, Soubry was called a Nazi. But it's just a name, it didn't hurt her."
Monday’s harassment led dozens of MPs to write to Met commissioner Cressida Dick expressing their “serious concerns” about the “deteriorating public order and security situation” outside parliament.
It comes as police investigate whether any criminal offence was committed during the abuse of Ms Soubry on Monday.
Anti-Brexit campaigners camped out near parliament said they were targeted on an almost daily basis by people linked to Yellow Vests UK.
One woman, who asked not to be named, told The Independent: "There's up to 35 at a time, which is quite intimidating.
"They put cameras in our faces, insulting us, trying to pull our flags down - just intimidation, basically. We've been called Nazis, paedophile enablers, traitors."
Five supporters of the group were arrested by police following scuffles during protests on Saturday and Monday.
Join our commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies