He was referring to a video posted on his social media account in June – under the hashtag “Rory Walks” – in which he met the three men.
One of the men responded “I don’t f*** with politics” when Mr Stewart told them he was a politician and running to be prime minister.
It was one of many filmed during his campaign to succeed Theresa May, and ran under the caption: “You can’t win ‘em all... Brick Lane.”
In comments reported by The Guardian, Mr Stewart, a former cabinet minister, said at the Emmanuel Centre in London on Wednesday: “One thing about social media is that it allows people to see politicians listening ... I can go to Brick Lane and three sort of minor gangsters can come up to me and spend a minute telling me I’m an idiot.
“And I can film it on my phone and put it up. And people love watching people being rude to politicians.”
Mr Stewart’s team said the comment was “obviously a light-hearted, self-deprecatory remarks – where Rory was reflecting on an amiable exchange he had when walking along Brick Lane”.
“Anyone who knows anything about Rory would know it’s absurd to suggest he was implying anything malign, and if any offence was caused he’d be the first to apologise.”
In response to the remarks, Labour’s shadow home secretary Diane Abbott, however, posted on her Twitter account: “This is racism”.
“It is completely shameful,” she added. “Especially when it comes from someone running to be the mayor of our diverse and multicultural capital city.”
Mr Stewart’s Liberal Democrat rival in the London mayoral contest, Siobhan Benita, also said his comments showed “how he’s out of touch with Londoners”.
She added: “His walks around the capital are cover up for his lack of knowledge about the city he claims he wants to run”.
The ex-international development secretary announced he will stand down as an MP earlier this month in order to run as an independent in the capital’s mayoral race next year.
He called for an end to the “suffocating embrace of our dying party politics” and vowed to defy the “mutual insults ... lazy habits, half-baked ideas and pointless compromises” of the current system.
Mr Stewart later apologised via social media, saying: “I am very sorry towards the guys and towards everyone else. I was wrong.”
Join our new commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies