He said: "There are people who are trying to feed fears and intolerance for a broad range of reasons... I will remain positive and remain pulling people together, pulling communities together right across this country."
But he stopped short of apologising.
Mr Trudeau was responding to questions about a woman who interrupted him as he was speaking at the corn roast - a party at which green maize is roasted and eaten.
“I want to know when you are going to refund the $146m we paid for your illegal immigrants” she said to him in French - the dominant language in the Canadian province.
Replying in the same language: "Madame, one of the important things in politics to listen."
She shouted back: “You have not answered my question."
After a few more exchanges, the prime minister said to cheers from the crowd: "Intolerance towards immigrants has no place in Canada".
Later as he shook hands with members of the public, the heckler approached him and asked if he tolerated "Quebecois de souche" - which literally translates to “old stock Quebecers” - as much as “illegal immigrants”.
It is also a right-wing, nationalist party in the province
He replied that he too was a “proud Quebecer”, adding: “Your racism has no place here”.
A woman belonging to a right-wing group called Storm Alliance later reportedly identified herself as the heckler in a private Facebook post.
Both the Canadian Broadcasting Corporation and the Globe and Mail newspaper reported that she wrote that was she was "pretty happy that I participated in him blowing a gasket".
The group has been protesting the arrival of asylum seekers at a border crossing near St-Bernard-de-Lacolle in the province.
Approximately 30,000 people have illegally crossed into Canada in the last 18 months, where they have sought asylum. Many have ended up in the provinces of Quebec and Ontario.
Both have asked Mr Trudeau’s government to pay for the costs associated with the asylum seekers. It has allocated US$132.7m (£103m) towards US-Canada border security.
Of the $38.3m (£29.8m) total pledged by the government to help provinces bearing extra costs of asylum seekers, Quebec would receive approximately $27.6m (£21m).
Canadians appeared to be split on if they felt Mr Trudeau had handled the event well, with many agreeing with the Prime Minister
Conservative Party leader Andrew Scheer tweeted his reaction to Mr Trudeau’s lack of an apology, writing: "By sweeping away legitimate questions on his failed border policy with vile personal insults, it is Trudeau himself who is guilty of polarizing the debate. No one has done more to divide Canadians than he has."
Lisa McLeod, the Ontario immigration minister, echoed his comments and said when Mr Trudeau “fearmongers and calls people un-Canadian or racist, [he] really debases the debate that we're having” about immigration.
Join our new commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies