The home secretary made the comments in a conference call with Conservative MPs concerned about an increase in small boat crossings.
More than 1,000 men, women and children have reached the UK coast by sea in 10 days, according to analysis by the Press Association.
It takes the total for 2020 to at least 4,500 – more than double the number thought to have crossed in the whole of 2019.
One MP on the call told The Sun on Sunday: “Priti was asked why the migrants are so desperate to leave France and come here. She told us some believe racism to be an issue.
“They claim they feel discriminated against when, for example, looking for work in France. Others claimed they feared being tortured if they stayed in France or Germany.
“Priti stressed that she didn’t believe any of this to be true. She was merely trying to explain the pull factors.”
Official reports have cited “dire” conditions in migrant camps along the French coast as a driving factor for those attempting boat crossings, and authorities have conducted a series of forced evictions.
There have been complaints of brutality, harassment and intimidation by police in northern France, against both migrants and people supporting them.
White nationalist groups, including the French branch of Generation Identity, have targeted migrants around Calais and Dunkirk in the past.
A Home Office spokesperson said: “The home secretary is clearly frustrated by the increasing number of small boats crossing the Channel – this is compounded by the fact we are currently restricted in our response by EU regulations.
“That is why the home secretary is committed, along with other government departments and No 10, to have legislation ready following the end of the transition period.”
Last year, Ms Patel pledged that the migrant crossings would be an “infrequent phenomenon” by spring this year, and has since insisted she is working to make the route ”completely unviable”.
Earlier this week, she reportedly told Tory MPs that the asylum system was “broken” as she promised to bring forward legal reforms and claimed laws were being exploited by “leftie, Labour-supporting lawyers” trying to frustrate government efforts to deport people.
The prime minister has suggested he wants a legal change to make it easier to “send away” migrants who reach the UK.
Downing Street claimed Brexit would allow the UK to draw up a new framework for dealing with migrants outside the “inflexible and rigid” Dublin regulation.
The UK has already made hundreds of deportation requests under the law, which requires asylum seekers to claim asylum in the first safe country they arrive in.
Immigration minister Chris Philp promised a “new, comprehensive action plan” to stem the latest surge in crossings after talks with French officials in Paris on Tuesday.
Last month, the UK and France launched a new intelligence-sharing unit intended to crack down on smugglers, and millions of pounds have been spent in the last 10 years on bilateral arrangements aiming to stop migrants leaving France.
On Saturday, the Ministry of Defence announced that it has no current plans to deploy royal navy ships in the Channel.
It will deploy additional military aircraft for surveillance and provide personnel to help the Border Force after the home secretary lodged a request for support.
The UN Refugee Agency (UNHCR) warned the deployment of warships could cause ”fatal incidents“.
“Although increasing numbers of people have been crossing the Channel by boat this summer, the numbers remain low and manageable,” a statement said. “Saving lives should be the first priority.”
Ministers have been accused of “stoking tension and division” with comments on the crossings, after the prime minister called the journeys “very bad and stupid and dangerous and criminal”.
A former royal marine, Dan O’Mahoney, has been put in the new post of “clandestine Channel threat commander”, but the Home Office denied claiming migrants present a threat.
Mr Mahoney called the crossings “dangerous and unnecessary”, adding: “I have already met the French and reaffirmed our joint commitment to stopping these crossings and making this route totally unviable.
“I will continue to work with the French to tackle the criminals behind the crossings and the organised crime networks which put people’s lives at risk.
“Twenty-three people smugglers have been jailed this year and two more were charged in recent weeks.”
The government was warned in an official report by MPs nine months ago that its own policies were “pushing migrants to take more dangerous routes” across the Channel.
Ms Patel was a member of the Foreign Affairs Select Committee when it heard evidence that the number of migrants trying to reach the UK by sea would increase and current measures were not working.
The resulting report called for the government to increase legal routes to seek asylum, improve “dire” conditions in French camps, and address the root causes of migration.
Chris Bryant, a Labour member of the Foreign Affairs Committee, told The Independent that the current situation “wasn’t only predictable, but we predicted it”.
He said measures previously billed as “clampdowns” had been “notoriously ineffective”.
The Home Office said the UK had resettled more than 25,000 vulnerable refugees since September 2015, with almost half of them being children, and had presented a future offer to the EU on family reunification for asylum seekers.
Join our commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies