The US president gave the retired Marine Corps general a dressing down in an incident seen by a number of administration staff, it is claimed.
Mr Trump's temper is said to have flared after advisers suggested he should stop politicising normal issues of government following another blistering attack on the media at a rally in Phoenix, Arizona.
Sources told the New York Times that Mr Kelly, 67, reacted to the outburst calmly, but said he later told colleagues he had never been spoken to in such a way during 35 years of military service and would not put up with similar treatment again.
The veteran of army campaigns in Iraq and South America was appointed as the successor to Reince Priebus five weeks ago and has attempted to restore order during a period of instability in the White House.
On Friday Mr Trump seemingly downplayed reports of a rift between them by tweeting: "General John Kelly is doing a great job as Chief of Staff. I could not be happier or more impressed."
The New York billionaire is said have been frustrated by Mr Kelly's moves to limit the number of advisers who have unrestricted access to him - a policy which runs counter to Mr Trump's love of spontaneity and brashness.
Roger Stone, a longtime adviser to Mr Trump, said it was "inevitable" that he "was going to rebel against the latest manager who wanted to control him".
“Ultimately Donald Trump is his own man, and he’s going to resist all the control and regimented systems Kelly is trying to impose,” he told the Times.
Aides say Mr Trump admires Mr Kelly's credentials, respects his leadership and management skills, and praises him often, both in private meetings and at public events.
In a tax policy speech on Wednesday in Missouri, Trump singled out Mr Kelly's work to decrease the number of illegal border crossings when he was secretary of homeland security.
The pair travelled to meet survivors of Hurricane Harvey in Texas on Saturday following the devastating storm which killed more than 40 people, displaced more than one million and dumped as much as 50 inches (127 cm) of rain in some areas.
Sources close to Mr Trump have said he is simmering with displeasure over what he considers personal disloyalty from National Economic Council Director Gary Cohn, who criticised his responses to a deadly white supremacist rally in Charlottesville, Virginia, on 12 August.
He is also said to have grown increasingly frustrated with Secretary of State Rex Tillerson, who has clashed with the president on issues including Afghanistan troop levels, the blockade on Qatar and Cuba policy.