The London Mayor said Mr Johnson had not performed his "first duty" in the role, which is diplomacy.
Speaking on the Andrew Marr show he was talking about the case of Nazanin Zaghari-Ratcliffe who is currently imprisoned in Tehran on trumped up charges of sedition.
Ms Zaghari-Ratcliffe, who works for the charitable arm of the Reuters news agency, has been held in the country since April 2016 when she went to visit her parents for Persian New Year.
But Mr Johnson potentially made her plight even worse this week when he wrongly claimed she had been in the country "training journalists" while speaking to the Foreign Affairs Select Committee earlier this week.
Iranian state TV took it as "proof" that she had been there to train journalism as part of a plot "to topple the Iranian regime".
He later refused to apologise when asked to in Parliament even though Ms Zaghari-Ratcliffe, who is a joint British-Iranian citizen, faces new charges of spreading "propaganda" though later apologised "if any words of [his] have been taken out of context and misconstrued to cause anxiety to Nazanin's family."
He spoke to his Iranian counterpart in an attempt to undo the damage but Ms Zaghari-Ratcliffe's husband, Richard Ratcliffe, has called on him to take him to Iran to see his wife and daughter.
Mr Ratcliffe has been unable to see his wife or their now three-year-old daughter Gabriella who is now living in Tehran with her maternal grandparents.
Mr Khan said it was important that Mr Johnson "clarified" his comments because they had been used by Iranian State TV to justify their treatment of Ms Zaghari-Ratcliffe.
He called for Mr Johnson to apologise and get back to her home in north London or step down.
He said: "This is the latest gaffe made by own Foreign Secretary. He has offended the Libyans in relation to what he said about Sirte being 'the new Dubai should they get rid of the dead bodies'. He's offended the Americans by saying President Obama is anti-British because he's 'part Kenyan'.
"He's offended the Spanish, he's offended Sikhs with what he said about whisky tariffs in the Gurdwara. I think he has got to go.
"I think he is our Foreign Secretary whose job is diplomacy in the best interests of our country and if Theresa May was a strong Prime Minister she would have sacked him a long time ago.
"There are questions about why she appointed him in the first place, she did but sure he must have done enough to go".
Join our new commenting forum
Join thought-provoking conversations, follow other Independent readers and see their replies