The Prime Minister's jibe - mimicking the famous car insurance advert starring Michael Winner - was directed at shadow chief secretary to the Treasury Angela Eagle as she heckled him on the floor of the Commons.
Amid uproar from the Labour benches, Mr Cameron resisted calls for an immediate apology.
But a senior Labour source described the comment as "patronising, sexist, insulting and un-prime ministerial", adding: "He certainly should apologise."
A Downing Street aide tried to play down the row, telling reporters: "I think you will find it is a popular advert. I think you are maybe over-analysing a humorous remark."
Mr Cameron's comment came as he defended his Government's plans to reform the NHS, which he said were even backed by former Labour MP Howard Stoate, a practising GP.
Ms Eagle, seated on the Labour front benches, loudly retorted that Dr Stoate had stood down at last year's election, rather than being defeated as the PM claimed.
The Prime Minister told the Wallasey MP: "Calm down, dear, calm down. Calm down and listen to the doctor."
As the Labour benches erupted in outrage, shadow chancellor Ed Balls angrily pointed to Ms Eagle and his wife Yvette Cooper, apparently demanding to know who the PM had been referring to, while party leader Ed Miliband appeared to call for an apology.
But the Prime Minister told them: "I said calm down, calm down dear. I'll say it to you if you like ... I'm not going to apologise. You do need to calm down."
Speaker John Bercow had to step in to quieten the Labour benches, telling MPs: "There's far too much noise in this chamber, which makes a very bad impression on the public as a whole."
But Labour MP John Woodcock revived the row later, telling MPs that the Prime Minister was "losing his rag because he is losing the argument".
Ms Eagle told BBC News: "I don't think any modern man would have expressed himself in that way."
But asked if she wanted an apology from Mr Cameron, she said: "The Prime Minister is responsible for what he says in the Commons. I think if there is an apology to make, it should be for the dreadful growth figures we have seen today, which demonstrated that the economy has effectively flatlined for six months."
She added: "I have been patronised by better people than the Prime Minister... It is for the Prime Minister to decide whether he expressed himself appropriately in the Commons. It is up to him as to whether he wants to annoy 51% of the population."
Labour's Caroline Flint accused Mr Cameron of using the word "dear" to "put women down", and said she had been on the receiving end of the same tactic from Communities Secretary Eric Pickles in the chamber last December, when he told her to "just get behind the programme then, dear".
In a message on Twitter, the shadow communities secretary said: "PM isn't the only one fond of using 'dear' to put women down. Pickles used the same tactic on me. See Hansard 6.12.10."
Meanwhile, the PM's comment was a hot topic for debate on parenting website Mumsnet, where contributors were split between those seeing it as patronising and those who regarded it as acceptable banter.
One poster, MrsBaldwin, wrote "In my view David Cameron just knocked some gloss off his 'I'm a modern Tory' spin. Is that what he says to SamCam at home?" while another, squeakytoy, asked "Why is it appalling? I would say it to my female and male friends if they were getting hysterical or shouty. It is nothing to do with gender."
Labour's deputy leader Harriet Harman said: "David Cameron's contemptuous response to Angela Eagle MP at Prime Minister's Questions today shows his patronising and outdated attitude to women.
"Women in Britain in the 21st century do not expect to be told to 'calm down dear' by their Prime Minister."
Downing Street later said that "no formal request for an apology has been made".
A spokeswoman said: "Let's not over-analyse something that was clearly a humorous remark. He was mimicking a popular TV ad and clearly no offence was meant."
A Tory spokesman said: "Labour seem desperate to talk about anything other than the economy after the good news on growth figures and Ed Miliband's weak performance today."
It later emerged that Mr Cameron previously used the "calm down dear" line in the House of Commons to a male MP.
Responding as opposition leader to a statement on Afghanistan from then PM Gordon Brown in December 2007, Mr Cameron was interrupted by David Miliband.
After exchanging some banter with the then foreign secretary, he added: "Calm down dear, you can answer some more questions." It is unclear from the footage whether he was addressing Mr Brown or Mr Miliband, who was seated next to the PM.
Mr Winner said the uproar caused by Mr Cameron's remark was "ridiculous".
He said: "I find it unbelievable that Harriet Harman, the deputy leader of a supposedly important party, sees fit to go on about my phrase 'calm down dear', which the Prime Minister used in Question Time today.
"Which planet is Miss Harman on? Does she not know that this phrase which I created 10 years ago has become a part of the nation's language?
"It's used by everybody. People come up to me in the street and in restaurants saying it. It's a totally harmless bit of fun.
"The Prime Minister has used it before which shows he's in touch with the British public, which is more than Harriet Harman seems to be.
"There are many things demeaning to women - violence in the home, forced prostitution and hundreds of other serious matters which Harriet Harman should concern herself with if she was so inclined.
"To make an issue out of a bit of fun just makes her look ridiculous."
Comments