Lord Young decided he could not remain in his post after reflecting overnight on his comments and the furious reaction they had provoked.
He had told the Daily Telegraph that low interest rates meant home-owners were actually better off thanks to the "so-called recession".
He dismissed the 100,000 job cuts expected each year in the public sector as being "within the margin of error" in the context of a 30 million-strong workforce and said that complaints about spending cuts came from "people who think they have a right for the state to support them".
The 78-year-old former trade and industry secretary in Margaret Thatcher's government also said people would look back on the recession and "wonder what all the fuss was about".
The Prime Minister initially seemed determined to hang on to his adviser, despite Downing Street branding his remarks "offensive" and "inaccurate".
Speaking on a visit to flood-damaged areas in Cornwall today, Mr Cameron said: "Obviously he's extremely embarrassed. He's withdrawn what he said, he's apologised for what he said, and that's absolutely right.
"I mean, he's not a member of the Government, he doesn't speak for the Government and I think he'll be doing a bit less speaking in the future.
"He should get on with what he has been doing and he was obviously extremely embarrassed and he was very quick to retract completely what he said."
However, as the clamour for Lord Young to go showed little sign of dying away, his resignation was announced this afternoon.
In a letter to Mr Cameron Lord Young wrote: "I have considered my position overnight and in view of the reaction to the reporting of the interview I gave earlier this week feel that it would be right to resign forthwith from my position as your adviser.
"I am a very strong supporter of you and the Coalition Government and for what you are about to achieve.
"It was always my ambition to play a small part in helping you restore the fortunes of our nation and you should know that you will continue to have my support in every way."
Unusually, Downing Street did not issue any letter from the Prime Minister expressing regret about his departure and thanking him for his work - which has included a high-profile review of Britain's health and safety culture.
Labour leader Ed Miliband said the fact that Mr Cameron had appointed Lord Young in the first place reflected badly on him.
"Lord Young is right to go, I think his remarks are frankly disgraceful and many of the people who are struggling up and down this country with the consequences of the recession that we had, the consequences of the spending cuts that we are seeing, will be insulted by his comments," Mr Miliband said.
"I do have to say though that it reflects very badly on the Prime Minister that he made this appointment of someone who seems so out of touch with people up and down this country.
"Part of what this Government has to do is demonstrate they do understand the consequences of the decisions they are making and the way people are struggling every day in our economy.
"I'm worried that people like Lord Young indicate a Government that doesn't understand that."
However, Lord Young's former government colleague and friend, David Mellor, expressed anger that the peer had been "cut off at the knees" for making a mistake.
"The Prime Minister should be more careful not to cut off at the knees people who are only trying to do their best to serve him, because there will be plenty of others who will blunder into these traps for the unwary and the Prime Minister needs his friends about him, not to cast them adrift the moment they say anything that he finds remotely embarrassing," Mr Mellor told BBC Radio 4's PM programme.
"He remained at the age of 78 willing to be a public servant, willing to go into Downing Street unpaid, not looking for any personal aggrandisement for himself, just wanting to do the right thing for the country.
"I do not think he should have been forced in effect into a position where he felt he had to resign."
He added: "David Young may have spoken without balance, (but) what he tried to do was try to rebalance the argument being presented by Labour.
"I think with that he should have been supported, and just cutting him off like that is going to be a problem going forward."
And some commentators said Lord Young had a point.
The director general of the Institute of Economic Affairs, Mark Littlewood - speaking before Lord Young's resignation - said: "He is, in many respects, right. The coalition's cuts are modest, not enormous. In 2015, this Government will still be spending more than when it came to power, and in real terms the cuts amount to only 4%.
"And while we should be sensitive to the fact that 100,000 people a year are expected to lose their jobs in the public sector, we should remember that, in the context of a 30 million-strong workforce, they are a tiny minority."
Former Labour trade minister and ex-CBI director general Lord Digby Jones told BBC Radio 4's Today programme: "It is true that if you went into late 2007, were on a tracker mortgage and you were in a sustainable job, then for the next 18 months, you had disposable income and - if you forgive the analogy - you had a good recession.
"He was partly, in one aspect of the economy, right."
Comments