David Cameron today said he kicked Alan Duncan out of his top team after "reflecting" on his claim that MPs were being forced to live "on rations".
But the Tory leader insisted that Mr Duncan agreed it was right for him to make way for veteran Sir George Young.
"I had some time to think about it over the summer and parliament will be coming back soon," Mr Cameron told Sky News.
"One of the most important things I have to do is just make sure all the time and think all the time, have I got the right people in the right posts as a strong opposition and as an alternative government?
"On that basis, I came to the decision that that it was right - and Alan agreed actually - it was right for him to move from that job to another job outside the Shadow Cabinet, but an important job in terms of dealing with the nation's prisons."
Mr Duncan's move to shadow prisons minister represents the end of eight years' constant service in the shadow cabinet - the last nine months as shadow leader of the House.
The Conservative leader has faced calls to sack the Rutland and Melton MP since last month, when he was covertly filmed by campaigner Heydon Prowse.
Asked why people would no longer want to become MPs, Mr Duncan said: "Basically, it's being nationalised, you have to live on rations and are treated like s***."
His remarks were particularly embarrassing because his job meant he led the party on reform of the system of MPs' expenses and allowances.
Controversy has surrounded Mr Duncan since the Daily Telegraph revealed he claimed around £4,000 for gardening at his constituency home over three years, including £598 to service his ride-on lawnmower.
It is understood that he was forced to pay back more than £5,000 by Mr Cameron.
Last night Mr Duncan said it was a "sensible decision", adding: "You have to be realistic about how difficult the expenses issue has been.
"What matters most is winning the election and David Cameron becoming the Prime Minister.
"I don't want to be a brake on that by making a difficult issue more problematic. I am very happy to get stuck into another job."
One of the longest-serving Tory MPs, Sir George entered Parliament in 1974 and served as a minister under Margaret Thatcher and John Major, rising to Transport Secretary. He was previously shadow Commons leader under William Hague, but left the front benches in 2000 to mount the first of two unsuccessful campaigns to become Speaker.
The North West Hampshire MP has most recently been chairman of the Commons Standards and Privileges Committee, which acts as an internal watchdog overseeing MPs' behaviour, but he has not been at the forefront of demands for reform of expenses.
Mr Duncan takes over as shadow justice minister from Edward Garnier, who has now been appointed shadow attorney general.
That position was previously held by shadow justice secretary Dominic Grieve, who had both jobs.
Labour former minister Denis MacShane pointed out that Sir George was an Old Etonian like Mr Cameron, and accused the Tory leader of favouring "public school millionaires".
"Cameron has done too little, too late - why has he dithered for three weeks?" he said. "And if Alan Duncan's not fit to be on the frontbench then why not sack him?"