David said he was returning to the backbenches in order to avoid becoming a "distraction" to his brother's effort to provide Labour with a fresh start.
He added it would also allow him to "recharge my political and intellectual batteries" and spend more time with his family.
The widely-expected announcement came four days after he lost out to Ed in the knife-edge ballot for the Labour leadership, and was hailed by his brother as a "thoughtful and gracious" move.
Speaking at the Labour conference in Manchester, Ed Miliband said: "He is my brother and I am very clear that, as leader of this party, my door is always open for him to serve in the future, either in opposition or back in government."
David - who spent the day with his family at his north London home - said he would remain MP for South Shields and would devote himself to developing the network of community organisers he set up as part of his leadership campaign.
In a letter to the chairman of his constituency party, he said he was confident that stepping back from the frontline was "the right decision for Ed, for the party and for me and the family".
He made clear he was not ruling out an eventual return to the front bench but refused to be drawn on whether his brother had privately urged him to stay.
Ed described him as "a massive talent" and added: "We would have been a stronger team with him in it, but I think he is right to have made the decision he did."
David's departure from Labour's top team came as the deadline passed for nominations to the shadow cabinet.
Some 49 Labour MPs - ranging from big beasts like Alan Johnson, Ed Balls and Peter Hain to relatively unknown backbenchers - are standing for 19 elected positions in a ballot of MPs which concludes on October 7.
In a sign of the new leader taking control of his party, Rosie Winterton was elected unopposed as opposition chief whip after Ed Miliband asked incumbent Nick Brown - a close ally of former Prime Minister Gordon Brown - not to stand.
Allies of David Miliband praised his decision to quit the front benches.
Mr Johnson said it would make it easier for Ed to "flourish as leader" by avoiding "constant scrutiny of their relationship".
Former deputy prime minister Lord Prescott said the party owed David "an enormous debt of gratitude".
And fellow-Blairite Tessa Jowell said David had "done the right thing for himself, and for his family, and he has definitely done the right thing for the party".
She added: "This has now given his brother Ed the space to run the party."
But Conservative Party chairman Baroness Warsi said that the fact that a leading architect of New Labour did not want to serve in the shadow cabinet "speaks volumes about the direction in which the new leader is taking Labour".
"After being elected by the unions, this is further evidence that Ed Miliband is vacating the centre ground of British politics," she said.
Speculation over David's future overshadowed Labour's annual conference, with expectations growing as the days went by that he would decline to serve in his brother's top team.
Much of the press coverage of Ed's crucial first leader's speech yesterday was dominated by a critical comment from David picked up by a TV microphone.
In his letter today, David made clear his concern that if he remained in the shadow cabinet, Labour's efforts to supplant the coalition Government would be seen through the prism of his personal relationship with his brother.
"The party needs a fresh start from its new leader, and I think that is more likely to be achieved if I make a fresh start," he said.
"This is now Ed's party to lead and he needs to be able to do so as free as possible from distraction.
"Any new leader needs time and space to set his or her own direction, priorities and policies. I believe this will be harder if there is constant comparison with my comments and position as a member of the shadow cabinet.
"This is because of the simple fact that Ed is my brother who has just defeated me for the leadership. I genuinely fear perpetual, distracting and destructive attempts to find division where there is none and splits where they don't exist, all to the detriment of the party's cause.
"Ed needs a free hand but also an open field."
Mr Miliband said he had spent 16 years "in or around the top of politics" since joining Tony Blair's team in opposition in 1994 and wanted to "recharge my political and intellectual batteries to be of greater service to the party and the country".
He indicated he would pursue interests in education, the environment and foreign policy and devote himself to "understanding better the new challenges and new ideas that will dominate the next couple of decades, and figuring out how to put our values into practice".
He said that he wanted to give more time to his sons Isaac, five, and Jacob, two, and his wife Louise Shackleton.
"I have essentially been a Cabinet minister for the whole of Isaac and Jacob's lives," he wrote.
"That is tough for me and tough for them. One happy consequence of the leadership election will be more time with Louise and the boys."
The chairman of South Shields Constituency Labour Party wrote to the MP praising his "courageous and selfless act of leadership".
Alan Donnelly made public his formal reply to the former foreign secretary's announcement, which said: "Thank you for letting us know of your plans following the election of Ed as our new party leader.
"In South Shields we all realise how difficult the last few days have been for both Louise and you and it was quite right for you to take some time to consider the implications of the election result for your family and for the Labour Party.
"Ed and Labour's new parliamentary team must be free to focus all of their attention on challenging the coalition Government, and as you have recognised, your membership of the shadow cabinet at this time would simply act as a distraction.
"In withdrawing from the shadow cabinet elections today you are demonstrating a courageous and selfless act of leadership which, although very sad, I support completely.
"You said in your speech on Monday that we must all throw our weight behind Ed Miliband and as you know I saw Ed immediately after his speech yesterday to congratulate him and to invite him to visit South Shields in the very near future, an invitation he immediately accepted.
"It is important for you to know that we in your local party and I believe Labour Party members across the country are enormously proud of you and your achievements in government and grateful for all that you have done for us and will do in the future.
"That is why I am so pleased that you intend to continue to serve the people of South Shields and the Labour Party.
"Finally, you said on Monday that there should be no more cliques or factions.
"You were speaking on behalf of us all when you said that, like you we all now want to turn the page and look to the future, united behind our excellent new leader.
"We hope to see you and Louise back in South Shields soon."
Labour frontbencher Tessa Jowell said David had "absolutely done the right thing".
She told the BBC: "He has done the right thing for himself, and for his family, and he has definitely done the right thing for the party. This has now given his brother Ed the space to run the party."
Ms Jowell dismissed suggestions that the decision had anything to do with his wife Louise being angry with Ed.
Shadow justice secretary Jack Straw said that even if Mr Miliband had taken "Trappist vows of silence" it would still have generated a story.
"That is the media we live with. Everyone in the media knows that is the case," he added.
Mr Straw - who is himself stepping down from frontline politics - said he had initially believed Mr Miliband would stay on.
But he added: "The sensible thing to do is what David has done. As we have seen from these pictures on TV, he is a very relieved man."
Former deputy prime minister Lord Prescott paid tribute to David Miliband, saying that the party owed him "an enormous debt of gratitude".
"He helped create a record that we can be proud of and I respect the fact that during the leadership campaign he chose to defend it," he wrote on his blog.
In a round of television interviews, Mr Miliband said it was "tempting" to suggest reforms to Labour's system for electing a leader - which he dubbed the "selectorate".
But he insisted it was for his brother - whose victory came down to the votes of trade union members - to decide future policy.
He said he had no regrets at not seizing chances before the general election to challenge Gordon Brown for the leadership.
And he denied any sense of "betrayal", saying he had a sense as early as last year that Ed might challenge him and that he was "perfectly within his rights" to do so.
Asked why he thought he lost, he told Sky News: "There was a choice of two strong candidates and the electorate decided - the selectorate decided - by a narrow margin that Ed should be the leader.
"However tempting it is to offer some views (on reforming the system) it is important that I do not get in the way of that.
"If Ed wants to develop plans to reform then that's his own choice. One of the rules that is very important is that he must have an open field to lead as he sees fit."
Asked if he wished he had challenged Mr Brown, he said: "I always made the right decision for the party and for the country. I do not regret remaining loyal no, not at all.
"I was not willing to make a bad situation worse by joining my friend James Purnell in leaving government. I don't think that would have achieved anything."
He said he was glad the contest had not become a "bloodbath" and said "good humour and good cheer were kept up right the way through and good family relations as well".
"I got more of a sense last year that Ed wanted to run, and he was perfectly within his rights to do so, so it was not a complete shock," he said.
"Although once the campaign gets going it is then a long and arduous process, a gruelling process. But it didn't become the bloodbath that many people predicted."
Asked if his decision to step aside showed it was now a bloodbath, he said: "I am not dead - I am still here."
John Hannett, general secretary of the shopworkers' union Usdaw, which had supported David Miliband in the leadership election, said: "I respect David's decision but do hope his move to the back benches will only be temporary as he is a great asset to the party. Given the challenges facing both party and country, his talent and experience will be sadly missed."
Comments