The move came as part of a major reshuffle of the Home Office ministerial team by Home Secretary Jack Straw, who is anxious to restore confidence in his department.
Both of the troubled organisations have been placed in the care of new minister of state Barbara Roche, who was a Labour immigration spokesman in opposition.
Mr O'Brien, who it is understood had hoped to be promoted to minister of state, retains his previous responsibility for race relations matters in a new brief which also includes freedom of information, constitutional issues and treatment of animals.
Much of the new role has been inherited from George Howarth, who has become a Northern Ireland minister.
The chaos at the Passport Agency and the IND largely resulted from the installation of new computer systems, which underwent major teething problems.
Mr O'Brien argued that he had inherited a "shambolic" immigration system. But the ensuing problems caused Mr Straw to tell a House of Commons committee that he "greatly regretted" the situation at the directorate.
Mr O'Brien's supporters argued yesterday that he had successfully put together the controversial Immigration and Asylum bill, currently going through parliament, and was ready for fresh challenges.
But the Home Secretary was also seriously embarrassed over the breakdown of the Passport Agency this summer, which led to travellers having to wait for six weeks for documents which should be available within 10 days.
A backlog of 500,000 applications developed and Mr Straw felt obliged to let people travel on outdated passports by allowing them to obtain extensions at Post Offices. The problems are expected to last until September.
Mr O'Brien staked his job on sorting out the chaos, saying he would resign if the problems were not resolved. But from yesterday, the Passport Agency was no longer his responsibility.Reuse content