And Earl Attlee, grandson of the post-war Labour premier Clement Attlee, has joined it as a Whip, specialising on defence, transport and Northern Ireland.
Others quitting the Conservative front bench, mostly previously announced, were ex-Lord Chancellor Lord Mackay of Clashfern, former Environment Minister of State Earl Ferrers and Whips Baroness Trumpington and Viscount Long.
Former transport minister Viscount Goschen, whose resignation was announced by Tory chief whip Lord Strathclyde before the general election, is to remain as the party's transport spokesman.
Lady Chalker, made a peer after losing her Wallasey seat in 1992, served more than 20 years on front benches in both Houses of Parliament.
Earl Attlee, a crossbencher until his defection to Conservatives in March, is one of 11 newcomers to Tory peers' leader Viscount Cranborne's team, announced yesterday.
The others include former Tory Euro-MPs' leader Lord Kingsland, who will lead on legal affairs, and former London Boroughs Association leader Lord Bowness, who will lead on local government and the environment.
Viscount Astor returns to the front bench as a home affairs spokesman, while former sports minister Lord Moynihan is to speak on foreign affairs.
Lord McColl of Dulwich, a surgery professor and former aide to the former prime minsiter, John Major - is to speak on health. Lord Pilkington of Oxenford will speak on education and employment.
New to the Tory whips' office are Baroness Rawlings, Baroness Anelay of St Johns, Baroness Byford and Lord Burnham.