In a surprise move, Labour leaders are consulting staff about whether to leave the giant tower block, which is situated next to the Tate Gallery. It comes just three years after the party moved from its former headquarters in Walworth Road, south London, at a cost of pounds 2m.
The Millbank building includes the "war room" whereTony Blair's landslide election victory was masterminded in 1997. Since then, the ultra-loyal Blairite officials based there have been dubbed the "Millbank Tendency" by the Prime Minister's left-wing critics.
One Blair aide said yesterday: "I think we should move out. Millbank has now become too associated with an image we need to leave behind us."
Some Downing Street advisers favour setting up a small election "war room" for a tight-knit group of campaigners, with other staff based at a separate party headquarters.
Labour is expected to remain at Millbank for the next general election but must decide in the next few months whether to move out shortly afterwards or sign a new, 15-year lease.
Fishman Speyer Properties, which also manages New York's Rockefeller Center and Chrysler Building, has told Labour it plans to refurbish Millbank Tower, which was built in 1963 and is Grade II listed.
If Labour remains, it would probably face an increase in its pounds 250,000- a-year rent and have to share the cost of the improvements.
In a report sent to staff, David Pitt-Watson, Labour's director of finance, said the advantages offered by Millbank were unlikely to be replicated in any building bought by the party, since they were typical only of "skyscraper offices".
The pluses included the open-plan office with room for 100 people; a media centre for press conferences and the close vicinity to Parliament.
Although it provided adequate accommodation between general elections, "it does not offer enough room for expansion in the short or long term", said Mr Pitt-Watson. "Any property which was affordable by the party would be a single building on four or five floors like Conservative Central Office in Smith Square."
Last night Labour officials said no decision had been taken. However, some staff have already expressed a desire to move, with one saying: "I don't want to be stuck in this place for another 15 years."
Labour's rethink emerged as the Tories announced last night that they are to spend pounds 350,000 a year on expanding their "war room" at Conservative Central Office. It will become fully operational in the autumn as the party gears up for the next election.
A handful of staff will be made redundant as the Conservative research department becomes more pro-active in attacking Labour. Ironically, the money will be found from the extra state funding for the Opposition announced by Mr Blair. The overall number of research staff will be increased from 20 to 35, in the department's biggest expansion since the Seventies.
It will work more closely with the media operation headed by Amanda Platell, the former Fleet Street editor.