Timothy Clifford, director of the National Galleries of Scotland, where the sculpture, Reclining Figure, is now displayed, said the ruling was ludicrous and would rob the public of the chance to see great works of art.
Mr Prescott has ruled that works commissioned from Moore, Ben Nicholson and other artists for the Time & Life Grade 2 listed building in New Bond Street, London, were an integral part of the architecture and should never have been taken away.
English Heritage said yesterday: "It is extremely useful that this area of listed-building law has been tightened and it will have significance for cases up and down the country."
But Mr Clifford said the Moore piece was not designed for the Time & Life building at all but was "bought off the peg." He added: "This opens a whole can of worms. The Henry Moore is not a one-off. It is an edition bronze. Over 780,000 people have seen it on display in Edinburgh. In the Time & Life building hardly anyone will see it. Henry Moore would have been appalled ... He would have preferred his work to have been seen next to his drawings and maquettes, as it is in Edinburgh."
Mr Clifford said that where a work was an integral part of the building, such as the Rubens cycle in the Banqueting Hall in Whitehall or the Van Dyck at Wilton House, in Salisbury, it was clear the works should not be removed. But that did not apply in this case, nor to Canova's Three Graces, which was allowed to leave Woburn Abbey and has been exhibited at the National Gallery of Scotland and at the Victoria & Albert Museum in London.
In his role as Secretary of State for the Environment, Mr Prescott decided the Moore should never have been removed. Mr Prescott, whose ruling follows a long planning battle by Westminster City Council, also issued a two- month deadline for the sculpture's return.
And he ruled that other works must be returned to the Time & Life building. They are a Nicholson, The Spirit of Architecture, a clock made by the Ironside brothers and a Geoffrey Clarke sculpture called The Complexities of Man.
Westminster council said: "Important works of art can now be protected and preserved in settings for which they were designed, regardless of value and ownership. The owners cannot assume they can take them away, which is what they have done in the past."