In a highly unusual move, the Environment Minister Michael Meacher summoned him earlier this month to try to persuade him to resign as chairman of the Environment Agency. John Prescott, Deputy Prime Minister, also wants him to go. Another attempt to get him to quit is said to be made shortly.
The 56-year-old Winchester-educated baron, a friend and neighbour of John Major, who was appointed to the newly created job during his premiership, has come under sustained fire this autumn for his running both of the agency and of his estate at Abbots Ripton, near Huntingdon. A former chairman of the Country Landowners' Association - paid pounds 52,000 a year for a two- and-a-half day week - he has nearly two years to go of his five-year term at Britain's biggest quango which is responsible for control of both pollution and flooding.
In September, Friends of the Earth called for Lord de Ramsey to be sacked after he let part of his land to be used to test genetically modified beet and sold some for housing.
Last month five MPs called for his resignation after an independent report condemned the agency for its "poor" handling of the Easter floods disaster in the Midlands.
He has also been criticised for receiving hundreds of thousands of pounds in subsidies from the European Union without declaring his interest as a farmer in the House Of Lords register of members' interests.
But environment ministers are mainly concerned that he has failed to put enough pressure on them and to make the agency into a tough enough champion for green issues. Agriculture ministers are also unenthusiastic.
Tony Juniper, deputy campaigns director of Friends of the Earth, said: "The agency under his leadership is a pale shadow of what it should be." It had "a highly patchy record" of prosecuting polluters.
Lord de Ramsey declined to comment but the agency defended its record, saying that prosecutions had increased. Spokeswomen said he had declared an interest as a farmer when he had spoken in the Lords, but was not required to do so in the register. The genetically modified beet covered only a small area and the housing was of a high standard.