Wellingborough Prison will close by the end of this year with the loss of almost 600 prison places, Kenneth Clarke said today.
The Justice Secretary said closing the category C jail in Northamptonshire will save £10 million per year.
The prison population in England and Wales was 86,652 last Friday, with enough capacity for a further 3,500 inmates, the Prison Service said.
Along with the annual saving, closing the jail will also avoid costs of up to £50 million which would have been necessary to maintain the longer term viability of the prison, the Prison Service said.
It is expected staff will move to other jails and compulsory redundancies are not expected.
Mr Clarke said: "The public has the right to expect continuing improvement in the quality and efficiency of public services, without compromising public safety.
"Closing outdated and expensive prisons is an important step in our strategy to deliver a fit-for-purpose, modern custodial estate that can provide high quality, cost-effective and secure regimes that protect the public and reform prisoners.
"Closing this one prison alone will save millions of pounds for the taxpayer."
He went on: "The changes will reduce our current capacity by 588 places and I am confident that they can be safely managed within existing headroom, whilst maintaining our ability to cope with any potential increase in population."
The Prison Service added that jails will only close when capacity allows, saying: "We will always ensure that there are sufficient places for those offenders sentenced to custody by the courts, including a margin to manage fluctuations in the prison population."
Two new prisons - the G4S-run Oakwood jail near Wolverhampton and the Serco-run Thameside prison in south-east London - opened earlier this year, creating about 2,500 extra places.
Juliet Lyon, director of the Prison Reform Trust, said: "Today's announcement of the closure of Wellingborough Prison signals a slowing down in the inexorable growth of the prison population over the past two decades.
"Each prison place costs nearly £40,000 a year and half of all prisoners reoffend within a year of release.
"Before we sink even more taxpayers' money into large private jails, it's time to invest in effective measures to cut crime that give a better return for the public purse."
She went on: "There is sound evidence for a switch of investment to well-organised community sentences, restorative justice and appropriate treatment and care for people with mental health needs, learning disabilities and drug and alcohol addictions."
In the Commons, Wellingborough MP Peter Bone railed at the decision to close the prison, claiming it would cost 600 jobs in his constituency.
He applied for an emergency Commons debate before MPs return to their constituencies for the summer later today but was rebuffed by Speaker John Bercow.
Mr Bone said: "This is a good, improving prison which is now being closed without any consultation or appeal process.
"The reason given for its closure is we have too much space in our prisons. This is coming after years and years of being told they are overcrowded and we need more spaces.
"The previous governments allowed prisoners out early because there just was not enough space. There seems to be no consistency within the Justice Department.
"Hundreds of people losing their jobs in my constituency for a short-term dubious economic saving is plain wrong.
"This is the wrong prison being closed for the wrong reasons at the wrong time."
Steve Gillan, general secretary of the Prison Officers' Association (POA), said: "This is short-sightedness by ministers.
"They talk about rehabilitation but have given prison officers no space to do that key work that protects the public from re-offending," he said.
"The prison population is vastly overcrowded.
"Government claim they have 3,000 space headroom. That is just smoke and mirrors.
"The vast majority of our prisons are overcrowded."
Pete McParlin, the POA's national chairman, added: "There needs to be a proper debate regarding prisons and the future of the criminal justice system.
"The race to the bottom is clearly accelerating with no proper investment structures in place."