In a speech, Mr Clarke described the 85,000 prison population in England and Wales as "astonishing" and question whether it delivers value for money for taxpayers.
Keeping a prisoner in jail costs an average £38,000 - more than sending a boy to Eton - but has too often proved "a costly and ineffectual approach that fails to turn criminals into law-abiding citizens", Mr Clarke said. In the worst prisons, jail sentences do no more than produce "tougher criminals" and introduce petty offenders to hardened felons.
The Government has committed to a full review of sentencing policy to ensure that it is deterring crime, protecting the public, punishing offenders and cutting reoffending.
Mr Clarke made clear that he wants it to deliver a system with a far greater emphasis on rehabilitation and what he termed "intelligent sentencing".
"This means prisons that are places of punishment, but also of education, hard work and change," he said in his speech in London.
"It means rigorously enforced community sentences that punish offenders, but also get them off drugs and alcohol and into employment."
He spelt out plans to hire voluntary and private sector organisations on a payment-by-results basis to reduce reoffending, with financial rewards for keeping released prisoners on the straight and narrow.
And he said that he hopes the reforms will pay for themselves by reducing the costs of the criminal justice system.
Mr Clarke indicated he favours a reduction in the use of short sentences, which make it "virtually impossible" to rehabilitate or train prisoners but often cost them their jobs, their homes and their families, making them more likely to reoffend.
But he insisted that his "first priority" is public safety and protecting communities from truly dangerous criminals.
"There are some nasty people who commit nasty offences," Mr Clarke said. "They must be punished, and communities protected. My first priority is the safety of the British public.
"But just banging up more and more people for longer without actively seeking to change them is what you would expect of Victorian England."
Mr Clarke highlighted an 8% rise in reoffending rates between 2006-08 while prison populations were rising, along with figures showing that nearly half of those jailed offend again within a year, including 60% of the 60,000 prisoners freed after short sentences.
He argued that political debate on law and order has been dominated by an unproductive "numbers game", with parties competing to jail the most offenders.
"The measure of success has been solely about whether a Government has spent more public money and locked up more people for longer than its predecessor in the previous years," he said.
"The consequence is that more and more offenders have been warehoused in outdated facilities, and we spend vast amounts of public money on prison. But no proper thought has been given to whether this is really the best and most effective way of protecting the public against crime.
"So I ask this: how do we actually go about improving the safety and protecting the property of honest citizens in the most cost effective way?"
Mr Clarke challenged the "prison works" philosophy espoused by former Conservative Home Secretary Michael Howard and supported by many of the party's grassroots supporters.
And he said: "Prison is the necessary punishment for many offenders.
"But does ever more prison for ever more offenders always produce better results for the public? We have many more people in prison than many other countries with lower crime levels. We have one of the highest crime rates in Western Europe, and one of the highest prison populations.
"I think it is too simple to argue about tougher sentencing or softer sentencing, although it makes for good headlines.
"I believe in intelligent sentencing, which will seek to give better value for money and the effective protection that people want."
During his first visit to a prison since becoming Justice Secretary, in Leeds yesterday, Mr Clarke said that deciding the effectiveness of the justice system by counting the numbers of people in prison was "slightly childish".
"What you should ask is what is the prison system for?" he said.
"The prison system is to protect the public, the prison system is to punish people for whatever it is that has got them here in the first place, but it also should be to ensure you reduce the risk of people committing the same offences again, getting into trouble again and coming back."
He said that the amount of people currently in prison was dramatically high - HMP Leeds is close to its maximum capacity of 1,154 inmates with 1,095 at present - but said public protection was his priority.
Mr Clarke visited B wing of HMP Leeds, meeting prisoners in the textiles workshop.
Yesterday, David Cameron said short prison sentences are needed in some circumstances.
He was speaking at his first PM Direct meeting in Leeds.
The Prime Minister said short sentences are needed for some persistent offenders.
He said: "Too many of the community punishments right now we don't think are meaningful, do we?
"So we need good punishments then we can probably keep more people out of prison.
"You can't get rid of all the short-term sentences but I accept there is a good case for saying sending someone to prison for a few weeks or a few months, there's no time to reform them or rehabilitate them or train them, so what's the point?"
Mr Clarke rejected Labour claims that his approach was at odds with strong backing by Mr Cameron during the General Election campaign for short sentences for vandalism and disorder.
The Tory leader cited his mother's experience as a magistrate as he criticised the sentencing policy of Liberal Democrat leader Nick Clegg - now his coalition partner and Deputy PM - during one of the televised leaders' debates.
"When someone smashes up the bus stop, when someone repeatedly breaks the law, when someone is found fighting on a Friday or Saturday night, as a magistrate, you've got to have that power for a short prison sentence when you've tried the other remedies," Mr Cameron said.
"I am not at direct odds with my leader. You do need short sentences for the kind of nuisance criminal who keeps being a recidivist," Mr Clarke told the BBC Radio 4 Today programme.
Increased prisoner numbers had "not done much about" problems such as vandalism and "loutish" behaviour though, he said.
Shadow justice secretary Jack Straw accused Mr Clarke of a return to the "hand-wringing" approach to crime in existence before Mr Howard's regime.
"He (Howard) deserves credit for turning the tide, as does the opposition leader at that time, Tony Blair, who encapsulated the need for a balanced policy with his call to be 'tough on crime, tough on the causes of crime'.
"Now, though, in the crazy world of coalition government, Kenneth Clarke shows he has learnt nothing about fighting crime in the time since he was in charge of prisons 17 years ago," Mr Straw wrote in the Daily Mail.
"Labour home secretaries since 1997, including me, sought to strengthen this approach. And it has made a big difference to the peace and tranquillity of local communities.
"A key factor in reducing crime has been the increased number of offenders sentenced to prison," he wrote.
"Does anyone seriously believe that crime would have come down and stayed down without these extra prison places?"
Millions could be shaved from budgets by running prisons more efficiently, he said, with no need to reduce places.
