The criminal justice system can be opaque, impenetrable, too concerned with defendants and too little concerned about victims, Nick Herbert said today.
Justice "must be swift, sure and seen to be done, or it's not done at all", the Policing and Criminal Justice Minister said.
Backing the use of technology and calling for more efficiency, he said that the state too often acted "like a bad parent, neglectful in repeatedly tolerating bad behaviour and then inevitably harsh".
"As well as dealing with defendants swiftly, we need to get a proper grip on them, taking the right action to prevent them sliding into ever more serious crimes," Mr Herbert said.
Launching the Government's "swift and sure justice" White Paper, he said there has been a culture within the criminal justice system which has tolerated delay and which must change.
There were 53 separate processes for one common assault case, he said, leading to it taking 15 weeks despite the fact the case only needed six hours of work.
"Justice must be swift, sure and seen to be done, or it's not done at all," he said.
The system "can be opaque and impenetrable" and "we must open the system up", he added.
"It can be too concerned with defendants and too little concerned about victims."
Last year's riots showed how quickly the system could work in some cases and that should be the norm, not the exception, Mr Herbert said.
But he insisted he was not calling for "rushed justice".
Mr Herbert highlighted the case of a Lithuanian lorry driver who was caught drink-driving in Kent last month.
He was taken to the police station, was charged at 9.21am, appeared before magistrates via a virtual court, pleaded guilty and, by 11.35, was disqualified from driving for 36 months, fined £1,500 and ordered to pay costs.
"That's what I mean by swift justice," Mr Herbert said.
Court hours will be more flexible, technology will enable police officers to give evidence remotely and video links for defendants and witnesses will become routine, he added.
Police will also be given simpler guidance on how to deal with offenders, while magistrates will have the power to check officers' use of cautions and penalty notices following concerns that serious and persistent offenders were escaping justice.
Under the plans, magistrates will also be given a stronger role in community justice, with single magistrates sitting outside of courts, such as in community centres, "to dispense rapid and effective justice in low-level, uncontested cases".
Neighbourhood justice panels will also be brought in to deal with anti-social behaviour and low-level crime, with offenders making amends to victims and repairing any damage done.
Mr Herbert said it would be a "measured return of power and responsibility to communities to resolve less serious crimes quickly and rigorously".
"Citizens aren't a threat to justice," he added.
"They are integral to our justice system."
Graham Beech, director at the crime reduction charity Nacro, said: "Speedy justice makes it easier to connect the sentence with the crime.
"But speeding up the process shouldn't be at the expense of proper justice or compromise the crucial need for appropriate sentences."
Javed Khan, chief executive of Victim Support, broadly welcomed the moves, saying the justice system could be "painfully slow" for victims and witnesses.
Shadow justice secretary Sadiq Khan said: "I hope the Government are going to explain exactly how this is going to be funded.
"The fear is that these proposals are simply designed to save money.
"And if this results in the cutting of corners within our justice system, it increases the risk of miscarriages of justice, which will further erode the public's confidence."
Richard Atkinson, chairman of the Law Society's criminal law committee, said: "We are concerned by the Government's obsession with speed and its apparent belief that speed and efficiency is one and the same thing.
"In particular, we question whether there is any need for weekend courts at a time when the numbers of criminal cases are declining and when these proposals will cause problems for prisons and the availability of other professionals in the system.
"There will be significant costs at a time the Government says there is no more money, including for defence practitioners who are mostly small businesses and do not have the same flexibility as large employers, such as the CPS and the court service."
He added: "We are keen to discuss these with Government, but it is mistaken if it believes that these changes can be achieved in a way which is cost-neutral."