Children in care in England are forced to wait an average of 20 months to move in with adoptive parents, according to new figures.
Children's Minister Tim Loughton said the first-ever local authority scorecards were a "trigger for urgent, detailed discussions" to speed up the process.
It is part of an action plan for adoption which includes proposals to reduce the length of the approval process for would-be adopters to six months.
Council leaders and children's services professionals condemned the scorecards and warned they have the potential to cause "unnecessary and avoidable concern in communities where there shouldn't be any".
The figures show that 80 local authorities met the interim thresholds of 21 months from entering care to adoption and matching a child to a family within seven months of a court order.
But 72 councils did not meet one or both of these thresholds which will be lowered to 14 months and four months respectively within four years.
Children in care in Hackney, east London wait the longest in the country to move in with adoptive parents, at two years and nine months on average.
Merton, south London has the second longest time, at two years and eight months, and Liverpool is third at two-and-a-half years.
Mr Loughton said: "Hundreds of children are being let down by unacceptable delays right across the country and throughout the adoption process. Every month a child waits to be placed there is less chance of finding a permanent, stable and loving home. This cannot go on.
"There has been some real progress, with local authorities beginning to bear down on adoption delays and helping in the redesign of a faster but still-thorough adopter assessment process. But these statistics illustrate all too starkly the magnitude of the challenge which we face.
"I make no apology for shining a light on the system to hold local areas to account. I have been clear that we won't hesitate to intervene where the worst delays are not tackled effectively."
He described the scorecards as "not the be-all and end-all" but said more areas needed to strike a better balance between quality placements and the risk of long-term damage to children by leaving them with uncertain futures.
The scorecards feature three key indicators relating to authorities' performance on adoption:
:: The average time it takes for a child to be moved in with an adoptive family
:: The proportion of children waiting longer than they should, including those still in care
:: The average time it takes an authority to match a child with a family after a court has decided that adoption is the right course.
But council leaders and children's services professionals said local authorities could not risk shifting their focus from the quality of placements to speed.
The Local Government Association, Association of Directors of Children's Services and the Society of Local Authority Chief Executives released a joint statement which said: "Councils are passionate about helping children and take their responsibilities towards those in their care extremely seriously.
"The adoption scorecards have the potential to cause unnecessary and avoidable concern in communities where there shouldn't be any, and may put prospective adopters off. Children waiting for adoption will not benefit from government struggling to get its act together.
"The data fails to provide a sound basis for comparison across local authority areas. For example, one council's Ofsted-rated outstanding adoption service looks like a poor performer in the scorecard. This is simply not credible."
Mr Loughton told BBC Radio 4's Today programme there had been a need to "overhaul" the whole adoption system, "not just tinker around".
He said the delay was "the most damaging thing for a child who comes into care, before they eventually find a safe, stable, loving home, with an adoptive family if that's the best destination for them".
Mr Loughton said the adoption scorecards could "cast a spotlight" on the extent of the job to do in adoption and was "not about targets".
He added: "I am not going to introduce targets, that can have some very perverse consequences as the previous Government found.
"What the adoption scorecard does is to give complete transparency of information, it shows those aspects of the whole system that are working well in certain authorities and those that aren't."
Catherine McKinnell, shadow minister for children and young families, said the adoption system needed to perform well at every level.
She said: "Adoption scorecards need to be more sophisticated to ensure the needs of the child come first. That means reflecting the quality of the process as well as the speed.
"The Government must ensure prospective adopters are not put off from coming forward simply because of their local authority's scorecard, when the council may have an outstanding rating from Ofsted.
"Tackling delays is important, but the Government could take action more quickly without the need for more legislation. They also need to speed up decisions in the courts, where the most significant delays occur.
"The Government must give prospective adopters confidence that they will get the post-adoption support they need to ensure that placements endure."