Utterly wrong! Flawed! Academics deride league tables that guide Michael Gove's reforms

Model used to calculate triennial rankings contains 'profound' error, says mathematician

The world's most influential education league tables used by Michael Gove as a reason for "whole system reform" of the sector have been derided as "useless and meaningless" by two academic studies.

The Education Secretary regularly points to Britain's position in the "landmark" Programme for International Student Assessment (Pisa) rankings of 15-year-olds' science, maths and reading ability to justify his policies. In a speech last year to the Education World Forum Mr Gove even described the German mathematician who runs Pisa, Andreas Schleicher, as "the most important man in the British education system".

Dr Hugh Morrison, a mathematician from Queens University, Belfast, said that the model used to calculate the triennial rankings is "utterly wrong" because it contains a "profound" conceptual error that confuses objective and subjective probability. "Pisa will never work", he added.

Professor Svend Kreiner, a statistician from the University of Copenhagen in Denmark, also said the Pisa model is fundamentally flawed. In a paper published this summer, he challenges Pisa's reliability and shows how results fluctuate significantly according to which test questions are used. He also reveals how, in the 2006 reading rankings, Canada could have been positioned anywhere between second and 25th, Japan between eighth and 40th and the UK between 14th and 30th.

Mr Gove has said that Britain between 2000 and 2009 "plummeted in the world rankings from 4th to 16th for science, 7th to 25th for literacy and 8th to 28th for maths". Pisa's 2012 results are published later this year.

The questions used for Pisa vary between countries and between students participating in the same assessment. In the 2006 tests half the students were not asked any reading questions but were allocated "plausible" reading scores to help calculate their countries' rankings. Responding to the news of the studies in today's TES, one teacher said: "The first year our kids did the PISA test, the science content was aligned to aspects of 5-14 that we never taught. [One asked]: 'Sir, why did they ask about dinosaurs and geology when we don't do it?'"

Professor Kreiner says the Rasch model used by Pisa can only work if the questions are of the same level of difficulty for each country taking the test. He said his research proves that this is not the case and therefore the comparisons that Pisa makes between countries are "useless".

Professor Kreiner first raised the issue two years ago, but the Organisation for Economic Development and Co-Operation (OECD), which runs Pisa, countered by suggesting that he had been able to find such wild fluctuations in rankings only by deliberately selecting particular small groupings of questions to prove his point.

Professor Kreiner's latest paper uses the same groups of questions as Pisa and comes up with very similar results to his initial analysis. "It is meaningless to try to compare reading in Chinese with reading in Danish," he said today.

Dr Morrison said that the Rasch model made the "impossible" claim of being able to measure ability independently of the questions that students answer. "I am certain this (problem) cannot be answered," he told TES.

A spokesperson for the OECD claimed that country rankings take account of "the uncertainty that results from sample data" adding: "No model exactly fits the data. Pisa will always seek to improve and learn from the latest research methods."

Christine Blower, General Secretary of the National Union of Teachers, said: "Michael Gove's habitual simplification and half-truths about Pisa have for many years formed the central plank of his justifications for widespread reform. These new calls from academics should encourage all to take the Secretary of State's every utterance on standards with a large pinch of salt."

A Department for Education spokesperson said: “While there will always be academic debate over methodology, international benchmarking is vitally important to ensure we are competing with the world’s best-performing education systems.”

Flawed evidence: Gove’s gaffes

Michael Gove has often been accused of citing flawed or non-existent research.

A Freedom of Information request revealed that PR surveys by Premier Inn and UKTV Gold were the source of his evidence when he said British teenagers were ignorant about key historical events.

In April, he told a conference that in the Far East “school days are longer, school holidays are shorter”. In fact, English pupils spend around 7,250 hours in the classroom, far more than South Korea, at 6,000, and Japan, at 6,300.

Sport
Alexis Sanchez has completed a £35m move to Arsenal, the club have confirmed
sportGunners complete £35m signing of Barcelona forward
Voices
Poor teachers should be fearful of not getting pay rises or losing their job if they fail to perform, Steve Fairclough, headteacher of Abbotsholme School, suggested
voicesChris Sloggett explains why it has become an impossible career path
Sport
world cup 2014
Sport
Ray Whelan was arrested earlier this week
PROMOTED VIDEO
Have you tried new the Independent Digital Edition apps?
News
ebookA unique anthology of reporting and analysis of a crucial period of history
Arts and Entertainment
In a minor key: Keira Knightley in the lightweight 'Begin Again'
film
Arts and Entertainment
Celebrated children’s author Allan Ahlberg, best known for Each Peach Pear Plum
books
News
peopleIndian actress known as the 'Grand Old Lady of Bollywood' was 102
News
Wayne’s estate faces a claim for alleged copyright breaches
newsJohn Wayne's heirs duke it out with university over use of the late film star's nickname
Life and Style
It beggars belief: the homeless and hungry are weary, tortured, ghosts of people – with bodies contorted by imperceptible pain
lifeRough sleepers exist in every city. Hear the stories of those whose luck has run out
News
Mick Jagger performing at Glastonbury
people
Life and Style
fashionJ Crew introduces triple zero size to meet the Asia market demand
Sport
Santi Cazorla, Mikel Arteta and Mathieu Flamini of Arsenal launch the new Puma Arsenal kits at the Puma Store on Carnaby Street
sportMassive deal worth £150m over the next five years
Arts and Entertainment
Welsh opera singer Katherine Jenkins
musicHolyrood MPs 'staggered' at lack of Scottish artists performing
Life and Style
beautyBelgian fan lands L'Oreal campaign after being spotted at World Cup
Arts and Entertainment
Currently there is nothing to prevent all-male or all-female couples from competing against mixed sex partners at any of the country’s ballroom dancing events
Potential ban on same-sex partners in ballroom dancing competitions amounts to 'illegal discrimination'
News
business
Independent
Travel Shop
the manor
Up to 70% off luxury travel
on city breaks Find out more
santorini
Up to 70% off luxury travel
on chic beach resorts Find out more
sardina foodie
Up to 70% off luxury travel
on country retreats Find out more
Have you tried new the Independent Digital Edition apps?
Independent Dating
and  

By clicking 'Search' you
are agreeing to our
Terms of Use.

iJobs Job Widget
iJobs Education

Nursery Assistants Needed in Ely

£10000 - £15000 per annum: Randstad Education Cambridge: The JobWe are now rec...

Trainee Recruitment Consultants For Multiple UK Offices

£18000 - £25000 per annum + DOE, OTE £40000: SThree: LONDON - BRISTOL - DUBLIN...

Subject Leader of ICT & Computing

£21588 - £36756 per annum + negotiable: Randstad Education Chelmsford: ICT/Com...

English Teacher - Maternity cover

£110 - £120 per day + Competitive rates of pay: Randstad Education Reading: **...

Day In a Page

A History of the First World War in 100 Moments: Peace without magnanimity - the summit in a railway siding that ended the fighting

A History of the First World War in 100 Moments

Peace without magnanimity - the summit in a railway siding that ended the fighting
Scottish independence: How the Commonwealth Games could swing the vote

Scottish independence: How the Commonwealth Games could swing the vote

In the final part of our series, Chris Green arrives in Glasgow - a host city struggling to keep the politics out of its celebration of sport
Out in the cold: A writer spends a night on the streets and hears the stories of the homeless

A writer spends a night on the streets

Rough sleepers - the homeless, the destitute and the drunk - exist in every city. Will Nicoll meets those whose luck has run out
Striking new stations, high-speed links and (whisper it) better services - the UK's railways are entering a new golden age

UK's railways are entering a new golden age

New stations are opening across the country and our railways appear to be entering an era not seen in Britain since the early 1950s
Conchita Wurst becomes a 'bride' on the Paris catwalk - and proves there is life after Eurovision

Conchita becomes a 'bride' on Paris catwalk

Alexander Fury salutes the Eurovision Song Contest winner's latest triumph
Pétanque World Championship in Marseilles hit by

Pétanque 'world cup' hit by death threats

This year's most acrimonious sporting event took place in France, not Brazil. How did pétanque get so passionate?
Whelks are healthy, versatile and sustainable - so why did we stop eating them in the UK?

Why did we stop eating whelks?

Whelks were the Victorian equivalent of the donor kebab and our stocks are abundant. So why do we now export them all to the Far East?
10 best women's sunglasses

In the shade: 10 best women's sunglasses

From luxury bespoke eyewear to fun festival sunnies, we round up the shades to be seen in this summer
Germany vs Argentina World Cup 2014: Lionel Messi? Javier Mascherano is key for Argentina...

World Cup final: Messi? Mascherano is key for Argentina...

No 10 is always centre of attention but Barça team-mate is just as crucial to finalists’ hopes
Siobhan-Marie O’Connor: Swimmer knows she needs Glasgow joy on road to Rio

Siobhan-Marie O’Connor: Swimmer needs Glasgow joy on road to Rio

18-year-old says this month’s Commonwealth Games are a key staging post in her career before time slips away
The true Gaza back-story that the Israelis aren’t telling this week

The true Gaza back-story that the Israelis aren’t telling this week

A future Palestine state will have no borders and be an enclave within Israel, surrounded on all sides by Israeli-held territory, says Robert Fisk
A History of the First World War in 100 Moments: The German people demand an end to the fighting

A History of the First World War in 100 Moments

The German people demand an end to the fighting
New play by Oscar Wilde's grandson reveals what the Irish wit said at his trials

New play reveals what Oscar Wilde said at trials

For a century, what Wilde actually said at his trials was a mystery. But the recent discovery of shorthand notes changed that. Now his grandson Merlin Holland has turned them into a play
Can scientists save the world's sea life from

Can scientists save our sea life?

By the end of the century, the only living things left in our oceans could be plankton and jellyfish. Alex Renton meets the scientists who are trying to turn the tide
Richard III, Trafalgar Studios, review: Martin Freeman gives highly intelligent performance

Richard III review

Martin Freeman’s psychotic monarch is big on mockery but wanting in malice