Utterly wrong! Flawed! Academics deride league tables that guide Michael Gove's reforms

Model used to calculate triennial rankings contains 'profound' error, says mathematician

The world's most influential education league tables used by Michael Gove as a reason for "whole system reform" of the sector have been derided as "useless and meaningless" by two academic studies.

The Education Secretary regularly points to Britain's position in the "landmark" Programme for International Student Assessment (Pisa) rankings of 15-year-olds' science, maths and reading ability to justify his policies. In a speech last year to the Education World Forum Mr Gove even described the German mathematician who runs Pisa, Andreas Schleicher, as "the most important man in the British education system".

Dr Hugh Morrison, a mathematician from Queens University, Belfast, said that the model used to calculate the triennial rankings is "utterly wrong" because it contains a "profound" conceptual error that confuses objective and subjective probability. "Pisa will never work", he added.

Professor Svend Kreiner, a statistician from the University of Copenhagen in Denmark, also said the Pisa model is fundamentally flawed. In a paper published this summer, he challenges Pisa's reliability and shows how results fluctuate significantly according to which test questions are used. He also reveals how, in the 2006 reading rankings, Canada could have been positioned anywhere between second and 25th, Japan between eighth and 40th and the UK between 14th and 30th.

Mr Gove has said that Britain between 2000 and 2009 "plummeted in the world rankings from 4th to 16th for science, 7th to 25th for literacy and 8th to 28th for maths". Pisa's 2012 results are published later this year.

The questions used for Pisa vary between countries and between students participating in the same assessment. In the 2006 tests half the students were not asked any reading questions but were allocated "plausible" reading scores to help calculate their countries' rankings. Responding to the news of the studies in today's TES, one teacher said: "The first year our kids did the PISA test, the science content was aligned to aspects of 5-14 that we never taught. [One asked]: 'Sir, why did they ask about dinosaurs and geology when we don't do it?'"

Professor Kreiner says the Rasch model used by Pisa can only work if the questions are of the same level of difficulty for each country taking the test. He said his research proves that this is not the case and therefore the comparisons that Pisa makes between countries are "useless".

Professor Kreiner first raised the issue two years ago, but the Organisation for Economic Development and Co-Operation (OECD), which runs Pisa, countered by suggesting that he had been able to find such wild fluctuations in rankings only by deliberately selecting particular small groupings of questions to prove his point.

Professor Kreiner's latest paper uses the same groups of questions as Pisa and comes up with very similar results to his initial analysis. "It is meaningless to try to compare reading in Chinese with reading in Danish," he said today.

Dr Morrison said that the Rasch model made the "impossible" claim of being able to measure ability independently of the questions that students answer. "I am certain this (problem) cannot be answered," he told TES.

A spokesperson for the OECD claimed that country rankings take account of "the uncertainty that results from sample data" adding: "No model exactly fits the data. Pisa will always seek to improve and learn from the latest research methods."

Christine Blower, General Secretary of the National Union of Teachers, said: "Michael Gove's habitual simplification and half-truths about Pisa have for many years formed the central plank of his justifications for widespread reform. These new calls from academics should encourage all to take the Secretary of State's every utterance on standards with a large pinch of salt."

A Department for Education spokesperson said: “While there will always be academic debate over methodology, international benchmarking is vitally important to ensure we are competing with the world’s best-performing education systems.”

Flawed evidence: Gove’s gaffes

Michael Gove has often been accused of citing flawed or non-existent research.

A Freedom of Information request revealed that PR surveys by Premier Inn and UKTV Gold were the source of his evidence when he said British teenagers were ignorant about key historical events.

In April, he told a conference that in the Far East “school days are longer, school holidays are shorter”. In fact, English pupils spend around 7,250 hours in the classroom, far more than South Korea, at 6,000, and Japan, at 6,300.

Start your day with The Independent, sign up for daily news emails
PROMOTED VIDEO
Have you tried new the Independent Digital Edition apps?
ebooks
ebooksA year of political gossip, levity and intrigue from the sharpest pen in Westminster
Life and Style
A still from the 1939 film version of Margaret Mitchell's 'Gone with the Wind'
life
Sport
David Silva strokes home his and City's second goal
football
Arts and Entertainment
a clockwork orange, stanley kubrick
film
Extras
The Tesco Hudl2: An exceptional Android tablet that's powerful, well-built and outstanding value
indybest

News
Sarah Silverman (middle) with sister Reform Rabbi Susan Silverman (right) and sister actress Laura Silverman (left) at Jerusalem's Western Wall for feminist Hanuka candle-lighting ceremony
peopleControversial comedian stages pro-equality Hanukkah lighting during a protest at Jerusalem's Wailing Wall
Sport
After another poor series in Sri Lanka, Alastair Cook claimed all players go through a lean period
cricket
Arts and Entertainment
The Bach Choir has been crowned the inaugural winner of Sky Arts’ show The Great Culture Quiz
arts + ents140-year-old choir declared winner of Sky Arts' 'The Great Culture Quiz'
Life and Style
food + drinkAuthor DBC Pierre presents his guide to the morning after
Life and Style
food + drink
Latest stories from i100
Have you tried new the Independent Digital Edition apps?
Independent Dating
and  

By clicking 'Search' you
are agreeing to our
Terms of Use.

iJobs Job Widget
iJobs Education

Opilio Recruitment: Product Owner

£40k - 45k per year + Benefits: Opilio Recruitment: We are currently recruit...

Opilio Recruitment: Product Development Manager

£40k - 45k per year + Benefits: Opilio Recruitment: We are currently recruit...

Recruitment Genius: Qualified Nursery Practitioner - Sevenoaks

£15000 - £16000 per annum: Recruitment Genius: We currently have an opportunit...

Recruitment Genius: Room Leader - Nursery

£17000 - £18000 per annum: Recruitment Genius: We currently have an opportunit...

Day In a Page

Homeless Veterans appeal: 'You look for someone who's an inspiration and try to be like them'

Homeless Veterans appeal

In 2010, Sgt Gary Jamieson stepped on an IED in Afghanistan and lost his legs and an arm. He reveals what, and who, helped him to make a remarkable recovery
Could cannabis oil reverse the effects of cancer?

Could cannabis oil reverse effects of cancer?

As a film following six patients receiving the controversial treatment is released, Kate Hilpern uncovers a very slippery issue
The Interview movie review: You can't see Seth Rogen and James Franco's Kim Jong Un assassination film, but you can read about it here

The Interview movie review

You can't see Seth Rogen and James Franco's Kim Jong Un assassination film, but you can read about it here
Serial mania has propelled podcasts into the cultural mainstream

How podcasts became mainstream

People have consumed gripping armchair investigation Serial with a relish typically reserved for box-set binges
Jesus Christ has become an unlikely pin-up for hipster marketing companies

Jesus Christ has become an unlikely pin-up

Kevin Lee Light, aka "Jesus", is the newest client of creative agency Mother while rival agency Anomaly has launched Sexy Jesus, depicting the Messiah in a series of Athena-style poses
Rosetta space mission voted most important scientific breakthrough of 2014

A memorable year for science – if not for mice

The most important scientific breakthroughs of 2014
Christmas cocktails to make you merry: From eggnog to Brown Betty and Rum Bumpo

Christmas cocktails to make you merry

Mulled wine is an essential seasonal treat. But now drinkers are rediscovering other traditional festive tipples. Angela Clutton raises a glass to Christmas cocktails
5 best activity trackers

Fitness technology: 5 best activity trackers

Up the ante in your regimen and change the habits of a lifetime with this wearable tech
Paul Scholes column: It's a little-known fact, but I have played one of the seven dwarves

Paul Scholes column

It's a little-known fact, but I have played one of the seven dwarves
Fifa's travelling circus once again steals limelight from real stars

Fifa's travelling circus once again steals limelight from real stars

Club World Cup kicked into the long grass by the continued farce surrounding Blatter, Garcia, Russia and Qatar
Frank Warren column: 2014 – boxing is back and winning new fans

Frank Warren: Boxing is back and winning new fans

2014 proves it's now one of sport's biggest hitters again
Jeb Bush vs Hillary Clinton: The power dynamics of the two first families

Jeb Bush vs Hillary Clinton

Karen Tumulty explores the power dynamics of the two first families
Stockholm is rivalling Silicon Valley with a hotbed of technology start-ups

Stockholm is rivalling Silicon Valley

The Swedish capital is home to two of the most popular video games in the world, as well as thousands of technology start-ups worth hundreds of millions of pounds – and it's all happened since 2009
Did Japanese workers really get their symbols mixed up and display Santa on a crucifix?

Crucified Santa: Urban myth refuses to die

The story goes that Japanese store workers created a life-size effigy of a smiling "Father Kurisumasu" attached to a facsimile of Our Lord's final instrument of torture
Jennifer Saunders and Kate Moss join David Walliams on set for TV adaptation of The Boy in the Dress

The Boy in the Dress: On set with the stars

Walliams' story about a boy who goes to school in a dress will be shown this Christmas