Utterly wrong! Flawed! Academics deride league tables that guide Michael Gove's reforms

Model used to calculate triennial rankings contains 'profound' error, says mathematician

The world's most influential education league tables used by Michael Gove as a reason for "whole system reform" of the sector have been derided as "useless and meaningless" by two academic studies.

The Education Secretary regularly points to Britain's position in the "landmark" Programme for International Student Assessment (Pisa) rankings of 15-year-olds' science, maths and reading ability to justify his policies. In a speech last year to the Education World Forum Mr Gove even described the German mathematician who runs Pisa, Andreas Schleicher, as "the most important man in the British education system".

Dr Hugh Morrison, a mathematician from Queens University, Belfast, said that the model used to calculate the triennial rankings is "utterly wrong" because it contains a "profound" conceptual error that confuses objective and subjective probability. "Pisa will never work", he added.

Professor Svend Kreiner, a statistician from the University of Copenhagen in Denmark, also said the Pisa model is fundamentally flawed. In a paper published this summer, he challenges Pisa's reliability and shows how results fluctuate significantly according to which test questions are used. He also reveals how, in the 2006 reading rankings, Canada could have been positioned anywhere between second and 25th, Japan between eighth and 40th and the UK between 14th and 30th.

Mr Gove has said that Britain between 2000 and 2009 "plummeted in the world rankings from 4th to 16th for science, 7th to 25th for literacy and 8th to 28th for maths". Pisa's 2012 results are published later this year.

The questions used for Pisa vary between countries and between students participating in the same assessment. In the 2006 tests half the students were not asked any reading questions but were allocated "plausible" reading scores to help calculate their countries' rankings. Responding to the news of the studies in today's TES, one teacher said: "The first year our kids did the PISA test, the science content was aligned to aspects of 5-14 that we never taught. [One asked]: 'Sir, why did they ask about dinosaurs and geology when we don't do it?'"

Professor Kreiner says the Rasch model used by Pisa can only work if the questions are of the same level of difficulty for each country taking the test. He said his research proves that this is not the case and therefore the comparisons that Pisa makes between countries are "useless".

Professor Kreiner first raised the issue two years ago, but the Organisation for Economic Development and Co-Operation (OECD), which runs Pisa, countered by suggesting that he had been able to find such wild fluctuations in rankings only by deliberately selecting particular small groupings of questions to prove his point.

Professor Kreiner's latest paper uses the same groups of questions as Pisa and comes up with very similar results to his initial analysis. "It is meaningless to try to compare reading in Chinese with reading in Danish," he said today.

Dr Morrison said that the Rasch model made the "impossible" claim of being able to measure ability independently of the questions that students answer. "I am certain this (problem) cannot be answered," he told TES.

A spokesperson for the OECD claimed that country rankings take account of "the uncertainty that results from sample data" adding: "No model exactly fits the data. Pisa will always seek to improve and learn from the latest research methods."

Christine Blower, General Secretary of the National Union of Teachers, said: "Michael Gove's habitual simplification and half-truths about Pisa have for many years formed the central plank of his justifications for widespread reform. These new calls from academics should encourage all to take the Secretary of State's every utterance on standards with a large pinch of salt."

A Department for Education spokesperson said: “While there will always be academic debate over methodology, international benchmarking is vitally important to ensure we are competing with the world’s best-performing education systems.”

Flawed evidence: Gove’s gaffes

Michael Gove has often been accused of citing flawed or non-existent research.

A Freedom of Information request revealed that PR surveys by Premier Inn and UKTV Gold were the source of his evidence when he said British teenagers were ignorant about key historical events.

In April, he told a conference that in the Far East “school days are longer, school holidays are shorter”. In fact, English pupils spend around 7,250 hours in the classroom, far more than South Korea, at 6,000, and Japan, at 6,300.

Voices
The Sumatran tiger, endemic to the Indonesian island of Sumatra, is an endangered species
voicesJonathon Porritt: The wild tiger population is thought to have dropped by 97 per cent since 1900
Arts and Entertainment
Beast would strip to his underpants and take to the stage with a slogan scrawled on his bare chest whilst fans shouted “you fat bastard” at him
musicIndie music promoter was was a feature at Carter gigs
News
news
Arts and Entertainment
Story line: Susanoo slays the Yamata no Orochi serpent in the Japanese version of a myth dating back 40,000 years
arts + entsApplying the theory of evolution to the world's many mythologies
PROMOTED VIDEO
Have you tried new the Independent Digital Edition apps?
News
ebookA unique anthology of reporting and analysis of a crucial period of history
News
Performers dressed as Tunnocks chocolate teacakes, a renowned Scottish confectionary, perform during the opening ceremony of the 2014 Commonwealth Games at Celtic Park in Glasgow on July 23, 2014.
news
Life and Style
Popular plonk: Lambrusco is selling strong
Food + drinkNaff Seventies corner-shop staple is this year's Aperol Spritz
News
Gardai wait for the naked man, who had gone for a skinny dip in Belfast Lough
newsTwo skinny dippers threatened with inclusion on sex offenders’ register as naturists criminalised
News
Shake down: Michelle and Barack Obama bump knuckles before an election night rally in Minnesota in 2008, the 'Washington Post' called it 'the fist bump heard round the world'
newsThe pound, a.k.a. the dap, greatly improves hygiene
Arts and Entertainment
La Roux
music
Arts and Entertainment
Graham Fellows as John Shuttleworth
comedySean O'Grady joins Graham Fellows down his local Spar
News
people
News
Ross Burden pictured in 2002
people
News
Elisabeth Murdoch: The 44-year-old said she felt a responsibility to 'stand up and be counted’'
media... says Rupert Murdoch
Arts and Entertainment
tv
Extras
indybest
Sport
Arsenal signing Calum Chambers
sportGunners complete £16m transfer of Southampton youngster
Independent
Travel Shop
the manor
Up to 70% off luxury travel
on city breaks Find out more
santorini
Up to 70% off luxury travel
on chic beach resorts Find out more
sardina foodie
Up to 70% off luxury travel
on country retreats Find out more
Have you tried new the Independent Digital Edition apps?
Independent Dating
and  

By clicking 'Search' you
are agreeing to our
Terms of Use.

iJobs Job Widget
iJobs Education

Horticulture Lecturer / Tutor / Assessor - Derbyshire

£15 - £18 per hour: Randstad Education Nottingham: As a result of our successf...

Retail Lecturer / Assessor / Tutor - Derbyshire

£15 - £18 per hour: Randstad Education Nottingham: Randstad Education are succ...

Business Studies Tutor / Assessor / Lecturer - Tollerton

£15 - £18 per hour: Randstad Education Nottingham: Randstad Education are succ...

Year 1 Teacher

£12 - £90 per day: Randstad Education Hull: Randstad Education require a year ...

Day In a Page

The children were playing in the street with toy guns. The air strikes were tragically real

The air strikes were tragically real

The children were playing in the street with toy guns
Boozy, ignorant, intolerant, but very polite – The British, as others see us

Britain as others see us

Boozy, ignorant, intolerant, but very polite
Countries that don’t survey their tigers risk losing them altogether

Countries that don’t survey their tigers risk losing them

Jonathon Porritt sounds the alarm
How did our legends really begin?

How did our legends really begin?

Applying the theory of evolution to the world's many mythologies
Watch out: Lambrusco is back on the menu

Lambrusco is back on the menu

Naff Seventies corner-shop staple is this year's Aperol Spritz
A new Russian revolution: Cracks start to appear in Putin’s Kremlin power bloc

A new Russian revolution

Cracks start to appear in Putin’s Kremlin power bloc
Eugene de Kock: Apartheid’s sadistic killer that his country cannot forgive

Apartheid’s sadistic killer that his country cannot forgive

The debate rages in South Africa over whether Eugene de Kock should ever be released from jail
Standing my ground: If sitting is bad for your health, what happens when you stay on your feet for a whole month?

Standing my ground

If sitting is bad for your health, what happens when you stay on your feet for a whole month?
Commonwealth Games 2014: Dai Greene prays for chance to rebuild after injury agony

Greene prays for chance to rebuild after injury agony

Welsh hurdler was World, European and Commonwealth champion, but then the injuries crept in
Israel-Gaza conflict: Secret report helps Israelis to hide facts

Patrick Cockburn: Secret report helps Israel to hide facts

The slickness of Israel's spokesmen is rooted in directions set down by pollster Frank Luntz
The man who dared to go on holiday

The man who dared to go on holiday

New York's mayor has taken a vacation - in a nation that has still to enforce paid leave, it caused quite a stir, reports Rupert Cornwell
Best comedians: How the professionals go about their funny business, from Sarah Millican to Marcus Brigstocke

Best comedians: How the professionals go about their funny business

For all those wanting to know how stand-ups keep standing, here are some of the best moments
The Guest List 2014: Forget the Man Booker longlist, Literary Editor Katy Guest offers her alternative picks

The Guest List 2014

Forget the Man Booker longlist, Literary Editor Katy Guest offers her alternative picks
Jokes on Hollywood: 'With comedy film audiences shrinking, it’s time to move on'

Jokes on Hollywood

With comedy film audiences shrinking, it’s time to move on