Utterly wrong! Flawed! Academics deride league tables that guide Michael Gove's reforms

Model used to calculate triennial rankings contains 'profound' error, says mathematician

The world's most influential education league tables used by Michael Gove as a reason for "whole system reform" of the sector have been derided as "useless and meaningless" by two academic studies.

The Education Secretary regularly points to Britain's position in the "landmark" Programme for International Student Assessment (Pisa) rankings of 15-year-olds' science, maths and reading ability to justify his policies. In a speech last year to the Education World Forum Mr Gove even described the German mathematician who runs Pisa, Andreas Schleicher, as "the most important man in the British education system".

Dr Hugh Morrison, a mathematician from Queens University, Belfast, said that the model used to calculate the triennial rankings is "utterly wrong" because it contains a "profound" conceptual error that confuses objective and subjective probability. "Pisa will never work", he added.

Professor Svend Kreiner, a statistician from the University of Copenhagen in Denmark, also said the Pisa model is fundamentally flawed. In a paper published this summer, he challenges Pisa's reliability and shows how results fluctuate significantly according to which test questions are used. He also reveals how, in the 2006 reading rankings, Canada could have been positioned anywhere between second and 25th, Japan between eighth and 40th and the UK between 14th and 30th.

Mr Gove has said that Britain between 2000 and 2009 "plummeted in the world rankings from 4th to 16th for science, 7th to 25th for literacy and 8th to 28th for maths". Pisa's 2012 results are published later this year.

The questions used for Pisa vary between countries and between students participating in the same assessment. In the 2006 tests half the students were not asked any reading questions but were allocated "plausible" reading scores to help calculate their countries' rankings. Responding to the news of the studies in today's TES, one teacher said: "The first year our kids did the PISA test, the science content was aligned to aspects of 5-14 that we never taught. [One asked]: 'Sir, why did they ask about dinosaurs and geology when we don't do it?'"

Professor Kreiner says the Rasch model used by Pisa can only work if the questions are of the same level of difficulty for each country taking the test. He said his research proves that this is not the case and therefore the comparisons that Pisa makes between countries are "useless".

Professor Kreiner first raised the issue two years ago, but the Organisation for Economic Development and Co-Operation (OECD), which runs Pisa, countered by suggesting that he had been able to find such wild fluctuations in rankings only by deliberately selecting particular small groupings of questions to prove his point.

Professor Kreiner's latest paper uses the same groups of questions as Pisa and comes up with very similar results to his initial analysis. "It is meaningless to try to compare reading in Chinese with reading in Danish," he said today.

Dr Morrison said that the Rasch model made the "impossible" claim of being able to measure ability independently of the questions that students answer. "I am certain this (problem) cannot be answered," he told TES.

A spokesperson for the OECD claimed that country rankings take account of "the uncertainty that results from sample data" adding: "No model exactly fits the data. Pisa will always seek to improve and learn from the latest research methods."

Christine Blower, General Secretary of the National Union of Teachers, said: "Michael Gove's habitual simplification and half-truths about Pisa have for many years formed the central plank of his justifications for widespread reform. These new calls from academics should encourage all to take the Secretary of State's every utterance on standards with a large pinch of salt."

A Department for Education spokesperson said: “While there will always be academic debate over methodology, international benchmarking is vitally important to ensure we are competing with the world’s best-performing education systems.”

Flawed evidence: Gove’s gaffes

Michael Gove has often been accused of citing flawed or non-existent research.

A Freedom of Information request revealed that PR surveys by Premier Inn and UKTV Gold were the source of his evidence when he said British teenagers were ignorant about key historical events.

In April, he told a conference that in the Far East “school days are longer, school holidays are shorter”. In fact, English pupils spend around 7,250 hours in the classroom, far more than South Korea, at 6,000, and Japan, at 6,300.

Start your day with The Independent, sign up for daily news emails
PROMOTED VIDEO
Have you tried new the Independent Digital Edition apps?
ebooks
ebooksA year of political gossip, levity and intrigue from the sharpest pen in Westminster
Arts and Entertainment
Kara Tointon and Jeremy Piven star in Mr Selfridge
tvActress Kara Tointon on what to expect from Series 3
Voices
Winston Churchill, then prime minister, outside No 10 in June 1943
voicesA C Benson called him 'a horrid little fellow', George Orwell would have shot him, but what a giant he seems now, says DJ Taylor
News
i100
Sport
footballBrighton vs Arsenal match report
Arts and Entertainment
Benedict Cumberbatch has spoken about the lack of opportunities for black British actors in the UK
film
News
Latest stories from i100
Have you tried new the Independent Digital Edition apps?
Independent Dating
and  

By clicking 'Search' you
are agreeing to our
Terms of Use.

iJobs Job Widget
iJobs Education

Old Royal Naval College: ORNC Visitor Experience Volunteer

Unpaid voluntary work: Old Royal Naval College: Join our team of friendly volu...

Recruitment Genius: Nursery Practitioner - Faringdon

Negotiable: Recruitment Genius: We currently have an opportunity for you to jo...

Ashdown Group: Junior Developer - Cirencester - £29,000

£25000 - £29000 per annum + Benefits: Ashdown Group: The Ashdown Group have be...

Recruitment Genius: Primary School Sports Coach

£16000 - £20000 per annum: Recruitment Genius: Calling all talented Level 2 qu...

Day In a Page

Iraq invasion 2003: The bloody warnings six wise men gave to Tony Blair as he prepared to launch poorly planned campaign

What the six wise men told Tony Blair

Months before the invasion of Iraq in 2003, experts sought to warn the PM about his plans. Here, four of them recall that day
25 years of The Independent on Sunday: The stories, the writers and the changes over the last quarter of a century

25 years of The Independent on Sunday

The stories, the writers and the changes over the last quarter of a century
Homeless Veterans appeal: 'Really caring is a dangerous emotion in this kind of work'

Homeless Veterans appeal

As head of The Soldiers' Charity, Martin Rutledge has to temper compassion with realism. He tells Chris Green how his Army career prepared him
Wu-Tang Clan and The Sexual Objects offer fans a chance to own the only copies of their latest albums

Smash hit go under the hammer

It's nice to pick up a new record once in a while, but the purchasers of two latest releases can go a step further - by buying the only copy
Geeks who rocked the world: Documentary looks back at origins of the computer-games industry

The geeks who rocked the world

A new documentary looks back at origins of the computer-games industry
Belle & Sebastian interview: Stuart Murdoch reveals how the band is taking a new direction

Belle & Sebastian is taking a new direction

Twenty years ago, Belle & Sebastian was a fey indie band from Glasgow. It still is – except today, as prime mover Stuart Murdoch admits, it has a global cult following, from Hollywood to South Korea
America: Land of the free, home of the political dynasty

America: Land of the free, home of the political dynasty

These days in the US things are pretty much stuck where they are, both in politics and society at large, says Rupert Cornwell
A graphic history of US civil rights – in comic book form

A graphic history of US civil rights – in comic book form

A veteran of the Fifties campaigns is inspiring a new generation of activists
Winston Churchill: the enigma of a British hero

Winston Churchill: the enigma of a British hero

A C Benson called him 'a horrid little fellow', George Orwell would have shot him, but what a giant he seems now, says DJ Taylor
Growing mussels: Precious freshwater shellfish are thriving in a unique green project

Growing mussels

Precious freshwater shellfish are thriving in a unique green project
Diana Krall: The jazz singer on being friends with Elton John, outer space and skiing in Dubai

Diana Krall interview

The jazz singer on being friends with Elton John, outer space and skiing in Dubai
Pinstriped for action: A glimpse of what the very rich man will be wearing this winter

Pinstriped for action

A glimpse of what the very rich man will be wearing this winter
Russell T Davies & Ben Cook: 'Our friendship flourished online. You can share some very revelatory moments at four in the morning…'

Russell T Davies & Ben Cook: How we met

'Our friendship flourished online. You can share some very revelatory moments at four in the morning…'
Bill Granger recipes: Our chef serves up his favourite Japanese dishes

Bill Granger's Japanese recipes

Stock up on mirin, soy and miso and you have the makings of everyday Japanese cuisine
Michael Calvin: How we need more Eric Cantonas to knock some sense into us

Michael Calvin's Last Word

How we need more Eric Cantonas to knock some sense into us