Utterly wrong! Flawed! Academics deride league tables that guide Michael Gove's reforms

Model used to calculate triennial rankings contains 'profound' error, says mathematician

The world's most influential education league tables used by Michael Gove as a reason for "whole system reform" of the sector have been derided as "useless and meaningless" by two academic studies.

The Education Secretary regularly points to Britain's position in the "landmark" Programme for International Student Assessment (Pisa) rankings of 15-year-olds' science, maths and reading ability to justify his policies. In a speech last year to the Education World Forum Mr Gove even described the German mathematician who runs Pisa, Andreas Schleicher, as "the most important man in the British education system".

Dr Hugh Morrison, a mathematician from Queens University, Belfast, said that the model used to calculate the triennial rankings is "utterly wrong" because it contains a "profound" conceptual error that confuses objective and subjective probability. "Pisa will never work", he added.

Professor Svend Kreiner, a statistician from the University of Copenhagen in Denmark, also said the Pisa model is fundamentally flawed. In a paper published this summer, he challenges Pisa's reliability and shows how results fluctuate significantly according to which test questions are used. He also reveals how, in the 2006 reading rankings, Canada could have been positioned anywhere between second and 25th, Japan between eighth and 40th and the UK between 14th and 30th.

Mr Gove has said that Britain between 2000 and 2009 "plummeted in the world rankings from 4th to 16th for science, 7th to 25th for literacy and 8th to 28th for maths". Pisa's 2012 results are published later this year.

The questions used for Pisa vary between countries and between students participating in the same assessment. In the 2006 tests half the students were not asked any reading questions but were allocated "plausible" reading scores to help calculate their countries' rankings. Responding to the news of the studies in today's TES, one teacher said: "The first year our kids did the PISA test, the science content was aligned to aspects of 5-14 that we never taught. [One asked]: 'Sir, why did they ask about dinosaurs and geology when we don't do it?'"

Professor Kreiner says the Rasch model used by Pisa can only work if the questions are of the same level of difficulty for each country taking the test. He said his research proves that this is not the case and therefore the comparisons that Pisa makes between countries are "useless".

Professor Kreiner first raised the issue two years ago, but the Organisation for Economic Development and Co-Operation (OECD), which runs Pisa, countered by suggesting that he had been able to find such wild fluctuations in rankings only by deliberately selecting particular small groupings of questions to prove his point.

Professor Kreiner's latest paper uses the same groups of questions as Pisa and comes up with very similar results to his initial analysis. "It is meaningless to try to compare reading in Chinese with reading in Danish," he said today.

Dr Morrison said that the Rasch model made the "impossible" claim of being able to measure ability independently of the questions that students answer. "I am certain this (problem) cannot be answered," he told TES.

A spokesperson for the OECD claimed that country rankings take account of "the uncertainty that results from sample data" adding: "No model exactly fits the data. Pisa will always seek to improve and learn from the latest research methods."

Christine Blower, General Secretary of the National Union of Teachers, said: "Michael Gove's habitual simplification and half-truths about Pisa have for many years formed the central plank of his justifications for widespread reform. These new calls from academics should encourage all to take the Secretary of State's every utterance on standards with a large pinch of salt."

A Department for Education spokesperson said: “While there will always be academic debate over methodology, international benchmarking is vitally important to ensure we are competing with the world’s best-performing education systems.”

Flawed evidence: Gove’s gaffes

Michael Gove has often been accused of citing flawed or non-existent research.

A Freedom of Information request revealed that PR surveys by Premier Inn and UKTV Gold were the source of his evidence when he said British teenagers were ignorant about key historical events.

In April, he told a conference that in the Far East “school days are longer, school holidays are shorter”. In fact, English pupils spend around 7,250 hours in the classroom, far more than South Korea, at 6,000, and Japan, at 6,300.

Start your day with The Independent, sign up for daily news emails
Have you tried new the Independent Digital Edition apps?
ebooks
ebooksAn introduction to the ground rules of British democracy
Latest stories from i100
Have you tried new the Independent Digital Edition apps?
SPONSORED FEATURES
Independent Dating
and  

By clicking 'Search' you
are agreeing to our
Terms of Use.

iJobs Job Widget
iJobs Education

Recruitment Genius: Nursery Nurse and Room Leader - Hackney

£15000 - £20000 per annum: Recruitment Genius: Are you a qualified childcare p...

AER Teachers: PPA TEACHER/MENTOR

£27000 - £37000 per annum: AER Teachers: THE SCHOOL: This is a large and vibra...

AER Teachers: EYFS Teacher

£27000 - £37000 per annum: AER Teachers: EYFS TEACHERAn 'Outstanding' Primary ...

AER Teachers: YEAR 3 TEACHER - PREPARATORY SCHOOL

£27000 - £40000 per annum: AER Teachers: YEAR 3 TEACHER - PREPARATORY SCHOOLA ...

Day In a Page

Refugee crisis: David Cameron lowered the flag for the dead king of Saudi Arabia - will he do the same honour for little Aylan Kurdi?

Cameron lowered the flag for the dead king of Saudi Arabia...

But will he do the same honour for little Aylan Kurdi, asks Robert Fisk
Our leaders lack courage in this refugee crisis. We are shamed by our European neighbours

Our leaders lack courage in this refugee crisis. We are shamed by our European neighbours

Humanity must be at the heart of politics, says Jeremy Corbyn
Joe Biden's 'tease tour': Could the US Vice-President be testing the water for a presidential run?

Joe Biden's 'tease tour'

Could the US Vice-President be testing the water for a presidential run?
Britain's 24-hour culture: With the 'leisured society' a distant dream we're working longer and less regular hours than ever

Britain's 24-hour culture

With the 'leisured society' a distant dream we're working longer and less regular hours than ever
Diplomacy board game: Treachery is the way to win - which makes it just like the real thing

The addictive nature of Diplomacy

Bullying, betrayal, aggression – it may be just a board game, but the family that plays Diplomacy may never look at each other in the same way again
Lady Chatterley's Lover: Racy underwear for fans of DH Lawrence's equally racy tome

Fashion: Ooh, Lady Chatterley!

Take inspiration from DH Lawrence's racy tome with equally racy underwear
8 best children's clocks

Tick-tock: 8 best children's clocks

Whether you’re teaching them to tell the time or putting the finishing touches to a nursery, there’s a ticker for that
Charlie Austin: Queens Park Rangers striker says ‘If the move is not right, I’m not going’

Charlie Austin: ‘If the move is not right, I’m not going’

After hitting 18 goals in the Premier League last season, the QPR striker was the great non-deal of transfer deadline day. But he says he'd preferred another shot at promotion
Isis profits from destruction of antiquities by selling relics to dealers - and then blowing up the buildings they come from to conceal the evidence of looting

How Isis profits from destruction of antiquities

Robert Fisk on the terrorist group's manipulation of the market to increase the price of artefacts
Labour leadership: Andy Burnham urges Jeremy Corbyn voters to think again in last-minute plea

'If we lose touch we’ll end up with two decades of the Tories'

In an exclusive interview, Andy Burnham urges Jeremy Corbyn voters to think again in last-minute plea
Tunisia fears its Arab Spring could be reversed as the new regime becomes as intolerant of dissent as its predecessor

The Arab Spring reversed

Tunisian protesters fear that a new law will whitewash corrupt businessmen and officials, but they are finding that the new regime is becoming as intolerant of dissent as its predecessor
King Arthur: Legendary figure was real and lived most of his life in Strathclyde, academic claims

Academic claims King Arthur was real - and reveals where he lived

Dr Andrew Breeze says the legendary figure did exist – but was a general, not a king
Who is Oliver Bonas and how has he captured middle-class hearts?

Who is Oliver Bonas?

It's the first high-street store to pay its staff the living wage, and it saw out the recession in style
Earth has 'lost more than half its trees' since humans first started cutting them down

Axe-wielding Man fells half the world’s trees – leaving us just 422 each

However, the number of trees may be eight times higher than previously thought
60 years of Scalextric: Model cars are now stuffed with as much tech as real ones

60 years of Scalextric

Model cars are now stuffed with as much tech as real ones