Utterly wrong! Flawed! Academics deride league tables that guide Michael Gove's reforms
Model used to calculate triennial rankings contains 'profound' error, says mathematician
Paul Gallagher is a reporter for the Independent and Independent on Sunday having joined the group in 2012. He has previously worked for the European Voice, Daily Mirror and the Observer and been based in Brussels, Belfast, Tokyo and London.
Friday 19 July 2013
The world's most influential education league tables used by Michael Gove as a reason for "whole system reform" of the sector have been derided as "useless and meaningless" by two academic studies.
The Education Secretary regularly points to Britain's position in the "landmark" Programme for International Student Assessment (Pisa) rankings of 15-year-olds' science, maths and reading ability to justify his policies. In a speech last year to the Education World Forum Mr Gove even described the German mathematician who runs Pisa, Andreas Schleicher, as "the most important man in the British education system".
Dr Hugh Morrison, a mathematician from Queens University, Belfast, said that the model used to calculate the triennial rankings is "utterly wrong" because it contains a "profound" conceptual error that confuses objective and subjective probability. "Pisa will never work", he added.
Professor Svend Kreiner, a statistician from the University of Copenhagen in Denmark, also said the Pisa model is fundamentally flawed. In a paper published this summer, he challenges Pisa's reliability and shows how results fluctuate significantly according to which test questions are used. He also reveals how, in the 2006 reading rankings, Canada could have been positioned anywhere between second and 25th, Japan between eighth and 40th and the UK between 14th and 30th.
Mr Gove has said that Britain between 2000 and 2009 "plummeted in the world rankings from 4th to 16th for science, 7th to 25th for literacy and 8th to 28th for maths". Pisa's 2012 results are published later this year.
The questions used for Pisa vary between countries and between students participating in the same assessment. In the 2006 tests half the students were not asked any reading questions but were allocated "plausible" reading scores to help calculate their countries' rankings. Responding to the news of the studies in today's TES, one teacher said: "The first year our kids did the PISA test, the science content was aligned to aspects of 5-14 that we never taught. [One asked]: 'Sir, why did they ask about dinosaurs and geology when we don't do it?'"
Professor Kreiner says the Rasch model used by Pisa can only work if the questions are of the same level of difficulty for each country taking the test. He said his research proves that this is not the case and therefore the comparisons that Pisa makes between countries are "useless".
Professor Kreiner first raised the issue two years ago, but the Organisation for Economic Development and Co-Operation (OECD), which runs Pisa, countered by suggesting that he had been able to find such wild fluctuations in rankings only by deliberately selecting particular small groupings of questions to prove his point.
Professor Kreiner's latest paper uses the same groups of questions as Pisa and comes up with very similar results to his initial analysis. "It is meaningless to try to compare reading in Chinese with reading in Danish," he said today.
Dr Morrison said that the Rasch model made the "impossible" claim of being able to measure ability independently of the questions that students answer. "I am certain this (problem) cannot be answered," he told TES.
A spokesperson for the OECD claimed that country rankings take account of "the uncertainty that results from sample data" adding: "No model exactly fits the data. Pisa will always seek to improve and learn from the latest research methods."
Christine Blower, General Secretary of the National Union of Teachers, said: "Michael Gove's habitual simplification and half-truths about Pisa have for many years formed the central plank of his justifications for widespread reform. These new calls from academics should encourage all to take the Secretary of State's every utterance on standards with a large pinch of salt."
A Department for Education spokesperson said: “While there will always be academic debate over methodology, international benchmarking is vitally important to ensure we are competing with the world’s best-performing education systems.”
Flawed evidence: Gove’s gaffes
Michael Gove has often been accused of citing flawed or non-existent research.
A Freedom of Information request revealed that PR surveys by Premier Inn and UKTV Gold were the source of his evidence when he said British teenagers were ignorant about key historical events.
In April, he told a conference that in the Far East “school days are longer, school holidays are shorter”. In fact, English pupils spend around 7,250 hours in the classroom, far more than South Korea, at 6,000, and Japan, at 6,300.
Game of Thrones author George RR Martin says 'f*** you' to fans who fear he will die before finishing Westeros saga
Instagram of US airport security chiefs: Lipstick knives and IED training kits among items seized
‘Ryan Gosling got someone pregnant and it's not me. Brazil you think you’re devastated…’
Israel-Palestine crisis: ‘We just want it to end… We don’t deserve to live like this’
Israel-Palestine crisis: Eight killed in Gaza Strip cafe while watching World Cup semi-final
Sustained immigration has not harmed Britons' employment, say government advisers
Australia facing international condemnation after turning around Sri Lankans at sea
7/7 memorial defaced on anniversary of 2005 attacks with ‘Blair lied thousands died’ graffiti
Even when it brutalises one of its own teenage citizens, America is helpless against Israel
There’s a nasty smell in the political air – and it’s coming from the Tories
Vanessa Feltz criticises 'vile' reaction to Rolf Harris allegations
- 1 Howard Jacobson: Let's see the 'criticism' of Israel for what it really is
- 2 Game of Thrones author George RR Martin says 'f*** you' to fans who fear he will die before finishing Westeros saga
- 3 Belgium fan Axelle Despiegelaere lands L'Oreal campaign after World Cup viral photo
- 4 Britney Spears sings 'Alien' without Auto-Tune in embarrassingly brilliant leaked audio clip
- 5 PornHub begs users to stop uploading video clips of Brazil getting beaten 7-1
£10000 - £15000 per annum: Randstad Education Cambridge: The JobWe are now rec...
£18000 - £25000 per annum + DOE, OTE £40000: SThree: LONDON - BRISTOL - DUBLIN...
£21588 - £36756 per annum + negotiable: Randstad Education Chelmsford: ICT/Com...
£110 - £120 per day + Competitive rates of pay: Randstad Education Reading: **...