Slowly, joyfully, painfully, we are coming to terms with the results of the Research Assessment Exercise 2008. For some, it ruined Christmas. But world-leading work was found to be widely spread across the universities of the United Kingdom. That is worth knowing. Nevertheless, the 4* label, on a strict interpretation of its definition, was surely used too liberally, and the Higher Education Funding Council needs to remember that, in some subjects, critical mass is essential.
I am not convinced that this country should perform this kind of exercise. The really important research would get done without it. What an RAE principally does is boost the quantity of solid work. It does, admittedly, help some extremely good academics in medium-quality universities to acquire more resources. It also gives vice-chancellors useful ammunition to lay before those who run weak departments.
Yet all RAE-like exercises throw a disturbingly black shadow. They lead to short-term thinking; they make researchers focus on the names of journals rather than on ideas; they foster an unhealthy conventionality, and probably not a trivial amount of clinical depression; they encourage a generalised dishonesty where much thought in senate houses is put into how unpleasant truths can be covered up; and, perhaps most worryingly, they deter us from putting value on the quality of teaching. It would be better to forget RAE exercises and instead have a large number of independent universities with proper tenure-track positions and with high standards enforced, as for Woolworths, and soon for Tottenham Hotspur if they aren't careful, by the power of competition and what economists euphemistically call "exit".
However, if the country insists on evaluation exercises, the next one, known as the Research Excellence Framework (REF), should in my judgement use a blend of bibliometrics and what I would term "peer overview". In other words, peer review teams should be able to look over the bibliometric data to stop the mechanical nature of that approach getting out of hand.
Just as after RAE 2001, there has been murmuring about grade inflation in RAE 2008. What happens if we use a tough, objective standard for deciding on what counts as 4*? I was recently involved in a study of exactly this for the subject of economics. Because my discipline came close to top of all UK university subjects in the latest RAE, it seems a useful example. And the bibliometric method we hit upon can be used in almost any scholarly area.
The method works like this. I and colleagues examined objective data, for the period 2001-08, on the world's most influential economics articles. The aim was to design a practical way to measure the quality of 4* university research.
Following Elhanan Helpman's new report for the ESRC on the international benchmarking of economics, I collected data on the most-cited articles in 22 well-known economics journals. I found, by this criterion, that the UK produced 10 per cent of the most influential work in the world between 2001 and 2008. For its size, the UK economics profession thus came out well – though I do not think one would say stunningly – in the objective data. Among an elite set of 450 genuinely world-leading articles – the most-cited papers over the period – the UK was the source of 45 of them. As an additional exercise, I studied the rest of Europe. The number of influential articles from these countries combined was 56. In other words, the whole of continental Europe (hundreds of millions of citizens) produced only slightly more of the really important work than the UK (60 million). The US produced the great majority of these 450 articles.
Interestingly, more than a quarter of these objectively important UK articles emanated from departments of economics not normally considered to be in the UK top half-dozen. This suggests that outstanding work – genuinely world-leading economics articles – comes from a wide range of sources (in my data, 21 universities, albeit all pre-1992 research ones).
My findings on the performance of UK economics are redolent of those from the official RAE 2008 data for the whole UK. First-rate creativity is widely spread geographically. For this reason, it may be time to turn away from a modern concern in academia with "top" departments, "top" journals, and similar monopoly-creating devices. Competition – subject to important requirements of critical mass – works.
The writer is professor of economics at Warwick University
Comments