Jumping through hoops

The Government relies on it to give funding, but the research assessment exercise is more about playing games than scholarship, says A D Harvey

For the past few months, the chief preoccupation of academic staff at Britain's universities has been the forthcoming Research Assessment Exercise (RAE). The results will determine how much government money each will receive for research. In 1996, fewer than one-third of university departments obtained the top grade of 5; in 2001, more than half obtained 5 or the new top grade, 5*.

This time the grading system has been revised, but if the rate of improvement has been maintained, nearly three-quarters of university departments can expect full marks. Competition has been reported to be intense: "RAE trigger headhunt race," said one headline; "Star staff cosseted in bid to top up RAE rating," reported another.

There have, of course, been claims that the entire business is a con. After the 2001 results were published it was pointed out that all university departments of veterinary medicine, and 82 per cent of university departments of pure mathematics, received top marks, compared to 21 per cent of departments of social policy and administration, and 12 per cent of departments of environmental sciences. The environmental scientists even tried to overturn their results in the courts.

There were, perhaps, fewer grumbles than in 1992 and 1996 that money won by higher-scoring departments was spent chiefly on increasing the salaries of college heads, but the Association of University Teachers considered boycotting any future RAE exercise, and the National Association of Teachers in Further and Higher Education (the main union in post-1992 universities) passed a motion condemning the Government's use of the RAE to underfund universities.

Since the first RAE in 1992, there have been complaints that the assessment system involves a "commodification" of research, emphasising profit (the prospect of government funding) and downgrading need and usefulness.

Claims that the RAE has led to a narrowing of the range of topics seen as suitable for research, and encouraged hasty research and frequent publication at the expense of major projects that might take longer to complete than the period between RAE submissions, sound plausible but barely stand up to examination. Interdisciplinary research in the arts and humanities was very much in vogue (as an ideal rather than a practice) in the early 1970s, but it had became obvious long before the first RAE in 1992 that if one published a book on political and social history, and another on the evolution of poetic style, you weren't ever going to get a job in a history or an English department. As for the great masterpieces requiring 10 years or more of unhurried investigation, these seem if anything more numerous (though perhaps less masterly) than 20 years ago.

The real problem with the RAE is one that its campus critics back away from, for fear of shooting themselves in the foot: the issue of quality. As the "Guide to the 2001 Research Assessment Exercise" explained, researchers had to provide the details of no more than four items of research: "All forms of output (books, papers, journals, recordings, performances) are treated equally; panels are concerned only with the quality of the research." In other words, a 700-page book marshalling a massive range of unfamiliar source references to present a coherent argument would be regarded as the equivalent of a 14-page article focused on a narrow topic, with a bibliography of 30 items.

Not, of course, that the assessors intended to read pages chosen at random from long books and compare them with pages from short articles: "Panels review all submissions, and read selectively from the research outputs cited."

The obvious meaning of this is that they look at the title of an article, and the title of the journal where it was published, turn over a few pages (mainly to check there is print on them) and say: "That looks fine."

What "read selectively" clearly does not mean is "make a serious effort to assess the originality of a piece of research, the degree of technical proficiency involved in carrying it out, and the scholarly significance of its conclusions." What they seem to mean when they say that "all forms of output are treated equally" is that they are all treated with the same contempt for scholarly principles. But no one objects to this, because, of course, if most work submitted was examined for its originality, proficiency and significance, most of the work submitted even by top-scoring departments would be dismissed as commonplace - and the one thing academics don't want to hear is that they're not very good at their jobs.

Meanwhile, the principle of limiting each researcher to only four items gives the lie to reports that universities are desperate to recruit star scholars to boost their RAE ratings. A star scholar would produce far more than the four articles in half a dozen years needed for a top score. Exactly a century ago, Albert Einstein published his doctoral thesis and the four most seminal scientific articles of the 20th century within the space of a single calendar year. "Oh well," today's academics say, "he was Albert Einstein." My own record is a 395-page monograph and five articles in refereed journals in a calendar year. ("Oh well, you're AD Harvey.")

Departments are also required to submit figures of the amount and source of research funding, number and source of research studentships, number of research students and of research degrees awarded, and "indicators of peer esteem". What are indicators of peer esteem? Most articles in journals in the humanities are never cited in footnotes; most books by academics are reviewed in only three or four academic journals; anyone can get invited to an academic conference who can pay the fee; and as the object of these questions is to assess the "research culture" of a department, shouldn't they also ask about indicators of peer disesteem?

Even at Oxford, a doctoral dissertation might be supervised by someone who has never completed a doctorate or published more than a couple of articles. At Middlesex University, some senior staff include in the otherwise rather short string of letters after their name the initials FRSA (Fellow of the Royal Society of Arts), a qualification obtained in exactly the way one obtains membership of the Carlton Club; by being nominated and approved, and paying a subscription.

At York University, the strict criteria to be met before one was permitted to apply for one of their prestigious Fortieth Anniversary Professorships related not to candidates' research record, but to their incomes. At London University, certain professors require doctoral students to omit from their bibliographies all mention of scholars whose books are regularly reviewed more favourably than their own. Shouldn't RAE panels mark departments down for that sort of thing?

Of course not. The object of the RAE is simply to distribute money. It is welcomed by university managers as a means of harassing academic staff, and submitted to by academic staff because it enables them to fudge the distinction between hard work and worthwhile work. It generates lovely piles of paper while diverting attention from inconvenient details. It is one more instance of the principle of the lowest common denominator in higher education.

AD Harvey's books include 'Collision of Empires: Britain in Three World Wars 1793-1945' (1992), 'Warriors of the Rainbow' (2000) and 'Arnhem' (2001). His short stories have been translated into German, Italian and Japanese, and his poetry has appeared in 'London Magazine' and 'Contemporary Review'