European Trends

NUMBER 56, SUMMER 2009

The Impact of the UK Research Assessment Exercise

Michael Shattock
Michael Shattock is visiting professor at the Institute of Education, University of London. E-mail: shattock@he.u-net.com.

The results of the latest, and probably the last, Research Assessment Exercise (RAE) in the United Kingdom were announced in December 2008, and the financial outcomes for universities were confirmed in March 2009. Each of the previous RAEs (1986, 1989, 1992, 1996, 2001) have cited winners and losers. This year, in addition to the usual concerns about the ranking of individual disciplines, the controversy has intensified over the translation of the RAE results into financial allocations.

A Restructuring Device
It is important to recognize that in the United Kingdom the RAE is not, as national research assessments are in some other countries, an exercise associated with quality assurance carrying reputational consequences only. It is a resource allocation device that determines the institutional recurrent grant contribution to the dual funding system for research—the “R” element in the block grant (about 20% of the whole) and funding awarded by the research councils for individual research projects. The RAE was introduced in 1985/86, following a Cabinet Office review of funding for research and development across all government departments. This assessment program quickly became, however, a key restructuring device within the university system, identifying (and rewarding, financially) universities successful in research and penalizing less successful ones. With the results incorporated into league tables, the RAE conveyed reputational advantage (“the research intensive university”) as well as benefits over time in research concentration. The 1992 RAE coincided with the legislation abolishing the binary line between universities and polytechnics and served to confirm a systemic hierarchy with the post-1992 universities ranked below any of the pre-1992 institutions.

The RAE Methodology
Behind the broad principles of research funding, intense controversies have risen about the methodology of measuring research excellence. From the beginning, the RAE has ranked disciplines not institutions. The institutional rankings and the financial allocations have been derived from the aggregation of subject rankings. The rankings have been undertaken by peer-group subject panels based on institutional submissions. These submissions, discipline by discipline, include research outputs (mostly not more than four publications per individual academic and listed so that the panel can consult them), a description of the research environment (research grants, number of research postgraduates, etc.), and indicators of esteem.

The particular details and the weightings have varied from RAE to RAE. In the early RAEs the presumption was that universities would submit almost 100 percent of their academic staff in the expectation of attracting higher financial allocations. However, as successive resource allocation models delivered less for lower scores, universities have reduced their lists to high-performing staff only. This emphasizes the extent to which “game playing” has developed. Thus, in 2008 Manchester University achieved sixth place in the multifaculty university ranking list by submitting only 75 percent of its eligible staff when most of its peers in the top 10 submitted around 90 percent. The RAE has been constantly criticized for encouraging head hunting (“poaching”) of research stars to win RAE inclusion (with the inevitable inflationary impact on academic rank and salary), although statistically based inquiry has suggested that gossip may have greatly exaggerated the actual transfers.

The RAE has become not just a piece of restructuring machinery but also a major cultural phenomenon of the UK higher education system. Academics’ publication rates may be planned around RAE cycles. Staff are recruited for their RAE potential. Institutional prestige is tied to RAE success, and highly ranked departments are magnets for research students. Membership of RAE panels represent an individual reputational ranking, while exclusion from an RAE submission in a research-active institution form an academic death warrant or, at least, a condemnation to a high teaching load. The publication of the RAE results can represent a defining point in the career of a vice-chancellor, pro-vice-chancellor (for research), or head of department.

The 2008 RAE
An element of predictability had invaded the RAE by 2007/08, which is why the results of its 2008 assessments have provoked surprise and much debate. Instead of the previous seven-point scale, the 2008 RAE adopted a five-point ranking: 4 star (world ranking), 3 star (internationally excellent but falls short of the highest standards of excellence), 2 star (recognized internationally), 1 star (recognized nationally), and unclassified. For the first time international scholars were invited, 50 in all, to join the assessment panels. Pertaining to the ranking, instead of summative ratings for each university, individual “quality profiles” of each discipline were to be identified and ranked. A “world-class” department would, theoretically, need everyone ranked 4 star—whereas in 2001 a 5-star department (then the highest grade) needed 50 percent of its staff rated as at “international standard”—but a generally non-research-active department with one or two 4-star performers would receive credit for their ranking.

In this way, pockets of excellence were recognized in a much more dispersed set of institutions than in previous RAEs. Due to the aggregated ratings, although the top 10 institutions—Cambridge, London School of Economics, Oxford, Imperial College, University College London, Manchester, Warwick, York, Essex, and Edinburgh (in that order)—did not differ markedly from previous RAEs, the table turned more fluid with some universities moving up many places (Queen Mary University, London from 46th to 11th, Nottingham from 35th to 24th) and some others fell equally sharply. The pockets of excellence spread widely across the system, and three post-1992 universities (Hertfordshire, Brighton, and De Montfort) were for the first time ranked above some pre-1992 institutions.

These results raised serious funding issues. The government had always liked that the RAE methodology chimed with its policy of investment in and concentration of STEM (science, technology, engineering, and mathematics) research to support national economic ambitions. This policy also helped maintain the United Kingdom’s position in worldwide citation tables. For institutions concerned about the resource base, however, the major issues have revolved around the gradient of the reward structures for the different rankings and the size of the “pots of gold” allocated to each discipline. The greater dispersal of former pockets of excellence—the majority in non-STEM subjects—produced in a fixed budget a theoretical redistribution of funding away from the major centers of research concentration and drove a coach and horses through the government’s policy. Rumors of large cuts in high-ranked institutions abounded. To accommodate the difficulty, the size of the fixed sum had to be expanded, and a switch of funding into the STEM “pots of gold” had to be undertaken. Thus, in England, whereas in 2001 90 percent of the R funding was shared among 38 universities, the figure will be 48 in 2008—25 institutions receiving research funding for the first time. There have been significant winners and losers: in spite of their ranking, Imperial College has lost 5 percent of its R money and London School of Economics 13 percent (because of the switch of funding to STEM subjects); Nottingham, on the other hand, which is strong in STEM subjects, gained 23 percent.

The Future of the RAE
A compromise may have been achieved. The advocates of concentration can point to 75 percent of the funds going to 26 institutions only, with Cambridge, Oxford, Imperial College, and University College London receiving more than 25 percent. However, the post-1992 universities in particular and many individuals in unfashionable institutions can claim to be vindicated in the exposure of a much greater spread of research talent than was apparent in the past. Nevertheless, the 2008 RAE has created aspirations that will be hard to meet. Another danger is that the new Research Excellence Framework, which is planned to succeed the RAE and will be much more metrics based, will be more heavily steered by government and less likely to reward excellence wherever it is found.

[Online] Available: http://www.bc.edu/bc_org/avp/soe/cihe/newsletter/Number56/p18_Shattock.htm