The long and winding road

It was with a certain amount of trepidation that I entered HEFCE’s offices in August 2004 to take up the role of RAE Manager. After all, having worked in three HEIs and seen the RAE 2001 from the perspective of an institutional research manager, it seemed that, at best, this would be a thankless task or, at worst, a poisoned chalice.

As the results of RAE 2008 are announced, four years on, I’m happy to admit that I was mistaken. There have been times when it has been arduous, not least when rumours circulated of the premature demise of the exercise after the first mention of the "Research Excellence Framework" surfaced. However, as with many things, reaching an end point makes one realise how much was good about the past momentous four years.

The RAE builds a tremendous spirit of a shared community and a strong sense of team (perhaps borne out of adversity); it is also an activity on a scale way beyond almost any other project in the HE sector. The strong collective work ethic has been evident in the work of the panels–at least one panel is reconvening as a research seminar group, and another has been photographed in team rugby shirts. For me, more importantly, that team ethos has been very evident among my colleagues in the RAE team who have worked tirelessly to help bring the exercise to fruition.

From the outset, the overriding objective of the project has been to produce a robust set of results that could be used by the funding bodies to allocate research funds. I firmly believe that we have delivered the former and, in due course, I trust that this confidence will be borne out in QR allocations that reflect the high quality of research undertaken in HEIs across the UK.

Moreover, we have, I think, delivered on the those things that the chairs of our main panels were most concerned to ensure–that the process rewarded excellence across all forms of research, and that those departments that nurtured the next generation of research talent could be identified and recognised.

There have inevitably been twists and turns along the way, some predictable, others less so. Colleagues in the sector were understandably sceptical about the changes introduced to the exercise. So, we ran roadshows up and down the UK to explain the new terminology of quality profiles and two-tier panel structures to colleagues in HEIs.

There were also doubts that our IT systems would withstand the battering of the submission deadline. So, we had it checked over by the people who test the systems for Children in Need, and it was fine.

Unanticipated hurdles came in the form of the complexities of working across all four UK funding bodies–there was a lot more confidence building to be done than we expected.

Issues, such as how best to assess outputs of significant scale and scope (so-called ‘superbooks’), and how to ensure that the contribution of early career researchers could be properly identified, were also challenges that the panels worked hard to address in a consistent way. The insights of our 15 main panel chairs were crucial.

The first part of the exercise was akin to being on a high-speed train, as things whistled by at great pace. Within a few months, the main panel chairs, sub-panel chairs and panel members had been identified, invited, and had sat down to start discussing the criteria. At the same time, we had drafted the guidance to panels, thrashed out definitions for quality levels, and started work on the data collection systems. Then the rollercoaster of panel meetings got underway…but, at least by the middle of 2005, we felt in control of our own destiny.

As we planned the assessment phase, more like an assault on the summit of Everest than a train ride through the English countryside (of which there were many), the breadth of the undertaking became clearer. A repository for storing the 60,000 or so outputs not available electronically had to be found and fitted out; the IT systems to collect submissions and make them available to the panels were developed; a licence agreement with publishers concluded; a small army of panel secretaries recruited and trained…the list could go on.

Throughout, the dedication of panel members and the panel secretariat (who were seconded from institutions across the sector) has been impressive. Whereas many panels set out to assess in detail a percentage of the submitted work, a large number ended up assessing in detail all, or virtually all, the outputs submitted. Often two or even three readers would assess each output–and some panels had 10,000 outputs to assess!

The determination of the panels to put into action the criteria that they had developed was evident. Our international panel members, drawn from more than a dozen countries, helped to ensure that the quality levels agreed for the exercise were applied consistently and appropriately across the main panels–even when it meant dialling into panel meetings at 5am New York time.

It has been an eventful and, at times, exhausting project. But now that we are nearing the end of the road, there is both the relief of having completed the journey in one piece, and a recognition that it has been an enormous privilege to have shared this endeavour with so many colleagues, both in the UK and abroad. Without them, the exercise simply could not take place.

This may be the end of the road for the RAE, but whatever follows it in the form of the REF will be able to draw on this experience so that research quality in the UK can continue to be assessed effectively.
Ed Hughes is Manager of RAE 2008 at the Higher Education Funding Council for England.