An RAE with something for everyone, but exactly what?

The RAE2008 results have produced more information about the spread of the UK’s research achievement than we could have hoped for back in the 1980s.  The profiling that Gareth Roberts proposed has overcome some of the worst deficiencies of the old grading system.  But it is pretty clear that the data avalanche has buried a lot and it will take a while to dig ourselves out.

Evidence planned to get a post on the blog by Thursday lunchtime last week, and we failed.  We failed because, in the words of Morris Zapp, ‘every decoding is another encoding’.  Each time we ran a query we came up with new questions, and very few answers.

The best bit about the data is the profiling, and that is also the trip-wire for analysts.  Profiling is great because it shows ‘what lies beneath’.  For research, most activity is skewed.  Some people earn a lot of money, train a lot of post-grads and publish a lot of papers.  Most people are less Stakhanovite.  Most papers get a couple of citations, or none.  Some papers get hundreds of citations.  So, wherever you look, you see a distribution that is anything but normal and that means that indices that use averages are not telling you much about the data spread and the centre of the distribution.  The Roberts profile takes us away from those averages, and it takes us away from single grades with their terrible funding cliffs.

But how do you analyse a profile?  One profile is OK.  We can look and see the spread, and reflect on the balance of national and international and whether that means ‘good’ in our dictionary.

A set of profiles across one subject is even better – more work to do the comparisons but we can look up and down the list and see how things shift about.  Interesting to see that one institution got a lot of 4* material but actually got less on [4* plus 3*] than another institution.  Interesting to speculate on whether those top end 4* outputs are the most critical element or whether it is an overall grade point average that establishes a ranking.   And how would you weight the elements in this subject, and would it be the same in that subject?

And that is what makes it very difficult when you start to try and create a combined picture across an institution, and then to put institutions into a single table.  We work on the data and start to pick out some strange differences.  Can it really be true that Anthropology, History of Art, Music and Drama all really have more than 20% of their output at 4* while Education, Psychology and Agriculture are under 10%?  In media subjects and the arts we get 4* values as high as 65% ‘international leading’.  I’m very happy for the institutions picked out but I do not really believe it.  That means two-thirds of the activity was at the international cutting edge and that is an almost infeasible standard to meet.

Biology, Chemistry, Physics and Mechanical Engineering are in the low teens.  Why so different?  Two things.  One is a much greater familiarity with the concept of categorising a portfolio of evidence about research activity.  It’s a pretty basic part of science culture but it doesn’t come so easily to humanities and arts (check the scars on the Warden of Goldsmiths if you don’t believe me).  So, for scientists, this is ‘what we do’.

The second thing is dialogue and consensus.  Running through 2007, at a lot of meetings and on visits to institutions, I heard people talking about what they thought the RAE2008 outcome would be like.  The view, which became a common one, was that selective submission meant few 1* items, a lot of 2*, not too difficult to hit 3* but bloody difficult to get a 4* except on the very best.   I thought that view was universal but I now think it was much more the view of scientists, and of scientists in pre-92 universities, than it was of academia as a whole.  But because the dialogue was going on, the scientists’ and the engineers’ consensus was reflected fairly consistently in the outcomes for the bigger subject areas.

More thoughts follow, but one thing is clear.  Any league table you create now may look pretty shaky by Twelfth Night.

Jonathan Adams

Director, Evidence Ltd

Keep the back-slapping in perspective

So, once again we have the gift of RAE results to brighten our festive cheer. But, in the spirit of the season, should we not pause a moment to reflect on what the outcomes actually tell us? There will, inevitably, be much obsessive interrogation of the outcomes and their implications in the days and weeks ahead but, in the midst of that, let us not forget the limitations of the RAE.

Much might have been done to refine the assessment process for 2008, building on the experience of panels in previous exercises. For example, this exercise could have tried to improve the consideration of interdisciplinary research, applied work, and the lot of early career researchers.

I have no doubt that all panel members in 2008 will have worked hard at these issues, and yet the framework they were given to work within looks very familiar and increasingly anachronistic.

Research has moved on apace: it is ever more collaborative, trans-national, interdisciplinary; it is increasingly done in knowledge sharing and translational partnerships with its non-academic users and beneficiaries; it is disseminated through a multitude of formats and media.

Yet, the RAE remains stubbornly disciplinary in its structure and, as I am sure closer examination of submissions will show, concerned with a rather narrow range of ‘safe’, conventional, academic outputs.

Now is not really the time to talk about the prospective Research Excellence Framework, but it is hard to see how the REF can be other than a retrogressive step in these areas as it is currently proposed. Research is complex and sophisticated: it deserves the same qualities in the systems by which it is evaluated. The RAE is not perfect, but it is a subtle and mature assessment tool, pioneered in the UK and recognised internationally as a model of good practice that commands confidence among the research community. It could be further improved, drawing on the skills and experience of panel members and our own growing research-based understanding of peer review processes. Should we really abandon it so lightly in favour of the questionably cheaper and undoubtedly nastier blunt instrument of bibliometrics?

The only genuine major change introduced for the RAE 2008 is the one that is proving most problematic, in many ways, as we grapple with the results. The currency has changed: graded profiles replace numerical values. Moreover, the conversion rate from the old grading scale is far from clear, and significant questions remain over whether, and to what extent, the currency has been debased in the process.

Equally, the mix of base and precious metals does not appear, on first glance at the results at least, consistent across the different territories in the RAE landscape. There are variations in the grade profiles awarded by some panels that seem rather at odds with what we might otherwise expect of the disciplines they have assessed.

The results of previous RAEs have been broadly endorsed after the event by comparative bibliometric analyses and independent expert opinion. We will need the same tests to be applied to the 2008 results, I suspect, before we can have full confidence in their robustness and consistency.

Perhaps the most worrying aspect of the change in rating scale is that the familiar hallmarks of quality, the 5 and 5*s, which are understood and relied on by research partners, sponsors, prospective research students and staff, have disappeared. As a sector, we face major challenges ahead in communicating the meaning of the results to those outside our walls who are not steeped in the mysteries of the RAE and for whom the ‘ready reckoners’ of quality have gone.

The need to offer clear, unambiguous quality labelling of the UK research base to industry, the public, voluntary services, policy makers and potential international students and staff recruits, will only increase as we face the difficult financial times ahead.

The season calls for celebration, and there is undoubtedly much to celebrate in the outcomes of the 2008 RAE. Research in UK higher education institutions is bigger, better and more exciting than ever before, so let’s party. But as we do so, let’s remember that there is also much exciting, relevant, valuable and high quality research that is not included within these results. They are only one indicator of the quality of a limited selection of the work that is going on in the UK research base. Keep them in perspective!
John Rogers, Director of Research and Knowledge Transfer at the University of Stirling, was the Manager of RAE 2001.

Final chapter in turbulent year for physics

Last February, the Council of the Royal Astronomical Society concluded that they had "no-confidence" in the new Science and Technology Research Council, which a few months earlier had taken on the responsibility of funding UK astronomy.

"The STFC’s Delivery Plan pays lip-service to the need to foster the UK academic community, who play the key role in delivery of all of STFC’s outputs–first class science, facility design and usage, and knowledge exchange, but has shown no evidence in its public statements or actions that it recognises this duty," announced the RAS, a normally cautious and diplomatic learned society. "The 25-per-cent decline in grants across the CSR period, with no sign of any intention or even desire to level this out in later years, has filled the community with deep pessimism and anger."

In a broad-ranging review of the current status of UK Physics, published in August, the Wakeham Panel reported the discipline to be in generally good health. However, as the review was set up in response to concerns such as those arising from the RAS Council, it was significant that the panel found that many of the strongest areas of UK physics were highly dependent on STFC support.

Furthermore, given concerns over a long-standing fall in physics A-level entries, it was important that the research areas in STFC’s remit, such as astronomy, space science and particle physics, are often also those most attractive to students–of both genders.

Adding to this turbulent scene, we now have another major input, with the results of the RAE 2008 RAE. Profiling all university physics departments on the percentage of staff judged to be of national or international repute, the overall picture confirms that university research is indeed competing very well at the (critical) global level. Or, at least, was recently performing well, given that the RAE is necessarily backward looking (over the period from 2002 to 2007).

So, how important is this final RAE for university physics?

In addition to providing another assessment of the discipline in a globally competitive research context, departmental profiles will, for several years ahead, determine the level of QR funding from HEFCE. And this second funding stream is more important than ever, in my view, given the erratic performance of the STFC and the enthusiasm of the Engineering and Physical Sciences Research Council for top-down thematic programmes.

Although representing a smaller fraction of university research income than it did before Fixed Economic Costing (fEC) was introduced, pre-FEC, QR funding is likely to form around 30 per cent of research ‘overhead’ income for a typical research-active department such as my own. It provides ‘seed corn’ for funding of young researchers not yet on the research council radar, and a useful counter to a reduction of external support for ‘blue skies’ research in response to government pressures for wealth-creation.

Translating a departmental research profile into the HEFCE grant will, of course, depend on the still-to-be announced funding formula. If HEFCE follows earlier practice, this will be skewed to disproportionately favour 3* and 4* researchers. If used that way, the QR funding will indeed be a significant cushion against STFC cuts, as it will benefit particularly those same strong departments that, paradoxically, are being hardest hit by the 25 per cent reduction in STFC grants.

However, recalling the damage done to the broader university physics community by previous RAEs, I would caution against straying too far from a linear funding formula, whereby a 4* researcher attracts 4 times the funding of a 1* researcher. Broadly spread QR funding could be crucial in avoiding more physics department closures, and an expansion of the regional ‘research deserts’ identified by the Institute of Physics.

Prestige is also at stake in the RAE results, in addition to funding. In that respect, the current exercise appears badly flawed.

To give a true picture of a department’s research strength, it is clearly important to know whether all eligible staff were submitted for assessment, or whether substantial numbers were ‘hidden away’. That normalisation has been denied in the 2008 RAE by HEFCE deciding not to reveal non-submission data. More sinister is the rumour that HEFCE’s decision was made under threat of legal action by a number of universities who chose the more ‘selective’ approach.

In response, we are likely to see conflicting ‘league tables’, with the most meaningful, of research ‘intensity’, using previous staff numbers to normalise the ‘quality’ tablebased only on submitted researchers.

So, should we welcome or regret the passing of the RAE?

I believe the RAE brought a new rigour to university research, and substantially raised the overall standards. However, as noted above, the harsher light led to some weak (and not so weak) departments being closed or merged. In physics, the number of university departments fell from 79 to 51 over the period of the 1991, 1996 and 2001 RAEs, with the undoubted loss of some important potential. On the other hand, the fall in student numbers probably made some shrinkage inevitable.

The Wakeham Report’s criticism that UK physics departments are sometimes too narrowly focussed has some validity, and has roots partly in the RAE. Another undesirable consequence of the non-linear funding of QR has been its influence on university appointments, with ambitious VCs competing–and often over-paying-for star performers in the manner of chairmen of Premiership football clubs.

Looking ahead, it seems that some metrics-based quality assessment will be used to guide the HEFCE component of dual research funding. No doubt there will be complaints that the particular metric is unfair to some. However, all universities must surely prefer to keep an imperfect system rather than to lose the dual-funding lifeline that must be in some danger now that both HEFCE and the research councils find themselves in the same part of Whitehall.
Ken Pounds is Emeritus Professor of Physics at the University of Leicester and former Chief Executive of the old Particle Physics and Astronomy Research Council.

Graded profiles make finer assessments

Hertfordshire’s performance in the RAE 2008 is good news and demonstrates that our selective strategy of submitting 14 Units of Assessment was successful. The graded profiles are very pleasing, particularly with six Units of Assessment showing 50 per cent or more internationally excellent and world-leading research (combined 3* and 4* ratings). Graded profiles are a fairer approach, compared with the single quality numerical values of RAE 2001, in the way that they identify the range of research activity in percentage terms over the five categories.

STEM research at Hertfordshire has performed very well with strong outcomes in Physics, Computer Science and Informatics, and General Engineering. The recently established Pharmacy provision also records a very respectable performance. Furthermore, both Nursing and Midwifery, and History show exceptional performances, which pus them towards the top of the university sector.

In these Units of Assessment, Hertfordshire has performed not only better than most of its post-1992 contemporaries but compares favourably with many longer established universities.

More generally, a number of post-1992 universities have shown real improvements in performance in specific Units of Assessment, and it is apparent that the 2008 RAE signals a further breakthrough in research for this sector. The relatively small investment to support research in the non-intensive research universities has resulted in excellent results overall.

Hertfordshire’s strategy over the past seven years has been to focus our somewhat limited resources on developing international excellence in selected research areas. This has not only proved successful but, in combination with the finer tuning in the assessment methodology, it now provide a platform for the university to build further its research profile.

The additional information that will be provided when the sub profiles of the assessment become available to individual universities in January 2009 will be invaluable. It will enable individual Units of Assessment to better understand the rationale for their overall profile, and to more fully identify their strengths and any weaknesses associated with their research activities.

Clearly, this will be particularly important for those Units of Assessments at universities that appear to have under-performed since the RAE 2001.

In addition to receiving our RAE results directly, we have benefited from using the Research Fortnight’s RAE benchmarking software. This has enabled us to assess our position in quality research terms against other universities.

We are delighted to be placed number 53 in a table of 117 universities, which again is indicative of the quality of our submissions in relation to the overall sector. There will, of course, follow a very interesting period of reflection, and a requirement that the funding algorithm applied is fair in its support of quality research wherever it is found. Assuming this is the case, then the RAE 2008 will have set a new benchmark in the assessment of research in the UK.

The challenge for the now delayed Research Excellence Framework will be to surpass this exercise by further enhancing the recognition of cross-disciplinary research and research aligned to societal and economic impact.
John Senior is Pro-Vice-Chancellor (Research) and Graham Galbraith is Deputy Vice-Chancellor at the University of Hertfordshire.

Roll out lifeline for arts and humanities

Reputation is one thing, money is another. For research in the arts and humanities, the RAE is of enormous importance because it doesn’t simply signal the reputation of departments, it is also fundamental to the funding of research in those departments. In the arts and humanities, the QR income that comes as a result of an institution’s performance in the RAE constitutes over 80 per cent of the funding from the dual support system, against about 50 per cent across the rest of the science and research base. So, RAE 2008 is not just a beauty parade; it’s one where the money that flows from that parade is fundamental to being able to do research.

I’m writing this at the end of a very long day, one that was probably just as long in every university round the UK, certainly every university where research is an important part of their mission, and at many others as well.

All institutions will see strengths and weaknesses in their performance but, at Goldsmiths, we’re pleased at consolidating our position within the research-intensive sector that makes up the Russell Group and the 1994 Group. Ranked by the top 4* grade alone, we’re especially pleased to have come ninth of the non-specialist institutions.

So, as it was for many in university senior management and their planning and research offices, yesterday was exhausting. And a day that hasn’t left much time for detailed analysis of the national picture, uncoloured by the need to know how well one’s own institution has done.

Nonetheless, a quick sketch of reflections might be worthwhile, partly because the RAE is so materially important for the arts and humanities, and also because the performance across institutions doesn’t always match the performance across subjects as a whole.

The arts and humanities have a good number of specialist institutions, for sure, but then specialist institutions exist in the sciences as well and they’ve mostly come out well from the RAE. More distinctive, perhaps, is the extent to which post-1992 institutions have climbed the league tables that, in general terms, are dominated by the older and research-intensive institutions.

In a simple league table based on Grade Point Averages, the top 60 institutions contain just two post-1992 universities (the University of the Arts, perhaps better seen still as an aggregation of specialist institutions, and Ulster). If one just looks at the arts and humanities, however, there are seven in the top 60, and 10 in the top 63!

Some of these institutions have done well by submitting relatively few staff for assessment, but others, such as Brighton (at 28, the top-ranking new university in the arts and humanities table, and one with real strengths in the arts) and Westminster (at 47) provided fairly substantial submissions. This is even more the case with the large submission from the University of the Arts (ranked at 58).

Such rankings reflect the strength of the creative and performing arts in the newer part of the sector, but not exclusively so. The success has emerged from the current levels of research funding and research concentration, and underlines the importance of funding excellence wherever it is found.

Position in the arts and humanities frequently doesn’t align with the overall position of an institution. Some research-intensive institutions do a lot better in the arts and humanities than they do overall–York, Warwick, Queen Mary, King’s College London, University of East Anglia, City and Royal Holloway are all striking examples of that, though there are more. Others do less well in the arts and humanities–UCL, Edinburgh, Bristol, Bath and Surrey among them.

It is too soon to speculate on the reasons, and probably unwise because there will be many counter-examples. It does nonetheless appear to be that case that smaller, 1994 Group institutions are likely to be stronger in the arts and humanities than they are overall, and this is probably a function of the challenge to these smaller research intensives of doing well across the board in big science subjects.

While chief executive of the Arts & Humanities Research Board, I learned that it is dangerous to lump together the arts and the humanities as if they could become a single homogeneous block by repeating the phrase often enough.

Just look at the relative success rates of research-intensive institutions in the RAE 2008 units of assessment for the creative and performing arts. In the top 20 institutions in Art & Design, just 8 are research-intensive, with 4 other pre-1992 institutions, 6 post-1992 and 1 specialist. Communications & Media match this achievement, with 10 from research-intensive institutions, 1 other pre-1992 institution and 9 post-1992.

Here is a real breadth of strong performance across the sector. But neither Drama & Dance (13 research-intensive and just 2 post-1992 in the top 20) nor Music (17 and 1 respectively) allow us to say that it is a pattern associated with the practice-based Panel O disciplines. Perhaps there is a larger historical and theoretical component of strong departments in Drama and Music than in Art & Design or Media? I’m not at all sure.

What is clear, however, is the overwhelming domination of the top 20 in the humanities by research-intensive institutions, for example, English 17, History 16, History of Art 17, Classics 19, French 19, German 19, Philosophy 18.

It is also clear that the arts and humanities allow players outside the research-intensive institutions to draw on old achievements or new opportunities, and to position themselves more highly than in the overall league tables. But only in some of the arts are they generally able to make a mark in the top 20.

There are, of course, exceptions, such as De Montfort in English and Oxford Brookes in History, even if the latter couldn’t repeat its famous 2001 achievement of outperforming the University of Oxford in that subject. But overall, it is in the arts where new universities can successfully make their mark, which signals an older capacity on which research culture and performance have been built.

My final comment may not be peculiar to the arts and humanities, and that concerns differential levels of generosity and meanness. The arts subjects spread their marks: the top 20 in Art & Design stretched from GPAs of 3.40 to 2.55, and these were matched for Communications & Media (3.60 to 2.55), Drama & Dance (3.50 to 2.55) and, a little less so, for Music (3.45 to 2.85).

There were, on the whole, lower top marks and more bunching in the humanities, with English’s spread from 3.15 to 2.85 for the institution in 20^th place, and History from 3.0 (if one omits the tiny Imperial submission that headed the table) to 2.75. Philosophy had a little more breadth.

Are differential spreads such as these the result of differences of quality or differences of discernment in the marking? And why in some humanities subjects did no department achieve a GPA of even 3.0–French, German, Linguistics, Archaeology and Asian Studies among them?

I suspect that there will be much mulling over differential severities of grading in different disciplines. However, at least the new profiling system’s elimination of the cliff-edge changes between grades meant that assessors were unlikely to shoot their own discipline in the foot in the way that the Art & Design panel did in 2001 by deciding that no submission was worthy of a 5*. That haunted the discipline for a long time, which is probably why their successors used the full mark range this time.

So, there is much to digest in these results for the arts and humanities. We see the continuing strength of the research-intensive universities in the humanities, and the expanding presence of strong research in the post-1992 institutions in parts of the Creative & Performing Arts sector. More broadly, institutions from across the old binary divide were able to sustain international quality research in the arts and humanities in a way that was not repeated across the RAE submissions as a whole.

If the research-intensive part of the sector dominates, then that is why it is the research-intensive part of the sector. But we should celebrate the diversity of places where at least pockets of research excellence are to be found. It is important for the health of the arts and humanities, and it is why government and HEFCE would be well-advised not to seek greater levels of concentration of funding than exist at present.
Geoffrey Crossick is Warden of Goldsmiths, University of London.

Did goal posts move before REF got on?

After the RAE 2001, Sunderland was crowned the "best new university for research", based on the overall grade point average per FTE of academic staff, and then our QR funding was cut by 40 per cent, from about £1.6 million to £1 million.

I suppose it was our own fault. Ratings of 3bs had attracted funding in the RAE 1996 and, even given the sector’s increased performance, we had naturally assumed that, with 3as, we wouldn’t lose out. I wonder what will happen this time, with the interesting graded profiles?

While we are well aware that we do not reside on the point of a golden multi-angled two-dimension shape, we do think that we undertake research that is of a volume and quality that is appropriate to and required for our mission. Like many other institutions, we believe that research informs teaching and enhances all levels of the student experience, which is core to our mission. As a result, we undertake research across the full range of our subject areas.

Unlike some new universities, we have made submissions to the RAE in nearly every one of our teaching areas, and we have been rated internationally in all and world-leading in many.

We value all levels of research and, as in 2001, we also submitted a relatively high proportion of staff, over 40 per cent, which is more than many of our peers. With relatively little peril of cliff-edge effects in funding, unless we have been unlucky with the 5-per-cent roundings, this approach seems pretty risk free.

We will expect to see our profiles fairly low in the various subject grading tables that will inevitably develop. But we are thick skinned, and can always rely on someone to calculate the average profile–be it grade point average or one of the myriad of other possibilities–for the whole university, including those staff not submitted. And this is where we do well against many of our peers.

Although they may have a better profile and, in some cases, even a better power rating (multiplying by the FTE submitted), our submission of 40 per cent of academic staff lifts the average rating in the institution as a whole. Of course, it is also where we fall further behind the ‘research high fliers’ (or whatever the de rigueur phrase is this week), but we recognise that that is not our mission.

We also know that, as a small institution with fewer than 500 academic members of staff, we will languish near the bottom of the power tables unless our relative size is taken into account.

However, it appears that many HEIs could not agree on how to count their staff, and so the HESA data will not be readily available (and of course was not collected at RAE submission time). Luckily though, the tables that include the percentage of staff submitted in the calculation will tend to favour Russell Group members, and their kind, which submit high proportions of their staff. The league tables that develop from these data will once again return us to the upper echelons of the rankings of the the new universities, which is where our mission requires us to be. For Sunderland, for once, it does seem to be a question of "if you can’t beat them, join them".

Of course, it is not all doom and gloom. Whatever funding regime HEFCE comes up with, the graded profile system seems sure to deliver us some QR allocation in most of our subject areas, with all of the knock-on benefits in terms of other funding streams (research capital, postgraduate research, charities’ support, and so on)

Overall, the RAE 2008 certainly seems to have levelled the playing field in terms of opportunity for universities such as Sunderland to follow their mission and be rewarded for it. Notwithstanding the issues of league tables, we do hope that we will be properly rewarded for our efforts this time. In 2001, we gained 3 grade 4s and 9 grade 3as, but the latter netted QR for only one in a capability area.

But our confidence has recovered. The only way our mission could hinder us this time is if only 4*s are funded–but surely HEFCE can’t have generated all these lovely profiles to use only one element of them in the funding formula. I think we will score this time!
Simon Kerridge is Head of Graduate Research Support at the University of Sunderland and Secretary of the Association of Research Managers and Administrators.

The king is dead, long live the king

The final set of RAE results is with us. The RAE is no more; on to the REF. But how much of a transition is there going to be from RAE to REF?

I suggest that the change from a single grade in RAE2001 to a three-part profile is as important as the future transition to the REF. The latter may well see greater use of metrics, directly and indirectly, but the provision of the profiles gives a much richer set of information than we’ve had before. (The profiles, published today, show the cumulative result for each submission. In January, institutions will receive the sub-profiles–for output, environment and esteem–that combine to make the cumulative result. The sub-profiles and the submissions will be generally available in Spring next year).

As before, we’ll be able to compare the input information (publications, postgraduate students per FTE and research income per FTE), once the data are released. Now, however, we’ll also be able to see more directly what the panels thought of those data and the other elements of the submission.

It’s probable that some submissions will have good ratings of their outputs, but that their environment can be seen to have pulled down their final rating. How might this additional information be used? Will it lead to greater attention on or support for departments to deliver on specific aspects of performance?

The profiles also introduce interesting opportunities for the funding councils, in terms of the funding mechanism that they choose to adopt. It goes without saying that there will be differential funding across the profile, but will that differential be the same for all subjects, as it has been in the past?

For example, if a subject was seen, nationally, to be internationally strong (that is, lots of 2*, 3* and 4* research), would it be reasonable not to fund 1* research? Equally, if a subject were weak, would it be reasonable to fund 1* and 2* at a higher rate, in order to try to support the development of the subject? Perhaps more controversially, if this is about funding institutions rather than subjects, per se, should the same approach be taken per institution?

A related issue is the allocation of the total funding across the subject areas. The approach differs between funding councils, but it needs to be based on the weighted combination of quality and volume in a subject, taking account of the relative cost of research, so that subjects get funded appropriately.

I wonder how much more informed we are now about the relative costs, which some believe are in need of updating.

A thought on presentational matters. It’s primarily the sector itself that gets uptight about the RAE and consequent league tables. But others do take note of the results. Nationally, we need to ensure that the transition from a seven point, discrete scale to a 0-4 continuum is adequately explained. Previously, ratings of 5 (with or without a multitude of stars) were "good". Now, a grade point average (I’m not promoting them, but they will be used) of 2.5 might be "good" in one subject, even though quantitatively it is half the value, and 2.8 in another. How will we ensure that the presentation of the results to external audiences is adequately explained? What will be the new currency in adverts and promotional material? I imagine that it could be proportions or volumes of staff assessed as being internationally recognised (that is, 2* and above) or internationally excellent (3* and above). Whatever we use, we need to ensure that the change in rating scale does not disadvantage the UK’s external reputation.

The RAE has been criticised because of its burden and its behavioural effects. I’m yet to be convinced that the REF will significantly reduce burden, partly because the sector does not seem to like simplicity. Also, it will have behavioural effects; all systems do, even more so if there’s money or career prospects attached.

However, I am a believer in the RAE and the effects it has had on institutions taking more seriously their responsibilities towards understanding and managing of their research activity. This is not necessarily about being more directive towards researchers’ activity, but about recognising what needs to be in place to achieve excellence and impact, in the widest sense.

Finally, a reminiscence on my part. My first RAE was in 1996, when I managed the process for the University of Glasgow. The results also came out just before Christmas, and the day they were released happened to coincide with the University Court’s annual dinner. Having arrived back from the funding council bearing the results and an initial analysis done in the car, I met with the Principal and Vice-Principals, all in their dinner jackets and black ties. Somewhat surreal!
Ian Carter is Director of Research and Enterprise at the University of Sussex, and chairs the Association of Research Managers and Administrators.

Prioritise funding for research intensives

The latest, and last of its kind, Research Assessment Exercise contains a single powerful message for the UK: we have an outstanding higher education sector by any international standard and our research base is in very good order.

RAE 2008 has confirmed that universities across the country are carrying out high quality internationally competitive research. This is exactly what is required to help the UK out of recession, provided we continue recent trends to translate discovery into product via the spin-out of small- and medium-sized companies.

Despite the efforts it has demanded of universities, which submitted very large quantities of information about their research activities, this form of assessment is extremely worthwhile. It is a robust tool that weighs the research output of the UK and, crucially, allows us to distinguish the best from the good or average.

This selectivity is of paramount importance. The UK should, of course, be proud of and nurture all of its universities–heterogeneity in their focus, mission and overall strategy is to be welcomed. They are the bedrock of our economic competitiveness, and the source of innovative research that improves our quality of life and expands our understanding of the world we live in.

But it is vital to recognise that some universities are more research intensive than others and to selectively support those institutions that are the most competitive internationally, and in terms of translating research to the benefit of UK plc. Thus, we must make a funding priority of those research-intensive universities that have consistently performed at the highest levels, are truly world-leading and internationally excellent.

These are not just national treasures but international treasures. They are also our best defence and, indeed, line of attack against the pressing global challenges of the 21st century.

Nations can and must agree protocols to reduce carbon emissions in order to avert catastrophic climate change, to take the most obvious example. But the effectiveness of such protocols will depend on scientists and engineers coming up with reliable and sustainable clean energy alternatives. The expertise needed to develop solutions can be found in the UK's world class universities.

The same is true for other threats and concerns, from global health to global security. At Imperial College, our governmental, industrial and NGO partners choose to work with us because they know that there is no better place than a world-class university to unravel knotty problems and develop answers to major challenges.

The UK is in the enviable position of being able to boast of four universities in the global top 10, as judged by the 2008 Times Higher Education world rankings. And I am very proud that Imperial has the highest proportion of staff, at 73 per cent, whose research is rated as world-leading or internationally excellent in this RAE. Cambridge and Oxford are close behind with 71 per cent and 70 per cent respectively.

The most urgent advice that I can give to government and industry in these uncertain economic times is this: do not fail to invest in the UK's most valuable and most prized intellectual assets. We must sustain the pipeline of innovation that feeds translation and spin-out engines to support employment and the future growth of the UK economy.

Do not be tempted by short-term cost-cutting to reduce funding or cancel contracts for discovery and innovation. Do not cause the UK to fall behind its competitors or shirk its global responsibilities. The key to long-term health and prosperity in the UK economy is world-class university research.

Today, we celebrate the strength of UK universities and perhaps breathe a sigh of relief that all of the hard work was worthwhile. But we must also look to the future. The RAE judges university research by a robust and tested method. Its successor, the Research Excellence Framework, must be equally robust, fit for purpose and able to distinguish the very best from the very good and, indeed, to measure openly both quality and volume.

Most of all, it must send a clear signal to the world that continued excellence is prized above all else in the UK's universities and that our contribution to discovery and innovation will continue to be supported by government selectivity.
Roy Anderson is Rector of Imperial College London.

Phew! Now all we need are New Year details and Spring allocations...

Research managers across the country are by now bent over the mass of data that has been produced by the RAE 2008, seeking to spin the best story possible out of it for their institution. And as everyone has by now realised, the 2008 RAE system offers much scope for precisely that–making the most of it. Universities will be able to take advantage of the more detailed rating system on offer to play to their individual strengths.

The exercise has cost HEFCE £12 million to run and many sleepless nights. It must be a frightful relief to David Eastwood, the council’s departing chief executive (to run Birmingham university), that the main thing we can all take away from the 2008 RAE is that UK research is really great, and it’s great everywhere! Phew.

The results have shown that 17 per cent of our research was rated as internationally excellent, with a 4* rating, and a further 37 per cent landed in the 3* category.

However, HEFCE assures us that this excellence has not come at the expense of research volume. The council rejects the suggestion that universities have been leaving out staff to boost their average quality rating, and it notes that the number of academics reviewed under the RAE has increased by 12 per cent.

Someone has been playing the system though, because actual submissions dropped by 9 per cent.

While today’s outcomes will provide ample food for thought, universities will have to wait until 5 January to get fuller feedback on their performance, and until 4 March to find how much cash they have secured for their efforts.

Forget submission rates, funding is the issue

So, it’s all over bar the money. A day spent in an ‘operations room’, first waiting for the download, then the team dissecting the results and letting people know what they got–not so different from our students hanging on their degree results, though the collectiveness of the occasion makes it more like an election night. So, who will be singing "Things can only get better" tonight?

While the funding councils would like to think of the RAE as an absolute measure of quality, the reality is that it is all about relative performance. It could hardly be otherwise when the primary purpose is to drive a funding allocation.

There are also many reputational spillovers as other funders, research students and collaborators see the outcome as a convenient signpost of quality. Foreigners might raise an eyebrow at the claims, which we are likely to see made, that the great majority of UK research is of "international quality". But this is the national Premiership, not the European Champions league, so that does not really matter. What it comes down to are rankings.

Until the funding formula is announced (of which more later), the focus is likely to be on league tables. Since these do not exist officially, it has become the prerogative of the press to fill the gap. Research Fortnight’s online service gets the 4* this time with its extensive analyses and do-it-yourself weightings facility.

An interesting feature has been the positioning and lobbying in the past few weeks as universities try to influence the presentation in ways that they believe will favour them.

Perhaps the most irrelevant argument has been the one about submission rates. Encouraged by some vice-chancellors, some newspapers claim that there has been a plot to prevent them from resurrecting the data available in previous RAEs, on the proportion of research active staff submitted, by using HESA data. Heated language has suggested that this outcome undermines the results because profiles can be massaged to omit staff who are likely to lower the ratings.

To some extent this is true, though a price has to be paid in reduced volume (and a lower position in research power tables). The fact is that when such data were available in 2001, the press and most academics largely ignored them. Furthermore, at the level of an institution, comparisons are so unreliable as to be useless.

Without going into the technicalities, there are at least two reasons why such data would tell you little or nothing. Firstly, large numbers of research-only staff who met the criterion of ‘independent researcher’ were submitted. For this group, inclusion or exclusion is largely discretionary, and could easily obscure the numbers of omitted academics.

Secondly, exclusions are largely concentrated in subjects where a proportion of staff are unlikely to perform research, for example, because they are professional trainers in vocational subjects, or are clinicians. At an institutional level, the proportion of staff submitted is more likely to reflect the mix of units of assessment than major policy differences. I would be surprised if the top 20 institutions had very different submission rates. It would be a pity if some misguided attempt to second guess these data distorts the presentation of results.

That leaves the data on profiles and FTEs, provided by the funding councils, as the basis for the league tables.

The most likely basis for the construction of tables is the Grade Point Average (GPA). When the funding councils chose to give the categories numeric labels, it was an easy jump to assume that these could somehow be read as a scale and arithmetically averaged. This move is no more justified as an assumption than any other weighting would be. The previous history of geometric distributions between grades suggests that GPAs almost certainly will not reflect the eventual distribution of funding, so why use them to construct tables?

The most fundamental divide in the analysis of results is one of scale–the question here is whether it is better to have a high average or a high total amount of quality in a given subject in an institution.

This is the difference between GPAs and research power indicators. Both approaches have their strengths and weaknesses. The difference hinges on the question of critical mass. Does it matter how many researchers of quality are in that subject in a given institution? In theory, the highest GPA could go to an institution with only a handful of researchers that have very little impact in total.

The smart reader of league tables will separate specialist institutions from the rest and consider them in the context only of their own subjects.

Research power tells you that there is a concentration of excellence but, equally, that there may be, in the same place, a concentration of the less excellent. Critical mass is important but it can often lie in the ability to configure teams across different disciplines rather than within a disciplinary block. You pays your money and you takes your choice.

What next? More detailed profiles and some stylised feedback in January, but the really important outcome is the announcement of the distribution of QR funds in March. The RAE is an answer to a question that is finalised after the results are known.

The challenge is to meet the funding needs of cash-strapped institutions and still have enough change to reward success and dynamism, but not so much as to destabilise large parts of the sector. A lot will come down to how many policy goals QR is intended to meet.

To remain at the international forefront, the top institutions need serious funding support. The previous skewed distribution of UK research funding has recognised this need and is largely supported by similar distributions being found in competitive grant funding. However, the clear blue water of research power that exists in the upper parts of the table–first between Oxbridge and the rest, then between Manchester plus UCL and the Russell Group pack, and also for the outstanding specialists, such as LSE is offset by a system which potentially rewards excellence found in small quantities virtually all the way down the table. It will be very difficult to design a formula which meets all needs and demands.

Could we see a proliferation of special objective funds to target some QR at institutions that would otherwise suffer losses surely unintended by the Funding Councils? This would depart from the stated objectives, but it has happened before.

And beyond that? A few months ago, the RAE appeared to be a dinosaur heading for extinction but its intended replacement, the Research Excellence Framework, is proving to be an even less flexible and agile creature whose survival may depend upon it becoming more and more like the exercise it was intended to replace.

Peer review is like democracy–the worst solution except for all the others. So, we may well see the RAE survive under another name. The biggest risk is that squabbles about the relative merits of different assessment systems could prompt politicians to pull the plug on QR–and that would be a tragic ending to the story for UK research.
Luke Georghiou is Professor of Science and Technology Policy and Management in the Manchester Institute of Innovation Research at Manchester Business School.