Planned num3rical obsolescence

We all use math every day, to forecast weather, to handle money. We also use math to analyse crime: reveal patterns; predict behaviours. Using numbers we can solve the biggest mysteries we know.
(Voice-over, opening credits, Numb3rs, Series 1. CBS television)

Wake up this morning to the latest RAE results – obsolete before the ink was even applied. Of course, RAE results were always out of date by the time they were published: staff leave, staff join, departments are closed after the census date and so on. Critically, pipeline delays in journals and the year-long slog of examining all those submissions mean that the snapshot we have is largely based on research submitted for publication at least two years ago.

This time round the funding machine that drives this quality supertanker determined in 2007 that this would be the RAE’s last sailing. All of this leads me to wonder whether this RAE has hit an iceberg, leaving a bit of funding flotsam that will float around until the Research Excellence Framework comes down the slipway.

This RAE is different from its predecessors in that it gives a ‘profile’ of each unit of assessment submitted, rather than a single score. Notwithstanding this, today’s press and many university websites will be filled with computations, league tables, formulae, commentary and analyses which turn the profiles into single figures in order to demonstrate who is the ‘best’.

For the numerically minded, I know of at least four different ways of calculating any unit of assessment’s ‘score’, all of which produce (often markedly) differently ordered league tables. The methods include Research Fortnight’s own ‘Quality Index’ which is

RF QI = [ (% in 4* x 16) + (% in 3* x 9) + (% in 2* x 4) + (% in 1* x 1) ] / 16

This formula – which seems to have changed since earlier briefing documents given out by some universities – gives significantly greater weighting to the percentage of 4*s achieved by any unit of assessment than in its own earlier iterations.

What a fun party game we have for Christmas – see how you can calculate the highest and lowest single figure scores for your favourite (or most hated) unit of assessment, with extra points for rationally legitimising each method devised. Unfortunately, the funding councils will be playing this game to their own rules – which they haven’t disclosed yet – to allocate the money that they give to universities for research. This is a pity, as this was supposed to be the raison d’être for the whole exercise and ironic given that the RAE has a pretence of objectivity, transparency and exactitude.

In principle, profiles offer greater potential for nuanced description and therefore a more robust qualitative evaluation of research ‘performance’. Good profiling should let us know the strengths, weaknesses and trajectories of research groups. Unfortunately the rather crude series of numbers published today will do little to facilitate that. Individual submissions will be published – but some time after we have all become bored with the endless game of calculation and recalculation of these simple Numb3rs.

Perhaps this is why it is planned that the REF will go for metrics – it avoids all of that nonsense about nuanced views, any pretence to be anything other than supporting and sustaining established, received and often recycled wisdoms, and cuts straight to the heart of what globalised and commodified higher education is all too frequently about: profit for publishers (if you are not sure about this, see Ciancanelli 2007), on whose citation indices the REF will significantly depend.

There are even more problems with this data. Some weeks ago it was announced that the Higher Education Statistics Agency (HESA) was unable to provide sufficiently reliable numbers of total academic staff in each institution. And in this RAE there was no obligation to provide complete lists of all academic staff in each unit of assessment. This means that it is now impossible to calculate, for each unit of assessment, the proportion of staff actually entered. This very basic error leads to two very serious problems.

First, the inability to calculate the percentage of staff returned means that highly selective departments may have successfully elbowed their way up all the league tables however you choose to calculate them. Proving this is hard, but let me give you some examples. Cambridge University comes third in education if your chosen method is to rank units of assessment by summing the percentage of 3*s and 4*s. On Cambridge’s website yesterday there were 79 academics listed in the education department. In the RAE, Cambridge returned 49.6 FTEs. Now this ratio is crude and, of course, Cambridge will have part time staff and leavers and joiners since the census date. But submitting about 63% of staff? When you are Cambridge? Similarly, and with the same caveats, Manchester Metropolitan University came ninth in the this league table and submitted 22.8 FTEs compared with 85 staff on their website yesterday, or just 27%. If we had really useful profiling data, we might think that MMU was considerably better value for research money than Cambridge. But of course you’re not going to find out.

Second, the failure to make available data on the proportion of staff submitted means that it will be impossible to produce reliable figures for women’s inclusion or, indeed, exclusion from the RAE. So much for the much vaunted ‘gender proofing’ exercise. Instead we will have to rely on assurances that all institutions properly audited their selection criteria for gender bias. Of course, the women will be content with this.

Despite all of these problems, the RAE is important – especially for those who claw themselves to the top of whichever league table the funding councils use to allocate the money. What is alarming is the number of social scientists judged to be excellent who are hoist by their own funding petard: so invested in gaining cash in competition from others that they fail to exercise their own disciplinary practice of critique and, thereby, significantly fail to shout out that the Emperor has No Clothes!
Rebecca Boden is Professor of Critical Management at the University of Wales Institute, Cardiff. She used to be an inspector of taxes.
Ciancanelli, P (2007) “(Re)producing universities: knowledge dissemination, market power and the global knowledge commons”, in Epstein D, Boden R, Deem, R, Rizvi, F and Wright S (eds) World Yearbook of Education 2008. Geographies of Knowledge, Geometries of Power: Framing the Future of Higher Education, New York and London: Routledge.