An RAE with something for everyone, but exactly what?
by Jonathan AdamsThe RAE2008 results have produced more information about the spread of the UK’s research achievement than we could have hoped for back in the 1980s. The profiling that Gareth Roberts proposed has overcome some of the worst deficiencies of the old grading system. But it is pretty clear that the data avalanche has buried a lot and it will take a while to dig ourselves out.
Evidence planned to get a post on the blog by Thursday lunchtime last week, and we failed. We failed because, in the words of Morris Zapp, ‘every decoding is another encoding’. Each time we ran a query we came up with new questions, and very few answers.
The best bit about the data is the profiling, and that is also the trip-wire for analysts. Profiling is great because it shows ‘what lies beneath’. For research, most activity is skewed. Some people earn a lot of money, train a lot of post-grads and publish a lot of papers. Most people are less Stakhanovite. Most papers get a couple of citations, or none. Some papers get hundreds of citations. So, wherever you look, you see a distribution that is anything but normal and that means that indices that use averages are not telling you much about the data spread and the centre of the distribution. The Roberts profile takes us away from those averages, and it takes us away from single grades with their terrible funding cliffs.
But how do you analyse a profile? One profile is OK. We can look and see the spread, and reflect on the balance of national and international and whether that means ‘good’ in our dictionary.
A set of profiles across one subject is even better – more work to do the comparisons but we can look up and down the list and see how things shift about. Interesting to see that one institution got a lot of 4* material but actually got less on [4* plus 3*] than another institution. Interesting to speculate on whether those top end 4* outputs are the most critical element or whether it is an overall grade point average that establishes a ranking. And how would you weight the elements in this subject, and would it be the same in that subject?
And that is what makes it very difficult when you start to try and create a combined picture across an institution, and then to put institutions into a single table. We work on the data and start to pick out some strange differences. Can it really be true that Anthropology, History of Art, Music and Drama all really have more than 20% of their output at 4* while Education, Psychology and Agriculture are under 10%? In media subjects and the arts we get 4* values as high as 65% ‘international leading’. I’m very happy for the institutions picked out but I do not really believe it. That means two-thirds of the activity was at the international cutting edge and that is an almost infeasible standard to meet.
Biology, Chemistry, Physics and Mechanical Engineering are in the low teens. Why so different? Two things. One is a much greater familiarity with the concept of categorising a portfolio of evidence about research activity. It’s a pretty basic part of science culture but it doesn’t come so easily to humanities and arts (check the scars on the Warden of Goldsmiths if you don’t believe me). So, for scientists, this is ‘what we do’.
The second thing is dialogue and consensus. Running through 2007, at a lot of meetings and on visits to institutions, I heard people talking about what they thought the RAE2008 outcome would be like. The view, which became a common one, was that selective submission meant few 1* items, a lot of 2*, not too difficult to hit 3* but bloody difficult to get a 4* except on the very best. I thought that view was universal but I now think it was much more the view of scientists, and of scientists in pre-92 universities, than it was of academia as a whole. But because the dialogue was going on, the scientists’ and the engineers’ consensus was reflected fairly consistently in the outcomes for the bigger subject areas.
More thoughts follow, but one thing is clear. Any league table you create now may look pretty shaky by Twelfth Night.
Director, Evidence Ltd