Exquisite Life Exquisite Life Research Europe Research Fortnight

December 22, 2008

An RAE with something for everyone, but exactly what?

by Jonathan Adams

The RAE2008 results have produced more information about the spread of the UK’s research achievement than we could have hoped for back in the 1980s.  The profiling that Gareth Roberts proposed has overcome some of the worst deficiencies of the old grading system.  But it is pretty clear that the data avalanche has buried a lot and it will take a while to dig ourselves out.

Evidence planned to get a post on the blog by Thursday lunchtime last week, and we failed.  We failed because, in the words of Morris Zapp, ‘every decoding is another encoding’.  Each time we ran a query we came up with new questions, and very few answers.

The best bit about the data is the profiling, and that is also the trip-wire for analysts.  Profiling is great because it shows ‘what lies beneath’.  For research, most activity is skewed.  Some people earn a lot of money, train a lot of post-grads and publish a lot of papers.  Most people are less Stakhanovite.  Most papers get a couple of citations, or none.  Some papers get hundreds of citations.  So, wherever you look, you see a distribution that is anything but normal and that means that indices that use averages are not telling you much about the data spread and the centre of the distribution.  The Roberts profile takes us away from those averages, and it takes us away from single grades with their terrible funding cliffs.

But how do you analyse a profile?  One profile is OK.  We can look and see the spread, and reflect on the balance of national and international and whether that means ‘good’ in our dictionary.

A set of profiles across one subject is even better – more work to do the comparisons but we can look up and down the list and see how things shift about.  Interesting to see that one institution got a lot of 4* material but actually got less on [4* plus 3*] than another institution.  Interesting to speculate on whether those top end 4* outputs are the most critical element or whether it is an overall grade point average that establishes a ranking.   And how would you weight the elements in this subject, and would it be the same in that subject?

And that is what makes it very difficult when you start to try and create a combined picture across an institution, and then to put institutions into a single table.  We work on the data and start to pick out some strange differences.  Can it really be true that Anthropology, History of Art, Music and Drama all really have more than 20% of their output at 4* while Education, Psychology and Agriculture are under 10%?  In media subjects and the arts we get 4* values as high as 65% ‘international leading’.  I’m very happy for the institutions picked out but I do not really believe it.  That means two-thirds of the activity was at the international cutting edge and that is an almost infeasible standard to meet.

Biology, Chemistry, Physics and Mechanical Engineering are in the low teens.  Why so different?  Two things.  One is a much greater familiarity with the concept of categorising a portfolio of evidence about research activity.  It’s a pretty basic part of science culture but it doesn’t come so easily to humanities and arts (check the scars on the Warden of Goldsmiths if you don’t believe me).  So, for scientists, this is ‘what we do’.

The second thing is dialogue and consensus.  Running through 2007, at a lot of meetings and on visits to institutions, I heard people talking about what they thought the RAE2008 outcome would be like.  The view, which became a common one, was that selective submission meant few 1* items, a lot of 2*, not too difficult to hit 3* but bloody difficult to get a 4* except on the very best.   I thought that view was universal but I now think it was much more the view of scientists, and of scientists in pre-92 universities, than it was of academia as a whole.  But because the dialogue was going on, the scientists’ and the engineers’ consensus was reflected fairly consistently in the outcomes for the bigger subject areas.

More thoughts follow, but one thing is clear.  Any league table you create now may look pretty shaky by Twelfth Night.

Jonathan Adams

Director, Evidence Ltd

June 04, 2008

You are the REF goes live

by Jonathan Adams

Have you worked out what the Research Excellence Framework means for you? Not easy, is it?

Even after the consultation by the Higher Education Funding Council for England and the announcements from the Department for Innovation, Universities and Skills, most people are still wondering “What do the numbers mean in my subject? What is HEFCE actually going to include? What happens when the quality indicator meets the data on funding and research students?”

You are the REF lets you play with the kind of model HEFCE might come up with, using the best data available now. There are three basic indicators, the three that HEFCE has said that the REF will be structured around: research funding, research student training and research publications (and their citation impact). Department by department, You are the REF lets you decide how those indicators should be combined into a single final score, and then see where your university comes in the league table. Then you can tweak your settings and see what the impact is. It’s an invitation to play with your destiny.

The kind of question you might ask yourself is, “Is funding a good indicator of who is cutting the mustard?” If so, you would give funding a high weighting compared with that of the other variables. But research funding comes from more than one source, and maybe you don’t value all those sources equally. Is a research council grant as good a measure of research quality, for your discipline, as money from industry? You can weight them differently.

You are the REF doesn’t cover all subjects; the data in some areas are coarse grained or almost absent. (That in itself is interesting: it tells you which subjects are going to be a challenge for HEFCE to map as well.)

Is this useful, or is it only fun? We think that the important point is to let a lot of people find out what makes sense for them. If you cannot manipulate the numbers yourself, it’s difficult to join the debate. And what is important for the UK research base is that the people who decide what happens to research funding should be the people who turn that money into knowledge.

The researchers must decide what the model should look like, not the bibliometricians. We would like to know which version of the model you prefer, and we might even let HEFCE know what you think when You are the REF...

May 28, 2008

Welcome to the ‘You Are The REF’ blog

by Jonathan Adams

The site is open for business and you have no doubt been trying some scenarios to see if they make sense.

Playing with the data may sound like an odd suggestion for me to make. After all, isn’t using data very seriously our business? But we know that you need to see how the data ‘behave’. So, push the envelope and see if the model still produces a meaningful outcome. Try a tweak and find out how quickly it looks like garbage.

The debate about the future of UK research assessment will be a lot more productive if lots of people have a good feel for how the possible indicators might work, how they can be manipulated and what works for them.  For the engineers we are fairly sure that the answer will look different from the biologists’ version.

The main site won’t be launched until next week and I will be posting a longer piece then. In the meantime, please post your comments here.

You Are the REF, so feel free to have fun.

Jonathan Adams
Evidence Ltd

October 10, 2007

Can Britain take the pace in the global talent race?

by Jonathan Adams

The lasting impact of David Sainsbury’s report, The Race to the Top: A review of government’s science and innovation policies, published last week, will depend critically on three things. Is the picture of the UK’s economy right? Because if it isn’t then the solutions won’t match future needs. Is the picture of global knowledge constraints right? Because if it isn’t we will be out of sync with action elsewhere. And, are the disparate parts of government going to respond in a constructive and supportive way? Because many of the outcomes depend on those disparate parts getting their collaborative act together.

Many people believe that the UK has a history of great invention but poor application—a deficit in knowledge transfer. Sainsbury says that “in a number of critical areas we are doing better than is commonly thought” and “there has been a dramatic increase in KT from British universities”. That’s good. Creaky, oak-panelled mechanisms are getting slicker.

But, outside, we have a global war for talent, identified by studies such as the US National Academies’ report in 2005, Rising Above The Gathering Storm: Energizing and employing America for a brighter economic future.  Stars are rising in the East. Other countries see the risks of a dearth of competent science-literate professionals. And some of the responses are massive. The US Competitiveness Initiative commits $5.9 billion (£2.9bn) in 2007 to increase research investments, encourage entrepreneurship and build school education. The total budget request for R&D in the US this year is $137bn, more than 50 per cent up on 2001. Compare that with what’s on offer in the UK.

The focus across the globe is partly about knowledge creation but mostly about its use. KT mechanisms are important but you can source knowledge all over the place. Bright and able people are the best way of transferring and exploiting knowledge. The single most important constraint is the supply of those people.

Back home, Sainsbury notes two cross-cutting performance issues: the different requirements of manufacturing and services; and the role of HEIs in supporting a diversity of excellence. The UK economy is low on manufacturing, which is a mismatch with our research strengths in sciences. We are much stronger in the knowledge-based industries, including services and financials, which appear to be light on R&D. What this means is that measures that address traditional research throughput to manufacturing are only partially effective across the value base of the economy. What would most benefit the KBIs is, wait for it, a strong supply of bright and able people. That puts us smack centre in the global talent war.

The review also addresses that third question, the weaknesses that lead the disparate parts of government to go their own way with little sharing and coordination. Whitehall has not been a learning organisation when it comes to KT. But the proposed simplification and strengthening of structures and processes will concentrate resources and make life easier for entrepreneurs.

So why is it difficult to feel enthusiastic? It’s because these ideas are a bit mechanistic. Sure, we need the right structures and processes, not least because the wrong ones are barriers that drag innovation down. But the right ones work only when there are people to work them.

For me, the most important section is Chapter 7: Educating a new generation of young scientists and engineers. This chapter is about the supply of people trained in science and engineering. It is about better incentives, training and support for school teachers to increase the supply of talented young people to higher education. That is the most important thing. It is no good providing advice about future careers to people disaffected by the present. We need more, better teachers working in enriched facilities with a decent, stimulating curriculum freed from the dead hand of 20th century science.

People will win the race to the top, not organisations. The rest of the review is about clearing the track; Chapter 7 is about putting the athletes in place. Sainsbury sees what needs to be done to match other countries, I believe. If government departments respond, and manage their infrastructure, then a significant boost to the UK’s innovation strengths is possible—so long as we allow the next generation of innovators a good run-up. People who entered secondary school in September will do their GCSEs in 2012. It takes time to get a change in the supply of talent—and the sooner that the government acts on Sainsbury the better.

Jonathan Adams

Jonathan Adams is a director of Evidence, which specialises in research performance analysis.