I am sometimes asked about impact case studies (ICS) which are underpinned by a single research output. Is this risky? Do I have any examples? My answer is generally that they exist and this is completely within the rules (case studies must “include references to one or more key research outputs” and underpinning research “may be the output(s) of a particular project”). But can I point to any particular examples? No. And are single output ICS a risky proposition? I don’t know.
Last week when I received another query about this, I wondered if I could give better answers. So I did some analysis of the REF2021 impact case studies.
My method
- Downloaded the spreadsheet of all REF2021 ICS.
- Used a formula in Excel to count the line breaks in the “3. References to the research” column for each ICS. The number of line breaks loosely indicates the number of items in the refs to the research section.
- Starting with ICS with the least line breaks in the refs to the research column, I manually counted the number of outputs for each ICS. The manual count was needed due to the variation in how authors formatted their case studies, e.g. some listed multiple outputs one after the other without any line breaks and some had extra line breaks where they provided additional information. I did this diligently for ICS with up to 8 line breaks. Those with 8 line breaks all had at least 2 outputs and a quick look at those with 10 line breaks said they had at least 2 outputs too. So I think I reached saturation.
- I then went through and noted the scores of the submissions to which these single output ICS belonged.
And what did I find?
- Here is my final list with scores REF 2021 Impact Case Studies – single output
- I found 28 single output ICS in the REF2021 database. There are 6361 ICS in the database (out of a total of 6781 submitted so a few hundred aren’t publicly available). So 0.44% of the publicly available 6361 ICS are single output.
- These were unevenly spread across main panels and three times more likely to appear in main panels C and D combined than in A and B combined (though the small numbers may mean this isn’t significant). See Figure 1 below.
- Scores look pretty good. It’s hard (for me) to analyse this robustly against multiple output ICS but 14 of the single output ICS sat within submissions that scored 3*-4* overall and another 5 were in units that scored mostly 3*-4* with a small % of 2*. There was nothing glaring about the overall profile of scores.
- 11 single output ICS (that’s 39% of them) were from Russell Group universities. This is in line with Russell Group representation in the REF2021 ICS database as a whole (2723/6361 = 42%). So there doesn’t seem to be anything significant in the type of university that submitted single output ICS.
Figure 1 – number of single output REF2021 ICS by main panel
Conclusions
Can I now point to examples of single output ICS? Yes I can. Here they are (same link as above), with scores and from a range of disciplines and universities.
Are they a risk? I don’t know. That’s for you to decide. From my point of view I’d say no, there is clearly no distinction in the rules or criteria for single output ICS and there doesn’t seem to be any obvious deviation in overall quality/score. There may be something in the disciplinary split – single output ICS are three times more likely to appear in AHSS disciplines than STEM but that may well be to do with publishing patterns in the different areas rather than the merit of any impact case study.
The sheer small number of single output ICS may make some of you think that they are outliers and therefore inherently risky. To try and contextualise the small number of 28/6361, I counted how many ICS scored U and I came to 26 – a remarkably similar number and a set of outliers to which no one wants to belong. Make of that what you will…
We also need to bear in mind that these are from REF2021, which had a 2* threshold for research underpinning impact case studies so there may have been a natural tendency for universities to err on the side of caution when it came to (not) submitting single output ICS. In REF2029 the 2* threshold is removed for the express purpose of encouraging universities to submit their best impacts, whatever the nature of the research. So does this open the field more for single output ICS?
To finish, I would always advise authors of single output ICS to beef up their underpinning research where possible. I think most people would say the same. But there is certainly room for single output ICS.
What do you think?
