Author: Stephen Kemp

REF impact case studies need to include evidence to corroborate claimed impact. This evidence can take many forms, quantitative or qualitative. Statements from research users, partners and beneficiaries can be a powerful form of evidence, both quantitative and qualitative, as seen in REF 2014 and REF2021.

An analysis of REF2014 impact case studies (Collecting Research Impact Evidence, 2016) showed that testimonials were the most commonly used evidence sources in all areas except in Main Panel A where they were the second most popular form of evidence after “reports” (which makes sense as medical-type impacts are often described in quantitative terms and formal evaluation is important).

There is the temptation to wait until closer to the REF2029 deadline in 2028 to start thinking about/gathering testimonials but it is a better idea to start at least planning this aspect now. What testimonials might you need, who can provide them and what will your approach be? This guide (a significant update on my 2018 post) will go through some key points including what makes a strong testimonial, what to include, how to gather them and a comparison of some indicative weaker vs stronger examples from across sectors/impact types.

Why are testimonials important?

1. Testimonials verify the link between the underpinning research and the impact. This attribution or contribution is important as it can be hard to show definitively in other ways.
2. Testimonials corroborate or confirm your impact claims. You may be able to state what changed but testimonials back this up in a way that other sources don’t.
3. Strong testimonials include information on the reach and significance of the impact. What the change was, what was the resulting benefit or outcome, who was affected and why this matters. This description is best presented as a mix of quantitative and qualitative information. Broadly, quantitative information shows an effect. The qualitative aspect describes the importance of this effect.
4. Following on from the previous point, while other sources may show a change or effect, testimonials can put this into context. Consider a 1% decrease in power consumption of some industrial process. Is this a significant impact? It’s impossible to say unless you (or the reader/panel member) know the context! Normally you might say 1% is in the noise and therefore it isn’t even clear if a change has happened, let alone whether it is significant. But a testimonial has the scope to provide context, maybe in this case that the industry partner usually considers a 0.1% improvement to be a big step (with an example where possible). Against this backdrop, the 1% improvement based on your research is a massively significant advance! The testimonial could then give further detail about the significance of this, e.g. across the global operations of this organisation, the change means an £XXXX saving in costs and a reduction of XXXX tonnes of CO₂.
5. A testimonial may well include information you can’t get elsewhere. For example, a partner may be the only organisation that gathers the required kind of data or performs regular monitoring/evaluation. They may provide proprietary or confidential information that you can’t get anywhere else (if you do need to use confidential information, REF has a number of measures to make this possible and reassure partners – see section 11 in the REF2029 guidance). Even where data is publicly available, it may be cleaned, sanitised, only published periodically, etc whereas a testimonial has the potential to include fuller, richer, tailored, up to date information.
6. Testimonials usually come from someone with authority, responsibility or ownership of an organisation, process, service, etc. This gives credibility to the corroboration and impact information.

Features of a strong testimonial

• Specificity – exactly what changed, when, and how
• Link to research – researcher and study clearly identified, pathway described
• Significance – details of changes in policy, practice, behaviour, cultural, economic, etc
• Reach – details of who benefited: people, departments, audience, geographic scope, etc
• Credibility – written by senior staff or influential beneficiaries

What to include – a checklist

In general testimonials should:

• Be written on the external organisation’s headed paper (or a suitably professional-looking email).
• Be signed by someone at an appropriate level. This will vary by case study but considerations should include seniority vs connection to the research (e.g. should the statement come from your direct contact, the person with responsibility for the area or the head of the organisation?), maturity of relationship, reputation and conflicts of interest.
• Name the researcher and refer to the research (could be in descriptive terms, citation of a research output, name of research programme – whatever comes across as most fitting).
• Describe how the organisation “found” or interacted with the research/researcher.
• Describe how the work fits with the organisation’s activities, strategy, needs, challenges, opportunities and other drivers.
• Describe how the research/knowledge/skills were put into action or used, e.g. did the organisation work with the academic (maybe through commissioned research, consultancy, knowledge transfer grant, advisory work, other joint activities), did they use the research in their decision making, did they train their staff according to the research?
• Describe what happened as a result of using the research/knowledge/skills or working with the researcher, e.g. did they produce guidelines for practice, were they better informed in making strategic decisions, was their service provision directly improved?
• Describe the resulting impact of this work. What was the benefit of using the research/knowledge or working with the researcher? Include quantitative or qualitative indicators to show the impact. That is, how they know the change was beneficial. They could also say where they’d be if they hadn’tused the research. This is the most important parts of the statement as it’s where the impact is really articulated (and any quantitative/qualitative evidence the organisation provides can be quoted in the case study and woven into the narrative).
• Describe who benefitted, e.g. people, departments, audience, geographic scope, etc
• Describe the context, both why this chance matters and the organisational context such as the scope and scale of their activities.
• Include dates. This is important as REF will only credit impact that occurred from 1 August 2020 to 31 July 2028. Any vagueness about when things happened means panels may not recognise the associated impact.
• Say something about the future – what’s next in this line of work? Do they foresee continued and growing benefits? Will they work with the researcher again? Will they be more open to using academic research in the future? Maybe they’ll change the way they operate as a result of the impactful piece of work.

How to gather testimonials

The first (and by far the best) option is to put the above list into your own words (so it doesn’t sound so much like a checklist) and use this to prompt a statement from your partner. You could either put it in writing and let them respond accordingly or you could use this as the basis of a conversation/interview. The beauty of the latter approach is that you can explore and clarify and it may uncover other relevant information.

Secondly, you could use the above list as a checklist/questionnaire. I wouldn’t recommend this approach. It may save time but you are unlikely to get the richness or authenticity of a more personal/tailored interaction. It won’t strengthen your relationship with the organisation and it may even damage it.

Finally, in some cases it may be necessary to essentially write the statement yourself and hand it over to the partner to sign. This is not recommended as you lose the authentic voice and you may miss some aspect of what made the work so valuable (including possibly some extra information the partners would have included if they’d had to write it themselves). On top of this there is a very real risk that if the academic writes a number of such letters for different partners to sign, they could all end up looking fundamentally the same which undermines credibility and authenticity. Consider how you’d view this as an assessor…

Other considerations

• Make sure the person giving the statement knows what it is for and has the authority to give it.
• Observe the relevant data management and ethics policies as you gather, hold and use this information.
• In previous REF exercises, some organisations were overwhelmed with requests for testimonials to the point where relationships were affected and in some cases they simply refused to provide testimonials. This is where strong relationships really count so focus on lasting, rather than superficial, interactions with stakeholders and partners.
• Some organisations will be concerned about confidentiality. There are a number of provisions for this. Panel members under confidentiality obligations as standard. If there is a conflict of interest with any individual involved in the assessment, you can alert the REF team and the information will not be made available to them. There is the facility to redact information from case studies when they are made publicly available on the REF impact database. Or they could be completely withheld from the database and not published publicly at all. See section 11 in the REF2029 guidance.
• Don’t wait to get testimonials. The details, nature and value of impacts may become forgotten over time so jump on them while they’re fresh. Plus you never know where people will be in a couple of years.

Using testimonials

The main thing to remember here is that REF panel members do not routinely read the corroborating sources themselves. They will look at a small sample of sources for verification purposes only, i.e. is the evidence really what the author is claiming? For the actual assessment, panel members have to rate the impact case study as it is presented to them. Impact case studies must be self-contained. This means that any important information in the sources must be included in the case study itself, usually in the “Details of the impact” section. Include quantitative information in your description of the impact and use direct quotes where necessary, for example to explain the significance of a change.

 (Fictitious) Examples of weaker vs stronger testimonials

Note: these examples would be the central elements of testimonials. The whole testimonial will be in a formal latter/email format with introduction, signoff, etc.

1. Industry/commercial impact:

• Weaker: “Dr Zhang’s work on sustainable materials has been very useful to our R&D team. We have incorporated some of her findings into our product development discussions, and her research continues to shape our thinking about greener production methods.”
• Stronger: “Between September 2020 and July 2022, Dr Zhang’s research on lignin-reinforced biodegradable polymers informed the development of our EcoPack™ packaging line.
  Her experimental data and material formulations were incorporated into our R&D process, leading to a reformulated packaging material that reduced petroleum-based polymer use by 35% while maintaining required strength and durability standards.
  EcoPack™ was launched in June 2022 and achieved EU compostability certification in April 2023. In its first 18 months on the market, it generated £8.6 million in revenue.
  Dr Zhang’s research was critical to the technical viability and regulatory compliance of this product line and continues to guide ongoing sustainable packaging development.

2. Policy impact

• Weaker: “Dr Alvarez’s research on urban transport policy has informed our city planning work. His recommendations were discussed at several policy meetings, and they have influenced how we think about congestion and public transport investment. His research continues to be part of our planning discussions.”
• Stronger: “Between March 2021 and June 2022, Dr Alvarez’s research on congestion management and sustainable urban transport directly informed the City Council’s Transport Strategy 2022–2027.
  His modelling of traffic flows and evaluation of low-emission zones was discussed in three formal policy workshops with senior planning officers and was incorporated into the Council’s decision to expand the Low-Emission Zone to cover five additional districts.
  Following implementation in September 2022, average peak-hour traffic congestion decreased by 12% across the newly included districts, as measured by city traffic sensors, and public transport usage in these areas increased by 8%.
  Dr Alvarez’s research was the principal evidence underpinning these changes and continues to guide the Council’s urban mobility planning.”

3. Third sector/NGO impact

• Weaker: “Professor Simmons’ research on youth engagement has informed our programme design. Her insights have been helpful for our staff and have shaped the way we run workshops. We continue to refer to her work when developing new projects.”
• Stronger: “Between February 2021 and November 2022, Professor Simmons’ research on youth engagement strategies informed the design of our “Youth Voices” programme across five community centres.
  Using her evidence-based framework, we revised workshop formats, introduced peer-led discussion sessions, and implemented structured feedback mechanisms. These changes directly impacted 412 young participants aged 14–18.
  Evaluation data collected in December 2022 show a 42% increase in participants reporting improved confidence in civic engagement, compared to the previous programme model.
  Professor Simmons’ research directly shaped these programme improvements and continues to guide the design of new youth engagement initiatives.”

4. Cultural/heritage impact

• Weaker: “Dr Rahman’s research on local migration history has informed the design of our exhibitions. Her work has helped us interpret our collections differently, and it continues to influence how we plan public engagement activities.”
• Stronger: “Between June 2021 and September 2022, Dr Rahman’s research on post-war South Asian migration informed the design of our exhibition “Arrivals and Afterlives: South Asian Stories in the City”, which ran from October 2022 to February 2023.
  Her work directly shaped the interpretive framework, guided the thematic structure of the galleries, and informed the inclusion of 43 newly recorded oral histories.
  The exhibition attracted 21,486 visitors, making it the second most visited temporary exhibition in our history. Visitor survey data (n=1,204) indicate that 68% of respondents reported an increased understanding of the history of local migration communities.
  Following the exhibition, we revised our Collections Development Policy (July 2023) to prioritise community co-curation and oral history acquisition in under-represented migrant histories.”

And finally…

I hope this is useful!

I am sometimes asked about impact case studies (ICS) which are underpinned by a single research output. Is this risky? Do I have any examples? My answer is generally that they exist and this is completely within the rules (case studies must “include references to one or more key research outputs” and underpinning research “may be the output(s) of a particular project”). But can I point to any particular examples? No. And are single output ICS a risky proposition in terms of how panels see them? I don’t think so and I don’t see why they should be but I can’t back that up.

Last week when I received another query about this, I wondered if I could give better answers. So I did some analysis of the REF2021 impact case studies.

My method

1. Downloaded the spreadsheet of all REF2021 ICS.
2. Used a formula in Excel to count the line breaks in the “3. References to the research” column for each ICS. The number of line breaks loosely indicates the number of items in the refs to the research section.
3. Starting with ICS with the least line breaks in the refs to the research column, I manually counted the number of outputs for each ICS. The manual count was needed due to the variation in how authors formatted their case studies, e.g. some listed multiple outputs one after the other without any line breaks and some had extra line breaks where they provided additional information. I did this diligently for ICS with up to 8 line breaks. Those with 8 line breaks all had at least 2 outputs and a quick look at those with 10 line breaks said they had at least 2 outputs too. So I think I reached saturation.
4. I then went through and noted the scores of the submissions to which these single output ICS belonged.

And what did I find?

• Here is my final list with scores REF 2021 Impact Case Studies – single output
• Identified 28 single output ICS in the REF2021 database. There are 6361 ICS in the database (out of a total of 6781 submitted – a few hundred aren’t publicly available). So 0.44% of the publicly available 6361 ICS are single output.
• These were unevenly spread across main panels and three times more likely to appear in main panels C and D combined than in A and B combined (though the small numbers may mean this isn’t significant). See Figure 1 below.
• Scores looked pretty good at first glance – there was nothing glaringly out of kilter about the overall profile of scores. This was backed up when I looked in a little more detail at the numbers. The average impact UoA GPA overall is 3.13. The average impact GPA for UoAs containing single output ICS is astoundingly… 3.14!  14 of the single output ICS sat within submissions that scored 3*-4* overall and another 5 were in units that scored mostly 3*-4* with a small % of 2*.
• 11 single output ICS (that’s 39% of them) were from Russell Group universities. This is in line with Russell Group representation in the REF2021 ICS database as a whole (2723/6361 = 42%). So there doesn’t seem to be anything significant in the type of university that submitted single output ICS.

Figure 1 – number of single output REF2021 ICS by main panel

Conclusions

Can I now point to examples of single output ICS? Yes I can. Here they are (same link as above), with scores and from a range of disciplines and universities.

Are they a risk? I don’t know. That’s for you to decide. My own point of view, as I said at the start, is I don’t see why they would be – there is no distinction in the rules or criteria for single output ICS.  And with the information we now have, there is no distinction in how panels saw them. On average a UoA containing single output ICS scores pretty much exactly the same for impact as the overall average UoA. There may be something in the disciplinary split – single output ICS are three times more likely to appear in AHSS disciplines than STEM but that may well be to do with publishing patterns in the different areas rather than the merit of any impact case study.

The sheer small number of single output ICS may lead some to think that they are outliers and therefore inherently risky. To try and contextualise the small number of 28/6361, I counted how many ICS scored U and I came to 26 – a remarkably similar number and a set of outliers to which no one wants to belong. Make of that what you will…

We also need to bear in mind that these are from REF2021, which had a 2* threshold for research underpinning impact case studies so there may have been a natural tendency for universities to err on the side of caution when it came to (not) submitting single output ICS. In REF2029 the 2* threshold is removed for the express purpose of encouraging universities to submit their best impacts whatever the nature of the research. So does this open the field more for single output ICS?

To finish, I would generally advise authors of single output ICS to beef up their underpinning research where possible. I think most people would say the same. But there is certainly room for single output ICS and there doesn’t seem to be any risk.

What do you think?

According to the Adam Smith Institute the “buy one, get one free” deal was pioneered by Josiah Wedgwood in the 1700s.  Despite being hundreds of years old, this concept is alive and well – and not only in supermarkets. In the past two REF exercises, researchers could get two (or more) impact case studies from one piece of work. How? Well, let’s check the guidance:

“104.c. Where a submitted impact is underpinned by collaborative research, each submitting unit whose research made a distinct and material contribution to the impact may submit that impact”
REF2021 Guidance on Submissions

Ok, excuse my silly introduction. This isn’t a secret voucher code to multiply your impact case studies. It’s a neat measure in REF to assess impact coming from  collaborative research. We don’t know whether this will still be a feature of REF2029 but nothing has been said otherwise so it would be reasonable to assume no change. Besides, if REF scrapped this provision then collaborations would succumb to in-fighting out over impact honours. All the work that’s been done to promote and enable collaborative/interdisciplinary research would be undone at a stroke. Well, there’d be misaligned incentives at the least. So this facility will probably remain.

Anyway, this is all leading up to another list of REF2021 impact case studies. Here we have pairs of case studies that are shared across UoAs or HEIs (yes! in the same way, if research from two or more HEIs has underpinned some impact, both HEIs can submit the same impact separately). I often get asked by authors, managers, REF leads, etc if I have examples of how shared case studies have been handled. Well, here are those examples.

I found them by looking for duplicated impact case study titles. Nothing clever. There are probably other shared case studies with different titles but these are obviously much harder to find so I don’t have any in my list.

First we have “impact case studies across UoAs” – these are probably of widest interest and there are 33 pairs.

Further down we have “impact case studies across HEIs”. There are only six pairs of these – and two of them are in the cross-UoA list too. Either there are really fewer examples or it’s less likely that a title will be duplicated across HEIs.

I’ve organised this in a way that makes sense to me – hopefully it makes sense to others too… You get the title, UoAs and HEIs. Clicking a dropdown reveals links to each of the joint case studies with an indication of score.

1. Impact case studies across UoAs

A new class of anti-infective drugs from DNA Minor Groove Binders
(UoAs 3 & 8, University of Strathclyde)

Abolishing Employment Tribunal Fees
(UoAs 16 & 18, University of Oxford)

Accelerating pharmaceutical manufacturing development using automated self-optimisation platforms
(UoAs 8 & 12, University of Leeds)

Active learning during primary school physical education in the West Midlands
(UoAs 23 & 24, Newman University)

Creating change by raising the awareness and understanding of the emotional needs of children in school as explained by attachment theory
(UoAs 20 & 23, University of Chester)

Decolonising the University Across Disciplines
(UoAs 19 & 34, School of Oriental and African Studies)

Electrical Impedance Tomography
(UoAs 3 & 11, Middlesex University)

Harnessing the draw of professional sports clubs to deliver improvements in health and wellbeing among at-risk groups
(UoAs 2 & 20, University of Glasgow)

HIV policy formulation and prevention: driving the decision for and implementation of publicly-funded pre-exposure prophylaxis in Scotland
(UoAs 2 & 20, University of Glasgow)

Horizon Proteins: Circular economy innovation from whisky by-product to fish feedstock
(UoAs 8 & 12, Heriot-Watt University)

How our electrical stimulation devices have improved long-term medical conditions
(UoAs 3 & 12, Bournemouth University)

Implementing high quality stratified care for low back pain
(UoAs 2 & 3, University of Keele)

Improved diagnosis and treatment of patients with lymphoma globally using Positron Emission Tomography (PET)
(UoAs 1 & 12, King's College London)

Improving COVID-19 symptom identification and infection spread in the UK with research-based App technology
(UoAs 1 & 3, King's College London)

Integrating Ethics into Policy and Practice for Issues on the Margins of Life
(UoAs 18 & 30, University of Keele)

Leeds Virtual Microscope: A New Tool for Pathologists Diagnosing Cancer
(UoAs 1 & 11, University of Leeds)

Managing Climate Risk and the Cost of Capital in Climate-Vulnerable Countries
(UoAs 17 & 22, School of Oriental and African Studies)

PARP inhibitors as a therapeutic to treat BRCA-defective cancers
(UoAs 1 & 5, University of Sheffield)

Pheromone Traps: providing new Tools across Europe for the Prevention of Spread of Pine Wood Nematode, an Invasive Disease of Pine Forests
(UoAs 6 & 8, University of Greenwich)

Pretext Drama: Creating ‘Third Spaces’ for Understanding
(UoAs 23 & 33, University of Chester)

Protecting pollinators: Influencing policy and retail to reduce and remove harmful pesticides
(UoAs 5 & 8, University of Sussex)

Raising the international quality of osteoarthritis management in primary care
(UoAs 2 & 3, University of Keele)

Raising the Minimum Wage in Greece
(UoAs 17 & 25, Brunel & LSE)

RECOVERY Trial: Global adoption of effective COVID-19 treatments to save lives
(UoAs 1 & 2, University of Oxford)

Reducing childhood unintentional injuries in the home
(UoAs 2 & 3, University of Nottingham)

Revitalising place through interdisciplinary art practice at a time of environmental change
(UoAs 27 & 32, Sheffield Hallam & Leeds)

Revolutionising the understanding of childhood food allergy to halt the epidemic
(UoAs 1 & 3, King's College London)

Science and Religion: Exploring the Spectrum
(UoAs 4 & 31, Newman University)

Shaping the future of robotic surgical systems for rectal cancer
(UoAs 1 & 2, University of Leeds)

Specialist cancer services improve outcomes for Teenagers and Young Adults with cancer
(UoAs 1 & 3, University of Leeds)

Supporting the development of a safer, more robust financial system for the Eurozone (Sovereign Bond Backed Securities)
(UoAs 16 & 17, London School of Economics and Political Science)

The 21st Century Public Servant: Changing the Public Service Workforce
(UoAs 19 & 20, University of Birmingham)

The Care and Management of Gout in Primary Care
(UoAs 2 & 3, University of Keele)

Transforming the development and manufacture of medicine through continuous processing and advanced manufacturing technology research
(UoAs 3 & 12, University of Strathclyde)

2. Impact case studies across HEIs

‘Taking Care’: Enhancing UK nursing training using mixed-methods drama research to reconceptualise, teach and promote embodied ‘care’ in clinical interactions
(UoAs 33 Music, Drama, Dance, Performing Arts, Film and Screen Studies, Kingston & Guildhall)

Consuming Prehistory
(UoA15 Archaeology, Cardiff & York)

Raising the Minimum Wage in Greece
(UoAs 17 & 25, Brunel & LSE)

Revitalising place through interdisciplinary art practice at a time of environmental change
(UoAs 27 & 32, Sheffield Hallam & Leeds)

Shaping Global Accounting Standards
(UoA17 Business and Management Studies, Bristol/LSE)

Structural and Regulatory Change in the Irish Credit Union Sector
(UoA17 Business and Management Studies, QUB/St Andrews)

That’s all for now. I hope this is useful. Please let me know if any of this is wrong, doesn’t work or you have other examples to add.

Not so hot on the heels of my other REF2021 impact case study spreadsheet-based lists (4* impact case studies from REF2021 and top 5 impact submissions in every UoA), here is a list of all impact case studies with known scores.

The impact case study database can be searched and filtered in a number of ways but (beyond sitting with a spreadsheet of REF results) there is no easy way to pick out examples to compare higher and lower scoring case studies. I’ve done the spreadsheet-sifting and the results are below. Each UoA dropdown reveals links to the case studies by score. These are only the case studies we can assign unambiguously – i.e. where the submission scored 100% 4*, 3*, etc. There other ways of inferring scores, for example in a submission that scored 50% 4* and 50% 2*, it would be feasible to infer which case studies scored 4* and which scored 2* but this would involve more time and insight – and you couldn’t know absolutely that that’s how the panel saw it .

1. Clinical Medicine

2. Public Health, Health Services and Primary Care

3. Allied Health Professions, Dentistry, Nursing and Pharmacy

4. Psychology, Psychiatry and Neuroscience

5. Biological Sciences

6. Agriculture, Food and Veterinary Sciences

7. Earth Systems and Environmental Sciences

8. Chemistry

9. Physics

10. Mathematical Sciences

11. Computer Science and Informatics

12. Engineering

13. Architecture, Built Environment and Planning

14. Geography and Environmental Studies

15. Archaeology

16. Economics and Econometrics

17. Business and Management Studies

18. Law

19. Politics and International Studies

20. Social Work and Social Policy

21. Sociology

22. Anthropology and Development Studies

23. Education

24. Sport and Exercise Sciences, Leisure and Tourism

25. Area Studies

26. Modern Languages and Linguistics

27. English Language and Literature

28. History

29. Classics

30. Philosophy

31. Theology and Religious Studies

32. Art and Design: History, Practice and Theory

33. Music, Drama, Dance, Performing Arts, Film and Screen Studies

34. Communication, Cultural and Media Studies, Library and Information Management

I hope this is useful. If you have any issues or spot any mistakes, please leave a comment or get in touch.

Thanks to David Steynor (currently Research Impact Manager at Queen’s University Belfast) for his original “all known scores” spreadsheet of REF2014 impact case studies. I’ve effectively just updated that here.

This post follows on from my recent list of 4* impact case studies from REF2021. As I acknowledged there, there aren’t unambiguously identifiable 4* impact case studies in all the UoAs. What if it’s one of these UoAs that you’re interested in? That’s the point of this post – to list the top 5(-ish) impact submissions in every UoA, including those where there were no clear 4* examples.

The impact case study database can be searched and filtered in a number of ways but (beyond sitting with a spreadsheet of REF results) there is no easy way to pick out the highest scoring examples. That’s where the list below comes in – click on the UoA of interest and you’ll uncover links to the highest scoring (publicly available) impact submissions in that area. The number in brackets is each institution’s impact GPA (scored out of 4). You’ll notice that many of my UoA lists below actually have more than a top 5 – this is where the last GPA is held by more than one institution.

1. Clinical Medicine

2. Public Health, Health Services and Primary Care

3. Allied Health Professions, Dentistry, Nursing and Pharmacy

4. Psychology, Psychiatry and Neuroscience

5. Biological Sciences

6. Agriculture, Food and Veterinary Sciences

7. Earth Systems and Environmental Sciences

8. Chemistry

9. Physics

10. Mathematical Sciences

11. Computer Science and Informatics

12. Engineering

13. Architecture, Built Environment and Planning

14. Geography and Environmental Studies

15. Archaeology

16. Economics and Econometrics

17. Business and Management Studies

18. Law

19. Politics and International Studies

20. Social Work and Social Policy

21. Sociology

22. Anthropology and Development Studies

23. Education

24. Sport and Exercise Sciences, Leisure and Tourism

25. Area Studies

26. Modern Languages and Linguistics

27. English Language and Literature

28. History

29. Classics

30. Philosophy

31. Theology and Religious Studies

32. Art and Design: History, Practice and Theory

33. Music, Drama, Dance, Performing Arts, Film and Screen Studies

34. Communication, Cultural and Media Studies, Library and Information Management

I hope this all makes sense and helps you find some useful impact case study examples, whatever your area. Any issues, just leave a comment or get in touch!

We all know there’s more to impact than REF impact case studies. However, they represent a lot of hard work and the new REF2021 database of 6,781 impact case studies is so big and wide-ranging that it’s a major resource for anyone interested in research impact. A look at these impact case studies can help academics and research managers understand what impact may look like in a particular area, plan and articulate impact, scope relevant stakeholders and engagement strategies, and consider what evidence and indicators might be relevant when describing different kinds of impact.

The database can be searched and filtered in a number of ways but (beyond sitting with a spreadsheet of REF results) there is no easy way to pick out the highest scoring examples. That’s why many REF and impact managers in universities across the UK will be making lists of 4* impact case studies from REF2021. Here’s mine – click on the UoA of interest below and you’ll uncover links to the (publicly available) 4* impact case studies in that area. These are case studies that are unambiguously identifiable as scoring 4* – there are many other 4* examples but they are buried in submissions.

Of course, it is highly reductionist to focus on these 246 publicly available, identifiable 4* case studies – they represent less than 4% of the whole REF2021 impact case study database – but I know (from my own work with academics and research managers) that this list is a useful entry point to exploring the database more widely.

It’s worth noting that not every UoA had unambiguously identifiable 4* case studies. In case the UoA you are interested in doesn’t appear below, watch this space because I’ll be putting together an accompanying list of the top 5 impact submissions in every UoA.

2. Public Health, Health Services and Primary Care

4. Psychology, Psychiatry and Neuroscience

6. Agriculture, Food and Veterinary Sciences

8. Chemistry

10. Mathematical Sciences

11. Computer Science and Informatics

13. Architecture, Built Environment and Planning

14. Geography and Environmental Studies

15. Archaeology

18. Law

19. Politics and International Studies

20. Social Work and Social Policy

21. Sociology

22. Anthropology and Development Studies

23. Education

24. Sport and Exercise Sciences, Leisure and Tourism

25. Area Studies

26. Modern Languages and Linguistics

27. English Language and Literature

28. History

29. Classics

30. Philosophy

31. Theology and Religious Studies

32. Art and Design: History, Practice and Theory

33. Music, Drama, Dance, Performing Arts, Film and Screen Studies

34. Communication, Cultural and Media Studies, Library and Information Management

I hope this all makes sense and helps you find some useful impact case study examples, whatever your area. Any issues, just leave a comment or get in touch!

Earlier in the year I set out a list of 4* REF2014 impact case studies to make it easy to find examples of good practice.

It’s a useful resource but there is a snag: 11 of the 36 UoAs have no obvious 4* impact case studies so it leaves something to be desired if (a) you’re looking for good examples in UoAs…

• 7. Earth Systems & Environmental Sciences
• 8. Chemistry
• 9. Physics
• 10. Mathematical Sciences
• 11. Computer Science & Informatics
• 12. Aeronautical, Mechanical, Chemical and Manufacturing Engineering
• 15. General Engineering
• 19. Business & Management Studies
• 21. Politics & International Studies
• 31. Classics
• 33. Theology and Religious Studies

… or (b) you want to look at a wider selection of high-scoring case studies than the limited view the 4* list gives you (for example, there’s only one case study for UoA34 on the 4* list).

That’s why I’ve put together a new list setting out the top 5 impact submissions in each UoA along with their scores. Now you can find high-scoring examples in any UoA.

As before, my starting point is that a browse of high-scoring REF2014 impact case studies can help authors articulate their own impact, get them over the tyranny of the blank page, provide inspiration around evidence and indicator types, and so on. Though things may have moved on in REF2021, the REF2014 database is still our best resource when it comes to writing REF impact case studies. The REF results did not attach scores directly to impact case studies so the easier it is for authors and impact geeks to find relevant high-scoring examples, the better.

Click on the UoA of interest below and you’ll uncover the top 5 impact submissions in that UoA with scores (out of a possible 4.00, obviously) and links to the REF2014 results site where you can access the impact case studies for each submission.

Note: for UoA 17 which unhelpfully combines Geography and Archaeology, I have split these into Geography-only and Archaeology-only top 5s.

Top five impact submissions by REF 2014 UoA

1. Clinical Medicine

2. Public Health, Health Services and Primary Care

3. Allied Health Professions, Dentistry, Nursing and Pharmacy

4. Psychology, Psychiatry and Neuroscience

5. Biological Sciences

6. Agriculture, Veterinary and Food Science

7. Earth Systems & Environmental Sciences

8. Chemistry

9. Physics

10. Mathematical Sciences

11. Computer Science & Informatics

12. Aeronautical, Mechanical, Chemical and Manufacturing Engineering

13. Electrical and Electronic Engineering, Metallurgy and Materials

14. Civil and Construction Engineering

15. General Engineering

16. Architecture, Built Environment and Planning

17. Geography, Environmental Studies and Archaeology

17. Archaeology only

17. Geography only

18. Economics and Econometrics

19. Business & Management Studies

20. Law

21. Politics & International Studies

22. Social Work and Social Policy

23. Sociology

24. Anthropology and Development Studies

25. Education

26. Sport and Exercise Sciences, Leisure and Tourism

27. Area Studies

28. Modern Languages and Linguistics

29. English Language and Literature

30. History

31. Classics

32. Philosophy

33. Theology and Religious Studies

34. Art and Design: History, Practice and Theory

35. Music, Drama, Dance and Performing Arts

36. Communication, Cultural and Media Studies, Library and Information Management

Things to bear in mind

1. The score for each submission includes the impact template as well as the impact case studies. This can throw up some odd-looking results, for example in UoA5, St Andrews is 2nd in the top 5 (with an overall impact score of 3.90 – including the impact template) but it also appears in the 4* list as the only UoA5 submission to get 4* for all impact case studies!
2. Just because a submission has a score at the top end of the 3s, you can’t count on all the case studies therein being 3* and 4*. For example, Imperial’s UoA12 submission comes 2nd in class with a score of 3.69 which looks great. However, the impact profile (detailed breakdown of scores – available by clicking “view submission” in the breadcrumb trail of any of the linked REF submission pages) shows that one of the case studies in this submission scored 1.5… It just so happens that the average was buoyed by all 17 other case studies scoring 4*! It should be possible to spot the 1.5 among all the 4s but my point is you can’t always select a high-scoring submission and be sure everything is within it is high-scoring (a bit like journal impact factors…). This isn’t a common issue – I had to search for this example – but it does come up so check those impact profiles if you want to ensure everything is 3* and 4*. The impact profiles can be tricky to interpret – if in doubt ask a friendly REF person to help you decode them! Just to complete this story, all four other submissions in the UoA12 top 5 contained exclusively 3* and 4* case studies.

I hope this all makes sense and helps everyone find some good impact case study examples, whatever your area. Any issues, just leave a comment or get in touch!

(2026 updated/expanded version here)

The REF impact assessment requires impact case studies to include evidence of impact. This evidence can take many forms, quantitative or qualitative. Statements from research users, stakeholders and beneficiaries can be a powerful form of qualitative evidence, as seen in REF 2014 (see here for an attempt to analyse links between evidence types and impact case study scores). For the next REF we have the opportunity to collect these statements as we go along and this guide should facilitate such statement-gathering.

What to include

In general statements should:

1. Be written on the external organisation’s headed paper (or a suitably professional-looking email).
2. Be signed by someone at an appropriate level. This will vary by case study but considerations should include seniority vs connection to the research (e.g. should the statement come from your direct contact, the person with responsibility for the area or the head of the organisation?), maturity of relationship, reputation and conflicts of interest.
3. Name the researcher and refer to the research (could be in descriptive terms, citation of a research output, name of research programme – whatever comes across as most fitting).
4. Describe how the organisation “found” the research/researcher.
5. Describe how it fits with the organisation’s activities, strategy, needs, challenges, opportunities and other drivers.
6. Describe how the research/knowledge/skills were put into action or used – e.g. did the organisation work with the academic (maybe through commissioned research, consultancy, knowledge transfer grant, advisory work, other joint activities), did they use the research in their decision making, did they train their staff according to the research?
7. Describe what happened as a result of using the research/knowledge/skills or working with the researcher – e.g. did they produce guidelines for practice, were they better informed in making strategic decisions, was their service provision directly improved?
8. Describe the resulting impact of this work – what was the benefit of using the research/knowledge or working with the researcher? Include quantitative or qualitative indicators to show the impact – i.e. how they know it was beneficial. They could also say where they’d be if they hadn’t used the research. This is the most important parts of the statement as it’s where the impact is really articulated (and any quantitative/qualitative evidence the organisation provides can be quoted in the case study and woven into the narrative).
9. Say something about the future – what’s next in this line of work? Do they foresee continued and growing benefits? Will they work with the researcher again? Will they be more open to using academic research in the future? Maybe they’ll change the way they operate as a result of the impactful piece of work.

How to gather testimonials

The first (and by far the best) option is to put the above list into your own words (so it doesn’t sound so much like a checklist) and use this to prompt a statement from your partner. You could either put it in writing and let them respond accordingly or you could use this as the basis of a conversation/interview. The beauty of the latter approach is that you can explore and clarify and it may uncover other relevant information.

Secondly, you could use the above list as a checklist/questionnaire. I wouldn’t recommend this approach. It may save time but you are unlikely to get the richness or authenticity of a more personal/tailored interaction. It won’t strengthen your relationship with the organisation and it may even damage it.

Finally, in some cases it may be necessary to essentially write the statement yourself and hand it over to the partner to sign. This is not recommended as you lose the authentic voice and you may miss some aspect of what made the work so valuable (including possibly some extra information the partners would have included if they’d had to write it themselves). On top of this there is a very real risk that if the academic writes a number of such letters for different partners to sign, they could all end up looking fundamentally the same which undermines the credibility of the messages. Consider how you’d view this as an assessor…

Other considerations

• Make sure the person giving the statement knows what it is for and has the authority to give it.
• Observe the relevant data management and ethics policies as you gather, hold and use this information.
• In REF2014, some organisations were overwhelmed with requests for testimonials to the point where relationships were affected and in some cases they simply refused to provide testimonials. This is where strong relationships really count so focus on lasting, rather than superficial, interactions with stakeholders and partners.
• Some organisations will be concerned about confidentiality. Although we don’t currently know how this will work for impact case studies and evidence in REF2021, there were provisions for this in REF2014 so we can expect similar in REF2021.
• Don’t wait to get testimonials. The details, nature and value of impacts may become dull with time so jump on them while they’re fresh. Plus you never know where people will be in a couple of years.

Examples

These statements are not only a source to be referenced – you can use choice quotes to tell the story and illustrate the outcomes. Here are some examples from REF2014 showing the kinds of things that were said in testimonials and how they can be weaved into the narrative. They all happen to be from UoA 22 (social work and social policy) just because they were handy. You will find similar examples across the UoAs.

1. From “Improving evidence-based policy and programming for AIDS-affected children in Sub-Saharan Africa“:

​​”This is evidenced by a comment from a Regional HIV and AIDS Advisor at Save the Children: `By sharing rigorous evidence on how children affected by HIV faced increased vulnerabilities over time, Dr Cluver has engaged major policy makers, donors and program implementers in critically “rethinking” and redesigning programs with an emphasis on effectively measuring results. Based on Dr Cluver’s research, UNICEF, PEPFAR have launched new technical guidance (UNICEF: From Evidence to Impact; PEPFAR: OVC Guidance: 2012) for program implementers. Her work is a testimony of how rigorous research is the foundation for effective programming…a clear example how a true partnership between researchers, policy makers and implementers can result in programs that actually make a difference in children’s lives.’ [C9] A senior advisor for PEPFAR also stated: `I have quoted Dr Cluver’s data more than any other research to support our work and also to justify and set policy for our global portfolio. All of the data presented from these studies has been influential, however of particular influence has been the data discussed in Cluver [et al] (2011) [R2] and 2012.’ [C10]”

2. From “Reducing child anti-social behaviour through effective parenting interventions: international impact on policy, practitioners and families“:

“As explained by the then Deputy Director of the Social Exclusion Task Force at the Cabinet Office [C2], who played a major part in driving New Labour policies on early parenting interventions in the late 2000’s: “Support from the Start” and Gardner’s UK trials [R1-3] were highly influential in creating momentum towards these major policy developments, including the very substantial roll out led by the National Academy of Parenting Practitioners (NAPP) from 2008.

​The founder and research director of NAPP [C3] adds to this: Gardner’s research was pivotal in helping persuade the Cabinet Office to set up NAPP. Her trial [R1] was the first in the world to show that parenting programmes could be effective in reducing severe antisocial behaviour, outside the narrow confines of child mental health services, showing that the voluntary sector could do just as good a job with difficult cases. Influenced by these findings, the government awarded £35 million to NAPP to disseminate evidence-based parenting programmes across the voluntary sector; this is estimated to have benefited over 150,000 children; Gardner was an important member of its steering group to ensure this research was implemented.

Gardner’s team’s research is frequently cited in influential systematic reviews, guidelines, and policy documents [see C4-7 for Cochrane, Campbell and other reviews]. For example, four of the trials [R1-6] have been cited in the NICE Guidelines (National Institute of Health and Clinical Excellence); the Chair confirms their policy influence [C3]: As Chair of the NICE Guideline on antisocial behaviour and conduct disorders, I oversaw the marshalling of the evidence which has to be relevant to British practice, and Gardner’s work was important in shaping the recommendations, since it showed that the interventions work in Britain — not all psychosocial interventions developed in the USA do this. Their work also added to the plausibility of the recommendations since it demonstrated that the mediating mechanism was an increase in positive parenting.”

3. ​​​​From “Situational crime prevention policy and practice“:

“A letter from the Home Office states: “The work of the Department and its advocacy for, and expertise in, situational crime prevention strategies are well known by officials responsible for Crime policy within the Home Office and to the experts within my own area, Home Office Science, whose task it is to ensure that we provide the best evidence on which to base policy developments“. As an example it is noted that “at a recent Forum `away-day’ he [Wortley] led a session on preventing theft from the person, which is one of the only crime types to have increased recently, and in particular focussed on the need to convert good ideas for prevention into concrete policies and practices” [2].

The New Zealand Police explicitly credits SCS with a significant role in shaping their crime strategies and helping to reduce crime: “The Jill Dando institute (JDI) has made a significant contribution to the development of crime science and intelligence led policing in New Zealand. We are grateful to JDI staff and associates who have increased our knowledge base, contributed to the development of our key products and fundamentally helped evolve the mindset within the New Zealand Police over the last 10 years…Over the past 4 years we have achieved sizeable reductions in crime against a backdrop of very high (and improving) levels of public trust and confidence in Police” [4].

The practical value of the research is acknowledged by the Agency: “Of particular value in recent years has been the work done by the Department around the topic of Internal Child Sex Trafficking (ICST)… In 2010, SOCA actively supported two research projects carried out by Professor Gloria Laycock with two masters students, Ella Cockbain and Helen Brayley. One project examined the social networks of offenders and victims and the other deconstructed the offence into a `crime script’. The research has been influential in helping to shape SOCA’s response to ICST” [6].” Taken from this case study.

And finally…

I hope this is useful. If you have any experiences or tips of your own then please share!

In the run-up to REF2021 a lot has been written about how to put together a top-scoring impact case study. However, at some point we need to skip the theory and get hands-on. So what’s the best way to get an idea of what makes great impact and what makes a great case study?

The answer is easy: seek out 4* impact case studies from REF2014. I don’t mean that we should be blindly trying to repeat what worked in the last REF – as Digital Science and King’s College London found, there are thousands of routes to impact and the sector will surely have moved on in terms of generating, reporting and assessing impact. However, a lot of effort went into the REF2014 impact case studies, leaving a valuable reservoir of impact knowledge.

The REF team did not publish scores for individual impact case studies, only scores for each Unit of Assessment (UoA) impact submission. But it isn’t hard to find 4* examples – simply look for those submissions that scored 4* overall and you know that every impact case study contained therein will be a 4* example.

This approach will net you 120 publicly-available examples of 4* impact case studies across 19 of the 36 UoAs. That’s an impressive and highly useful resource and most universities will have such a list. But what if your “native” Unit of Assessment isn’t represented among these? Sure, you can find examples from other areas that may be broadly relevant but it isn’t ideal. And besides, even if your area is represented in the current 4* impact corpus, you can always use more examples.

Well I have good news. The list below is an extended library of 4* impact case studies. It contains:

• 198 4* impact case studies across 25 UoAs.
• That’s an extra 78 4* examples with an additional 6 UoAs now represented where there were no 4* examples before.
• These “new” UoAs are: Biological Sciences; Electrical and Electronic Engineering, Metallurgy and Materials; Architecture, Built Environment and Planning; Geography, Environmental Studies and Archaeology; Philosophy; and Art and Design – History, Practice and Theory.
• A number of UoAs now have more 4* examples than before: Clinical Medicine; Public Health, Health Services and Primary Care; Allied Health Professions, Dentistry, Nursing and Pharmacy; Agriculture, Veterinary and Food Science; Law; Social Work and Social Policy; Modern Languages and Linguistics; Communication, Cultural and Media Studies, Library and Information Management.

Note: for UoAs that still don’t have any obvious 4* case studies, see my more recent post on the top 5 REF 2014 impact submissions in each and every UoA.

Extended list of 4* Impact Case Studies by REF 2014 Unit of Assessment

1. Clinical Medicine
2. Public Health, Health Services and Primary Care
3. Allied Health Professions, Dentistry, Nursing and Pharmacy
4. Psychology, Psychiatry and Neuroscience
5. Biological Sciences
6. Agriculture, Veterinary and Food Science
13. Electrical and Electronic Engineering, Metallurgy and Materials
14. Civil and Construction Engineering
16. Architecture, Built Environment and Planning
17. Geography, Environmental Studies and Archaeology
18. Economics and Econometrics
20. Law
22. Social Work and Social Policy
23. Sociology
24. Anthropology and Development Studies
25. Education
26. Sport and Exercise Sciences, Leisure and Tourism
27. Area Studies
28. Modern Languages and Linguistics
29. English Language and Literature
30. History
32. Philosophy
34. Art and Design: History, Practice and Theory
35. Music, Drama, Dance and Performing Arts
36. Communication, Cultural and Media Studies, Library and Information Management

How was the extended list generated?

As noted above, the basic set of 4* impact case studies can be compiled from those submissions that scored 4* overall. The enhanced list also takes into account two other factors. Firstly, in the REF2014 impact scoring scheme, the impact template (REF3a) was worth 20% of the total impact score. Secondly, impact elements were scored in half-steps, e.g. 2.0, 2.5, 3.0, 3.5, etc. We can use these two constraints to unearth the extra 4* impact case studies.

Let’s take a couple of examples:

1. University College London scored 3.70 (i.e. 80% 4*, 10% 3*, 10% 2*) in UoA 32 (impact case study quota of 3). This score could only have come from scores of 4* for each case study and 2.5 for the impact template (REF3a).
2. Cardiff University scored 3.90 (i.e. 90% 4*, 10* 2*) in UoA 16 (impact case study quota of 2). This score could only have come from scores of 4* for each case study (making up 80% of the overall score) and 3.5 for the impact template (the other 20% of the overall score).

In general, submissions with an odd number of impact case studies (or an even number not divisible by 4) AND impact profiles with a 4* bucket of 90% or 80% must have perfect 4* impact case studies, as the loss of 10% or 20% could only have been caused by a non-perfect impact template (except for ).

I hope this proves useful for REF and beyond!

Thanks to David Steynor (currently Research Impact Manager, Queen Mary University of London) for inspiration on the impact and Emer O’Leary for inspiration on the 4* daffodil sketch.

Edited on 23rd March 2018: I originally included University of York in UoA2 and King’s College London in UoA15 which I shouldn’t have! Both scored 3.80 (i.e. 80% 4*, 20% 3*)  with 6 impact case studies. This could have come from a 4* impact template, 3 x 4* case studies, and 3 x 3.5 case studies. Thanks again to David Steynor!