Blog

All known scores for REF2021 impact case studies

Not so hot on the heels of my other REF2021 impact case study spreadsheet-based lists (4* impact case studies from REF2021 and top 5 impact submissions in every UoA), here is a list of all impact case studies with known scores.

The impact case study database can be searched and filtered in a number of ways but (beyond sitting with a spreadsheet of REF results) there is no easy way to pick out examples to compare higher and lower scoring case studies. I’ve done the spreadsheet-sifting and the results are below. Each UoA dropdown reveals links to the case studies by score. These are only the case studies we can assign unambiguously – i.e. where the submission scored 100% 4*, 3*, etc. There other ways of inferring scores, for example in a submission that scored 50% 4* and 50% 2*, it would be feasible to infer which case studies scored 4* and which scored 2* but this would involve more time and insight – and you couldn’t know absolutely that that’s how the panel saw it .

1. Clinical Medicine
2. Public Health, Health Services and Primary Care
3. Allied Health Professions, Dentistry, Nursing and Pharmacy
4. Psychology, Psychiatry and Neuroscience
5. Biological Sciences
6. Agriculture, Food and Veterinary Sciences
7. Earth Systems and Environmental Sciences
8. Chemistry
9. Physics
10. Mathematical Sciences
11. Computer Science and Informatics
12. Engineering
3* De Montfort University
2* Anglia Ruskin University (no publicly available impact case studies)
13. Architecture, Built Environment and Planning
14. Geography and Environmental Studies
15. Archaeology
16. Economics and Econometrics
U University of Northampton (no publicly available impact case studies)
17. Business and Management Studies
18. Law
19. Politics and International Studies
20. Social Work and Social Policy
21. Sociology
22. Anthropology and Development Studies
23. Education
24. Sport and Exercise Sciences, Leisure and Tourism
25. Area Studies
26. Modern Languages and Linguistics
27. English Language and Literature
28. History
29. Classics
30. Philosophy
31. Theology and Religious Studies
32. Art and Design: History, Practice and Theory
33. Music, Drama, Dance, Performing Arts, Film and Screen Studies
34. Communication, Cultural and Media Studies, Library and Information Management

I hope this is useful. If you have any issues or spot any mistakes, please leave a comment or get in touch.

Thanks to David Steynor (currently Research Impact Manager at Queen’s University Belfast) for his original “all known scores” spreadsheet of REF2014 impact case studies. I’ve effectively just updated that here.

REF2021: Top 5 impact submissions by UoA

This post follows on from my recent list of 4* impact case studies from REF2021. As I acknowledged there, there aren’t unambiguously identifiable 4* impact case studies in all the UoAs. What if it’s one of these UoAs that you’re interested in? That’s the point of this post – to list the top 5(-ish) impact submissions in every UoA, including those where there were no clear 4* examples.

The impact case study database can be searched and filtered in a number of ways but (beyond sitting with a spreadsheet of REF results) there is no easy way to pick out the highest scoring examples. That’s where the list below comes in – click on the UoA of interest and you’ll uncover links to the highest scoring (publicly available) impact submissions in that area. The number in brackets is each institution’s impact GPA (scored out of 4). You’ll notice that many of my UoA lists below actually have more than a top 5 – this is where the last GPA is held by more than one institution.

1. Clinical Medicine
2. Public Health, Health Services and Primary Care
3. Allied Health Professions, Dentistry, Nursing and Pharmacy
4. Psychology, Psychiatry and Neuroscience
5. Biological Sciences
6. Agriculture, Food and Veterinary Sciences
7. Earth Systems and Environmental Sciences
8. Chemistry
9. Physics
10. Mathematical Sciences
11. Computer Science and Informatics
12. Engineering
13. Architecture, Built Environment and Planning
14. Geography and Environmental Studies
15. Archaeology
16. Economics and Econometrics
17. Business and Management Studies
18. Law
19. Politics and International Studies
20. Social Work and Social Policy
21. Sociology
22. Anthropology and Development Studies
23. Education
24. Sport and Exercise Sciences, Leisure and Tourism
25. Area Studies
26. Modern Languages and Linguistics
27. English Language and Literature
28. History
29. Classics
30. Philosophy
31. Theology and Religious Studies
32. Art and Design: History, Practice and Theory
33. Music, Drama, Dance, Performing Arts, Film and Screen Studies
34. Communication, Cultural and Media Studies, Library and Information Management

I hope this all makes sense and helps you find some useful impact case study examples, whatever your area. Any issues, just leave a comment or get in touch!

REF2021: 4* impact case studies

We all know there’s more to impact than REF impact case studies. However, they represent a lot of hard work and the new REF2021 database of 6,781 impact case studies is so big and wide-ranging that it’s a major resource for anyone interested in research impact. A look at these impact case studies can help academics and research managers understand what impact may look like in a particular area, plan and articulate impact, scope relevant stakeholders and engagement strategies, and consider what evidence and indicators might be relevant when describing different kinds of impact.

The database can be searched and filtered in a number of ways but (beyond sitting with a spreadsheet of REF results) there is no easy way to pick out the highest scoring examples. That’s why many REF and impact managers in universities across the UK will be making lists of 4* impact case studies from REF2021. Here’s mine – click on the UoA of interest below and you’ll uncover links to the (publicly available) 4* impact case studies in that area. These are case studies that are unambiguously identifiable as scoring 4* – there are many other 4* examples but they are buried in submissions.

Of course, it is highly reductionist to focus on these 246 publicly available, identifiable 4* case studies – they represent less than 4% of the whole REF2021 impact case study database – but I know (from my own work with academics and research managers) that this list is a useful entry point to exploring the database more widely.

It’s worth noting that not every UoA had unambiguously identifiable 4* case studies. In case the UoA you are interested in doesn’t appear below, watch this space because I’ll be putting together an accompanying list of the top 5 impact submissions in every UoA.

2. Public Health, Health Services and Primary Care
4. Psychology, Psychiatry and Neuroscience
Early years’ experience and social mobility: Research evidence promoting policy changes to young children’s education and home learning environments (Birkbeck College)
Informing practice and legal frameworks for adoption by LGBT parents (Birkbeck College)
The impact of Interpretative Phenomenological Analysis on training and practice in academia, applied clinical and health research, and commercial consultancy (Birkbeck College)
Improving Literacy Attainments for Young People Using Prosody, Morphology, and Phonology (Coventry University)
Tackling FGM across the EU though a Community Behaviour-Change Approach (Coventry University)
A: Design and use of a rapid assessment tool that improves detection of delirium benefits patients and health services (University of Edinburgh)
B: Extending the indications for the clot-busting drug alteplase for treatment of acute ischaemic stroke increases patient survival and independent living worldwide (University of Edinburgh)
C: A standardised cognitive screening tool to underpin personalised care for people with motor neurone disease worldwide (University of Edinburgh)
D: Adding Functional Neurological Disorders to the global neurology agenda: transforming information, attitudes, diagnosis and patient experience (University of Edinburgh)
E: A novel teaching method based on the theory of Interactive Alignment is in widespread use in schools and universities in China and results in improved levels of English (University of Edinburgh)
F: Reversal in recommended standard of care of treating immobile stroke patients prevents deep vein thrombosis and improves survival after stroke (University of Edinburgh)
G: Improving the reproducibility of preclinical research through more rigorous and robust policy and practice at publishers, funders and industry (University of Edinburgh)
H: Identifying cost-effective imaging strategies to diagnose acute stroke and to prevent secondary disabling stroke (University of Edinburgh)
I: Uncovering a causative link between recombinant interferon beta therapy and thrombotic microangiopathy leads to international safety alerts and risk mitigation measures (University of Edinburgh)
J: Change in UK blood donation policy around plasma and platelets following accurate definition of vCJD transmission risks leads to simpler logistics and cost savings (University of Edinburgh)
Do Something Different: achieving positive behaviour change in health and business settings and among individuals (University of Hertfordshire)
New clinical approaches for the effective diagnosis and treatment of obsessive-compulsive and related disorders (University of Hertfordshire)
Improving identification and support of individuals with movement and handwriting difficulties through development of assessment tools for practitioners (Oxford Brookes University)
Patient-Reported Outcome Measures (PROMs) benefit patient quality of life and the economy (Royal Holloway and Bedford New College)
Psychological research improves practice in the management of chronic pain (Royal Holloway and Bedford New College)
Using the science of reading to change literacy policy and practice (Royal Holloway and Bedford New College)
Advancing multimodal evidence-based treatment of ADHD worldwide (University of Southampton)
Changing our understanding of the lasting impact of severe early deprivation on development: The impact of the English and Romanian Adoption study (University of Southampton)
LifeGuide – Developing Internet-based Support for Healthcare (University of Southampton)
POWeR – cost effective online support for weight management (University of Southampton)
6. Agriculture, Food and Veterinary Sciences
8. Chemistry
10. Mathematical Sciences
11. Computer Science and Informatics
Fighting the Malicious Web (University of Birmingham)
Providing accessibility to scientific documents to visually impaired readers through new technology (University of Birmingham)
Securing Banking Infrastructure with a New Cryptographic Protocol Testing Tool (University of Birmingham)
Code size reduction enables cost savings and enhances functionality of modem and WiFi chipsets (University of Edinburgh)
Effect handlers for increased reach, effectiveness, and future-proofing in software products and services (University of Edinburgh)
Neural machine translation improves translation quality (University of Edinburgh)
Neural network speech recognition algorithms lead to accurate speech-to-text transcription deployed in diverse products and services (University of Edinburgh)
Novel architecture for high-performance computing accelerates data intensive software applications and enables release of new server lines (University of Edinburgh)
Ouroboros provable proof-of-stake protocol ensures success of new blockchain platform and cryptocurrency (University of Edinburgh)
The novel testing tool DexFuzz enables a smooth transition to Google’s Android Runtime (ART) and its continuous bug-free development (University of Edinburgh)
Two billion devices enabled annually by optimised low power embedded processor (University of Edinburgh)
World-leading audio animation research enables continued growth of spinout company and novel features in computer gaming (University of Edinburgh)
Blended Spaces – Creating Commercial and Social Benefit by Transforming Human Computer Interaction (Edinburgh Napier University)
Enabling Sensitive Personal Data to be Shared with Trust using Novel Digital Security Methods (Edinburgh Napier University)
Protecting Employees, Children and Sensitive Data using Innovative Approaches to Cyber Security (Edinburgh Napier University)
Algorithms for Paired Kidney Donation: Increasing Living Kidney Transplantation in the UK (University of Glasgow)
Increased scalability of Erlang for improved telecommunications and internet server capability (University of Glasgow)
WebRTC standards improve interactive audio/visual communications via the world-wide web (University of Glasgow)
GraphicsFuzz: Reliable and Secure Graphics Programming (Imperial College of Science, Technology and Medicine)
Infer: Scalable Static Analysis at Facebook and Beyond (Imperial College of Science, Technology and Medicine)
IXICO: Automated Biomedical Image Analysis for Clinical Trials (pharma) and Healthcare (Imperial College of Science, Technology and Medicine)
SLAM: Enabling Algorithms for Robotics and Augmented Reality (Imperial College of Science, Technology and Medicine)
Causal Bayesian Reasoning Tools for Critical Decision Making (Queen Mary University of London)
Improving Mental Healthcare through a Tool for Effective Clinical Communication (Queen Mary University of London)
The Bela Open-source Hardware Platform for Music Makers and Artists (Queen Mary University of London)
The Infer Tool for Automatic Verification of Memory Violations in Software Systems (Queen Mary University of London)
Cost-effective assurance for industrial software systems through open source formal modelling and verification tools (University of Southampton)
LifeGuide: Online Behavioural Interventions (University of Southampton)
Novel image capture, processing and analysis methods for widespread web publishing, new commercial products and cultural heritage preservation (University of Southampton)
Pioneering transnational open data innovation ecosystems to develop AI and the data economy in a secure way (University of Southampton)
13. Architecture, Built Environment and Planning
14. Geography and Environmental Studies
15. Archaeology
18. Law
19. Politics and International Studies
20. Social Work and Social Policy
Achieving justice and improving support and protection for victims-survivors of gender-based violence (University of Bristol)
Effecting policy and practice change in England to improve the educational outcomes of Children in Care (University of Bristol)
Evidence of premature deaths in people with learning disabilities informs major reform of health policy and practice in England (University of Bristol)
Multidimensional measures of poverty and deprivation have informed policy makers and policy initiatives globally (University of Bristol)
Making adoptions more open: enabling birth family contact nationally and internationally (University of East Anglia)
Transforming the lives of children in state care: implementation of the UEA Secure Base model of therapeutic caregiving in the UK and internationally (University of East Anglia)
Better measurement of income inequality levels and trends (London School of Economics and Political Science)
Housing Plus – giving social housing tenants a voice (London School of Economics and Political Science)
Improving the fairness and efficiency of Austria’s social insurance and healthcare system (London School of Economics and Political Science)
Improving the lives of disadvantaged people through better measurement of poverty and inequality (London School of Economics and Political Science)
Informing fairer, more transparent, and cost-effective social care policy (London School of Economics and Political Science)
Supporting policy and practice change for better mental health (London School of Economics and Political Science)
Better support for victims and survivors of domestic violence and abuse in LGBT relationships (University of Sunderland)
Influencing international government policy on and public attitudes towards assisted suicide (University of Sunderland)

21. Sociology

22. Anthropology and Development Studies
23. Education
Shaping the policy and practice of character education (University of Birmingham)
Transforming higher education access and service provision for people with vision impairment (University of Birmingham)
Transforming How Autistic Children and Young People are Understood, Taught and Treated in Mainstream Education Settings (University of Birmingham)
Enhancing educational experiences for care experienced children and young people in Wales (Cardiff University)
Reshaping the effective delivery of early years and primary education in Wales (Cardiff University)
Transforming national policy and practice in sex and relationships education in Wales, England and internationally (Cardiff University)
Defending and improving schooling for disadvantaged pupils (University of Durham)
Improving the educational outcomes of children starting school in five countries (University of Durham)
The Dyslexia Debate and its relevance for policy and practice (University of Durham)
The Pupil Premium Toolkit: Evidence for Impact in Education (University of Durham)
Enhancing gender equality in national and international educational contexts (University of Lancaster)
Evaluating and enhancing the quality of learning and teaching in higher education (University of Lancaster)
Changing policy and practice on the Education of Children Looked After (CLA) (University of Oxford)
Improving national literacy levels in the Republic of the Sudan (University of Oxford)
Improving the Quality of Educational Assessments (University of Oxford)
Reforming Initial Teacher Education in Wales (University of Oxford)
Transforming Early Education Policy and Practice (University of Oxford)
Changing family literacy practices in prisons, schools and community settings (University of Sheffield)
Transforming children’s digital literacy practices in formal and non-formal learning spaces (University of Sheffield)
Education and Peacebuilding in Conflict-Affected Contexts: Influencing UNICEF’s and key international actors’ approaches to peace-promoting education policy and practice (University of Sussex)
Enabling second chance education for out-of-school children in Ghana and Ethiopia (University of Sussex)
Improving policy and practice on Integration, Education and Language for refugees and migrants (University of Sussex)
Fairer Access for UK Postgraduates (University of York)
Transforming Modern Foreign Languages Pedagogy in England (University of York)
Transforming Teacher Selection Practices in Developed and Developing Countries (University of York)
24. Sport and Exercise Sciences, Leisure and Tourism
25. Area Studies
26. Modern Languages and Linguistics
27. English Language and Literature
28. History
29. Classics
30. Philosophy
31. Theology and Religious Studies

32. Art and Design: History, Practice and Theory

33. Music, Drama, Dance, Performing Arts, Film and Screen Studies
Changing the lives of people living with dementia (PWD) and their families through music therapy research (Anglia Ruskin University)
Improving music therapy for children with autism and their families by developing and embedding an innovative interactive approach (Anglia Ruskin University)
‘Taking Care’: Enhancing UK nursing training using mixed-methods drama research to reconceptualise, teach and promote embodied ‘care’ in clinical interactions (Guildhall School of Music & Drama)
Cultural inclusion, cross-curricular education, and new inter-institutional production mechanisms through practice-based research in opera and libretto composition (Guildhall School of Music & Drama)
Lesions in the Landscape: Stimulating Understanding and Debate around Amnesia and Memory Loss Through the Intersection of Arts and Science (University of Kent)
The Power of Drawing: Transforming Public Perception of Raphael (University of Kent)
Women We See: Improving Representation in Advertising on London’s Transport Network (University of Kent)
In Place of War (IPOW) – Supporting, developing and promoting artists from conflict zones: from responsive research to internationally significant arts organisation (University of Manchester A)
Multi-Story Water: Cultivating environmental citizenship in West Yorkshire through sited performance research (University of Manchester A)
Performing Development: Progressing the United Nations Sustainable Development Goals in Fragile Territories through Collaborative Arts-based Research Projects (Queen Mary University of London A)
The Verbatim Formula: Making Care-Experienced Young People Heard in Care and Education Contexts in the United Kingdom (Queen Mary University of London A)
Dance for Parkinson’s: evidencing, developing and embedding dance initiatives for people with Parkinson’s Diseases internationally (Roehampton University A)
Dancing with ageing, illness and loss through dialogic research (Roehampton University A)
Augmenting Audio Drama: innovation in the form and preservation of cultural memory (Roehampton University B)
Doing Time (Roehampton University B)
Curating new expressions of Mali’s musical heritage: Trio De Kali and the Kronos Quartet (School of Oriental and African Studies)
Reviving Endangered Music to Empower Marginalized Communities in Mexico and Namibia (School of Oriental and African Studies)
ICS1 Inclusive Creativity – driving inclusion in music for disabled artists using creative technologies (University of Ulster)
ICS2 Arts Data Impact: using arts data to drive organisational and policy change in the creative industries sector (University of Ulster)
34. Communication, Cultural and Media Studies, Library and Information Management

I hope this all makes sense and helps you find some useful impact case study examples, whatever your area. Any issues, just leave a comment or get in touch!

Top-scoring impact case studies by REF 2014 UoA

Earlier in the year I set out a list of 4* REF2014 impact case studies to make it easy to find examples of good practice.

It’s a useful resource but there is a snag: 11 of the 36 UoAs have no obvious 4* impact case studies so it leaves something to be desired if (a) you’re looking for good examples in UoAs…

  • 7. Earth Systems & Environmental Sciences
  • 8. Chemistry
  • 9. Physics
  • 10. Mathematical Sciences
  • 11. Computer Science & Informatics
  • 12. Aeronautical, Mechanical, Chemical and Manufacturing Engineering
  • 15. General Engineering
  • 19. Business & Management Studies
  • 21. Politics & International Studies
  • 31. Classics
  • 33. Theology and Religious Studies

… or (b) you want to look at a wider selection of high-scoring case studies than the limited view the 4* list gives you (for example, there’s only one case study for UoA34 on the 4* list).

That’s why I’ve put together a new list setting out the top 5 impact submissions in each UoA along with their scores. Now you can find high-scoring examples in any UoA.

As before, my starting point is that a browse of high-scoring REF2014 impact case studies can help authors articulate their own impact, get them over the tyranny of the blank page, provide inspiration around evidence and indicator types, and so on. Though things may have moved on in REF2021, the REF2014 database is still our best resource when it comes to writing REF impact case studies. The REF results did not attach scores directly to impact case studies so the easier it is for authors and impact geeks to find relevant high-scoring examples, the better.

Click on the UoA of interest below and you’ll uncover the top 5 impact submissions in that UoA with scores (out of a possible 4.00, obviously) and links to the REF2014 results site where you can access the impact case studies for each submission.

Note: for UoA 17 which unhelpfully combines Geography and Archaeology, I have split these into Geography-only and Archaeology-only top 5s.

Top five impact submissions by REF 2014 UoA

1. Clinical Medicine
2. Public Health, Health Services and Primary Care
3. Allied Health Professions, Dentistry, Nursing and Pharmacy
4. Psychology, Psychiatry and Neuroscience
5. Biological Sciences
6. Agriculture, Veterinary and Food Science
7. Earth Systems & Environmental Sciences
8. Chemistry
9. Physics
10. Mathematical Sciences
11. Computer Science & Informatics
12. Aeronautical, Mechanical, Chemical and Manufacturing Engineering
13. Electrical and Electronic Engineering, Metallurgy and Materials
14. Civil and Construction Engineering
15. General Engineering
16. Architecture, Built Environment and Planning
17. Geography, Environmental Studies and Archaeology
17. Archaeology only
17. Geography only
18. Economics and Econometrics
19. Business & Management Studies
20. Law
21. Politics & International Studies
22. Social Work and Social Policy
23. Sociology
24. Anthropology and Development Studies
25. Education
26. Sport and Exercise Sciences, Leisure and Tourism
27. Area Studies
28. Modern Languages and Linguistics
29. English Language and Literature
30. History
31. Classics
32. Philosophy
33. Theology and Religious Studies
34. Art and Design: History, Practice and Theory
35. Music, Drama, Dance and Performing Arts
36. Communication, Cultural and Media Studies, Library and Information Management

Things to bear in mind

  1. The score for each submission includes the impact template as well as the impact case studies. This can throw up some odd-looking results, for example in UoA5, St Andrews is 2nd in the top 5 (with an overall impact score of 3.90 – including the impact template) but it also appears in the 4* list as the only UoA5 submission to get 4* for all impact case studies!
  2. Just because a submission has a score at the top end of the 3s, you can’t count on all the case studies therein being 3* and 4*. For example, Imperial’s UoA12 submission comes 2nd in class with a score of 3.69 which looks great. However, the impact profile (detailed breakdown of scores – available by clicking “view submission” in the breadcrumb trail of any of the linked REF submission pages) shows that one of the case studies in this submission scored 1.5… It just so happens that the average was buoyed by all 17 other case studies scoring 4*! It should be possible to spot the 1.5 among all the 4s but my point is you can’t always select a high-scoring submission and be sure everything is within it is high-scoring (a bit like journal impact factors…). This isn’t a common issue – I had to search for this example – but it does come up so check those impact profiles if you want to ensure everything is 3* and 4*. The impact profiles can be tricky to interpret – if in doubt ask a friendly REF person to help you decode them! Just to complete this story, all four other submissions in the UoA12 top 5 contained exclusively 3* and 4* case studies.

I hope this all makes sense and helps everyone find some good impact case study examples, whatever your area. Any issues, just leave a comment or get in touch!

Guidance on testimonials and statements to corroborate impact

The REF impact assessment requires impact case studies to include evidence of impact. This evidence can take many forms, quantitative or qualitative. Statements from research users, stakeholders and beneficiaries can be a powerful form of qualitative evidence, as seen in REF 2014 (see here for an attempt to analyse links between evidence types and impact case study scores). For the next REF we have the opportunity to collect these statements as we go along and this guide should facilitate such statement-gathering.

What to include

In general statements should:

  1. Be written on the external organisation’s headed paper (or a suitably professional-looking email).
  2. Be signed by someone at an appropriate level. This will vary by case study but considerations should include seniority vs connection to the research (e.g. should the statement come from your direct contact, the person with responsibility for the area or the head of the organisation?), maturity of relationship, reputation and conflicts of interest.
  3. Name the researcher and refer to the research (could be in descriptive terms, citation of a research output, name of research programme – whatever comes across as most fitting).
  4. Describe how the organisation “found” the research/researcher.
  5. Describe how it fits with the organisation’s activities, strategy, needs, challenges, opportunities and other drivers.
  6. Describe how the research/knowledge/skills were put into action or used – e.g. did the organisation work with the academic (maybe through commissioned research, consultancy, knowledge transfer grant, advisory work, other joint activities), did they use the research in their decision making, did they train their staff according to the research?
  7. Describe what happened as a result of using the research/knowledge/skills or working with the researcher – e.g. did they produce guidelines for practice, were they better informed in making strategic decisions, was their service provision directly improved?
  8. Describe the resulting impact of this work – what was the benefit of using the research/knowledge or working with the researcher? Include quantitative or qualitative indicators to show the impact – i.e. how they know it was beneficial. They could also say where they’d be if they hadn’t used the research. This is the most important parts of the statement as it’s where the impact is really articulated (and any quantitative/qualitative evidence the organisation provides can be quoted in the case study and woven into the narrative).
  9. Say something about the future – what’s next in this line of work? Do they foresee continued and growing benefits? Will they work with the researcher again? Will they be more open to using academic research in the future? Maybe they’ll change the way they operate as a result of the impactful piece of work.

How to gather testimonials

The first (and by far the best) option is to put the above list into your own words (so it doesn’t sound so much like a checklist) and use this to prompt a statement from your partner. You could either put it in writing and let them respond accordingly or you could use this as the basis of a conversation/interview. The beauty of the latter approach is that you can explore and clarify and it may uncover other relevant information.

Secondly, you could use the above list as a checklist/questionnaire. I wouldn’t recommend this approach. It may save time but you are unlikely to get the richness or authenticity of a more personal/tailored interaction. It won’t strengthen your relationship with the organisation and it may even damage it.

Finally, in some cases it may be necessary to essentially write the statement yourself and hand it over to the partner to sign. This is not recommended as you lose the authentic voice and you may miss some aspect of what made the work so valuable (including possibly some extra information the partners would have included if they’d had to write it themselves). On top of this there is a very real risk that if the academic writes a number of such letters for different partners to sign, they could all end up looking fundamentally the same which undermines the credibility of the messages. Consider how you’d view this as an assessor…

Other considerations

  • Make sure the person giving the statement knows what it is for and has the authority to give it.
  • Observe the relevant data management and ethics policies as you gather, hold and use this information.
  • In REF2014, some organisations were overwhelmed with requests for testimonials to the point where relationships were affected and in some cases they simply refused to provide testimonials. This is where strong relationships really count so focus on lasting, rather than superficial, interactions with stakeholders and partners.
  • Some organisations will be concerned about confidentiality. Although we don’t currently know how this will work for impact case studies and evidence in REF2021, there were provisions for this in REF2014 so we can expect similar in REF2021.
  • Don’t wait to get testimonials. The details, nature and value of impacts may become dull with time so jump on them while they’re fresh. Plus you never know where people will be in a couple of years.

Examples

These statements are not only a source to be referenced – you can use choice quotes to tell the story and illustrate the outcomes. Here are some examples from REF2014 showing the kinds of things that were said in testimonials and how they can be weaved into the narrative. They all happen to be from UoA 22 (social work and social policy) just because they were handy. You will find similar examples across the UoAs.

1. From “Improving evidence-based policy and programming for AIDS-affected children in Sub-Saharan Africa“:

​​”This is evidenced by a comment from a Regional HIV and AIDS Advisor at Save the Children: `By sharing rigorous evidence on how children affected by HIV faced increased vulnerabilities over time, Dr Cluver has engaged major policy makers, donors and program implementers in critically “rethinking” and redesigning programs with an emphasis on effectively measuring results. Based on Dr Cluver’s research, UNICEF, PEPFAR have launched new technical guidance (UNICEF: From Evidence to Impact; PEPFAR: OVC Guidance: 2012) for program implementers. Her work is a testimony of how rigorous research is the foundation for effective programming…a clear example how a true partnership between researchers, policy makers and implementers can result in programs that actually make a difference in children’s lives.’ [C9] A senior advisor for PEPFAR also stated: `I have quoted Dr Cluver’s data more than any other research to support our work and also to justify and set policy for our global portfolio. All of the data presented from these studies has been influential, however of particular influence has been the data discussed in Cluver [et al] (2011) [R2] and 2012.’ [C10]”

2. From “Reducing child anti-social behaviour through effective parenting interventions: international impact on policy, practitioners and families“:

“As explained by the then Deputy Director of the Social Exclusion Task Force at the Cabinet Office [C2], who played a major part in driving New Labour policies on early parenting interventions in the late 2000’s: “Support from the Start” and Gardner’s UK trials [R1-3] were highly influential in creating momentum towards these major policy developments, including the very substantial roll out led by the National Academy of Parenting Practitioners (NAPP) from 2008.

​The founder and research director of NAPP [C3] adds to this: Gardner’s research was pivotal in helping persuade the Cabinet Office to set up NAPP. Her trial [R1] was the first in the world to show that parenting programmes could be effective in reducing severe antisocial behaviour, outside the narrow confines of child mental health services, showing that the voluntary sector could do just as good a job with difficult cases. Influenced by these findings, the government awarded £35 million to NAPP to disseminate evidence-based parenting programmes across the voluntary sector; this is estimated to have benefited over 150,000 children; Gardner was an important member of its steering group to ensure this research was implemented.

Gardner’s team’s research is frequently cited in influential systematic reviews, guidelines, and policy documents [see C4-7 for Cochrane, Campbell and other reviews]. For example, four of the trials [R1-6] have been cited in the NICE Guidelines (National Institute of Health and Clinical Excellence); the Chair confirms their policy influence [C3]: As Chair of the NICE Guideline on antisocial behaviour and conduct disorders, I oversaw the marshalling of the evidence which has to be relevant to British practice, and Gardner’s work was important in shaping the recommendations, since it showed that the interventions work in Britain — not all psychosocial interventions developed in the USA do this. Their work also added to the plausibility of the recommendations since it demonstrated that the mediating mechanism was an increase in positive parenting.

3. ​​​​From “Situational crime prevention policy and practice“:

“A letter from the Home Office states: “The work of the Department and its advocacy for, and expertise in, situational crime prevention strategies are well known by officials responsible for Crime policy within the Home Office and to the experts within my own area, Home Office Science, whose task it is to ensure that we provide the best evidence on which to base policy developments“. As an example it is noted that “at a recent Forum `away-day’ he [Wortley] led a session on preventing theft from the person, which is one of the only crime types to have increased recently, and in particular focussed on the need to convert good ideas for prevention into concrete policies and practices” [2].

The New Zealand Police explicitly credits SCS with a significant role in shaping their crime strategies and helping to reduce crime: “The Jill Dando institute (JDI) has made a significant contribution to the development of crime science and intelligence led policing in New Zealand. We are grateful to JDI staff and associates who have increased our knowledge base, contributed to the development of our key products and fundamentally helped evolve the mindset within the New Zealand Police over the last 10 years…Over the past 4 years we have achieved sizeable reductions in crime against a backdrop of very high (and improving) levels of public trust and confidence in Police” [4].

The practical value of the research is acknowledged by the Agency: “Of particular value in recent years has been the work done by the Department around the topic of Internal Child Sex Trafficking (ICST)… In 2010, SOCA actively supported two research projects carried out by Professor Gloria Laycock with two masters students, Ella Cockbain and Helen Brayley. One project examined the social networks of offenders and victims and the other deconstructed the offence into a `crime script’. The research has been influential in helping to shape SOCA’s response to ICST” [6].” Taken from this case study.

And finally…

I hope this is useful. If you have any experiences or tips of your own then please share!

More 4* REF impact case studies

In the run-up to REF2021 a lot has been written about how to put together a top-scoring impact case study. However, at some point we need to skip the theory and get hands-on. So what’s the best way to get an idea of what makes great impact and what makes a great case study?

The answer is easy: seek out 4* impact case studies from REF2014. I don’t mean that we should be blindly trying to repeat what worked in the last REF – as Digital Science and King’s College London found, there are thousands of routes to impact and the sector will surely have moved on in terms of generating, reporting and assessing impact. However, a lot of effort went into the REF2014 impact case studies, leaving a valuable reservoir of impact knowledge.

The REF team did not publish scores for individual impact case studies, only scores for each Unit of Assessment (UoA) impact submission. But it isn’t hard to find 4* examples – simply look for those submissions that scored 4* overall and you know that every impact case study contained therein will be a 4* example.

This approach will net you 120 publicly-available examples of 4* impact case studies across 19 of the 36 UoAs. That’s an impressive and highly useful resource and most universities will have such a list. But what if your “native” Unit of Assessment isn’t represented among these? Sure, you can find examples from other areas that may be broadly relevant but it isn’t ideal. And besides, even if your area is represented in the current 4* impact corpus, you can always use more examples.

Well I have good news. The list below is an extended library of 4* impact case studies. It contains:

  • 198 4* impact case studies across 25 UoAs.
  • That’s an extra 78 4* examples with an additional 6 UoAs now represented where there were no 4* examples before.
  • These “new” UoAs are: Biological Sciences; Electrical and Electronic Engineering, Metallurgy and Materials; Architecture, Built Environment and Planning; Geography, Environmental Studies and Archaeology; Philosophy; and Art and Design – History, Practice and Theory.
  • A number of UoAs now have more 4* examples than before: Clinical Medicine; Public Health, Health Services and Primary Care; Allied Health Professions, Dentistry, Nursing and Pharmacy; Agriculture, Veterinary and Food Science; Law; Social Work and Social Policy; Modern Languages and Linguistics; Communication, Cultural and Media Studies, Library and Information Management.

Note: for UoAs that still don’t have any obvious 4* case studies, see my more recent post on the top 5 REF 2014 impact submissions in each and every UoA.

Extended list of 4* Impact Case Studies by REF 2014 Unit of Assessment

1. Clinical Medicine
2. Public Health, Health Services and Primary Care
3. Allied Health Professions, Dentistry, Nursing and Pharmacy
4. Psychology, Psychiatry and Neuroscience
5. Biological Sciences
6. Agriculture, Veterinary and Food Science
13. Electrical and Electronic Engineering, Metallurgy and Materials
14. Civil and Construction Engineering
16. Architecture, Built Environment and Planning
17. Geography, Environmental Studies and Archaeology
18. Economics and Econometrics
20. Law
22. Social Work and Social Policy
23. Sociology
24. Anthropology and Development Studies
25. Education
26. Sport and Exercise Sciences, Leisure and Tourism
27. Area Studies
28. Modern Languages and Linguistics
29. English Language and Literature
30. History
32. Philosophy
34. Art and Design: History, Practice and Theory
35. Music, Drama, Dance and Performing Arts
36. Communication, Cultural and Media Studies, Library and Information Management

How was the extended list generated?

As noted above, the basic set of 4* impact case studies can be compiled from those submissions that scored 4* overall. The enhanced list also takes into account two other factors. Firstly, in the REF2014 impact scoring scheme, the impact template (REF3a) was worth 20% of the total impact score. Secondly, impact elements were scored in half-steps, e.g. 2.0, 2.5, 3.0, 3.5, etc. We can use these two constraints to unearth the extra 4* impact case studies.

Let’s take a couple of examples:

  1. University College London scored 3.70 (i.e. 80% 4*, 10% 3*, 10% 2*) in UoA 32 (impact case study quota of 3). This score could only have come from scores of 4* for each case study and 2.5 for the impact template (REF3a).
  2. Cardiff University scored 3.90 (i.e. 90% 4*, 10* 2*) in UoA 16 (impact case study quota of 2). This score could only have come from scores of 4* for each case study (making up 80% of the overall score) and 3.5 for the impact template (the other 20% of the overall score).

In general, submissions with an odd number of impact case studies (or an even number not divisible by 4) AND impact profiles with a 4* bucket of 90% or 80% must have perfect 4* impact case studies, as the loss of 10% or 20% could only have been caused by a non-perfect impact template (except for ).

I hope this proves useful for REF and beyond!

Thanks to David Steynor (currently Research Impact Manager, Queen Mary University of London) for inspiration on the impact and Emer O’Leary for inspiration on the 4* daffodil sketch.

Edited on 23rd March 2018: I originally included University of York in UoA2 and King’s College London in UoA15 which I shouldn’t have! Both scored 3.80 (i.e. 80% 4*, 20% 3*)  with 6 impact case studies. This could have come from a 4* impact template, 3 x 4* case studies, and 3 x 3.5 case studies. Thanks again to David Steynor!

Writing Pathways to Impact

The UK Research Councils have required applicants to submit impact plans (“Pathways to Impact”) as part of research proposals since 2009 so they’ve been around for a while now. Still, researchers often struggle to understand what they’re being asked for and how to fulfil these requirements. Pathways to Impact is often the last piece that’s written, squeezed in just before submission as one of the lower-priority parts of a proposal. I have written about why we need to give more priority to Pathways to Impact and treat it as a major opportunity rather than another piece of bureaucracy. This post sets out how to write a Pathways to Impact statement.

The UK Research Councils are responsible for spending around £3 billion on research each year. This is public money which might otherwise go into schools, police, health/social care and so on. Thus there is an expectation that researchers bidding for these funds show the wider relevance and potential benefit (beyond academia) of their work and plot appropriate steps to begin to deliver on this potential. It is these steps which should be captured in the Pathways to Impact.

Many other funders have their own requirements for impact/dissemination/exploitation plans. This guidance is written with the Research Councils’ Pathways to Impact statement clearly in mind but the principles and logic apply wherever you need to present impact plans.

Expectations

Academics are often uneasy with the idea of planning impact. You can’t predict the future. You don’t know how or when your work might become relevant. You don’t control the external context. Even if there is some chance of impact, you can’t deliver it – other people do. The benefits of research often unfold over a very long timescales. How can you be asked to plan for all this? Isn’t impact planning essentially just lying to get funding? These would all be valid concerns IF funders were asking you to predict the future and make it happen, but they’re not.

Funders are asking you to take a reasonable view of who might potentially benefit from your work and what you can do to increase the chances of that benefit happening. It’s all about this potential and how you can help things along. They understand it may not pan out as plotted in your Pathways to Impact but whatever direction and form things take, if you’ve considered potential benefits and beneficiaries (and budgeted accordingly) then you’ll be better placed to create and respond to opportunities. This way the benefits are more likely to be realised than if your research sits in a journal waiting to be found.

Most academics I speak with have some inkling of their potential impact. This guidance is about developing those initial thoughts into plans. If you have no idea where to start in terms of your potential impact then I’d recommend you speak with colleagues (academics and impact/funding development staff in your university) or someone like me.

Components

The impact component of proposals is addressed in two parts:

  1. Impact Summary: a text box in the Je-S online form. This is a 4000 character statement of who (beyond academia) might benefit from the proposed research and how.
  2. Pathways to Impact: a separate document, up to 2 sides of A4 (font size 11, 2cm margins, etc). It should build on the who and how in the Impact Summary and set out what will be done to maximise potential benefits.

I recommend focusing on the Pathways to Impact (hence this post) and the Impact Summary should hopefully drop out afterwards.

Structure

I don’t want to set out a template for Pathways to Impact as each plan should be authentic, specific and relevant to the proposed research. What I will do is suggest an outline structure which will help you address the required points in a logical flow.

1. Introduction

Start with a summary (around 1/3 of a page) of the main objectives or deliverables of the proposed research. Say which of these have the potential for non-academic impact, stating broadly who (what types of stakeholder, sectors, etc) will benefit and how. This introduction may repeat what you write elsewhere in the proposal but often the Pathways to Impact is the first rich, substantial document that appears in the long pdf that the Je-S system generates for reviewers. It makes sense to have a reasonable introduction here. Also, repeating certain key ideas (e.g. overarching objectives or vision) in the same terms throughout the proposal reinforces the point of your proposal, helps reviewers to quickly “get it” and means panel members can easily pitch your proposal to colleagues on prioritisation panels.

2. Development and engagement activities for each stakeholder group

Having set the scene, you now need to go into detail about how you will engage with each stakeholder group. This needs to be broken down by group and the way you do this will depend on the nature of your work. As a starting point I’d suggest breaking this into 2-3 sections using headings such as “Impact on industry” and “Impact on policy” or you could choose to break it down by sector, e.g. research on sea-level rise might have sections for impact on marine management, coastal infrastructure and coastal communities. If you have too many sections, you won’t be able to do justice to each and it may seem that you lack focus. If you have too few, it may appear that you haven’t fully thought through the potential. It’s up to you to find the best way of presenting this – you may well need a few iterations.

For each of these sections:

  • Identify the key organisations and people you need to engage with and how they will benefit from the research. You should at least be naming the organisations, even better if you can name the right departments and people. You want to show that you have a real route in, you’re not just speculating that they’ll be interested. Ultimately you will be able to show how these organisations have already been involved in planning the research. If this is the case and they are willing to provide a letter supporting the project then they can be entered in Je-S as a Project Partner. The letter should show how they are contributing to the project. This could be in the form of direct (i.e. cash) or indirect (e.g. providing advice, data, equipment, undertaking development or dissemination) contributions.
  • Say what you will do to engage with these partners, make the research useful to them and help them embed it. These activities need to be tailored to the stakeholder and might involve meetings, placements, reports and briefings, training, developing research outputs to make them more usable (e.g. building prototypes, toolkits, resources, putting a user interface on software), public engagement, attending conferences and events, etc. These activities will usually have costs associated with them. Specify these costs as you describe each activity, include them in your costing and account for them in the Justification of Resources.
  • Discuss timing. Some activities and stakeholders need to be engaged from the start (e.g. if they have a stake in the research design or if they hold critical data), others will only be engaged later when you have something to show them. The way you treat timing issues is often a marker of how serious your plans are. Hurriedly written impact plans often don’t consider timing or default to leaving all the impact and engagement activities to the end of the project.

3. A final section on costs, milestones/timescales, management and evaluation

This section (around 1/3 of a page) should round things out by describing some of the project management aspects.

  • Costs. Summarise the total costs associated with the Pathways to Impact plan and make sure this tallies with the Justification of Resources and the costing in the Je-S form.
  • Milestones and timescales. If you have considered timing alongside each activity as suggested above then there’s no need to repeat all that here. However, it can be helpful to reviewers (and for you and fellow investigators) to include a simple Gantt-type chart to show how things fit together. Ideally your impact plans will be truly integrated with the research elements, in which case include the impact plans in the overall project Gantt chart/workplan (maybe part of the Case for Support, maybe a separate attachment depending on scheme and Research Council). If you do this, make sure you provide suitable signposting in the Pathways to Impact.
  • Management. The plans described in the Pathways to Impact are effectively a sub-project or work package of the proposal so state who is responsible for overseeing and carrying out impact activities. Again, ideally the impact elements should be integrated into the research project meaning any management plan in the proposal should consider impact management alongside management of the research. If you do this, again make sure you provide suitable signposting in the Pathways to Impact.
  • Evaluation, success measures, key performance indicators. How will you track the progress of your impact plan? How will you know if you’ve succeeded in having impact? How will you evaluate the extent of your impact? In my experience, this kind of information won’t make or break your Pathways to Impact but given that impact is a big factor in the REF, it seems a good idea to give some thought to these questions. Thinking about evaluation at this stage will help you to gather the right kinds of information as you go along. It may be that your partners are routinely collecting or generating such information which means you don’t need to.

What makes a great Pathways to Impact?

  1. Clear identification of beneficiaries, activities and deliverables.
  2. Good consideration of a range of relevant potential impacts. Think about potential benefits and beneficiaries across industry/business, policy/practitioners/regulation and the public.
  3. Appropriate track record of impact and engagement where this applies to the stakeholders and activities identified in the current plan. Don’t simply state your track record as an indicator that good things will happen. Your track record is only meaningful when it shows you have the links and experience to deliver on the current plans.
  4. Co-production and involvement of stakeholders right from the planning/scoping stages of the research. This shows their buy-in and commitment. If they are present as Project Partners (providing a letter stating their direct or indirect contribution to the research) then even better. NOTE: the ideal kinds of relationship with stakeholder don’t just happen when you’re writing a proposal. You need to nurture these links over the long term and from time to time you may call on them to be involved in your research proposals. If you’re an early career researcher then start building links now.
  5. Timing activities so that impact and engagement develop over the course of the project, maybe as key research outputs are delivered or as required to best fit the needs of stakeholders.
  6. Active language (e.g. we will…, this will enable…).
  7. A focus on how your research will be used, rather than how it will be disseminated.

What if your research doesn’t have impact?

Once upon a time I was leading a training session on Pathways to Impact and a physicist asked me what possible use there might be for research on gravitational waves. A pure mathematician responded, “You think you’ve got problems? I can’t even explain my research to non-mathematicians!”

RCUK states that the “pathways to impact will apply for the vast majority of proposals” and that exceptions will be few and far between. Most academics I deal with have some idea of their potential impact, some starting point. However in a small number of areas this really is challenging. Still, RCUK are clear that they expect you to give it a shot.

There is a get-out clause (“In the few exceptions where this is not the case, the Pathways to Impact statement should be used to fully justify the reasons why this is not possible”) but don’t be tempted to take this option as the Research Council will almost certainly come back to you asking for an “acceptable” Pathways to Impact.

In my experience, there is always some way of putting together an “acceptable” impact plan even in those tricky areas, it just requires close attention on a case-by-case basis.

And finally…

If I’ve missed anything or if you have your own top tips then I’d love to hear from you in the comments or feel free to get in touch.

Celebrating Pathways to Impact

Next year the UK Research Councils’ infamous “Pathways to Impact” will be 10 years old. But who will be celebrating? Who even realises?

I started working at the Research Councils in 2009 when “Impact Plans”, as they were known in the beginning, were born. The idea was that given the amount of public money going into research, partly on the back of lobbying placing universities as engines of the economy, academics should be prepared to articulate the wider potential benefits of their work and how they might help things along.

To me, fresh out of postdoctoral research, the impact agenda was new but it didn’t seem unreasonable. However, academics took a lot of coaching to understand what they were being asked to do and why. Other funders have followed suit in asking for impact plans so has the community come to embrace Pathways to Impact and its kind? From my point of view, as someone who works with academics to help them win funding and increase the impact of their research, I would say… partially. But there’s still a long way to go.

I believe impact is an inherent part of a socially-aware, publicly-funded academia. However, if the idea of impact for its own sake didn’t grab people, the inclusion of impact in REF2014 definitely focussed minds. If universities want to take part in REF (and reap the rewards in terms of reputation and funding) then they need to show their contribution to the world. So the UK has a research system where academics are able to plan and cost impact at the outset of their research and then later get rewarded for demonstrating the impact they’ve achieved. It’s almost as if someone designed it.

Still, I feel we’re not making the most of the opportunity presented by the Pathways to Impact statement (and equivalent impact/dissemination/exploitation plans required by other funders). Everyone focuses on the assessment end – REF impact case studies – rather than the inputs. So why should we give impact plans more love?

  1. The best way to achieve impact is to build it in from the start. Answering the right questions in the right ways, building partnerships, engaging stakeholders, exchanging knowledge, translating research outputs into other forms – these are all more effective when they are properly integrated and resourced.
  2. “Impact thinking” often means higher-quality, more fundable research proposals. When you’re talking to stakeholders and considering the wider context around your research, you’re better equipped to show importance and timeliness, you benefit from the input of industry, policy and other experts to help plan and steer the project, you can access unique resources (facilities, data, field sites, etc) and the proposal just feels more “together” and well-thought-out to reviewers. Who knows – your partners may even offer a cash contribution to the research!
  3. Two grants for the effort of one. If you don’t consider impact now, you may well find yourself applying for a separate “impact” grant later. A whole new proposal when you could have just written it into the original! Definitely a case of working smarter not harder. Get two grants for the effort of one while making it more fundable? Surely a no-brainer. Don’t worry if you feel you’re making a commitment you’re not sure you can keep. This is all about spotting and nurturing potential, not delivering on promises. You can use the impact funds whatever direction your impact takes.
  4. It saves rewriting later. Research Councils ask that you provide an “acceptable” Pathways to Impact statement before they release any funding. Even if they are desperate to fund your research, you can’t start the grant if your Pathways to Impact isn’t up to scratch. Instead you’ll have to rewrite the Pathways until the funder is happy with it. At this point, you can’t include more costs so you’ll have to conjure up an impact plan with no supporting resources! Avoid this frustration by getting it right in the first place.

Whatever your view of impact plans (maybe “unnecessary bureaucracy” or “important but not a priority”), it’s hard to deny there’s a big opportunity here. As we approach this tenth anniversary, I say it’s time to truly embrace Pathways to Impact.

This piece was written for Emerald’s Real World Impact blog.

You can find my take on how to write a Pathways to Impact statement here.