Ref 2014 Impact Case Study

Impact case studies

As part of the 2014 Research Excellence Framework exercise, UK higher education institutions (HEIs) submitted 6,975 impact case studies demonstrating the impact of their research on wider society.

These case studies provide a unique and invaluable source of information on the impact of UK research. UK higher education (HE) research has wide and varied benefits on the economy, society, culture, policy, health, the environment and quality of life — both within the UK and overseas.

Universities engage with a range of public, private and charitable organisations and local communities. Analysis found that these wider impacts and benefits often stem from multidisciplinary work.

Latest report: Patterns in research outputs

'Publication patterns in research underpinning impact in REF2014' describes patterns in research outputs submitted by UK higher education institutions (HEIs) to the 2014 Research Excellence Framework (REF) and to previous Research Assessment Exercises (RAEs). 

The REF impact case study database

The impact case study database is a searchable tool that will make the impact case studies widely available, and will enable analysis and automated text mining.

Read the licence arrangements relating to the use of this database.

An initial analysis of the REF impact case studies is captured in the report: ‘The nature, scale and beneficiaries of research impact’.

The REF impact case studies were analysed by Digital Science, a division of Macmillan Science & Education, working in conjunction with its sister company Nature Publishing Group and the policy institute at King’s College London. This analysis was co-funded by the UK Funding Bodies, Research Councils UK and Wellcome Trust.

Maps of impact case studies

The maps of impact case studies indicate the local and global spread of research impact for UK higher education institutions (HEIs) by impact type and research area.

They are based on the names of locations referenced in REF2014 impact case studies and categorised in the REF impact case study database.

By Prof Mark Reed @profmarkreed

8 points for writing an effective impact case study

1. Create a coherent narrative that explains clearly the relationship between the underpinning research and the impact. It may be useful to briefly explain what was original or distinctive about the research that contributed to the impacts. Consider starting from the perspective of the beneficiaries – how did they benefit and why is that benefit so important to them? Be as specific as possible if you want the link between the research and the impact to be credible, including specific details about the names of researchers, their position and dates and locations of the research activity. Choose a strong headline for your case study and ensure your summary starts with a powerful opening sentence that summarises the case study and draws the reader in

2. Be as clear as possible about exactly what the impact was, adding some sort of precise quantification with numbers wherever possible. Quantitative data and indicators need to be meaningful and contextualised to clearly support the case being made, not used as a substitute for a clear narrative. Avoid generalised or exaggerated statements about your impact.

3. Clearly identify specifically who has benefited from the work or which groups/organisations have changed something as a result of the research. Bear in mind that this may include ‘intermediary’ organisations as well as your intended ‘end users’ or audiences.

4. Be concise. A concise case study that pulls out the key points easily for readers has far greater impact than one that is dense and rambling. The UK’s Research Excellence Framework gives people strict page limits, but if you don’t have this constraint, then you could consider imposing some sort of constraint on your case studies.

5. Keep your language simple and direct. If possible, get advice from a science writer or communications specialist, but try and avoid introducing inaccuracies. Remember your audience, and that your audience is probably not other academics. Readers should not have to have in-depth expert or prior knowledge to be able to understand your case study. It is essential that impact case studies avoid academic jargon, so that they are accessible to all your readers. 

6. If you are writing multiple case studies, identify key features of best practice and be consistent about all your case studies covering these aspects. You might for example want to consider a particular list of sub-headings for each case study to follow. However, avoid making all your case studies sound too similar, as though they are all written by the same person, if you want them to also feel authentic.

7. Related to this, make sure you provide detailed, specific and independent evidence to support every claim you make. All material required to make a judgement about the impact needs to be contained within the case study. Avoid anecdotal evidence or evidence that might perceived as such. If possible, link to published evidence that demonstrates the impact of your research. It is possible to commission work to demonstrate impacts yourself, but you will need to consider carefully how this will be published, to ensure that it is perceived to be sufficiently independent and credible. Rather than just listing sources of evidence, explain how each source of evidence supports a specific aspect of the impact that has been claimed.

8. Bring your case studies to life with quotes that illustrate the impact with greater resonance than could otherwise be done with formal language. If these quotes are from people with high profile and relevant job titles, then this adds significant credibility to your case study, as well as some lived experience. Finding quotes from people years after an impact has occurred can be tricky however, so it is recommended to collect these from people as the impact unfolds, and if nothing else, keep track of contact details so that people can be easily contacted later. Bear in mind that key people who could attest to the impact of your work may retire or pass away before you need to write your case study. Usually it is best to aim for a good balance of quantitative data and quotes to support impacts. However, in case studies where there is no quantitative data to corroborate an impact (e.g. a policy which has been developed but not yet implemented), quotes may be the primary source of corroboration to evidence your impact. 

What do we know about impact in REF2020?

We don’t yet know the rules and weighting that will be associated with the assessment of research impact in HEFCE’s Research Excellence Framework in 2020 (REF2020), but we do know four important things:

  • Impact was worth 20% of an institution’s score in 2014, and most commentators expect this to either remain the same or increase slightly to 25% for REF2020
  • Some impact case studies submitted to REF2014 that have continued to generate further impacts during the current REF period are likely to be eligible for re-submission to REF2020. HEFCE are currently consulting on this, to try and get the balance right between incentivising Higher Education Institutes to continue developing and building on existing impacts, whilst having sufficient incentives to develop new impacts
  • It is likely that the eligibility criteria and guidelines for submitting impact case studies in 2020 will be broadly similar to the 2014 assessment, although these are currently under consultation too
  • Assessment criteria may be expanded (e.g. including impacts on public discourse and attitudes) or become more fine-grained (e.g. considering different elements of reach such as number versus the types or geographical spread of people adopting a new technology or behavior) in response to feedback from REF2014 panels, but they are still likely to address aspects of significance and reach.

HEFCE commissioned an assessment of the potential for metrics to inform REF2020, which reported in early 2015. It concluded that metrics could not replace case study narratives for the assessment of impact in 2020. Research commissioned by HEFCE about the impact case studies submitted to REF2014 somewhat uncritically accepted the breadth and depth of impacts captured in the assessment, implying that a similar approach in 2020 could yield similarly authoritative “evidence” of impact. This approach to impact may be criticized as biased towards a narrow set of self-reinforcing perceptions of what constitutes valid and significant impacts, created primarily between different sectors of the research community, rather than in the eyes of beneficiaries. However, although research has been proposed to address these issues, if funded, it will not report until after procedures have been set in place for REF2020. It is therefore reasonable to assume that REF2020 impact case studies will be broadly similar to REF2014.

This is what Universities were required to submit in 2014, which is likely to be similar in 2020:

  • Each Unit of Assessment (UOA) had to submit an impact statement (REF3a, Figure 1)
  • Each UOA also had to submit between 2 and 7 impact case studies, depending on the number of academics submitted in the Research Outputs section of the assessment (REF3b, Figure 1)
  • Underpinning research had to be “excellent” (2* quality or above) and must have been conducted by University members of staff to be eligible.

Underpinning research could include published reports (e.g. final reports to funders) where it was clear that they were of 2* or above quality, but it is highly risky to submit a case study with underpinning research that has not undergone peer-review in an international journal, in case panel members do not deem it to be of sufficient quality for the impact to be eligible. If you are submitting grey literature as the underpinning research, provide copies or links to any assessments that may have been made by the funders as a form of peer-review, and if this isn’t possible then make sure you get it reviewed by a number of experts in the field to ensure the panel is likely to judge it as 2* or above in quality. Depending on how long there is before REF2020, you may be able to turn a report from the grey literature into a peer-reviewed journal article and submit both as underpinning evidence (the original report to show that the research preceded the impact and the peer-reviewed version to demonstrate 2* quality). The underpinning research has to be conducted (not just published) at the submitting institution, so members of staff appointed between now and the 2020 assessment will need to be able to conduct significant new research to further underpin and extend existing impacts if these are to be claimed by their new institution.

Research has to have made a distinct and material contribution to any impacts that are claimed, such that the impact would not have occurred or would have been significantly reduced without the research. If this link cannot be adequately established, then the case study may be graded as “unclassified”. Research from a number of different individuals, UOAs and institutions may have contributed towards a particular impact, and as a result, there are a number of REF2014 case studies that cover the same impact in different ways, based on the specific impacts related to the research conducted by each of the different submitting institutions. If you are in this position however, it is worth considering whether the impacts that you are able to claim linked to your work will be as far-reaching and significant as those claimed by a competing institution who is submitting to the same UOA. Panels assessing the same impact submitted by different institutions will inevitably compare case studies, and you want to make sure that if this happens, your grade doesn’t suffer as a result.

0 comments

Leave a Reply

Your email address will not be published. Required fields are marked *