Applications for REF 2028 must contain at least 5% non-traditional materials.

The recent publication of a series of reviews and early decisions for REF2028 highlighted an increased focus on research culture and environment beyond traditional researchers and research publications. James Baker, Lindsey Ballantyne, Neil Chu Hong, Gemma Derrick, Andy Dixon, Georgina Fletcher, Jude Fransmann, Simon Hattrick, Emma Karun, Simon Kerridge, Kirsty Pringle, Tony Roche (The Hidden REF Committee) states that applications must include 5% non-traditional results and introduces the Hidden REF 5% Manifesto as a starting point to achieve this goal.

In the recently announced early decisions for REF2028.

First, the new outcome measure, now known as Contribution to Knowledge and Understanding, continues to support a broader range of outcomes than is normally presented in the REF. As stated in the decision document, these may be: “audio and video content, scientific publications, software, datasets and artifacts”.

Second, departments are now required to submit structured explanatory statements for evaluation along with their submissions. This allows them more flexibility in recognizing and evaluating how unconventional results contribute to the overall quality of research and researchers.

Third, in an unprecedented step in global research evaluation, there will be an audit of research in the UK. “permit submission of results by any employee, including non-academic staff”, this disconnection between individual employees and the results presented allows departments to choose different kinds of results more freely.

As we wrote for this blog in 2020, “we believe that if we recognize all who are vital to research, we will create the right environment for them to advance“. Therefore, we welcome these developments, even if they are not revolutionary, but highlight the rules that existed for filing in 2014 and 2021. However, as previous experience has shown, simply allowing for more diverse contributors and outcomes does not necessarily mean that they will emerge spontaneously or on an equal footing. Welcome to the new goals and objectives of Hidden REF.

How risky are unconventional exits?

One area of ​​concern with recent decisions is related to the decline in the cost of results in REF from 65% in 2014 to 60% in 2021 and possibly to 45% or less in 2028. The argument goes that presenting unconventional results may come at the expense of a more profitable journal publication or double-weighted book. In addition, the inclusion of any employee’s work may come at the expense of professional academic researchers, who traditionally receive high marks in peer reviews. Thus, the decision on what and whom to include in the work depends on the strategic decision-making in institutions.

The perceived risk of evaluating unconventional products may not be as high as some people think. Based on REF2014 and REF2021 data, unconventional results account for only 3.0% (2014) and 2.6% (2021) of all reported results. Preference was given to the presentation of journal articles and books in both exercises. However, in both REF2014 and REF2021, non-traditional results performed just as well as “traditional” results in terms of the proportion rated 4*. In fact, journal articles are worse in terms of 4* ratings than average publication results, only marginally better than physical artefacts, exhibitions and performances, and significantly worse than digital artefacts and “other”. So, in the run-up to REF2028, including most of these outputs may not pose as much of a risk.

Rice. 1: REF2021 result type mapping has changed compared to REF 2021 views

Rice. 2: REF2021 output quality profile by output type

Learning to evaluate differently

Latent REF was first launched in 2020 and focused on celebrating the diversity of ways traditionally non-REF people and results can contribute to the research environment and how knowledge is produced. Exercise identified a wide range little-known people and research results that are important producers of knowledge. It also required the development of new approaches to assessing and understanding how non-traditional people and results collectively contribute to the development of modern research culture and are an integral part of it.

The first exercise Hidden REF made significant progress in this area. As with the official REF exercise, our evaluation panels were composed of the world’s leading subject matter experts and experts with experience in research evaluation processes. Assessors were able to find clues and characteristics on which to evaluate these materials, despite the lack of formal training and prior evaluation of these types of materials. When evaluating the Hidden Roles category, one of the panellists reported that “(In the submission) the emphasis is on recognizing the need for networking, bridging and eavesdropping; skills that are not often recognized or rewarded in academia.” Finding these new categories and panel methods and communicating them to reporting institutions is an important part of demystifying the potential risk of unconventional results. This is a topic that we will explore further in Hidden REF Festival in Bristol on Thursday 21 September.

How much to submit?

Recent REF decisions provide an opportunity to create a more inclusive research culture, but how much non-traditional work should institutions strive to include in their proposals? We encourage all Higher Education Institutions (HEIs) to work towards ensuring that at least 5% of submissions under the Contribution to Knowledge and Understanding criterion for REF2028 are non-traditional outcomes.

We called it “Hidden manifest REF 5%” and by signing it, universities will reaffirm their commitment to ensure that the diverse research outputs that make their research environment great are recognized and take another step towards having a wider range of people fully appreciated for their contributions.

Figure 3: Percentage of non-traditional results by total size of institutions represented at REF2021

A threshold of 5% is quite possible, many smaller universities and 29.3% of all universities have already reached this threshold during the 2021 experiment (Fig. 3). What prevents a wider representation of universities from recognizing this diversity is the fear of being uncompetitive, not the excuse that it is against the “rules”. The continued lack of recognition of unconventional findings in REF2028 submissions threatens to undermine the laudable goals of boosting research culture across the sector. Let’s make sure these hidden exits are noticed in 2028.

Hidden REF will take place in Bristol on Thursday 21 September. To be part of the change register here and to show your support for the Hidden REF 5% manifesto, please sign Here.

The content created on this blog is for informational purposes only. This article represents the views and opinions of the authors, but does not reflect the views and opinions of the Impact of Social Science blog (blog) or the London School of Economics and Political Science. Please see our comment policy if you have any concerns about posting a comment below.

Image Credit: LSE Impact Blog via Canva.