Does REF add any value to research in the UK?
As the UK chose to link research evaluation to research funding, there were criticisms that the competitive nature of REF evaluation created the conditions for a winner. Although it is difficult to assess Banal-Estanol and others. use a new method to evaluate the effectiveness of research in the UK compared to counterfactual research in the US. They found that overall research output in the UK increased compared to the comparator, but research productivity did not increase. The REF did reinforce the pre-existing concentration of research output in some institutions, although mostly at the expense of non-elite universities.
Public research policy in the UK over the last 35 years has revolved around nationwide evaluation activities, first the Research Evaluation (RAE) and then the Research Excellence Framework (REF). The research performance rankings obtained from these exercises determine the allocation of the university’s core research funding. Thus, they have a great influence on the state of British universities, both directly financially and indirectly through prestige.
This raises a simple question: how comparable are these rankings across universities? After all, British universities have historically had different missions and research trajectories. It has been argued from the start of these exercises that these comparisons risk creating a competitive “rich get richer” dynamic that reinforces inequality and works against ambitions to level the system. Has the REF in its current form added value by improving the overall research capacity and efficiency of UK universities, or has it stimulated efforts to relocate existing research capacity?
Direct comparison of universities with each other in a highly heterogeneous domestic higher education (HE) sector can be misleading. In our recent article, we advocate a new and potentially fairer approach whereby comparisons should be made with similar international benchmarks. Ideally, such benchmarks should follow the same trends as UK universities before the assessment. Given that such comparable international units may not exist, we applied a synthetic control method to identify an appropriate control group. In essence, we created a unit for each UK university as a weighted average of US universities. Thus we find, for example, that in the field of economics the best match for City University London is a combination of City University of Delaware, Florida Atlantic University, and the University of Georgia. Once we have a comparable unit, we can compare a UK university’s research activity with its counterfactual to determine the causal effect of REF on that university’s results.
We applied our approach to evaluate the impact of REF 2014 on the research activities of UK universities in the field of economics and business. We focused on REF 2014 because we expected that its additional impact on research effectiveness – compared to the previous RAE 2008 – would be greater than in the previous and subsequent rounds. Notably, significant changes have been made to REF 2014, such as providing higher rewards for world-leading research (4*) and eliminating payments for research recognized only internationally (2*) or nationally (1*).
Some of the university performance measures we use include the number of publications, the number of publications in leading journals, the number of publications per author, and the number of publications in leading journals per capita. As shown in Figure 1, the number of publications, as well as publications in leading journals, increased on average for UK universities throughout the 2014 REF evaluation period. However, the same measures also increased on average for universities in other countries, such as the US. The key is to determine whether the research activity of UK universities has increased more than it would have in the absence of the REF. Hence the need to find (or construct!) suitable counterfactuals for every British university.
Figure 1. Evolution of research outcome measures over time. This figure shows the evolution over time of the mean annual values of the outcomes of interest, separated by the UK (solid line) and the US (dashed line).
Our results show that REF 2014 significantly increased UK university research results on average compared to their intended results. Indeed, the number of UK chapter publications increased compared to their US controls throughout the 2009-2014 treatment period, but especially towards the end of that period (2012-2014). The high quality of research, as measured by the number of publications in leading journals, also increased, albeit to a lesser extent, again, especially towards the end of the evaluation period. However, the number of publications per author and the number of publications in leading journals per author did not change relative to the counterfactual, as the number of authors in British universities also increased.
This suggests that the REF has not resulted in an overall increase in UK universities’ research productivity or productivity in the area of excellence. Instead of succeeding in improving their research environment by investing in processes and support strategies that would enable their existing scientists to become more productive, it seems that universities have had to hire more researchers either from other UK universities or from the international market in order to work better. . While this may not be a bad outcome, apart from the question of the sustainability of such strategies, this behavior could lead to the migration of research output from one university to another or from one region to another, resulting in a reallocation rather than an overall improvement in the research capacity of the UK research system.
Our results support concerns that REF may have increased the concentration of research results at elite universities. REF 2014 really strengthened the already strong position of the Russell Group Universities. When comparing Russell Group universities with non-Russell Group universities, our results show that while non-Russell Group universities on average experienced greater growth in the share of publications in leading economics journals, Russell Group universities collectively experienced higher growth in the total number of publications on economics and business and publications in leading business journals. However, the results in each group are not the same, as shown in Figure 2, and only a very few universities in each group outperformed their counterfactual performance in at least one performance measure in a statistically significant way. The rest of the universities were either unaffected or affected by the REF (see table A5 in the appendix to the article for more details).
Fig2. Distribution of annual REF treatment effects for all UK universities and Russell and non-Russell groups. On fig. Figure 2 shows the distribution of the number of British university publications that exceed their counterfactuals (left) and the number of publications in leading British university journals that exceed their counterfactuals (right) for all universities and separately for the Russell and non-Russell groups of universities.
This points to the need for future evaluation activities (and other international performance research funding schemes) to encourage and encourage a more inclusive culture of collaboration between domestic higher education institutions rather than to create a competitive environment. A situation that widened the productivity gap and prompted the “losers” to stop applying for REFs altogether (as happened in the economy).
In addition, our approach provides a robust data-driven methodology for assessing university research performance against comparable international benchmarks. Allocation of funding based on results meeting these benchmarks would ensure fairer competition that could level the playing field between higher education providers. In addition, research support for non-elite universities can also provide access to research-based and research-oriented education for the less well-off students attending them, offering a real mechanism to “level up” the current system.
The content created on this blog is for informational purposes only. This article represents the views and opinions of the authors, but does not reflect the views and opinions of the Impact of Social Science blog (blog) or the London School of Economics and Political Science. Please see our comment policy if you have any concerns about posting a comment below.
Image credit: Edward Howell via Unsplash.com.
