TY - JOUR
T1 - Is big team research fair in national research assessments?
T2 - The case of the UK Research Excellence Framework 2021
AU - Thelwall, Mike
AU - Kousha, Kayvan
AU - Makita, Meiko
AU - Abdoli, Mahshid
AU - Stuart, Emma
AU - Wilson, Paul
AU - Levitt, Jonathan
N1 - Funding Information:
This study was funded by Research England, Scottish Funding Council, Higher Education Funding Council for Wales, and Department for the Economy, Northern Ireland as part of the Future Research Assessment Programme ( https://www.jisc.ac.uk/future-research-assessment-programme ). The content is solely the responsibility of the authors and does not necessarily represent the official views of the funders.
Publisher Copyright:
© 2023 Mike Thelwall et al., published by Sciendo.
PY - 2023
Y1 - 2023
N2 - Collaborative research causes problems for research assessments because of the difficulty in fairly crediting its authors. Whilst splitting the rewards for an article amongst its authors has the greatest surface-level fairness, many important evaluations assign full credit to each author, irrespective of team size. The underlying rationales for this are labour reduction and the need to incentivise collaborative work because it is necessary to solve many important societal problems. This article assesses whether full counting changes results compared to fractional counting in the case of the UK's Research Excellence Framework (REF) 2021. For this assessment, fractional counting reduces the number of journal articles to as little as 10% of the full counting value, depending on the Unit of Assessment (UoA). Despite this large difference, allocating an overall grade point average (GPA) based on full counting or fractional counting gives results with a median Pearson correlation within UoAs of 0.98. The largest changes are for Archaeology (r=0.84) and Physics (r=0.88). There is a weak tendency for higher scoring institutions to lose from fractional counting, with the loss being statistically significant in 5 of the 34 UoAs. Thus, whilst the apparent over-weighting of contributions to collaboratively authored outputs does not seem too problematic from a fairness perspective overall, it may be worth examining in the few UoAs in which it makes the most difference.
AB - Collaborative research causes problems for research assessments because of the difficulty in fairly crediting its authors. Whilst splitting the rewards for an article amongst its authors has the greatest surface-level fairness, many important evaluations assign full credit to each author, irrespective of team size. The underlying rationales for this are labour reduction and the need to incentivise collaborative work because it is necessary to solve many important societal problems. This article assesses whether full counting changes results compared to fractional counting in the case of the UK's Research Excellence Framework (REF) 2021. For this assessment, fractional counting reduces the number of journal articles to as little as 10% of the full counting value, depending on the Unit of Assessment (UoA). Despite this large difference, allocating an overall grade point average (GPA) based on full counting or fractional counting gives results with a median Pearson correlation within UoAs of 0.98. The largest changes are for Archaeology (r=0.84) and Physics (r=0.88). There is a weak tendency for higher scoring institutions to lose from fractional counting, with the loss being statistically significant in 5 of the 34 UoAs. Thus, whilst the apparent over-weighting of contributions to collaboratively authored outputs does not seem too problematic from a fairness perspective overall, it may be worth examining in the few UoAs in which it makes the most difference.
KW - Collaboration
KW - REF
KW - REF2021
KW - Research assessment
KW - Research quality
KW - Scientometrics
UR - http://www.scopus.com/inward/record.url?scp=85149858960&partnerID=8YFLogxK
U2 - 10.2478/jdis-2023-0004
DO - 10.2478/jdis-2023-0004
M3 - Article
SN - 2543-683X
VL - 8
SP - 9
EP - 20
JO - Journal of Data and Information Science
JF - Journal of Data and Information Science
IS - 1
ER -