Estimating expert review quality scores for journal articles with bibliometrics and artificial intelligence

Mike Thelwall, Kayvan Kousha, Paul Wilson, Mahshid Abdoli, Meiko Makita, Emma Stuart, Jonathan Levitt

Research output: Contribution to conferencePaperpeer-review

4 Downloads (Pure)


Bibliometric indicators are often used to help estimate the quality of academic journal articles, but with little systematic evidence about the relationship between the two. This talk summarises the findings of studies of this relationship for quality scores from the UK Research Excellence Framework (REF) 2021, the world’s most financially important systematic academic expert review exercise. It also reports the extent to which machine learning can estimate the quality of journal articles from bibliometric information and metadata. These studies drew on an almost complete set of individual article quality scores from REF2021 – the largest science-wide data set of this nature. The bibliometric analyses surprisingly suggest that there may be a positive association between quality scores and article or journal citation rates in all broad fields. This relationship is never very strong and is weak in some social sciences and in the arts and humanities. The machine learning analyses developed a range of strategies to predict quality scores alongside expert reviewers to reduce the human labour needed for reviewing, then asked a sample of the original expert reviewers to comment on the desirability of the strategies. Although a technically desirable strategy was developed, its ethical implications led to it not being recommended for future REFs.
Original languageEnglish
Publication statusPublished - 12 Oct 2023
Event28th Nordic Workshop on Bibliometrics and Research Policy: NWB2023 - Chalmers University of Technology, Gothenburg, Sweden
Duration: 11 Oct 202313 Oct 2023
Conference number: 28


Workshop28th Nordic Workshop on Bibliometrics and Research Policy
Internet address


  • Research Excellence Framework (REF)
  • machine learning
  • Artificial intelligence and machine learning
  • Nordic Workshop on Bibliometrics and Research Policy


Dive into the research topics of 'Estimating expert review quality scores for journal articles with bibliometrics and artificial intelligence'. Together they form a unique fingerprint.

Cite this