Evaluation of large language models with clinical guidance for vetting outpatient magnetic resonance imaging lumbar spine referrals

TY - JOUR

T1 - Evaluation of large language models with clinical guidance for vetting outpatient magnetic resonance imaging lumbar spine referrals

AU - Clackett, William

AU - Alsusa, Hatim

AU - Watson, Hannah

AU - Kascenas, Antanas

AU - Scott, David

AU - Kanodia, Avinash K.

AU - Barry, Oliver T.

AU - O’Neil, Alison Q.

N1 - Publisher Copyright: © The Author(s) 2026

PY - 2026/5

Y1 - 2026/5

N2 - Objectives: Accurate triage of lumbar spine magnetic resonance imaging (MRI) referrals for sciatica is important for patient assessment, diagnosis and surgical planning. This study evaluates the accuracy and speed of large language models (LLMs) in automatically vetting lumbar spine MRI referrals from general practice. Methods: Three LLMs (GPT-4, Claude Opus, Gemini) were tasked with assigning an outcome (Accept – Routine, Accept – Urgent, Reject) and flagging MRI contraindications for lumbar spine referrals. Three prompts of increasing detail, including clinical guidelines and training examples, were used. Two radiology registrars synthesised 120 referrals, vetted by two board-certified radiologists, with a third resolving disagreements. Performance was assessed using accuracy, precision, recall and F1 scores. Results: Inter-rater agreement between radiologists was substantial for vetting outcome (Cohen's κ = 0.76) and contraindication detection (κ = 0.68). Claude Opus with the full prompt achieved the highest accuracy (0.86) for vetting outcomes. GPT-4 with the instruction-only prompt achieved the highest F1 score (0.88) for contraindication detection. LLMs completed the task substantially faster than radiologists (9.8 ± 1.0 vs 135.0 ± 45.0 min). Conclusions: LLMs demonstrate promising performance in vetting radiological referrals for sciatica, particularly with detailed context. All models identified all urgent referrals, suggesting potential for prioritising vetting worklists and improving timeliness of care.

AB - Objectives: Accurate triage of lumbar spine magnetic resonance imaging (MRI) referrals for sciatica is important for patient assessment, diagnosis and surgical planning. This study evaluates the accuracy and speed of large language models (LLMs) in automatically vetting lumbar spine MRI referrals from general practice. Methods: Three LLMs (GPT-4, Claude Opus, Gemini) were tasked with assigning an outcome (Accept – Routine, Accept – Urgent, Reject) and flagging MRI contraindications for lumbar spine referrals. Three prompts of increasing detail, including clinical guidelines and training examples, were used. Two radiology registrars synthesised 120 referrals, vetted by two board-certified radiologists, with a third resolving disagreements. Performance was assessed using accuracy, precision, recall and F1 scores. Results: Inter-rater agreement between radiologists was substantial for vetting outcome (Cohen's κ = 0.76) and contraindication detection (κ = 0.68). Claude Opus with the full prompt achieved the highest accuracy (0.86) for vetting outcomes. GPT-4 with the instruction-only prompt achieved the highest F1 score (0.88) for contraindication detection. LLMs completed the task substantially faster than radiologists (9.8 ± 1.0 vs 135.0 ± 45.0 min). Conclusions: LLMs demonstrate promising performance in vetting radiological referrals for sciatica, particularly with detailed context. All models identified all urgent referrals, suggesting potential for prioritising vetting worklists and improving timeliness of care.

KW - Large language models

KW - lumbar spine magnetic resonance imaging

KW - natural language processing

KW - referral prioritisation

UR - https://www.scopus.com/pages/publications/105036229274

U2 - 10.1177/00369330261441582

DO - 10.1177/00369330261441582

M3 - Article

C2 - 41989203

AN - SCOPUS:105036229274

SN - 0036-9330

VL - 71

SP - 43

EP - 53

JO - Scottish Medical Journal

JF - Scottish Medical Journal

IS - 2

ER -