TY - JOUR
T1 - Variability between human experts and artificial intelligence in identification of anatomical structures by ultrasound in regional anaesthesia
T2 - a framework for evaluation of assistive artificial intelligence
AU - Bowness, James S.
AU - Morse, Robert
AU - Lewis, Owen
AU - Lloyd, James
AU - Burckett-St Laurent, David
AU - Bellew, Boyne
AU - Macfarlane, Alan J.R.
AU - Pawa, Amit
AU - Taylor, Alasdair
AU - Noble, J. Alison
AU - Higham, Helen
N1 - Publisher Copyright:
© 2023 The Author(s)
PY - 2024/5
Y1 - 2024/5
N2 - Background: ScanNavTM Anatomy Peripheral Nerve Block (ScanNav™) is an artificial intelligence (AI)-based device that produces a colour overlay on real-time B-mode ultrasound to highlight key anatomical structures for regional anaesthesia. This study compares consistency of identification of sono-anatomical structures between expert ultrasonographers and ScanNav™. Methods: Nineteen experts in ultrasound-guided regional anaesthesia (UGRA) annotated 100 structures in 30 ultrasound videos across six anatomical regions. These annotations were compared with each other to produce a quantitative assessment of the level of agreement amongst human experts. The AI colour overlay was then compared with all expert annotations. Differences in human–human and human–AI agreement are presented for each structure class (artery, muscle, nerve, fascia/serosal plane) and structure. Clinical context is provided through subjective assessment data from UGRA experts. Results: For human–human and human–AI annotations, agreement was highest for arteries (mean Dice score 0.88/0.86), then muscles (0.80/0.77), and lowest for nerves (0.48/0.41). Wide discrepancy exists in consistency for different structures, both with human–human and human–AI comparisons; highest for sartorius muscle (0.91/0.92) and lowest for the radial nerve (0.21/0.27). Conclusions: Human experts and the AI system both showed the same pattern of agreement in sono-anatomical structure identification. The clinical significance of the differences presented must be explored; however the perception that human expert opinion is uniform must be challenged. Elements of this assessment framework could be used for other devices to allow consistent evaluations that inform clinical training and practice. Anaesthetists should be actively engaged in the development and adoption of new AI technology.
AB - Background: ScanNavTM Anatomy Peripheral Nerve Block (ScanNav™) is an artificial intelligence (AI)-based device that produces a colour overlay on real-time B-mode ultrasound to highlight key anatomical structures for regional anaesthesia. This study compares consistency of identification of sono-anatomical structures between expert ultrasonographers and ScanNav™. Methods: Nineteen experts in ultrasound-guided regional anaesthesia (UGRA) annotated 100 structures in 30 ultrasound videos across six anatomical regions. These annotations were compared with each other to produce a quantitative assessment of the level of agreement amongst human experts. The AI colour overlay was then compared with all expert annotations. Differences in human–human and human–AI agreement are presented for each structure class (artery, muscle, nerve, fascia/serosal plane) and structure. Clinical context is provided through subjective assessment data from UGRA experts. Results: For human–human and human–AI annotations, agreement was highest for arteries (mean Dice score 0.88/0.86), then muscles (0.80/0.77), and lowest for nerves (0.48/0.41). Wide discrepancy exists in consistency for different structures, both with human–human and human–AI comparisons; highest for sartorius muscle (0.91/0.92) and lowest for the radial nerve (0.21/0.27). Conclusions: Human experts and the AI system both showed the same pattern of agreement in sono-anatomical structure identification. The clinical significance of the differences presented must be explored; however the perception that human expert opinion is uniform must be challenged. Elements of this assessment framework could be used for other devices to allow consistent evaluations that inform clinical training and practice. Anaesthetists should be actively engaged in the development and adoption of new AI technology.
KW - artificial intelligence
KW - machine learning
KW - medical devices
KW - regional anaesthesia
KW - sono-anatomy
KW - ultrasonography
KW - validation
UR - http://www.scopus.com/inward/record.url?scp=85175342873&partnerID=8YFLogxK
U2 - 10.1016/j.bja.2023.09.023
DO - 10.1016/j.bja.2023.09.023
M3 - Article
C2 - 39492288
AN - SCOPUS:85175342873
SN - 0007-0912
VL - 132
SP - 1063
EP - 1072
JO - British journal of anaesthesia
JF - British journal of anaesthesia
IS - 5
ER -