On clinical agreement on the visibility and extent of anatomical layers in digital gonio-photographs

Andrea Peroni (Lead / Corresponding author), Anna Paviotti, Mauro Campigotto, Luis A. Pinto, Carlo A. Cutolo, Yue Shi, Caroline Cobb, Jacintha Gong, Sirjhun Patel, Stewart Gillan, Andrew J. Tatham, Emanuele Trucco

Research output: Contribution to journalArticlepeer-review

1 Downloads (Pure)

Abstract

Purpose: To quantitatively evaluate the inter-annotator variability of clinicians tracing the contours of anatomical layers of the iridocorneal angle on digital gonio photographs, thus providing a baseline for the validation of automated analysis algorithms.

Methods: Using a software annotation tool on a common set of 20 images, five experienced ophthalmologists highlighted the contours of five anatomical layers of interest: iris root (IR), ciliary body band (CBB), scleral spur (SS), trabecular meshwork (TM), and cornea (C). Inter-annotator variability was assessed by (1) comparing the number of times ophthalmologists delineated each layer in the dataset; (2) quantifying how the consensus area for each layer (i.e., the intersection area of observers’ delineations) varied with the consensus threshold; and (3) calculating agreement among annotators using average per-layer precision, sensitivity, and Dice score.

Results: The SS showed the largest difference in annotation frequency (31%) and the minimum overall agreement in terms of consensus size (∼28% of the labeled pixels). The average annotator’s per-layer statistics showed consistent patterns, with lower agreement on the CBB and SS (average Dice score ranges of 0.61–0.7 and 0.73–0.78, respectively) and better agreement on the IR, TM, and C (average Dice score ranges of 0.97–0.98, 0.84–0.9, and 0.93–0.96, respectively).

Conclusions: There was considerable inter-annotator variation in identifying contours of some anatomical layers in digital gonio photographs. Our pilot indicates that agreement was best on IR, TM, and C but poorer for CBB and SS.

Translational Relevance: This study provides a comprehensive description of inter-annotator agreement on digital gonio photographs segmentation as a baseline for validating deep learning models for automated gonioscopy.

Original languageEnglish
Article number1
Number of pages10
JournalTranslational Vision Science and Technology
Volume10
Issue number11
DOIs
Publication statusPublished - 1 Sep 2021

Keywords

  • inter-annotator variability
  • automated gonioscopy
  • AI software validation
  • Inter-annotator variability
  • Automated gonioscopy

Fingerprint

Dive into the research topics of 'On clinical agreement on the visibility and extent of anatomical layers in digital gonio-photographs'. Together they form a unique fingerprint.

Cite this