The accuracy and reliability of crowdsource annotations of digital retinal images

Danny Mitry (Lead / Corresponding author), Kris Zutis (Lead / Corresponding author), Baljean Dhillon, Tunde Peto, Shabina Hayat, Kay-Tee Khaw, James E. Morgan, Wendy Moncur, Emanuele Trucco, Paul J. Foster, UK Biobank Eye and Vision Consortium

Research output: Contribution to journalArticle

108 Downloads (Pure)


PURPOSE: Crowdsourcing is based on outsourcing computationally intensive tasks to numerous individuals in the online community who have no formal training. Our aim was to develop a novel online tool designed to facilitate large-scale annotation of digital retinal images, and to assess the accuracy of crowdsource grading using this tool, comparing it to expert classification.

METHODS: We used 100 retinal fundus photograph images with predetermined disease criteria selected by two experts from a large cohort study. The Amazon Mechanical Turk Web platform was used to drive traffic to our site so anonymous workers could perform a classification and annotation task of the fundus photographs in our dataset after a short training exercise. Three groups were assessed: masters only, nonmasters only and nonmasters with compulsory training. We calculated the sensitivity, specificity, and area under the curve (AUC) of receiver operating characteristic (ROC) plots for all classifications compared to expert grading, and used the Dice coefficient and consensus threshold to assess annotation accuracy.

RESULTS: In total, we received 5389 annotations for 84 images (excluding 16 training images) in 2 weeks. A specificity and sensitivity of 71% (95% confidence interval [CI], 69%-74%) and 87% (95% CI, 86%-88%) was achieved for all classifications. The AUC in this study for all classifications combined was 0.93 (95% CI, 0.91-0.96). For image annotation, a maximal Dice coefficient (∼0.6) was achieved with a consensus threshold of 0.25.

CONCLUSIONS: This study supports the hypothesis that annotation of abnormalities in retinal images by ophthalmologically naive individuals is comparable to expert annotation. The highest AUC and agreement with expert annotation was achieved in the nonmasters with compulsory training group.

TRANSLATIONAL RELEVANCE: The use of crowdsourcing as a technique for retinal image analysis may be comparable to expert graders and has the potential to deliver timely, accurate, and cost-effective image analysis.

Original languageEnglish
Article number6
Number of pages9
JournalTranslational Vision Science and Technology
Issue number5
Early online date21 Sep 2016
Publication statusPublished - 21 Sep 2016


Cite this

Mitry, D., Zutis, K., Dhillon, B., Peto, T., Hayat, S., Khaw, K-T., Morgan, J. E., Moncur, W., Trucco, E., Foster, P. J., & UK Biobank Eye and Vision Consortium (2016). The accuracy and reliability of crowdsource annotations of digital retinal images. Translational Vision Science and Technology, 5(5), [6].