Calibration of Deep Medical Image Classifiers: An Empirical Comparison using Dermatology and Histopathology Datasets

Jacob Carse, Andres Alvarez Olmo, Stephen McKenna (Lead / Corresponding author)

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)
95 Downloads (Pure)

Abstract

As deep learning classifiers become ever more widely deployed for medical image analysis tasks, issues of predictive calibration need to be addressed. Mis-calibration is the deviation between predictive probability (confidence) and classification correctness. Well-calibrated classifiers enable cost-sensitive and selective decision-making. This paper presents an empirical investigation of calibration methods on two medical image datasets (multi-class dermatology and binary histopathology image classification). We show the effect of temperature scaling with temperature optimized using various measures of calibration replacing the standard negative log-likelihood. We do so not only for networks trained using one-hot encoding and cross-entropy loss, but also using focal loss and label smoothing. We compare these with two Bayesian methods. Results suggest little or no advantage to the use of alternative calibration metrics for tuning temperature. Temperature scaling of networks trained using focal loss (with appropriate hyperparameters) provided strong results in terms of both calibration and accuracy across both datasets.
Original languageEnglish
Title of host publicationUncertainty for Safe Utilization of Machine Learning in Medical Imaging
Subtitle of host publication4th International Workshop, UNSURE 2022, Held in Conjunction with MICCAI 2022, Singapore, September 18, 2022, Proceedings
EditorsCarole H. Sudre, Christian F. Baumgartner, Adrian Dalca, Chen Qin, Ryutaro Tanno, Koen Van Leemput, William M. Wells III
Place of PublicationSwitzerland
PublisherSpringer Nature
Pages89-99
Number of pages11
Edition1
ISBN (Electronic)9783031167492
ISBN (Print)9783031167485
DOIs
Publication statusPublished - 2022
EventUncertainty for Safe Utilization of Machine Learning in Medical Imaging - , Singapore
Duration: 22 Sept 202222 Sept 2022
https://unsuremiccai.github.io/

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume13563 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

ConferenceUncertainty for Safe Utilization of Machine Learning in Medical Imaging
Abbreviated titleUNSURE2022
Country/TerritorySingapore
Period22/09/2222/09/22
Internet address

Keywords

  • Calibration
  • Classification
  • Deep learning
  • Uncertainty

ASJC Scopus subject areas

  • Theoretical Computer Science
  • General Computer Science

Fingerprint

Dive into the research topics of 'Calibration of Deep Medical Image Classifiers: An Empirical Comparison using Dermatology and Histopathology Datasets'. Together they form a unique fingerprint.

Cite this