TY - JOUR
T1 - Standardized Assessment of Automatic Segmentation of White Matter Hyperintensities
T2 - Results of the WMH Segmentation Challenge
AU - Kuijf, Hugo J.
AU - Casamitjana, Adrià
AU - Collins, D. Louis
AU - Dadar, Mahsa
AU - Georgiou, Achilleas
AU - Ghafoorian, Mohsen
AU - Jin, Dakai
AU - Khademi, April
AU - Knight, Jesse
AU - Li, Hongwei
AU - Lladó, Xavier
AU - Biesbroek, J. Matthijs
AU - Luna, Miguel
AU - Mahmood, Qaiser
AU - Mckinley, Richard
AU - Mehrtash, Alireza
AU - Ourselin, Sebastien
AU - Park, Bo Yong
AU - Park, Hyunjin
AU - Park, Sang Hyun
AU - Pezold, Simon
AU - Puybareau, Elodie
AU - De Bresser, Jeroen
AU - Rittner, Leticia
AU - Sudre, Carole H.
AU - Valverde, Sergi
AU - Vilaplana, Veronica
AU - Wiest, Roland
AU - Xu, Yongchao
AU - Xu, Ziyue
AU - Zeng, Guodong
AU - Zheng, Guoyan
AU - Heinen, Rutger
AU - Chen, Christopher
AU - Van Der Flier, Wiesje
AU - Barkhof, Frederik
AU - Viergever, Max A.
AU - Biessels, Geert Jan
AU - Andermatt, Simon
AU - Bento, Mariana
AU - Berseth, Matt
AU - Belyaev, Mikhail
AU - Cardoso, M. Jorge
AU - Zhang, Jianguo
N1 - Funding Information:
The work of H. J. Kuijf was supported by The Netherlands Organization for Health Research and Development (ZonMW) through the Off Road Grant under Grant 451001007. The work of S. Andermatt was supported by the MIAC AG, Basel, Switzerland. The work of M. Bento and L. Rittner was supported in part by the Hotchkiss Brain Institute and in part by CAPES process PVE underGrant 88881.062158/2014-01. The work of A. Casamitjana was supported in part by the Spanish Ministerio de Econom?a y Competitividad through the project MALEGRA under Grant TEC2016-75976-R, in part by the European Regional Development Fund (ERDF), and in part by the Spanish Ministerio de Educacin, Cultura y Deporte FPU Research Fellowship. The work of D. Jin and Z. Xu was supported by the National Institute of Allergy and Infectious Diseases, USA, through the Intramural Research Program. The work of A. Khademi and J. Knight was supported in part by the Natural Science and Engineering Research Council of Canada (NSERC CGS-M) and in part by the Ontario Ministry of Advanced Education and Skills Development (OGS-M). The work of H. Li and J. Zhang was supported by the National Natural Science Foundation of China under Grant 61628212. The work of X. Llad? and S. Valverde was supported in part by the Ministerio de Ciencia y Tecnolog?a, Spain, under Grant TIN2014-55710-R and Grant DPI2017- 86696-R. The work of M. Luna and S. H. Park was supported by the National Research Foundation of Korea (NRF) through the Basic Science Research Program funded by the Ministry of Education under Grant 2018R1D1A1B07044473. The work of R. McKinley and R. Wiest was supported by the Swiss Multiple Sclerosis Society. The work of A. Mehrtash was supported in part by the U.S. National Institutes of Health under Grant P41EB015898, in part by the Natural Sciences and Engineering Research Council (NSERC) of Canada, and in part by the Canadian Institutes of Health Research (CIHR). The work of C. H. Sudre was supported by the Alzheimer's Society Junior Research Fellowship under Grant AS-JF-17-011. The work of V. Vilaplana was supported in part by the SpanishMinisterio de Econom?a y Competitividad through the project MALEGRA under Grant TEC2016-75976-R and in part by the European Regional Development Fund (ERDF). The work of G. Zeng and G. Zheng was supported in part by the Swiss National Science Foundation under Project 205321-163224. The work of F. Barkhof was supported by the NIHR UCLH Biomedical Research Centre. The work of G. J. Biessels was supported by The Netherlands Organization for Scientific Research (NWO) through the VICI under Grant 918.16.616.
Funding Information:
Manuscript received January 24, 2019; revised March 11, 2019; accepted March 13, 2019. Date of publication March 19, 2019; date of current version October 25, 2019. The work of H. J. Kuijf was supported by The Netherlands Organization for Health Research and Development (ZonMW) through the Off Road Grant under Grant 451001007. The work of S. Andermatt was supported by the MIAC AG, Basel, Switzerland. The work of M. Bento and L. Rittner was supported in part by the Hotchkiss Brain Institute and in part by CAPES process PVE under Grant 88881.062158/2014-01. The work of A. Casamitjana was supported in part by the Spanish Ministerio de Economía y Competitividad through the project MALEGRA under Grant TEC2016-75976-R, in part by the European Regional Development Fund (ERDF), and in part by the Spanish Ministerio de Educacin, Cultura y Deporte FPU Research Fellowship. The work of D. Jin and Z. Xu was supported by the National Institute of Allergy and Infectious Diseases, USA, through the Intramural Research Program. The work of A. Khademi and J. Knight was supported in part by the Natural Science and Engineering Research Council of Canada (NSERC CGS-M) and in part by the Ontario Ministry of Advanced Education and Skills Development (OGS-M). The work of H. Li and J. Zhang was supported by the National Natural Science Foundation of China under Grant 61628212. The work of X. Lladó and S. Valverde was supported in part by the Ministerio de Ciencia y Tecnología, Spain, under Grant TIN2014-55710-R and Grant DPI2017-86696-R. The work of M. Luna and S. H. Park was supported by the National Research Foundation of Korea (NRF) through the Basic Science Research Program funded by the Ministry of Education under Grant 2018R1D1A1B07044473. The work of R. McKinley and R. Wiest was supported by the Swiss Multiple Sclerosis Society. The work of A. Mehrtash was supported in part by the U.S. National Institutes of Health under Grant P41EB015898, in part by the Natural Sciences and Engineering Research Council (NSERC) of Canada, and in part by the Canadian Institutes of Health Research (CIHR). The work of C. H. Sudre was supported by the Alzheimer’s Society Junior Research Fellowship under Grant AS-JF-17-011. The work of V. Vilaplana was supported in part by the Spanish Ministerio de Economía y Competitividad through the project MALEGRA under Grant TEC2016-75976-R and in part by the European Regional Development Fund (ERDF). The work of G. Zeng and G. Zheng was supported in part by the Swiss National Science Foundation under Project 205321_163224. The work of F. Barkhof was supported by the NIHR UCLH Biomedical Research Centre. The work of G. J. Biessels was supported by The Netherlands Organization for Scientific Research (NWO) through the VICI under Grant 918.16.616. (Corresponding author: Hugo Kuijf.) Please see the Acknowledgment section of this paper for the author affiliations.
Publisher Copyright:
© 2019 IEEE.
PY - 2019/11
Y1 - 2019/11
N2 - Quantification of cerebral white matter hyperintensities (WMH) of presumed vascular origin is of key importance in many neurological research studies. Currently, measurements are often still obtained from manual segmentations on brain MR images, which is a laborious procedure. Automatic WMH segmentation methods exist, but a standardized comparison of the performance of such methods is lacking. We organized a scientific challenge, in which developers could evaluate their method on a standardized multi-center/-scanner image dataset, giving an objective comparison: the WMH Segmentation Challenge (https://wmh.isi.uu.nl/).Sixty T1+FLAIR images from three MR scanners were released with manual WMH segmentations for training. A test set of 110 images from five MR scanners was used for evaluation. Segmentation methods had to be containerized and submitted to the challenge organizers. Five evaluation metrics were used to rank the methods: (1) Dice similarity coefficient, (2) modified Hausdorff distance (95th percentile), (3) absolute log-transformed volume difference, (4) sensitivity for detecting individual lesions, and (5) F1-score for individual lesions. Additionally, methods were ranked on their inter-scanner robustness.Twenty participants submitted their method for evaluation.This paper provides a detailed analysis of the results. In brief, there is a cluster of four methods that rank significantly better than the other methods, with one clear winner. The inter-scanner robustness ranking shows that not all methods generalize to unseen scanners.The challenge remains open for future submissions and provides a public platform for method evaluation.
AB - Quantification of cerebral white matter hyperintensities (WMH) of presumed vascular origin is of key importance in many neurological research studies. Currently, measurements are often still obtained from manual segmentations on brain MR images, which is a laborious procedure. Automatic WMH segmentation methods exist, but a standardized comparison of the performance of such methods is lacking. We organized a scientific challenge, in which developers could evaluate their method on a standardized multi-center/-scanner image dataset, giving an objective comparison: the WMH Segmentation Challenge (https://wmh.isi.uu.nl/).Sixty T1+FLAIR images from three MR scanners were released with manual WMH segmentations for training. A test set of 110 images from five MR scanners was used for evaluation. Segmentation methods had to be containerized and submitted to the challenge organizers. Five evaluation metrics were used to rank the methods: (1) Dice similarity coefficient, (2) modified Hausdorff distance (95th percentile), (3) absolute log-transformed volume difference, (4) sensitivity for detecting individual lesions, and (5) F1-score for individual lesions. Additionally, methods were ranked on their inter-scanner robustness.Twenty participants submitted their method for evaluation.This paper provides a detailed analysis of the results. In brief, there is a cluster of four methods that rank significantly better than the other methods, with one clear winner. The inter-scanner robustness ranking shows that not all methods generalize to unseen scanners.The challenge remains open for future submissions and provides a public platform for method evaluation.
KW - brain
KW - evaluation and performance
KW - Magnetic resonance imaging (MRI)
KW - segmentation
UR - http://www.scopus.com/inward/record.url?scp=85074378885&partnerID=8YFLogxK
U2 - 10.1109/TMI.2019.2905770
DO - 10.1109/TMI.2019.2905770
M3 - Article
C2 - 30908194
AN - SCOPUS:85074378885
SN - 0278-0062
VL - 38
SP - 2556
EP - 2568
JO - IEEE Transactions on Medical Imaging
JF - IEEE Transactions on Medical Imaging
IS - 11
M1 - 8669968
ER -