Predicting Radiologists' Gaze with Computational Saliency Models in Mammogram Reading

TY - JOUR

T1 - Predicting Radiologists' Gaze with Computational Saliency Models in Mammogram Reading

AU - Lou, Jianxun

AU - Lin, Hanhe

AU - Young, Philippa

AU - White, Richard

AU - Yang, Zelei

AU - Shelmerdine, Susan

AU - Marshall, David

AU - Spezi, Emiliano

AU - Palombo, Marco

AU - Liu, Hantao

N1 - Copyright: © 2023 IEEE.

PY - 2024

Y1 - 2024

N2 - Previous studies have shown that there is a strong correlation between radiologists' diagnoses and their gaze when reading medical images. The extent to which gaze is attracted by content in a visual scene can be characterised as visual saliency. There is a potential for the use of visual saliency in computer-aided diagnosis in radiology. However, little is known about what methods are effective for diagnostic images, and how these methods could be adapted to address specific applications in diagnostic imaging. In this study, we investigate 20 state-of-the-art saliency models including 10 traditional models and 10 deep learning-based models in predicting radiologists' visual attention while reading 196 mammograms. We found that deep learning-based models represent the most effective type of methods for predicting radiologists' gaze in mammogram reading; and that the performance of these saliency models can be significantly improved by transfer learning. In particular, an enhanced model can be achieved by pre-training the model on a large-scale natural image saliency dataset and then fine-tuning it on the target medical image dataset. In addition, based on a systematic selection of backbone networks and network architectures, we proposed a parallel multi-stream encoded model which outperforms the state-of-the-art approaches for predicting saliency of mammograms.

AB - Previous studies have shown that there is a strong correlation between radiologists' diagnoses and their gaze when reading medical images. The extent to which gaze is attracted by content in a visual scene can be characterised as visual saliency. There is a potential for the use of visual saliency in computer-aided diagnosis in radiology. However, little is known about what methods are effective for diagnostic images, and how these methods could be adapted to address specific applications in diagnostic imaging. In this study, we investigate 20 state-of-the-art saliency models including 10 traditional models and 10 deep learning-based models in predicting radiologists' visual attention while reading 196 mammograms. We found that deep learning-based models represent the most effective type of methods for predicting radiologists' gaze in mammogram reading; and that the performance of these saliency models can be significantly improved by transfer learning. In particular, an enhanced model can be achieved by pre-training the model on a large-scale natural image saliency dataset and then fine-tuning it on the target medical image dataset. In addition, based on a systematic selection of backbone networks and network architectures, we proposed a parallel multi-stream encoded model which outperforms the state-of-the-art approaches for predicting saliency of mammograms.

KW - Measurement

KW - Deep learning

KW - Visualization

KW - Computational modeling

KW - Transfer learning

KW - Mammograms

KW - Predictive models

KW - Radiology

KW - Mammography

KW - Saliency

KW - Task analysis

KW - Medical diagnostic imaging

KW - transfer learning

KW - mammograms

KW - radiology

KW - saliency

UR - https://www.scopus.com/pages/publications/85153352689

U2 - 10.1109/TMM.2023.3263553

DO - 10.1109/TMM.2023.3263553

M3 - Article

SN - 1941-0077

VL - 26

SP - 256

EP - 269

JO - IEEE Transactions on Multimedia

JF - IEEE Transactions on Multimedia

M1 - 10089554

ER -