Predicting Radiologists' Gaze with Computational Saliency Models in Mammogram Reading

Jianxun Lou, Hanhe Lin, Philippa Young, Richard White, Zelei Yang, Susan Shelmerdine, David Marshall, Emiliano Spezi, Marco Palombo, Hantao Liu

Research output: Contribution to journalArticlepeer-review

1 Citation (Scopus)
2 Downloads (Pure)


Previous studies have shown that there is a strong correlation between radiologists' diagnoses and their gaze when reading medical images. The extent to which gaze is attracted by content in a visual scene can be characterised as visual saliency. There is a potential for the use of visual saliency in computer-aided diagnosis in radiology. However, little is known about what methods are effective for diagnostic images, and how these methods could be adapted to address specific applications in diagnostic imaging. In this study, we investigate 20 state-of-the-art saliency models including 10 traditional models and 10 deep learning-based models in predicting radiologists' visual attention while reading 196 mammograms. We found that deep learning-based models represent the most effective type of methods for predicting radiologists' gaze in mammogram reading; and that the performance of these saliency models can be significantly improved by transfer learning. In particular, an enhanced model can be achieved by pre-training the model on a large-scale natural image saliency dataset and then fine-tuning it on the target medical image dataset. In addition, based on a systematic selection of backbone networks and network architectures, we proposed a parallel multi-stream encoded model which outperforms the state-of-the-art approaches for predicting saliency of mammograms.
Original languageEnglish
Article number10089554
Pages (from-to)256-269
Number of pages14
JournalIEEE Transactions on Multimedia
Publication statusE-pub ahead of print - 31 Mar 2023


  • Visualization
  • Predictive models
  • Task analysis
  • Computational modeling
  • Mammography
  • Measurement
  • Medical diagnostic imaging
  • Deep learning
  • Transfer learning
  • Mammograms
  • Radiology
  • Saliency

ASJC Scopus subject areas

  • Signal Processing
  • Electrical and Electronic Engineering
  • Media Technology
  • Computer Science Applications


Dive into the research topics of 'Predicting Radiologists' Gaze with Computational Saliency Models in Mammogram Reading'. Together they form a unique fingerprint.

Cite this