Generating Chest Radiology Report Findings Using a Multimodal Method

Chenyu Wang; Vladimir Janjic; Stephen McKenna

doi:10.1007/978-3-031-66955-2_13

Generating Chest Radiology Report Findings Using a Multimodal Method

Chenyu Wang, Vladimir Janjic, Stephen McKenna (Lead / Corresponding author)

Computing

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Automatic report generation from chest x-ray imaging (CXR) could potentially alleviate the workload of radiologists and improve clinical efficacy. We introduce a multimodal approach, that integrates radiology images with text describing patients’ indications, to generate the findings section in radiology reports. We instantiate this approach by building on two existing methods, R2Gen and CvT2DistilGPT2. We report experiments on two public datasets, MIMIC-CXR and IU X-ray, using evaluation metrics for natural language generation and clinical efficacy assessment. Results show that improvements across all metrics are obtained through the incorporation of indications text. For example, we obtain 35% and 8% increases in BLEU-4 and F1 scores, respectively.

Original language	English
Title of host publication	Medical Image Understanding and Analysis
Subtitle of host publication	28th Annual Conference, MIUA 2024, Proceedings
Editors	Moi Hoon Yap, Connah Kendrick, Ardhendu Behera, Timothy Cootes, Reyer Zwiggelaar
Publisher	Springer
Pages	188-201
Number of pages	14
Volume	1
ISBN (Print)	9783031669545
DOIs	https://doi.org/10.1007/978-3-031-66955-2_13
Publication status	Published - 24 Jul 2024
Event	28th Annual Conference on Medical Image Understanding and Analysis - Manchester Metropolitan University, Manchester, United Kingdom Duration: 24 Jul 2024 → 26 Jul 2024 https://miua2024.github.io/ (Link to Conference Website)

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	14859 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	28th Annual Conference on Medical Image Understanding and Analysis
Abbreviated title	MIUA 2024
Country/Territory	United Kingdom
City	Manchester
Period	24/07/24 → 26/07/24
Internet address	https://miua2024.github.io/ (Link to Conference Website)

Keywords

Chest X-ray
Convolutional neural network
Multimodal Learning
Radiology report generation
Transformer

ASJC Scopus subject areas

Theoretical Computer Science
General Computer Science

Access to Document

10.1007/978-3-031-66955-2_13

Cite this

Wang, C., Janjic, V., & McKenna, S. (2024). Generating Chest Radiology Report Findings Using a Multimodal Method. In M. H. Yap, C. Kendrick, A. Behera, T. Cootes, & R. Zwiggelaar (Eds.), Medical Image Understanding and Analysis: 28th Annual Conference, MIUA 2024, Proceedings (Vol. 1, pp. 188-201). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14859 LNCS). Springer . https://doi.org/10.1007/978-3-031-66955-2_13

Wang, Chenyu ; Janjic, Vladimir ; McKenna, Stephen. / Generating Chest Radiology Report Findings Using a Multimodal Method. Medical Image Understanding and Analysis: 28th Annual Conference, MIUA 2024, Proceedings. editor / Moi Hoon Yap ; Connah Kendrick ; Ardhendu Behera ; Timothy Cootes ; Reyer Zwiggelaar. Vol. 1 Springer , 2024. pp. 188-201 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{e432f44dcb1b4b4caaa7088afc28b346,

title = "Generating Chest Radiology Report Findings Using a Multimodal Method",

abstract = "Automatic report generation from chest x-ray imaging (CXR) could potentially alleviate the workload of radiologists and improve clinical efficacy. We introduce a multimodal approach, that integrates radiology images with text describing patients{\textquoteright} indications, to generate the findings section in radiology reports. We instantiate this approach by building on two existing methods, R2Gen and CvT2DistilGPT2. We report experiments on two public datasets, MIMIC-CXR and IU X-ray, using evaluation metrics for natural language generation and clinical efficacy assessment. Results show that improvements across all metrics are obtained through the incorporation of indications text. For example, we obtain 35% and 8% increases in BLEU-4 and F1 scores, respectively.",

keywords = "Chest X-ray, Convolutional neural network, Multimodal Learning, Radiology report generation, Transformer",

author = "Chenyu Wang and Vladimir Janjic and Stephen McKenna",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.; 28th Annual Conference on Medical Image Understanding and Analysis, MIUA 2024 ; Conference date: 24-07-2024 Through 26-07-2024",

year = "2024",

month = jul,

day = "24",

doi = "10.1007/978-3-031-66955-2_13",

language = "English",

isbn = "9783031669545",

volume = "1",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer ",

pages = "188--201",

editor = "Yap, {Moi Hoon} and Connah Kendrick and Ardhendu Behera and Timothy Cootes and Reyer Zwiggelaar",

booktitle = "Medical Image Understanding and Analysis",

url = "https://miua2024.github.io/",

}

Wang, C , Janjic, V & McKenna, S 2024, Generating Chest Radiology Report Findings Using a Multimodal Method. in MH Yap, C Kendrick, A Behera, T Cootes & R Zwiggelaar (eds), Medical Image Understanding and Analysis: 28th Annual Conference, MIUA 2024, Proceedings. vol. 1, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 14859 LNCS, Springer , pp. 188-201, 28th Annual Conference on Medical Image Understanding and Analysis, Manchester, United Kingdom, 24/07/24. https://doi.org/10.1007/978-3-031-66955-2_13

Generating Chest Radiology Report Findings Using a Multimodal Method. / Wang, Chenyu ; Janjic, Vladimir ; McKenna, Stephen (Lead / Corresponding author).
Medical Image Understanding and Analysis: 28th Annual Conference, MIUA 2024, Proceedings. ed. / Moi Hoon Yap; Connah Kendrick; Ardhendu Behera; Timothy Cootes; Reyer Zwiggelaar. Vol. 1 Springer , 2024. p. 188-201 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14859 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Generating Chest Radiology Report Findings Using a Multimodal Method

AU - Wang, Chenyu

AU - Janjic, Vladimir

AU - McKenna, Stephen

N1 - Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.

PY - 2024/7/24

Y1 - 2024/7/24

N2 - Automatic report generation from chest x-ray imaging (CXR) could potentially alleviate the workload of radiologists and improve clinical efficacy. We introduce a multimodal approach, that integrates radiology images with text describing patients’ indications, to generate the findings section in radiology reports. We instantiate this approach by building on two existing methods, R2Gen and CvT2DistilGPT2. We report experiments on two public datasets, MIMIC-CXR and IU X-ray, using evaluation metrics for natural language generation and clinical efficacy assessment. Results show that improvements across all metrics are obtained through the incorporation of indications text. For example, we obtain 35% and 8% increases in BLEU-4 and F1 scores, respectively.

AB - Automatic report generation from chest x-ray imaging (CXR) could potentially alleviate the workload of radiologists and improve clinical efficacy. We introduce a multimodal approach, that integrates radiology images with text describing patients’ indications, to generate the findings section in radiology reports. We instantiate this approach by building on two existing methods, R2Gen and CvT2DistilGPT2. We report experiments on two public datasets, MIMIC-CXR and IU X-ray, using evaluation metrics for natural language generation and clinical efficacy assessment. Results show that improvements across all metrics are obtained through the incorporation of indications text. For example, we obtain 35% and 8% increases in BLEU-4 and F1 scores, respectively.

KW - Chest X-ray

KW - Convolutional neural network

KW - Multimodal Learning

KW - Radiology report generation

KW - Transformer

U2 - 10.1007/978-3-031-66955-2_13

DO - 10.1007/978-3-031-66955-2_13

M3 - Conference contribution

AN - SCOPUS:85200682033

SN - 9783031669545

VL - 1

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 188

EP - 201

BT - Medical Image Understanding and Analysis

A2 - Yap, Moi Hoon

A2 - Kendrick, Connah

A2 - Behera, Ardhendu

A2 - Cootes, Timothy

A2 - Zwiggelaar, Reyer

PB - Springer

T2 - 28th Annual Conference on Medical Image Understanding and Analysis

Y2 - 24 July 2024 through 26 July 2024

ER -

Wang C , Janjic V , McKenna S. Generating Chest Radiology Report Findings Using a Multimodal Method. In Yap MH, Kendrick C, Behera A, Cootes T, Zwiggelaar R, editors, Medical Image Understanding and Analysis: 28th Annual Conference, MIUA 2024, Proceedings. Vol. 1. Springer . 2024. p. 188-201. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-66955-2_13