Knowledge-Driven Subspace Fusion and Gradient Coordination for Multi-modal Learning

TY - GEN

T1 - Knowledge-Driven Subspace Fusion and Gradient Coordination for Multi-modal Learning

AU - Zhang, Yupei

AU - Wang, Xiaofei

AU - Meng, Fangliangzi

AU - Tang, Jin

AU - Li, Chao

N1 - Conference code: 27th

PY - 2024/10/14

Y1 - 2024/10/14

N2 - Multi-modal learning plays a crucial role in cancer diagnosis and prognosis. Current deep learning based multi-modal approaches are often limited by their abilities to model the complex correlations between genomics and histopathology data, addressing the intrinsic complexity of tumour ecosystem where both tumour and microenvironment contribute to malignancy. We propose a biologically interpretative and robust multi-modal learning framework to efficiently integrate histopathology images and genomics by decomposing the feature subspace of histopathology images and genomics, reflecting distinct tumour and microenvironment features. To enhance cross-modal interactions, we design a knowledge-driven subspace fusion scheme, consisting of a cross-modal deformable attention module and a gene-guided consistency strategy. Additionally, in pursuit of dynamically optimizing the subspace knowledge, we further propose a novel gradient coordination learning strategy. Extensive experiments demonstrate the effectiveness of the proposed method, outperforming state-of-the-art techniques in three downstream tasks of glioma diagnosis, tumour grading, and survival analysis. Our code is available at https://github.com/helenypzhang/Subspace-Multimodal-Learning.

AB - Multi-modal learning plays a crucial role in cancer diagnosis and prognosis. Current deep learning based multi-modal approaches are often limited by their abilities to model the complex correlations between genomics and histopathology data, addressing the intrinsic complexity of tumour ecosystem where both tumour and microenvironment contribute to malignancy. We propose a biologically interpretative and robust multi-modal learning framework to efficiently integrate histopathology images and genomics by decomposing the feature subspace of histopathology images and genomics, reflecting distinct tumour and microenvironment features. To enhance cross-modal interactions, we design a knowledge-driven subspace fusion scheme, consisting of a cross-modal deformable attention module and a gene-guided consistency strategy. Additionally, in pursuit of dynamically optimizing the subspace knowledge, we further propose a novel gradient coordination learning strategy. Extensive experiments demonstrate the effectiveness of the proposed method, outperforming state-of-the-art techniques in three downstream tasks of glioma diagnosis, tumour grading, and survival analysis. Our code is available at https://github.com/helenypzhang/Subspace-Multimodal-Learning.

KW - Cancer diagnosis and prognosis

KW - Molecular Pathology

KW - Multi-modal learning

UR - https://www.scopus.com/pages/publications/85207641088

U2 - 10.1007/978-3-031-72083-3_25

DO - 10.1007/978-3-031-72083-3_25

M3 - Conference contribution

AN - SCOPUS:85207641088

SN - 9783031720826

T3 - Lecture Notes in Computer Science

SP - 263

EP - 273

BT - Medical Image Computing and Computer Assisted Intervention

A2 - Linguraru, Marius George

A2 - Dou, Qi

A2 - Feragen, Aasa

A2 - Giannarou, Stamatia

A2 - Glocker, Ben

A2 - Lekadir, Karim

A2 - Schnabel, Julia A.

PB - Springer Nature Switzerland AG

CY - Switzerland

T2 - 27th International Conference on Medical Image Computing and Computer-Assisted Intervention

Y2 - 6 October 2024 through 10 October 2024

ER -