ArXiv Preprint
Clinical decision-making in oncology involves multimodal data such as
radiology scans, molecular profiling, histopathology slides, and clinical
factors. Despite the importance of these modalities individually, no deep
learning framework to date has combined them all to predict patient prognosis.
Here, we predict the overall survival (OS) of glioma patients from diverse
multimodal data with a Deep Orthogonal Fusion (DOF) model. The model learns to
combine information from multiparametric MRI exams, biopsy-based modalities
(such as H&E slide images and/or DNA sequencing), and clinical variables into a
comprehensive multimodal risk score. Prognostic embeddings from each modality
are learned and combined via attention-gated tensor fusion. To maximize the
information gleaned from each modality, we introduce a multimodal
orthogonalization (MMO) loss term that increases model performance by
incentivizing constituent embeddings to be more complementary. DOF predicts OS
in glioma patients with a median C-index of 0.788 +/- 0.067, significantly
outperforming (p=0.023) the best performing unimodal model with a median
C-index of 0.718 +/- 0.064. The prognostic model significantly stratifies
glioma patients by OS within clinical subsets, adding further granularity to
prognostic clinical grading and molecular subtyping.
Nathaniel Braman, Jacob W. H. Gordon, Emery T. Goossens, Caleb Willis, Martin C. Stumpe, Jagadish Venkataraman
2021-07-01
