ArXiv Preprint
Automated radiology report generation aims at automatically generating a
detailed description of medical images, which can greatly alleviate the
workload of radiologists and provide better medical services to remote areas.
Most existing works pay attention to the holistic impression of medical images,
failing to utilize important anatomy information. However, in actual clinical
practice, radiologists usually locate important anatomical structures, and then
look for signs of abnormalities in certain structures and reason the underlying
disease. In this paper, we propose a novel framework AGFNet to dynamically fuse
the global and anatomy region feature to generate multi-grained radiology
report. Firstly, we extract important anatomy region features and global
features of input Chest X-ray (CXR). Then, with the region features and the
global features as input, our proposed self-adaptive fusion gate module could
dynamically fuse multi-granularity information. Finally, the captioning
generator generates the radiology reports through multi-granularity features.
Experiment results illustrate that our model achieved the state-of-the-art
performance on two benchmark datasets including the IU X-Ray and MIMIC-CXR.
Further analyses also prove that our model is able to leverage the
multi-grained information from radiology images and texts so as to help
generate more accurate reports.
Yuhao Wang, Kai Wang, Xiaohong Liu, Tianrun Gao, Jingyue Zhang, Guangyu Wang
2022-11-21
