ArXiv Preprint
Multimodal models trained on large natural image-text pair datasets have
exhibited astounding abilities in generating high-quality images. Medical
imaging data is fundamentally different to natural images, and the language
used to succinctly capture relevant details in medical data uses a different,
narrow but semantically rich, domain-specific vocabulary. Not surprisingly,
multi-modal models trained on natural image-text pairs do not tend to
generalize well to the medical domain. Developing generative imaging models
faithfully representing medical concepts while providing compositional
diversity could mitigate the existing paucity of high-quality, annotated
medical imaging datasets. In this work, we develop a strategy to overcome the
large natural-medical distributional shift by adapting a pre-trained latent
diffusion model on a corpus of publicly available chest x-rays (CXR) and their
corresponding radiology (text) reports. We investigate the model's ability to
generate high-fidelity, diverse synthetic CXR conditioned on text prompts. We
assess the model outputs quantitatively using image quality metrics, and
evaluate image quality and text-image alignment by human domain experts. We
present evidence that the resulting model (RoentGen) is able to create visually
convincing, diverse synthetic CXR images, and that the output can be controlled
to a new extent by using free-form text prompts including radiology-specific
language. Fine-tuning this model on a fixed training set and using it as a data
augmentation method, we measure a 5% improvement of a classifier trained
jointly on synthetic and real images, and a 3% improvement when trained on a
larger but purely synthetic training set. Finally, we observe that this
fine-tuning distills in-domain knowledge in the text-encoder and can improve
its representation capabilities of certain diseases like pneumothorax by 25%.
Pierre Chambon, Christian Bluethgen, Jean-Benoit Delbrouck, Rogier Van der Sluijs, Małgorzata Połacin, Juan Manuel Zambrano Chaves, Tanishq Mathew Abraham, Shivanshu Purohit, Curtis P. Langlotz, Akshay Chaudhari
2022-11-23
