Reproduce “Masked Vision and Language Pre-training with Unimodal and Multimodal Contrastive Losses for Medical Visual Question Answering” on OpenResearch

About this paper

Pengfei Li, Gang Liu, Jinlong He, Zixu Zhao, Shenjun Zhong

Researchers at Harbin Engineering University and Monash Biomedical Imaging developed MUMC, a self-supervised pre-training framework for Medical Visual Question Answering that integrates unimodal and multimodal contrastive losses with an efficient masked image strategy. This approach achieved new state-of-the-art accuracy, including a 14.7% absolute improvement on the PathVQA dataset, by learning enhanced cross-modal and within-modal feature representations.

Key insights

• Simultaneously learning strong representations within each modality (unimodal) and robust alignments between modalities (multimodal) through contrastive losses synergistically enhances feature representation quality for medical VQA.
• A masked image strategy that functions purely as a data augmentation technique, without reconstruction loss, offers an efficient and effective way to enhance visual representations during pre-training.
• The model's ability to attend to clinically relevant image regions, as shown by Grad-CAM visualizations, is crucial for building trust and enabling the adoption of AI systems in medical contexts.

Summary by alphaXiv.