Reproduce “Best of Both Worlds: Multimodal Contrastive Learning with Tabular and Imaging Data” on OpenResearch

About this paper

Paul Hager, Martin J. Menten, Daniel Rueckert

Researchers from Technical University of Munich and Imperial College London developed a novel multimodal contrastive learning framework that effectively integrates medical imaging and tabular data, learning powerful representations from unlabeled datasets. This approach enables superior unimodal (image-only) prediction for tasks like cardiac risk assessment, particularly improving performance in scenarios with limited labeled data, and introduces a flexible method for incorporating class labels into contrastive learning.

Key insights

• Morphometric tabular features, related to size and shape (e.g., ventricular volume, car length), contribute most significantly to the learned representations due to their direct correlation with image content.
• The proposed 'Label as a Feature' (LaaF) method provides a simple and effective way to integrate ground truth labels into multimodal contrastive learning, outperforming or rivaling existing supervised contrastive strategies, particularly in imbalanced or low-data settings.
• Multimodal pretraining yields superior feature representations for downstream tasks, requiring fewer labeled samples to achieve high performance, especially in data-scarce medical contexts.

Summary by alphaXiv.