Reproduce “MeDSLIP: Medical Dual-Stream Language-Image Pre-training with Pathology-Anatomy Semantic Alignment” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wenrui Fan, Mohammod N.I. Suvon, Shuo Zhou, Xianyuan Liu, Samer Alabed, Venet Osmani, Andrew J. Swift, Chen Chen and 1 more
MeDSLIP is a medical dual-stream language-image pre-training framework that explicitly disentangles pathology and anatomy semantics from medical images and reports. The approach achieves state-of-the-art performance in zero-shot classification and grounding tasks, and demonstrates superior transferability for fine-tuning classification and segmentation, particularly with limited data.
Key insights
- Explicitly disentangling pathology and anatomy semantics in both images and texts leads to a more granular and clinically relevant understanding of medical conditions.
- Modeling the intricate relationships between disentangled pathology and anatomy, through both cross-modal and intra-image contrastive learning, is crucial for enhanced VLM performance.
- Aggregating all positive examples into a prototype vector for contrastive learning improves data efficiency and interaction modeling in complex medical scenarios.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.