Reproduce “Fleming-VL: Towards Universal Medical Visual Reasoning with Multimodal LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yan Shu, Chi Liu, Robin Chen, Derek Li, Bryan Dai
Fleming-VL advances medical visual reasoning by introducing a unified multimodal large language model capable of interpreting diverse medical imaging, including 2D, 3D, and temporal video data. The system addresses data imbalance through strategic data curation and multi-stage training, achieving state-of-the-art performance across various medical VQA and report generation tasks while demonstrating balanced generalization across modalities.
Key insights
- A unified architecture with a shared vision encoder can effectively process heterogeneous medical modalities (2D, 3D, video) by transforming diverse inputs into a compatible representation, eliminating the need for modality-specific encoders.
- Targeted data synthesis for underrepresented modalities and the creation of holistic video datasets are crucial for achieving balanced performance and robust generalization across the full spectrum of clinical imaging.
- A multi-stage training approach, combining interleaved natural/medical pre-training with instruction tuning and medical-oriented reinforcement learning, fosters genuine clinical reasoning and mitigates shortcut learning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.