Reproduce “Medical Large Vision Language Models with Multi-Image Visual Ability” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xikai Yang, Juzheng Miao, Yuchen Yuan, Jiaze Wang, Qi Dou, Jinpeng Li, Pheng-Ann Heng
Researchers at The Chinese University of Hong Kong introduced the Medical Multi-IMage (Med-MIM) instruction dataset and benchmark, featuring 83.2K samples, to address the multi-image visual understanding gap in medical Large Vision-Language Models (LVLMs). Their instruction-tuned models, Med-Mantis and MIM-LLaVA-Med, showed superior performance in multi-image medical question answering tasks requiring temporal understanding, reasoning, comparison, and co-reference.
Key insights
- Each distinct multi-image visual ability (temporal understanding, reasoning, comparison, co-reference) contributes significantly and uniquely to overall model performance, as shown by ablation studies.
- Model performance in multi-image tasks directly correlates with the size of the Med-MIM instruction dataset used for fine-tuning, with pronounced gains observed in specific abilities.
- Integrating a "composed" multi-image instruction dataset, derived from adapting single-image QA pairs, substantially enhances both closed-type and open-type multi-image question answering.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.