Reproduce “II-MMR: Identifying and Improving Multi-modal Multi-hop Reasoning in Visual Question Answering” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jihyung Kil, Farideh Tavazoee, Dongyeop Kang, Joo-Kyung Kim
This work introduces II-MMR, a framework that identifies and quantifies multi-hop reasoning complexity in Visual Question Answering (VQA) benchmarks and enhances V&L model performance on these challenging questions. It achieves this by developing novel prompting strategies that generate high-quality, answer-related multi-modal reasoning paths, leading to notable accuracy improvements on complex multi-hop VQA tasks.
Key insights
- Current VQA benchmarks, such as GQA and A-OKVQA, are heavily skewed towards 0-hop or 1-hop questions, leading to an overestimation of V&L models' true multi-hop reasoning abilities.
- Vision & Language models struggle significantly with genuine multi-hop reasoning, as evidenced by drastic performance drops on 2-hop questions (e.g., BLIP-2 accuracy dropping from ~45% on 0/1-hop GQA questions to 7.66% on 2-hop questions).
- Traditional Chain-of-Thought (CoT) prompting is largely ineffective or even detrimental for multi-modal VQA tasks due to models struggling to visually ground rationales, whereas answer prediction-guided CoT consistently improves performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.