Reproduce “Retrieval Meets Reasoning: Even High-school Textbook Knowledge Benefits Multimodal Reasoning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Cheng Tan, Jingxuan Wei, Linzhuang Sun, Zhangyang Gao, Siyuan Li, Bihui Yu, Ruifeng Guo, Stan Z. Li

A training-free framework, Retrieval Meets Reasoning (RMR), integrates multimodal retrieval with in-context learning to cultivate cognitive reasoning abilities in vision-language models. This method utilizes a high-school knowledge library, leading to performance increases of up to +33.67% on ScienceQA and +18.9% on A-OKVQA.

Key insights

  • Integrating retrieval with rationales from simple, well-structured knowledge (like high-school textbooks) can significantly enhance multimodal reasoning in VLMs without requiring fine-tuning.
  • The RMR framework successfully encourages models to move beyond direct answer replication by providing structured QRA triplets, prompting the generation of coherent rationales.
  • An optimal retrieval size of k=3 contextual examples was identified, balancing sufficient information with avoiding noise from excessive or less relevant data.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.