Reproduce “LLaVA-CoT: Let Vision Language Models Reason Step-by-Step” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Guowei Xu, Peng Jin, Ziang Wu, Hao Li, Yibing Song, Lichao Sun, Li Yuan

LLaVA-CoT trains an 11-billion-parameter vision-language model to answer visual questions through four stages: task summary, relevant image description, reasoning, and conclusion. Its SWIRES search can regenerate an earlier stage when later candidates score poorly. On six reasoning-focused benchmark scores, the model with test-time scaling averages 66.3, versus 56.9 for its base model, a benchmark gain rather than evidence of universal superiority.

Key insights

  • Explicitly structured reasoning, guided by distinct stages and tags, is crucial for imbuing VLMs with systematic, in-depth thinking abilities.
  • A high-quality, multistage reasoning dataset (LLaVA-CoT-100k) is essential for training VLMs to adopt and effectively utilize structured thought processes.
  • Dynamic, stage-aligned self-correction during inference, as implemented by SWIRES, significantly improves VLM robustness and accuracy by recovering from suboptimal intermediate steps, outperforming traditional test-time scaling methods.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.