Reproduce “Rationale-Enhanced Decoding for Multi-modal Chain-of-Thought” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shin'ya Yamaguchi, Kosuke Nishida, Daiki Chijiwa

Rationale-Enhanced Decoding (RED), introduced by NTT researchers, improves multi-modal Chain-of-Thought (CoT) reasoning in large vision-language models (LVLMs) by explicitly integrating generated rationales during inference. This plug-and-play method consistently enhances reasoning accuracy across various benchmarks and models, better leveraging rationale content and scaling effectively with larger LVLMs.

Key insights

  • Existing multi-modal CoT approaches in LVLMs often fail to effectively leverage generated rationales, showing reduced contribution of rationale tokens and a lack of semantic grounding.
  • Decoupling next-token prediction into image-conditional and rationale-conditional probabilities, then combining them multiplicatively, forces LVLMs to ground predictions on both inputs.
  • The derived Rationale-Enhanced Decoding (RED) provides a theoretically optimal, training-free strategy to enhance rationale utilization during inference.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.