Reproduce “When to Think and When to Look: Uncertainty-Guided Lookback” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jing Bi, Filippos Bellos, Junjia Guo, Yayuan Li, Chao Huang, Yolo Y. Tang, Luchuan Song, Susan Liang and 3 more
A systematic analysis quantifies the impact of test-time "thinking" in large vision-language models (LVLMs) on visual reasoning across different model sizes and task types. The work introduces an uncertainty-guided lookback decoding strategy that achieves accuracy improvements of 2-6.5 points with a 35-45% reduction in token usage across benchmarks including MMMUval, MathVista, and MMBench.
Key insights
- Both increasing reasoning path breadth (sampling) and depth (explicit thinking modes) consistently improve LVLM accuracy, with sampling being particularly beneficial for smaller models and thinking for reasoning-centric tasks.
- 'More thinking' is not always efficient; it can double token usage for smaller models on easy tasks without proportional accuracy gains, and 'long-wrong' trajectories represent a notable inefficiency in recognition-heavy domains.
- Token-level visual sensitivity can be effectively probed to identify textual cues (pause phrases) that signal model uncertainty or drift from visual grounding, providing a basis for adaptive decoding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.