Reproduce “VLA-Thinker: Boosting Vision-Language-Action Models through Thinking-with-Image Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chaoyang Wang, Wenrui Bao, Sicheng Gao, Bingxin Xu, Yu Tian, Yogesh S. Rawat, Yunhao Ge, Yuzhang Shang
VLA-Thinker introduces a framework that integrates visual perception as an active, dynamically invocable reasoning action within Vision-Language-Action models, moving beyond text-based Chain-of-Thought reasoning. This approach achieved a state-of-the-art 97.5% average success rate on the LIBERO benchmark (+6.5% over baseline) and demonstrated improved robustness in long-horizon tasks on RoboTwin 2.0.
Key insights
- Treating visual perception as a dynamically invocable reasoning action facilitates a more interactive and adaptive perception-reasoning-action loop, enabling robust subgoal tracking and ambiguity resolution, akin to human cognitive processes.
- Synthesizing explicit Chain-of-Thought data from powerful large multimodal models for keyframes and intermediate steps can effectively activate foundational reasoning and structured tool-use priors in VLA models.
- The combination of an SFT cold-start for initial reasoning activation and GRPO for trajectory-level reinforcement learning is complementary and essential for effectively training multimodal reasoning systems under sparse rewards.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.