Reproduce “R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, Cho-Jui Hsieh
VisualThinker-R1-Zero successfully replicates the "aha moment" of emergent reasoning for multimodal AI by applying reinforcement learning to a 2B non-supervised fine-tuned model. The approach achieved a 59.47% accuracy on CVBench, outperforming supervised fine-tuned models, and demonstrated spontaneous self-reflection and increased reasoning length during training.
Key insights
- Emergent, DeepSeek R1-like reasoning, including an "aha moment" with spontaneous self-reflection, is achievable in multimodal models, even relatively small ones (2B parameters), when using a non-supervised fine-tuned base model.
- Applying simple reinforcement learning techniques to instruction-tuned (SFT) models often leads to superficial reasoning patterns and "reward hacking" rather than genuine, deeper emergent intelligence.
- A "clean slate" non-SFT model is more conducive to fostering autonomous, exploratory reasoning dynamics through simple reinforcement learning, which can result in superior performance on vision-centric tasks compared to SFT models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.