Reproduce “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu and 192 more

The paper tests whether reinforcement learning from verifiable answers can shape reasoning without human-written solution traces. DeepSeek-R1-Zero starts from pretrained DeepSeek-V3-Base, skips supervised fine-tuning, and uses answer and format rewards with group-relative updates. On AIME 2024, reported pass@1 rises from 15.6% to 77.9%, while self-consistency reaches 86.7%; this applies to verifiable tasks, not open-ended writing.

Key insights

  • Sophisticated reasoning capabilities can organically emerge in LLMs through outcome-based reinforcement learning, rather than requiring explicit human-annotated reasoning trajectories.
  • Reinforcement learning incentivizes emergent behaviors such as self-reflection, verification, and dynamic strategy adaptation, with models generating longer 'thinking chains' for harder problems.
  • Bypassing conventional supervised fine-tuning for reasoning tasks can allow LLMs to discover novel and potentially superior problem-solving strategies beyond human cognitive biases.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.