Reproduce “Native Reasoning Models: Training Language Models to Reason on Unverifiable Data” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuanfu Wang, Zhixuan Liu, Xiangtian Li, Chaochao Lu, Chao Yang

Native Reasoning Training (NRT) is a framework that trains language models to reason using only question-answer pairs, circumventing the need for human-annotated reasoning traces or external verifiers. This method, which intrinsically rewards reasoning that increases the model's confidence in correct answers, boosted average performance by 10.2 points over SFT on Llama-3.1-8B across nine benchmarks, achieving a 76.0 score on GSM8K compared to SFT's 29.0.

Key insights

  • Principled reward shaping, particularly weighted-sum aggregation that targets difficult tokens, is crucial for cultivating robust reasoning abilities without explicit trace supervision.
  • Language models can intrinsically learn to resolve their own predictive uncertainties by generating reasoning that specifically boosts confidence in previously high-entropy, hard-to-predict tokens.
  • NRT enables the autonomous emergence of a structured reasoning vocabulary and format, allowing models to clearly differentiate between internal thought processes and final answers.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.