Reproduce “Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, Noah D. Goodman

Researchers from Stanford University found that a language model's initial capacity to exhibit cognitive behaviors such as verification and backtracking dictates its ability to self-improve through reinforcement learning. This self-improvement capability can be induced by priming models with data rich in these behaviors or by curating pretraining data, even when the priming data contains incorrect solutions.

Key insights

  • A model's initial policy exhibiting specific cognitive behaviors (e.g., verification, backtracking) is a strong predictor and causal factor for its capacity to self-improve through reinforcement learning.
  • The structure and presence of these reasoning patterns, rather than the correctness of the solutions in priming data, are crucial for enabling subsequent self-improvement.
  • The ability for LLMs to self-improve can be engineered by targeted fine-tuning with behavior-rich data or by augmenting pretraining datasets to amplify desired cognitive behaviors.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.