Reproduce “ReNIO: Reweighting Negative Trajectory Importance for LLM On-Policy Distillation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Chen Lin, Kedi Chen, Wei Zhang

Researchers introduce ReNIO, a reweighting mechanism for on-policy distillation (OPD) in large language models that emphasizes negative (incorrect) reasoning trajectories. ReNIO consistently improved mathematical reasoning and code generation performance by up to 15.44% relative to baselines, doing so while preserving the efficiency of short-prefix training.

Key insights

  • Training LLMs exclusively on incorrect SGOs in an on-policy distillation setting consistently outperforms training only on correct SGOs, revealing a strong learning signal from mistakes.
  • Models trained on incorrect SGOs exhibit more exploratory and self-corrective reasoning behaviors, characterized by longer responses and more reflection-style markers.
  • ReNIO effectively identifies "correctable errors" by assigning higher weights to SGOs where the student deviates but the teacher provides a sharp, confident corrective signal (low entropy).

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.