Reproduce “Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mengjie Ren, Jie Lou, Boxi Cao, Xueru Wen, Hongyu Lin, Xianpei Han, Le Sun, Xing Yu and 1 more
Correction-Oriented Policy Optimization (CIPO) introduces a method to enhance large language models' reasoning and error correction capabilities by converting on-policy failed trajectories into explicit, directional supervision. This approach led to a 4.55% improvement in mathematical reasoning and a 1.43% gain in code generation over existing baselines, alongside substantial boosts in error correction performance across various tasks.
Key insights
- Converting ambiguous failure signals into precise, directional guidance by requiring the model to 'self-correct' its errors yields a richer and denser learning signal for policy optimization.
- It is possible to derive fine-grained, corrective supervision directly from the model's own on-policy failed rollouts, eliminating the need for external annotations, process reward models, or LLM-based critics.
- Adaptive control mechanisms for replay ratios and risk-averse reward shaping are crucial for maintaining learning stability and preventing the model from 'unlearning' correct behaviors while acquiring new error-correction skills.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.