Reproduce “Training Language Models to Fail Gracefully” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Daman Arora, Xuecheng Liu, Himanshu Gaurav Singh, Fahim Tajwar, Abitha Thankaraj, Ruslan Salakhutdinov, Haiwen Feng, Yuda Song and 1 more
GracefulRL trains models to favor success over safe fallback and fallback over error, while preserving a learning signal for success. It optimizes a combination of the log probability of success and the log probability of avoiding error, without labels or an inference-time verifier. On AIME, Qwen3-4B-Base reduced errors by 88.8% with a 3.1-point success decrease, in tested settings.
Key insights
- Expected-reward policy gradients can lose their learning signal near policies concentrated on a single outcome; the paper’s analysis shows the proposed population objective retains a gradient near fallback-only or error-only policies under stated assumptions.
- Fallback is not always a simple tradeoff against success: in Minesweeper, learning to fall back was associated with both fewer errors and more completed boards.
- Caution can transfer beyond the math training domain, though benchmark abstention labels do not establish whether a question is beyond a particular model’s capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.