Reproduce “Beyond Outcome Rewards: Post-Rationale Distribution Optimization for Selective Prediction in Reasoning Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhimin Gao, Meibo Hu, Jiamian Wang, Zhiqiang Tao, Pichao Wang

PRDO trains a model’s post-rationale answer distribution with proper scoring rules, alongside a token-level KL penalty that keeps rationale generation near the initial model. On four MMLU-Pro backbones, it improved Brier score and error ranking over GRPO with temperature scaling. However, verification accuracy sometimes fell, so the evidence supports better selective decisions, not generally better reasoning or accuracy.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.