Reproduce “Post-Training Shifts Confidence: A Three-Stage Analysis of How SFT, RL, and OPD Shape CoT Calibration” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shuhao Li, Guodong Du, Anhao Zhao, Wanyu Lin, Tianyu Yuan, Xiaoyu Shen
This study introduces a three-stage framework to analyze how supervised fine-tuning (SFT), reinforcement learning (RL), and on-policy distillation (OPD) reshape large language model (LLM) confidence calibration during chain-of-thought (CoT) reasoning. It reveals distinct, stage-dependent shifts in confidence reliability across these post-training methods, enabling tailored inference strategies that improve efficiency and accuracy, with the PosConf strategy enhancing RL answer aggregation by 6.1 percentage points over majority voting.
Key insights
- Reasoning-oriented post-training induces distinct and stage-dependent shifts in confidence calibration, meaning confidence signals are useful for different inference-time decisions depending on the post-training paradigm.
- Confidence reliability is position-dependent within CoT traces; SFT maintains a stable confidence gap, RL shows strong post-commitment separation, while OPD exhibits inverse calibration later in reasoning.
- Tailoring confidence-based inference strategies to a model's specific post-training history and the reasoning stage (e.g., using OPD's Pre-CoT for difficulty estimation, SFT's Intra-CoT for early stopping, RL's Post-CoT for aggregation) significantly enhances performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.