Reproduce “UCPO: Uncertainty-Aware Policy Optimization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xianzhou Zeng, Jing Huang, Chunmei Xie, Gongrui Nan, Siye Chen, Mengyu Lu, Weiqi Xiong, Qixuan Zhou and 4 more

UCPO introduces an adaptive reinforcement learning framework that enables large language models to express uncertainty by dynamically balancing rewards for correct, incorrect, and uncertain responses. This approach consistently achieved the highest average Precision on Answered Questions (PAQ), reaching up to 79.63% for Qwen3-8B, while stably preventing overconfidence and reward hacking across various tasks.

Key insights

  • Fixed intermediate rewards for uncertain responses are brittle and lead to failure modes like uncertainty suppression or reward hacking, preventing stable training of uncertainty-aware policies.
  • Decoupling advantage estimation for deterministic (correct/incorrect) and uncertain responses is essential to maintain clear policy gradients and prevent interference between these distinct learning signals.
  • Dynamically adjusting the incentive for uncertainty based on the model's current error rates and performance enables an adaptive equilibrium, avoiding both excessive overconfidence and uninformative abstention.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.