Reproduce “PRISM: A Unified Framework for Post-Training LLMs Without Verifiable Rewards” on OpenResearch

About this paper

Mukesh Ghimire, Aosong Feng, Liwen You, Youzhi Luo, Fang Liu, Xuan Zhu

Researchers at Arizona State University and Amazon Web Services developed PRISM, a unified framework for post-training large language models (LLMs) on complex reasoning tasks without requiring verifiable ground-truth rewards. PRISM integrates process reward models with internal self-certainty signals, leading to stable training, prevention of "reward hacking," and improved performance on math and code generation benchmarks, often matching or exceeding models trained with ground-truth supervision.

Key insights

• Purely internal feedback mechanisms, such as self-certainty or entropy-based rewards, are unstable and unreliable for long-term LLM training, leading to "reward hacking" and actual performance degradation.
• Process Reward Models (PRMs) offer a more robust and reliable signal for evaluating step-by-step reasoning, showing a significantly stronger correlation with true correctness compared to internal confidence metrics.
• A combined approach within PRISM, integrating dense, external process-level feedback from PRMs with a weighted internal confidence signal (self-certainty), creates a stable and effective learning signal that mitigates the weaknesses of individual methods.

Summary by alphaXiv.