Reproduce “Rewarding Better Thinking for LLM Preference Alignment” on OpenResearch

About this paper

Xubo Liu, Wenya Guo, Ruxue Yan, Xinying Qian, Ying Zhang

A new process-oriented reward, Thinking Checklist Reward (TCR), improves Large Language Model (LLM) alignment for open-ended tasks by providing trajectory-level supervision based on sample-specific "thinking checklists" derived from preference data. This method employs an Exponential Moving Average (EMA)-based residual formulation to capture "thinking surplus" and consistently enhances LLM performance across various models and benchmarks.

Key insights

• While outcome quality and process quality show moderate correlation, process-level checklist scores provide a finer resolution, especially for responses with similar final outcomes, demonstrating the need for explicit process supervision.
• The EMA-based residual formulation for process rewards is crucial for effectively capturing a complementary "thinking surplus," providing a targeted signal for reasoning improvement without redundancy with outcome rewards.
• Sample-specific, preference-derived thinking checklists are more effective than generic or global process-level rewards for guiding nuanced reasoning behaviors in open-ended tasks.

Summary by alphaXiv.