Reproduce “Critique-out-Loud Reward Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D. Chang, Prithviraj Ammanabrolu
Critique-Out-Loud (CLoud) reward models from Databricks, MIT, and UC San Diego integrate explicit natural language critiques into the reward modeling process. This approach enhances transparency and interpretability while demonstrating improved performance on preference classification and Best-of-N selection benchmarks compared to traditional reward models.
Key insights
- On-policy training, where the reward head is trained on critiques generated by the model itself, is critical for performance, showing substantial improvements over off-policy training (e.g., 6.35 percentage points on RewardBench for the 8B model).
- Self-consistency decoding, by generating multiple critiques and averaging rewards, benefits performance on reasoning tasks but offers minimal or no gains for non-reasoning tasks or Best-of-N selection.
- The CLoud approach bridges traditional black-box reward models with transparent LLM-as-a-Judge frameworks by enabling explicit reasoning during the evaluation process.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.