Reproduce “Improving Reward Models with Synthetic Critiques” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zihuiwen Ye, Fraser Greenlee-Scott, Max Bartolo, Phil Blunsom, Jon Ander Campos, Matthias Gallé

This research from the University of Oxford and Cohere demonstrates that integrating synthetic natural language critiques into reward model (RM) training significantly enhances RM performance and data efficiency. The approach improves test accuracy, particularly for complex reasoning and adversarial tasks, showing that 5,000 critique-augmented preference pairs can be as effective as up to 200,000 standard preference pairs when using high-quality critiques.

Key insights

  • The quality of synthetic critiques, derived from stronger LLM critique generators (e.g., GPT4-Turbo, Command R+), is crucial for achieving substantial performance gains in reward models.
  • Critiques significantly boost data efficiency, especially in low-resource settings, allowing fewer labeled examples to yield competitive RM performance.
  • Critiques provide a more pronounced benefit to RMs initialized from weaker base models and for tasks requiring complex reasoning or robustness against adversarial examples.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.