Reproduce “Theoretical guarantees on the best-of-n alignment policy” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander D'Amour, Jacob Eisenstein, Chirag Nagpal, Ananda Theertha Suresh
This research from Google DeepMind and Google Research provides a comprehensive theoretical analysis of the best-of-n alignment policy, correcting a widespread misconception by proving that the commonly cited KL divergence formula is an upper bound rather than the exact value. The paper introduces a new, more accurate KL divergence estimator and characterizes the win rate, offering a more precise understanding of best-of-n's performance and its favorable reward-KL tradeoffs compared to other rejection sampling methods.
Key insights
- The widely accepted formula for KL divergence of best-of-n is an upper bound, not the true value, implying best-of-n causes less drift than previously thought.
- The gap between the old formula and true KL divergence varies significantly: it is small for low-probability outcomes but can be large when the highest-reward outcome has a high probability.
- Best-of-n offers more favorable win rate vs. KL divergence tradeoffs compared to 'rewind-and-repeat' policies.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.