Reproduce “WARP: On the Benefits of Weight Averaged Rewarded Policies” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Alexandre Ramé, Johan Ferret, Nino Vieillard, Robert Dadashi, Léonard Hussenot, Pierre-Louis Cedoz, Pier Giuseppe Sessa, Sertan Girgin and 2 more

WARP, developed by Google DeepMind, is an LLM alignment technique that leverages a three-stage model merging process to achieve a superior balance between human preference alignment and preservation of pre-trained knowledge. It results in improved reward-KL trade-offs compared to standard Reinforcement Learning from Human Feedback methods.

Key insights

  • A dynamic Exponential Moving Average (EMA) anchor for KL regularization enables more stable training and better reward outcomes than a static SFT anchor.
  • Task vectors of independently fine-tuned policies are often nearly orthogonal, indicating they capture distinct alignment aspects that can be effectively combined through spherical interpolation.
  • Model merging provides a method to discover superior points on the KL-reward Pareto front that are not easily reachable through direct reinforcement learning optimization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.