Reproduce “RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignment” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuhao Du, Zhuo Li, Pengyu Cheng, Zhihong Chen, Yuejiao Xie, Xiang Wan, Anningzhe Gao

Researchers developed Variational Alignment with Re-weighting (VAR), an offline framework that reformulates LLM alignment as a reward-driven supervised fine-tuning problem. This method achieves comparable or superior performance to existing RLHF techniques, improving helpfulness and harmlessness metrics by an average of 7.16% over DPO, while training approximately 5 times faster than online methods like GRPO.

Key insights

  • The optimal policy for RLHF can be theoretically expressed in a closed form, allowing the complex RLHF problem to be reframed as a simpler weighted supervised fine-tuning task.
  • Employing non-negative, exponentially scaled reward weights directly addresses the limitations of clipping-based methods by better preserving fine-grained reward distinctions during optimization.
  • Stable and efficient LLM alignment is achievable by avoiding negative weights and unbounded loss landscapes, ensuring the optimization process remains well-behaved.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.