Reproduce “Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment” on OpenResearch

About this paper

Yueqin Yin, Zhendong Wang, Yujia Xie, Weizhu Chen, Mingyuan Zhou

The Self-Augmented Preference Optimization (SAPO) framework introduces an off-policy learning paradigm to align Large Language Models with human preferences without requiring pre-collected paired preference data. It achieves this by dynamically generating rejected responses from chosen data using an Exponential Moving Average (EMA) model and a replay buffer, matching or surpassing the performance of leading offline methods like DPO and ORPO on various benchmarks, including the Open LLM Leaderboard and MT-Bench.

Key insights

• Off-policy learning, facilitated by an EMA model and replay buffer, is crucial for stability and consistent learning signals, significantly outperforming on-policy sampling.
• Segment-level supervision for generating rejected responses ensures contextual integrity and efficiency, providing more targeted negative samples than generating full responses from scratch.
• The replay buffer's FIFO nature and inverse frequency sampling naturally introduce a curriculum learning effect and promote balanced exploration of historical and recent data, enhancing learning efficiency and robustness.

Summary by alphaXiv.