Reproduce “Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedback” on OpenResearch

About this paper

Runlong Zhou, Maryam Fazel, Simon S. Du

Extragradient Preference Optimization (EGPO) introduces an algorithm for Nash Learning from Human Feedback (NLHF) to align large language models with non-transitive human preferences. It ensures last-iterate linear convergence to the Quantal Response Equilibrium, demonstrating robustness to noisy gradients and outperforming existing methods in win-rates during LLM fine-tuning.

Key insights

• Aggregated human preferences often exhibit non-transitivity, validating the necessity of game-theoretic Nash Learning from Human Feedback (NLHF) over traditional RLHF models.
• Achieving last-iterate convergence is crucial for the practical deployment of large language models, contrasting with computationally intensive average-iterate convergence requirements.
• Complex extragradient updates can be faithfully translated into single-step, direct gradient computations for neural networks, bridging the gap between theoretical guarantees and practical implementation.

Summary by alphaXiv.