Reproduce “Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuheng Zhang, Dian Yu, Baolin Peng, Linfeng Song, Ye Tian, Mingyue Huo, Nan Jiang, Haitao Mi and 1 more
Iterative Nash Policy Optimization (INPO) aligns Large Language Models with general human preferences by formulating the alignment problem as a two-player game and learning a Nash policy through an online no-regret algorithm. This method innovates by directly optimizing on binary preference signals, bypassing the need for computationally expensive win rate estimations, and consistently outperforms existing iterative alignment techniques on standard benchmarks.
Key insights
- LLM alignment can be framed and robustly solved as a symmetric two-player game, with the goal of finding a Nash policy that performs well against any other policy.
- Online no-regret learning, specifically Online Mirror Descent (OMD), provides an effective and theoretically sound iterative approach for learning this Nash policy from queryable preference oracles.
- It is possible to efficiently optimize for general human preferences by introducing a novel loss objective that bypasses the computationally expensive estimation of individual response win rates, relying instead on direct binary preference signals.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.