Reproduce “MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tianze Wang, Dongnan Gui, Yifan Hu, Shuhang Lin, Linjun Zhang

MPO combines policies already aligned to separate preferences by taking a normalized weighted geometric mean of their response probabilities, with batch stochastic mirror descent learning the mixing weights instead of running reinforcement learning on an aggregated reward. In Helpful Assistant, on Qwen 2.5-7B, its minimum GPT-judged win rate is 53.1%, versus 46.1% for MaxMin-RLHF, under the tested setup.

Key insights

  • The optimal policy for aggregating diverse, normalized rewards (especially in a max-min setting) can be expressed as a log-linear combination of single-objective policies.
  • Normalizing reward functions relative to a reference policy provides stability and a shared scale for effective multi-objective aggregation.
  • Multi-objective LLM alignment can be achieved efficiently as a post-processing step, rather than through complex and unstable reinforcement learning fine-tuning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.