Reproduce “MGDA-Decoupled: Geometry-Aware Multi-Objective Optimisation for DPO-based LLM Alignment” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Andor Vári-Kakas, Ji Won Park, Natasa Tagasovska

Researchers at Prescient Design, CS CoE, Genentech | Roche developed MGDA-DECOUPLED, a geometry-aware multi-objective optimization algorithm to align Large Language Models (LLMs) using the Direct Preference Optimization (DPO) framework. This method balances conflicting objectives by decoupling coefficient calculation from gradient aggregation, achieving statistically significant overall alignment improvements (+2.3% on Gemma-2B, +4.7% on Qwen-0.5B) and enhanced per-objective performance on the UltraFeedback dataset.

Key insights

  • Decoupling coefficient calculation using loss-normalized gradients from the application of these coefficients to raw gradients provides a more robust and effective strategy for multi-objective optimization than standard geometry-aware or magnitude-sensitive methods.
  • The ratio `L_i / ||∇L_i||` serves as a first-order proxy for the distance to the optimum, allowing MGDA-DECOUPLED to implicitly prioritize objectives estimated to be geometrically further from convergence.
  • The approach effectively mitigates "procedural unfairness" and scale-sensitivity issues common in multi-objective LLM alignment, distributing model capacity more equitably across diverse alignment goals.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.