Reproduce “Provably Convergent Primal-Dual DPO for Constrained LLM Alignment” on OpenResearch

About this paper

Yihan Du, Seo Taek Kong, R. Srikant

A primal-dual Direct Preference Optimization (DPO) framework is introduced for constrained Large Language Model alignment, efficiently maximizing helpfulness while adhering to safety constraints by training only two models. The method provides provable convergence guarantees for suboptimality and constraint violation, and it demonstrates competitive empirical performance on benchmarks like TruthfulQA and PKU-SafeRLHF.

Key insights

• DPO-based objectives can formally achieve the same optimal policies as their corresponding RLHF-style objectives, validating direct policy optimization for constrained alignment.
• Implicitly leveraging reward information from a single DPO-trained model allows for a memory-efficient two-model architecture, significantly reducing computational overhead.
• Adaptive learning of the Lagrange multiplier, guided by cost estimates, removes the need for prior knowledge of optimal parameters, enhancing practicality.

Summary by alphaXiv.