Reproduce “Private Direct Preference Optimization for LLM Alignment” on OpenResearch

About this paper

Yangfan Jiang, Fei Wei, Ergute Bao, Xiaokui Xiao, Yaliang Li, Bolin Ding

PrivDPO introduces a privacy-preserving Direct Preference Optimization (DPO) method that formally protects sensitive human preference signals in Large Language Model (LLM) alignment. This approach maintains alignment utility comparable to non-private DPO and scales efficiently to models with up to 32 billion parameters, significantly outperforming existing privacy-preserving baselines.

Key insights

• The core privacy leakage in DPO gradients is effectively contained within a one-dimensional shift along a "preference axis," which is determined by the textual content and not the sensitive preference bit itself.
• Perturbing only the scalar coefficient along this preference axis is sufficient to achieve strong privacy guarantees, leading to more efficient and less utility-degrading noise injection than full gradient perturbation.
• Unbiased perturbation of the sensitive preference signal can be achieved by randomly rescaling the DPO objective function, allowing seamless integration with modern LLM training paradigms without requiring computationally intensive per-example gradient manipulations.

Summary by alphaXiv.