Reproduce “Entropy Controllable Direct Preference Optimization” on OpenResearch

About this paper

Motoki Omura, Yasuhiro Fujita, Toshiki Kataoka

Researchers from The University of Tokyo and Preferred Networks introduce Entropy Controllable Direct Preference Optimization (H-DPO), which refines the DPO framework by modifying its regularization term to explicitly control the entropy of the learned policy via a new hyperparameter, alpha. This method consistently improves single-sample task performance and enhances pass@k coverage on mathematical and coding tasks by promoting a more effective mode-seeking behavior.

Key insights

• Contrary to common assumptions, reverse KL divergence can sometimes lead to "mode-covering" behavior when fitting a multimodal target, failing to adequately sharpen the policy distribution onto a single preferred mode.
• Explicitly weighting the entropy component in the DPO regularization provides direct, learnable control over the policy's distribution sharpness, directly addressing the mode-covering limitation.
• The optimal level of policy entropy (controlled by alpha) is task-dependent; tasks requiring precision benefit from sharper policies (alpha<1), while those benefiting from diverse sampling for coverage (e.g., pass@k for coding) may benefit from broader policies (alpha>1).

Summary by alphaXiv.