Reproduce “Distribution Preference Optimization: A Fine-grained Perspective for LLM Unlearning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kai Qin, Jiaqi Wu, Jianxiang He, Haoyuan Sun, Yifei Zhao, Xu Wang, Bin Liang, Yongzhe Chang and 3 more
Distribution Preference Optimization (DiPO) presents a fine-grained, distribution-level approach to large language model unlearning, intrinsically generating preference signals via logit modulation. This method effectively unlearns specific information while preserving general model utility, achieving state-of-the-art forget quality on the TOFU benchmark and demonstrating superior scalability on MUSE compared to existing optimization-based techniques.
Key insights
- Shifting unlearning from the vast "response-level" to the finite "distribution-level" (next-token probability) offers a more constrained and manageable space for defining preferences.
- Preference signals for unlearning can be intrinsically generated directly from the model's own logits via modulation, eliminating the need for external data, auxiliary models, or domain-specific input.
- The derived DiPO loss function is theoretically consistent, ensuring that optimization pushes the model away from dispreferred distributions while pulling it closer to preferred ones.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.