Reproduce “Large Language Model Unlearning for Source Code” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xue Jiang, Yihong Dong, Huangzhao Zhang, Tangxinyu Wang, Zheng Fang, Yingwei Ma, Rongyu Cao, Binhua Li and 4 more
Researchers from Peking University and Alibaba Group developed PROD, a token-level unlearning method for Large Language Models generating source code, which precisely removes undesirable code snippets while maintaining general code generation capabilities. PROD achieved an average relative gain of 124% in Pareto Dominance Ratio over baselines across tasks like copyrighted, insecure, and deprecated API code unlearning, minimizing legal and security risks.
Key insights
- Existing LLM unlearning methods are ill-suited for source code due to its rigid syntax, causing severe collateral damage to code generation utility.
- A token-level approach like PROD, which sculpts the target output distribution by suppressing undesirable tokens and redistributing probabilities, is effective for precise code unlearning.
- Key components like noise elimination via nucleus sampling and an $\alpha$-suppression mechanism are critical for achieving stable unlearning that balances forget quality and model utility.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.