Reproduce “The More Popular, The Harder to Forget: Adaptive Popularity for LLM Unlearning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Anna Borisiuk, Andrey Savchenko, Alexander Panchenko, Elena Tutubalina
AdaPop tests popularity-aware machine unlearning for factual knowledge in language models. It uses an external popularity estimate to adjust forgetting pressure across facts and a feedback controller to limit damage to retained knowledge. On RWKU’s Level-3 adversarial subset, Llama-3.1-8B-Instruct scores 0.262 in forget-set ROUGE-L Recall, versus WGA’s 0.396; this is a prompt-based result, not proof of complete deletion.
Key insights
- The depth of a fact's memorization in an LLM correlates with its pretraining popularity, requiring an adaptive unlearning strategy that differentiates pressure based on this signal.
- External popularity proxies offer a more effective measure of unlearning difficulty than internal model confidence, which can merely indicate output suppression rather than genuine knowledge erasure.
- A dual-ascent optimization framework can automate the dynamic balancing of competing unlearning (forget) and preservation (retain) objectives, improving stability and reducing the need for manual hyperparameter tuning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.