Reproduce “Measure, Don't Optimize: Forecasting Recovery in LLM Unlearning” on OpenResearch

About this paper

Zirui Song, Huaxing Liu, Xiang Wang, Shuai Li, Xinye Li, Lang Gao, Jinghui Zhang, Zheng Lu and 3 more

This research from MBZUAI and Alibaba Group investigates the persistence of forgotten knowledge in Large Language Models after unlearning procedures, introducing J-Access to audit internal representations. It demonstrates that behavioral forgetting is often insufficient, with 85% of unlearned models retaining internal access, and shows that J-Access can predict model-level knowledge recovery under subsequent attacks, though direct optimization of J-Access can lead to audit evasion rather than genuine knowledge deletion.

Key insights

• Behavioral forgetting is insufficient; 85% of audited unlearned models retained internal access to forgotten knowledge at levels exceeding a retain-only gold model, despite passing behavioral checks.
• Pre-attack J-Access reliably predicts model-level susceptibility to knowledge recovery under subsequent relearning attacks, providing a valuable diagnostic for assessing the long-term stability of unlearning.
• Directly optimizing J-Access during unlearning leads to 'audit evasion,' where the measured signal decreases but post-attack knowledge recovery increases, indicating suppression of the audit metric rather than genuine knowledge deletion.

Summary by alphaXiv.