Reproduce “Mapping Post-Training Forgetting in Language Models at Scale” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jackson Harmon, Andreas Hochlehnert, Matthias Bethge, Ameya Prabhu
The authors map post-training changes by testing the same questions before and after training, classifying retention, backward transfer, forgetting, and non-acquisition, then adjusting transitions for multiple-choice guessing. Across nearly 30 model–training combinations, aggregate scores can hide item-level losses. QwQ-32B reports 5.4 ± 0.3% forgetting and 10.3 ± 0.6% backward transfer, not proof of erasure or new knowledge.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.