Reproduce “Causal Estimation of Memorisation Profiles” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Pietro Lesci, Clara Meister, Thomas Hofmann, Andreas Vlachos, Tiago Pimentel
Researchers at the University of Cambridge and ETH Zürich developed a principled and efficient causal method, based on the difference-in-differences design, to estimate language model memorization profiles. Their approach dynamically characterizes how memorization evolves throughout training and demonstrates that memorization trends are predictable across different model sizes.
Key insights
- Memorization in language models is a dynamic process, with distinct instantaneous, persistent, and residual phases influenced by training stage, learning rate schedules, and data order.
- The Difference-in-Differences (DiD) design offers a robust and computationally efficient causal framework to quantify instance-level memorization throughout the entire training trajectory.
- Memorization trends can be predicted across different model scales, suggesting that insights from smaller models can inform the design and training of larger, more resource-intensive language models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.