Reproduce “Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
USVSN Sai Prashanth, Alvin Deng, Kyle O'Brien, Jyothir S V, Mohammad Aflah Khan, Jaydeep Borkar, Christopher A. Choquette-Choo, Jacob Ray Fuehne and 4 more
Researchers from EleutherAI, Google DeepMind, Microsoft, and several universities introduce a taxonomy of language model memorization: Recitation, Reconstruction, and Recollection. This framework demonstrates that different factors influence distinct memorization types, with the memorization of rare sequences ("recollection") showing the fastest proportional growth with increased model scale and training, providing a more granular view of how models retain information.
Key insights
- LM memorization is a multifaceted phenomenon, where different factors (like perplexity, duplication, and templating) exert distinct influences on Recitation, Reconstruction, and Recollection categories.
- The 'recollection' category, which involves memorization of rare and less predictable sequences, exhibits the fastest proportional growth as models scale in size and undergo longer training.
- The proposed intuitive taxonomy significantly improves the predictability of memorization, outperforming both a generic model and an automatically partitioned baseline.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.