Reproduce “Understanding the Interplay between Memorization and Learning in Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bishwamittra Ghosh, Soumi Das, Qinyuan Wu, Mohammad Aflah Khan, Deepak Garg, Krishna P. Gummadi, Evimaria Terzi
The authors distinguish contextual memorization from recall by comparing a training string’s loss with the lowest loss reached when training without it. They propose memorization-aware training, which dynamically down-samples low-loss strings and restores them if later forgotten. Across tested formal languages, it reduces test-loss variability, with mean loss lower or comparable to base; these results do not establish universal gains.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.