Reproduce “How new data permeates LLM knowledge and how to dilute it” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Chen Sun, Renat Aksitov, Andrey Zhmoginov, Nolan Andrew Miller, Max Vladymyrov, Ulrich Rueckert, Been Kim, Mark Sandler

Research from Google DeepMind reveals how new knowledge permeates through large language models, demonstrating that surprising information (measured by token probability) leads to broader changes across model behavior, while introducing "stepping-stone" text augmentation and "ignore-k" update pruning techniques that reduce undesirable priming effects by 50-95%.

Key insights

  • Degree of priming after learning can be predicted by measuring token probability before learning
  • There exists a probability threshold below which significant priming occurs
  • Changes in memorization and priming are coupled in some models
  • Surprising information tends to cause broader changes across model behavior

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.