Reproduce “Differential learning kinetics govern the transition from memorization to generalization during in-context learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Alex Nguyen, Gautam Reddy

This research investigates how transformer models transition between memorizing (in-weights learning) and generalizing (in-context learning) by proposing that the relative learning rates of distinct sub-circuits govern this shift. The study develops a quantitative theory of "differential learning kinetics" that explains various observed ICL phenomena and provides experimentally validated scaling laws for ICL acquisition time and task diversity thresholds.

Key insights

  • The transition from memorization to generalization is governed by the relative learning kinetics of distinct sub-circuits, where slower memorization for diverse tasks allows ICL to be acquired first.
  • ICL acquisition time is exponentially sensitive to initial conditions and scales linearly with context length (N), explaining the long-tailed distribution of acquisition times.
  • ICL transience, or the loss of ICL capability over long training, can be explained by L2 regularization on attention parameters combined with the MLP's continued, slow memorization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.