Reproduce “Scaling Laws and Interpretability of Learning from Repeated Data” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds and 10 more

The authors test how repeated training text affects language-model loss by training transformers for 100B tokens with fixed token budgets, varying a randomly sampled subset’s size and repetition frequency while keeping tokens unique. An 800M-parameter model with 10% repeated tokens can approach a 340M model’s test loss at a middle repetition level; this result concerns loss, not downstream tasks or duplication.

Key insights

  • Data repetition induces a strong double descent phenomenon in test loss, with peak degradation occurring at intermediate repetition frequencies where the repeated data is successfully memorized.
  • Repeated data causes models to diverge from expected power-law scaling, defining a predictable region in the (model size, repetition frequency) plane where performance degradation is severe.
  • The model's ability to generalize copying patterns and the functionality of specific internal mechanisms like induction heads are disproportionately harmed, indicating a shift from generalization towards memorization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.