Reproduce “First Learn, Then Memorize: The Spectral Bias of Diffusion Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Raphaël Urfin, Tony Bonnaire, Giulio Biroli, Marc Mézard

The work explains delayed memorization in diffusion models through the spectrum of the neural tangent kernel (NTK) on repeatedly noised training examples. It identifies faster-learning modes associated with generalization and slower modes linked to individual samples, and reports that suppressing the latter can reduce measured memorization while retaining test-loss performance.

Key insights

  • The spectrum separates into larger, faster-learned modes associated with data-distribution features and smaller, slower modes associated with noise differences among copies of the same sample.
  • The slower memorization component arises from repeated noising and is absent in the single-copy setting; early stopping or ridge regularization can suppress it before full interpolation.
  • In the polynomial sample-complexity regime, generalization modes can also appear in degree-related groups, yielding a hierarchy of learning timescales.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.