Reproduce “Generalization, memorization, and overfitting for diffusion models trained in the lazy high-dimensional regime” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hugo Latourelle-Vigeant, Sinho Chewi, Aram-Alexandre Pooladian, John Sous, Theodor Misiakiewicz

The authors analyze denoising score matching with inner-product kernel models in a high-dimensional lazy-training setting. At fixed noise, training first learns covariance structure, then reaches zero training risk while test risk approaches the pure-noise score risk; only at super-polynomial times does the empirical Bayes denoiser memorize. In a proved early-stopped window, sampling is approximately Gaussian, not necessarily the target distribution.

Key insights

  • The empirical DSM problem is "effectively noiseless" in high dimensions, meaning the target for denoising is nearly deterministic, which challenges classical early stopping rationales.
  • Kernel nonlinearity introduces a "self-induced regularization" (a per-sample correction δn) that locally averages the estimator, rather than a global ridge penalty.
  • The learned nonlinear score functions become effectively indistinguishable from their linearized counterparts during sampling due to "delocalization," where reverse diffusion trajectories maintain minimal overlap with training samples.
  • Overfitting the training objective (zero training loss) does not automatically imply memorization of training samples; memorization only occurs at super-polynomially long training times.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.