Reproduce “Understanding Generalization in Diffusion Distillation via Probability Flow Distance” on OpenResearch

About this paper

Huijie Zhang, Zijian Huang, Siyi Chen, Jinfan Zhou, Zekai Zhang, Peng Wang, Qing Qu

Researchers from the University of Michigan and collaborators introduced the Probability Flow Distance (PFD), a novel metric for quantifying generalization and memorization in diffusion distillation models. Their work reveals universal scaling laws for these phenomena, along with observations of epoch-wise double descent and early learning behaviors.

Key insights

• PFD accurately captures the memorization-to-generalization (MtoG) transition in diffusion distillation, outperforming other practical metrics that conflate quality with generalization.
• Both memorization and generalization errors in distilled diffusion models follow a universal scaling law governed by the ratio of distillation samples ($N$) to the square root of model parameters ($\sqrt{|\theta|}$).
• Generalization dynamics in diffusion distillation exhibit complex patterns, including epoch-wise double descent in generalization regimes and early learning followed by increased memorization in data-limited scenarios.

Summary by alphaXiv.