Reproduce “Understanding Representation Dynamics of Diffusion Models via Low-Dimensional Modeling” on OpenResearch

About this paper

Xiao Li, Zekai Zhang, Xiang Li, Siyi Chen, Zhihui Zhu, Peng Wang, Qing Qu

A theoretical framework explains the "unimodal representation dynamics" in diffusion models, where feature quality for downstream tasks peaks at intermediate noise levels, by linking it to the model's ability to learn data distributions and generalize. This research, primarily from the University of Michigan and Ohio State University, provides a mechanistic understanding of diffusion models as representation learners.

Key insights

• The unimodal dynamics arise from a trade-off: initial noise increase aids denoising irrelevant information, but excessive noise reduces the model's class confidence, ultimately degrading representation quality.
• Diffusion models process information from "fine-to-coarse," progressively removing class-irrelevant details while retaining class-specific, coarse attributes as noise levels increase.
• The unimodal characteristic is inherent to the diffusion model's internal processing across timesteps, persisting even when features are extracted from clean inputs, not merely from input noise.
• Unimodal representation dynamics serve as a reliable indicator of model generalization, transitioning to a monotonic decreasing trend when models enter a memorization regime.

Summary by alphaXiv.