Reproduce “Mathematical analysis of singularities in the diffusion model under the submanifold assumption” on OpenResearch

About this paper

Yubin Lu, Zhongjian Wang, Guillaume Bal

Researchers at Illinois Institute of Technology, Nanyang Technological University, and the University of Chicago mathematically characterized the strong, pointwise singularity of diffusion model score functions when data resides on low-dimensional manifolds. They proposed a Conditional Expectation Model (CEM) that re-parameterizes the target function to remain bounded, leading to improved approximation accuracy, reduced "pollution errors" on manifold data, and a 12% lower FID score on MNIST compared to DDPM.

Key insights

• The score function (and its $\epsilon$-function variant) exhibits a strong, pointwise singularity of $O(1/t)$ as $t \to 0$ when data is concentrated on a lower-dimensional manifold, providing a mathematical basis for observed numerical instabilities.
• When approximating unbounded score functions, neural networks may implicitly learn a bounded, 'polluted' version of the data distribution, explaining why generated samples often appear slightly diffused from true manifold data.
• Re-parameterizing the target function as the conditional expectation of the initial data, $E_{X_0|X_t=X}[X_0|X_t=X]$, creates a bounded learning target, circumventing the singularity issue for neural network approximation.

Summary by alphaXiv.