Reproduce “Recursively Trained Diffusion Models: Limiting Collapse Distribution and Spectral Characterization” on OpenResearch

About this paper

Naïl B. Khelifa, Richard E. Turner, Ramji Venkataramanan

The authors show recursive diffusion training can drift when rounds mix real data with generated samples and reverse sampling stops at a positive time. This truncation adds Gaussian smoothing, so rounds converge to a smoothed mixture of the original data. On CIFAR-10, FID falls from about 33 to 5 across eight generations, approaching a proxy limit, not the original distribution.

Key insights

• Truncation of the reverse diffusion process at a positive time `t_0` is an inherent and primary mechanism of model collapse in diffusion models, even with perfect score estimation and exact sampling.
• Recursive training acts as a low-pass filter on the true data distribution, systematically attenuating higher-order Hermite modes (fine-scale features) more strongly, leading to loss of detail and diversity.
• Annealed truncation schedules, where `t_0` converges to zero over generations, can asymptotically eliminate the compounding collapse effect and enable the recovery of the original data distribution in the error-free regime.

Summary by alphaXiv.