Reproduce “Diffusion As Self-Distillation: End-to-End Latent Diffusion In One Model” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xiyuan Wang, Muhan Zhang

Researchers at Peking University developed Diffusion As Self-Distillation (DSD), a framework unifying the encoder, decoder, and diffusion network into a single end-to-end trainable Vision Transformer. This approach achieves stable training and strong generative performance on ImageNet 256x256, with the DSD-B model (205 million parameters) outperforming larger multi-component latent diffusion models in efficiency.

Key insights

  • Latent collapse in end-to-end LDM training is analogous to issues in self-distillation and stems from two root causes: latent variance suppression by the standard diffusion loss and failure of rank differentiation due to predicting high-rank noisy targets.
  • Decoupling the target latent's gradient path and re-formulating the diffusion objective to predict the *clean* latent are crucial for stabilizing training and preventing latent space degradation.
  • Leveraging self-distillation principles like EMA target encoders and data augmentation can further enhance the stability and efficiency of learning expressive latent representations within a unified generative architecture.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.