Reproduce “Adversarial Score identity Distillation: Rapidly Surpassing the Teacher in One Step” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Mingyuan Zhou, Huangjie Zheng, Yi Gu, Zhendong Wang, Hai Huang

The SiDA framework integrates adversarial learning and real-world data into the data-free Score Identity Distillation (SiD) process, enabling single-step diffusion models to generate images with significantly improved quality and faster convergence. This method establishes new state-of-the-art FID scores, such as 1.366 on ImageNet 512x512, often surpassing larger multi-step teacher models.

Key insights

  • Adversarial training with real data can compensate for the inherent limitations of a teacher diffusion model's approximated score, allowing a distilled student generator to surpass the teacher's performance.
  • An existing U-Net architecture, specifically the fake score network's encoder, can be repurposed to serve dual functions of score estimation and discrimination without introducing additional model parameters.
  • Pre-initializing the generator with a data-free SiD model (SiD²A) significantly improves training stability and robustness for subsequent adversarial distillation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.