Reproduce “Sprint: Sparse-Dense Residual Fusion for Efficient Diffusion Transformers” on OpenResearch

About this paper

Dogyun Park, Moayed Haji-Ali, Yanyu Li, Willi Menapace, Sergey Tulyakov, Hyunwoo J. Kim, Aliaksandr Siarohin, Anil Kag

SPRINT (Sparse–Dense Residual Fusion) introduces an efficient training algorithm for Diffusion Transformers (DiTs) that enables aggressive token dropping (up to 75%) while preserving generative quality. This method achieves up to 9.8x lower training GFLOPs and nearly 50% reduced inference FLOPs with improved sample quality on ImageNet-1K 256x256, addressing the high computational costs of large-scale DiT pretraining.

Key insights

• The dense-shallow path and sparse-deep path play complementary roles, with shallow layers capturing local textures and deep layers modeling global semantics, both crucial for high-quality outputs.
• Aggressive token dropping (up to 75%) can be effectively managed without significant performance degradation by employing structured group-wise subsampling and a residual fusion architecture.
• A brief fine-tuning stage (e.g., 20K–50K iterations) is sufficient to effectively adapt the model to dense inputs after sparse pre-training, recovering fine-grained local details.

Summary by alphaXiv.