Reproduce “DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjie Bai, Yangqing Jia, Ming-Yu Liu and 2 more

Researchers from MIT, Princeton, NVIDIA, and Lepton AI developed DistriFusion, a distributed parallel inference method for high-resolution diffusion models that leverages "displaced patch parallelism." This approach achieves up to a 6.1x speedup for 3840x3840 images on 8 NVIDIA A100 GPUs, reducing generation time from 140 seconds to 22.9 seconds, while maintaining image quality comparable to single-GPU inference.

Key insights

  • The high similarity between inputs of consecutive denoising steps in diffusion models can be exploited to reuse previous step's activations, effectively providing inter-patch context with minimal communication overhead.
  • By performing sparse computations only on relevant portions of feature maps and using asynchronous communication patterns, the latency of communication can be hidden within computational operations.
  • Accurate approximation of global statistics for Group Normalization is crucial for maintaining image quality in a distributed setting without requiring costly synchronous aggregation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.