Reproduce “DiG: Scalable and Efficient Diffusion Models with Gated Linear Attention” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Lianghui Zhu, Zilong Huang, Bencheng Liao, Jun Hao Liew, Hanshu Yan, Jiashi Feng, Xinggang Wang

Researchers from Huazhong University of Science & Technology and ByteDance developed DiG, a diffusion model architecture based on Gated Linear Attention (GLA) that offers enhanced efficiency for visual content generation. DiG achieves up to 4.2x faster training and 75.7% GPU memory savings compared to Transformer-based baselines, while maintaining or improving image generation quality across various resolutions.

Key insights

  • Gated Linear Attention can be effectively adapted as a diffusion model backbone, offering superior efficiency without sacrificing generation quality, especially for large-scale and high-resolution tasks.
  • The Spatial Reorient & Enhancement Module (SREM) successfully addresses the inherent limitations of 1D-centric linear attention for 2D visual data by combining simple yet effective strategies for spatial context (block-wise scanning) and local awareness (depth-wise convolution).
  • Identity initialization for the depth-wise convolution layer within SREM is crucial for convergence and performance, providing a neutral starting point that preserves local information.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.