Reproduce “FourTune: Towards Fully 4-Bit Efficient Post-Training for Diffusion Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Bowen Xue, Zihan Min, Xingyang Li, Zhekai Zhang, Haocheng Xi, Lvmin Zhang, Maneesh Agrawala, Jun-Yan Zhu and 3 more

The FourTune framework introduces a fully 4-bit (W4A4G4) post-training paradigm for large diffusion models, enabling efficient adaptation across tasks like customization, reinforcement learning, and distillation. This approach achieves generation quality comparable to full-precision methods while reducing memory consumption by approximately 2.25 times and accelerating training throughput by up to 2.79 times.

Key insights

  • Fully 4-bit precision for weights, activations, and gradients (W4A4G4) can be achieved in diffusion model post-training, matching full-precision quality when numerical stability is explicitly addressed.
  • Decomposing model weights and using a small, frozen full-precision stabilizer branch is effective in mitigating gradient explosions and ensuring numerical stability in an aggressive low-bit training regime.
  • Block-wise quantization of weights allows for efficient 4-bit computation during the backward pass by enabling direct 4-bit transposed matrix multiplications, avoiding costly dequantization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.