Reproduce “Post-Training Pruning for Diffusion Transformers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chengzhi Hu, Xuewen Liu, Jing Zhang, Mengjuan Chen, Zhikai Li, Qingyi Gu
Researchers introduce DiT-Pruning, an efficient post-training method tailored for Diffusion Transformers (DiTs) that significantly reduces computational overhead and parameter count while preserving image generation quality. This method integrates a novel Squared Transformation on Weights (STW) saliency metric and a Clustering-Aware Granularity (CAG) strategy, outperforming existing pruning baselines like Wanda and Magnitude at high sparsity rates (e.g., 50%) across state-of-the-art DiT models.
Key insights
- Traditional LLM pruning methods are ineffective for DiTs due to a significant scale mismatch between DiT weights (larger magnitudes) and activations (smaller L2 norms) compared to LLMs, leading to biased sensitivity estimations.
- The loss increase from pruning has a quadratic dependency on parameter magnitude, similar to elastic potential energy, which is not accurately captured by linear approximations used in existing pruning methods, especially for DiTs' larger weights.
- DiT parameter importance exhibits distinct clustering patterns within different layers, necessitating an adaptive pruning granularity rather than a uniform approach to effectively preserve model performance under sparsification.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.