Reproduce “Taming Outlier Tokens in Diffusion Transformers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiaoyu Wu, Yifei Wang, Tsu-Jui Fu, Liang-Chieh Chen, Zhe Gan, Chen Wei
Research from Rice University and Apple characterized outlier tokens in Diffusion Transformers and their ViT-based encoders, which compromise image generation quality, and developed Dual-Stage Registers (DSR) to effectively mitigate them, resulting in an ImageNet-256 FID reduction from 5.89 to 4.58 and improved training efficiency.
Key insights
- Outlier tokens are pervasive in both ViT-based representation encoders (peaking in later layers) and Diffusion Transformers (concentrated in intermediate layers), with distinct patterns depending on their role in the generative pipeline.
- The negative impact of these outliers on generation quality is due to degraded local patch semantics and spatial structure, not merely isolated extreme loss values.
- Dual-Stage Registers (DSR) effectively suppress both encoder-side and denoiser-side outliers, providing a robust mechanism to enhance global capacity and improve local fidelity in generative Transformers.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.