Reproduce “One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao and 2 more
UniDiffuser uses one diffusion model for image–text marginal, conditional, and joint generation by setting separate noise levels for the two modalities. It trains a shared network to predict noise in paired data, then uses modality timesteps to choose what is generated or conditioned on. On MS-COCO, it reaches text-to-image FID 9.71, below Versatile Diffusion’s 10.09 but above Stable Diffusion’s 8.59.
Key insights
- Learning marginal, conditional, and joint distributions in diffusion models can be unified by framing them as noise prediction on perturbed data, differentiated solely by modality-specific timesteps.
- Classifier-Free Guidance can be naturally integrated without extra training, as the model's unified objective inherently provides both conditional and unconditional noise predictions.
- A Transformer architecture, specifically U-ViT, provides a scalable and effective backbone for modeling complex interactions and dependencies across multiple data modalities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.