Reproduce “eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala and 5 more
NVIDIA Research's eDiff-I improves text-to-image diffusion models by employing an ensemble of expert denoisers and comprehensive multimodal conditioning, achieving an FID-30K score of 6.95 on MS-COCO 2014 (256x256) and demonstrating superior text comprehension and visual fidelity.
Key insights
- The behavior of diffusion models shifts temporally, focusing on text alignment at high noise levels and visual fidelity at low noise levels, suggesting a need for specialized denoisers.
- An ensemble of denoisers allows for scaling model capacity by specializing functions across time steps without increasing the computational cost per inference step.
- Combining diverse text embeddings (T5 for detail, CLIP for global concepts) and integrating CLIP image embeddings provides complementary strengths for robust generation and controllable style transfer.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.