Reproduce “Separate-and-Enhance: Compositional Finetuning for Text2Image Diffusion Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhipeng Bao, Yijun Li, Krishna Kumar Singh, Yu-Xiong Wang, Martial Hebert
A new compositional finetuning method for Text-to-Image diffusion models, dubbed "Separate-and-Enhance," directly modifies cross-attention mechanisms to mitigate mask overlaps and ensure adequate attention activation for all prompt objects. This approach achieved a 95% higher success rate and improved image realism compared to Stable Diffusion on individual compositional prompts, also exhibiting strong generalization to novel object combinations.
Key insights
- Compositional misalignment in Text-to-Image models stems from a dual problem of low attention activation for individual objects and excessive attention mask overlaps between multiple objects.
- The "Separate" and "Enhance" losses work synergistically; decoupling overlapping objects facilitates better enhancement of individual low-activated objects, and ensuring high activation makes overlaps more identifiable and resolvable.
- Efficient and effective finetuning can be achieved by strategically targeting only the key mapping functions (to_k) in cross-attention modules, allowing for enhanced compositional capacity without compromising image quality or increasing inference time.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.