Reproduce “Freestyle Layout-to-Image Synthesis” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Han Xue, Zhiwu Huang, Qianru Sun, Li Song, Wenjun Zhang
FreestyleNet introduces a framework for Freestyle Layout-to-Image Synthesis (FLIS), combining precise spatial control from semantic masks with the open-vocabulary generation capabilities of large text-to-image models. This approach leverages Rectified Cross-Attention to achieve superior image quality, with a Fréchet Inception Distance (FID) of 14.4 on COCO-Stuff, while enabling the generation of novel objects and styles within specified layouts.
Key insights
- Rectified Cross-Attention effectively injects fine-grained spatial control into pre-trained diffusion models by explicitly constraining text embeddings to exert influence only within regions defined by corresponding semantic masks.
- By leveraging the vast generative prior learned by large text-to-image models from internet-scale data, FreestyleNet can synthesize a wide array of unseen semantics (objects, attributes, styles) onto arbitrary spatial layouts.
- Selective application of rectified attention allows for versatile applications such as binding novel attributes to objects, specifying global artistic styles, or generating entirely new objects within designated layout regions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.