Reproduce “CoMPaSS: Enhancing Spatial Understanding in Text-to-Image Diffusion Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gaoyang Zhang, Bingtao Fu, Qingnan Fan, Qi Zhang, Runxing Liu, Hong Gu, Huaqi Zhang, Xinguo Liu
CoMPaSS enhances text-to-image diffusion models' spatial understanding by curating a high-quality spatial dataset (SCOP) and augmenting text encoders to retain token ordering (TENOR). This framework achieves state-of-the-art performance on spatial benchmarks, demonstrating up to a 131% relative improvement on GenEval Position for the FLUX.1 model with minimal computational overhead.
Key insights
- Flawed and ambiguous spatial information in widely used image-text training datasets is a primary underlying cause of text-to-image models' difficulties with spatial understanding.
- Explicitly preserving the absolute token ordering and sequential structure of text prompts within the text encoder's embeddings is crucial for effective spatial interpretation by diffusion models.
- A holistic approach combining targeted data curation with a lightweight, plug-and-play module can achieve substantial improvements in complex AI capabilities like spatial reasoning with negligible computational cost.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.