Reproduce “TokenCompose: Text-to-Image Diffusion with Token-level Supervision” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zirui Wang, Zhizhou Sha, Zheng Ding, Yilin Wang, Zhuowen Tu

TOKENCOMPOSE improves the compositional abilities of Latent Diffusion Models by integrating token-level and pixel-level supervision during finetuning. This enables text-to-image models to more accurately generate multiple specified objects in complex prompts while leveraging vision foundation models for automated segmentation and incurring no additional inference time.

Key insights

  • Automated, dense supervision derived from vision foundation models can effectively instill fine-grained, token-level understanding into generative models without the need for manual annotation.
  • Explicitly optimizing cross-attention maps during training to ground text tokens to specific visual regions significantly enhances a diffusion model's compositional capabilities and object accuracy.
  • Improved compositional fidelity can be achieved through a training-based method without increasing inference computational costs, offering an efficient and scalable solution compared to inference-time manipulation techniques.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.