Reproduce “MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Dewei Zhou, You Li, Fan Ma, Xiaoting Zhang, Yi Yang

Multi-Instance Generation Controller (MIGC) enhances pre-trained Stable Diffusion models to generate images with multiple distinct objects, each adhering to specific attributes and positions. The approach significantly improves instance success rate on the new COCO-MIG benchmark from 32.39% to 58.43% and boosts attribute, count, and spatial success rates on DrawBench to 97.50%, 67.50%, and 93.13% respectively, while maintaining inference speed.

Key insights

  • Decomposing complex multi-instance generation into independent instance shading subtasks within Cross-Attention layers is an efficient and effective strategy for leveraging pre-trained T2I models.
  • Augmenting text embeddings with positional tokens effectively resolves instance merging issues for identically described objects at different locations.
  • A dedicated Enhancement Attention layer can successfully force the generation of specified instances, overcoming the problem of missing objects that the base diffusion model might otherwise fail to produce.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.