Reproduce “Enhancing MMDiT-Based Text-to-Image Models for Similar Subject Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tianyi Wei, Dongdong Chen, Yifan Zhou, Xingang Pan

Researchers from S-Lab, NTU and Microsoft GenAI developed a test-time optimization method to improve MMDiT-based text-to-image models' ability to generate multiple similar subjects. The approach achieved success rate gains of up to +52.7% on prompts with three similar subjects and +46.2% for four similar subjects compared to Stable Diffusion 3, as validated by GPT-4o evaluation, and was preferred by 73.38% of users in a study.

Key insights

  • MMDiT-based text-to-image models like Stable Diffusion 3 suffer from specific architectural ambiguities (inter-block, text encoder, semantic) that are the root cause of failures when generating multiple similar subjects.
  • Analyzing and manipulating cross-attention maps during early denoising steps provides a powerful, architecturally-informed way to resolve these ambiguities and steer the generation towards correct subject differentiation.
  • An adaptive strategy involving online overlap detection and re-sampling with an explicit conflict mask-guided restriction loss is crucial for effectively addressing stubborn semantic ambiguities in complex multi-subject prompts.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.