Reproduce “Grounded Text-to-Image Synthesis with Attention Refocusing” on OpenResearch

About this paper

Quynh Phung, Songwei Ge, Jia-Bin Huang

Researchers at the University of Maryland, College Park developed the Attention Refocusing framework to enhance the controllability of text-to-image diffusion models, improving compositional accuracy for complex prompts. This training-free method, which combines LLM-generated spatial layouts with novel cross- and self-attention refocusing losses during sampling, achieved consistent quantitative gains of up to 16% in spatial accuracy and up to 7% in F1 score for object counting across various state-of-the-art baselines.

Key insights

• Addressing both cross-attention (token-to-region alignment) and self-attention (intra-image feature propagation) is crucial for comprehensive compositional control, as issues in both layers contribute to T2I failures.
• Leveraging LLMs for automatic spatial layout generation effectively bridges the semantic gap between complex natural language and pixel-level image synthesis, making the grounding process user-friendly and scalable.
• Applying gradient-based guidance on latent features during inference, derived from attention-refocusing losses, can effectively steer the generative process in a training-free and model-agnostic manner, making it a versatile plug-and-play solution.

Summary by alphaXiv.