Reproduce “Paragraph-to-Image Generation with Information-Enriched Diffusion Model” on OpenResearch

About this paper

Weijia Wu, Zhuang Li, Yefei He, Mike Zheng Shou, Chunhua Shen, Lele Cheng, Yan Li, Tingting Gao and 1 more

ParaDiffusion enables high-quality image generation from extensive textual paragraphs up to 512 words by integrating an adapted Llama V2 with a diffusion model and utilizing a new, information-rich ParaImage dataset. The system achieved a 60.3% human voting rate for direct wins in text faithfulness on challenging long-text prompts.

Key insights

• Integrating and adapting powerful decoder-only Large Language Models (LLMs) like Llama V2 via LoRA allows diffusion models to achieve superior semantic comprehension for long-form text inputs, moving beyond simple keyword matching to deep contextual understanding.
• The creation of extensive, high-quality datasets with detailed paragraph-level descriptions, such as ParaImage, is critical for training models to accurately capture complex scene elements, attributes, and spatial relationships.
• A multi-stage training methodology, progressively focusing on general text-image alignment, then long-text specifics with LLM adaptation, and finally aesthetic quality tuning, optimizes both visual appeal and precise semantic faithfulness.

Summary by alphaXiv.