Reproduce “UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wei Li, Xue Xu, Jiachen Liu, Xinyan Xiao
UNIMO-G is a multimodal conditional diffusion framework that unifies text-driven and zero-shot subject-driven image generation by efficiently processing interleaved vision-language inputs. The system achieved an FID-30K of 8.36 on MS-COCO, outperforming leading text-to-image models, and demonstrated superior visual fidelity and relevance in multi-entity subject-driven generation on a new benchmark, MultiBench.
Key insights
- Leveraging a frozen MLLM for multimodal prompt encoding effectively unifies text and visual guidance, providing a flexible and powerful conditioning mechanism for diffusion models.
- A two-stage training approach, starting with broad text-to-image pre-training and then instruction tuning with constructed multimodal prompts, is effective for instilling both general generation capabilities and precise visual faithfulness.
- The introduced visual enhancement learning component during instruction tuning significantly improves image similarity and text-image alignment by guiding attention to specific object regions, reducing issues like entity blending or omission.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.