Reproduce “MUSES: 3D-Controllable Image Generation via Multi-Modal Agent Collaboration” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yanbo Ding, Shaobin Zhuang, Kunchang Li, Zhengrong Yue, Yu Qiao, Yali Wang
MUSES is an AI system that generates images with precise 3D control from text by orchestrating multi-modal agents for layout planning, 3D model acquisition, and image synthesis. It achieves superior accuracy in object count, orientation, 3D spatial relationships, and camera views compared to existing state-of-the-art models, and introduces the T2I-3DisBench for evaluating 3D-controllable image generation.
Key insights
- Orchestrating specialized AI agents for distinct stages of 3D content creation (planning, modeling, rendering) effectively bridges the gap between 2D generative models and complex 3D scene control.
- Leveraging Large Language Models with in-context learning and Chain-of-Thought prompting can accurately interpret complex natural language 3D spatial and orientational cues into structured 3D layouts.
- Precisely aligning retrieved or generated 3D models to specific orientations (e.g., "facing right") using a fine-tuned classifier is crucial for faithful 3D scene rendering.
- Generating detailed 3D-to-2D condition images (depth and Canny maps) from a simulated 3D environment enables diffusion models to maintain high visual quality while strictly adhering to complex 3D scene compositions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.