Reproduce “Omni123: Exploring 3D Native Foundation Models with Limited 3D Data by Unifying Text to 2D and 3D Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chongjie Ye, Cheng Cao, Chuanyu Pan, Yiming Hao, Yihao Zhi, Yuanming Hu, Xiaoguang Han
Chongjie Ye et al. introduce Omni123, a 3D native foundation model that unifies text-to-2D and text-to-3D generation within a single autoregressive framework, leveraging abundant 2D data as a geometric prior to address 3D data scarcity. The model outperforms existing methods in text-to-3D shape generation, achieving superior semantic alignment, and demonstrates enhanced instruction-based 3D editing with lower Chamfer Distance.
Key insights
- Cross-modal generative consistency between images and 3D can act as an implicit structural constraint, effectively leveraging rich 2D data as a geometric prior to fortify 3D generation.
- Representing text, images, and 3D geometry as discrete tokens in a shared sequence space enables native 3D reasoning and editing without relying on intermediate 2D proxies.
- A multi-stage training recipe, including pre-training, continued training for viewpoint awareness, and supervised fine-tuning, is essential for coordinating diverse generative tasks and mitigating interference.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.