Reproduce “You See it, You Got it: Learning 3D Creation on Pose-Free Videos at Scale” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Baorui Ma, Huachen Gao, Haoge Deng, Zhengxiong Luo, Tiejun Huang, Lulu Tang, Xinlong Wang
This work introduces See3D, a framework that learns 3D creation from abundant, pose-free internet videos, overcoming the limitations of scarce 3D datasets. It curates WebVi3D, a large-scale multi-view dataset, and employs a visual-conditional learning mechanism to achieve state-of-the-art performance in tasks like single-view-to-3D generation, sparse-view-to-3D reconstruction, and 3D editing.
Key insights
- High-quality 3D priors can be effectively learned from abundant, pose-free Internet videos, analogous to human 3D perception, without requiring explicit 3D annotations.
- The proposed time-dependent visual-condition serves as a robust '2D-inductive' alternative to '3D-inductive' camera pose conditions for guiding multi-view diffusion models.
- Scalable and accurate data curation is critical; filtering dynamic content and insufficient viewpoint variations from raw video data substantially improves 3D generation performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.