Reproduce “ReplicateAnyScene: Zero-Shot Video-to-3D Composition via Textual-Visual-Spatial Alignment” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mingyu Dong, Chong Xia, Mingyuan Jia, Weichen Lyu, Long Xu, Zheng Zhu, Yueqi Duan
ReplicateAnyScene offers a zero-shot framework that generates compositional 3D scenes from monocular videos by progressively aligning information from various vision foundation models across textual, visual, and spatial dimensions. It achieves higher visual fidelity, geometric accuracy, and semantic coherence in reconstructed scenes compared to previous methods, effectively reducing common artifacts.
Key insights
- Effective compositional 3D reconstruction requires progressive, multi-modal alignment of VFM priors, rather than direct integration, to overcome information fragmentation.
- Geometric proximity in 3D space is essential for robustly deduplicating fragmented 2D object tracks and establishing consistent instance identities across video frames.
- Optimizing 3D asset generation by selecting views that maximize visible 3D surface area leads to higher fidelity meshes than methods relying on 2D pixel area.
- Combining iterative visual-spatial alignment with semantic-aware, rule-based geometric corrections effectively resolves physical implausibilities and produces coherent 3D scene compositions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.