Reproduce “One Video, One World: Turning Monocular Video into Physical 4D Scenes” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Junhao Chen, Boran Zhang, Mingjin Chen, Henghaofan Zhang, Saining Zhang, Congcong Zhu, Hao Zhao, Ruqi Huang and 2 more
OVOW converts monocular video into separately labeled 4D object meshes by segmenting instances, reconstructing canonical or deformable meshes, estimating metric poses, and assembling ground and contact relationships. On a benchmark of static and rigid-motion scenes, it reaches 0.440 oriented-box Scene-IoU versus 0.200 for TabletopGen. Deformable motion is evaluated only on validation data and qualitatively on real videos.
Key insights
- Composing pre-trained foundation models allows for comprehensive, structured 4D scene reconstruction from monocular video without requiring end-to-end task-specific training.
- Decoupling global rigid motion from local non-rigid deformation, achieved through a two-stage pose and deformation recovery, uniformly handles diverse object dynamics and leads to more robust 4D scene models.
- Physics-grounded assembly, including ground plane estimation and robust inter-object contact resolution, is essential for generating physically plausible and simulation-ready 4D scene assets.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.