Reproduce “Video2Game: Real-time, Interactive, Realistic and Browser-Compatible Environment from a Single Video” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hongchi Xia, Zhi-Hao Lin, Wei-Chiu Ma, Shenlong Wang
Video2Game presents a framework that transforms a single real-world video into a real-time, interactive, and browser-compatible 3D environment, integrating a large-scale neural radiance field with mesh baking and physics modeling. The system enables free exploration and physically plausible interactions, achieving over 100 FPS in browser-based environments and outperforming state-of-the-art neural rendering and mesh-based baselines in novel view synthesis and geometry reconstruction.
Key insights
- Monocular 2D priors (depth, normals, semantics) effectively regularize large-scale NeRF training, significantly improving geometry accuracy and novel view synthesis from sparsely captured, unbounded video data.
- Distilling a complex NeRF into a game-engine compatible mesh with a neural texture allows for a balance between photo-realistic rendering and the efficiency required for real-time interaction and broad compatibility.
- Decomposing the reconstructed scene into individual, actionable entities with assigned rigid body physics attributes enables robust physical interactions (e.g., collisions, manipulation) within the generated virtual environment.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.