Reproduce “MoVerse: Real-Time Video World Modeling with Panoramic Gaussian Scaffold” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yang Zhou, Ziheng Wang, Yuqin Lu, Haofeng Liu, Jun Liang, Shengfeng He, Jing Li

MoVerse creates interactively navigable 3D environments from a single narrow-field-of-view image by separating offline world construction from real-time video rendering. The system generates high-fidelity, temporally coherent video outputs at 8 frames per second on an NVIDIA RTX 4090 GPU.

Key insights

  • Expanding the initial narrow-field-of-view input into a complete 360-degree panorama provides comprehensive context, reducing the under-constrained nature of 3D scene reconstruction.
  • Employing 3D Gaussian Splatting as a dense, directly renderable 3D scaffold offers a robust and controllable geometric anchor for the scene, ensuring spatial persistence.
  • Distilling a bidirectional conditional video diffusion transformer into a causal autoregressive student enables real-time, streaming high-fidelity video generation during interaction.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.