Reproduce “Pantheon360: Taming Digital Twin Generation via 3D-Aware 360° Video Diffusion” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ting-Hsuan Chen, Ying-Huan Chen, Tao Tu, Jie-Ying Lee, Cho-Ying Wu, Fangzhou Lin, Hengyuan Zhang, David Paz and 5 more

Pantheon360 introduces a framework for generating high-fidelity, temporally consistent 360° videos from sparse inputs along user-defined camera trajectories. It achieves precise geometric control by integrating a 3D geometric cache with a diffusion model, outperforming existing baselines in quality and 3D consistency on real-world datasets like Web360, with an FVD of 356.151 and MET3R of 0.2840 for single view-to-video generation.

Key insights

  • Decoupling complex 3D geometric reasoning from the generative process via an explicit 3D Cache allows the diffusion model to focus on photorealistic texture synthesis while rigorously enforcing global geometric consistency.
  • Leveraging the full panoramic field-of-view of 360° videos provides holistic scene understanding and reduces the need for hallucination of unseen regions, leading to better cross-view and temporal consistency compared to narrow FoV methods.
  • A dual-anchor latent fusion technique, involving bidirectional geometric conditioning and averaging predictions in latent space, effectively mitigates geometric inconsistencies and ensures smoother interpolation, particularly for challenging real-world scenarios.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.