Reproduce “GeoVideo: Introducing Geometric Regularization into Video Generation Model” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yunpeng Bai, Shaoheng Fang, Chaohui Yu, Fan Wang, Qixing Huang

GeoVideo introduces an explicit geometric regularization framework into latent diffusion video generation models by incorporating per-frame depth prediction and a cross-frame consistency loss. The method improves 3D reconstruction quality and video coherence, outperforming baselines with an MVCS of 77.2 and achieving lower FVD (122.7) for text-to-video generation.

Key insights

  • Explicitly modeling 3D geometry using depth maps and enforcing multi-view consistency is crucial for overcoming temporal artifacts and generating physically plausible videos.
  • Integrating depth as an additional modality into the latent space of existing 2D video diffusion models allows them to learn structured 3D representations without sacrificing visual fidelity.
  • A reprojection-based loss, which aligns globally consistent 3D point clouds with per-frame depth predictions, effectively guides the generative model towards producing coherent and reconstructable 3D scenes.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.