Reproduce “Martian World Model: Controllable Video Synthesis with Physically Accurate 3D Reconstructions” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Longfei Li, Zhiwen Fan, Wenyan Cong, Xinhang Liu, Yuyang Yin, Matt Foutter, Panwang Pan, Chenyu You and 5 more

A system generates realistic, 3D-consistent, and controllable videos of Martian landscapes by first creating a physically accurate 3D dataset from sparse rover imagery, then fine-tuning a video diffusion transformer. The approach achieves superior visual fidelity (lowest FID, FVD) and 3D consistency (lowest 2D Warp Error) compared to existing models.

Key insights

  • A dedicated data engine capable of robustly reconstructing physically accurate, metric-scale 3D models from sparse, challenging planetary imagery is foundational for successful domain-specific video synthesis.
  • Integrating robust 3D geometric foundation models with specialized refinement techniques (e.g., depth rescaling, metric-aware initialization) is essential for overcoming the limitations of traditional 3D reconstruction methods on Martian data.
  • Fine-tuning a scalable Video Diffusion Transformer on a 3D-rich, multimodal Martian dataset significantly enhances visual fidelity, 3D consistency, and camera controllability compared to models trained solely on terrestrial data.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.