Reproduce “VaViM and VaVAM: Autonomous Driving through Video Generative Modeling” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Florent Bartoccioni, Elias Ramzi, Victor Besnier, Shashanka Venkataramanan, Tuan-Hung Vu, Yihong Xu, Loick Chambon, Spyros Gidaris and 9 more

This research introduces VaViM and VaVAM, an end-to-end perception-to-action system for autonomous driving that leverages large-scale self-supervised video generative pre-training. It demonstrates that learning rich visual representations from raw video data can enable state-of-the-art performance in frontal driving scenarios within a closed-loop simulator, while also revealing complexities in scaling generative models for real-world safety.

Key insights

  • Large-scale self-supervised video pre-training enables generative models to learn rich, driving-relevant features from raw video, evident through qualitative semantic grouping in VaViM's representations.
  • A "scaling conflict" was observed where increasing model size or data improved generative quality and open-loop trajectory following but sometimes led to degraded closed-loop safety, suggesting an overfitting to expert behavior rather than adaptive decision-making.
  • The NeuroNCAP-based closed-loop evaluation, complemented by proposed new metrics for trajectory adherence and goal-progress, proved crucial for uncovering real-world safety issues not captured by traditional metrics.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.