Reproduce “SUV: Future Scene Understanding as Video Generation for End-to-End Driving” on OpenResearch

About this paper

Yibo Yuan, Jiacheng Fu, Jiangtong Zhu, Yi Li, Jianhua Han, Meng Tian, Zhuohan Liu, Zhiwei Xiong and 3 more

SUV (Future Scene Understanding as Video Generation) unifies end-to-end autonomous driving by generating synchronized multi-modal future scene videos (RGB, semantics, depth, instance tracks) using a single, shared video expert derived from a pretrained foundation model. This framework enables an action expert to directly leverage these rich latent future representations for trajectory planning, achieving 91.0 EPDMS on NAVSIM-v2 navtest, outperforming prior state-of-the-art methods.

Key insights

• Casting diverse future scene understanding as unified video generation, leveraging a shared video expert, eliminates the need for stream-specific visual prediction heads, enhancing scalability.
• Direct access to comprehensive, multi-modal future latent representations significantly improves trajectory planning performance, especially in challenging long-tail scenarios.
• Generative pretraining of the video expert is crucial for robust and accurate prediction of structured future scene properties (segmentation, depth, tracking), outperforming random initialization by substantial margins (e.g., 14.5-point mIoU increase).

Summary by alphaXiv.