Reproduce “COME: Adding Scene-Centric Forecasting Control to Occupancy World Model” on OpenResearch

About this paper

Yining Shi, Kun Jiang, Qiang Meng, Ke Wang, Jiabao Wang, Wenchao Sun, Tuopu Wen, Mengmeng Yang and 1 more

The COME framework introduces scene-centric forecasting control to diffusion-based occupancy world models, explicitly disentangling ego-vehicle motion from scene evolution. It establishes new state-of-the-art performance on the nuScenes-Occ3D dataset, showing up to a 26.3% improvement in mIoU and 34.6% in IoU over prior methods while enhancing spatial consistency and long-term prediction capabilities.

Key insights

• Explicitly disentangling ego-motion from scene evolvement significantly improves the spatial consistency and prediction accuracy of occupancy world models, as demonstrated by the 44% mIoU improvement of a scene-centric U-Net over a SOTA ego-centric model.
• A multi-stage training approach, where a pre-trained generative world model is guided by a scene-centric forecasting module through a ControlNet, effectively combines imaginative generative capabilities with geometric coherence.
• Implementing a visibility-aware masking strategy on control features is essential for balancing quantitative performance and qualitative realism, preventing noise from unobserved areas from degrading overall generation quality.

Summary by alphaXiv.