Reproduce “RenderWorld: World Model with Self-Supervised 3D Label” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ziyang Yan, Wenzhen Dong, Yihua Shao, Yuhang Lu, Liu Haiyang, Jingwen Liu, Haozhe Wang, Zhe Wang and 3 more

RenderWorld presents a vision-only end-to-end autonomous driving framework that generates self-supervised 3D occupancy labels using Gaussian Splatting. It achieves state-of-the-art results in 3D occupancy prediction, 4D occupancy forecasting, and motion planning on the NuScenes dataset.

Key insights

  • Gaussian Splatting can efficiently generate accurate 3D occupancy labels from 2D images in a self-supervised manner, overcoming the memory limitations of NeRF-based approaches.
  • Explicitly separating and encoding 'air' and 'non-air' elements within a VAE significantly enhances scene reconstruction accuracy and reduces positional errors, improving fine-grained scene understanding.
  • A purely vision-only end-to-end framework, when meticulously designed with efficient 3D perception and scene representation, can achieve performance competitive with or superior to LiDAR-based systems in AD tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.