Reproduce “Occupancy Learning with Spatiotemporal Memory” on OpenResearch

About this paper

Ziyang Leng, Jiawei Yang, Wenlong Yi, Bolei Zhou

ST-Occ introduces a unified temporal modeling paradigm for vision-centric 3D occupancy prediction, employing a scene-centered spatiotemporal memory with explicit uncertainty and dynamic awareness. This approach achieves a 42.13 mIoU, outperforming prior state-of-the-art by 3 mIoU, and dramatically improves temporal consistency by over 25% while significantly enhancing computational efficiency on the Occ3D benchmark.

Key insights

• Moving from ego-vehicle-centered, frame-wise temporal queues to a unified, scene-centered spatiotemporal memory dramatically improves efficiency for high-resolution 3D occupancy.
• Explicitly modeling and leveraging voxel-level uncertainty and dynamic motion during temporal fusion are crucial for robust and consistent predictions in complex autonomous driving environments.
• Storing rich temporal attributes (class activation, log variance, occupancy flow) within the spatiotemporal memory provides essential context for informed decision-making during fusion.

Summary by alphaXiv.