Reproduce “SToRe3D: Sparse Token Relevance in ViTs for Efficient Multi-View 3D Object Detection” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sandro Papais, Lezhou Feng, Charles Cossette, Lingting Ge
SToRe3D enhances multi-view 3D object detection for autonomous driving by implementing joint, end-to-end sparsity across Vision Transformer image tokens and Detection Transformer 3D object queries, guided by planner-aligned relevance. This approach achieves up to 3x faster inference speed, enabling real-time performance (e.g., ~18 FPS with ViT-B) on benchmarks like nuScenes, while preserving accuracy for agents critical to motion planning.
Key insights
- Jointly sparsifying both 2D image tokens and 3D object queries significantly reduces overall latency and FLOPs while maintaining detection accuracy more effectively than pruning only one stream.
- Aligning computational sparsity with "planner-aligned relevance" allows perception systems to efficiently focus resources on objects critical for downstream autonomous driving planning tasks.
- The "store-reactivate" mechanism, by reintroducing previously filtered information, significantly mitigates performance degradation under aggressive sparsity budgets compared to hard pruning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.