Reproduce “PTT: Point-Trajectory Transformer for Efficient Temporal 3D Object Detection” on OpenResearch

About this paper

Kuan-Chih Huang, Weijie Lyu, Ming-Hsuan Yang, Yi-Hsuan Tsai

The Point-Trajectory Transformer (PTT) introduces an efficient temporal 3D object detection framework that leverages only current-frame point clouds and historical object trajectories, rather than dense past point clouds. This approach allows the model to effectively incorporate up to 64 frames of temporal context, leading to improved accuracy (e.g., +5.6% 3D mAPH LEVEL 2 for vehicles) while significantly reducing memory overhead and achieving fast inference speeds (150ms for 64 frames) on the Waymo Open Dataset.

Key insights

• Storing compact historical object trajectories (sequences of 3D bounding box proposals) is significantly more memory-efficient than retaining dense point clouds from past frames, enabling the utilization of much longer temporal contexts (e.g., 64 frames).
• Sophisticated attention-based fusion, as implemented in the Point-Trajectory Transformer, is essential for effectively integrating current-frame spatial point features with diverse temporal cues from long-term, short-term, and future-aware trajectory information.
• Anticipating future object states, even based on inferred velocity, can provide valuable context that enhances current-frame 3D object detection accuracy and robustness.

Summary by alphaXiv.