Reproduce “Robots That Can See: Leveraging Human Pose for Trajectory Prediction” on OpenResearch

About this paper

Tim Salzmann, Lewis Chiang, Markus Ryll, Dorsa Sadigh, Carolina Parada, Alex Bewley

The Human Scene Transformer (HST) integrates 3D skeletal keypoints and head orientation into a Transformer-based architecture for human trajectory prediction in human-centric environments. This method improves prediction accuracy by up to 11% when rich visual cues are available, particularly for scenarios with limited historical positional data, and achieves competitive performance on standard pedestrian datasets.

Key insights

• Rich 3D vision-based human features (skeletal keypoints, head orientation) provide a substantial prediction improvement, especially in scenarios with limited historical positional data (e.g., up to 11% improvement at first detection).
• The proposed Agent Self-Alignment mechanism effectively enables full self-attention in Transformer models, allowing for comprehensive interaction modeling across agents and time while maintaining permutation invariance.
• Explicitly modeling interactions between agents is critical for accurate trajectory prediction in human-centric environments, showing significantly better performance compared to models that ignore interactions.

Summary by alphaXiv.