Reproduce “GSNet: Joint Vehicle Pose and Shape Reconstruction with Geometrical and Scene-aware Supervision” on OpenResearch

About this paper

Lei Ke, Shichao Li, Yanan Sun, Yu-Wing Tai, Chi-Keung Tang

GSNet reconstructs detailed 3D vehicle shapes and 6DoF poses from a single monocular image, outperforming prior regression-based methods and achieving comparable accuracy to fitting-based methods with significantly faster inference times (0.45s vs 3.38s+). The Hong Kong University of Science and Technology research improves 3D perception for autonomous driving by integrating diverse features and a novel hybrid loss function.

Key insights

• Fusing diverse visual features, including ROI, transformed bounding box, 2D keypoint heatmaps, and precise 2D keypoint coordinates, significantly improves 3D pose and shape prediction accuracy.
• Geometric consistency losses, derived from 2D-3D keypoint projection, and novel scene-aware losses (inter- and intra-instance co-planarity) provide unambiguous supervision, effectively regularizing network training for the ill-posed monocular 3D problem.
• Jointly learning dense 3D shape reconstruction and 6DoF pose estimation offers synergistic benefits, where detailed shape enables richer loss functions that, in turn, boost pose accuracy.

Summary by alphaXiv.