Reproduce “Improving 3D Labeling in Self-Driving by Inferring Vehicle Information using Vision Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Steven Chen, Shivesh Khaitan, Nemanja Djuric
A training-free method leverages Vision Language Models (VLMs) to infer detailed vehicle make, model, and generation (VMMGR) information, directly improving the accuracy of 3D bounding box dimensions for self-driving data. This approach provides more precise initial labels for human annotators, achieving a 2D Intersection-over-Union (IoU) of 0.8831 and 98.1% VMMGR accuracy using Gemini Pro 2.5 on a proprietary dataset.
Key insights
- Leveraging specific vehicle identity (make, model, generation) as a strong prior for 3D dimensions can significantly enhance bounding box accuracy beyond general vehicle type classification.
- Advanced prompt engineering with chain-of-thought elements (e.g., assessing occlusion, identifying configuration/modifications) substantially improves VLM performance for both VMMGR and dimension inference.
- VLMs can generate 3D vehicle dimensions that are sometimes more accurate than initial lidar-aided human annotations in challenging cases, such as heavily occluded vehicles.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.