Reproduce “Talk2Sensors: 3D Visual Grounding in Autonomous Driving via Sensor-Adaptive Physical Cue Matching” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Runwei Guan, Di Tian, Ningwei Ouyang, Ruixiao Zhang, Shaofeng Liang, Haocheng Zhao, Lianqing Zheng, Xiaokai Bai and 4 more
Researchers from The Hong Kong University of Science and Technology (Guangzhou) and collaborators developed Talk2Sensors, the first tri-sensor 3D visual grounding dataset for autonomous driving, which explicitly aligns natural language with visual, geometric, and kinematic cues from cameras, LiDAR, and 4D radar. They also introduced TSFormer, a Transformer-based framework that leverages these distinct physical properties through language-guided fusion, achieving 51.000 mAP on the dataset's Entire Annotated Area, outperforming leading fusion baselines by over 8 mAP.
Key insights
- Explicitly aligning natural language with distinct physical cues from heterogeneous sensors (visual appearance from camera, 3D geometry from LiDAR, object kinematics from radar) is crucial for robust and query-adaptive 3D visual grounding.
- A dynamic, language-guided fusion approach with sparse-preserving arbitration prevents dense modalities from overwhelming critical sparse signals, significantly improving grounding accuracy for attribute-rich queries.
- A coarse-to-fine, property-aware fusion strategy enables monotonic performance improvements in 3D visual grounding as more complementary sensor modalities are integrated, contrasting with limitations of naive fusion methods.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.