Reproduce “WildRefer: 3D Object Localization in Large-scale Dynamic Scenes with Multi-modal Visual Data and Natural Language” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhenxiang Lin, Xidong Peng, Peishan Cong, Ge Zheng, Yujin Sun, Yuenan Hou, Xinge Zhu, Sibei Yang and 1 more
WildRefer introduces a one-stage framework for 3D object localization from natural language in large-scale, dynamic, human-centric outdoor scenes using multi-modal LiDAR-camera data. The method achieves superior performance on two newly developed datasets, STRefer and LifeRefer, reaching 62.01% [email protected] on STRefer and 38.89% [email protected] on LifeRefer.
Key insights
- Effectively fusing multi-modal data (LiDAR geometry, image appearance, language semantics) by prioritizing 3D-critical information (point-language interaction first) is essential for robust 3D localization.
- Capturing temporal information through a spatial-temporal attention mechanism in a Dynamic Visual Encoder (DVE) significantly enhances performance in dynamic scenes.
- A one-stage grounding framework can outperform two-stage methods in complex 'in-the-wild' scenarios by directly predicting bounding boxes without relying on potentially noisy 3D object proposals.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.