Reproduce “KM-ViPE: Online Tightly Coupled Vision-Language-Geometry Fusion for Open-Vocabulary Semantic SLAM” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zaid Nasser, Mikhail Iumanov, Tianhao Li, Maxim Popov, Jaafar Mahmoud, Malik Mohrat, Ilya Obrubov, Ekaterina Derevyanka and 2 more
KM-ViPE introduces an online Simultaneous Localization and Mapping (SLAM) framework that tightly couples vision, language, and geometry using uncalibrated monocular RGB video. The system creates a robust, open-vocabulary 3D semantic map, achieving an average Absolute Trajectory Error of 1.84 cm on dynamic TUM RGB-D sequences.
Key insights
- Integrating high-level, view-invariant DINO features directly into the geometric bundle adjustment framework enhances robustness in dynamic scenes without requiring explicit object masks.
- The adaptive robust kernel, guided by DINO feature similarity, provides a mechanism to dynamically down-weight residuals from inconsistent or dynamic points, leading to more stable pose and depth estimates.
- Achieving real-time, open-vocabulary semantic mapping from uncalibrated monocular input is feasible through the synergistic combination of foundation models, online intrinsic estimation, and robust geometric optimization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.