Reproduce “Open-vocabulary Mobile Manipulation in Unseen Dynamic Environments with 3D Semantic Maps” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dicong Qiu, Wenzong Ma, Zhenfu Pan, Hui Xiong, Junwei Liang
The Hong Kong University of Science and Technology (Guangzhou) developed a training-free framework for open-vocabulary mobile manipulation, enabling robots to navigate and interact in unseen, dynamic environments using a novel 3-layer 3D semantic map and adapting to misleading human instructions. Real-world experiments on a mobile manipulator achieved an overall success rate of 73.3% and demonstrated robust recovery from mislocated objects or erroneous hints.
Key insights
- A training-free approach orchestrating off-the-shelf foundation models (LLMs, VLMs) with classical robotics components enables complex mobile manipulation in unseen, dynamic environments without extensive fine-tuning.
- The novel 3-layer 3D Semantic Map (3DSMap) effectively bridges low-level geometry with high-level semantic understanding, facilitating intelligent perception, reasoning, and planning.
- Reliability of open-vocabulary object detection is significantly improved by a 'proposal-approval' workflow where a VLM verifies detection proposals, reducing false positives for real-world deployment.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.