Reproduce “Open-World Task and Motion Planning via Vision-Language Model Generated Constraints” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nishanth Kumar, William Shen, Fabio Ramos, Dieter Fox, Tomás Lozano-Pérez, Leslie Pack Kaelbling, Caelan Reed Garrett
OWL-TAMP integrates Vision-Language Models (VLMs) with Task and Motion Planning (TAMP) systems, enabling robots to solve complex, natural language-specified manipulation tasks. It uses VLMs to dynamically generate both discrete action-ordering constraints and continuous code-based constraints for off-the-shelf TAMP planners, achieving a 92% success rate across 10 simulated open-world tasks and successfully completing 19 diverse real-world tasks.
Key insights
- Dynamically generating both discrete (plan sketches) and continuous (code-based classifiers) constraints from natural language instructions allows TAMP systems to overcome their traditional "closed-world" limitations.
- A structured integration of VLM common sense with TAMP's hybrid discrete-continuous search and backtracking capabilities enables robust, zero-shot generalization for complex robot manipulation tasks.
- Providing VLMs with a clear contract in the form of constraints offers an effective and generalizable interface for combining foundation models with established, domain-independent robot planners.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.