Reproduce “Programmatically Grounded, Compositionally Generalizable Robotic Manipulation” on OpenResearch

About this paper

Renhao Wang, Jiayuan Mao, Joy Hsu, Hang Zhao, Jiajun Wu, Yang Gao

PROGRAMPORT introduces a modular, program-based framework for robotic manipulation that interprets natural language instructions through a domain-specific language and leverages frozen pre-trained vision-language models for visual grounding. This architecture achieves strong zero-shot and compositional generalization, often outperforming baselines with 10x less training data, and demonstrates real-world applicability on physical robots.

Key insights

• Explicitly disentangling vision-language grounding from action policy learning via a programmatic, modular framework is crucial for achieving superior generalization and data efficiency in robotic manipulation.
• Pre-trained Vision-Language models can be effectively leveraged for robust, task-agnostic visual grounding when kept frozen, preserving their general semantic understanding without being diluted by action-specific learning.
• Neuro-symbolic approaches, by combining neural perception with symbolic reasoning for instruction parsing, enable robots to better mirror the compositional nature of language and tasks, leading to more interpretable and adaptable systems.

Summary by alphaXiv.