Reproduce “Zero-shot Interactive Perception” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Venkatesh Sripada, Frank Guerin, Amir Ghalamzan
Researchers from the University of Surrey and University of Sheffield developed the Zero-Shot Interactive Perception (ZS-IP) framework, enabling robots to extract hidden information and resolve semantic queries in cluttered environments through physical interaction. The system integrates Vision-Language Models with multi-strategy manipulation and memory-guided action planning, achieving high success rates in interactive perception tasks, often outperforming existing methods like MOKA.
Key insights
- Specialized visual augmentations, such as "pushlines" derived from object geometry, are effective in capturing affordances for contact-rich actions, significantly improving pushing performance in cluttered environments.
- Integrating a memory-guided action module with a VLM allows robots to retain temporal context from previous interactions, enabling more robust multi-step reasoning, dynamic action selection, and efficient task termination.
- Zero-shot interactive perception can be achieved by tightly coupling high-level VLM reasoning with enhanced visual affordances and memory, allowing robots to resolve complex semantic queries in partially observable scenes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.