Reproduce “Automating Robot Failure Recovery Using Vision-Language Models With Optimized Prompts” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hongyi Chen, Yunchao Yao, Ruixuan Liu, Changliu Liu, Jeffrey Ichnowski
Researchers at Carnegie Mellon University developed a method for automating robot failure recovery by intelligently leveraging existing Vision-Language Models through optimized prompt engineering. The approach significantly improves the VLM's spatial reasoning and generalizability for robot control, achieving superior performance over a VLA model in motion-level tasks and high accuracy in detecting and analyzing task-level failures, without requiring model fine-tuning or extensive data collection.
Key insights
- Optimizing visual inputs by highlighting key elements and providing reference images significantly enhances VLM's ability to interpret robot states for recovery.
- Decomposing VLM queries into smaller, focused questions (e.g., 1D movements for spatial reasoning, separate checks for task-level failures) dramatically improves the VLM's precision and interpretability.
- Framing language prompts to focus on relative positions instead of absolute coordinates mitigates VLMs' inherent difficulty with precise 3D spatial reasoning, making them more effective for robot control.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.