Reproduce “Diagnose, Correct, and Learn from Manipulation Failures via Visual Symbols” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xianchao Zeng, Xinyu Zhou, Youcheng Li, Jiayou Shi, Tianle Li, Liangming Chen, Lei Ren, Yong-Lu Li
Researchers from Beihang University, Southern University of Science and Technology, and Shanghai Jiao Tong University developed ViFailback, a framework for diagnosing, correcting, and learning from real-world robotic manipulation failures. It introduces explicit visual symbols for efficient data annotation and provides multimodal corrective guidance, resulting in up to a 23.8% increase in task success rates for a leading VLA model in real-world experiments.
Key insights
- Existing state-of-the-art VLMs exhibit significant limitations in robotic manipulation failure diagnosis and correction, particularly in open-ended reasoning and multi-step guidance generation.
- Utilizing explicit visual symbols significantly streamlines the annotation of complex real-world failure data, making it more practical to collect and scale such datasets.
- Multimodal corrective guidance, combining textual explanations with precise visual symbols, offers a more actionable and interpretable form of feedback for VLA models, enabling effective policy recovery.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.