Reproduce “Fail-RAG : A Retrieval Augmented Generation Informed Framework for Robot Failure Identification” on OpenResearch

About this paper

Ameya Salvi, Jie Hu

Researchers at Hitachi America, Ltd. developed Fail-RAG, a Retrieval Augmented Generation-informed framework for identifying robot operational failures without extensive model fine-tuning. This framework improved failure detection accuracy by an average of 25 percentage points across five robot operations compared to a baseline Vision-Language Model, providing semantic insights into failure causes.

Key insights

• The effectiveness of the RAG component is tied to the distinctness of failure scenario embeddings, where higher mean cosine distance among vectors generally correlates with improved performance.
• Fail-RAG's performance gains were more pronounced in physical robot experiments compared to simulations, suggesting the RAG context is more critical in real-world environments with varying factors like lighting.
• The choice of distance metric for RAG retrieval showed varied performance across different operations, with cosine distance selected for its equivalent multi-modal embedding performance.

Summary by alphaXiv.