Reproduce “Unsupervised Discovery of Failure Taxonomies from Deployment Logs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Aryaman Gupta, Yusuf Umut Ciftci, Somil Bansal

Researchers at Stanford University developed an unsupervised framework to automatically discover failure taxonomies from large volumes of raw, multimodal robotic deployment logs. This method produced semantically coherent and actionable failure modes across various robotic domains, achieving an 85.53% weighted F1 score for assigning failure episodes and enhancing downstream safety workflows.

Key insights

  • Semantic observation downsampling is more effective than fixed-rate sampling, preserving crucial information for VLM reasoning by focusing on changes in CLIP embeddings.
  • General-purpose VLMs like Gemini 2.5 Pro can outperform specialized, fine-tuned models for failure reasoning in real-world scenarios due to better generalization and reduced hallucination.
  • LLMs can effectively act as optimizers for robust, semantically coherent clustering of failure explanations, yielding taxonomies that align well with expert knowledge and are actionable for system improvements.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.