Reproduce “Holistic Evaluation and Failure Diagnosis of AI Agents” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Netta Madvil, Gilad Dym, Alon Mecilati, Edo Dekel, Jonatan Liberman, Rotem Brazilay, Liron Schliesser, Max Svidlo and 7 more

A new evaluation framework integrates top-down and bottom-up analyses to provide fine-grained, localized failure diagnosis for AI agents, demonstrating superior localization accuracy (up to 3.5x) and joint accuracy (up to 12.5x) on the TRAIL benchmark compared to existing methods, while scaling to arbitrary trace lengths.

Key insights

  • Evaluation methodology, not merely the raw capability of foundation models, is a primary bottleneck in achieving effective AI agent diagnosis, as evidenced by the performance gap between monolithic and framework-integrated frontier models.
  • Decomposing evaluation into independent per-span assessments allows the framework to scale robustly to arbitrarily long execution traces, resolving context window limitations and maintaining stable localization accuracy across varied trace lengths.
  • Combining top-down and bottom-up analytical perspectives is crucial for a complete diagnostic picture, revealing both localized implementation errors and broader, aggregated behavioral failures that necessitate a holistic view.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.