Reproduce “FALAT: Tracing Failures in LLM Agent Trajectories via Dependency-Guided Search” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Md Nakhla Rafi, Md Ahasanuzzaman, Dong Jae Kim, Zhijie Wang, Tse-Hsun Chen
FALAT is a diagnostic framework that reformulates failure attribution in LLM agent trajectories as a hierarchical, dependency-guided search problem, designed to accurately identify the responsible agent and the decisive error step. The framework demonstrates improved accuracy over existing methods, achieving up to 73.0% agent-level and 29.1% step-level accuracy on challenging hand-crafted trajectories.
Key insights
- Reformulating failure attribution as a hierarchical dependency-guided search significantly enhances the ability to precisely locate decisive errors in complex LLM agent trajectories.
- Using an external prior of expected behavior, constructed independently of the agent's potentially flawed trajectory, is crucial for preventing reliance on incorrect internal reasoning during diagnosis.
- Explicitly modeling typed dependency relations among candidate steps is effective for distinguishing between errors that originate a fault and those that merely propagate its effects downstream.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.