Reproduce “From Flat Logs to Causal Graphs: Hierarchical Failure Attribution for LLM-based Multi-Agent Systems” on OpenResearch

About this paper

Yawen Wang, Wenjie Wu, Junjie Wang, Qing Wang

CHIEF, a framework for LLM-based Multi-Agent Systems, transforms flat execution logs into hierarchical causal graphs to attribute task failures. It achieved agent-level accuracies up to 77.59% and step-level accuracies up to 52.00% on the Who&When benchmark, outperforming most baseline methods with moderate computational overhead.

Key insights

• Explicitly reconstructing a hierarchical causal graph from flat execution logs significantly enhances observability and interpretability, providing a structured representation essential for tracing error propagation.
• The combination of oracle-guided backtracking, for efficient search space reduction, with rigorous counterfactual reasoning is crucial for distinguishing root causes from propagated symptoms in complex multi-agent interactions.
• The full CHIEF framework, integrating structural graph representation, efficient search, and causal verification, demonstrates that these components are complementary and indispensable for precise failure attribution in MAS.

Summary by alphaXiv.