Reproduce “Aegis: Automated Error Generation and Attribution for Multi-Agent Systems” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fanqi Kong, Ruijie Zhang, Huaxiao Yin, Guibin Zhang, Xiaofei Zhang, Ziang Chen, Zhaowei Zhang, Xiaoyuan Zhang and 2 more
The Aegis framework automates error generation and attribution for multi-agent systems, constructing a dataset of nearly 10,000 annotated trajectories, and enabling fine-tuned models to significantly outperform baselines in identifying faulty agents and error modes. This approach addresses the data scarcity bottleneck in MAS reliability research, providing a scalable method for creating diagnostic data.
Key insights
- Automated synthetic data generation effectively overcomes the data scarcity bottleneck for MAS error attribution, providing realistic and verifiable training examples.
- Fine-tuning open-source LLMs on the domain-specific Aegis synthetic dataset yields superior error attribution performance, surpassing proprietary models an order of magnitude larger.
- The effectiveness of Chain-of-Thought (CoT) prompting for open-source models is enhanced after GRPO training, suggesting it depends on cultivated underlying reasoning capabilities rather than being a standalone technique.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.