Reproduce “Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneity” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hangfan Zhang, Zhiyao Cui, Jianhao Chen, Xinrun Wang, Qiaosheng Zhang, Zhen Wang, Dinghao Wu, Shuyue Hu
The authors evaluate multi-agent debate (MAD) against chain-of-thought (CoT) and self-consistency (SC) across four models and nine benchmarks. Existing debate methods do not reliably outperform these single-agent approaches. In a two-model experiment, heterogeneous Society of Minds averaged 76.7 ± 0.4 across nine benchmark scores, above 72.1 for separate-model SoM; this limited result does not establish general gains.
Key insights
- Multi-agent debate (MAD) frameworks frequently underperform simpler single-agent baselines like Chain-of-Thought (CoT) and Self-Consistency (SC) across diverse benchmarks, despite higher computational costs.
- The inefficiency of current MAD methods stems from an inability to consistently correct errors without introducing new ones, rather than issues with hyperparameters like agent count or debate rounds.
- Incorporating model heterogeneity, where agents use different foundational LLMs, consistently improves MAD performance, with simpler MAD frameworks benefiting most from this approach.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.