Reproduce “DarkForest: Less Talk, Higher Accuracy for Multi-Agent LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yi Li, Songtao Wei, Dongming Jiang, Zhichun Guo, Qiannan Li, Bingzhe Li
DarkForest, a controlled-communication framework for multi-agent Large Language Model (LLM) systems, enhances reasoning accuracy by up to 5.4 absolute points on MATH while reducing token consumption by up to 6.5x. It achieves this through independent candidate generation, calibrated evidence aggregation, and strategic, compact information disclosure to a final coordinator.
Key insights
- Less inter-agent communication, when strategically controlled and focused on compact, calibrated evidence, can lead to higher accuracy and significantly reduced computational cost compared to extensive, unrestricted dialogue.
- Empirically calibrating factors like individual agent reliability, support-pattern reliability, and independence provides a more robust basis for aggregating diverse evidence than simple voting mechanisms.
- A deterministic guardrail can effectively prevent a multi-agent system from discarding strongly supported correct answers derived from its internal belief state, enhancing overall robustness at zero additional inference cost.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.