Reproduce “Auditing Multi-Agent LLM Reasoning Trees Outperforms Majority Vote and LLM-as-Judge” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wei Yang, Shixuan Li, Heng Ping, Peiyu Zhang, Paul Bogdan, Jesse Thomason

AgentAuditor selects among multi-agent reasoning traces by organizing their steps into a tree and auditing evidence at points where paths diverge. In majority-wrong subsets with a correct minority answer, it achieves 65.35% correctness on GSM8K and 81.82% on AMC, versus 0% for voting by definition. Recovery requires a correct, distinguishable candidate path; these are conditional rates, not overall benchmark scores.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.