Reproduce “Minority Sentinel: When to Overturn Majority Voting in Multi-Agent LLM Debates” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Chuan He, Zebin Chen, Zhengyi Yang, Shaobo Qiao, Mingchen Ju, Jiate Liu, Dong Wen, Guanfeng Liu

Researchers at the University of New South Wales and Euler AI developed Minority Sentinel, a framework that identifies and recovers correct minority opinions in multi-agent LLM debates by analyzing behavioral signals. This method achieved a Net Gain of +1.71% across six benchmarks, recovering 22.3% of "Minority Truth" cases with 81.2% flip precision, outperforming an LLM-as-judge baseline.

Key insights

  • "Minority Truth" is a quantifiable problem, with 25.5% of divergent debate outcomes actually having the correct answer in the minority opinion.
  • Rich behavioral signals embedded in debate logs, beyond mere answers, can effectively discriminate between correct and incorrect majority consensuses.
  • Using a non-LLM meta-classifier provides "cognitive orthogonality," avoiding the correlated error problem inherent in LLM-as-Judge paradigms for adjudicating LLM debates.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.