Reproduce “Demystifying Multi-Agent Debate: The Role of Confidence and Diversity” on OpenResearch

About this paper

Xiaochen Zhu, Caiqi Zhang, Yizhou Chi, Tom Stafford, Nigel Collier, Andreas Vlachos

Research from the University of Cambridge improves Multi-Agent Debate (MAD) for Large Language Models by integrating diversity-aware initialization and a confidence-modulated debate protocol. These interventions demonstrate consistent accuracy gains on reasoning tasks, outperforming vanilla MAD and majority voting, and transform the debate's theoretical dynamics from a martingale to a submartingale, enabling systematic progress toward correct solutions.

Key insights

• Diversity-aware initialization improves the prior probability of the debate containing a correct hypothesis without altering the subsequent debate dynamics.
• Confidence-modulated debate, through calibrated confidence expression and utilization, transforms the belief propagation from a martingale into a strict submartingale, enabling systematic drift towards the correct answer.
• The combination of diverse initial viewpoints and confidence-modulated belief updates provides a principled approach to enhance collective intelligence in LLMs, drawing parallels to effective human deliberation.

Summary by alphaXiv.