Reproduce “Mitigating Cultural Bias in LLMs via Multi-Agent Cultural Debate” on OpenResearch

About this paper

Qian Tan, Lei Jiang, Yuting Zeng, Shuoyang Ding, Xiaohua Xu

Researchers from the University of Science and Technology of China and NVIDIA developed Multi-Agent Cultural Debate (MACD), a training-free framework employing culturally-personified agents in a debate process to generate more culturally neutral LLM responses. This method increased the "No Bias Rate" for GPT-4o on their new bilingual CEBiasBench from 47.6% to 57.6% (LLM-as-judge) and from 69.0% to 86.0% (Multi-Agent Vote), consistently performing well across English, Chinese, and Arabic languages.

Key insights

• Prompting LLMs in a high-resource non-Western language like Chinese does not eliminate cultural bias but instead shifts it towards an East Asian perspective.
• Single LLM evaluators exhibit significant systematic bias, necessitating an aggregated, culturally diverse judging mechanism like MAV for stable and reliable cultural bias assessment.
• Explicitly assigning cultural personas to agents and orchestrating their deliberation through a 'Seeking Common Ground while Reserving Differences' strategy is effective for mitigating cultural bias in LLM outputs.

Summary by alphaXiv.