Reproduce “An Empirical Analysis on Large Language Models in Debate Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xinyi Liu, Pinxin Liu, Hangfeng He
Researchers empirically analyzed advanced large language models (LLMs) for debate evaluation, demonstrating that models like GPT-4 can match human performance in determining debate outcomes in a zero-shot setting. The study revealed that these LLMs are susceptible to distinct biases, including positional, lexical, and order effects, which influence their evaluative judgments.
Key insights
- Advanced LLMs like GPT-3.5 and GPT-4 can achieve performance levels comparable to human evaluators and surpass fine-tuned state-of-the-art models in zero-shot debate evaluation.
- LLMs exhibit consistent and quantifiable biases in debate evaluation, including a preference for the second presented argument (positional bias) and the concluding side (order bias).
- Prompt design, particularly the choice of label verbalizers and phrasing, significantly influences LLM performance and can introduce or shift existing biases rather than eliminate them.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.