Reproduce “The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bolei Ma, Xinpeng Wang, Tiancheng Hu, Anna-Carolina Haensch, Michael A. Hedderich, Barbara Plank, Frauke Kreuter
This research systematically reviews and categorizes existing methods for evaluating Attitudes, Opinions, and Values (AOVs) in Large Language Models (LLMs), identifying a prevalent Western-centric bias and significant methodological inconsistencies across current approaches. It establishes a unified framework and points to critical challenges in robust human-AI alignment and reliable social science applications.
Key insights
- LLMs frequently demonstrate Western-centric and left-leaning political biases, often failing to accurately represent diverse global human values and opinions.
- Current AOV evaluation methods suffer from significant inconsistencies, particularly substantial mismatches (over 50% in some models) between logits-based and text-based output extractions.
- Models are highly sensitive to minor input perturbations, such as option reordering or rephrasing, resulting in unreliable and unstable AOV assessments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.