Reproduce “Benchmarking Overton Pluralism in LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Elinor Poole-Dayan, Jiayi Wu, Taylor Sorensen, Jiaxin Pei, Michiel A. Bakker
Researchers at MIT, Brown, Stanford, and U. Washington developed OVERTONSCORE and OVERTONBENCH, a human-grounded metric and benchmark, to quantify how well large language models represent a spectrum of reasonable perspectives. Their findings indicate that current state-of-the-art LLMs cover only 39-48% of the relevant "Overton window" of human viewpoints.
Key insights
- Current LLMs exhibit a substantial "pluralism gap," capturing only a fraction (39-48%) of the Overton window, with even the combination of best responses from all models leaving 23-31% of viewpoints unrepresented.
- Overton pluralism is domain-dependent; models performing well on politically contentious questions may underperform on general values-based questions, indicating it is not a monolithic capability.
- A moderate negative correlation was observed between Overton pluralism and perceived political neutrality, suggesting that striving for a 'neutral' answer might inadvertently omit legitimate minority viewpoints and sacrifice pluralism.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.