Reproduce “Polar: A Benchmark for Evaluating Political Bias in LLMs” on OpenResearch

About this paper

Sangho Kim, Heejin Kim, Yoonhee Park, Hyunggeun Jeon, Jaejin Lee

The Polar benchmark, developed by researchers at Seoul National University, offers a reproducible, likelihood-based framework to evaluate political bias in 38 Large Language Models across U.S. and South Korean contexts. Its findings indicate that LLM political preferences are context-dependent, varying by issue category and influenced by the language of presentation, with U.S. political content consistently showing a left-progressive lean.

Key insights

• Political bias in LLMs is highly context-dependent; models exhibit a consistent left-progressive bias on U.S. political content but show more centered and mixed ideological distributions on South Korean content.
• Overall neutrality scores can mask underlying category-specific biases, as some issue areas within a country may lean opposite to the average, such as a right-leaning tendency in 'Market Economy' for South Korea alongside a left-leaning 'Labor' tendency.
• The presentation language of political content can alter measured biases, demonstrating that linguistic and cultural cues influence how models interpret and assign likelihoods to political statements.

Summary by alphaXiv.