Reproduce “Fairness in LLM-Generated Surveys” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Andrés Abeliuk, Vanessa Gaete, Naim Bro
A framework is developed to measure socio-demographic biases in Large Language Models when generating survey responses. The study reveals that LLMs consistently perform better on U.S. survey data compared to Chilean data, with biases manifesting differently across countries and for intersecting demographic groups.
Key insights
- LLMs exhibit a consistent U.S.-centric bias, performing better on U.S. datasets, which persists even after normalizing for inherent data predictability.
- LLMs are adept at capturing population-level patterns (Jensen-Shannon Similarity) but show limitations in individual-level prediction accuracy, particularly for marginalized groups.
- Socio-demographic biases manifest differently across countries (e.g., gender and age in Chile, political identity and race in the U.S.) and are exacerbated for intersectional subgroups.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.