Reproduce “SocioBench: Modeling Human Behavior in Sociological Surveys with Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jia Wang, Ziyu Zhao, Tingjuntao Ni, Zhongyu Wei
Researchers from multiple Chinese universities developed SocioBench, a large-scale, cross-national benchmark using International Social Survey Programme (ISSP) data, to evaluate how well Large Language Models (LLMs) simulate human behavior in sociological surveys. Experiments show LLMs achieve an average accuracy of 30-40%, with notable biases in performance across different continents, genders, and age groups.
Key insights
- Current LLMs achieve an average accuracy of only 30-40% when simulating individual human behavior in complex sociological surveys, highlighting limitations in accurately reflecting real-world social attitudes.
- Significant demographic biases exist in LLM simulations, with lower accuracy for respondents from African regions and varying performance across genders and age groups.
- Model performance is influenced by domain, parameter scale, and the specific architecture, with explicit "thinking modes" offering limited or no improvement in accuracy.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.