Reproduce “LLM-as-a-Demographic: Whom Sociodemographic Prompting Helps, and Whom It Hurts” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Daniela Occhipinti, Andrea Piergentili, Marco Guerini

The authors test whether demographic prompts make large language model judges match real annotators’ rating distributions. They compare predicted five-point ratings with judgments from matching profiles across three English tasks in the DEMO corpus. Prompting improves most base models but can reduce instruction-tuned models’ alignment, especially on offensiveness: Woman × Black loses 0.087 Δs, within this study’s tasks and profiles.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.