Reproduce “Large Language Models Can Be a Viable Substitute for Expert Political Surveys When a Shock Disrupts Traditional Measurement Approaches” on OpenResearch

About this paper

Patrick Y. Wu

Patrick Y. Wu demonstrates that Large Language Models (LLMs) can reliably substitute for expert political surveys in measuring latent attributes, particularly when disruptive events compromise traditional data collection. The research validates LLM-derived measures against existing expert data and successfully uncovers a novel "knowledge institution" perception, both of which predicted federal agency layoffs during a simulated political "shock" event.

Key insights

• LLMs can effectively capture aggregate political perceptions of latent attributes, showing high correlation with traditional expert survey consensus and providing temporal flexibility for retrospective analysis.
• LLMs possess the capacity to estimate and validate novel theoretical constructs (e.g., perception as a "knowledge institution") that are not measurable by traditional means and can independently predict political outcomes.
• A principled "two-part theory-gap criterion" is proposed, guiding researchers on when to responsibly use LLMs as substitutes for expert surveys based on strong theoretical backing and a clear measurement gap.

Summary by alphaXiv.