Reproduce “Prompting for Policy: Forecasting Macroeconomic Scenarios with Synthetic LLM Personas” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Giulia Iadisernia, Carolina Camassa
This study from Banca d’Italia systematically replicated the European Central Bank's Survey of Professional Forecasters using GPT-4o with over 2,000 synthetic personas. It demonstrated that while LLMs achieve competitive macroeconomic forecasting accuracy compared to human experts, sophisticated persona descriptions do not provide a measurable improvement in performance.
Key insights
- Sophisticated persona descriptions do not yield a statistically significant improvement in LLM macroeconomic forecasting accuracy, suggesting they can be omitted to save computational resources.
- LLM-generated forecasts exhibit remarkably low internal disagreement (near-zero dispersion) compared to human expert panels, indicating a strong tendency towards consensus regardless of persona diversity.
- LLMs can maintain competitive forecasting accuracy on out-of-sample data when provided with relevant real-time contextual information, implying effective utilization of current economic data rather than reliance on memorized training data patterns.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.