Reproduce “Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters” on OpenResearch

About this paper

Xiao Ye, Jacob Dineen, Evan Zhu, Shijie Lu, Kevin Song, Ben Zhou

Researchers at Arizona State University introduce HINDCAST, a methodology to rigorously evaluate Large Language Models (LLMs) for probabilistic forecasting by simulating an LLM agent at a past timestamp and restricting its information access to pre-event data. The study demonstrates that retrieval-augmented LLMs generally improve forecasting accuracy and calibration on fact-based questions but can degrade performance in speculative domains by misinterpreting hype as predictive signal.

Key insights

• Retrieval-augmented forecasting generally improves LLM performance by lowering Brier scores and increasing accuracy, especially when the pre-event corpus contains strong, fact-anchored evidence.
• Retrieval can be detrimental in domains prone to speculative hype, leading LLMs to misinterpret opinion as predictive signals and making them more confidently wrong.
• The quality and nature of information in the retrieval corpus critically influence its effectiveness, highlighting the need for LLMs to discern factual indicators from mere speculation.

Summary by alphaXiv.