Reproduce “Prediction Arena: Benchmarking AI Models on Real-World Prediction Markets” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jaden Zhang, Gardenia Liu, Oliver Johansson, Hileamlak Yitayew, Kamryn Ohly, Grace Li

Arcada Labs and Harvard University researchers introduced Prediction Arena, a benchmark where AI models autonomously trade on real prediction markets with real capital. The benchmark revealed that current models generally struggled, averaging a 13.8% loss in initial evaluations, and that platform design significantly influenced performance, with some next-generation models demonstrating early positive returns in discovery-based markets.

Key insights

  • Real-market evaluation with financial stakes provides crucial insights into AI model differences in prediction accuracy, risk discipline, and strategic decision-making not discernible from traditional, low-stakes benchmarks.
  • The quality of initial predictions and strategic capital allocation, rather than the quantity of research or computational throughput, are the primary drivers of trading success.
  • Benchmark design, particularly the structure of market access and opportunity identification (curated vs. discovery), profoundly influences which AI capabilities are expressed and successful.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.