Reproduce “JockeyBench: Evaluating LLM Agents on Horse-Racing Decision-Making” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhongwei Xie, Xingsu Li, Jingqiao Yu, Zihao Wang, Hong ChiTat, Zhaowei Wang, Weizhi Du, Yuxuan Liu and 3 more
JockeyBench evaluates two language-model workflows for repeated Japanese horse-race forecasting using historical information cut-offs and sealed predictions. It measures forecast quality, revision, research costs, and retrospective capped-Kelly betting separately. Across 324 races, some revisions improved forecasts, but all 28 configuration means lost under the specified betting rule, showing predictive signal did not ensure positive returns.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.