Reproduce “Market-Bench: Evaluating Large Language Models on Introductory Quantitative Trading and Market Dynamics” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Abhay Srivastava, Sam Jung, Spencer Mateega
AfterQuery researchers introduce MARKET-BENCH, a new benchmark designed to evaluate large language models (LLMs) on introductory quantitative trading tasks, specifically their ability to generate executable backtesters from natural language descriptions. The benchmark reveals that while some LLMs can produce executable code for these tasks, they often struggle with numerical accuracy and complex financial reasoning, leading to significant discrepancies from reference implementations.
Key insights
- High executability of LLM-generated code does not guarantee numerical accuracy; models can produce runnable code that significantly deviates from correct financial logic (semantic failures).
- LLMs exhibit varying capabilities across strategy complexity, with multi-asset interactions and risk hedging posing greater challenges in terms of reasoning about market mechanics and numerical precision.
- Significant inconsistency in LLM performance was observed across different evaluation rounds, highlighting that reliability cannot be assumed even when a model performs well in some instances.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.