Reproduce “Market Making via Reinforcement Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Thomas Spooner, John Fearnley, Rahul Savani, Andreas Koukorinis
The authors train a market maker on reconstructed order-book data, using asymmetric reward damping to reduce credit for profitable inventory exposure while retaining its losses. In an HSBA.L out-of-sample simulation, the consolidated agent reports normalized PnL of 15.43 ± 13.01, versus 2.80 ± 10.30 for a fixed strategy, and lower mean absolute inventory; this result does not establish live-trading profitability.
Key insights
- Eligibility traces are essential for temporal-difference reinforcement learning to function effectively in noisy, partially observable financial market-making environments, addressing issues of reward attribution.
- Careful design of the reward function, specifically using an asymmetrically dampened PnL, can dramatically improve learning stability and risk-adjusted returns by shaping agent behavior towards genuine market making and away from speculation.
- Complex, high-dimensional state spaces in financial markets can be effectively managed using a Linear Combination of Tile Codings (LCTC), which allows for efficient learning by leveraging approximate independence among state components.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.