Reproduce “Learning from Expert Factors: Trajectory-level Reward Shaping for Formulaic Alpha Mining” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Junjie Zhao, Chengxi Zhang, Chenkai Wang, Peng Yang
This work introduces Trajectory-level Reward Shaping (TLRS) alongside a reward centering mechanism to improve the efficiency and stability of reinforcement learning for automated discovery of interpretable formulaic alpha factors. The proposed method leads to faster and more stable convergence during training and boosts the Rank Information Coefficient by 9.29% over existing potential-based shaping algorithms.
Key insights
- Exact subsequence matching offers a computationally efficient and robust alternative to distance-based methods for reward shaping in symbolic generation, reducing complexity from linear to constant with respect to feature dimensions.
- Setting the discount factor to 1 in the RL formulation for symbolic expression generation is crucial to avoid length bias and ensure the discovery of optimal, potentially longer, formulas.
- Decomposing the value function and applying a reward centering mechanism can significantly stabilize RL training by enabling the approximator to focus on relative state values.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.