Reproduce “In-Context Symbolic Regression: Leveraging Large Language Models for Function Discovery” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Matteo Merler, Katsiaryna Haitsiukevich, Nicola Dainese, Pekka Marttinen
ICSR uses a language model to propose equation forms, while nonlinear least squares fits their coefficients; feedback from scored candidates guides revisions. On four symbolic regression benchmark families, with Llama 3 8B, it reports mean test R² of 0.993 and mean complexity of 6.7 nodes, matching uDSR’s fit at lower complexity. It leads at the largest domain expansions, not guaranteeing extrapolation.
Key insights
- Large Language Models possess inherent mathematical reasoning and pattern discovery capabilities that can be effectively harnessed for symbolic regression without requiring specialized pre-training or fine-tuning.
- Integrating a complexity penalty (Occam's Razor) into the fitness function is crucial for guiding the LLM to discover simpler, more interpretable equations, which directly correlates with improved out-of-distribution generalization.
- The iterative optimization loop, where the LLM continually refines its guesses based on prior results, is essential for achieving high accuracy and identifying valid functional forms, significantly outperforming single-pass generation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.