Reproduce “Searching the Space of Feed-Forward Neural-Network Weight-Update Rules with Fixed Depth Symbolic Regression” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Charles Brum, Edward Finkelstein
Symbolic regression can automatically discover explicit and compact weight-update rules for feed-forward neural networks. On small symbolic regression benchmarks, these algorithmically generated rules consistently outperformed established, hyperparameter-tuned optimizers, achieving an aggregate 44.47% reduction in Mean Squared Error across improved cases.
Key insights
- Symbolic regression is a viable method for automatically discovering explicit, compact, and interpretable weight-update rules for neural networks.
- Effective update rules exhibit structural diversity but often incorporate recurring motifs, such as nonlinearities, rational expressions, and elements resembling momentum or adaptive normalization.
- Numerous algebraically distinct expressions can achieve comparable predictive performance, suggesting the existence of large equivalence classes of functionally similar update dynamics.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.