Reproduce “Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alexander Camuto, George Deligiannidis, Murat A. Erdogdu, Mert Gürbüzbalaban, Umut Şimşekli, Lingjiong Zhu
Researchers model stochastic optimization algorithms as Iterated Function Systems to connect generalization error to the Hausdorff dimension of their invariant measures, showing this fractal dimension can serve as a capacity metric with empirical support.
Key insights
- Generalization error can be effectively bounded by the Hausdorff dimension of the invariant measure of the optimization algorithm, providing a more refined capacity metric than ambient dimension.
- A compact term, E_U,W[log ||J_h_U(W)||], explicitly links algorithm choices, hyperparameters (step-size, batch-size), and loss function geometry to the generalization bound.
- For specific problems, the Hausdorff dimension is analytically shown to decrease with increasing step-size and batch-size, suggesting a relationship between these parameters and improved generalization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.