Publications · Conference paper · NeurIPS 2021
Fractal structure and generalization properties of stochastic optimization algorithms
Advances in Neural Information Processing Systems (NeurIPS), 34, pp. 18774–18788, 2021. Gürbüzbalaban and Şimşekli are corresponding authors
In brief
Asks what controls how well a stochastic optimizer generalizes, and answers with geometry: the iterates of an algorithm such as SGD accumulate on a set that is typically a fractal, and the paper bounds the generalization error by the Hausdorff dimension of that set rather than by the number of parameters. For algorithms driven by heavy-tailed noise the dimension is tied to the tail index — heavier tails, lower dimension, better generalization — and the dimension can be estimated numerically and tracks the generalization gap in experiments.
Topics
Cite
@inproceedings{camuto2021fractal,
title = {Fractal structure and generalization properties of stochastic optimization algorithms},
author = {Alexander Camuto and George Deligiannidis and Murat A. Erdogdu and Mert Gürbüzbalaban and Umut Şimşekli and Lingjiong Zhu},
year = {2021},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
volume = {34},
pages = {18774--18788},
note = {arXiv:2106.04881},
}Relatedsame topics