Publications · Conference paper · ICML 2019
A tail-index analysis of stochastic gradient noise in deep neural networks
International Conference on Machine Learning (ICML), PMLR 97, pp. 5827–5837, 2019. Best Paper Honorable Mention, ICML 2019.
In brief
Challenges the Gaussian picture of stochastic gradient noise: empirically, the noise in deep network training is heavy-tailed and better modeled by α-stable laws, which reframes SGD as a discretization of a Lévy-driven differential equation — with consequences for how the algorithm explores the loss landscape and escapes local minima.
Go further
Interactive
Heavy tails and basin exits → — a playground built around the ideas in this paper.
Topics
Cite
@inproceedings{simsekli2019tailindex,
title = {A tail-index analysis of stochastic gradient noise in deep neural networks},
author = {Umut Şimşekli and Levent Sagun and Mert Gürbüzbalaban},
year = {2019},
booktitle = {International Conference on Machine Learning (ICML)},
series = {Proceedings of Machine Learning Research},
volume = {97},
pages = {5827--5837},
url = {http://proceedings.mlr.press/v97/simsekli19a.html},
note = {arXiv:1901.06053},
}Relatedsame topics