A tail-index analysis of stochastic gradient noise in deep neural networks

Umut Şimşekli; Levent Sagun; Mert Gürbüzbalaban

Conference paper, 2019

A tail-index analysis of stochastic gradient noise in deep neural networks

Umut Şimşekli, Levent Sagun, Mert Gürbüzbalaban

International Conference on Machine Learning (ICML), PMLR 97, pp. 5827–5837, 2019. Best Paper Honorable Mention, ICML 2019.

Publisher page arXiv:1901.06053

In brief

Challenges the Gaussian picture of stochastic gradient noise: empirically, the noise in deep network training is heavy-tailed and better modeled by α-stable laws, which reframes SGD as a discretization of a Lévy-driven differential equation — with consequences for how the algorithm explores the loss landscape and escapes local minima.

Go further

The playground Heavy tails and basin exits lets you try the ideas of this paper in the browser.

The research note Why heavy-tailed SGD leaves narrow basins explains the paper informally.

Topics

Heavy tails in SGD

Cite

@inproceedings{simsekli2019tailindex,
  title   = {A tail-index analysis of stochastic gradient noise in deep neural networks},
  author  = {Umut Şimşekli and Levent Sagun and Mert Gürbüzbalaban},
  year    = {2019},
  booktitle = {International Conference on Machine Learning (ICML)},
  series  = {Proceedings of Machine Learning Research},
  volume  = {97},
  pages   = {5827--5837},
  url     = {http://proceedings.mlr.press/v97/simsekli19a.html},
  note    = {arXiv:1901.06053},
}