The heavy-tail phenomenon in SGD

Mert Gürbüzbalaban; Umut Şimşekli; Lingjiong Zhu

Conference paper, 2021

The heavy-tail phenomenon in SGD

Mert Gürbüzbalaban, Umut Şimşekli, Lingjiong Zhu

International Conference on Machine Learning (ICML), PMLR 139, pp. 3964–3975, 2021.

arXiv:2006.04740

In brief

Shows that SGD can produce heavy-tailed, power-law iterate fluctuations even from light-tailed data: multiplicative gradient noise drives a Kesten-type random recursion whose stationary law has a tail index controlled by the stepsize-to-batch-size ratio — connecting algorithm hyperparameters to the tail behavior, and thereby the generalization properties, of the learned solutions.

Go further

The playground Heavy tails and basin exits lets you try the ideas of this paper in the browser.

The research note Why heavy-tailed SGD leaves narrow basins explains the paper informally.

Topics

Heavy tails in SGD

Cite

@inproceedings{gurbuzbalaban2021heavytail,
  title   = {The heavy-tail phenomenon in SGD},
  author  = {Mert Gürbüzbalaban and Umut Şimşekli and Lingjiong Zhu},
  year    = {2021},
  booktitle = {International Conference on Machine Learning (ICML)},
  series  = {Proceedings of Machine Learning Research},
  volume  = {139},
  pages   = {3964--3975},
  note    = {arXiv:2006.04740},
}