The heavy-tail phenomenon in SGD

Mert Gürbüzbalaban; Umut Şimşekli; Lingjiong Zhu

Publications · Conference paper · ICML 2021

The heavy-tail phenomenon in SGD

Mert Gürbüzbalaban, Umut Şimşekli, Lingjiong Zhu

International Conference on Machine Learning (ICML), PMLR 139, pp. 3964–3975, 2021.

arXiv:2006.04740

In brief

Shows that SGD can produce heavy-tailed, power-law iterate fluctuations even from light-tailed data: multiplicative gradient noise drives a Kesten-type random recursion whose stationary law has a tail index controlled by the stepsize-to-batch-size ratio — connecting algorithm hyperparameters to the tail behavior, and thereby the generalization properties, of the learned solutions.

Go further

Interactive

Heavy tails and basin exits → — a playground built around the ideas in this paper.

Topics

Heavy tails in SGD

Cite

@inproceedings{gurbuzbalaban2021heavytail,
  title   = {The heavy-tail phenomenon in SGD},
  author  = {Mert Gürbüzbalaban and Umut Şimşekli and Lingjiong Zhu},
  year    = {2021},
  booktitle = {International Conference on Machine Learning (ICML)},
  series  = {Proceedings of Machine Learning Research},
  volume  = {139},
  pages   = {3964--3975},
  note    = {arXiv:2006.04740},
}

Relatedsame topics

Algorithmic stability of stochastic gradient descent with momentum under heavy-tailed noise (arXiv, 2025)
Heavy-tail phenomenon in decentralized SGD (IISE Trans., 2025)
Rényi differential privacy for heavy-tailed SDEs via fractional Poincaré inequalities (arXiv, 2025)
Differential privacy of noisy (S)GD under heavy-tailed perturbations (arXiv, 2024)
Algorithmic stability of heavy-tailed SGD with general loss functions (ICML, 2023)

← All publications · Research program · Mert Gürbüzbalaban