Heavy-tail phenomenon in decentralized SGD

Mert Gürbüzbalaban; Yuanhan Hu; Umut Şimşekli; Kun Yuan; Lingjiong Zhu

Publications · Journal article · IISE Trans. 2025

Heavy-tail phenomenon in decentralized SGD

Mert Gürbüzbalaban, Yuanhan Hu, Umut Şimşekli, Kun Yuan, Lingjiong Zhu

IISE Transactions, 57(7), pp. 788–802, 2025.

arXiv:2205.06689 Code

In brief

Extends the heavy-tail theory of SGD to the decentralized setting, showing how network communication interacts with multiplicative gradient noise to shape the tail behavior of the iterates.

Topics

Heavy tails in SGD Distributed & decentralized learning

Cite

@article{gurbuzbalaban2025decentralizedheavytail,
  title   = {Heavy-tail phenomenon in decentralized SGD},
  author  = {Mert Gürbüzbalaban and Yuanhan Hu and Umut Şimşekli and Kun Yuan and Lingjiong Zhu},
  year    = {2025},
  journal = {IISE Transactions},
  volume  = {57(7)},
  pages   = {788--802},
  note    = {arXiv:2205.06689},
}

Relatedsame topics

Algorithmic stability of stochastic gradient descent with momentum under heavy-tailed noise (arXiv, 2025)
Rényi differential privacy for heavy-tailed SDEs via fractional Poincaré inequalities (arXiv, 2025)
Differential privacy of noisy (S)GD under heavy-tailed perturbations (arXiv, 2024)
Algorithmic stability of heavy-tailed SGD with general loss functions (ICML, 2023)
Algorithmic stability of heavy-tailed stochastic gradient descent on least squares (ALT, 2023)

← All publications · Research program · Mert Gürbüzbalaban