Algorithmic stability of heavy-tailed SGD with general loss functions

Anant Raj; Lingjiong Zhu; Mert Gürbüzbalaban; Umut Şimşekli

Publications · Conference paper · ICML 2023

Algorithmic stability of heavy-tailed SGD with general loss functions

Anant Raj, Lingjiong Zhu, Mert Gürbüzbalaban, Umut Şimşekli

International Conference on Machine Learning (ICML), pp. 28578–28597, 2023.

Publisher page arXiv:2301.11885

In brief

Extends the heavy-tailed stability analysis from least squares to general, non-convex loss functions by modeling SGD with a heavy-tailed stochastic differential equation. It derives algorithmic-stability bounds in terms of the tail index and the loss geometry, giving generalization bounds that depend on the tails rather than only on the number of iterations.

Topics

Heavy tails in SGD

Cite

@inproceedings{raj2023general,
  title   = {Algorithmic stability of heavy-tailed SGD with general loss functions},
  author  = {Anant Raj and Lingjiong Zhu and Mert Gürbüzbalaban and Umut Şimşekli},
  year    = {2023},
  booktitle = {International Conference on Machine Learning (ICML)},
  pages   = {28578--28597},
  url     = {https://proceedings.mlr.press/v202/raj23a.html},
  note    = {arXiv:2301.11885},
}

Relatedsame topics

Algorithmic stability of stochastic gradient descent with momentum under heavy-tailed noise (arXiv, 2025)
Heavy-tail phenomenon in decentralized SGD (IISE Trans., 2025)
Rényi differential privacy for heavy-tailed SDEs via fractional Poincaré inequalities (arXiv, 2025)
Differential privacy of noisy (S)GD under heavy-tailed perturbations (arXiv, 2024)
Algorithmic stability of heavy-tailed stochastic gradient descent on least squares (ALT, 2023)

← All publications · Research program · Mert Gürbüzbalaban