Publications · Conference paper · NeurIPS 2021
Convergence rates of stochastic gradient descent under infinite noise variance
Advances in Neural Information Processing Systems (NeurIPS), 34, pp. 18866–18877, 2021.
In brief
Provides convergence guarantees for SGD when gradient noise is so heavy-tailed that its variance is infinite: under a p-positive definiteness condition on the Hessian, plain SGD still converges to the global optimum of strongly convex problems, with rates in the p-th moment — no gradient clipping or robustification needed.
Topics
Cite
@inproceedings{wang2021infinite,
title = {Convergence rates of stochastic gradient descent under infinite noise variance},
author = {Hongjian Wang and Mert Gürbüzbalaban and Lingjiong Zhu and Umut Şimşekli and Murat A. Erdogdu},
year = {2021},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
volume = {34},
pages = {18866--18877},
note = {arXiv:2102.10346},
}Relatedsame topics