Fractional underdamped Langevin dynamics: retargeting SGD with momentum under heavy-tailed gradient noise

Umut Şimşekli; Lingjiong Zhu; Yee Whye Teh; Mert Gürbüzbalaban

Conference paper, 2020

Fractional underdamped Langevin dynamics: retargeting SGD with momentum under heavy-tailed gradient noise

Umut Şimşekli, Lingjiong Zhu, Yee Whye Teh, Mert Gürbüzbalaban

International Conference on Machine Learning (ICML), PMLR 119, pp. 8970–8980, 2020.

Publisher page arXiv:2002.05685

In brief

When the gradient noise is heavy-tailed, SGD with momentum no longer samples from the distribution one would expect; it is attracted to a different, heavier-tailed one. Fractional underdamped Langevin dynamics is a modification of the momentum iteration whose invariant measure is the intended Gibbs distribution despite α-stable noise, derived with fractional calculus and tested on deep networks.

Topics

Sampling & Langevin algorithms Heavy tails in SGD

Cite

@inproceedings{simsekli2020fractional,
  title   = {Fractional underdamped Langevin dynamics: retargeting SGD with momentum under heavy-tailed gradient noise},
  author  = {Umut Şimşekli and Lingjiong Zhu and Yee Whye Teh and Mert Gürbüzbalaban},
  year    = {2020},
  booktitle = {International Conference on Machine Learning (ICML)},
  series  = {Proceedings of Machine Learning Research},
  volume  = {119},
  pages   = {8970--8980},
  url     = {http://proceedings.mlr.press/v119/simsekli20a.html},
  note    = {arXiv:2002.05685},
}