Asymmetric heavy tails and implicit bias in Gaussian noise injections

Alexander Camuto; Xiaoyu Wang; Lingjiong Zhu; Chris Holmes; Mert Gürbüzbalaban; Umut Şimşekli

Conference paper, 2021

Asymmetric heavy tails and implicit bias in Gaussian noise injections

Alexander Camuto, Xiaoyu Wang, Lingjiong Zhu, Chris Holmes, Mert Gürbüzbalaban, Umut Şimşekli

International Conference on Machine Learning (ICML), PMLR 139, pp. 1249–1260, 2021.

arXiv:2102.07006

In brief

Explains a side effect of adding Gaussian noise to activations during training. The gradient noise this induces is heavy-tailed and, more unusually, asymmetric, which gives SGD an implicit bias that can degrade generalization; modeling the dynamics with asymmetric α-stable processes identifies the cause, and correcting for the asymmetry recovers the regularization benefit that the noise injection was meant to provide.

Topics

Heavy tails in SGD

Cite

@inproceedings{camuto2021asymmetric,
  title   = {Asymmetric heavy tails and implicit bias in Gaussian noise injections},
  author  = {Alexander Camuto and Xiaoyu Wang and Lingjiong Zhu and Chris Holmes and Mert Gürbüzbalaban and Umut Şimşekli},
  year    = {2021},
  booktitle = {International Conference on Machine Learning (ICML)},
  series  = {Proceedings of Machine Learning Research},
  volume  = {139},
  pages   = {1249--1260},
  note    = {arXiv:2102.07006},
}