Fractional moment-preserving initialization schemes for training deep neural networks

Mert Gürbüzbalaban; Yuanhan Hu

Conference paper, 2021

Fractional moment-preserving initialization schemes for training deep neural networks

Mert Gürbüzbalaban, Yuanhan Hu

International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 130, pp. 2233–2241, 2021.

arXiv:2005.11878 Code

In brief

Standard initialization schemes keep the variance of activations constant across layers — which assumes the variance exists. When signals are heavy-tailed it may not, and the paper proposes initializations that preserve fractional moments of order below two instead, generalizing the Xavier and He schemes to the heavy-tailed regime, with experiments showing the difference in training deep networks.

Topics

Heavy tails in SGD

Cite

@inproceedings{gurbuzbalaban2021init,
  title   = {Fractional moment-preserving initialization schemes for training deep neural networks},
  author  = {Mert Gürbüzbalaban and Yuanhan Hu},
  year    = {2021},
  booktitle = {International Conference on Artificial Intelligence and Statistics (AISTATS)},
  series  = {Proceedings of Machine Learning Research},
  volume  = {130},
  pages   = {2233--2241},
  note    = {arXiv:2005.11878},
}