HyLo: a hybrid low-rank natural gradient descent method

Baorun Mu; Saeed Soori; Bugra Can; Mert Gürbüzbalaban; Maryam Mehri Dehnavi

doi:10.1109/SC41404.2022.00052

Conference paper, 2022

HyLo: a hybrid low-rank natural gradient descent method

Baorun Mu, Saeed Soori, Bugra Can, Mert Gürbüzbalaban, Maryam Mehri Dehnavi

International Conference for High Performance Computing, Networking, Storage and Analysis (SC), 2022.

Publisher (DOI)

In brief

A practical natural-gradient method for training large networks on many GPUs. HyLo keeps two Khatri–Rao-based low-rank approximations of the Fisher information — one more accurate, one cheaper — and a gradient-based rule for switching between them, so the curvature information that makes natural gradient converge in fewer iterations no longer costs more time than it saves. It converges 1.4–2.1× faster than the state-of-the-art distributed implementation of KFAC, with up to 350× less computation and 10.7× less communication on ResNet-50.

Topics

Distributed & decentralized learning

Cite

@inproceedings{mu2022hylo,
  title   = {HyLo: a hybrid low-rank natural gradient descent method},
  author  = {Baorun Mu and Saeed Soori and Bugra Can and Mert Gürbüzbalaban and Maryam Mehri Dehnavi},
  year    = {2022},
  booktitle = {International Conference for High Performance Computing, Networking, Storage and Analysis (SC)},
  doi     = {10.1109/SC41404.2022.00052},
}