Robust distributed accelerated stochastic gradient methods for multi-agent networks

Alireza Fallah; Mert Gürbüzbalaban; Asuman Ozdaglar; Umut Şimşekli; Lingjiong Zhu

Journal article, 2022

Robust distributed accelerated stochastic gradient methods for multi-agent networks

Alireza Fallah, Mert Gürbüzbalaban, Asuman Ozdaglar, Umut Şimşekli, Lingjiong Zhu

Journal of Machine Learning Research, 23(220), pp. 1–96, 2022.

arXiv:1910.08701

In brief

Takes the speed–robustness analysis of accelerated methods to networks. Agents hold local strongly convex objectives, exchange information only with neighbors, and see noisy gradients; the paper proposes distributed accelerated stochastic gradient methods, works out how fast they converge and how much gradient noise they amplify as a function of stepsize, momentum, and the network’s connectivity, and shows how to choose the parameters to sit on the resulting trade-off.

Topics

Distributed & decentralized learning Momentum & acceleration Risk & robustness

Cite

@article{fallah2022multiagent,
  title   = {Robust distributed accelerated stochastic gradient methods for multi-agent networks},
  author  = {Alireza Fallah and Mert Gürbüzbalaban and Asuman Ozdaglar and Umut Şimşekli and Lingjiong Zhu},
  year    = {2022},
  journal = {Journal of Machine Learning Research},
  volume  = {23(220)},
  pages   = {1--96},
  note    = {arXiv:1910.08701},
}