Publications · Conference paper · NeurIPS 2023
Uniform-in-time Wasserstein stability bounds for (noisy) stochastic gradient descent
Advances in Neural Information Processing Systems (NeurIPS), 2023.
In brief
Stability bounds for SGD usually degrade as training runs longer; this paper proves Wasserstein stability bounds for (noisy) SGD that are uniform in time, so they do not grow with the number of iterations. The key is to view the iterates as a Markov chain and exploit its contraction properties, which yields time-uniform generalization bounds in convex and non-convex settings.
Cite
@inproceedings{zhu2023wasserstein,
title = {Uniform-in-time Wasserstein stability bounds for (noisy) stochastic gradient descent},
author = {Lingjiong Zhu and Mert Gürbüzbalaban and Anant Raj and Umut Şimşekli},
year = {2023},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
url = {http://papers.nips.cc/paper_files/paper/2023/hash/05d6b5b6901fb57d2c287e1d3ce6d63c-Abstract-Conference.html},
note = {arXiv:2305.12056},
}Relatedsame topics