Why random reshuffling beats stochastic gradient descent

Mathematical Programming, 186, pp. 49–84, 2021.

Publisher (DOI)Code

Resolves a long-standing open question about without-replacement sampling: random reshuffling — the sampling scheme practitioners actually use — provably converges faster than i.i.d. sampling for SGD, with rate Θ(1/k^2s) over epochs against the Ω(1/k) barrier of with-replacement sampling.

The playground Why random reshuffling beats SGD lets you try the ideas of this paper in the browser.

The research note Why random reshuffling beats SGD explains the paper informally.

Incremental methods & data ordering

• Randomness and permutations in coordinate descent methods (Math. Program., 2020)
• Convergence rate of incremental gradient and incremental Newton methods (SIAM J. Optim., 2019)
• Global convergence rate of proximal incremental aggregated gradient methods (SIAM J. Optim., 2018)
• Surpassing gradient descent provably: a cyclic incremental method with linear convergence rate (SIAM J. Optim., 2018)
• A double incremental aggregated gradient method with linear convergence rate for large-scale optimization (ICASSP, 2017)