Notes

Accessible companions to selected papers and ongoing research themes: each note takes a result from the papers and walks through the idea at conversational pace, and links the work it accompanies.

Subscribe via RSS

October 20265 min read

Why heavy-tailed SGD leaves narrow basins

Gradient noise in deep learning is heavy-tailed, and that changes where SGD ends up: a Gaussian walker has to climb a barrier, a heavy-tailed one jumps it. The exit time from a basin is then set by the basin's width, not its depth — which is why heavy-tailed SGD prefers wide minima.

Companion to the papers in NeurIPS 2019, ICML 2019 and ICML 2021

October 20264 min read

Three ways to reach a saddle point

Minimizing over one variable while maximizing over another is what robust learning, adversarial training and constrained problems come down to — and the plain gradient method fails at it. Why it fails, what extragradient and SAPD do instead, and why doing it fast and doing it quietly pull in opposite directions.

Companion to the papers in SIAM J. Optim. 2024, J. Mach. Learn. Res. 2024 and NeurIPS 2022

October 20264 min read

Why random reshuffling beats SGD

Everyone shuffles the data once per epoch instead of sampling it with replacement, and for years nobody could prove it helps. It does: for strongly convex problems the rate goes from 1/k to 1/k² in the number of epochs. Here is where the extra factor comes from.

Companion to the papers in Math. Program. 2021, SIAM J. Optim. 2019 and Math. Program. 2020

July 20264 min read

How fragile is acceleration?

Momentum buys a quadratic speedup — and pays for it in sensitivity to gradient errors. A guided tour of why that trade-off is a theorem, not an engineering accident, and how to design on the frontier instead of falling off it.

Companion to the papers in SIAM J. Optim. 2020, Optim. Methods Softw. 2025 and Math. Oper. Res. 2025