Notes

Accessible companions to selected papers and ongoing research themes: each note takes a result from the papers and walks through the idea at conversational pace, and links the work it accompanies.

Subscribe via RSS

October 202618 min read

A line search without a leader

Every first-order method needs a stepsize, and the one the theory hands you is built from constants nobody knows. On one machine a line search fixes that. On a network of agents who share a momentum parameter and only talk to their neighbors, a stepsize one agent rejects changes everyone's step. The paper behind this note lets each agent run its own backtracking test and synchronizes the network with a single number per iteration, keeping the O(1/K) guarantees and dropping the constants.

Companion to the papers in J. Nonlinear Var. Anal. 2026

October 202611 min read

Can momentum survive a delay?

In an asynchronous system the gradient you receive was computed a few steps ago. No retuning of heavy ball or Nesterov's method survives that: with any fixed delay, both fall back toward the gradient-descent rate. A momentum method that extrapolates its gradient query by more than the delay keeps the acceleration, and the reason can be read off a characteristic polynomial.

Companion to the papers in arXiv 2026

October 20265 min read

Why heavy-tailed SGD leaves narrow basins

Gradient noise in deep learning is heavy-tailed, and that changes where SGD ends up: a Gaussian walker has to climb a barrier, a heavy-tailed one jumps it. The exit time from a basin is then set by the basin's width, not its depth — which is why heavy-tailed SGD prefers wide minima.

Companion to the papers in NeurIPS 2019, ICML 2019 and ICML 2021

October 20264 min read

Three ways to reach a saddle point

Minimizing over one variable while maximizing over another is what robust learning, adversarial training and constrained problems come down to — and the plain gradient method fails at it. Why it fails, what extragradient and SAPD do instead, and why doing it fast and doing it quietly pull in opposite directions.

Companion to the papers in SIAM J. Optim. 2024, J. Mach. Learn. Res. 2024 and NeurIPS 2022

October 20264 min read

Why random reshuffling beats SGD

Everyone shuffles the data once per epoch instead of sampling it with replacement, and for years nobody could prove it helps. It does: for strongly convex problems the rate goes from 1/k to 1/k² in the number of epochs. Here is where the extra factor comes from.

Companion to the papers in Math. Program. 2021, SIAM J. Optim. 2019 and Math. Program. 2020

July 20264 min read

How fragile is acceleration?

Momentum buys a quadratic speedup — and pays for it in sensitivity to gradient errors. A guided tour of why that trade-off is a theorem, not an engineering accident, and how to design on the frontier instead of falling off it.

Companion to the papers in SIAM J. Optim. 2020, Optim. Methods Softw. 2025 and Math. Oper. Res. 2025