Research
Interests and program
Optimization algorithms for machine learning, treated as dynamical systems driven by imperfect information: how fast methods stay robust to noisy, biased, or adversarial gradients; the stochastic dynamics behind heavy tails and sampling; minimax optimization and primal–dual methods; and learning over networks — with tools from convex optimization, probability, and control.
How the areas fit together
- Robust and risk-sensitive optimization. How should fast algorithms be designed when gradients are noisy, biased, or adversarial?
- Stochastic dynamics and sampling. Which stochastic dynamics make optimization and sampling faster, more stable — or heavy-tailed?
- Minimax optimization and primal–dual methods. How should saddle-point problems — the form that robust learning, adversarial training, and constrained problems take — be solved fast and reliably from stochastic gradients?
- Distributed learning and large-scale computation. How do communication, computation, robustness, and stochasticity interact over networks?
One dynamical system, three norms
A running theme of my current work: an optimization algorithm is a feedback dynamical system, and once you take that literally, its behavior under imperfect gradients becomes something you can compute — not just bound. Average-case fragility is an H2 norm. Worst-case amplification of adversarial errors is an H∞ norm, with an explicit worst-case noise achieving it. Rare-event behavior is a risk-sensitive index whose convex conjugate is a large-deviations rate function. Each is computable for the generalized momentum family, and each can be designed for.
H2 norm + risk measures
Honest but noisy gradients. Iterates equilibrate to a stationary distribution; its size solves a Lyapunov equation, and entropic risk measures capture its tails.
H∞ norm
Worst-case errors with finite energy. The H∞ norm of the algorithm is computable in closed form on quadratics, together with the worst-case noise itself.
Risk-sensitive index + LDP
Systematically biased gradient estimates. A Riccati reduction yields exact risk-sensitive indices, and large deviations of the running suboptimality follow by convex conjugacy.
Speed–robustness–risk trade-offs in first-order methods are fundamental — and designable.
Robustness and risk of accelerated methods
Momentum makes first-order methods fast — and fragile. Nesterov acceleration buys a √κ improvement over gradient descent on smooth strongly convex problems, but the classical theory assumes exact gradients. In practice gradients carry errors from subsampling, quantization, delays, or adversarial perturbation, and momentum, which extrapolates the past, also extrapolates those errors.
This line of work analyzes and designs momentum methods with control-theoretic and risk-theoretic tools. Writing the generalized momentum family (which contains gradient descent, heavy ball, and Nesterov’s method as parameter choices) as a linear feedback system makes its robustness computable: a Heisenberg-like inequality shows that speed times robustness is bounded below, so the right question is not “fast or robust?” but where to sit on the Pareto frontier. Entropic risk and entropic value-at-risk then extend the analysis from average behavior to tails, giving risk-averse parameter design; H∞ analysis covers adversarial errors with matching worst-case noise constructions; and risk-sensitive indices with large deviation principles govern rare events under biased noise.
The recurring model: a generalized momentum method as a feedback dynamical system driven by gradient error w.
Drive these methods yourself in the momentum playground → — sliders for stepsize, momentum, and conditioning, with the rate ρ and the H₂ amplification computed live.
Key papers
- Robust accelerated gradient methods for smooth strongly convex functions (SIAM J. Optim., 2020)
- A universally optimal multistage accelerated stochastic gradient method (NeurIPS, 2019)
- Accelerated linear convergence of stochastic momentum methods in Wasserstein distances (ICML, 2019)
- Entropic risk-averse generalized momentum methods (Optim. Methods Softw., 2025)
- Robustly stable accelerated momentum methods with a near-optimal L2 gain and H∞ performance (Math. Oper. Res., 2025)
- Accelerated gradient methods with biased gradient estimates: risk sensitivity, high-probability guarantees, and large deviation bounds (J. Nonlinear Var. Anal., 2026)
Heavy tails in stochastic optimization
Stochastic gradient noise in deep learning is not Gaussian. Empirically it is heavy-tailed, and — more strikingly — SGD can generate heavy-tailed behavior even from light-tailed data: multiplicative noise turns the iteration into a Kesten-type random recursion whose stationary distribution has power-law tails.
The tail index is not a fixed property of the problem; it is controlled by algorithmic choices such as the ratio of stepsize to batch size, and by the stepsize schedule. This gives a mechanism connecting hyperparameters to the geometry of the solutions SGD finds, with consequences for exit times from basins, algorithmic stability, generalization, and the design of initialization schemes — and it extends to decentralized training over networks.
Kesten’s mechanism: multiplicative randomness produces power-law tails with an algorithm-dependent index α.
Key papers
- A tail-index analysis of stochastic gradient noise in deep neural networks (ICML, 2019)
- First exit time analysis of stochastic gradient descent under heavy-tailed gradient noise (NeurIPS, 2019)
- The heavy-tail phenomenon in SGD (ICML, 2021)
- Cyclic and randomized stepsizes invoke heavier tails in SGD than constant stepsize (Trans. Mach. Learn. Res., 2023)
- Algorithmic stability of heavy-tailed SGD with general loss functions (ICML, 2023)
Sampling and Langevin algorithms
Langevin-type algorithms sit at the interface of optimization and sampling: adding calibrated noise to (stochastic) gradient methods yields samplers for Bayesian learning and global optimizers for non-convex problems. This thread develops their non-asymptotic theory — momentum-based acceleration through Hamiltonian dynamics, provable speedups from breaking reversibility, penalty methods for sampling under constraints, and decentralized samplers that run over networks of agents.
Key papers
- Global convergence of stochastic gradient Hamiltonian Monte Carlo for nonconvex stochastic optimization: nonasymptotic performance bounds and momentum-based acceleration (Oper. Res., 2022)
- Breaking reversibility accelerates Langevin dynamics for non-convex optimization (NeurIPS, 2020)
- Decentralized stochastic gradient Langevin dynamics and Hamiltonian Monte Carlo (J. Mach. Learn. Res., 2021)
- Penalized overdamped and underdamped Langevin Monte Carlo algorithms for constrained sampling (J. Mach. Learn. Res., 2024)
Sampling without replacement and incremental methods
Practitioners shuffle their data once per epoch; classical SGD theory assumes independent sampling with replacement. Closing that gap resolved a long-standing open question: random reshuffling provably beats SGD, converging at Θ(1/k2s) over epochs where with-replacement sampling is stuck at Ω(1/k). The surrounding body of work builds the convergence theory of incremental gradient, incremental aggregated gradient, incremental Newton, and coordinate descent methods — including when deterministic cyclic orders outperform randomized ones.
Key papers
- Why random reshuffling beats stochastic gradient descent (Math. Program., 2021)
- On the convergence rate of incremental aggregated gradient algorithms (SIAM J. Optim., 2017)
- A globally convergent incremental Newton method (Math. Program., 2015)
- Global convergence rate of proximal incremental aggregated gradient methods (SIAM J. Optim., 2018)
- When cyclic coordinate descent outperforms randomized coordinate descent (NeurIPS, 2017)
- Randomness and permutations in coordinate descent methods (Math. Program., 2020)
Minimax optimization and primal–dual methods
Many modern learning problems are games rather than minimizations: distributionally robust learning, adversarial training, constrained and multi-agent problems all take the form of saddle-point (minimax) problems, where one player minimizes while another maximizes. This thread develops first-order algorithms for them — accelerated primal–dual methods such as SAPD and SAPD+, and smoothed alternating gradient descent–ascent schemes — and builds their convergence theory for the stochastic, large-scale setting.
The guarantees come in the forms practitioners need: rates in expectation; high-probability bounds and risk-averse guarantees that certify a single run rather than the average run; complexity bounds for non-convex–concave problems; and an understanding of the speed–robustness trade-off that gradient noise imposes on accelerated primal–dual dynamics, in parallel with the momentum theory. On the modeling side, distributionally robust formulations based on mean–semideviation risk come with finite-sample guarantees. The work is supported by two Office of Naval Research grants on primal–dual algorithms for saddle-point and distributionally robust learning problems.
The saddle-point template behind robust learning, adversarial training, and constrained problems.
Key papers
- Robust accelerated primal-dual methods for computing saddle points (SIAM J. Optim., 2024)
- High probability and risk-averse guarantees for a stochastic accelerated primal-dual method (J. Mach. Learn. Res., 2024)
- High-probability complexity bounds for stochastic non-convex minimax optimization (NeurIPS, 2024)
- SAPD+: an accelerated stochastic method for nonconvex-concave minimax problems (NeurIPS, 2022)
- Mean-semideviation-based distributionally robust learning with weakly convex losses: convergence rates and finite-sample guarantees (Math. Program., 2026)
Distributed and decentralized learning
Large-scale learning is increasingly a multi-agent affair. This thread designs communication-efficient and resilient algorithms for networks: robust accelerated methods for multi-agent optimization, distributed quasi-Newton methods with superlinear local rates, decentralized augmented Lagrangian schemes, decentralized Langevin samplers, and consensus-based methods that remain resilient to faulty or adversarial agents.
Key papers
- Robust distributed accelerated stochastic gradient methods for multi-agent networks (J. Mach. Learn. Res., 2022)
- DAve-QN: a distributed averaged quasi-Newton method with local superlinear convergence rate (AISTATS, 2020)
- IDEAL: inexact decentralized accelerated augmented Lagrangian method (NeurIPS, 2020)
- RESIST: resilient decentralized learning using consensus gradient descent (Trans. Mach. Learn. Res., 2026)
- Decentralized stochastic gradient Langevin dynamics and Hamiltonian Monte Carlo (J. Mach. Learn. Res., 2021)
Nonsmooth spectral optimization and pseudospectra
The earliest thread — from the Courant years with Michael Overton — concerns optimization problems whose objectives are spectral quantities: pseudospectral abscissa and radius, polynomial root optimization, and fast computation of the H∞ norm via spectral value sets. It resurfaces inside the robustness program: computing the real stability radius of an optimization algorithm is exactly a problem of this kind.
Key papers
- Fast approximation of the H∞ norm via optimization over spectral value sets (SIAM J. Matrix Anal. Appl., 2013)
- Some regularity results for the pseudospectral abscissa and pseudospectral radius of a matrix (SIAM J. Optim., 2012)
- Approximating the real structured stability radius with Frobenius-norm bounded perturbations (SIAM J. Matrix Anal. Appl., 2017)
- Explicit solutions for root optimization of a polynomial family with one affine constraint (IEEE Trans. Autom. Control, 2012)
- Polynomial root radius optimization with affine constraints (Math. Program., 2017)
- Robustly stable accelerated momentum methods with a near-optimal L2 gain and H∞ performance (Math. Oper. Res., 2025)
Grants
Office of Naval Research — PI. “Primal-Dual Algorithms for Minimax Problems with Applications to Distributionally Robust Learning.” $400,000.
Office of Naval Research — PI. “Robust Primal-Dual Algorithms for Saddle Point Problems with Applications to Multi-Agent Systems.” $326,647.
NSF, Division of Mathematical Sciences (DMS-2053485) — PI. “Langevin MCMC Methods for Machine Learning.” $180,009.
NSF, Division of Computing and Communication Foundations (CCF-1814888) — Single PI. “Communication-Efficient Distributed Algorithms for Machine Learning.” $464,412.
NSF, Division of Mathematical Sciences (DMS-1723085) — Single PI. “Beyond With-replacement Sampling for Large-Scale Data Analysis and Optimization.” $125,001.