סמינר בחומר מעובה: An Effective Theory of High-Dimensional SGD

Stochastic optimization methods are central to modern machine learning, yet their remarkable empirical success remains only partially understood. This talk presents a theoretical framework for analyzing stochastic gradient descent (SGD) in the high-dimensional regime where both the sample size and parameter dimension grow proportionally. Statistical-physics approaches have long provided exact descriptions of learning dynamics in simple neural networks with isotropic Gaussian data. Building on this perspective, I will present a deterministic-equivalent framework that applies to broader covariance structures, loss functions, model classes, and stochastic optimization schemes. In the high-dimensional limit, the microscopic stochastic dynamics close onto a low-dimensional set of macroscopic order parameters.
These reduced dynamics describe learning in large set of problems and yield equations for their generalization error. The framework also reveals how SGD selects among possible solutions i.e. its implicit bias provides explicit step-size conditions for convergence and stability, as well as scaling laws for the learning dynamics.