Overparameterization as a General Mechanism for Accelerating Gradient Descent

Abstract

Overparameterization is often viewed as a defining feature of modern machine learning, where gradient descent is the dominant training algorithm. This talk argues that many advanced fast solvers in scientific computing can be understood from a similar perspective. By enlarging the representation space through overlap and redundancy, one can expose hidden structures and create more effective descent directions, leading to substantially faster gradient-based algorithms. Examples from numerical PDEs, together with MgNet and large language models, illustrate this perspective and suggest that overparameterization may provide a general mechanism for accelerating gradient descent in both scientific computing and modern machine learning.