Benign overfitting

Speaker: Professor Peter Bartlett
Affiliation: University of California, Berkeley (USA)

Abstract

Deep learning has revealed some major surprises from the perspective of statistical complexity: even without any explicit effort to control model complexity, these methods find prediction rules that give a near-perfect fit to noisy training data and yet exhibit excellent prediction performance in practice. This talk surveys recent work on methods that predict accurately in probabilistic settings despite fitting too well to training data. We give a characterization of this phenomenon in linear regression and in ridge regression, and we present a fundamentally nonlinear setting where benign overfitting occurs: a two-layer neural network trained using gradient descent on a classification problem.

References: 

• Benign overfitting in linear regression. Bartlett, Long, Lugosi, Tsigler. PNAS 117(48):30063–30070, 2020. arXiv:1906.11300
• Benign overfitting in ridge regression. Tsigler, Bartlett. arXiv:2009.14286
• Deep learning: a statistical viewpoint. Bartlett, Montanari, Rakhlin. Acta Numerica 30:87-201, 2021.  arXiv:2103.09177
• Benign overfitting without linearity. Chatterji, Frei, Bartlett. arXiv:2202.05928