Mean-field limits of trained weights in deep learning: A dynamical systems perspective
Title | Mean-field limits of trained weights in deep learning: A dynamical systems perspective |
Publication Type | Journal Article |
Year of Publication | 2022 |
Authors | Smirnov, A, Hamzi, B, Owhadi, H |
Journal | Dolomites Research Notes on Approximation |
Volume | 15 |
Issue | 3 |
Pagination | 125-145 |
Date Published | 10/2022 |
Publisher | Padova University Press |
Place Published | Padova, IT |
ISSN Number | 2035-6803 |
Abstract | Training a residual neural network with L2 regularization on weights and biases is equivalent to minim- izing a discrete least action principle and to controlling a discrete Hamiltonian system representing the propagation of input data across layers. The kernel/feature map analysis of this Hamiltonian system suggests a mean-field limit for trained weights and biases as the number of data points goes to infinity. The purpose of this paper is to investigate this mean-field limit and illustrate its existence through numerical experiments and analysis (for simple kernels). |
URL | https://drna.padovauniversitypress.it/2022/3/12 |
DOI | 10.14658/pupj-drna-2022-3-12 |