Gaussian mean field regularizes by limiting learned information
- Kunze, Julius Computer Science, University College London, London WC1E 6BT, UK
- Kirsch, Louis Computer Science, University College London, London WC1E 6BT, UK - Istituto Dalle Molle di studi sull'intelligenza artificiale (IDSIA), Facoltà di scienze informatiche, Università della Svizzera italiana, Svizzera
- Ritter, Hippolyt Computer Science, University College London, London WC1E 6BT, UK
- Barber, David Computer Science, University College London, London WC1E 6BT, UK - Alan Turing Institute, London NW1 2DB, UK
-
03.08.2019
Published in:
- Entropy. - 2019, vol. 21, no. 8, p. 758
English
Variational inference with a factorized Gaussian posterior estimate is a widely-used approach for learning parameters and hidden variables. Empirically, a regularizing effect can be observed that is poorly understood. In this work, we show how mean field inference improves generalization by limiting mutual information between learned parameters and the data through noise. We quantify a maximum capacity when the posterior variance is either fixed or learned and connect it to generalization error, even when the KL-divergence in the objective is scaled by a constant. Our experiments suggest that bounding information between parameters and data effectively regularizes neural networks on both supervised and unsupervised tasks.
- Language
-
- English
- Classification
- Computer science and technology
- License
- License undefined
- Identifiers
-
- RERO DOC 328287
- DOI 10.3390/e21080758
- ARK ark:/12658/srd1318902
- Persistent URL
- https://n2t.net/ark:/12658/srd1318902
Statistics
Document views: 60
File downloads:
- Kunze_E_2019.pdf: 138