Deterministic Variational Inference for Robust Bayesian Neural N
需积分: 0 105 浏览量
2022-08-03
13:26:06
上传
评论
收藏 3.2MB PDF 举报
Published as a conference paper at ICLR 2019
DETERMINISTIC VARIATIONAL INFERENCE FOR
ROBUST BAYESIAN NEURAL NETWORKS
Anqi Wu
1∗
, Sebastian Nowozin
2†
, Edward Meeds
4
,
Richard E. Turner
3,4
, Jos
´
e Miguel Hern
´
andez-Lobato
3,4
&
Alexander L. Gaunt
4
1
Princeton Neuroscience Institute, Princeton University
2
Google AI Berlin
3
Department of Engineering, University of Cambridge
4
Microsoft Research, Cambridge
anqiw@princeton.edu, nowozin@google.com,
{ret26,jmh233}@cam.ac.uk, {ted.meeds, algaunt}@microsoft.com
ABSTRACT
Bayesian neural networks (BNNs) hold great promise as a flexible and principled
solution to deal with uncertainty when learning from finite data. Among approaches
to realize probabilistic inference in deep neural networks, variational Bayes (VB)
is theoretically grounded, generally applicable, and computationally efficient. With
wide recognition of potential advantages, why is it that variational Bayes has seen
very limited practical use for BNNs in real applications? We argue that variational
inference in neural networks is fragile: successful implementations require careful
initialization and tuning of prior variances, as well as controlling the variance of
Monte Carlo gradient estimates. We provide two innovations that aim to turn VB
into a robust inference tool for Bayesian neural networks: first, we introduce a novel
deterministic method to approximate moments in neural networks, eliminating
gradient variance; second, we introduce a hierarchical prior for parameters and
a novel Empirical Bayes procedure for automatically selecting prior variances.
Combining these two innovations, the resulting method is highly efficient and
robust. On the application of heteroscedastic regression we demonstrate good
predictive performance over alternative approaches.
1 INTRODUCTION
Bayesian approaches to neural network training marry the representational flexibility of deep neural
networks with principled parameter estimation in probabilistic models. Compared to “standard”
parameter estimation by maximum likelihood, the Bayesian framework promises to bring key
advantages such as better uncertainty estimates on predictions and automatic model regularization
(MacKay, 1992; Graves, 2011). These features are often crucial for informing downstream decision
tasks and reducing overfitting, particularly on small datasets. However, despite potential advantages,
such Bayesian neural networks (BNNs) are often overlooked due to two limitations: First, posterior
inference in deep neural networks is analytically intractable and approximate inference with Monte
Carlo (MC) techniques can suffer from crippling variance given only a reasonable computation
budget (Kingma et al., 2015; Molchanov et al., 2017; Miller et al., 2017; Zhu et al., 2018). Second,
performance of the Bayesian approach is sensitive to the choice of prior (Neal, 1993), and although
we may have a priori knowledge concerning the function represented by a neural network, it is
generally difficult to translate this into a meaningful prior on neural network weights. Sensitivity to
priors and initialization makes BNNs non-robust and thus often irrelevant in practice.
In this paper, we describe a novel approach for inference in feed-forward BNNs that is simple to
implement and aims to solve these two limitations. We adopt the paradigm of variational Bayes (VB)
for BNNs (Hinton & van Camp, 1993; MacKay, 1995c) which is normally deployed using Monte
∗
Work done during an internship at Microsoft Research, Cambridge.
†
Work done while at Microsoft Research, Cambridge.
1
剩余23页未读，继续阅读
评论0