Journal of Machine Learning Research 1 (2001) 211{244 Submitted 5/00; Published 6/01
Sparse Bayesian Learning and the Relevance Vector Machine
Michael E. Tipping
mtipping@microsoft.com
Microsoft Research
St George House, 1 Guildhal l Street
Cambridge CB2 3NH, U.K.
Editor:
Alex Smola
Abstract
This paper introduces a general Bayesian framework for obtaining
sparse
solutions to re-
gression and classication tasks utilising mo dels linear in the parameters. Although this
framework is fully general, we illustrate our approach with a particular specialisation that
we denote the `relevance vector machine' (RVM), a mo del of identical functional form to
the p opular and state-of-the-art `support vector machine' (SVM). We demonstrate that by
exploiting a probabilistic Bayesian learning framework, we can derive accurate prediction
models whichtypically utilise dramatically fewer basis functions than a comparable SVM
while oering a numb er of additional advantages. These include the b enets of probabilis-
tic predictions, automatic estimation of `nuisance' parameters, and the facility to utilise
arbitrary basis functions (
e.g.
non-`Mercer' kernels).
We detail the Bayesian framework and asso ciated learning algorithm for the RVM, and
give some illustrative examples of its application along with some comparative b enchmarks.
We oer some explanation for the exceptional degree of sparsity obtained, and discuss
and demonstrate some of the advantageous features, and potential extensions, of Bayesian
relevance learning.
1. Intro duction
In
supervised learning
we are given a set of examples of input vectors
f
x
n
g
N
n
=1
along with
corresp onding targets
f
t
n
g
N
n
=1
, the latter of which might be real values (in
regression
)
or class labels (
classication
). From this `training' set we wish to learn a mo del of the
dep endency of the targets on the inputs with the ob jective of making accurate predictions
of
t
for previously unseen values of
x
. In real-world data, the presence of noise (in regression)
and class overlap (in classication) implies that the principal mo delling challenge is to avoid
`over-tting' of the training set.
Typically,we base our predictions up on some function
y
(
x
) dened over the input space,
and `learning' is the pro cess of inferring (p erhaps the parameters of ) this function. A exible
and p opular set of candidates for
y
(
x
)isthatoftheform:
y
(
x
;
w
)=
M
X
i
=1
w
i
i
(
x
)=
w
T
(
x
)
;
(1)
where the output is a linearly-weighted sum of
M
, generally nonlinear and xed, basis func-
tions
(
x
)=(
1
(
x
)
;
2
(
x
)
;:::;
M
(
x
))
T
. Analysis of functions of the typ e (1) is facilitated
c
2001 Michael E. Tipping.
剩余33页未读，继续阅读
资源评论
