高斯过程回归-经典文章拜读

Gaussian Processes for Regression:

A Quick Introduction

M. Ebden, August 2008

Comments to mebden@gmail.com

1 MOTIVATION

Figure 1 illustrates a typical example of a prediction problem: given some noisy obser-

vations of a dependent variable at certain values of the independent variable x, what is

our best estimate of the dependent variable at a new value, x

∗

?

than claiming f(x) relates to some specific models (e.g. f(x) = mx + c), a Gaussian

process can represent f(x) obliquely, but rigorously, by letting the data ‘speak’ more

clearly for themselves. GPR is still a form of supervised learning, but the training data

are harnessed in a subtler way.

As such, GPR is a less ‘parametric’ tool. However, it’s not completely free-form,

and if we’re unwilling to make even basic assumptions about f(x), then more gen-

eral techniques should be considered, including those underpinned by the principle of

maximum entropy; Chapter 6 of Sivia and Skilling (2006) offers an introduction.

Figure 1: Given six noisy data points (error bars are indicated with vertical lines), we

are interested in estimating a seventh at x

∗

= 0.2.

, . . . , y

imagined as a single point sampled from some multivariate (n-variate) Gaussian distri-

bution, after enough thought. Hence, working backwards, this data set can be partnered

with a GP. Thus GPs are as universal as they are simple.

Very often, it’s assumed that the mean of this partner GP is zero everywhere. What

relates one observation to another in such cases is just the covariance function, k(x, x

A popular choice is the ‘squared exponential’,

exp

, (1)

where the maximum allowable covariance is defined as σ

0

this maximum, meaning f(x) is nearly perfectly correlated with f(x

for our function to look smooth, neighbours must be alike. Now if x is distant from

ment errors and so on. Each observation y can be thought of as related to an underlying

function f(x) through a Gaussian noise model:

), (2)

something which should look familiar to those who’ve done regression before. Regres-

sion is the search for f (x). Purely for simplicity of exposition in the next page, we take

the novel approach of folding the noise into k(x, x

0

), by writing

k(x, x

) = σ

+ σ

δ(x, x

), (3)

where δ(x, x

0

) is the Kronecker delta function. (When most people use Gaussian pro-

cesses, they keep σ

). However, our redefinition of k(x, x

equally suitable for working with problems of the sort posed in Figure 1. So, given n

observations y, our objective is to predict y

∗

∗

; their expected values

are identical according to (2), but their variances differ owing to the observational noise

, x

) k(x

, x

)

, x

) k(x

, x

)

.

.

.

K

∗

=

k(x

∗

, x

K

= k(x

, x

∗

). (5)

Confirm for yourself that the diagonal elements of K are σ

+ σ

, and that its extreme

off-diagonal elements tend to zero when x spans a large enough domain.

∗

∗

where T indicates matrix transposition. We are of course interested in the conditional

probability p(y

?”. As

explained more slowly in the Appendix, the probability follows a Gaussian distribution:

y

∗

∗

K

y, K

∗∗

∗

K

K