3
Part I
Linear Regression
To make our housing example more interesting, lets consider a slightly richer
dataset in which we also know the number of bedrooms in each house:
Living area (feet
2
)
#bedrooms Price (1000$s)
2104 3 400
1600
3 330
2400
3 369
1416
2 232
3000
4 540
.
.
.
.
.
.
.
.
.
Here, the x’s are two-dimensional vectors in R
2
. For instance, x
(i)
1
is the
living area of the i-th house in the training set, and x
(i)
2
is its number of
bedrooms. (In general, when designing a learning problem, it will be up to
you to d ecide what features to choose, so if you are out in Portland gathering
housing data, you might also decide to include other features such as whether
each house has a fireplace, the number of bathrooms, and so on. We’ll say
more about feature selection later, but for now lets take the features as given.)
To perform super vised learning, we must decide how we’re going to rep-
resent functions/hypotheses h in a computer. As an initial choice, lets say
we decide to approximate y as a linear function of x:
h
θ
(x) = θ
0
+ θ
1
x
1
+ θ
2
x
2
Here, the θ
i
’s are the parameters (also called weights) parameterizing the
space of linear functions mapping from X to Y. When there is no risk of
confusion, we will drop the θ subscript in h
θ
(x), and write it more simply as
h(x). To simplify our notation, we also introduce the convention of letting
x
0
= 1 (this is the intercept term), so that
h(x) =
n
X
i=0
θ
i
x
i
= θ
T
x,
where on the right-hand side above we are viewing θ and x both as vectors,
and here n is the number of input variables (not counting x
0
).
Now, given a training set, how do we pick, or learn, the parameters θ?
One reasonable method seems to be to make h(x) close to y, at least for
