模式识别第四版答案

Notes and Solutions for: Pattern Recognition by

Sergios Theodoridis and Konstantinos Koutroumbas.

John L. Weatherwax

∗

January 19, 2006

Introduction

Here you’ll find some notes that I wrote up as I worked through this excellent book. I’ve

worked hard to make these notes as good as I can, but I have no illusions tha t they are perfect.

If you feel that that there is a better way to accomplish or explain an exercise or derivation

for helping improve these notes and solutions.

All comments (no matter how small) are much appreciated. In fact, if you find these notes

useful I would appreciate a contribution in the form of a solution to a problem that I did

not work, a mathematical derivation of a statement or comment made in the book that was

unclear, a piece of code that implements one of the algorithms discussed, or a correction to

a typo (spelling, grammar, etc) about these notes. Sort of a “take a penny, leave a penny”

type of approach. Remember: pay it forward.

∗

wax@alum.mit.edu

1

with feature vector x is observed. Now we only observe the feature vector x and not the true

class label k. Since we must still perform an action when we observe x let λ

loss associated with the event that the object is truly from class k and we decided that it is

from class i. Define r

r

k

=

M

λ

ki

P (we classify this object as a member of class i)

which is the books equation 2.14. Thus the total risk r is the expected value of the class

dependent risks r

taking into account how likely each class is or

r =

M

k=1

r

P (ω

)

k

=

M

i=1

R

The decision rule that leads to the smallest total risk is obtained by selecting R

i

to be the

region of feature space in which the integrand above is as small as possible. That is, R

i

should be defined as the values of x such that for that value of i we have

M

k=1

M

2

When the covariance matrices for two classes are the same and diagonal i.e. Σ

i

= Σ

j

= σ

then the discrimination functions g

(x) are given by

ij

(x) = w

j

)

T

) , (2)

since the vector w is w = µ

point on the decision hyperplane it also must satisfy g

(x) = 0 from the functional form for

g

ij

T

) = 0 .

This is the statement that the line connecting µ

and µ

j

is orthogonal to the decision hy-

perplane. In the same way, when the covariance matrices of each class are not diagona l but

i

j

to t he respective class means µ

i

probability will have a “larger” region of R

P (ω

) < P (ω

j

) then ln

P (ω

=

1

2

+ µ

)

)

i

Then taking the derivative with respect to µ, setting the result equal to zero, and calling

that solution ˆµ gives

σ

σ

µ

Maximum Entropy Est imation

As another method to determine distribution parameters we seek to maximize the entropy

H or

L

x

x

4

above is equivalent to

L

x

x

Solving for the integral of ln(p(x)) we get

x

) .

Ta ke the x

e

λ−1

(x

) = 1 ,