277
as m
−p/(2p+d)
, where m is the sample size, the data lies in R
d
, and
where the regression function is assumed to be p times differentiable.
We can get a very rough idea of the impact of sample size on the rate
of convergence as follows. Consider a particular point in the sequence
of values corresponding to the optimal rate of convergence: m =10,000
samples, for p = 2 and d = 10. Suppose that d is increased to 20; what
number of samples in the new sequence gives the same value? The
answer is approximately 10 million. If our data lies (approximately) on
a low dimensional manifold L that happens to be embedded in a high
dimensional manifold H, then modeling the data directly in L rather
than in H may turn an infeasible problem into a feasible one.
The purpose of this monograph is to describe the mathematics and
key ideas underlying the methods, and to provide some links to the
literature for those interested in pursuing a topic further.
3
The sub-
ject of dimension reduction is vast, so we use the following criterion
to limit the discussion: we restrict our attention to the case where the
inferred feature values are continuous. The observables, on the other
hand, may be continuous or discrete. Thus this review does not address
clustering methods, or, for example, feature selection for discrete data,
such as text. This still leaves a very wide field, and so we further limit
the scope by choosing not to cover probabilistic topic models (in par-
ticular, latent Dirichlet allocation, nonnegative matrix factorization,
probabilistic latent semantic analysis, and Gaussian process latent vari-
able models). Furthermore, implementation details, and important the-
oretical details such as consistency and rates of convergence of sample
quantities to their population values, although important, are not dis-
cussed. For an alternative, excellent overview of dimension reduction
methods, see Lee and Verleysen [62]. This monograph differs from that
work in several ways. In particular, while it is common in the litera-
ture to see methods applied to artificial, low dimensional data sets such
as the famous Swiss Roll, in this monograph we prefer to use higher
dimensional data: while low dimensional toy data can be valuable to
lim
n
sup
Θ
P (
ˆ
T
n
− θ≥cb
n
)=0; {b
n
} is called an optimal rate of convergence if it is
both a lower rate of convergence and an achievable rate of convergence. Here the inf
ˆ
T
n
is
over all possible estimators
ˆ
T
n
.
3
This monograph is a revised and extended version of Burges [17].
