To Isa
Preface
Taken literally, the title “All of Statistics” is an exaggeration. But in spirit,
the title is apt, as the book does cover a much broader range of topics than a
typical introductory b ook on mathematical statistics.
This book is for people who want to learn probability and statistics quickly.
It is suitable for graduate or advanced undergraduate students in computer
science, mathematics, statistics, and related disciplines. The book includes
modern topics like nonparametric curve estimation, bootstrapping, and clas-
sification, topics that are usually relegated to follow-up courses. The reader is
presumed to know calculus and a little linear algebra. No previous knowledge
of probability and statistics is required.
Statistics, data mining,andmachine learning are all concerned with
collecting and analyzing data. For some time, statistics research was con-
ducted in statistics departments while data mining and machine learning re-
search was conducted in computer science departments. Statisticians thought
that computer scientists were reinventing the wheel. Computer scientists
thought that statistical theory didn’t apply to their problems.
Things are changing. Statisticians now recognize that computer scientists
are making novel contributions while computer scientists now recognize the
generality of statistical theory and methodology. Clever data mining algo-
rithms are more scalable than statisticians ever thought possible. Formal sta-
tistical theory is more pervasive than computer scientists had realized.
Students who analyze data, or who aspire to develop new methods for
analyzing data, should be well grounded in basic probability and mathematical
statistics. Using fancy tools like neural nets, boosting, and support vector
viii Preface
machines without understanding basic statistics is like doing brain surgery
before knowing how to use a band-aid.
But where can students learn basic probability and statistics quickly? Nowhere.
At least, that was my conclusion when my computer science colleagues kept
asking me: “Where can I send my students to get a good understanding of
modern statistics quickly?” The typical mathematical statistics course spends
too much time on tedious and uninspiring topics (counting methods, two di-
mensional integrals, etc.) at the expense of covering modern concepts (boot-
strapping, curve estimation, graphical models, etc.). So I set out to redesign
our undergraduate honors course on probability and mathematical statistics.
This book arose from that course. Here is a summary of the main features of
this book.
1. The book is suitable for graduate students in computer science and
honors undergraduates in math, statistics, and computer science. It is
also useful for students beginning graduate work in statistics who need
to fill in their background on mathematical statistics.
2. I cover advanced topics that are traditionally not taught in a first course.
For example, nonparametric regression, bootstrapping, density estima-
tion, and graphical models.
3. I have omitted topics in probability that do not play a central role in
statistical inference. For example, counting methods are virtually ab-
sent.
4. Whenever possible, I avoid tedious calculations in favor of emphasizing
concepts.
5. I cover nonparametric inference before parametric inference.
6. I abandon the usual “First Term = Probability” and “Second Term
= Statistics” approach. Some students only take the first half and it
would be a crime if they did not see any statistical theory. Furthermore,
probability is more engaging when students can see it put to work in the
context of statistics. An exception is the topic of stochastic processes
which is included in the later material.
7. The course moves very quickly and covers much material. My colleagues
joke that I cover all of statistics in this course and hence the title. The
course is demanding but I have worked hard to make the material as
intuitive as possible so that the material is very understandable despite
the fast pace.
8. Rigor and clarity are not synonymous. I have tried to strike a good
balance. To avoid getting bogged down in uninteresting technical details,
many results are stated without proof. The bibliographic references at
the end of each chapter point the student to appropriate sources.
Preface ix
Data generating process
Observed data
Probability
Inference and Data Mining
FIGURE 1. Probability and inference.
9. On my website are files with R code which students can use for doing
all the computing. The website is:
http://www.stat.cmu.edu/∼larry/all-of-statistics
However, the book is not tied to R and any computing language can be
used.
Part I of the text is concerned with probability theory, the formal language
of uncer tainty which is the basis of statistical inference. The basic problem
that we study in probability is:
Given a data generating pro cess, what are the properties of the out-
comes?
Part II is about statistical inference and its close cousins, data mining and
machine learning. The basic problem of statistical inference is the inverse of
probability:
Given the outcomes, what can we say about the process that gener-
ated the data?
These ideas are illustrated in Figure 1. Prediction, classification, clustering,
and estimation are all special cases of statistical inference. Data analysis,
machine learning and data mining are various names given to the practice of
statistical inference, depending on the context.
