Chapter 7
AdaBoost
Zhi-Hua Zhou and Yang Yu
Contents
7.1 Introduction ........................................................... 127
7.2 The Algorithm ......................................................... 128
7.2.1 Notations ....................................................... 128
7.2.2 A General Boosting Procedure .................................. 129
7.2.3 The AdaBoost Algorithm ....................................... 130
7.3 Illustrative Examples ................................................... 133
7.3.1 Solving XOR Problem .......................................... 133
7.3.2 Performance on Real Data ...................................... 134
7.4 Real Application ....................................................... 136
7.5 Advanced Topics ....................................................... 138
7.5.1 Theoretical Issues ............................................... 138
7.5.2 Multiclass AdaBoost ............................................ 142
7.5.3 Other Advanced Topics ......................................... 145
7.6 Software Implementations .............................................. 145
7.7 Exercises .............................................................. 146
References .................................................................. 147
7.1 Introduction
Generalization ability, which characterizes how well the result learned from a given
training data set can be applied to unseen new data, is the most central concept in
machine learning. Researchers have devoted tremendous efforts to the pursuit of tech-
niques that could lead to a learning system with strong generalization ability. One
of the most successful paradigms is ensemble learning [32]. In contrast to ordinary
machine learning approaches which try to generate one learner from training data,
ensemble methods try to construct a set of base learners and combine them. Base
learners are usually generated from training data by a base learning algorithm which
canbeadecisiontree,aneuralnetwork,orotherkindsofmachinelearningalgorithms.
Just like “many hands make light work,” the generalization ability of an ensemble
is usually significantly better than that of a single learner. Actually, ensemble meth-
ods are appealing mainly because they are able to boost weak learners, which are
127
© 2009 by Taylor & Francis Group, LLC
评论0