AdaboostSVM
5星 · 超过95%的资源 需积分: 32 37 浏览量
2012-07-04
15:51:00
上传
评论 2
收藏 334KB PDF 举报 
Engineering Applications of Artificial Intelligence 21 (2008) 785–795
AdaBoost with SVM-based component classifiers
Xuchun Li
, Lei Wang, Eric Sung
School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore
Received 28 May 2006; received in revised form 22 May 2007; accepted 13 July 2007
Available online 14 September 2007
Abstract
The use of SVM (Support Vector Machine) as component classifier in AdaBoost may seem like going against the grain of the Boosting
principle since SVM is not an easy classifier to train. Moreover, Wickramaratna et al. [2001. Performance degradation in boosting. In:
Proceedings of the Second International Workshop on Multiple Classifier Systems, pp. 11–21] show that AdaBoost with strong
component classifiers is not viable. In this paper, we shall show that AdaBoost incorporating properly designed RBFSVM (SVM with the
RBF kernel) component classifiers, which we call AdaBoostSVM, can perform as well as SVM.
Furthermore, the proposed AdaBoostSVM demonstrates better generalization performance than SVM on imbalanced classification
problems. The key idea of AdaBoostSVM is that for the sequence of trained RBFSVM component classifiers, starting with large s values
(implying weak learning), the s values are reduced progressively as the Boosting iteration proceeds. This effectively produces a set of
RBFSVM component classifiers whose model parameters are adaptively different manifesting in better generalization as compared to
AdaBoost approach with SVM component classifiers using a fixed (optimal) s value. From benchmark data sets, we show that our
AdaBoostSVM approach outperforms other AdaBoost approaches using component classifiers such as Decision Trees and Neural
Networks. AdaBoostSVM can be seen as a proof of concept of the idea proposed in Valentini and Dietterich [2004. Bias-variance analysis
of support vector machines for the development of SVM-based ensemble methods. Journal of Machine Learning Research 5, 725–775]
that Adaboost with heterogeneous SVMs could work well. Moreover, we extend AdaBoostSVM to the Diverse AdaBoostSVM to address
the reported accuracy/diversity dilemma of the original Adaboost. By designing parameter adjusting strategies, the distributions of
accuracy and diversity over RBFSVM component classifiers are tuned to maintain a good balance between them and promising results
have been obtained on benchmark data sets.
r 2007 Published by Elsevier Ltd.
Keywords: AdaBoost; Support Vector Machine; Component classifier; Diversity
1. Introduction
One of the major developments in machine learning in
the past decade is the Ensemble method, which finds a
highly accurate classifier by combining many moderately
accurate component classifiers. Two of the commonly used
techniques for constructing Ensemble classifiers are Boost-
ing (Schapire, 2002) and Bagging (Breiman, 1996).
Compared with Bagging, Boosting performs better when
the data do not have much noise (Opitz and Maclin, 1999;
Bauer and Kohavi, 1999). As the most popular Boosting
method, AdaBoost (Freund and Schapire, 1997) creates a
collection of component classifiers by maintaining a set of
weights over training samples and adaptively adjusting
these weights after each Boosting iteration: the weights of
the training samples which are misclassified by current
component classifier will be increased while the weights of
the training samples which are correctly classified will be
decreased. Several ways have been proposed to implement
the weight update in Adaboost (Kuncheva and Whitaker,
2002).
The success of AdaBoost can be attributed to its ability
to enlarge the margin (Schapire et al., 1998), which could
enhance the generalization capability of AdaBoost. Many
studies that use Decision Trees (Dietterich, 2000) or Neural
Networks (Schwenk and Bengio, 2000; Ratsch, 2001)as
component classifiers in AdaBoost have been reported.
These studies show good generalization performance of
ARTICLE IN PRESS
www.elsevier.com/locate/engappai
0952-1976/$ - see front matter r 2007 Published by Elsevier Ltd.
doi:10.1016/j.engappai.2007.07.001
Corresponding author. Tel.: +65 9092 7335.
E-mail addresses: uchunli@pmail.ntu.edu.sg, xuchunli@pmail.ntu.
edu.sg (X. Li).
剩余10页未读，继续阅读
评论6
最新资源