Developing an effective target appearance model is a challenging task due to the influence of factors such as partial occlusion, illumination variations, fast motion, etc. Existing appearance models usually utilize the tracking results from previous frames as target templates upon which the target appearance model is built by linear combinations of the templates. With such kind of representation, visual tracking is not robust when drastic appearance variations occur. We propose a simple but effe