IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, VOL. 19, NO. 4, AUGUST 2015 575
A Discrete Particle Swarm Optimization for
Covering Array Generation
Huayao Wu, Changhai Nie, Member, IEEE, Fei-Ching Kuo, Member, IEEE,
Hareton Leung, Member, IEEE, and Charles J. Colbourn
Abstract—Software behavior depends on many factors.
Combinatorial testing (CT) aims to generate small sets of test
cases to uncover defects caused by those factors and their
interactions. Covering array generation, a discrete optimization
problem, is the most popular research area in the field of CT.
Particle swarm optimization (PSO), an evolutionary search-based
heuristic technique, has succeeded in generating covering arrays
that are competitive in size. However, current PSO methods for
covering array generation simply round the particle’s position
to an integer to handle the discrete search space. Moreover, no
guidelines are available to effectively set PSOs parameters for
this problem. In this paper, we extend the set-based PSO, an
existing discrete PSO (DPSO) method, to covering array gen-
eration. Two auxiliary strategies (particle reinitialization and
additional evaluation of gbest) are proposed to improve perfor-
mance, and thus a novel DPSO for covering array generation
is developed. Guidelines for parameter settings both for conven-
tional PSO (CPSO) and for DPSO are developed systematically
here. Discrete extensions of four existing PSO variants are devel-
oped, in order to further investigate the effectiveness of DPSO
for covering array generation. Experiments show that CPSO can
produce better results using the guidelines for parameter settings,
and that DPSO can generate smaller covering arrays than CPSO
and other existing evolutionary algorithms. DPSO is a promising
improvement on PSO for covering array generation.
Index Terms—Combinatorial testing (CT), covering array
generation, particle swarm optimization (PSO).
I. INTRODUCTION
A
S SOFTWARE functions and run-time environments
become more complex, testing of modern software sys-
tems is becoming more expensive. Effective detection of
Manuscript received May 12, 2013; revised December 29, 2013 and
May 18, 2014; accepted September 28, 2014. Date of publication October 9,
2014; date of current version July 28, 2015. This work was supported in part
by the National Natural Science Foundation of China under Grant 61272079,
in part by the Research Fund for the Doctoral Program of Higher Education
of China under Grant 20130091110032, in part by the Science Fund for
Creative Research Groups of the National Natural Science Foundation of
China under Grant 61321491, in part by the Major Program of National
Natural Science Foundation of China under Grant 91318301, and in part
by the Australian Research Council Linkage under Grant LP100200208.
(Corresponding author: Changhai Nie.)
H. Wu and C. Nie are with the State Key Laboratory for Novel
Software Technology, Nanjing University, Nanjing 210023, China (e-mail:
hywu@outlook.com; changhainie@nju.edu.cn).
F.-C. Kuo is with the Faculty of Information and Communication
Technologies, Swinburne University of Technology, Hawthorn, VIC 3122,
Australia (e-mail: dkuo@swin.edu.au).
H. Leung is with the Department of Computing, Hong Kong Polytechnic
University, Hong Kong (e-mail: hareton.leung@polyu.edu.hk).
C. J. Colbourn is with Arizona State University, Tempe, AZ 85287-8809,
USA (e-mail: colbourn@asu.edu).
Digital Object Identifier 10.1109/TEVC.2014.2362532
failures at a low cost is a key issue for test case genera-
tion. Combinatorial testing (CT) is a popular testing method
to detect failures triggered by various factors and their interac-
tions [1]. By employing covering arrays as test suites, the CT
method aims to sample the large combination space with few
test cases to cover different interactions among a fixed num-
ber of factors. Kuhn and Reilly [2] shows that more than 70%
of the failures in certain software were caused by the interac-
tions of one or two factors, and almost all the failures could
be detected by checking the interactions among six factors.
Therefore, CT can be an effective method in practice.
Generating a covering array with fewest tests (minimum
size) is a major challenge in CT. In general, the minimum
size of a covering array is unknown; hence, methods have
focused on finding covering arrays that have as few tests as
possible at reasonable search cost. The many methods that
have been proposed can be classified into two main groups:
1) mathematical methods and 2) computational methods [1].
Mathematical (algebraic or combinatorial) methods typically
exploit some known combinatorial structure. Computational
methods primarily use greedy strategies or heuristic-search
techniques to generate covering arrays, due to the size of the
search space.
Mathematical methods yield the best possible covering
arrays in certain cases. For example, orthogonal arrays used
in the design of experiments provide covering arrays with a
number of tests that is provably minimum. However, all known
mathematical methods can be applied only for restrictive sets
of factors. This limitation has led to an emphasis on compu-
tational methods. Greedy algorithms have been quite effective
in generating covering arrays, but their accuracy suffers from
becoming trapped in local optima.
In recent years, search-based software engineering (SBSE)
has focused on using search-based optimization algorithms
to find high-quality solutions for software engineering prob-
lems. Inspired by SBSE, many artificial intelligence-based
heuristic-search techniques have been applied to software
testing. For example, simulated annealing (SA) [3]–[7],
genetic algorithm (GA) [8]–[10], and ant colony
optimization (ACO) [9], [11], [12] have all been applied
to covering array generation. These techniques can generate
any types of covering arrays, and the constraint solving
and prioritization techniques can be easily integrated. Their
applications have been shown to be effective, producing
relatively small covering arrays in many cases. Particle swarm
optimization (PSO), a relatively new evolutionary algorithm,
1089-778X
c
2014 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.