3
highly relevant, set of primary publications (cf. Section 2.3).
Finally, we have developed various metrics and reviewed
the selected primary publications against these metrics
through full paper examination. After the examination, we
cross-check the extracted results to ensure their correctness
and eventually we report on the findings to the research
community (cf. Section 2.4).
2.1 Initial research questions
Given the common challenges enumerated in the Introduc-
tion section, which have motivated several research lines in
Android apps, we investigate several research questions to
highlight how and which challenges have been focused on
in the literature. In particular, with regards to the fact that
Android has programming specificities (e.g., event-based
mechanisms, GUI), we categorize test concerns targeted by
the research community. With regards to the challenge of
ensuring scalability, we study the tests levels which are
addressed in research works. With regards to the challenge
of generating test cases, we investigate in details the funda-
mental testing techniques leveraged. Finally, with regards
to the fragmentation of the Android ecosystem, we explore
the extent of validation schemes for research approaches.
Overall, we note that testing Android apps is a broad
activity that can target a variety of functional and non-
functional requirements and verification issues, leverage
different techniques and focus on different granularity levels
and phases. Our investigation thus starts with the following
related research questions:
• RQ1: What are the test concerns? With this research
question, we survey the various objectives sought by An-
droid app testing researchers. In general, we investigate
the testing objectives at a high level to determine what
requirements (e.g., security, performance, defects, energy)
the literature addresses. We look more in-depth into the
specificities of Android programming, to enumerate the
priorities that are tackled by the community, including
which concerns (e.g., GUI and ICC mechanism) are fac-
tored in the design of testing strategies.
• RQ2: Which test levels are addressed? With the second
research question, we investigate the levels (i.e., when
the tests are relevant in the app development process)
that research works target. The community could indeed
benefit from knowing to what extent regression testing is
(or is not) developed for apps which are now commonly
known to evolve rapidly.
• RQ3: How are the testing approaches built? In the third
research question, we process detailed information on the
design and implementation of test approaches. In par-
ticular, we investigate the fundamental techniques (e.g.,
concolic testing or mutation testing) leveraged, as well as
the amount of input information (i.e., to what extent the
tester should know about the app prior to testing) that
approaches require to perform.
• RQ4: To what extent are the testing approaches vali-
dated? Finally, the fourth research question investigates
the metrics, datasets and procedures in the literature for
measuring the effectiveness of state-of-the-art approaches.
Answers to this question may shed light on the gaps in the
research agenda of Android testing.
2.2 Search Strategy
We now detail the search strategy that we applied to harvest
literature works related to Android app testing.
Identification of search keywords. Our review focuses
on two key aspects: Testing and Android. Since a diversity
of terms may be used by authors to refer, broadly or pre-
cisely, to any of these aspects, we rely on the extended set
of keywords identified in Table 1. Our final search string
is then constructed as a conjunction of these two categories
of keywords (search string = cat1 & cat2), where each
category is represented as a disjunction of its keywords
(cat = kw1 | kw2 | kw3).
TABLE 1: Search Keywords
Category Keywords
Android
android, mobile, portable device,
smartphone, smart phone, smart device
Test
test, testing, measure, measurement, measuring,
check, checking, detect, detecting, detection
Online repository search. We use the search string on
online literature databases to find and collect relevant pa-
pers. We have considered four widely used repository for
our work: ACM Digital Library
2
, IEEE Xplore Digital Li-
brary
3
, SpringerLink
4
, and ScienceDirect
5
. The “advanced”
search functionality of the four selected online repositories
are known to be inaccurate, which usually result in a huge
set of irrelevant publications, noising the final paper set [22].
Indeed, those irrelevant publications do not really match our
keywords criteria. For example, they may not contain any of
the keywords shown in the Test category. Thus, we develop
scripts (combined with Python and Shell) to perform off-line
matching verification on the papers yielded by those search
engines, where the scripts follow exactly the same criteria
that we have used for online repository search. For example,
regarding the keywords enumerated in the Test category,
if none of them is presented in a publication, the scripts
will mark that publication as irrelevant and subsequently
exclude it from the candidate list.
Major venues search. Since we only consider a few
repositories for search, the coverage can be limited given
that a few conferences such as NDSS
6
and SEKE
7
do not
host their proceedings in the aforementioned repositories.
Thus, to mitigate the threat to validity of not including all
relevant papers, we further explicitly search in proceedings
of all major venues in computer science. We have chosen the
comprehensive CCF-ranking of venues
8
and leveraged the
DBLP
9
repository to collect the Document Object Identifiers
(DOI) of the publications in order to crawl abstracts and
all publication metadata. Since this search process considers
major journal and conference venues, the resulting set of
2. http://dl.acm.org/
3. http://ieeexplore.ieee.org/Xlpore/home.jsp
4. http://link.springer.com
5. http://www.sciencedirect.com
6. The Network and Distributed System Security Symposium
7. International Conference on Software Engineering & Knowledge
Engineering
8. http://www.ccf.org.cn/sites/ccf/paiming.jsp, we only take into
account software engineering and security categories, as from what have
observed, the majority of papers related to testing Android apps.
9. http://dblp.uni-trier.de
评论0