2018 6th International Conference in Software Engineering Research and Innovation
Edroid: A Mutation tool for Android Apps
Eduardo Luna
Department of Computer Science
Pennsylvania State University
Harrisburg, PA, USA
eal5376@psu.edu
Abstract— Android applications have experienced an
outstanding growth in recent years with a projected figure of
378 billion in the number of application downloads by 2020. As
a result, verification and validation of Android apps and their
graphical user interface through testing is crucial for quality
assurance. One way to assess quality assurance is through the
use of mutation testing. Mutation testing modifies the code of the
System Under Test in small ways in order to produce faulty
versions of the program called mutants and compare them to
the original version. However, the cost of mutation testing can
be expensive and time consuming. We propose a graphical user-
friendly Android mutation tool called Edroid whose primary
purpose is to mutate Android’s main components such as
activities, services, content providers and broadcast receivers
using the source code of XML files. We introduce fourteen
Android mutation operators, where ten of them are new, that
are derived from operators used in different fields such as
mutation testing for Graphical User Interface and XML
schema. We evaluate Edroid on five real world Android
applications using testing methodologies such as random testing
and black box testing. Results from these experiments
demonstrate that Edroid facilitates the generation of mutants.
In addition and equally important, the set of proposed mutation
operators are effective in revealing errors that improve the
quality of a given test suite.
Keywords—mutation testing, mobile application testing,
verification & validation, fault-based testing, software testing
I. INTRODUCTION
Mutation testing, as a fault-based technique, modifies the
code of the System Under Test (SUT) in small changes in
order to produce faulty versions of the program called
mutants. This approach was originally proposed in the early
1970’s by Richard Lipton [1]. Each modification is based on
a well-defined syntactic rule called a mutation operator. Tests
cases are then executed on these mutants and their result
(actual output) is compared to the original one (expected
output). If the actual and expected output differ, then we
imply that the mutant is killed. As a result, the quality of a
test suite is measured by using the mutation score which is
the percentage of killed mutants. The higher the mutation
score, the better the test cases. In other words, the main goal
of mutation testing is to evaluate an existing test suite and
give directions on how to design more effective test cases.
978-1-5386-6577-0/18/$31.00 ©2018 IEEE
DOI 10.1109/CONISOFT.2018.00022
Authorized licensed use limited to: University of the Cumberlands. Downloaded on August 09,2021 at 17:58:15 UTC from IEEE Xplore. Restrictions apply.
Omar El Ariss
Department of Computer Science
Texas A&M University-Commerce
Commerce, TX, USA
Omar.El.Ariss@tamuc.edu
In order to reduce the laborious process of manually
generating mutants for a system under test, automated
mutation tools have been developed to target different
programming languages. Some of the popular tools are
Proteum for the C language [2], Mutpy for Python [3], Mutant
for Ruby [4] and Mujava for Java[5].
The importance of an Android Mutation tool lies on the
fact that Android mobile applications have experienced a
noticeable growth in the last decade with a promising
projected figure of nearly 378 billion apps downloads by
2020 [6]. Companies that develop Android apps tend to
accelerate the development process and apply rapid
development techniques such as Agile to increase
productivity and reduce cost efforts [7]. Unfortunately, due
to the eagerness of releasing the apps before the competitors,
most of the time, the testing process falls short and is not
given enough time. Appbrain shows that nearly 13% of
Android apps have low quality [8]. This can be due to the fact
that the apps need to be tested on different screen sizes and it
is considered to be a tedious process. In addition, there is
insufficient knowledge among developers on how the apps
should be properly tested. Google provides various tools to
test android apps such as UIAutomator [9], Espresso [9] and
AndroitJUnitRunner [9]. These tools are very useful to ease
the testing process and generate test suites faster.
Furthermore, it is important to evaluate the effectiveness of
the test suite of the app being tested.
In this paper, we propose Edroid, an automated mutation
testing tool whose goal is the automatic generation of mutants
for Android apps from their graphical user interface (GUI).
Edroid applies strong mutation testing also called traditional
mutation testing with a full generation of source code for
mutants. Edroid as a tool allows the user to mutate their
Android app using a list of fourteen mutation operators. Our
list of mutation operators includes operators that are used in
traditional GUI applications, XML (Extensible Markup
Language) code, and from recent research done on Android
apps [10-11]. Android mutation operators from previous
work focus mainly on mutating the GUI and configuration
files of Android apps. In addition, we use mutation operators
applied from different fields such as operators used to mutate
99
XML schemas [12-14] and Java programs with GUI [15-17]
and use these operators to design new mutation operators for
Android apps.
The proposed study makes the following contributions:
• Ten newly introduced mutation operators for
Android apps.
• Evaluate Android apps through the GUI and
configuration rather than the java code. To handle
specific components such as Activity, Services and
Content providers, only present in the Android
development platform.
• Development of Edroid with the goal of enabling
module extension and future improvements in the
Android Mutation testing field.
The rest of the paper is organized as follows. Section II
describes related work done in the mutation testing field such
as the development of previous mutation systems. Section III
introduces our proposed mutation operators. Section IV
describes Edroid, its functionalities and limitations. Section
V describes the methodology used in order to apply mutation
testing on Android apps and how the mutants are evaluated.
Section VI evaluates Edroid on three Android mobile apps
and Section VII concludes the paper with vision for future
work.
II. RELATED WORK
In this section we focus on recent research done in four
areas: Android testing, mutation testing for GUI,
configuration files for android apps, and an overview of
existing mutation testing tools.
The Android Development environment [9] incorporates a
testing framework that includes testing automation tools such
as Espresso, UI Automator, and AndroidJUnitRunner. The
Espresso test recorder tool allows the creation of User
Interface (UI) tests for Android apps, which is suitable for
functional UI testing [18]. The UI Automator provides a set
of APIs to design UI tests that allows the tester to perform
interactions on the Android device and apps such as
inspecting the layout hierarchy of the target device or
touching a button within an installed app. These interactions
are suitable for cross-app UI testing across the android device
operating system. AndroidJunitRunner extends Junit [19]
functionality and is suitable for unit testing. In addition,
commercial tools such as Robotium [20] are often used for
the design of UI tests. Robotium allows the recording of UI
tests faster and simpler than most existing Android testing
tools and is suitable for functional UI testing. All of the
previous mentioned tools are useful for the design of test
cases, and will be used in order to evaluate our proposed tool.
Mutation testing was initially proposed in 1971 by
100
Authorized licensed use limited to: University of the Cumberlands. Downloaded on August 09,2021 at 17:58:15 UTC from IEEE Xplore. Restrictions apply.
Richard Lipton [1], and was later developed and published by
DeMillo et al [21]. Pilot mutation System (PIMS) [22] was
the first recognized Mutation tool that pioneered the process
of automated mutation testing. PIMS consisted of three
phases. First, the mutants are generated from the original
code. Second, the mutation tool accepts the test cases as input
from the user. Finally, the test cases are executed on all the
mutants and the number of killed mutants is determined.
Since then, most of the mutation testing tools that are
developed by various researchers, and which we will follow
here in this paper, try to follow this approach.
Much of the mutation testing tools developed up to date
target the Java programming language, being MuJava [5] the
mostly used tool among them. MuJava uses traditional
statement level mutation operators and class level operators
to apply mutation testing on java programs. The mutation is
done directly at bytecode level to improve the performance
of the tool. Mujava allows the tester to select the mutation
operators through an easy-to-use interface. It has been
utilized by a great amount of research studies due to its
accessibility and simplicity to accomplish the mutation
process. For this reason, Edroid’s layout is influenced by
Mujava’s interface and design.
Few research have explored the field of mutation testing
on GUIs. Alegroth et al [15], introduced a set of eighteen
mutation operators and divided them into three different
classes: “removing”, “modifying” and “adding” code in the
system under test. The authors applied their proposed
operators on two GUI tools: Guitar (an automated
component-based GUI) [15] and Vgt Guitar (a visual GUI
prototype) [23]. The latter one being not useful in practice but
only for the purpose of the authors’ research. Oliveira et al.
[16] evaluated the same mutation operators on two different
applications: a regular Java calculator program and a real-
world application. Alsmadi [17] on the other hand, proposed
seven mutation operators that switch, delete and add GUI
components. These operators were evaluated using four small
open source applications. We believe that testing the GUI of
an Android app is an essential step for the assurance of system
quality, since the majority of the interactions of the users
occur through its GUI. We therefore concluded that GUI
mutation operators should also be applied on Android apps
and will be included in our tool.
Communication between Android applications is crucial
and needs to be considered while testing an Android app. The
Android Application Framework layer provides high-level
services. Services that are used to perform actions in the
background such as playing music through a music player.
Applications made by developers can use these services,
communicate and respond to them using a broadcast receiver.
Avancini and Ceccato [12], proposed an approach to test the
inter-application communication in mobile devices. The
authors modified the intent filters that are required for
android apps to have a correct communication. Therefore, we
introduce two mutation operators based on intent filter
communication for services and receivers.
Although XML files present in an android app source code
are slightly different and have a distinct purpose from
traditional XML files, the presentation semantics of
structured data on both is very similar. Li and Miller [13],
proposed a set of eighteen mutation operators based on
common faults that can occur when developing XML
Schema. Franzotte and Vergilio [14] on the other hand,
introduced ten mutation operators for XML Schema
documents. These operators are also based on common faults
made by developers and are divided into two groups that
produce elementary and structural mutations. Elementary
mutation operators modify elements and attribute values in
the document while structural mutation operators modify the
tree structure of the XML schema. In both of those work, the
authors’ objective is to reveal possible faults in XML
schemas through the use of mutation analysis. In this paper,
we benefit from this structural similarity in XML schemas
and make use of some of the previously proposed mutation
operators and apply them to the XML of an Android app.
As a starting point on the field of Android mutation testing,
Deng et al. [11] introduced an Android mutation tool that uses
eleven Android mutation operators along with the traditional
java operators in Mujava. These android mutation operators
are based on common faults made by developers. Their
operators also target the GUI and configuration files of
Android apps. The GUI operators target individual elements
of the interface while the configuration operators only target
the application permissions. Their mutation tool is then
evaluated using eight applications from Google Play
Marketplace. Their work implements the first android
mutation tool. In addition, Linares-Vasquez et al. [32] [33]
proposed a framework for Android mutation testing called
MDroid+. This framework implement 38 mutation operators
based on android and java faults. Furthermore, the authors
produced a taxonomy of android faults and grouped them into
android, java and mixed faults. These works have similar
approach to our goal. However, in our work, the proposed
mutation operators mutate the layout of a group of elements
in the GUI, and mutate the broadcast receivers and services
used by the Android app using the source code of XML files.
Further, our proposed mutation tool focuses on generating
mutants through the utilization of an easy to use interface.
By analyzing the related work, we conclude that the
process of automated mutation testing implemented to PIMS
can also be implemented to the development of Edroid. In
addition, the various set of existing mutation operators
mentioned in this section, which have been used in previous
101
Authorized licensed use limited to: University of the Cumberlands. Downloaded on August 09,2021 at 17:58:15 UTC from IEEE Xplore. Restrictions apply.
mutation tools, can also be included to Edroid’s mutation
operator list. We conclude that some of the mutation
operators used in other languages such as C can also be
applied in the context of Android Mutation testing.
III. EDROID DESIGN
Edroid, is an Android mutation system that supports
automated generation of mutants for Android apps. The
purpose of Edroid is to reduce the cost of mutating Android
applications, which tends to be time consuming and highly
expensive to achieve. This section describes Edroid’s
functions, and its limitations.
A proper understanding of the organization of an Android
project is needed in order to design Android mutation
operators. Android app is built using four main components
such as activities, services, content providers and broadcast
receivers [24]. These components and their interaction within
the app are described in the Android Manifest file [25]. Fig.
1 summarizes the function of each component.
Fig 1. Android Application components
Fig. 2 demonstrates the mutation testing procedure that
Edroid follows. First, mutants are generated by modifying the
source files of the Android apps by either using a subset or
our entire proposed mutation operators. Edroid allows the
generation of first order or high order mutants. Second, the
Mutant Executor uses a test suite to evaluate the mutants. The
source files of the mutant app are compiled as an APK file.
The APK file is then run on a mobile device. The behavior
and output of this APK file are then documented as the actual
results. Finally, the expected results are compared with the
actual results to determine which mutants are killed. The
mutation score is finally calculated by the number of killed
mutants in proportion to the total number of mutants.
In its current version, some of Edroid‘s components are
not yet automated. The modules with blue color in Fig. 2
indicate the ones that are currently automated while the rest
are done manually. Edroid’s set of mutation operators
currently only mutates the source code of XML files of
android apps. The future goal of Edroid is to cover the entire
mutation testing process, to include java mutation operators
from [11, 32, 33] and to work for both standard Java programs
and Android apps. Edroid does not detect equivalent mutants a folder named after the name of the mutant and its current
during any part of the mutation process nor does it allow them generation number.
to be removed manually through the interface.
IV. ANDROID MUTATION OPERATORS

Edroid includes fourteen mutation operators, eleven of

them target the graphical user interface while three of them
target the configuration files. Out of these fourteen operators,
eight GUI operators and two configuration operators are
newly introduced by us. Table I describes the list of newly
proposed Android Mutation operators from this research
along with its abbreviation and its type.

TABLE I. List of newly proposed Android Mutation Operators

# Abbreviation Description
1 BWI Set Button Widget Invisible (GUI)
2 TWI Set Edit Text Widget Invisible (GUI)
3 VGI Set View Group Layout Invisible (GUI)
4 VGT Change View Group Type (GUI)
5 BWA Change Button Widget Appearance (GUI)
6 TWA Change Edit Text Appearance (GUI)
7 BWX Button Widget Text Switch (GUI)
8 BWP Button Widget Duplicate (GUI)
Fig. 2 Edroid mutation testing procedure
9 RFC Change Intent Filter Receiver (config.)
Edroid consists of an interface that allows the user to 10 SFC Change Intent Filter Service (config.)
generate Android mutants. The interface includes three
components: 1) the list of files to be mutated, 2) the list of
The rest of Edroid’s mutation operators are based on the
mutation operators available to select from and 3) a console
work of Deng et al. [11]. APD (Activity Permission Deletion)
that updates the user regarding the current state of the
which mutates the Android manifest file and BWD (Button
mutation process.
Widget Deletion), BWS (Button Widget Switch) and TWD
(EditText Widget Deletion) which mutate the GUI. These
Edroid first requires the user to enter the location of the
operators also mutate the GUI and configuration files of
folder that contains the Android source files. Then, the next
Android apps.
step is for the user to select the list of mutation operators to
be applied to the Android app. Mutants are generated by
In the following section, we describe in detail how the
pressing the “Generate” button. Edroid detects whether the
Android mutation operators work using two running
type of file is a XML Layout or Android Manifest file and
examples: Calculator for Android and Forecastie apps.
generates the mutants that are selected by the user according
Calculator for Android [27], is a simple calculator that
to its type. The console shows the type and number of
provides basic operations and some advanced mathematical
generated mutants. For every mutant that is generated, the
functions such as sine, cosine and tangent. It has one activity,
mutated file and the rest of the files are also generated with
one Android Manifest file and two Layout files to display the
it. The mutated file shows which excerpt of the source code
GUI of the activity. The leftmost screen of Fig. 3a illustrates
was modified by the mutant.
the original GUI of the main activity. We chose the
Calculator app as a running example for its simplicity and
In order to generate each mutant, Edroid follows a four-
large number of GUI elements. This implies that a large
steps workflow similar to the methodology used by Oliviera
number of mutants will be generated by the GUI Mutation
et al. [16]: (1) Each XML file is read as a text file; (2) a
operators. Forecastie [28] is a weather app that updates the
specific keyword or set of keywords are found on the source
user with the weather conditions of surrounding areas. It has
code; (3) the keywords are replaced one at a time and the part
four activities, one Android Manifest file and eight Layout
of code being replaced is commented with the name of the
files. Forecastie is chosen here as an example to demonstrate
mutant; (4) the newly modified version of the file is saved in
the configuration mutation operators. It receives three

102

Authorized licensed use limited to: University of the Cumberlands. Downloaded on August 09,2021 at 17:58:15 UTC from IEEE Xplore. Restrictions apply.
broadcast messages and offers one service. These messages
are called implicitly, which follow the assumption that we
need to test the mutation operators.
A. XML Layout Mutation Operators
An XML Layout file is used to define the visual layout
of the GUI of an Android app. In this file we declare the GUI
components such as buttons, text fields and screen layouts.
There exist seven different types of layout: Linear Layout,
Relative Layout, Table Layout, Absolute Layout, Frame
Layout, List View, and Grid View. Each layout has a set of
properties that determine the visual features of that layout.
Linear Layout, Relative Layout, Table Layout, and Absolute
Layout are considered common layouts used to organize the
elements that are contained within them. In contrast, List
View and Grid View Layouts are used to display a list of
items. Usually these layouts are filled at runtime to display
dynamic content. They use an adapter which acts like a
middleman between the data source and the layout. The
adapter extracts content from an external source such as a
database or an internal source such as an array and transform
each record into a list item [26].
Without a proper declaration of a layout and the elements
inside of it, the expected GUI may not have the desired
output. Because of this, we introduce the following operators
that mutate the syntax of XML Layout files specifically the
type of layout used, and elements such as buttons and text
fields.
• Set Button Widget Invisible (BWI): This operator
removes one button at a time by making it
inaccessible from the GUI while the functionality
specified in the Java code and the space it occupies
still remains
• Set Edit Text Widget Invisible (TWI): Similar to
BWI, an Edit Text widget is removed one at a time
from the apps’ GUI maintaining its code
functionality and space on the screen.
The purpose of the previous mutation operators is to
assert that the test case fails if the widget is not accessible
on the application GUI [16]. To kill a mutant of this type,
the tester must verify that each widget is properly shown
on the screen.
• Set View Group Layout Invisible (VGI): A view
group layout can contain one or more widgets such
as text fields, buttons, checkboxes and radio buttons.
Each element inside a layout is treated as a child.
The objective of doing this is to dictate the way that
children are arranged on the screen. The VGI
operator removes the entire layout and all the
elements inside of it. This is done for every View
103
Authorized licensed use limited to: University of the Cumberlands. Downloaded on August 09,2021 at 17:58:15 UTC from IEEE Xplore. Restrictions apply.
Group Layout on the screen. The purpose of this
operator is to assert that widgets such as
checkboxes, radio buttons and every other widget
inside the View Group are accessible on the
application GUI.
• Change View Group Type (VGT): This operator
changes the display properties of the elements in a
view group layout. This modification may
sometimes lead to improper arrangement of the
elements on the screen. VGT operator is applied to
every layout on the screen. The purpose of the VGT
operator is to ensure that the tester checks for the
desired distribution and location of the elements on
the screen. Fig. 3b shows an example of a VGT
mutant where the buttons on the screen are displayed
incorrectly.
• Change Button Widget Appearance (BWA):
BWA changes the appearance of a button widget
one at a time. The appearance can be modified by
selecting a different text color, background color,
text-style or typeface. Fig. 3c illustrates an example
of a BWA mutant where the button “4” text color
has been modified.
• Change Edit Text Appearance (TWA): Similar to
BWA, this operator changes the appearance of an
edit text widget one at a time. An example of a TWA
mutant is shown in Fig. 3d where the result text field
background color is changed to a white color.
The purpose of BWA and TWA operators is to obligate
the tester to build test cases that check whether the buttons
and text fields inside of the GUI are displayed as desired. This
involves that the expected behavior depends on the change
being intentional or unintentional [16]. Utilizing the BWA
mutant displayed in the third screen of Fig 3c as an example,
a test should fail if the desired text color of button “4” is not
black, otherwise the test should pass.
• Button Widget Text Switch (BWX): BWX
switches the text of a button widget with the text of
every other button on the screen one at a time. The
underlying functionality of both buttons remains but
the text displayed in each button is swapped
• Button Widget Duplicate (BWP): The BWP
operator replaces and overwrites the text of a button
widget to the text of every other button on the
screen. The functionality of the buttons is not
modified but the text displayed on both buttons
changes the GUI and makes it look as if two equal
buttons are on the screen.
Similar to the BWA operator, the purpose of BWX and
BWP operators is to check whether the buttons inside of the
 Fig. 3a Original GUI Fig. 3b VGT Mutant
GUI have the desired behavior. This implies that the expected
action of a button depends on the change made by the mutant
as intentional or unintentional.
B. Android Manifest Mutation Operators
The Android Manifest file stores essential information
regarding an Android app. Its purpose is to define the
organization and interaction between the Android main
application components and their attributes. In addition, any
permission required by the device such as Wi-Fi, Bluetooth
or location services is also specified in the Manifest file [24].
Fig. 4 shows an excerpt of the Alarm Klock Manifest file. It
begins with the root manifest tag and its attributes. The
manifest tag contains the uses-permission tag that specifies
the permissions needed by the Alarm Klock app. For
example, WAKE_LOCK keeps the screen turned on while
READ_EXTERNAL_STORAGE allows the application to
read from an external storage such as an SD card. Further, the
manifest tag also includes alarm clock activity services, such
as alarm notification and receivers such as Alarm trigger.
Fig. 4 Excerpt of Alarm Klock Manifest file
104
Authorized licensed use limited to: University of the Cumberlands. Downloaded on August 09,2021 at 17:58:15 UTC from IEEE Xplore. Restrictions apply.
Fig. 3c BWS Mutant Fig.3d TWA Mutant
As mentioned in the Related Work Section, this
configuration is regulated by using intent filters on services
and receivers. A service is a long-running operation that is
performed in the background of the Android operating
system. An example of a service is a user playing music while
browsing other applications. In the other hand, a broadcast
receiver is usually sent by the Android operating system and
received by the application. An example of a broadcast
receiver is the arrival of a new text message. Without an
appropriate configuration of the services and receivers in the
Android manifest file, the application might have issues
starting and using services such as connecting to the Internet
and accessing the user’s location with the Global Positioning
System (GPS). Therefore, we introduce the following
mutation operators that mutate the syntax of intent filters on
services and receivers specified in Android manifest files.
• Change Intent Filter Receiver (RFC): RFC
changes the intent filter action of a receiver into any
other action that is not defined in the intent filter.
Fig. 5 illustrates an example of an RFC mutant
where the action name of the intent filter is
modified. In this example the type of intent in the
original file is “App widget update”, which is sent
when the application widget needs to be updated.
The mutant modifies the type of intent using any
other type of intent. For this particular case, it is
modified with “App widget deleted” which deletes
the application widget. An application widget in
Android is the miniature version of an Android
application. It is usually placed in the home screen
and includes limited functionality.
• Change Intent Filter Service (SFC): Similar to
RFC, this operator modifies the intent filter action
of a service with another action that is not present in
the intent filter.

Fig. 5 Examples of RFC Mutants
V. EVALUATION
A. Subjects
In order to evaluate our proposed approach, we select five
Android apps from the Google Play Marketplace: Calculator
for Android [27], Forecastie [28], Antenna Pod [29], Alarm
Klock [30] and Tipster [31]. The source code of all five apps
can be found on Github.
Calculator for Android and Forecastie provide basic
operations with some advanced mathematical functions and
weather condition updates respectively. Antenna Pod is a
podcast manager and player that has instant access to
hundreds of free and paid podcasts, from independent
podcasters to big stations such as CNN (Cables News
Network) and BBC (British Broadcast Corporation). In
addition, the user can import and export the sources in a
simple way using RSS (Really Simple Syndication) or Itunes.
It has twenty-two activities along with their respective layout
files and one Android manifest file.
In addition, we select two Android apps that are used in
other research. Alarm Klock, which is a typical alarm clock
app that includes some advanced features such as the use of
mp3/media library songs as alarms. It has two activities, one
service, two receivers, one Android manifest file and nine
layout files. Tipster is an app used to calculate tips and split
payments or bills among several customers. It includes one
activity, one Android manifest file and one layout file.
B. Experiments
For our experiments, we use the main XML layout files of
each app along with their Android manifest file. Additional
layout files are selected based on the type and number of
elements included in their GUI. For example, to test the
Alarm Klock app, we select the notification layout file which
includes a good number of desired elements and can
potentially generate a large set of mutants. We ignore layout
files such as slider, time_picker and range_bar that do not
include any of the desired GUI elements that our mutation
105
Authorized licensed use limited to: University of the Cumberlands. Downloaded on August 09,2021 at 17:58:15 UTC from IEEE Xplore. Restrictions apply.
operators target. For evaluation, we use three test suites that
are built using different testing strategies: random testing,
adhoc testing, and black box testing.
For our first set of experiments, we apply random testing
using Monkey tool from the Android SDK on the five android
apps. Each experiment consists of 5,000 pseudo random
events such as clicks, touches or system-level events. For
each run we specify a seed value. In addition, we add a delay
of 500ms between events to guarantee that the screen
rendering is complete before the next event is executed. For
the second set of experiments, we apply Adhoc testing. An
experienced Android tester manually experiments with the
apps for a considerable amount of time. The tester was
encouraged to test all the features of every app as much as
possible. We recorded the tester’s input with the Robotium
Recorder tool. For the third set of experiments we use black
box testing focusing on testing the requirements,
functionalities, and usability of the app through the elements
of the applications’ user interface.
We check mutant equivalency manually by hand. For GUI
testing, an equivalent mutant is a mutant that generates an
identical GUI in contrast to the original GUI. For
configuration testing, an equivalent mutant is one that does
not make any changes to the original functionality of the app.
The only mutation operator that generated equivalent mutants
was VGT. This mutant changes the arrangement of elements
inside a view group layout. An example of such a case is
when the original code uses Linear Layout and the mutated
code uses instead a Table Layout. Both layouts share similar
attributes since Table Layout extends Linear Layout. The rest
of the GUI mutation operators did not generate any
equivalent mutant. This means that the GUI operators can
generate a representative and useful set of mutants.
In addition, none of the configuration mutation operators
generated any equivalent mutant. This indicates that all the
permissions, services and receivers specified in the Android
manifest file are necessary for the functionality of each app.
We also notice that some mutants did not allow the app to be
started and forced a sudden close. We named these invalid
mutants “stillborn”. The only mutation operator that
generated stillborn mutants was the VGT operator. An
example of such a case is when the original layout is a
common layout such as Linear Layout and the mutated layout
is of the kind that uses an Adapter such as List View. This
caused a syntactic error and was directly detected by the
compiler.
Table II lists the apps and the respective files that are tested.
It also shows the number of equivalent, stillborn and total
number of mutants generated. Edroid generated a total of
2035 mutants from the nineteen files tested using the fourteen
mutation operators. The number of Android mutants per file
ranged from 7 mutants in the Android manifest file of Alarm
Klock to 1837 mutants in the XML file of the Calculator app.
 TABLE II. Mutants generated
App Name File
Calculator calculator.xml
Forecastie AndroidManifest.xml
Antenna Pod addfeed.xml
feedinfo.xml
AndroidManifest.xml
Alarm Klock notification.xml
AndroidManifest.xml
Tipster main.xml
Total
The Android manifest files for the Calculator and Tipster app
did not have any mutants generated therefore we did not
include them in the table. Although the Calculator app
consisted of only one file to be tested, our system generated
the largest set of mutants among the rest of the apps. This is
because the Calculator GUI has a group of thirty-four
elements that includes buttons and text fields.
To apply mutation testing and to obtain the mutation
scores for the different testing techniques of every Android
app we used a Macbook with 2.7 Ghz Intel i5 processor and
a Samsung Galaxy Tab 4. The average compilation time for
each mutant was around ten seconds.
C. Results and Discussion
The results of our experiments are shown in Table III and
Table IV. Table III summarizes the results obtained for each
mutation operator among the five apps tested. Table IV shows
the results generated for each Android app.
The configuration operators were only applied to the
Android manifest files of three apps: Forecastie, Antenna Pod
and Alarm Klock. Seven of the fifteen APD mutants
generated were killed by the Android monkey tool. Two of
the five APD mutants of Forecastie app were stillborn. This
is due to the fact that Forecastie requested two permissions
for its initialization (Internet, and access_network_state). The
APD mutant that deleted the BOOT_COMPLETED
permission was killed in all three testing approaches. This
means that for Android apps, it is always necessary to receive
notifications of the Android Device system, such as the Wi-
Fi has been disconnected. Table III shows the mutation score
of all configuration operators: 27% for Random testing, 38%
for Adhoc testing and 73% for Black Box testing. For this
study, the mutation score is calculated by the formula: D / (N-
E-S), where D is the number of mutants killed, N is the total
number of mutants, E is the number of equivalent mutants
and S is the count of stillborn mutants. It can be seen from
Table IV that none of the SFC and RFC mutants were killed
using Android monkey tool. We believe the reason for this is
that most of the receivers and services communicate with the
106
Authorized licensed use limited to: University of the Cumberlands. Downloaded on August 09,2021 at 17:58:15 UTC from IEEE Xplore. Restrictions apply.
# Mutants # Equiv Mut # Stillborn
1,837 1 14
9 0 2
55 1 2
30 3 6
12 0 0
39 3 6
7 0 0
46 0 6
2,035 8 36
widget version of the app. Since a widget is a simple
application extension and is not an activity, it cannot be tested
by the Android monkey tool. On the other hand, some of the
mutants were killed using Adhoc testing and nearly all of
them using black box testing. In order to achieve a 100%
mutation score for these mutation operators, it is necessary to
test each service and broadcast receiver at least once.
The GUI mutation operators were only applied in the
XML Layout files. These operators are divided in three
categories: 1) ones that mutate a button such as BWI, BWA,
BWX, BWP, BWD, BWS; 2) ones that mutate a text field
such as TWI and TWD; 3) ones that mutate a group of
elements such as VGT and VGI. Table III displays the
mutation operators of all GUI mutants: 26% for Random
testing, 65% for Adhoc testing and 100% for black-box
testing. Random testing performed poorly to reveal faults in
the GUI and Adhoc testing made an acceptable performance.
On the other hand, black-box testing reached a perfect
mutation score. For this to occur, the tester needs to check for
the correct location, desired appearance and desired visibility
of every element in the GUI structure of the Android app.
Table III shows a mutation score of 0% for the BWI, TWI,
BWA and TWA operators using Random testing and Adhoc
testing. BWI and TWI operators mutate the visibility of an
element such as removing it from the GUI but leaving the
space it occupies. BWA and TWA operators mutate the
appearance of an element such as the change of its color. The
0% mutation score is due to the fact that Random testing and
Adhoc testing approaches do not check for the desired
visibility and desired appearance of an element in the GUI. In
contrast, black-box testing examines the desired visibility and
appearance of an element in the GUI at least once, leading to
a perfect mutation score for these operators. Table IV shows
the total mutation score of all the applications tested: 21% for
random testing, 56% for Adhoc testing and 99% for black-
box testing. All XML layout files have a mutation score of
100% using black-box testing and the configuration files have
an overall better performance using this testing approach. In
addition, it can be seen from Table IV that Adhoc testing
outperformed Random testing in revealing faults in all the
files of the five applications.
 TABLE III. Results of each mutation operator

Random Adhoc Black-Box

Testing Testing Testing
Operator Total Equivalent Stillborn Killed MS Killed MS Killed MS
Mutants Mutants Mutants Mutants Mutants Mutants
Configuration Operators
APD 15 0 2 7 0.54 8 0.62 10 0.76
SFC 3 0 0 0 0 1 0.33 2 0.66
RFC 10 0 0 0 0 1 0.10 7 0.70
TOTAL 28 0 2 7 0.27 10 0.38 19 0.73
GUI Operators
BWI 46 0 0 0 0 0 0 46 1
TWI 8 0 0 0 0 0 0 8 1
VGI 15 0 0 3 0.20 6 0.40 15 1
VGT 90 8 34 17 0.35 29 0.60 48 1
BWA 46 0 0 0 0 0 0 46 1
TWA 8 0 0 0 0 0 0 8 1
BWX 580 0 0 155 0.27 388 0.67 580 1
BWP 580 0 0 173 0.30 426 0.73 580 1
BWD 46 0 0 12 0.26 31 0.67 46 1
BWS 580 0 0 148 0.25 409 0.70 580 1
TWD 8 0 0 5 0.62 6 0.75 8 1
TOTAL 2007 8 34 513 0.26 1295 0.65 1965 1
TABLE IV. Results of each testing approach

Total Random AdHoc Black-Box

Mutants Testing Testing Testing
App Name File Killed MS Killed MS Killed MS
Mutants Mutants Mutants
Calculator calculator.xml 1837 387 0.21 1021 0.56 1822 1
Forecastie AndroidManifest.xml 9 2 0.28 3 0.42 6 0.86
Antenna addfeed.xml 55 4 0.08 29 0.55 52 1
Pod feedinfo.xml 30 4 0.19 12 0.57 21 1
AndroidManifest.xml 12 4 0.33 4 0.33 9 0.75
Alarm notification.xml 39 5 0.17 16 0.53 30 1
Klock AndroidManifest.xml 7 1 0.14 3 0.42 5 0.71
Tipster main.xml 46 9 0.22 27 0.675 40 1
TOTAL 2035 416 0.21 1115 0.56 1985 0.99

code and configuration files. Therefore, Edroid makes this

VI. CONCLUSIONS
We proposed in this work Edroid. A set of fourteen
Android mutation operators, of which ten were newly
introduced. These operators modified the functionality, the
GUI and the configuration of Android apps. The generated
mutants were designed with a central goal of improving the
quality of Android application development through a
rigorous evaluation of the test cases used. The focus was on
mutating the GUI and XML schemas of an app. Android
mutation testing is an exhaustive process and can become
complex when testing larger apps with more files, lines of
process feasible and can ease and accelerate the mutation
testing process.
We applied mutation testing on five real world Android
apps. The results showed the effectiveness of our proposed
mutation technique in evaluating a set of the tests cases. We
found out that all three testing techniques need more test
cases to improve mutation coverage. It was revealed that the
mutants proposed can be killed using test cases that: check
the expected output of each element in the screen at least
once, check if the elements are shown, check the correct

107

Authorized licensed use limited to: University of the Cumberlands. Downloaded on August 09,2021 at 17:58:15 UTC from IEEE Xplore. Restrictions apply.
location of each element, check for the desired appearance of
every element, check the functionality of each service and
broadcast receiver at least once.
On the other hand, it was revealed that Random testing with
Android monkey tool was not as effective to reveal faults on
Android apps GUI or its configuration. However, Adhoc
testing could be effective to reveal certain faults in the
functionality of the app if the user tested the proper
combinations of buttons. It was revealed that Random testing
had the lowest percentage of mutation score among all the
testing approaches applied. Furthermore, it was shown that
black-box testing had a perfect mutation score for all GUI
mutation operators.
For future work, we intend to diminish some of the
Edroid’s limitations by designing a new module that accepts
test cases as input from the user; and allows the user to
execute the test cases on mutants to determine the mutation
score. We also propose to extend Edroid to include traditional
Java mutation operators and new Android mutation operators.
Furthermore, we realize that additional Android mutation
operators are needed to achieve a better accurate result for
testing the quality of an android app. We intend to add
mutation operators that can mutate exclusive features of the
mobile device such as the orientation and GUI elements that
were not targeted by this study such as checkboxes and radio-
buttons. In addition, we plan to evaluate Edroid on more
testing strategies.
VII. REFERENCES
[1] R. Lipton, “Fault diagnosis of computer programs,” Student Report,
Carnegie Mellon University,1971.
[2] J. C. Maldonado, M. E. Delam\aro, S. C. Fabbri, A. da Silva Simao, T.
Sugeta, A. M. R. Vincenzi, and P. C. Masiero. Proteum: A family of
tools to support specification and program testing based on mutation.
In Mutation testing for the new century. 2001.
[3] Konrad Halas, “Mutpy 0.4.0 mutation testing tool for python 3.x,”
https://pypi.python.org/pypi/MutPy/0.4.0, last access Feb 2017
[4] M. Schirp, “Mutation testing tool for Ruby,” Online, 2012,
https://github.com/mbj/mutant, last access Feb 2017.
[5] Y.-S. Ma, A.J. Offutt, Y.-R. Kwon, "MuJava: An Automated Class
Mutation System", Software Testing Verification and Reliability, vol.
15, no. 2, pp. 97-133, June 2005.
[6] Ovum: App Revenue to Double by 2020, Outpacing Download
Growth, 2016, https://www.ovum.com/press_releases/ovum-app-
revenue-to-double-by-2020-outpacing-download-growth/, last access
September 2016.
[7] Agile Methodology in Mobile Application Development, 2017,
https://project-management.com/agile-methodology-in-mobile-
application-development/, last access February 2017
[8] Google Play stats, http://www.appbrain.com/stats last access
September 2016.
[9] Testing Support Library. https://developer.android.com/topic/
libraries/testing-support-library/index.html, last access December
2016
[10] Y.-S. Ma, J. Offutt, " Description of muJava’s Method-level Mutation
Operators", November 2005, https://cs.gmu.edu/~offutt/mujava/
mutopsMethod.pdf.
[11] L. Deng, N. Mirzaei, P. Ammann, and J. Offutt, “Mutation operators
for testing Android apps”, Information and Software Technology, 81,
pp.154-168, January 2017.
108
Authorized licensed use limited to: University of the Cumberlands. Downloaded on August 09,2021 at 17:58:15 UTC from IEEE Xplore. Restrictions apply.
[12] A. Avancini and M. Ceccato, "Security testing of the communication
among Android applications," Automation of Software Test (AST),
2013 8th International Workshop on, San Francisco, CA, 2013, pp. 57-
63.
[13] Jian Bing Li and J. Miller, "Testing the semantics of W3C XML
schema," 29th Annual International Computer Software and
Applications Conference (COMPSAC'05), 2005, pp. 443-448 Vol. 2.
[14] L. Franzotte and S. R. Vergilio, “Applying Mutation Testing in XML
Schemas,” in SEKE, 2006, pp. 511–516
[15] E. Alegroth, Z. Gao, R. Oliveira and A. Memon, "Conceptualization
and Evaluation of Component-Based Testing Unified with Visual GUI
Testing: An Empirical Study," 2015 IEEE 8th International Conference
on Software Testing, Verification and Validation (ICST), Graz, 2015,
pp. 1-10.
[16] R. A. P. Oliveira, E. Alégroth, Z. Gao and A. Memon, "Definition and
evaluation of mutation operators for GUI-level mutation analysis,"
Software Testing, Verification and Validation Workshops (ICSTW),
2015 IEEE Eighth International Conference on, Graz, 2015, pp. 1-10.
[17] I. M. Alsmadi, "Using Mutation to Enhance GUI Testing Coverage,"
in IEEE Software, vol. 30, no. 1, pp. 67-73, Jan.-Feb. 2013.
[18] Android developers guide, https://developer.android.com/studio/test/
espresso-test-recorder.html, last access February 2016.
[19] JUnit. Last access Februrary 2017. http://junit.org/junit5/
[20] Robotium, https://github.com/RobotiumTech/robotium, last access
February 2017.
[21] A. T. Acree, T. A. Budd, R. A. DeMillo, R. J. Lipton, and F. G.
Sayward. Mutation analysis. Technical report GIT-ICS-79/08, School
of Information and Computer Science, Georgia Institute of
Technology, Atlanta GA, September 1979.
[22] D. M. S. Andre. Pilot mutation system (PIMS) user's manual. Technical
report GIT-ICS-79/04, Georgia Institute of Technology, April 1979.
[23] B. N. Nguyen, B. Robbins, I. Banerjee, and A. Memon, “Guitar: an
innovative tool for automated testing of gui-driven software,”
Automated Software Engineering, vol. 21, no. 1, pp. 65–105, 2014
[24] Projects Overview. Last access February 2017.
https://developer.android.com/studio/projects/index.html
[25] Android App Manifest, 2017. Last access February 2017.
https://developer.android.com/guide/topics/manifest/manifest-
intro.html
[26] Android – UI Layouts. Last access February 2017.
https://www.tutorialspoint.com/android/android_user_interface_layou
ts.html
[27] Calculator for Android app. Last access December 2016.
https://github.com/CMDann/Simple-Android-Calculator
[28] Forecastie Weather app. Last access December 2016.
https://github.com/martykan/forecastie
[29] Antenna Pod app. Last access December 2016.
https://github.com/AntennaPod/AntennaPod
[30] Alarm Klock app. Last access December 2016.
https://github.com/kraigs-
android/kraigsandroid/tree/master/android/alarmclock
[31] Tipster app. Last access December 2016.
https://github.com/IanDarwin/Android-Cookbook-
Examples/tree/master/Tipster
[32] Kevin Moran, Michele Tufano, Carlos Bernal-Cárdenas, Mario
Linares-Vásquez, Gabriele Bavota, Christopher Vendome,
Massimiliano DiPenta, and Denys Poshyvanyk, “MDroid+: A
Mutation Testing Framework for Android”, in Proceedings of the 40th
IEEE/ACM International Conference on Software Engineering
(ICSE'18), Formal Research Demonstrations Track, Gothenburg,
Sweden, May 27-June 3, 2018
[33] M. Linares-Vásquez, G. Bavota, M. Tufano, K. Moran, M. Di Penta,
C. Vendome, C. Bernal-Cárdenas, and D. Poshyvanyk, “Enabling
mutation testing for android apps,” in Proceedings of the 2017 11th
Joint Meeting on Foundations of Software Engineering, ser. ESEC/FSE
2017. New York, NY, USA: ACM, 2017, pp. 233–244. [Online].
Available: http://doi.acm.org/10.1145/3106237.3106275