2015 IEEE 2015 International Conference on Computer, Communication, and Control Technology (I4CT 2015), April 21 - 23 in
Imperial Kuching Hotel, Kuching, Sarawak, Malaysia
Exploratory Software Testing in Agile Project
Beni Suranto
Department of Informatics, Faculty of Industrial Technology
Universitas Islam Indonesia
Yogyakarta, Indonesia
beni.suranto@uii.ac.id
Abstract—Software testing is an important process in a software
development life-cycle as validation and verification mechanisms
to guarantee the quality of the intended software product.
Exploratory testing is a software testing where the testers may
interact with the system in whatever way they want and use the
information the system provides to react and generally explore
the system functionalities without restraint. Exploratory testing
allows the full power of the human brain to be brought to bear on
finding bugs and verifying functionality without preconceived
restrictions. This paper discuss about enacting exploratory
software testing in a software project which was developing
Student Marking System (SMS) in University XYZ.
Keywords-software testing; exploratory; agile
I. EXPLORATORY SOFTWARE TESTING
Software testing is an important process in a software
development life-cycle as a “validation and verification”
mechanism that has following goals:
• Validating whether the software is the right product
(i.e. meeting the applicable standards and customer
needs, whether it works as expected) [1]. A software
can be labeled as a right product when it provide the
functionalities needed by the customer, behave as the
customer’s expectation, and give high level usability
to help the customer solve their problems. Without
testing process it is impossible for developer and
customer to judge whether a software is a right
product or not. In the context of Student Mark System
(SMS) project, the developer team must perform the
testing process to validate the system whether it is be
able to help the stakeholders (i.e. teachers, assessment
coordinators, faculty staff) to perform all marking
stuff in an effective and efficient way or not.
• Verifying whether the software is right (i.e. matching
the requirement specification)[2]. Testing as a
verification mechanism is used to judge whether the
functionalities embedded in the software is developed
as specified in the user requirement specification. In
the SMS project, the stakeholders use every weekly
meeting to verify whether the demonstrated features
correctly represent the requirements specified in the
previous meeting or not.
• To discover faults, defects, errors, bugs in the
software where the behaviour of the software is
978-1-4799-7952-3/15/$31.00 ©2015 IEEE 280
incorrect, undesirable or does not conform to its
specification[3]. Testing also very important to detect
all possible undesirable system behaviour, such as
incorrect computations, system crashes, data
corruption, etc. Since the SMS deals with many
computations and processes of “very important” data
(i.e. student’s marks) , the developer team must
perform the testing process to ensure that the system
doesn’t bring any risk of unacceptable errors or bugs.
Exploratory software testing is a software testing process
with the scripts are removed entirely (or, the scripts’s rigidness
relaxed) [4]. In this method, testers may interact with the
system in whatever way they want and use the information the
system provides to react and generally explore the system’s
functionality without restraint. Advocates argue that
exploratory testing allows the full power of the human brain to
be brought to bear on finding bugs and verifying functionality
without preconceived restrictions [5].
As one of software testing methods, exploratory testing has
following goals [5]:
• To gain an understanding of how an application
works, what its interface looks like, and what
functionality it implements: In the SMS project, this
goal is achieved with perform two different testing
process. First, the developer team perform exploratory
testings in the group meetings. Second, the developer
team with the stakeholders together perform
exploratory testing in the weekly client meetings.
These two are complementary processes. Some bugs
that were not discovered in the first process may be
discovered in the second process. It is very often since
the developer team and the stakeholders sometimes
use different ways to explore the functionalities of the
system.
• To force the software to exhibit its capabilities: The
idea is to make the software work hard and to ask it
hard questions that put it through its paces. This may
or may not find bugs, but it will certainly provide
evidence that the software performs the function for
which it was designed and that it satisfies its
requirements.The main purpose of the SMS project is
to help the stakeholders perform all marking stuff
related processes in an effective and efficient way. It
has a lot of specific requirements since different
faculty might have different “ways” to do the marking
 process for their students. To deal with this issue,
some staff from different schools (i.e. School of
Education and School of Chemistry and Physics) were
involved in the exploratory testing in the weekly client
meeting sessions to help the developer team to verify
whether the system is satisfy their requirements or not.
• To find bugs: Exploring the edges of the application
and hitting potential soft spots is a specialty of
exploratory testing. The goal is purposeful which is
exploration to identify untested and historically buggy
functionality. Exploratory testing is well suited to be
used by developer team which is developing modern
web system using agile methods[3]. In an agile
project, development cycles are short, leaving little
time for formal script writing and maintenance. Also,
system features often evolve quickly, so minimizing
dependent artifacts (i.e. pre-prepared test cases) is a
desirable attribute. The SMS project is using agile
method with one week iteration process and each
week it is very often that the stakeholders ask for
changing some features the way they want.
The drawback to exploratory testing is that testers risk
wasting a great deal of time wandering around an application
looking for things to test and trying to find bugs[4]. The lack of
preparation, structure, and guidance can lead to many
unproductive hours and retesting the same functionality over
and over, particularly when multiple testers or test teams are
involved. This is where guidance comes into play. The
developer team of the SMS project uses a technique called
Session-Based Team Management (SBTM) to perform
exploratory testings during the system development life cycle.
about 60-90 minutes for a exploratory testing session
for each charter.
• Reviewable Result: takes the form of a session sheet, a
page of text that follows a formal structure. This
structure includes charter, coverage areas (not code
coverage; typically product areas, product elements,
quality criteria, or test techniques); start time, tester
name(s); time breakdown; data files; test notes; bugs
(where a “bug” is a problem that the tester reasonably
believe represents a threat to the value of the product);
issues (where an “issue” is a problem that threatens
the value of the testing process—missing information,
tools that are unavailable, expertise that might be
required, questions that the tester might develop
through the course of the session)
• Debriefing: a conversation between the tester who
performed the session and someone else—ideally a
test lead or a test manager, but perhaps simply another
tester. In the debrief, the session sheet is checked to
make sure that it’s readable and understandable; the
testers discuss the bugs and issues that were found; the
testers makes sure that the protocol is being followed;
and collaboration happen. The developer team of the
SMS project usually spent about 15-30 minutes to
perform debriefing for every exploratory testing
session.

II. SESSION-BASED TEST MANAGEMENT (SBTM)

SBTM which was invented by James and Jonathan Bach in
2000 is a well-known guidance that can be used in exploratory
testing method[6]. SBTM has four elements:
• Charter: a one- to three-sentence mission for a testing
session. The charter is designed to be open-ended and
inclusive, encouraging the tester to explore the
application and affording opportunities for variation.
Charters are not aimed to be comprehensive
descriptions of what should be done, but the total set
of charters for the entire project should include
everything that is reasonably testable. Figure 1 shows
two examples of charters that were used in
exploratory testing for the SMS project.
• Time Box: some period of time between 45 minutes
and 2 ¼ hours, where a short session is one hour (+/-
15 minutes), a long session is two, and a normal
session is 90 minutes. The intention here is to make
the session short enough for accurate reporting,
changes in plans (such as a session being impossible Fig. 1. Examples of charters in the exploratory testing for the SMS project
due to a broken build, or a session changing its charter
because of a new priority), but long enough to
perform appropriate setup, to get some good testing Figure 2 shows an example of reviewable result of
in, and to make debriefing efficient. Software exploratory testing for the SMS project.
engineers working on the SMS project usually spent

281

Fig. 2. An example of reviewable result in exploratory testing for the SMS
Project
III. OBJECTS IN EXPLORATORY SOFTWARE TESTING
A. User Input
A general definition of an input is “..a stimulus generated
from an environment that causes the system under test to
respond in some manner.”[3].
The key point of the definition above is that an input
originates from outside the system (e.g. given by the user) and
causes some code in the system to execute. Following are some
example of user inputs that were be considered in exploratory
testing for the SMS :
• Username and password input by the user in the login
page when the user want to login into the system.
• Every click done by the user in all clickable things in
every page (e.g. login button, SMS main menus, SMS
tab menus, course link in the course navigation page,
column header in the class list page, add/edit/delete
button in assessment configuration page)
• All data that the user must input to configure the
component marking for a particular class (i.e.
282
components/assessments/parts names and their
maximum mark)
• Actual marks input by the user using manual input or
using the “mark upload” feature.
• Every value that user select from all GUI input
elements in every page (e.g. dropdown list for grading
scheme, dropdown list for manual grade, dropdown
list for supplementary exam, dropdown list for
histogram data set, radio button for histogram bin-
interval)
B. State
In exploratory testing context, a state of software is defined
as “..a coordinate in the state space that contains exactly one
value for every internal data structure.”[5]
The state space is the cross-product of every internal
variable in the system. It contains large number of different
internal states that represent whether the system provide the
right or wrong response to a particular input.
Software state is a result of interaction between the
software itself with its history of receiving input and its
environment. The software’s state changes when inputs are
applied and stored internally. Exploratory testing aims to test
these changes. Is the software state updated properly
according to the inputs ? Does the state of the software cause
undesirable behavior ? Is there any inputs lead the software
getting into states it should never be in ?
Following are some examples of software states for the
SMS which were tested with exploratory testing method :
• Different types of user (e.g. admin, teacher) should
get different states of the main page.
• When the teacher upload new marks for a particular
assessment, the states of all student data in the
corresponding class should be updated.
• When the admin change the grading scheme option,
the states of all class in the corresponding term
should be updated (i.e. the final grades in the class
list shoud be reproduced based on the new grading
scheme)
• When the admin append the enrolment data with a
new feed, the states of all affected classes should be
updated (i.e. some new students will be included in
the classes and some existing students in the classes
will be labelled as “unenrolled”)
C. User Data
Current software generally interact with a large data store
using database management system and/or file management
system. The most challenging task in the user data testing
process is how the testers be able to create a database with
specific data as similar as possible to the type of data we expect
our users to have[7].
 Some important characteristics of real user database that • Network (localhost and network server)
must be considered by the testers in exploratory testing are :
From the results of the previous exploratory testing
• Since data is added and modified the real user processes, the developer team of the SMS project found that
database will evolve and it can grow very large. browser was the element which brought many bugs and issues
(i.e. compability problems).
• Very often that there are data relationships and
structures in the real user database that the testers
unable to infer. IV. CONCLUSION
Exploratory testing is one of software testing methods that
• The real user database often require a very big storage
use “explore and find” approach to verify the functionalities
space that is not available in the testing process.
and detect bugs in the system. This method is well-suited to be
Following are user data that were tested with exploratory adopted in agile projects since it doesn’t need a lot of time to
testing in the SMS project : create the test plans (compared to the script-based methods)
and also it encourages collaboration among software engineers
• Enrolment Data: This data is used to populate the in the developer team and between the developer team and the
database, especially for the tables related with the stakeholders of the system.
enrolment processes. The developer team of the SMS
project use anonymous files as enrolment feeds (given SBTM give well-designed guidance to be used in
by the stakeholders) to test following functionalities : exploratory software testing in order to manage the testing
process. Some important objects that must be considered in
o Upload enrolment data exploratory software testing are input, user data, software state,
o Append enrolment data and environment.
o Data reconciliation
REFERENCES
• Marks Data: This data contains marks for a particular
assessment(s) in a particular class. This data is [1] Sommerville, Ian 2007, Software Engineering 8th Edition, Addison-
uploaded using “upload mark” feature which is Wesley, New York.
available in the class list page. The developer team of [2] Ammann, Oaul 2008, Introduction to Software Testing, Cambridge
the SMS project use marks data to test following University Press, New York
functionalities : [3] R.D. Craig and S.P. Jaskiel, Systematic Software Testing, Boston:
Artech House Publishers, 2002.Reference 5
o Marks data preview [4] "Exploratory software testing." SciTech Book News Dec.
2009. Academic OneFile. Web. 25 Sept. 2012.
o Identifier and assessment columns mapping
[5] Bach, J. (2003). Exploratory Testing Explained.
o Class list data update (http://www.satisfice.com/articles/et-article.pdf)
[6] J. Bach, "Session-Based Test Management", STQE, vol. 2, no. 6, 2000.
The developer team of the SMS project use comma- [7] J. Bach, "Exploratory Testing", in The Testing Practitioner, Second ed.,
separated values (csv) files for both enrolment data and marks E. van Veenendaal Ed., Den Bosch: UTN Publishers, 2004, pp. 253-265.
data in the testing processes.

D. Environment
The definition of an environment is depends on the type of
the system being tested. There are some elements contruct an
environment of a particular system: the operating system which
the system is running, the various applications that might
interact with the system, file and setting that may affect the
system, the network that the system is connected, and so on
and so forth. In exploratory testing, the testers must consider
anything that can affect the behavior of the system as part of
the environment[7].
Following are some elements that were considered as parts
of the environment in exploratory testing for the SMS :
• Operating system (Windows and Linux)
• Browser (Internet Explorer, Google Chrome, Mozilla
Firefox, Safari, Opera)
• Hardware (different machines, different displays)

283