The Magazine for Agile Developers and Agile Testers

April 2011

www.agilerecord.com free digital version made in Germany ISSN 2191-1320 issue 6

© iStockphoto.com/GarySandyWales

Test Planning and Execution in a Mo-
bile Game Development Project using
SCRUM
by José Carréra Alvares Neto
The information technology industry increasingly realizes the im-
portance of conducting, in a careful and efficient manner, verifi-
cation and validation activities, which includes software testing.
Here the testing individuals, along with the rest of the team, work
to assure that the developed software meets all clients’ needs
and is of a high quality standard. To achieve this goal, an effec-
tive test plan is indispensable. From the beginning of the project,
a test engineer should be present, because this allows us to plan
ahead and to find and fix defects as soon as possible in the de-
velopment life cycle. After all, as mentioned in testing literature,
the sooner defects are found, the lower will be the costs to fix
them and the higher is the probability that their correction won’t
cause new bugs (Glenford J. Myers, “The art of software testing”).
This article, describes how testing activities were performed in
a mobile game development project using SCRUM as the man-
agement process. It will describe in detail the testing strategies
used, along with the best practices identified and the learned
lessons. The main goal of the article is to assist other test en-
gineers who are starting in game development projects, so that
they can easily and rapidly adapt to the differences compared
to standard software development projects. This will also allow
them to contribute to the creation of new and even more effec-
tive testing techniques.
The project
All the techniques and learned lessons described in this article
were experienced during a project developed at C.E.S.A.R. (Re-
cife’s Center of Advanced Studies and Systems) from December
2007 to March 2008.
Since the project included, amongst others, elements like short
duration, a small team, frequent client involvement, and con-
stant requirement changes, it was decided to apply SCRUM and
an agile development methodology to help manage all activities
during the project. In accordance with SCRUM, all tasks to be de-
veloped were listed as backlog items (BLI). These were elected to
26 www.agilerecord.com
© iStockphoto.com/uriy2007
be developed in short development cycles (“sprints”), whereby at
the end of each sprint a new version of the product was released
to the client. In our project, each sprint lasted 10 work days, and
the items to be developed were chosen by the team during sprint
planning meetings.
During the sprint planning meeting, all team members had to pri-
oritize all BLIs, in order to help decide which tasks had to be de-
veloped during the following sprint. From a test engineer’s point
of view, the approach was to always try to anticipate the features
that appeared to be critical for system behavior and for meeting
the client’s needs. Prioritization could be assigned due to com-
plexity or importance; the intention was to prevent bugs relating
to these features from being found late in the project life cycle.
Testing strategy
Planning – First sprint
Testing activities started before the end of the project’s first
sprint with the arrival of a test engineer. As a first task, a com-
plete analysis of the Basic Game Design Specification (BGDS)
was made. This document summarizes all basic game features.
After evaluating the BGDS and all client’s requests and needs,
a simple test strategy was defined which involved documenting
manual test cases as soon as the sprint started using a simple
priority scheme based on the complexity of the selected story and
the implementation order. At this initial stage we also identified
and solved all test environment needs, like available hardware,
SIM cards, bug tracking software, etc. Finally, a set of general
test cases made available by the client were also evaluated prior
to starting test execution.
Test case design
One of the initially defined constraints to the project was that all
documented BLIs should be covered by test cases, and the BGDS
were considered as the test oracle. Based on project characteris-
tics like short duration, scarce BLI documentation, and frequent
changes, we decided to design test cases in a more general way
with a focus on testing the game’s basic functionalities for each
BLI. This resulted in a set of test cases similar to those used for
“sanity” tests. This way we expected to maximize the time spent
on test execution and to avoid spending excessive time on docu-
mentation.
Test execution
The test specification consisted of a spreadsheet with a set of
test cases sent by the customer and a group of specific scenar-
ios designed specifically for the game by the test engineers., A
MANTIS bug tracker (http://www.mantisbt.org/) was used as the
defect management tool.
During the execution phase, the following activities were planned
to be executed in each sprint: The main focus was the execution
of the exploratory tests as features were released throughout the
sprint, along with the execution of the client set of test cases and
the game’s specific group of test cases. The use of exploratory
testing is generally encouraged for projects with characteristics
similar to ours, and when executed by experienced test engi-
neers, such”exploratory tests can be much more efficient than
the tests performed following scripted test cases” (James Bach).
During the course of each sprint, an important task performed
by the test engineer was to effectively monitor the progress of
all developers’ activities on the current sprint. This was done in
order to define the best time to request the release of intermedi-
ate versions of the game for component testing and also to avoid
defects or change requests being raised for features that had not
been fully released by the development team. This monitoring of
activities was made easier by the SCRUM methodology where,
during the daily meetings, we could easily follow project activities
through the burndown graph.
As previously mentioned, the decision to prioritize the explora-
tory tests was made due to the project’s main characteristics,
such as a lack of a extensive documentation at the beginning of
the project, and frequent changes of client’s needs and require-
ments.
Formal tests
The complete set of test cases consisted of the client’s standard
test cases together with game- specific test cases which came to
a total of around 15O tests. However, not all specific tests were
executed in the initial sprints, since most of the features were
not yet developed. A complete test execution cycle would take
an average of three days. During the rest of the sprint other test-
ing activities like test design and maintenance were performed
along with exploratory testing and change request validation.
The client’s set of test cases mainly focused on assuring that
the company’s standards were being followed by our team. This
concerned features like key mapping, performance, interaction
and user interface. Taking this into consideration, it was manda-
tory to run these tests for each sprint in order to assure that the
developed game suited all clients’ demands and, above all, that
they wouldn’t interfere with the mobile phone’s basic features.
At the end of each sprint an intermediate version of the game
was released to the client, who could analyze the delivery and
provide feedback to our team. This usually involved aspects like
game play, game design and defects. Through an analysis of this
feedback we could figure out which areas were more relevant to
our client, adjust our test strategy accordingly, and then focus on
the missed defects in the next sprint.
Exploratory tests
Exploratory tests, which were chosen as our main test execu-
tion strategy due to the project profile, began early in the initial
sprints. In a traditional approach, informal test charters were pre-
pared focusing on a specific area or BLI to be tested. During the
course of the project, as the Game Design Specification became
more mature, we started using two new approaches for running
the exploratory tests, which will be described below.
Test case based
In this approach the actual test cases were used as the focus
area for each exploratory test, whereby the steps of the test case
were run in an unusual way. The tester is encouraged to diverge
as much as possible from the specified test steps, and to try and
think of alternative paths that could be taken instead of the one
suggested by the test case. The main idea is to use the existing
test cases just as a reference in order to cover all the features
of the application and to leave it to the test engineer to evalu-
ate relevance of the features to the system. By doing this, we
encourage the test engineer to think creatively to find new ways,
or ways not previously foreseen, to break the software. This ap-
proach achieved very good results and certainly increased the
total number of relevant defects found.
If this approach is to achieve a high degree of success, it is very
important for the test engineer to know all existing features of
the system, the market and the client’s expectations. He needs
to fully concentrate on his work in order to notice details that may
have escaped before. It is also important that the engineer can
work in a comfortable environment during test execution without
being constantly interrupted, thus allowing an efficient analysis
of the existing test cases and unexplored possibilities.
GDS scanning
A different approach, which we used in the later sprints, was
based on the execution of the exploratory tests through a com-
plete scan of the GDS. This approach couldn’t be applied fully in
the initial sprints, because only the BGDS was available, which
didn’t contain enough information to allow a more detailed ex-
ecution.
For this technique to be applied successfully, the test engineer
needs to have already read and understood the document,, and
there should be no open questions. The main idea of this ap-
proach is to make sure that every description included in the
GDS is correctly coded in the game. By simultaneously analyz-
ing the document and exploring the game, it becomes visible if
any important scenario isn’t well described in the text. With this
approach we can combine the benefits of static testing of docu-
www.agilerecord.com 27
 mentation with the advantages of exploratory testing of the ap-
plication. Any gaps between game and documentation can easily
be found and the game designer can clarify the type of any bugs
found.
Registered defects
During the course of the project, 122 change requests (CRs)
were registered, which were simply classified for this article into
four categories, according to the type of defect, as follows: func-
tional, user interface, art, and sound effects. Later on in this
article is described how each of these areas presented unique
important characteristics. In addition to these descriptions, two
graphs will indicate the amount of CRs per type and the severity
of the registered defects.
The severity of each bug was classified as follows: (i) minor (for
defects that do not block the game’s correct behavior, e.g., the
phone doesn’t vibrate when a new level is unlocked); (ii) normal
(for defects that affect important elements of the game, but
do not block the game’s execution, e.g., a special effect lasts
shorter than the value specified in the GDS); (iii) major (for de-
fects that directly affect gameplay or user satisfaction, that have
a direct impact on the game’s level design, and that prohibit the
player from proceeding, e.g., scenarios where the game freezes);
(iv) critical (for defects similar to major defects, but with an even
higher impact on the game’s correct operation, e.g., level is not
unlocked after completing tasks or phone doesn’t receive calls
while game is started).
Functional
The CRs from this group are related to inconsistencies regarding
the game’s designed rules and logic. This type of defect, even
those with lower severity, have a direct impact on the game’s
success, because they usually get in the way of a smooth un-
derstanding of the game’s objectives. They may even block the
player from overcoming the challenges presented, turning the
game into an impossible mission.
Scenarios like score limits, lousy player and excellent player
approach, and other possible features that involve testing the
game’s features and limits, were tests that usually detected this
kind of defect. These scenarios were not always well described in
the GDS, and developers didn’t take them into consideration or
unit test them properly.
User interface
Interface defects were connected to failures during display of
texts, opening of pop-up windows, screen limits, etc. These is-
sues could be easily noticed by any player, and would definitely
give the idea of a poorly developed game, without care for details.
These defects, although easily detected, frequently escape the
developing team and even the initial test cycles. This can happen
because developers usually run tests using a simulator and not a
real device. It is part of the test engineer’s job to assure that all
game screens and texts are checked for the supported devices.
28 www.agilerecord.com
Art
All change requests of the “art” category are related to features
like image rendering, lightening, and any other aspects related to
the elements produced by the art team (although some of them
may also be performed by the development team).
This type of defect varies widely from huge perceptible failures,
which can be easily noticed, to specific scenarios that are caused
only by a defined sequence of actions. This kind of defect can be
found not only by focusing on this aspect during testing, but also
while running any other type of test. All that is needed is a highly
alert tester who pays attention to details.
It is highly important for the test engineer, especially if he is not
experienced with this kind of defect, to interact with the art team
to clarify the questions related to possible defects, and in doing
so beginning to understand the features, their limits and their
solutions.
Sound effects
We also assigned defects to a sound effect category, because
it turned out to be a key area where initially we did not expect
to find a relevant amount of bugs. However, testing showed that
this assumption was wrong. Several defects were found which
demanded considerable work from the development team to get
them fixed. One aspect observed, the concurrent events execu-
tion, caused complications in the game’s general behavior. This
concerns scenarios such as executing a sound while another one
is already playing, user-initiated pauses of the game, disabling
and enabling the sound. In addition, severe performance prob-
lems could result from some of the game’s sound events.
Throughout the course of the project, this kind of the defect,
which initially was underestimated, gained higher priority and at-
tention. We found that in this area we had a higher probability of
causing other defects while fixing one.
At first, test execution for this aspect was impacted by the quality
of the available hardware, which presented a bad sound quality.
Later on, with the arrival of new hardware, tests could be easily
executed and showed better results. Therefore it is important for
the testing team to ensure the availability of the correct hardware
at the beginning of the project.
Figure 1 – Amount of defects registered by type

Figure 2 – Severity of reported defects
Best practices
Below we describe some of the practices that were applied in our
project and that presented good results. These could therefore
even be applied to projects in different areas.
BLI defect tracking
Every time a new change request was submitted during the
course of the project, along with its short description, a tag was
added to identify to which BLI it was related. At the start, this was
only used to help the Configuration Control Board (CCB) with the
CR assignment, but later on it aided the team in evaluating which
BLIs presented more defects of higher severity.
On the basis of this information we could plan test execution fo-
cusing on two aspects: (i) validate if BLIs with few or no defects
were sufficiently tested; (ii) analyze BLIs that showed a greater
amount of defects, their characteristics and which test scenarios
could be re-tested or added to allow the discovery of new bugs.
Greater emphasis was applied to the second aspect, as de-
scribed by Myers “The probability of the existence of more errors
in a section of a program is proportional to the number of errors
already found on that section”.
This approach proved to be efficient as new errors were found.
Some adjustments were made to make this task faster. For ex-
ample, instead of adding the BLI identification in the description
header, we added a new text field to the defect management tool
so we could identify the BLI and later on easily identify the de-
fects found.
Greater level of detail in defect description
One of the most important activities of the test engineer is re-
cording the defects, found in a defect management tool. Never-
theless, some people didn’t perform this task as expected. Is-
sues were sometimes not completely described, making it harder
for managers and developers to understand the error. Later on
this generated several interruptions for the test engineer in order
to clarify the issue description or, even worse, to discard the de-
fect. Therefore, it is very important for the defects to be reported
in a detailed and didactic manner, making everyone’s job easier,
including other testers that might be involved in retesting after
the issue gets fixed.
To assist in this task, one real benefit is to add a recorded video
of the issue or, if that is not possible, at least a screenshot. If the
issue can’t be reproduced on the simulator, use a regular camera
to capture it. Just remember that this is not a rule. It’s up to you
to evaluate whether a CR should be improved by adding some
extra resource (after all, not all issues have a visual feedback).
Defect management tool
Another highly important asset to assist the test engineer’s work
is the defect management tool. In our project the open source
tool MANTIS was used. It provides the engineer with effective
control of the registered defects, allows trends analysis and,
most of all, enhances team communication.
Taking notes
Keeping an electronic notepad always open or even a piece of
paper and pen on your desk can really help during test activities.
With the time pressure and tight deadlines always present , it is
not always possible to test all the scenarios that we foresee dur-
ing testing. This may happen because of the need of keeping fo-
cused on the current test cycle or other tasks being performed. If
not written down somewhere, these other activities or even hints
observed during team meetings may get lost and never tested.
Making a habit of note-taking for any relevant information that
might help in future activities (for example, a new test scenario
or system characteristic, some user feedback, etc.), will help the
test engineer to avoid forgetting interesting investigations that
could be performed and assist in finding new bugs.
LEASSONS LEARNED
Throughout the project, many new experiences were faced and a
lot was learned by working in a project with very different charac-
teristics to those found in regular software development projects.
Test case design
After gaining some experience in test case design for games,
we observed that tests that were related to functionalities didn’t
need to be repeated on different game scenarios because the
error found applied to the whole application (e.g. pressing a spe-
cific key doesn’t produce the expected behavior). On the other
hand, when considering the user interface and art, this type of
test needed to be repeated for each scenario of the application,
since some errors can be found only in specific scenarios.
Using cheat codes
As the game evolved, a couple of cheat codes (coding that pro-
vided special game advantages that would not be available on
the final version, such as “invincibility”),were created to make
some tests easier to execute. for both developers and testers.
However, we need to be extremely careful when deciding to use
this type of assistance. If they are misused, it can lead to hidden
bugs or, conversely, bugs that only appear due to the existence
of the cheat code. An example that happened to us was a cheat
code that unlocked a game level, which blocked developers from
reproducing a bug reported by testers.
www.agilerecord.com 29
 If correctly used, cheat codes can increase the team’s perform-
ance and even help anticipating bugs.
Team communication
It is also important for the testing team to define specific time
intervals throughout the project where reports from test execu-
tion cycles will be sent for the rest of the team. This helps us to
make our work visible and understandable for the entire team.
Although SCRUM already makes tasks communication easier
among team members by applying the daily meetings and burn-
down graphs, we still need a more formal type of communication.
A recommended moment for these reports is at the end of each
sprint.
Activities followed up using the burndown graph
As test engineers we commonly need to assure that we are us-
ing the latest available versions for testing. During the course
of a sprint, the time for requesting new partial versions can be
determined through the daily meetings and the burndown graph,
where we can be aware of the stage of each BLI and set up with
the developers the number of features available for testing. The
test engineer needs to keep constantly in touch with the team
leader to assure that these intermediate versions get released
for component and exploratory testing.
BLIs changes closely followed
Generally on Agile projects, but especially on ours, a great
amount of change happened to the BLIs which directly impacts
on the previously designed test cases. Therefore, it is highly rel-
evant to stay tuned to changes caused by client’s feedbacks, us-
ability tests and reports, meetings and also technical limitations.
This follow-up is made easier when the Game Designer works
closely with the rest of the team and keeps everyone informed
when changes occur. This way, any questions related to any fea-
ture of the game can be easily discussed and clarified with the
Game Designer.
Informally reported defects don’t get fixed
Just like a tester forgets to test scenarios that he doesn’t docu-
ment, by informally reporting a defect to a team member, (devel-
oper, GD or artist), there is no guarantee of a fix. No matter what
the severity of the report is, the informal communication creates
a high risk that it doesn’t get fixed or, if fixed, that it may not be
validated.
Retest with different devices
Throughout the project several errors were found which were re-
lated to the game’s interaction with specific devices or external
applications such as phone calls or SMS. This kind of error caus-
es considerable effort for the development team to investigate
the issue and to find out the cause. However, this kind of error
can’t be solved by our team since it is not related to our game.
Therefore, it is a good approach to retest with different devices or
different games with similar characteristics. This extra informa-
tion won’t eliminate the CR, but developers can start analyzing
with this aspect in mind.
30 www.agilerecord.com
This approach can also be applied to different type of scenarios,
such as performance, sound effects, rendering and other fea-
tures which can be compared with similar games.
CONCLUSION
No matter what stage we are at in the development life cycle
or the experience level of the developing team, there are many
causes for software bugs. Most of them are not related to the
code itself, but to other problems, such as incomplete or unclear
requirements, hardware issues and integration. Therefore, con-
sidering the practices and lessons learned from this project, we
are convinced that software quality is a high priority for modern
software products, like mobile games, and that achieving this
shall be a common goal for the entire team with a clear division
of responsibilities. Making sure that there are no conflicts be-
tween developers, testers and other team members regarding
quality, everyone must work together to deliver a high quality
product. Only by placing priority on quality we will be able to de-
liver products that fully meet our clients’ needs and expectations.
> About the author
José Carréra
MSc, has been test engi-
neer at C.E.S.A.R. (Recife’s
Center for Advanced Stu-
dies and Systems) since
2006 and Professor of
Computer Science at FA-
TEC (Faculdade de Tec-
nologia de Pernambuco),
Brazil, since 2010. He ob-
tained his master degree
in software engineering (2009), graduated in computer
science (2007), and is a Certified Tester - Foundation
Level (CTFL) by the ISTQB (2009). His main research in-
terests include quality assurance, agile methodologies,
software engineering and performance testing.