You are on page 1of 15

2

TestComplete
by SMARTBEAR
Introduction ............................................................................................ 4
User Expectations ................................................................................. 5
Methods of Testing for Video Games .................................................... 7
Black Box Testing .................................................................................. 8
White Box Testing .................................................................................. 8
Automated Testings Role in Video Game Development ....................... 9
Video Game Testing on a Mobile Platform .......................................... 11
Mobile ............................................................................................................. 11
Data ......................................................................................................... 11
Network Availability .................................................................................. 12
Gestures .................................................................................................. 12
GUI ........................................................................................................... 12
Portrait and Landscape Views ................................................................. 12
Processing Power .................................................................................... 12
Memory .................................................................................................... 13
Battery ...................................................................................................... 13
Fragmentation of devices ......................................................................... 13
Gaming ........................................................................................................... 14
Audio ........................................................................................................ 14
Graphics ................................................................................................... 14
Accessibility ............................................................................................. 14
Databases ................................................................................................ 14
Conclusion ........................................................................................... 15
Contents
3
TestComplete
by SMARTBEAR
Introducton
Weve all heard the stories of video game testersrecent
high school graduates sitting in a room, playing video
games for twelve hours straight, drinking Mountain Dew
and documenting any bugs they can fnd. Many kids think
of this as a dream job. But what they dont understand is
that its probably one of the most tedious, mind-numbing
jobs one might fall into. As a software tester, its not par-
ticularly the scene you want to think of when you imagine
your day-to-day work life, yet this is the culture of testing
video games. There isnt any test automation tool that can
help minimize the tedium. Its all manual user acceptance
testing and, the truth is, video game testing probably ruins
video games for most of the people who attempt this voca-
tion.
This eBook is about the exploding industry of mobile gam-
ing and the reality of testing video games for mobile de-
vices. If the nuances of mobile app testing and video game
testing werent bad enough, combine the two and you have
a perfect storm for any software tester. Even the most
seasoned software tester might be hesitant to approach
mobile gaming.
If you go into the Apple store or the Google Play store, ev-
ery day there is a new list of games that have come to the
market. A recent study by eMarketer showed that by 2015,
approximately 162 million people will be playing games
on mobile devices. To put that into perspective, it amounts
to about half of the entire US population playing mobile
games in some fashion. So, despite the fact that mobile
gaming can be a testing nightmare, the benefts of produc-
ing a hit game could outweigh the risks. The eMarketer
4
TestComplete
by SMARTBEAR
study also claims that user growth could push revenues for
the mobile gaming market to $1.78 billion in 2013. Clearly,
there is money to be made!
User Expectatons
So youve decided that developing a mobile game is a
great way to make money. Before jumping right into game
development, you must understand who your end us-
ers are and what genre and platform caters to the users
gaming needs. As with all software
development, the user will be your
best friend and worst enemy. By
listening to your users feedback on
design and mechanics you increase
the chances of being successful.
However, by ignoring the needs of
your users, you risk alienating them
and destroying your credibility as a
video game developer.
Here is an example:
The main point of a Diablo gamewhat users came to
love the Diablo games forwas the constant slaughtering
of monsters and demons with the sole purpose of level-
ing up, customizing your character, and fnding cool loot to
which you could sell off or trade with other players.
Yes, Diablo III had cool loot, the graphics were better,
performance was fne, but there was something very wrong
with this iteration of Diablo. With Diablo III, Blizzard intro-
duced the auction house as a way to make real life and
in-game money off of the rare loot a player could fnd, with
5
TestComplete
by SMARTBEAR
a small fraction of all real-money transactions going into
Blizzards pockets.
The issue here was that the gear a character needed to be
strong enough to progress further in the game became a
bottleneck when you had no choice but to use the auction
house to get the gear your character needed to progress.
Essentially, Blizzards attempt to add a feature that was
supposed to be optional became the sole basis of the
game. Users found themselves spending more time look-
ing for gear on the auction house instead of going out and
fghting monsters.
We are writing tell you about an important change to
Diablo III: were going to be removing the gold and real-
money auction house system from the game.

When we initially designed and implemented the auction
house system, the driving goal was to provide a convenient
and secure system for trades. But after much review and
player feedback, it became increasingly clear that despite
the benets of the AH system and the fact that many play-
ers around the world use it, it ultimately undermines Dia-
blos core game play: kill monsters to get cool loot.

Were working out the details of how the auction house
system will be shut down, but we wanted to share the news
as soon as we made the decision in order to give everyone
as much advance notice as possible. Please note that the
nal shutdown will occur onMarch 18, 2014. We will keep
everyone informed as we work through this process, but
feel free to check out our blog poston the subject, and stay
tuned toDiablo 3.comfor further details.
TheDiablo III team
6
TestComplete
by SMARTBEAR
The point is that you must be in touch with whom you are
creating the game for and the game must be tested not
only for bugs but for fundamental issues, such as un-
wanted features and/or requirements that are irrelevant to
stakeholders. Sometimes, simplicity is the best approach.
In the case of mobile games, you can always add features
later via patching once you have a better idea what your
users really want. However, this should not be an excuse
for releasing buggy software and patching later since this
will eliminate your chances at being reputable. Those bad
reviews dont go away after you release version 2.
Methods of Testng for Video Games
Youve built game requirements based on feedback from
all stakeholders. The development team is about to start
developing the game and the testing team is about to start
testing the developers work. Welcome to the world of
video game testing! Your users are going to pick apart your
game and tell you everything that is wrong with it, so test
for anything that would ruin the user experience.
Every gaming company tests their game differently, be-
cause there are so many different genres of games and
since software testing methods are always dependent on
the context of the project. Despite the differences in testing
methodologies from team to team there are two main cat-
egories of testing that always seem to be used for testing
games.
Black box testing Testing functionality without
checking the code
White box testing - Testing and examining code as
opposed to functionality
7
TestComplete
by SMARTBEAR
It is important to understand that these are generic test
strategies and not types of testing. For instance, func-
tional testing would be a specifc type of testing that falls
under the black box testing strategy. Likewise, unit testing
would fall under white box testing.
Black Box Testng
Black box testing is the method of choice for testing video
games. This testing strategy is done without access to the
code. It is done by the tester to interpret how a user would
play the video game in normal use. When you hear about
the scenarios and picture a group of recent high school
graduates in a room drinking Mountain Dew and playing
and testing video games, they are black box testing.
Not only is black box testing an easy way to emulate a
user, it is also a cheap way. Black box testers need mini-
mal technical knowledge, but good hand-eye coordination
from heavy gaming is highly sought after. You need testers
who can play a game all the way through quickly and then
play again going a different path through the game.
Black box testing is advantageous in that it can separate
the players perspective from that of the developers. How-
ever, since the tester does not see the code, if a defect is
found it is harder for the developer to pinpoint the segment
of code that is erroneous.
White Box Testng
White box testing is testing with access to the code and is
the technical side of video game testing. For example, unit
testing falls under this testing strategy. White box testing
8
TestComplete
by SMARTBEAR
is advantageous since its easier to see what inputs can
be effective in testing. Cleaning up code that could cause
bugs is done in this strategy.
Even though only a small amount of white-box testing
directly relates to game play or user experience, it is still
important in optimizing the code for better game perfor-
mance and having better coverage overall for the gaming
application.
Automated Testngs Role in Video Game
Development
So how about automated testing? Wouldnt implementing
a test automation tool such as TestComplete help increase
velocity amongst video game testers? Well, yes and no.
Maybe your game has a database, like Diablo, in which
there is a list of items and the possible properties and rarity
those items can have. Using TestComplete for data-driven
testing in a video game database would work wonders and
save time for your team. Its not that test automation cant
be used, but rather the context determines how it is used.
Simply put, because there are so many variables in a video
game such as user interaction, 3D environments and AI,
a test automation tool could not easily emulate gameplay.
Some companies even have used automated bots (AI
to emulate users) to speed up the testing process. The
problem with this approach is that, for example, a bot goes
to click on an item in the game but then a creature runs in
front of the bot as it goes to click on that item, the test will
fail. A user would just move and click the item if the crea-
ture got in the way.
9
TestComplete
by SMARTBEAR
In another situation, you could request what amounts to
a temporary easter egg from the dev team that your tool
can use to warp right to the shop. This would allow you to
set variables such as player level so your tool can check
to see if the correct items are available for you based on
those variables.
So it is important to realize that automated
testing may not be applicable for all projects
but that does not mean there arent ben-
efts to using test automation for
game development. Some of
the main benefts are to
understand the risks
of game components
and fnding defects
faster and earlier.
Since video games are
becoming massive in scale, test
automation will also be a primary
approach for regression testing
as game code gets larger with
each update.
Then there is always the case of the
platform in which your game is developed for. TestCom-
plete has been proven as a terrifc tool for automated func-
tional testing for HTML5 and Flash applications, and video
games for those web technologies are no exception.
In actuality, developing games for HTML5 is becoming
more popular due to the advancements of the technol-
ogy as a multi-platform approach to gaming. For example,
10
TestComplete
by SMARTBEAR
instead of porting a web game from the web to a mobile
device, you could simply develop a game for HTML5.
Since HTML5 is supported across almost every popular
platform from web to mobile - you would have a multiplat-
form game without the hassle of porting the code over for
each platform.
Video Game Testng on a Mobile Platorm
Video games and mobile devices have a lot in common
when it comes to testing; they both have many variables
the limit how testers can approach them. You can imagine
that developing a game on a mobile platform is twice as
diffcult. Not only are you dealing with the nuances of video
game testing, you basically throw a wrench at software
testing with the variables involved in mobile development,
for example, having no idea what the attributes are of the
device your game will be deployed tosince the most
popular devices on the day your game is released may
not have even been announced when you start your test-
ing. This is the main reason that mobile gaming is a work
in progress and many games are based off of a simplistic
GUI.
Below is a list of variables that need to be considered
when developing a game for a mobile platform.
Mobile
Data
How much data? What data?
What are your privacy policies?
What are you accessing on their device?
What is your end users tolerance for privacy
11
TestComplete
by SMARTBEAR
encroachment?
Network Availability
How does your app perform on various carrier
networks as opposed to Wif?
Will your app be a data hog that causes users to
watch their bandwidth usage as a result?
Is the performance on Wif so markedly improved
as to discourage the end user from using the app in
any other environment?
Gestures
Does your app give the user gesture options that are
confusing or that create confict with other apps?
If they are not willing to adapt to the gestures you
have incorporated in your app is the app navigable
without them?
GUI
Does your app have a command line feel to it or is
it appealing to even the most non-technical of users
(who will be a vast majority of most app audiences
anyway)?
Portrait and Landscape Views
What happens to your app when your end user turns
his or her device on its side or on its head?
Does it respond quickly?
Is it responsive at all?
Are there any felds that suddenly become
impossible to navigate because of the limited space
due to the keyboard expanding for use?
Processing Power
12
TestComplete
by SMARTBEAR
The last thing you want your app to be associated with
is making your users device so slow because it is a tre-
mendous resource hog. If within those frst few hours of
downloading your app your customer notices that noth-
ing is working as well on their device as it did before they
downloaded your app then your app will be immediately
removed.
Memory
Lets be honest. Most mobile device users are pushing
their memory limits to the nth degree on any given day.
How will your app impact device memory?
Battery
What impact will your app have on the battery life of a
mobile device? If an end user downloads your app then
suddenly sees their battery life fall through the foor you
may be viewed as a troublemaker. Not good.
Fragmentation of devices
There are an ever growing list of devices, screen sizes,
form factors, etc. that make up the mobile marketplace.
Its not cheap to make design alterations for every make
and model. Also, once you make accommodations for one
you may end up breaking something with another.
Will you need to pick and choose which devices are going
to represent the largest portion of your target market then
go from there? This is a critical decision that must be taken
seriously.
13
TestComplete
by SMARTBEAR
Gaming
Audio
Do sound effects play at the correct time?
Does music play at the correct time?
Do sound effects and music combined cause
cancellation at certain frequencies?
Is the sound overall clear on the device in which it is
playing?
Graphics
Are devices that supposedly meet hardware
requirements able to render graphics at a stable
frame rate?
Are there unwanted graphical errors appearing on
the screen, such as, artifacts?
Is the layering correct?
Accessibility
Is your game accessible to your target market?
Databases
Is the game accessing the correct data from the
correct server?
14
TestComplete
by SMARTBEAR
Conclusion
I hope you now have a better idea what to look for when
developing a mobile game. As stated before, there isnt a
right or wrong approach to testing a mobile game but you
need to at least try to consider all the possible variables
both physically and virtually. Sometimes its best to try
to list every possible situation a player would be in when
playing a game. Even if you do manage to identify every
possible variable, scenario and their interactions, testing
them will be a combination of targeting the moving parts of
mobile apps and games. With practice and determination,
you should be able to accomplish great testing on great
new mobile games without breaking the budget or letting
your competitors beat you to market, ensuring your games
are easy to understand for the players and easy on the
eyes.
About SmartBear Software
More than one million developers, testers and operations profession-
als use SmartBear tools to ensure the quality and performance of their
APIs, desktop, mobile, Web and cloud-based applications. SmartBear
products are easy to use and deploy, are affordable and available for
trial at the website. Learn more about the companys award-winning
tools or join the active user community at http://www.smartbear.com, on
Facebook or follow us on Twitter @smartbear and Google+.
SmartBear Software, Inc. 100 Cummings Center, Suite 234N Beverly, MA 01915 +1 978.236.7900
www.smartbear.com 2013 by SmartBear Software, Inc. Specifcations subject to change
EBOK-TC-20131004-V1-GAMETESTING
Get started with
automated testing
today
Try TestComplete