You are on page 1of 24

Total War Arena Usability Report for August 2017

Report Information

Study Date 14/8/2017 – 16/8/2017
Report Date 23/8/2017
For Product Team Wargaming.net
Researchers Dr. S.Stafford; A.Alvarez; K.Correa; C.Koelndorfer;
J.Han;
Report Author Dr. S.Stafford et. al.
Build August 2017

Table of Content

Summary 1

Methodology 1

Top Findings 2

Detailed Findings 6

Finding 1: It was difficult to find commander upgrades 6
Finding 2: There was confusion on how to swap out units either one at a time or all at once 7
Finding 3: It took a while for players to understand the tech tree 8

Additional Findings 9

Likert Scale Statistics 10

1. Which commander did you select after the first part of the tutorial? 10
2. After your first battle (non-tutorial), did you change your commander? 11
3. While using the user interface, how difficult was it to change your units? 12
4. Which upgrading system did you use primarily? 13
5. While using the user interface, was it clear on how to replace multiple units at a time? 14
6. While using the user interface, was it clear on how to upgrade your commander? 15
7. While using the user interface, was it clear to you how to unlock abilities for your commander? 16
8. While using the user interface, was it clear on how to add consumables for your squad units? 17
9. Did you change the sensitivity, map size, or any other in-game interface settings? 18

Referenced Universal Principles of Design 19
1 | PAGE
TOTAL WAR ARENA REPORT
Summary
This report highlights the findings from a usability study for Total War Arena. This usability study was
conducted to test the user interface over the course of a 60-minute session. The report serves to record
any features of the interface and how the players received them.

Overall, the game’s interface was received with mixed reviews. Participants liked the aesthetics of the
interface both in-game and out of game, but did have issues with the actual usability of it. The specific
issues are covered in the top findings section.

Any design principles mentioned in the report are included in the referenced universal principles of
design section.

Methodology
Beginning on August 10th and concluding on August 18th, a usability study was conducted with a total of
12 male participants with ages ranging from 20-30 years old. Before playing, participants began the
survey and answered a section on demographics and their preferred games.

After finishing the demographics and preferred games section, participants were asked to complete the
tutorial and first officer training battle. After completing a sub-section of the tutorial that mentions
upgrades, participants were briefed on what they would be doing for the remainder of the test.

Participants were instructed to play through 3 officer training matches, where they were encouraged
and reminded to speak their thoughts out loud. Between each battle, participants were told that they
could spend the 3 currencies made available to them (gold, silver, and experience points) on any units
they wanted, up to tier 4. As well, they were told that they could upgrade any of the three default
commanders (Germanicus, Cynane, and Arminius), up to tier 4.

After finishing the entire duration of the study, the participants filled out a survey regarding Total War
Arena’s user interface. The survey included 5 different images identifying different locations in the
interface, as follows: Main Menu, Tech Tree (abilities), Squad (upgrading units), Squad (replacing units),
In-Game (officer training start). Participants were asked to use these images to mention any feedback on
the user interface elements. As well, participants answered additional questions and text fill-in and were
prompted about which upgrading system they used.

2 | PAGE
TOTAL WAR ARENA REPORT
Top Findings

Clicking on a Number will redirect you to the corresponding finding.

# Description of Issue Recommendations Priority
1 Players struggled to find the Adding a brief tutorial that covers the Red – Very
commander upgrades tree. All variety of ways to upgrade, including large
participants struggled at first to find it, commander upgrades, will help priority
with some taking longer to find it on players remember how to upgrade.
their own.
2 There was confusion on how to swap Either emphasizing the squad switch Red – Very
out units either one at a time or all at menu that pops up or offering a short large
once. tutorial will help players see this priority
All participants struggled at first to menu and understand how to use it.
switch their units in and out, and some
couldn’t figure out how to do so on their
own.
3 It took a while for players to Keeping both upgrade systems will Yellow –
understand the tech tree. All increase flexibility of the game, and Medium
participants struggled to upgrade their including a brief tutorial about the priority
selected units or commander due to the different upgrade methods will help
page not including a brief tutorial. players understand both systems.
4 Players had a hard time noticing Adding a flash to the consumable Yellow –
consumables. Most participants did not icon that shows up after purchasing a Medium
realize that consumables were in the unit with a consumable slot, or priority
game and instead focused on unit offering a pop-up/tutorial about
abilities. them will help players understand the
value of them and what they are.

3 | PAGE
TOTAL WAR ARENA REPORT
Research Questions

Main Interface Test Findings
Players notice the Battle Players would go automatically to the Battle button at first before we
button. had a chance to instruct them that they would be testing using the
Officer Training screen. The Battle button is very obvious and
explanatory.
Players can change As this is a standard feature for most games, participants had no issues
graphics settings. with finding the settings options.
Players can view their Tabs were clear and self-explanatory.
profile statistics.
Players can explain that the Players found the panel to be clear and informative without any
army panel features both explanation needed as to what they meant.
commander orders and
unit abilities.
Players can return from Players did not have any difficulty navigating back to the main screen
any screen to the default from the other screens.
one.

Units Test Findings
Players can replace units It took a while for the participants to learn how to replace units unless
from the Squad section. they stumbled across the method to do so by accident early on. Some
only learned how to do this one unit at a time while others learned that
you can swap up to three units with a desired unit type at a time.
Players can replace several Same as above.
units at once.
Players can explain that Players are able to differentiate between abilities that are passive and
there are active and those that are meant to be activated when the criteria are met.
passive unit abilities.
Player can purchase
consumables for units.
Players understand how
consumables work.
Players use the Restock
automatically button.
Players can research a new Players somewhat struggled to unlock units through purchasing items
unit. for the previous units in the upgrade tree. This was somewhat affected
by the confusion of having to replace units once they had unlocked
them.
Players can explain that the Players were confused at first as to why they could not unlock units in
unit and commander tiers the next tier, but after seeing the red notification that said their
correlate in terms of commander was too low tier. Some users did not realize this until
researching enough time had passed and they were prompted to check out their
commander’s tier.
4 | PAGE
TOTAL WAR ARENA REPORT
Players can explain what Players understood the currencies were earned somehow and found
can be obtained with silver out how to spend them but were confused about having to spend both
and experience experience and silver to unlock/purchase items.
Players can explain the
difference between free
and unit experience.
Players can view the Players noticed the experience at the top right after being briefed
amount of unit experience. through the tutorial.
Players can explain unit Players generally found unit types to match their game role preferences
types. and would explain them as such when talking about usage of their
units.
Players understand that Most players did not use the tech-trees section to change commanders
after selecting a new and would instead select their commander through the squad selection
-commander on the Armies menu that showed a brief explanation of the commander being
screen (Tech Trees chosen/used.
section) the current
commander is replaced.
Players understand that Players didn’t typically pick a premium unit even after being limited to
there are premium units in just Tier 4 units. They did not know the value of a premium unit being
the unit tree. free to use over a regular unit costing silver to use.

Commanders Test Findings
Players can purchase a new Players were eager to explore the commander lists and were
commander. disappointed when told they could only use the first unlocked
commanders.
Players can switch the Players had no difficulty learning how to switch between commanders.
commander.
Players can boost the As stated in Finding 1, it was difficult for players to find the commander
commander's tier. upgrade screen. However, once they did find it, players had no trouble
boosting the commander’s tier or upgrading their abilities.
Players can view how much The experience tab clearly indicated the required amount of XP needed
experience is required for and was understood by the players.
the commander to reach
the next tier.
Player can explain that Players understood that commanders could utilize all of the different
commanders within one types of units but that certain commanders had specific orders and
faction only differ in their abilities.
orders.
Players understand that a Players understood the Tier system as it applies to commanders
unit researched for one needing to be a certain level before the next tier of units could be
commander is unavailable researched.
for another
commander of the same

5 | PAGE
TOTAL WAR ARENA REPORT
faction with a lower tier.
Players can upgrade the Upon finding the tree, players had no issue with upgrading the orders
current commander order.
Players drag the tree with Players understood how to move the tree and view the orders.
orders to view all the
orders.
Players understand that Players understood the Tier system applied to the orders and abilities
different orders are as well.
available at different tiers.
Players can see that the Players were interested in the different commander options and would
Greece faction comprises 4 scroll down the list to look at all of the other commanders of the
commanders. factions.

Post Battle Screen Test Findings
Players understand what Players saw that they would gain general experience and silver after
their rewards for a battle each match but did not notice the free experience they had earned for
are. using a specific unit, nor did they notice if they had spent it.
Players understand what Players saw their score as a reflection of how well they performed as a
their personal score in a member of their team.
battle is.
Players understand what Most players saw victory as the prime way to gain more rewards.
actions in a battle lead
them to rewards.

Profile Test Findings
Players can find a list of Players were not aware that they had the ability to check their
their achievements. achievements after battle and only noticed them as they popped up
after the end of a battle when they were in the lobby.
Players understand the Players mostly understood Gold to be the premium currency, whereas
overview of their account silver represented the free-to-earn currency and experience was
statistics. experience earned by playing the game. Some users were confused on
the ways to earn either.
Players understand the Players did not question the statistics of their commander at all, with
statistics of their some players not even being aware of their commander actually
commanders. participating in the battle until they had already played several battles.
Players understand the Players were somewhat confused as to what the different
statistics of their units. classifications of stat weights stood for (i.e. being different than typical
“Attack”, “Defence”, etc.) and needed to read the tooltip next to the
stat weight to fully understand what it meant. Most users understood
weights due to their respective icons.

Other Test Findings
Players read tooltips. Depending on the player’s value of informative versus actual learning,
they would take the extra time to read the tooltips over and over.
6 | PAGE
TOTAL WAR ARENA REPORT
Some users preferred testing out items first, then reading any tooltip
related to it, almost ignoring them at first until they figured out what
they meant.
Player can close a modal Players would quickly read through any informative windows that pop-
window. up and would continue on with what they were doing, generally being
annoyed with getting too much information given to them in a tutorial.
Players read boxes with If a player was restricted from continuing on in the game before
textual instructions completing an action, they would then read what the instructions were
and then comply to them.

Detailed Findings

Finding 1: It was difficult to find commander upgrades

Explanation
The players struggled to find out how to upgrade their selected commander due to the page being in the
tech tree, whereas they were taught how to use the squad selection upgrade method.
Supporting Verbatim

● Participant 3 - “I would like to know what doing each upgrade means before I do it and
congratulate me on having the availability to buy new upgrades when they are available.”
● Participant 5 - “it may help to have the commander portrait or the menu button for the
commander upgrade tree display a border that flashes or shines to draw the players attention to
it.”
● Participant 8 - “Put the upgrade button on the left side of the screen because I didn't realize
there's a left panel at first.”
Priority of this Issue

Red – Needs to be fixed to prevent confusion about players upgrading their commander.

Recommendation

Emphasizing the commanders’ upgradeability by allowing players to upgrade through the preview-
commander section (much like upgrading units through the squad replacement page), will increase the
intuitive nature of the upgrading system.

Alternatively, at least moving the upgrade tier button to the top left area of the screen and adding a
picture above it would make it easily visible. Adding a button that leads users to the tech-tree in this
same area would be useful.

7 | PAGE
TOTAL WAR ARENA REPORT
According to the Gutenberg Diagram principle of design, the user wouldn’t notice the upgrade button if
it’s a small button in the top right area of the screen due to the direction that users typically read a
page.

According to the Iconic Representation principle of design, the usage of an icon next to the upgrade
button would draw the user’s attention much better than just a button.

According to the Layering principle of design, placing items of relevance together, like a large icon and
the upgrade button would help reinforce information retention.

8 | PAGE
TOTAL WAR ARENA REPORT
Finding 2: There was confusion on how to swap out units either one at a time or
all at once

Explanation

Players struggled to see the replace unit box (once their squad was full) towards the bottom-center of
the screen. Due to a lack of a correlating tutorial, they would miss this box multiple times before finding
it.

Supporting Verbatim

● Participant 3 - “I only changed out my units when I upgraded my unit into a new form of the
same unit (archers into archers, etc.)”
● Participant 4 - “It was a little confusing at first changing the units”
● Participant 7 - “At times, I'd get confused if I had actually changed out that unit to begin with.”
● Participant 10 - “I was a little confused at first on how to switch my unit because I didn't realize I
need to select a unit to replace before hitting replace.”
Priority of this Issue

Red - Needs to be fixed to prevent confusion and frustration in players trying to swap out their units.

Recommendation

Offering the replace-unit window (when a squad is full), in the center of the screen may increase
players’ awareness of how to change out their units.

As well, requiring players to confirm which units they wish to replace before letting them change to a
different menu or screen will prevent them from wondering why their units did not replace or what they
did wrong. This will reduce the amount of player frustration and confusion with replacing units, and in-
turn, the game overall.

According to the Confirmation principle of design, making a user confirm the unit replacement before
proceeding to another page will allow users to determine whether they have or have not successfully
replaced a unit and whether they wanted to or not.

According to the Operant Conditioning principle of design, reinforcing how a player changes out their
unit through repetition and an attached positive reward, like a pop-up that emphasizes their new squad,
will increase their understanding of how the unit replacement system works.

According to the Depth of Processing principle of design, giving a player more of a critical decision when
choosing to replace units by emphasizing what their squad may have or lack will in-turn increase their
understanding of what they want to bring to a match.

9 | PAGE
TOTAL WAR ARENA REPORT
Finding 3: It took a while for players to understand the tech tree

Explanation

Players were able to find the individual unit upgrades quicker and easier than utilizing the Tech
Tree. Once they found the Tech Tree, players who were not used to seeing all the upgrade branches of
a Tech Tree in a game were confused and a little overwhelmed by all the information there. This made
them not understand the reason for the Tech Tree when they could upgrade individual units on their
own without the use of the Tech Tree.

Supporting Verbatim

● Participant 3 - “I had little knowledge of the other window's existence. The one I used was more
readily available.”
● Participant 4 - “I completely forgot that there was a tech tree in the first place after getting into
squads and upgrading.”
● Participant 8 - “I don't think I can unlock required equipment in the Tech Tree.”
● Participant 10 - “The other screen [tech-tree] seemed more basic and I didn’t know how to
upgrade my unit.”
● Participant 11 - “easier to access despite the lack of an ability to go back and look at previous
units.”
Priority of this Issue

Yellow – Should be looked at after red issues are addressed, as to teach players how to properly use the
tech tree so that they do not blame the game for not teaching them.

Recommendation

Players typically preferred the squad-version of upgrading due to it being easily accessible, but it’s
recommended to keep both systems as this will allow users to use whichever system they feel
comfortable with.

Though, if both systems are used for upgrading, a quick tutorial on the tech-tree before placing players
in their first real players-versus-player battle might allow them to better understand both versions.
Having a quick tutorial will let players decide which system they’d like to use before coming to an
assumption of either systems’ value.

According to the Visibility principle of design, the tech-tree would be more usable if it included obvious
visible feedback to the user on the status of how upgraded a unit is and how to continue upgrading that
unit.

According to the Control principle of design, the tech-tree could use a somewhat extended tutorial
(longer than the squad upgrade tutorial after Officer training) to explain the depth of how it can be
used. This idea of control allows users to learn both systems first, and then decide which they want to
use.

10 | PAGE
TOTAL WAR ARENA REPORT
According to the Aesthetic-Usability Effect principle of design, the version of upgrading units through
the squad page was perceived as more usable than the tech-tree, simply because the squad page offers
more pictures and images to the user than the tech-tree.

11 | PAGE
TOTAL WAR ARENA REPORT
Finding 4: Players had a hard time noticing consumables.
Explanation:

Due to a lack of information in the tutorial about consumables, most players did not realize that
consumables were in the game and instead focused on unit abilities. Players felt that the UI did not do
enough to communicate that there were consumables to use within the game.

Supporting Verbatim

● Participant 4 – Didn’t notice that you could add consumables and later when mentioned thought
that they were extremely unclear. Mentioned that if there was a way to do add them that they
need to be more visible
● Participant 8 – Found that consumables were extremely hard to understand and were very
unclear.
● Participant 9 – Didn’t notice that you could add consumables and later when mentioned thought
that they were extremely unclear.
● Participant 10 – Didn’t add any consumables.
● Participant 11 - Didn’t notice that you could add consumables and later when mentioned
thought that they were extremely unclear.
● Participant 12 - Didn’t seem to notice that you could add consumables and later when mentioned
thought that they were somewhat clear. Noticed later after the tasked was passed (and toward
the end of the test), that you could add them. Once he saw them he wished he knew of the
sooner.
● Participant 13 - Didn’t seem to notice that you could add consumables and later when mentioned
thought that they were neither clear nor unclear. Later he explained that he didn’t get to see
how to add consumables.
[[Will add actual quotes by participants tomorrow when videos can be watched]]

Priority of this Issue

Yellow – Should be looked at after red issues are addressed, as a unit that has been unlocked that offers
a consumable slot could include a corresponding pop-up that explains what consumables are.

Recommendation

Changing or adding a visual that allows participants to see the benefits of consumables could help
explain how using their consumables tactically can drastically change a battle for them.

For players that had a hard time seeing the option to add consumables, and for those who didn’t know
that they could, the consumables could be designed to stand out a little bit more visually, especially if
these items have a decently strong impact in battles.

12 | PAGE
TOTAL WAR ARENA REPORT
Additional Findings

# Finding Recommendation Priority of Issue
1 Base capture notification was not Based on observations and comments Green – Not a
as obvious as other UI elements from participants on other UI elements, priority
and was missed. increasing the color brightness, adding a
strobing effect, and enlarging the
notification would help catch the players
attention. It would also add to the
urgency of the situation.
2 Friendly base capture notification Setting the ‘Friendly Fire’ notification to Red – Needs to
can get covered by friendly fire be lower on the screen when the ‘Base be fixed ASAP
notification. Capture’ notifications are already on
screen will help avoid this issue.
3 Threat level indicators on units Based on confused comments by Red – Needs to
were not fully understood. participants, explaining the threat be fixed ASAP
notifications in the tutorial will help
players understand the information they
represent.
4 The purpose of the capturable By describing the point of these towers in Yellow –
towers on the field were not the tutorial and/or labeling the towers, Somewhat of a
quite clear. players will better understand the point priority
of the towers and not confuse them with
objectives that need to be taken.
5 There was some confusion Based on other team-based games, Green – Not a
between player-controlled units making the player’s units color different priority
and friendly units. from
6 Little attention was directed to By covering the uses of these icons in Yellow –
the pin and marker icons, even tutorials, as well as enlarging the overall Somewhat of a
though they were flashing. size of the icons, players will better priority
understand and notice them.

13 | PAGE
TOTAL WAR ARENA REPORT
Likert Scale Statistics

1. Which commander did you select after the first part of the tutorial?

# Answer % Count

1 Cynane 50.00% 5
2 Germanicus 10.00% 1
3 Arminius 40.00% 4
Total 100% 10

14 | PAGE
TOTAL WAR ARENA REPORT
2. After your first battle (non-tutorial), did you change your commander?

# Answer % Count

1 Yes 20.00% 2
2 No 80.00% 8
Total 100% 10

15 | PAGE
TOTAL WAR ARENA REPORT
3. While using the user interface, how difficult was it to change your units?

# Answer % Count

1 1 (Really Easy) 20.00% 2
2 2 0.00% 0
3 3 0.00% 0
4 4 40.00% 4
5 5 10.00% 1
6 6 20.00% 2
7 7 10.00% 1
8 8 0.00% 0
9 9 0.00% 0
10 10 (Really Difficult) 0.00% 0
Total 100% 10

16 | PAGE
TOTAL WAR ARENA REPORT
4. Which upgrading system did you use primarily?

# Answer % Count

1 Tech Tree 10.00% 1
2 Squad Selection 90.00% 9
Total 100% 10

17 | PAGE
TOTAL WAR ARENA REPORT
5. While using the user interface, was it clear on how to replace multiple units at a
time?

# Answer % Count

1 Yes 70.00% 7
2 No 10.00% 1
3 I did know this was an option, but didn't do it. 0.00% 0
4 I didn't know this was an option. 20.00% 2
Total 100% 10

18 | PAGE
TOTAL WAR ARENA REPORT
6. While using the user interface, was it clear on how to upgrade your commander?

# Answer % Count

1 Yes 90.00% 9
2 No 10.00% 1
3 I did know that this was an option, but didn't do it 0.00% 0
4 I didn't know this was an option. 0.00% 0
Total 100% 10

19 | PAGE
TOTAL WAR ARENA REPORT
7. While using the user interface, was it clear to you how to unlock abilities for your
commander?

# Answer % Count

1 Extremely clear 60.00% 6
2 Somewhat clear 0.00% 0
3 Neither clear nor unclear 30.00% 3
4 Somewhat unclear 0.00% 0
5 Extremely unclear 10.00% 1
Total 100% 10

20 | PAGE
TOTAL WAR ARENA REPORT
8. While using the user interface, was it clear on how to add consumables for your
squad units?

# Answer % Count

1 Extremely clear 10.00% 1
2 Somewhat clear 10.00% 1
3 Neither clear nor unclear 10.00% 1
4 Somewhat unclear 0.00% 0
5 Extremely unclear 70.00% 7
Total 100% 10

21 | PAGE
TOTAL WAR ARENA REPORT
9. Did you change the sensitivity, map size, or any other in-game interface settings?

# Answer % Count

1 Yes 10.00% 1
2 No 60.00% 6
3 I did know that that was an option, but didn't do it. 10.00% 1
4 I didn't know that was an option. 20.00% 2
Total 100% 10

22 | PAGE
TOTAL WAR ARENA REPORT
Referenced Universal Principles of Design

Aesthetic-Usability Effect: People perceive more-aesthetic designs as easier to use than less-
aesthetic designs, whether they are or not.

Confirmation: Confirmation provides a means for verifying that an action is intentional before it
is performed. Confirmations slow task performance, and should be reserved for use with critical
or irreversible operations only.

Control: People should be able to exercise control over what a system does, but the level of
control should be related to their proficiency and experience using the system.

Depth of Processing: Thinking hard about information improves the likelihood that the
information will be recalled at a later time.

Gutenberg Diagram: According to the diagram, Western readers naturally begin at the primary
optical area (top left of a screen) and move across and down the display in a series of sweeps to
the terminal area (bottom right of a screen). The strong and weak fallows (top right, bottom
left) lie outside this path and receive minimal attention unless visually emphasized.

Iconic Representation: Iconic representation is the use of pictorial images to make actions,
objects and concepts in a display easier to find, recognize, learn and remember.

Layering: Layering involves organizing information into related groupings and then presenting
or making available only certain groups at any one time. Layering is primarily used to manage
complexity, but can also be used to reinforce information.

Operant Conditioning: A technique used to modify behavior by reinforcing desired behaviors,
and ignoring or punishing undesired behaviors.

Visibility: The usability of a system is improved when its status and methods of use are clearly
visible.

23 | PAGE
TOTAL WAR ARENA REPORT