You are on page 1of 33

See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/313461225

Game Play Differences by Expertise Level in Dota 2, A Complex Multiplayer


Video Game

Article  in  International Journal of Gaming and Computer-Mediated Simulations · October 2016


DOI: 10.4018/IJGCMS.2016100101

CITATIONS READS
5 2,089

4 authors, including:

Lisa Castaneda Manrita Kaur Sidhu


foundry10 Radia and foundry10
9 PUBLICATIONS   80 CITATIONS    50 PUBLICATIONS   868 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Dota 2 Research View project

All content following this page was uploaded by Lisa Castaneda on 25 February 2019.

The user has requested enhancement of the downloaded file.


Game Play Differences by Expertise Level in Dota 2, A Complex Multiplayer Video Game

Lisa Castaneda Manrita Sidhu Thomas Swanson Jon Azose

Abstract

Dota 2, a complex team based video game, was used to study expertise and attentional allocation
in a multiplayer online battle arena (MOBA) setting. Pre- and post-play survey questions and
eye-tracker data were collected from 67 video game players during a session of Dota 2 play.
Questions explored abstract versus concrete conceptualizations of game-play and individual
versus team focus. Quantitative eye-tracker data was evaluated for differences in visual attention
and scan patterns. We noted that novices reflected on more concrete game elements and were
likely to look back at the same location twice in a row. There was no difference among player
categories in amount of time looking at mini-map or in self vs. team focus; however, experts
were more able to reflect on abstract game concepts. Expert-novice differences in this study are
similar to expertise research findings from other domains. The qualitative and unique
quantitative metrics that can be gathered from complex games may provide insight into the
development of expertise.

Keywords: Complex Games, Dota 2, Expertise, Eye-tracking, Scan Patterns, Video Game, Visual
Attention, Multiplayer, MOBA

Introduction

Expertise studies have long been an important tool for investigating how individuals develop
skills in a given domain. A common approach for studying skill acquisition is to examine
individuals who are identified as being at the “top” of their field (Bloom, 1985). However,
gaining a clear sense of how an expert accomplishes a task is a challenge. Experts are not always
consciously aware of all the knowledge and skills they use within their domain, making self-
reports of expert technique somewhat unreliable (Feldon, 2007; Sullivan, Yates, Inaba, Lam &
Clark, 2014). Eye-tracking is a methodology that has been used to examine expertise with regard
to visual processing in a variety of domains such as medicine (Kundel, Nodine, Conant &
Weinstein, 2007), games (Almeida, Veloso, Roque & Mealha, 2011), sports (North,Williams,
Hodges, Ward & Ericsson, 2009) and aviation (Kasarkis, Stehwien, Hickox & Aretz, 2001). As
eye-tracking demonstrates exactly where participants are focusing, it eliminates some of the
ambiguity of what experts are actually doing at a given moment in time. In games research,
proprietary in-game metrics of performance give another quantitative measure of an individual’s
performance relative to other players. Often, these metrics include complex algorithms that
encompass far more than just a measure of time played since total experience is a component of,
but not necessarily a sufficient indicator for, expert performance (Ericsson & Lehmann, 1996;
Gegenfurtner, Lehtinen & Saljo, 2011; Feltovich, Prietula & Ericsson, 2006). Utilizing methods
that are not solely based on self-reports, but also include objective measures of skill may aid in
better understanding how expertise develops (Tan, Leong & Shen, 2014). Such a mixed-method
approach, as applied in this study, involving game metrics, eye-tracking and in-person
interviews, can provide a range of measures through which to study expertise.

Games have long been used to study skill acquisition in performance among novices and experts.
Early work by Chase and Simon (1973) and de Groot (1978), in chess, laid the groundwork for
games and expertise studies by demonstrating novice-expert differences in a highly trackable
environment. A commonly utilized definition of expert performance was proposed by Ericsson
and Lehman (1996), “as consistently superior performance on a specified set of representative
tasks for a domain,” and that is how we will operationally use the definition here. Ericsson and
Smith (1991) highlighted the need to design studies that observe experts in a controlled setting,
analyze their cognitive processes, and propose explicit learning mechanisms for skill acquisition.
The introduction of computerized games, both chess and others, enables us to more specifically
capture some of the elements Ericsson suggests and thereby better understand expertise within
complex systems. In addition, since 97% of teens and 49% of adults play video games (Duggan,
2015; Lenhart ,Kahne, Middaugh, Macgill, et al., 2008), research in this area is socially and
developmentally relevant (Greenfield, DeWinstanley, Kilpatrick, & Kaye, 1994). Video games
provide a fertile environment in which to study the development of expertise in a domain that has
significant cultural capital. Greenfield et al. (1994) have argued that video games provide an
excellent platform for studying skill acquisition because they involve goal-oriented behavior as
well as instantaneous feedback.

Though expertise is often contextual and may not transfer between tasks (Boot, Blakely, &
Simons, 2011), there is some evidence to suggest transference between related domains. For
example, studies from both the U.S. military and healthcare have observed that many skills
needed to perform in the battlefield or in the operating room overlap with those utilized when
playing video games (Nawrocki & Winner, 1983; Rosser et al., 2007), suggesting that there are
important applications that extend beyond the games themselves. The military has used
traditional, commercial games to develop strategic thinking and team-based skill building in its
officer and non-commissioned officer training (Macedonia, 2002). In addition, video games, and
especially more complex team strategy games such as Multiplayer Online Battle Arenas
(MOBAs), may provide an interesting domain in which to explore links between specific game
attributes and learning outcomes. This is particularly true if we can start to make specific
connections between attributes of the games and performance (Wilson, Bedwell, Lazzara, Salas,
et al., 2009). In addition,their digital nature allows for a much more precise measure of attention
than, say, a traditional chess board, as each action taken by the player can be automatically
digitally recorded, measured and evaluated as events take place in real time around a team of
players. Demonstrating that expertise in the domain of MOBA video games mirrors that of other
domains provides yet another access point for the expertise research in an area of growing
relevance and commercial application. As noted by Becker (2007), commercial video games are
not only engaging for players, they also tap into highly effective mechanisms for learning. Game
designers might find this information useful because it could help them understand how to better
control the growth of player skill. In competitive games, where retention over time is important
to the popular micro-transaction revenue model, this knowledge may help reduce player
frustration through smooth skill development. Through the study of attention and expertise, we
can examine why those in-game mechanisms are successful, how players can improve their
skills, and how designers can make headway in the in-game and pre-game tutorials. As noted by
Boot (2011), process strategy approaches to data collection, including think-aloud protocols and
eye-tracking studies, may better help to flesh out the development of expert performance in
gaming.

In recent years, research in video games has continued to expand, particularly in the realm of
attention and training. A number of studies have used video games to examine aspects of
attention, such as divided attention (Greenfield et al.,1994 ), visual attention (Almeida, Mealha &
Veloso, 2011; Seif El-Nasr & Yan, 2006), visualization (Almeida et al., 2010), and attentional
control (Chisholm, Hickey,Theeuwes, & Kingston, 2010; Hubert-Wallander, Green & Bavelier,
2010). Modern video games, particularly online competitive games, also provide a host of
metrics that are tracked throughout game-play, as mentioned above, providing a novel and rich
way to decrease reliance on self-report and observation, both of which may be relatively more
subjective and prone to systematic error.

Much of the current research on video games is focused on first person shooters (FPS), puzzle
games, and occasionally real-time strategy (RTS) games (Green and Bavelier, 2003; Green, Li &
Bavelier, 2010; Boot, Kramer, Simons, Fabiani & Gratton, 2008; Almeida et al., 2010; Isokoski
& Martin, 2006) but we feel that complex multiplayer games, such as Dota 2, offer another
useful avenue for exploring expertise. When using the term “complexity”, particularly in
reference to a video game,, it is important to consider the elements that actually constitute
complexity. Latham and Yukl (1975) argued that complexity is derived from the challenge of
figuring out which performance arrangement is the optimal one, particularly if the performance
tasks are interrelated in complicated ways and difficult to separate. Though these characteristics
are certainly true of a ten player online battle arena strategy game like Dota 2, we believe the
additional element, identified by Campbell (1988), of the task having a high level of cognitive
demand due to all of the potentially simultaneous and rapidly presenting input and decision
pathways, is an essential element that makes Dota 2 an interesting environment in which to
explore expertise development in a more complex game environment.

This notion of the complexity of Dota 2 is also represented in the views of the wider MOBA
gaming community itself. Dota 2 is recognized as having a steep learning curve (Hernandez,
2014; Thursten, 2015) among the MOBAs available now, even to players who are familiar with
the genre. The game involves two teams of five players who work to invade their opponent’s
territory, destroying enemy fortifications called “towers”, and ultimately destroying the enemy
“ancient” (a large rock-like structure on the opposing team’s base). With over 100 heroes to
choose from, each with a unique variety of traits and strengths, a complex array of items to
purchase to enhance the abilities of one’s hero, and a variety of roles that one can fill while
playing, the player has much information to master while under fast-paced, multi-dimensional
attentional and strategic demands.

In many games, including Dota 2, continually updated information is projected over the main
screen in a heads-up-display (HUD), to inform the player on points accrued and other game
attributes. Caroux and Isbister (2016) note that the HUDs in RTS games are visually more
complex and contain a great deal more information than the more visually minimalist FPS
games. We believe that the HUDs in RTS-type games are visually similar to the HUD in Dota 2,
suggesting again that the complexity of the display may place different demands on players than
FPS screens. In addition, to our knowledge, very little research has focused on the highly
complex genre of MOBA games. Given the complexity of gameplay, the wider relevance of
video games to other fields, the popularity of competitive video games (eSports), and that Dota 2
is a free-to-play game, making it widely accessible to more than half a million active players per
month, we feel it is a relevant and interesting medium to explore. Looking at differences on the
novice-expert continuum in a complex decision making environment, may better enable us to
examine specific elements of task performance, with an aim to eventually improve performance
in a variety of domains and/or to improve visual layout or game tutorials.

In exploratory interviews with professional Dota 2 players (Castaneda, 2015), professionals


suggested that novices who are able to focus on selective elements rather than becoming
completely overwhelmed by all the stimuli in-game have an inherent advantage and are likely to
develop skills more rapidly. Professionals stated that one of the main traits skilled players have is
knowing what to focus on and what to ignore (Castaneda, 2015). In particular, the Dota experts
in our initial study felt there would be differences in how attention was distributed to the mini-
map (a real-time small scale overview map of game action) by novices versus expert players.
While investigating the HUD use of players with various skill levels, in both an FPS and RTS
setting, Caroux and Isbister (2016) found no significant differences between novice and experts
in number of fixations on the HUDs. However, they did find differences in duration of fixations,
fixation location, and in how players of various skill levels self-reported conscious attention to
and usefulness of such displays. Some of the limitations of the previous research on HUDs,
however, include the relatively unreliable definitions of novices and experts, and small sample-
size (Caroux & Isbister, 2016). MOBAs provide a unique game-play environment compared with
FPS or RTS games. They have more HUDs than the average FPS to contend with during game-
play, and, in contrast to RTS games, have a team-based aspect which must be attended to
throughout the game. We wondered whether a larger sample size, with players who truly
represent the skill spectrum from novice to expert (based on in-game metrics), in a complex
MOBA game, might show interesting differences in attentional allocation.
Research on cognitive load, which refers to the amount of mental effort utilized in working
memory to complete a task, and the ability of experts in a domain to circumvent processing
limitations by utilizing specific skills or techniques (Sweller, 1988), is also relevant to the
allocation of attention in-game. For instance, if players can efficiently use their oculomotor
system during tasks that are fast paced and stimulating, then they will be better able to control
and limit the influx of both visual and cognitive information, reducing cognitive load, and
allowing their attention to be preserved for the most important tasks (Shapiro & Raymond,
1987). Research in simulated military battle scenarios has also highlighted the importance of
avoiding “information overload” so that performance can at least be maintained or even
improved, even in a fast-paced, high information-volume setting (Eccles, 2008). Similar types of
distractions can occur in an online video game battle scenario, particularly with multiple players
interacting simultaneously. One of the great challenges of a game like Dota 2 is that a large
amount of information must be processed simultaneously and cannot necessarily be broken out
and addressed in succession, thus requiring parallel processing. One way in which experts are
able to reduce cognitive load is through attentional control so that extraneous information does
not impede processing (Eccles, 2006). Posner (1980) argues that attention represents a system for
routing information and that the orienting of one’s attention can occur covertly either through a
cognitive shift of resources, or more overtly through shifts of the head and eyes. Therefore, eye
tracking can provide a measure of overt attentional selection (Armstrong & Olantunji, 2012).

When individuals are visually scanning in complex environments, such as games, they need to
learn where the optimal focal points are within a given scene (Hayhoe & Ballard, 2005). In a
video game environment, including Dota 2, although the overall strategy and timing are highly
complex, often the minute-to-minute task structures within a given scene are clear, at least to the
experienced player. In addition to the traditional user inputs which can be studied in video games
(mouse, controller, keyboard), eye-tracking can provide an additional source of information
about where subjects are focusing attention (Isokoski & Martin, 2006). Using video software,
researchers can isolate points in a match to connect findings in the eye tracker data to specific
interactions or settings as they appear in-game. During visual scanning of an environment, there
are several different types of eye movements. Fixations, which are pauses or stops, last between
100 and 600 milliseconds as the brain processes the information perceived by the eyes (Matos,
2010). These are contrasted with the quick, jerky movements between fixations, known as
saccades, during which visual information is not registered but the gaze is moved quickly to a
different location (Isokoski & Martin, 2006). These varying types of movements and the pattern
in which they take place (scan pattern) can provide a great deal of information about to what a
subject is attending when looking at a computer screen, and what information is being ignored.

Differences in scan patterns have been identified between experts and novices in several domains
including driving, aviation, gaming and medicine (Kasarkis, Stehwien, Hickox, & Aretz, 2001;
Krupinski, Graham, & Weinstein, 2013; Sullivan, Yang, Day, & Kennedy, 2011). Shapiro and
colleagues (1987) note that when considering scan patterns it is notable that an ill-placed saccade
to an area of lesser importance on a screen can actually decrease the detectability of more salient
information elsewhere on the screen. A study of pathologists evaluating slides for evidence of
cancer found that with increasing expertise, scan patterns became more efficient. Fixations were
faster with content grasped quicker, less time spent on unimportant areas, and less need to revisit
or double-check previous locations (Krupinski, Graham, & Weinstein, 2013). Therefore, using
eye-tracking data and looking at scan patterns across a skill spectrum ranging from novice to
expert may provide interesting data about attentional changes that occur as expertise level
increases.

In addition to looking at the quantitative data of scan patterns, qualitative data provide additional
information about how a player is thinking about the game. Players’ comments about their
attention can help us to understand the differences between how novices and experts think about
in-game decisions. Sweller (1994) suggested that intrinsic cognitive load is very high in complex
problem scenarios and therefore novices may simply not be able to attend to all of the complex
elements successfully. However, novices might be more attuned to the more basic concrete
elements of game-play and therefore more apt to recall them during interviews-concrete elements
that, in contrast, may have become so automated by higher skill players they they are not readily
available for recall by them. It has been noted in the expertise literature that, in fact, experts are
significantly more likely to speak in abstract terms about their domain and thus, are
paradoxically less effective at explaining concepts to novices than are other novices (Hinds,
Patterson, & Pfeffer, 2001; Robinski & Stein, 2013). We anticipated that expert level players,
when asked to reflect on their overall game-play, would be more apt to focus on higher-level
strategic/abstract elements whereas novices would likely highlight more basic game-play
concepts.

Based on our initial study involving professional players, we theorize that some of the
differences between novices and experts would emerge in measurable ways. In this study we
attempt to demonstrate that MOBAs, such as Dota 2, offer yet another domain for exploring skill
acquisition and progression into expertise. Three hypotheses addressed are:

Hypothesis #1: Experts will spend significantly more time than novices fixating on key
areas of the screen, such as the mini-map. Additional key areas of the screen that are
important to game-play (Health, Mana, HP) may also show differences in attention
depending on skill level.

In Dota 2, players have two different screen views that focus on the movement of player
characters available at all times: a third-person view occupying the main portion of the screen,
and an abbreviated mini-map located on the left lower side of the screen, presenting a global
overview using icons representing only key features of the battle arena,. Previous research has
demonstrated that HUDs, such as a mini-map used in-game, can increase the amount of
information a player must process, which may be particularly difficult for beginners who have
not automated some tasks (Wilson, 2006). In Castaneda (2015), professional players expressed
the belief that the ability to allocate attention to the mini-map was an acquired skill gained with
expertise and that novices, instead, would be more apt to focus on the action on the main screen.
The professionals also felt that experts would be more focused on the strategic elements of hero
placement and battle/farming strategies using information from the mini-map than the novices. If
true, the reasons for this may include: (a) The main screen communicates immediately vital
information about individual players. As novice players may be using all their attentional
resources simply to survive, they would likely focus almost solely on the main screen; (b) the
main screen’s aesthetic aspects, with large scale, detailed, rapidly changing images, may be more
attention grabbing than the small mini-map in the corner; (c) novices may be focused on their
own individual performance, looking primarily at the main screen which contains their hero and
their own stats, rather than the mini-map which provides information on entire game and their
team. In addition, experts in the previous study indicated that they used shortcuts (such as
hotkeys) which decreased the need for attention to specific areas such as the Shop, and we expect
these shifts in attention to be evident in the eye-tracker data.

Hypothesis #2: Experts will demonstrate different scan patterns than novices.

As noted above, previous research on expertise across several domains has found that experts
utilize different scan patterns than novices, because they know, based on previous knowledge
and experience, where and when to look. This makes their performance more accurate and
efficient, while streamlining cognitive resources. In addition, the Dota 2 HUD has a great deal of
shifting information on screen that may not require attention in-game at all times. Experts tend to
be more skilled at ignoring extraneous or redundant information and can determine when
attention should be directed there, whereas this same type of information may adversely affect a
more novice player’s ability to process effectively (Haider and Frensch, 1999). Even though
Dota 2 is a game that has not previously been studied in this way to our knowledge, we
anticipate a similar pattern will emerge here. We hypothesize that expert players will scan the
areas of interest (AOIs) in different sequences than the less skilled players. We also anticipate
that experts might be more efficient in their scan patterns as measured by the sequence,
repetitiveness, or length of fixations on key AOIs.

Hypothesis #3: Experts’ verbal reports will focus relatively more on strategy, including
teamwork, while novices will focus on concrete game elements and active tracking steps.

As skill level increases, we predict that players will engage in higher-level/abstract thinking
about the game and this will be evident in the comments they make while reflecting post-game.
In addition, we anticipate that this increased focus on more abstract game elements by more
highly skilled players will also be evidenced by a reflection not only of “self” as player but by
the inclusion of comments that reference the team. Experts may not actively comment on more
rudimentary aspects of game play and attention as these steps may be more automated, as
demonstrated in previous studies in other fields (Feldon, 2007; Sullivan et al., 2014). Novices
may be more actively attending to rudimentary elements of game-play and thus may be more
likely to call those pieces out as they reflect on their game-play.

Methods

Participants

Sixty-seven volunteers were recruited from the greater Seattle area at The International 4 (TI4),
the largest Dota 2 tournament in the world. Players self-selected for participation by visiting a
sign-up booth at the tournament or signing up online after the event. Of the 67 participants, three
were excluded from analysis: two had manually reversed their on-screen set-up and one had no
reported ranking. Of the remaining subjects, 52 were males, 12 were females and all were over
the age of 18. All had Dota 2 game-play experience and had online gaming profiles through
Steam, the online gaming platform that hosts Dota 2, which allowed verification of their total
playtime and ranking.

Software and Hardware

All participants played one public match of Dota 2 while logged into their normal Steam
accounts. This allowed them to be matched to people at their level via the game’s internal rating
system. A custom-built desktop PC was used to host the game. The eye-tracker utilized was a
Tobii t60-XL which allowed gameplay to occur on a 24 inch monitor without any additional
external hardware.

Measures

Both qualitative and quantitative measures were utilized. Open-ended interviews were collected
from all participants, focused here on the post-play questions (Appendix 1). Eye-tracker data was
collected from the beginning of the match through the end of the match. If a player inadvertently
hit “escape” during game-play, the game was paused and the eye-tracker restarted. Player skill
level was determined by a proprietary in-game rating mechanism called Matchmaking Rating
(MMR). MMR is computed by examining in-game statistics, such as wins/losses and is a
proprietary modified version of the Elo algorithm. MMR may be based on solo or team play and
includes a number of variables that measure levels of skill. It provides a dynamic and systematic
way to rank skill levels on a novice to expert level play spectrum. MMR includes an uncertainty
factor that decreases as more games are played. The algorithm is better able to more accurately
predict ability level with increasing data. This change is a function of the player performance in
the match, which gives much more information about skill than just total amount of time played
(M. Ambinder, personal communication, March 25, 2016). The MMR system is divided
into“ranked” and “unranked” divisions, selected by the player before joining a game. The only
difference is that unranked play does not have a persistent, tracked ranking number, whereas
ranked does. An additional division occurs for players who queue for games alone versus with a
pre-set team (called “solo” and “team” MMR, respectively). MMR is calculated in the same way
across all divisions. Since the study was focused on individual performance, solo-play MMRs
were used for all participants. Ranked MMR was used whenever possible. If ranked MMR was
not available for a player, unranked MMR was used. A paired samples t-test showed no evidence
of a systematic difference between an individual’s solo ranked vs. unranked MMR t(50) = 0.896,
p = 0.375.

Expertise in Dota 2

Definitions of novice-expert skill levels relative to MMR scores were based on personal
communication with an experimental psychologist from the game’s developer (M. Ambinder,
personal communication, July 5, 2014). Novices were defined as participants who are familiar
with the game but have an MMR ranking below but not exceeding 2000. Intermediate players
were players with an MMR of 2001 to 3000. Advanced players fell into the range of an MMR of
3001-4000. Experts were players with an MMR higher than 4000. Among our participants, the
breakout by skill level was: 7 novice, 23 intermediate, 25 advanced, and 9 expert.

Table 1. Player stat breakdown.

Procedure

The study was performed in a dedicated testing area at Valve Software headquarters.Participants
were seated in stationary chairs in front of the eye tracker. All participants went through the
Tobii manufacturer-suggested calibration. A laboratory assistant sat next to the participants
viewing a monitor that mirrored the game play and showed the eye-tracking scan patterns. Each
player played one complete match. Even in normal gameplay, matches can vary significantly in
length and are not indicative of any particular expertise level, so we did not include
measurements of match-length. Eye-tracking data as well as the entire Dota 2 match were
recorded. Recordings began once subjects had been calibrated and were sufficiently comfortable
with the instructions and set-up.
At the conclusion of the game-play, all participants verbally responded to a series of questions
about their game-play experience, in particular, how they thought the game went, where they
thought they focused their attention during the game, and whether or not they thought the game
was representative of their usual play-style. Correlation of eye-tracking data with the qualitative
responses was not possible as the eye-tracker data indicates only the AOI visualized, not what
the player was observing or thinking, whereas the open-ended questions referred to more
complex concepts which did not end up translating directly into regions of the screen.

Analysis of Eye-Tracker Data

The Dota 2 game screen was subdivided into nineteen areas of interest (AOI) for analysis (Figure
1). The main body of the screen outside the labeled areas in Figure 1 comprises a twentieth
region, which we label “Rest of Screen”. The AOIs were defined as key informational locations
on the screen both by one of the co-authors who is an experienced Dota 2 player and an
experimental psychologist at the game’s production company, Valve software. These AOIs
already stand out on-screen due to the HUD used in-game (they are already illuminated in
special boxes so that attention can be directed towards them in-game). They were subdivided by
content (such as mini-map, store, player statistics, and monetary tracker), and individually
tagged. The mini-map was the second largest AOI measured at 96,000 pixels. The locations of
the AOIs remain static during the game even as the main-screen moves.

Figure 1. Dota 2 game screen with AOIs highlighted.


Estimates of the proportion of time spent looking at each AOI were computed by aggregating
data across players within each expertise category. Confidence intervals for these estimates were
obtained via bootstrap resampling of individuals within each expertise category. We also
estimated contrasts between experts and novices in proportion of time spent looking at individual
AOIs. Confidence intervals were obtained via bootstrap resampling with a Bonferroni correction
to account for multiple comparisons.

An expert/novice difference in scan patterns would be evidenced by a difference in observed


sequences of fixations. While it’s possible that there are long strings of fixations that are
indicative of expertise, such as looking from Mini-map to Gold (the currency in the game) to
Items (objects that supply additional functions to the hero), to Shop (purchase of items to
strengthen one’s hero), to HP/Mana (measures of health and magic power), for clarity our
analysis was restricted to short patterns of two consecutive fixations (e.g. Gold to Items or Mini-
map to Mini-map). Any sequential events with more than 100 ms missing data between them
were excluded from analysis.

In addition, the rate at which a player transitioned between different AOIs may provide a
measure of fluency in switching mental tasks. Therefore, transition data were analyzed with a
quasibinomial GLM. Each player, i, was assumed to have an individual probability, p_i, of
transitioning between areas of interest. We assumed a standard logistic mean model of the form
logit(p_i)=beta_0 + beta_1*MMR_i. The parameter of interest is beta_1, which governs the
relationship between MMR and cross-AOI transitions. If experts transitioned between different
AOIs more frequently than novices, beta_1 would be positive.

It is also informative to examine expert/novice differences in transition probabilities between


specific pairs of AOIs. Within each expertise category, we estimated conditional probabilities
that, given a current fixation in region r_1, the subsequent fixation would occur in region r_2.
These were computed by taking the ratio of observed r_1-to-r_2 transitions over r_1-to-anywhere
transitions across each expertise group. Confidence intervals for these estimates were obtained
via bootstrap resampling of participants within each category. Contrasts between experts and
novices were also estimated via bootstrap resampling with a Bonferroni correction to confidence
intervals.

Analysis of survey questions

The assessment of concrete versus abstract strategy was achieved by analyzing the players’ free-
response answers to a battery of eight post-game questions. Previous research (Caroux and
Isbister, 2016; Castaneda, 2015) suggests that players of different expertise levels might look at
different areas for different reasons. To add a qualitative element in order to explore those
reasons, we asked players very basic questions that emphasized why those specific locations
should be related to strategy. Tan, Leong and Chen’s research (2014) highlighted the importance
of gathering think-aloud protocol responses in addition to quantitative captures of data. Since our
focus was on novice versus expert attention we used very open-ended questions (e.g., “When
you looked at the mini-map, what were you looking for, why were you looking at it
specifically?) to get at a subject’s metacognitive reflections of their own attention. The questions
were broad and exploratory in nature so that we could begin to look for trends in player
commentaries and were done post-game so as not to interfere with game-play.

The responses were categorized into one of four ranked levels, each within two domains:
Abstract focus and Team focus. Two reviewers who were experienced Dota 2 players, blind to
the expertise level of the subjects, generated ratings for each scale independently, weighted
Kappa = .71 for abstract focus and = .65 for team focus. In instances of disagreement in rating, a
third reviewer, blinded to the expertise level of the subjects, met with the other reviewers and
obtained a consensus rating through discussion.

In order to explore skill level and its relationship to abstract/strategic thinking versus
concrete/mechanical thinking, players answered eight free response post-game questions (see
Appendix 2). Abstract focus was rated by reviewers on a 1-4 scale based on themes they
identified in responses for each question. Reviewers considered the series of questions and
generated a score from 1-4 based on the level of abstractness.

Table 2. Abstract vs. Concrete Focus.

Another metric was designed to detect relationships between skill level and either self-focus or
team-focus. Questions were also scored for this domain (Appendix 2). Team Focus was also
rated by reviewers on a 1-4 scale based on themes they identified in responses to each question.
Reviewers considered the series of questions and generated a score from 1-4 based on the type of
focus.
Table 3. Team vs. Individual Focus.

Results

Hypothesis #1: Experts will spend significantly more time than novices fixating on key
areas of the screen, such as the mini-map

Figure 2: Average attention time to the Mini-map across expertise categories. Estimates for each
quantity are shown with 80% and 95% confidence intervals.
Figure 2 shows extensive overlap between lower and higher MMR categories in attention given
to the mini-map. This overlap indicates that our data are insufficient to confidently estimate
either expert/novice differences or progression across skill levels. If systematic variation in the
amount of time spent looking at the mini-map existed across expertise categories, one would
expect little or no overlap between the novice and expert 95% confidence intervals which is
reflected in no significant difference in mini-map attention time across expertise levels (p > 1).

In contrast to the mini-map, in which there is only weak evidence of novice/expert differences in
attention, our results are suggestive of expertise differences in the allocation of attention to the
HP/Mana (p=.28; Figure 3), and Shop Button (p=.27; Figure 4) regions. In the case of HP/Mana,
we estimate that experts spend more time looking at the region than do any of the other expertise
categories. In the case of the Shop Button, experts spend less time than do the other categories.
Estimates and confidence intervals for attention time are provided in Table 4.

Figure 3: Average attention time to the HP/Mana region across expertise categories. Estimates
for each quantity are shown with 80% and 95% confidence intervals.
Figure 4: Average attention time to the shop button across expertise categories. Estimates for
each quantity are shown with 80% and 95% confidence intervals.

Table 4: Estimates and 95% confidence intervals for fraction of time spent looking at AOIs.
All estimates are given in units of seconds per minute. Confidence intervals represent 95%
confidence that each individual estimate falls within the specified range, not 95% confidence
that all estimates fall within their respective ranges.

The width of the confidence bounds in these plots is dictated by the amount of between-player
variability within each expertise group. The wide expert bounds in the HP/Mana plot indicate
more person-to-person variability in the expert group than in the other groups. Likewise, more
person-to-person variability is present in the amount of time novices look at the Shop Button
with a steep decline in variability as we move through intermediate, advanced, and to expert.

In all three of these regions, tests for non-zero differences between experts and novices fall short
of statistical significance after correcting for multiple comparisons (Table 4). However, testing
for statistical significance is not of primary interest here since the null hypothesis, that there is
precisely zero difference in attention between novices and experts, is implausible. Of much
greater practical interest is providing estimates of attention time along with a measure of
uncertainty in those estimates. Despite a lack of statistical significance, we estimate with high
confidence that experts pay more attention to HP/Mana and less attention to the Shop Button
than novices.

Table 5: Estimates and 95% confidence intervals for novice/expert contrasts in fraction of time
spent looking at AOIs. All estimates are given in units of seconds per minute. Confidence
intervals and effective p-values have been adjusted for multiple comparisons using the
Bonferroni correction. (The Bonferroni correction can produce effective p-values larger than 1.)

Hypothesis #2: Experts will demonstrate different scan patterns than novices.

Experts demonstrated a higher number of gaze transitions than novices. The observed
relationship between p_i and MMR_i is plotted in Figure 5. There is a significant positive
relationship between MMR and the probability of transitioning between different AOIs
(p=0.0075). An increase of 1000 MMR points is associated with an increase in 0.079 to the log-
odds of cross-AOI transitions (95% CI: 0.023 to 0.135). In a player with MMR of 2000, we
would expect 24.1% of fixations to be in different AOIs than their previous fixations. This rises
to 25.6% probability for an MMR of 3000, and 27.2% for an MMR of 4000. Another way to
examine this is that a typical player with 2000 MMR would look at the same AOI for 4.1
fixations in a row on average, compared to 3.7 fixations in a row for a 4000-MMR player. This
suggests that experts extract information more quickly than novices, with less need to double-
check within the same AOI.
Figure 5: Relationship between individuals’ MMR and estimated probability of transitioning
between different areas of interest on subsequent transitions. Dashed-line is the fitted curve from
a logistic regression.

With high confidence, experts were more likely than novices to follow a fixation in the Mini-
map with a fixation in the Rest of Screen region (Figure 6). A bootstrap test on the difference
between expert and novice transition probabilities found that the expert transition probability was
higher (p < 0.001). Likewise, experts were significantly less likely than novices (p < 0.001) to
follow a Mini-map fixation with another Mini-map fixation (Figure 7). These differences are
consistent with other expertise studies’ findings that experts are more adept at switching between
mental tasks, again with less need to double check their decisions on the mini-map.
Figure 6: Estimated probability that a fixation in the Mini-map region will be followed by a
fixation in the “Rest of Screen” region. Estimates for each quantity are shown with 80% and
95% confidence intervals.
Figure 7: Estimated probability that a fixation in the Mini-map region will be followed by
another fixation in the Mini-map region. Estimates for each quantity are shown with 80% and
95% confidence intervals.

Estimates and confidence intervals for transition probabilities are given in Table 6. Significant
correlations between expertise and transitions were observed for mini-map to “rest of screen”
and “mini-map to mini-map” (ps < 0.001; see Table 7).

Table 6: Estimates and 95% confidence intervals for between-region transition probabilities,
given in percentages. Confidence intervals represent 95% confidence that each individual
estimate falls within the specified range, not 95% confidence that all estimates fall within their
respective ranges.

Table 7: Estimates and 95% confidence intervals for novice/expert contrasts in between-region
transition probabilities. Confidence intervals and effective p-values have been adjusted for
multiple comparisons using the Bonferroni correction.

Hypothesis #3: Experts’ verbal reports will focus relatively more on strategy, including
teamwork, while novices will focus on concrete game elements and active tracking steps.

The analysis of expertise levels versus categories, showed a significant positive correlation (ρ =
0.321, p = 0.008) was present between Abstract Focus and MMR but there was no correlation (ρ
= 0.080) with Team Focus and MMR. Categorized survey answers were analyzed and resulting
correlations were plotted. Spearman correlations investigating the relations between Abstract
Thinking and Team Focus found a significant positive correlation (ρ = 0.607, p < .01). The
statistical evidence shows that both Abstract and Team scores were correlated suggesting that
although they were scored separately they are each measuring some aspect of player
performance which likely indicates they are measuring some level of strategic reasoning.

Below are sample participant responses given to the question: “When you looked at your hero,
what were you looking for, why were you looking at it specifically?” by a player with an MMR
of 2364 (intermediate level) and another player with an MMR of 4390 (expert level). The
responses received an overall rating of a “1” and a “4” respectively by the coders in the category
Abstract Focus.

2364 MMR Response -- I was trying to see where my character was for positioning.
(concrete/mechanical)
4390 MMR Response -- I didn’t watch my hero all that much, I just watch things like last-
hitting, debuffs and keeping track of health, mana and cool-downs. (abstract/strategic)

Another example of qualitative responses from two players, one with an MMR of 3204
(advanced) and the other with an MMR of 5396 (expert), in response to being asked “When you
looked at the enemies, what were you looking for, why were you looking at it specifically?”
Again, each player received an overall rating of a “1” and a “4” respectively for Abstract Focus

3204 MMR Response -- Looking at their positioning and their aggressiveness.


(concrete/mechanical)
5396 MMR Response -- I look for what items they have, because that is really important to see if
we are strong enough to fight them. I was aware of the strategies certain players would use and
when they would use them. There are ways to play around them, also what spells they are using.
(abstract/strategic)

In the category of Team focus, there was no correlation with MMR. Below are examples of
responses from players that were given a “1” and “4” respectively overall for Team focus. The
first is a 2364 (intermediate) MMR player and the second a 3226 (advanced) player in response
to the question: “When you looked at the mini-map what were you looking for, why were you
looking at it specifically?”

2364 MMR Response -- Colors


(individual focus)
3226 MMR Response -- Looking for things to counter with strategy; looking for other players, if
they’re going to Roshan.
(team focus)

In another example, a player with an MMR of 3949 (advanced) and another with an MMR of
6523 (expert), whose overall responses were ranked “4” and “2” respectively illustrate the
opposite trend, where the higher-level player gave a more individually-focused response to the
prompt, “When you looked at your hero, what were you looking for why were you looking at it
specifically?” And “When you looked at the mini-map what were you looking for, why were you
looking at it specifically?”

3949 MMR Response -- Just to see where on screen I was. I don’t usually look directly at my
hero. Usually my hero isn’t on my screen. On the mini-map I was looking for the location of my
teammates and enemies. (team focus)
6523 MMR Response --I wanted to see my own position relative to the game. (individual focus)

Discussion

In our previous work, expert players believed that one key difference between themselves and
novices was that they attended to the mini-map regularly throughout game-play and novices did
not. Given this, we hypothesized that experts would spend more time looking at the mini-map
because, as they could more rapidly grasp the active information on the main screen, they would
be able to expend more attentional resources on the strategic information on the mini-map
instead. We anticipated that, in contrast, novices might have their attention drawn to rest of the
screen. A surprising amount of similarity occurred between experts and novices with regard to
attention to various regions of the screen. There was no evidence in this study for expert/novice
differences in the amount of time spent looking at the mini-map from the eye-tracker data. The
eye-tracker findings were further supported by the qualitative responses we received from
players. Every single participant in the study referenced attending to the mini-map during
gameplay, across the skill-band when asked during their post-game interviews. In fact, one
expert level player (MMR = 5653) said, with regard to the mini-map, “I don’t look at the mini-
map a lot. I don’t think it is all that important. When I do it’s to see where the players are at and
how the towers are doing when we need to attack.” A very novice level player (MMR = 1300),
when asked the same question said, “I look at hero icons to see where our team is and where the
enemy is. It’s also a good point for orienting oneself.” All players on the skill spectrum attended
to the map to some degree but future research might try to refine the question and look more
specifically at the types of information various levels of player are gleaning from the mini-map.
For instance, in our above example a lower-level player (MMR = 2364) when asked what he
looks for on the mini-map simply said, “Colors.”

It is possible that our initial assumption concerning the map was overly broad and that video
game players in general are conditioned to look at on-screen maps (maps can be found in a huge
array of games and are often a key component of successful game-play). It is possible that
experts are extracting different information when they look at the mini-map than novices. Much
like Eccles (2008) noted, it is possible that the experts are more likely to look at “useful”
information on the map than novices. For example, an expert might be able to look at the mini-
map and discern not only the position of visible enemies, but also to gather other information
allowing a prediction of the most likely position of invisible enemies. In contrast, a novice may
only notice visible enemy positions and therefore be unable to anticipate ambushes or surprise
attacks. If this is the case, one would suspect that novices may have to look back at the map
again, which is indeed what we found, as discussed below.

The strongest evidence found in terms of differences in fixations is that experts spend less time
looking at the Shop button and more time looking at HP/Mana than novices. The Shop button
and the specific items for sale within the Shop have shortcuts associated with them. For skilled
players who know what to buy and when, use of shortcuts allows for efficiency and automation,
given the fluid state of the game. We believe that experts’ decreased fixations in this area may be
due to their utilization of specific strategies such as shortcuts to help them actively reduce
cognitive load. Conversely, HP and Mana are elements of the game that are dynamic and thus
must be actively attended to, regardless of skill level. The data regarding fixations with HP/Mana
suggests low attention to this AOI except at the expert level. At that level attention can
presumably be allocated more fully to this area of the screen, as less is required for other areas,
such as the shop, which may be more automated. Further examination of these concepts,
especially in a longitudinal study may provide a useful area for future research.

It is also interesting to consider whether the deployment of efficient attention using


heuristics/shortcuts emerges or develops in a particular sequence by skill level. For instance,
when assessing the Shop button, a sequential decrease in usage is present as the skill spectrum
moves from novice to experts. The explanation may be that progressive acquisition of knowledge
and use of shortcuts for the shop helps experts circumvent processing limitations in an
incremental fashion, resulting in less need for active attention to this area with increasing
experience. The qualitative results support this idea that more advanced players do not need to
actively attend to aspects of the game, such as the store. A player with an MMR of 2095 was
asked what he attended to in the store and he responded, “Mostly just to see the recommended
items and how much more money I would need to buy those items.” This response is contrasted
with a player MMR = 6523 who stated, “ I don’t look at the store, I open it up.” This expert-level
player is communicating that he does not actually go to the store and look, he is able to utilize
hot-keys to both open the store and get his items.

Previous research on novice versus expert visual attention has demonstrated that experts utilize
different scan patterns when attending to visual stimuli than their less-skilled counterparts, and
that experts may be more efficient in their ability to switch between tasks (Karsarkis et al.,
2001). This ability to switch tasks quickly suggests that experts glean information more quickly
than novices. Experts in our study did seem to be able to switch mental tasks faster than novices.
We quantify this by looking at the proportion of observations of two subsequent fixations which
occur in different AOI. An additional 1000 points of MMR is associated with an increase of
0.080 in log-odds of cross-AOI transition probability, on average. In other words, the higher the
MMR the less likely experts were to have two subsequent fixations in the same area than lower
level MMR participants.

Scan patterns were identified by measuring the probability that successive fixations transitioned
between different AOIs. Each AOI conveys unique information, so we would expect that experts
should be able to transfer their attention between different AOIs more frequently than novices,
which did in fact turn out to be the case. As Figure 6 shows, the estimates for a transition
probability of mini-map to the main body of the screen highlighted differences in scan pattern
across the skill-band. Although not a completely linear association, it is clear that novices were
less apt to transition from mini-map to the rest of the screen than experts. This suggests that
novices may not have been able to extract information from the mini-map as quickly. Looking at
the mini-map for a brief period and extrapolating that information to look at the main screen is
likely a skill that must be acquired. The difference at the expert level might suggest that experts
were able to extrapolate information from the mini-map with one fixation whereas less-skilled
players may have needed to “double-check” the map. Similarly, in a flight simulation study,
expert pilots were noted to have shorter fixations in general with more accurate landings,
suggesting the ability to extract information more rapidly (Kasarkis et al., 2001). The data are not
as clear for the intermediate and advanced players, but in terms of novice versus expert, there
does seem to be an effect.
A related trend in the scan patterns is shown in Figure 7, the mini-map to mini-map transition.
Novices were also more likely to have a scan pattern of fixations from mini-map back to mini-
map suggesting that they had to look back at the mini-map a second time to verify or review
information. Although it appears to be a marginal effect, the trend from novices to experts is
roughly linear, suggesting this attentional shift may be an acquired skill. However, the data from
the mid-level players make it difficult to say how the transition probabilities develop. It would be
interesting to explore the types of information from the mini-map retained by individuals at
various positions in the skill band. It is intriguing how the data reveal progressive decreases in
the mini-map to mini-map transition from novice to intermediate to advanced, but then a jump
from advanced to expert. If particular scan patterns are typical for experts or novices, it might
suggest that players learn through experience that they should look at AOIs in a specific
sequence. The sequencing of attentional focal points is not something explicitly taught in the
tutorials for the game. These are likely developed as players learn to automate some processes
and determine where the most useful places to allocate attention are at given points within the
game.

Including qualitative elements provided additional insight into how players considered their own
game play. We anticipated that a greater focus on one’s team would emerge in the post-game
reflections with increased skill level. In our previous interviews with professional players, they
often spoke about the team element being vitally important to competitive play. Given the
difficulty of the game itself, we anticipated that lower-level players might be more self-focused
in an attempt to simply survive. Surprisingly, however, a correlation between Team Focus and
MMR was not present. This may be because we did not specifically address the “team” during
our post-questions and simply looked to see if the players referenced the team on their own.
Conversely, in related work from a later study (Bonny & Castaneda, 2016a) our data suggests
that there are differences that emerge between novice and experts when controlling for Party
(playing with friends on a team) vs. Solo (queueing up alone) game-play. In the current study, all
players queued up as Solo players so a potential “Team Focus” expert-novice difference may
have been diminished in our study due to the unfamiliarity with the other players in the game as
a solo player, versus what we might see from expert players when playing competitively with
their own teammates.

In addition, higher level MMR players had a positive correlation with abstract references to their
game-play. Previous research (Feldon, 2007; Sullivan et al., 2014) has demonstrated that experts
are not able to articulate upon or reflect as clearly on specific strategies they utilize, partially due
to automation of basic tasks. This is in line with our assertion, and expertise research in other
domains, that suggests experts are less likely to discuss the more concrete elements of game-
play.

Since the study was cross-sectional, we cannot formally make inferences about learning curves.
However, since the observed changes in transition probabilities vary with expertise category, it is
possible that this is due to learning. We hope that this work might form the foundation for further
longitudinal studies of how in-game skills are acquired. In fact, this study served as the impetus
for a second and more focused study that examined what players at various skill-levels recalled
specifically about the mini-map in a variety of structured trials (Bonny & Castaneda, 2016b).
The aim was to try to illuminate what information players of various skill levels may be
extracting from the mini-map.

One of the great challenges with expertise research is highlighted by Sullivan et al. (2014),
“From a training perspective, it is not clear which cognitive strategies improve with expertise
level. The improvement is considered to be accrued with practice, but the learning mechanism is
yet to be uncovered.” We believe that complex games, such as MOBAs like Dota 2, provide an
interesting and innately trackable environment for studying expertise which may shed additional
light on the specific types of cognitive strategies that improve with acquired skill. Since similar
patterns, with regard to expertise, are found in the Dota 2 environment as in other domains, the
game can provide another avenue for researchers to study expertise. Given that previous research
has demonstrated some transfer from games and simulations into real world settings in the
military, medicine, and emergency preparedness scenarios (Nawrocki & Winner, 1983; Rosser et
al., 2007; Rose, Seater, & Norige, 2015) we are hopeful that there may be potential for transfer
and in-game learning from complex MOBAs like Dota 2 for training scenarios, collaborative
team-decision making or in visual attention during complex tasks. In addition, other mixed
methods research using both in-game metrics and qualitative measures can give us more detailed
data through which to explore both learning and player performance. The inclusion of qualitative
interviews enables us to explore the quantitative findings from a more metacognitive perspective.
If elements of learning from video games can be transferred effectively into other domains, or
conversely if findings from video game studies replicate findings on skill acquisition in other
domains, there may be important implications for both learning and training in game
environments.

As video game studies continue to evolve, we think it is important to include games from a wider
variety of genres, both to mirror player interest as well as gain a more solid understanding of
how skill acquisition occurs. Although chess is often used for game-based expertise studies, we
believe that complex strategy video games may be the next generation of tools for exploring
learning and skill development. In addition, though video games often do an adequate job of
teaching players how to play through in-game tutorials or teachable moments early in gameplay,
examining actual player data across the skill-spectrum may help game developers better target
which skills should be explicitly taught and when. As Caroux and Isbister (2016) noted, there
may also be implications for user design in terms of the HUD based on player interaction with
the screen. Future studies might try to further identify the types of information players of
different skill levels are extracting from the mini-map. In addition, further exploration of the
team-play aspect, across the skill spectrum, might provide interesting socio-cognitive directions
for future research and applications for other fields.
This study suggests that expertise effects in a video-game environment may be analogous to
those found in other domains. Our data also raise the possibility that in-game learning may have
points during which there are significant jumps as skill level progresses which may have
implications for gaming tutorials and learning in other environments. The qualitative components
hint at transitions in the types of thinking and reflection that occur across the skill spectrum. We
believe that studying subjects with varying levels of skill in the domain of complex strategy
oriented video games allows a powerful insight into the development of expertise.

References

Almeida, S., Mealha, O. & Veloso, A.I. (2010). Interaction behavior of hardcore and
inexperienced players: Call of Duty: Modern Warfare context. In Proceedings of ‘SBGames
2010 - IX Brazilian Symposium on Computer Games and Digital Entertainment - Arts & Design
Track, 106-115. http://www.sbgames.org/papers/sbgames10/artanddesign/Full_A&D_13.pdf

Almeida, S., Veloso, A., Roque, L. and Mealha, O. (2011). The eyes and games: A survey of
visual attention and eye tracking input in video games. In Proceedings of the Symposium of the
Special Commission of Games and Digital Entertainment of the Computing Brazilian Society
SBGames, 1-10.

Armstrong, T. & Olatunji, B.O. (2012). Eye tracking of attention in the affective disorders: A
meta-analytic review and synthesis. Clinical Psychology Review, 32(3), 704-723.

Becker, K. (2007). Pedagogy in commercial video games. Games and simulations in online
learning: Research and development frameworks, Hershey, PA: Information Science Publishing.

Bloom, B. (1985). Developing Talent in Young People. Ballantine: New York.

Boot, W. R., Kramer, A. F., Simons, D. J., Fabiani, M., & Gratton, G. (2008). The effects of
video game playing on attention, memory, and executive control. Acta psychologica, 129(3),
387-398.

Boot, W. R., Blakely, D. P., & Simons, D. J. (2011). Do action video games improve perception
and cognition?. Frontiers in psychology, 2, 226.

Bonny, J.W., & Castaneda, L.M. (2016a) Number processing ability is connected to longitudinal
changes in multiplayer online battle arena skill. Manuscript Submitted for Publication.
Bonny, J. & Castaneda, L.M. (2016b). Impact of the Arrangement of Game Information on
Recall Performance of Multi-Player Online Battle Arena Players. Applied Cognitive Psychology,
in-press.

Campbell, D. J. (1988). Task complexity: A review and analysis. Academy of management


review, 13(1), 40-52.

Caroux, L. & Isbister, K. (2016). Influence of head-up displays’ characteristics on user


experience in video games. International Journal Human-Computer Studies, 87, 65-79.

Castaneda, L.M. (2015). Dota 2 exploratory interviews: Skill acquisition and the professional
player. Foundry10, retrieved from: http://foundry10.org/newsite/wp-
content/uploads/2014/11/Dota-2-Exploratory-Interviews-2013.pdf

Chase, W.G. & Simon, H.A. (1973). Perception in chess. Cognitive Psychology, 4, 55-81.

Chisholm, J.D., Hickey, C., Theeuwes, J., & Kingston, A. (2010). Reduced attentional capture in
video game players. Attention, Perception, Psychophysiology, 73(3), 667-671.

Dota 2, Icefrog (2011). Bellevue WA: Valve Software (P.O. Box 1688, Bellevue WA 98009,
USA).

de Groot, A. (1978). Thought and choice in chess. The Hague: Mouton. (Original work
published in 1946).

Duggan, M. (2015, December 2015). Pew Research Center: Gaming and Gamers.
Retrieved from: http://www.pewinternet.org/2015/12/15/gaming-and-gamers/

Eccles, D.W., Walsh, S.E., & Ingledew, D.K. (2006). Visual attention in orienteers at different
levels of experience. Journal of Sports Sciences, 24(1), 77-87.

Eccles, D.W. (2008). Experts’ circumvention of processing limitations: An example from the
sport of orienteering. Military Psychology, 20(1), S103-2121.
http://dx.doi.org/10.1080/08995600701804822

Ericsson, K.A. & Smith, J. (1991). Prospects and limits of the empirical study of expertise: An
introduction. In: Towards a general theory of expertise: Prospects and Limits. Ed: Ericsson,
K.A. & Smith, J. New York: Cambridge University Press.

Ericsson, K.A. & Lehmann, A.C. (1996). Expert and exceptional performance: Evidence of
maximal adaptation to task constraints. Annual Review of Psychology, 47, 273-305.
Feldon, D. (2007). The Implications of Research on Expertise for Curriculum and Pedagogy.
Educational Psychology Review, 19(2), 91-110.

Feltovich, P. J., Prietula, M. J., & Ericsson, K. A. (2006). Studies of Expertise from
Psychological Perspectives. In K.A. Ericsson, N. Charness, P.J. Feltovich, & R.R. Hoffman
(Eds.), Cambridge handbook of expertise and expert performance (pp. 41-67). Cambridge:
Cambridge University Press.

Gegenfurtner, A., Lehtinen, E., & Säljö, R. (2011). Expertise differences in the comprehension
of visualizations: A meta-analysis of eye-tracking research in professional domains. Educational
Psychology Review, 23(4), 523-552.

Green, C. S., & Bavelier, D. (2003). Action video game modifies visual selective attention.
Nature, 423(6939), 534-537.

Green, C.S., Li, R. & Bavelier, D. (2010). Perceptual learning during action video game playing.
Topics in Cognitive Science, 2, 202-216.

Greenfield, P. M., DeWinstanley, P., Kilpatrick, H., & Kaye, D. (1994). Action video games and
informal education: Effects on strategies for dividing visual attention.Journal of Applied
Developmental Psychology, 15(1), 105-123.

Haider, H., & Frensch, P. A. (1999). Eye movement during skill acquisition: More evidence for
the information-reduction hypothesis. Journal of Experimental Psychology: Learning, Memory,
and Cognition, 25(1), 172.

Hayhoe, M. & Ballard, D. (2005). Eye movements in natural behavior. TRENDS in Cognitive
Science, 9 (4), 188-194.

Hernandez, P. (2014). 93 hours in and I’m still not sure Dota 2 is fun. Kotaku, Retrieved from:
http://kotaku.com/93-hours-in-im-still-not-sure-if-dota-2-is-fun-1611226542.

Hinds, P.J., Patterson, M., & Pfeffer, J. (2001). Bothered by abstraction: The effect of expertise
on knowledge transfer and subsequent novice performance. Journal of Applied Psychology, 86,
1232 -1243.

Hubert-Wallander, B., Green, C.S., & Bavelier, D. (2011). Stretching the limits of visual
attention: The case of action video games. Cognitive Science, retrieved from:
http://greenlab.psych.wisc.edu/documents/Stretching-the-limits-of-visual-attention(Hubert-
Wallander-etal-2011).pdf.
Isokoski, P. & Martin, B. (2006). Eye tracker input in first person shooter games. In Proceedings
Communication by Gaze Interaction: COGAIN Conference, 76-79.

Kasarkis, P., Stehwien, J., Hickox, J.,& Aretz, A. (2001). Comparison of Expert and Novice Scan
Behaviors During VFR Flight. Paper presented at the 11th International Symposium on Aviation
Psychology, Columbus, Ohio.

Krupinski, E., Graham, A., Weinstein, R. (2013). Characterizing the Development of Visual
Search Expertise in Pathology Residents Viewing Whole Slide Images. Human Pathology, 44(3),
357-64.

Kundel, H. L., Nodine, C. F., Conant, E. F., & Weinstein, S. P. (2007). Holistic component of
image perception in mammogram interpretation: gaze-tracking study 1. Radiology, 242(2), 396-
402.

Latham, G. P., & Yukl, G. A. (1975). A review of research on the application of goal setting in
organizations. Academy of management journal, 18(4), 824-845.

Lenhart, A., Kahne, J., Middaugh, E., Macgill, A., Evans, C. & Vitak, J. (2008, September 16).
Pew: Teens, Video Games and Civics. Retrieved from:
http://www.pewinternet.org/2008/09/16/teens-video-games-and-civics/

Macedonia, M. (2002). Games, simulation, and the military education dilemma. In Internet and
the University: 2001 Forum (pp. 157-167).

Matos, R. (2010). Designing eye-tracking experiments to measure human behavior. Tobii,


Retrieved from:
http://www.measuringbehavior.org/files/tutorials/Tobii_MB2010_tutorial_handouts.pdf.

Nawrocki, L.H. & Winner, J.L. (1983). Video games: Instructional potential and classification.
Journal of Computer-Based Instruction, 10, 80-82.

North, J. S., Williams, A. M., Hodges, N., Ward, P., & Ericsson, K. A. (2009). Perceiving
patterns in dynamic action sequences: Investigating the processes underpinning stimulus
recognition and anticipation skill. Applied Cognitive Psychology, 23(6), 878-894.

Posner, M.I. (1980). Orienting of Attention. Quarterly Journal of Experimental Psychology, 32,
3-25.

Rensink, R.A., O’Regan, J.K., & Clark, J.J. (1997). To see or not to see: The need for attention
to perceive changes in scenes. American Psychological Society, 8(5), 368-373.
Robinski, M.& Stein M. (2013). Tracking Visual Scanning Techniques in Training Simulation
for Helicopter Landing. Journal of Eye Movement Research, 6(2):3, 1-17.

Rose, C., Seater, R. & Norige, A. (2015). Skill transfer and virtual training for IND response
decision making. MIT Lincoln Laboratory. Retrieved from:
http://www.sts.rpi.edu/public_html/ruiz/EGDFall2015/__EDG%20LECTURES/big%20question
s%20and%20game%20responses%20possibly%20could%20be_files/IND%20Decisions%20and
%20Skills_FINAL.pdf

Rosser J.C, Lynch P.J., Cuddihy, L., Gentile, D.A., Klonsky, J.,& Merell, R. (2007). The Impact
of Video Games on Training Surgeons in the 21st Century. Archives of Surgery, 142(2), 181-
186.

Seif El-Nasr, M. & Yan, S. (2006). Visual attention in 3D games. In Proceedings of the 2006
ACM SIGCHI International Conference on advances in computer entertainment technology.
Hollywood, CA: ACE.

Shapiro, K. & Raymond, J.E., (1987). Attentional strategies for sport training. In: Vicker, J.N. &
Cancic, M.J. Eds., Proceedings of Eye Movements in Sport and Psychomotor Activity (30-40).
Calgary, Canada: Canadian Society for Psychomotor Learning and Sport Psychology.

Sullivan, J., Yang, J.H., Day, M., Kennedy, Q. (2011). Training simulation for helicopter
navigation by characterizing visual scan patterns. Aviation, Space, and Environmental Medicine,
82(9) 871-878.

Sullivan M., Yates K., Inaba K., Lam, L,. Clark, R. (2014). The Use of Cognitive Task Analysis
to Reveal the Instructional Limitation of Experts in the Teaching of Procedural Skills. Academic
Medicine, 89, 811-816.

Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive
Science, 12(2), 257-285.

Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design. Learning
and instruction, 4(4), 295-312.

Tan, C. T., Leong, T. W., & Shen, S. (2014, April). Combining think-aloud and physiological
data to understand video game experiences. In Proceedings of the SIGCHI Conference on
Human Factors in Computing Systems (pp. 381-390). ACM.
Taylor, R. (2016 January 16). Rise of eSports is a Game Changer. HuffPost. Retrieved from:
http://www.huffingtonpost.com/rich-taylor/rise-of-esports-is-a-game_b_6784174.html

Thursten, C. (2015). Game is hard: How Dota 2 changed my view of the average gamer. PC
Gamer, Retrieved from: http://www.pcgamer.com/game-is-hard-how-three-years-of-dota-2-
changed-my-view-of-the-average-gamer/

Wilson, G. (2006). Off with their HUDs: Rethinking the heads-up display in console game
design. Gamasutra. Retrieved from:
http://www.gamasutra.com/view/feature/130948/off_with_their_huds_rethinking_.php

Wilson, K. A., Bedwell, W. L., Lazzara, E. H., Salas, E., Burke, C. S., Estock, J. L., & Conkey,
C. (2009). Relationships between game attributes and learning outcomes review and research
proposals. Simulation & Gaming, 40(2), 217-266.

Endnotes

**** would like to acknowledge that we receive funding from *********** but the opinions
and research expressed in this paper are our own.

Appendices

Appendix 2.
Post-Questions
What are some of the key things you paid attention to in this match? (heroes, enemies, mini-map,
health, store, powers)
When you looked at your hero, what were you looking for and why were you looking at it
specifically?
When you looked at the enemies, what specifically were you looking for and why were you
looking at it specifically?
When you looked at the mini-map, what were you looking for, why were you looking at it
specifically?
When you looked at your health, what were you looking for, why were you looking at it
specifically?
When you looked at the store, what were you looking for, why were you looking at it
specifically?
When you looked at your skills, what were you looking for, why were you looking at it
specifically?
Was there anything you thought you should have paid attention to that you didn’t?
View publication stats

You might also like