Addressing the Impact of learning Analytics
Increasing the Impact of Learning Analytics
Shane Dawson Srecko Joksimovic
Centre for Change and Centre for Change and
Complexity in Learning Complexity in Learning
University of South University of South
Australia Australia
Australia Australia
shane.dawson@unisa.edu.au srecko.joksimovic@unisa.edu.au
ABSTRACT
Learning Analytics (LA) studies the learning process in order to
optimize learning opportunities for students. Although LA has
quickly risen to prominence, there remain questions regarding the
impact LA has made to date. To evaluate the extent that LA has
impacted our understanding of learning and produced insights that
have been translated to mainstream practice or contributed to the-
ory, we reviewed the research published in 2011-2018 LAK con-
ferences and Journal of Learning Analytics. The reviewed studies
were coded according to five dimensions: study focus, data types,
purpose, institutional setting, and scale of research and implemen-
tation. The coding and subsequent epistemic network analysis indi-
cates that while LA research has developed in the areas of focus
and sophistication of analyses, the impact on practice, theory and
frameworks have been limited. We hypothesize that this finding is
due to a continuing predominance of small-scale techno-centric ex-
ploratory studies that to date have not fully accounted for the multi-
disciplinarity that comprises education. For the field to reach its
potential in understanding and optimizing learning and learning en-
vironments, there must be a purposeful shift to move from explor-
atory models to more holistic and integrative systems-level re-
search. This necessitates greater effort applied to understanding the
research cycles that emerge when multiple knowledge domains co-
alesce into new fields of research.
CCS CONCEPTS
• Applied computing~Education • Applied computing~Learning
management systems
Permission to make digital or hard copies of all or part of this work
for personal or classroom use is granted without fee provided that
copies are not made or distributed for profit or commercial advantage
and that copies bear this notice and the full citation on the first page.
Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy oth-
erwise, or republish, to post on servers or to redistribute to lists, re-
quires prior specific permission and/or a fee. Request permissions
from Permissions@acm.org
LAK19 March, 2019, Tempe, AZ, USA
© 2019 Association for Computing Machinery.
ACM ISBN 978-1-4503-6256-6/19/03…$15.00
https://doi.org/10.1145/3303772.3303784
446
LAK’19, March 2019, Arizona, USA
Oleksandra Poquet George Siemens
School of Computing Centre for Change and
National University of Complexity in Learning
Singapore University of South
Singapore Australia
sspoquet@gmail.com Australia
george.siemens@unisa.edu.au
KEYWORDS
Learning Analytics, epistemic network analysis, adoption
ACM Reference Format:
Dawson, S., Joksimovic, S., Poquet, S. & Siemens, G. (2019). In-
creasing the Impact of Learning Analytics. In The 9th Interna-
tional Learning Analytics and Knowledge Conference (LAK19),
March, 2019, Tempe, AZ, USA. ACM, New York, NY, USA. 10
pages. https://doi.org/10.1145/3303772.3303784
1 INTRODUCTION
It is commonly noted that learning analytics (LA) has the potential
to address many of the challenges confronting contemporary edu-
cation [20]. Cited challenges include the impact of reduced govern-
ment funding and resources on teaching quality, increased account-
ability, growing student diversity, alongside demands for flexible
offerings and global competition [27]. LA promises to bring new
insights into the learning process to enable practices that can en-
hance student success. Alongside its proclaimed potential for im-
pact, LA is also noted to be a developing domain of research. The
rhetoric of researchers in a young field, against the backdrop of
broad changes in education, requires greater scrutiny.
The first LA conference (LAK11) was held in Banff, Canada,
bringing together some one hundred researchers from around the
world with an intent to explore and shape learning analytics as a
discipline. Almost a decade later, it is timely to reflect on the impact
LA has made: can we, as frequently claimed, demonstrate that LA
is addressing the often-cited challenges facing education? Is the
field of LA beginning to realize its potential?
This paper argues that to date, LA research has failed to fully
deliver on its promise to impact education and educational institu-
tions. For the field of LA to achieve its stated goals, there is a ne-
cessity for research to make a purposeful and transformative shift
from a predominance of exploratory studies that are often limited
in scale to evaluative research that explores the impact of LA ac-
tions and outcomes at scale such as institutional adoption or sys-
temic transformations. In making this argument, we first explain
our approach to research maturity through the concept of research
cycles: from conceptualization and exploration to evaluation stud-
ies that provide the necessary foundation for refining, revising or
generating theory. Next, we outline how access to new sets of data
and methods influence the types of questions posed and subse-
 LAK’19, March 2019, Arizona, USA S. Dawson, et al .

quently the progression of LA research. Finally, we detail the spe- order to analyze these problems in the context of the changing ed-
cific methods of this study and its findings, before discussing the ucational landscape. At the same time, the combination of multiple
need for replication and evaluation studies that contribute to the re- disciplines converging on addressing such problems raises the pos-
fining or developing theory. sibility to access new data sets or apply novel combinations of pre-
viously disparate disciplinary methodologies. This process prompts
2 RESEARCH MATURITY the initiation of numerous smaller scale exploratory works as re-
searchers interrogate the affordances of alternate methods applied
LA is commonly labeled as a bricolage field, emerging from the
into new domains. An example is the application of machine learn-
convergence of multiple disciplines [11]. While this entanglement
ing models to predict student academic performance. The studies
of ontologies, methodologies and theories generates many benefits
within the exploratory phase can lead to a higher level of evaluation
it is also accompanied with many challenges. The advantages of
and replication research that enables revisions and new contribu-
multi-disciplinary research stems from the new perspective and in-
tions to theory. Figure 2 depicts the process of research maturation
sights different theoretical and computational approaches can de-
from the surrounding environmental pressures to conceptualiza-
velop. Evidence of the impact such convergences have made can
tion, exploration, evaluation and theory building. Although the
be found in the prominence of behavioral economics. This line of
model depicts a linear set of phases, we recognize that the research
research evolved from the joining of psychology and economics
is cyclical and will constantly inform all phases from conceptual-
and has profoundly impacted on research into human decision mak-
ization to exploration, evaluation and contributions to theory. The
ing [33]. Conversely, the challenges lie in the ability to establish a
progression of research maturation that has transpired in LA is
common understanding of the strengths and weaknesses of differ-
noted by Ferguson [11], and recently by Mangarsoka and Gianna-
ent methodologies or articulating and accepting various discipli-
kos [26]. The prominence of LA grew from government and fund-
nary theories and related assumptions. However, the benefits of
ing pressures alongside the increased availability of student learn-
bringing multiple perspectives, approaches and theoretical frame-
ing data. The increased adoption of technology into the learning-
works to bear on complex problems far outweighs the transitional
teaching process generated new and accessible sets of learning data.
difficulties linked to such convergences.
The availability of learning data was notably accelerated with the
In evaluating its impact, LA can learn from other established
development of open online courses such as MOOCs. Additional
multi-disciplinary fields. Similar to LA, Information Systems (IS)
challenges such as reduced resourcing and increased accountability
research involves an interplay between a wide range of factors, in-
requirements by government agencies, helped to distinguish LA
cluding people, technology, organizational structures and culture.
and raise the importance of LA through new opportunities for re-
For example, Hevner [21] proposed a conceptual research model to
search. The transitions and progression of LA is reflected in the
understand, execute, and evaluate IS research and provide a unify-
initial works being strongly positioned around defining and con-
ing framework. Hevner and colleagues [21, 22], consider three in-
ceptualizing
ter-related cycles of IS research that are applicable to the field of
LA (Figure 1). These cycles are termed as relevance, design, and
rigor. The relevance cycle represents the problem identification
phase, whereby the conceptualization of these problems is heavily
influenced by multiple environmental factors (e.g. funding, data ac-
cess, computational methods, government and organizational pol-
icy, technologies). The design cycle is based on iterations of work
that seek to build or explore in order to gather data and insights to
refine and evaluate outcomes. Linked to the outcomes of evaluation
studies, the rigor cycle relates to feedback into theory and additions
to new knowledge. The boundaries of the research cycles are not
discrete, they are blurred and integrated. Hevner’s model illustrates
there is a level of maturity of work as research transitions from the the field and associated research possibilities [9]. In this paper we
Figure 1: Design science research cycles [20]
conceptualization of problems and the initiation of exploration
studies to the replication and evaluation of outcomes. The develop-
ment of the field of LA is analogous to the model of IS research
cycles.
The application of these research cycles to LA can enable an
understanding of how the field is maturing (Figure 2, modified from
Hevner [21]). The emergent environmental factors generate novel
pressures that require new approaches and methods for scientific
inquiries – often collaborative and multi-disciplinary [11]. The
range of environmental pressures impacting LA also invoked a nec-
essary definitional and conceptualization phase. A re-framing of Figure 2: Conceptual model of research maturation phases
problems such as student retention and success were necessary in review the state of field in terms of the types and scale of research
undertaken to determine the level of maturation of the field and

447
 Addressing the Impact of learning Analytics LAK’19, March 2019, Arizona, USA

overall impact to date. In this context we view maturation as studies focus groups or interviews), or viii) other data sources (e.g., litera-
building on outcomes of prior works, rather than simply the number ture reviews). Finally, we observed the research settings - whether
of studies conducted. This process mirrors health research and drug the study collected data from i) university, ii) other formal educa-
trials. Modeling and small-scale trials lead to human trials and fi- tion organizations (e.g. k-12), or iii) informal learning contexts (e.g.
nally broad population studies, applying prior works and research. workplace).
This results in ongoing evolution of research models and focal The coding was performed through several stages. Initially,
points. three coders reviewed 50 studies for training purposes and to gain
familiarity with the literature, process and coding scheme. Next,
3 METHODS each of the coders independently coded a further 40 studies (in two
iterations – 2x20), until reaching the average agreement of 90 per-
This paper aims to identify if learning analytics research is address-
cent, with average Fleiss’ Kappa [15, 16] of 0.61. The agreement
ing its goal of developing insights and understanding into the learn-
varied from 100 percent (Fleiss’ Kappa 1) for items such as “stu-
ing process and then making an impact through improving learner
dent focus” or “other educational settings” to 77 percent (Fleiss’
success at scale. If LA research is fulfilling its self-stated potential
Kappa 0.32) for “other data”. The consistency issue was due to the
of improving student success there will be strong evidence of in-
limited number of studies for a particular coded dimension, hence,
creased research activity evaluating the interventions or support
reaching high agreement was challenging. Therefore, a decision
processes that build-on established predictive models or prior ex-
was made to revisit the definition for each data-related item (di-
ploratory work. It is feasible that as these evaluation and replication
mension) before continuing the coding process. Moreover, as there
studies are undertaken there will be an increased focus on a research
were very few studies coded as “evaluate”, “understand”, and
cycle where feedback and contributions of new knowledge provide
“feedback” (Figure 3), agreement would be revisited and consid-
insight and refinements to theory and established frameworks. It is
ered for each of the studies coded with at least one of these three
at this juncture that the field can begin to demonstrate influence on
dimensions. Finally, the remaining studies were equally split be-
student success and learning. The following research question un-
tween the three coders. Overall, each coder independently reviewed
derpins this broad aim:
approximately 150 studies.
• Has the scope and focus of LA research progressed the
field in terms of understanding the learning process and 3.2 Epistemic Network Analysis
optimizing learning for student success?
Post the coding process, the relationships between the five dimen-
3.1 Literature Analysis Procedure sions described in our coding scheme (i.e., Focus, Purpose, Scale,
Data, and Settings) were explored using Epistemic Network Anal-
To address the research question 522 papers published in LAK
ysis (ENA) [30]. ENA is a graph-based method for analyzing logs
Conference proceedings and Journal for Learning Analytics (JLA) of coded data that reveals the occurrence of relationships between
were analyzed. The process included all research and practitioner
the constructs that emerge from the underlying dataset [30]. ENA
papers between 2011 and 2018 inclusive. Editorials and posters is often used to model complex and highly dynamic problem do-
were excluded from the analyses. All studies were coded according
mains [28, 29]. Specifically, ENA models an epistemic network us-
to five overarching dimensions – Focus, Purpose, Scale, Data and ing the codes assigned to each of the items in the dataset, and where
Settings (Figure 3). Specifically, the papers were noted for their
the codes represent nodes in the network. The method has been
study Focus, that is whether the study concentrated on i) using stu- used to understand collaboration patterns across various, formal
dent data to improve learning outcome, ii) improving teaching effi-
and informal, educational domains [14, 18]. In this study, ENA is
cacy, or iii) addressing policy issues related to learning analytics used to explore the relationships between the five coded dimensions
adoption. Moreover, we also coded the reviewed papers for their
(i.e., Focus, Purpose, Scale, Data, and Settings), extracted from the
Purpose. The study purpose was adopted from Dawson, et al [9] papers reviewed in the LAK conferences and JLA.
and explains whether the goal of the study reflects i) exploratory
There are two central points of interest when modeling epis-
(i.e., predict/build), ii) evaluation, iii) conceptual (i.e., understand), temic networks - namely weights of the connections between the
iv) solution validation (i.e., feedback) types of research, or v) opin-
nodes and stanzas (i.e., the data subsets where the co-occurrence of
ion papers (i.e., other). A focus of the work was to examine the
codes is established) [29]. In addition to using binary codes (the
scale of data used in the published LA studies. Each paper was
code does or does not exist), ENA allows for specifying numeric
coded for i) having more than 100 data points (i.e., N>100); ii)
values that represent the strength of a given code. Stanza further
whether data were collected at the course or project, iii) program, defines the analysis unit or the subset of data where the links be-
iv) institution, or v) multiple-institution level. The nature and type
tween the codes are expected to be established. For example, ana-
of Data collected was also coded. This included i) instrument/pro- lyzing collaboration within a discussion forum, we can model the
tocol (e.g., survey, questionnaire), ii) multimodal data sources, iii)
epistemic network with the stanza at the level of each discussion
institutional data (e.g., learning environment, LMS, or student in-
formation system, library data), iv) third-party data (e.g., Publish-
ers, gaming environments or mobile apps); v) assessment data
(grades, GPA or pre-post test scores), vi) social media and social
interaction data (e.g. twitter, facebook), vii) qualitative data (e.g.

448
 LAK’19, March 2019, Arizona, USA S. Dawson, et al .

participant or perhaps at the level of discussion thread, as being

more commonly used [14, 18]. In this study, each coded paper was
treated as stanza for ENA. As the coders reviewed each paper they
assigned a binary variable 0,1 for each of the sub-dimensions (Fig-
ure 3). For example, if the study was undertaken in a university
setting it was coded as 1 for this dimension. Specifically, we ob-
served whether a specific code (e.g., “Focus.Student” or “Pur-
pose.Evaluate”) occurred at the level of a single study. The use of
studies as a unit of analysis enabled us to explore the connections
between the five dimensions for each study. ENA in this instance,
provided a means to visualize and statistically analyze the relation-
ships between the dimensions both in terms of distance (how close
they are to appearing together) and strength (how often they appear
together). Figure 4 visualizes the relationships between the coded
dimensions plotted separately for each year. The nodes in the graph
are colored with respect to the five broad dimensions noted in fig-
ure 3 (i.e. Focus, Purpose, Scale, Data, and Settings). For the pur-
pose of simplifying the visual representation, we applied a thresh-
old to filter out the least prominent nodes in the epistemic network
(Figure 4).

4 RESULTS

4.1 General Overview

The findings from the coding process demonstrate there is a relative
consistency in the quantity of papers aligned to the categories of
focus, purpose, scale and settings from LAK conference proceed-
ings (2011-2018) and JLA publications (2014-2018). As demon-
strated in Figures 2 and 3, throughout the years, the Focus dimen-
sion of the vast majority of LA studies is on the student. This find-
ing was also observed by Knobbout and van der Stappen [24] in
their systematic review of how LA measures affects learning. The
authors noted that the majority of research were related to student
performance or learning process. In the present study, there was an
observed subtle increase in LA work directed towards educators or
instructors for the 2018 published literature (LAK conference and
JLA) from prior years. The number of studies involving policy, par-
ticularly in the area of privacy and ethics has been relatively stable
from 2011-2018 (see Figure 3).
The second category is related to the purpose of the published
studies (Figure 3). In this instance we can see that the majority of
research undertaken has been exploratory. That is, the researchers
are aiming to build a tool or through analysis of particular data sets
predict a discrete student outcome (e.g. performance,
retention). This category of research can be considered as novel but
early in development. Papers related to understanding can also be
Figure 3: Percentage of papers per year for the coded dimen-
classified as early research approaches. These studies are often con-
sions. Each paper coded may include more than one
ceptual and seeking to unpack a particular theoretical model in the
dimension.
context of LA. The number of evaluation studies peaked in 2017.
These types of studies aim to build upon the foundations of earlier studies, i.e. studies that actively provide feedback into theory for
exploratory works. For instance, the prior research may have estab-
the purposes of validation, refinement or revision, have been a rel-
lished a predictive model and suggested support actions. In evalua- atively minor focus of LAK and JLA work. For example, self-reg-
tion studies this model is tested and applied in situ. The evaluation
ulated learning is an often-used theory to interpret the findings ob-
studies, defined as seeking evaluation of implemented models or served in LA. However, using SRL as an example, there are a very
tools, can be considered as more mature and provides evidence to
limited number of studies that have sought to further revise related
lead towards the feedback category of research. To date, feedback

449
 Addressing the Impact of learning Analytics
theoretical models. An indication of maturity of research and ap-
plied methods relates to the scale of the studies in applied research
and practice [21, 34]. In the subset of exploratory studies reviewed,
the scale of the research tended to be confined to the course
(S.Course) or project level (S.Project). As noted in Figure 3, a mi-
nority of studies were undertaken at the level of program or institu-
tion. Studies conducted at the scale of an institution would repre-
sent data from numerous programs of study or data that has been
collected across the entire institution (e.g. Admissions). The coding
revealed that the bulk of published works have been confined to a
course level or project. This finding is further evidence of the small
number of institutions globally that have large scale LA instantia-
tions [8, 10]. The lack of scale also highlights the difficulty for re-
searchers to access data that lies outside a course or program of
study often due to security and privacy concerns and institutional
challenges of accessing data [8, 10]. The coding of multi-institution
data are representative of studies such as ASSISTments that involve
multiple schools or education institutions. While this category ap-
pears to reflect further scale the studies involved were also at a class
level – though involving multiple classes from multiple organiza-
tions. The coding in this study also highlights the broad range of
data sources. There is a strong emphasis on data from institutional
“learning environments” such as the LMS, Student Information
System and Assessment. Third party tools from publishers, ITS or
gaming environments are also common.
The final coding category relates to the setting or context
wherein the study was undertaken. As detailed in Figure 2, the Uni-
versity setting continues to be the major site for LA research. Since
2011 the number of studies from “other education settings” such as
K-12 have been steadily increasing. This may suggest that LA is
broadening its reach beyond university and is gaining traction in K-
12. Interestingly, the number of studies involving informal settings
such as the workplace have decreased over time.
4.1 Network Relationships
Visual and statistical analysis of the change in the focus of reviewed
learning analytics literature was undertaken using ENA. The results
of the ENA (Figure 4) illustrate how the reviewed studies addressed
the aspects of Focus, Purpose, Scale, Data, and Settings over the
past eight years of learning analytics research and practice. The
group average network for 2011 indicates that the main focus of the
presented papers related to the “other” classification of papers. That
is, the presented papers were of a conceptual nature reflecting the
discussions pertaining to the definition or framing of learning ana-
lytics in university settings.
In 2012, the focus of the research tended to be on student suc-
cess and retention in university settings. However, the scale of the
reported studies were of a small population size indicative of a sin-
gle project or course. The papers reported findings from explora-
tory studies or the development of predictive models or LA tools.
Although minor, in 2012 there was an introduction of policy papers
related to privacy and ethics as well as studies of LA in informal
learning settings. However, 2013, saw an observed increase in the
use of third party and assessment data to build models that would
predict student retention or success. There remained a predomi-
450
LAK’19, March 2019, Arizona, USA
nance of studies in university settings though there was an emer-
gence of work in alternate education settings. Namely, this is re-
flective of research in K-12 education incorporating the ASSIST-
ments platform. However, these studies tended to remain at the
course and project level.
For 2014, the relationships between categories can be seen to be
relatively consistent with the prior year. Although, there was an ob-
served increase in the scale of studies undertaken (i.e. larger popu-
lation size) as well as a growing use of data from institutional sys-
tems such as the LMS, Student Information Systems (learning en-
vironments) and student assessment. The number of evaluation
studies was more prominent than prior years. This is an important
focus as LA research matures beyond exploration and development
towards evaluation and impact on theory and practice.
The pattern observed in 2013, 2014 was replicated in 2015.
However, there was an increase in the number of studies involving
a substantial population size (N>100). This lies in part to the num-
ber of MOOC studies undertaken. While the population size of the
research increased the studies were predominantly limited to single
MOOC instance. Research undertaken in “other education settings”
(outside the university context) also gained further significance.
Additionally, an increased use of third-party tools perhaps indicates
the growth of commercial LA products. The number of Evaluation
studies were consistent with 2014 and remained a minority of the
total works. Conceptual papers describing a proposed framework
or understanding a model further decreased from its peak in 2012.
An increase in the number of presented papers incorporating
data collected from survey instruments and self-reported data was
observed in 2016. Although these methods were employed in ear-
lier conferences and papers, there was a substantial increase in
2016. This may be reflective of an interest in studies of learner dis-
positions and cognitive psychology such as attention and mind
wandering, as well as a sign of the recognition that LA studies were
lacking the ‘social’ element of the socio-technical systems where
learning occurs. There was also a focus on mapping qualitative data
and self-reported psychological surveys to clickstream data. The
number of evaluation studies remained consistent.
For 2017 there was a rise in the number of evaluation studies.
This category is representative of more mature studies that seek to
assess and refine earlier forms of exploratory research. For exam-
ple, the evaluation of a predictive model and noted support inter-
ventions through field and case studies. Although an increase in
2017 was observed, for 2018 the quantity of evaluation studies de-
clined. The network structure linking relationships between student
focused studies that undertake predict/ build research in university
settings identified from 2013 has remained consistent over the
years. In order to evaluate the differences between the years with
respect to the first dimension of the ENA space (x-axis Figure 3),
we performed the non-parametric Kruskal-Wallis test. The test was
2(1, 7) = 15.61, p=0.03. However, the further post hoc analysis us-
ing Dunn’s test of multiple comparisons using rank sums, showed
that only the difference in ENA space between 2011 and 2015 was
statistically significant.
LAK’19, March 2019, Arizona, USA S. Dawson, et al .

Figure 4: Epistemic networks between the groups of coded dimensions. Each label is constructed using a letter to represent the five
coded dimensions (Focus - F, Purpose - P, Scale - S, Data - D, and Settings – St) and the subcategory (e.g., student, learning environ-
ment). Nodes are colored according to the dimension. X-axis represents the first dimension of the epistemic space (svd1), and the y-
axis represents the second (svd2) dimension of the epistemic space.

451
 Addressing the Impact of learning Analytics
5 DISCUSSION
This study identifies the extent learning analytics research, as
evidenced in LAK conference proceedings and JLA articles, is ad-
dressing the goal of developing insights into, and an understanding
of, the learning process. The findings indicate that the majority of
LA studies can be classified as exploratory research, focusing on
students enrolled in independent courses or small projects. The data
types tend to be derived from institutional technologies such as the
Learning Management System (LMS), Student Information Sys-
tems, and assessment data. Example studies include investigating
factors that influence student success outcomes leading to the de-
velopment of a particular predictive model or suggested interven-
tions and support actions. However, as the findings demonstrate,
there are relatively few studies to date that have attempted to extend
exploratory works to evaluate the results or suggested actions to
interrogate the proposed model and feedback to theory. The re-
search reviewed for this paper suggests the field has yet to under-
take a substantial engagement with what Hevner [21] described as
the rigor cycle, which would indicate development of the field and
growing impact on theoretical models. While the maturation of LA
has transitioned from conceptualization, it remains mired within a
phase of exploratory analyzes.
We argue that the field requires further impetus to facilitate the
transition from technical exploratory works to the evaluation of LA
actions within the broader socio-technical system LA operates in.
As illustrated in Figures 1 and 2, a focus on evaluation research can
fuel the development of new insights into theory and frameworks
to support the ongoing maturation of the field. These findings re-
flect Gasevic, et al’s [17] assertion that LA research has all too fre-
quently centered on the identification of attributes and behavioral
characteristics of students that are associated with discrete learning
outcomes. LA must extend beyond individual analysis and provide
rigor in evaluating assumptions, interventions and actions to refine
established or generate new theories of learning. If the factors un-
derpinning a predictive model of student success holds true, then,
neither educators nor students have sufficiently changed practice to
improve learning outcomes.
Our premise for this paper is that to date, LA has not, and will
not, make a significant impact at scale while research and deploy-
ment are undertaken at the level of an individual course instructor
or in small-scale experimental studies, in lieu of a more systems-
oriented approach. Further research is required for replication and
evaluation studies to refine, revise or establish new theories of
learning, adoption or design. To achieve this goal a systems orien-
tation is needed to identify how LA can improve student success in
complex organizational systems [25].
5.1 LA and Socio-technical Systems
Similar to information systems research or health sciences, LA
operates at the intersection of multiple disciplines in solving com-
plex problems that involve the confluence of social and technical
factors. Instead of improving the efficiency of organizational pro-
cesses or the effectiveness of health services, LA focuses on the
specific problem of understanding and optimizing factors that lead
to a successful educational experience for all learners [26]. At a
452
LAK’19, March 2019, Arizona, USA
simplistic level, this is a two-part problem requiring researchers to
first understand the learning process before it can be optimized.
However, the complexity of this inquiry is reflected in the existing
literature, as the majority of LA publications to date have failed to
reach the adequate maturity of the research rigor cycle to be able to
make a broader impact on the existing theories, frameworks, and
models (see Figure 4). For LA to reach its potential as a research
domain, it is necessary to untangle the complex interactions be-
tween social, technical, and environmental factors at the systems
level [10].
A student’s learning experience rarely starts and ends in one
course. An integrative, complex, and holistic view is needed to un-
derstand the dynamics outside of a specific course that influence
learning performance. However, the focus, purpose, and the scale
of the existing research, seem to be impacted by the current suite of
institutional technologies employed and the associated data sets
(Figure 4). The reviewed publications suggests that the abundance
of, and access to, technology-mediated learning data has encour-
aged the deployment and advancement of computational methods.
Furthermore, the access to large sets of institutional learning data
has also influenced the scoping of research problems and hence
overall direction and progression of LA.
Technologies and teaching practices can provide a rich data
source at the same time a limited source of data. For instance, the
reviewed papers demonstrate the heavy use of trace data collected
through various technologies such as Intelligent Tutoring Systems
(ITS) (e.g. ASSISTments) or Learning Management Systems
(LMS). These learning technologies produce large volumes of stu-
dent interaction data. While we are witnessing an ever-growing so-
phistication of analyses of these forms of trace data such as statis-
tical network analysis, soft clustering of student learning strategies
and transitions, the data lacks the personal, social and cultural in-
formation that also impacts on student learning. In short, the access
to the data obtained from these and similar systems can be auto-
mated and scaled, making such data readily accessible for the ap-
plication of statistical and machine learning methods. In contrast,
despite the recent advances in multimodal LA [3], data concerning
social and personal dimensions such as motivations, emotions,
health and culture are reliant on self-reports or collected from ex-
pensive and intrusive equipment. Even when physiological and so-
cio-emotional data are captured through technologies there are
challenges in synchronizing such discrete data sources. Although
there is a lack of richness in trace data, the reviewed works demon-
strate that LA research has well developed numerous predictive
models of student success. The capacity to predict student out-
comes from behavioral and assessment data affords educators early
and timely opportunity to intervene [17]. The ability for early inter-
vention to aid student success drives LA practice and research, re-
flecting a pursuit to understand both learning process and context
with the intent to improve student outcomes. Hence, the ability to
generate actionable, timely formative and personalized feedback is
a noted goal of LA research and practice. However, to achieve an
understanding of the learning process requires more than the gen-
eration of interventions. The notion that such actionable intelli-
gence is generated through LA is central to much of the existing
research. Greller and Drachsler [19], and Chatti, et al. [5], proposed
 LAK’19, March 2019, Arizona, USA
a set of dimensions that influences both LA uptake from a socio-
technical and organizational perspective. Clow [6], and Campbell
and Oblinger [4] were more direct to articulate the process of LA
as a research cycle involving discrete but interrelated activities.
Clow’s [7] LA cycle is the most generic. The cycle commences
with learners – as students undertake various course activities, they
generate data that is then analyzed to provide the basis for some
action. Campbell and Oblinger [4] stress the importance of includ-
ing an additional step involving the testing of the prescribed support
actions to further revise and refine the findings
The ENA results (see Figure 4) illustrate that there is predomi-
nance of small-scale exploratory studies and a deficit of research
undertaking evaluative studies that attempt to verify findings and
provide explicit feedback to revise, refine or develop new theories
and models (see Figure 2). This requires researchers to shift from
small-scale research projects to areas that address and target sys-
temic impact and outcomes. The achievement of this aim will re-
quire a foundation of, and culture for, supporting replication stud-
ies. Such an approach is difficult to achieve as research into social
phenomena are often contradictory [35]. Results identified in one
study are frequently contrasted by others. Learning analytics is no
exception [1]. What has been demonstrated to improve or predict
student success in one instance, classroom or institutional context,
may or may not be replicated in alternate studies. An explanation
for the lack of replicable outcomes lies in the dynamics of the sys-
tems within which LA operates. This is not to say that the research
reviewed in this paper is not rigorous, but rather that researchers
frequently adopt alternate theories and models and statistical ap-
proaches which in turn influence interpretations and outcomes.
Hence, achieving the goal of improving learning processes and im-
pacting outcomes can be difficult as much of the data being col-
lected is not only limited in scope, but also fails to account for so-
cial, cultural, and contextual variables, requiring researchers to
adopt proxies related to grades and completion rates, rather than
more sophisticated approaches to optimize learning for all learners
[23]. The research cycles provide a high level framework for as-
sessing LA work to date. The results highlight the influence that
surrounding environmental factors and access to data plays on the
types of problems that are being explored and methodologies
adopted. Due to these constraints, LA as a field has been stifled in
terms of addressing complex social challenges and has not devel-
oped the types of evaluative models that are required for ongoing
impact and growth of the field.
5.2 Implications
Learning analytics like social science cannot be reduced to simplis-
tic analytical problems. In recent years there has been momentum
in developing more complex models of research and thinking that
utilize simulations, alternate agents, biological networks, adjacent
possibilities and evolutionary pathways to name but a few. In es-
sence, these approaches aim to identify how the multitude of inter-
connected nodes within a system impact and influence each other.
These relationships are not necessarily linear nor predictable. Yet,
as Herbert Simon [32] remarked, an observed theme in complex
systems is the concept of hierarchy. That is actors within a system
are often comprised of inter-related sub-systems that can be seen to
453
S. Dawson, et al .
fall into a hierarchical structure. For LA this may also provide a
useful model for thinking how learning manifests within education
settings and how LA can be deployed for deep impact in education
organizations. For LA, the research is yet to interrogate how actors
in complex systems interact – both in terms of a) the learning pro-
cess and b) the application of LA.
The findings in this study raise several important implications
for LA research and practice. Research does not operate in isolation
of other systems nor is it de-contextualized from practical applica-
tion. All research is impacted and influenced at numerous levels
such as politics, funding sources from international, national and
local sources, availability and access to data, opportunities and lim-
itations of methods and access to expertise through our own collab-
orative networks. As a practical example of the impact of external
factors on the nature of LA research, consider the influence of re-
duced government funding to education and establishment of reten-
tion as a performance metric in driving a research focus towards
models of student success and retention. To date though, research
consideration for how LA manifests within a complex system has
been minimal.
Education is highly representative of a complex system residing
within multiple broader system structures. Our models of teaching
from on campus to fully online, the diversity of students, the tech-
nical ecosystem required to support the myriad of academic prac-
tices and policies, local and global economic forces and societal
expectations all play a role in shaping how education functions, pri-
oritizes actions, and responds to such environmental pressures. The
inclusion of new practices or processes within such a complex sys-
tem requires highly skilled leadership, substantial and persistent ef-
fort and organizational agility [31]. Of equal importance in driving
the broader impact of LA is an understanding of how these systems
dynamics influence the adoption of insights gained from more gran-
ular LA research such as cognitive profiles of learners, predictive
models of learner success, and feedback and support interventions.
Over the past decade there have been a number of LA institu-
tional adoption models proposed that attempt to reflect the role data
and data analytics plays in education systems [see for example: 2,
8, 13, 19]. In particular, Greller and Drachsler [19] argued that any
successful large scale LA instantiations must capture the interre-
lated nature that operates between the identified dimensions that
can mediate adoption. Similarly, Ferguson, et al., [13] proposed the
use of the RAPID Outcomes Mapping Approach (ROMA) as a
model to identify the factors influencing adoption and how these
factors and agents intersect within a dynamic system. Despite the
number of proposed models to there remain very few large scale
LA instantiations. LA research is clearly multi-disciplinary and de-
ploying a LA initiative similarly comprises numerous interactions
with the many sub-units that make up an education organization.
For example, LA adoption will involve, technical systems, aca-
demic systems, teaching and workload models, ethics and privacy,
assessment policies, retention practices, communication processes
and so on. The ability to call on replicated studies that demonstrate
the impact of LA can only serve to aid uptake. However, without a
purposeful intent to move LA works from decontextualized and ex-
ploratory settings to replicated and evaluation, there will not be a
substantial impact on theory, adoption, nor sector wide transfor-
 Addressing the Impact of learning Analytics
mation. This problem is not solely borne by LA. Social science re-
search has long been confronted with issues of replication and val-
idation of research findings [35]. We can collectively begin to
tackle this challenge through deeper collaborations and investiga-
tions into the types of questions posed, and data and methods ap-
plied to LA. Incorporating for example, enhanced data sets beyond
technology mediated trace data to embrace other design, cognitive
and socio-emotional dimensions at scale.
The findings from this study reflect the complexity in address-
ing the optimization of learning for a field established from the con-
vergence of multiple disciplines. However, the exploratory studies
undertaken can and do provide a substantial foundation in order to
transition to more evaluation and replication studies. Such a step
may require stimulus from existing research organizations, funding
agencies, collaborations and the active promotion of established
works such as the LACE initiative [12]. Collectively, the field must
pursue and engage in the complexity of these systems and in ad-
vancing understanding of how principles of dynamic systems influ-
ences adoption.
6 CONCLUSIONS
While LA research has not yet reached its potential, it is advancing
and is on the right path to fulfil its stated promise of generating
sector wide transformations. The realist in this situation would flag
this will only come to fruition through understanding and engineer-
ing the influential actors that reside in the education system. Tar-
geting research towards LA as a complex model may help craft the
understandings required to transition from the current deep insights
about learning behavior to knowledge about the application of LA
for action and impact within the hierarchical sub-units of a complex
education system. Now is the opportune time to achieve this goal
by engineering and influencing the direction of our future research
to consider LA within a more holistic model. That is, to undertake
research that aims to understand the learning process as well as op-
timizing that process for all students across all contexts. Research
directions and priorities can be influenced. The strength of our work
from individual researchers to international collaborations and
noted research organizations (such as SoLAR and IEDM) can col-
lectively present a powerful voice to generate the opportunities to
have a profound influence on the types of research that are funded,
supported and ultimately undertaken in order to encourage new di-
rections in the field. Discussions with granting agencies and direct
funding of PhD and early career scholarships and research grants
can provide the impetus to expand the level of LA research into
evaluation and replication studies to fuel the knowledge base and
revise, refine and contribute new theory. This will require a pur-
poseful shift to target, promote and enact such research.
REFERENCES
[1] Andres, J. M. L., Baker, R. S., Siemens, G., Gašević, D. and Spann, C.
A. (2017). Replicating 21 findings on student success in online learning.
Technology, Instruction, Cognition, & Learning, 10, 4, 313-333.
[2] Bichsel, J. (2012). Research computing: The enabling role of
information technology. EDUCAUSE Center for Applied Research.
454
LAK’19, March 2019, Arizona, USA
[3] Blikstein, P. and Worsley, M. (2016). Multimodal learning analytics
and education data mining: Using computational technologies to measure
complex learning tasks. Journal of Learning Analytics, 32, 2, 220-238.
[4] Campbell, J. and Oblinger, D. (2007). Academic analytics.
EDUCAUSE Quarterly Magazine, 2007, 25th October.
[5] Chatti, M. A., Dyckhoff, A. L., Schroeder, U. and Thüs, H. (2012). A
reference model for learning analytics. International Journal of
Technology Enhanced Learning, 4, 5-6, 318-331.
[6] Clow, D. (2012). The learning analytics cycle: closing the loop
effectively. In Proceedings of the Proceedings of the 2nd International
Conference on Learning Analytics and Knowledge (Vancouver, British
Columbia, Canada). ACM.
[7] Clow, D. (2013). An overview of learning analytics. Teaching in
Higher Education, 18, 6, 683-695.
[8] Colvin, C., Rogers, T., Wade, A., Dawson, S., Gašević, D.,
Buckingham Shum, S. and Fisher, J. (2015). Student retention and
learning analytics: A snapshot of Australian practices and a framework
for advancement. Australian Office for Learning and Teaching, Sydney,
Australia.
[9] Dawson, S., Gašević, D., Siemens, G. and Joksimovic, S. (2014).
Current state and future trends: a citation network analysis of the learning
analytics field. In Proceedings of the Fourth International Conference on
Learning Analytics And Knowledge (LAK '14) (Indianapolis, USA). ACM,
New York, NY, USA, .
[10] Dawson, S., Poquet, O., Colvin, C., Rogers, T., Pardo, A. and
Gasevic, D. (2018). Rethinking learning analytics adoption through
complexity leadership theory. ACM, NY, USA, City.
[11] Ferguson, R. (2012). Learning analytics: Drivers, developments and
challenges. International Journal of Technology Enhanced Learning, 4,
5/6, 304-317.
[12] Ferguson, R. and Clow, D. (2016). Learning analytics community
exchange: evidence hub. In Proceedings of the Sixth International
Conference on Learning Analytics & Knowledge (Edinburgh, UK). ACM.
[13] Ferguson, R., Macfadyen, L., Clow, D., Tynan, B., Alexander, S. and
Dawson, S. (2014). Setting Learning Analytics in Context: Overcoming
the Barriers to Large-Scale Adoption. Journal of Learning Analytics, 1, 3,
120-144.
[14] Ferreira, R., Kovanović, V., Gašević, D. and Rolim, V. (2018).
Towards Combined Network and Text Analytics of Student Discourse in
Online Discussions. In Proceedings of the International Conference on
Artificial Intelligence in Education. Springer, Cham.
[15] Fleiss, J. L. and Cohen, J. (1973). The equivalence of weighted kappa
and the intraclass correlation coefficient as measures of reliability.
Educational and Psychological Measurement, 33, 613–619.
[16] Freelon, D. (2013). ReCal OIR: Ordinal, interval, and ratio intercoder
reliability as a web service. International Journal of Internet Science, 8, 1,
10-16.
[17] Gašević, D., Dawson, S. and Siemens, G. (2015). Let’s not forget:
Learning analytics are about learning. TechTrends, 59, 1, 64-71.
[18] Gasevic, D., Joksimovic, S., Eagan, R. and Shaffer, D. W. (2018).
SENS: Network analytics to combine social and cognitive perspectives of
collaborative learning. Computers in Human Behavior.
[19] Greller, W. and Drachsler, H. (2012). Translating learning into
numbers: A generic framework for leanring analytics. Educational
Technology & Society, 15, 3, 42-57.
[20] Haythornthwaite, C., de Laat, M. and Dawson, S. (2013).
Introduction to the Special Issue on Learning Analytics. American
Behavioral Scientist, 57, 10, 1371-1379.
[21] Hevner, A. R. (2007). A Three Cycle View of Design Science
Research. Scandinavian Journal of Information Systems, 19, 2, 87-92.
[22] Hevner, A. R., March, S. T., Park, J. and Ram, S. (2004). Design
Science in Information Systems Research. Management Information
Systems Research Center, 28, 1, 75-105.
[23] Joksimović, S., Poquet, O., Kovanović, V., Dowell, N., Mills, C.,
Gašević, D., Dawson, S., Graesser, A. C. and Brooks, C. (2018). How do
we model learning at scale? A systematic review of research on MOOCs.
Review of Educational Research, 88, 1, 43-86.
 LAK’19, March 2019, Arizona, USA S. Dawson, et al .

[24] Knobbout, J. and van der Stappen, E. (2018). Where Is the Learning
in Learning Analytics? A systematic literature review to identify measures
of affected learning. In Proceedings of the European Conference on
Technology Enhanced Learning. Springer, Cham.
[25] Macfadyen, L. P., Dawson, S., Pardo, A. and Gaševic, D. (2014).
Embracing big data in complex educational systems: The learning
analytics imperative and the policy challenge. Research & Practice in
Assessment, 9, 17-28.
[26] Mangaroska, K. and Giannakos, M. N. (2018). Learning analytics for
learning design: A systematic literature review of analytics-driven design
to enhance learning. IEEE Transactions on Learning Technologies.
[27] Ognjanovic, I., Gasevic, D. and Dawson, S. (2016). Using
institutional data to predict student course selections in higher education.
The Internet and Higher Education, 29, 49-62.
[28] Shaffer, D. W. (2006). Epistemic frames for epistemic games.
Computers & Education, 46, 3, 223-234.
[29] Shaffer, D. W., Collier, W. and Ruis, A. R. (2016). A tutorial on
epistemic network analysis: Analyzing the structure of connections in
cognitive, social, and interaction data. Journal of Learning Analytics, 3, 3,
9-45.
[30] Shaffer, D. W., Hatfield, D., Svarovsky, G. N., Nash, P., Nulty, A.,
Bagley, E., Frank, K., Rupp, A. A. and Mislevy, R. (2009). Epistemic
Network Analysis: A Prototype for 21st-Century Assessment of Learning.
International Journal of Learning and Media, 1, 2, 33-53.
[31] Siemens, G., Dawson, S. and Eshleman, K. (2018). Complexity: A
Leader’s Framework for Understanding and Managing Change in Higher
Education. EDUCAUSE Review, 53, 6.
[32] Simon, H. A. (1962). The Architecture of Complexity. Proceedings
of the American Philosophical Society, 106, 6, 467-482.
[33] Thaler, R. H. (2016). Behavioral economics: past, present, and future.
American Economic Review, 106, 7, 1577-1600.
[34] Torraco, R. J. (2002). Research Methods for Theory Building in
Applied Disciplines: A Comparative Analysis. Advances in Developing
Human Resources, 4, 3, 355-376.
[35] Watts, D. J. (2017). Should social science be more solution-oriented?
Nature Human Behavior, 1.

455