finlandmath.com eduten.

com

Finland Math Guide

to Learning Analytics
Part 1: Introduction to Learning Analytics
Table of contents:

01 10
Introduction Learning
Analytics Process

02 12
What Do We Mean Examples of
by Learning Analytics? Learning Analytics

04 19
Stakeholders in the Considering
Learning Analytics Process Ethics and Privacy

Introduction
Learning analytics is one of
the current trends in modern
education. According to the
advocates, learning analytics:
• can improve learning results of
students
• enable early interventions for
learning problems
• reduce the workload of teachers
• improve the transparency of
learning for all involved in the
process.
In fact, learning analytics is often
seen as a potential solution for
improving PISA scores, and, in
general, the overall quality of
education. However, most of the
people actually involved in the
process know very little about
01
learning analytics, let alone being
able to utilize it successfully. As
Finland Math and its principles
are well connected to learning
analytics, we decided that it is time
to publish a practical guide about
analytics. This is part 1 of 2 of our
series of guides, meant to help you
to understand learning analytics.
WE ANSWER QUESTIONS SUCH AS:
What does it mean?
Who are the people
involved?
What are the benefits?
What kind of ethical
issues arise?

We’ve also collected several
examples about learning analytics
at different levels. This guide is
written for anyone interested
in learning analytics, including
teachers, students and school
administrators. We strongly believe
that all people involved in the
learning process are also the key
people in learning analytics.
The second part of the guide, to be
published later this year, will focus
on the intersection of learning
analytics and the principles of
Finland Math. It’s designed to help
you understand how learning
analytics is easily utilized in the
classroom. We’ll also focus on
different levels of learning analytics
and take a closer look at the data
sources. If you like the first part, you
won’t want to miss the second one.
These guides are based on
practices we’ve collected while
conducting experiments and
research about learning analytics
for more than 15 years in the
University of Turku, Finland and in
the University of Helsinki.
We hope you enjoy reading
the guides and find learning
analytics to be a valuable asset
for enhancing the possibilities of
Finland Math even further. We start
this guide by looking at the multiple
definitions of learning analytics
provided by scientists - if you’re
eager to see what analytics in
action means, you may want to skip
to the latter sections.
However you read this guide, have
fun!
What Do We Mean by
Learning Analytics?
As a concept, learning analytics
is both old and new. The core idea
of analysing information (i.e. data)
collected from learners in form of
assessment is at least as old as the
concept of formal education. But
the modern definition of learning
analytics relies more on the current
concept of data analytics. In fact,
some sources define learning
analytics as data analytics done
in the learning domain. But others
(including us), argue that it is much
more than that.
To fully understand the concept,
let’s take a look at some of the ways
to define learning analytics in a
scientific context.
02
Early definitions of learning
analytics described it as:
The use of data and models
to predict student progress
and performance (or success),
and the ability to act on that
information.
Although the definition does
capture the core processes of
learning analytics, it was deemed
as insufficient quite early (for
example by the founding president
of the Society for Learning Analytics
Research, George Siemens). Some
other authors have argued that the
words “progress”, “performance”
and “success” can be seen as

ambiguous: different students have
different goals and they follow
different paths to reach those
goals.
The reference model designed by
Chatti et al. (2012) provides quite
a holistic view on the concept of
learning analytics. In the model, the
analytics is defined as an umbrella
that for example contains all areas
of technology enhanced learning
(TEL). Some of these areas form the
basis of what should be understood
as learning analytics:
ACADEMIC ANALYTICS:
Application of business intelligence
analytics in the higher education
systems, i.e. the usage of data-
analytics in educational context.
EDUCATIONAL DATA MINING:
Mining and exploring data that has
been gathered from educational
context. The educational data is
often unique to all other data.
PERSONALIZED ADAPTIVE
LEARNING:
Adaptive learning systems can
change their content (exercises
or other materials) based on the
user’s performance, preferred
learning style or misconceptions.
In contrast to adaptivity, term
adaptability is used to describe
learning environments which can
be changed by the user, but which
do not automatically adapt to
user’s needs.
03
RECOMMENDER SYSTEMS:
Systems that collect data from user’s behaviour and
based on the analysis of the data, recommend actions
or items that are the best fit to the behaviour.
There are other definitions of learning analytics, such as a “model
that is based on educational data and statistical models” or as an
“intersection of data science, theory and design”. Still, all definitions
seem to somewhat agree on the main principles: collecting and
analyzing educational data and acting according to analysis. The
1st International Conference on Learning Analytics and Knowledge
provided perhaps the most conclusive definition so far:
Learning analytics is the measurement, collection, analysis
and reporting of data about learners and their contexts, for
purposes of understanding and optimizing learning and the
environments in which it occurs.
Not much to add there, we think.
 04
Stakeholders in the Learning
Analytics Process
Before considering concrete examples of analytics on different levels, let’s discuss
the roles of people involved in learning analytics. In learning analytics, we can
isolate at least three different groups of stakeholders. Although the primary goal
of all groups is the same (improving the quality of education, that is), there are
different nuances and different means for reaching the goals.

Group 1: Students
Students are at the heart of the
analytics process. The data collected
in learning analytics is mostly
connected directly to students’
actions. From the students’ point of
view, the goals of analytics can be
split into three main areas.
interested in improving their learning
results. Again, the motivation behind
this is often both external and
internal: better grades mean better
changes in education and work
opportunities later, but better learning
can be a reward itself.

01 02
First, like all stakeholders, the students The second possible benefit of
are (or at least they should be) analytics for students is improving

the learning experience. By analyzing
the process, it’s possible to recognize
the culprits and try to overcome
them by changing the process. In
a learning context, the improved
experience can mean for example
a better working environment, less
stress (for example in the form of
decreased math anxiety), improved
motivation or shorter school days. It
should be noted that the goals are
not separate: an improved learning
process is very much connected to
better learning performance and
better results in general.
03
The third benefit of analytics
for students is the improved
understanding of their own learning,
or metacognition. If the results and
visualizations provided by analytics
are offered to students, they enable
new means of self reflection and
05
can drastically improve students’
understanding of the whole
learning process and their role in
it. Emphasizing the central role of
students in the process is highly
important, and also one of the key
principles of Finland Math. Moreover,
better understanding of the whole
process can lead to growth mindset,
another key principle of Finland Math.
In our opinion, the role of the students
in learning analytics should be a lot
bigger than what it is at the moment.
A major share of the data that is
currently served for teachers only,
could be offered to students as well.
Let’s discuss this further when we talk
about the examples and the privacy
issues of analytics.

Group 2: Teachers
Typically, learning analytics is
considered as something consumed
mainly by teachers and conducted
01
primarily by researchers. Although
this is not (and shouldn’t be) the
case anymore, teachers are still the
group that can conveniently perform
analytics. They traditionally have
the best access to data and obvious
ways to utilize it in their everyday
work.
However, many teachers lack the
skills and/or the time to properly utilize
analytics - one of the main reasons
02
we are publishing this guide.
Learning analytics is often tightly
connected with assessment and
evaluation. Analyzing learning data
collected from students’ activities
can indeed improve the quality of
06
assessment, but there are other
benefits for teachers too.
First, analytics provides an excellent
tool for critically assessing one’s
own teaching process. For example,
data collected from students’ time
allocation between different tasks
can help to identify too difficult or
too easy tasks. Concrete example
of utilizing learning analytics in
assessing the teaching process is
provided in the Examples section of
this guide.
The second benefit of utilizing
analytics is the better allocation
of resources. Mainly, this means
that a teacher can utilize analytics
to identify the students in need of
assistance.
 07
In a perfect world, we would have one teacher for each student. In real life,
several students compete for teacher’s attention. For best allocation of time,
it is important to know which students require the help the most.

Although technology enables automatization of some assistance processes,

we are still far away from having AI powered teaching systems that can truly
assist their students with the same expertise and empathy that humans can.

TO SUMMARIZE:

Teachers want to understand their students as well as possible, and

learning analytics can be the perfect solution for this. If this can be done
with fewer exams and less paperwork than before, who would object?

Group 3: Administrators
The third interest group is school administrators. This group includes all
decision makers. The actual demography of the group varies strongly
between different countries and cultures, nevertheless, the stakeholders in
this group are the ones responsible for resources and their allocation, which
in practice often means for example hiring teachers and acquiring materials.

From an administrators’ point of view,
learning analytics has much to offer.
Collecting and analyzing versatile
data about learning and teaching
processes enables data-driven
decision making.
For example, the administrators can
provide more teaching resources
08
Teachers can sometimes find
learning analytics performed by
Primary goals of learning analytics
administration intimidating, as the
analytics can be used to measure
for different stakeholders.
their performance. While learning
analytics can pinpoint the need for
more resources, the results of the
analysis can likewise reveal opposite
cases. STUDENTS TEACHERS ADMINISTRATORS

to certain groups, if their learning

performance, motivation or However, the data-driven decision Improve student Early detection of Data-driving
well-being seems to be sub-par making should be seen as a positive perfomance, problems. decision making:
according to analytics. experience. properly utilized, it will motivation and better allocation
Better
even out the workload between well-being. of resources with
understanding of
Better learning performance is teachers. This will also enable equal better results.
learning process.
naturally one of the biggest goals chances for better guidance for all
of administrators, as higher grades students. Evaulation of

attract more (and better) students. teaching process.

As we have discussed before,

clever use of learning analytics can
potentially lead to improved learning The one shared goal is improved learning
performance. performance of all students.

What About Parents?
In addition to the three main groups
of stakeholders, there is one group
that’s easy to forget, but equally
important. That is, the parents.
Parents as a stakeholder group, share
some properties from both teachers
and administrators. Parents’ main
focus (or at least should be) is on
student wellbeing and improved
learning performance.
But the solutions to potential
problems may not be as obvious
as they are for teachers. Parents
are often not pedagogical experts
in terms of formal education, while
their responsibility in education
is in different areas. Hence, close
collaboration with parents and
teachers is important.
09
Nevertheless, there are plenty of
examples of learning analytics
we can provide for parents. The
most important ones are of course
the visualizations of students’
progress in different topics as well
as their possible problems and
misconceptions. One particularly
useful area is time usage. If students’
time spent among different topics
and concepts is visualized for
parents as well, they can make better
decisions based on the data. Not only
to allocate more time for studying,
but often the opposite. Learning
happens while resting, and student
well-being and mindset are highly
dependent on their wellbeing. Hence,
it’s important to ensure they have
enough free time as well.

Learning Analytics Process
The process of utilizing learning analytics can be divided into three phases:
data collection, analysis and reporting, and pedagogical interventions.
10
The process can also be seen as cyclic, as it can be improved by learning from
the successes and mistakes. Note the differences in two images: in the cyclic
illustration interpretation is separated into its own phase while in the other one
it is included in the interventions step.

PEDAGOGICAL INTERVENTIONS
Capture

Refine Reporting

ANALYSIS AN D REPORTING

Act Interpretation
(intervention) / Prediction
COLLECTING DATA

Learning analytics as a sequential three-step process. Learning analytics as cyclic five-step process.
 11
Conducting the steps is not always easy. In fact, there are several well-known
difficulties associated with all the steps. Some key-actions and related possible
problems are described in the table below.

CAP TURE REPORT INTERPRETATION ACTING

What data? To whom? What variables Who acts?

and techniques?
How often? How often? When?
What can be
Where to store? What is deduced from How?
reported? data?
In what form What are the
collected? How visualized? How to ensure expected
correctness? results?
Who collects?
What are the What are the
Safety and error margins? observed
privacy results?

As explained in detail later in this guide, utilizing a comprehensive tool, such as

Eduten Playground, can help you overcome many difficulties associated with
different stages of the analytics, as many of these issues are already solved in
the tools.
 12
Examples of • Exercises,
• Other tasks,
• Course assignments and
• Lecture attendances.
Learning Analytics • Seminar work,

To further illustrate what we mean by learning analytics, let’s consider some
examples. We have selected four very different examples from different stages
and school levels to illustrate versatile use of analytics in education. For each
example, a short description on actions of all four stages (capture, report,
interpretation and acting) is presented.
An example of data collected in a course is visualized in the image below.
The students are color coded by their final grade. In this case, we are mostly
interested in the students who are not going to pass the course.

Example 1: Predicting Learning Results. best

total points
As the first example, let’s see how the data gathered from learning results can
be used to build a model, which can then be used to predict the results in the
latter instances. worst

fail

First, let’s consider a typical eight-week long university course. The level and
the topic are not important, any course (or similar unit) at any level will do, as
tasks completed
long as all actions the students take during the course can be recorded. In
this case, a digital education tool is used to keep track of students’ scores and Visualization of course performance at the end of the course.
The students are classified into groups based on their final grades.
submissions in all course areas, including for example:

What is important is that by using the data and continuous assessment we
can visualize the students’ current progress in the course in realtime. When
we combine the data of their progress day by day with the knowledge about
their final grades, we can build a model that predicts students’ progress before
they have completed the course. In the future instances of the course it is then
possible to get somewhat reliable results about the final grades as early as in
the second or third week of the course.
best
total points
worst
fail
tasks completed
Visualization of course performance at the end of the second week.
Students who will likely not pass the course are represented as red dots.
13
In the image above, the state of a course is visualized at the end of the second
week. By using the data gathered before, and the model built upon that data,
we can reliably predict as much as 80 or 90 % of the students who are not
going to pass the course. This opens up new possibilities for interventions to
get the students back on track. The actual numbers and percentages are
dependent on the amount of data and the methods used to analyze the data.
CAPTURE:
Data collected automatically with digital learning environment.
REPORT:
Visualization, built for teacher(s) and/or student counselor.
INTERPRETATION:
Locate the dots indicating students at risk of failing.
ACTING:
Contacting students, identifying students who want to stay at course but
are failing, supplementary exercises and assistance.
Our other example of predicting learning outcome is from a wider concept.
Instead of trying to predict the outcome of a single course instance, we can
 14
use data gathered from the whole curriculum. When there is enough data, it is
CAPTURE:
again possible to build a model that predicts students’ success at the end of
Data from study registry (several years).
the school year, months before that.

REPORT:
Visualization, built for student counselors or administrators.

INTERPRETATION:
Locate the dots indicating students at risk of failing.
credits completed

ACTING:
Contacting students, identifying students who want to stay at course but
are failing, supplementary courses and assistance.

cources completed

Example visualization of students’ collected credits and completed courses. In

the example, the red dots represent students who are likely not going to reach
the minimum required credit limit by the end of the school year.

In the example visualization the red dots represent students who are unlikely to
reach the required study credits per year, and the blue dots show the students
who are likely going to reach the limit. The model used in the example is based
Example 2:
on the number of credits collected and the grades obtained. Teacher Self-Reflection.
 15
Example 2: Teacher Self-Reflection. 10

7
Next, let’s see an example of how teachers can utilize learning analytics to
6
improve the quality of learning experience and the effectiveness of teaching. 5

3
In this example, we utilize data collected automatically by Eduten Playground,
2
but similar data collected by any means can be utilized similarly. (Although 1

0
collecting data from students’ time usage might be difficult without digital

14
10

15
12

13
5
4

8
2

11
9
7
1
se

se
se

se

se
se

se
se

se

se

se

se

se
se

se
ci

ci
ci

ci

ci
ci

ci
ci

ci

ci

ci

ci

ci
ci

ci
tools).

er

er
er

er

er
er

er
er

er

er

er

er

er
er

er
ex

ex
ex

ex

ex
ex

ex
ex

ex

ex

ex

ex

ex
ex

ex
The visualization seems to hint, that exercise 7 was way easier than the others,
In particular, we are going to utilize three different variables: time usage, score and exercises 4, 5 and 9 were more difficult than the others.
and number of times the exercises were submitted for automated evaluation.
Let’s look at the average time used and the number of times the exercises were

The idea is to enable self-reflection by observing exercises connected to the submitted next.

lesson. Now, whether the exercises were completed at school or as homework 350

is not important, as long as they are about the topic that was taught during the 300

250
lesson observed. The idea is that the exercises prepared for the lesson are all
200

about different sub-topics. 150

100

50
Now, by observing the individual exercises we can analyze how well different 0

sub-topics were learned. Let’s look at the average scores collected from each

15
10

14
13
12
4

8
5

6
3
2

11
9
7
1
se

se
se

se

se
se

se

se

se

se

se

se

se
se
se
ci
of the submissions first:

ci
ci

ci

ci
ci

ci

ci

ci

ci

ci

ci

ci
ci
ci
er

er
er

er

er
er

er

er

er

er

er

er

er
er

er
ex

ex
ex

ex

ex
ex

ex

ex

ex

ex

ex

ex

ex
ex

ex

These visualizations somewhat confirm our previous findings, but when
compared to previous figure, there seems to be some inconsistencies. The best
visualization from this data comes when we combine all variables into same
figure:
500
16
scores. So, what does this mean? First, we can use the information to analyze
the difficulty level of individual exercises.
Second, if we assume that the exercises are on the same level of difficulty, we
can use the results to evaluate students’ knowledge on different subtopics.

450
This way, it is possible to determine the weaknesses in our teaching and
400
improve the overall quality of education.
350

300

250
CAPTURE:
200
Data collected automatically with digital learning environment.
150

100
REPORT:
50
Visualization, built for teacher.
0

14

15
10

13
12
4

8
6
3

11
5

9
2

7
1
se

se

se
se
se

se

se

se
se

se

se

se

se
se

se
ci

ci

ci
ci
ci

ci

ci

ci
ci

ci

ci

ci

ci
ci

ci
INTERPRETATIOIN:
er

er

er
er
er

er

er

er
er

er

er

er

er
er

er
ex

ex

ex
ex
ex

ex

ex

ex
ex

ex

ex

ex

ex
ex

ex

Identifying topics with abnormal time to score ratio.

It’s quite easy to see that exercise 7 took a lot of time to complete, but probably ACTING:
was not that difficult (as the number of submissions and the average score Evaluating and modifying materials, topics and methods.

are quite high). Moreover, exercises 4, 5 and 9 seem to be the most difficult
ones, while 1 and 8 were probably easier to complete. It should be noted that all
exercises are scored in the same scale, which is a prerequisite for comparing
 17
Example 3: Student Perceptions connections between individual items. For this, correlations between items are
found. A correlation means a mutual relationship or connection between two
and Wellbeing. things. This is illustrated in the image below.

As the next and final example, let’s see how survey data can be analyzed to
draw more information from the results. Our example survey is about study

I get enough sleep at night

habits of students. By study habits, in this context, we mean things that are
not directly connected to formal education, but still have an effect on learning
experience and results.

Examples of the things included under the topic include:when students

complete their homework, how much they sleep at night, and what their social
connections are like. The survey items are typically presented as claims (such
I do my homework immediately after school
as “I sleep at least 8 hours per night during school week”), and the answers are
given in a Likert scale of 1 to 5 , where 1 represented “completely disagree” and 5
“completely agree”. The figure shows the relationship of answers to two questions:

Do you do homework straight Do you get enough sleep

Typically, a “conventional” analysis is first done to each of the questions
away after the school ends? at night?
separately. This includes calculating the average and standard deviation for
each of the items.
The visualization of example data illustrates the relationship between the two
However, from learning analytics’ point of view, we can extract more (and questions. By connecting individual variables like this, we can create a network
more interesting) information from the survey answers by finding out the (or a mesh) where the items are connected with red or green connectors,

illustrating either negative or positive correlation between the items. This kind
of network can be extremely useful when trying to improve students’ mindset
and mental and physical wellbeing.
An example network illustrating the correlations between individual questions in a study
habits survey. Green color indicates a positive correlation, and red color a negative one.
The thickness of the line denotes the strength of the correlation
18
CAPTURE:
Data collected via surveys.
REPORT:
Visualization, built for all stakeholder groups.
INTERPRETATION:
Locate positive and negative correlations.
ACTING:
Improving student grouping with formal and informal methods,
discussions. For example by enabling student group activities by offering
department funding, premises and other support.
What About Digital Education Tools?
Digital tools are definitely among the most important sources of data for
learning analytics. However, unlike some of the previous examples, we do not
need to analyze the data ourselves, as the digital tools often have dedicated
analytics built inside them. Moreover, digital systems can potentially collect
huge amounts of data about learners. Analyzing this kind of data can provide
useful information about individual learners and about learners collectively.
 19
In some cases, we can combine the data (for example to find out how the data Different areas have different rules and laws for privacy, so it is difficult to
of an individual student relates to the average of a larger number of similar provide generally binding advice. However, there are some general guidelines
students). you should follow when conducting any learning analytics experiments:

Examples of analytics in digital education tools are provided in part 2 of the

guide, where we discuss the relationship of learning analytics and Finland Ensure that the parents are well informed about the experiments, and if
required, get a formal permit from them.
Math.

Take the students as stakeholders into the whole process, whenever

Considering
possible. As we have discussed before, presenting visualizations and
other analytics results to students is often a good idea and can assist

Ethics and Privacy

in their self-reflection. This can lead to better mindset and to improved
learning experiences.

The data used in learning analytics
is usually sensitive and confidential.
Since there are often under-age
children involved, we should be
limitations and rules set by laws
and regulations (such as the
General Data Protection Regulation
of European Union or local data and
Make the whole process as transparent as possible. There is (usually)
no need to hide details from students, so tell them what kind of data is
collected, how it is collected, how it is analyzed and visualized and how
and to whom it is reported.

even more cautious when handling privacy laws). Second, whenever

the data. There are two kinds of
issues that should be properly
addressed when analyzing
learning data. First, there are the
dealing with issues connected to
When analyzing the data, anonymize (or pseudonymize)* the data as
education, it is necessary to follow
early as possible. In addition to providing security, this can make the
strict ethical guidelines. Almost
assessment less biased.
always, these two intersect.
 20

ANONYMOUS OR PSEUDONYMOUS DATA?

The terms anonymous and pseudonymous are used quite

In Eduten Playground we have paid special attention to data security and
often when discussing data and learning analytics. In this
ethical considerations. This means that if you adapt Finland Math and the
context, anonymous data means that there is not any
learning analytics feature in Eduten Playground, you can trust that the
identifying information (i.e. student’s name, email address,
utilization is based on the best and the most secure practices. Since Finland
social security number or any information that can be used
Math is strongly connected to learning analytics, we will do our share of
to connect student’s personal information with the data)
bringing together students, teachers and administrators to learn mutually
and no such information can be attached to the document.
about learning analytics in the future as well.
Pseudonymous data means data that is anonymized, but
there is a way to connect the data points to the original user.
This information is usually accessible to a restricted group of

You can read more about this from our users only. If you don’t have access to that information, the
website, at www.eduten.com pseudonymous data is effectively anonymous for you.

IN A NUTSH ELL:

In anonymous data, all identifying records are permanently

destroyed, while in pseudonymous data, identifying records
are replaced with made-up identifiers.
 21

Authors
UNI V E R S I T Y L E C T U R E R HEAD OF PEDAGOGY
Erkki Kaila, PhD Eikka Kurvinen
University of Helsinki M.Sc. CS, MA Education
erkki.kaila@helsinki.fi eikka@eduten.com

Erkki specializes in informatics education, research and Eikka has trained thousands of teachers in the use of Eduten

pedagogy. He is one of the original developers of Eduten Playground. He’s also one of the founding developers from way

Playground. back in 2012.

Some other team-players behind Finland Math are:

A S S O C I A T E P R O F ESSOR
M I N D S E T E XP E R T UNIV ERSITY TEA CH ER
Mikko -Jussi Laakso, PhD
Makke Leppänen Jussi-Pekka Järvinen
Director of the Centre for Learning Analytics,
M.Sc. of Psychology, MBA MA Education
University of Turku
makke@eduten.com jpejar@utu.fi
milaak@utu.fi

Makke has studied growth mindset and motivation psychology Mikko-Jussi has 20 years of experience in research based Jussi-Pekka is a class teacher with special education proficiency. He

at the University of Sydney. He has run a variety of teacher development of Finnish education with pedagogical and is currently working at the University of Turku as a university teacher

training projects over the last 20 years. technological aspects, namely Learning Analytics. focusing on digital pedagogy. In 2019, he was elected as the University

Teacher Of The Year together with his colleague Aleksi Lahti.