Technology, Pedagogy and Education

ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/rtpe20

Teaching with learning analytics:how to connect

computer-based assessment data with classroom
instruction?

Wilfried Admiraal, Jordi Vermeulen & Jacquelien Bulterman-Bos

To cite this article: Wilfried Admiraal, Jordi Vermeulen & Jacquelien Bulterman-Bos
(2020) Teaching with learning analytics:how to connect computer-based assessment data
with classroom instruction?, Technology, Pedagogy and Education, 29:5, 577-591, DOI:
10.1080/1475939X.2020.1825992

To link to this article: https://doi.org/10.1080/1475939X.2020.1825992

© 2020 Technology, Pedagogy and Published online: 13 Oct 2020.

Education Association

Submit your article to this journal Article views: 2631

View related articles View Crossmark data

Citing articles: 3 View citing articles

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=rtpe20
TECHNOLOGY, PEDAGOGY AND EDUCATION
2020, VOL. 29, NO. 5, 577–591
https://doi.org/10.1080/1475939X.2020.1825992

Teaching with learning analytics:how to connect computer-based

assessment data with classroom instruction?
a
Wilfried Admiraal , Jordi Vermeulenb and Jacquelien Bulterman-Bosc
a
Graduate School of Teaching, Leiden University, Leiden, The Netherlands; bDepartment of Teacher Education,
Amsterdam University of Applied Sciences, Amsterdam, The Netherlands; cOpen Doors Education, Amsterdam, The
Netherlands

ABSTRACT ARTICLE HISTORY

Computer-based assessments can provide students with feedback to guide Received 26 July 2018
their learning as well as inform teachers who extract information to prepare Accepted 20 September 2019
their teaching. Five secondary school teachers were monitored during one KEYWORDS
school year to answer the following research questions: (1) What kind of Computer-based
student data do teachers use for their teaching practice? (2) How do assessment; learning
teachers use student data for their instructional practice? and (3) How are analytics; formative
these classroom instructions evaluated by students? To use data for their assessment; assessment for
teaching practice, teachers preferred data with detailed information about learning; language teaching
each student, task and response. In class, teachers focused on providing
low-performing students with feedback, either by themselves or by a high-
performing peer student. Subsequently, low-performing students evalu­
ated this formative assessment practice more positively than high-
performing students as they valued the chance to practise and the addi­
tional feedback and instruction they received in class. Implications for
teacher support are discussed.

Introduction
Computer-based assessments give students the opportunity to practise and generate data that
provide students with feedback and inform teachers about details of their students’ performance.
These so-called learning analytics can make teaching and learning more efficient and effective. What
these computer-based assessments have in common is that they support online adaptive testing for
students and offer hints and – in some cases – scaffolds to support students’ performance and
learning. They also provide information about students’ performance including test scores, online
student behaviour and qualitative details of student answers. Most research on computer-based
assessments suggests benefits of direct feedback on students’ learning performance, but little
information is available on how teachers can connect computer-based assessment data to classroom
instruction. In order to enhance the effects of the use of computer-based assessments and to
improve classroom teaching in general, more insights are needed into how teachers’ classroom
instruction could benefit from computer-based assessment data. Teachers might use these student
data to prepare their classroom instruction, to inform their assessments of student performance and
to expand their pedagogies. The objective of the current study is to provide insights into how
teachers’ classroom instruction could benefit from computer-based assessment data.

CONTACT Wilfried Admiraal w.f.admiraal@iclon.leidenuniv.nl Graduate School of Teaching, Leiden University, Leiden,
The Netherlands
© 2020 Technology, Pedagogy and Education Association
578 W. ADMIRAAL ET AL.

Computer-based assessment, learning analytics and teaching

Computer-based assessments offer standardised tests for a particular school subject with an adapta­
tion to students’ ability level and automated feedback about students’ test performance (their test
scores as well as details of their answers), hints and – sometimes – tutor scaffolds. Computer-based
assessments can support the formative function of assessment (Bennett, 2011; Black & Wiliam, 2009).
Assessment can have both a summative function – evaluating students’ performance – and
a formative function – informing students about their performance and how to improve this.
Generally, two features appear to be important in designing formative assessment that will support
learning (Wiliam, 2011). First, the evidence should be more than information about the gap between
current and desired performance; it must also provide feedback about what kinds of instructional
activities are likely to result in improving performance. In their landmark meta-analysis of feedback,
Hattie and Timperley (2007) identified three types of feedback: (1) Feed up – Where am I going, (2)
Feed back – How am I going? and (3) Feed forward – Where to next? A second feature of formative
assessment supporting student learning is that students should actually be engaged in actions to
improve learning, which might be remedial activities by the teacher, peer support or reflecting on
different ways to move their learning forward. For both features, the role of the teacher is crucial.
Teachers not only can provide different types of feedback, they can also engage students in learning
activities. Yet not much is known about the role of the teacher using data of computer-based
assessments for their teaching.
These data of computer-based assessments can be understood as learning analytics, which can be
defined as ‘the measurement, collection, analysis and reporting of data about learners and their
contexts, for purposes of understanding and optimizing learning and the environments in which it
occurs’ (Siemens & Gasevic, 2012, p. 1). Learning analytics refers to learner activities such as learner
participation in discussion forums, learner online behaviour and computer-based assessments. Two
types of learning analytics can be distinguished: embedded and extracted analytics (Greller &
Drachsler, 2012). Embedded analytics refers to data that are used to inform the learner and/or to
adapt tasks to the ability levels of students without teacher intervention. An advantage of embedded
analytics is that these can be used seamlessly during participation in the learning activity itself.
A weakness is that these analytics can be ignored by the learner. Extracted analytics refers to data
that are presented for interpretation and provide teachers with information about learning process
and outcomes, which they use to improve their classroom instruction (see Figure 1).

Embedded analytics
Computer-based learning environments differ in the way they incorporate automated assessment,
hints, scaffolds and feedback, how they adapt to the ability level of individual students and what kind
of information they present to a teacher. In a meta-analysis of item-based feedback in computer-based
environments, Van der Kleij et al. (2015) examined the findings of 40 studies. The authors classified
feedback into three categories, based on the review of Shute (2008): (1) knowledge of results – which
indicates whether the answer is correct or incorrect without providing the correct answer or any
additional information, (2) knowledge of correct response – which is similar to the former, but also
provides the correct answer, and (3) elaborated feedback – which can be understood as additional
instruction combined with knowledge of results or knowledge of correct response and can take many
forms such as hints, explanation of the correct answer, additional information and extra study material.
The meta-analysis showed larger effect sizes for elaborated feedback compared to knowledge of
results and knowledge of correct response, especially for higher order learning outcomes.
Embedded analytics with simple feedback in the form of knowledge of results or correct responses
are included in most computer-based assessments. Faber et al. (2017) examined the effects of one of
the most commonly used adaptive computer-based learning environments in Dutch education,
Snappet. In this study with 79 primary schools (40 in the treatment group, 39 in a control group,
 TECHNOLOGY, PEDAGOGY AND EDUCATION 579

Figure 1. Embedded and extracted analytics in computer-based assessments.

with 1808 Grade 3 students), positive effects of working with the Snappet environment have been
found on students’ maths performance, measured by standardised maths tests. In addition, the authors
report that the high-performing students (top 20%) benefitted the most. A similar tool was studied by
De Witte et al. (2015). These authors examined data from Got it Maths, an adaptive computer-based
assessment tool, of 9898 students in grades 1–3. In this explanatory study, the authors conclude that
the more exercises students completed, the higher their test scores. These positive effects of these
adaptive computer-based learning environments with in-between feedback have been confirmed for
secondary school students (Haelermans & Ghysels, 2017).
In the studies mentioned above, in-between feedback for students mainly consisted of knowledge
of results or correct responses and some general hints – if needed. Wang (2011) evaluated the
Graduated Prompting Assessment Module as part of remedial teaching of junior high school mathe­
matics. When students answer an item incorrectly, they get instructional prompts in a graduated way.
These prompts (with a maximum of three for each item) gradually reveal the answer by guiding
students to find it step by step. The number of prompts depends on how many tries students need to
give the correct answer, but the content of the prompts is the same for all students. In a quasi-
experimental design, Wang examined the effects of this graduated prompting assessment module on
maths performance, compared to students taking a web-based test or a paper-and-pencil test. In the
latter two conditions, students answered all items first and received the correct answers at the end of
the test, with a delay for the paper-and-pencil condition. All students first received mathematics
instruction in traditional classroom setting from the same teacher with the same learning materials.
Significant differences were found between the graduated prompting assessment, on the one hand,
and both other assessment forms showing a larger growth in maths scores in the former condition.
580 W. ADMIRAAL ET AL.

Extracted analytics
Students’ assessment data not only provide direct feedback and instruction to students, they also
can help teachers to better diagnose changes in student learning and adjust their classroom
instruction. In an expert study on future scenarios of technology-enhanced assessment, Perrotta
(2014) distinguished a ‘enhanced instructional planning’ scenario, in which teachers act as instruc­
tional designers, using data to shape their teaching. Yet the question remains to what extent
teachers do so in practice. Ysseldyke and Bolt (2007) examined the effects of the Accelerated
MathTM program on elementary school students’ maths achievement. Teachers in 80 classroom
settings were randomly assigned to an experimental or a control condition. Teachers in the experi­
mental condition used progress monitoring Accelarated MathTM (Renaissance Learning, 1998), in
which students completed an adaptive test as pre-test and then were assigned to appropriate
instruction levels. Students responded to these exercises and got immediate feedback. Teachers
were provided with summary information detailing the quantity and quality of maths performance
for each student in their class in order to provide support and appropriate instruction. Effects were
examined on students’ maths performance. The authors found a main effect of condition for only
one of the two maths performance indicators, and condition effects for both indicators in only some
schools. Further analyses showed that almost 40% of the students in the experimental condition
were not involved in progress monitoring at all, although the teachers of the experimental condition
were asked to implement the program with all of the students in their classes. Analyses with
achievement for students who were high, low and non-implementers showed consistently large
and positive gains for the high-implementation group and no differences between the low-
implementation and no-implementation groups. No qualitative information is provided on how
teachers adapt instruction, individualise instruction or group students for instruction.
The finding from the study of Ysseldyke and Bolt (2007) that the usage of the formative assessments
is the dominant factor that explains differences in student learning gains is confirmed by a study by
Koedinger et al. (2010). These authors examined effects of ASSISTments, which is a web-based maths
tutor providing automated assessments, corrective feedback, scaffold questions, hints and summary
information of maths performance for each student. In a quasi-experimental study, the authors found
students in the schools that used ASSISTments significantly outperformed students in the comparison
school. They also found that the greater usage of the system by students was associated with better
performance. Finally, and probably the most interesting for the context of the current study, they
found an interaction effect of teacher and student use, which meant that increased teacher use (i.e.
having more students use the system more often) was associated with learning gains among students
with little or no use. This latter finding suggests that students who did not use the system of
assessments have benefitted from teachers adapting their whole-class instruction based on what
they learned from the summary information from ASSISTments.

Purpose of this study

Greater usage of computer-based learning environments by both teachers and students seems to be
associated with better student performance. A limitation of all the studies is that little to no information
is available on how teachers used student assessment data for their classroom instruction. In their
evaluation of the Formative Assessment Multimedia Learning Environment (FAMLE), Zhan and So
(2017) reported that teachers believed that insufficient feedback was automatically generated from
FAMLE to integrate the information in their teaching. The teachers also experienced difficulties with
understanding which assessment task assessed what, despite the researchers’ effort to provide rich
assessment information in a user-friendly way. Neither teachers nor students seemed to understand
the quantity of data generated.
 TECHNOLOGY, PEDAGOGY AND EDUCATION 581

In the current study, a case study has been carried out on formative assessment of five
secondary school language teachers using data from the computer-based environment Got it
Language (https://www.thiememeulenhoff.nl/got-it) to adapt their lesson plans and instruction
in the following lessons. Got it Language is one of the adaptive assessment environments
described above with both embedded and extracted analytics. Three research questions were
formulated:

(1) What kind of student data do teachers use for their teaching practice?
(2) How do teachers use student data for their instructional practice?
(3) How are these classroom instructions evaluated by students?

Method
Participants and procedure
The participants were five Dutch-language teachers, one information technology (IT) support officer
and 47 Grade 7 students from one secondary school. During one school year (September 2014 –
May 2015), 12 cycles of two weeks were studied with weekly meetings in the computer room at
school, in which students completed online Dutch-language tests (guided by the IT support officer).
These two weekly meetings were followed by a team meeting with the teachers to prepare one
particular lesson of each teacher in the following week. This preparation was based on learning
analytics. After the particular lesson, the teachers participated in an evaluation team meeting, which
started a new cycle. Data collection included (1) teachers’ lesson-preparation forms, (2) reports of the
12 team meetings and a start-up meeting, (3) video recording of 55 lessons (of all 5 teachers with 114
students), (4) interviews with 47 students, and (5) interviews with the five teachers and the IT support
officer. The 47 students were selected because they were directly involved in one of the feedback
forms teachers used in their classes.
The software Got it Language (Got it Taal) was used for the language tests. This online adaptive
learning environment includes student tests in the domain of reading, spelling and grammar of
Dutch language, each with themes (such as grammar concepts within the domain of grammar) and
subthemes (such as subject and direct and indirect object within the theme of grammar concepts).
For all students, taking tests followed the same procedure (see Figure 2):

Figure 2. Simplified schedule of student use of Got it (grid lines in the Practice box stand for different item difficulty). Passing
a final test leads to an entry test in the next subtheme; failing a final test leads to more practising on the same subtheme.
582 W. ADMIRAAL ET AL.

(1) Students completed an entry test on a particular subtheme.

(2) Based on their score on the entry test, students practised on items that differed in difficulty (grid
lines in the Practice box of Figure 2), starting at the difficulty level set by their entry test score.
(3) If students passed the practice items at a particular difficulty level, they were directed to a final
test on that particular subtheme.
(4) If students passed the final test on a particular subtheme, they were directed to an entry test
on a new subtheme.
(5) If students failed the final test on a particular subtheme, they returned to the practice items of
the same subtheme.

The tests were multiple-choice tests and open-answer tests. With multiple-choice items, students got
a second chance if an answer was incorrect. If the second choice was again incorrect, the tool
provided students with the correct option. After completing the open-answering items, all correct
answers were kept and incorrect answers were deleted (based on programmed text recognition); in
the next step, students only revised the incorrect answers. Students could ask for help from the IT
support officer and consult additional written clarifications or explanations presented in a video clip.
Students completed the computer-based assessments in weekly sessions in school.

Data
In Table 1, we summarise the relationship between the five data sources and the three research questions.

Lesson-preparation forms
Although teachers could access the student scores in Got it, they complained that they could not
manage the standard reports. Therefore, the researchers created a report in Excel to inform the
teachers. Teachers completed a form to prepare each lesson that was based on students’ assessment
scores. In each form, teachers filled in the names of the students who would receive additional
feedback or instruction, students’ scores and the way teachers planned feedback (by fellow students
or themselves) and their instruction in class. In total, 55 forms were completed.

Team meetings
Once every two weeks the teachers had aone-hour team meeting in which they not only discussed
the way they would like to provide feedback in next week’s lessons but also reflected on the lesson of
the previous week. Video clips or collegial observations were used for sharing and reflection. In total,
12 meetings were reported.

Video recordings
In total, 55 lessons by the five teachers are recorded on video by one of their colleagues. From these
lessons, 60 feedback sessions were selected (in some cases, feedback was provided to two students
in different sessions). The video clips were used to check whether teaching practices matched the
lesson preparation forms, to select the students for an interview and to prepare the student
interviews.

Table 1. Relationship between three research questions (RQs) and five data sources.
Data sources RQ1 RQ2 RQ3
Lesson preparation forms X X
Reports of team meetings X X
Video recordings of the lessons X
Student interviews X
Teacher interviews X X X
 TECHNOLOGY, PEDAGOGY AND EDUCATION 583

Student interviews
Of the total 114 students who were taught by the five teachers, 47 students were interviewed during
the project period. These 47 students were selected as they were directly involved in the feedback
from their teachers in at least one of the lessons. Ten students were interviewed twice or three times,
which resulted in a total of 60 student interviews. Further criteria to select students for an interview
included the sound and image quality of the video recordings of a lesson and a variety of students
based on their gender and language scores in Got it.
Students were interviewed during a break or right after the final lesson of that day. The interviews
lasted about 15 minutes each and were videotaped. After a short introduction, the interviews started
with a video clip of a feedback session in class to remind students of the particular class situation.
Then students were asked to describe (1) what was happening, (2) what the teacher did, (3) whether
the feedback helped them and how, (4) how they felt about the situation, and (5) suggestions to
improve the feedback they received.

Interviews with teachers and mentor

At the end of the project period, the five language teachers and the IT support officer who facilitated
the Got it sessions were interviewed. After a short introduction to the aim of the interviews, the
participants were asked to reflect on the procedure of the two-week cycle, the Got it tool and Got it
sessions, the way they used test scores for the team meetings, the lesson preparation and in class,
and how they evaluated the effects on the language performance of their students.

Analyses
All data are qualitative data. Each data source has been analysed using an inductive data-grounded
approach in which transcripts were repeatedly read and analysed to inductively determine the main
themes (Braun & Clark, 2006). In the next step, the results of the thematic analysis were then
summarised for their relevance to the three research questions. The resulting materials were further
analysed with a two-column format method (Argyris, 1993), including the summarised materials in
the first column and an interpretative description of the materials in the second column. This
interpretation entailed teachers’ evaluation of the data presentation from the tool, their goals with
these data in class, their actual data use in class, the way teachers provide feedback in class and
students’ evaluation of the intervention with formative assessment. Three researchers were respon­
sible for the summaries in the first columns and the analyses in the second column. Differences in
researchers’ interpretation were discussed until agreement; different interpretations are not
included in the findings.

Findings
Presentation of assessment data
Before answering the three research questions, we first present issues with the presentation of data
teachers reported in their team meetings and interviews. Although the editors of Got it indicated
that teachers are provided with direct insight in information that makes appropriate feedback
possible, teachers had difficulties with the interpretation of the information on language test scores
that were provided by Got it. These difficulties relate to six issues:

(1) The standard presentation of the scores was at the level of themes, but teachers wanted to see
the scores on subthemes as these would give better indications as to what teachers should
address in their class instruction. For example, students’ scores on the use of grammar
concepts do not provide any information on how students deal with a subject or direct or
indirect object in a sentence. The presentation was adapted to include test scores on
subthemes as well.
584 W. ADMIRAAL ET AL.

(2) The standard presentation only included the final test scores for each theme, whereas
students spent most of the time on practising. In the team meetings, teachers reported
that they found that some final student scores were meaningless, as some students did not
practise seriously. The presentation was adapted to include scores of these practice tests to
provide teachers with insights into how students progressed towards the final test scores for
each theme.
(3) Teachers had to put a lot of effort into the comparison of student scores at the same ability
level by selecting the correct themes, subthemes and level of the test assignments. Teachers
complained that they ‘could not see the relevant information right away’ and ‘had to be
supported to read the correct information in the system’ (both quotes from the notes of the
team meetings). The presentation was slightly adapted to make it easier to compare similar
test scores.
(4) Initially, teachers had to create their own overview of all students from one particular group.
As teachers wanted to use the test scores to prepare their classes for a particular group, the
presentation has been adapted to provide an overview of student groups in addition to the
individual student scores.
(5) Initially, no qualitative information was available about the content of the particular test item
and individual student response to that item. In the team meetings, teacher indicated that they
needed more information about an incorrect answer in order to label it as a typo, a mistake,
a misunderstanding or a foolish attempt. For example, the same incorrect response to
a particular prompt might refer to a mistake in either spelling or suffix use. During the project
period, this qualitative information about the item and student response became available.
(6) No dates were linked to the completed tests, which meant that for the teachers it was unclear
when students completed a particular test. This could mean that the teachers addressed
a particular student’s test score in their class, while the student completed the particular test
a while ago. This issue was not resolved during the project period.

In sum, the editor of Got it adapted the presentation tool during the project period with more details
about the tasks and how students completed these tasks (instead of only presenting their scores and
group scores), and an explicit link between the scores and the specific part of the language tests.
These changes helped the teachers to better interpret the learner data. But other issues were not
resolved, and teachers had to use workarounds to get the information they wanted. One of these
workarounds was the Excel overviews the researchers provided. On the other hand, teachers
reported that they did not need some information that was provided, such as the number of
attempts students used for a correct response and how long this took.

Teachers’ use of student data

Teachers mainly used the following information to prepare their classes (based on the lesson
preparation forms and the individual interviews):

(1) individual student scores on the practice tests (instead of the final test scores) per subtheme;
(2) group mean scores per subtheme;
(3) difference between 1 and 2;
(4) the number of practice tests students took per subtheme;
(5) for each individual student, qualitative information about the content of the item and student
response (only the last month).

In Got it, students can choose which themes and subthemes within a domain they would like to start
with. However, the teachers in this school decided to prescribe the sequence of themes and
subthemes within a domain. At the beginning of the school year, teachers indicated which domain,
 TECHNOLOGY, PEDAGOGY AND EDUCATION 585

themes and subthemes would be covered in which period of the year. In this way, they wanted to
link the assessment scores with the classes they planned during the school year. Yet the level and
pace of students varied, and tests were adapted to the ability level of the students. This meant that
after some time students differed in themes and subthemes they worked on in Got it, and teachers
had more difficulties aligning their classes with the work of the students in Got it. More detailed
information about student scores (see above) supported teachers’ decision on which data could be
used to prepare their classes.
Based on the information listed above, teachers selected one or two individual students who (mostly
negatively) deviated from the group mean. But the selection of the individual students for individual
feedback in class was not only based on these scores. In the interviews, teachers reported that they also
used information from their own class experiences and previous regular tests to prepare their lessons.
This means that the teachers tried to integrate the data sources they had about students’ performance.
In sum, for class preparation teachers used the information from Got it at the most detailed level:
per subtheme, per individual student and per practice test. Which subthemes they chose was mainly
based on their school-year plan for what they wanted to address in class. This way of working was
quite time-consuming, which meant that teachers focused on the most extremely deviating scores.

Teachers’ instructional practice

Teachers’ main aim of using Got it as a means of formative assessment was to improve students’
acquisition of the content of their language instructions. They reported three benefits of the use of
Got it. Firstly, they mentioned that students, especially the low-performing students, were triggered
to put some effort into language learning by completing the computer-based assessments.
Receiving feedback from the tool about their low scores because they did not practise enough
helped these students to become aware that effort pays off. In the interview, a teacher mentioned
that ‘now they [students] understand that it [practising] is not useless, their achievements will be
monitored. Importance is placed on their practising.’ Secondly, the teachers mentioned that the
computer-based test provided additional learning opportunities as students practised their lan­
guage skills, received corrective feedback, asked for hints and read the instructions. Thirdly, teachers
valued the detailed information on individual item responses as they learned about misconceptions
of their students they could address in class. In a team meeting, a teacher mentioned that she
learned from some incorrect answers and adapted her plan for the next lesson to focus on some
general grammar rules she thought were underlying these incorrect answers.
Based on the assessments and the team meetings, the teachers used various forms of feedback
during their lesson, ranging from individual feedback and additional tasks via working in pairs
(mostly one low-performing student with one high-performing student) to whole-class instruction
based on the learner data from the online tests (see left column of Table 2 for an overview).
From Table 2, we can see that teachers mainly provided individual students or pairs of students
with a different task than the other students in the group. In most cases, the other students of the
group were not actively involved in the tasks of the selected students. Sometimes teachers made
explicit in class why some students got other tasks, but in most cases teachers did not do so, to
prevent student stigmatisation. In an interview, a teacher explained, ‘you should be careful with
making explicit which students had low scores. Students are so sensitive for this kind of things. Both
the low and high scoring students.’ The frequency of these eight feedback forms is not equally
distributed among the five teachers. For example, one of the teachers applied the form with a pair of
students who completed a different task than the other students in the group (type 6) in 9 of the 14
lessons that he based on the Got it scores. He explained: ‘I wanted to experiment with this, and it
went on well. So, I stuck to this form.’ Another teacher reported holding on to plenary forms of
instruction (types 1 and 2):
586 W. ADMIRAAL ET AL.

Table 2. Types of feedback in class and student evaluations of each type (between bracket the number of students interviewed,
Ntotal = 60).
Student evaluations
Type of feedback in class Students with scores < mean Students with scores > mean
1. Inquiring test scores for a small selection . . . stimulates to practise more often (2) . . . not necessary, better direct feedback
of students (3) during test meeting (1)
2. Giving selected student a turn (7) . . . stimulates thinking, also when . . . does not add much (1)
other student gets a turn (6)
3. Selected individual students complete . . . valuable (5)
the same task as other students, but
with teacher support (questioning,
providing information) (5)
4. Pairs of selected students complete the . . . valuable (3) . . . does not add much (1)
same tasks as other students, with one
low- and one high-performing student (4)
5. Selected individual students complete . . . understand better the underlying
other tasks different from those of the rules and get more practice (16)
other students with teacher support (16)
6. Pairs of selected students complete the . . . understand better the underlying . . . understand better because I have
other tasks as other students, with one rules as my peer uses other explained underlying rules. Feel
low- and one high-performing student wording. Feel more safe than in proud and confident. Miss class
(23) class (17) instruction (6)
7. Inviting a student to give instruction in . . . valuable, though scary (1)
class (1)
8. Inviting a high-performing student to . . . understand better because I have to
give instruction in class and to help explain and instruct. Feel proud (1)
other students individually (1)

I like to have control and give directions. I also do not believe that a well-performing student can instruct a poor-
performing student. These forms might be en vogue, but you really need to have some competences to do this
in a right way.

Actually, other teachers also chose for repetition of successful feedback forms. In the interviews,
four teachers indicated that they needed more support to experiment with other forms of
providing feedback based on the assessment scores by, for example, workshops or instructions
based on literature study. Yet one teacher cynically mentioned that this kind of support would not
have helped much, as she thought the teachers were not eager enough to find out how they
could give feedback. This lack of intrinsic motivation might be related to the top-down way
teachers were selected for this project. Actually, one of the four teachers who mentioned that
they needed more support also stated that he favoured teacher-centred ways of teaching and
found it difficult to individualise instruction and to provide students with different tasks.
Apparently, the in-between team meetings did not support teachers enough to vary their
teaching methods.

Students’ evaluations
In Table 2, we have summarised student evaluations of each of the eight feedback forms in class. In
Table 3, we summarised the answers on two main interview questions: ‘How do you feel about what
the teacher did in class?’ and ‘Did feedback in class help you?’
Generally, the low-performing students were satisfied with the individual approach of their
teacher during the lessons. They valued the additional feedback, the opportunity to practise more
and the additional instruction, either by the teacher or a high-performing peer, about underlying
grammar rules. In the peer-feedback constellation (with one low-performing student and one high-
performing student), they valued the safety of this constellation (in contrast to getting a turn in class)
 TECHNOLOGY, PEDAGOGY AND EDUCATION 587

Table 3. Summary of the student answers on two interview questions.

Low-performing students High-performing students Total
(n = 50) (n = 10) (n = 60)
Teacher in class
Positive 33 4 37
Negative 2 1 3
Neutral/don’t know 15 5 20
Feedback helped
Yes 34 3 37
No 10 5 15
Don’t know 6 2 8

and the way their high-performing peer formulated the explanations, which aligned more with how
students would formulate explanations.
The high-performing students generally were less satisfied with the feedback in class based on
the assessment scores from Got it. They did not see the additional value for themselves, except when
they were invited to support and instruct other students. Then they felt proud of being invited to
explain and instruct in class and help other students. They also mentioned that in these occasions
they learned better to understand the underlying rules of grammar as they were forced to explain
these in a very explicit way.
Yet students also mentioned some disadvantages. Both low-performing and high-performing
students who got a turn in class felt that they got too much attention and were placed in a special
position, although the low-performing students also indicated that they learned a lot from it. These
perceptions could be related to the youth culture in Dutch secondary schools in that a student does
not want to be special but likes to be an equal member of the group. Other issues students raised
were the lack of teacher control of the accuracy of answers when pairs of students worked separately
on a task, missing class instructions when students worked in pairs, and feelings of insecurity when
a low-performing student got a turn in class. Finally, most students who indicated that the feedback
did not help them (see lower row of Table 3) mentioned that they already knew the correct answers
and the underlying grammar rules and that they just did not do the tests seriously.

Discussion and conclusions

Computer-based assessments can be used for direct feedback to the students (embedded analytics)
as well as indirect feedback through teachers’ instructional practices (extracted analytics). Both types
of feedback are included in the current study on formative assessment, although the focus of
research was on (1) the data teachers used for the preparation of their language classes, (2) their
feedback during class and (3) how this was evaluated by the students.
With respect to the data teachers used from the computer assessments (RQ1), it was clear that the
Got it software did provide teachers with a lot of information, but not exactly the information they
needed. The individual students were able to attend a fully adapted test program and received
detailed feedback about their achievements (embedded analytics). The presentation for the teachers
was at a quite general level providing overviews of individual scores and group means. On request of
the teachers, the way information was presented to the teachers was adapted during the project
period into more detailed scores of students and more qualitative information of both item and
individual student responses. The elaboration of more details fitted the goals teachers had with their
feedback during class: they want to select particular students and provide them with either detailed
feedback and instruction or support them with a tailor-made new assignment. These goals could
only be reached if the teachers received the information on a detailed level. Yet this asks for a change
in views on the use of computer-based assessment: from a tool for individualised digital practice (cf.
Haelermans & Ghysels, 2017) to a tool for formative assessment providing both students and
teachers with information for Feed up, Feed back and Feed forward (Hattie & Timperley, 2007).
588 W. ADMIRAAL ET AL.

With respect to the way teachers used student data for their instructional practice (RQ2), teachers
reported three benefits of the use of Got it Language. Firstly, they mentioned that students,
especially the low-performing students, were triggered to put some effort into language learning
by completing the computer-based assessments. Receiving feedback from the tool about their low
scores because they did not practise enough helped these students to become aware that effort
pays off. Secondly, the teachers mentioned that the computer-based test provided additional
learning opportunities as student practised their language skills, received corrective feedback,
asked for hints and read the instructions. Both benefits (engaging students with learning and
providing additional learning activities) are confirmed in other empirical studies on computer-
based assessments (Faber et al., 2017; Haelermans & Ghysels, 2017). Thirdly, teachers valued the
detailed information on individual item responses as they learned about misconceptions of their
students they could address in class. This ‘learning effect’ for teachers can have long-term effects
with also changing teacher practices that are not directly based on computer-based assessment
data. This finding can be understood as an addition to the literature on computer-based assessments
as most of it is about student learning.
Furthermore, teachers had a clear preference for the selection of one or two students (mostly low
performing) who received an additional assignment on the subtheme related to the assessment as
well as additional feedback on the completion of this new assignment, from either the teacher or
a high-performing peer. Teachers preferred feedback and instruction that was aligned with the way
they were used to teach and their beliefs about effective teaching and learning strategies. In a study
on the implementation of tablets in secondary schools, Admiraal et al. (2017) already found that
beliefs about teaching and learning were strongly related to how teachers preferred to teach with
tablets.
With respect to the student evaluations of the formative assessment intervention (RQ3), low-
performing students – who were the majority with 50 out of the 60 feedback sessions – were
generally quite satisfied with the formative assessment teacher provided. They valued the additional
feedback from the teacher or a high-performing peer.
In the current study, the computer-based assessment program Got it Language did provide
embedded analytics as well as extracted analytics (cf. Figure 1). Yet its main function is to provide
learners with opportunities to practise their skills at their own ability level and provide – mainly
corrective – feedback. This function is similar to many other computer-based assessment programs
such as Snappet, Muiswerk or Accelatered Maths to name a few. Similar to these programs, Got it is
less suited for formative assessment in the sense that it does not provide teachers with adequate
information about student assessments to prepare and perform their teaching in class. As Wiliam
(2011) emphasised, evidence should be more than information about the gap between current and
desired performance; it must also provide indications how to improve performance. Much more
detail seems to be necessary to fulfil this formative function. Yet students indicated that they really
valued the additional feedback from their teachers or peers, which aligns with the results of studies
that did examine the effects of teacher use in class. These studies (e.g. Koedinger et al., 2010;
Ysseldyke & Bolt, 2007) found significant learning gains arising from teachers’ use of the information
from assessments in class.
Another conclusion from the current study is that formative assessment based on computer-
based assessments seems to benefit low-performing students mostly. For these students, more
opportunities to practise and more feedback and instruction have additional value for their language
learning. Most studies on computer-based assessments do not mention a differential effect for low-
and high-performing students. Other studies do so, but the findings are inconsistent (Faber et al.,
2017; Wang, 2011). Obviously, the current study included student evaluations and did not measure
effects on students’ language performance, neither short term nor long term.
A final conclusion from the current study is that teachers need to be supported in their formative
assessment. Not only do they need accurate and relevant information from the computer-based
assessment tools, they also should be informed better how to deal with this information, in what
 TECHNOLOGY, PEDAGOGY AND EDUCATION 589

way they could vary in the way they provide feedback in class and how they can see the effects of their
pedagogy. This conclusion aligns with the findings from other studies on teachers’ competency in
formative assessment in either initial teacher education programmes (Brevik et al., 2017) or continuing
professional development (Livingston & Hutchinson, 2017). These studies confirm that teachers should
be better prepared for the interpretation of assessment scores and the use of formative assessments in
class. In the current study, the in-between meetings with the teachers were meant to support them
with group discussions and collegial support. Apparently, these meetings did not have this supportive
function, and the group of teachers could not provide each other with enough variety in their teaching
methods. However, this could also be caused by the fact that these teachers were selected by their
school leader to participate in this project, which may hinder their intrinsic motivation to participate.

Implications
Based on the findings, we formulate three implications for formative assessment with computer-based
assessment. First, the tool (i.e. Got it Language) should be redesigned in order to provide accurate
assessment information for teachers to prepare and perform their feedback in class. This means easy-
to-get overviews of student scores, quantitative and qualitative, and with enough details. Editors of
these computer-based assessment tools should obtain user data and evaluations in order to adapt
their tools to both learner (embedded analytics) and teacher (extracted analytics) needs. Otherwise, it
might be that this kind of tools will not find their way into the classroom.
The second implication concerns the selection of the students for feedback in class. Although
formative assessment in this way might benefit low-performing students mostly, the high-
performing students could also benefit. By forms of peer teaching or peer feedback, these high-
performing students gain confidence and feel proud, which might motivate them for higher levels of
language learning. In addition, as concluded in many peer-assessment studies, teaching is the best
way of learning; providing feedback and instruction requires a deep understanding of the assess­
ment rules and criteria.
The third implication is related to the support of teachers in more ways than providing them with
relevant and accurate assessment data. In the current study, teachers preferred to be more supported
in their pedagogy in class. Adaptive or individualised instruction is a complex way of teaching and
even more so, if this is based on student data from computer-based assessments. Discussion meetings
with teachers (but probably more structured or with more input than in the current study), constella­
tions with teacher design teams or teacher data teams, workshops or master classes on individualised
instruction, (collaborative) action research or lesson studies are forms of professional development
that could help to improve teachers’ pedagogy in class with the aim of differentiating on the basis of
assessment scores. In this way, teachers could make well-informed decisions on how to differentiate in
class with the focus of improving the performance of all their students.
In addition to these implications for practice, future research can be directed on validating the
findings in other school subjects. Many different tools for computer-based assessment are used in
learning mathematics, including one of the same editor (Got it – maths). Our study does not suggest
subject-specific outcomes: the way teachers should be supported in their teaching with computer-
based assessment data and the various forms of teaching they use in class are probably similar for all
school subjects. Yet validation is necessary with larger sample sizes, a variety of school subjects and
educational contexts in order to provide generic conclusions about how teachers can effectively use
computer-based assessment in class.

Concluding remarks
Formative assessment based on computer-based assessment gives students and teachers the
opportunity to improve learning processes with students practising on their own ability level,
receiving direct feedback about their performance and learning in class that is supportive and
590 W. ADMIRAAL ET AL.

challenging. Yet the tools should be more focused on providing teachers with adequate information,
and teachers should be more supported in how to handle these assessments and how to connect
these to classroom teaching. Then research about both short-term and long-term effects should
make explicit the additional value of this kind of formative assessment, compared to just student
practice or just regular teacher instruction.

Disclosure statement
No potential conflict of interest was reported by the authors.

Funding
This work was supported by the Nederlandse Organisatie voor Wetenschappelijk Onderzoek [grant number 405-14-
516].

Notes on contributors
Wilfried Admiraal is full professor of educational sciences and director of ICLON Leiden University Graduate School of
Teaching.
Jordi Vermeulen is a teacher educator in social sciences at the teacher education department of the University of
Applied Sciences Amsterdam.
Jacquelien Bulterman-Bos is managing director of Open Doors Education, a company focused on the integration of
knowledge and action in education and educational research.

ORCID
Wilfried Admiraal http://orcid.org/0000-0002-1627-3420

References
Admiraal, W., Louws, M., Lockhorst, D., Paas, T., Buynsters, M., Cviko, A., Janssen, C., de Jonge, M., Nouwens, S., Post, L.,
van der Ven, F., & Kester, L. (2017). Teachers in school-based technology innovations: A typology of their beliefs on
teaching and technology. Computers & Education, 114, 57–68. https://doi.org/10.1016/j.compedu.2017.06.013
Argyris, C. (1993). On organizational learning. Blackwell Publishers.
Bennett, R. E. (2011). Formative assessment: A critical review. Assessment in Education: Principles, Policy & Practice, 18(1),
5–25. https://doi.org/10.1080/0969594X.2010.513678
Black, P. J., & Wiliam, D. (2009). Developing the theory of formative assessment. Educational Assessment, Evaluation and
Accountability, 21(1), 5–31. https://doi.org/10.1007/s11092-008-9068-5
Braun, V., & Clark, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101.
https://doi.org/10.1191/1478088706qp063oa
Brevik, L. M., Blikstad-Balas, M., & Lyngvær Engelien, K. (2017). Integrating assessment for learning in the teacher
education programme at the University of Oslo. Assessment in Education: Principles, Policy & Practice, 24(2), 164–184.
https://doi.org/10.1080/0969594X.2016.1239611
De Witte, K., Haelermans, C., & Rogge, N. (2015). The effectiveness of a computer-assisted math learning program.
Journal of Computer Assisted Learning, 31(4), 314–329. https://doi.org/10.1111/jcal.12090
Faber, J. M., Luyten, H., & Visscher, A. J. (2017). The effects of a digital formative assessment tool on mathematics
achievement and student motivation: Results of a randomized experiment. Computers & Education, 106, 83–96.
https://doi.org/10.1016/j.compedu.2016.12.001
Greller, W., & Drachsler, H. (2012). Translating learning into numbers: A generic framework for learning analytics.
Educational Technology & Society, 15(3), 42–57.
Haelermans, C., & Ghysels, J. (2017). The effect of individualized digital practice at home on math skills – Evidence from
a two-stage experiment on whether and why it works. Computers & Education, 113, 119–134. https://doi.org/10.1016/
j.compedu.2017.05.010
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/
10.3102/003465430298487
 TECHNOLOGY, PEDAGOGY AND EDUCATION 591

Koedinger, K. R., McLaughlin, E. A., & Heffernan, N. T. (2010). A quasi-experimental evaluation of an online formative
assessment and tutoring system. Journal of Educational Computing Research, 43(4), 489–510. https://doi.org/10.2190/
EC.43.4.d
Livingston, K., & Hutchinson, C. (2017). Developing teachers’ capacities in assessment through career-long professional
learning. Assessment in Education: Principles, Policy & Practice, 24(2), 290–307. https://doi.org/10.1080/0969594X.2016.
1223016
Perrotta, C. (2014). Innovation in technology-enhanced assessment in the UK and the USA: Future scenarios and critical
considerations. Technology, Pedagogy and Education, 23(1), 103–119. https://doi.org/10.1080/1475939X.2013.838453
Renaissance Learning. (1998). Accelerated MathTM (computer program). http://www.renaissancelearning.com
Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189. https://doi.org/10.3102/
0034654307313795
Siemens, G., & Gasevic, D. (2012). Guest editorial – Learning and knowledge analytics. Educational Technology & Society,
15(3), 1–2.
Van der Kleij, F. M., Feskens, R. C. W., & Eggen, T. J. H. M. (2015). Effects of feedback in a computer-based learning
environment on students’ learning outcomes: A meta-analysis. Review of Educational Research, 85(4), 475–511.
https://doi.org/10.3102/0034654314564881
Wang, T.-H. (2011). Implementation of web-based dynamic assessment in facilitating junior high school students to
learn mathematics. Computers & Education, 56(4), 1062–1071. https://doi.org/10.1016/j.compedu.2010.09.014
Wiliam, D. (2011). What is assessment for learning? Studies in Educational Evaluation, 37(1), 3–14. https://doi.org/10.
1016/j.stueduc.2011.03.001
Ysseldyke, J., & Bolt, D. M. (2007). Effect of technology-enhanced continuous progress monitoring on math
achievement. School Psychology Review, 36(3), 453–467. https://doi.org/10.1080/02796015.2007.12087933
Zhan, Y., & So, W. W. M. (2017). Views and practices from the chalkface: Development of a formative assessment
multimedia learning environment. Technology, Pedagogy and Education, 26(4), 505–515. https://doi.org/10.1080/
1475939X.2017.1345783