Factual Knowledge Acquisition and Remembering
Using Digital Flashcard Tool
Matea Markić Vučić Suzana Tomaš
SPARK school Faculty of Humaties and Social
Mostar, Bosnia and Herzegovina Sciences, University of Split
mateamarkic@gmail.com Split, Croatia
stomas7@gmail.com
Abstract— Teaching for teachers is a particularly important
paradigm of modern society in which information and
communication technology is the leading generic technology.
During the COVID-19 pandemic, we conducted an online course
for primary and secondary school teachers (n=24) from Bosnia
and Herzegovina. The course entitled E-learning: Teaching for
Teachers took place during May and June 2020, and is designed
for continual improvement of teacher's professional knowledge,
including helping teachers switch from face to face to online
teaching. This paper presents the results of a one group
experiment in which participant quiz scores were analyzed in
two different course cycles – cycle one presented the basic
elements and configuration of the e- learning systems, while
cycle two presented the concepts and models of hybrid learning.
The research was implemented with a traditional learning
management system Moodle (moodle.org) and a digital
flashcard system Memory (memory.com). The goal of the
research is to determine whether digital flashcard technology in
the process of learning and teaching improves the cognitive
process of remembering categorie of factual knowledge.
Keywords—teaching for teachers, online course, factual
knowledge, cognitive process remembering, digital flashcards,
self-assessment
I. INTRODUCTION
Teaching for teachers is a particularly important paradigm
of modern society in which information and communication
technology is the leading generic technology. Information and
communication technology brings changes in all levels of
education, which requires the design and delivery of teaching
content with significant involvement of e-learning and e-
learning systems. As a result, changes must take place in
teacher education curricula for beginning teachers as well as
for continuing education curricula for teachers that are already
teaching in the education system. Let's look at what the
changes are manifested in and how they were initiated?
Primarily it is the transformation of pedagogical approaches,
the predominant share of hybrid learning that combines
traditional and online learning.
Hybrid learning combines the best elements of online and
traditional education, and it is likely to be the dominant model
of teaching in the future [1]. The proportion of traditional and
online teaching in a hybrid model is expressed by the amount
of relative time devoted to each type of teaching. An online
course is one in which 80% or more of the total number of
teaching hours are delivered online, while the hybrid course is
one in which the share of online teaching ranges from 39 to
79% [2].
The aforementioned share percentages can have
exceptions to the rule and that is exactly what happened in
many instructional environments during the COVID-19
pandemic. Faced with the necessity of social distancing, many
education processes around the world had to be carried out in
Slavomir Stankov
University of Split
Split, Croatia
slavomirstankov@gmail.com
a manner that would completely avoid contact of the
participants in the teaching process. The resulting solution in
many areas was 100% online teaching process. When
considering online teaching and learning, it is important to
mention a few “myths” about unfolding in higher education
[3].
A. Online learning is easier than face-to-face learning. The
difficulty in learning is not determined by the method of
delivery of teaching content, but by the content itself and
the expected learning outcomes.
B. Online teachers spend less time teaching online than face-
to-face. Online teacher engagement, unlike traditional
teaching, involves extra effort and some new roles. Often
the teacher takes on multiple roles that traditionally occur
in online teaching (content creator, administrator,
designer). Also, the virtual classroom is open 24 hours a
day, making teachers more accessible to learners than in
the school building where learners can access teachers
only during building hours.
C. Online learners spend less time learning online than face-
to-face. The Internet and mobile devices have enabled
learners to learn anywhere, anytime. Accordingly, learners
are constantly learning, as opposed to face-to-face
teaching where resources and time are limited. The
multitude of tools offered by online learning creates an
environment to increase learner engagement and interest
in specific topics.
During the COVID-19 pandemic, we conducted an online
training course for primary and secondary school teachers
from Bosnia and Herzegovina. The course entitled E-
learning: Teaching for Teachers took place during May and
June 2020, organized by the SPARK School Association from
Mostar (www.spark.ba). Most of the course participants were
beginning teachers with no previous experience with online
teaching and this background level determined the structure of
teaching content. The teaching content included basic
knowledge about e- learning and e-learning systems which are
especially prevalent in the formal education system of Bosnia
and Herzegovina. This paper presents the teaching scenario
and the achieved results of one part of the course.
Research, development, and application of online and
hybrid teaching is the result of many years of practice, both
scientific and teaching [4]-[6]. We delivered online teaching
content for the training course with previously prepared
curricula. In this work, we focused on research and analysis of
the learner's assessment in an online course. We created
assumptions that both during learning and teaching (formative
assessment) and knowledge testing (knowledge evaluation)
we enable a more layered picture of learners knowledge. In
such an approach, we will conclude on the most important
question Know how the learner learns [7].
 Within the context of Bloom's taxonomy are focus upon
factual knowledge and the cognitive process - remembering.
The taxonomy of knowledge addresses the relationships
between the dimensions of knowledge and cognitive
processes. For the research presented in this paper, we utilized
the most popular taxonomy of knowledge - the revised
Bloom's taxonomy for the cognitive domain [8].
Following the title of the research, we are oriented towards
the dimension of knowledge - factual knowledge and the
cognitive process - remembering. Factual knowledge is one of
the most fundamental types of knowledge and is often
described as knowledge representing the basic building block
about a particular subject matter [9]. Factual knowledge
includes the terminology, specific details or structure elements
of a subject matter. In terms of motivation and retention of
learner's factual knowledge, digital flashcards are an excellent
method of studying and learning information, not only for
memorize facts quickly, but to process and store the
information over the long-term [10]. Globally, it enables
learning and testing by the method of spaced repetition, the
process of learners self-assessment, or act of self- reflection
known as metacognition [11]-[13].
Our research was conducted with the help of a traditional
learning management system Moodle (moodle.org) and a
digital flashcard system Memory (memory.com) to determine
the relation with the factual knowledge and digital flashcards.
The paper is divided into several sections. Following this
introduction, we describe the research methodology that relies
on the online delivery of teaching content in Moodle and
Memory systems. In the thrid section we present the results
and offer a discussion. The manuscript ends with concluding
remarks.
II. RESEARCH METHODOLOGY
The course considered in this manuscript is designed for
continual improvement of teacher's professional knowledge
including helping teachers switch from face to face to online
teaching. The experiment involved a single group [14] of 24
primary and secondary school teachers from Bosnia and
Herzegovina. The teachers participating in the course were
from the fields of natural, technical, and social sciences.
Experimental factors are learning activities consisting of a
learning period and subsequent assessment, conducted
through two cycles.
TABLE I. SCHEDULE OF LEARNING AND ASSESSMENT CYCLES
Learning period Assessment period
First cycle 25.05.2020.-27.05.2020. 27.05.2020.
Second cycle 27.05.2020.-29.05.2020. 29.05.2020.
After the first phase of the course, we selected two cycles
of learning, teaching and assessing the knowledge of the
participants (Table I). Our research involved the defining of
learning outcomes, the development of learning activities for
each of two cycles, and defining a learning assessment. A brief
description of each of these three elements is provided below.
A. Expected learning outcomes
a. Describe the basic elements and configuration of
e-learning systems
b. Describe and analysis of the hybrid learning
B. Learning activities
a. First cycle (two Moodle lessons)
 Basic structural elements of the e-learning system
(authoring tools, computer-mediated communication,
navigation tools, components for course management,
achievement measure)
 E-learning system configuration (learning management
system, content management system, learning content
management system) and e-learning objects (e-learning
standards, main actors in e-learning standards, Advanced
Distributed Learning, SCORM).
b. Second cycle (Memory flashcard with 67
questions)
 Hybrid learning (face-to-face instruction, online
instruction, definitions of hybrid learning, teacher and
learner role in face-to-face and online instruction)
 Models of hybrid learning (rotation, flexible, a la carte,
enriched virtual)
C. Learning Assessment
a. Delivery of learning content
The delivery of learning content in the first cycle is based
on the learning management system Moodle, while in the
second cycle, the digital flashcard technology was
implemented using the Memory system.
The Moodle learning management system provides the
framework to handle all aspects of the e-learning process. The
infrastructure within Moodle: delivers and manages
instructional content; provides learning and teaching
assessment; tracks and traces learner progress; and presents
learning data activities in the first experiment cycle.
One of the reasons that Moodle was chosen as the control
and testing environment is that it offers rich analytics, the
analysis of which for our course are included in the
manuscript. The Memory digital flashcard system does not
offer teachers the ability to create their own quizzes. Instead,
all quizzes in the Memory flashcard system are automatically
generated using artificial intelligence algorithms.
There are various modes of self-assement in the Memory
system. The smart mode is a combination of all modes which
are intelligently rotated based on the learner's knowledge and
skill level. The practice mode revises questions that the
learner has already completed. The multiple-choice mode is a
great way to learn and practice fast as the learner simply
selects the correct answer from a list of possible answers for
testing their own memory. The GAPS mode allows the learner
to provide 'fill-in-the-blank' answers to the provided
questions. Together, these modes help learners retain
information, boost comprehension, obtain grades, provide
motivation, and offers a means of learner engagement.
There are advanced algorithms embedded in the Memory
flashcard app which further optimize the learning process.
Memory does not use traditional SuperMemo
(SuperMemo.com) or Leitner algorithm [15]; but instead uses
a new proprietary system for delivering questions based on the
use of artificial intelligence. Additionally, Memory
incorporates various inputs such as your hesitation time,
number of backspace hits, flashcard flips, average typing
speed, and other variables to automatically calculate a learners
performance in order to know the best time for repeat
questions and setting due dates for praciting at correct time
intervals [16].
b. Testing structure
After each learning cycle, each learner took the Moodle
quiz to determine the level of acquired knowledge. The Quiz
activity module allows the teacher to design and build quizzes
consisting of a large variety of question types, including
multiple-choice, true-false, short answer, drag and drop
images and text. These questions are kept in the Question bank
and can be re-used in different quizzes [17]. During the
creation of quizzes, we used the same type of questions, all in
accordance with the teaching content to which the testing
refers. Table II shows the distribution of the number and type
of questions in both quizzes.
Quiz 1 was related to the first cycle and two Moodle
lessons, and Quiz 2 was related to Memory flashcards used to
present learning content in the second cycle of learning. We
used four types of questions: True/False, Embedded Answers
(Cloze), Matching and Multiple-choice.
TABLE II. QUESTION TYPES ON MOODLE QUIZZES
Quiz 1 Quiz 2
Question type (number of (number of
questions) questions)
True/False 1 1
Embedded Answers (Cloze) 5 6
Matching 5 6
Multiple-choice 4 2
In connection with all the above, the goal of the research
is to determine whether digital flashcard technology in the
process of learning and teaching improves the cognitive
process of remembering the category of factual knowledge.
 Research question (RQ): Do findings support
instructional efficiency of flashcards and correspond
with the learner’s self-assessment?
III. RESULTS AND DISCUSSION
In this section we present the assessment results which
include quiz statistics, quiz structure analysis, and the
correlation between learning activities and assessment.
A. Quiz statistics
Moodle quiz statistics offered us useful information for
helping us learn about how well the quiz was assisting
learners. It can be observed as a tool for learning analytics.
This report gives a statistical (psychometric) analysis of the
quiz, and the questions within it. Table III provides a
summary of the entire quizzes. By analyzing and interpreting
each indicator we can improve the quizzes, and thus the
learning experience.
For discriminating, deferred feedback, tests aim for the
value of average grade, between 50% and 75%. Values
outside these limits requires revision. Interactive tests with
multiple tries invariably lead to higher averages. A high
average grade may indicate that the quiz was easy for most of
the learners. Values of both quizzes were in the satisfactory
range. Learners scored slightly higher results on Quiz 1 than
on Quiz 2.
TABLE III. QUIZ STATISTICS
Quiz name Quiz 1 Quiz 2
Total number of attempts 19 18
Average grade 69,54% 67,17%
Median grade 72,50% 67,19%
Standard deviation 15,35% 19,66%
Score distribution skewness -0,0899 -0,3227
Score distribution kurtosis -1,1973 -0,6041
Coefficient of internal consistency 73,95% 85,17%
Error ratio 51,04% 38,52%
Standard error 7,84% 7,57%
The standard deviation shows how much variation there
was in the quiz scores among the learners. Aimed values are
between 12% and 18 %. A smaller standard deviation
indicates that quiz scores were similar among the learners.
Value of the Quiz 1 is in the aimed range, and for Quiz 2, it's
slightly above the upper value, suggesting that some learners
had high quiz scores and other participants had quite low
scores.
Skewness is usually described as a measure of a dataset’s
symmetry – or lack of symmetry. Both values are in the range
between -0.5 and 0.5, so the data are fairly symmetrical.
Kurtosis is a measure of the flatness of the distribution. A
normal, bell-shaped, distribution has a kurtosis of zero. A
value for Quiz 1, which is greater than 1 may indicate that the
test is not discriminating very well between very good or very
bad learners and those who are average.
A low value of the Coefficient of Internal Consistency
(CIC) indicates either that some of the questions are not very
good at discriminating between learners of different ability
and hence that the differences between total scores owe a good
deal to chance or that some of the questions are testing a
different quality from the rest and that these two qualities do
not correlate well – i.e. the test as a whole is inhomogeneous.
Higher value on Quiz 2 indicates questions with better
discrimination, i.e. more homogenous quiz.
Error ratio (ER) is related to CIC and it estimates the
percentage of the standard deviation which is due to chance
effects rather than to genuine differences of ability between
learners. Value on the Quiz 1 is slightly above 50%, and
cannot be regarded as satisfactory, it implies that less than half
the standard deviation is due to differences in ability and the
rest to chance effects.
B. Quiz structure analysis
Quiz structure analysis in Moodle is oriented towards
questions, and it is presented in a table format. For this
research, we selected the parameters that are in the focus of
our research, ie facility and discrimination index.
Discrimination index is the correlation between the
weighted scores on the question and those on the rest of the
test. It indicates how effective the question is at sorting out TABLE VI. AVERAGE GRADE PER QUESTION TYPE
able learners from those who are less able [18]. The results Quiz 1 Quiz 2
were interpreted according to the five categories (Table IV). Question type
(average) (average)
General conclusion is that the questions on Quiz 2 have better
True/False 36,84% 50,00%
discrimination. 66.67% of the questions on Quiz 2 have very
good discrimination, unlike the Quiz 1, where only 26.7% of Embedded Answers (Cloze) 38,77% 55,81%
questions belong to the same category of discrimination Matching 69,37% 67,67%
index.
Multiple-choice 95,05% 87,50%
TABLE IV. DISCRIMINATION INDEX DISTRIBUTION
C. Correlation between learning assessment and activities
Quiz name Quiz 1 Quiz 2 After the completion of all activities mentioned in the
upper section, we refined data, and filtered learners that had
Adequate discrimination 5 2 all activities completed. The total number of learners that
Very good discrimination 4 10 completed all required activities is 14. We used the Pearson
correlation coefficient to examine the correlation between
Very weak discrimination 5 1 learner activities on Moodle and Memory and quiz results.
Weak discrimination 1 2 Correlation is an effect size and so we can verbally describe
the strength of the correlation using the interpretation of its
absolute value [20].
The facility index is a measure of how easy or difficult a
question is for quiz-takers. There are nine in total categories The results presented in Table VII included the correlation
of the discrimination index (seven on both quizzes), and in between activities that are in the focus of the research. The
Table V are sorted questions on both quizzes according to the analysis aimed to examine is there a correlation between the
given categories. We notice that the results on Quiz 2 were number of logs on each system and quiz results. Each
interaction of the learners on the Moodle is written in the form
distributed over the three groups, and on Quiz 1 over six
of Moodle logs. Logs in Moodle presents activity reports, and
groups.
we observe them as indicators of learner's engagement on the
course.
TABLE V. FACILITY INDEX DISTRIBUTION
An indicator similar to Moodle logs in Memory is called
Quiz name Quiz 1 Quiz 2 the number of answers. It includes the total number of answers
given by the learner on the system, regardless of the accuracy
Difficult 1 0 and mode in which he learned.
Moderately difficult 1 0
TABLE VII. CORRELATION BETWEEN MOODLE QUIZZES AND LEARNER
About right for the average learner 6 8 ACTIVITY ON EACH SYSTEM

Fairly easy 2 1 Number of Number of

Quiz 1 Quiz 2 Moodle answers in
Easy 0 5 logs Memory

Very easy 4 0 Quiz 1 1

Extremely easy 1 0
Quiz 2 0.0033 1
One part of the analysis relates to learners'
achievement/average grade on particular types of questions. Number of
-0.4609 0.1912 1
As presented in Table VI learners achieved their lowest Moodle logs
average grades on the True/False questions. The second Number of
hardest type of questions were Embedded Answers (Cloze) answers in 0.0421 0.5100 -0.0270 1
Memory
type of questions. Matching and multiple-choice questions
were easier for learners, as the average grade on these
In addition to the discussion, we provide a review of the
questions was high, above 60%. It is notable that the order of
obtained results, as well as the correlation of knowledge and
difficulty of questions types was the same on the both
logs, especially related to the learner instructional
quizzes. This result is related to the fact that learners had
productivity in flashcards/Memory environment.
offered answers to these questions, which are easier to
recognize and thus achieve better results. Formulating
Results presented in Table VII indicates moderate
multiple-choice and matching questions usually includes
correlation for both, Quiz 1 and Moodle logs, as well as the
distractors answers. Distractors are the incorrect answers in
Quiz 2 and Number of answers in the Memory. However,
multiple-choice and matching questions. Choosing the right
although both results belong to the same category in terms of
distractors can make multiple-choice and matching questions
the interpretation of the correlation index, there is a higher
much easier or harder than intended [19]. Multiple-choice
value when we observe the relationship between the results
questions in Memory are generated using artificial
on Quiz 2 and the number of answers in Memory. The result
intelligence technology by using a combination of answers in
indicates that learners who had more activities on Memory
the current course and the course topic [16].
scored better results on the quiz - and that there is a higer
correlation between learner's activity on Memory, than on
Moodle regarding the quiz results. Flashcards as a learning
environment enable self-assessment. The learner himself
assesses whether he has learned something and confirms this
by turning the card. Another confirmation of the correlation
between total answers and results on Quiz 2, is that a learner
with the highest score on Quiz 2 had the highest number of
Total answers in Memory.
One of the reasons why learners achieved high results on
multiple-choice and matching question (Table VI) lies in the
fact that they were able to directly prepare for learning using
multiple-choice mode on Memory.
IV. CONCLUSION
One experimental group research, which included two
different knowledge fields, and conducted during two cycles
of learning and assessment, has confirmed the achievement
of learning outcomes. It should be noted that not all
participants were successful in the course. Of the initial 24
participants, 14 completed all of the obligations during the
learning, teaching, and assessment cycles. There are several
reasons for this. Certainly, the teaching obligations that the
participants had during the course are of utmost importance.
Of all the participants, only three teachers are professionally
oriented towards informatics, while the rest are from social
sciences. Also, for most teachers, this is the first encounter
with online teaching, which is still structured in such a way
that it required a lot of engagement in both, learning and
assessment. It should be noted that the new knowledge was
acquired solely by the online delivered content. This was the
first experience of using flashcards technology for most
participants, who accepted it as an effective learning and self-
assessment environment and mostly praised it. Future
research can go in several different directions. One of the
possible directions should include an experiment in which
participants will have the task of designing their flashcards
and implementing them with students from current teaching
practice. There is a room for the introduction of a two groups
experiment, in which the acquisition of the same field
knowledge on different learning platforms would be
compared. By introducing technologies that enable self-
assessment, they can thus influence the better performance of
learners’ during an assessment, which was confirmed by this
research. Flashcards can be introduced in the learning
independently, but also in combination with other
technologies and resources.
ACKNOWLEDGMENT
This paper is a part of the work supported by the United
States Office of Naval Research grant, N00014-20-1-2066
„Enhancing Adaptive Courseware based on Natural Language
Processing“.
REFERENCES
[1] J. Watson, “Blended Learning: The Convergence of Online and Face-
to-Face Education ,” Vienna , 2008.
[2] I. E. Allen and J. Seaman, “Changing Course: Ten Years of Tracking
Online Education in the United States,” 2013. Accessed: Jun. 26, 2020.
[Online]. Available: www.favazza.com.
[3] R. Graetz, “7 myths about online learning in higher ed,” Mar. 12, 2019.
https://www.ecampusnews.com/2019/03/12/7-myths-about-online-
learning-higher-ed/ (accessed Jun. 26, 2020).
[4] S. Stankov, T. Volarić, and M. Markić, “The Use of Blended Learning
and Peer Tutoring Models at the Faculty of Education,” Suvremena
pitanja, no. 27, pp. 29–57, 2019.
[5] M. Markić, “Evaluation of Quiz Using a Statistical Calculation in
Learning Management System,” Nov. 2017, doi:
10.23919/SOFTCOM.2017.8115535.
[6] S. Stankov and S. Tomaš, “Teachers for the future – key messages and
issues from the first and second round table held within preparations for
the Presidency of the Republic of Croatia of the Council of the European
Union,” Ministry of Science and Education. Jul. 11, 2019, Accessed:
Jun. 26, 2020. [Online]. Available: https://mzo.gov.hr/news/teachers-
for-the-future-key-messages-and-issues-from-the-first-and-second-
round-table-held-within-preparations-for-the-presidency-of-the-
republic-of-croatia-of-the-council-of-the-european-union/3121.
[7] A. Grubišić, S. Stankov, and B. Žitko, “Enhancing Adaptive Courseware
based on Natural Language Processing, Office of Naval Research grant,
N00014-20-1-2066 ,” 2020. https://www.acnltutor.net/ (accessed Jun.
26, 2020).
[8] L. W. Anderson and D. R. Krathwohl, A taxonomy for learning,
teaching, and assessing : a revision of Bloom’s taxonomy of educational
objectives. Longman, 2001.
[9] K. F. Hew and W. S. Cheung, “Enhancing Students’ Learning of Factual
Knowledge,” in Using Blended Learning, 2014, pp. 97–107.
[10] “Free Flashcards from K5 Learning | K5 Learning.”
https://www.k5learning.com/free-flashcards (accessed Jul. 14, 2020).
[11] R. Schmidmaier, R. Ebersbach, M. Schiller, I. Hege, M. Holzer, and M.
R. Fischer, “Using electronic flashcards to promote learning in medical
students: Retesting versus restudying,” Med. Educ., vol. 45, no. 11, pp.
1101–1110, 2011, doi: 10.1111/j.1365-2923.2011.04043.x.
[12] N. Kornell and R. Bjork, “Optimising self-regulated study: The benefits-
and costs-of dropping flashcards,” Memory, vol. 16, no. 2, pp. 125–136,
Feb. 2008, doi: 10.1080/09658210701763899.
[13] “These Flashcard Apps Will Help You Study Better in 2020.”
https://collegeinfogeek.com/flashcard-apps/ (accessed Jul. 14, 2020).
[14] V. Mužić, An introduction to the methodology of research in education
(Uvod u metodologiju istraživanja odgoja i obrazovanj)a, 2nd ed.
Zagreb: Educa, 2004.
[15] “Leitner system - supermemo.guru.”
https://supermemo.guru/wiki/Leitner_system (accessed Jun. 26, 2020).
[16] “Help Centre & FAQ - Memory.com.” https://www.memory.com/help
(accessed Jun. 26, 2020).
[17] “Quiz activity - MoodleDocs.”
https://docs.moodle.org/39/en/Quiz_activity (accessed Jun. 26, 2020).
[18] P. Butcher, “Quiz report statistics,” 2010.
https://docs.moodle.org/dev/Quiz_report_statistics (accessed Jun. 15,
2016).
[19] W. M. I. Maplesoft, “Picking Distractors for Multiple Choice
Questions.” 2008.
[20] J. Evans, Straightforward statistics for the behavioral sciences. Pacific
Grove: Brooks/Cole Pub. Co., 1996.