Vol 20 No 8 August 2021

IJLTER.
ORG
International Journal
of
Learning, Teaching
And
Educational Research
p-ISSN: 1694-2493
e-ISSN: 1694-2116
Vol.20 No.8
International Journal of Learning, Teaching and Educational Research
(IJLTER)
Vol. 20, No. 8 (August 2021)
Print version: 1694-2493
Online version: 1694-2116
IJLTER
International Journal of Learning, Teaching and Educational Research (IJLTER)
Vol. 20, No. 8
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part
of the material is concerned, specifically those of translation, reprinting, re-use of illustrations,
broadcasting, reproduction by photocopying machines or similar means, and storage in data banks.
Society for Research and Knowledge Management

The International Journal of Learning, Teaching and Educational

Research is a peer-reviewed open-access journal which has been
established for the dissemination of state-of-the-art knowledge in the
fields of learning, teaching and educational research.
Aims and Objectives
The main objective of this journal is to provide a platform for educators,

teachers, trainers, academicians, scientists and researchers from over the
world to present the results of their research activities in the following
fields: innovative methodologies in learning, teaching and assessment;
multimedia in digital learning; e-learning; m-learning; e-education;
knowledge management; infrastructure support for online learning;
virtual learning environments; open education; ICT and education;
digital classrooms; blended learning; social networks and education; e-
tutoring: learning management systems; educational portals, classroom
management issues, educational case studies, etc.
Indexing and Abstracting

Research is indexed in Scopus since 2018. The Journal is also indexed in
Google Scholar and CNKI. All articles published in IJLTER are assigned
a unique DOI number.
Foreword
We are very happy to publish this issue of the International Journal of

Learning, Teaching and Educational Research.

Research is a peer-reviewed open-access journal committed to
publishing high-quality articles in the field of education. Submissions
may include full-length articles, case studies and innovative solutions to
problems faced by students, educators and directors of educational
organisations. To learn more about this journal, please visit the website
http://www.ijlter.org.
We are grateful to the editor-in-chief, members of the Editorial Board

and the reviewers for accepting only high quality articles in this issue.
We seize this opportunity to thank them for their great collaboration.
The Editorial Board is composed of renowned people from across the

world. Each paper is reviewed by at least two blind reviewers.
We will endeavour to ensure the reputation and quality of this journal

with this issue.
Editors of the August 2021 Issue

VOLUME 20 NUMBER 8 August 2021
Table of Contents
Washback Impact of the MUET: The Before and After Effect of a High-stake University English Test in Malaysia 1
Najihah Mahmud, Nor Hazwani Munirah Lateh, Nazirah Mahmud, Ariezal Afzan Hassan, Amaal Fadhlini Mohamed, Siti
Amirah Ahmad Tarmizi
The Efficacy of Promoting Financial Literacy with MOOC among Economics Pre-Service Teachers ...................... 18
Khoo Yin Yin, Derek Watson, Rohaila Yusof
The Relationships between Experience, Qualification and Subject Specialization and Content Knowledge Mastery
of Economic and Management Sciences Teachers: A Case of Accounting Teaching ................................................... 36
Habasisa Molise
Effect of the Use of WebQuest in a Chinese for Tourism Course at a Thai University: An Exploratory Study ....... 50
Pairin Srisinthon
Multiple Means of Engagement Strategies for Maximising the Learning of Mathematics in Pandemic-regulated
Classrooms............................................................................................................................................................................. 68
Matshidiso M. Moleko
Online Design Thinking Problems for Enhancing Motivation of Gifted Students....................................................... 91

Ayed H. Ziadat, Mohammad Abed Sakarneh
Teaching-Learning Strategies to Production Planning and Control Concepts: Application of Scenarios to

Sequencing Production with Virtual Reality Support ................................................................................................... 108
Fernando Elemar Vicente dos Anjos, Luiz Alberto Olibveira Rocha, Rodrigo Pacheco, Debora Oliveira da Silva
Medical Students’ Evaluation of Online Assessment: A Mixed-Method Account of Attitudes and Obstacles ..... 126
Abed Alkarim Ayyoub, Oqab Jabali
The Systematic Implementation of an Innovative Postgraduate Online Learning Model in the Middle East ....... 139
Nessrin Shaya, Laila Mohebi
Autonomy and Creative Thinking Skills of Prospective Elementary School Teacher Students in Learning
Mathematics with Science Phenomena assisted by the Learning Management System ........................................... 160
Suprih Widodo, Turmudi ., Rizki Rosjanuardi
E-Learning Implementation Barriers during COVID-19: A Cross-Sectional Survey Design .................................... 176
Ishaq Al-Naabi, Abdullah Al-Abri
Implications of Social Media Addiction on Academic Performance among Generation Z Student-athletes during
COVID-19 Lockdown ......................................................................................................................................................... 194
Wan Ahmad Munsif Wan Pa, Muhammad Sofwan Mahmud, Mohd Syazwan Zainal
The Role of Non-Academic Service Quality in an Academic-Oriented Context: Structural Equation Modeling
amongst Parents of Secondary Education Students ....................................................................................................... 210
Thao-Thanh Thi Phan, Ha-Giang Thi Tran, Le-Huong Thi Nguyen, Tam-Phuong Pham, Thang-The Nguyen, Hiep-Hung
Pham, Thai-Quoc Cao
Developing A Framework Peace Education for Primary School Teachers in Indonesia ........................................... 227
Sahril Buchori, Sunaryo Kartadinata, Syamsu Yusuf, Ilfiandra Ilfiandra, Nurfitriany Fakhri, Sofwan Adiputra
The Measurement of Knowledge Construction in A Course of Diagnostic Evaluation of Learning Disorders in

Psychology Students........................................................................................................................................................... 240
Guadalupe Elizabeth Morales-Martinez, Michelle Garcia-Torres, Maria del Carmen Castro-Gonzalez, Yanko Norberto
Mezquita-Hoyos
The Effect of Resource-Based Instructions on Pre-service Biology Teachers’ Attitudes towards Learning Biology
............................................................................................................................................................................................... 262
Josiane Mukagihana, Catherine M. Aurah, Florien Nsanganwimana
The Effectiveness of a Self-Questioning Strategy at Developing Academic Achievement and Critical-Thinking

Skills among Secondary-School Students in Saudi Arabia ............................................................................................ 278
Ali Tared Aldossari, Mohammed Msnhat Aldajani
The Implementation of Formal Assessments in Intermediate Phase Mathematics at Primary Schools in South
Africa .................................................................................................................................................................................... 300
Senzeni Sibanda, Awelani M. Rambuda
1

Vol. 20, No. 8, pp. 1-17, August 2021
https://doi.org/10.26803/ijlter.20.8.1
Received May 31, 2021; Revised Aug 16, 2021; Accepted Aug 30, 2021
Washback Impact of the MUET: The Before and

After Effect of a High-stake University English
Test in Malaysia
Najihah Mahmud and Nor Hazwani Munirah Lateh

Universiti Malaysia Kelantan, Malaysia
https://orcid.org/0000-0001-7938-1499
https://orcid.org/0000-0002-6968-0513
Nazirah Mahmud
Universiti Sultan Azlan Shah, Malaysia
https://orcid.org/0000-0001-6343-6555
Ariezal Afzan Hassan, Amaal Fadhlini Mohamed

and Siti Amirah Ahmad Tarmizi
Universiti Malaysia Kelantan, Malaysia
https://orcid.org/0000-0002-9806-0874
https://orcid.org/0000-0001-8559-4618
https://orcid.org/0000-0002-4932-6007
Abstract. This study seeks to explore the washback effect of the Malaysian
University English Test (MUET), a high-stakes compulsory university
entry test in the context of Malaysia. As simple and linear as it commonly
appears, washback has been found to be far more complex than simply
looking at the impact that a test might or might not have on the
stakeholders. Therefore, this study aims to fill in this knowledge gap by
systematically re-examining the beliefs on washback by investigating the
relationship between the students’ perceptions of the MUET in terms of
its importance and difficulty, with their language learning strategies
whilst preparing for the test and after sitting the test. Using a mixed
methods approach, a student questionnaire and student interview were
utilised to elicit data from 30 male and 46 female students. The students
were further divided into two groups, specifically those who were
preparing for the MUET and those who had already sat the MUET. The
findings suggest that preparing for the MUET encouraged the students to
utilise a certain language learning strategy more compared to the others,
specifically focusing on language skills that had not been formally tested
before. Although the students’ perception did not significantly shape the
students’ course of action when preparing for the test, it still had an
impact on their overall view of the whole test-taking matter. This study is
©Authors
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0
International License (CC BY-NC-ND 4.0).
2
intended to add more insights to the less explored areas of washback,

specifically the students’ perceptions and washback over time.
Keywords: washback; language assessment; high-stake test; language

testing; backwash
1. Introduction
In the field of education, it is well known that tests, especially high stakes tests,
have a huge influence on the teaching and learning process. Specifically, in
language education, the influence that the tests exert over teaching and learning
is known as ‘washback’ or ‘backwash’. Washback has always been associated with
the negative consequences of tests. However, this phenomenon was not
empirically investigated until the 1980s. Since then, language education
researchers have been looking at washback differently as the findings from the
previous research on washback continually reveal how complex it is (Dong, 2020;
Khan et al., 2019; Hughes, 2021).
Alderson and Wall (1993), in their pioneering study on washback in Sri Lanka,
hypothesised that “Tests that have important consequences will have washback”
and conversely “Tests that do not have important consequences will have no
washback” (p. 120). In other words, the higher the stakes of a test, the more of an
impact it will have on the teaching and learning process. According to Qi (2007),
the authorities are always tempted to resort to manipulating high stakes testing
in the name of ‘curriculum innovation’ for an immediate outcome as it is claimed
to be a “quick and most cost-effective way to improve education” (p.52). Studies
on the washback of high-stakes testing reported that the test could change the
students’ learning behaviour by motivating them to put more effort into their
learning (Cho, 2004; Pan & Newfields, 2012; Thomas, 2005) while also promoting
learner autonomy (Pan, 2014; Stecher, 2002) when preparing for the test if the
stakes are sufficiently high.
2. Literature Review
The connection between (1) testing, and (2) the teaching and learning practices
has been commonly explored based on the research in the field (Barrows et al.,
2013; Cheng et al., 2011; Gebril & Eid, 2017; Luong-Phan & Effeney, 2015) sparked
by the ground-breaking study on washback by Alderson and Wall in the late
1980s. Alderson and Wall (1993), in their significant publication on washback,
raised the notion of the complexity of washback and the needs for more in-depth
research attempting to not only describe what washback looks like but also to
account for what occurs. They also pointed out that although it is widely known
that a relationship between testing, teaching and learning practices does exist, the
complexity of the washback concept itself makes it difficult to prove how directly
tests affect the teaching and learning practices without considering other
mediators or variables that may or may not contribute to the change. Similarly,
Stoneman (2006) observed that the washback phenomenon has yet to be clearly
explained and deeply explored despite the abundant literature on language
assessments focusing on the impact of testing on teaching and learning.
http://ijlter.org/index.php/ijlter
3
In a doctoral study by Mahmud (2018), it was hypothesised that the students’

perception of the test has an impact when it comes to determining the students’
course of action when preparing for the test as opposed to their actual proficiency
level. For instance, even if a student has a weak command of the English language,
if the students think that they can perform well in the test, they will be very
motivated when preparing for the test and vice versa. In a more recent study
undertaken in China involving 3,105 high school students, Dong (2020) found that
the students’ positive perception of the test increased the positive washback of the
test. Test preparation activities may help to increase the students’ probability of
success but it only works in the short-term. If it is too excessive, it can be
counterproductive regarding the students’ achievements in terms of their test
score. Aside from being a waste of the students’ time and energy, Dong (2020)
emphasised that it defeats the purpose of learning and in turn, creates negative
washback.
The MUET was first introduced in 1999 with the aim (1) of “bridge the gap in
English language needs between secondary and tertiary education (Malaysian
Examination Council); and (2) to consolidate and enhance the English proficiency
of students preparing to enter Malaysian public universities” (Lee, 2004, p. 1). The
four language skills of reading, writing, listening and speaking are tested in the
MUET. It is designed and administered by the Malaysian Examination Council
and it is recognised in Malaysia and Singapore (Othman & Nordin, 2013). This
test is significant for pre-degree students as it serves as an indicator of their
English language proficiency which enables them to enrol on their desired course
(Kaur & Nordin, 2006). This study aimed to investigate the washback effect of a
high-stake language test, the Malaysian University English Test (MUET), while
preparing for the test and after sitting the test.
This study investigated the Malaysian students’ perception of the MUET by

exploring its relationship with the language learning strategies that they
employed when preparing for the test. In addition, it also explored how long the
washback effect of MUET lasted after the students sat the test. The research
questions guiding this study were as follows:
1. What are the students’ perceptions of the MUET?
2. To what extent do the students’ perceptions seem to have a washback
impact on the students’ language learning strategies?
3. Is there a difference in the washback impact before and after sitting for the
MUET?
3. Methodology
The use of both quantitative and qualitative data gathering techniques assisted
the researcher in illuminating different aspects of the same issue and providing a
more complete picture of the study (Denscombe, 2014). As this study aimed to
generate data on the students’ perception of the test and their language learning
strategies in relation to the washback effect of the MUET over time, an analytical
comparison was carried out between the students who were preparing for the
MUET and the students who had already sat the test. To provide multiple
perspectives and methods, the data was gathered through the collection of the
4
students’ perceptions using a questionnaire and through interviews. Two groups

of students were recruited for this study, specifically students who were
preparing for the MUET (Group A, n=30) and students who had already sat the
MUET and who were currently undertaking a general English language course at
a public university in Malaysia (Group B, n=46).
Table 1: Participants
Gender Group A Group B Total
Male 11 13 24
Female 19 33 52
Data from Group A was used to describe the washback effect of the MUET whilst
data from Group B was used to explore the washback impact over time, also
known as the washback length.
For this study, the questionnaire was adapted from established questionnaires
from relevant previous studies as detailed in Table 2.
Table 2: Questionnaire
Cronbach’s
Sections Sources
alpha (α)
Background information N/A N/A
Section 1 : Perception of the Xie & Andrews N/A
Test (2013)
Section 2 : Perception of the Bodas (2006) .833
Test Importance
Section 3 : Perception of the Bodas (2006) .805
Test Consequences
Section 4 : Language Learning Oxford (1990) .890
Strategies Green (2007)
In general, the questionnaire consisted of two main sections. The first section
covered the demographic questions such as gender, field of study, English
proficiency level etc. Section two consisted of four sub-sections, dealing with the
students’ (1) perception of the test (MUET) in general, (2) their perception of the
test importance, (3) their perception of the test consequences and (4) Language
Learning strategies (Oxford, 1990). All of the variables in section two were
assessed using a Likert scale.
This study also utilised the students’ individual interviews to gather the
qualitative data. As pointed out by Atkins and Wallace (2012), interviews not only
allow the researchers to engage with the participants individually, but it also
allows them to collect various types of in-depth data, for example, factual data,
views and opinions, personal narratives and histories. According to Ary et al.
(2013), there is no general rule for determining the number of participants for the
purpose of collecting qualitative data. For this study, 12 open-ended questions
with several probing questions on the students’ experiences of learning English
were prepared for the student interviews. The questions were adapted based on
5
the previous studies on washback (Hsu, 2010; Mahmud 2018; Shih, 2013) focusing
on the learners’ general perception of their English language learning and the test
under investigation, the MUET.
4. Findings
Item analysis of the mean scores of the students’ perceptions for all constructs was
carried out to better understand how the students responded to each item
individually before analysing the items according to their grouping. The student
questionnaire consisted of four constructs: (i) the perception of the MUET in
general, (ii) the perception of the test importance, (iii) the perception of the test
consequences and (iv) the language learning strategies. In the item analysis, the
constructs were treated as the dependent variables, with gender as fixed factors,
and English language proficiency and experience within the test as covariates.
Taking into consideration that the data was not normally distributed, non-
parametric analyses, the Mann-Whitney U-test and the Kruskall-Wallis test were
employed to analyse the aforementioned constructs both as individual items and
as an overall scale. To ensure a greater level of reliability, only significant
differences in the mean scores (p < .05) were taken into consideration. The
findings were arranged according to the sub-sections as follows.
4.1 Students’ Perception of the MUET

4.1.1 Perception of the MUET in general
The mean scores for all items were calculated and arranged in descending order
as illustrated in Table 3. A Likert scale (comprised 5 for strongly agree, 4 for agree,
3 for undecided, 2 for disagree and 1 for strongly disagree) was used to score the
items.
Table 3: The MUET in general

Items Std.
Mean Deviation
1.4 The MUET made me practise my listening skills more than before. 4.30 .611
Strongly
Agree
1.12 More speaking activities should be conducted in MUET 4.29 .629

preparation class.
1.5 The MUET made me practise my speaking skills more than before. 4.24 .671
1.11 More listening activities should be conducted in MUET 4.16 .590
preparation class.
1.6 The MUET made me practise my reading skills more than before. 4.11 .723
Agree
1.13 More writing activities should be conducted in MUET preparation 4.11 .759
class.
1.10 More reading activities should be conducted in MUET preparation 4.03 .765
class.
1.3 The MUET made me practise my writing skills more than before. 3.99 .721
Agree
1.2 My English language learning was improved by practising MUET 3.89 .665
past year questions.
1.9 I think that the MUET preparation class that I took was not very 2.46 1.113
Disag
ree
helpful.
6
The trend seems to be that the highest-ranking items with mean scores that are
within the range of ‘Strongly Agree’ are those related to the listening and speaking
skills. These two skills are seldom tested in other standardised English language
test in Malaysia as opposed to reading and writing. This clearly indicates the
washback that the MUET had on these students, especially towards the two skills
which were not tested before. The mean scores for the set of items regarding the
students’ increase in effort connected to their reading and writing skills ranges
from 3.99 to 4.11, which are ranked very high. Item 1.9 has the lowest mean score
(2.46) which falls under the ‘Disagree’ range. This indicates that most of the
students perceived the MUET preparation class that they took as being beneficial
to them.
Next, the analysis of the overall scale was carried out. The Mann-Whitney U-test
was run and the analysis revealed there to be no significant differences in terms
of the students’ general perception of the MUET in relation to their (i) gender, p =
.906, (ii) experience with the test, p = .074, and (iii) proficiency level, p = .442. The
result suggests that the students' general perception of the MUET did not change
even after they had already taken the test. The students’ proficiency level in
English did not affect their general perception of the MUET as well.
4.1.2 Perception of Test Importance

The perception of the test importance construct consisted of five items as shown
in Table 4.
Table 4: Test Importance

Std.
Items
Mean Deviation
2.3 It is very important for me that I do well in the MUET 4.42 .753
Strongly
Agree
2.4 It is very important for my future undertakings that I do 4.38 .816

well in the MUET
2.5 Every student who wants to get into the university should 4.13 .900
pass the MUET
Agree
2.2 It is very important for my teacher that I do well in the 3.97 1.045
MUET
2.1 It is very important for my parents that I do well in the 3.70 .980
MUET
The top two items are items relating to the students’ priority of their own self, as
opposed to external factors like their teachers (item 2.2) and parents (item 2.1),
which were ranked as the bottom two.
At the item level, both the Mann-Whitney U-test and the Kruskal-Wallis test
showed there to be no significant differences across gender (p = .506), English
proficiency level (p = .619), and experience with the test (p = .619) for each item in
the perceived test importance construct. This shows that the students of all
English proficiency levels regarded the MUET as an important test. This finding
confirms the status of the MUET as a high-stakes test. The students were also
7
asked about their perception of the consequences that they had to face if they
failed the MUET, which has been presented in the next section.
4.1.3 Perception of the Consequences of Exam Failure

The perception of the test consequences construct consisted of six items as shown
in Table 5. A Likert scale (comprising 5 for extremely to 1 for not at all) was used
to score the items.
Table 5: Consequences of Exam Failure

Std.
Items
Mean Deviation
3.2 How upset would you be for letting your teachers down? 4.14 .948
3.1 How upset would you be for letting your parents down? 3.99 1.052
Very
3.3 Will your chance to get into top universities be affected? 3.93 1.037
3.4 Will your chance to enrol on your desired course be 3.91 1.048
affected?
3.6 Will your ability to communicate in the English language 3.03 1.376
Undecided
affected?
3.5 Will your motivation to learn the English language be 3.03 1.366
affected?
Interestingly, the top two items with the highest mean score for the test
consequences construct are the items that are extrinsic in nature, pertaining to
living up to others’ expectations, namely their teachers (item 3.2) and parents
(item 3.1). Following very closely are the items related to the main objective of the
MUET, which is where it acts as an entry test into university. The students agreed
and were aware that their application to their desired university (item 3.3) and
courses (item 3.4) would be affected if they performed poorly on the MUET.
However, they appeared to be unsure whether their ability to communicate and
their motivation to learn English would be affected if they did poorly in the
MUET. Based on the overall ranking, it is safe to assume that the severity of the
consequences of exam failure for the MUET is perceived to be quite threatening
to the students.
Next, the Mann-Whitney U-test was run to determine whether there were any
differences in the perception of the test consequences between the three
independent variables of gender (p = .699), experience with the test (p = .321) and
English language proficiency (p = .496). The findings revealed that there were
differences between the variables, but they were not statistically significant.
Similar to the findings for perceived test importance, the students’ proficiency
level in English did not seem to affect their perception of the test consequences.
The findings thus far show that there were no significant differences reported
between the students’ perception of the test and the three independent variables.
However, at the item level, the Kruskal-Wallis test revealed there to be significant
differences across the English language proficiency levels for item 3.6 (Will your
ability to communicate in the English language affected?, p=.042) with the mean
8
ranking across the grouping in an ascending manner being as follows: Low

(22.83), Average (39.60) and High (47.06). Interestingly, the findings reveal that
the students might regard the MUET as one of the main determinants of their
English language ability, especially for high proficiency students.
4.2 Relationship between the students’ perceptions and washback impact on

language learning strategies
In order to explore the washback impact of the MUET on the students’ learning,
item analysis was carried out to determine the students’ pattern of responses
based on the frequency of their perceived language learning strategy usage. The
mean scores for all items were calculated and arranged in descending order as
illustrated in Table 6. A Likert scale (comprising 5 for all the time, 4 for most of
the time, 3 for undecided, 2 for sometimes and 1 for never) was used to score the
items.
Table 6: Language Learning Strategies

Std.
Items
Mean Deviation
4.15 When writing in English, I tried to translate from my
4.08 1.043
language.
4.10 When reading in English, I tried to translate it into my
4.00 1.007
language to help me understand.
4.6 If I couldn’t think of an English word, I used a word or
3.99 1.026
phrase that means the same thing.
4.16 I thought about the goals that I wanted to achieve in this
3.97 1.078
English language course.
4.18 When I received corrected work from the teacher, I
3.89 1.078
thought about how to improve next time.
4.14 To understand unfamiliar English words, I tried to guess
3.82 1.186
High
their meaning.
4.8 I encouraged myself to use English even when I was afraid
3.82 1.055
of making a mistake.
4.17 I tried to improve my writing by analysing the work of
3.71 1.198
other writers.
4.2 I tried to find better ways of learning English. 3.70 1.178
4.3 I tried to improve my English by asking others to correct my
3.66 1.250
mistakes.
4.12 I used new English words in sentences so then I could
3.54 1.259
remember them.
4.13 When I learned a grammar rule, I tested myself to make
3.51 1.301
sure that I really knew it.
4.1 I memorised English words by saying or writing them
several times. 3.09 1.246
4.11 I was NOT sure how to improve my English skills. 2.92 1.393
Medium
4.4 I did the MUET practice tests in my free time. 2.83 1.320
4.7 I reviewed my English class notes or textbook in my free
2.82 1.230
time.
4.9 I read English without looking up every new word. 2.75 1.406
4.5 I studied extra English outside of my MUET preparation
2.67 1.258
class.
9
The findings revealed that the students reported a moderate-to-high frequency of

using language learning strategies to prepare for the MUET. Specifically, the
students used meta-cognitive strategies (items 4.2, 4.11, and 4.18), often associated
with learning success, at a high frequency. A high frequency of usage was also
reported for cognitive strategies related to rote-translation (item 4.15 and 4.10). As
depicted in Table 6, in terms of the individual learning strategies for all of the
strategy items belonging to the meta-cognitive (planning, organising and
evaluating learning) and compensation (to overcome difficulties in
communication) categories, the students reported a high frequency of use with a
mean score ranging from 3.70 to 3.99. Meanwhile, none of the language learning
strategies were reported at a low frequency of use. For the strategies used at
moderate frequencies, the findings show that most of the strategies were related
to the learning activities that the students did in their free time, for example, item
4.4 (I did MUET practice tests in my free time) and item 4.7 (I reviewed my English
class notes or textbook in my free time). Finding opportunities to practice outside
of the classroom (item 4.5) was the least used strategy among the students, which
is not surprising. This strategy requires self-initiation and courage, which many
young learners in a second language context might not possess.
To determine the relationship between the students’ perception of the test

importance and their language learning strategies, Kendall's tau-b correlation was
carried out. There was found to be a weak positive association that was not
statistically significant between perceived test importance and language learning
strategies as a whole, τb = .140, p = .089, and direct learning strategies, τb = .149,
p = .076, and a very weak positive association between perceived test importance
and indirect language learning strategies, τb = .091, p = .283. There was no
statistically significant association between the students’ perceived test
importance and their use of language learning strategies.
Next, there was found to be a weak positive association, which was not
statistically significant, between the perceived test consequences with language
learning strategies, τb = .154, p = .058 and direct learning strategies, τb = .153, p =
.065, and a very weak positive association between the perceived test
consequences and indirect language learning strategies, τb = .082, p = .326. There
was no statistically significant association found between the students’ perceived
test consequences and their use of language learning strategies. Therefore, the null
hypothesis cannot be rejected, and the alternative hypothesis cannot be accepted.
The data suggests that the students’ perception of test importance and the test
consequences did not seem to affect the students’ usage of language learning
strategies.
4.3 The before and after effect of the MUET

This section presents the analysis of the individual interviews with two students
from Group B, which consisted of students who had already taken the MUET. The
two students were labelled R1 (Female, MUET Band 2) and R2 (Male, MUET Band
3) to ensure anonymity. Students R1 and R2 were randomly chosen from a list of
the students who volunteered to be interviewed.
10
When it comes to English language learning, it was evident from the qualitative
data that both students seemed to be particularly concerned with grammar:
“For me it’s difficult, in terms of grammar. And also, the usage of the
words.” (R2)
“Even when we enter the university, we still have to learn grammar

because it’s easy to forget. We have to really focus in the class.” (R1)
This finding can be attributed to the washback effect of the 11 years of formal
English language education in Malaysia. This is because grammar comes first in
the syllabus before any of the four language skills. Therefore, most Malaysian
students are under the impression that before they work on their language skills,
they must make sure that their grammar is good.
Another skill that they were concerned with when it comes to English language
learning was speaking skills. Both agreed that for them to improve their speaking
skills, they had to practice:
“We have to practice it a lot. We have to make it a habit. For example,
when communicating with the teacher, we are supposed to use the English
language.” (R2)
“For speaking skills, we have to always use it. If we want to improve our
speaking skills, we should mix around with the Chinese students, that’ll
help.” (R1)
The interview participants were also asked about their current English language
learning experience. When asked to describe how their current English language
class is, the following are their descriptions:
“Normally the teacher would emphasize on the syllabus from the textbook.
For example, we were involved in drama production for an event called
‘Drama Night’, so the teacher asked us to develop our script which took
months to finished. Then, after the event, we continued with the syllabus
from the textbook.” (R1)
“The teacher taught us based on the textbook, and if there was any
question, we would discuss it together. Then, the teacher also asked us to
present in front of the class.” (R2)
Obviously, emphasis was given to the activities in the textbook. However, these
particular students prefer something more interactive as opposed to being
constrained or restricted by the syllabus. They mentioned the activities that were
conducted in their previous semester’s English class, which was English I. They
seemed to like it and perceived it as helping them to improve their English
language skills:
“I remember last semester’s English teacher, for example, when we were
learning about grammar, the teacher would make up a story about it or
the teacher would create songs about the grammar that we were learning.
11
That way, we could memorize it better. Well actually we memorized the

song, but at the same time, we were learning grammar as well.” (R1)
“My teacher last semester conducted some sort of game. In my opinion,

after we played that game, it has somehow helped improve my English
because throughout the game, the teacher made it compulsory for us to
use English language, forcing us to speak in English. I think that’s quite
helpful.” (R2)
Moving on to the MUET, the students were asked to recall their experience
preparing for the high-stakes language test. Questions pertaining to what they did
inside and outside of their classes were asked to see what kind of learning
activities and strategies they used, and to determine the intensity of the washback
of the MUET. The participants were first asked about their perception of the
MUET in general compared to the other English language tests that they had taken
before. The following were their responses:
“For me, MUET is important because it helps us improve especially our
speaking skills, it encouraged the students to speak in English more
because they definitely did not want to get low marks, so they would try
to speak with other people to practice. And then for listening skills, for
example my friends, if previously they were mostly listening to Malay
songs, but because listening was tested in MUET, they started to listen
to English songs more.” (R1)
R1 described the MUET as having more challenging questions for the writing
component and more questions for the reading component. She also mentioned
that the MUET encourages students to speak more in English because they do not
want to get low marks for the speaking component. Being tested on all four
language components forced the students to work on all four skills. Improvising
the four skills required different learning strategies as mentioned by R2 below:
“MUET does not consist of only one test, but it has different papers for
listening, speaking, writing, and reading, so for each of these skills,
different preparation strategies are required. For example, when I took
SPM in the past, it was more on the writing skills only, but for MUET,
we must be able to grasp all of the skills, and for speaking, we have to be
able to communicate well.” (R2)
R2 mentioned that the previous English language test that he took mainly focused
on writing skills. Since the MUET also tested his speaking and listening skills, he
had to apply different learning strategies to prepare himself for it. He stated that
his teacher devoted one whole period of the English lesson to teaching each
language skill separately:
“For example, if for this period, the teacher decided to focus only on
listening skills, for other period, the teacher will focus only on speaking
skills and so on.” (R2)
He also mentioned the use of textbooks in the MUET preparation class. In R1’s
case, her teacher emphasised the listening and speaking activities more in the
classroom:
12
“The teacher focuses more on the listening and speaking skills. For
listening skills, the teacher provided us with MUET text book. The teacher
would play the CD in the classroom for us, then the teacher would ask us
to answer the questions in the text book. For speaking skills, the teacher
divided us into groups of four, because in MUET speaking test, the
students are divided in the same way, so the teacher would imitate the real
situation in MUET speaking test.” (R1)
Similar to R2’s MUET preparation class, R1’s teacher also utilised textbooks to
prepare the students for the test. The textbook that R1 mentioned is a
commercialised MUET preparation textbook sold at ‘Popular’, one of the main
bookstore chains in Malaysia. In her comment, R1 mentioned that for speaking,
her teacher mimicked the real MUET situation to familiarise the students with the
format. However, her teacher only did this when the date of the MUET drew
closer, which was two weeks before the actual test took place:
“The teacher would act as the examiner. But, during that time, it was just
two weeks before the actual test took place. Before that, the teacher just
asked us to practice on our own.” (R1)
The researcher also asked the students to comment on the importance of the
MUET to them. R1 initially was not aware of the purpose of the MUET or the use
of the MUET result, not until after she had taken the test:
“When I sat for MUET at the matriculation college, I did not even know
that MUET result would affect my university admission result. My
teacher did not say anything about it. My teacher just mentioned that it
is important, that is all.” (R1)
R1 stated that the reason why her teacher did not say anything on the importance
of the MUET was because her teacher did not want to pressure her students. Her
teacher thought that there would not be any problem for the students when it
came performing well in the MUET as most of them managed to score above Band
3 in the mock-MUET practice test.
“My teacher did not tell us about it because he did not want us to be
stressed out. It was also because when we did a pre-test for MUET, he told
us that all of us would be able to get Band 3 and above. That was why my
teacher did not say anything.” (R1)
For R2, the only thing that he knew about the objective of the MUET was that it
was one of the requirements of applying to tertiary education in Malaysia.
“All I know is MUET is compulsory to gain entry into the university,
that’s all.” (R2)
R1 specifically described her difficulties when applying for her desired course as
most of the degree courses required at least Band 3 results in the MUET:
“When I was applying for the university, there were a lot of courses that
I could not apply because these courses require at least Band 3. It was
quite difficult for me. So, I just chose Band 2 courses because I only got
Band 2 in MUET.” (R1)
13
She ended up choosing courses with a set minimum requirement of MUET Band
2, as those were her only options. However, despite getting a low MUET Band,
she did not let her MUET Band define her English language proficiency:
“For speaking skills, the percentage for the overall score is less than
reading and writing, hence, those who like to read books, or overly focused
when answering the questions, they would be able to perform well. I mean,
unless if the percentage for all skills are equal, then maybe MUET can be
used to really measure the overall English language ability. For me, no.
Imagine those who manage to get Band 4, but are not able to speak
fluently, it’s still the same.” (R1)
She stated that since the MUET has different weightages for the different
components, like the reading component contributes the highest percentage to the
overall Band, students who have good reading skills might be able to perform
better. For her, getting a higher Band in the test with no ability to speak the
language well would still be pointless.
5. Discussion
5.1 Students’ Perception of the MUET
As can be seen in the findings of this study, although limited in scope, there is
clear evidence of the washback effect of the MUET especially for the listening and
speaking skills as indicated by the quantitative data. Although the MUET consists
of all four language skills, reading and listening have not been formally tested
before in any standardised English language test in Malaysia other than the
MUET. Therefore, once these two skills were added to the test, the learners started
to pay more attention to those two skills as well. A scrutiny of the qualitative data
showed that the students were particularly concerned with their speaking and
listening skills as well as opposed to their writing and reading skills. They also
mentioned that their teachers focused more on these two skills during the English
lessons. As suggested by Nambiar and Ransirini (2012), both the teachers and
students tend to focus more on the tasks that they consider to be imperative to the
outcome of the test. Different washback effects depend on the perceived task
importance. Although the MUET tested all four skills, it is safe to assume that due
to the novelty effect, the students and teachers decided to focus more on the
listening and speaking skills because they have been dealing with reading and
writing skills for the past 11 years of formal education in Malaysia. This was also
the case in Dong’s (2020) study where the students were found to rarely engage
in communicative learning, for instance speaking, as it was not tested.
The quantitative findings also suggest that the students’ general perceptions
about the MUET were not influenced by either their gender or their English
proficiency level. Their perceptions of the MUET also did not change even after
they had taken the test. Similarly, when it comes to the students’ perception of the
importance of the MUET to them, their gender, English proficiency level, and their
experience with the MUET did not seem to affect their perception as well. This
shows that regardless of their English language proficiency, the students regard
the MUET as an important test. This confirms the status of it as a high-stakes test.
The findings also revealed that failing the test threaten the students’ chances of
enrolling on their desired course and to their intended university. This frustration
14
was clearly expressed by one of the interview respondents as her choices when
applying to tertiary education were limited due to her low MUET score.
However, when it comes to the test consequences, the students who were
preparing for the MUET appeared to be more worried about the consequences of
failing the test and how it would affect their motivation to learn and their ability
to use the English language. This can be attributed to a phenomenon known as
test anxiety, as they had yet to sit for the MUET and did not know what to expect.
Those who had already sat the MUET seemed less concerned. This might be due
to the fact that they have gone already through the whole experience of the MUET
and managed to pull through well in the tertiary level as they were currently
studying in a university at the time that this study was conducted.
5.2 Language Learning Strategies

One of the objectives of this study was to see if the MUET encourages students to
employ more language learning strategies when preparing for it. The quantitative
findings revealed that the students reported a moderate-to-high frequency of
language learning strategy use to prepare for the MUET. None of the language
learning strategies were reported at a low frequency of use. A closer analysis of
the quantitative data disclosed that specifically, the students used cognitive
strategies related to rote-translation at a high frequency. The qualitative data
revealed similar findings in that the students utilised translation techniques to
help them learn. Cognitive strategies like rote-translation are forms of direct
learning strategies that, according to Pan (2014), are not deep learning strategies
that can really help the students acquire the necessary language skills. In his
study, he found that most of the students reported to have frequently used
traditional language learning activities such as reading textbooks, memorising
vocabulary and idioms, and practicing sentence patterns to name a few. A
moderate obsession with grammar was also observed in the responses given by
the interview respondents in this study. Similar findings were reported by Shih
(2013). He found that most of the students in his study seemed to employ more
surface strategies in their English language learning process rather than deep
strategies. This type of strategy is mostly geared towards scoring on the test and
will not benefit the students in the long run.
To see whether the students’ perception of the MUET could be associated with
their language learning strategy use, correlation analysis was carried out on
perceived test importance and perceived test consequences in relation to the
language learning strategies. The results suggest that the students’ perception of
the test importance and test consequences are not statistically significantly
associated with the students’ usage of language learning strategies.
5.3 The before and after effect of the MUET

As reported by the qualitative data in this study (see section 4.3), it can be seen
that the students were made to practice the skills that had not been tested before
in their formal education in Malaysia more, which were speaking and listening
skills. This is because the MUET tests all four language skills. However, they
perceived that their speaking skills had particularly improved since they
practiced both at school and in real life. Although the students expressed a
15
reservation when speaking in English, it was encouraging to see a slight self-

realisation regarding the need to speak the language more in order to help them
improve their skills.
As a high-stakes test, the MUET results affect the students significantly, as the
MUET is one of the requirements for university entry in Malaysia. Certain courses
like medicine, engineering and TESL require a slightly higher MUET band
compared to others. Not being able to score the minimum MUET band
requirement for their desired course would result in the student having to take
another course at the university. This explains the strong washback impact that
the MUET can impose on the students due to its high stakes. If utilised correctly
by the stakeholders, especially the educators, the problems related to encouraging
the students to practice their language skills more can be tackled in due time.
5.4 Limitation of the study

It should be noted that the number of participants in this study was fairly small
and that the grouping was not normally distributed. The findings from the
analysis should not be taken at face value. More holistic data needs to be collected
when attempting to explore a complex phenomena like washback, hence, more
student interviews need to be carried out for both groups, not only two Group B
students.
5.5 Implications of the study

It can be seen in this study that the perception of the test could be one of the
important factors involved in determining test washback. According to Dong
(2020), among the stakeholders, the teacher is determined to be the most
important individual affecting the teaching and learning process. Hence, the
teacher can promote the proper perspective of the test among the students which
in return could help them to promote positive washback and improve the
students’ performance in the long run. Several studies tapping into washback
have indicated that washback changes over time, hence more research should
investigate this. The present study attempted to explore a part of the washback
length element by comparing the before and after effect of washback. This study
is hoped to add more insights to the less explored areas on washback, specifically
the students’ perceptions and washback over time.
6. Conclusion
It was apparent from the findings that the students’ actual proficiency in the
English language did not have a washback impact on the students’ course of
action when preparing for a high-stakes test like the MUET. Although the
quantitative data revealed a similar result in relation to the students’ perceptions,
the qualitative data appeared to reveal a glimpse into the relationship between
the students’ perceptions and their influence on their selected language learning
strategies when preparing for a test. This calls for further exploration regarding
the washback impact of a test utilising a much bigger sample with more diverse
participants and instruments.
16
Acknowledgments
This work was supported by a research grant provided by Universiti Malaysia
Kelantan (R/SGJP/A0400/01060A/001/2019/00596).
7. References
Alderson, J. C., & Wall, D. (1993). Does washback exist? Applied Linguistics.
https://doi.org/10.1093/applin/14.2.115
Ary, D., Jacobs, L., Sorensen, C., & Walker, D. (2013). Introduction to research in education:
Cengage Learning.
Atkins, L., & Wallace, S. (2012). Qualitative research in education. Sage Publications.
Barrows, J., Dunn, S., & Lloyd, C. A. (2013). Anxiety, self-efficacy, and college exam
grades. Universal Journal of Educational Research, 1(3), 204-208.
Bodas, J. (2006). Intra-individual and Extra-individual predictors of text anxiety in Indian
children: A cross-cultural perspective (Doctoral dissertation, Virginia Tech).
Cheng, L., Andrews, S., & Yu, Y. (2011). Impact and consequences of school-based
assessment (SBA): Students’ and parents’ views of SBA in Hong Kong. Language
Testing, 28(2), 221-249.
Cho, D. (2004). Use of standardized tests as university graduation requirement. English
Teaching, 59(1), 251–266.
Denscombe, M. (2014). The Good Research Guide: For Small-scale Social Research Projects (5th
Edition). UK: McGraw-Hill Education.
Dong, M. (2020). Structural relationship between learners’ perceptions of a test, learning
practices, and learning outcomes: A study on the washback mechanism of a high-
stakes test. Studies in Educational Evaluation, 64, 100824.
https://doi.org/10.1016/j.stueduc.2019.100824
Gebril, A., & Eid, M. (2017). Test Preparation Beliefs and Practices in a High-Stakes
Context: A Teacher’s Perspective. Language Assessment Quarterly, 14(4), 360–379.
https://doi.org/10.1080/15434303.2017.1353607
Green, A. (2007). IELTS washback in context: Preparation for academic writing in higher
education (Vol. 25). Cambridge University Press.
Hsu, H. F. (2010). The impact of implementing English proficiency tests as a graduation
requirement at Taiwanese universities of technology (Doctoral Dissertation, University
of York).
Hughes, L. (2021). Washback and the assessment practices of ESL instructors at Japanese
university. Language Literacy: Journal of Linguistics, Literature, and Language
Teaching, 5(1), 1-9. https://doi.org/10.30743/ll.v5i1.3238
Kaur, N., & Nordin, R. (2006). A case for reconstruction of the pedagogy of the Malaysian
University English test (MUET) through thematic units instruction. Journal of
Institutional Research South East Asia, 4(1), 5–16.
Khan, A. B. M. A., Aziz, M. S. A., & Stapa, S. H. (2019). Examining the factors mediating
the intended washback of the English language school-based assessment: Pre-
service ESL teachers’ accounts. Pertanika Journal of Social Sciences & Humanities,
27(1), 51-68.
Lee, K. S. (2004). Exploring the connection between the testing of reading and literacy: The
case of the MUET. GEMA Online Journal of Language Studies, 4(1), 41–50.
Luong-Phan, N. H., & Effeney, G. (2015). TOEFL iBT and language learning motivation:
An investigation into teaching styles and influential factors for Vietnamese
adolescents. International Journal of Research Studies in Language Learning, 4(3), 3-18.
Mahmud, N. (2018). Investigating the Washback Effect of the MUET as a University Entry Test
on Students in Malaysia (Doctoral dissertation, University of York).
17
Nambiar, M., & Ransirini, S. (2012). Teaching MUET, not English: A study of the washback
effect of the Malaysian University English Test (MUET). English in Multicultural
Malaysia. Pedagogy and Applied Research, 49-62.
Othman, J., &Nordin, A. B. (2013). MUET as a predictor of academic achievement in ESL
teacher education. GEMA Online Journal of Language Studies, 13(1), 99–111.
Oxford, R. L. (1990). Language learning strategies: What every teacher should know. Boston:
Heinle & Heinle Publishers.
Pan, Y. C. (2014). Learner washback variability in standardized exit tests. TESL-EJ:
Teaching English as a Second or Foreign Language, 18(2), 1–30.
Pan, Y. C., & Newfields, T. (2012). Tertiary EFL proficiency graduation requirements in
Taiwan: A study of washback on learning. Electronic Journal of Foreign Language
Teaching, 9(1), 108–122.
Qi, L. (2007). Is testing an efficient agent for pedagogical change? Examining the intended
washback of the writing task in a high‐stakes English test in China. Assessment in
Education: Principles, Policy & Practice, 14(1), 51–74.
https://doi.org/10.1080/09695940701272856
Shih, P. C. (2013). The English benchmark policy for graduation: An investigation of perception,
motivation, and approaches to learning at a university of technology in Central Taiwan
(Doctoral dissertation, Durham University).
Stecher, B. M. (2002). Consequences of large-scale, high stakes testing on school and
classroom practice. Making Sense of Test-Based Accountability in Education, 79–100.
Stoneman, B. W. H. (2006). The impact of an exit English test on Hong Kong undergraduates: a
study investigating the effects of test status on students' test preparation behaviours
(Doctoral dissertation, The Hong Kong Polytechnic University).
Thomas, R. M. (2005). High-stakes testing: Coping with collateral damage. Routledge.
Xie, Q., & Andrews, S. (2013). Do test design and uses influence test preparation? Testing
a model of washback with Structural Equation Modeling. Language Testing, 30(1),
49-70.
18

Vol. 20, No. 8, pp. 18-35, August 2021
Received Jun 29, 2021; Revised Aug 16, 2021; Accepted Aug 30, 2021
The Efficacy of Promoting Financial Literacy

with MOOC among Economics Pre-Service
Teachers
Khoo Yin Yin

Sultan Idris Education University
https://orcid.org/0000-0003-4850-2184
Derek Watson
University Sunderland
https://orcid.org/0000-0002-1944-3544
Rohaila Yusof
Sultan Idris Education University
https://orcid.org/0000-0001-5304-8970
Abstract. It is important to educate financial literacy to pre-service

teachers to manage their money well and to disseminate the knowledge
to their students. The study aimed to examine the effectiveness of MOOC
in promoting financial literacy among pre-service teachers. This paper
also examines the correlation among the variables of financial literacy,
collaborative skills, problem-solving, accountability, and decision-
making. Pre-service teachers' views were discussed after attending the
MOOC. The study adopted a mixed-method research design. A total of
100 pre-service teachers were selected through a random-sampling
technique to participate in the survey. A purposive sampling technique
was used to pick five of the participants for the follow-up interview. The
quantitative data were analyzed with descriptive and inferential
statistics, while the qualitative data were analyzed on the basis of content
analysis. The quantitative method yielded favourable descriptive
statistics; and they also showed a significant relationship between
financial literacy and MOOC. The variables indicated some correlations.
The qualitative results also revealed the positive responses of financial
literacy, collaborative skills, problem-solving, accountability, and
decision-making. The course design and participants' engagement are the
key factors of the success of MOOCs. Future research should emphasize
the incorporation of financial literacy into different subjects. Educators
and researchers could well provide more online learning in addition to
those activities that combine financial literacy in other courses.
Keywords: financial literacy; Economics pre-service teachers; MOOCs;

collaborative skills; accountability; problem-solving
©Authors
19
1. Introduction
Massive open online course (MOOC) is a widespread application of information-
technology education (Haron et al., 2019). Initially introduced in Malaysia in 2015,
MOOCs are now considered a new e-learning platform that was later
implemented by the Malaysian Higher Education Institutions (Kumar & Al-
Samarraie, 2018). By utilizing the platform of MOOCs, it is possible to create a
conducive learning forum that allows learning to happen anywhere and in any
place (Hassan et al., 2015 Qaffas et al., 2020). It is also recognized that MOOC
provides open learning to the enrolled participants; and the students can follow
the course at their own place and pace.
Most of the MOOCs in Malaysia offer academic courses; nevertheless, a minority

of courses have failed to embrace the essential skills, such as financial literacy. If
financial literacy is not prioritized, there could well be negative consequences. For
example, a total of 47% of bankruptcies in Malaysia involved young adults (Nurul
Afiqah, 2016; Khoo & Fitzgerald, 2017). There is a significant tendency for
millennials to spend beyond their means; since they are often easily influenced by
the social media; and, in consequence, they are inclined to buy expensive and
branded goods (Caixeta et al., 2016; Khan et al., 2019). For example, 38.8% of
young people between the ages of 20 and 24 repeatedly upgrade their
smartphones to the latest model (Malaysian Communications and Multimedia
Commission, 2015).
In contrast, the consequences of the financial crisis have elevated the importance
of financial literacy. The Organization of Economic Co-Operation and
Development (2014) raised the importance of financial literacy; and it was highly
recommended that it should be included within the school curriculum. The OECD
further stressed the importance of educating financial literacy to all citizens, in
order to instil better financial management of their own funds and to enhance
their financial well-being. The Malaysian Government supported this study; and
it is evident in other countries, such as the United States and Australia, which
have integrated these recommendations (Cordero & Pedraja, 2019).
Financial literacy has been the only cross-curricular element in Malaysia's

curriculum since 2017, resulting in the Financial Education Network being
launched in 2019 (Financial Education Network, 2019). Teachers are encouraged
to include financial literacy in their classes, in order to enhance their students'
awareness of financial literacy (Sawatzki & Sullivan, 2017). Evidence suggests that
teachers no longer regard financial literacy as not being a key priority; since it is
not a standalone subject in the Malaysian secondary-schooling system (Mohd
Aziz & Kassim, 2020). This view has influenced students’ perception in learning
financial literacy. Furthermore, teachers rarely attend developmental courses;
and consequently, they struggle to integrate this critical skill into their schemes of
work effectively. However, research in this area is rather limited; and this reflects
the research gap. In addition, the academic debate of utilizing MOOCs in financial
literacy is limited; and in consequence, there is a research gap to be filled. This
issue is the research gap that is yet to be filled in this study. Pre-service teachers,
who will become qualified teachers, should be exposed to an incorporated
financial literacy, which would later be transmitted to the students during class.
20
Unfortunately, some teachers claimed that they could not cover financial literacy
in class, due to time constraints (Rakow, 2019). The same study also mentioned
that many teachers are frequently pressed for time; and they are forced to omit
specific topics, such as financial literacy. These barriers could be bridged if
teachers were to undertake proper training on the pedagogical application and
positive impacts of financial literacy. However, due to the lack of financial literacy
for graduates, they must seek advice from financial planning services for personal
financial management.
On a more positive note, researchers have indicated that students, who studied
financial education developed financial literacy, when compared to those who did
not attend the class (Murugiah, 2016; Zhu, 2018; Haneger & Cude, 2019). The
OECD further stated that education could help bridge the financial-literacy gap,
and that financial education programmes have been found to successfully
improve students' and adolescents' financial knowledge and attitudes (OECD,
2013).
Many researchers only focused on pedagogical skills and on the content

knowledge of teacher-training courses; in fact, some essential life skills should
also be emphasized among them (Prajapati et al., 2017). Since every pre-service
teacher has both generic and bespoke learning styles, the practical learning
platform should be tailored, in ordered to meet the learners' needs. The learning
platform must have the necessary flexibility to encompass various learning
pedagogies, such as live-case studies, presentations, etc. MOOC is a popular
learning mode that offers such flexibility; and furthermore, it is an affordable way
to learn.
In addition, prior research showed evidence that financial literacy can be

effectively taught, together with the help of digital tools (Angel, 2018; Kuntze et
al., 2019). Angel (2018) and Kuntze et al. (2019) shared the same views that online
learning using different devices, could enhance financial literacy. Therefore,
MOOCs, together with words and videos, are the most suitable platform from
which to implement financial literacy.
The main contribution of the current study to the practical gap, is to help pre-
service teachers to acquire important skills, such as collaborative skills, and to
incorporate financial literacy into their lessons through practical activities. In
addition, this study has also contributed to the literature on Economics education
that focuses on community pre-service teachers. Despite the lack of financial
literacy among young adults, there has also been a lack of research that could
improve financial knowledge and promote financial awareness among pre-
service teachers. The population of pre-service teachers concerning the
effectiveness of financial literacy through digital platforms is the significant result
of this study.
2. Objective
This paper aimed to answer the following research objectives:
1. To explore the effectiveness of MOOC in promoting financial literacy among
pre-service teachers.
21
2. To investigate the correlation among financial-literacy variables, collaborative

skills, accountability, problem-solving skills and decision-making.
3. To examine pre-service teachers' views on financial literacy after attending the
MOOC.
3. The Literature review

3.1 The Conceptual Framework
This study is based on Mayer's Multimedia Learning Theory. In Mayer's
Multimedia Learning Theory, the student engages in five important cognitive
processes. Mayer (2014) stated that meaningful learning from words and pictures
occurs, when the learner engages in five cognitive processes, which are:
1. Choosing the relevant words for processing in verbal-working memory;
2. Selecting relevant images for processing in visual-working memory;
3. Organizing the selected words into a verbal model;
4. Organizing selected images into a pictorial model; and
5. Integrating the verbal and pictorial representations with prior knowledge.
These cognitive processes determine what information is to be drawn upon in the

working memory, when knowledge is retrieved from one’s long-term memory and
integrated with new information, in order to construct new knowledge. The new
knowledge in the working memory will be relocated and transferred to long-term
memory through the process of encoding (Mayer, 2014). Refer to Figure 1. MOOCs
have met the criteria of Multimedia Learning; since they are colorful, animated
pictures and can transform words into pictures. Furthermore, students can cope with
the knowledge of financial literacy by utilising their tacit knowledge.
Multimedia Sensory Working Memory Long-Term

Presentation Memory Memory
Selecting Organizing Verbal
Ears Sound
Words Words Words Mode
Integrating Prior
Knowledge
Selecting Organizing Pictoral

Pictures Eyes Images
Images Images Mode
Figure 1: Conceptual Framework: Mayer Multimedia Learning Theory
3.2 Learning with MOOCs

Learners worldwide have benefitted from MOOCs (Sidek et al., 2019; Gabaree et
al., 2020;). During the Covid-19 pandemic, learners were able to evaluate their
courses at home. Consequently, MOOCs were elevated as a new teaching method
that overcame the limits of traditional teaching platforms, by allowing students to
learn from any place and at any time (Wang, 2021). Learners invariably benefit
from observing videos or by reading lecture notes online. This is a type of remote
learning that enables students to learn outside the physical classroom. As a result,
22
MOOCs are created under the open-concept or open-education umbrella

(Quiliano Terreros et al., 2009). This platform has demonstrated that it supports
lifelong learning, in acquiring key knowledge and skills and the much-needed
life-skill interactions with peers and live-project exposure (Sonwalkar &
Maheshkar, 2015).
Researchers have identified the key approaches to incorporate MOOCs into

classroom lesson plans and schemes of work, including explicitly defining the
content, selecting material and subjects, specifying the MOOC and its range, and
ensuring that the scope of the MOOC is easily accessible to learners. Additional
strategies, such as learning aim, objectives, teaching activities, and assessment, are
also central in the preplanning and in delivering effective lessons (de Jong et al.,
2019).
MOOCs grades, behavioral (Phan et al., 2016) outcomes and social engagement
were found to have favorable connections (Gillani & Eynon, 2014; Torres & Beier,
2018). According to these studies, educators and learning designers are
encouraged to include learning activities in MOOCs relevant to career demands,
in order to increase learners' extrinsic motivations (Deng et al., 2020). In this
report, the researchers also said that learners' engagement might be further
stimulated by cognitively engaging the course and by inserting suitable formative
assessment into such tasks. The length of MOOCs should likewise be kept to a
minimum, in order to maintain pupils’ attention. As with all online programmes,
the most challenging aspect of MOOCs is student dropout. Students would
invariably leave the course if they become demotivated; and if they fail to manage
their time effectively. To compensate for this, MOOCs should last between 1 and
16 weeks. However, the MOOCs’ inventor strongly suggests that programmes
should last no longer than seven weeks.
Various studies reported that financial literacy could be effectively enhanced

through online learning (Kalmi, 2017). All previous studies suggested that using
digital tools, such as videos or games, could effectively enhance students' financial
literacy. Additional studies also revealed that videos, or other digital tools, could
effectively improve financial knowledge, whether in college or in high school
(Zhu, 2019; Popovich et al., 2020). MOOCs with multimedia, in line with the
conceptual framework of this research, were selected as being the most suitable
platform for promoting financial literacy in this paper.
4. Methodology
4.1 Research Design
A mixed-methods research design was used in this study. A survey design was
used in quantitative research; whereas, in qualitative research, an interview was
used. The researchers selected the mixed-method research method; since they felt
a need to comprehend this topic throughout various research phases, and as part
of the validation process (Creswell & Clark, 2017).
23
4.2 Population and Sampling technique

The target population of this study is 100 pre-service Economics teachers in
Peninsular Malaysia. Consequently, a total of 100 pre-service teachers, majoring
or minoring in Economics, were selected as quantitative samples. Each sample
selected was between the ages of 20 and 21. A random sampling procedure was
used to select the respondents. The target respondents were represented with a
code number, and the selection was based on the randomly generated number by
using Microsoft Excel.
A method known as purposive sampling was employed for the interviewed

respondents. Five teacher candidates were also chosen for a follow-up interview.
The goal of the follow-up interview was to ensure the triangulation of the data.
The participants were selected on the basis of their daily expenditure. Although
gender was not considered in this study, two boys and three girls were selected
to participate in the interview.
4.3 Instrument
The principal researcher created a 50-item closed-ended questionnaire, with five
Likert scales, which were then checked by professionals: Financial literacy,
collaborative skills, accountability, problem-solving, and decision-making were
the five main components of this set of questionnaires. There were ten questions
in each section. The questionnaire's reliability was tested in a pilot study with 53
pre-service teachers; while a group of information-technology experts tested the
MOOC platform's functionality. All the items in this survey were found to be
reliable and acceptable, with a value of 0.70 or above (Sekaran & Bougie, 2010);
because the questionnaire's Cronbach Alpha was 0.832. The pre-service teachers
were given an online assessment with 20 questions regarding financial literacy, in
order to test their content understanding.
4.4 The Research Procedure

All of the pre-service teachers attended a six-week MOOC course. During hands-
on activities, the candidates were divided into 25 groups of four each. This
learning activity should improve their collaborative learning via peer dialogue
and debate.
The following was the schedule for the six-week course:

All pre-service teachers were required to attend a two-week course on
understanding fundamental financial literacy for the first two weeks. The pre-
service teachers progressed to learning how to incorporate financial literacy into
their lessons. Financial planning was also taught to the pre-service teachers. On
week three, the pre-service teachers were asked to choose a secondary school
economics sub-topic and to construct teaching activities relevant to financial
literacy. From weeks three to week six, the researchers peer-shared videos that
they had produced.
The pre-service teachers were responsible for designing teaching aids, based on
their lesson plan in week four. Pre-service teachers could construct a game on
conserving money and knowledge about trust and bonding for money and
banking. They evaluated their peers' work from different groups and provided
24
constructive feedback and remarks in week five. During the final unit of week six,
the pre-service teachers were required to prepare a short case relating to their
daily life, together with a financial literacy component. During this week, pre-
service teachers were required to complete an online quiz and a questionnaire
regarding financial literacy. Finally, five pre-service teachers, from various
groups, were selected to participate in the interview in week seven.
The five pre-service teachers were all third-year students at a public university.
Three females and two males were selected as interviewees. They all participated
in the MOOC as self-directed learners, devoting at least one hour every week to
the course. Although they were from various groups, they all completed and
submitted the tasks on time.
4.5 The Data-Collection Method

After watching all of the video clips, the questionnaires were handed out in
person, and the data were collected anonymously by using a code-number
system. This study was also conducted prior to the global pandemic; and it was
modelled on a real-life classroom scenario.
Five pre-service teachers were chosen to attend the interview session separately,
and each session lasted about 30 minutes. With the permission of the pre-service
teachers, the interviews were recorded. The following were the open-ended
interview questions:
• How do you incorporate financial literacy into teaching?
• Do you feel that your collaborative skills have improved after attending
MOOC?
• Could you understand the importance of accountability in financial
literacy?
• Could you tell me about the problem-solving skill that you learnt, after
attending the course?
• Do you agree that decision-making is an essential skill in financial literacy?
And if so, why?
4.6 The Data Analysis
The quantitative data were analyzed with descriptive and inferential statistics.
The five main components of the questionnaire were used to compare the mean
score; and a t-test was used to determine the significant difference after
implementing MOOC among the pre-service teachers. An unstandardized
coefficient model was developed to calculate the coefficient. The Pearson
correlation was also employed, in order to check the relationship among the
variables. SPSS version 23 was used to compute the data.
The qualitative data were then analyzed via content analysis. Content analysis
techniques were used to interpret the transcripts of each subject, both explicitly
and implicitly throughout the analytical process (Krippendorff, 2019). It
commenced with the process of making transcriptions for five interviews, after
the completion of three interview sessions. According to Miles and Huberman
(2019), the data that were collected from the interviews should be systematically
compiled after transcriptions had been made. The interview recording was
25
listened to several times, in order to ensure the comprehension thereof; and then
the data were stored electronically.
5. The Results
5.1 To examine the effectiveness of MOOC for promoting financial literacy
among pre-service teachers
The survey results were examined and summarized in Table 1. According to the
questionnaire summary, the subscales of the surveys were financial literacy,
collaborative skills, accountability, problem-solving, and decision-making. The
highest average mean score for ten categories was problem-solving (M=3.80,
SD=.776), followed by financial knowledge (M=3.75, SD=0.772). Accountability
received the lowest mean score (M=3.41, SD=0.934).
Table 1: Summary of the Questionnaire on MOOC

Sub-scale Item Average Mean Score S.D
Financial Knowledge 10 3.75 0.772
Collaborative Skills 10 3.36 1.033
Accountability 10 3.41 0.934
Problem Solving 10 3.80 0.776
Decision-Making 10 3.52 0.846
The results in Table 2 revealed that both the standardized and the unstandardized
beta coefficients were significant. The value of the unstandardized coefficients
was 69.238 (SD=6.837), and the t value was 10.097 (p<0.05). On the other hand, the
standardized coefficients showed a value of 0.073 with a t value of 0.721 (p<0.05).
According to the unstandardized coefficient, increases in the financial literacy of
one unit should boost students' use of MOOC. The variance in financial literacy is
examined through the semi-partial correlation = 20% (0.4522). The Tolerance value
is 0.751,which higher than 0.63 (1- R2), which showed that there could not be any
multi-collinearity.
Table 2: Coefficients
Model Unstandardized Standardized t Sig. Correlations Collinearity
Coefficients Coefficients Statistic
Β Std. Beta Zero- Partial Part Tolerance VIF
error order
Constant 69.238 6.857 10.097 0.00
MOOC 0.067 0.093 0.073 0.72 0.00 0.478 0.45 0.37 0.751 1.315
* significant at p < .05

DV: Financial literacy
Adjusted R2 = .37
5.2 To investigate the correlation among the variables of financial literacy,

collaborative skills, accountability, problem-solving and decision-making
The results of the Pearson correlation are shown in Table 3. There were ten
different correlations, which means that ten coefficients were statistically
significant. Financial literacy was significantly correlated with collaborative skills,
r = 0.118 (p < 0.05). In addition, financial literacy also correlated with
accountability significant, r = 0.216 (p < 0.05). In addition, financial literacy
26
correlated with decision-making, r= 0.227 (p < 0.001) and problem-solving r = -

0.240 (p > 0.001).
On the other hand, collaborative skill was significantly correlated with all the
variables. For example, collaborative skills correlated with accountability, with r
= 0.288 (p < 0.001), problem-solving with r = 0.202 (p < 0.001), decision-making
with r = 0.346 (p < .001). Accountability was correlated with all the variables, such
as problem-solving with r = 0.369 (p <0.001), accountability with r = 0.327 (p
<0.001) and decision-making r = 0.494, (p <0. 001).
Table 3. Pearson Correlations among the variables

Correlations
Financial Collaborative Accountability Problem Decision
literacy skills Solving Making
Pearson 1 0.118* 0.216** -0.240** 0.227**

Financial Correlation
literacy Sig. (2-tailed) 0.018 0.000 0.427 0.000
N 100 100 100 100 100
Pearson 0.118* 1 0.288** 0.202** 0.346**
Collaborative Correlation
skills Sig. (2-tailed) 0.018 0.000 0.000 0.000
N 100 100 100 100 100
Pearson 0.216* 0.288** 1 0.369** 0.494**
Correlation
Accountability
Sig. (2-tailed) 0.000 0.000 0.000 0.000
N 100 100 100 100 100
Pearson -0.040 0.202** 0.369** 1 0.327**
Problem Correlation
Solving Sig. (2-tailed) -0.240** 0.000 0.000 0.000
N 100 100 100 100 100
Pearson .227** .346** .494** .327** 1
Decision Correlation
Making Sig. (2-tailed) 0.000 0.000 0.000 0.000
N 100 100 100 100 100
*. Correlation is significant at the 0.05 level (2-tailed).
**. Correlation is significant at the 0.01 level (2-tailed).
5.3 Pre-service teachers' views after attending MOOC

A total of five pre-service teachers attended the interview. To protect their
identities, they used pseudonyms. Their opinions on financial literacy,
collaboration, accountability, problem-solving, and decision-making were polled.
All of these enquiries were related to the questionnaires.
5.3.1 Financial literacy

Eighty per cent of the participants said that they strongly believed their financial-
literacy understanding had improved, and that they could incorporate the lessons'
skills. They were well-versed in how to instil financial literacy in students; and
they had created appropriate teaching aids. When asked how to incorporate
financial literacy into teaching, these individuals said:
27
"… select a suitable topic, like demand and supply, by using a video clip." (Male
1)
"… I would start with a story; and I would then give the class a problem-solving
task." (Female 2)
"… set an online platform, with an element of financial literacy, and then allow
students to have self-directed learning." (Female 3)
This cohort of pre-service teachers had mastered many skills through MOOC; and
they expressed their enthusiasm for MOOC learning. They had also gained
financial knowledge through the platform (Deng & Benckendorf, 2021). Pre-
service teachers created teaching aids, shared their ideas with their colleagues,
and commented on their peers' work.
5.3.2 Collaborative Skills

During group discussions, one can always improve collaborative skills. MOOC
provides a venue for pre-service teachers to learn and share. To finish the
assignment, they had to work together. Each of the participants has a certain role
to play, based on the work at hand. Each week, the group leaders were rotated
among the pre-service teachers. The job had to be distributed equitably among the
pre-service teachers by the leaders (Duret et al., 2019). For example, during a
session of creating teaching aids, one person would prepare the video's storyline
and script, two members would prepare the video, and a third member would get
the task of editing the video. The successful completion relied heavily on team
collaboration. When asked whether they had developed collaborative skills after
attending the MOOC, they expressed their delight; and they noted that they each
had contributed something:
"… I did my work, according to our leader's instruction, and I completed
my task within the allocated timeframe. We learnt how to help each other
every week, and we learnt from each other too. Everybody acquired
different strengths." (Female 1)
Aside from collaborative abilities, pre-service teachers were accountable for their
work. As a result, responsibility is a necessary skill in the teaching profession. It
also has a significant impact on financial literacy.
5.3.3 Accountability
Accountability is an important element in financial literacy (Rakow, 2019). All the
pre-service teachers agreed that they understood the importance of accountability
in financial literacy. They also practised it when they had completed the task given
during MOOC learning. They learnt the meaning of accountability in financial
literacy through the weekly videos provided. Some of them tried to incorporate
the value of accountability, when designing the teaching aids.
"… I put the value of accountability into my teaching aid. I made a music
video related to financial literacy for the chapter on money and banking;
then I incorporatedl this value." (Male 2)
All of the pre-service teachers agreed that accountability in financial literacy is an

important element. They practised it as well; since they completed the MOOC
learning activity. Through the weekly videos, they learned about the need for
28
accountability in financial literacy. When creating instructional tools, some of the

pre-service teachers attempted to incorporate the concept of accountability.
5.3.4 Problem-solving
One of the pre-service teachers provided a proactive problem-solving example.
During the MOOC, she used a case study, as a teaching tool. She asked the
students to assume the scenario that they were alone, without savings or
investments. The scenario included the visualization of their payday, in which a
snatch robber took their handbag/wallet. Meanwhile, they had to imagine that
they had received a call from their hometown, informing them that their mother
was gravely ill, and required financial assistance. The question that was posted
among pre-service teachers was: 'What are your options for resolving this issue?'
"… I try to figure out the suitable situation for high-school students who
had just started to learn financial literacy. I made the question a little
trickier; and I let them think that I had set a few learning activities for me
to incorporate into the lessons during my teaching practicum." (Female
1)
Other pre-service teachers provided a good example of problem-solving, too. The

activities that the pre-service teachers devised could be used during their
practicum. By doing so, the pre-service teachers gained further insights and
content knowledge through practical exposure and case-study analysis (Wright et
al., 2017).
5.3.5 Decision-Making
The comments of the pre-service teachers suggested that one of the components
they had to acquire in financial literacy and collaborative work in MOOC was
decision-making. They all agreed that financial literacy requires good decision-
making skills. Decision-making is an essential skill in financial literacy
(Sonwalker & Maheshkar, 2015). They also expressed their opinions on the
necessity of deciding to complete the work. For example, they chose their lesson
plan or learning activities and the teaching aids they created were based on
mutual agreement and decision-making. In order to meet the requirements of
incorporating financial literacy into lessons, they had to decide on a topic, what
materials to use as teaching aids, and what learning activities to implement.
"… I kept on discussing matters with my members, when I was the leader
of the group. I cannot decide on my own; I have to consider all their
abilities and strengths. Some of them are good in making videos, but
others are good in animation or singing." (Male 2)
6. Discussion
The effectiveness of MOOC in increasing financial literacy was investigated in this
study. In descriptive statistics, the quantitative results demonstrated positive
results for financial literacy. they showed a medium-high average, ranging from
3.41 to 3.80, because the MOOC was implemented in a short duration. On a short-
term basis, the influence of learning, particularly value, could not be seen.
However, in terms of regression, the results were significant (t value 10.097)
(p<0.05) for an unstandardized model and a t value of 0.721 (p<.05) for a
standardized model.
29
The Pearson coefficient revealed a correlation for financial literacy. The results
showed that financial literacy was statistically related to collaborative skills,
accountability, problem-solving, and decision-making. The results are consistent
with the literature review of Bakar and Bakar (2020) and Compen et al. (2020). It
was discovered that financial literacy might well be improved through MOOCs
and digital learning. MOOC is a learning platform that allows learners to learn
from professionals worldwide, and thereby go enhance their abilities (Clark et al.,
2017). In Mayer’s Learning Theory, students could learn new knowledge better by
combining words and pictures, for example, videos in MOOC.
The qualitative results revealed that MOOC could improve collaboration skills,
problem-solving skills, accountability, and decision-making skills. Previous
research had found that online or blended learning improves collaborative skills
(Compen et al., 2020). Studies also reported that online learning is able to enhance
problem-solving (Yeen-Ju et al., 2015). On the other hand, online learning also
promotes accountability (Pulukuri & Abrams, 2020), as well as the decision-
making ability of learners (Galvis, 2018; Compen et al., 2020).
During the course, pre-service teachers were required to create videos, in order to
obtain practical experience and to learn how to include financial literacy into the
curriculum. The videos were created by pre-service teachers; and they included
technology, animation, and problem-solving activities. These videos serve as a
wake-up call for students interested in learning about financial literacy. Pre-
service teachers also chose a problem that was relevant to their students' daily
lives. These qualitative results supported the conclusions of Kuntze et al. (2019)
that videos are an effective online method for delivering financial literacy
knowledge outside the classroom, and for reinforcing it.
Additionally, videos can be used as post-class reinforcement tools for increasing

students' learning interest, allowing them to study more interactively (Pulukuri &
Abram, 2020). The MOOC course also provided a forum for pre-service teachers
to socialize, share ideas, and engage in practical exercises. When pre-service
teachers were involved in the learning process, they developed a positive outlook
(Gurvitch & Lund, 2014). This mindset should benefit them in both their
behaviour and in their future teaching. This finding is consistent with the research
of Amagir et al. (2019).
In addition, previous research mentioned that ICT significantly benefits learners

(Gaboy et al., 2020), especially in financial literacy (Lusardi et al., 2015), which is
similar to the results found in this study. Both studies also mentioned visual tools;
videos that can enhance financial literacy more interactively. In addition, with the
assistance of fingertip technology, students can learn anywhere and at any time
(Shah et al., 2019). Moreover, universities can engage with schools and invite
students to engage in their learning platforms with the assistance of mobile phone
technology (Che Kob et al., 2020).
Furthermore, previous research has found that ICT benefits learners (Gaboy et al.,
2020), particularly in financial literacy (Amagir et al., 2019), which is consistent
30
with the results of this study. Both types of research addressed visual aids, as well
as videos that can help people to learn more about money. Furthermore, students
can learn anywhere and at any time, by using finger-tip technology (Shah et al.,
2019). With the advancement of mobile phone technology, universities may
engage with schools and ask students to participate in their learning platforms
(Che Kob et al., 2020).
The use of mobile phone technology might be a solution for teachers who claim
that they do not have enough time to incorporate financial literacy into the
curriculum because it explores a learning platform that allows self-directed
learning beyond the classroom.
7. Limitations
The following were some noticeable limitations in this study:
Firstly, the samples of this study were restricted to pre-service teachers from
major or minor economics only. Pre-service teachers from other courses were not
included in this study. Therefore, studies should be conducted among different
courses before any generalizations of financial literacy can be made. Secondly, in-
service teachers were excluded from this study. This scenario has limited the
results regarding the generalization of the efficacy of financial literacy for
MOOCs.
8. Future Research
Pre-service teachers should produce and share projects on various topics about
financial literacy across the curricula in various subjects when using MOOCs. A
possible issue to investigate the incorporation of financial literacy in different
subjects, such as Accounting and Business Management will probably become the
future research direction. Financial planning is another future research direction;
since this issue is only a small segment of the current research. In future research,
the researchers can consider doing in-depth financial literacy training for pre-
service and in-service teachers in terms of investment and possible retirement
plans.
9. Recommendations
From the results, we would like to make some recommendations that might be
useful to educators and researchers. These engagements were divided into two
categories: course design and participation engagement.
• Course design
Course design is an essential aspect of attracting and retaining
participants. A robust course considering the students to get the basic
knowledge and make the students learn with pleasure would result in a
better success percentage. The course content could consider elements
related to students' prior knowledge, whereas learning activities can
design familiar activities. When educators design any courses related to
financial literacy, learning activities such as videos, quizzes, and games
could be a good choice for initial learning.
• Participants’ engagement in participation
31
To make the course successful, one should try to include as many

activities as possible that need to be implemented in the participation.
Hand-on activities are one way to let students engage in the learning
activities; for example, students can work in a group to produce a video
related to financial literacy. Teachers' lively teaching approach, or group
activities coulc also attract students' participation. Students could gain
basic financial knowledge through a platform that attracts their interest.
Another option is to send a notification or a push button to participants
to inform them of the task's deadline.
10. Conclusion
In this study, pre-service teachers were exposed to various events that put them
in perspective and inspired them to be innovative in teaching their students'
financial-planning skills and principles. Alternatively, financial planning is a vital
part of children's discipline, which ought to be instilled. This study researched the
various approaches to teach pre-service teachers how to instil supportive ideals
in their students to become responsible for themselves. The positive results of this
study are encouraging, given the literature review related to financial literacy
related to collaborative skills, accountability, problem-solving and decision-
making. The mentioned skills are transferable skills that are needed to enhance
financial literacy. From the results, the study suggests that MOOC coulc promote
financial literacy among pre-service teachers effectively. Therefore, educators and
researchers could provide more online learning opportunities that combine
financial literacy in Economics, Accounting, Mathematics, or even in languages.
Personal finance is rarely discussed at school or at home; educators should seize

the opportunity to teach the future generation the importance thereof. We
are assisting parents in protecting the future of their children by shaping
the next generation.
Acknowledgement
This research has been carried out under the Fundamental Research Grants
Scheme (FRGS/1/2019/SS08/UPSI/02/3) provided by the Ministry of Education
of Malaysia. The authors would like to extend their gratitude to Universiti
Pendidikan Sultan Idris (UPSI), that helped to manage the grant. The authors also
would like to thank Vasiliki Kondou Watson for editing in this paper.
11. References
Angel, S. (2018). Smart tools? A randomized controlled trial on the impact of three
different media tools on personal finance. Journal of Behavioral and Experimental
Economics, 74, 104-111. https://doi.org/10.1016/j.socec.2018.04.002
Amagir, A., Groot, W., Brink, H. M. V. D., & Wilschut, A. (2019). SaveWise: the design of
a financial education program in the Netherlands. Citizenship Social & Economics
Education, 18(2), 100-120. https://doi.org/10.1177/2047173419870053
Bakar, M. Z. A., & Bakar, S. A. (2020). Prudent financial management practices among
Malaysian Youth: The moderating roles of financial education. The Journal of Asian
Finance, Economics and Business, 7(6), 525-535.
http://doi.org/10.13106/jafeb.2020.vol7.no6.525
32
Caixeta, C. C., Nascimento, J. A., & Abrem, A. M. (2016). The influence of social media on
consumer perception on brand. Espacious, 37(27), 17-26.
http://dx.doi.org/10.5195/emaj.2019.173
Che Kob, C. G., Kannapiran, S., & Shah, A. (2020). The usage of mobile learning:
comparative studies among technical and vocational education students in
selected universities. International Journal of Interactive Mobile Technologies, 14(5),
203-209. https://doi.org/10.3991/ijim.v14i05.13355
Clark, K. R., Vealé, B. L, Watts, L. K. (2017). A Review of the Use of Massive Open Online
Courses (MOOCs) in Medical Imaging Education. The Internet Journal of Allied
Health Sciences and Practice, 23, 15(2), 1-6.
https://nsuworks.nova.edu/cgi/viewcontent.cgi?article=1659&context=ijahsp
Compen, B., Witte, K. D., & Schelfhout, W. (2020). The impact of teacher engagement in
an interactive webinar series on the effectiveness of financial literacy education.
British Journal of Educational Technology, 1-15. http://doi.org. 10.1111/bjet.13013
Cordero, J. M., & Pedraja, F. (2019). The effect of financial education training on the
financial literacy of Spanish students in PISA. Applied Economics, 51(16), 1679-1693.
http://doi.org.10.1080/00036846.2018.1528336
Creswell, J. W., & Clark, V. L. P. (2017). Designing and conducting mixed-methods research
(3rd ed.). California: Sage Publication.
de Jong, P. G. M., Pickering, J. D., Hendriks, R. A., Swineerton, B. J., Goshtasbpour, F., &
Reinders, M. E. J. (2019). Twelve tips for integrating massive open online course
content into classroom teaching. Medical Teacher, 42(4), 393-397.
https://doi.org/10.1080/0142159X.2019.1571569
Deng, R., Benckendorff, P., & Gannaway, D. (2020). Linking learner factors, teaching
content and engagement patterns with MOOC learning outcome. Journal of
Computer Assisted Learning, 1-18. https://doi.org/10.1111/jcal.12437
Deng, R., & Benckendorff, P. (2021). What are the key themes associated with the positive
learning experience in MOOCs? An empirical investigation of learners' ratings
and reviews. International Journal of Education Technology in Higher Education, 18
(9), 1-28. https://doi.org/10.1186/s41239-021-00244-3
Duret, D., Christley, R., Denny, P., & Senior, A. (2019). Collaborative learning with
peerwise. Research in Learning Technology, 26, 1-14. http://doi:0.25304/rlt.v26.1979
Financial-Education Network. (2019). Empowering financial freedom.
https://www.fenetwork.my/about/
Gabaree, L., Rodeghiero, C., Presicce, C., Rusk, N., & Jain, R. (2020). Designing creative
and connected online learning experiences. Information and Learning Sciences, 121
(7/8), 655-663. https://doi.org/10.1111/jcal.12437
Gaboy, R. G., Mabalay, M. C., Mananghaya, M. E., Mercado, M. G. M., & Romblon, B. M.
(2020). Coping with the new norm: ICT-Pedagogy integration awareness and
competency of TEI faculty. Journal of Research, Policy & Practice of Teachers & Teacher
Education, 10(2), 49-62. http://doi.org.10.37134/jrpptte.vol10.2.4.2020
Galvis, A. H. (2018). Supporting decision-making processes on blended learning in higher
education: literature and good practices review. International Journal of Educational
Technology in Higher Education, 15(25), 2-38. http://
10.37134/jrpptte.vol10.2.4.2020
Gillani, N., & Eynon, R. (2014). Communication patterns in massively open online courses.
The Internet and Higher Education, 23, 18-26.
http://dx.doi.org/10.1016/j.iheduc.2014.05.004
Gurvitch R., & Lund J. (2014). Animated video clips: Learning in the current generation.
Journal of Physical Education, Recreation and Dance, 85(5), 8–17.
https://doi.org/10.1080/07303084.2014.897566
33
Haron, H., Yusof, A. R. M., Samad, H., Ismail, N., Juanita, A., & Yusof, H. (2019). The
platform of MOOC (Massive Open Online Course) on open learning: Issues and
challenges. International Journal of Modern Education, 1(3), 01-09.
http://doi.org/10.35631/ijmoe.13001
Hassan, H., Mailok, R., & Johan, R. (2015). UPSI students' perceptions on the use of ICT in
learning: comparison between knowledge testing and task-based testing on ICT
competency. Journal of ICT in Education, 2, 55-
77. https://ejournal.upsi.edu.my/index.php/JICTIE/article/view/2600
Haneger, R., & Cude, B. J. (2019). Financial literacy of high school graduates: Long and
short-term financial behaviour by age group. Journal of Family and Economic Issues,
40, 564-575. http://10.1891/1052-3073.27.1.3
Kalmi, P. (2017). The effects of financial education: Evidence from Finish lower secondary
schools. Economic notes, 9999 (9999), 1-34. http://dx.doi.org/10.1111/ecno.12114
Krippendorff, K. (2019). Content analysis: An Introduction to its methodology. Sage
publication. https://us.sagepub.com/en-us/nam/content-analysis/book 258450
Khan, H. U. R., Lim, C. K., Tan, K. L. & Khan, M. R. (2019). Location-based social media
content: A conceptual paper on travellers’ purchase decisions. Management
Research Journal, 8, 30-40. https://doi.org/10.37134/mrj.vol8.3.2019
Khoo, Y. Y., & Fitzgerald, R. (2017). Pocket cartoons: Learning financial literacy with
mobile cartoons in Malaysia. International Journal of Mobile and Blended Learning,
9(4), 49-64. https://doi.org/10.37134/mrj.vol8.3.2019
Kumar, J. A., & Al-Samarraie, H. (2018). MOOC in the Malaysian higher education
institutions: The instructors' perspectives. The Reference Librarian, 1-16.
https://doi.org/10.1080/02763877.2018.1458688
Kuntze, R., Wu, C., Ross, W., & Whuang, Y. (2019). Improving financial literacy in college
of business students: Modernizing delivery tools. Emerald Insight, 37(4), 976-990.
https://doi.org/10.1108/IJBM-03-2018-0080
Lusardi, A., Samek, A., Kapteyn, A., Glinert, L., Hung, A., Heinberg, A. (2015). Visual tools
and narratives: New ways to improve financial literacy. Journal of Pension
Economics and Finance, 16(3), 297-323.
https://doi.org/10.1017/S1474747215000323
Malaysian Communications and Multimedia Commission. (2015). Hand phone user
survey 2014. Kuala Lumpur. https://www.mcmc.gov.my/ms/home.
Mayer, R. E., Lee, H., & Peebles, A. (2014). Multimedia Learning in a Second Language: A
Cognitive Load Perspective. Applied Cognitive Psychology, 28(5), 653–
660. https://doi.org/10.1002/acp.3050
Miles, M. B., & Huberman, A. M (2019). Qualitative Data Analysis 4th Edition. USA: Sage
Publications.
Mohd Aziz, N. I., & Kassim, S. (2020). Does financial literacy really matter for Malaysians?
A review. Advanced International Journal of Banking, Accounting and Finance, 2(2),13-
20. http://dx.doi.org/10.35631/AIJBAF.22002
Murugiah, L. (2016). The level of understanding and strategies to enhance financial
literacy among Malaysians. International Journal of Economics and Financial, 6(S3),
130-139. https://www.econjournals.com/index.php/ijefi/article/view/2621
Nurul Afiqah. (2016). The relationship between the level of financial literacy and
investment by decision-making millennials in Malaysia. Taylor's Business Review,
6, 39-47. http://dx.doi.org/10.18488/journal.137.2020.42.110.126
Phan, T., McNeil, S. G., & Robin, B. R. (2016). Students' patterns of engagement and course
performance in a massive open online course. Computers & Education, 95, 36-44.
https://doi.org/10.1016/j.compedu.2015.11.015
34
Popovich, J. J., Loibl, C., Zirkle, C., & Whittington, M. S. (2020). Community-college
students' response to a financial literacy intervention: An exploratory study.
International Review of Economics Education, 34 (2020), 1-15.
http://dx.doi.org/10.1016/j.iree.2020.100182
Prajapati, R., Sharma, F. B., & Sharma, F. D. (2017). Significance of life skills education.
Contemporary Issues in Education Research, 10(1), 1-6.
https://eric.ed.gov/?id=EJ1126842
Pulukuri, S., & Abrams, B. (2020). Incorporating an online interactive video platform to
optimize active learning and improve student accountability through educational
videos. Journal of Chemical Education, 97, 4505-4514.
https://pubs.acs.org/doi/10.1021/acs.jchemed.0c00855
Rakow, K. C. (2019). Incorporating financial literacy into the accounting curriculum.
Accounting Education, 28(4), 384-400.
https://doi.org/10.1080/09639284.2019.1578247
Sawatzki, C., & Sullivan, P. (2017). Teachers' Perceptions of financial literacy and the
implications for professional learning. Australian Journal of Teacher Education,
42(5), 51-65. http://dx.doi.org/10.14221/ajte.2017v42n5.4
Sekaran, U., & Bougie, R. (2010). Research methods for business: A skill building approach
(5th ed). New York, NY: John Wiley & Sons Ltd. Sidek, S. F., Mohamad Yatim, M.
H., Ariffin, S. A., & Nurzid, A. (2019). The acceptance factors and effectiveness of
MOOC in the blended learning of computer architecture and organization course.
Universal Journal of Education Research, 8(3), 909-915.
https://doi.org/10.30534/ijatcse/2019/3281.32019
Shah, A., Suhailiezana, Che Kob, C. G., & Khairudin, M. (2019). Effectiveness of M
Learning applications for design and technology subject. International Journal of
Interactive Mobile Technologies, 13(10), 120-133.
http://dx.doi.org/10.3991/ijim.v13i10.11324
Torres, W. J., & Beier, M. E. (2018). Adult development in the wild: The determinants of
autonomous learning in a Massive Open Online Course. Learning and Individual
Differences, 65, 207–217. https://doi.org/10.1016/j.lindif.2018.06.003
Qaffas, A. A., Kaabi, K., Shadiev, R., & Essalui, F. (2020). Towards an optimal
personalization strategy in MOOC. Smart Learning Environment, 7(14), 1-18.
https://doi.org/10.1186/s40561-020-0117-y
OECD. (2013). Education at a glance 2013: OECD indicators. OECD.
http://www.oecd.org/education/eag2013%20(eng)FINAL%2020%20June%2020
13.Pdf
OECD. (2014). Education at a glance. http://www.oecd.org/daf/fin/financial-
education/financial-education-for-youth.htm
Quiliano Terreros, A., Del Carmen, R., Jesús, P., & Durán, B. (2009). Systematic mapping
study 2012-2017: Quality and effectiveness measurement in MOOC. Turkish
Online Journal of Distance Education, 20 (1), 223-
247.http://hdl.handle.net/11285/632793
Wang, Z. (2021). Financial management teaching mode based on MOOC under the
background of informatization. In: Sugumaran V., Xu Z., Zhou H. (Eds.)
Application of Intelligent Systems in Multi-modal Information Analytics. MMIA
2020. Advances in Intelligent Systems and Computing, 1233. Retrieved from
https://link.springer.com/book/10.1007/978-3-030-15740-1
Wright, M. C., Bergom, I., & Bartholomew, T. (2017). Decreased class size, increased active
learning? Intended and enacted teaching strategies in smaller classes. Active
Learning in Higher Education, 1-17. https://doi.org/10.1177%2F1469787417735607
35
Yeen-Ju, T., H, Neo, M., & Salvaretnam, B. (2015). Enhancing problem-solving skills in an
authentic blended learning environment: a Malaysian context. International Journal
of Information and Education Technology, 5 (11), 841-846.
http://dx.doi.org/10.7763/IJIET.2015.V5.623
Zhu, A. Y. F. (2018). Financial literacy among Hong Kong's Chinese adolescents: testing
the validity of a scale and evaluating two conceptual models. Youth & Society, 1–
26. https://doi.org/10.1177%2F0044118X17753813
Zhu, A. Y. F. (2019). School financial education and parental financial socialization:
Findings from a sample of Hong Kong adolescents. Children and Youth Service
Review, 107, 1-10. http://doi.org/10.1016/j.childyouth.2019.104532
36

Vol. 20, No. 8, pp. 36-49, August 2021
Received Mar 29, 2021; Revised Jul 30, 2021; Accepted Aug 30, 2021
The Relationships between Experience, Qualification

and Subject Specialization and Content Knowledge
Mastery of Economic and Management Sciences
Teachers: A Case of Accounting Teaching
Habasisa Molise*
University of Limpopo, Polokwane, South Africa
Department of Education Studies, School of Education
https://orcid.org/0000-0002-3971-0718
Abstract. This paper investigates the relationship between teaching

experience, qualification, and subject specialization on the content
mastery of teachers of financial literacy to Grade 9 learners. The sample
consisted of 89 Grade 9 teachers of financial literacy in the Thabo
Mofutsanyana education district of South Africa. Data was collected
using a questionnaire that elicited biographical data and required
teachers of financial literacy to answer questions on a variety of
accounting topics, to diagnose the effect of teaching experience,
qualification, and subject specialization on the content knowledge
mastery of the teachers. The data revealed three distinct themes, namely,
solving the accounting equation, understanding accounting concepts,
and the connection between topics and application of accounting
principles. The items of the questionnaire were clustered into three
constructs: accounting equation, accounting concepts, and accounting
principles and subsidiary journals. The results show that having
experience of teaching the subject, having relevant qualifications, and
having a minor or major subject specialization in financial literacy
enhances teachers’ content knowledge and competence. This result
suggests that teachers of financial literacy should have a minimum of
three years of teaching experience before they teach senior classes,
possess a relevant diploma or degree, and have had accounting as a
subject until at least the second to the third year of their studies. In
general, teachers should not be expected to teach subjects they did not
specialize in, and their level of experience should be a determining factor
in workload/subject allocation for teachers, especially for accounting
(also known as financial literacy).
Keywords: accounting teaching; content knowledge; teaching experience;

subject specialization; financial literacy
* Corresponding author: Habasisa Molise; Email: habasisa.molise@ul.ac.za
©Author
37
1. Introduction
The paper explores the correlation between teaching experience, qualifications
and subject specialization and the content mastery of teachers of financial literacy
(FL) to Grade 9 learners. FL is a component of the subject economic and
management sciences (EMS), commonly known as accounting in South Africa
(Molise, 2020). Generally, teaching experience, qualifications and subject
specialization are viewed as important variables, which contribute to the content
mastery of teachers (Antony et al, 2019; Bamidele & Adekola, 2017). When
teachers are selected for employment, these variables may serve to determine the
suitability of the teachers for teaching certain subjects (Bamidele & Adekola,
2017). Much research has been done on the effect of teaching experience,
qualifications and subject specialization on learners’ academic achievement
(Antony et al., 2019; Donkoh, 2017; Ladd & Sorensen, 2017), yet we do not know
whether teaching experience, qualification, and subject specialization are related
to accounting content mastery by teachers (Emmanue & Ambe, 2014; Musau &
Abere, 2015). Subject specialization level refers to the highest qualification
obtained in the subject by a teacher (Myrberg et al., 2018). I argue that, in order to
teach accounting, teachers should have studied accounting for at least two or three
years of a diploma or degree qualification.
The South Africa, the minimum qualification for a teacher is a Bachelor’s degree.
However, most teachers at rural schools lack relevant qualifications to teach EMS,
and they teach EMS because of their teaching experience in subjects such as
economics and business studies (Ngwenya, 2020). Thus, EMS teachers struggle to
teach FL, because schools do not consider teachers’ qualifications and subject
specializations when subject allocations are done (Modise & Letlhoenyo, 2020).
Studies have investigated the effect of teachers’ experience of teaching, and their
qualifications, to determine if these characteristics contribute to learner
achievement (Nixon et al., 2016; Podolsky et al., 2019; Papay & Kraft, 2015), and
have found that the number of years spent teaching a particular subject, and
qualifications relating to this subject, do not necessarily translate into academic
achievement of learners (Nixon et al., 2016; Samad & Nurusus, 2015). The studies
found that several factors, in addition to experience and qualifications, could
contribute significantly to academic achievement of learners, and the teaching
strategies and methods teachers use (Podolsky et al., 2019; Hatlevik, 2017;
Bamidele & Adekola, 2017).
Studies found that teachers teach subjects in which they did not major, and that
they lack the requisite experience and qualifications to teach those subjects
(Modise & Letlhoenyo, 2020). This situation compromizes learner achievement,
also in the subject of FL. Ngwenya (2019) appreciates the importance of teachers’
content knowledge mastery of accounting, which ensures that content is correctly
transferred to learners. Accordingly, Samad and Nurusus (2015) argue that
learners develop negative attitudes towards learning accounting, because it is
presented in an abstract manner. Donkoh (2017) acknowledges that teachers are
passionate about teaching, and suggests that teachers work tirelessly to make sure
that EMS learners are taught, even though teachers may have poor content
mastery of the subject. The findings reported above motivated this researcher to
38
explore whether teachers’ experience of teaching accounting, their teaching

qualifications and whether they had specialized in accounting contribute to their
content knowledge mastery.
This article focuses on the relationship that teaching experience, qualification, and
subject specialization have with content mastery of teachers of FL to Grade 9
learners, as expressed by knowledge of the accounting equation, accounting
concepts, and accounting principles.
2. Literature review
In this section, literature is presented to determine whether teaching experience,
qualification, and subject specialization in accounting contribute to content
knowledge mastery of teachers of EMS to Grade 9 learners. Shulman (1986)
distinguishes between different kinds of knowledge that an effective teacher
should possess. These include knowledge about the subject matter they are
teaching (content knowledge), knowledge of specific strategies for teaching a
particular subject matter (pedagogical content knowledge, PCK), and knowledge
of the materials and media by which instruction and assessment are carried out
(curricular knowledge). Content knowledge involves knowledge of the substance
of the field; specialization concepts, principles, and procedures, and the
relationships between these elements. Thus, “teachers must have an in-depth
knowledge of the specific accounting topics that they teach as well as the
accounting that their learners will learn in the future as teachers’ subject
knowledge impacts their behavior and thus indirectly affects learner
achievement” (Molise, 2020, p. 460).
Several studies identified factors that can be associated with problems relating to
content knowledge mastery of accounting teachers (Melo et al., 2020; Modise,
2016; Peter et al., 2017). These factors are classified as misconceptions, difficulties
associated with defining accounting concepts, inability to manipulate accounting
equations, and difficulty in applying accounting principles and procedures
(Modise & Letlhoenyo, 2020). These studies confirm that misconceptions are
generally undesirable and must be identified and addressed (Melo et al., 2020;
Peter et al., 2017). It is assumed that the reason for insisting on identifying and
addressing misconceptions is to enhance the process of teaching and learning
through effective content mastery by accounting teachers (Molise, 2020). I, thus,
argue that, to improve learners’ academic achievement, teachers should be able to
present content in an appropriate way, and should ensure that the content is
simplified by using context-based examples that learners can relate to when they
learn about the accounting equation, concepts and principles.
Antony et al. (2019, p. 1) sought to assess secondary school teachers’

characteristics in relation to qualifications and experience and found that teacher
qualifications and teaching experience have a significant effect on biology
teachers’ TPACK. Findings of research into teacher qualifications (that is, the type
of degree) are inconclusive. Some studies found a correlation between learner
achievement and qualifications (Croninger et al., 2007; Brante, 2013), while others
found negative effects (Musau & Abere, 2015). Thus, “some argue that the
39
requirement of a second degree raises the cost in terms of teacher education and
the time it involves and may prevent quality candidates from choosing this
profession” (Nixon et al., 2016, p. 1636). Teacher qualification refers to skills and
knowledge an individual brings to improve teaching and learning conditions
(Samad & Nurusus, 2015), and it is suggested that a teacher has the ability to
transfer knowledge to learners based on institutional recognition (Emmanue &
Ambe, 2014). In this study, teaching qualification relates to formal qualifications
possessed by teachers, at any tertiary level.
Several studies confirm the importance of subject specialization (Nixon et al.,

2015; Emmanue & Ambe, 2014). These studies report that teachers have limited
accounting content knowledge because they are generalists rather than specialists
(Nixon et al., 2015), and they choose to spend their time and effort teaching other
subjects. Secondary school teachers have to complete a specialization in their
specialty subject before teaching it (Modise & Letlhoenyo, 2020); the area of study
is considered to be the teacher’s area of expertise. In the senior phase, teachers are
not required to have specialized in a subject, and it is doubtful whether teachers
can show the same level of competence in all subjects (Modise, 2016). When they
teach accounting, teachers need to focus on helping learners visualize accounting
concepts and procedures (Qhosola, 2015) by using pictures and diagrams, rather
than focusing only on doing calculations correctly (Molise, 2020). Several research
studies report a positive relationship between teachers' preparation in the subject
matter they later teach, and student achievement (Podolsky et al., 2016, p. 88),
while others have less unequivocal results (Modise & Letlhoenyo, 2020). Darling-
Hammond et al. (2017) argue that there is a correlation between teacher
preparation and student achievement, and it can sometimes be negative. Thus,
Myrberg et al. (2018, p 12) “find a positive relationship in mathematics, but none
in science”.
Also, Nixon et al. (2016, p 38) “report a positive relationship between student
achievement and teachers’ majoring in mathematics”. Musau and Abere (2015, p.
84), however, found that having majored in mathematics does not affect learner
performance, and could have a significant negative effect on teachers with more
coursework in physical science. I argue that, when teachers have a limited
conceptual understanding of accounting content knowledge, it influences their
teaching effectiveness and ability to refine their mathematics instruction to match
students’ achievement. Therefore, teachers must teach the content in the simplest
manner, to enable learners to understand it.
The next section will focus on the PCK model, which was adapted to interrogate
the effect that teaching experience, qualification, and subject specialization have
on the content knowledge mastery of accounting of teachers teaching FL to Grade
9 learners.
3. Conceptual framework
The PCK model is used in this paper to investigate the content mastery of teachers
of FL, which the proponent of this model, Shulman, refers to as the “missing
paradigm” in research on teaching and teacher knowledge (1986). PCK
40
emphasizes representations and conceptions/misconceptions (McKlin et al.,

2019), as used in this paper, “broadened ideas about how knowledge might matter
to teaching” (Kuhn et al., 2016, p. 9), which shows that “it is not just knowledge
of content, on the one hand, and knowledge of pedagogy” (Ngwenya, 2019, p. 22).
There is a need for a relationship between content and pedagogy to exist (Hung-
Hsi, 2017).
Content knowledge comprises “knowledge of the subject and its organizational

mechanisms” (Qhosola, 2015). Modise and Letlhoenyo (2020) believe that
comprehending a teaching subject involves more than knowing its facts and
concepts. According to Ngwenya (2020), educators must have full knowledge of
institutional expectations of what is right and wrong to teach (Rahmi, 2018;
Qhosola, 2015, p. 216). Moreover, the instructor is required to grasp why a specific
topic is essential to a field, whilst another topic may be peripheral (Molise, 2020).
This framing assisted me to explore how teaching experience, qualifications, and

subject specialization contribute to the content mastery of EMS teachers in relation
to the accounting equation, accounting concepts, and principles, as entrenched
throughout the EMS curriculum and South Africa’s Curriculum and Assessment
Policy Statement (CAPS). The framework assisted me to understand how teachers'
orientations to content influenced the way they taught that content.
4. Method
This descriptive study employed a quantitative research approach to investigate
the effect teaching experience, qualifications, and subject specialization have on
the content knowledge mastery of FL teachers. A descriptive research study,
according to Hlalele and Mosia (2020, p. 112), depicts “the particular
characteristics of a scenario, a social context, a connection, or a depiction of a
phenomena as it occurs naturally”. The approach used in this study helped “to
answer questions about the relationships among the measured variables such as
teaching experience, qualifications and subject specialization with the purpose of
explaining, predicting and controlling phenomena” (Bamidele & Adekola, 2017,
p. 3). It also helped me to determine the relationship between independent and
dependent variables, with the aim of confirming a hypothesis about a
phenomenon. The research was carried out in the Free State province's Thabo-
Mofutsanyane education district. All Grade 9 teachers were requested to
participate by completing a content-based FL questionnaire. I informed the
participants of their rights, such as that they could disengage from the study
without consequences, and that data generated through this research would be
used for research purposes only (Touron et al., 2018). Therefore, the purpose of
this study was to investigate the relationship between teaching experience,
qualifications and subject specialization and the content mastery of FL teachers.
The following sections will provide more details about the participants.
4.1 Profile of participants

The population comprised Grade 9 EMS teachers (n=97) at rural intermediate
schools in Thabo Mofutsanyane education district in the Free State province,
South Africa; the study had 89 participants who were randomly selected. Data
41
was generated by administering a questionnaire to EMS teachers. The

questionnaire comprised two sections; the first section had 25 accounting
questions, and the second elicited information on the biographical characteristics
of teachers.
Table 1: Participants’ demographic information

Demographic variables Category n %
Experience Less than 3 years 64 72%
3 to 5 years 15 17%
10 to 15 years 7 9%
15 years and above 3 3%
Qualification(s) Degree/Diploma 58 65%
Degree plus PGCE* 29 33%
Honors degree 2 2%
Accounting level No accounting 25 28%
First Year 29 32%
Second Year 21 24%
Third Year 14 16%
*Postgraduate Certificate in Education
4.2 Instrumentation
A closed-ended questionnaire was administered to collect data on the
relationships teaching experience, qualifications and subject specialization have
on the content mastery of teachers who teach FL to Grade 9 learners. The
questionnaire teased the problem areas of accounting topics, and was easy to
complete, yet effective in collecting data.
After revision, the 23-item content-based questionnaire, with its four

subscales/domains, covered the following subscales/domains: accounting
equation; accounting concepts; accounting principles and FL (questions 9, 10, 12,
16, 18). The questionnaire elicited biographical information (gender, age, teaching
experience, qualifications, and subject specialization), to contextualize the
teachers’ answers and content challenges.
4.3 Data analysis

I used Statistica for data analysis. The mode of data analysis (Gaertner & Brunner,
2018) underpinned the goal of the study, which was determining the relationships
between teaching experience, qualification and subject specialization and the
content knowledge mastery of teachers of FL to Grade 9 learners. Furthermore,
the independent and dependent variables were also determined to either confirm
or dispute the hypothesis relating to the phenomenon (Chang, et al., 2020;
Gudmundsdottir & Hatlevik, 2018; Mosia, 2014).
4.4 Validity and reliability

The tools used for data collection by this study were developed by a scholar at the
University of the Free State, a senior accounting subject advisor who has 25 years
of experience teaching and researching in the field of education. Officials of the
Free State Education Department and the Educational, Training and Practices
42
Development Sector Education and Training Authority evaluated and

commented on the questionnaire. These government authorities were asked to
verify whether the instrument did actually relate to the material necessary for FL
education in Grade 9. The Cronbach’s Alpha coefficient was used to assess the
survey's reliability (Hlalele & Mosia, 2020). A reliability value of 0.870 or above is
deemed acceptable, and a reliability coefficient greater than 0.90 indicates that
some questions are redundant, as they are evaluating the same question, but in a
different form.
5. Results
This section presents data on the relationships between the variables of teaching
experience, qualifications, and subject specialization level and the content mastery
of teachers teaching FL to Grade 9 learners, concerning the constructs of
accounting equation, accounting concepts, and accounting principles.
5.1 Solving of the accounting equation

Figure 1 depicts the proportion of teachers who gave incorrect answers in
response to questions on solving accounting equations – a theme in Grade 9 FL –
according to participants’ teaching experience, qualifications, and subject
specialization. The teaching experience of the teachers ranged from two years to
more than 15 years, while their qualifications ranged from junior degrees with
and without a teaching qualification, to honors degrees. The university level to
which they had studied accounting ranged from zero to third-year level, with few
of the teachers having studied accounting up to second-year level.
Figure 1. Incorrect answers on questions relating to accounting equation
Data in Figure 1 shows that, firstly, teachers with little teaching experience (no
more than 2 years), and who did not have accounting as their subject
specialization, had the highest number of incorrect answers - between 9 and 12.
This can be compared to teachers who had done accounting to first-year level,
who had between 3 and 6 incorrect answers. In contrast, teachers with the same
43
teaching experience (up to 2 years), a degree plus PGCE as qualification, but who
had studied accounting up to their third year, had the lowest proportion of
incorrect answers – at most 3. Teachers with up to 2 years of teaching experience,
with a degree but no accounting specialization, had a higher proportion of
incorrect answers than teachers with the same experience and years of accounting
specialization, who had degrees plus PGCE.
Secondly, teachers with teaching experience of maximum 3 years, with a degree

plus PGCE, but without accounting as subject specialization, had 6 to 9 incorrect
answers, while teachers with the same teaching experience and qualifications,
who had specialized in accounting up to third year, gave the smallest proportion
of incorrect answers – maximum 3. Teachers with the same teaching experience,
with an honors degree and accounting at first-year level, had a lower proportion
of incorrect answers than similarly experienced and qualified teachers who had
not specialized in accounting at first-year level.
Thirdly, teachers with teaching experience of up to 5 years, with an honors degree

and accounting at first-year level, gave the fewest incorrect answers – no more
than 3. Teachers with the same qualification and first-year accounting, and
between 11 and 15 years’ experience, gave no more than 3 incorrect answers.
Similarly, teachers with more than 15 years of experience, with an honors degree,
but no specialization in accounting, gave few incorrect answers.
5.2 Comprehension of accounting concepts and relationships between topics

Figure 2 depicts the proportion of teachers who gave incorrect answers to
questions in the questionnaire on comprehension of accounting concepts and
relationships between topics and themes of Grade 9 FL, in relation to teaching
experience, qualifications and subject specialization.
Figure 2. Incorrect answers on questions relating to understanding of accounting concepts

Data in Figure 2 shows that, firstly, teachers who up to two years teaching
experience and degrees plus PGCE, and who had specialized in accounting up to
44
third-year level, had between 6 and 9 wrong answers. Secondly, teachers with up
to three years teaching experience and degrees, but who had not had accounting
as their subject specialization, had the highest proportion of incorrect answers,
namely between 9 and 12.
Thirdly, teachers with up to five years teaching experience and a degree plus
PGCE, but who had not done accounting at first-year-level, had the highest
proportion of incorrect answers (between 9 and 12). In turn, teachers with the
same teaching experience and qualification, but with accounting at third-year
level, had a lower proportion of incorrect answers – maximum 3. In contrast,
teachers with the same teaching experience but with accounting at third-year
level, and a degree, had a higher proportion of incorrect answers than those with
the same experience and specialization, but with a degree plus PGCE. Fourthly,
teachers with teaching experience of between 11 and 15 years, holding a degree
plus PGCE, who had not specialized in accounting, had a higher proportion of
incorrect responses than teachers with the same teaching experience and
accounting subject specialization level, yet with a teachers’ diploma.
Lastly, teachers with teaching experience of more than 15 years, holding a teacher
diploma and having done third-year-level accounting subject specialization, had
the lowest proportion of incorrect answers – no more than 3 – compared to
teachers with the same teaching experience and qualification, but without
accounting as their subject specialization.
5.3 Application of accounting principles

Figure 3 depicts the proportion of teachers who chose incorrect answers in the
questionnaire on the theme application of accounting principles of Grade 9 FL
teaching, according to teaching knowledge, qualifications, and subject
specialization.
Figure 3. Incorrect answers on questions relating to accounting principles
45
Data in Figure 3 shows that, firstly, teachers who have at most two years of
teaching experience, and who hold a degree plus PGCE, and who had not
specialized in accounting, had the highest proportion of incorrect answers.
Secondly, teachers with teaching experience of up to 3 years, who have a degree

and had third-year level accounting as their subject specialization, had a lower
proportion of incorrect answers, at most 3, compared to teachers with the same
teaching experience and qualification, who had a high number of incorrect
answers – between 6 and 9. Similarly, teachers with the same teaching experience,
no more than 3 years, with a degree plus PGCE, but who did not specialize in
accounting at all, had the highest number of incorrect answers, that is, between 9
and 12.
Thirdly, teachers who have up to five years teaching experience, who hold a
degree, and who specialized in accounting up to third-year level, had the lowest
proportion of incorrect answers, that is, at most 3. Other teachers, with the same
teaching experience and qualification, yet do not have accounting subject
specialization, had a lower number of incorrect answers, that is, between 3 and 6.
Similarly, teachers with the same teaching experience, and holding a degree plus
PGCE, with no accounting specialization, had a low proportion of incorrect
answers, that is, between 3 and 6.
Fourthly, teachers who had teaching experience of between 11 to 15 years, who

have a degree plus PGCE, but who had not specialized in accounting, had a lower
proportion of incorrect answers than teachers with only a degree, but the same
experience and first-year accounting.
Lastly, teachers with teaching experience of more than 15 years, who hold a
diploma and who had specialized in accounting at third-year level, had the lowest
proportion of incorrect answers – no more than 3. Similarly, teachers with the
same teaching experience and qualification, yet do not have accounting subject
specialization, had a lower number of incorrect answers, that is, no more than 3.
In contrast, teachers with the same years of teaching experience and who had no
accounting subject specialization, and a degree plus PGCE, had a higher
proportion of incorrect answers than teachers with a diploma.
6. Discussion of results
This section discusses results concerning solving the accounting equation,
comprehension of accounting concepts, and relations between topics and
application of accounting principles. The reviewed literature confirms that
teachers must have a thorough comprehension of the specific accounting topics
they teach (Molise, 2020). However, most of the teachers who teach EMS are
confronted with serious accounting content misconceptions, which threaten
effective teaching of FL topics to Grade 9 learners (Melo et al., 2020; Modise, 2016;
Peter et al., 2017). Modise and Letlhoenyo (2020) confirm that misconceptions are
generally undesirable and must, therefore, be identified and addressed. The
majority of the teachers in the study struggled to analyze transactions in the
46
accounting equation, lacked comprehension of accounting concepts and how they

link with other concepts, and they were unable to apply accounting principles.
Firstly, the results show that teaching experience, qualifications, and subject
specialization play pivotal roles in the content mastery of EMS teachers in relation
to solving the accounting equation. It was evident from the analysis that teachers
with a degree plus PGCE, and who had specialized in accounting at third-year
level, had a lower number of incorrect answers – at most 3 – and the number
decreased further as they gained more teaching experience. Similarly, the number
of wrong answers of teachers with a degree plus honors, with accounting at first-
year level, decreased from 3 to 0 as they gained more experience, first from 3 to 5
years and then from 11 to 15 years. Even teachers with no accounting
specialization, with a degree plus honors, are likely to experience a decrease in
the proportion of incorrect answers when they reach 15 years of teaching
experience. Therefore, more teaching experience is acquired by accounting
teachers who have a relevant degree and who had specialized in accounting for
three years – they had a lower proportion of incorrect answers.
Secondly, it is evident from the analysis that teachers with a degree plus PGCE,
with a third-year level accounting subject specialization, decreased the proportion
of incorrect answers (from 3 to 0), as they gained more teaching experience (from
3 to 5 years). Similarly, teachers with a diploma, with a third-year accounting
specialization, showed a considerable decrease in the proportion of incorrect
answers (from 3 to 0), as they reached 15 years of experience.
Thirdly, it is evident from the analysis that there was a considerable decrease in
the proportion of incorrect answers, from 3 to 0, for teachers holding a degree with
a third-year accounting subject specialization as they gained more teaching
experience (from 3 to 5 years). Similarly, teachers with a diploma, with no
accounting specialization, show a considerable decrease in the proportion of
incorrect answers, from 3 to 0, as they reach 15 years of experience. I recommend
appointing EMS teachers who have a degree or diploma that included the study
of accounting, economics, and business studies up to second to third-year level
for them to ensure content mastery, especially the accounting part (referred to as
FL in the CAPS document) of EMS.
The author argues that content misconceptions by teachers (as illustrated by

incorrect answers on the questionnaire) are not unique to accounting, and that
intervention strategies through content-specific workshops and cluster teaching
collaboration must be implemented to improve teachers’ content skills in all
subjects.
7. Limitations
The research was done in a single education district of the Free State province,
South Africa, and data was generated using a single tool, a closed-ended
questionnaire.
47
8. Conclusion
This paper aimed to explore the relationships between teaching experience,
qualification, and subject specialization on the content mastery of teachers of FL
(the accounting part of EMS) to Grade 9 learners. The results show that the
experience of the teacher in teaching the subject, relevant qualifications, and
having specialized in accounting to the second or third-year level enhances
teachers' content knowledge and competence. It is worth noting that teachers who
had relevant teaching qualifications with accounting as their subject of
specialization at either second or third-year level contributed less to the
proportion of incorrect answers regarding the accounting equation, accounting
concepts, and principles. However, most of the teachers who contributed to a
higher proportion of incorrect answers did not have a relevant accounting
teaching qualification and had not specialized in the subject during their training.
Similarly, teachers who had 3–15 years of teaching experience had fewer incorrect
answers because they have been teaching the subject for more than 3 years.
I, therefore, recommend that teachers strive to gain at least three years of teaching
experience before they teach senior classes, that they have a relevant diploma or
degree that involved studying accounting to at least the second or third year.
Teachers should refrain from teaching subjects in which they did not specialize;
the teacher’s level of experience should be a determining factor in doing
workload/subject allocation for EMS teachers, especially for accounting/FL.
Teachers must receive continuous professional development support, to help
address inherent content knowledge misconceptions they may have.
9. References
Antony, M. K., Paidy, H., Subali, B., Pradana, S. P., Hapsari, N., & Astuti, F. E. C. (2019).
Teacher’s TPACK profile: The effect of teacher qualification and teaching
experience. Journal of Physics: Conference Series, 6, 1-6.
https://doi.org/10.1088/1742-6596/1397/1/012054
Bamidele, A. D., & Adekola, F. F. (2017). Effects of teacher’s qualifications and teaching
experience on students’ academic achievement in basic science in junior
secondary school. International Journal of Education and Evaluation, 3(2), 1–9.
Brante, T. (2013). The professional landscape: the historical development of professions in
Sweden. Professions and Professionalism, 3(2), 1-18.
https://doi.org/10.7577/pp.558
Chang, S. H., Ward, P. & Goodway, J. D. (2020) The effect of a content knowledge teacher
professional workshop on enacted pedagogical content knowledge and student
learning in a throwing unit. Physical Education and Sport Pedagogy, 25(5), 493-508.
https://doi.org/10.1080/17408989.2020.1743252
Croninger, R. G., King Rice, J. K., Rathbun, A., & Nishio, M. (2007). Teacher qualifications
and early learning: effects of certification, degree, and experience on first-grade
student achievement. Economics of Education Review, 26(3), 312–324.
Darling-Hammond, L., Burns, D., Campbell, C., Goodwin, A. L., Hammerness, K., Low,
E. L. McIntyre, A., Sato, M., & Zechner, K. (2017). Empowered educators: How high-
performing systems shape teaching quality around the world. John Wiley & Sons.
Donkoh, S. (2017). Investigating the effect of teaching experience on teacher knowledge.
International Journal of Scientific and Research Publications, 7(6), 319–327.
48
Emmanue, I. A. & Ambe, B. A. (2014). Influence of teachers, professional qualification and

area of specialisation on the implementation of environmental education
curriculum in Cross River State – Nigeria. International Conference on Chemical,
Environment & Biological Sciences (CEBS) (155–160).
http://dx.doi.org/10.15242/IICBE.C914120
Gaertner, H., & Brunner, M. (2018). Once good teaching, always good teaching? The
differential stability of student perceptions of teaching quality. Educational
Assessment, Evaluation and Accountability, 30, 159-182.
https://doi.org/10.1007/s11092-018-9277-5
Gudmundsdottir, G. B. Hatlevik, O. E. (2018). Newly qualified teachers’ professional
digital competence: Implications for teacher education. European Journal of
Teacher Education, 41, 214-231. https://doi.org/10.1080/02619768.2017.1416085
Hatlevik, I. K. (2017). The impact of prospective teachers’ perceived competence on
subsequent perceptions as schoolteachers. Teachers and Teaching: Theory and
Practice, 23(7), 810–828. https:/doi.org/10.1080/13540602.2017.1322056.
Hlalele, D., & Mosia, M. (2020). Teachers’ sense of community in rural learning ecologies.
Alternation, 27(2), 101–124. https://doi.org/10.29086/2519-5476/2020/v27n2a6
Hung-Hsi, W. (2017). The content knowledge mathematics teachers need. In Y. Li, W. J.
Lewis, & J. Madden (Eds.), Mathematics Matters in Education. Essays in honour of
Roger E. Howe (43–91). Springer.
Kuhn, C., Alonzo, A.C., & Zlatkin‑Troitschanskaia, O. (2016). Evaluating the pedagogical
content knowledge of pre-and in‑service teachers of business and economics to
ensure the quality of classroom practice in vocational education and training.
Empirical Research in Vocational Education and Training, 8(5), 1-18.
https://doi.org/10.1186/s40461‑016‑0031‑2
Ladd, H. F., & Sorensen, L. C. (2017). Returns to teacher experience: Student achievement
and motivation in middle school. Education Finance and Policy, 12(2), 241–279.
http://direct.mit.edu/edfp/article-pdf/12/2/241/1691525/edfp
McKlin, T., Lee, T., Wanzer, D., Magerko, B., Edwards, D., Grossman, S., Bryans, E., &
Freeman, J. (2019). Accounting for pedagogical content knowledge in a theory of
change analysis. In ICER ’19: International Computing Education Research Conference
(12-14). Association for Computing Machinery.
https://doi.org/10.1145/3291279.3339412
Melo, L. Cañada-Cañada, F., González-Gómez, D., & Jeong, J. S. (2020). Exploring
pedagogical content knowledge (PCK) of physics teachers in a Colombian
secondary school. Education Science, 10(12), 362.
https://doi.org/10.3390/educsci10120362
Molise, H. (2020). Exploring the content knowledge of accounting teachers in rural
contexts: A call for a decoloniality approach. International Journal of Learning,
Teaching and Educational Research, 19(8), 447–458.
Modise, M. A. (2016). Pedagogical content knowledge challenges of accounting teachers.
International Journal of Educational Science, 13(3), 291–297.
Modise, M. A. & Letlhoenyo, S. (2020). Pedagogical content knowledge of accounting
teachers in South Africa: Readiness of Grade 9 for accounting in Grade 10.
Proceedings of ADVED 2020- 6th International Conference on Advances in Education,
(79–80).
Mosia, M. S. (2014). Gender differentials and the mathematics performance of Grade 10 to
12 learners at the Dilanedi Schools. Mediterranean Journal of Social Sciences, 5(23),
1426-1431. https://doi.org/10.5901/mjss.2014.v5n23p1426
49
Musau, L. M., & Abere, M. J. (2015). Teacher qualification and students’ academic
performance in science mathematics and technology subjects in Kenya.
International Journal of Educational Administration and Policy Studies, 7(3), 83–89.
Myrberg, E., Johansson, S., & Rosén, M. (2018). The relation between teacher specialisation
and student reading achievement. Scandinavian Journal of Educational Research,
63(5), 1–15. https://doi.org/10.1080/00313831.2018.1434826
Ngwenya, J. (2019). Accounting teachers’ experiences of communal feedback in rural
South Africa. South African Journal of Education, 32(2), S1–S10.
https://doi.org/10.15700/saje.v39ns2a1502
Ngwenya, J. C. (2020). Assets teachers identify for the teaching of Accounting Education
in a rural school in KwaZulu-Natal. Perspectives in Education, 38(1), 72–87.
http://dx.doi.org/10.18820/2519593X/pie.v38i1.6
Nixon, R. S., Campbell, B. K., & Luft, J. A. (2016). Effects of subject-area degree and
classroom experience on new chemistry teachers’ subject matter knowledge.
International Journal of Science Education, 2016(38), 1636–1654.
https://doi.org/10.1080/09500693.2016.1204482
Papay, J. P., & Kraft, M. A. (2015). Productivity returns to experience in the teacher labor
market: Methodological challenges and new evidence on long-term career
improvement. Journal of Public Economics, 130, 105–119.
https://doi.org/10.1016/j.jpubeco.2015.02.008
Peter, I., Ward, P., & Li, W. (2017). Effects of improved content knowledge on pedagogical
content knowledge and student performance in physical education. Physical
Education and Sport Pedagogy, 22(1), 71–88.
Podolsky, A., Kini, T., Bishop, J. and Darling-Hammond, L. (2016), Solving the teacher
shortage: How to attract and retain excellent educators. Learning Policy Institute.
https://learningpolicyinstitute.org/sites/default/files/product-
files/Solving_Teacher_Shortage_Attract_Retain_Educators_REPORT.pdf
Podolsky, A., Kini, T., & Darling-Hammond, L. (2019). Does teaching experience increase
teacher effectiveness? A review of US research. Journal of Professional Capital and
Community, 4, 286–308. https://doi.org/ 10.1108/JPCC-12-2018-0032
Qhosola, M. R. (2015). Accounting teacher preparation: A critical accounting perspective.
Journal of Higher Education Association/Research in Education Southern Africa,
13(1&2), 213–222.
Rahmi, E. (2018). The analysis of pedagogical content knowledge of teacher candidates.
Advances in Economics, Business and Management Research, 57, 604–609.
Samad, A., & Nurusus, E. (2015). The influence of teaching experience, school location and
academic background on teachers’ beliefs in teaching grammar. Asian Social
Science, 11(15), 257–266.
Shulman, L. S. (1986). Those who understand: Knowledge growth in teaching. Educational
Researcher, 15(2), 4–14.
Touron, J., Martin, D. R., Navarro Asencio, E., Pradas, S., & Inigo, V. (2018). Construct
validation of a questionnaire to measure teachers’ digital competence (TDC).
Spanish Journal of Pedagogy, 76, 25–54. https://doi.org/10.22550/REP76-1-2018-02
50

Vol. 20, No. 8, pp. 50-67, August 2021
Effect of the Use of WebQuest in a Chinese for

Tourism Course at a Thai University:
An Exploratory Study
Pairin Srisinthon
Walailak University, Nakhon Si Thammarat, Thailand
https://orcid.org/0000-0003-3693-3163
Abstract. In trying to move away from teacher-centred approaches towards

student-centred approaches amidst the global Covid-19 pandemic,
integrating technology into foreign-language classes has been considered.
Under these conditions, this study implemented WebQuest technology in a
Chinese for Tourism class, followed by adopting a mixed-methods research
approach to investigate the student learning. This study aimed to 1) assess
the learning achievement of 63 students as part of a Chinese for Tourism
course using WebQuest to provide the instructional materials alongside the
face-to-face format, and 2) to collect the learners’ opinion on the WebQuest-
based activities. SPSS was used for the quantitative analysis and thematic
analysis was used to analyse the interview data. The development of the
WebQuest lessons began with a critical and analytical review of the existing
literature related to the WebQuest course design. The WebQuest lessons were
then implemented for six weeks after which the students’ learning
achievements were evaluated. The findings revealed that after using
WebQuest, the learners’ Chinese language ability test scores were higher than
before utilizing the instrument. The students were satisfied with the
WebQuest lessons with varying satisfaction levels for the questionnaire
items. The benefits of teaching and learning in connection to the WebQuest
lessons got the highest score. From the 15 learners’ semi-structured
interviews, it was found that the majority of the learners agreed that teaching
and learning through the WebQuest lessons was a modern method offering
convenient access due to the lack of time limitations.
Keywords: tourism WebQuest; Chinese majoring students; Chinese teaching;

Chinese language; mixed methods
©Author
51
1. Introduction
According to the statistics of the Thai Tourism Department, approximately 8.8 million
Chinese travelled to Thailand in 2016. This number is higher than the number of
ASEAN (Association of Southeast Asian Nations) tourists travelling to Thailand. The
need to use Chinese in the tourism industry is high. Based on the reports from the
Centre of Cooperative Education and Career Development of Walailak University
(2016-2018), most students majoring in Chinese chose to do their work-integrated
learning in companies related to the tourism industry such as hotels, airports, and
tour companies. Many of Thailand’s universities offer Chinese for Tourism courses
to support the growth of the tourism industry.
The Chinese major curricula try to meet the requirements of the tourist industry by
offering students Chinese for Tourism courses to prepare them for real-world careers.
Chinese for Tourism is a Chinese for Specific Purposes course dealing with specific
Chinese vocabulary and expressions in the field in the areas of business, travel, and
an introduction to tourist attractions (Traveling Chinese Glossary Program Group,
Foreign Chinese College, Shanghai Normal University, 2008). This course assists the
students in improving their Chinese language skills when it comes to travel and
tourism. The Chinese for Specific Purposes course involves the teaching and learning
of Chinese as a second or foreign language of which the aim is the ability to use
Chinese in their career activities. Teaching and learning Chinese for specific purposes
requires a methodology that differs entirely or partially from the one used when teaching for
the purpose of language skills (Traveling Chinese Glossary Program Group, Foreign
Chinese College, Shanghai Normal University, 2008). The instructors of such a course
face the challenges of structure, technical vocabulary and field-specific knowledge
(Zheng, 2018). They also encounter the challenge of a paradigm shift from the teacher-
centred approach to the student-centred approach (Wang, 2021; Zheng, 2018). It is
important to determine the teaching processes that suit the changing world and the
newer generations of students (Chen, 2018). The designers of this course must
integrate the expected Chinese language skills with the intended content knowledge
to support the learners’ needs (Baker, 2018). In order to move away from the teacher-
centred approach to concentrate on the student-centred approach, most educators
suggest that teachers integrate technology into the Chinese language class. Since
technology plays an important role in society today, how to take advantage of
technology as part of supporting Chinese teaching is an issue worth studying.
Under these conditions, WebQuest technology is an effective tool with which to

engage the learners throughout the process of learning (Kaur and Kauts, 2018). In the
context of teaching and learning Chinese in Thailand, a few detailed studies were
conducted on the utilization of WebQuest in the Chinese classroom. Meanwhile, due
to the many years of experience in teaching the Chinese language, the researchers
found that the students were rarely actively involved when it came to finding
educational information on the Internet. The level of knowledge seeking from the
Chinese Internet network is very low. The management of the Chinese for Tourism
course was made more effective by connecting to both Thai and Chinese networks,
52
allowing the learners to practise their language skills and learn more effectively
through interactions with their teachers and peers. This study attempted to find a
way to enhance the level of student learning. Exploring how WebQuest can be best
used will benefit many groups of people, be they course designers, course lecturers,
or students.
2. Literature Review and Theoretical Frameworks

Given the technological advancements in recent years, conducting research on the
application of technology in teaching that had been done over the past ten years, and
integrating online technology into the learning process were found to have a positive
influence on the students’ learning engagement (Dunlap & Lowenthal, 2018; Bognar,
Sablic & Skugor, 2019). In order to improve the teaching methods and classroom
environment, many educational institutions use new technology (e.g., tech-based
learning) as part of their language classrooms (Berezova, Mudra & Yakushko, 2018;
Dousti, Amirian & Nejadansari, 2021). Technology has the potential to offer quite a
number of free learning materials online for learning activities (Teng, 2017; Gao, Shen
& Wang, 2020). Given the advancements in technology and the ways in which the
new generation of students learn (Martin & Bolliger, 2018), education institutions
need to provide access to web-based teaching and stimulate the learners to utilize
Internet resources as part of the learning process. Learning through authentic
materials from the Internet offers an active classroom environment as part of a
learner-centred activity and a level of communication among the instructors, learners,
and their peers. The differences between tech-based learning and traditional learning
are compared in Table 1 (see Wasim, Sharma & Siddiqui, 2014, p. 448).
Table 1: Comparison of Tech-based Learning and Traditional Learning

Tech-based learning Traditional Learning
- Student-centred instruction - Teacher-centred instruction
- Multi-sensory stimulation - Single-sense stimulation
- Multi-path progression - Single-path progression
- Multimedia - Single media
- Collaborative work - Isolated work
- Information exchange - Information delivery
- Active/exploratory/inquiry-based learning - Passive learning
- Critical thinking and informed decision- - Factual, knowledge-based learning
making - Isolated, artificial context
- Authentic, real-world context
Based on Table 1, WebQuest is more ideal for teaching and learning as it offers what
the students should experience, for example, student-centred instruction,
collaborative work, inquiry-based learning, active learning and critical thinking
(Ebadi & Rahimi, 2018), which is a more suitable approach to learning for the current
world of working.
53
2.1 What is WebQuest?

Web-based lessons (WebQuest) are designed based on the tech-based learning model
created by Professor Bernie Dodge and Tom March of San Diego State University as
a learning tool to gain information from the Internet or long-distance networks. It
offers access to online resources under the supervision of the instructor (Sakadineca
& Jansone, 2018). The learners utilize appropriate and relevant content which the
instructors provide to complete their learning tasks. Dodge (2001:?:) describes
WebQuest as “an inquiry-oriented activity in which some or all of the information
that learners interact with comes from resources on the Internet”. March (2003) also
notes that WebQuest is an effective instrument with links to web resources and tasks
that encourage the students to find the answers to the open-ended questions to
develop their personal expertise and group collaboration skills in order to transfer
new information, leading to deeper learning.
According to Kazakova and Klyoster (2018) and Kaur and Kauts (2018) two forms of
WebQuest exist, namely (i) short-term WebQuest, which provides one to three
lessons, focused on the learning and application of specific skills, and (ii) long-term
WebQuest which takes as long as three weeks to approximately one month to master.
The learners will analyse the new knowledge in a classroom setting with their peers
and will learn to apply it to real-life situations.
Based on the tech-based learning model, WebQuest includes five parts, namely an
introduction, a task, process, evaluation and conclusion (Millsion & Downey, 2001;
Sakadineca & Jansone, 2018; Zendler & Klein, 2018; Berezova et al., 2018). These
warrant a brief description:
a. Introduction: Briefly explains the information and activities, and tries to stimulate
the students to solve the problems.
b. Task: Clearly explains the tasks and important results that the students must find.
c. Process: Clarifies what activities the students must perform in order to achieve
their tasks, and provides information sources available on the Internet and other
resources, in order for the students to use appropriate sources to solve their
assigned problems, focusing on multiple sources of knowledge and diversity.
d. Evaluation: Focuses on the criteria for assessing the learning tasks.
e. Conclusion: Summarizes the main objectives and experiences of the students
during the learning process.
Several studies have been conducted using the WebQuest approach (Kazakova &
Kloyster, 2018; Ebadi & Rahimi, 2018; Sakadineca & Jansone, 2018; Chen, 2019;
Adanan, Adanan & Herawan, 2020). These studies have shown that the WebQuest
process supports the learners’ level of learning engagement and the level of
enjoyment of their studies. WebQuest also enhances the learners’ critical thinking
skills by tasking them with analysing information and completing tasks. For example,
Sakadineca and Jansone (2018) designed WebQuest tasks for the purpose of teaching
the Latvian language and literature. The results showed that all the students in the
54
class positively worked in groups and were active when it came to using WebQuest
to complete their tasks. Liang and Fung (2020) implemented WebQuest in an English
classroom to stimulate the students’ critical thinking skills. Their findings revealed
that after implementing the WebQuest-based critical thinking programme, the
students’ thinking skills improved, especially in terms of their vocabulary when
expressing their opinions. Similarly, Wang (2021) used WebQuest for the purpose of
supporting the students to actively participate in the information technology (IT)
classroom. They found that the students engaged with their learning more and that
their problem-solving abilities increased.
Researchers claim that the WebQuest technology, as an interactive method, can be

successfully implemented in the field of teaching foreign languages. They believe that
after the implementation of WebQuest and other learning methods such as
collaborative projects, problem-based classrooms or flipped classrooms in the field of
language acquisition, the learners’ language competency performance will be
significantly improved (Awada & Dihab, 2018; Berezova et al., 2018; Adanan et al.,
2020; Synekop, 2020). Awada and Dihab (2018) investigated student’s academic
writing performance after integrating Student Team Achievement Division (STAD)
and WebQuest into their learning process. The results of the study showed that these
web-based interventions improved the students’ academic writing performance.
Furthermore, all the teacher participants developed more positive perceptions of
web-based instruction. Teng (2017) introduced the concept of a flipped classroom, by
dividing three EFL (English Foreign Language) classes into a structured flipped
classroom, a semi-structured flipped classroom, and a traditional classroom. It was
found that using WebQuest in the flipped classroom was a most effective way to
increase the learners’ academic performance in the pre-post test score comparison.
Zheng (2018) supported using WebQuest as a teaching and learning tool in the
Chinese classroom. He conducted a comparative study of two classes. The
experimental group was a trial group that used online lessons, while the control
group used lecture-based teaching. The instructor designed the course content
according to Hanban's “International Chinese Language Course Layout” and
provided the video clips about the Chinese content and culture from the Internet. The
web lessons were also designed to provide sufficient interactivity between the
learners and the lecturer through an online group chat. At the end of the semester,
the learners took an HSK test (Chinese Proficiency Test for assessing non-native
Chinese speakers’ Chinese skills). It was found that the HSK scores of the learners
who followed lectures through the web were significantly higher than those of the
learners in the control group. The results gathered from the questionnaires indicated
that most of the learners from the web-learning group were satisfied with the self-
controlled learning process. They also strongly agreed to watching video clips in
small groups. The study proved that WebQuest encourages learners to learn, analyse,
synthesize and provide solutions that have been discovered by themselves or in small
groups. In addition, the group that learned Chinese online were confident when
learning with Chinese students. On the contrary, the students who studied in the
control group were not confident when studying with Chinese students in their own
55
majors. This is because they did not understand the lectures given by the Chinese
instructors. A similar study by Chen (2018) showed that patients with hearing
problems who studied Chinese pronunciation through online classes got higher
scores after having used web lessons. They progressed in their listening performance
and were able to discuss interactions with a medical professional about their hearing
problems. The web lessons made it easier for the learners to communicate effectively.
The integration of WebQuest in classrooms provides learners with the ability to

gather information from various websites. This can help the learners to understand
both the linguistic and cultural aspects, and to improve their intercultural
competence (Awada and Diab, 2018; Kazakova & Klyoster, 2018; Ebadi & Rahimi,
2018). Awada and Diab (2018) divided EFL learners from eight different countries
into experimental and control groups. They provided the experimental group with
the Inquiry-based Intercultural Communication Technological Model. The control
group received regular instructions. The findings show that the participants in the
experimental group had a strong intrinsic motivation towards conducting culturally
based debates in a setting that ensured intercultural communication.
Learning through WebQuest allows learners to learn without time limitations. The
students have to be self–directed, active and autonomous when it comes to searching
for information before referring to the instructor who acts as the facilitator. This
solves the problem of over-dependence on the instructors to deliver knowledge and
to identify problems (Synekop, 2020). It has been proven that the students improved
their self-esteem and self-confidence by taking part in a more participative course.
The reviewed studies show that most of the studies discussed, were conducted on the
use of WebQuest in EFL classrooms. Further studies are recommended to investigate
the use and outcomes of WebQuest technology in other subjects. More studies,
therefore, will be required to determine the effects of the WebQuest approach on
Chinese teaching. This will help us to understand the advantages of this technology
in a holistic manner.
3. Study Objectives
This study aimed to investigate the effectiveness of WebQuest technology when
implemented in a Chinese for Tourism class. The research questions were as follows:
1. What were the learners’ achievements after the Chinese for Tourism course
through WebQuest?
2. What were the learners’ opinions about the WebQuest lessons?
4. Research Methodology
4.1 Research design

The mixed-methods design was used to explore the learners’ achievements and their
opinion of learning via the WebQuest lessons. The independent variable was the
implemented WebQuest lessons which included the Chinese for Tourism course
56
content. The dependent variables included the learners’ achievements and their
opinion of the WebQuest instruction. The first dependent variable was measured by
comparing the pre-test and post-test scores. The second dependent variable was
measured using the post-instruction questionnaires and informal group interviews.
The quantitative data from the questionnaires and the pre-post-test scores were
analysed using SPSS and an analysis of variance (ANOVA) respectively. The
qualitative data from the informal group interviews were analysed using thematic
analysis.
4.2 Research site and participants

This research was conducted at a university in southern Thailand. The university
emphasizes teaching; all of the teaching staff had passed the UK-based Advanced HE
Training Programme (the active teaching and learning programme used for training
university instructors provided by the Higher Education Academy). On this basis, its
lecturers emphasize their teaching, paying close attention to the learning of
individual students, adopting formative assessments and feedback, and focusing on
analytical and critical thinking. Moreover, the university encourages its instructors to
teach using active-based methods and to employ empirical research. The participants
in this study were 63 undergraduate Thai students majoring in Chinese who were
enrolled in the Chinese for Tourism course offered in the first term of the 2019
academic year. This course was selected because its objective was the use of Chinese
in their professional activities or areas of specialization. The students attended five
50-minute class periods per week in a smart classroom where each student was
allowed to use his/her personal computer or mobile phone. Fifteen interviewees were
selected using the volunteering system.
4.3 Research instruments

The instruments used in this study included WebQuest for the Chinese for Tourism
Class, the pre-test and post-test, and the post-instruction questionnaire. The learners’
Chinese proficiency and the learning objectives were considered before WebQuest
was created specifically for the course. The course was offered to third-year Chinese
majors who had studied approximately 1,000 words for everyday use. The WebQuest
lessons consisted of six steps: 1) Introduction; 2) Task; 3) Process; 4) Resources; 5)
Evaluation, and 6) Conclusion (Dodge, 2001). The WebQuest lessons were divided
into two main parts, one for the administrator or instructor and one for the learners.
The WebQuest lessons covered communication contexts that the learners could
genuinely apply elsewhere. The contents consisted of Ticket Booking, Welcome to
Thailand, Check-in Procedures, Room Reservation, and Checking in and Checking out
at a hotel. WebQuest on Chinese for Tourism was provided to the participants for the
purpose of learning in the classroom and outside of it. It consisted of Chinese content
on tourism, online exercises, links to useful internal and external websites, and
communication suggestions via e-mail and Facebook.
The performance test consisted of the pre-test and post-test developed from the
course content. This was used to evaluate the learners’ proficiency regarding their
Chinese for Tourism knowledge and their Chinese language proficiency before and
57
after studying the course using the WebQuest lessons. Similarly, the pre-test and
post-test consisted of ten multiple-choice questions, twenty gap-filling items, ten
true-false questions and five sentences to write using the words provided. The total
score for the performance test was 50 points.
The post-instruction questionnaire was developed by the researcher after conducting

a literature review on the effectiveness of WebQuest in supporting the learning
process. It was used to measure the learners’ attitude towards the use of web
resources for learning as part of the Chinese for Tourism Course and it was
administered at the end of said course. The questionnaire was divided into three main
aspects: the WebQuest Design, the WebQuest Lesson Content, and the Usefulness of
WebQuest. It used a Likert scale for the items ranging from 1 (not at all satisfied) to 5
(extremely satisfied). Three invited Item Objective Congruence IOC experts validated
the questionnaire. The score obtained was over 0.67.
Subsequent to the questionnaire response session, fifteen learners from the class
attended the semi-structured interviews on a voluntary basis. Each interviewee had
approximately ten minutes to respond to a set of open-ended questions. Notes were
taken by the interviewer and a research assistant, in order to record the key ideas as
part of the data analysis. All proper interview protocols were adhered to at all times,
including confidentiality.
4.4 Data collection and data analysis

The quantitative data were collected using a questionnaire and the qualitative data
were gathered using semi-structured interviews. The Statistical Package for the Social
Sciences software ( SPSS) was used for the data analysis. The data collected were
analysed using the mean, standard deviation and t- test. Analysis of variance
(ANOVA) was conducted to compare the pre-test and post-test score differences with
a significance level of 0. 05. The questionnaire items were interpreted using the
included Likert scale ranges based on Best’s criteria (Best, 1986).
1.00 - 1.80 = Not at all satisfied

1.81 - 2.60 = Slightly satisfied
2.61 - 3.40 = Moderately satisfied
3.41 - 4.20 = Very Satisfied
4.21 - 5.00 = Extremely satisfied
Thematic analysis was used to analyse the data from the interviews (see Braun &
Clarke, 2019; Terry & Hayfield, 2021). The interview data were identified and
coded. Five major themes emerged from the data.
4.5 Research procedure

The research procedure is depicted in the following table (Table 2):
58
Table 2: Research procedure

Item Research Procedure Purpose Timeline
1 Designing the WebQuest -To prepare the learning Before
lessons, performance tests materials and research conducting the
and opinion questionnaire instruments research
2 Evaluating the students’ -To measure the learners’ Before the
Chinese proficiency using background knowledge of experiment
the pre-test Chinese for Tourism.
3 Implementing WebQuest -To introduce how to use the 6 weeks
for Chinese for Tourism WebQuest lessons, learning
tasks, and objective goals.
-To provide the specific
Chinese skills needed in the
tourism industry, focusing on
the specific vocabularies
around accommodation, travel
methods, sightseeing and
holidays.
4 Evaluating the student -To evaluate their learning 7th week
learning using the post-test achievements
5 Collection of the data -To measure the students’ 8th week
through a questionnaire opinions of/on the WebQuest
lessons
6 Collection of the data -To gain more information 8 - 9th week
through interviews about the effectiveness of the
WebQuest lessons
4.6 Ethical considerations

Before beginning the experiment, the details of the purpose and methodology of the
research were explained to the participants for them to feel respected and
appreciated after they had left the project. All the participants were requested to
return a copy of their completed consent form before conducting the research. They
could withdraw from the study at any time. The researcher treated the information
gathered from the individuals confidentially without disclosing the participants’
identity. In addition, all the data gathered from the participants were only available
to the researcher and were kept in a secure location.
5. Results
5.1 Learners’ academic achievements
The first research question aimed to determine the learners’ level of
achievement after studying the Chinese for Tourism course through
WebQuest. Table 3 depicts a comparison of the students’ pre-test and post-
test scores.
59
Table 3: Comparison of the learners’ pre-test and post-test scores

Mean N Std. Deviation Std. Error Mean Sig.
Pair 1 Pre-test 7.30 63 4.01 0.52 .000
Post-test 21.30 63 6.22 0.81
The statistics for academic performance in Table 3 indicate that the

learners’ achievements were positive with higher scores after studying the
Chinese for Tourism course through the WebQuest lessons at a significance
level of 0.000 (Sig. < 0.005).
5.2 Learners’ opinions
The second research question was answered by the questionnaire and the
interview data, the findings of which are shown in the following table.
Table 4: Learners’ opinions about the WebQuest lessons

Item Aspect Mean S.D. Level of Satisfaction
1 Diverse formats of information 3.00 0.53 Moderately satisfied
and content presentation
2 Interesting presentation 2.95 0.66 Moderately satisfied
3 Creative design 3.47 2.57 Very satisfied
4 The font used 3.26 0.72 Moderately satisfied
5 Web colours 3.38 0.72 Moderately satisfied
6 Linked tabs 3.01 0.65 Moderately satisfied
7 Easy to communicate between the 3.03 0.73 Moderately satisfied
learners and the instructors
8 Easy access to other resources 3.00 0.71 Moderately satisfied
9 Appropriate tasks and evaluations 3.14 0.64 Moderately satisfied
10 Online feedback of the learners 2.84 0.65 Moderately satisfied
The overall average value 3.11 0.65 Moderately satisfied
Table 4 shows that the overall average value of the learners’ satisfaction
with the WebQuest design was 3.11. When considering the individual
aspects, the highest mean score was for Item 3 (mean=3.47). This means that
the learners seemed to be quite satisfied with the design of WebQuest even
though they had never used web-based lessons before. Because of the
technical problems, the aspect of online feedback got the lowest mean score.
60
Table 5: Learners’ Opinions about the WebQuest Lesson Content

Item Aspect Mean S.D. Level of
Satisfaction
1 Web structure 3.92 0.58 Very satisfied
2 Suitable content according to the 3.85 0.65 Very satisfied
learning objective
3 Modern application of the 4.20 0.71 Very satisfied
content
4 Genuine application of the 4.25 0.71 Extremely satisfied
content
5 Language and content accuracy 4.06 0.64 Very satisfied
and reliability
6 Enough resources to use 3.73 0.75 Very satisfied
The overall average value 4.00 0.68 Very satisfied
According to Table 5, it can be concluded that the learners were highly satisfied with
the content of the WebQuest lessons. They were satisfied with the authentic
application of the content as indicated by giving it the highest mean score (mean =
4.25). The reason behind this might simply be that the learners are much more
concerned about utilizing their Chinese for Tourism knowledge in real life. The
learners also focused on the modern content that was beneficial for them to learn. The
learners were highly satisfied with the accuracy and reliability of the language and
content of the web lessons.
Table 6: Learners’ Opinions of the usefulness of WebQuest

Item Aspect Mean S.D. Level of Satisfaction
1 Provide self-study-based learning 4.25 0.83 Extremely satisfied

2 Enable the learners to learn anytime and 4.57 0.82 Extremely satisfied
anywhere
3 Motivate the learners to study 4.04 0.67 Very satisfied
4 Support the learners to solve problems better 4.23 0.64 Extremely satisfied
5 Develop the relationship among the 4.15 0.55 Very satisfied
instructor, learners, and their peers
6 Develop their learning performance 4.45 0.54 Extremely satisfied
7 Learners can control their learning process 4.04 0.71 Very satisfied
8 Support critical thinking 4.34 0.53 Extremely satisfied
9 Gain knowledge that meets the learning 4.50 0.61 Extremely satisfied
objective
10 Appropriate tasks that meet the learning 4.54 0.65 Extremely satisfied
objective
11 Discipline in learning 3.73 0.88 Very satisfied
The overall average value 4.25 0.68 Extremely satisfied
61
As shown in Table 6, the learners were satisfied with the usefulness of the Chinese
for Tourism WebQuest at a high level (mean = 4.25). The aspect of learning anytime
and anywhere got the highest average score of 4. 57. This indicates that the students
were able to study without the limitations of time and space. The tasks provided by
the Chinese for Tourism WebQuest were suitable for the learning objectives, thus the
satisfaction of the learners was found to be at the highest level (mean = 4.54).
5.3 Opinions on the WebQuest lessons from the interview data

Based on the thematic analysis (Braun & Clarke, 2019), the opinions of the students
on the WebQuest lessons collected from fifteen interviewees were coded into the
following categories:
- Promoting autonomous learning
- Supporting modern and up-to-date learning content
- Practising technological skills
- Enjoyed working in cooperative groups
- Internet connectivity problems
Table 7: Learners’ responses in each category
Item Category Total (person)

1 Internet connectivity problems 15
2 Supporting modern and up-to-date learning content 15
3 Practising technological skills 13
4 Enjoyed working in cooperative groups 13
5 Promoting autonomous learning 12
Based on Table 7, all the interviewees found the Internet connectivity problem to be
a big obstacle. However, the good points were that WebQuest supports modern and
up-to-date learning content, that it gave them the opportunity to practise their
technological skills, that they enjoyed working in cooperative groups and that it
promoted autonomous learning.
Promoting autonomous learning

Twelve learners responded that integrating the WebQuest approach into the Chinese
for Tourism Class was a new and modern learning method that enhanced their
Chinese language skills development. They were able to learn anytime and anywhere
after class. One interviewee said, “I spent a lot of time preparing for this course apart
from classroom presentations. The assignments provided in the web helped me to
better understand knowledge of Chinese for Tourism”. Another interviewee added,
“I preferred to depend on my own ability to search knowledge from Internet rather
than following the teacher’s guidance. My Chinese reading and translating skills also
improved while working with Chinese posted articles”. Only three learners said that
they preferred not to participate in autonomous learning, and that they liked learning
from the instructor’s lectures.
62
Supporting modern and up-to-date learning content

Studying through WebQuest encouraged the students to interact with new
information from the web resources. As one commented, “The WebQuest
contents are modern and useful, stimulate me to engage with learning”. All
interviewees commented that the information provided on WebQuest
widened their knowledge of the course and that they could apply the
information to real-life situations and in the workplace. Most of the
interviewees reported that they saved time by getting the learning materials
from the WebQuest lessons.
Practising technological skills

The students had no experience with the WebQuest instruction method
prior to the course. Through WebQuest, they obtained technological skills
and knowledge about the use of Internet-enhanced learning. Most of the
interviewees noted that submitting the tasks online was easy and
comfortable. Approximately 60% of the interviewees reported that the
WebQuest Chinese for Tourism course had detailed and clear steps to help
them carry out the learning tasks, presenting the tasks well and providing
the related materials. They were able to practise more technological skills
when completing their learning tasks.
Enjoy working in cooperative groups

Many of the learning tasks on WebQuest required being accomplished by
cooperative groups. According to thirteen learners, they enjoyed working
in cooperative groups to complete the assigned tasks. For example, one
responded, “I was happy to work with my peers because we will help each
other to complete the assigned tasks”. Another interviewee responded, “I
like to brainstorm with my classmates, this helped us to do the tasks
better”. Only two interviewees in the class reported that some of their peers
were not responsible enough to do the tasks, meaning that they did not
engage with their assigned cooperative groups.
Internet connectivity problems

Fifteen learners gave feedback on the technical problems including the weak wi-fi
connection and the webpage not being stable. This problem interrupted the process
of learning. Some of the students were worried about submitting the tasks late and
getting no scores. Providing more explanations for selected vocabularies’ meanings
as part of the WebQuest lessons should be considered. Some of the students asked
the instructor to explain the tasks provided on the webpage again in class.
6. Discussion and Implications

A number of studies focused on using WebQuest in English classrooms to develop
the students’ academic performance (Adanan et al., 2020; Baker, 2018; Kazakova &
Kloyster, 2018), but of these only a few studies (e.g., Zheng, 2018) have been
conducted on implementing WebQuest in Chinese classrooms. This study attempted
63
to find a way to improve the teaching and learning of Chinese using the principles of
long-term WebQuest use (Dodge, 2001) to design web lessons for the Chinese for
Tourism Course. The participants studied the web lessons for six weeks. The findings
from this research indicate that learning through the Chinese for Tourism WebQuest
resulted in higher scores in the learner’s post-test compared to the pre-test scores at a
significant level. This result agrees with the study by Zheng (2018) who revealed that
the HSK (Chinese Proficiency Test) scores of the students taught through using web
lessons were significantly higher. Overall, the investigation reflected the learners’
positive satisfaction with the Chinese for Tourism WebQuest lessons at a high level.
They enjoyed working with their peers to complete the learning tasks. These findings
are compatible with the findings on using WebQuest to enhance English writing and
reading performance, conducted by Berezova et al. (2018). They found that the
students were actively engaged with the authentic and collaborative tasks provided
through the Internet.
The students believed that the knowledge that they gained from practising Chinese
for tourism through the web lessons could be applied in their daily life and in the
workplace. Learning through web lessons is a new and modern style of Chinese
learning. Through this mode of learning, the learners can improve their language
skills. Dudeney (2003) points out that WebQuest technology is an effective tool to
provide access to the Internet in a language classroom, and that it provides the
learners with an opportunity to study with no time limits. Practising various tasks
enabled the learners to understand the Chinese for Tourism content and to increase
their knowledge of the vocabulary and expressions suitable for use in many situations
when travelling. Similar results were reported by Awada and Diab (2018) and Liang
and Fung (2020), namely that after implementation of the WebQuest in the EFL
classrooms, the students’ language competency improved.
This learning method fosters positive motivation and critical thinking. The results of
this study confirm the inferences drawn from the previous studies that employing
technological devices in the Chinese learning process enhances the learners’
motivation and their enjoyment when studying (Seitkazy et al., 2016; Chen, 2019;
Zheng, 2018, Wang, 2021). By learning through WebQuest lessons, the learners
managed their own learning process with no time limitations (Mitsikopoulou, 2014;
Liu, Huang & Wosinski, 2017; Zeng & Fu, 2019).
Although integrating the selected technology into the Chinese learning context in
higher education contributed various advantages, problems did occur. The learners
faced difficulties when trying to get connected to the Internet as they were fully
dependent on the wi-fi connection provided by the university, both in the classrooms
and in the dorms. A large number of users were utilizing the same access point at the
same time, which was mostly at night. The Internet streaming became poor as a result,
and sometimes the online connection was lost. As a multimedia programme,
WebQuest lessons require a high storage capacity to run. This was the biggest
problem when it came to the learners participating actively in the Chinese for
64
Tourism WebQuest activities created by the instructor. Similar findings were also
made in previous studies by Seitkazy et al. (2016), and Yarmakeev, Valiakhmetova,
Akhmadullina and Terane (2019), who reported that the main problem with learning
online was the weak Internet connection and poor condition of the computer
equipment. This included that the mobile phone system was not effective enough to
run the Chinese for Tourism WebQuest programme, thus preventing the learners
from interacting. In this study, the instructors could not contact their students through
WebQuest and they changed their method of contact to using Line or e-mail for the
submission of assignments instead.
7. Conclusion
This research revealed that integrating the WebQuest lessons and technology into the
Chinese learning process enhanced the development of Chinese regarding their
tourism knowledge and Chinese language skills. The difference between the pre-test
(7.30) and post-test scores (21.30) corroborates the learner’s academic achievements.
As for the questionnaire and interview results, the learners were positive regarding
the use of WebQuest lessons to help them with the process of learning Chinese for
tourism. According to the questionnaire responses, the learners were satisfied with
the usefulness of WebQuest at a high level (mean = 4.25). The instructors should
motivate and encourage the learners to utilize the Internet effectively by providing
useful online resources related to the learning process. According to the interview
data, the learners reported in their feedback on technical problems such as a weak wi-
fi connection or WebQuest not being stable. The university should improve its
Internet connection in order to encourage students to engage with the online learning
environment. To solve the wi-fi connection problem, a WebQuest design allowing it
to be used on both laptops and mobile phones should be considered.
8. Acknowledgments
This paper is part of the research study titled “The Innovation of Learning Chinese
for Tourism by Using WebQuest”, funded by the Innovation and Research
Department of Walailak University, Thailand. This research and the informed
consent were reviewed and approved by the Human Research Ethics Committee of
Walailak University, based on the Declaration of Helsinki (Approval No. WUEC-19-
029-01).
9. Recommendations
It is recommended that more in-depth investigative studies and more research design
types, such as experimental and control group research designs, be used to
investigate technology-supported education on a wider scale. The scope should also
be broadened concerning the examination of the students’ achievements after
learning through WebQuest. WebQuest should be implemented in other Chinese
courses such as Business Chinese, Chinese Literature and Chinese Translation to
gather more essential and useful information to fill the gaps in the literature for
Chinese foreign language teaching that will benefit other Chinese language teachers
and students.
65
10. References
Adanan, H., Adanan, M., & Herawan, T. (2020). M-WebQuest development: Reading
comprehension of senior high school students in Indonesia. International Journal of
Emerging Technologies in Learning (IJET), 15(3), 74-92.
https://doi.org/10.3991/ijet.v15i03.10628
Awada, G., & Diab, H. (2018). Effect of WebQuest media on learners’ intercultural
communication debate and motivation. 19th International CALL Conference (2018).
Call Paper.
Baker, D. M. (2018). USA and Asia hospitality and tourism students' perceptions and
satisfaction with online learning versus traditional face-to-face instruction. E-Journal of
Business Education and Scholarship of Teaching, 12(2), 40-54.
https://doi.org/10.33915/etd.2960
Berezova, L. М., Mudra, S. М., & Yakushko, K. H. (2018). The effect of WebQuests on the writing
and reading performance of university students. Information Technologies and Learning
Tools, (64,2), 110-118. https://doi.org/10.33407/itlt.v64i2.1979
Best, J. W. (1986). Research in Education 1986. Prentice-Hall.
Bognar, B., Sablić, M., & Škugor, A. (2019). Flipped learning and online discussion in higher
education teaching. Didactics of smart pedagogy (pp. 371-392). Springer, Cham.
https://doi.org/10.1007/978- 3-030-01551-0_19
Braun, V., & Clarke, V. (2019). Reflecting on reflexive thematic analysis. Qualitative Research in
Sport, Exercise and Health, 11(4), 589-597.
https://doi.org/10.1080/2159676x.2019.1628806
Chen, J. (2019). Designing online project-based learning instruction for EFL learners: A
WebQuest approach. MEXTESOL Journal, 43(2), 1-7.
Chen, X. (2018). Hearing rehabilitation training system based on Web. Beijing Biomedical
Engineering. 37(1), 86-90.
Dodge, B. (1995). WebQuests: A technique for Internet-based learning. Distance Educator, 1(2),
10-13.
Dodge, B . (2001). Site overview:The WebQuests Page. http://webquest.sdsu.edu/overview.htm.
Dousti, M., Amirian, Z., & Nejadansari, D. (2021). Application of WebQuest-based Instruction
in Higher Education Context: EFL Students’ Achievement in Writing Skill. Journal of
English Language Teaching and Learning, 13(27), 113-136.
https://doi.org/10.22034/elt.2021.42866.2312
Dudeney, G. (2003). The quest for practical web usage. Global Issues Special Interest Group.
IATEFL ISSUES, 8-9.
Dunlap, J., & Lowenthal, P. (2018). Online educators’ recommendations for teaching online:
Crowd sourcing in action. Open Praxis, 10(1), 79-89.
https://doi.org/10.5944/openpraxis.10.1.721
Ebadi, S., & Rahimi, M. (2018). An exploration into the impact of WebQuest-based classroom
on EFL learners’ critical thinking and academic writing skills: A mixed-methods study.
Computer-Assisted Language Learning, 31(5), 617-
651. https://doi.org/10.1080/09588221.2018.1449757
66
Gao, H., Shen, H. D., & Wang, Y. L. (2020). Diversified teaching mode reform of mechanical
principle and design based on Internet plus. Life-Long Education, 9(5), 225.
http://dx.doi.org/ 10.18282 /le.v9i5.1257
Hanban. (2004). Hanban's International Chinese Language Course Layout. Hanban Explained.
https//everything.explained.today/Office_of_Chinese_Language_International/
Kaur, S., & Kauts, A. (2018). Impact of WebQuest on student engagement. Indian Journal of
Public Health Research & Development, 9(12), 1665-1669. https://doi.org/10.5958/0976-
5506.2018.02230.1
Kazakova, O. P., & Kloyster, A. M. (2018). Educational tasks modelled on the basis of the Web-
Quest technology (from the Experience of Teaching Foreign Languages). Astra
Salvensis.
Liang, W., & Fung, D. (2020). Development and evaluation of a WebQuest-based teaching
programme: Students’ use of exploratory talk to exercise critical
thinking. International Journal of Educational Research, 104, 101652.
https://doi.org/10.1016/j.ijer.2020.101652
Liu, D., Huang, R.., & Wosinski, M. (2017). Smart learning in smart cities. Singapore: Springer
Nature.
March, T. (2003). What WebQuests Are (Really). https://tommarch.com/writings/what-
WebQuests-are/
Martin, F., & Bolliger, D. U. (2018). Engagement matters: Student perceptions on the importance
of engagement strategies in the online learning environment. Online Learning, 22(1),
205-222. https://doi.org/10.24059/olj.v22i1.1092
Millson, A. J., & Downey, P. (2001). WebQuest: using Internet resources for cooperative inquiry.
https://rc.library.uta.edu/uta-
ir/bitstream/handle/10106/11730/WebQuest%20using%20Internet%2020resources
.pdf?sequence=1
Mitsikopoulou, B. (2014). Introduction to Webquests. www.opencourses.uoa.gr/modules/docu
Sakadineca, O., & Jansone, A. (2018). Students' attitudes toward learning activities organized
with the means of WebQuest method. Baltic Journal of Modern Computing, 6(3), 271278.
https://doi.org /10.22364/bjmc.2018.6.3.04
Seitkazy, P. B., Toleubekova, R. K., Amanova, A. K., Tashetov, A. A., Iskakova, G., &
Demissenova, S. S. (2016). A Web-Quest as a teaching and learning tool. International
Electronic Journal of Mathematics Education, 11(10), 3537-3549.
Teng, M. F. (2017). Flipping the classroom and tertiary level EFL students' academic
performance and satisfaction. Journal of Asia TEFL, 14(4), 605.
https://doi.org/10.18823/asiatefl.2017.14.4.2.605
Synekop, O. (2020). WebQuest as technology of differentiated ESP instruction at university
level. Journal of Teaching English for Specific and Academic Purposes,? 043-052.
https://doi.org/ 10.22190 /JTESAP2001043S
Terry, G., & Hayfield, N. (2021). Conceptual foundations of thematic analysis. Essentials of
thematic analysis , Number? Issue? pp. 3–14. American Psychological
Association. https://doi.org/10.1037/0000238-001
Travelling Chinese Glossary Programme Group, Foreign Chinese College, Shanghai Normal
University. (2008). Luyou Hanyu dagang [Outline of Traveling Chinese Function].
Shanghai Normal University.
67
Walailak University. (2016-2018). Reports from the Centre of Cooperative Education and
Career Development of Walailak University, Nakhon Si Thammarat Province,
Thailand.
Wang, Y. H. (2021). Integrating modified WebQuest activities for programming
learning. Journal of Computer Assisted Learning. https://doi.org/10.1111/jcal.12537
Wasim, J., Sharma, S. K., Khan, I. A., & Siddiqui, J. (2014). Web-based learning. International
Journal of Computer Science and Information Technologies, 5(1), 446-449.
Yarmakeev, I., Valiakhmetova N., Akhmadullina, R., & Terane, G. (2019). WebQuest
technologies as learning motivation. Helix. 9(5), 5307-
5310. https://doi.org/10.29042/2019-5307-5311
Zendler, A., & Klein, K. (2018). The effect of direct instruction and WebQuest on learning
outcomes in computer science education. Education and Information Technologies, 23(6),
2765-2782.
Zeng, Q., & Fu, J. (2019). Teaching mode of WebQuest in college English flip class under the
mobile learning environment. Proceedings of 2019 5th International Conference on
Education Technology, Management and Humanities Science (ETMHS 2019). 1177-
1181. http://doi.org/1025236/etmhs.2019.253
Zheng, Q. (2018). The teaching reform study based on Internet plus communicative task for
elementary comprehensive course of TCFL. Jinhua Polytechnic Journal, 18(3), 21-27.
68

Vol. 20, No. 8, pp. 68-90, August 2021
Multiple Means of Engagement Strategies for

Maximising the Learning of Mathematics in
Pandemic-regulated Classrooms
Matshidiso M. Moleko
University of the Free State, South Africa
https://orcid.org/0000-0002-1437-7218
Abstract. Many learners find mathematics learning challenging. In

response to that actuality, this paper highlights mathematics teachers’
experiences of, and insights into how they adopted and implemented the
principle of “Multiple Means of Engagement” (MME) to maximise
learning in pandemic-regulated classrooms (in the context of the study,
characterised by alternative weeks of attendance, social distancing and
wearing of masks). The MME principle is one of the three universal
design for learning (UDL) key principles, which guides on how diverse
groups of learners can be effectively catered for. The empirical processes,
premised on a phenomenological case study, commenced with focus
group discussions with 8 high school mathematics teachers from a
previously disadvantaged area, who have prior-training in MME. A free
attitude interview (FAI) technique was used, to afford the teachers the
opportunity to share their insights into the application of MME in their
pandemic-regulated classrooms. The content analysis of the teachers’
reflections revealed the following aspects: clear instructions, step-by-step
guides, checklists to enhance self-regulation, varying demand and
resources to meet challenges, fostering collaboration, providing
corrective feedback to sustain effort and persistence, addressing
mathematical vocabulary and using real-life situations to recruit interest.
These strategies were found not only essential in maximising learning in
mathematics under normal circumstances, but also indispensable during
the prevailing conditions of the pandemic. The findings therefore suggest
MME as a suitable mathematical approach during this Covid19 period.
Keywords: learner engagement; Mathematics teaching; maximising

learning; multiple means of engagement
1. Introduction
The purpose of the study was to highlight mathematics teachers’ experiences of
and insights into how they implemented the multiple means of engagement
(MME) principle to maximise learning in pandemic-regulated classrooms. MME
©Author
69
is one of the key principles of the universal design for learning (UDL) teaching
framework, which encompasses strategies to engage diverse groups of learners in
meaningful activities meant to motivate them, address anxiety, and maximise
learning (Dalton, 2017). Many learners find mathematics learning challenging and
become demotivated, anxious, and disengaged in the classroom. With respect to
this, Smith (2004) notes that most mathematics classrooms are occupied by
learners with negative thoughts and feelings towards mathematics. These learners
often develop anxiousness about mathematics that causes them to be less likely to
continue working on mathematical problems, especially when they fail to
understand these the first time (Smith, 2004). This makes the teaching of
mathematics even more challenging, since teachers have to ensure that they not
only teach mathematics effectively but at the same time also deal with issues
pertaining to a lack of motivation, learners’ negative thoughts and feelings
towards mathematics, and learners’ state of being disengaged. According to
Filgona et al. (2020), learners who are unmotivated tend to be disengaged, which
often leads to behaviour that makes the teaching of mathematics difficult.
However, Gray and Madson (2007) and Hake (1998) maintain that mathematics
teachers have a responsibility to ensure that learners are not disengaged in the
classroom by using interactive approaches that have proven to be useful in terms
of engaging learners and providing them with actual gain in learning.
Learner engagement is regarded as a way of addressing problems and emotional

dispositions such as demotivation, anxiousness and disengagement that learners
often experience in various school subjects and specifically in mathematics
(Padgett et al. 2019). In short, learner engagement is seen as a potential solution
for addressing these emotional dispositions in an endeavour to encourage
learners to accomplish learning tasks, persist in the midst of challenge, develop
academic satisfaction, and remain engaged with the given tasks (Halverson and
& Graham, 2019).
Williams and Williams (2011) view engaging learners in learning as an important

ingredient for success and a key to academic motivation and persistence. Kuh et
al. (2011) hold the act of engaging learners in the learning process in high esteem
and regard it as crucial in terms of promoting learner success. According to
Yengin et al. (2010), engaging learners in the learning process is a strategy that
puts learners at the centre of their learning and inspires them to participate and
take on an active role in the learning process.
Learner engagement is further regarded as an essential precursor for maximising

learning (Zyngier, 2007). Engaging learners in the learning process provides many
benefits, including increased “attention and focus, motivation to practise higher-
level critical thinking skills and promoting meaningful learning experiences”
(Park, 2003). Three domains of engagement have been identified through the
literature, namely “cognitive, emotional, and behavioural engagement”
(Fredricks et al. 2004). Behavioural engagement is often observed through actions
such as learners paying attention and cooperating and participating in class
activities (Hughes, 2012), while cognitive engagement is exhibited through self-
efficacy, motivation, focus, and psychological effort from an individual (Duchesne
70
& Ratelle, 2010). “Emotional engagement” is noticeable through a sense of

community and belonging, support in the classroom, identifying with school, and
one’s feelings towards school and learning (Fredricks et al. 2004). Research shows
that learners who are highly engaged often demonstrate active involvement in
learning and usually have high academic achievements (Appleton et al., 2008;
Fredricks et al., 2004; Leonard, 2008; Smith et al., 2010; Sirin & Rogers-Sirin, 2005).
Moreover, they demonstrate a positive attitude and behaviour towards their
schoolwork (Smith et al., 2010).
Although learner engagement positively influences learning, many teachers seem

to have difficulty in engaging learners, and this often results in low learner
participation in class activities, demotivation, and disengagement (Parsons et al.,
2014). Teachers’ challenges with regard to engaging learners were recently
exacerbated by the Covid-19 pandemic that forced schools to close (during the
lockdown) and then apply an alternating attendance system for learners to be able
to maintain social distancing inside the classroom. All of these changes had a
significant impact on the teaching of mathematics and raised the levels of
demotivation and learner disengagement in the classroom. The fact that teachers
had not been trained to teach under the prevailing conditions and lack knowledge
of inclusive teaching strategies (Bansilal & Ubah, 2019) makes it even more
challenging for mathematics teachers to teach and effectively engage learners
under these conditions. Furthermore, the nature of teaching mathematics that
necessitates learner engagement, which is currently impeded by the restrictions
of the pandemic, makes it difficult for teachers to teach and create a teaching and
learning space that is conducive for diverse learner population (Klein, 2000).
Balwant (2018) stresses that disengagement in the classroom is a serious challenge
that should be addressed in order to maximise learning. In line with Balwant’s
statement, Thoonen et al. (2011) suggest the need for improving teaching practices
and, more so, ways to engage learners in order to maximise learning. The fact that
learner engagement serves as a precursor for maximising learning (Zyngier, 2007)
raises, even more, the need for teachers to consider strategies to meaningfully
engage learners in order to maximise learning. Taking all of this into account, the
current study sought to highlight mathematics teachers’ experiences of and
insights into how they implemented the MME to maximise learning during the
Covid-19 period (in their pandemic-regulated classrooms).
MME is one of the UDL principles that provides guidelines for enhancing
motivation and learner engagement in order to maximise learning (Burgstahler,
2008). It addresses issues of learner variability (e.g. different learning styles and
preferences) and provides flexible choices for engagement in the learning process
(Dalton, 2017). Since learners have different learning preferences and needs
(Boothe et al., 2018), they also differ in the manner in which they have to be
engaged. MME thus requires mathematics teachers to use varied engagement
strategies applicable to all types of learners, regardless of their background to
ensure that learners do not undergo the psychological and emotional distress
(Pokhrel, & Chhetri, 2021).
71
1.1 Significance of the study

The MME principle serves as a guide for assisting teachers in terms of applying
strategies to engage diverse groups of learners in meaningful activities meant to
motivate them, address anxiety, and maximise their learning. Learners differ
distinctly in terms of how they should be engaged and motivated to learn. For
instance, some learners engage spontaneously and through innovation, while
others frequently disengage and thus prefer a “stringent routine”. Some learners
prefer to solve mathematical problems individually, while others prefer to
embark on collaborative problem solving activities. This means that there is no
single “means of engagement that will suit all learners in all contexts”;
consequently, there is a need for teachers to “provide multiple options for
engagement”. MME thus provides a comprehensive guide for engaging learners
by varying “options for self-regulation, options for sustaining effort and
persistence, and options for recruiting interest”. The study thus has at least two
theoretical inferences: firstly, the findings contribute to the advancement of
literature on learner engagement practices to be used in mathematics classrooms;
and secondly, the study serves as a guide for informing inclusive planning of
mathematics teaching.
1.2 Study Context

The Thabo Mofutsanyane district is in the central part of South Africa in the Free
State province. The district has low socio-economic status and is densely
populated, with about 647 schools. Poor performance, especially in mathematics,
is often reported in this district, and many teachers there are in need of training
in more effective methods of learning and teaching. Three years prior to the
outbreak of the Covid-19 pandemic, the researcher was a teaching and learning
coordinator at the local university and was involved in the in-service training of
mathematics teachers. She introduced them to the concepts of UDL and MME,
which captured their interest and led to the implementation of an action research
project with the purpose to empower them with teaching strategies aimed at
improving their teaching practices in order to maximise learning in mathematics.
The action research project ran for two years (2017-2018) through four phases,
namely planning, action, observation, and reflection. Through this, the
participants were empowered with knowledge and skills on how to implement
UDL principles in their mathematics classrooms by the end of the project in 2018.
When the pandemic started, the researcher became interested in how these
mathematics teachers would adopt the MME principle during one year of
pandemic regulations that had a large impact on the learning environment, such
as the lockdown period (no school attendance) and social distancing, which led to
a system of alternating school weeks for learners. The interest was sparked by the
fact that the school in which the research was conducted was of lower socio-
economic (deprived of the luxury of online facilities).
The school in which the study was conducted had approximately 1 600 learners
who attended school on a fortnightly basis (i.e., two weeks per month). The
teachers engaged in focus group discussions in which they shared their
experiences of and insights into how they used the MME principle to engage
72
learners in learning, thereby “providing options for self-regulation, for sustaining

effort and persistence, and for recruiting interest”.
The school in which the study was conducted, was in a developing environment
wherein poverty levels were high, with poor infrastructure and lack of computers
which meant that the online learning was not an option. Teachers therefore had
to be creative in terms of thinking of ways in which they can support learning
even when they were not physically present to teach learners. Their efforts were
exemplary to others in similar environment and with a possibility of adaptation
to online learning as well.
1.3 Overview of the article

The article commences with a brief overview of perspectives from the literature
on the learner engagement theory and MME as a UDL principle. The
methodology section provides detail on the design and execution of the study. In
the discussion of the findings, the experiences and insights of the participants are
shared to indicate how they adopted the MME principle during one year of Covid-
19 regulations, especially the lockdown (no school attendance) and social
distancing that led to a system of alternating school weeks, which had a significant
impact on the learning environment. All these views lead to a conclusion on the
use of MME in the learning of mathematics by diverse groups of learners in
challenging learning environments.
2. Literature review on Multiple Means of Engagement

The UDL is a teaching framework that comprises of three principles, namely
“multiple means of representation, multiple means of action and expression, and
MME” (Center for Applied Specialised Technology, 2011). It is an approach to
teaching that requires teachers to rethink their teaching strategies in order to
address the needs of all learners. It affords learners opportunities to demonstrate
what they know in several ways. The UDL refers to “an approach to teaching that
involves the proactive design and use of inclusive teaching strategies that benefit
a broad range of learners, including those with disabilities (Scott et al., 2002, p.1)”.
Although UDL comprises of the three principles as already mentioned, this article
focuses only on one of these principles, namely; MME. The MME principle is
expedited through the application of numerous and flexible methods of
engagement of students’ learning that are incorporated into teaching to support
affective learning (Boothe et al., 2018; Dalton, 2017; Evmenova, 2018). It requires
teachers to use diverse strategies that allow learners to be engaged and motivated
to learn (Dalton et al., 2012).
According to cognitive science, the affective network is part of the brain that is
linked to MME, which addresses the “why” of learning (Rose & Meyer, 2002).
Affective networks help regulate emotional involvement with learning such as
learners’ motivation and their ability to focus on and remain engaged with a task
(Darling-Hammond et al., 2020). MME, which is linked to affective networks,
requires teachers to be reflective about their teaching practices in order to devise
flexible engagement strategies to enhance mathematics learning.
73
It should be noted though that the MME principle does not only address the issue
of why learners engage but also why they engage in the manner in which they do.
For this reason, Hartnett (2020) stresses the need to consider it in teaching and
learning contexts. Courey et al. (2013) also recommend the use of MME, as it
enables teachers to recognise that no single option of engagement works for all
learners and provides a comprehensive guide in terms of how teachers should
provide learners with a range of engaging learning materials to maximise their
learning and engage them in meaningful mathematics learning. Capp (2020) notes
that MME is vital to help stimulate learners’ interests and increase their
motivation to learn and, accordingly, recommends teachers to consider it.
When teachers apply the MME principle, they are often advised to look into
finding ways to link material to learners’ lives, interests, and experiences by
giving them choices (Dalton, 2017). This principle guides engagement by
providing three types of options, namely; “options for self-regulation, for
sustaining efforts and persistence, and for recruiting interest” (Capp, 2020; Center
for Applied Specialised Technology, 2014). In this study, teachers reflected on
how they used this principle to maximise mathematics learning. The study
therefore highlights the teachers’ experiences of and insights into how they
provided these three options. Table 1 reflects the MME principle and the
guidelines it consists of. These are the guidelines that guided or informed the
study.
Table 1: Multiple means of engagement principle (Center for Applied Specialised

Technology, 2014)
Provide multiple means of engagement
Provide options for self-regulation
+ Promote expectations and beliefs that
optimise motivation
+ Facilitate personal coping skills and strategies
+ Develop self-assessment and reflection
Provide options for sustaining effort
and persistence
+ Heighten salience of goals and
objectives
+ Vary demands and resources to
optimise challenge
+ Foster collaboration and a sense of
community
+ Increase mastery-orientated feedback
Provide options for recruiting interest
+ Optimise individual choice and autonomy
+ Optimise relevance, value, and authenticity
+ Minimise threats and distractions
3. Learner Engagement Theory

The learner engagement theory was adopted as the lens underpinning the study.
Engagement refers to an individual’s active participation and involvement in a
particular activity, where motivation expresses impetus behind the activity
(Appleton et al., 2006). Sinatra et al. (2015) postulate that the “concept of
engagement” can be explored from different levels in the learning process. The
74
first level is “micro-level engagement”, which they define as a learner’s state of

“engagement in a moment, task, or learning activity”. This is the level that is
typically found in a class with a time constraint in the order of minutes to hours.
The second level is “macro-level engagement” which characterises engagement
across time (e.g., days and weeks), contexts (e.g., classwork, homework, and
groupwork) and activities associated with the learning tasks that take place before
and after learning events. According to Csikszentmihalyi (1997), the highest
degree of micro-level engagement is represented by the flow concept, which is
experienced when interaction with the learning task results in the learner being
“in a state of effortless attention, arising through an interaction between positive
affect and high attention” (De Manzano et al., 2010, p. 301). According to Engeser
(2012), flow is observable when learners become “totally absorbed in” and focus
on the given tasks. The improved “state of flow” takes place under three main
conditions, namely; clearly defined task goals provided to the learners, prompt
constructive feedback provided to the learners regarding their performance and
progress made, and the learners’ current skill levels being “matched to the
challenge” they are faced with (Engeser & Rheinberg, 2008). The learner
engagement theory was deemed suitable for the study because it highlights
different types of engagement, explains what engagement should look like, and
outlines what makes it successful.
4. Research Design and Methodology

In this phenomenological case study, eight teachers (with teaching experience
ranging from 8 to 18 years) of one high school in the Thabo Mofutsanyane district
were purposively selected to participate in the research. Of the eight teachers who
participated in the study, five were males and three were females. The age of the
teachers ranged between 36 to 52 years. The school in which the research was
conducted is one of the biggest high schools in the district, with an enrolment of
approximately 1 600 learners. The school was selected because the mathematics
teachers in this school had previously undergone UDL training and, therefore,
were familiar with all of its principles, including MME. These teachers were
chosen because of their lengthy experience in teaching mathematics and their
knowledge of the application of UDL principles in the classroom.
A series of six recorded focus group discussions, each session lasting for two
hours, were conducted to generate data. As the teachers had previously
undergone UDL training and had implemented the principles in their classes
prior to the outbreak of the pandemic, the researcher became interested in finding
out how they were using these principles (especially the MME principle) given
the pandemic that had forced schools to observe the rules of the national disaster
(wearing masks, social distancing, etc.).
Focus group discussions were conducted with the mathematics teachers so that
they could provide first-hand information on how they had implemented the
MME principle to maximise learning. The themes that were used were informed
by the guidelines of MME as shown in Table 1. The first theme was based on the
teachers’ implementation of MME to “provide options for self-regulation”; the
second theme was based on the teachers’ implementation of MME to “provide
75
options for sustaining effort and persistence”; and the third theme was based on
the teachers’ implementation of MME to “provide options for recruiting interest”.
The subthemes and categories were also informed by the MME guidelines as
shown in Table 1. The themes and subthemes were used to understand how the
teachers had adopted MME to maximise mathematics learning.
The qualitative content analysis techniques were used to analyse the data; they
were partly directed and partly inductive. The techniques involved numerous
steps for analysing data, beginning with the reading of the data texts, placing
different labels on the texts to differentiate it (open-coding), and clustering the
emerging concepts into bigger clusters (categories). The bigger clusters were
further grouped into themes that came out from the data (Hsieh & Shannon, 2005).
The analysis of data in this study was directed because the general themes were
determined a priori. Since the study sought to explore a phenomenon using a few
participants in their school setting, the findings of this study cannot be
generalised. However, in a context similar to that of this study, these results could
apply. Furthermore, the findings of this study may contribute to the body of
knowledge by illuminating some ways in which MME can be implemented to
effectively engage learners and maximise learning.
The free attitude interview (FAI) technique was used for data collection during
the focus group discussions. In line with FAI, the open-ended questions that were
posed during the focus group discussion (based on how the teachers had
implemented the MME principle by providing “options for self-regulation, for
sustaining effort and persistence, and for recruiting interest” to maximise
learning) made it possible for the participants to engage in lengthy discussions
leading to the generation of rich data. In order to eliminate researcher bias,
member checking was conducted to ensure that the data were analysed and the
findings interpreted in a manner that correctly captured and portrayed the
participants’ explanations, ideas, and opinions.
The University of the Free State issued ethical clearance to conduct the research.
The Free State Department of Education and the principal of the school where the
study was conducted were approached in order to ask for their permission to
conduct the study and they gave approval. The participants (teachers) in the
research signed consent forms. Their right to confidentiality was emphasised, and
anonymity was guaranteed. The participants were requested to participate on a
voluntary basis, and they were informed that they would not be penalised should
they decide not to proceed with participation at any stage of the research project.
5. Findings and Discussion

The following sections highlight the mathematics teachers’ experiences of and
insights into how they adopted the MME principle and applied it to provide the
“options for self-regulation, options for sustaining effort and persistence, and
options for recruiting interest”.
76
5.1 Provide Options for Self-regulation

Self-regulation is of extreme importance, especially in the pandemic context in
which learners were given tasks to perform individually. This necessitated the
teachers to develop self-regulation strategies to enable learners to learn
independently and to maximise learning. During a focus group discussion, the
teachers reflected on how they had implemented the MME principle to provide
options for self-regulation. They highlighted some of the strategies they had
employed to “promote expectations and beliefs that optimise motivation” (Center
for Applied Specialised Technology, 2011). They reflected as follows:
It is important to give mathematical tasks with clear instructions on what
[is] to be done. Learners usually respond better when they know what they
are expected to do. They also respond better when they are given the
opportunities to tackle tasks in their own ways. (Teacher 3)
It also helps to let learners know what you want from them … what is it
that you want them to do … what you expect of them to do with respect
to tasks that you give them. (Teacher 4)
I found it useful to give learners problems which they understand. They

become motivated when they work with such problems. (Teacher 2)
The above extracts indicate the significance of making learners understand

expectations in order to successfully engage them and maximise their learning.
According to Teacher 3, teachers can do so by providing learners with clear
instructions on what is expected from them so that they know how to approach
the given tasks. According to both Teachers 3 and 4, learners respond better when
they understand the tasks they are given and know what is expected from them.
According to Teacher 2, it is vital to give learners mathematical problems that they
understand, as this will motivate them to attempt to solve these tasks or problems.
What also seems to be important to do as part of engaging learners and
maximising their learning is to give them opportunities to solve problems using
their own approaches. Such opportunities will make learners feel in charge and
responsible for their own learning processes and thus raise their motivation to
engage in the given tasks. The form of teaching that is espoused from the extracts
in which expectations are promoted is regarded as a “high impact practice”,
which Ambrose et al. (2010) deem significant in terms of motivating learners to
learn so that they can make the progress necessary to contribute towards their
metacognitive gains. Teaching that promotes high expectations and beliefs not
only improves learning but also elevates motivation, inculcates a sense of self-
efficacy in learners, and results in the higher levels of engagement necessary for
meaningful mathematics learning experiences and maximising learning.
As learners were given tasks to work on individually whilst at home during the
lockdown, teachers had to use strategies that helped facilitate the learners’ coping
skills. The teachers reflected as follows:
What I tried to do during lockdown was to give my learners detailed
examples and step-by-step guidelines for solving problems so that they
can [sic] do the problems on their own wherever they were. (Teacher 4)
77
I found giving tasks together with the checklists helpful because the
learners are able to check how far they are with the task and also what is
still outstanding. This also eliminates distractions they might come across
and ease the anxiety that comes with engaging them with the task while
working alone. (Teacher 1)
Giving them different ways to solve the mathematical problems helps,

especially when I give similar tasks because the learners were able to apply
the methods that worked for them and not rely on the ones that I
introduced them to. (Teacher 5)
The extracts above indicate that engagement is not only a “classroom business”
but an important practice that should also be implemented when learners are
working on their own outside the mathematics classroom. The pandemic had thus
forced teachers to use strategies that would help learners engage with the learning
tasks while working individually without their teacher’s physical presence.
According to Teacher 4, learners can be engaged outside the classroom with the
tasks given to them by providing them with detailed, explicit examples, serving
as a guide on how to solve the problems, and step-by-step guidelines on how to
solve mathematical problems before they have to solve similar problems
independently. According to Lipscomb et al. (2004), a step-by-step guide serves
as an important practice and a useful scaffold for supporting learning by helping
to bridge the task demand in light of the learners’ current level, thus enabling
learners to be more successful than they would have been otherwise. Step-by-step
task-orientated scaffolding also provides learners with ample support and
guidance necessary to manage the main areas of difficulty when they are working
independently (Smirnova, 2016). According to Teacher 1, checklists are also useful
when learners work independently, as these help learners to monitor their own
work and to establish where they are with the given tasks by identifying the
completed and outstanding parts of the tasks. Checklists therefore serve as an
adaptive strategy for helping learners to monitor, manage, and direct their
emotional responses to external events, while also helping to reduce task-
irrelevant distracters. Hyppönen, Hirsto & Sointu (2019) note that learners who
monitor themselves and also have the ability to self-regulate their “learning
processes are more likely to perform well in their academic tasks”. Moreover,
Teacher 5 alluded that exposing learners to different ways of solving
mathematical problems not only optimises learning but can also serve as an
opportunity for teachers to assist learners in coping with the demands of the
mathematical content. Introducing different methods of solving mathematical
problems provides learners with an opportunity to select and use simple methods
for them, which, in turn, makes it possible for learners to cope with the demands
of the given tasks.
Developing the self-assessment and reflection strategy was also highlighted as a

good strategy for self-regulation. The teachers reflected as follows:
Although we are able to monitor the learners’ progress when giving them
problems to solve in class, we still have to make sure that we empower our
learners to be able to monitor their own progress when they are working
on their own … Eeeerrrrr, like during the lockdown. (Teacher 7)
78
Bearing in mind that our learners differ in so many ways, I think we need
to give clear guidelines to all of them, especially those who cannot be able
[sic] to do this on their own. (Teacher 8)
I assisted my learners to do self-assessments and reflections, and I realised

that that helped them realise areas where they were struggling as well as
realise the progress they were making, and that was really motivating for
them. (Teacher 6)
Monitoring is an important teaching and learning practice that teachers have to

inculcate in learners so that they are able to monitor their progress. This was
emphasised by Teacher 7, who pointed out that teachers had the responsibility to
enable learners to monitor their own progress. This means that teachers have to
develop self-assessment strategies and explicitly teach and guide learners on how
to do self-assessment. By doing so, the teachers will develop a better capacity for
self-regulation. According to Bercher (2012), self-assessment and reflection are
important because they enable learners to monitor not only their progress but also
their emotions and reactivity sensibly. In accordance with Teacher 7, Teacher 8
added that learners differed in many ways and, on the basis of this, teachers had
to provide a great deal of explicit instruction and guidelines for learners for them
to learn how to successfully self-assess. The need for self-assessment and
reflection was further stressed by Teacher 6, who encouraged teachers to help
learners self-assess and reflect because the mere recognition of one’s own progress
towards greater independence could be highly motivating.
5.2 Provide Options for Sustaining Effort and Persistence

The discussion on how teachers implemented the MME principle revealed the
need to offer alternatives for sustaining effort and persistence as another
important strategy to engage learners and maximise learning. The teachers
pointed out the significance of reminding learners about the lesson outcome
expected to be achieved and how to achieve it. They reflected as follows:
I think we need to keep on reminding learners of the lesson outcome they
are expected to achieve. When learners are given a task to do, they must
be given timeframes to complete it and in-between be reminded of what
they have to do to achieve the lesson outcome. (Teacher 3)
I often try to show various ways in which the goal can be achieved.
(Teacher 1)
… the same way I do in class under normal circumstance[s] to encourage

learners and motivate them to learn, I found that working for me,
especially when giving them meaningful feedback to enable them to
correct their mistakes and address misconceptions. (Teacher 5)
According to the extracts above, one way of sustaining effort and persistence
when engaging learners, as highlighted by Teacher 3, is by regularly reminding
learners about the intended lesson outcome to be achieved regarding the given
mathematical tasks. The timeframe for completing a task must be communicated,
79
and teachers should be there to remind the learners about the lesson outcomes to
be achieved and guide them so that the outcome can eventually be achieved. Such
support is required to help sustain the effort and concentration of the learners in
the face of distracters. Teacher 1 explained that the lesson outcome (i.e., intended
goal) to be achieved should be displayed or demonstrated in multiple ways; hence
the statement “I often try to show various ways in which the goal can be achieved”. What
Teacher 1 pointed out is important in order to cater for a diverse learner
population. The practice of making a goal clear to the learners, in an endeavour to
maximise learning, is advocated by Anderson and Stritch (2016), who avow that
even if the goal may be difficult to achieve, when learners know it and are guided
well on how to achieve it, they will pursue it. Hatip (2020) also espouses the notion
of making a goal clearer so to maximise learning. Being guided on how to achieve
the goal thus helps in terms of raising learners’ motivation levels and inspires
them to make an effort and persist in engaging in the given mathematical tasks,
even in the face of challenges. According to Teacher 5, another way to sustain
effort and persistence, especially in the time of the pandemic when learners are
working independently, is to continually motivate and encourage the learners by
providing them with immediate corrective feedback that will enable them to
correct their mistakes and address misconceptions. Febrianto et al. (2020) also
stress the significance of motivating learners when they are working
independently particularly during the pandemic when teachers are not present.
In engaging learners, teachers should keep in mind the fact that learners differ in
terms of how they learn and their levels of understanding content. This implies
that learners should be engaged in different ways and be given tasks that match
their learning abilities and different levels of understanding. Consequently,
teachers should vary the demand and resources to meet challenges when
engaging learners in mathematical activities. This point was raised by Teachers 2
and 3, who narrated as follows:
We teach learners who learn differently, and therefore we need to vary the
resources and also give tasks that learners will be able to tackle. (Teacher
2)
Knowing my learners helped me a lot because I was able to give them tasks
that match their abilities. (Teacher 3)
The practice of varying demand and resources to meet challenges when engaging
learners is supported by the Center for Applied Specialised Technology (2011),
which asserts that no single means of engagement is optimal for all learners in all
contexts. According to Teacher 3, knowing the types of learners in her class was
helpful in terms of devising activities that matched the abilities of her learners.
Matching abilities with task demands thus helps to sustain motivation for learners
to remain engaged with activities.
Fostering collaboration and a sense of community was also highlighted as a useful

strategy for engaging learners and maximising learning (Mishra et al.,2020).
Although during the lockdown, the teachers were not able to implement the
strategy, they did manage to implement it shortly after the lockdown, during the
period of alternating groups of learners coming to school every week. The
teachers commented as follows:
80
Although during [the] lockdown I could not be able [sic] to create groups
for learners to work individually, I managed, though, to create the [sic]
small groups during the period of alternating groups of learners that came
to school after [the] lockdown. Creating such small groups in class was
helpful. It gave learners opportunities to work together and learn from
each other, although I had to ensure that they observe the social distancing
rule and also wear masks during their interaction in class. (Teacher 6)
I formed the groups in such a way that learners were able to work well
together so that they can benefit from each other. (Teacher 4)
I found it helpful to change the members of the groups from time to time.
(Teacher 7)
Fostering collaboration among learners was not possible during the lockdown
period; however, when the learners went back to school the teachers managed to
get them to work in small, intimate groups while observing the rules of the
pandemic. The teachers deemed the practice of fostering collaboration among the
learners important as part of engaging learners and maximising their learning.
According to Teacher 6, fostering collaboration can be done by creating small,
intimate groups. The teacher regarded group work as an important practice that
should be implemented to afford learners opportunities to work cooperatively
and learn from one another. Teacher 4 pointed out that the groups of learners
should be carefully formed so that they could derive maximum benefit by
learning from one another. Teacher 7 added that members of the groups should
be changed from time. Changing the members of groups frequently allows
flexibility and exposes learners to diverse ideas from time to time, which is vital
for maximising learning. The practice of peer cooperation is highly espoused in
teaching and learning contexts and is a practice that is highly recommended in
the 21st century. According to Poellhuber et al. (2008), peer collaboration helps to
“create a sense of community” that is much needed to sustain learner motivation
and persistence and support learners’ commitment to studying and remaining
engaged. McKeen (2019) notes that flexible grouping enables better differentiation
and manifold roles, while also providing opportunities for learners to work
collaboratively. Flexible grouping also allows mathematics teachers to adapt their
teaching and this enables learners to achieve the learning objectives.
The teachers also reflected on increasing mastery-orientated feedback as a key

strategy in implementing the MME principle of “providing options for sustaining
effort and persistence” (Center for Applied Specialised Technology, 2011). The
teachers reflected as follows:
When I engage my learners, it really helped to give them assessments and
provide feedback from time to time, especially feedback that is encouraging
and helpful … I think we must give them feedback that makes them realise
their mistakes so that they can be able [sic] to correct the mistakes, and
this needs to be done on time. (Teacher 8)
Sometimes when we give feedback, we tend to compare and create

unhealthy competitions amongst the learners. Again, we give feedback
81
that focuses only on the marks instead of addressing the errors which
learners often commit. I think this is what we need to change if we want
our learners to remain engaged. (Teacher 1)
According to Teacher 8, one way in which teachers can provide mastery-

orientated feedback is by frequently giving assessment and feedback. The
expression “feedback that is encouraging and helpful” refers to feedback that not only
makes it possible for learners to realise their mistakes and what causes these
mistakes but also indicates areas where they are doing well. This means that
assessment should not only focus on weaknesses but also point out the strengths
of the learners as part of motivating them. The practice of giving constructive and
corrective feedback is crucial in terms of sustaining the learners’ motivation and
effort towards learning (Al-Hattami, 2019).
According to Teacher 1, ensuring that learners remain engaged requires teachers

to give constructive and corrective feedback that is aimed at building the learners
and not unfair feedback that focuses on comparing their strengths and
weaknesses. The feedback should not focus only on the marks awarded to learners
but also on worked-out problems. This should be done by paying attention to how
the problems were solved (i.e., steps for solving the problems) in order to identify
errors made so as to share them with the learners at a later stage in an endeavour
to enlighten and correct them. This type of feedback is regarded as mastery-
orientated feedback because it serves to guide learners towards “mastery rather
than a fixed notion of performance compliance” (Darling-Hammond et al., 2020).
This type of feedback is vital for sustained engagement because it accentuates the
importance of “process” over “product”, thus encouraging learners to embrace
the notion of focusing on the attainment of the lesson objectives and appreciating
the progress they are making in terms of learning content rather than focusing on
the marks. This approach to feedback is also vital in terms of nurturing “mastery
orientation on the part of the learners” where they seek not only to advance their
understanding of content, skills and learning strategies, but also to identify
relevancy of the mathematical concepts which they are learning and
“meaningfulness” in the task itself (Ames, 1992; Hattie & Gan, 2011).
The suggestions made by the teachers necessitate teachers to carefully look into
the feedback they provide to ensure that it inspires persistence in learners to be
able to remain engage in problem solving.
5.3 Provide Options for Recruiting Interest

The pandemic has made learning difficult for most learners, as they had to
struggle individually working on their tasks. The issue of working individually
necessitated teachers to devise strategies for recruiting learners’ interest so they
could remain engaged and motivated to work on the given mathematical tasks.
The teachers thus highlighted some strategies that they used to optimise
individual choice and autonomy as a way of recruiting learners’ interest and
maximise learning. They narrated as follows:
One of the things that worked for me was to try to relate content and real
life. Relating content to real life helps because it makes learners realise the
importance of what they are learning. (Teacher 5)
82
… just because we present content in a manner which other learners find

easy to relate to, that should not make us assume that all learners prefer
to be taught in that way. We need to vary the activities and present
content in different interesting ways. Eeeehhhh … we really need to come
up with different strategies that will stimulate the learners’ interest.
(Teacher 7)
Teacher 5 highlighted the usefulness of real-life situations when engaging learners

(“because it makes learners realise the importance of what they are learning”) as one way
in which teachers could maximise learning, engage learners in meaningful
learning, and also recruit their interest in learning mathematical concepts.
According to the teacher, the relevance of the concepts taught in class and
demonstrating this relevance through authentic, meaningful activities maximise
learning. Moleko and Mosimege (2020) avow that teaching mathematics in
context increases engagement. This form of teaching enables learners to apply the
learnt concepts in real life, helps develop their new perceptions and skills to be
able to use mathematical reasoning to problems, and enhances their capability to
partake in the “economy of today and tomorrow” (Umugiraneza et al., 2017).
According to Teacher 7, teachers should not “assume that all learners will find the
same mathematical activities or content equally relevant or valuable”; therefore,
they should accordingly vary the activities or forms of content representations “in
order to cater for the diverse learning preferences and styles of all the learners”.
Teachers should also invest time in devising various strategies that will assist in
terms of stimulating learners’ interest so they will remain engaged. However, it
should be noted that teachers can only successfully do so if they know their
learners. For instance, teachers can design visual activities for learners who best
assimilate information when represented in visual formats and do the same with
other groups of learners with different learning styles and preferences (auditory
learners, tactile learners, etc.).
In order for effective learner engagement to be realised and for learning to be

maximised, teachers have to ensure that they minimise threats and distractions
that may impede learning (Center for Applied Specialised Technology, 2011). In
line with this, the teachers commented as follows:
I have realised that one of the things that kill our learners, especially when
we do not address it, is the mathematical vocabulary. It is difficult for
learners to solve problems when they do not understand some of these
words. For example, words such as “at least”, “compound”, etc. should
be taught to learners so that they can know what they mean. (Teacher 2)
Our teaching should not create misunderstandings because this later

makes it difficult for learners to solve problems on their own. For example,
two raised to exponent three should be pronounced as such and not as two
to power three. (Teacher 3)
Teacher 2 highlighted the significance of addressing the insufficient knowledge of

mathematical vocabulary in order to eliminate knowledge gaps that could
hamper the learning process. According to the teacher, lacking an understanding
83
of the mathematical vocabulary could be a potential threat that can impede

mathematical problem solving. The teacher gave an example of terms such as “at
least” and “compound”, which are not only used within a mathematical context
but also in other contexts. These are words that teachers should explain so that
their meanings can be understood in context to avoid confusion (Moleko &
Mosimege, 2020).
Furthermore, Teacher 3 noted that teaching that causes misunderstanding should

be avoided in order for learners to be engaged in meaningful mathematics
learning. The example the teacher gave is that of “two raised to exponent three”
and “two to power three”. According to the teacher, these two expressions are
often used interchangeably; yet they do not mean the same thing. For example,
two raised to exponent three can be represented numerically as 23 (where 2 is a
base and 3 an exponent); and 23 (a base together with an exponent) is called a
“power”, mathematically speaking. Consequently, failure on the part of teachers
to mind how they teach these mathematical expressions in class may result in
teaching that causes distractions, which could subsequently impede learning. The
teaching of mathematics therefore requires teachers to avoid or eliminate
elements of distractions at all times in order to maximise learning.
6. Summary of examples of multiple means of engagement strategies to

maximise learning of mathematics in pandemic-regulated classrooms
6.1 Offer the Alternatives for Self-regulation
The findings of the study indicated that during the lockdown period learners
worked on their own without the physical presence of the educators to guide
them. This necessitated the clear expectations regarding the given mathematical
tasks to be communicated so that learners could understand what is expected of
them. The significance of giving learners the mathematical problems, which they
possess knowledge of, was highlighted as a good strategy to motivate and ensure
that they remain engaged. This therefore means that the teachers have to design
mathematical examples that are clear to follow, and also provide the step-by-step
guidelines in order to make it easy for learners to solve similar given problems
independently. Lipscomb et al. (2004) regard the step-by-step guide as an
important practice and a useful scaffold to support learning and help bridge the
task demand in light of the learners’ current level. When learners are expected to
learn independently, it also helps to provide them with checklists so that they can
be able to monitor their own progress. Inculcating the culture of self-monitoring
and empowering learners with skills to monitor their own progress is a
worthwhile practice that helps facilitate independent learning (Bellini & Peters,
2008). Self-assessment and reflection were also found to be useful in terms of
enabling learners to identify areas in which they were struggling with as well as
those they were making good progress on. Bercher (2012) regards self-assessment
and reflection practices worthwhile in terms of assisting learners not only to
monitor their progress but also to monitor their emotions and reactivity sensibly.
A blend of these strategies emerging from literature and empirical data thus point
to self-regulation, which is constituted within the MME framework. Since learners
“differ markedly in the ways in which they could be engaged”, this necessitates
teachers to vary the strategies and to offer alternatives for self-regulation.
84
6.2 Offer the Alternatives for Sustaining Effort and Persistence

Many learners find mathematics challenging and often develop senses of
helplessness and resentment towards the subject. These behavioural dispositions
are often exhibited by learners in classrooms, in the presence of teachers who are
there to guide them. Solving mathematical problems on their own (without the
presence of the teachers) thus require teachers to use strategies that will help
sustain effort and persistence. Although this may be difficult to achieve, findings
of the study indicate several MME strategies that can be used to sustain effort and
persistence. These strategies include; regularly reminding learners about the
intended lesson outcomes which should be achieved, communicating the
timeframes for completing the tasks and guiding the learners on how to achieve
the learning goal. Anderson and Stritch (2016), maintain that the practice of
guiding learners on how to achieve the learning goal is significant because even if
the learning goal may be difficult to achieve, when learners know it and are
guided well on how to achieve it, they will pursue it regardless. Providing instant
corrective feedback is also vital in terms of enabling learners to realise their
mistakes and misconceptions soon and to correct them. Giving tasks that match
the learners’ learning abilities and different levels of understanding is also
important in terms of motivating learners when they are working independently.
This therefore necessitates teachers to vary the demand and resources to meet
challenges when engaging learners in mathematical activities. Fostering
collaboration amongst the learners affords learners with opportunities to work
together to learn from one another. However, the groups should be carefully
formed so that learners could derive maximum benefit by learning from one
another. Changing the members of groups frequently is also important as it allows
flexibility and exposes learners to diverse ideas from time to time. Poellhuber et
al. (2008) espouse the notion of peer collaboration since it helps to “create a sense
of community that is much needed to sustain learner motivation and persistence
and support learners” commitment to studying and remaining engaged.
Flexibility grouping on the other hand enables better differentiation (McKeen,
2019) and also allows teachers to tailor mathematics instruction such that they
meet the needs of diverse learners. The practice of providing the constructive and
empowering corrective feedback is crucial in terms of sustain motivation and
effort (Al-Hattami, 2019). This should be the type of feedback that not only focus
on making learners aware of their mistakes but also shows areas in which they are
doing well. A mixture of these strategies emerging from literature and empirical
data thus point to the notion of sustaining effort and persistence which is also
constituted within the MME framework. Since learners differ in terms of
motivation levels, MME thus advocates the notion of offering varied strategies
and alternatives for sustaining effort and persistence.
6.3 Offer the Alternatives for Recruiting Interest

Recruiting interest is important to ensure that learners remain focused on the
given mathematical tasks. One way to recruit interest is through the use of real-
life situations. Learners learn best when real life situations which they are familiar
with are used and related to the content they are being taught. This form of
teaching enables learners to apply the learnt concepts in real life, to “develop new
perceptions and skills to apply mathematical reasoning to problems”, and to
85
enhance their “capacity to participate” in the economy of today and tomorrow

(Umugiraneza et al., 2017). Since learners may not find “the same mathematical
activities or content equally relevant or valuable”; it is therefore advisable for
teachers to vary the activities or forms of content representations in order to cater
for the diverse learning preferences and styles of all the learners. Lack of
understanding the mathematical vocabulary should be addressed since it can be
a potential threat that can impede mathematical problem solving. Teaching that
causes misunderstanding and confusion should therefore be avoided in order for
learners to be engaged in meaningful mathematics learning.
The findings of the study revealed two significant contributions (made by the
study): firstly, the findings of the study contribute to the expansion of the teaching
and learning theories in mathematics to promote; active learning, inclusivity and
sustainability. Secondly, the study serves as a guide for promoting learner
engagement by proposing a “new idea” of multiple means of engagement (MME)
approach in mathematics, which should be regarded as an essential strategy to
promote active learning.
The summary of the MME strategies highlighted above were deemed useful to
provide all learners a better platform for overcoming their problems with
mathematics and the challenges brought about by the pandemic regulations. It
should be noted though that, different environments may require different
combinations/combinations of the above strategies.
7. Conclusion
The study highlighted how mathematics teachers had adopted MME to maximise
learning. It was revealed that “no single means of engagement will be ideal for all
learners in all contexts”; therefore, it is essential for teachers to vary their teaching
strategies to provide multiple options for engagement. The teachers in the study
maximised mathematics learning by using MME guidelines, namely; “providing
options for self-regulation, options for sustaining effort and persistence, and
options for recruiting interest”. Providing options for self-regulation in the study
involved promoting high expectations for optimising learner motivation to
engage with the given mathematical tasks, facilitating personal coping skills and
strategies for learners to sustain engagement with the tasks even in the face of
difficulties, and developing self-assessment strategies and teaching learners how
to use them so they were able to monitor their progress. Providing “options for
sustaining effort and persistence” required the teachers to indicate the learning
goal to be achieved and guide their learners on how to achieve it, to foster peer
collaboration among the learners so they could engage and learn from one
another, and to increase mastery-orientated feedback by providing immediate
feedback that was constructive and corrective. Providing “options for recruiting
interest” required the teachers to reinforce relevance and value by using real-life
examples so that learners could recognise the applicability of the concepts they
were learning in reality and also by eliminating elements of threats and
distractions that might hamper learning. The MME approach in mathematics was
also found to be useful in terms of developing purposeful and motivated learners.
One of the greatest lessons the study provided is that teachers are “a crucial part
86
of curriculum implementation” and therefore they cannot detach themselves from

the learning experiences of learners (Mapuya, 2021). Hence they have to strive to
monitor, assess and engage learners even when learners are not in class due to
Covid pandemic.
8. Recommendations
The findings of this study suggest the need for further research to be conducted
on different topics in mathematics in order for teachers to maximise learning
through MME. It should be noted that different mathematics topics necessitate
different ways in which learners can be engaged in the learning processes. The
MME principle will therefore serve as a guideline for engaging learners in
meaningful mathematics learning as well as maximising the learning thereof.
Also, MME will enable teachers to cater for and provide alternative ways in which
to engage a broad range of learners in meaningful mathematics learning
experiences.
Funding: This research is funded by the National Research Foundation (NRF)
Conflicts of Interest: The author declares no conflict of interest. The study was
conducted primarily for scholarly purposes. The conceptualisation of the study
and all the empirical processes were not influenced by own personal interest or
anticipated gain.
10. References
Al-Hattami, A. A. (2019). The perception of students and faculty staff on the role of
constructive feedback. International Journal of Instruction, 12(1), 885-894.
Ambrose, S. A., Bridges, M. W., DiPietro, M., Lovett, M. C., & Norman, M. K. (2010). How
learning works: Seven research-based principles for smart teaching. John Wiley & Sons.
Ames, C. (1992). Classrooms: Goals, structures, and student motivation. Journal of
Educational Psychology, 84(3), 261. https://psycnet.apa.org/doi/10.1037/0022-
0663.84.3.261
Anderson, D. M., & Stritch, J. M. (2016). Goal clarity, task significance, and performance:
Evidence from a laboratory experiment. Journal of Public Administration Research
and Theory, 26(2), 211-225. https://doi.org/10.1093/jopart/muv019
Appleton, J. J., Christenson, S. L., & Furlong, M. J. (2008). Student engagement with school:
Critical conceptual and methodological issues of the construct. Psychology in the
Schools, 45(5), 369-386. https://doi.org/10.1002/pits.20303
Appleton, J. J., Christenson, S. L., Kim, D., & Reschly, A. L. (2006). Measuring cognitive
and psychological engagement: Validation of the Student Engagement
Instrument. Journal of School Psychology, 44(5), 427-445.
https://doi.org/10.1016/j.jsp.2006.04.002
Balwant, P. T. (2018). The meaning of student engagement and disengagement in the
classroom context: Lessons from organisational behaviour. Journal of Further and
Higher Education, 42(3), 389-401. https://doi.org/10.1080/0309877X.2017.1281887
Bansilal, S., & Ubah, I. (2019). The use of semiotic representations in reasoning about
similar triangles in Euclidean geometry. Pythagoras, 40(1), 1-10.
https://doi.org/10.4102/pythagoras.v40i1.480
Bellini, S., & Peters, J. K. (2008). Social skills training for youth with autism spectrum
disorders. Child and adolescent psychiatric clinics of North America, 17(4), 857-873.
https://doi.org/10.1016/j.chc.2008.06.008
87
Bercher, D. A. (2012). Self-monitoring tools and student academic success: When

perception matches reality. Journal of College Science Teaching, 41(5), 26-32.
Boothe, K. A., Lohmann, M. J., Donnell, K. A., & Hall, D. D. (2018). Applying the principles
of universal design for learning (UDL) in the college classroom. The Journal of
Special Education Apprenticeship, 7(3), 1–13.
Burgstahler, S. (2008). Equal access: Universal design of instruction. DO-IT, University of
Washington. edu/doit/Brochures/Academics/equal_access_udi.htm
Capp, M. J. (2020). Teacher confidence to implement the principles, guidelines, and
checkpoints of universal design for learning. International Journal of Inclusive
Education, 24(7), 706-720. https://doi.org/10.1080/13603116.2018.1482014
Center for Applied Special Technology. (2014). UDL guidelines – Version 2.0: Principle III
provide multiple means of engagement.
http://www.udlcenter.org/aboutudl/udlguidelines/principle3
Center for Applied Specialised Technology [CAST]. (2011).Universal Design for Learning
Guidelines version 2.0. Wakefield, MA: Author.
Courey, S. J., Tappe, P., Siker, J., & LePage, P. (2013). Improved lesson planning with
universal design for learning (UDL). Teacher Education and Special Education, 36(1),
7-27. https://doi.org/10.1177%2F0888406412446178
Csikszentmihalyi, M. (1997). Flow and education. NAMTA Journal, 22(2), 2-35.
Dalton, E. M. (2017). Beyond universal design for learning: Guiding principles to reduce
barriers to digital & media literacy competence. Journal of Media Literacy Education,
9(2), 17-29. http://digitalcommons.uri.edu/jmle/
Dalton, E. M., Mckenzie, J. A., & Kahonde, C. (2012). The implementation of inclusive
education in South Africa: Reflections arising from a workshop for teachers and
therapists to introduce universal design for learning. African Journal of Disability,
1(1), 1–7. https://doi.org/10.4102/ajod.v1i1.13.
Darling-Hammond, L., Flook, L., Cook-Harvey, C., Barron, B., & Osher, D. (2020).
Implications for educational practice of the science of learning and development.
Applied Developmental Science, 24(2), 97-140.
https://doi.org/10.1080/10888691.2018.1537791
De Manzano, Ö., Theorell, T., Harmat, L., & Ullén, F. (2010). The psychophysiology of flow
during piano playing. Emotion, 10(3), 301. DOI: 10.1037/a0018432
Duchesne, S., & Ratelle, C. (2010). Parental behaviors and adolescents’ achievement goals
at the beginning of middle school: Emotional problems as potential mediators.
Journal of Educational Psychology, 102(2), 497. DOI: 10.1037/a0019320
Engeser, S. E. (2012). Advances in flow research. Springer Science+ Business Media.
https://psycnet.apa.org/doi/10.1007/978-1-4614-2359-1
Engeser, S., & Rheinberg, F. (2008). Flow, performance and moderators of challenge-skill
balance. Motivation and Emotion, 32(3), 158-172. DOI 10.1007/s11031-008-9102-4
Evmenova, A. (2018). Preparing teachers to use universal design for learning to support
diverse learners. Journal of Online Learning Research, 4(2), 147-171.
https://www.learntechlib.org/p/181969/
Febrianto, P. T., Mas' udah, S., & Megasari, L. A. (2020). lmplementation of Online
Learning during The Covid-19 Pandemic on Madura lsland,
lndonesia. lnternational Joumal of Learning, Teaching and Educational Research, 19(8),
233-254. https://doi.org/10.26803/ijlter.19.8.13
Filgona, J., Sakiyo, J., Gwany, D. M., & Okoronka, A. U. (2020). Motivation in learning.
Asian Journal of Education and Social Studies, 16-37.
https://doi.org/10.9734/ajess/2020/v10i430273
88
Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of
the concept, state of the evidence. Review of Educational Research, 74(1), 59-109.
https://doi.org/10.3102%2F00346543074001059
Gray, T., & Madson, L. (2007). Ten easy ways to engage your students. College Teaching,
55(2), 83-87. https://doi.org/10.3200/CTCH.55.2.83-87
Hake, R. R. (1998). Interactive engagement versus traditional methods: A six-thousand-
student survey of mechanics test data for introductory physics courses. American
Journal of Physics, 66(1), 64-74. https://doi.org/10.1119/1.18809
Halverson, L. R., & Graham, C. R. (2019). Learner engagement in blended learning
environments: A conceptual framework. Online Learning, 23(2), 145-178.
https://doi.org/10.24059 /olj.v23i2 .1481
Hartnett, M. (2020). Relationships between online motivation, participation, and
achievement: More complex than you might think. Journal of Open, Flexible and
Distance Learning, 24(1), 75-88.
https://search.informit.org/doi/10.3316/informit.195515786038745
Hatip, A. (2020). The Transformation Of Learning During Covid-19 Pandemic Towards
The New Normal Era. PROCEEDING UMSURABAYA.
Hattie, J., Gan, M., & Brooks, C. (2011). Instruction based on feedback. Handbook of Research
on Learning and Instruction, 249-271.
Hsieh, H. F., & Shannon, S. E. (2005). Qualitative health research. Three approaches to
qualitative content analysis. Qualitative Health Research, 15(9), 1277-1288.
https://doi.org/10.1177/1049732305276687
Hughes, K. (2012). A behavioural understanding of privacy and its implications for
privacy law. The Modern Law Review, 75(5), 806-836.
https://doi.org/10.1111/j.1468-2230.2012.00925.x
Hyppönen, L., Hirsto, L., & Sointu, E. (2019). Perspectives on University Students’ self-
regulated learning, task-avoidance, time management and achievement in a
flipped classroom context. International Journal of Learning, Teaching and
Educational Research. https://doi.org/10.26803/ijlter.18.13.5
Klein, M. (2000). How active involvement in learning mathematics can preclude
meaningful engagement: Contributions from Foucault. Pedagogy, Culture &
Society, 8(1), 69-83. https://doi.org/10.1080/14681360000200079
Kuh, G. D., Kinzie, J., Schuh, J. H., & Whitt, E. J. (2011). Student success in college: Creating
conditions that matter. John Wiley & Sons.
Leonard, S. H. (2008). Measuring cognitive and psychological engagement in middle school
students [Doctoral dissertation. University of South Dakota].
Lipscomb, L., Swanson, J., & West, A. (2004). Scaffolding. Emerging perspectives on learning,
teaching and technology.
http://projects.coe.uga.edu/epltt/index.php?title=Scaffolding.
Mapuya, M. (2021). First-Year Accounting Student Teachers’ Constructivist Learning
Experiences, The Lecturer’s Role and Implications for Curriculum
Implementation. International Journal of Learning, Teaching and Educational
Research, 20(1), 103-119. https://doi.org/10.26803/ijlter.20.1.6
McKeen, H. (2019). The impact of grade level flexible grouping on math achievement
scores. Georgia Educational Researcher, 16(1), 48-62.
https://files.eric.ed.gov/fulltext/EJ1206047.pdf
Mishra, D. L., Gupta, D. T., & Shree, D. A. (2020). Online Teaching-Learning in Higher
Education during Lockdown Period of COVID-19 Pandemic. International Journal
of Educational Research Open, August, 100012.
https://doi.org/10.1016/j.ijedro.2020.100012
89
Moleko, M. M., & Mosimege, M. D. (2020). Teachers’ and learners’ experiences for guiding
effective teaching and learning of mathematics word problems. Issues in
Educational Research, 30(4), 1375-1394.
ttps://search.informit.org/doi/10.3316/informit.606279637425764
Padgett, J., Cristancho, S., Lingard, L., Cherry, R., & Haji, F. (2019). Engagement: what is
it good for? The role of learner engagement in healthcare simulation contexts.
Advances in Health Sciences Education, 24(4), 811-825.
https://doi.org/10.1007/s10459-018-9865-7
Park, C. (2003). Engaging students in the learning process: The learning journal. Journal of
Geography in Higher Education, 27(2), 183-199.
https://doi.org/10.1080/03098260305675
Parsons, S. A., Nuland, L. R., & Parsons, A. W. (2014). The ABCs of student engagement.
Phi Delta Kappan, 95(8), 23-27. https://doi.org/10.1177%2F003172171409500806
Poellhuber, B., Chomienne, M., & Karsenti, T. (2008). The effect of peer collaboration and
collaborative learning on self-efficacy and persistence in a learner-paced
continuous intake model. International Journal of E-Learning & Distance Education /
Revue Internationale du e-Learning et la Formation à Distance, 22(3), 41-62.
Pokhrel, S., & Chhetri, R. (2021). A literature review on impact of COVID-19 pandemic on
teaching and learning. Higher Education for the Future, 8(1), 133-141.
https://doi.org/10.1177%2F2347631120983481
Rose, D. H., & Meyer, A. (2002). Teaching every student in the digital age: Universal design for
learning. Association for Supervision and Curriculum Development.
Scott, S., McGuire, J. M., & Embry, P. (2002). Universal design for instruction fact sheet.
University of Connecticut, Center on Postsecondary Education and Disability.
https://www.westga.edu/student-services/counseling/assets-
counseling/docs/universal_design_fact_sheet.pdf
Sinatra, G. M., Heddy, B. C., & Lombardi, D. (2015). The challenges of defining and
measuring student engagement in science. Educational Psychologist, 50(1), 1-13.
https://doi.org/10.1080/00461520.2014.1002924
Sirin, S. R., & Rogers-Sirin, L. (2005). Components of school engagement among African
American adolescents. Applied Developmental Science, 9(1), 5-13.
https://doi.org/10.1207/s1532480xads0901_2
Smirnova, L. A. (2016). Scaffolding preparation for a reading exam. Journal of Teaching
English for Specific and Academic Purposes, 4(2), 391-398.
Smith, D. C., Ito, A., Gruenewald, J., & Yeh, H. L. (2010). Promoting school engagement:
Attitudes toward school among American and Japanese youth. Journal of School
Violence, 9(4), 392-406. https://doi.org/10.1080/15388220.2010.509308
Smith, M. R. (2004). Math anxiety: Causes, effects, and preventative measures [Senior honors
thesis. Liberty University]. https://digitalcommons.liberty.edu/honors/255
Thoonen, E. E., Sleegers, P. J., Oort, F. J., Peetsma, T. T., & Geijsel, F. P. (2011). How to
improve teaching practices: The role of teacher motivation, organizational factors,
and leadership practices. Educational Administration Quarterly, 47(3), 496-536.
https://doi.org/10.1177%2F0013161X11400185
Umugiraneza, O., Bansilal, S., & North, D. (2017). Exploring teachers’ practices in teaching
mathematics and statistics in KwaZulu-Natal schools. South African Journal of
Education, 37(2). https://doi.org/10.15700/saje.v37n2a1306
Williams, K. C., & Williams, C. C. (2011). Five key ingredients for improving student
motivation. Research in Higher Education Journal, 12, 1.
Yengin, İ., Karahoca, D., Karahoca, A., & Yücel, A. (2010). Roles of teachers in e-learning:
How to engage students & how to get free e-learning and the future. Procedia-
90
Social and Behavioral Sciences, 2(2), 5775-5787.

https://doi.org/10.1016/j.sbspro.2010.03.942
Zyngier, D. (2008). (Re)conceptualising student engagement: Doing education not doing
time. Teaching and Teacher Education, 24(7), 1765-1776.
https://doi.org/10.1016/j.tate.2007.09.004
91

Vol. 20, No. 8, pp. 91-107, August 2021
Online Design Thinking Problems for Enhancing

Motivation of Gifted Students
Ayed H. Ziadat
Al-Balqa’a Applied University, Princess Rahma University College,
Department of Special education, Al-Salt, Jordan
https://orcid.org/0000-0002-7920-2419
Mohammad Abed Sakarneh

Al-Balqa’a Applied University, Princess Rahma University College,
Department of Special education, Al-Salt, Jordan
https://orcid.org/0000-0001-8683-7941
Abstract. Motivation is important for harnessing abilities and

competencies, even if students are naturally or potentially gifted. This
study used design thinking pedagogy as an innovative approach to
motivate students holistically to learn and attend school. Design thinking
is an approach to acquire and apply knowledge in a real situation that
involves five stages: empathy, define, ideate, prototype, and test. The
study was a quantitative quasi-experimental study with a one-group
design and pre and post-tests. The study evaluated intrinsic and four
extrinsic motivation categories – integrated, identified, introjected, and
external regulation – to evaluate gifted students’ motivation to do
coursework, and used a motivation scale to gauge their willingness to
attend school, and an attitude questionnaire to determine students’
satisfaction with and engagement in the design thinking class. The
experimental group consisted of 77 randomly selected gifted students at
the King Abdullah II School for Excellence. After engaging in the design
thinking class, the gifted students’ scores on all motivation categories
were higher in the post-test than they had been in the pre-test. However,
gifted students’ motivation levels were not associated with their gender,
grades, and SAT scores. Thus, the design thinking approach is a
promising approach for educating gifted students; students found it
satisfying and they exhibited high levels of engagement behavior. The
study results recommend that a design thinking approach is worth
pursuing to increase gifted students’ motivation. The researchers
recommend considering both gifted and non-gifted students in future
studies involving the design thinking approach.
Keywords: design thinking approach; extrinsic motivation; gifted

students; motivation; intrinsic motivation
©Authors
92
1. Introduction
Achieving the aim of developing 21st century skills and capabilities, namely,
creative thinking, critical thinking, communication, and collaboration, requires
teachers to adopt a suitable pedagogy to meet students’ unique needs in the class
(Sakarneh & Al-Swelmyeen, 2020; Samat & Ismail, 2020; Stith et al., 2020). Gifted
students, specifically, encounter a lack of teaching support in the regular
classroom, which causes them to experience more motivational issues in relation
to learning and school than their non-gifted peers. These motivational issues have
commonly been suggested to be a cause of underachievement (Hornstra et al.,
2020). Samat and Ismail (2020) explain, furthermore, that gifted students require
specially designed programs that motivate them to participate in acquiring and
seeking knowledge development. The motivation of gifted students positively
impacts their giftedness development (Barabwd et al., 2017). Burns and Martin
(2021) and McCoach and Flake (2018) have the same perception about the role of
motivation in harnessing abilities and competencies, even for students who are
naturally or potentially gifted. They stated that motivation is the catalyst of
giftedness; an adequate motivation level is a necessity for developing giftedness
for both naturally and potentially gifted students.
A study in Jordan detected a clear difference between the intrinsic motivation

levels of gifted female students, their gifted male peers, and ungifted peers
(female and male). Gifted male students did not exhibit a higher level of
motivation than their ungifted peers of either gender. Despite gifted female and
male students being recruited from the same school for excellence, gifted female
students acquired substantially greater benefits than their male counterparts
(Heilat et al., 2019). This finding suggests that gifted male students gain fewer
benefits from education programs that do not focus on fostering their motivation
levels.
This study investigated the design thinking pedagogy, as an innovative approach

to motivate students about learning and to attend school, so that teachers can
provide gifted students with the expected levels of support, satisfaction, and
relatedness. The study intended to contribute to supporting classroom
organization and meeting gifted students’ needs. Studies have found that the
majority of gifted students fail to recognize their abilities (Hebert, 2020; Siegle &
McCoach, 2018). Implementing design thinking in the classroom may contribute
to supporting students’ recognition of their cognitive abilities and competencies.
Personal recognition of their abilities is likely to serve gifted students
professionally, socially, and emotionally (Hebert, 2020).
This study focused on harnessing the potential benefits and features of a design
thinking approach to enhance and upgrade gifted students’ motivation, since
motivation plays an important role in students’ progress and excellence. The
benefits of a design thinking approach can be assessed by measuring the
difference between students’ motivation levels before and after the
implementation of the approach. Furthermore, design thinking, as a new learning
experience, can introduce students to a new learning concept, and prepare them
for the post-high school stage of learning.
93
1.1 Study Objectives

The study investigated the application of a design thinking approach as a
motivational tool. Thus, the study aims were to determine
- Differences between pre and post-tests of gifted students’ motivation to
learn and attend school;
- Differences in students’ motivation that were attributable to their
demographic features (gender, grade, SAT score); and
- Whether students developed positive attitudes to learning by approaching
problems with design thinking.
2.1 Motivation
The word motivation is derived from the Latin word meveo, meaning move. A
simple dictionary definition of motivation is having a reason to engage,
accomplish, or do a particular thing. Motivation carries out a force to act or move
to action (McCoach & Flake, 2018). According to self-determination theory, this
means stimulating attitudes, values, and tendencies towards action. According to
this theory, there are two motivation categories: intrinsic and extrinsic motivation
(Hornstra et al., 2020). Intrinsic motivation is the ingrained, inherent, and innate
inclination to engage, seek or participate in challenges to feel satisfaction or
enjoyment (Heilat et al., 2019). Extrinsic motivation refers to external stimuli or
forces that induce a move to action or to taking action (Hornstra et al., 2020; Fleith,
2016).
External stimulation sources have various origins. The current study identifies
four types of extrinsic motivation, namely, integrated, identified, introjected, and
external regulation (Ryan & Deci, 2000). Integrated motivation refers to the
internalization of the stimuli to the self, through which the action becomes self-
determined. Identified motivation is the inclination to act by recognizing the
personal significance of an act and accepting the act as personal regulation.
Introjected motivation is a controlling incentive to act, such as acting due to a
feeling of obligation; in other words, it is about acting in order to avoid guilt or
anxiety (e.g., students study because their parents expect them to). External
regulation refers to the motivation or stimulus to act to satisfy external demands,
or to acquire external rewards through external causality. There is a further
motivation category that reflects the concept of lack of intention to act, and
personal causation (Ryan & Deci, 2000).
Motivation and giftedness are interrelated, and motivation is considered a
fundamental catalyst for developing giftedness (Burns & Martin, 2021; Hornstra
et al., 2020; Samat & Ismail, 2020; McCoach & Flake, 2018; Barabwd et al., 2017).
Thus, scholarly interest has turned to investigating the motivation level of gifted
students, and comparing it to that of non-gifted students, for example, the
Jordanian study of Heilat et al. (2019). Another research interest is the role of
motivation and its impact on gifted students’ performance and progression. For
example, Barabwd et al. (2017) surveyed gifted students' perceptions of the role
of motivation in developing giftedness and found that they had highly positive
perceptions in this regard, and believed it played a significant role in developing
giftedness. In contrast, research studies (e.g., Steenbergen-Hu et al., 2020;
Rubenstein et al., 2012) argue that lack of motivation can be a predictor of
94
underachievement by gifted students. Burns and Martin (2021) discuss recent

research into the motivation of gifted students and suggest practical strategies
teachers can apply to foster giftedness development of gifted students. They also
provide effective strategies to address motivational issues. This study supports
these suggestions and uses a design thinking approach to foster gifted students'
motivation.
2.2 Design Thinking

Design thinking relates to design theory, which is a philosophy that clarifies
concepts and criticizes current models and practices. However, this theory is
widely known in industrial settings, and not in the education field (Simeon et al.,
2020). In education, design thinking is a teaching approach or environment that
can be distinguished from conventional learning approaches for acquiring
knowledge, which goes beyond acquiring knowledge, to the implementation and
application of knowledge in the real situation. Design thinking is not only
applicable to a single discipline, but to multidisciplinary contexts that include
engineering, social sciences, medicine, education, and so forth (Tsai & Wang, 2020;
Wrigley et al., 2018; Henriksen et al., 2017). In education, design thinking is
considered to be a cognitive process that includes creation, experimentation,
feedback, and redesign of whatever the field of the subject comprises (Li et al.,
2019).
Design thinking is a holistic approach to acquiring and applying knowledge in
real situations. It involves five stages: empathy, define, ideate, prototype, and test.
Empathy involves learning more about the problem and people involved. Define
is about clarifying the problem and setting goals for solving it, as expected. Ideate
is about investigating, doing research, and getting ideas to achieve the goals.
Prototype is about selecting the best solution and creating prototypes that can be
justified. Lastly, in the test stage, the prototype is tested in the real situation and
evaluated with experiments (Stith et al., 2020).
Studies investigated the role of a design thinking approach for different aspects
or dimensions, such as increasing creative thinking and innovation (Henriksen et
al., 2017), self-efficacy (Tsai & Wang, 2020), and so forth. Simeon et al. (2020)
applied an afterschool-program-based design thinking approach in one discipline
(physics) and evaluated its potential to increase students’ achievement in
developing concepts. Statistical results detected an increase in the level of
students’ achievement, with a difference between genders in favor of male
students. Educators had positive perceptions of the utility of a design thinking
approach for educating gifted students. Teachers said that design thinking tasks
foster the development of 21st century skills, has psychological benefits, and
increases the motivation level of students (Stith et al., 2020). Teachers explained
that a design thinking approach increases motivation. Firstly, students exhibit
active engagement, find the approach exciting and fun, and feel that they are
making a contribution to society, because they have the opportunity to develop
solutions on their own. The second reason why design thinking increases
motivation, according to teachers, is that the task context is authentic and
transferable to other contexts. Thirdly, the problems students solve are from the
real word, which students experience, feel, see, and live. In the fourth place,
95
students are not constrained to particular ways of finding solutions; instead, they
can create their own solutions in their own ways (Stith et al., 2020).
A study of design thinking in a subject relating to the human–computer interface
assessed students’ motivations and found that doing design thinking tasks
increased students’ motivation to do coursework and attend school. Furthermore,
students showed higher levels of self-efficacy and lower test anxiety, and their
ability to control, manage, and regulate their effort and time improved. The study
linked this increase in motivation to the facilitation of the flow of knowledge in
the course (Ahmad et al., 2017). Similar results were obtained by a study that
targeted university students, in which more than 80% of students stated that they
had the self-motivation to pursue success in a design thinking course. Students
ascribed their perseverance to the significance of these skills for their self-
development, which caused their engagement to become self-determined (Wei et
al., 2020). A second study used design thinking methodology to enhance
university students’ motivation and performance in their graduate projects.
Results confirm the significance of a design thinking approach for fostering
motivation (Bordel et al., 2019).
In summary, studies found that learning with a design thinking program
increased self-motivation of students, and improved teachers’ perceptions about
design thinking (Wei et al., 2020; Ahmad et al., 2017). However, there is no
empirical evidence for this association. Thus, the current study intended to find
empirical evidence for this claim and, furthermore, determine students’ attitudes
toward a design thinking program.
3. Method
The study used a quasi-experimental approach with a one-group design and pre
and post-tests.
The study examined the effect of design thinking as a learning approach on
students’ motivation in a multidisciplinary context. The study investigated
afterschool activities, because the setting of the school’s conventional activities
was not conducive to setting design thinking problems. The researchers played a
teacher role in the study, as the usual teacher was unqualified to carry out a design
thinking approach. Furthermore, for students to engage in design thinking
problems, they need to have integrated and resource-rich settings. All students
could access various online laboratories and online research databases. The
students could also use a university chemistry library or computer facilities under
the researchers’ guidance.
3.1 Study Participants and Learning Environments

The study used online learning management software at the King Abdullah II
School for Excellence (Grade 11 to 12 (Tawjihi1)) in Jordan. Seventy-seven students
were selected randomly from the secondary grades – 45 female and 32 male
students. They provided signed informed consent, and parental approval was
obtained. Fifty-two students were in Grade 11, and 25 students were in Grade 12.
SAT scores were obtained for all students from their student profiles at the school.
The SAT is a foreign test that is taken for three main subjects: reading, writing,
1
The General Secondary Education Certificate Examination in Jordan and Palestine.
96
and mathematics. SAT scores were used, rather than grade point averages, to
avoid biasing the results due to school administration or other factors, and to
ensure homogeneity. The majority of students (N=72) obtained average SAT
scores in excess of 90%; only five students obtained average scores ranged
between 80.4% and 90%.
Design thinking problems were posed via online learning modules that the school
had adopted for learning purposes. The study opted to use the online learning
modules because, at the time the study was administered, the school used online
learning due to global health circumstances.
3.2 Procedure
All students engaged in three weekly multidisciplinary sessions via Microsoft
Teams (online platform). Each session was a one-hour class period that was
scheduled after the end of the school schedule. These sessions differed from
conventional school lessons and involved the authentic discipline of design
thinking about a problem. Most students selected tasks from a pool of design
thinking problems based on their preferences. Only the first five problems were
reported by the study (see Table 1).
Each task was discussed in five online sessions. The first session was an
introductory session, in which the problem was either stated by a teacher or
sparked by trigger questions. In the second session, students presented
information that they had collected during searches relating to the problem;
students presented their perceptions to their peers. In the third session, students
were encouraged to reveal the solutions they suggest for solving the problem, and
to explain how they planned to implement the solutions. The fourth session
involved the teacher encouraging students to report on their progress and discuss
any obstacles they faced, so that they could overcome the obstacles, either by
suggestions by group members, or by the teacher’s suggestions and guidance. In
the fifth session, each group assumed complete responsibility for presenting their
solution and a report to their classmates. In conclusion, the solutions were driven
and offered by participants and approved by the teacher.
Each problem was solved by students in a group of three gifted students. New
groups were formed between design thinking problems. This setting is consistent
with a student-centered classroom and student-driven environment.
Table 1. Tasks list of online design thinking problems developed during the study
Online design
Description Related content
thinking problem
High rate of A graphic presentation of the water Chemistry, social
consumption of consumption rate by each activity of sciences, material
water for daily life, and a classification of grey research, life sciences
irrigation and black water. Identifying the best
purposes water sample to reuse for irrigation
according to Jordanian standards for
irrigation water. Build a simple filter to
reuse grey water sample for irrigation.
Waste Feasibility study of recent solutions for Administration
management waste management mechanisms, sciences, engineering,
97
enabling features that facilitate life sciences, computer

management, and a survey of and information
residents’ perceptions. sciences, social
sciences
High rate of Suggest application features or Computer and
dependency on interface to reduce time spent on social information sciences,
social media media. social sciences,
psychology
Lowering the Study solutions and provide reports Life sciences, social
unemployment related to other countries that sciences, psychology,
rate succeeded in lowering its financial sciences,
unemployment rate; discuss probable administrative
and unlikely solutions. sciences
Developing Design a presentation suggesting ways Life sciences, social
awareness of and to educate communities about bullying sciences, psychology
reducing bullying and suggest ways to cope with
bullying.
3.3 Instrument
A student motivation questionnaire, in English, was used for pre and post-
assessment. It comprises scales measuring academic motivation (four items) and
motivation for school (18 items distributed over four dimensions). All items of
both scales are answered on a five-point Likert scale ranging from “totally not
applicable to me” to “totally applicable to me”. The academic motivation scale
consists of four items developed by Vallerand et al. (1992). The motivation for
school scale consists of four facets: external regulation, introjected, intrinsic, and
identification motivation, and was developed by Rayan and Connell (1989). Both
scales have high psychometric properties. The academic motivation scale and
motivation for school scale scored above the recommended reliable score
(Cronbach’s alpha >0.70) (Hornstra et al., 2020; Utvaer & Haugan, 2016). The
psychometric properties of the scale used in the current study also have acceptable
validity, according to the Pearson correlation coefficient recorded for the scale, in
which all items of the motivation scale correlate with their dimensions, and all
statements’ correlation values were in excess of 0.31, which is the lowest
acceptable value (Core et al., 2021).
The study used SPSS software to analyze participants’ responses, and embedded
descriptive and inferential statistics to validate study objectives.
Table 2. Pearson correlation test to show construct validity

Amotivation External Introjection Identification Intrinsic
# Correlation # Correlation # Correlation # Correlation # Correlation
1 .802** 1 .379** 1 .470** 1 .353** 1 .854**
2 .860** 2 .565** 2 .539** 2 .528** 2 .770**
3 .824** 3 .400** 3 .326** 3 .583**
4 .863** 4 .494** 4 .373** 4 .409**
5 .323** 5 .225* 5 .509**
6 .355**
98
Furthermore, the reliability of the scale is also acceptable, as it recorded a

Cronbach’s alpha higher than the threshold value (>0.70). See Table 3.
Table 3. Cronbach Alpha Coefficient for motivation scale

Dimension Cronbach’s alpha
Amotivation 0.856
External 0.776
Introjection 0.812
Identification 0.760
Intrinsic 0.704
Motivation (Total) 0.873
The second measure the study used is the attitude scale, which was implemented
after the completion of the design thinking classes, to assess students’ attitudes
about the design thinking problem approach. The scale has two dimensions,
namely, satisfaction with program (four items, responses evaluated using five-
point Likert scale, from “strongly disagree” to “strongly agree”) and behavioral
engagement in the session (five items with a five-point Likert scale ranging from
“never” to “always”) developed by Nie and Lau (2009). The scale achieved a good
reliability score, since Cronbach’s alpha was higher than the accepted threshold
(>0.70). See Table 4.
Table 4. Cronbach’s alpha coefficient for attitude scale

Dimension Cronbach alpha
Satisfaction 0.775
Behavior 0.741
Attitude (Total) 0.830
3.4 Normality and Homogeneity Test

Before starting to test the hypotheses, the researcher made sure that the data
followed the normal distribution, by applying a set of special tests. See Table 5.
Table 5. Normal distribution and homogeneity tests for the study data
Kolmogorov-Smirnov
Kurtosis Skewness
Variables Value Sig. Tolerance VIF
Amotivation 0.957 0.562 0.564 -0.541 0.685 2.001
External 1.567 0.264 0.318 -0.214 0.432 2.684
Introjection 0.874 0.587` 0.584 -0.454 0.384 2.854
Identification 0.698 0.541 0.461 -0.605 0.491 1.540
Intrinsic 1.002 0.121 0.657 -0.241 0.426 2.366
Satisfaction 1.068 0.103 0.746 -0.326 0.654 1.451
Behavior 0.888 0.354 0.555 -0.567 0.441 2.214
Note: Sig. = 0.05; VIF = 10
It is clear from Table 5 that the statistical significance values on the Kolmogorov-
Smirnov test are in excess of α ≤ 0.05, and that all the values for skewness are
below -1. Kurtosis is less than 7, the tolerance values are greater than 0.05, while
the VIF values are below 10. These tests confirm that all values approximate a
normal distribution and, thus, allow the use of parametric methods.
99
4. Results
4.1 Group Heterogeneity
To validate the heterogeneity of the participants of the study, inferential statistics,
including a T-test, was used, and heterogeneity was validated. According to Table
6, an independent sample T-test for gender, grade, and SAT score found that
motivation and related facets (amotivation, external, introjection, identification,
intrinsic) were not significant at the pre-test; t-values for gender were 1.448, 0.502,
-1.110, -0.002, 0.242, 0.845, for grade were 0.436, 0.049, -1.449, 1.036, 0.768, 0.534,
and for SAT score were 0.376, -1.054, -0.221, -1.095, -0.861, -0.860, respectively).
None of these values are significant at the 0.05 level, which indicates the
equivalence of all members of the study sample at the time of the pre-test.
Table 6. Independent sample T-test to show the variance between motivation and its
dimensions according to gender, grade, and SAT score at the pre-test
Motivation facets Gender N Mean Std. deviation (t) P value
Male 32 2.56 0.95 1.448 .152
Amotivation
Female 45 2.27 0.80
Male 32 3.61 0.32 .502 .617
External
Female 45 3.56 0.44
Male 32 2.74 0.36 -1.110- .271
Introjection
Female 45 2.84 0.36
Male 32 3.17 0.28 -.002- .999
Identification
Female 45 3.17 0.35
Male 32 2.11 0.64 .242 .809
Intrinsic
Female 45 2.08 0.50
Motivation Male 32 2.95 0.21
.845 .401
(total) Female 45 2.91 0.21
Grade N Mean Std. deviation (t) P value
11 52 2.42 0.88 .436 .664
Amotivation
12 25 2.33 0.88
11 52 3.58 0.41 .049 .961
External
12 25 3.58 0.38
11 52 2.76 0.38 -1.499- .138
Introjection
12 25 2.89 0.31
11 52 3.12 0.32 1.036 .245
Identification
12 25 3.16 0.31
11 52 2.13 0.55 .768 .445
Intrinsic
12 25 2.02 0.59
Motivation 11 52 2.93 0.21
.534 .595
(total) 12 25 2.90 0.22
SAT
N Mean Std. deviation (t) P value
score
> 90 72 2.40 0.86 .376 .708
Amotivation
> 80.2 5 2.25 1.15
> 90 72 3.57 0.40 -1.054- .295
External
> 80.2 5 3.76 0.30
> 90 72 2.80 0.36 -.221- .826
Introjection
> 80.2 5 2.83 0.39
> 90 72 3.16 0.32 -1.095- .277
Identification
> 80.2 5 3.32 0.33
100
> 90 72 2.08 0.55 -.861- .392

Intrinsic
> 80.2 5 2.30 0.67
Motivation > 90 72 2.92 0.21
-.860- .393
(total) > 80.2 5 3.00 0.16
4.2 Descriptive Statistics

According to the descriptive analysis results, the students had a higher average
mean (M=4.04; very high motivation level) on the motivation scale in the post-test
than in the pre-test (M=2.92; medium motivation level). In the pre-test, students’
scores indicated a medium level of motivation: external regulation and
identification motivation levels were high and other motivation categories were
medium. The statistical data of the post-test measurement show a tangible
difference from the pre-test results for almost all the motivation categories.
Table 7. Mean and standard deviation for (pre and post) motivation and its dimensions
Pre-test Post-test
Dimensions Std. Std.
Mean Mean
Deviation Deviation
Honestly, I don’t know; I really feel that I am wasting
2.34 1.07 4.03 0.73
my time in school
I once had good reasons for going to school;
2.49 1.13 4.03 0.69
however, now I wonder whether I should continue
I can’t see why I go to school and frankly, I couldn’t
2.51 0.93 3.83 0.64
care less
I don’t know; I can’t understand what I am doing in
2.23 1.04 4.10 0.66
school
Amotivation 2.39 0.87 4.00 0.38
Because I'll get in trouble if I don't 3.66 1.02 4.25 0.65
Because that's what I'm supposed to do 3.69 0.92 4.05 0.76
So that the teacher won't yell at me 3.40 0.63 3.75 0.67
Because that's the rule 3.45 1.08 4.13 0.66
So others won't get mad at me 3.69 0.89 3.79 1.06
External regulation 3.58 0.40 3.99 0.42
Because I want the teacher to think I'm a good
2.70 1.06 4.29 0.48
student
Because I will feel bad about myself if I don't 2.70 1.08 4.27 0.62
Because I'll feel ashamed of myself if I don't 3.23 0.71 4.38 0.49
Because I want the other students to think I'm smart 2.74 1.04 4.13 0.78
Because it bothers me when I don't 2.01 0.92 3.68 0.66
Because I want people to like me 3.40 0.71 4.53 0.50
Introjection 2.80 0.36 4.22 0.25
Because I want to understand the subject 3.55 0.64 4.62 0.49
Because 1 want to learn new things 3.53 0.66 4.62 0.49
To find out if I'm right or wrong 3.45 0.70 4.57 0.50
Because I think it is important to work on my
3.31 0.80 4.52 0.50
schoolwork
Because I wouldn't want (like) to do that (negative
2.00 0.74 1.42 0.64
behavior)
Identification 3.17 0.32 3.95 0.26
I1: Because it's fun 2.14 0.76 3.90 0.77
I2: Because I enjoy it 2.04 0.62 3.95 0.67
101
Intrinsic 2.09 0.56 3.92 0.60

Motivation (total) 2.92 0.21 4.04 0.16
4.3 Difference in Motivation Between Pre and Post-tests

The study examined if there was a difference between the motivation, measured
by pre and post-test scores, of gifted students. The descriptive data of the pre and
post-tests indicate a clear difference. Foremost, statistically significant differences
have to be tested in order to validate differences statistically. Therefore, a paired
sample T-test was applied to determine the statistically significant differences
between the pre and post-test on motivation (see Table 8).
Table 8. Difference between (pre and post) motivation levels (paired sample T-test) (N=77)
Std. Paired
Mean (t) P value
deviation differences
Amotivation pre 2.39 0.87
-1.60 -13.374- 0.00*
Amotivation post 4.00 0.38
External pre 3.58 0.40
-0.42 -5.845- 0.00*
External post 3.99 0.42
Introjection pre 2.80 0.36
-1.42 -29.693- 0.00*
Introjection post 4.22 0.25
Identification pre 3.17 0.32
-0.78 -28.996- 0.00*
Identification post 3.95 0.26
Intrinsic pre 2.09 0.56
-1.83 -19.009- 0.00*
Intrinsic post 3.92 0.60
Motivation pre total 2.92 0.21
-1.12 -38.000- 0.00*
Motivation post total 4.04 0.16
The results of the paired sample T-test in Table 8 indicate that there are statistically
significant differences in the results of the pre and post-tests for motivation and
its related dimensions (amotivation, external, introjection, identification,
intrinsic). The paired differences for these dimensions and total degree were -1.60,
-0.42, -1.42, -0.78, -1.73, -1.12 respectively, and, with the significance level less than
(>0.01), the differences were in favor of the post-test, which illustrates the high
level of the mean values in that measurement.
4.4 Attributing Differences in Motivation to Demographic Differences

The study assessed whether gender, grade, or SAT score had an effect on the
motivation level of students after they had attended a design thinking class. The
study applied an independent sample T-test; the results are given in Table 9.
Table 9. Independent sample T-test to determine variance between motivation and its
dimensions according to gender, grade, and SAT score at the post-test
Std.
Gender N Mean (t) P value
deviation
Male 32 3.97 0.37
Amotivation -.547- .586
Female 45 4.02 0.39
Male 32 3.99 0.39
External -.128- .898
Female 45 4.00 0.45
Male 32 4.25 0.26
Introjection .841 .403
Female 45 4.20 0.25
102
Male 32 3.94 0.26

Identification -.378- .706
Female 45 3.96 0.26
Male 32 3.98 0.55
Intrinsic .771 .443
Female 45 3.88 0.63
Motivation Male 32 4.04 0.17
.101 .920
(total) Female 45 4.04 0.16
Std.
Grade N Mean (t) P value
deviation
11 52 3.97 0.41 -.857- .394
Amotivation
12 25 4.05 0.30
11 52 3.98 0.43 -.421- .675
External
12 25 4.02 0.40
11 52 4.21 0.23 -.288- .774
Introjection
12 25 4.23 0.31
11 52 3.96 0.25 .346 .730
Identification
12 25 3.94 0.28
11 52 3.92 0.60 .021 .983
Intrinsic
12 25 3.92 0.61
Motivation 11 52 4.03 0.17
-.512- .610
(total) 12 25 4.05 0.16
Std.
Sat score N Mean (t) P value
deviation
> 90 72 3.98 0.38
Amotivation -1.567- .121
> 80.2 5 4.25 0.25
> 90 72 4.01 0.43
External .852 .397
> 80.2 5 3.84 0.26
> 90 72 4.21 0.25
Introjection 1.303 .243
> 80.2 5 4.37 0.32
> 90 72 3.96 0.26
Identification -1.068- .341
> 80.2 5 3.88 0.18
> 90 72 3.94 0.60
Intrinsic .860 .393
> 80.2 5 3.70 0.57
Motivation > 90 72 4.04 0.17
-.220- .826
(total) > 80.2 5 4.05 0.04
The result of the independent sample T-test for the association of motivation with
gender, grade, and SAT score indicates that motivation and its dimensions
(amotivation, external, introjection, identification and intrinsic with total score)
were not significant at post-test (t) values: -0.547, -0.128, 0.841, -0.378, 0.771, 0.101,
-0.857, -0.421, -0.288, 0.346, 0.021, -0.512, -1.567, 0.852, 1.303, -1.068, 0.860, -0.220
respectively. None of these values are significant at the level of 0.05, which
indicates that there were no statistically significant differences in motivation in
relation to gender, grade and SAT score.
4.5 Gifted Students’ Attitudes to the Design Thinking Problem Class

To determine gifted students’ perceptions of and attitudes toward the design
thinking approach, all participants completed an attitude questionnaire to rate
their satisfaction and engagement behavior. The descriptive statistics of their
responses is presented in Table 10.
103
Table 10. Attitudes of gifted students towards design thinking (N=77)

Attitudes Mean Std. deviation
I am glad to be in this program 4.26 0.73
I think it is nice to participate and study in this program 3.94 0.73
If I could, I would rather back to conventional study
2.49 0.60
settings
If I had to move to another settings, I would still want
4.43 0.57
to study through such setting
Satisfaction 3.78 0.32
I pay attention well 4.30 0.74
I keep my attention on the work during the entire
4.42 0.68
lesson
I listen carefully when the teacher explains something 4.27 0.74
I try my best to complete class work 4.44 0.68
I try my best to answer the teacher’s questions 4.42 0.69
Engagement behavior 4.37 0.27
Attitude (total) 4.11 0.21
The descriptive statistics indicate that students’ satisfaction was high, with a total
mean of 3.78 and standard deviation of 0.57. The statement with the highest score
(If I had to move to another settings, I would still want to study through such setting) has
a mean of 4.43 and standard deviation of 0.57, which is high. The statement which
stipulated (If I could, I would rather back to conventional study settings.
The result for engagement behavior is at a high level, with a mean of 4.37 and a
standard deviation of 0.21. The statement, I try my best to complete class work,
ranked first, with a mean of 4.44 and a standard deviation of 0.6; it is at a high
level. In contrast, the statement, I listen carefully when the teacher explains something,
ranked last, with a mean of 4.27 and a standard deviation of 0.74 – which is still at
a high level. Total attitude obtained a mean of 4.11, with a standard deviation of
0.21, which is at a high level.
5. Discussion
Descriptive statistics indicate that gifted students attend school because they are
forced to do so by external regulations: they wish to avoid getting into trouble, or
to avoid punishment, and they obey school regulations. Other factors that
stimulated students to attend school were self-identified stimuli, such as the
desire to acquire knowledge, and recognizing that attending would help them
meet their needs in the future Not surprisingly, gifted students who participated
in the study exhibited a medium level of association between academic
motivation and their level of external regulation motivation. This indicates that, if
students are not obligated to attend school, they will not do so. Their
unwillingness may be attributed to the absence of fun and enjoyable activities
(thus, an absence of high intrinsic motivation). This finding confirms the study by
Heilat et al. (2019), which found that gifted students in Jordan have the same level
of motivation as non-gifted students, which hinders gifted students’ giftedness
development.
The statistical inferential tests validated the significant differences found for all
motivation categories levels, in favor of the post-test measurement. It is clear that
104
the online design thinking problems played a significant role in fostering the
motivation levels of gifted students. This confirms the research results of Stith et
al. (2020), Wei et al. (2020), Bordel et al. (2019) and Ahmad et al. (2017). Gifted
students found the design thinking class fun and enjoyable. They were stimulated
by attending the class and the learning they underwent, and their participation
was self-determined (see Table 5), probably because the design thinking class
gave students opportunities to develop solutions on their own (Stith, et al., 2020).
These opportunities were turned into self-determined participation and fostered
a feeling of belonging in relation to the subject, since students dealt with real-life
situations – some students lived with these problems (Wei et al., 2020).
Furthermore, introjected motivation increased through the design thinking class
– students became more interested in external appreciation and pursuing a good
image in the eyes of their peers or teachers (see Table 5). This finding resonates in
their high satisfaction with the design thinking approach, and their engagement
behavior. Their responses indicate that they shared an interest in and were
committed to solving the design thinking problems they discussed in class (see
Table 10).
Moreover, the design thinking problems presented to students in this study were
appropriate for all gifted students (female and male, Grades 11 and 12, and any
SAT score band). No differences were found between motivation levels and
gender, class, or SAT score categories of gifted students. This was expected, since
the participants chose their preferred problem from a pool of suggested real-life
problems.
6. Conclusion
Motivation plays a significant role in harnessing students’ abilities and
competencies, even if students are naturally or potentially gifted. The study used
design thinking pedagogy as an innovative approach to motivating students to
learn and attend school. Design thinking is a holistic approach to acquiring
knowledge and applying this knowledge in real situations through five stages:
empathy, define, ideate, prototype, and test. The study was a quantitative quasi-
experimental study with a one-group design and pre and post-tests. The study
evaluated intrinsic and four extrinsic motivation categories, namely integrated,
identified, introjected, and external regulation, to evaluate gifted students’
motivation to do coursework. A motivation scale was used to gauge their
motivation to attend school, and an attitude questionnaire explored students’
satisfaction and engagement with the design thinking class. The experimental
group consisted of 77 randomly selected gifted students at the King Abdullah II
School for Excellence. After engaging in the design thinking class, the students
achieved higher scores than in the pre-test for all motivation categories.
Furthermore, the students did not exhibit any differences in their post-test
motivation levels in relation to gender, grade, and SAT scores. Thus, the design
thinking approach is a promising approach for motivating gifted students; the
students found it satisfying and high engagement behavior was recorded.
The study design and results justify pursuing implementing a design thinking
approach to increase gifted students’ motivation for different subjects and at
different grade levels. Furthermore, the study did not attempt to find differences
105
between gifted and non-gifted students, since gifted students who were enrolled
in a school for excellence were engaged in various authentic learning
environments, which hindered attempts to distinguish the effect of design
thinking on motivation between two groups, as the two groups were engaged in
different activities. Thus, the study was limited to one group pre and post-test,
and future studies are encouraged to involve two groups, to validate the current
results. The researchers recommend involving both gifted and non-gifted
students in future studies that use the design thinking approach, which would
enable comparisons between the two groups of students. Furthermore,
longitudinal studies are recommended, to track the giftedness development of
gifted students using a design thinking approach to learning.
7. References
Ahmad, W. F., Sarlan, A., Rokhmah, S., & Ahmad, R. (2017). Integrating design thinking
approach to increase student’s motivation. 7th World Engineering Education Forum
(WEEF) (pp. 268-272). IEEE. https://doi.org/10.1109/weef.2017.8467148
Barabwd, H., Nor, M., & Ishak, N. (2017). Gifted students' motivation and its impact on
the development of their giftedness. Journal of Education in Black Sea Region, 3(1).
https://doi.org/10.31578/jebs.v3i1.118
Bordel, B., Alcarria, R., Manso, M., & Robles, T. (2019). Enhancing students' motivation and
academic results in the context of final projects through design-thinking methodologies.
23th International Technology, Education and Development Conference.
INTED2019 Proceedings (pp. 8235–8239). IATED.
Burns, E., & Martin, A. (2021). Motivational issues in gifted education: understanding the
role of students’ attribution and control beliefs, self-worth protection and growth
orientation. In S. R. Smith (Ed.), Handbook of giftedness and talent development in the
Asia-Pacific (pp. 339–357). Springer. https://doi.org/10.1007/978-981-13-3041-
4_15
Core, R., Waring, M., Hedges, L., & Ashley, L. (2021). Research methods and methodologies in
education (3rd ed.). SAGA.
Fleith, D. de S. (2016). Creativity, motivation to learn, family environment, and giftedness:
A comparative study. Psicologia: Teoria e Pesquisa, 32, 1–9.
https://doi.org/10.1590/0102-3772e32ne211
Hebert, T. (2020). Understanding the social and emotional lives of gifted students. Sourcebooks.
Heilat, M. Q., Seifert, T., & Qian, M. (2019). Mental motivation, intrinsic motivation and
their relationship with emotional support sources among gifted and non-gifted
Jordanian adolescents. Cogent Psychology, 6(1), 1–13.
https://doi.org/10.1080/23311908.2019.1587131
Henriksen, D., Richardson, C., & Mehta, R. (2017). Design thinking: A creative approach
to educational problems of practice. Thinking Skills and Creativity, 26, 140–153.
https://doi.org/10.1016/j.tsc.2017.10.001
Hornstra, L., Bakx, A., Mathijssen, S., & Denissene, J. A. (2020). Motivating gifted and non-
gifted students in regular primary schools: A self-determination perspective.
Learning and Individual Differences, 80, 1–13.
https://doi.org/10.1016/j.lindif.2020.101871
Li, Y., Schoenfeld, A. H., diSessa, A. A., Graesser, A. C., Benson, L. C., English, L. D., &
Duschl, R. A. (2019). Design and design thinking in STEM education. Journal for
STEM Education Research, 2, 93–104. https://doi.org/10.1007/s41979-019-00020-z
106
McCoach, D. B., & Flake, J. K. (2018). The role of motivation. In S. I. Pfeiffer, E. Shaunessy-
Dedrick, & M. Foley-Nicpon (Eds.), APA handbook of giftedness and talent (pp. 201-
213). American Psychological Association. https://doi.org/10.1037/0000038-013
Nie, Y., & Lau, S. (2009). Complementary roles of care and behavioral control in classroom
management: The self-determination theory perspective. Contemporary
Educational Psychology, 34(3), 185–194.
https://doi.org/10.1016/j.cedpsych.2009.03.001
Rayan, R. M., & Connell, J. P. (1989). Percevied locus of causality and internalization:
Examining reasons for acting in two domains. Journal of Personality and Scial
Psychology, 57(5), 749-761. http://doi.org/10.1037/0022-3514.57.5.749
Rubenstein, L., Siegle, D., Reis, S., Mccoach, D. B., & Burton, M. (2012). A Complex quest:
The development and research of underachievement interventions for gifted
students. Psychology in the Schools, 49(7), 678-694.
https://doi.org/10.1002/pits.21620
Ryan, R., & Deci, E. (2000). Intrinsic and extrinsic motivations: Classic definitions and new
directions. Contemporary Educational Psychology, 25(1), 54–67.
https://doi.org/10.1006/ceps.1999.1020
Sakarneh, M. A., & Al-Swelmyeen, M. B. (2020). The extent to which the Jordanian
inclusive basic school teachers use the constructivism theory in teaching. Journal
of Educational and Social Research, 10(1), 182–197. https://doi.org/10.36941/jesr-
2020-0017
Samat, F., & Ismail, E. S. (2020). Meeting the needs of Malaysian mathematically talented
students by an enrichment program: A module for primary school students.
Madrosatuna: Journal of IslamicElementary School, 4(1), 23–25.
https://doi.org/10.21070/madrosatuna.v4i1.55
Siegle, D., & McCoach, B. (2018). Underachievement and the gifted child. In S. Pfeiffer, E.
Shaunessy-Dedrick, & M. Foley-Nicpon, APA handbook of giftedness and talent (pp.
445–473). American Psychological Association.
Simeon, M. I., Samsudin, M. A., & Yakob, N. (2020). Effect of design thinking approach on
students’ achievement in some selected physics concepts in the context of STEM
learning. International Journal of Technology and Design Education.
https://doi.org/10.1007/s10798-020-09601-1
Steenbergen-Hu, S., Olszewski-Kubilius, P., & Calvert, E. (2020). The effectiveness of
current interventions to reverse the underachievement of gifted students:
Findings of a meta-analysis and systematic review. Gifted Child Quarterly, 64(2),
132–165. https://doi.org/10.1177%2F0016986220908601
Stith, K. M., Potts, M. L., Rubenstein, L. D., Shively, K. L., & Spoon, R. (2020). Perceptions
of K-12 teachers on the cognitive, affective, and conative functionalities of gifted
students engaged in design thinking. Journal of STEM Teacher Education, 55(1), 41–
57. https://doi.org/10.30707/J
Tsai, M.-J., & Wang, C.-Y. (2020). Assessing young students’ design thinking disposition
and its relationship with computer programming self-efficacy. Journal of
Educational Computing Research, 59(3), 410-428.
https://doi.org/10.1177/0735633120967326
Utvaer, B. K., & Haugan, G. (2016). The academic motivation scale: Dimensionality,
reliability, and construct validity among vocational students. Nordic Journal of
Vocational Education and Training, 6(2), 17-45. https://doi.org/10.3384/njvet.2242-
458x.166217
Vallerand, R., Pelletier, L., Blais, M., Briere, N., Senecal, C., & Vallieres, E. (1992). The
academic motivation scale: A measure of intrinsic, extrinsic, and amotivation in
107
education. Educational and Psychological Measurement, 52, 1003–1017.

https://doi.org/10.1177/0013164492052004025
Wei, L. C., Shamsuddin, W. N., Rozee, I. M., & Vesudevan, M. (2020). Degree students'
self-motivation toward design thinking project. International Journal of Education
and Pedagogy, 2(4), 185-192.
http://myjms.mohe.gov.my/index.php/ijeap/article/view/11606
Wrigley, C., Mosely, G., & Tomitsch, M. (2018). Design thinking education: A comparison
of massive open online courses. She Ji: The Journal of Design, Economics, and
Innovation, 4(3), 275–292. https://doi.org/10.1016/j.sheji.2018.06.002
108

Vol. 20, No. 8, pp. 108-125, August 2021
Teaching-Learning Strategies to Production

Planning and Control Concepts: Application of
Scenarios to Sequencing Production with Virtual
Reality Support
Fernando Elemar Vicente dos Anjos

Instituto Federal de Educação, Ciência e Tecnologia do Rio Grande do Sul
(IFRS), Campus Caxias do Sul. Caxias do Sul, Brazil; and
UNISINOS –University of Vale do Rio dos Sinos
https://orcid.org/0000-0002-9096-2806
Luiz Alberto Oliveira Rocha

UNISINOS – University of Vale do Rio dos Sinos, São Leopoldo, Brazil
https://orcid.org/0000-0003-2409-3152
Rodrigo Pacheco
Instituto Federal de Educação, Ciência e Tecnologia do Rio Grande do Sul
(IFRS), Campus Caxias do Sul, Caxias do Sul, Brazil
https://orcid.org/0000-0002-9399-4475
Débora Oliveira da Silva

UNISINOS – University of Vale do Rio dos Sinos, São Leopoldo, Brazil
https://orcid.org/0000-0001-7023-4927
Abstract. This paper aims to present scenarios to be applied in higher

education to the theme of production planning and control, addressing
factors of the production system and indicators arising from this process
and the application of virtual reality to support the process. The applied
method combines the development of six scenarios for virtual reality
application and the discussion about the impacts in indicators from the
production planning and control, for example, inventory in the process,
manufacturing lead-time, use of equipment, and punctual delivery
attendance. Findings revealed that the teaching-learning process of
production planning and control, when applied through scenarios,
generates opportunities for students to learn the impact in the
indicators. The virtual reality in this environment supports creating
differentiated teaching-learning environments to generate the most
significant knowledge for students which positively impacts the future
in the world of work. In addition, it allows people involved in the
teaching-learning processes of production engineering to apply the
©Authors
109
concepts presented in the sequencing process, lean about the impacts of

decisions on production sequencing indicators and appreciate the
support of virtual reality to generate an environment more cognitive for
students.
Keywords: teaching-learning in production engineering; engineering

dducation; planning and production control; virtual reality in teaching-
learning process; production sequencing
1. Introduction
When teaching-learning strategies are approached, several paths can be traced.
Mizukami (1992) presented five teaching approaches: traditional, behavioural,
humanistic, cognitive, and socio-cultural. Santos (2006) highlighted that
cognitivism proposes that knowledge is generated through experiments in the
world, analyzing aspects through intervention in the processes, rescuing the
discussion on theoretical studies of cognitive psychology presented by Piaget
and Vigotsky. They explained the theory of learning called constructivism.
In 2019, the ASEE Annual Conference & Exposition (promoted by the American
Society for Engineering Education) took place, in which a conference proceeding
was published that presented a discussion of cognitive teaching approaches to
Engineering. In this discussion, Crawley et al. (2019) described that the new MIT
program, called New Engineering Education Transformation (NEET), brings
information that alumni are better prepared to work as innovators, creators,
entrepreneurs, and future leaders, when knowledge is developed through
cognitive approaches, for the formation of critical, systemic and humanistic
thinking. They also supported professionals to learn and think more effectively
on their initiative throughout their lives.
The creation of environments that seek to prioritize cognition can be organized

and explored in different ways. Prensky (2001) highlighted the importance of
applying technologies in the teaching-learning process, and Wang et al. (2017)
pointed out in their research that the application of technologies can positively
impact students' learning. Some researches present the application of mobile
technology, virtual reality, and augmented reality as part of the teaching-
learning strategies in cognitive environments. Examples of such uses are the
investigation of crime scenes (Mayne & Green, 2020), the teaching of chemistry
(Frevert & Di Fuccia, 2019), functional spectroscopy (Lamb et al., 2018), and
medical surgery (Żechowicz et al., 2018), among others. Positive cases of the
application of virtual reality and augmented reality in the learning and
satisfaction of students can be perceived in research in several areas of
engineering, such as, , data structuring (Akbulut et al., 2018), machining
manufacturing processes, and robotics (Grodotzki et al., 2018), along with
construction of buildings and construction environments and equipment
(Sánchez et al., 2015; Shirazi & Behzadan, 2015).
Anjos et al. (2020) described in their research that there are some applications of
virtual reality in the teaching-learning processes of production engineering, and
110
there are many other opportunities for the application of virtual reality that
measure the results in the teaching-learning process or also measure the
satisfaction of students when using such a method. Thus, they indicated future
research in production systems management, production planning, control,
material handling, production simulation, production process management,
metrological quality organization, work organization, accident risk analysis and
prevention, work safety, process, and product ergonomics.
Through the perception of the importance of teaching-learning environments

with cognitive bias in the training of students and impacts on their professional
career and the application of technologies in the formation of the cognitive
environment and the opportunities for new research with the application of
virtual reality highlighted by Anjos et al. (2020), the following research questions
arise:
1- What concepts of production planning and control could be developed in
virtual reality?
2- Which theoretical bases applied to the virtual reality model should be taken
into account?
3- What are the model's requirements, and why would it strengthen the studied
concepts?
The general objective of this study is to develop a virtual reality application

model with different scenarios, to be used as a teaching-learning strategy for
production planning and control concepts.
This research is justified because applying virtual reality models, in some cases,
increases the level of knowledge retention of the subjects studied, such as from
25% to 80% after three weeks of studies carried out with virtual reality, when
compared to students who used only the traditional teaching method (Laseinde
et al., 2016). Students learn more using virtual reality than students who only
took classes with the traditional teaching method (Inayat et al., 2016) and were
more satisfied and engaged with the subjects studied (Fonseca et al., 2016). The
more increase in knowledge retention, the highest level of learning, satisfaction,
and engagement positively impact organizations because professionals entering
the job market arrive entirely with more acquired knowledge, supporting
organizations and making them more competitive in the market than its
competitors (Anjos et al., 2020).This article is organized in the following sections:
the theoretical framework, methods, results, analysis of results, and conclusion,
along with some limitations and future research suggestions.
2. Theoretical Background
2.1. Cognitive Environments
According to Lefrançois (2016), expectations that drive behaviour are formed by
cognitions, and are developed after experiences with styles and rewards. This
strategy is a way for teachers to stimulate students through their expectations
and learning objectives formed in a cognitive environment. It is also highlighted
that an essential part of the cognitive environment is related to the perception
(generated by significant experiences) of the formation of concepts, memories,
111
languages, thinking, problem-solving, and decision making. In the same line of

thought, Kanakana-Katumba and Maladzhi (2019) described that the cognitive
teaching-learning environment must take into account some characteristics, such
as (I) interrogative approach, (II) experimental learning, (III) problem-based
learning, (IV) case-based learning, (V) project-based learning, (VI) research-
based learning, and (VII) competency-based learning.
According to Mestrinho and Cavadas (2018), the interrogative approach is

innovative in higher education environments and becomes a good practice for
introducing the collaborative approach and bringing energy to the teaching-
learning environment. Bates (2015) claimed that the experimental approach is
widely used in engineering, allowing students to practice the concepts and
theories developed. The problem-based methodology has the characteristic of
organizing students into groups to deal with previously defined problems as
well as presenting solutions through studied concepts (Bates, 2015; Tsai et al.,
2015).
Case-based learning uses extensive discussion in the teaching-learning processes

by discussing cases and examples of possible solutions (Bates, 2015). Project-
based learning is similar to case-based learning. However, according to Bates
(2015) and Mestrinho and Cavadas (2018), the scope of the work is broader and
presents real challenges and tasks that often generate manual work and not
theorists only. Research-based learning is similar to project-based learning, yet
the instructors control the situation in the learning environment (Hwang et al.,
2015; Soudien, 2010). Finally, competency learning allows students depending
on their level of knowledge to learn more quickly. This approach allows one to
demonstrate competence and permits students to control their learning (Bates,
2015; Kreamer et al., 2015). Quadir et al. (2019) highlighted a positive existence
of cognitive learning environments if interactive activities with feedback and
multimedia components are used compared to subjective and objective learning
of students.
2.2. Virtual Reality and Cognitive Environments

Virtual reality can create artificial environments that can be used for different
purposes. According to Lamb et al. (2018), virtual reality environments happen
through the interaction between man and machine, as there is a simulation of a
real environment that can create interaction and communication between them.
Martins and Guimarães (2012) described that these environments use
multisensory technology that uses elements of computer graphics to create
virtual environments. They also highlighted that these environments have
characteristics related to immersion, interaction, and involvement, which work
according to the relationship between the user and the virtual environment.
Such characteristics are shown in Table 1.
112
Table 1. Basic characteristics related to the virtual world

Immersion No
immersionwhenthecomputationallydevelopedenvironmentisviewedthr
ough a desktop screenorprojectedvisualization.
Withimmersion, whentheuserfeelstotallyinside a
computationalenvironment, usuallyusingvisualizationglassesorCAVEs
(Cave Automatic Virtual Environment).
Interaction No interactionwhentheuser does notinteractwiththe virtual
environment.
Withinteraction, whenconnectedtothecomputer'sabilitytodetectuser
input andinstantlymodifythe virtual world and its actions.
Involvement It isrelatedtotheconditionofthelevelofinvolvementof a userwith a
givenaction, whichcanbe passive, such as, receiving training,
viewingthe virtual environment, oractive, such as reading a book,
participating in virtual surgery.
Adapted from Martins and Guimarães (2012).
Schlemmer and Backes (2015) emphasized that virtual reality applications bring
a very realistic environment involving the participants. When applied in the
teaching-learning processes, this possibility allows students to enjoy presence,
immersion, interaction, and involvement, which are all combined in enhancing
the learning results. In the same vein, Duncan et al. (2012) described all the
benefits of the virtual environment in the teaching-learning processes, merely in
the option of formative laboratory, collaborative work, socialization, and
entertainment. Gilbert (2004) explained that science subjects can be abstract, and
therefore, for deep learning, the virtual world is a valuable tool. Smutny et al.
(2019) highlighted that, in higher education, the application of virtual reality in
curricula positively impacts student engagement and motivation to learn.
2.3 Virtual Reality in the Teaching-Learning Processes of Production Planning

and Control
Production engineering is an extensive area. In some regions of the world, it is
even called industrial engineering or management engineering. In Brazil, the
guidelines for this area are organized by the Brazilian Association of Production
Engineering - ABEPRO. The ABEPRO (2008) described the curricular guidelines
to be applied in production engineering, organizing them in: I) Production
Management; II) Quality Management; III) Economic Management; IV)
Ergonomics and Workplace Safety; V) Product Management; VI) Operational
Research; VII) Strategic and Organizational Management; VIII) Organizational
Knowledge Management; IX) Environmental Management; and X) Education in
Production Engineering. As a sub-topic of item I, the theme of production
planning and control is in the same document. This theme is fundamental for
organizations because it is responsible for processing all sales information and
future demands and generating information on production needs for the
productive, supply, logistics, and purchasing departments. (Lage Junior, 2019).
The production planning and control process has another extremely relevant
function. Through the information generated for all subsystems of the
organization, deliveries are generated to customers within the agreed terms at a
113
reasonable cost (Corrêa et al., 2019). The production scheduling technique

impacts the use of equipment idleness and delays in scheduled deliveries (Habib
et al., 2015). Some factors need to be considered to execute production
sequencing, namely rules and guidelines of the productive system and
performance indicators (Corrêa et al., 2019). Among the rules and guidelines for
sequencing production, some researchers highlight relevant factors in
production planning and control, such as, fixed production batches or dynamic
production batches (Brahimi et al., 2017; Suzanne et al., 2020), organization of
production systems for discrete, continuous or project production (Armbruster
et al., 2012; La Marca et al., 2010), equipment failure rate (Göttlich & Knapp,
2019), setup times and manufacturing lead time (Allahverdi & Soroush, 2008),
processing time and Take time (Ayough et al., 2020), production capacity
available for the execution of the planned production (Babaei et al., 2014;
Oliveira & Costa, 2018), organization of the layout, according to the processing
and material flow scripts (Caicedo et al., 2019).
When evaluating the performance indicators of a production system, the results

of decisions of the production sequencing impact in the indicators, for example,
delivery attendant, delay of order, production advance, production lead time,
number of overdue orders, inventory in the process, and use of equipment
(Lustosa et al., 2008).
3. Method
The method applied to this research is organized in six stages as described in
Figure 1.
Figure 1. Method applied in the research
As displayed, steps 1, 2, and 3 of the method are described in the introduction

sections and the theoretical background on the developed theme. Stage 4 of the
method was segregated into two stages: defining the model's characteristics and
which software to use.
The virtual reality model used has a combination of factors that influence
decisions about production sequencing, which will be organized in scenario
formats to collect different results and how each factor impacts the results from
114
the production sequencing process. Each scenario was a mix of these guidelines:
the size of production lots, equipment failure rate (MTFB and MTTR), setup
times, processing times, production capacity, and demand. The scripts for the
processing of materials and the layout did not vary between scenarios. The
transfer batches will be identical to the production batches of scenarios 1 to 4.
The remaining scenarios will be informed in table 7 and 8 in the results section.
The evaluated indicators are: inventory in the process, punctual delivery,
manufacturing lead-time, and use of equipment.
The layout is organized in line. It consisted of six material processing

equipment, six assembly departments, and a shipping department. With this
equipment, eight different components will be processed. The processing
equipment, assembly departments, and the materials that are processed are
shown in Table 2.
Table 2. Equipment, assembly departments and parts processed in the scenarios

elaborated
MachinesandEquipment Assembly Department Processedparts
Eq1 MTG1 Xa
Eq2 MTG2 Xb
Eq3 MTG3 Xc
Eq4 MTG4 Xd
Eq5 MTG5 Xe
Eq6 MTG6 Xf
-- -- Xg
-- -- Xh
The organization of the equipment layout and the material processing flow are described
in Figure 2.
Figure 2. Layout of the production environment and material flow
115
For executing the scenarios, the Siemens Tecnomatix Real NC® (2020) - Plant
Simulation software will be used. The decision was made because this software
can simulate dynamic and productive environments in virtual reality. The
characteristic of virtual reality can be observed through the desktop, or its
images can be transferred to viewing glasses with characteristics of immersion in
the virtual environment. An example of the images generated in virtual reality
by the chosen software is shown in Figure 3.
Figure 3. Example of virtual reality application with (Siemens Tecnomatix RealNC,

2020) - Plant Simulation
Source: http://www.engusa.com/pt_br/product/siemens-tecnomatix-plant-simulation
Proposals on the scenarios used to organize production planning and control are
presented in the results section.
4. Results
As described in the method, it was necessary to organize a mix of guidelines that
impact the production planning and control process results for the preparation
of the scenarios. During the creation of Scenario 1, it was defined that the
available capacity of the processing equipment would be 24 hours a day and that
the manufacturing batches would correspond to seven days of demand from the
assembly departments. The other information applied to this scenario is
available in Table 3.
116
Table 3. Scenario 1 for the application of virtual reality in the teaching processes of
production planning and control
The objective of this scenario is to demonstrate the impact of large

manufacturing batches and high setup times on production sequencing
indicators. With this scenario, lead time indicators, inventory in processes, and
punctuality of deliveries (for the assemblies) will suffer impacts due to the
production lot and setup times.
The second scenario was elaborated from the data used in Scenario 1. The same
demand, capacity, cycle times, and failure rate data described in Scenario 1 were
used. The manufacturing batches were changed to four days of demand and the
setup times with an average reduction of 30%. The data for Scenario 2 are
revealed in Table 4.
Table 4. Scenario 2 for the application of virtual reality in production planning and
control teaching processes
The objective of Scenario 2 is to demonstrate the impacts on lead time and in-
process inventory with the reduction of manufacturing batches, setup times, and
punctuality of deliveries in the evaluated indicators.
For the development of Scenario 3, the same database as Scenario 2 was used,
changing only the different data on failure rate and the available capacity among
117
the evaluated equipment. The structured data for the simulation of Scenario 3
are described in Table 5.
Table 5. Scenario 3 for the application of Virtual Reality in the teaching processes of
In Scenario 3, there was a reduction in the failure rate of approximately 35%. The
available capacity was modified in five of the six pieces of equipment in this
production system, in which Eq1 was kept available 24 hours a day; the Eq2 /
Eq3 / Eq4 / Eq6 available 16 hours a day; and Eq5 available for 8 hours a day.
The reduction in the failure rate increases the availability of the equipment to
process materials, and the difference in the available capacity of the equipment
generates high indexes of stocks awaiting processing and shortage of materials
for later processes.
Scenario 4 was elaborated from the previous scenario, changing processing

times and material demand parameters to assess the impacts of these parameters
on the indicators of manufacturing lead time, stock in process, efficiency, and
punctuality of deliveries. The parameters used are displayed in Table 6.
118
In the scenario described in Table 6, there was a reduction in cycle times by

approximately 20% and demand by 10% compared to that described in Table 5.
The reduction in cycle times increases in the quantities produced in the same
period, but negatively impacts the manufacturing lead time and an inventory
increase. The demand reduction reinforces this impact because the consumption
of subsequent processes is lower, generating greater inventory in the process.
For elaborating scenarios 5 and 6, we tried to change the production planning

decision and applied the production logic pulled into the system, in which the
decision of what to produce depends on the planned stock level between
operations. For the execution of Scenario 5, it was decided to use the same data
as Scenario 1, but with supermarkets between operations. For Scenario 6, the
data is identical to Scenario 4, also with supermarkets between operations. The
layout of the production environment and the material flow for Scenarios 5 and
6 are shown in Figure 4.
Figure 4. Layout of the production environment and material flow for Scenarios 5 and 6
When evaluating the layout and material flow as shown in Figure 4, it is possible
to notice a change compared to that presented in Figure 2. That is, it appears that
Eq1 has its production managed by the stock levels of supermarkets called SA
and SB. The inventory levels demand the production of Eq3 of supermarkets SC
and SD. The production of the Eq4, Eq5, and Eq2 equipment (which has a
continuous flow with the Eq6 equipment) is managed by the inventory level of
the SE supermarket. The data defined for Scenario 5 are described in Table 7.
Scenario 5 uses rules to control inventory levels in supermarkets to determine

which components should be manufactured. When any part reaches the
minimum inventory level, a new manufacturing batch can be started. The level
119
of in-process inventory will be reduced because the equipment worked less,

even with production batches much larger than the daily demand (production
batches with seven days of daily demands). On the other hand, the equipment
efficiency will be reduced due to the demand lack for production in some
moments. In Scenario 6, data from Scenario 4 of virtual reality simulation were
used, according to Table 8.
Similar to Scenario 5, Scenario 6 brings all the production planning management

related to the supermarket stock levels. Unlike the previous scenario, however,
this one has setup times, maximum stock levels, and smaller production batches,
and the production lead time and stock in process indicators show considerable
improvements. Indicators of equipment efficiency and punctuality of delivery
will only be possible if they are evaluated after the simulation of the model in
virtual reality, using the software chosen for this task. All theoretical evidence
indicates that punctuality of delivery should have better rates in the leanest
scenarios (shorter setup times, shorter cycle times, shorter production batches),
but the quantification of these rates is possible only after the scenarios
simulation.
120
5. Analysis of Results
It appears that the application of virtual reality in the teaching-learning
processes is used in several areas of performance, demonstrating superior
learning results by students who use virtual reality when compared to those
who only use the traditional teaching method. This application is verified, for
example, in research by the authors like Quadir et al. (2019), Li et al. (2018), and
Skarka et al. (2015). They demonstrated the efficiency of the approach and
emphasized that it is essential to develop a learning environment that generates
the opportunity to increase learning through virtual reality to have a relevant
result.
Deciding on the production planning and control area generates unprecedented

research in an area of great relevance in production engineering. According to
Lage Junior (2019), the processes in this area impact various organizational
indicators, such as on-time delivery rate, inventory levels, idleness or lack of
production capacity, organization of internal work at factories, and direct
activities of purchasing, receiving, and storing materials. In addition, the
research opportunity presented by Anjos et al. (2020) demonstrated the
importance of creating differentiated teaching environments to generate the
most significant possible knowledge for students. In addition, Kanakana-
121
Katumba and Maladzhi (2019) emphasized that the cognitive environment must
consider some characteristics to be elaborated. In the scenarios proposed by the
authors, an environment with an experimental approach (through virtual
reality) is perceived because it generates for the student the opportunity to
practice, the concepts and theories developed in a practical way.
The scenarios to be applied in virtual reality were organized through a mix of

guidelines. According to the guidelines change, there was a change in some
indicators resulting from production planning and control. The relevance of each
scenario can be highlighted; in Scenario 1, the impacts of large production lots
and transfers between the equipment and the high setup times are shown ; in
Scenario 2, data similar to scenario 1 are applied, but with smaller production
batches and setup time. When comparing the results, impacts of indicators are
perceived, such as reducing production lead time, inventory levels in processes,
and punctuality of delivery. Similarly, Scenario 3 uses the reference data from
Scenario 2 but with a reduction in the available capacity for production
(reduction of the available time). Through this action, it will be possible to
evaluate improvements in the efficiency indexes. Scenario 4 was elaborated on
data from the previous scenario, but with reduced processing times for
manufactured materials and reduced demand. This change influences all the
indicators monitored during the virtual reality simulation. Scenarios 5 and 6 use
the data applied in Scenarios 1 and 4, respectively, however, with a fundamental
change, the production planning system starts to be drawn from the inventory
levels for the assembly processes (SE supermarket).
The validity of this model is based on the relationship between the factors that
make up the production system, for example, fixed production batches or
dynamic production batches (Brahimi et al., 2017; Suzanne et al., 2020),
organization of production systems for discrete, continuous or project
production (Armbruster et al., 2012; La Marca et al., 2010), equipment failure
rate (Göttlich & Knapp, 2019), setup times and manufacturing lead time
(Allahverdi & Soroush, 2008), processing time and take time (Ayough et al.,
2020), production capacity available for the execution of the planned production
(Babaei et al., 2014; Oliveira & Costa, 2018), organization of the layout, according
to the processing and material flow scripts (Caicedo et al., 2019) and the
evaluation the performance indicators of a production system, the results of
decisions of the production sequencing impact in the indicators , for example,
delivery attendant, delay of order, production advance, production lead time,
number of overdue orders, inventory in process, and use of equipment (Lustosa
et al., 2008).It is noticed that, in addition to the realism of the virtual
environment Schlemmer and Backes (2015), the benefits of the training
environment in virtual reality Duncan et al. (2012) and the cognition generated
by the training environment Lefrançois (2016), scenarios proposed for
application in virtual reality manage to articulate the guidelines that influence
the planning and control of production, impact the monitored indicators, and
finally support students to develop more clearly and concisely knowledge on the
topic addressed.
122
6. Conclusion
The discussion on the teaching-learning processes in production planning and
control allows us to see that the combination of scenarios with virtual reality and
the focus of the cognition of the teaching environment generate a combination of
factors that, if well-organized, lead to positive results in student learning
through the combination of dynamic data simulation and virtual reality, in
which the virtual world brings students closer to the studied subject. The
different scenarios provide students with two opportunities: (i) the ability to
understand which factors related to the production system influence the
production sequencing indicators, and (ii) with what impact each factor
influences these evaluated indicators. Scenario 1 demonstrates the impact of
factors (large production batches and high setup times) on the analyzed
indicators and how changing the factors applied to scenario 2 improves the
indicators. Scenarios 3 and 4 discuss other factors (failure rate and available
capacity) and how the quantitative change of factors influences the evaluated
indicators. Finally, they change the logic applied to production sequencing,
transforming it into a pull system, in which production is managed by demand
and all the benefits that this logic brings to the indicators of the production
sequencing process.
7. Limitations
Some limitations of the research are the lack of variation in process flows and
factory layout and the lack of application of variability in the data used in the
simulation, such as, production times, failure rates, setup times, and demand,
because it is known that these types of data are usually not static.
8. Future Research
For future research, the authors suggest testing other production sequencing
approaches, such as drum, lung, and rope (from the theory of constraints), and
developing a system with production for product stock by adding to this
decision the variability of the data of the applied factors, for example, setup
times, capacity and failure rate, to the sequencing of production.
9. References
ABEPRO. (2008). Áreas e Sub-áreas de Engenharia de Produção [Production engineering areas
and subareas] .
http://www.abepro.org.br/interna.asp?p=399&m=424&ss=1&c=362
Akbulut, A., Catal, C., & Yildiz, B. (2018). On the effectiveness of virtual reality in the
education of software engineering. Computer Applications in Engineering
Education, 26(4), 918–927. https://doi.org/10.1002/cae.21935
Allahverdi, A., & Soroush, H. M. (2008). The significance of reducing setup times/setup
costs. European Journal of Operational Research, 187(3), 978–984.
https://doi.org/10.1016/j.ejor.2006.09.010
Anjos, F. E. V. dos, Rocha, L. A. O., Silva, D. O. da, & Pacheco, R. (2020). Virtual and
augmented reality application in production engineering teaching-learning
processes. Production, 30, 1–16. https://doi.org/10.1590/0103-6513.20190088
Armbruster, D., Fonteijn, J., & Wienke, M. (2012). Modeling production planning and
transient clearing functions. Logistics Research, 5(3), 133–139.
https://doi.org/10.1007/s12159-012-0087-8
123
Ayough, A., Zandieh, M., & Farhadi, F. (2020). Balancing, sequencing, and job rotation
scheduling of a U-shaped lean cell with dynamic operator performance.
Computers and Industrial Engineering, 143.
https://doi.org/10.1016/j.cie.2020.106363
Babaei, M., Mohammadi, M., & Ghomi, S. M. T. F. (2014). A genetic algorithm for the
simultaneous lot sizing and scheduling problem in capacitated flow shop with
complex setups and backlogging. The International Journal of Advanced
Manufacturing Technology, 70, 125–134. https://doi.org/10.1007/s00170-013-
5252-y
Bates, A. W. (2015). Teaching in a digital age: Guidelines for designing teaching and learning
(1st ed.). Tony Bates Associates Ltd.
Brahimi, N., Absi, N., Dauzère-Pérès, S., & Nordli, A. (2017). Single-item dynamic lot-
sizing problems: An updated survey. European Journal of Operational Research,
263(3), 838–863. https://doi.org/10.1016/j.ejor.2017.05.008
Caicedo, A. J., Parra, J. W., & Rivera, L. (2019). Mathematical model for production
sequencing in a manufacturing company. Journal of Physics: Conference Series,
1388(1). https://doi.org/10.1088/1742-6596/1388/1/012001
Corrêa, Henrique Luiz; Gianese, Irineu Gustavo Nogueira; Caon, M. (2019). Planejamento,
programação e controle da produção: MRP II / ERP :Conceitos, uso e implantação : Base
para SAP, oracle applications e outros softwaresintegrados de gestão [Production
planning, scheduling and control: MRP II / ERP: Concepts, use and implementation:
Basis for SAP, oracle applications and other integrated management software] (6th ed.).
Atlas.
Crawley, E., Hosoi, A., Long, G., Kassis, T., Dickson, W., & Mitra, A. (2019, June). Moving
forward with the New Engineering Education Transformation (NEET) program at MIT
- building community, developing projects, and connecting with industry. 2019 Paper
presented at ASEE Annual Conference & Exposition Proceedings, Tampa,
Florida. https://doi.org/10.18260/1-2--33124
Duncan, I., Miller, A., & Jiang, S. (2012). A taxonomy of virtual worlds usage in
education. British Journal of Educational Technology, 43(6), 949–964.
https://doi.org/10.1111/j.1467-8535.2011.01263.x
Fonseca, D. E., Redondo, E. D., & Valls, F. (2016). Motivación y mejora académica
utilizando realidad aumentada para el estudio de modelos tridimensionales
arquitectónicos [Motivation and academic improvement using augmented
reality for the study of architectural three-dimensional models]. Education in the
Knowledge Society (EKS), 17(1), 45–64. https://doi.org/10.14201/eks20161714564
Frevert, M., & Di Fuccia, D.-S. (2019). Virtual reality as a means of teaching
contemporary chemistry. Proceedings of the 2019 The 3rd International
Conference on Digital Technology in Education, 34–38.
https://doi.org/10.1145/3369199.3369218
Gilbert, J. K. (2004). Models and modelling: Routes to more authentic science education.
International Journal of Science and Mathematics Education, 2(2), 115–130.
https://doi.org/10.1007/s10763-004-3186-4
Göttlich, S., & Knapp, S. (2019). Load-dependent machine failures in production network
models. SIAM Journal on Applied Mathematics, 79(4), 1197–1217.
https://doi.org/10.1137/18M1193177
Grodotzki, J., Ortelt, T. R., & Tekkaya, A. E. (2018). Remote and virtual labs for engineering
education 4.0: Achievements of the ELLI project at the TU Dortmund university. Paper
presented at the 26th SME North American Manufacturing Research
Conference,NAMRC 46,Texas, USA.
https://doi.org/10.1016/j.promfg.2018.07.126
124
Habib, A., Jilcha, K., & Berhan, E. (2015). Performance Improvement by Scheduling
Techniques: A Case of Leather Industry Development Institute. Paper presented at
Afro-European Conference for Industrial Advancement,261–269).
https://doi.org/10.1007/978-3-319-13572-4_21
Hwang, G.-J., Chiu, L.-Y., & Chen, C.H. (2015). A contextual game-based learning
approach to improving students’ inquiry-based learning performance in social
studies courses. Computers & Education, 81, 13–25.
Inayat, I., Inayat, Z., & ul Amin, R. (2016). Teaching and learning object-oriented analysis
and design with 3D game. Proceedings of 14th International Conference on
Frontiers of Information Technology , 46–51.
https://doi.org/10.1109/FIT.2016.16
Kanakana-Katumba, M. G., & Maladzhi, R. (2019). Online learning approaches for science,
engineering and technology in distance education. Paper presented at the IEEE
International Conference on Industrial Engineering and Engineering Management,
930–934. https://doi.org/10.1109/IEEM44572.2019.8978892
National Association of State Directors of Career Technical Education Consortium , &
Achieve. (2015, November 5). Building a strong relationship between
competency-based pathways and career technical education [Blog post].
https://careertech.org/sites/default/files/CTE-
CompetencyBasedPathways.pdf
La Marca, M., Armbruster, D., Herty, M., & Ringhofer, C. (2010). Control of continuum
models of production systems. IEEE Transactions on Automatic Control, 55(11),
2511–2526. https://doi.org/10.1109/TAC.2010.2046925
Lage Junior, M. (2019). Planejamento e controle da produção : teoria e prática [Production
planning and control: Theory and practice] (1st ed.). LTC.
Lamb, R., Antonenko, P., Etopio, E., & Seccia, A. (2018). Comparison of virtual reality
and hands on activities in science education via functional near infrared
spectroscopy. Computers & Education, 124, 14–26.
Laseinde, O. T., Adejuyigbe, S. B., Mpofu, K., & Campbell, H. M. (2016). Educating
tomorrows engineers: Reinforcing engineering concepts through Virtual Reality (VR)
teaching aid. Paper presented at the IEEE International Conference on Industrial
Engineering and Engineering Management, IEEM 2015, 2016.
https://doi.org/10.1109/IEEM.2015.7385894
Lefrançois, G. R. (2016). Teorias da Aprendizagem: O que o professor disse [Theories of human
learning] (6th ed.). Cengage Learning.
Li, Y., Zhang, D., Guo, H., & Shen, J. (2018). A novel virtual simulation teaching system
for numerically controlled machining. International Journal of Mechanical
Engineering Education, 46(1), 64–82. https://doi.org/10.1177/0306419017715426
Lustosa, L. J., Mesquita, M. A., Quelhas, G., Osvaldo, L., & Oliveira, R. J. de. (2008).
Planejamento e controle da produção [Planning and production control] (1st ed.).
Elsevier.
Martins, V. F., & Guimarães, M. de P. (2012). Desafios para o uso de Realidade Virtual e
Aumentada de maneira efetiva no ensino [Challenges for using virtual and
augmented reality effectively in teaching]. Anais Do Workshop de Desafios Da
Computação Aplicada à Educação[Proceedings of the Challenges of Computing
Applied to Education Workshop], 100–109.
Mayne, R., & Green, H. (2020). Virtual reality for teaching and learning in crime scene
investigation. Science and Justice, 60(5), 466–472.
https://doi.org/10.1016/j.scijus.2020.07.006
125
Mestrinho, N., & Cavadas, B. (2018). Innovation in teacher education: An integrative

approach to teaching and learning science and mathematics. Proceedings, 2(21),
1343. https://doi.org/10.3390/proceedings2211343
Mizukami, M. da G. N. (1992). Ensino: A Abordagens do Processo [Teaching: Process
Approaches] (1st ed.). LTC.
Oliveira, F. S., & Costa, M. L. G. (2018). Capacity expansion under uncertainty in an
oligopoly using indirect reinforcement-learning. European Journal of Operational
Research, 267(3), 1039–1050. https://doi.org/10.1016/j.ejor.2017.11.013
Prensky, M. (2001). Digital natives, digital immigrants part 1. On the Horizon, 9(5), 1–6.
https://doi.org/10.1108/10748120110424816
Quadir, B., Yang, J. C., & Chen, N. S. (2019). The effects of interaction types on learning
outcomes in a blog-based interactive learning environment. Interactive Learning
Environments, 1–14. https://doi.org/10.1080/10494820.2019.1652835
Sánchez, A., Redondo, E., Fonseca, D., & Navarro, I. (2015). Academic performance
assessment using augmented reality in engineering degree course. Proceedings -
Frontiers in Education Conference, 1–7.
https://doi.org/10.1109/FIE.2014.7044238
Santos, J. (2006). Teorias da aprendizagem: Comportamentalista, cognitivista e
humanista [Learning theories: Behaviorist, cognitivist and humanist]. Revista
Cientifica Sigma [Sigma Scientific Journal], 2, 97–111.
Schlemmer, E., & Backes, L. (2015). Learning in Metaverses. IGI Global.
https://doi.org/10.4018/978-1-4666-6351-0
Shirazi, A., & Behzadan, A. H. (2015). Content delivery using augmented reality to
enhance students’ performance in a building design and assembly project.
Advances in Engineering Education, 4(3), 1–24.
Skarka, W., Wylezol, M., Januszka, M., Rzydzik, S., & Targosz, M. (2015). Improving the
ability of future engineers by using advanced interactive 3D techniques in
education. In R. Curran, N. Wognum, M. Borsato,J. Stjepandic & W.J.C.
Verhagen (Eds.), Transdisciplinary lifecycle analysis of systems ( pp. 647–656). IOS
Press. https://doi.org/10.3233/978-1-61499-544-9-647
Smutny, P., Babiuch, M., & Foltynek, P. (2019). A review of the virtual reality
applications in education and training. Proceedings of the 2019 20th
International Carpathian Control Conference, ICCC.
https://doi.org/10.1109/CarpathianCC.2019.8765930
Soudien, C. (2010). Transformation in higher education- A briefing paper. Policy Studies
Journal, 44(1), 29. https://doi.org/10.1007/1-4020-4006-7
Suzanne, E., Absi, N., Borodin, V., & van den Heuvel, W. (2020). A single-item lot-sizing
problem with a by-product and inventory capacities. European Journal of
Operational Research, 287(3), 844–855. https://doi.org/10.1016/j.ejor.2020.05.017
Tsai, C. W., Shen, P. Di, & Lu, Y. J. (2015). The effects of problem-based learning with
flipped classroom on elementary students’ computing skills: A case study of the
production of Ebooks. International Journal of Information and Communication
Technology Education, 11(2), 32–40. https://doi.org/10.4018/ijicte.2015040103
Wang, H.-Y., Liu, G.-Z., & Hwang, G.-J. (2017). Integrating socio-cultural contexts and
location-based systems for ubiquitous language learning in museums: A state of
the art review of 2009-2014. British Journal of Educational Technology, 48(2), 653–
671. https://doi.org/10.1111/bjet.12424
Żechowicz, M., Gromadziński, L., Dudziński, L., Grabarczyk, Ł., Żechowicz, T.,
Zabłocki, M., & Zaborowski, P. (2018). Development and implementation of
methods for training doctors and dental surgeons using virtual devices and
simulation. Polish Annals of Medicine. https://doi.org/10.29089/2017.17.00002
126

Vol. 20, No. 8, pp. 126-138, August 2021
Received Jul 10, 2021; Revised Aug 11, 2021; Accepted Aug 25, 2021
Medical Students’ Evaluation of Online

Assessment: A Mixed-Method Account of
Attitudes and Obstacles
Abed Alkarim Ayyoub*

Faculty of Educational Sciences and Teacher Training, An-Najah National
University, Nablus, Palestine
https://orcid.org/0000-0001-9111-4465
Oqab Jabali
Language Center
Faculty of Humanities, An-Najah National University, Nablus, Palestine
https://orcid.org/0000-0003-1156-6205
Abstract. This mixed-method account explores the medical students’

perceptions and attitudes regarding online assessment in two Palestinian
universities. The researchers aimed at identifying the way medical
students look at online assessment, as well as the pitfalls and the
drawbacks of such an atypical evaluation method at the university level.
Of the large number of medical students enrolled in the two universities,
302 completed and returned the survey, and 61 students were selected to
interpret their open responses qualitatively. The study findings suggest
that the evaluation of Al-Quds University in Jerusalem was better and
higher than that of the An-Najah National University in Nablus.
Statistically significant differences were found, when some demographic
variables, i.e., gender and year of study interaction were selected. Finally,
the study respondents highlighted a number of drawbacks for online
assessment, which the researchers sorted into eight main categories that
varied in percentages. The medical students showed their dissatisfaction
with this type of assessment, for many considerations, including, but not
limited to, technical and administrative matters in nature.
Keywords: evaluation; medical students; online assessment; perceptions;

technical problems
* Corresponding author: Abed Alkarim Ayyoub; Email: ayyoub@najah.edu
©Authors
1. Introduction
A lot of factors have recently emphasised the necessity to opt for the adoption of
online teaching and learning. The tremendous technological advances, educators’
and students’ high expectations, and the rise of unexpectedly urgent conditions
and demands, all dictate that the majority of educational institutions should
seriously consider the use of modern technology in learning/teaching situations.
Consequently, a large number of higher educational institutions worldwide have
currently been incorporating computer technology for a considerable time.
Pedagogically, education is not merely communicating information to learners in
a passive way. Rather, students acquire information, or knowledge, by being
actively engaged socially, psychologically and cognitively, as they construct
meanings. This is deeply rooted in the constructivist model, in which such factors
as socio-cultural and emotional beliefs, values and attitudes significantly impact
students’ learning outcomes and their interaction in schools (Ambrose et al., 2010).
As such, the main premise of the constructivist approach revolves around
bringing the learner to the forefront of the teaching/learning situation, while
adopting relevant pedagogical strategies that build on students’ diverse needs,
interests, strengths, and expectations, and that are conducive to their academic
success and social development (Richardson & Fleer, 2003).
Students are very likely to develop meaningful knowledge, based on their
experiences away from stringent, firm classroom regulations; and they might then
well suggest their own relevant assessment techniques that are very necessary in
the teaching/learning process (Kottail, 2009); they also learn better when they are
provided with opportunities to be part of the assessment approach that is used in
the classroom. It has been argued that the application of various evaluation and
assessment techniques, within the boundaries of the constructivist model,
promotes students’ critical thinking skills and increases their academic
achievement.
Additionally, involving students in educational assessment may have future
positive outcomes in their lives. It should also reduce instructors’ anxieties or
concerns. In fact, relevant and meaningful forms of assessment tend effectively to
measure and “evaluate judgment, attitude and behaviour, in addition to
knowledge, and skills” (Harris et al., 2017, p. 605).
The quality of educational assessment processes should be based on universal
standards, as well as on students’ unique and preferred learning styles and
intelligence levels (Harris at al., 2017). While this can be a challenging process, it
might be facilitated by integrating alternative assessment strategies, including the
online assessment technique (Bennett, 2011). This new type of assessment could
well result in improving students’ performance outcomes. In fact, this approach
can help in overcoming the gaps in traditional assessment that might not be
suitable for assessing higher-level cognitive and affective skills (Kuh et al. 2014).
Using technology in the process of assessment has become a reality that ranges
from developing examinations to storing results for future use. It is intended for
assessing learners’ prior knowledge, skills, and abilities; it also aims to create and
manage materials and resources, in addition to providing feedback (Cakiroglu et
al., 2017). E-assessment provides new methods and opportunities for various
types of assessment related to various types of knowledge (Alruwais, Wills &
Wald, 2018; Chang et al., 2013; Crews & Curtis, 2010; Kuriakose & Luwes, 2016).
Furthermore, electronic assessment helps to reduce the work load of instructors
and students; since it is likely to be accessed at anytime and anywhere (Cukusic
et al., 2014). Accordingly, it is easy for educators and school administrators to
gather data, to conduct statistical analyses and to test the results (Broughton, 2013;
Douglas, 2012); and consequently, to make good decisions (Duran et al., 2013).
Furthermore, e-assessment can be easily reproduced and utilized; since it needs
no more than a simple computer, or a smart phone with an internet connection.
This would make it possible to measure students’ outcomes, and to
instantaneously, help them to get direct and immediate feedback on their
performance (Alruwais, Wills & Wald, 2018; Betlej, 2013).
It is intended to promote authentic assessment, as well as to facilitate testing and
collaboration (Guo et al., 2014; Johnson & Davies, 2012). Electronic testing is more
cost-effective, and easier to handle, to administer and to score, more reliable and
easier to replicate; and sometimes, it is more authentic, objective, and bias-free,
when marking students’ responses (Jordan & Mitchell, 2009; Khare & Lam, 2008).
It also allows instructors to use questions that promote interactivity and the use
of multimedia (Lahad et al., 2004), which would undoubtedly benefit students
who are subjected to this form of evaluation (Ali et al., 2021).
Online assessment may face some challenges and obstacles. For instance, some
students may be inexperienced in handling computer programs and online
assessment techniques (Alruwais, Wills & Wald, 2018). Another challenge may
relate to computer availability and internet connection (Ridgway et al., 2004), as
well as to the lack of sufficient and good infrastructure (Ridgway et al., 2004),
especially in poor countries like Palestine.
It is also possible for some teachers to face problems, when they are forced to use
e-assessment techniques, especially for the first time; some instructors may lack
the necessary experience, or they may not be familiar with the technology; and
consequently, this may pose problems for students, such as delaying the loading
of exams (Ogletree et al., 2014; Russell & Shepherd, 2010).
Furthermore, there is the open-question issue; marking open questions, such as

explaining, and listing items would then be difficult to handle (Stodberg, 2012).
Student identification is another major challenge that exacerbates the reluctance
to adopt online assessment; needless to say, learners cannot be verified easily,
although being remote. Cheating is another pitfall of online assessment; when
students take an exam that is not invigilated. Cheating more than doubles in such
situations (Osuji, 2012; Stodberg, 2012; Yates & Beaudrie, 2009).
Students have different attitudes and perspectives towards their instructors, the
teaching/learning methodologies, as well as the assessment techniques used to
evaluate their performance. They also have their own reservations about the
effectiveness in certain circumstances. The current study aimed to identify
medical students’ evaluation of online assessment, which had been used for a
whole year in two Palestinian universities. The researchers attempted to answer
the following two questions: 1) Do medical students' evaluation of the electronic
assessment differ, according to students’ gender, level (academic year), and total
average?: and 2) What are the major obstacles that face educators, when using
online/electronic assessment?
2. Methods
To carry out the current quantitative/qualitative study, institutional consent and
approval had to be obtained from the two university vice-presidents of academic
affairs, the directors of the Electronic-Learning Centre, the deans of Scientific
Research Departments, as well as the research Ethics Committees. Quantitatively,
a descriptive online questionnaire was compiled to explore the way medical
students evaluate and perceive online assessment during the COVID-19
pandemic, in the academic year of 2019/2020. As the researchers were interested
in examining the relevant information efficiently, e-assessment as an evaluation
tool, which was conceived comprehensively, in order to include any possible
advantage, or to exclude any disadvantage that might be relevant to instruction
and/or education at the university level.
The survey included 100 constructed declarative statements, which were sent to
40 arbitrators, whose scores were collected; and the correlation between each
paragraph and the total was calculated; and then the paragraphs, the correlation
of which was found to be less than 0.6, were deleted. Then, the mean scores for
the first and last quarters were calculated by the arbitrators, for rating the
paragraphs. A t-test of two independent samples between the means of the two
quarters for each paragraph was calculated, and the paragraphs for which the
differences were not statistically significant, were deleted – simply because the
researchers were merely interested in the paragraphs that are distinguished by
high discrimination. Finally, 35 high discrimination items that were of significant
relevance and correlation to the concept of e-assessment were retained.
To ensure the questionnaire’s validity, factorial validity was calculated by using
the Kaiser-Mayer-Olkin test; this was found to be 0.0955 which ensured that the
items were suitable for exploratory-factor analysis. To exclude orthogonal items,
the principal-component method and the Oblimin rotation method were used;
consequently, items (9, 10, 11, 24, 29, 34) the communality degree, or factor loading
of which was less than 0.3, were excluded. In total, 29 items were retained, as
shown in the scree plot below.
Figure 1: Scree plot for the number of dimensions in the questionnaire
Clearly here, there was a dominant dimension; since there was one infliction
point, which represented 41% of the explained variance, as shown in Table 1
below. Consequently, the questionnaire was considered suitable, according to the
factorial-validity analysis.
Table: Total Variance explained for the dominant dimension of the questionnaire
Extraction Sums of Squared
Initial Eigenvalues Loadings
% Cumulative % Cumulative
Component Total Variance % Total Variance %
1 11.812 40.733 40.733 11.812 40.733 40.733
To ensure the reliability of the questionnaire, Cronbach’s Alpha was calculated.

Cronbach’s Alpha coefficient was 0.946 for the total of 29 items. The alpha value
was higher than 0.7; this shows that the questionnaire is reliable. Eventually, the
questionnaire was posted for medical students to complete online. 302 survey
instruments were completed and returned for analysis.
Qualitatively, the principal study tool also required students to provide

information about their attitude towards online assessment, the various obstacles
they faced and the ways to overcome them, in order to complement and inform
the quantitative findings by providing valuable data that would help to
understand online assessment and the students’ attitudes towards it. Sixty-one
students from each university were randomly selected; and their responses were
analyzed by using MAXQDA. The researchers sorted through the obstacles and classified
them into eight different codes, as follows: ethical, administrative, training, time,
technical, negative attitudes, substantive issues, and psychological problems.
2.1. Sample
The population of this study included those students who study medicine at An-
Najah National University in Nablus and those at the Al-Quds University in
Jerusalem. However, few students answered the survey (n = 302), with 151 from
each university. A total of 61 students’ responses were randomly selected, in order
to account for the qualitative analysis of the data.
2.2. Procedures
Eventually, the questionnaire was posted to the university students online; it was
written in the students’ native language (Arabic), to ensure that all the students
fully understood the survey items. A total of 302 survey instruments were
completed and returned for analysis. Data collection was carried out during the
first semester of the academic year of 2019/2020.
Quantitatively, the data were normally distributed and analyzed by using
descriptive statistics; factorial ANOVA was used to calculate the mean differences
between the demographic elemental scores, by using the SPSS version 26. The
principal study tools also required students to provide information about their
attitudes, opinions and perspectives on e-assessment, in order to complement and
inform the quantitative findings by providing valuable data that examined
students’ attitudes towards these matters.
Qualitatively, MAXQDA was used by the researchers themselves, in order to
calculate the frequencies, percentages and students’ responses, and their opinions,
as well as the information provided on the open-ended questions. MAXQDA is a
software program designed for computer-assisted qualitative methods, data and
text analysis; it offers tools for the organization and analysis of qualitative data,
especially those obtained as texts, in order to attain a valuable explanation and a
comprehensive understanding or interpretation of a phenomenon, or a tendency
[MAXQDA: The Art of Data Analysis, n.d.].
2.3. Analysis
Quantitatively, the data were normally distributed and analyzed by using
descriptive statistics; and factorial ANOVA was used to calculate the mean
differences between the demographic elemental scores by using the SPSS version
26. Qualitatively, MAXQDA was used to calculate the frequencies, percentages,
and students’ responses to the open questions.
3. The Results
3.1. Demographic characteristics and the evaluation of the online assessment
To answer the first question, (302) students studying medicine and health sciences
at An-Najah National University (n= 151) and Al-Quds University (n= 151)
responded to the questionnaire. The descriptive statistics of the students’
responses were calculated, based on the demographic variables; the results are
shown in Table 2.
Table 2: Demographic features of respondents and the results
An Najah University Al Quds University
N M S.D. N M S.D.
Gender Male 51 3.10 0.85 46 3.45 0.55
Female 100 3.07 0.63 105 3.21 0.50
Average Excellent 23 2.79 0.81 20 3.33 0.75
Very good 57 3.25 0.60 60 3.30 0.56
Good 57 3.03 0.77 61 3.25 0.42
Fair 14 3.08 0.53 10 3.28 0.38
Year Freshman 17 2.78 1.12 18 3.25 0.57
Sophomore 35 3.11 0.52 35 3.34 0.50
Junior 56 3.14 0.70 39 3.30 0.64
Senior 34 3.13 0.65 32 3.17 0.53
Super senior 9 3.01 0.62 27 3.33 0.33
Total 151 3.08 0.73 151 3.28 0.53
The results in Table 2 show that medical students’ evaluation of the online
assessment was medium; as the mean squares ranged between 2.6 – 3.4, based on
a Likert Scale. However, the evaluation of the students at the Al-Quds University
was higher than those of the An-Najah University (0.2). Furthermore, there were
differences among the students’ demographic variables. To ensure whether these
differences were statistically significant, the researchers used Factorial ANOVA,
as shown in Table 3 below.
Table 3: Tests of Between-Subjects Effects (Factorial ANOVA) for demographic

variables
Mean
Source SS Df Square F P η2
University 1.35 1 1.35 3.78 0.04 0.02
Gender 1.09 1 1.09 3.07 0.08 0.01
Average 0.04 3 0.01 0.04 0.99 0.00
Year 0.58 4 0.14 0.41 0.80 0.01
university * gender 0.15 1 0.15 0.43 0.51 0.00
university * average 0.63 3 0.21 0.59 0.62 0.01
university * year 0.89 4 0.22 0.63 0.64 0.01
gender * average 1.61 3 0.54 1.51 0.21 0.02
gender * year 3.73 4 0.93 2.62 0.04 0.04
average * year 5.41 12 0.45 1.27 0.24 0.06
university * gender * average 0.06 3 0.02 0.06 0.98 0.00
university * gender * year 0.72 4 0.18 0.51 0.73 0.01
university * average * year 5.33 10 1.03 1.48 0.20 0.02
gender * average * year 4.65 9 0.52 1.45 0.17 0.05
university * gender * average * year 0.54 2 0.27 0.76 0.47 0.01
Error 84.42 237 0.36
Corrected Total 119.59 301
The results showed that there was a statistically significant main effect in the evaluation
of online assessment attributed to university, F (1, 237) 3.87, p 0.04, ŋ2 0.02;
looking back at Table 1, one sees that the evaluation of Al-Quds University was
better than that of An-Najah University. However, no statistically significant
differences were found, based on the other variables. With respect to interactions
among the demographic variables, the study results showed that there were no
statistically significant differences for most of them, except the interaction
between gender and the year of study, F (4, 237) 2.62, p 0.04, ŋ2 0.04.
Figure 2 below illustrates the trends of these differences.
Figure 2: Trend of interaction between student’s gender and year of study
Figure 2 above shows that the evaluation of male students is better than that of
the female students in all the years of study, except for the freshmen, whose
evaluation was higher. However, the higher the level of the student, the lower
their evaluation. This is inconsistent with the findings of a study conducted in
Romania, which showed acceptance of online assessment increases, as the
students move to higher levels (Marius et al., 2016).
To answer the second question, which required students to provide information

about their attitudes towards online assessment, the various obstacles faced, and
the ways to overcome them, sixty-one students from each university were
randomly selected; and their responses were analyzed by using MAXQDA. The
researchers sorted the obstacles and classified them into eight different codes, as
follows: ethical, administrative, training, time, technical, attitudinal, substantive,
and psychological problems, as outlined in Figure 3. It can be noticed that the
students in both universities believe that most of the obstacles were
administrative in nature; the way the two universities manage online assessment
has many pitfalls, such as question types, mark distribution, marking, and the lack
of any feedback. Technical problems (e.g., poor internet connection, lack of
infrastructure, etc.) were also detected in both universities.
Attitudinal and psychological problems were more at An-Najah University than

at Al-Quds University; and this explains why the evaluation of Al-Quds
University was better than that of An-Najah University.
18
16
14
12
10
8
6 AlQuds Un
4 An Najah Un
2
0
Figure 3: Codes of students’ attitudes to online assessment
4. Discussion
The main study findings showed that the evaluation of Al-Quds University (Mean
is 3.28) was better and higher than those of the other university (Mean is 3.08), as
shown in Table 2 above. This could be explained by various reasons. Firstly, the
marking system used in the two universities varies. An-Najah University
administers at least three exams per semester, in addition to a set of assignments,
research papers, projects, discussions, or presentations; while Al-Quds University
administers very few exams. This means that it is very likely for students to lose
marks in each exam, or in any other evaluation activity. In addition, the high
percentage of the participation mark is offered to Al-Quds students; it is 20% of
the total mark of each course at this university; while it is 5%-10% in very few
courses at An-Najah University. Secondly, An-Najah University adopts the
National Board of Medical Exams (NBME), which forces students to sit for these
exams at different levels; and it contributes 40% of its weight to the final mark of
many courses. Some of these exams are very demanding (Internal Medicine and
Advanced Surgery); and many students do not do well in them; since they were
originally prepared for students whose native language is English, but not for
students that use English as a foreign language.
Furthermore, An-Najah students study a course called Medical Ethics in the first
year; and they are tested in this course in the fourth year, as part of the NBME;
added to this, is the fact that these exams are paid for by the students themselves.
On the other hand, most of these exams are held locally at Al-Quds University;
and they are free of charge.
The study findings also showed that both gender and year interacted; and
consequently, this resulted in statistically significant differences. Responses to the
questionnaire items showed that the evaluation of fresh male students was lower
than that of their female counterparts; female students in the first year of study
were mostly more apprehensive and more anxious, due to being in a different
environment, and this, together with their inability to find the necessary
textbooks, resources and materials, increased their concerns. However, the
evaluation of females improved considerably in the second year; but it was still
lower than that of their male counterparts.
This could be attributed to the fact that these females started to become familiar
with the university and its system; they made considerably more effort; and they
also showed more satisfaction with the exam marking and teaching
methodologies. Furthermore, the study findings showed that in the third year,
female students showed more dissatisfaction with online assessment; since they
started to lose marks, due to adopting the online assessment approach, which
lacks precision, fairness, and an explicit grading system. Similar results were
reached by Wiggins (1990). Third year students began to lose marks, due to
examiners’ high level of subjectivity, as stated by (Moni et al., 2002).
With respect to the main obstacles that face online assessment, the researchers
sorted them into eight main categories, which varied in abundance. A lot of
students in the two universities stated that the time allocated to online exams was
barely sufficient; consequently, these students were forced to exert more effort to
study and gain higher marks. This finding is inconsistent with the findings of
previous research done by Cukusic et al. (2014). Added to this is the timer on the
exam; it creates more stress (Khan and Khan, 2019). This is also contrary to Betlej’s
(2013) findings that online assessment is easy to handle and enables students to
get feedback very speedily.
The current study showed that e-assessment was not that easy, due to the lack of
adequate infrastructure for online assessment and poor internet connection
(Ridgway et al., 2004). Students at both universities also stated that in most online
exams, students could not go back to questions that they had already answered,
due to examiners’ intervention; and this relates indirectly to the marking of these
exams. While these results are consistent with those of Stodberg’s (2012) research
findings, they contradict those of other researchers, such as Jordan & Mitchell
(2009), Khare & Lam (2008), who undermined the role of examiners.
Furthermore, students argued that sitting for many exams, doing many
assignments, and being tested on topics they had never studied, was very difficult
and challenging – so much so, that exams become irrelevant, and arbitrary; and
they are done just to acquire marks (Hawe, 2002); while most of the question items
required deeper understanding; and this further reduced their motivation.
Another major technical obstacle, according to students, is related to poor internet

connection, which results in slow logging into the exams, and sometimes delayed
loading of these exams by instructors (Russell & Shepherd, 2010). Finally, some
students believed that online assessment was not fair; since it possibly allows
some students to cheat, due to insufficient invigilation and examinee-identity
verification; such a result is consistent with other studies, including those of Osuji
(2012); Russell & Shepherd, (2010); Yates & Beaudrie (2009). Consequently, online
assessment reduces motivation among smart, hard-working students, who would
eventually disapprove of this type of assessment.
5. Conclusion
The researchers found that applying new types of assessment without careful,
prior strategic planning, is very likely to create different attitudes among students,
irrespective of their gender, year of study or average marks. Students’ evaluations
differ, based on where they study. Medical students showed dissatisfaction with
this type of assessment, for many reasons, including, but not limited to, various
technical and administrative aspects. Consequently, students’ perceptions and
attitudes, regarding online assessment, should be considered, in order to ascertain
a smooth educational process that effectively incorporates technology.
University administrations are supposed to prepare well for such a transition to

online assessment, in order to persuade students of the usefulness thereof. This
study showed that to convince students to accept online assessment, universities
should plan well, provide adequate infrastructure, and consider students’
circumstances. Finally, the highlighted pitfalls of online assessment should be
addressed seriously and carefully, in order to integrate this type of assessment –
not only at universities – but also in all the other educational institutions.
This study was carried out in two Palestinian universities; it could be improved if
other universities or university staff members, as well as decision-makers, were
involved. Added to this limitation, of course, was the limited access to students in
the other university, which is located in a place that was not easy for the
researchers to access, without having a permit from the Israeli occupation. The
small number of participants involved in the qualitative survey was attributed to
this fact. The time allocated to respond to the online questionnaire played a
negative role; and it resulted in having a small, unrepresentative sample of
participants, who completed the survey. Consequently, a future longitudinal
study could shed more light on the benefits, challenges, and shortcomings of
online assessments, as well as their impact on students’ attitudes and their
academic achievements.
Contributors: Both authors made substantial contributions to the writing of this

paper; they were both involved in drafting this article, or in revising it critically
for important intellectual content; and they both gave their final approval of the
version to be published.
6. References
Ali, L., Abidal Hamid. N., & Damour, H. (2021). The Shift to Online Assessment Due to
COVID-19: An Empirical Study of University Students, Behaviour and
Performance, in the Region of UAE. International Journal of Information and
Education Technology, 11(5), 220-228. https://doi.org/10.18178/ijiet.2021.11.5.1515
Ambrose, S. A., Bridges, M. W., DiPietro. M., Lovett, M. C., & Norman, M. K. (2010). How
Learning Works-Seven Research-Based Principles for Smart Teaching. San Francisco,
CA: Jossey Bass.
Bennett, R. E. (2011). Formative assessment: a critical review. Assessment in Education:
Principles, Policy & Practice, 18(1), 5-25.
https://doi.org/10.1080/0969594X.2010.513678
Betlej, P. (2013). E-examinations from student's perspective – The future of knowledge
evaluation. Studia Ekonomiczne, 153, 9–22.
Broughton, S., Hernandez-Martinez, P., & Robinson, C. (2013, July 28th - August 2nd). A
definition for effective assessment and implications on computer-aided
assessment practice 37th Conference of the International Group for the
Psychology of Education, (PME 37), (Vol. 2, pp. 113-120). Kiel, Germany.
Cakiroglu, U., Erdogdu, F., Kokoc, M., & Atabay, M. (2017). Students’ preferences in
online assessment process: Influences on academic performances. Turkish Online
Journal of Distance Education-TOJDE, 18(1), 132-142.
Chang, C. C., Liang, C., & Chen, Y.H. (2013). Is learner self-assessment reliable and valid
in a Web-based portfolio environment for high school students? Computers &
Education, 60(1), 325-334. https://doi.org/ 10.18178/ijiet.2021.11.5.1515
Crews, T., & Curtis, D. (2010). Online course evaluations: Faculty perspective and
strategies for improved response rates. Assessment & Evaluation in Higher
Education, 36(7), 965–878. https://doi.org/10.1080/02602938.2010.493970
Cukusic, M., Garaca, Z., & Jadric, M. (2014). Online self-assessment and students' success
in higher education institutions. Computers & Education, 72, 100-109.
Douglas, M., Wilson, J., & Ennis, S. (2012). Multiple-choice question tests: a convenient,
flexible and effective learning tool? A case-study. Innovations in Education and
Teaching International, 49(2), 111–112.
https://doi.org/101080/147032972012677596
Duran, M., Mihladiz, G., & Balliel, B. (2013). The competency level of elementary school
teachers towards the alternative assessment methods. Mehmet Akif Ersoy
Universitesi Egitim Bilimleri Enstitusu Dergisi, 2(2), 26-37.
Guo, R., Palmer-Brown, D., Lee, S., & Cai, F. (2014). Intelligent diagnostic feedback for
online multiple-choice questions. Artificial Intelligence Review, 42(3), 369–383.
http://doi.org/101007/s10462-013- 9419-6.
Harris, P., Bhanji, F., Topps, M., Ross, S., Lieberman, S., Frank, J. R., Snell, L., Sherbino, J.,
& ICBME Collaborators. (2017). Evolving concepts of assessment in a
competency-based world. Medical Teacher, 39(6), 603–608.
http://doi.org/10.1080/0142159X.2017.1315071
Hawe, E. M. (2002). Assessment in a pre‐service teacher education program: The rhetoric
and the practice of standards‐based assessment. Asia‐Pacific Journal of Teacher
Education, 30(1), 93–106. http://doi.org/10.1080/13598660120115002
Johnson, G., & Davies, S. (2012). Unsupervised Online Constructed-Response Tests:
Maximizing Student Learning and Results Integrity (pp. 400–408) Wellington:
Paper presented at the Ascilite Conference.
Jordan, S., & Mitchell, T. (2009). E-assessment for learning? The potential of short-answer
free-text questions with tailored feedback. Br J Educ Technol, 40(2), 371–385.
https://doi.org/10.1111/j.1467-8535.2008.00928.x
Khan, S., & Khan, R. (2019). Online assessment. Exploring perspectives of university
students. Education and Information Technologies, 24, 661–677.
https://doiorg/101007/s10639-018-9797-0.
Khare, A., & Lam, H. (2008). Assessing student achievement and progress with online
examinations: Some pedagogical and technical issues. International Journal on E-
learning, 7(3), 383–402.
Kottail, N. K. (2009). Meaning making and self-evaluation. ERIC.
http://filesericedgov/fulltext. Accessed August 10, 2020.
Kuh, G. D., Jankowski, N., Ikenberry, S. O., & Kinzie, J. (2014). Knowing What students
Know and Can Do: The Current State of Student Learning Outcomes Assessment
in US Colleges and Universities. Urbana: University of Illinois and Indiana
University, National Institute for Learning Outcomes Assessment (NILOA).
Kuriakose, R. B., & Luwes, N. (2016). Student perceptions to the use of paperless
technology in assessments–a case study using clickers. Procedia - Social and
Behavioral Sciences. 228(2016), 78-
85. https://doi.org/10.1016/j.sbspro.2016.07.012.
Lahad, N., Dafoulas, G., Kalaitzakis, E., & Macaulay, L. (2004). Evaluation of online
assessment: The role of feedback in learner-centered e-learning Proceedings of the
37th Hawaii International Conference on System Sciences, 2004, 1-10.
Marius, P., Marius, M., Dan, S., Emilian, C. & Dana, G. (2016). Medical students’
acceptance of online assessment systems. Acta Medica Marisiensis, 62(1), 30–32.
https://doiorg/101515/amma-2015-0110
MAXQDA: The Art of Data Analysis (n.d.). What is MAXQDA.
https://www.maxqda.com/what-is-maxqda
Moni, K. B., van Kraayenoord, C. E., & Baker, C. D. (2002). Students’ perceptions of
literacy assessment. Assessment in Education, 9(3), 319–342.
Nuha, A., Wills, W., & Wald, M. (2018). Advantages and Challenges of Using e-
Assessment International Journal of Information and Education Technology, 8(1),
34-37. https://doiorg/10.18178/ijiet.2018.8.1.1008
Ogletree, A., Ogletree, S., & Allen, B. (2014). Transition to online assessments: A personal
perspective of meeting common core state standards in an elementary school in
Georgia. Georgia educational researcher, 2014, 11(1), 170–187.
https://doiorg/10.20429/ger.2014.110107
Osuji, U. (2012). The use of e-assessments in the Nigerian higher education system. Turkish
Online J Distance Educ, 13(4), 1–13.
Richardson, C., & Fleer, M. (2003). Collective Mediated Assessment: Moving towards a
Socio-cultural Approach to assessing children’s learning. Journal of Australian
Research in Early Childhood Education, 10, 41-55.
Ridgway, J., McCusker, S., & Pead, D. (2004). Literature review of e-assessment. University
of Durham: Durham, England.
Russell, C., & Shepherd, J. (2010). Online role-play environments for higher education.
British Journal of Educational Technology, 41(6), 992-1002.
https://doiorg/101111/j1467-8535200901048x
Stodberg, U. (2012). A Research Review of E-Assessment. Assessment & Evaluation in
Higher Education, 37(5), 591-604.
https://doi.org/10.1080/02602938.2011.557496
Yates, R., & Beaudrie, B. (2009). The impact of online assessment on grades in community
college distance education mathematics courses. American Journal of Distance
Education, 23(2), 62–70. https://doi.org/10.1080/08923640902850601
139

Vol. 20, No. 8, pp. 139-159, August 2021
The Systematic Implementation of an Innovative

Postgraduate Online Learning Model in the Middle East
Nessrin Shaya
American University in the Emirates, Dubai, United Arab Emirates
https://orcid.org/0000-0003-4201-3945
Laila Mohebi
Zayed University, Dubai, United Arab Emirates
https://orcid.org/0000-0003-2640-4532
Abstract. This paper aims at examining the factors contributing to effective

implementation of online learning in the Middle East higher education
sector, through investigating the success and learning effectiveness of an
innovative online learning model offered jointly by three prestigious
universities in two different Arab countries. A mixed-method research
approach was employed to triangulate data collected from key stakeholders
engaged with the programme, namely senior managers and enrolled
students, to derive findings that would inform managers, trainers and
educators, from a systemic implementation, faculty-development and
course-design perspective. Qualitative data gathered from face-to-face, semi-
structured interviews with a number of senior managers took place
delineated on the necessary enabling conditions to create, design and offer a
quality online programme. Quantitative data collected from a student
satisfaction survey examined factors contributing to perceived learning and
student satisfaction in an online learning context. Following thematic
analysis, five major factors were identified as critical for effective online
learning implementation, namely: Rationale and Motivating Factors,
Technology Infrastructure, E-Learning Pedagogy and Support
Infrastructure, Course Design and Delivery and Lessons Learned. Results
showed that Course Structure/Organisation, Learner Interaction, Student
Engagement and Instructor Presence appeared to be the major factors
contributing to high satisfaction and perceived learning. This paper argues
that the shift towards blended and online modes of learning is inevitable,
advocating the fundamental conception that online education is
instrumental in expanding access to tertiary education. If properly designed
and implemented, online education has the potential to alleviate academic
rigor through improved faculty productivity at reduced tuition costs.
Implications for further research and practical recommendations are also
discussed.
Keywords: course delivery; course design; higher education; online

education; perceived learning; student satisfaction; success factors
©Authors
140
1. Introduction
While the education culture across the Middle East values campus-based learning,
the shift towards blended and online modes of learning is inevitable. Despite
reported benefits in holding a repertoire of learning opportunities, the absence of
legal frameworks supporting recognition of online degrees is the biggest hurdle
towards embracing online education (The Open University, 2021). An
examination of the dominant features of education and academic institutions of
the third millennium shows that chief characteristics are “flexibility,
inclusiveness, collaboration, authenticity, relevance and extended institutional
boundaries” (Yuksel, 2010, p. 1). Responsibilities of learners and lecturers have
transformed significantly as educational goals have expanded to include self-
directed learning, digital literacy, continuous global dialogue, attainment of meta-
cognitive skills and processes that comprise holistic curricula, critical thinkers and
problem-solvers. Accordingly, higher education must undergo necessary changes
to adapt the traditional educational structures to the uprising knowledge age,
represented through the integration of information technology denoted by the
‘digital age’. This research advocates the fundamental conception that online
education is instrumental in expanding access to tertiary education, yet, has the
potential to alleviate academic rigor and standards through improving faculty
productivity at reduced tuition costs (Carey & Trick, 2013; Meyer, 2012).
Therefore, it focuses on an innovative online programme that is the first of its kind
launched in the nation, offered in a joint collaboration between three American
universities in Lebanon and Egypt, breaking down a number of cultural, social
and political barriers. In a country where online education is yet to be approved
and recognised, this programme is revolutionary for the Lebanese community
(Malaeb, 2020), capturing the attention of significant stakeholders and official
regulators, monitoring results and effectiveness, poising for further growth and
supporting the uprising of the nation. Therefore, the current research understudy
will focus on the Lebanese context.
What is more, is the fact that only a small number of studies have been carried out
in the Middle East to investigate the future prospects and barriers towards
implementing online education (Baytiyeh, 2017; El Turk & Cherney, 2016).
However, with the constant growth of the Web influences and changes in how
online courses are designed and implemented, continuation of studies of learners’
perspectives of online learning environments are needed to build effective Web-
based instruction that can optimise the learning experience within this ever-
changing landscape. This study seeks to add to the theoretical body of knowledge
and literature, the experiences of online education in Lebanon, the prospects and
challenges.
Over the past years, distance education has remarkably altered the landscape of
modern education, advancing significant changes in the offered learning
opportunities. A myriad of literature ascertains that online education can emerge
as a powerful socio-economic force in combating education inequalities and
poverty and driving economic development (Burns, 2017; Li & Lalani, 2020).
Despite reported benefits, institutional efforts to experiment with online
modalities of teaching and learning were frequently assessed by the government
141
(El Ghali & Nauffal, 2020). Accordingly, this paper reports on an innovative online
post-graduate engineering diploma programme that is considered to be
revolutionary for Lebanon, monitored by many significant stakeholders
(Baytiyeh, 2017) such as the Ministry of Education and Higher Education in
Lebanon. This is done to better understand the readiness and commitment of
students and the impact of the educational experience in terms of learning
effectiveness, academic integrity and student satisfaction. The success of the
programme will contextualise the country as a possible market for online
education and poises for further growth and possible approval of online
education. Accordingly, this study aims to present the first approved
postgraduate online diploma as a foundational stage for potential growth of the
industry in Lebanon. This mixed-method case study approach aims to answer the
following research questions:
- What are the main factors contributing to effective implementation of
online learning in the higher education sector in Lebanon?
- How satisfied were students with their fully online courses?
- What factors contributed to students’ satisfaction with online learning?
In an attempt to get on with global progress and support the current efforts, the
findings will be framed and presented as a model on the most appropriate mode
of implementation of online education in Lebanon.
2. Significance of Research
At times where the West has adopted and implemented online learning since long
time and is continuously experimenting with and trying out innovative models as
learning solutions, till now Lebanon does not approve online education and
applications for equivalency of online degrees are faced with refusal, despite how
prominent and reputable the graduating universities are (Baytiyeh, 2017; El Turk
& Cherney, 2016). A number of hindering factors towards implementation of
online education have been reported, among which is the lack of trust in
underlying teaching effectiveness and academic rigor, the absence of the Lebanese
Quality Assurance Agency to audit quality and abundance of higher educational
institutions, relative to a small country in geographical area like Lebanon (El
Amine, 2017). Therefore, it is not surprising that investments in e-learning in the
Middle East fall second to last behind Africa, equivalent to $683 million by 2016,
whereas North America ranks first with $23 billion worth of investments in the
same fiscal year (El Amine, 2017). The Ministry of Education and Higher
Education has put forward some efforts in embracing online education, such as
creating a taskforce of different expertise to draft a national strategy and create a
framework of action to recognise and accredit distance learning, yet the most
serious efforts remain from two leading American universities in Beirut and the
Arab Open University, a branch of the online learning network (El Turk &
Cherney, 2016).
Shaya (2018) argues that primitive systems continue to override the work of
higher education legislative academic decision-makers in Lebanon which is not
on par with advancements happening in the education system worldwide, hence,
denying institutions the chance to maintain a competitive advantage in a fast-
changing market and provide better quality and access to education. Those
142
universities which do not embrace online education will be left behind in the race
for globalisation and technological development. It is anticipated that the study
of the effectiveness of current online practices will bring about the quality of
online education and bring confidence among the regulatory authorities on the
need to accredit and approve this revolutionary mode of education on par with
the West. On the other hand, the perception offered by faculty members teaching
at leading institutions in Lebanon has the potential to expand awareness among
students and parents and lead to a change in the attitudes in perceiving the need
for online learning. Finally, and most importantly, the results of the study present
as an implementation model for private institutions on the best ways to deliver
quality online instruction in the most cost-effective ways that can meet learners,
market and business needs.
3. Theoretical Framework and Literature Review

The current research understudy is part of a larger stakeholder analysis study that
reports on the country’s readiness to accredit and implement online modes of
teaching and learning. The stakeholders are mainly higher education students,
faculty, executive leaders of private universities and the federal legislative
authorities. It is governed by the theoretical conception of Aparicio et al. (2016)
that perceives any successful e-learning system as a function of three main pillars:
e-learning stakeholders, e-learning technologies and e-learning activities. Hence,
this study presents the experience of the first postgraduate online programme as
a foundational ground to build upon, making use of the lessons learned, bringing
confidence in the capacity of private higher education institutions to carry out
such innovative models and unravel the prospects and challenges. These elements
guided the investigation seeking perceptions pertaining to faculty, students and
developers as main stakeholders, the e-learning team in particular, to deeply
understand the status of online learning technologies incorporated, the
instructional designers and faculty to report on pedagogy and online activities
and students to unravel the quality of learning and instruction versus
expectations.
The study is the first official postgraduate online programme to be offered in

Lebanon. Eighteen credits are required towards the completion of the diploma,
with quality of delivery and content sought to be equivalent to face-to-face
instruction, as indicated by the Ministry of Education and Higher Education in
Lebanon (Baytiyeh, 2017). Many internal policies and regulations govern the
diploma such as admission and registration procedures, duration of the study,
curriculum and syllabus. Hence, it is deemed necessary to document the
systematic implementation of the programme and assess student satisfaction.
4. Related Work
A number of international studies (Archambault & Crippen, 2014; Albrahim, 2020;
Basilaia & Kvavadze, 2020; Fauzi & Khusuma, 2020) have documented academic
rigor following online delivery. For instance, Shutimarrungson et al. (2014)
studied the implementation of the constructivist model in an online learning
course and results indicated that participants achieved high score achievement
and high critical thinking skills on post-tests. Studying the nature of the learning
143
experience it was revealed that the activities challenged the learners to develop
higher-order thinking skills and encouraged them to connect with former
experience and respond to the diversity of the learning needs, which lead to
authentic learning.
This study, among many other studies in literature, supports the conception that
pedagogy shapes the efficiency of online programmes, and if properly
implemented can lead to significant achievement gains and better learning
outcomes. Scrutinising the most appealing pedagogical features in online
education, the literature reveals the following factors. First, teachers serve
primarily as guiders and facilitators of learning, not instructors. Learning is
learner-centred (Brookfield, 1995; Chen, 1997; Huang, 2002; Schell & Janicki, 2012;
Spitzer, 1998; Wagner & McCombs, 1995). Second, learning takes the form of real-
world scenarios emphasising authentic learning (Carwile, 2007; Doolittle, 1999;
Jonassen et al., 1994; Koohang et al., 2009). Third, the social presence of students
by way of incorporating negotiations, discussions and debates should also be
considered (Chickering & Gamson, 1991; Chickering & Ehrmann, 1996). Fourth,
content should be meaningful and students should build on their pre-existing
knowledge. Fifth, students should be prompted to hold themselves accountable
for their learning, hence, to become self-mediated and self-aware, given that a safe
environment for questioning and learning is provided. Finally, teachers should
provide and encourage multiple perspectives and representations of content
(Chickering & Gamson, 1991; Chickering & Ehrmann, 1996). On the other hand,
upon reviewing the literature, the trends for 2017 to 2018 (Jasmini, 2017) appear
to be:
- Contextualised learning, represented through modernisation of current e-
learning system designs and approaches.
- Two-way conversation in e-learning, where learners’ needs will inform
content rather than available resources or classical approaches. For instance,
an ‘e-learning authoring’ tool could be used to fetch needs and elicit responses
through sending out polls and questionnaires that will then in turn help to
shape strategy and content.
- Better use of data, where option such as ‘Elucidat’ can allow viewing analytics
that would help in realising pitfalls and draw plans for personalising and
improving learning content.
- Extensive use of videos in terms of social learning and video learning in online
courses. Options like ‘Elucidat’ and ‘Periscope’ provide a variety of features
to build interactive videos and for better use in designing content.
- Microlearning will be used to personalise e-learning content.
- Social e-learning or ‘informal learning’ provided through e-learning
experiences.
- Mobile learning will continue to rise and learning content will be more
accessible through mobile devices.
- Performance support vs learning experience.
4.1 The Growing Market of E-Learning in the Middle East

The 21st century in the Middle East has witnessed growing interest in e-learning
through considerable investments in innovation leading to better e-learning
144
solutions, while the latter market revenue has risen to $560.7 million by 2016
(Edarabia, 2016). The e-learning and online education market was valued at $558.1
million in 2016 and is expected to reach $237.1 million by 2023. The Kingdom of
Saudi Arabia holds the largest shares in the Middle East and North African
(MENA) market, with expectations of earnings reaching $237 million by 2023 as
well. As a result, the Middle East Online Education and E-Learning Market Size,
Demand, Opportunity and Growth Outlook 2023 (Research and Markets, 2017)
report has been issued from Dublin to reflect and accompany the spanned growth.
The report presents historical market data for the previous year, and at the same
time reflects revenue estimates and forecasts till 2023. The scope includes market
trends, strategical management and development issues and so on.
Internationally, recruitment for online education has achieved numerous gains
and progress, with a reported enrollment of 308,000 at the University of Phoenix,
leader of online learning in higher education, 78,000 at Kaplan University, and
about 62 universities offering at least one course on Massive Open Online Courses
(MOOCS) platforms, including Stanford (Bonvillan & Singer, 2013).
4.2 The Programme Under Study
In October 2012, the main hosting university in Lebanon received a grant from the
European Union to offer an online joint/dual professional diploma and a degree
in green technologies (Uni-Med, 2021). The project originally aimed at widening
access to Information and Communication Technology ICT through incorporating
the use of new technologies in teaching and learning.
Three main aims resulted, with the first the development of ‘Visio-conferencing’,
then online courses and an online joint postgraduate diploma.. Three leading
American universities in Lebanon and Cairo partnered in offering an international
diploma. The funding process was completed by August 2016 and the diploma
has currently three main specialisations, namely renewable energies, green
buildings and water resources. The project aimed at developing programme
structure and curriculum for a postgraduate degree in green technologies,
creating a platform for the development of blended learning and a training
faculty.
5. Methodology
Charmaz (2006) defines grounded theory as “an inductive, comparative
methodology that provides systematic guidelines for gathering, synthesising,
analysing and conceptualising qualitative data for the purpose of theory
construction” (p.2). Hence, an inductive grounded theory methodology has been
utilised in the current understudy aiming at developing theoretical models with
high levels of rigor. Therefore, the research design covered two main stages:
(i) Qualitatively identifying the main themes, through thematic analysis, that
contribute to learning effectiveness from programme design perspective, to
delineate on the necessary enabling conditions to create, design and offer a
quality online programme. Particularly, the opinions offered by the managing
director of the programme, teaching faculty members, the e-learning team
supporting the programme comprising the general manager, instructional
designers and multimedia designers, were instrumental.
145
(ii) Quantitatively assessing the factors contributing to student satisfaction and

perceived learning.
Data was collected primarily through document analysis, interviews and surveys.
Fifteen audio-recorded semi-structured interviews were carried out with
professionals holding senior positions in the design, development and
implementation stage of the programme. The subjects’ qualifications ranged
between Masters’ and PhD holders who have worked closely on developing the
programme. Four of the subjects were senior college Deans and a total of twelve
were faculty with ranks ranging between assistant and full professors. Ten were
females and five were males. In addition, among the interviewees were senior
instructional designers and IT specialists. Interviews were first piloted, where
interview question guides and procedures were verified and cross-checked by
two academicians who are experts in the field, at the rank of full professor and
associate professor, to ensure question rigor through avoiding any potential
ambiguity, bias or leading questions. Each interview lasted between 60 and 90
minutes. There were 10 to 17 questions based on each interviewee’s position and
the interview questions were categorised under main themes in an attempt to
facilitate data analysis. A qualitative approach was adopted to identify emerging
factors under the assigned main themes and finally data was scrutinised and
interpreted iteratively leading to formation of codes and themes. Subjects’
responses were organised in a table form design comprising each subject’s code
name, main themes, codes revealing subjects’ individual perceptions and
corresponding quotes. Then, thematic analysis was utilised to look for patterns
and themes across datasets. Findings took the form of locating emerging themes
and sub-themes, where emerging codes were considered as potential themes in a
manner where various codes merge towards forming dominating themes.
On the other hand, for the purpose of data on student satisfaction, this research
study employed the Student Learning and Satisfaction in Online Learning
Environments (SLS-OLE) survey that was developed by Gray and DiLoreto
(2016), which has been proven for validity and reliability. A convenience sampling
approach was followed. A five point Likert scale for the questionnaire was used,
prompting students to assign their level of satisfaction with the programme. The
questionnaire was circulated electronically to the students enrolled in the online
diploma programme. A total of sixty responses were collected, representing the
majority of enrolled population with a 60% response rate , given that it is a new
and unique programme. Data was analysed using SPSS to generate descriptive
statistical results while being continuously cross checked to verify validity of
responses. The questionnaire was first factor analysed, with minimal loading
cutoff score to be  0.5 (Suliman 2001). The factors were successfully loaded
scoring 0.59 and above on the varimax rotation. Knowing that the determinant of
the R-matrix should be greater than 0.00001 (Field 2005), the determinant in this
study was found to be .002, that is larger than the necessary value. The value of
KMO is .733 meaning that factor analysis results in reliable factors (2005), and the
Bartlett test appears to be significant with value .03, and accordingly factor
analysis yielded reliable findings. Given the scarcity of available information as
the diploma is the first of its kind in the country that is online and approved, and,
while the whole nation has their eyes open, namely regulatory authorities and
146
different university councils, to understand the pre-requisites for successful

implementation and lessons learned, every single response will be of great value
and worth. Many studies acknowledge the importance of the online diploma
programme understudy as a European funded project that will revolutionise the
education system in Lebanon, hence, it is deemed important.
6. Demography of Respondents in Quantitative Data

A total of 60 students responded with meaningful answers on the survey. Around
70% of them are of the age group 24 to 34 years old, one is below 24 and the
remainder are above 35. The majority of the respondents were married and were
males, around 82% and 75% of the total respectively, and the remaining were
unmarried. As expected, the largest group lead full time jobs (88%), whereas the
rest were equally distributed as either working a part-time job or were
unemployed by the time data was collected. All respondents reported CGPA
between 2.00 and 4.00, and three quarters of the group scored above 3.00 and the
remainder below this. Finally, 82% were of Lebanese nationality, while 18% were
non-Lebanese.
7. Qualitative Findings on Implementation of Programme

Five main factors emerged as important to the implementation of the online
learning programme in Lebanon, namely:
- Rationale and Motivating Factors
- Technology Infrastructure
- E-Learning Pedagogy and Support Infrastructure
- Course Design and Delivery
- Lessons Learned
A- Rationale and Motivating Factors
Given the European project funding, the initial aim of the project was to develop
a number of online courses that would lead to a certificate. However, the desire
was high and findings from feasibility studies brought about a niche in the market
requiring postgraduate diploma studies in green technologies, and accordingly
the decision was taken.
RR: There was no initial intention to develop a diploma, rather online
courses in green energy, but aspirations were high and market research
showed that there was a niche to develop this into a diploma.
The leading American university hosting the programme has been offering
blended learning delivery for 10 years, since 2008, and has the capacity through
its e-learning team comprised of 12 experts to develop quality online modules.
Three universities embraced the project, namely three American universities, two
in Lebanon and another in Cairo. The international partners helped the three main
universities in turning the fragmented parts into one full programme, which is
offered completely online, to be the first in Lebanon and some of its neighbouring
countries.
RR: The main leading university has an advantage because they have a
small group of specialised people instructional designers to help in
assessment and testing in online education, do frequent testing rather
than one midterm. They helped the faculty.
147
The diploma programme was launched even before the funding was complete.
Presently, more than 27 online courses have been designed and delivered, 90% of
students are new to the world of online education, students allocate at least 5 to 7
hours per week to study per course, and they are residents from different parts of
the world. Accordingly, the regulatory authority has approved the programme
and is is aware of the success of the online diploma programme
RR: Our degree is almost equivalent to Masters’ courses, and the ministry
is very attentive to it because we are the first.
Offering a post-graduate online programme extends far beyond the online
diploma programme for the main leading American university, where shifting to
online modes of delivery is part of the university’s strategic planning. The
university aims towards increasing the international student body and to increase
student enrollment by half. Given available space and resources, going virtual is
therefore one element of the Master Plan. Therefore, solid efforts are placed on
intensifying e-learning resources, such as reliable cloud hosting, an e-learning
platform and learning management system.
WW: Our strategical plan is to increase enrollment by 50% without
increasing the space, without increasing facility, and the only way is to
go virtual. This one of the items in the Master Plan. Therefore, we are
placing extensive efforts in our e-learning resources, on reliable cloud
hosting stable, even though I think we have accomplished a success story
since 2005.
B- Technology Infrastructure
From an operational perspective, the programme functions along with three main
components: admission, registration and learning management system. Moodle
and Blackboard are the main learning management systems utilised. To overcome
the problem of slow and low quality of Internet service, the leading university had
a British partner offering an external hosting cloud in addition to the Moodle
partner.
WW: Our institution has a team dedicated to implementing online
courses, we contacted different companies, looked for Moodle partners, not
only hosting cloud, so we took a SAS, software as a service, we contacted
different companies and we selected a partner in UK.
All the implementation of Moodle happens locally at the leading university,
where every semester the online diploma programme has 10 to 15 courses running
through Moodle, that are reliable and efficient, with no incidents registered such
as hacking and server problems. The e-learning team acknowledges the
importance of training for students, where upon enrollment, new students receive
intensive training either through Skype or Adobe Connect. The support team is
available almost all the time and issues are usually resolved within two hours.
C- E-Learning Pedagogy and Support Infrastructure
The concept of integrating technologies into teaching and learning is not new at
the main leading American university, rather an extension of efforts that stretched
for several years back. Training sometimes took the shape of a full programme
that would last many weeks, on best practices, reading content, then finally
148
authoring their own courses that would meet certain objectives. At least twelve
different faculty members are called for training every semester.
Regarding the online diploma pedagogy, the fundamentals of online pedagogy
are well governed. A number of specialised instructional designers, who are
experts in the field of online education, are on the board. The designers’ duties, in
collaboration with multimedia designers and e-learning system managers, are to:
- Train faculty members in using e-learning technologies and associated
pedagogies prior to any activity.
- Through one-on-one and collective group meetings, to closely assist faculty
members in desigining their courses, then to transform them into online
learning modes.Test the courses and verify their quality prior to publishing
and delivery. In that case, many courses were turned down because they were
not cleared by the instructional designers.
- Follow up on faculty members during the semester.
Custom-designed training sessions, in addition to one-on-one support, is offered
to every new faculty member joining the diploma for three to four months,
whereby training and designing of the course takes place. In essence, almost three
quarters of the course should be ready prior to offering it up for study. The
instructional design team helps in finding the best methodologies for teaching the
course, relying on the most suitable learning activities, graphics and assessment,
and accordingly the syllabus might need to be modified and other types of
interactive engagement assignments included.
HH: Every time new faculty joining, individual work of 1-1 for about 3 –
4/months to train and design a course takes place.
MM: They come with topic learning outcomes and syllabus, we assist
them in methodologies of teaching, design activities and learning
modules, we help to choose learning activities, assessment, etc. They write
the preliminary syllabus and textbook then we work on redesigning
syllabus to develop appropriate assessment, as they are used to classical
quizzes and exams, whereas we train them on different kinds of
assessment and interactive engagement documents.
With time faculty members become proficient in online learning systems, where
members reported better student evaluation reports and general internal
satisfaction after teaching online courses for some time.
D- Course Design and Delivery
Reflections on course design and delivery were brought about through the
perceptions of faculty members teaching online courses in the online diploma
programme. The first faculty member is an associate professor in Engineering,
with teaching experience of more than 10 years. He has been engaged in a blended
learning approach at the leading university since 2013 and received a total
training of 10 to 12 hours after joining the online diploma programme. The
workshops were just a start, rather, it is the on-the-job training and exposure that
reinforce the necessary skills to deliver high quality teaching. He teaches two
courses, each two credit hours, and each requires around nine to 10 sets of lectures
each lasting for about 40 minutes on average.
149
II: For each of these courses there is like 9 or 10 set of lectures, about 40
mins each.
II: It is the OJT, on job training, when you start actually experimenting
with online, there is a lot of learning that takes place.
Content development happens locally, where the faculty member gets the
syllabus with learning outcomes along with assigned topics. Then instructional
designers will undergo some modifications on instructional strategies, replace
paper books with e-books and alter assessment to be quiz-based into a set of
different assessment techniques such as position paper, presentation, critique and
others.
With the help of instructional designers, the faculty prepares the material using a
PowerPoint Presentation with very interactive slides, supplemented with videos,
graphics, etc. Then with the help of multimedia specialists, voice over is recorded,
then through a e-learning technologies manager, the slides would be turned into
online material using specialised software. A simple link would allow students to
download and access the slides.
Regarding assessment, the faculty confirms that conducting online quizzes
remains a problem in the diploma programme as it is difficult to verify the identity
of the user taking the quiz and to check that cheating is not happening. However,
what helps is the nature of the candidates who are pursuing the diploma for career
advancement and self-satisfaction, hence any kind of plagiarism wouldn’t help
them. In all cases, minor weight is assigned to online quizzes (20%) and the main
weight is on assignments. The faculty gives about eight assignments every
semester, in the form of mini-projects. They are asked to undertake some reading
of a material or research/paper and then a report and exercises are then shaped
so that students will have to comment on each other’s work. Eventually, they are
learning and critiquing and students are interacting where questions go back and
forth between them, while the faculty members monitor the interaction and
interfere only when required. An assignment that initiates dynamic discussion
and keeps the students engaged would help in meeting the learning outcomes,
rather than the grade itself.
Issues like monitoring attendance are not hard, as the system allows the faculty to
keep track of the activity of the student, slides visited, how much time is spent
and when it is necessary to initiate and contact the student regarding attendance.
E- Challenges Faced and Lessons Learned
Findings were obtained from different interviewees on the lessons learned over
the three years since the diploma was offered at the university, as follows:
(i) Offering online degrees (or diplomas) by any reputable university takes long
extensive effort, and as one senior executive mentioned “it might look like
something small, but the effort is like introducing a PhD programme. The
difficulty lies in meeting local, regional and international registration and
accreditation of enrolling universities, which is almost equivalent to
reaccreditation”. Hence, from an administrative perspective, serious
commitment is required.
(ii) GG: We worked on the administration to make sure the team in our university is doing
its part in admission and registration, it might look something small, but the effort is
150
like introducing PhD program, getting IDs, online tuition fees, everything online, to
get all this done banner oracle financial system huge amount of money.Despite its
operational difficulties, the online diploma is very profitable. It has achieved
profit from the first day of launching, indicating a good market for the online
diploma, although it is typically new.
(iii) GG: This is a diploma that made money from the very first semester, it actually made
profit, making good money for all the 3 partners. Tuition fees is basic, very affordable,
the way we designed it, the way we shared our resources, the university charges
overhead and yet we are still making money. It is a very good opportunity.The
presence of strong technological infrastructure is essential to ensure smooth
automated progression between admission, registration and course
navigation.
WW: All transitions between the three units Admission, Registration and
Moodle is automated, and the units are from the leading university staff,
who are experienced with more than 15 years of experience to increase
chances of success, hence there should be a reliable hardware and software
information technology.
(iv) The presence of specialised instructional designers in abundance is a must,
where their main role is faculty development and to provide course design
support. Training faculty members might seem brief, however, despite how
knowledgeable they are, intensive training on teaching methodologies,
formulation of their teaching identity and proper follow up on the students is
hard to reach.
(v) Ample time of no less than three to four months should be given to prepare,
test and pilot online courses before they are up and running. On average 20 to
30 slides will require 18 hours of extensive work to prepare, then three hours
for audio -recording and two hours for post-production, i.e. publishing and
posting. In addition, expectations should be set from the beginning for faculty
members regarding due dates, deliverables and consequences if the
commitment isn’t fulfilled.
HH: Give the course enough ample time, minimum 3-4 months of
preparation and testing, we pilot it user-testing, we do that with our
media specialist. It takes long to develop interactive quality lectures.
(vi) Faculty members’ social presence is very important and is underestimated,
otherwise there is a risk of isolation of learners and low completion rates of
courses. Online learning students need to continuously feel the presence of
faculty members.
(vii) A reliable auditing system helps in controlling quality, where auditors are
assigned to each online course and conduct satisfaction and progress surveys
twice in a semester, in the middle of the semester and towards the end.
DD: Each course has assigned auditor. They conduct surveys middle of
the semester and by the end. Auditors make sure learning outcomes are
met, and whether students are properly progressing.
(viii) Carrying out live sessions, whether for the purpose of synchronised
learning or meeting the students in chat rooms and getting to know them
better, appears to be effective in strengthening the faculty member-learner
bond.
(ix) The personal effort required from a faculty member to teach an online course
is by far greater compared to face-to-face instruction. Hence, monetary
151
rewards as extrinsic factors will help, however, the faculty member should
have intrinsic motivational factors to ‘buy-in’ to online courses and diminish
resistance and reluctance.
KK: Yes, a lot, I am suffering in the diploma, though it is my second year
of teaching online. Each semester I am changing the course. The courses I
am teaching here are not introductory, rather advanced and not available.
So, I have to always search for ways to make students very interested in
the subject. It is nice, but very challenging. Switching online by itself is
not hard but switching and making it interactive and interesting is not
easy.
(x) Not all students are digital natives and ‘tech-savvy’.
YY: My lessons learned that never assume that students are digital
natives. Many times we take it for granted they feel at ease with
technology, not all students do. We assume, if we give them any online
platform they will naturally figure out what to do, that’s not the case.
Whether we like it or not there is a digital divide, especially in Lebanon.
8. Results of Student Satisfaction Survey

The questionnaire aims to assess student learning and satisfaction in the first and
only accredited online programme in Lebanon, offered by partnering American
universities in Lebanon and Cairo. Around 100 students were enrolled in total by
the time the study was conducted.
The students’ learning and satisfaction were measured based on six dimensions,
namely: Course Structure/Organisation, Learner Interaction, Student
Engagement, Instructor Presence, Student Satisfaction and Perceived Learning,
and comprising 34 questions. The results are as follows:
A- Course Structure/Organisation:
Students’ rating of the organisation of the course and structure yielded a mean of
4.15 and a standard deviation of .65. Strength appeared in clear statement of
learning outcomes, ease of navigation and organised layout of course, where
means were 4.25, 4.31 and 4.35 respectively. Moderate satisfaction was in the
articulation of clear instructions on student participation and alignment of
learning outcomes with activities.
Table 1: Course Structure/Organisation Results
A- Course Structure/Organisation Mean SD
During your diploma courses,
1 student learning outcomes was 3.9375 0.57
aligned to the learning activities.
2 Course navigation was logical. 4.3125 0.79
The layout of the course was
3
organised. 4.4375 0.63
Instructions about student
4 participation were clearly 3.8125 0.65
presented.
The purpose of the course was
5 4.25 0.57
clearly presented.
Total 4.15 0.65
152
B- Learner Interaction:
The overall rating of student satisfaction on interaction was moderate to high with
total mean of 3.83. Items such as on opportunities to introduce oneself and the
chances of engaging in active participation rated relatively high, but the rest of the
items rated moderate with the least pertaining to exchange of peer comments. In
general, social interaction scored moderate to high.
Table 2: Learner Interaction Results
B- Learner Interaction Mean SD
I frequently interacted with other
6 3.97 0.81
students in the course.
There were opportunities for active
7 3.94 0.57
learning in this course.
The learning activities promoted
8 3.88 0.95
interaction with others.
I had the opportunity to introduce
9 4.72 0.79
myself to others in the class.
I communicated often with other
10 3.89 1.01
students within the course.
I regularly communicated with the
11 3.07 0.92
instructor of the course.
I received ongoing feedback from
12 3.00 0.81
my classmates.
Total 3.84 0.84
C- Student Engagement
The whole factor as a total scored moderate with a mean of 3.1125 and standard
deviation .90. The mean in the below table indicates that the least scoring item was
the learner-instructor interaction with a low mean of 2.88, and the highest was in
the learner-content interaction with a high mean of 4.19, implying dedication and
commitment from the students’ side, but lack of engagement from the faculty
members’ side.
Table 3: Student Engagement Resul ts
C- Student Engagement Mean SD
I frequently interacted with my
13 2.88 0.80
instructor of this course.
I discussed what I learned in the
14 3.63 0.88
course outside of class.
I completed my readings as
15 3.69 0.94
assigned during the course.
I participated in synchronous
16 and/or asynchronous chat 3.32 0.79
sessions during the course.
I was actively engaged in the 4.19 0.75
17
activities required in the course.
Total 3.55 0.89
153
D- Instructor Presence
Student rating on instructor presence in total was 3.2 indicating moderate
satisfaction levels. The five items measuring this factor rated between low (mean
2.0) to moderate and moderate-high (3.7). Satisfaction on items pertaining to
instructor feedback from assignments was the highest, however, the quality of the
comment was low. The feeling of being properly followed up on by the instructor
also rated moderate in satisfaction.
Table 4: Instructor Presence Results
D- Instructor Presence Mean SD
The instructor’s feedback on
18 3.82 0.40
assignments was clearly stated.
The instructor's feedback on
19 2.07 0.57
assignments was constructive.
The instructor provided timely
20 feedback about my progress in the 3.25 0.85
course.
The instructor cared about my
21 3.19 0.83
progress in this course.
I learned from the feedback that
22 3.69 0.60
was provided during the course.
Total 3.2 0.65
E- Satisfaction
The overall satisfaction of the e-learner who took the survey was high with a mean
of around 4.00. They appeared to be highly satisfied with the learning content and
would recommend it to their peers. Less satisfaction was in the instructor
himself/herself and student-student interaction.
Table 5: Student Satisfaction Results
E- Student Satisfaction Mean SD
I am satisfied with my overall
23 3.98 0.71
experience in this course.
I would recommend this course to
24 4.57 0.81
other students.
I am satisfied with the level of
25 student interaction that occurred in 3.97 1.09
the course.
I am satisfied with my learning in
26 4.00 0.73
the course.
I am satisfied with the instructor of
27 3.97 0.51
the course.
I am satisfied with the content of
28 4.00 0.36
the course.
Total 4.05 0.65
154
F- Perceived Learning
The total mean of items on students’ perceived learning was 4.00. High scoring
appeared in satisfaction with learning, learning tasks and understanding of
content, acquisition of skills for their future and career. Learning, however, was
not much better than what was anticipated.
Table 6: Perceived Learning Results
F- Perceived Learning Mean SD
I am pleased with what I learned in
29 4.17 0.44
the course.
The learning tasks enhanced my
30 3.94 0.44
understanding of the content.
I learned more in the course than I 1.19
31 3.99
anticipated.
I learned skills that will help me in
32 4.07 0.57
the future.
The learning activities promoted
33 the achievement of student 3.75 0.57
learning outcomes.
The course contributed to my
34 4.125 0.88
professional development.
Total 4.01 0.68
9. Discussion of Findings
Findings revealed that five main factors emerged as important to the
implementation of the online learning programme in Lebanon, namely: Rationale
and Motivating Factors, Technology Infrastructure, E-Learning Pedagogy and
Support Infrastructure, Course Design and Delivery and Lessons Learned. The
rationale behind offering the online diploma programme was significant in better
understanding the current context of universities in Lebanon. It brought forth
serious attempts to consider online education as a strategic objective and current
experimentation to reach ultimate successful modes of implementation. In their
widely spread work, “Barriers to Adoption of On-Line Learning Systems in US
Higher Education,” Bacow et al. (2012) highlighted four main rationales among
American universities to offer online courses and programmes. The leading
American universities in Lebanon are in line with two of the aforementioned
rationales, namely “the desire to generate new revenue streams by reaching
students who wouldn’t otherwise enroll in traditional programs” (p. 9) such as
international students, and “responding to space constraints” (p. 12) through
avoiding expanding or creating expensive additional facilities. On the other hand,
improving learning outcomes and increasing retention didn’t surface in the
conducted interviews as Lebanon is still far from considering online education as
a viable option that can generate better learning outcomes and hence become a
strategic objective. The presence of online education in the leading university’s
Master Plan provides clear support at the level of leadership and belief in the
value of online learning. The diploma programme places a major focus on the
social presence of faculty members, reinforced through high levels of student–
faculty interaction, which is another attestation of the close follow up of the
155
managing team on the progress and satisfaction of students as it is highly

correlated to learning and retention rates (Drouin et al., 2015; Govindesamay,
2012), and a fundamental pillar of online education best practices (2015). The
team’s fear of isolated and unhappy learners, hence, calling for more social
interaction, concurs with leading universities’ attempts, such as SUNY (Pickett et
al., 1999). Analysis revealed high technology infrastructure and technical
capacities among addressed institutions where the managing team was able to
overcome all anticipated problems prior to launching the programme, revealing
the operational feasibility of implementation. Among the major contributing
factors is the resourcefulness of the leading university, equipped with all the
needful resources, from qualified faculty, strong technology capacity that was
able to link the processes of admission, registration and course offering among the
three universities, to open educational resources with a helpful digital library and
experienced instructional designers.
The programme was not only able to overcome perpetuated challenges, but also
documented barriers in Lebanon pertaining to inconsistent Internet access and
poor network (El Turk & Cherney, 2016), doubts in the feasibility of
implementation (Nasser & Abouchedid, 2005), resistance among faculty members
to teach online (Tarhini et al., 2013), general doubt in perceived usefulness (Nasser
and Abouchedid, 2005), and so on. Four main factors contributed to high student
learning and satisfaction, namely: Course Structure and Organisation, Learner
Interaction, Student Engagement and Instructor Presence. These results are in line
with SUNY findings, where the main factors impacting satisfaction and learning
effectiveness could be summarised as: faculty-student interaction, peer
interaction, social presence and resolving individual difficulties through proper
course structure and strong technical infrastructure. Findings reveal that the
Lebanese higher education system is on the right track towards paving the way
for more creative, innovative and advanced growth in the field of online
education in Lebanon. And as the e-learning team manager said during the
interview, “we don’t wait for the ministry of education to come to us, we go to
them.”
10. Strategical Recommendations on Modes of Implementation

This research reported on the prospects and barriers in designing and
implementing an innovative online learning model in the Middle East that proved
to be attractive, not because of the machine-guided online instruction element,
rather due to the potential of offering improved learning outcomes while bending
the cost curve, poising for the growth in the industry. As leaders are
contemplating the future of education, particularly given the rapid increase in
adoption of online technologies worldwide, the below presents recommendations
for universities aiming at developing online learning programmes in the near
future.
(i) The presence of strong technological infrastructure is essential to ensure
smooth automated progression between admission, registration and course
navigation.
156
(ii) The presence of specialised instructional designers in abundance is a must,

to extensively train faculty members, design faculty development
programmes and provide course design support.
(iii) Ample time of no less than three to four months should be given to prepare,
test and pilot online courses before publishing. On average 20 to 30 slides
will require 18 hours of extensive work to prepare, three hours for audio
recordingand two hours for post-production, i.e. publishing and posting. In
addition, expectations should be set from the beginning for faculty members
regarding due dates, deliverables and consequences if the commitment isn’t
fulfilled.
(iv) Faculty members’ social presence is very important and is underestimated,
otherwise there is a risk of isolation of the learner and low completion rates
of courses. Online learning students need to continuously feel the presence
of faculty members.
(v) A reliable auditing system helps in controlling quality, where auditors are
assigned to each online course and conduct satisfaction and progress
surveys twice in a semester, in the middle of the semester and towards the
end.
(vi) Carrying out live sessions, whether for the purpose of synchronised learning
or meeting the students in chat rooms and getting to know them better,
appears to be effective in strengthening the faculty member-learner bond.
(vii) The personal effort required from a faculty member to teach an online course
is by far huge compared to face-to-face instruction. Hence, monetary
rewards as extrinsic factors will help, however, the faculty member should
have intrinsic motivational factors to ‘buy-in’ to online courses and diminish
resistance and reluctance.
(viii) Student readiness may seem high, but not all students are tech savvy, hence,
students’ development programmes are a must.
11. Conclusion, Limitations and Future Research

This study aims to present the first approved postgraduate online diploma as a
foundational stage for potential growth of the industry in Lebanon. It explores the
main factors leading to effective implementation of online program in the higher
education sector in Lebanon, students’satisfaction with online learning, and
contributing factors to high satisfaction and perceived learning. It has brought
forward findings that are critical in the Lebanese context as it presents a departure
from the current pervasive classical modes of delivery, overcoming many
obstacles documented in literature. Despite known challenges relevant to poor
Internet and power quality, lack of funds to finance such projects and
fundamental reluctance to teaching online courses not owned by faculty, the
mentioned online programme was able to highlight and capture official and
public attention. Moreover, despite public doubt regarding the feasibility of
implementation of any online programme, the mentioned online programme
showed high levels of satisfaction and perceived learning among e-learners. Thus,
it could be argued that implementation of online education at undergraduate or
post-graduate level is feasible. The rationale and motivation factors, lessons
learned from the initial developmental and implementation phases along with the
student satisfaction survey will present as a critical report on what has been
157
achieved in online education at reputable universities in Lebanon. Findings

revealed that five main factors emerged as important to the implementation of the
online learning programme in Lebanon, namely: Rationale and Motivating
Factors, Technology Infrastructure, E-Learning Pedagogy and Support
Infrastructure, Course Design and Delivery and Lessons Learned. A number of
factors appeared to contribute to the high student learning and satisfaction,
namely: Course Structure and Organisation, Learner Interaction, Student
Engagement and Instructor Presence. Findings allowed for the proposal of
strategical recommendations on modes of implementation that would optimise
potential success pertinent to pedagogy, technology infrastructure, quality control
system and key stakeholders’ perceptions and readiness. Hence, this study
suggests that future research would further investigate online learning in
Lebanon in light of that. Moreover, this study recommends including
participation of a larger sample of subjects in the student satisfaction survey,
while qualitatively re-examining senior stakeholders’ opinions post Covid-19.
12. References
Abouchedid, K., & Eid, G. M. (2004). E‐learning challenges in the Arab world:
revelations from a case study profile. Quality Assurance in Education, 12(1), 15-
27. https://doi.org/10.1108/09684880410517405
Albrahim, F. A. (2020). Online Teaching Skills and Competencies. TOJET: The Turkish
Online Journal of Educational Technology, 19(1), 9-20.
Basilaia, G., & Kvavadze, D. (2020). Transition to Online Education in Schools during
a SARS-CoV-2 Coronavirus (COVID-19) Pandemic in Georgia. Pedagogical
Research, 5(4). doi:https://doi.org/10.29333/pr/7937
Fauzi, I., & Khusuma, I. H. (2020). Teachers’ Elementary School in Online Learning
of COVID-19 Pandemic Conditions. Jurnal Iqra’ : Kajian Ilmu Pendidikan, 5(1),
58-70. https://doi.org/10.25217/ji.v5i1.914
Aparicio, M., Bacao, F., & Oliveira, T. (2016). An e-Learning Theoretical Framework.
Journal of Educational Technology & Society, 19(1), 292-307.
https://www.jstor.org/stable/jeductechsoci.19.1.292
Bacow, L. S., Bowen, W. G., Guthrie, K., M., Lack, K. A., & Long, M. P. (2012). Barriers to
Adoption of Online Learning Systems in U.S. Higher Education. New York: Ithaka.
Basilaia, G., & Kvavadze, D. (2020). Transition to Online Education in Schools during a
SARS-CoV-2 Coronavirus (COVID-19) Pandemic in Georgia. Pedagogical Research,
5(4). https://doi.org/10.29333/pr/7937
Baytiyeh, H. (2017). The flipped classroom model: when technology enhances professional
skills. The International Journal of Information and Learning Technology, 34 (1), 51-62.
https://doi.org/10.1108/IJILT-07-2016-0025
Brookfield, S. (1995). Adult learning: An overview. International encyclopedia of education,
10, 375-380.
Burns, M. (2017). Online Education Business as a Tool to Boost Economy. Business
Community. https://www.business2community.com/tech-gadgets/online-
education-business-tool-boost-economy-01972787
Carey, T., & Trick, D. (2013). How online learning affects productivity, cost and quality in higher
education: An environmental scan and review of the literature. Higher Education
Quality Council of Ontario.
Carwile, J. (2007). A constructivist a roach to online teaching and learning. Inquiry, 12(1),
68-73. https://eric.ed.gov/?id=EJ833907
158
Charmaz, K. (2006). Constructing grounded theory: A practical guide through qualitative

analysis. Sage.
Chen, L. L. (1997). Distance delivery systems in terms of pedagogical considerations: A
reevaluation. Educational Technology, 37(4), 34-37.
https://www.jstor.org/stable/44428405
Center for Education Innovations. (2016).
https://www.educationinnovations.org/p/center-for-education-innovations-
cei
Chickering, A. W., & Ehrmann, S. C. (1996). Implementing the seven principles:
Technology as lever. AAHE bulletin, 49, 3-6
Chickering, A. W., & Gamson, Z. F. (1991). Appendix A: Seven principles for good practice
in undergraduate education. New directions for teaching and learning, 1991(47), 63-
69.
Doolittle, P. E., & Camp, W. G. (1999). Constructivism: The career and technical education
perspective. Journal of vocational and technical education, 16(1), 23-46.
https://eric.ed.gov/?id=EJ598590
Drouin, M., Stewart, J., & Gorder, K. V. (2015). Using methodological triangulation to
examine the effectiveness of a mentoring program for online instructors. Distance
Education, 36(3), 400-418. https://doi.org/10.1080/01587919.2015.1081735
Edarabia. (2016). Innovating Education Technology in the Middle East.
https://www.edarabia.com/115413/innovating-education-technology-in-the-
middle-east/
El Amine, A. (2017). Culture of law at Arab universities. Contemporary Arab Affairs, 10(3),
392-407. https://doi.org/10.1080/17550912.2017.1350368
El Ghali, H., & Nauffal, D. 2020. COVID19: Is Lebanon Ready for Online Higher
Education. https://www.aub.edu.lb/ifi/news/Pages/20200330-covid19-is-
lebanon-ready-for-online-higher-education.aspx
El Turk, S., & Cherney, I.D. (2016). Perceived online education barriers of administrators
and faculty at a US university in Lebanon. Creighton Journal of Interdisciplinary
Leadership, 2(1), 15-31. https://eric.ed.gov/?id=EJ1152181
Field, J. (2005). Social capital and lifelong learning. Policy Press.
Gray, J. A., & DiLoreto, M. (2016). The effects of student engagement, student satisfaction,
and perceived learning in online learning environments. International Journal of
Educational Leadership Preparation, 11(1), n1. https://eric.ed.gov/?id=EJ1103654
Huang, H. M. (2002). Toward constructivism for adult learners in online learning
environments. British journal of educational technology, 33(1), 27-37.
https://doi.org/10.1111/1467-8535.00236
Jonassen, D.H., Campbell, J.P., & Davidson, M.E. (1994). Learning with media:
Restructuring the debate. Educational technology research and development, 42(2), 31-
39. https://www.jstor.org/stable/30218685
Jasmini. (2017). Online Learning Statistics And Trends.
https://elearningindustry.com/online-learning-statistics-and-trends
Koohang, A., Riley, L., Smith, T., & Schreurs, J. (2009). E-learning and constructivism: from
theory to application. Interdisciplinary Journal of E-learning and Learning Objects, 5,
91-109. https://www.learntechlib.org/p/44824/
Li, C., & Lalani, F. (2020). The COVID-19 pandemic has changed education forever. World
Economic Forum. https://www.weforum.org/agenda/2020/04/coronavirus-
education-global-covid19-online-digital-learning/
159
Meyer, K. A. (2012). The influence of online teaching on faculty productivity. Innovative

Higher Education, 37(1), 37-52. https://link.springer.com/article/10.1007/s10755-
011-9183-y
Malaeb, A. (2020). Amid coronavirus, students forced online, but Lebanon won't recognize
online degrees. https://english.alarabiya.net/features/2020/12/11/Lebanon-
crisis-Amid-coronavirus-students-forced-online-but-Lebanon-won-t-recognize-
online-degrees
Nasser, R., & Abouchedid, K. (2005). Graduates' perception of university training in light
of occupational attainment and university type: The case of Lebanon. Education+
Training.
Pickett, A., & Shea, P. (1999). The evolution of faculty development and course design at
the SUNY Learning Network. In WebNet World Conference on the WWW and Internet
(pp. 1393-1394). Association for the Advancement of Computing in Education
(AACE). https://www.learntechlib.org/p/7721/
Research and Markets. (2017). Middle East Online Education and E Learning Market Size,
Demand, Opportunity & Growth Outlook 2023.
https://www.researchandmarkets.com/reports/4326517/middle-east-online-
education-and-e-learning
Schell, G.P., & Janicki, T.J. (2012). Online course pedagogy and the constructivist learning
model. Journal of the Southern Association for Information Systems, 1(1), 26-36.
http://dx.doi.org/10.3998/jsais.11880084.0001.104
Shaya, N. M. (2018). Adoption and Implementation of Online Learning Systems in Lebanon:
Prospects and Barriers (Doctoral dissertation, The British University in Dubai
(BUiD)). https://bspace.buid.ac.ae/handle/1234/1291
Shutimarrungson, W., Pumipuntu, S., & Noirid, S. (2014). A model of e-learning by
constructivism approach using problem-based learning to develop thinking skills
for students in Rajabhat University. Educational Research and Reviews, 9(21), 1165-
1172. https://doi.org/10.5897/ERR2014.1800
Spitzer, D.R. (1998). Rediscovering the social context of distance learning. Educational
Technology, 38(2), 52-56. https://www.jstor.org/stable/44428438
Suliman, A. M. (2001). Work performance: is it one thing or many things? The
multidimensionality of performance in a Middle Eastern context. International
Journal of Human Resource Management, 12(6), 1049-1061.
https://www.tandfonline.com/doi/abs/10.1080/713769689
Tarhini, A., Hone, K., & Liu, X. (2013). User acceptance towards web-based learning
systems: Investigating the role of social, organizational and individual factors in
European higher education. Procedia Computer Science, 17, 189-197.
https://doi.org/10.1016/j.procs.2013.05.026
The Open University. (2021). Annual Report. http://www2.open.ac.uk/about/annual-
report-2018-19/
Wagner, E.D., & McCombs, B.L. (1995). Learner centered psychological principles in
practice: Designs for distance education. Educational technology, 35(2), 32-35.
https://www.jstor.org/stable/44428961
Yuksel, U. (2010). Integrating curriculum: Developing student autonomy in learning in
higher education. Journal of College Teaching & Learning (TLC), 7(8).
https://doi.org/10.19030/tlc.v7i8.138
160

Vol. 20, No. 8, pp. 160-175, August 2021
Received Apr 29, 2021; Revised Jul 22, 2021; Accepted Jul 31, 2021
Autonomy and Creative Thinking Skills of

Prospective Elementary School Teacher Students
in Learning Mathematics with Science Phenomena
assisted by the Learning Management System
Suprih Widodo
School of Post Graduate, Universitas Pendidikan Indonesia, Bandung, Indonesia
https://orcid.org/0000-0003-2709-4804
Turmudi
https://orcid.org/0000-0001-7976-211X
Rizki Rosjanuardi
https://orcid.org/0000-0001-5280-6901
Abstract. This research intends to explain the autonomy and creative

thinking skills of prospective elementary school student teachers in
mathematics with the learning management system (LMS)-assisted
science phenomenon. The data collection was done by using test
instruments in the form of essay questions to capture the creative thinking
skills and a non-test in the form of a questionnaire was used to capture the
autonomy skills of prospective elementary school student teachers. The
data were analysed through several stages, namely data reduction, data
presentation, data conclusions, and data validation using triangulation.
The result of the research shows that using the LMS can improve the
creative thinking skills of prospective elementary school teachers, while
their autonomy skills in experimental classes and control classes indicate
no difference. This happens because students were given the same
opportunity to explore the material independently. Thus, a way to
develop autonomy through mathematics learning needs to be explored.
Keywords: autonomy; creative thinking skills; learning management

system; science phenomenon
1. Introduction
Principals in Indonesia have indicated that the new graduates of educators have
©Authors
161
low levels of competence (Widodo, 2016b). Teachers with teaching experience also
have low levels of pedagogical and professional competencies (Giarti, 2016). In
addition to having hard skills, the competence of elementary school teachers must
also include soft skills and 21st century skills (KA21) which are often referred to as
the 4Cs, namely critical thinking, collaboration, communication and creativity that
are useful in their personal and professional lives (P21, 2011). In mathematics
learning, the Partnership for 21st Century Skills also defines the skills of the 21st
century through a 21st century skill mathematics map in the field of learning and
innovation skills for K12 consisting of creativity, critical thinking, communication
and collaboration (P21, 2011).
Based on the literary search, not much research in mathematics learning has been
done related to the development of 21st century skills. Nevertheless, Salehudin et
al. (2015) explained that through mathematics learning educators can provide
support for students to master the skills of the 21st century. Mathematics learning
research in Indonesia and some countries is still focused on mathematical skills as
proposed by the National Council of Teachers of Mathematics (NCTM) (2000a).
These skills consist of five process skills that students must master through
mathematics learning, namely (i) problem solving (Mushlihuddin & Nurafifah,
2018; Maslukha et al., 2018; Fuchs et al., 2004); (ii) reasoning and proof (Lestari,
2019; Kramarski & Mevarech, 2003); (iii) connections (Nabilah, 2019; Fauzi, 2015);
(iv) communication (Fauzi & Priatna, 2019; Brendefur & Frykholm, 2000;
Buchholz, 2005) and (v) representation (Effendi, 2012).
Two research studies in mathematics learning related to 21st century skills have
been conducted by Turmudi et al. (2017) and Afifah (2019). In the research by
Turmudi et al. (2017), they applied mathematics learning with didactic science to
develop 21st century skills consisting of critical thinking skills, creativity,
communication and collaboration. Afifah (2019), on the other hand, developed the
21st century skills of grade VIII students in mathematics learning with project-
based learning. Another 21st century skills development was carried out by
Muhajir (2018) who trained 21st century skills in physics learning and other
experiments in Malaysia by testing several 21st century skills in the fourth grade
(Salehudin et al., 2015).
This research will discuss the development of creative thinking skills as one aspect
of the 21st century skills. Creative thinking skills are related to the skill of
discovering new things that have not existed before and are original, developing
new solutions to each problem, and involving the ability to generate new, varied,
and unique ideas (Chiam, 2014).
Mathematics as an important knowledge mastered by students can also

contribute to the development of 21st century skills. However, in reality, many
consider mathematics to be an area of knowledge that is often regarded as a
difficult and intimidating lesson for students (Ashcrafts, 2002; Siregar, 2017). This
is caused by 10 factors (Widodo, 2016a), among which is the fact that "there are not
many mathematics books published in Indonesia that present problems in the
form of context, as a result, mathematics feels abstract and difficult to learn". He
162
also explained that mathematics is considered difficult because of the perception

of the student him- or herself who considers mathematics as a difficult lesson.
Many parents impart to their children the notion that mathematics is difficult so
that from childhood until adulthood, they have the perception that mathematics is
difficult and intimidating.
This relates to self-regulation, anxiety levels and student pressures studied in

relation to psychological well-being. The concept of psychological well-being was
originally introduced by Bernice Neugarten in 1961. She stated that psychological
well-being is a psychological condition at a time when a person reaches old age
(Tobin & Neugarten, 1961). The two main approaches to understanding
psychological well-being, according to Ryff (2013), are, firstly, the focus on
happiness in the sense of achieving happiness and preventing pain; and the
second approach is the awareness of being an overall functional or whole person,
including a healthy body and state of mind (Stern, 2007). Research on the well
being of children has been conducted in several parts of the world. In the United
Kingdom (UK), child welfare was researched (Rees et al., 2010) while in the
United States (US), the well-being of children was the topic of research (Huebner
& McCullough, 2010). This was also the case in Indonesia (Muqodas, 2019);
however, mathematics education was not included.
Taking into consideration the background above and the Covid-19 pandemic, this
research adopted a 21st century learning approach. There are nine principles
related to 21stcentury learning styles, namely i) being relevant to student life; ii)
applying cross-disciplinary learning; iii) developing critical thinking skills; iv)
encouraging the transfer of knowledge; v) teaching students how to learn; vi)
conveying misunderstandings directly; vii) collaborating; viii) exploiting
technology to support learning and ix) developing student creativity (Savedra &
Ofter, 2012). As an implementation of the 21st century learning style, this research
integrated technology and mathematical learning approaches using scientific
phenomena to develop creative thinking skills and autonomy.
In 2000, the National Council of Teachers of Mathematics (NCTM) made

recommendations for integrating technology in mathematics learning through
technology principles (NCTM, 2000b). The technology principle states that
technology is important in mathematics learning and teaching activities since it
influences how mathematics is taught and improves student learning activities.
Furthermore, the NCTM states that technology can facilitate the process of solving
mathematical problems, communication, reasoning and evidence. This is in line
with the research of Ayers et al. (1988).
Many research findings conclude that technology is the most suitable instrument
to develop mathematics learning and teaching activities. The activity of writing,
processing, and studying computer algorithm outputs can improve the
understanding of mathematical concepts and principles, numeracy skills, and
problem-solving abilities of students (Hatfield & Kieren, 1972). Technology is the
infrastructure used in mathematics education (Kaput et al., 2019). Furthermore,
they contend that to realize the technological potential, a new pedagogical method
163
is needed, and teacher development needs to be conducted. Kaput et al. (2019)

also shared the results of their research on classroom connectivity (CC), namely
classes involving technology.
However, the use of technology in learning mathematics is still a rare occurrence.

Some people think that learning mathematics using technology is not effective
enough to increase the passion for learning. Usually, the technology used in
learning mathematics comprises Cabri Geometry (Kösa et al., 2002), Kahoot for
quizzes (Göksün & Gürsoy, 2019; Prieto, 2019), Maple (Blyth & Labovic, 2009),
SPSS to process data (Xiao et al., 2015), calculators (Penglase & Arnold, 1996) and
PhotoMath for proof (Hamadneh & Al–Masaeed, 2015). However, it is still rare
for researchers to consider the use of mathematics learning technology for
distance learning.
It needs to be borne in mind that all students globally who are impacted by
Covid-19 must learn remotely. Therefore a remote management system called the
learning management system (LMS) is necessary. LMS is a management system
for distance learning, online learning or e-learning. It enables students to study
any time and any where. In an emergency, LMS functions as a distance learning,
online learning or e-learning infratstructure By using LMS, teachers and lecturers
can monitor student activities through a student attendance management system,
provide diverse learning resources (text, video, audio, animation) and connect to
other sources on the Internet such as structured assignments and quizzes with
interesting forms, as well as enabling discussion forums. Furthermore, LMS can
provide tools for collaboration and for both asynchronous learning (teachers and
students access the system at different times) and synchronous (teachers and
students access the system together and real-time) so that they can converse
directly (Widodo, 2017). Studies in the use of LMS in learning have been carried
out in universities by Bealty and Ulasewicz (2006), Kelley et al. (2016), McGill and
Klobas (2009) and Weaver et al. (2008). However, research reports on the use of
LMS in the field of mathematics learning are still rare, considering some of the
limitations of the features in LMS that can be used.
The basis of the selection of scientific phenomena in this research is that the nature
of mathematics is non-independent knowledge that can help humans in
understanding and mastering social, economic and natural problems (Kline, 1973).
Mathematics is an emerging science, evolving from real life and serving as a
practical tool for real problems (Boyer, 2011). Studies on phenomenon-based
learning to improve the cognitive and affective abilities of students through
mathematics learning have been conducted related to the development of
mathematical abilities of students by Aisah (2015), Faturohman (2015), Hidayat
(2015) and Rosyid (2015). The results of these research studies show that
phenomenon-based mathematics learning can improve students' mathematical
abilities (mathematical communication skills and mathematical connections) and
critical thinking. Meanwhile, research that has been trying to develop 21st century
skills required by students through mathematics learning has been conducted by
Turmudi (2017), who developed the didactic design of mathematics learning
through science phenomena and applied it to junior high school students. In
164
addition, 21st century skills development has been widely applied to STEM-based
integrative mathematics learning by Becker and Park (2011), Bell (2010) and Kim
and Park (2012).
Research Problem
Departing from empirical-conceptual conditions as described above, this research
examines the development of autonomy as one of the aspects in psychological
well-being and creative thinking skills in mathematics learning with the
phenomenon of LMS-assisted science in prospective elementary school teachers.
Conceptual Framework
Starting with the background, the research problem, and the pandemic of Covid-
19, a 21st century learning approach was adopted for this study. As an
implementation of the 21st century learning style, this research integrates
technology and mathematical learning approaches using scientific phenomena to
develop creative thinking skill and autonomy. For technology issues, the National
Council of Teachers of Mathematics (NCTM) made recommendations for
integrating technology in mathematics learning through technology principles
(NCTM, 2000b). In terms of scientific phenomena issues, the nature of
mathematics as a non-independent knowledge can help humans in understanding
and mastering social, economic and natural problems (Kline, 1973). Mathematics
is an emerging science, evolving from real life and serving as a practical tool for
real problems (Boyer, 2011). Moreover, mathematics explains phenomena and
their occurrence via teachers to students (Turmudi, 2014).
2. Theoretical Framework
2.1 Autonomy and Creative Thinking Skill
Autonomous people are independent people who can determine the best for
themselves (Keyes & Ryff, 1999). These individuals have an internal locus of
evaluation, i.e. not seeking the consent of others but rather evaluating themselves
by personal standards. Therefore, they do not think about the expectations and
judgments of others towards them. Furthermore, autonomous individuals do not
rely on the judgment of others to make important decisions. They do not conform
to social pressures to think and act in any particular way.
The first dimension of learning and innovation skill for K12 is creativity.
Creativity relates to discovery, trying new approaches to solving problems and
being innovative (P21, 2011). In line with the views of Chiam et al. (2014),
creativity is explained as a skill for discovering new things that have not existed
before, being original, developing new solutions to each problem, and involving
the ability to generate new, varied, and unique ideas (Chiam et al., 2014).
Creativity is the ability to uncover new relationships, to see things from a new
point of view, and to form new combinations of two or more concepts that have
been mastered before (Suryadi, 2005).
Based on the explanation above, creative and innovative thinking skills in this
research is the ability to generate new ideas that are original and different from
others, and to provide many solutions to problems. The indicators used in
165
measuring creative thinking skills in this study refer to indicators of creative

thinking skills (Torrance, 1966, 1974) known as TTCT and presented in Table 1.
Table 1: Torrance creative thinking skills indicators (1966, 1974)

1 Fluency Mention many problem-solving solutions
2 Flexibility Provide different answers of solutions
3 Originality Create new solutions or answers that have
never been used before to solve problems
4 Elaboration Mention the complete solutions
2.2 Mathematics Learning with Science Phenomena

Science phenomenon-based mathematics learning is derived from Freudenthal’s
didactical phenomenology theory (Freudenthal, 2002). Didactic phenomenon-
based mathematics learning in mathematical learning introduces appropriate and
suitable phenomena for students in learning mathematical concepts (Freudenthal,
2002; Turmudi, 2014). In this research, the phenomena introduced to students are
phenomena related to science. The science phenomenon that is introduced to
students is learning that emphasizes the importance of contextual issues to
introduce mathematical topics to students.
Phenomenology comes from the Greek phaninomenon meaning ‘emerging’, and

logos meaning ‘science’ (Turmudi, 2017). The philosophy of phenomenology is
learning about consciousness as the experience of the subject (or the view of the
first person) (Turmudi, 2017). The term ‘phenomenology’ is usually followed by
the word ‘didactic’, which means that learning begins with or departs from an
experience and consciousness. In mathematics learning, he illustrated that
students can be invited to walk on the floor (tiles) then explained to raise
awareness that the floor tiles have mathematical value. Then the students pause,
try to observe, look deeply, and imagine the mathematical value that can be
derived from the experience. Furthermore, students can try to conjecture in the
form of questions, then collect data or information to support the notion in order
to make abstractions or conclusions (Turmudi, 2017).
2.3 Learning Management System

The Learning management system (LMS) is a system or software as an
infrastructure in electronic, online or distance learning (McGill & Klobas, 2009;
Watson & Watson, 2007). The existence of this LMS cannot be separated from the
opportunities for learning using Internet technology and networks. The
capabilities and features of the LMS depend on the needs of the organization that
uses the system to manage learning. Dye and Fagerberg (2004) explain that in
general an LMS must have the following capabilities:
1. Registration for administrators, teachers and students on curriculum or
learning;
2. Linking curriculum and learning administration to learning content;
3. Supporting traditional virtual learning;
4. Supporting various kinds of analysis and assessment, tracking team or
individual progress;
5. Ability to administer test results, assessments and reports; and
166
6. Capability management so that an organization can measure learning needs

and identify development needs.
LMS is a powerful technology that has yet to reach its full potential and is
important for the Information Age paradigm of education (Watson & Watson,
2007). Because of its importance, greater care and understanding needs to be used
when applying the term in research literature. In its implementation, the LMS that
used was known as blended learning. Blended learning is a formal education
program in which a student learns, at least in part, through online delivery of
content and instruction with some element of student control over time, place,
path, and pace (Staker & Horn, 2012). In the blended learning model, students
study teaching materials at home, communicating with other students and
teachers through online discussions. Concepts can be reinforced by the teacher
through student involvement. Therefore, owing to the Covid-19 pandemic, this
research study was conducted online.
3. Methods
3.1 Design
The experimental design to obtain KA21 and PWB data used in this research is the
posttest-only design with a nonequivalent control group as presented in Figure 1.
In this design, the experimental group and the control group were given a post-
test, with the experimental class being given the treatment of mathematical
learning with a filtered-s least mean square (FSLMS) algorithm. The pre-
experiment design used is depicted below:
Group A X1 → O
Group B X → O
Figure 1: KA21 quantitative pre-experimental research design
Note:
X1: Treatment to the experimental class
X : Treatment to the control class
O : Post-test
This design is used to determine how LMS variables affect creative thinking skills
in KA21 and autonomy in PWB.
3.2 Participants
The population in this study were all the semester 1 students of prospective
elementary school teacher level 2 at a university located in two provinces, namely
in West Java and Banten. The samples taken to test the model in this research were
four (4) classes of diverse students, two classes of which were used as
experimental classes, and the remaining two (2) classes were comparison classes.
The students in all four classes numbered approximately 156 students. The
sampling technique used was a purposive sampling technique. Purposive
sampling is a sampling technique based on knowledge of the characteristics of the
population and the purpose of the research (Fraenkel, 2012). In the experimental
classes from the sample, mathematics learning was carried out using LMS-assisted
science phenomena mathematics learning (FSLMS). In this research, mathematics
167
learning with science phenomenon was applied to rational number material to the
prospective elementary school student teachers level II.
3.3 Research Instruments

The instruments used in this research were questionnaires related to the
autonomy or independence of learning that has been validated and tested. There
were 14 statements which were based on the theory of Ryff (1999) and the
development of autonomous instruments (Muqodas, 2019). Tests related to
rational numbers of three essay questions were also given to students to test their
creative thinking skills. These had been previously validated and empirically
tested. The test question refers to indicators 1, 2, 3 of Torrance's creative thinking
skills (Torrance, 1966, 1974).
3.4 Settings
Learning was conducted in a blended way. At each meeting, lecturers presented
learning plans, teaching materials in various forms such as audio, video and
presentations as well as discussion materials on the LMS. At the beginning of the
lesson, the lecturer initiated the learning synchronously by providing the details,
conveyed the learning objectives, scientific phenomena to be studied and quizzes.
Then the lecturer gave an explanation related to the activities that would be
carried out next, accompanied by a question and answer session. After that,
students studied the teaching materials. This was followed by carrying out
activities asynchronously. The lecturer guided learning activities through LMS
and a WhatsApp group. Once the asynchronous session had been completed, the
learning continued with a synchronous discussion. Students delivered practical
results in turn, then shared ideas, notions and questions and reflected on the
learning that had taken place.
4. Results and Discussion

4.1 Autonomy of Prospective Elementary School Student Teachers
The descriptive analysis of the autonomy of prospective elementary school
teachers students is described in Table 2.
Table 2: Description of PWB-A prospective elementary school teachers

Description A(1) A(2) Category F(1) %f(1) f(2) %f(2)
Average Score 52.17 51,48 Very High 0 0.00 0 0,00
Highest Score 71.00 71,00 High 54 62.07 39 44,83
Lowest Score 41.00 30,00 Low 33 37.93 48 55,17
Ideal Score 84.00 84,00 Very Low 0 0.00 0 0,00
StDev 6.26 6,92 Total 87 100.00 87 100,00
Information:
A(1): PWB-A results in the FSLMS mathematics learning group
A(2): PWB-A results in the TFSLMS mathematics learning group
F(1) : Frequency of students in each category in the FSLMS mathematics learning group
F(2) : Frequency of students in each category in TFSLMS mathematics learning group
168
Based on Table 2, it was found that PWB-A students are prospective elementary
teachers in mathematics learning groups with FSLMS and TFSLMS for the
following reasons:
1. There was an average score difference of 0.69 where the learning group with
FSLMS was higher than the TFSLMS. However, the standard deviation score
of the learning group with FSLMS was lower by 0.66 than the mathematics
learning group with TFSLMS;
2. The highest score in the mathematics learning group with FSLMS was the
same as that of the TFSLMS group; however, the lowest score was 11 points
higher than the TFSLMS math learning group. Therefore the range of PWB-A
scores in the mathematics learning group with FSLMS was 9 points less than
that of the TFSLMS group;
3. After treatment, neither of the learning groups with FSLMS or TFSLMS had
students with very high PWB-A scores, or very low scores; and
4. The percentage of prospective elementary school student teachers who have a
low PW-A score is higher in the mathematics learning group with TFSLMS.
Based on the PWB-A normality and homogeneity test, it was found that PWB-A
data of prospective elementary school student teachers in the mathematics
learning groups with FSLMS and TFSLMS are normal and homogeneous. On that
basis, the two different tests were conducted on average by using t-test using =
0.05. The test results are shown in Table 3.
Table 3: Test differences in PWB-A of prospective elementary school student teachers
Result t-test for Equality of Means

t df Sig. (2- Mean Difference Std. Error
tailed) Difference
PWB-A Equal variances -0.169 172 0.866 -0.184 1.091
POSTES assumed
Based on Table 3, t-test results indicate that the sig score. is 0.866 > 0.05. Thus H0 is
accepted, which means that there is no difference in the average score of PWB-A
data of prospective elementary school student teachers in the mathematics
learning group with FSLMS and TFSLMS.
4.2 Creative Thinking Skills (KKf) of Prospective Elementary School Student

Teachers
The description of creative thinking skills of prospective elementary school
teacher students is shown in Table 4.
Table 4: Description of KKf achievement of prospective elementary school student teachers

Class
KKf Achievement
FSLMS TFSLMS
Average 7,58 3,67
Highest Score 12 10
Lowest Score 0 0
St Dev 3,64 7,61
Ideal Score 16
169
Based on Table 4, the achievement score of KKf of prospective elementary school

student teachers in the mathematics learning group with FSLMS and TFSLMS can
be explained as follows:
1. There was an average score difference of 3.91, whereas the KKf of the learning
group with FSLMS was higher than the group with TFSLMS. However, the
standard deviation score of the learning group with FSLMS was lower by 3.97
than the math learning group with TFSLMS; and
2. The highest score in the math learning group with FSLMS was 2 points
higher, but the lowest score was the same as that of the TFSLMS group.
Based on the data, the KKf achievement of prospective elementary school student
teachers in the FSLMS group is better than that of the TFSLMS group. In addition,
inferential tests were conducted for differences in creative thinking skills between
the FSLMS and TFSLMS groups. Based on the normality test, it was found that
the results of KKf achievement data of prospective elementary school student
teachers in the mathematics learning groups with FSLMS and TFSLMS are not
normally distributed. On that basis, the difference test was conducted using the
Mann-Whitney test using  = 0,05. The test results are shown in Table 5.
Table 5: Test differences in KKf achievement of prospective elementary school

student teachers
KKf
Mann-Whitney U 2853,500
Wilcoxon W 6681,500
Z -3,188
Asymp. Sig. (2-tailed) 0,001
4.3 Discussion
Autonomy is one aspect of psychological well-being that can affect student
learning outcomes (Winch, 2002). By having good autonomy, students will have
high levels of independence in learning as well. People who are positive in terms
of psychological well-being (PWB) are happy, healthy, productive, and have
satisfying interpersonal relationships (Ryff, 2013). In a learning situation that is
affected by the Covid-19 pandemic as it is today, learning is mostly directed at
blended learning or learning that combines synchronous and asynchronous
techniques. These are highly dependent on the independence of students' learning
because the teacher only facilitates and provides motivation from a distance. In
this study, LMS-assisted mathematics learning shows that the students in the
class with mathematics learning with LMS-assisted science phenomena
descriptively have a higher level of autonomy. This is indicated by a higher
average score. The percentage of students who have higher levels of autonomy is
greater than in the class without scientific phenomena. This indicates that
computers can help students learn mathematics and problem-solving (Bell, 2010;
Hatfield & Kieren, 1972; Robitaille et al., 2020).
However, inferentially the results show that there is no significant difference. In

this research, LMS is needed as a learning tool during a pandemic to manage the
learning (Robitaille et al., 2020). Learning is strongly influenced by the subject
170
material, the teacher, the novelty effect, and the learning content. Although the
LMS offers various teaching materials such as audio-video, audio, and
presentations, independent learning will determine whether students can choose
one, or study all of the teaching materials presented, or even not choose any of the
teaching materials delivered. Students who study independently will be able to
determine the learning resources they want to learn or even study other teaching
materials as enrichment. Meanwhile, students with low autonomy will depend on
the instruction and supervision of the teacher or instructor.
Creative thinking skills is one of the most important skills of this century, and are
the highest-level skills in Bloom's taxonomy (Krathwohl, 2010). Therefore, these
skills play a very important role in individual success. In this research, learning
mathematics with LMS-assisted science phenomena has shown to have an impact
on students' creative thinking skills as shown in Table 4.1. Descriptively, the
average creative thinking skills of students who study mathematics with LMS-
assisted science phenomena are almost twice those of the comparison classes’
average. This is also reinforced by the results of the inferential test which shows
that there is a significant difference in the alpha of 5%.
Furthermore, learning with LMS-assisted science phenomena applied in this

research provides opportunities for students to explore the surrounding
environment through didactic phenomena that appear in students' daily lives. In
this learning, students conduct experiments that can provide opportunities for
them to develop their imagination, and ways and methods of completing and
reporting assigned experiments. In group experiments, students can also share
ideas in solving problems. This is in line with previous preliminary research on
LMS which also has a positive influence on mathematical creative thinking skills
(Widodo, 2017). This is supported by research by Garrison and Anderson (2003),
who found that technology is a learning tool used in formal education practices to
disseminate, illustrate, communicate, or support students and teachers in activities
designed specifically to sustain learning (Gunawan et al., 2019). This indicates
that learning using multimedia electronics is more significant than conventional
learning (Munir, 2010).
The analysis was continued to determine whether there is a relationship between

autonomy and creative thinking skills. Based on the correlation coefficient, a score
of 0.24 was obtained. Therefore, it can be concluded that there is a low relationship
between the two variables. This is not according to the views of Winch (2002)
who stated that autonomy is in line with critical thinking skills. Therefore, further
research needs to be done on other aspects of psychological well-being related to
creative thinking skills as part of 21st century skills.
5. Conclusion
Based on the analysis and discussion of these results, it can be concluded that the
autonomy of prospective elementary school teachers in learning mathematics with
LMS-assisted science phenomena is no different from learning mathematics
without LMS-assisted science phenomena. Both of these lessons produce
autonomy for students because they provide equal opportunities to study a
171
variety of teaching materials. The creative thinking ability of students in the

experimental classes is better than that of students in the control classes because
students in the experimental classes are more likely to improve their creative
thinking skills and are allowed to explore the surrounding environment through
didactic phenomena that appear in students' daily lives. This research has shown
that what needs to be followed up in further research is how elementary school
candidate teachers can explore additional teaching materials independently that
are not available in LMS. Therefore, to improve their ability of autonomy and
creative thinking, teachers need to adopt innovative learning such as mastering
LMS as learning management in selecting materials and implementing the
teaching and learning process effectively. The results reported here should be
interpreted in the light of these considerations.
6. References
Afifah, N. M. (2019). Keterampilan abad 21 dalam pembelajaran matematika berbasis proyek [21st
century skills in project-based mathematics learning]. UIN Sunan Ampel
Surabaya.
Aisah, L. S. (2015). Pembelajaran Matematika Berbasis Fenomena Didaktis Melalui Pendekatan
Saintifik Untuk Meningkatkan Kemampuan Koneksi Matematis Dan Self-Esteem Siswa
SMP [Didactic Phenomenon-Based Mathematics Learning Through a Scientific
Approach to Improve Mathematical Connection Ability and Self-Esteem of Junior
High School Students]. Universitas Pendidikan Indonesia.
Ashcrafts, M. (2002). Math anxiety: Personal, educational, and cognitive consequences.
Current Directions in Psychological Science, 11(5). https://doi.org/10.1111/1467-
8721.00196
Ayers, T., Davis, G., Dubinsky, E., & Lewin, P. (1988). Computer experiences in learning
composition of functions. Journal for Research in Mathematics Education, 19, 246–259.
Bealty, B., & Ulasewicz, C. (2006). Online teaching and learning in transition: Faculty
perspectives on moving from blackboard to the moodle learning management
system. Journal Techtrends, 50(4), 36–45.
Becker, K., & Park, K. (2011). Effects of integrative approaches among science, technology,
engineering, and mathematics ( STEM ) subjects on students’ learning : A
preliminary meta-analysis. Journal of STEM Education, 12(5), 23–38.
https://doi.org/10.1037/a0019454
Bell, S. (2010). Project-based learning for the 21st century skills.
http://doi.org/10.1080/00098650903505415
Blyth, B., & Labovic, A. (2009). Using Maple to implement eLearning integrated with
computer-aided assessment. International Journal of Mathematical Education in
Science and Technology, 40(7), 975–988.
https://doi.org/10.1080/00207390903226856
Brendefur, J., & Frykholm, J. (2000). Promoting mathematical communication in the
classroom: Two pre-service teachers’ conceptions and practices. Journal for Research
in Mathematics Education, 3(125). https://doi.org/10.1023/A:1009947032694
Chiam, L. C., Hong, H., Ning, F. H., & Tay, W.Y. (2014). Creative and Critical Thinking in
Singapore Schools. Office of Education Research, National Institute of Education,
Nanyang Technological University.
Dye, A., & Fagerberg, T. (2004). Exploring online service in a mobile environment.
Technical Working Paper.
Effendi, L. A. (2012). Pembelajaran Matematika Dengan Metode Penemuan Terbimbing
Untuk Meningkatkan Kemampuan Representasi Dan Pemecahan Masalah
172
Matematis Siswa SMP [Learning Mathematics with Guided Discovery Methods to

Improve Representation and Mathematical Problem Solving Skills for Junior High
School Students]. Jurnal Penelitian Pendidikan, LPPM UPI, 13(2).
Faturohman, D. R. (2015). Pembelajaran Matematika Berbasis Fenomena Didaktis Melalui
Pendekatan Scientific untuk Meningkatkan Kemampuan Berpikir Kritis dan Kemandirian
Belajar Siswa SMP [Didactic Phenomenon-Based Mathematics Learning Through a
Scientific Approach to Improve Critical Thinking Ability and Independent
Learning of Junior High School Students]. Universitas Pendidikan Indonesia.
Fauzi, M. R., & Priatna, N. (2019). Analysis of student’s mathematical connection and
communication in algebra: the exponential equations. Journal of Physics: Conference
Series, 1211 012065. https://doi.org/10.1088/1742-6596/1211/1/012065
Fauzi, A. (2015). The enhancement of student’s Mathematical connection ability and self-
regulation learning with metacognitive learning approach in junior high school. In
Proceedings of the International Conference on Research and Education in
Mathematics (ICREM7). https://doi.org/10.1109/ICREM.2015.7357048
Fraenkel, J. R., & N. E. W. (2012). How to design and evaluate research in education (8th ed.).
McGraw-Hill Higher Education.
Freudenthal, H. (2002). Phenomenology, didactical mathematical, O F. Springer Netherlands.
Fuchs, L. S., Fuchs, D., Prentice, K., Hamlett, C. L., Finelli, R., & Courey, S. J. (2004).
Enhancing mathematical problem solving among third-grade students with
schema-based instruction. Journal of Educational Psychology, 96(4), 635–647.
https://doi.org/https://doi.org/10.1037/0022-0663.96.4.635
Giarti, S. dan Astuti, S. (2016). Implementasi TQM Melalui Pelatihan Model in House
Training untuk Meningkatkan Kompetensi Pedagogik Guru SD [Implementation
of TQM through Model in House Training to Improve Pedagogic Competence of
Elementary School Teachers]. Scholaria Jurnal Pendidikan Dan Kebudayaan, 6(12).
https://doi.org/10.24246/j.scholaria.2016.v6.i2.p80-91
Göksün, D. O., & Gürsoy, G. (2019). Comparing success and engagement in gamified
learning experiences via Kahoot and Quizizz. Journal Computers & Education, 135,
15–29. https://doi.org/10.1016/j.compedu.2019.02.015
Gunawan, G., Sahidu, H., Susilawati, S., Harjono, A., & Herayanti, L. (2019). Learning
management system with Moodle to enhance creativity of candidate physics
teacher. Journal of Physics: Conference Series, 1417(1), 0–6.
https://doi.org/10.1088/1742-6596/1417/1/012078
Hamadneh, I. M., & Al–Masaeed, A. (2015). Math teachers’ attitudes towards photo math
application in solving mathematical problem using mobile camera. Educational
Research and Reviews, 10(14), 1930–1936. https://doi.org/10.5897/ERR2015.2181
Hatfield, L. L., & Kieren, T. E. (1972). Computer-assisted problem solving in school
mathematics. Journal for Research in Mathematics Education, 3(2).
https://doi.org/10.2307/748668
Hidayat, A. (2015). Perbandingan Kemampuan Berpikir Kreatif Matematis Serta Self-Efficacy
Antara Siswa MTs yang Memperoleh Pembelajaran Berbasis Fenomena Didaktis dan
Berbasis Kurikulum 2013 Melalui Pendekatan Invesitigasi [Comparison of
Mathematical Creative Thinking Ability and Self-Efficacy Between MTs Students
Who Received Didactic Phenomenon-Based Learning and 2013 Curriculum-Based
Through Investigation Approach]. Universitas Pendiidkan Indonesia.
Huebner, S. E., & McCullough, G. (2010). Correlates of school satisfaction among
adolescents. Journal of Educational Research, 93(5).
https://doi.org/10.1080/00220670009598725
Kaput, J., Hegedus, S., & Lesh, R. (2019). Foundations for the future in mathematics education.
Routledge.
173
Kelley, T. R., Knowles, J. G., Abell, S., Lederman, N., Bandura, A., Bandura, A., Williams,
D. (2016). A conceptual framework for integrated STEM education. International
Journal of STEM Education. https://doi.org/10.1186/s40594-016-0046-z
Keyes, C. L. M., & Ryff, C. D. (1999). Psychological well-being in midlife. In S. L. Willis & J.
D. Reid (Eds.), Life in the middle: Psychological and social development in middle age.
Academic Press. https://doi.org/10.1016/B978-012757230-7/50028-6
Kim, Y., & Park, N. (2012). The effect of STEAM education on elementary school student’s
creativity improvement. Communications in Computer and Information Science, 339
CCIS, 115–121. https://doi.org/10.1007/978-3-642-35264-5_16
Kline, M. (1973). Why JoHnny can’t add: The failure of the new math. Vintage Books.
Kösa, T., & Karakuş, F. (2010). Using dynamic geometry software Cabri 3D for teaching
analytic geometry. Procedia - Social and Behavioral Sciences, 2(2), 1385–1389.
https://doi.org/10.1016/j.sbspro.2010.03.204
Kramarski, B., & Mevarech, Z. R. (2003). Enhancing mathematical reasoning in the
Cclassroom: The effects of cooperative learning and metacognitive training.
American Educational Research Journal, 40(1), 281–310.
https://doi.org/10.3102/00028312040001281
Lestari, S. A. P. (2019). Mathematical reasoning ability in relations and function using the
problem solving approach. Journal of Physics: Conference Series, 1188(012065).
https://doi.org/10.1088/1742-6596/1188/1/012065
Maslukha, M., Lukito, A., & Ekawati, R. (2018). Refractive thinking profile in solving
mathematical problem reviewed from students math capability. Journal of Physics:
Conference Series, 947(012022).
McGill, T. J., & Klobas, J. (2009). A task–technology fit view of learning management
system impact. Journal Computers & Education, 52(2), 496–508.
Merzbach, U. C., & Boyer, C. B. (2011). A history of mathematics. John Wiley & Sons.
https://doi.org/10.2307/3620228
Muhajir, S. (2018). Penerapan project oriented problem based learning dengan reading infusion.
Untuk melatihkan keterampilan abad Ke 21 Mahasiswa. Universitas Pendidikan
Indonesia.
Munir. (2010). Penggunaan learning management system (LMS). Di Perguruan Tinggi:
Studi Kasus Di Universitas Pendidikan Indonesia [Use of learning management
system (LMS). In Higher Education: A Case Study at the Indonesian University of
Education]. Cakrawala Pendidikan, 29(1), 109–119.
https://doi.org/10.21831/cp.v1i1.222
Muqodas, I. (2019). Konseling Kesejahteraan untuk Meningkatkan Kesejahteraan Psikologis
Mahasiswa [Welfare Counseling to Improve Student Psychological Well-being].
Universitas Pendidikan Indonesia.
Mushlihuddin, R., & Nurafifah, I. (2018). The effectiveness of problem-based learning on
students’ problem solving ability in vector analysis course. Journal of Physics:
Conference Series, 948(012028). https://doi.org/10.1088/1742-6596/948/1/012028
Nabilah, N. G. S., Suhendra, S., & Yulianti, K. (2019). The efforts of improving
mathematical connection ability of senior high school student with 7e learning
cycle model. Journal of Physics: Conference Series, 1157(042096).
NCTM. (2000a). Executive summary principles and standards for school mathematics.
https://www.nctm.org/uploadedFiles/Standards_and_Positions/PSSM_Executi
veSummary.pdf
NCTM. (2000b). Principles and standards for school mathematics overview. Journal of
Equine Veterinary Science.
P21. (2011). 21st century skill map. http://www.21.org
174
Penglase, M., & Arnold, S. (1996). The graphics calculator in mathematics education: A
critical review of recent research. Mathematics Educational Research Journal, 8, 58–90.
https://doi.org/10.1007/BF03355481
Rees, G., Goswani, H., & Bradshaw, J. (2010). Developing an index of children’s subjective
well being in England. The Children's Society.
Robitaille, D. F., Sherrill, J. M., & Kaufman, D. M. (2020). The effect of computer utilization
on the achievement and attitudes of ninth-grade mathematics students. Journal for
Research in Mathematics Education, 8(1), 26–32.
https://doi.org/10.5951/jresematheduc.8.1.0026
Rosyid, A. (2015). Pembelajaran Matematika Berbasis Fenomena Didaktis Melalui Pendekatan
Saintifik untuk Meningkatkan Kemampuan Komunikasi Matematis dan Self-Confindence
Siswa SMP [Didactic Phenomenon-Based Mathematics Learning Through a
Scientific Approach to Improve Mathematical Communication Skills and Self-
Confindence of Junior High School Students]. Universitas Pendidikan Indonesia.
Ryff, C. D. (2013). Psychological well-being revisited: Advances in the science and practice
of eudaimonia. Psychotherapy and Psychosomatics, 83(1), 10–28.
https://doi.org/10.1159/000353263
Salehudin, N. N., Hassan, N. H., & Hamid, N. A. A. (2015). Matematik dan Kemahiran
Abad Ke-21: Perspektif Pelajar [21st Century Mathematics and Skills: Student
Perspectives]. Jurnal Pendidikan Matematik., 3(1), 24–36.
Savedra, A. R & Ofter, V. D. (2012). Learning 21st century skills requires 21st century
teaching. The Phi Delta Kappan, 94(2), 8-13.
https://doi.org/10.1177/003172171209400203
Siregar, N.R. (2017). Persepsi siswa pada pelajaran matematika: studi pendahuluan pada
siswa yang menyenangi game [Students' perceptions of mathematics: a
preliminary study on game-loving students]. Prosiding Temu Ilmiah Nasional X
Ikatan Psikologi Perkembangan Indonesia, Vol 1.
Staker, H., & Horn, M. B. (2012). Classifying K–12 Blended Learning.
Stern, S. (2007). Factors that impact the health and psychological well-being of older adults shortly
following institutionalization. Case Western Reserve University.
Suryadi, D. (2005). Penggunaan Pendekatan Pembelajaran Tidak Langsung serta Pendekatan
Gabungan Langsung dan Tidak Langsung dalam Rangka Meningkatkan Kemampuan
Berpikir Matematik Tingkat Tinggi Siswa SLTP [The Use of Indirect Learning
Approaches and Combined Direct and Indirect Approaches in Order to Improve
High-Level Mathematical Thinking Skills for Junior High School Students].
Universitas Pendidikan Indonesia.
Tobin, S.S., & Neugarten, B. L. (1961). Life satisfaction and social interaction in the aging.
Journal of Gerontology, 16, 344–346. https://doi.org/10.1093/geronj/16.4.344
Torrance, E. P. (1966). The Torrance tests of creative thinking-norms-technical manual research
edition -Verbal tests, Forms A and B - Figural tests, Forms A and B. Personnel Press.
Torrance, E. P. (1974). The Torrance tests of creative thinking-norms-technical manual research
edition - Verbal tests, Forms A and B - Figural tests, Forms A and B. Personnel Press.
Turmudi, T. (2014). Pengembangan pembelajaran matematika berbasis fenomena didaktis
[Development of didactic phenomenon-based mathematics learning]. Universitas
Pendidikan Indonesia
Turmudi, T., Hidayat, A. S., & Sispiati, R. (2017). Mathematics instruction based on science
using didactical phenomenology approach in junior secondary school in
Indonesia. Intech Open. http://dx.doi.org/10.5772/intechopen.68437
Watson, W. R., & Watson, S. L. (2007). An argument for clarity: What are learning
management systems. What are they not, and what should they become?
TechTrends, 51(2), 28–34.
175
http://cardinalscholar.bsu.edu/handle/123456789/194513
Weaver, D., Spratt, C., & Nair, C. S. (2008). Academic and student use of a learning
management system: Implications for quality. Australian Journal of Educational
Technology, 24(1). https://doi.org/10.14742/ajet.1228
Widodo. (2016a). Profesor Ini Ungkap Mengapa Matematika Dianggap Sulit [This Professor
Reveals Why Mathematics Is Considered Difficult]. Suara.com: Edisi Rabu.
Suara.Com.
Widodo, A. (2016b). Teacher pedagogical content knowledge (PCK) and students
reasoning and wellbeing. Journal of Physics: Conference Series, 812.
http://doi.org/10.1088/1742-6596/812/1/012119
Widodo, S. (2017). Implementing Google Apps for education as learning management
system in math education. Journal of Physics: Conference Series, 895(1).
https://doi.org/10.1088/1742-6596/895/1/012053
Winch, C. (2002). Strong autonomy and education. Educational Theory, 52(1).
Xiao, X., Xu, H., & Xu, S. (2015). Using IBM SPSS modeler to improve undergraduate
mathematical modelling competence. Computer Applications in Engineering
Education, 23, 603–609. https://doi.org/10.1002/cae.21632
176

Vol. 20, No. 8, pp. 176-193, August 2021
E-Learning Implementation Barriers during

COVID-19: A Cross-Sectional Survey Design
Ishaq Al-Naabi and Abdullah Al-Abri

English Language Centre, University of Technology and Applied Sciences,
Nizwa, Oman
https://orcid.org/0000-0001-8829-2922
https://orcid.org/0000-0001-7198-9643
Abstract. Most higher education institutions have embraced emergency

remote teaching (ERT) as a response to school and university closures
due to the Covid-19 pandemic. We used a cross-sectional survey design
to examine teachers’ views on the implementation of e-learning for ERT
due to Covid-19 in Omani higher education. We examined four barrier
categories: teacher-related, institutional-related, curriculum-related and
student-related. We also assessed the relationship between these barrier
levels and measured the differences between e-learning and teachers’
gender, academic qualifications, teaching experience and prior
experience in e-learning. Through an investigation of 856 university
faculty members, we found that student-level barriers had the strongest
impact on the implementation of e-learning for ERT during Covid-19.
Furthermore, teachers’ academic qualifications and prior experience
influenced the success of e-learning practices. Based on these results, we
presented some implications with respect to student preparedness,
curriculum modifications and institutional readiness for the successful
implementation of ERT during Covid-19 or any similar future
pandemics.
Keywords: Covid-19; e-learning; emergency remote teaching; Oman;

higher education
1. Introduction
The Covid-19 outbreak has had a serious impact on educational systems around
the world. The majority of higher education institutions cancelled face-to-face
classes and embraced emergency remote teaching (ERT) in an attempt to
maintain social distancing. With millions of students around the world having
been affected by school and university closures during Covid-19 (UNESCO,
2020), many educational institutions implemented some type of ERT (Hodges et
al., 2020). To curb the spread of coronavirus, the Omani government announced
the suspension of all schools and universities on the 15th of March 2020, and
©Authors
177
consequently, most public and private institutions used what resources they had
available to shift to ERT.
Technology integration for online and distance learning is a crucial component
of providing quality education (UNESCO, 2014). Teachers at the tertiary level
must be ‘fluent users of technology’ if they are to support student learning (US
Department of Education, 2016, p. 34). Additionally, Trust (2017) argued that
future teachers should be capable of using technology to enhance students’
learning.
The shift to ERT has enabled higher education to provide instruction during
emergencies; however, the shift has caused serious challenges (Crawford et al.,
2020; Hodges et al., 2020). As many teachers were not prepared for the sudden
shift to online instruction, they faced issues in preparing online content,
adapting face-to-face materials, and learning new instructional pedagogies for
online teaching and learning (Crawford et al., 2020; Dhawan, 2020; Rapanta et
al., 2020). Gacs et al. (2020) and Karalis (2020) suggested that higher education
institutions should provide sufficient support to their faculty during
emergencies. Students faced issues with online learning during COVID-19
(Crawford, Butler-henderson, et al., 2020; Gaber et al., 2020; Zuo et al., 2020).
Apart from teachers and students preparedness for ERT, institutions were not
fully ready to shift to online instruction (Bao, 2020; Crawford et al., 2020).
Since ERT is mostly dependent on teachers’ use of and competency in e-learning
(Crawford et al., 2020; Demuyakor, 2020; Gacs et al., 2020; Hodges et al., 2020), it
is essential that we measure the barriers to technology integration for ERT
during Covid-19. Therefore, in this paper, we have sought to investigate the
barriers faced by university teachers in using e-learning for ERT during Covid-
19 in Omani public and private higher education institutions.
In addition to advancing knowledge in e-learning and online pedagogy, the
findings of this paper can inform higher education practitioners in overcoming
challenges associated with online instruction during emergencies. Also, it can
help institutions to better plan their instruction during similar pandemics in the
future. People in charge of professional development can use the results of this
study as a basis for needs analysis for teacher professional development and
student training provisions.
ERT is a temporary shift of instructional delivery to an alternative delivery mode
due to crisis circumstances (Hodges et al., 2020, p. 6). Remote teaching is not
new, and it has been argued that teachers should, in fact, be prepared to teach
and administer online courses (Baran & Alzoubi, 2020; Crawford et al., 2020;
Gacs et al., 2020; Hartshorne et al., 2020; Hodges et al., 2020; Karalis, 2020). While
responses of higher education providers to Covid-19 differed, the majority of
institutions in developed and developing economies chose to migrate to online
teaching (Crawford et al., 2020; Hodges et al., 2020).
Karalis (2020) proposed a model that was intended to provide a good response
to crises in education. He argued that a needs analysis that includes the needs of
students, instructors and available resources, as well as considering the nature of
178
the courses themselves, is required for the success of any emergency plan.
Furthermore, Baran and Alzoubi (2020) proposed a human-centred model for
transferring face-to-face courses online during Covid-19. Based on their model,
they asserted that human-centred design models can enhance student empathy,
engage students in pedagogical problem-solving and help them build a learning
community (Baran & Alzoubi, 2020). Despite these efforts, ERT is mainly aimed
at ensuring the continuity of education during emergencies, which entails some
sacrifices if it is to guarantee a rapid response to a crisis (Gacs et al., 2020;
Hodges et al., 2020).
ERT requires technology integration, which poses challenges and difficulties for
both teachers and students. Crawford et al. (2020) identified a variety of
challenges that can hinder institutions in their migration to online education
during ERT, including lack of infrastructure, teacher skillsets and readiness to
design, offer and implement online teaching and learning. Apart from these
challenges, many educational systems worldwide continue to lack online
instruction strategies (Crawford et al., 2020). Moreover, Bao (2020) reported that
higher academic institutions lacked the preparedness to shift to online teaching
and learning during Covid-19, calling for systematic planning and investments
from all sectors to ensure the ability to successfully shift to ERT. Similarly,
Chang and Fang (2020) reported that teachers in higher education were not
familiar with the teaching tools required for online teaching which posed serious
negative impact on the implementation of ERT. Nevertheless, this
unpreparedness, interruption and sudden shift, according to McMaster et al.
(2020), can provide great learning opportunities for teachers.
Previous research has established a set of common technology integration
barriers that have each been labelled, measured and rated differently, and that
overlap on certain occasions (Schoepp, 2005). Common challenges include
creating content for online spaces, learning new delivery tools, understanding
online pedagogy, engaging parents, addressing student mental health issues,
and attempting various pedagogical strategies to address both synchronous and
asynchronous teaching and learning (Hartshorne et al., 2020, p. 138).
Alqudah et al. (2020) used a survey design to measure the perceptions and
experiences of Jordanian academic ophthalmologists in e-learning for an
undergraduate course during Covid-19. The study revealed some advantages of
e-learning, including convenience, flexibility of time and place for both students
and teachers, increased motivation of shy students and overcoming the
circumstances of lockdown measures during Covid-19 (Alqudah et al., 2020).
Though the majority of their sample viewed their e-learning experience during
Covid-19 as positive and successful, Alqudah et al. (2020) did report some
disadvantages of e-learning, such as lack of interaction, discomfort of teaching
and learning without face-to-face interactions, and lack of practical and clinical
training. The study also identified some limitations to e-learning during Covid-
19, including poor infrastructure, shortage of e-learning training courses in
ophthalmology, students’ or teachers’ poor e-learning skills and inadequate
internet speed (Alqudah et al., 2020).
In another study using a survey design, Mailizar et al. (2020) examined the
secondary school teachers’ views of e-learning during Covid-19. This study
179
found that student-related barriers (skills and knowledge, motivation and e-

learning infrastructure) had a strong impact on teacher implementation of e-
learning (Mailizar et al., 2020). Student-related barriers to e-learning
implementation had a strong positive correlation with school-related barriers
(i.e. availability of software and hardware, internet connectivity, policies,
technical support) and curriculum-related barriers (i.e. content, assessments, e-
learning resources, curriculum alignment to online teaching and learning).
Due to the scarcity of research on this topic during Covid-19, for this study we
relied on the large and growing body of literature that has investigated the
barriers to e-learning implementation prior to Covid-19. Several studies
(Assareh & Hosseini, 2011; Naveed et al., 2017; Pelgrum, 2001) have identified
various barriers to e-learning implementation. Despite differences in labelling
and classifying these barriers, they were similar and overlapping. In general, the
barriers can be classified into four main categories: student-related barriers,
teacher-related barriers, curriculum-related barriers and institutional-related
barriers. The issues related to students included lack of ICT skills, lack of e-
learning knowledge, lack of English language proficiency, lack of motivation,
lack of technological infrastructure and poor assessment and online learning
skills (Assareh & Hosseini, 2011; Naveed et al., 2017; Pelgrum, 2001). Assareh
and Hosseini (2011), Naveed et al. (2017) and Pelgrum (2001) also identified
barriers related to teachers, including lack of ICT skills, lack of e-learning
knowledge, instructor resistance to change, lack of time to develop e-courses,
lack of motivation, difficulty in monitoring and assessing student learning and
difficulty in integrating technology into teaching. With respect to curriculum,
the literature identified a lack of sound instructional design, curriculum
ambiguity, poor curriculum quality, unavailability of resources for e-learning
and a mismatch between the existing teaching and learning process, current
evaluation procedures, and the online curriculum (Assareh & Hosseini, 2011;
Naveed et al., 2017; Pelgrum, 2001). Finally, studies have also reported
institutional-related barriers, such as inappropriate infrastructure, low internet
bandwidth, lack of technical support, lack of financial support, lack of adequate
policies and lack of training on e-learning and online pedagogy.
Several research studies have identified various determinants to successful e-
learning practices. Although gender was not a crucial factor influencing e-
learning use, research has reported that female teachers are more confident in
utilising e-learning in their teaching (González-gómez et al., 212; Mahdizadeh et
al., 2008). In addition to gender, teacher’s academic qualification and prior
experience in e-learning can affect the use of e-learning (Fryer & Bovee, 2016;
Jones, 2003; Sørebø et al., 2009; Wilson, 2012).
These studies highlight the barriers that teachers face when using e-learning in
teaching and learning at various levels. These barriers hinder the teaching and
learning process in ERT during Covid-19, as ERT is entirely based on either
synchronous or asynchronous online teaching and learning (Hodges et al., 2020).
Assareh and Hosseini's (2011) four-dimension classification of e-learning
barriers (learners, teachers, curriculum and school) were used in this study.
Since the context of the current study is higher education (i.e. colleges and
universities), we changed the school classification to ‘institution’. Learner-
180
related barriers included insufficient e-learning knowledge and skills, limited

accessibility and lack of motivation. Teacher-related barriers included
insufficient knowledge and skills in e-learning, attitudes and beliefs towards e-
learning, lack of confidence and previous experience. The barriers related to
curriculum that can hinder e-learning use included in this study are disparity
between curriculum and assessment, f2f curriculum that does not have e-
learning components and complex tasks that cannot be delivered through an e-
learning system. Finally, we included the following institutional-related barriers
in our study: e-learning infrastructure, policies and professional development in
e-learning.
Our research is guided by the following research questions:
1. What barriers do teachers face in implementing e-learning for ERT during
Covid-19 in Oman?
2. What is the relationship between each level of barrier to e-learning
implementation?
3. Are there any significant differences between e-learning implementation
barriers and teachers’ gender, academic qualifications, teaching experience
and prior experience in e-learning?
4. Are there any significant relationships between e-learning implementation
barriers and type of institution?
3. Methodology
3.1 Research Design
The study employed a cross-sectional survey research design. Creswell (2018)
defined survey research as “a set of procedures in quantitative research in which
investigators administer a survey to a sample or to the entire population of
people to describe the attitudes, opinions, behaviours, or characteristics of the
population” (p.376). Elsewhere, Fraenkel et al. (2012) stated that using a
questionnaire in survey research methodology could provide reliable, valid and
generalizable quantitative and qualitative results.
3.2 Research Instrument
We developed a questionnaire consisting of three main parts to gather teachers’
views on the implementation of e-learning during Covid-19. In Part A, the
participants’ demographic information, including gender, level of education,
teaching experience, institution, specialisation, teacher certification and devices
used for e-learning was obtained. The second part comprised 39 Likert-scale
items that were grouped into the four main dimensions (teacher-related barriers,
institution-related barriers, curriculum-related barriers and student-related
barriers). We adapted the items included in the scale from various studies (Flack
et al., 2020; Haney, 2002; Mailizar et al., 2020; Naveed et al., 2017) and added
some items for the purpose of this research. Participants were asked to mark
their responses on a five-level agreement scale ranging from strongly disagree to
strongly agree. The third part included two open-ended questions that asked
about other barriers that teachers faced in implementing e-learning during
COVID-19 and about their suggestions for better online pedagogy during
emergencies.
181
Five academics from different institutions reviewed the questionnaire for clarity
and relevance to ensure its content validity. Along with the questionnaire, a
content validation form was provided for the reviewers. The questionnaire
achieved satisfactory level of content validity (the content validity index for each
item ranged between 0.8 to 0.92 across the five reviewers. In addition, we piloted
the questionnaire with a group of university teachers from a private university
in Oman who had experienced ERT during Covid-19. Twenty-seven teachers
(55.9% males, 44.4% females) completed the survey. The majority of the
participants had a master’s degree (77.8%), while 22.2% had doctoral degrees.
The sample included teachers from different specialisations, such as business,
education, language and literature, information and communication technology
and English language teaching.
To ensure the reliability of the research instrument, we computed Cronbach’s
alpha reliability coefficient in SPSS, which was found to be 0.88. The reliability
coefficient of the four dimensions ranged from 0.42 to 0.85. Teacher-related
barriers had a lower coefficient (0.42) compared to other dimensions of the
questionnaire (teacher-related barriers = 0.75; curriculum-related barriers = 0.77;
student-related barriers = 0.85). Thus, we removed three items from the teacher-
related barriers dimension to increase the instrument’s reliability. Once these
items were removed, the coefficient increased to 0.89.
The questionnaire was distributed to teachers using Google Forms®.
Participation in the study was voluntary. Although participants were not asked
to fill out an informed consent, their submission of the questionnaire was
regarded as their consent. Both researchers stored the data and the data was
discarded once the data was analysed.
3.3 Sample
Following random sampling, we drew the actual study sample from public and
private higher education institutions in Oman (teachers from public institutions,
N = 726; teachers from private institutions, N = 122). Note that the numbers of
teachers from each sector is not equal as there are more public institutions
compared to private institutions in the country. The sample included 856
university teachers who experienced ERT during Covid-19 (63.3% males, 36.7%
females). As shown in Table 1, the sample included teachers with various
educational degrees. Exactly 70 of the participants held bachelor’s degrees
(8.2%), 535 held master’s degrees (62.5%) and 251 held doctoral degrees (29.3%).
Although the teachers’ teaching experience ranged from one to more than 15
years of experience, half of the sample had more than 15 years of teaching
experience. The sample also included teachers from different specialisations,
including social sciences, science, computer and information systems, education,
business and management, engineering, mathematics and health. Four hundred
and seventy-one of the teachers (55%) had completed teacher preparation
programmes, whereas 385 teachers (45%) had not taken part in a teacher
education programme. The majority of the participants (83.2%) had some
experience in e-learning prior to the implementation of ERT during Covid-19.
During ERT, the teachers used various devices for online teaching and learning.
182
Table 1: Demographic information of the research sample

Variable N %
Gender
Male 542 63.3
Female 314 36.7
Academic Qualification
Bachelor’s Degree 70 8.2
Master's Degree 535 62.5
Doctoral Degree 251 29.3
Teaching Experience
0 to 5 Years 32 3.7
5 to 10 Years 123 14.4
10 to 15 Years 237 27.7
More than 15 Years 464 54.2
Specialisation
Social Sciences 129 15.1
Science 40 4.7
Computer Science and Information 124 14.5
Systems
Education 110 12.9
Business and Management 88 10.3
Engineering 283 33.1
Mathematics 68 7.9
Health 14 1.6
Institution
Public 726 84.8
Private 122 14.3
Teacher Certification
Yes 471 55.0
No 385 45.0
Experience in E-Learning Prior to ERT
Yes 712 83.2
No 144 16.8
Devices Used for E-Learning in ERT
Mobile/Handheld Device 13 1.5
Computer/Laptop 249 29.1
Both 594 69.4
Total 856 100.0
4. Findings
We used a set of descriptive statistical tests including means, standard
deviations, Spearman’s correlation and one-way multivariate analysis of
variance (MANOVA) to analyse the quantitative data. On the other hand, we
used content analysis to analyse the qualitative data obtained through the open-
ended questions. First, we familiarised ourselves with the data by reading the
responses. Then, we defined grouping themes under which the initial data was
classified. For question 1, we defined six emerging themes: policies and
procedures, technological and infrastructure issues, student-related issues,
curriculum-related issues, time-related issues and teachers’ experience in online
183
pedagogy. We then read the data, categorised it into six groups, and followed
the same procedures to analyse the second open-ended question.
4.1 E-Learning Implementation Barriers
As indicated earlier, the barriers were classified into four main categories:
teacher-related barriers, institution-related barriers, curriculum-related barriers
and student-related barriers; the results of which are displayed based on this
classification. Our descriptive results revealed that teacher-related barriers did
not significantly hinder the implementation of e-learning during Covid-19 (see
Table 2). Overall, the participants believed that e-learning was useful (M =
1.99, SD = 1.11) and convenient (M = 2.24, SD = 1.07) for ERT during Covid-19.
The participants also held that they were confident (M = 2.12, SD = 1.06) as they
had sufficient knowledge (M = 2.15, SD = 1.06) and the necessary skills (M =
2.13, SD = 1.03) to use e-learning during the pandemic.
Table 2: Descriptive results of teacher-related barriers
Statement N M SD
1. I have sufficient knowledge to use e-learning for emergency
856 2.15 1.06
remote teaching during the Covid-19 pandemic.
2. I have the necessary skills to use e-learning for emergency
856 2.13 1.03
3. I am confident in using e-learning for emergency remote
856 2.12 1.06
teaching during the Covid-19 pandemic.
4. E-learning is useful for emergency remote teaching during
856 1.99 1.11
the Covid-19 pandemic.
5. The use of e-learning for emergency remote teaching during
856 2.24 1.07
the Covid-19 pandemic is convenient for me.
Overall, institution-related barriers did not considerably obstruct the use of e-

learning (see Table 3). The participants confirmed that they had enough training
(M = 2.22, SD = 1.11) and technical support (M = 2.16, SD = 1.16) from their
institutions in implementing e-learning during the pandemic. Moreover, they
agreed that the institutions provided them with a clear assessment policy (M =
2.42, SD = 1.13) and online conferencing tools (M = 2.21, SD = 1.11) for
synchronous communication with their students. However, the teachers
reported that their institutions encountered challenges in monitoring the quality
of online teaching (M = 2.94, SD = 1.11).
Table 3: Descriptive results of institution-related barriers
Statement N M SD
1. My institution has a policy for emergency remote teaching. 856 2.38 1.14
2. My institution’s policies and regulations support the use of e-
learning for emergency remote teaching during the Covid-19 856 2.21 1.12
pandemic.
3. My institution has a clear assessment policy for emergency remote
856 2.42 1.13
teaching during Covid-19.
4. My institution has online conference tools for synchronous
856 2.21 1.11
communication with students and teachers.
5. My institution provided technical support for e-learning use for
856 2.16 1.16
emergency remote teaching during Covid-19.
184
6. My institution provided training on e-learning for emergency

856 2.22 1.11
7. My institution allowed me to design my own learning experiences. 856 2.33 1.07
8. My institution faced difficulty in monitoring the quality of online
teaching in emergency remote teaching during the Covid-19 856 2.94 1.11
pandemic.
The curriculum-related barriers appeared to be more critical than the teacher-

related and institution-related barriers (see Table 4). The teachers reported that
though learning and teaching materials and textbooks were available (M = 2.24,
SD = 1.01) and suitable (M = 2.43, SD = 1.05), the nature of the courses made
them difficult to teach online (M = 2.95, SD = 1.22). They also stated that the
courses included materials that could not be delivered properly online (M = 2.94,
SD = 1.11). Furthermore, the participants stated that some courses required face-
to-face attendance for practical tasks (M = 2.52, SD = 1.32) or settings that
allowed for a community of learning, neither of which were attainable during e-
learning (M = 2.94, SD = 1.14).
Table 4: Descriptive results of curriculum-related barriers
Statement N M SD
1. Learning and teaching resources that are available in the e-learning
856 2.24 1.01
system are in accordance with the curriculum.
2. The textbooks that I use are suitable for e-learning use. 856 2.43 1.05
3. Student's assessments are in line with e-learning use. 856 2.47 1.04
4. The nature of my course makes it difficult to be taught through e-
856 2.95 1.22
learning.
5. My course has practical tasks that require students to attend in person at a
856 2.52 1.32
specific time.
6. My course has many materials that cannot be conveyed online. 856 2.98 1.27
7. The e-learning activities address different learning styles of students. 856 2.56 0.99
8. My course requires a community of learning, which was difficult to build
856 2.96 1.14
in the e-learning system.
For the student-related barriers, the results (see Table 5) indicated that English
language proficiency was a strong barrier (M = 3.00, SD = 1.11) in the use of e-
learning for ERT during Covid-19. Lack of training (M = 2.99, SD = 1.19),
resistance to participation due to cultural norms (M = 2.96, SD = 1.06), and
inadequate internet connection (M = 2.95, SD = 0.94) comprised the second, third
and fourth barriers, respectively. The next most significant barriers were
availability of devices (i.e. laptops and tablets) (M = 2.89, SD = 1.07), lack of
interest in using e-learning (M = 2.80, SD = 1.04), lack of sufficient knowledge (M
= 2.68, SD = 1.17) and lack of necessary skills (M = 2.66, SD = 1.13). On the other
hand, ability to access the e-learning system (M = 2.55, SD = 1.01), ability to
progress (M = 2.49, SD = 0.96), prior experience (M = 2.46, SD = 1.29), raising
questions (M = 2.43, SD = 0.99), and completing course assignments (M = 2.30,
SD = 1.00) were the least significant barriers, respectively.
185
Table 5: Descriptive results of student-related barriers
Statement N M SD
1. My students used e-learning prior to Covid-19. 856 2.46 1.29
2. My students have sufficient knowledge in the use of e-learning. 856 2.68 1.17
3. My students have the necessary skills for the use of e-learning. 856 2.66 1.13
4. My students have devices (i.e. laptops and tablets) for the use of e-
856 2.89 1.07
learning.
5. My students received training on the use of the e-learning system
856 2.99 1.19
prior to the Covid-19 pandemic.
6. My students are interested in using e-learning. 856 2.80 1.04
7. My students have an internet connection. 856 2.95 0.94
8. My students are able to access the e-learning system. 856 2.55 1.01
9. My students' English skills are a barrier to using the e-learning
856 3.00 1.11
system.
10. My students are able to do online assignments during Covid-19. 856 2.30 1.00
11. My students are able to raise questions and concerns during the
856 2.43 0.99
course in the e-learning system.
12. My students are able to progress in the course during Covid-19. 856 2.49 0.96
13. My students find online assessment challenging. 856 2.58 1.03
14. My students find e-learning hectic because of the number of
856 2.66 1.00
online courses during Covid-19.
15. My students are resistant to participate in e-learning activities
856 2.96 1.06
due to cultural norms.
4.2 Relationship between E-Learning Implementation Barriers

We conducted an analysis of Spearman’s correlation coefficient in order to
pinpoint the relationship between each category of barriers to e-learning
implementation. As Table 6 demonstrates, significant positive correlations
existed among all levels, but the correlations were of different strengths. The
results revealed that the strongest correlation was between teacher-related
barriers and institution-related barriers (rₛ = .605, p = .000, N = 856). Meanwhile,
the association between teacher-level barriers and student-related barriers was
moderate, but still statistically significant (rₛ = .553, p = .000, N = 856). We found
a moderate positive correlation between student-related barriers and institution-
related barriers (rₛ = .548, p = .000, N = 856) and between curriculum-related
barriers and institutional-related barriers (rₛ = .444, p = .000, N = 856). Moreover,
there was only a weak positive correlation between student-related barriers and
curriculum-related barriers (rₛ = .364, p = .000, N = 856), and the lowest positive
correlation among all the categories was between curriculum-related barriers
and teacher-related barriers (rₛ = .271, p = .000, N = 856).
186
Table 6: The relationship between each category of barriers to e-learning

implementation
Teacher-Level Institution- Curriculum- Student-Level
Barriers Level Barriers Level Barriers Barriers
Teacher-Level
1.000 .605** .271** .553**
Barriers
Institution-
1.000 .444** .548**
Level Barriers
Curriculum-
1.000 .364**
Level Barriers
Student-Level
1.000
Barriers
* Correlation interpretation scale: .00–.19 = very weak, .20–.39 = weak, .40–.59 = moderate, .60–
.79 = strong, .80–1.0 = very strong.
** Correlation is significant at the 0.01 level (2-tailed).
4.3 Differences in E-Learning Implementation Barriers in the Context of

Teachers’ Gender, Academic Qualifications, Teaching Experience and Prior
Experience in E-Learning
We administered MANOVA to investigate any possible significant differences in
e-learning implementation barriers in the context of the teachers’ gender,
academic qualifications, teaching experience, and prior experience in e-learning
(see Table 7). There was a statistically significant difference between e-learning
implementation barriers and academic qualifications: F (8, 1700) = 4.51, p < .05;
Wilk's Λ = 0.959, partial η2 = .021. The MANOVA also revealed significant
differences in e-learning implementation barriers due to teaching experience: F
(12, 2246) = 2.07, p < .05; Wilk's Λ = 0.971, partial η2 = .010. There were also
significant differences with prior experience in e-learning: F (4, 851) = 12.66, p <
.05; Wilk's Λ = 0.944, partial η2 = .056. Furthermore, the results did not indicate
any significant difference in e-learning implementation barriers resulting from
gender: F (4, 851) = 1.99, p < .05; Wilk's Λ = 0.991, partial η2 = .009. Although the
difference between these variable (gender, academic qualification, teaching
experience, prior experience in e-learning) are minor, the results indicate that
they influenced teacher implementation of e-learning during COVID-19.
Table 7: Results of the MANOVA
Partial Eta
Effect Value F Sig. Squared
Gender Wilks'
.991 1.992b .094 .009
Lambda
Academic Wilks'
.959 4.509b .000 .021
Qualification Lambda
Teaching Wilks'
.971 2.073 .016 .010
Experience Lambda
Prior Experience Wilks'
.944 12.662b .000 .056
in E-Learning Lambda
187
4.4 The Relationship between E-Learning Implementation Barriers and Type

of Institution
We conducted a Spearman’s correlation coefficient to investigate any possible
relationship between e-learning implementation barriers and type of institution.
The results revealed a very weak, but still statistically significant, positive
association between curriculum-level barriers and the type of institution (rₛ =
.071, p = .05, N = 848). This can be due to the different curriculum development
and evaluation frameworks and guidelines implemented at different
institutions. Also. Private higher education tend to have flexible curriculum
guidelines where adapting and enhancing curriculum is accessible. However,
curriculum change is more rigid and centralised in public higher education.
Table 8 shows the relationship between e-learning implementation barriers and
type of institution. The results of this test should be interpreted cautiously as the
number of teachers in the sample were more from public institutions.
Table 8: The relationship between e-learning implementation barriers and type of

institution
Teacher- Curriculum- Student-
Level Institution- Level Level
Barriers Level Barriers Barriers Barriers
Institution .000 -.067 .071* .019
* Correlation is significant at the 0.05 level (2-tailed).
4.5 Results of the Open-Ended Questions

There was a good response rate for the open-ended questions in the
questionnaire (the number of responses for question 1 = 590, and the number of
responses for question 2 = 582). The first open-ended question encouraged
participants to voice any issues they faced during ERT, while the second
question allowed participants to provide suggestions to improve e-learning
implementation practices for ERT during Covid-19.
The results revealed that teachers faced various issues with ERT during Covid-
19. First, the teachers reported facing issues related to technological
infrastructure, including poor internet connectivity and lack of hardware and
software for online teaching and learning. Consequently, these issues
contributed to the failure of many of the online classes conducted during Covid-
19.
Our results also indicated some student-related issues. According to the
teachers, many students encountered challenges due to poor network coverage,
and students were not motivated to attend their online classes. Teachers
attributed this issue to a lack of proper training on the use of live streaming
platforms, course management systems and online learning skills and strategies.
Moreover, the teachers reported issues with monitoring student attendance
during the online classes. Some respondents reported that these difficulties were
related to the students’ joining and leaving classes frequently during the same
online class due to internet connection issues.
188
In addition, the results revealed that skills and knowledge of online teaching
and learning were insufficient for some of the teachers. Accordingly, their
inexperience in online pedagogy was reported in two contexts. First, the
participants who faced these challenges were not competent in using the
required applications for online teaching and learning, such as course
management systems and video conferencing software. Second, they
demonstrated insufficient knowledge of online pedagogy. In other words, they
lacked knowledge of the teaching methods required to administer online
instruction.
Furthermore, the participants reported issues with the curriculum. A common
issue was that the existing face-to-face courses were not suitable for online
teaching; the courses contained materials that were challenging for teachers to
cover through online classes. Finally, the course assessments for the face-to-face
classes were not suitable for online classes.
Our analysis showed that there were issues in relation to institutions’ policies
and guidelines. The participants reported that their institutions lacked online
teaching and learning policies and guidelines. They also stated that their
institutions were unable to monitor and assess the quality of e-learning during
this period due to a lack of specific quality assurance measures for ERT.
Meanwhile, only a few respondents mentioned issues with time; those who
reported time as an issue stated that they found online teaching to be time-
consuming.
The teachers who participated in our study provided a variety of
recommendations to enhance online teaching and learning through ERT during
Covid-19. The majority of the respondents emphasised the need for proper
student training, which would benefit the quality of both teaching and learning
in the online context. They recommended student training in various areas,
including video streaming software, learning management systems and screen
and voice recording software.
The teachers also recommended enhancements to the technological
infrastructure. Their recommendations can be divided into two main parts.
Some of the recommendations focused on enhancing internet connectivity at
both the national and institutional levels. In addition, the participants
recommended providing and upgrading software, hardware and network
facilities at the institutional level. Some teachers also recommended establishing
video recording studios to aid in recording professional videos for online
classes.
Furthermore, the teachers recommended some amendments to teaching and
learning policies and institutional guidelines. Some of the teachers
recommended establishing new policies that are specific to ERT, including
changes to attendance policies, quality assurance and quality enhancement
policies, and student support services.
Finally, the participants recommended a redesign of courses and assessments.
Teachers who taught practical courses asserted the need to change assessment
patterns to suit online instruction, and some of the teachers recommended
189
including short quizzes prior to and after classes to ensure student readiness and
to monitor and assess their understanding of the content.
5. Discussions
5.1 Students preparedness for Emergency Remote Teaching
The current study reveals that students were not prepared for ERT, and they
demonstrated insufficient online learning skills, strategies and knowledge.
Similar observations have been reported in previous research. Alqudah et al.
(2020) identified poor e-learning skills, unpreparedness and inadequate internet
accessibility for students as the main obstacles hindering the continuity of
learning in the ERT period in Jordan. Research has also confirmed that student
issues such as readiness, motivation, accessibility and equity negatively
impacted e-learning practices during Covid-19 (Adnan & Anwar, 2020; Bao,
2020; Hartshorne et al., 2020; Landrum, 2020; Mailizar et al., 2020). Prior Covid-
19 research has confirmed similar findings (Assareh & Hosseini, 2011; Naveed et
al., 2017; Pelgrum, 2001). Therefore, this suggests that students themselves might
pose a serious challenge to e-learning implementation in an ERT context.
Another possible explanation of this issue that should be considered, however, is
that institutions may not have had sufficient time to prepare their students for
the sudden shift to ERT.
5.2 Curriculum and E-Learning Implementation during COVID-19
The positive correlation between curriculum-related and institutional-related
barriers signifies that the existing curriculum is not suited for ERT. A possible
reason for this might be that most of the courses offered during ERT were not
subject to sound instructional design due to the emergency situation. This is
supported by Hodges et al.'s (2020) argument that online course design during
ERT is often rushed with minimum resources, and little attention is paid to
typical planning and preparation. This finding also confirms the work of
Crawford et al. (2020) and Bao (2020), both of whom claimed that shifting
courses to online delivery during Covid-19 lacked systematic course design and
evaluation. Furthermore, recent research has pointed out the need for systematic
planning when shifting to online instruction. Karalis (2020), for example,
emphasised that piloting – piloting the new course, assessment and teaching
materials – is an essential and integral part of shaping proper responses to
emergencies in education. Meanwhile, along the same line of thought, Gacs et al.
(2020) stressed the importance of a quick needs analysis to understand teacher
and student needs in relation to technology, workload, accessibility, equity,
readiness, motivation and prior knowledge.
5.3 Institutional Readiness for Emergency Remote Teaching
The positive correlation between teacher-related barriers and institution-related
barriers can explain the unpreparedness for the shift to ERT during Covid-19.
Our results confirm the lack of digital education policies and procedures at most
higher education institutions in Oman. Additionally, the dearth of quality
assurance measures and polices concerned with emergency teaching and
learning negatively affected ERT practices. This finding relates not only to
Omani higher education, but can be generalised to educational systems
worldwide (see Crawford et al., 2020). Findings from the current research
indicate the need for teacher professional development, especially since
190
inadequate ICT skills in teachers has been reported as a crucial hindrance of e-

learning implementation (Alqudah et al., 2020; Assareh & Hosseini, 2011; Bao,
2020; Chang & Fang, 2020; Crawford et al., 2020; Naveed et al., 2017; Pelgrum,
2001; Sharpe et al., 2003).
5.4 Other Influential Barriers
Academic qualifications and teaching experience played a significant role in e-
learning implementation for ERT during Covid-19. Although teachers’ academic
qualifications and teaching experience have a positive effect on their
implementation of e-learning, teachers advocated for training provisions
specifically related to the ERT context. Previous research confirms the finding
that prior experience and academic qualifications of teachers positively correlate
to the successful implementation of e-learning (Cidral et al., 2018; Hartshorne et
al., 2020; Mahdizadeh et al., 2008; Mailizar et al., 2020; Naveed et al., 2017;
Reeves & Pedulla, 2011).
Meanwhile, the type of institution did not lead to any significant differences in
e-learning implementation for ERT. In other words, the experience of public and
private educational providers was the same, which indicates that neither sector
was fully prepared for the sudden shift to ERT.
5.5 Requirements for Successful Emergency Remote Teaching
The evidence from this study suggests that higher education institutions should
pay attention to student training provisions. Higher education providers should
consider providing and/or enhancing students’ internet access to maximise their
ability to participate in online learning. Although our findings relate to e-
learning implementation for ERT during Covid-19, they are applicable to normal
teaching settings and face-to-face teaching that depends, to some extent, on e-
learning solutions. The current study also highlights the need for a national
policy for digital delivery of higher education and a national policy for ERT.
Although the shift to online instruction was accomplished rapidly, it is
important that institutions follow a systematic plan for the shift to online
instruction that considers the needs of both teachers and students. Additionally,
it is necessary that periodic and quick evaluation of courses that are shifted
online during ERT be implemented. While it might be argued that this may not
be possible or feasible during exceptional circumstances, institutions should
ensure that they review and pilot parts of their courses concurrently to online
instruction. Student, teacher and parent feedback should also be considered
when evaluating online instruction during ERT. Finally, this work demonstrates
the importance of enhancing institutional technological infrastructure.
5.6 Limitations and Future Research
A number of limitations need to be considered. First, the study only included
English-speaking teachers. The questionnaire was not translated into Arabic,
which is spoken by many university teachers in Oman. Second, the study
sample only included teachers. Including students in the study sample would
have affected the conclusions we were able to draw, especially with regard to
student-related barriers to e-learning implementation.
However, the findings of this study do bring up a call to teachers, institutions
and policy makers to further investigate issues related to e-learning in ERT
191
contexts. Future research should explore student training provisions in online

learning during ERT and could also investigate the professional development of
higher-education teachers in ERT contexts. Design-based research can be useful
for designing and piloting teacher- and student-training programmes, and
would be useful to assess institutions’ readiness for online instruction and
course design during COVID-19 and other unexpected emergencies in the
future.
6. Conclusion
Using a cross-sectional survey design, we examined teachers’ views on the
implementation of e-learning in ERT during COVID-19. Student-related barriers
had a strong impact on teachers’ use of e-learning with teachers’ academic
qualifications and prior experience having a less significant impact on teachers’
use of e-learning in ERT during COVID-19. The study recommends training
programmes for both students and teachers to prepare them for handling
learning and teaching in ERT contexts. In addition, the study recommends
higher education institutions to adapt their curriculum to suite ERT. As teachers
were not prepared for the sudden shift to ERT during COVID-19, higher
education providers need to implement professional development provisions
and amend their teaching and learning policies to suite the ERT context.
7. References
Adnan, M., & Anwar, K. (2020). Online learning amid the COVID-19 pandemic:
Students’ perspectives. Journal of Pedagogical Sociology and Psychology, 2(1), 45–51.
https://doi.org/10.33902/jpsp.2020261309
Alqudah, N. M., Jammal, H. M., Saleh, O., Khader, Y., Obeidat, N., & Alqudah, J. (2020).
Perception and experience of academic Jordanian ophthalmologists with E-
Learning for undergraduate course during the COVID-19 pandemic. Annals of
Medicine and Surgery, 59(September), 44–47.
https://doi.org/10.1016/j.amsu.2020.09.014
Assareh, A., & Hosseini, B. M. (2011). Barriers to E-teaching and E-learning. Procedia
Computer Science, 3, 791–795. https://doi.org/10.1016/j.procs.2010.12.129
Bao, W. (2020). COVID ‐19 and online teaching in higher education: A case study of
Peking University. Human Behavior and Emerging Technologies, 2(2), 113–115.
https://doi.org/10.1002/hbe2.191
Baran, E., & Alzoubi, D. (2020). Human-centered design as a frame for transition to
remote teaching during the COVID-19 pandemic. Technology and Teacher
Education, 28(2), 365–372.
Chang, C. L., & Fang, M. (2020). E-Learning and Online Instructions of Higher Education
during the 2019 Novel Coronavirus Diseases (COVID-19) Epidemic. Journal of
Physics: Conference Series, 1574(1). https://doi.org/10.1088/1742-
6596/1574/1/012166
Cidral, W. A., Oliveira, T., Felice, M. Di, & Aparicio, M. (2018). E-learning success
determinants: Brazilian empirical study. Computers and Education, 122, 273–290.
Crawford, J., Butler-henderson, K., Rudolph, J., Malkawi, B., Glowatz, M., Magni, P., &
Lam, S. (2020). COVID-19: 20 countries’ higher education intra-period digital
pedagogy responses. Journal of Applied Learning & Teaching, 3(1), 1–20.
https://doi.org/10.37074/jalt.2020.3.1.7
Creswell, J. W. (2018). Educational Research: Planning, Conducting and Evaluating
Quantitative and Qualitative Research (6th edition). Pearson.
192
Demuyakor, J. (2020). Coronavirus (COVID-19) and online learning in higher institutions

of education: A survey of the perceptions of Ghanaian international students in
China. Online Journal of Communication and Media Technologies, 10(3).
https://doi.org/10.29333/ojcmt/8286
Dhawan, S. (2020). Online learning: A panacea in the time of COVID-19 crisis. Journal of
Educational Technology Systems, 49(1), 5–22.
https://doi.org/10.1177/0047239520934018
Fryer, L. K., & Bovee, H. N. (2016). Supporting students’ motivation for e-learning:
Teachers matter on and offline. Internet and Higher Education, 30, 21–29.
https://doi.org/10.1016/j.iheduc.2016.03.003
Gaber, D. A., Shehata, M. H., & Amin, H. A. (2020). Online Team- Based Learning
Sessions as Interactive Methodologies During The Pandemic. Medical Education,
March, 14198. https://doi.org/10.1111/medu.14198
Gacs, A., Goertler, S., & Spasova, S. (2020). Planned online language education versus
crisis-prompted online language teaching: Lessons for the future. Foreign
Language Annals, May, 1–13. https://doi.org/10.1111/flan.12460
González-gómez, F., Guardiola, J., Martín, Ó., Ángel, M., & Alonso, M. (212 C.E.).
Gender differences in e-learning satisfaction. Computers & Education, 58(1), 283–
290.
Hartshorne, R., Baumgartner, E., & Kaplan-rakowski, R. (2020). Preservice and inservice
professional development during the COVID-19 pandemic. Journal of Technology
and Teacher Education, 28(2), 137–147.
Hodges, C., Moore, S., Lockee, B., Trust, T., & Bond, A. (2020). The difference between
emergency remote teaching and online learning. Educause Review, 7.
https://er.educause.edu/articles/2020/3/the-difference-between-emergency-
remote-teaching-and-online-learning
Jones, A. (2003). ICT and future teachers Are we preparing for e-learning. In IFIP
Advances in Information and Communication Technology (Vol. 132).
https://doi.org/10.1007/978-0-387-35701-0
Karalis, T. (2020). Planning and evaluation during educational disruption: Lessons
learned from Covid-19 pandemic for treatment of emergencies in education.
European Journal of Education Studies, 7(4), 2020.
https://doi.org/10.5281/zenodo.3789022
Landrum, B. (2020). Examining students’ confidence to learn online, self-regulation skills
and perceptions of satisfaction and usefulness of online classes. Online Learning
Journal, 24(3), 128–146. https://doi.org/10.24059/olj.v24i3.2066
Mahdizadeh, H., Biemans, H., & Mulder, M. (2008). Determining factors of the use of e-
learning environments by university teachers. Computers and Education, 51(1),
142–154. https://doi.org/10.1016/j.compedu.2007.04.004
Mailizar, Almanthari, A., Maulina, S., & Bruce, S. (2020). Secondary school mathematics
teachers ’ views on e-learning implementation barriers during the COVID-19
pandemic: The case of Indonesia. Eurasia Journal of Mathematics, Science and
Technology Education, 16(7). https://doi.org/10.29333/ejmste/8240
McMaster, D., Veremu, M., & Santucci, C. (2020). COVID-19: opportunities for
professional development and disruptive innovation. Clinical Teacher, 17(3), 238–
240. https://doi.org/10.1111/tct.13175
Naveed, Q. N., Muhammed, A., Sanober, S., Qureshi, M. R. N., & Shah, A. (2017).
Barriers effecting successful implementation of E-learning in Saudi Arabian
universities. International Journal of Emerging Technologies in Learning, 12(6), 94–
107. https://doi.org/10.3991/ijet.v12i06.7003
Pelgrum, W. J. (2001). Obstacles to the integration of ICT in education: Results from a
193
worldwide educational assessment. Computers and Education, 37(2), 163–178.

https://doi.org/10.1016/S0360-1315(01)00045-8
Rapanta, C., Botturi, L., Goodyear, P., Guàrdia, L., & Koole, M. (2020). Online university
teaching during and after the Covid-19 crisis: Refocusing teacher presence and
learning activity. Postdigital Science and Education, 2, 923–945.
http://link.springer.com/10.1007/s42438-020-00155-y
Reeves, T. D., & Pedulla, J. J. (2011). Predictors of teacher satisfaction with online
professional development: Evidence from the USA’s e-learning for educators
initiative. Professional Development in Education, 37(4), 591–611.
https://doi.org/10.1080/19415257.2011.553824
Schoepp, K. (2005). Learning and Teaching in Higher Education: Gulf Perspectives
Barriers to Technology Integration in a Technology-Rich Environment. Learning
and Teaching in Higher Education: Gulf Perspectives, 2, 1–24.
http://www.zu.ac.ae/lthe/vol2no1/lthe02_05.pdf
Sharpe, R., Benfield, G., Roberts, G., Francis, R., Garrison, D. R., Anderson, T., Welsh, E.
T., Wanberg, C. R., Brown, K. G., Simmering, M. J., Sun, P. C., Tsai, R. J., Finger,
G., Chen, Y. Y., Yeh, D., Selim, H. M., Guri-Rosenblit, S., Chan, T.-W., Roschelle,
J., … Walton, G. (2003). What drives a successful e-Learning? An empirical
investigation of the critical factors influencing learner satisfaction. Computers &
Education, 49(4), 245–258. https://doi.org/10.1111/j.1470-3327.2005.00614.x
Sørebø, Ø., Halvari, H., Gulli, V. F., & Kristiansen, R. (2009). The role of self-
determination theory in explaining teachers’ motivation to continue to use e-
learning technology. Computers and Education, 53(4), 1177–1187.
UNESCO. (2020). Global Education Coalition: Covid-19 Education Response.
https://en.unesco.org/covid19/educationresponse/globalcoalition
Wilson, A. (2012). Effective professional development for e-learning: What do the
managers think? British Journal of Educational Technology, 43(6), 892–900.
https://doi.org/10.1111/j.1467-8535.2011.01248.x
Zuo, L., Dillman, D., & Miller Juve, A. (2020). Learning At-Home During COVID-19: A
Multi-institutional Virtual Learning Collaboration. Medical Education, March, 1–2.
https://doi.org/10.1111/medu.14194
194

Vol. 20, No. 8, pp. 194-209, August 2021
Implications of Social Media Addiction on

Academic Performance among Generation Z
Student-athletes during COVID-19 Lockdown
Wan Ahmad Munsif Wan Pa, Muhammad Sofwan Mahmud* and

Mohd Syazwan Zainal
Universiti Kebangsaan Malaysia, Malaysia
https://orcid.org/0000-0001-6218-4376
https://orcid.org/0000-0002-0504-4622
https://orcid.org/0000-0002-1450-2760
Abstract. With the rapid advancement of internet technology, the

increasing prevalence of smartphone ownership at a young age, the
uncertainty surrounding the end of the COVID-19 pandemic, and the
effect of social media addiction on academic performance among
students, social media use will continue to be a major point of
contention. However, student-athletes are a distinct group of students
who should not be overlooked due to their concurrent commitments to
studying, training, and competing. As such, this study sought to
investigate the effect of social media addiction on academic performance
among Generation Z student-athletes during Malaysia's COVID-19
lockdown. The research design used in this study was a quantitative
cross-sectional survey. This study surveyed 91 respondents and utilized
the Social Media Addiction Questionnaire (SMAQ) and Grade Point
Average (GPA). The findings indicated that Facebook was the preferred
social media platform over Instagram, Twitter, and YouTube, and that
more than half of the participants were social media addicts. The
researchers concluded that social media addiction had a significant
impact on academic performance, accounting for 57.6 percent of the
variance. These findings imply that excessive social media use, leading
to addiction, has a significant negative effect on academic performance.
Solutions are desperately needed, such as turning off notifications,
silencing the phone, engaging in home workouts, reconnecting with
family, or taking up new hobbies.
Keywords: social media addiction; academic performance; student-

athletes
* Corresponding author: Muhammad Sofwan Mahmud; sofwanmahmud@ukm.edu.my
©Authors
195
1. Introduction
Coronavirus 2019 (COVID-19) was discovered for the first time in December
2019 in Wuhan, Hubei Province, China, causing the World Health Organization
(WHO) to declare it a global pandemic which poses a physical and psychological
threat to the public. The continued spread of COVID-19 is a public health
emergency and a global hazard. To prevent the disease from spreading further,
governments have issued emergency orders for people to stay at home, while
universities and schools have been closed. It has been estimated that, by March
26, 2020, the closures affected more than 150 million children and adolescents in
165 nations (UNESCO, 2020). Due to the COVID-19 pandemic, higher education
institutions have been forced to implement emergency online learning (Yahya,
Rahman & Siddiq, 2021). On top of that, this pandemic has become a source of
stress since it has had an impact on all sectors of human existence, including
economy, social activities, health, education, politics, culture, and
communication.
Based on statistics provided by the Ministry of Health (2021), Selangor was one
of the states in Malaysia with the highest number of daily positive COVID-19
cases during the movement control order 2.0 (MCO). During this pandemic
lockdown, communication has encompassed much more than just sending
messages to people. Communication is the process of conveying or exchanging
information through speech, writing, or any other medium (Buck, 2019).
Additionally, information and communication technologies (ICTs) are now
widely used in virtually every aspect of human life, including entertainment,
education, decision-making, information searching, commerce, and business. In
terms of benefits, ICTs can help people to save time, effort, and money (Filimon
et al., 2017).
The importance of communication through social media is obviously seen as a

funnel for information between family members, education, entertainment, and
other activities, as well as a vehicle for a child's socialization and education from
birth to adulthood. Social media platforms are one of the most widely used
forms of information on the planet. Social media platforms include Facebook,
Instagram, Twitter, Snapchat, YouTube, blogging platforms, Newsfeeds, Wikis,
and social gaming platforms (Kuss et al., 2014). On the other hand, excessive use
of social media has developed into a problem for some individuals who exhibit
signs of behavioral addiction such as salience, mood modulation, withdrawal,
tolerance, and conflict (Hawi & Samaha, 2019).
However, a study on social media addiction among Generation Z student-

athletes was limited by the fact that previous research focused exclusively on
sedentary Generation Z students (Curtis et al., 2019; Gazi & Çetin, 2017; Grau,
Kleiser & Bright, 2019). The requirements of a student-athlete's academic course
must be balanced with their athletic activity. Ryba et al. (2015) asserted that the
economic, structural, and institutional rules in place create a one-of-a-kind
structure that enables a student-athlete to pursue a successful dual profession.
These Generation Z student-athletes are not immune to the internet's rapid
growth, technological advancements, and social media proliferation.
196
2. Problem Statement
When COVID-19 isolates households from one another, parents and guardians
may become concerned about their own financial future, making learning at
home more difficult, particularly for children who lack motivation (Daniel,
2020). Students' lives have been significantly impacted by the COVID-19
pandemic in a variety of ways, depending on their level and field of study, as
well as their program of study. Additionally, several students have found
themselves unable to complete their university curriculum and assessments in a
timely manner, and in many cases, they have been immediately cut off from
their social group. Social media is critical as a communication tool for conveying
information in these circumstances. Individuals frequently turn to the media in
response to everyday annoyances and adversity. According to Zhao and Zhou
(2021), when confronted with COVID-19 problems, such as venting emotions for
mood management or joining online communities for social support, people
tended to turn to social media for problem-focused activities, such as searching
for health-related information and emotion-focused coping.
Despite the obvious benefits of social media in an emergency, such as COVID-

19, increased social media use is likely to result in social media addiction (Zhao
& Zhou, 2021). Social media's positive and negative effects on society are well
established (Bajpai, 2018). On the other hand, they promote a sense of belonging
and an excellent marketing or communication tool for businesses, political
parties, and governments. Interacting with others on social media aids in the
development of communication and technical abilities. Meanwhile, concerns
have been raised about possible links between frequent social media use and
anxiety and depression (Pa et al., 2020), as well as issues surrounding
cyberbullying, online harassment, and "trolling."
According to Hootsuite and We Are Social's digital usage studies, Malaysia has
the fifth-highest mobile social media penetration rate in the world and the
highest in Southeast Asia (Bernama, 2019). According to the same survey,
internet penetration has reached 81% in Malaysia, with users spending an
average of eight hours and five minutes online. During the same time period,
individuals spent two hours and 59 minutes on social media platforms like
Facebook, Instagram, and Twitter. On the other hand, many individuals take
advantage of these opportunities by devoting significant time to social media
(Schulze, Scholer & Skiera, 2015). Numerous studies have established a link
between social media use and academic achievement and performance among
international undergraduate university students. According to a nationwide
survey of 3000 college students, 90% of students use Facebook and 37% use
Twitter (Dahlstrom et al., 2011). As a result of the long hours they spend on
social media, students are more susceptible to addiction, which can result in
assignment procrastination, loss of focus, and can have a negative impact on
their academic performance.
Moreover, student-athletes are among those affected by the lockdown

pandemic. They had to cancel training and tournament for an uncertain period
of time. Furthermore, they have a lot of free time and can easily become addicted
197
social media users. This situation is supported by statistics reported by Kashif

and Aziz-Ur-Rehman (2020) in which most people are becoming addicted to
social media because of the spare time they have gained due to the suspension of
all outside activities; many also reported feeling agitated and nervous if they did
not use social media during the coronavirus lockdown. People have been
spending more time on social media, which causes them to be distracted and
delayed in their work.
Additionally, there is evidence of the devastating effect of social media on

academic success, particularly Facebook, and smartphone use (Paul, Baker, &
Cochran, 2012; Junco & Cotton, 2012; Samaha & Hawi, 2016; Lepp, Barkley, &
Karpinski, 2015; Samaha & Karpinski, 2016). When these tools are combined,
students spend more time interacting with them than they do learning. Recent
studies indicate that students' excessive use of social media can have a
detrimental effect on their academic performance (Skiera, Hinz, & Martin-Span,
2015; Al-Menayes, 2014; Skiera, Hinz, & Martin-Span, 2015). Due to their
pervasiveness, students are more susceptible to social media and smartphones
than the general population (Giunchiglia et al., 2018). On the other hand,
student-athletes are a category of students that should not be ignored because
they are committed to three main goals at the same time, which are studying,
training, and competing.
As a result of the difficulties, the researchers gained a better understanding of

the negative impact of social media addiction on academic performance.
Additionally, this type of study on Malaysian Generation Z student-athletes was
uncommon, as previous studies focused on the general university student
population. The most frequently used social media platform by Generation Z
student-athletes in Malaysia is unknown. It is critical to understand the
mediators of the relationship between social media addiction and academic
performance in order to identify risk factors and develop social media addiction
prevention strategies. Indeed, these issues inspired the researchers to conduct a
study on the impact of social media addiction on academic performance.
Therefore, the present research aims to:
1. To recognize the most frequently used social media platforms by
Generation Z student-athletes during MCO 2.0.
2. To identify the extent to which Generation Z student-athletes are addicted
to social media.
3. To determine the impact of social media addiction on academic
performance in Generation Z student-athletes.
The research questions:

i. What is the most common type of social media platform used by
Generation Z student-athletes during MCO 2.0?
ii. What is the level of social media addiction among Generation Z student-
athletes?
iii. What is the effect of social media addiction on academic performance
among Generation Z student-athletes?
198
The null hypothesis:

Ho: There is no significant impact of social media addiction on academic
performance among Generation Z student-athletes.
During the COVID-19 pandemic, excessive levels of stress and addiction to
social media are positively related (Brailovskaia et al., 2019). At the same time,
people are helpless because they need to maintain their social media activity and
communicate with their loved ones through social media (Perrin, 2015). Indeed,
many people have been spending an increasing amount of time on social media
platforms like Facebook and Twitter, in addition to remote work and study,
which may satisfy their needs for disaster-related information, entertainment,
and interpersonal communication (Merchant & Lurie, 2020). Zhao and Zhou
(2021) reported that when individuals are exposed to COVID-19-related stress,
they are more likely to actively engage with social media, such as sharing
personal feelings in order to alleviate unpleasant emotions.
According to statistics, the total number of users on social media has been
increasing by 10 percent on a yearly basis, (Hawi & Samaha, 2017). Numerous
researchers and academics are examining the effects of social media technologies
and applications on various aspects of people's lives, as they have become an
inseparable part of daily life in recent years (Filimon et al., 2017). With the
increasing use of smartphone technology, social media usage among adults aged
18 to 29 years in the United States of America has surged from 12 percent to 90
percent since 2005 (Sutherland et al., 2018). Due to the ease and low cost of
Internet access and the enormous number of registered users on social media,
these platforms are one of the simplest and most effective methods of spreading
and obtaining information. Whether it is a sporting event, disease, or natural
disaster, the overall response to important events is usually to search for
information (González-Padilla, 2020). According to a recent study, users use
social media for a variety of reasons, including seeking knowledge, seeking
social engagement, avoiding boredom and leisure, fleeing unpleasant emotions,
and seeking happy ones (Brailovskaia, Schillack & Margraf, 2020). Thus, social
media has a great impact on how people connect, work, and share information.
Social media networking via smartphone has also become the fastest growing
method of affiliation due to its ability to quickly transmit essential information.
Although social media has had an impact on a variety of businesses, it has had
the most significant impact on classroom teaching and the educational system as
a whole (Sivakumar, 2020). Some institutions use social media to promote
programs and engage with current and potential students, as well as alumni,
due to the popularity of social media. Some academics have embraced social
media to improve the educational process and cooperate on projects, as well as
to improve learning outcomes by communicating with students more effectively.
According to the literature, undergraduate students at bachelor level are the
ones who use social media platforms to engage with other students and
university professors (Duggan et al., 2015; Head, 2013; Kim, Sin & Tsai, 2014).
199
Students' use of social media applications has exploded in the last decade, and
the impact on academic performance has been mixed (Alnjadat et al., 2019). A
survey of 30 male students at the Petroleum Institute in Abu Dhabi, in the
United Arab Emirates (UAE), discovered that students with a high Grade Point
Average (GPA) spend a significant amount of time on social media websites and
applications (Tayseer et al., 2014). According to Alahmar (2017) and Boateng and
Amankwaa (2016), social media was used positively in the learning process in
China and Iraq, with students using the platform to discuss assignments and
other coursework; receive news and updates on class schedules and locations;
exchange information with peers; and express concerns about their academic
work. Similarly, research conducted in Australia in 2018 reported that 67 percent
of 106 undergraduate students polled cited technology as a factor in their
successful academic achievement (Sutherland et al., 2018). Thus, social media
has become ingrained in the lives of Internet users, and the increasing
pervasiveness of smartphones is just accelerating this trend (Giunchiglia et al.,
2018).
According to Al-Menayes (2014), while strolling, traveling by bus or train, or

even driving a car, smartphones can be used as instructional and communication
devices. A decade ago, micro time intervals during which users could engage in
a variety of online activities were not possible. Micro time slots, on the other
hand, can result in compulsive smartphone use, which prevents face-to-face
interaction and has a detrimental effect on academic performance. Students who
spent an excessive amount of time interacting on social networking sites were
more likely to fail their classes (Alnjadat et al., 2019). Additionally, students who
were heavily involved in social networking site activities while studying lacked
focus and were easily distracted (Okyeadie Mensah & Nizam, 2016).
On top of that, Generation Z is the generation that has grown up with a lot of
technology, gadgets, and access to the Internet. According to Glass (2007) in
Salleh, Mahbob and Baharudin (2017), a generation is defined as the length of
time that individuals live in a certain year or a cohort of that year. Generation Z
is the most recent generation to be born, and will rule the planet for the next
decades (Wiedmer, 2015; Salleh, Mahbob & Baharudin 2017). Singh (2014),
Jaleniauskiene and Juceviciene (2015) and Schroer (2016) defined Generation Z
as those born between 1995 and 2012. This generation appears to have been
genetically modified by the Internet since birth. This is because they were born
after the Internet became widely used around the world. Table 1 shows the
dominant characteristics of each generation in the 20th and 21st centuries
(MacKenzie & McGuire, 2016; Glass, 2007; Wiedmer, 2015):
Table 1: Generation dominant characteristic

Generation Range of birth Dominant behavioral
years characteristics
Traditionalist 1927-1945 Disciplined and loyal
Baby Boomers 1946-1964 Work ethics that are responsible
and strong
Generation X 1965-1980 Thinkers who are autonomous and
productive
200
Generation Y 1981-1994 More self-assured in social

situations, but less self-sufficient
Generation Z 1995-2012 Inadequate communication skills,
excessive reliance on technology
According to Salleh, Mahbob and Baharudin (2017), most of the characteristics

described have yet to manifest because this generation were born during
technological explosions. Table 2 illustrates the four specific characteristics of
Generation Z:
Table 2: Characteristics of Generation Z

Characteristics Description
Instant Generation Z has become impatient, requiring
gratification immediate gratification, introverted, and
disconnected from society because of extensive
internet exposure.
Hypertext Generation Z prefers to watch videos over reading
mindset hardcopy materials, such as books and manuals.
Overprotected Spoon-feeding by parents is a common occurrence
in Generation Z. They are likely to demand that
institutions create a similar environment in this
scenario.
Ineffective Generation Z is described as being more
communication individualistic, self-absorbed, and team-oriented
abilities than previous generations on a limited basis.
4. Theoretical Framework and Conceptual Framework

Active use of social media could contribute to the effects of COVID-19-related
stress on the relationship between social media addiction and academic
performance. Active use refers to activities that enable direct communication
with others, such as commenting on friends' posts, tagging, "liking," updating
one's status or sharing images or videos. By contrast, passive use refers to the act
of browsing news feeds or viewing other people's posts without engaging in
direct communication (Burke, Kraut & Marlow, 2011). A student-athlete,
according to Pato et al. (2014), is a full-time university or high school student
who participates in athletics or sports independently or as a member of a
federation, club, or sports organization. Therefore, student-athletes need to
divide their time between training, competition, and studies as well. In this
study, the researchers adopt Social Learning Theory (SLT) as a fundamental
knowledge to understand the addiction to social media among Generation Z
student-athletes. According to the SLT, children and adolescents learn through
observing their surroundings, especially those who are close to them (Bandura,
1977). People are devising new strategies and techniques to inform, prepare, and
teach others, which is a human phenomenon known as planned socialization.
Both natural and deliberate socializing, according to Ransome and Sampford
(2016), are beneficial to living a purposeful life. Figure 1 illustrates the theoretical
framework that consists of three components under SLT, adapted from Bandura
(1977) in Ghazali and Ghani (2018), while Figure 2 illustrates the conceptual
framework that consists of four sections.
201
Personal
factors
Social
Learning
Theory
Behavior Environment
Factors factors
Figure 1: Interconnectedness of behavioral, environmental, and personal aspects that

drive Generation Z behavior.
• Personal factors: huge amount of free time during pandemic lock down,
stress, previous media exposure, business interest.
• Environment factors: social reinforcement, availability, group norms,
peer pressure.
• Behavioral factors: Social media platforms enable students to share,
create, exchange, discuss, and obtain information and ideas online in
networks and virtual communities. However, if the students cannot
control the amount of time using the social media, this situation can
potentially lead to social media addiction behaviors.
COVID-19 Generation Social Academic

lockdown Z student- media performance
athletes addiction
Figure 2: The conceptual framework for the research
The conceptual framework is a generative framework that encapsulates the

research process in its essence (Adom, Hussain & Joe 2018). The conceptual
framework depicted in Figure 2 was created for each of the study's processes
and the underlying variables used to review the new search results. The study
concept is derived from the researchers’ reading of the literature.
5. Research Type, Sample, and Study Context

The researchers used a cross-sectional survey design in this study, combined
with quantitative methods. The sampling technique in this study was a
probability sampling, specifically cluster (area) random sampling. This is
202
because the respondents were located within a 10-kilometer radius, student-

athletes born between 1995 to 2012 and the data collection was performed after
the first phase of COVID-19 MCO 2.0 in Malaysia (19th February 2021 until 3rd
March 2021). This technique was ideal for us as researchers as we had limited
finances, time, and energy resources; while also being concerned about staying
safe from the threat of COVID-19 infection, and our accessibility to respondents,
specifically Generation Z student-athletes who resided on the campus of
Universiti Kebangsaan Malaysia. The University is a public institution of higher
education in Bangi, Selangor. The institutional research board of the University
granted permission for the investigation. Before volunteers began the survey,
they were given a consent form explaining the study's objective and assuring
them that the data collection, storage, and reporting processes would ensure
confidentiality and anonymity. The Statistical Package for the Social Sciences
(SPSS) version 25 was used for data entry and analysis. A simple descriptive
analysis was conducted in the form of numbers and percentages. To discover the
contribution between variables, simple linear regression was implemented as a
test of significance, with a threshold of less than 0.05 considered significant.
6. Data Collection Instrument

The survey was divided into three sections, one for demographic data collection
and the other two for independent research instruments. Gender and social
media type were provided in the demographic information section. The
remaining sections were composed of the Social Media Addiction Questionnaire
(SMAQ), adapted from Hawi and Samaha (2017) and Grade Point Average
(GPA) academic performance. The highest GPA awarded is an A, which is
worth ten points, and the lowest is an E, which is worth zero points. Meanwhile,
The Social Media Addiction Questionnaire (SMAQ) is an eight-item
questionnaire designed to assess social media addiction. Responses were
recorded on a seven-point Likert scale ranging from 1 (strongly disagree) to 7
(strongly agree), with higher scores indicating greater levels of addiction. Table 3
shows the seven-point Likert-type scale; Table 4 shows the three sections of the
research instrument; Table 5 shows the norms for SMAQ; and Table 6 shows the
breakdown of (GPA) academic performance.
Table 3: 7-point Likert-type scale

Strongly Disagree Somewhat Neutral Somewhat Agree Strongly
disagree disagree agree agree
1 2 3 4 5 6 7
Table 4: Three section of Research Instrument

Section Item Category
of data
Section Demographic Gender Nominal
A Type of social media
Section Social media I frequently consider social Ordinal
B addiction media when I am not actively (Likert
using it. scale)
I frequently use social media
for no discernible reason.
203
Conflicts with others have

arisen as a result of my social
media use.
When I feel the need to access
social media, I pause whatever
else I'm doing.
When I use social media, I feel
a sense of connection with
others.
I'm losing track of how much
time I spend on social media.
I'm distressed at the prospect
of being unable to access social
media.
I've been unable to decrease
my reliance on social media.
Section Academic GPA Ordinal
C performance (Rank)
Table 5: Norms for SMAQ

Level of social media addiction Score
Low <7
Moderate 8 – 50
High >51
Table 6: Breakdown of (GPA) academic performance

Grade Value Rank
A 4.00 10
A- 3.67 9
B+ 3.33 8
B 3.00 7
B- 2.67 6
C+ 2.33 5
C 2.00 4
C- 1.67 3
D+ 1.33 2
D 1.00 1
E 0.00 0
Due to lockdown and the 10 km radius of travel restrictions imposed by the

Malaysian government to break the COVID-19 chain, the instrument was
distributed by using email and the respondent required to answer and return to
the researchers through email as well. Previously, a pilot test had been
conducted with 76 Generation Z student-athletes from several public
universities in Malaysia. The reliability was tested using SPSS version 25, and
yielded a Cronbach alpha value between 0.67 – 0.89. According to Chua (2014),
Cronbach’s alpha coefficient should be between 0.65 to 0.95 which is high and
satisfactory. Meanwhile, preliminary analyses of all simple linear regression
analyses were conducted to ensure that the normality, linearity,
multicollinearity, and homoscedasticity assumptions were not violated.
204
7. Findings
The study questionnaire was completed by 91 Generation Z student-athletes out
of a total of 120 invitees (a response rate of 76 %); specifically, 57 (62.6 %) were
male student-athletes and 34 (37.4%) female student-athletes. The range of birth
year was 1995-1998 (36.3%) and 1999-2002 (63.7%). Descriptive analysis was
performed to answer the first and second research questions. Figure 3 showed
type of social media frequently used by respondents.
17.6%
5.5%
52.7%
24.2%
Figure 3: Type of social media frequently used by respondents during MCO 2.0.
Based on Figure 3 above, 52.7% of Generation Z student-athletes preferred to use

Facebook; 24.2 % preferred Instagram; 5.5% preferred to use Twitter; and 17.6 %
preferred YouTube. Figure 4 showed the level of social media addiction among
Generation Z student-athletes.
60.4%
39.6%
Figure 4: Level of social media addiction among Generation Z student-athletes.
205
Based on Figure 4, results showed that 39.6 % of Generation Z student-athletes

were moderately addicted to social media. However, 60.4 % of Generation Z
student-athletes were highly addicted to social media.
Table 6: Simple linear regression test of the impact of social media addiction on
academic performance
Sum of Mean
Model df F Sig.
Squares Square
1 Regression .014 1 .014 .032 .048
Residual 39.283 89 .441
Total 39.297 90
As shown in Table 6, the Simple Linear Regression yielded a significant value of

p = 0.048 (P <0.05). In conclusion, these findings indicate that the null hypothesis
(Ho1) was rejected, social media addiction had significantly contributed to
academic performance. Therefore, the researchers referred to the test of
Coefficients and R² value to determine the extent to which social media use
affects the academic performance of Generation Z student-athletes.
Table 7: Coefficients Test

Unstandardized Standardized
Coefficients Coefficients Percentage
Model t Sig. R R²
B Std. Beta (%)
Error
1 (predictor) 6.068 1.712 3.544 .001
Social 0.48 0.270 .019 .179 .0048 0.019 0.576 57.6%
media
addiction
Predictor: social media addiction
Dependent variable: GPA
As presented in Table 7, the Coefficients test showed that the correlation

between the dependent variable and the predictor variable was R = 0.019. Linear
Regression Analysis was found to be significant [F (1,89) = 0.032, p = 0.048 (P
<0.05)] and R² equivalent to 0.576 showed that 57.6% of the change in the
dependent variable of academic performance was due to social media addiction.
Thus, the researchers concluded that social media addiction significantly
impacted academic performance by 57.6%. This can result in procrastination
about assignments, loss of attention, and can have a negative impact on their
academic achievement. Okyeadie Mensah and Nizam (2016) stated that,
students who were immersed in social networking site activities while studying
lacked focus and were easily distracted. These findings suggest that social media
addiction has a significant negative effect on academic performance, and that
solutions, such as turning off notifications, silencing the phone, engaging in
home workouts, reconnecting with family, or discovering new hobbies are
urgently needed to overcome it. Interventions that can assist Generation Z
student-athletes regulate and lessen their social media addiction are also
critically needed.
206
8. Discussion
The COVID-19 pandemic has posed a serious threat to educational systems. This
perspective offers guidance to academics, administrators, and government
officials on how to address the crisis. With the rapid advancement of internet
technology, the widespread use of smartphones at a young age, and the
uncertainty surrounding the end of the COVID-19 pandemic, social media
addiction will continue to be a topic of discussion in relation to students'
academic performance. It is undeniable that social media has a huge impact on
students from a variety of perspectives, as it has become ingrained in their lives,
particularly social media platforms, such as Facebook, Instagram, Twitter,
Snapchat, and YouTube, as well as blogging sites, newsfeeds, wikis, and social
gaming platforms. Students benefit from their ease of use and accessibility
because it saves them time and energy. For instance, students can now access
Facebook or Instagram from anywhere and at any time via a smartphone. This
social media phenomenon has resulted in a massive information flow in
developed countries and Malaysia, ranging from remote areas to the middle of
cities and regardless of age or race. As a result, most university students can
gain access to a greater variety of information more easily and quickly.
This study investigated the effects of social media on Generation Z student-

athletes. The major goal of this study was to see if there was a correlation
between social media usage and academic performance. Our findings revealed
that more than 50% of Generation Z student-athletes preferred to use Facebook
as their main social media platform compared to Instagram, Twitter and
YouTube. Next, our findings showed that 60.4% of Generation Z student-
athletes were highly addicted to social media during MCO 2.0 in Malaysia. This
could be attributed to a huge amount of free time and stay at home restrictions
imposed by the government. These findings were also in line with previous
research by Kashif and Aziz-Ur-Rehman (2020), that stated most people are
becoming addicted to social media because of the extra free time they now have
due to the suspension of all outside activities, and many individuals are
becoming agitated and frightened if they do not use it during the coronavirus
lockdown. Moreover, our findings revealed that social media addiction
significantly contribute 57.6% to the respondents’ academic performance. Future
researchers can also expand the number of respondents and conduct interviews
using a variety of methods, such as a mix of quantitative and qualitative
techniques. It is also worth mentioning that the research found no gender
differences. Furthermore, results must be replicated for specific demographics,
such as gender and age differences.
In conclusion, no one can be left behind with their social media addiction. We do
not want Generation Z student-athletes to be unproductive citizens who interact
infrequently with other humans and rely heavily on technology, which will
inevitably result in a variety of symptoms in the future. In Malaysia, Generation
Z student-athletes must be media literate users in order to comprehend and
utilize social media to a degree. However, it is necessary to revert to a more
normal state of life in which one understands how to divide time between
oneself, the surrounding community, and spiritual needs.
207
9. Acknowledgements
This study was fully funded by FPEND Research Grant: GG-2019-003,
Pemerkasaan Penerbitan Ahli Pusat Pendidikan dan Kesejahteraan Komuniti, Faculty
of Education, Universiti Kebangsaan Malaysia.
10. References
Adom, D., Hussain, E. K., & Joe, A. A. (2018). Theoretical And Conceptual Framework :
Mandatory Ingredients Theoretical And Conceptual Framework : Mandatory
Ingredients Engineering. International Journal of Scientific Research, 7(1), 93–98.
Al-Menayes, J. (2014). The relationship between mobile social media use and academic
performance in university students. New Media and Mass Communication, 25, 23-
29.
Alahmar, A. T. (2017). The influence of social networks on the grades of medical students at
University of Babylon, Iraq. July. https://doi.org/10.5455/jmas.263824
Alnjadat, R., Hmaidi, M. M., Samha, T. E., Kilani, M. M., & Hasswan, A. M. (2019).
Gender variations in social media usage and academic performance among the
students of University of Sharjah. Journal of Taibah University Medical Sciences,
14(4), 390–394. https://doi.org/10.1016/j.jtumed.2019.05.002
Bajpai, P. (2018). Analyzing Effect of Social Media on Academic Performance of
University Graduates. In Proceedings of the 2018 The 3rd International Conference on
Information and Education Innovations (pp. 40-44).
Bandura, A. (1977). Self-efficacy: toward a unifying theory of behavioral
change. Psychological review, 84(2), 191.
Bernama. (2019). Malaysia Ranks Top 5 Globally in Mobile Social Media Penetration,
Highest in Region. News Straits Times. Retrieved July 13, 2019, from
https://www.nst.com.my/lifestyle/bots/2019/01/456119/malaysia-ranks-top-
5-globally-mobile-social-media-pen etration-highest
Boateng R., & Amankwaa, A. (2016). The impact of social media on students' academic
life in higher education. Glob J Hum Soc Sci: G Linguistics & Education 2016; 16(4).
Brailovskaia, J., Schillack, H., & Margraf, J. (2020). Tell me why are you using social
media (SM)! Relationship between reasons for use of SM, SM flow, daily stress,
depression, anxiety, and addictive SM use–An exploratory investigation of
young adults in Germany. Computers in Human Behavior, 113, 106511.
Brailovskaia, J., Rohmann, E., Bierhoff, H. W., Schillack, H., & Margraf, J. (2019). The
relationship between daily stress, social support and Facebook Addiction
Disorder. Psychiatry research, 276, 167-174.
Buck, R. (2019). Motivation, emotion, cognition, and communication: Definitions and
notes toward a grand theory. Advances in Motivation Science (Vol. 6, pp. 27-69).
Elsevier.
Burke, M., Kraut, R., & Marlow, C. (2011). Social capital on Facebook: differentiating
uses and users. In: Paper Presented at the Proceedings of the SIGCHI Conference on
Human Factors in Computing Systems. Vancouver, BC (2011).
http://doi.org/10.1145/1978942.1979023
Chua, Y. P. (2014). Research Method [Kaedah Penyelidikan]. Buku 1; Edisi Kedua [Book 1;
Second Edition]. McGraw Hill Malaysia Sdn. Bhd.
Curtis, B. L., Ashford, R. D., Magnuson, K. I., & Ryan-Pettes, S. R. (2019). Comparison of
smartphone ownership, social media use, and willingness to use digital
interventions between generation z and millennials in the treatment of substance
use: Cross-sectional questionnaire study. Journal of medical Internet research, 21(4),
e13050.
208
Dahlstrom, E., de Boor, T., Grunwald, P., & Vockley, M. (2011). ECAR: National study of
undergraduate students and information technology. http://net.educause.edu/
ir/library/pdf/ers1103/ers1103w.pdf
Daniel, J. (2020). Education and the COVID-19 pandemic. Prospects, 49(1), 91-96.
Duggan, M., Ellison, N. B., Lampe, C., Lenhart A., & Madden M. (2015). Social media
update. http://www.pewinternet.org/files/2015/01/PI_SocialMediaUpdate201
44.pdf.
Filimon, N., Girona, U. De, & Kiygi-calli, M. (2017). Social Media and Higher Education –
An International Perspective. 10(1), 181–191. https://doi.org/10.14254/2071-
789X.2017/10-1/13
Gazi, M. A., & Çetin, M. (2017). The research of the level of social media addiction of
university students. International Journal of Social Sciences and Education
Research, 3(2), 549-559.
Ghazali, S., & Ghani, N. M. (2018). Perception of Female Students towards Social Media-
Related Crimes. Pertanika Journal of Social Sciences & Humanities, 26(2).
Glass, A. (2007). Understanding generational differences for competitive
success. Industrial and commercial training.
González-Padilla, D. A., & Tortolero-Blanco, L. (2020). Social media influence in the
COVID-19 pandemic. International braz j urol, 46, 120-124.
Grau, S., Kleiser, S., & Bright, L. (2019). Exploring social media addiction among student
Millennials. Qualitative Market Research: An International Journal.
Giunchiglia, F., Zeni, M., Gobbi, E., Bignotti, E., & Bison, I. (2018). Mobile social media
usage and academic performance. Computers in Human Behavior, 82, 177-185.
Hawi, N. S., & Samaha, M. (2017). The relations among social media addiction, self-
esteem, and life satisfaction in university students. Social Science Computer
Review, 35(5), 576-586.
Hawi, N., & Samaha, M. (2019). Identifying commonalities and differences in personality
characteristics of Internet and social media addiction profiles: traits, self-esteem,
and self-construal. Behaviour & Information Technology, 38(2), 110-119.
Head, A. (2013). Learning the ropes: How freshmen conduct course research once they
enter college. SSRN, 2364080. http://dx.doi.org/10.2139/ssrn.2364080
Jaleniauskiene, E., & Juceviciene, P. (2015). Reconsidering university educational
environment for the learners of generation Z. Social Sciences, 88(2), 38-53.
Junco, R., & Cotten, S. R. (2012). No A 4 U: The relationship between multitasking and
academic performance. Computers & Education, 59(2), 505-514.
Kashif, M., & Aziz-Ur-Rehman, M. K. J. (2020). Social media addiction due to
coronavirus. International Journal of Medical Science in Clinical Research and
Review, 3(04), 331-336.
Kim, K. S., Sin, S. C. J., & Tsai, T. I. (2014). Individual differences in social media use for
information seeking. The journal of academic librarianship, 40(2), 171-178.
Kuss, D., M. Griffiths, L. Karila, & J. Billieux. (2014). “Internet Addiction: A Systematic
Review of Epidemiological Research for the Last Decade.” Current Pharmaceutical
Design, 20(25), 4026–4052.
Lepp, A., Barkley, J. E., & Karpinski, A. C. (2015). The relationship between cell phone
use and academic performance in a sample of US college students. Sage
Open, 5(1), 2158244015573169.
MacKenzie, J., & McGuire, R. (2016). The First Generation of the Twenty-First Century.
http://magid.com/sites/default/files/pdf/MagidPluralistGenerationWhitepap
er.pdf
Merchant, R. M., & Lurie, N. (2020). Social media and emergency preparedness in
response to novel coronavirus. Jama, 323(20), 2011-2012.
Ministry of Health. (2021). Situasi Terkini Covid-19 [Current Situation of Covid-19].
209
Okyeadie, M. S., & Nizam, D. (2016). The impact of social media on students’ academic.
Int J Educ Learn Train (IJELT), 1(1): 14e21.
Pa, W. A. M. W., Salamuddin, N., Zin, N. M., & Lian, D. K. C. (2020). Sports Massage
Therapy Towards Pre-Competition Anxiety Among Malaysian High
Performance Tennis Players. In Proceedings of the 1st Progress in Social Science,
Humanities and Education Research Symposium (PSSHERS 2019), 200824.
Pato, A. S., Isıdori, E., Calderón, A., & Brunton, J. (2017). An innovative European sports
tutorship model of the dual career of student-athletes. UCAM.
Paul, J. A., Baker, H. M., & Cochran, J. D. (2012). Effect of online social networking on
student academic performance. Computers in Human Behavior, 28(6), 2117-2127.
Perrin, A. (2015). Social media usage. Pew research center, 125, 52-68.
Ransome, W., & Sampford, C. (2016). Ethics and socially responsible investment: A
philosophical approach. Routledge.
Ryba, T. V., Stambulova, N. B., Ronkainen, N. J., Bundgaard, J., & Selänne, H. (2015).
Dual career pathways of transnational athletes. Psychology of Sport and
Exercise, 21, 125-134.
Salleh, M. S. M., Mahbob, N. N., & Baharudin, N. S. (2017). Overview of “Generation Z”
Behavioural Characteristic and its Effect Towards Hostel Facility. International
Journal of Real Estate Studies, 11(2), 59-67.
Samaha, M., & Hawi, N. S. (2016). Relationships among smartphone addiction, stress,
academic performance, and satisfaction with life. Computers in human
behavior, 57, 321-325.
Schroer, W. J. (2016). Generations X, Y, Z and the Others.
http://socialmarketing.org/archives/generations-xy-z-and-the-others/
Schulze, C., Schöler, L., & Skiera, B. (2015). Customizing social media marketing. MIT.
Singh, A. (2014). Challenges and issues of generation Z. IOSR Journal of Business and
Management, 16(7), 59-63.
Sivakumar, R. (2020). Effects of social media on academic performance of the
students. The Online Journal of Distance Education and e-Learning, 8(2), 90-97.
Skiera, B., Hinz, O., & Spann, M. (2015). Social media and academic performance: Does
the intensity of Facebook activity relate to good grades?. Schmalenbach Business
Review, 67(1), 54-72.
Sutherland, K. E., Davis, C., Terton, U., & Visser, I. (2018). University student social
media use and its influence on offline engagement in higher educational
communities. Student Success, 9(2), 13-24.
UNESCO. (2020). UNESCO rallies international organizations, civil society and private
sector partners in a broad coalition to ensure #Learning Never Stops. 26 March
2020. https://en.unesco.org/news/unesco-rallies-international-organizations-
civil-societyand-private-sector-partners-broad.
Wiedmer, T. (2015). Generations do differ: Best practices in leading traditionalists,
boomers, and generations X, Y, and Z. Delta Kappa Gamma Bulletin, 82(1), 51.
Tayseer, M., Zoghieb, F., Alcheikh, I., & Awadallah, M. N. (2014). Social network:
academic and social impact on college students. In ASEE 2014 Zone I
Conference (pp. 3-5).
Yahya, M. W. B. H., Rahman, T., & Siddiq, A. A. (2021). Online Learning in the Quran
Reading Class during Covid-19 Pandemic. International Journal of Learning,
Teaching and Educational Research, 20(5).
Zhao, N., & Zhou, G. (2021). COVID-19 stress and addictive social media use (SMU):
Mediating role of active use and social media flow. Frontiers in Psychiatry, 12, 85.
210

Vol. 20, No. 8, pp. 210-226, August 2021
The Role of Non-Academic Service Quality in an

Academic-Oriented Context: Structural Equation
Modeling amongst Parents of Secondary
Education Students
Thao-Thanh Thi Phan

Thanh Do University, Kim Chung, Hoai Duc, Hanoi, Vietnam
https://orcid.org/0000-0002-9511-9527
Ha-Giang Thi Tran

Hanoi Metropolitan University, Cau Giay, Hanoi, Vietnam
http://orcid.org/0000-0002-6061-5424
Le-Huong Thi Nguyen

Hoa Binh College of Education, Dan Chu, Hoa Binh, Vietnam
https://orcid.org/0000-0002-8103-2797
Tam-Phuong Pham
Can Tho University, Ninh Kieu, Can Tho, Vietnam
https://orcid.org/0000-0002-1745-9067
Thang-The Nguyen
Vietnam Institute of Educational Sciences, Hoan Kiem, Hanoi, Vietnam
http://orcid.org/0000-0002-3998-0099
Hiep-Hung Pham
Phu Xuan University, Phu Hoi, Hue, Vietnam
EdLab Asia Educational Research and Development Centre, Hanoi, Vietnam
https://orcid.org/0000-0003-3300-7770
Thai-Quoc Cao*
EdLab Asia Educational Research and Development Centre, Hanoi, Vietnam
Hanoi University of Social Sciences and Humanities, Hanoi, Vietnam
http://orcid.org/0000-0001-8762-4809
Abstract. In the education sector, the academic factor is considered the

primary focus when assessing the service quality of schools. However, the
* Corresponding author: Thai-Quoc Cao; Email: thaicao@edlabasia.org
©Authors
International License (CC BY-NC-ND 4.0)
211
traditional approach to service quality proposes that the supplemental

indicators of quality are just as important in meeting customer
expectations. As parents of secondary school students are not directly
involved in the academic service of the school, non-academic factors (e.g.,
administrative quality [ADQ] and physical environment quality [PEQ])
can be considered alternative service quality indicators, subsequently
affecting their satisfaction and loyalty. Using data obtained from a survey
questionnaire with 230 parents of secondary education students in
Vietnam, this study examined non-academic service factors as
antecedents of parent satisfaction and loyalty. Results of the exploratory
factor analysis (EFA) yielded two factors under the non-academic service
dimension: PEQ and ADQ. Results of the structural equation
modeling (SEM) showed that while PEQ affects loyalty both directly and
indirectly, ADQ affects loyalty only indirectly through satisfaction.
Implications for stakeholders, including school principals and
policymakers, have been drawn from the findings of this study.
Keywords: administrative quality; non-academic service quality; parent

loyalty; parent satisfaction; physical environment quality; secondary
education
1. Introduction
With a growing population and an ever-fiercer competition in the market,
Vietnam’s educational sector is striving to adapt to a fast-changing world and the
rapid development of the country. Amidst this situation, the secondary education
sector in Vietnam has to deal with two major obstacles. The first obstacle is unique
to the public secondary school, which emerged from the inevitable transition
toward fiscal autonomy of public education in Vietnam. Traditionally, with
governmental support, the tuition fee for general secondary education is
substantially lower than that of their private counterpart. However, with the
transition toward fiscal autonomy, the public education sector has to persuade
parents to become more accepting toward an increased tuition fee while
improving the quality of education to retain students (Nguyen, 2019). The second
obstacle is an alias of Vietnam’s booming economy and population. The
substantial profitability of the quasi-market of educational service in Vietnam has
led to ever more competition (Lan Anh, 2015). This tendency has pushed the
typically conservative public school into the context of the already competitive
educational market amongst private secondary schools. While various
stakeholders are contending the education market, emphasizing the importance
of applying marketing perspectives in the secondary education sector has
dramatically increased, since a slight advantage can decide the survivability of a
school (Kassim et al., 2010; Sharif & Kassim, 2012).
From the traditional marketing approach, customer satisfaction and loyalty are
critical factors related to the success of an organization (Kaura et al., 2015). This is
especially true in secondary education, where the decision regarding in which
school to enroll their children is a financial one made by parents. When compared
with lower education levels, the decision to choose the secondary school is valued
more greatly since this is the stepping stone for the children’s tertiary education
or future career (Rolfsman, 2020; Savolainen et al., 2008). The school with higher
212
parent satisfaction and loyalty performs more effectively in the educational

market (Li & Hung, 2009). Therefore, gaining the satisfaction and loyalty of
parents, who are also the purchasers of the service, is decisive to gaining the
competitive edge over competitors.
In traditional business, improving service quality is one way to increase customer

satisfaction and loyalty (Lee, 2017), a fact that also applies to educational service
(Hemsley-Brown et al., 2010; Pham & Lai, 2016). However, there are also inherent
differences between the education and other business sectors. While customer
perception about service quality in other businesses depends on both the tangible
(physical manifestation of the company) and intangible (the service provided by
the employees), the assessor of education quality mainly focuses on an ethereal
quality, which is the academic side of educational assistance. Indeed, previous
research on student perceptions of educational quality has pointed out that
academic factors have a more substantial role in the overall student assessment
compared with non-academic ones (Teeroovengadum et al., 2019). However,
considering parents as the purchasers of educational services, the oversight of
non-academic service quality can make the evaluation of service quality
imprecise.
The reasons for the potential overlook lie in the similarities and differences in the
service experience of parents and students in education. Like with the students,
parent satisfaction and loyalty are mainly based on the academic quality of the
school. However, unlike students, who directly experience the secondary school
education, parents’ perception about the academic quality is based on a more
indirect assessment, that of school image (Li & Hung, 2009), or their children’s
academic results and personal transformation (Harvey & Green, 1993;
Teeroovengadum et al., 2016). Meanwhile, most parents have to deal directly with
non-academic issues related to their children’s education, such as administrative
issues and financial matters. Therefore, from a marketing perspective, examining
the parents’ perceptions of non-academic service quality in secondary school
might provide valuable information both in terms of theory and practice
regarding parents’ assessment of service quality, and helpful advice for
educational institutions to increase their market competitiveness.
As there is currently no research on the effect of non-academic service quality on

the level of satisfaction and loyalty with secondary education students, this
research, to the best of our knowledge, may be the first to strive to address this
research issue. Therefore, the main goal of this study is to examine to what extent
non-academic service factors affect parent satisfaction and loyalty. Particularly,
two research questions have arisen from this research goal:
1. What components constitute the non-academic service factors in
the context of secondary education in Vietnam?
2. How do non-academic service factors affect parent satisfaction and
loyalty in the context of secondary education in Vietnam?
To address the research questions, exploratory factor analysis (EFA) was

conducted to examine and adapt an established questionnaire to measure parents’
213
assessment of non-academic service quality. Subsequently, confirmatory factor

analysis (CFA) and structural equation modeling (SEM) were performed to
examine the model fit and the role of non-academic service quality on parent
satisfaction and loyalty.
The findings of this study have both academic and practical implications. In terms
of academic implications, this study, to the best of our knowledge, is the first ever
effort to investigate non-academic service factors as determinants of parent
satisfaction and loyalty in a secondary education context. In terms of practical
implications, school principals and policymakers are amongst the key
stakeholders who may benefit from this study.
Parent Satisfaction and Loyalty
Satisfaction and loyalty are two prominent factors examined in marketing
research, and are direct antecedents of customer retention and business success
(Hallowell, 1996; Kandampully & Suhartanto, 2000; Kasiri et al., 2017).
Furthermore, satisfaction and loyalty are closely related concepts and should be
included conjointly to explain best success of service providers, including
educational institutions. Satisfaction is identified as one of the most viable direct
determinants of loyalty. This assertion has been verified in many different
contexts, such as tourism (Eid, 2015), retail (Kamran-Disfani et al., 2017), banking
(Bloemer et al., 1998), and e-commerce (Eid, 2011).
In the educational context, we follow the definition of Fantuzzo et al. (2006), which
operationalizes parent satisfaction according to the degree of expectation being
met. Satisfaction can also serve as the primary pathway to customer loyalty, which
is also an essential factor in the marketing field. Customer loyalty in the
educational context can be defined according to the long-term commitment to the
favored educational institution of parents (Li & Hung, 2009). As the concept of
loyalty can be specified by different theoretical approaches (e.g., behavioral
loyalty, intentional loyalty), in this study, we followed Li and Hung (2009) in
using the hybrid system to measure loyalty, which has been commonly applied in
educational contexts.
Non-Academic Service Quality

The taxonomy of the educational service quality concept has been defined in
various ways in literature. As summarized by Sharif and Kassim (2012), service
quality can be defined either by the excellence of service delivery or the success in
meeting customer expectations. In the educational context, the latter approach is
often the preferred avenue to examine service quality. As previous research has
mainly relied on student assessment, the expectation of service quality is
primarily defined by the added value from academic indicators, while neglecting
the non-academic aspect of the service (Sharif & Kassim, 2012). Teeroovengadum
et al.’s (2016) scale measuring both non-academic and academic service quality
has shown that the factors related to non-academic service quality have a lesser
effect on students’ assessment of service quality. However, from the parents’
214
perspective, the aspect of service quality that they often directly have exposure to
is primarily non-academic factors.
Previous research on the assessment of patients’ relatives on medical service

quality has shown that interpersonal relationships with the medical staff and the
physical environment are important indicators of service quality since the
relatives do not have direct exposure to the medical care and enough technical
knowledge about the service (Mohammad Mosadeghrad, 2013). Therefore, to
parents, the non-academic factors may also play an essential role in assessing
service quality. According to Sharif and Kassim (2012), non-academic service
quality is crucial in choosing educational institutions when the other academic
criteria are identical. To better distinguish the sub-concepts of academic and
non-academic service quality, we conceptualize non-academic service quality,
based on Sharif and Kassim (2012), as the factors related to the infrastructure and
management activities of the educational institution. Therefore, this study aimed
to explore the effects of non-academic service quality on parent satisfaction and
loyalty. EFA was used to unveil the factors of non-academic service quality.
Previous studies, using students as studied objects, have revealed a positive
association between non-academic service quality and satisfaction and loyalty
(see Kardoyo et al., 2019; Mulyono et al., 2020; Sultan & Yin Wong, 2013).
Subsequently, these factors were used to answer our main research hypotheses,
which expect non-academic service quality elements to significantly increase
parent satisfaction and loyalty.
Conceptual Model
Based on the above review of literature, we built our conceptual model as
illustrated in Figure 1. Specifically, it is suggested that satisfaction and
non-academic service factors both have a positive effect on loyalty. Meanwhile,
non-academic service factors are also proposed to have a direct effect on
satisfaction. In other words, satisfaction is supposed to have a partial mediating
effect on the link between non-academic service factors and loyalty.
Figure 1: Conceptual model of the research
215
3. Methodology
Data Collection
To answer the research questions, an online survey questionnaire was distributed
to Vietnamese parents of secondary school students. We used the snowball data
collection method to secure respondents. Specifically, we utilized our network of
educators and teachers in secondary schools across Vietnam to forward the survey
to the parents. Every two weeks during the data collection process, a follow-up
email was sent to remind the unresponsive parents to complete the survey. The
final sample comprised 230 respondents deemed suitable for SEM (Hoogland &
Boomsma, 1998). Table 1 shows an overview of the demographic profile of the
sample.
Table 1: The demographic background of the respondents
Characteristic Frequency (n & %)
Gender
Male 36 (15.7)
Female 188 (81.7)
Prefer not to say 6 (2.6)
Age
35 years old and under 18 (7.8)
36–40 years old 70 (30.4)
41–45 years old 83 (36.1)
46–50 years old 41 (17.8)
50 years old and over 18 (7.8)
Type of work
Public sector 131 (57.0)
Private sector 71 (30.9)
Other 28 (12.2)
Residency
Hanoi or Ho Chi Minh City 177 (77.0)
Other 53 (23.0)
Level of education
Lower than undergraduate 22 (9.6)
Undergraduate education 95 (41.3)
216
Characteristic Frequency (n & %)
Postgraduate education 109 (47.4)
Other 4 (1.7)
The grade of the children
Lower secondary education 147 (63.8)
Upper secondary education 83 (36.1)
Total 230
Instrument Development
A positivism paradigm was employed to help answer the research questions. The
research questionnaire was developed based on the Higher Education Service
Quality (HESQUAL) scale (Teeroovengadum et al., 2016). The measurement has
a total of five main factors and eleven sub-factors. From the conceptualization
mentioned above of non-academic service quality, we chose five sub-factors of the
HESQUAL scale to measure non-academic service factors, as follows:
(i) administrative processes (ADP); (ii) attitude and behavior of staff (ATB);
(iii) support facilities quality (SFQ); (iv) learning setting (LNS); and (v) support
infrastructure (SPI).
To use the questionnaire on Vietnamese parents, a translated version of the survey

was required. Following the recommendations of Epstein et al. (2015), we
employed back-translation to translate the English questionnaire into Vietnamese.
The original English version (E1) was translated into Vietnamese (V1) by a co-
author of this paper. After this, another co-author who had never seen the E1
version was asked to back-translate V1 into English (E2). The three versions of the
questionnaire (i.e., E1, V1, & E2) were compared and revised by all authors of the
paper, subsequently leading to the second Vietnamese version (V2). An
educational practitioner and a researcher were invited to examine the V2 and E1
versions for face validity. Some revisions on the terminology in the questionnaire
were made based on the comments of the two invited experts.
4. Results
Exploratory Factor Analysis
As the questionnaire had been translated into Vietnamese, we decided to conduct
EFA to examine the latent constructs of non-academic service quality. Following
the procedure of Hair et al. (1998), the items with factor loadings larger than 0.50
were considered significant and kept. Meanwhile, items with factor loadings
smaller than 0.4 were excluded. Table 2 shows the factor structure of the 13
observed variables examined by the EFA. As shown in Table 2, rather than having
five initial factors, this analysis revealed two main factors, labeled physical
environment quality (PEQ) and administrative quality (ADQ).
217
Table 2: The exploratory factor analysis results

Factor Component Factor Cronbach
(Latent loading alpha
variable)
SPI1: Availability of adequate cafeteria
0.840
infrastructure
SPI2: Availability of adequate sports
0.851
infrastructure
SPI3: Availability of adequate recreational
0.778
infrastructure
SPI4: Availability of adequate library
0.751
infrastructure
PEQ SFQ2: Amount of opportunity for sports 0.929
0.825
and recreational facilities
SFQ3: Availability of adequate IT facilities 0.791
LNS1: Having adequate lecture rooms 0.766
LNS2: Having quiet places to study within
0.530
the campus
LNS3: Availability of adequate teaching
tools and equipment (e.g., projector, 0.546
whiteboard)
ADP1: Well-standardized administrative
processes so that there is not much 0.853
bureaucracy and useless difficulties
ADP2: Administrative procedures are
clear and well structured so that service 0.878
ADQ 0.931
delivery times are minimum
ATB1: Willingness of administrative staff
0.923
members to help students
ATB2: Ability of administrative staff
0.848
members to solve students’ problems
Nine items loaded onto PEQ, with factor loadings ranging from 0.53 to 0.85. The
second factor, ADQ, had factor loadings ranging from 0.85 to 0.92. All items used
for the analysis are presented in Table 2. The EFA results therefore provide the
answer for research question 1 (What components constitute the non-academic
service factors in the context of secondary education in Vietnam?). There are two
specific factors pertaining to the non-academic service dimension of secondary
education in Vietnam, namely PEQ and ADQ.
Figure 2 represents our adjusted conceptual model based on the EFA results.
218
Figure 2: The adjusted conceptual model of the research after exploratory factor
analysis
Particularly, we propose that satisfaction has a positive effect on loyalty. At the

same time, we suggest that satisfaction partially mediates the relationships of
PEQ-loyalty and ADQ-loyalty. Under these circumstances, five hypotheses are
proposed:
H1: PEQ has a significant positive effect on satisfaction.
H2: ADQ has a substantial significant positive effect on satisfaction.
H3: PEQ has a significant positive effect on loyalty.
H4: ADQ has a significant positive effect on loyalty.
H5: Satisfaction has a significant positive effect on loyalty.
Confirmatory Factor Analysis

To assess how well the measurement model fits with the observed data, CFA was
conducted. Specifically, all the goodness of fit (GOF) indices were satisfied with
their acceptable level, indicating that the model fit with the observed data. Table 3
presents the GOF indices of the model and their respective cut-off levels for a good
fit.
Table 3: Results of multiple fit indices
Index Result Acceptable level

Chi-square 254.159 -
Degree of freedom 174 -
Chi-square/Degree of
1.461 <5
freedom
GFI 0.908 > 0.9
AGFI 0.878 > 0.8
NFI 0.949 > 0.9
RMSEA 0.045 < 0.08
CFI 0.983 > 0.9
219
Regarding the issues of convergent validity and discriminant validity, the factor
loadings, composite reliability (CR) scores, and average variance extracted (AVE)
scores of the model were examined (Tables 4 and 5).
Table 4: Results of factor loading for Confirmatory Factor Analysis
Factor Item
(Latent Factor loading
variable)
SFQ2: Amount of opportunity for sports and recreational

0.824
facilities
SFQ3: Availability of adequate IT facilities 0.803
LNS3: Availability of adequate teaching tools and

0.691
equipment (e.g., projector, whiteboard)
LNS2: Having quiet places to study within the campus 0.649

PEQ
LNS1: Having adequate lecture rooms 0.760
SPI4: Availability of adequate library infrastructure 0.767
SPI3: Availability of adequate recreational infrastructure 0.797
SPI2: Availability of adequate sports infrastructure 0.801
SPI1: Availability of adequate cafeteria infrastructure 0.768
ADP2: Administrative procedures are clear and well

0.958
structured so that service delivery times are minimum
ADP1: Well-standardized administrative processes so that

0.926
there is not much bureaucracy and useless difficulties
ADQ
ATB2: Ability of administrative staff members to solve
0.768
students’ problems
ATB1: Willingness of administrative staff members to help

0.786
students
SAT3. This school has met my expectations 0.941
SAT SAT2. My choice of this school was a wise one 0.964
SAT1. I am satisfied with the decision to choose this school 0.939
LOY5. When talking about school with my relatives or

LOY 0.898
friends, I will praise this school voluntarily.
220
Factor Item
(Latent Factor loading
variable)
LOY4. I will encourage my relatives or friends to let their

0.923
children study in this school
LOY3. When my relatives or friends need information

about secondary school, I will voluntarily recommend this 0.916
school
LOY2. When my child is entering secondary school, this

0.824
school will be my first choice
LOY1. If I have a child studying in secondary school, I will

0.819
continue to let him/her attend the same school.
Table 5: Convergent and discriminant validity
Factor correlation
Construct CR AVE
SAT PEQ ADQ LOY
PEQ 0.926 0.584 0.764
ADQ 0.921 0.746 0.622 0.864
SAT 0.964 0.899 0.519 0.646 0.948
LOY 0.943 0.769 0.587 0.535 0.770 0.877
Note: PEQ – physical environment quality; ADQ – administrative quality;

SAT – satisfaction; LOY – loyalty; CR – composite reliability; AVE – average
variance extracted
Regarding the factor loadings of the observed variables, the PEQ factor loadings
ranged from 0.649 to 0.824, ADQ from 0.768 to 0.958, satisfaction from 0.939 to
0.964, and loyalty from 0.819 to 0.923. Overall, the factor loadings of the observed
variables were relatively high. Regarding the convergent validity issue, as shown
in Table 4, the CR and AVE scores of the concepts were higher than their
thresholds, 0.7 and 0.5, respectively. The discriminant validity of the
questionnaire was examined by comparing the AVE scores and the squared
correlation coefficients between factors. The AVE scores were higher than the
squared correlation coefficients, which indicates that the criteria for discriminant
validity were satisfied.
Path Analyses and Hypotheses Testing

Table 6 shows the standardized regression estimates and the p-values of the
exogenous variables.
221
Table 6: Results of the structural equation modelling
Standardized β
p-value Hypothesis
coefficient
Dependent variable: SAT
PEQ 0.197 0.007 H1 is supported
ADQ 0.524 *** H2 is supported
R2 44.1%
Dependent variable: LOY
SAT 0.674 *** H5 is supported
PEQ 0.283 *** H3 is supported
ADQ -0.077 0.261 H4 is not supported
R2 64.5%
Regarding the effect of non-academic service constructs on satisfaction, the

regression paths of the SEM showed that PEQ (β = 0.20, p = 0.007) and ADQ
(β =0.52, p < 0.001) have a significant positive effect on satisfaction, indicating that
higher non-academic service constructs come with higher parent satisfaction.
Therefore, hypotheses H1 and H2 are supported. Higher PEQ is also directly
associated with higher loyalty (β = 0.28, p < 0.001), indicating that increasing the
PEQ heightens parents’ commitment. This relationship supports hypothesis H3.
However, ADQ does not significantly affect loyalty (p = 0.261), and therefore
hypothesis H4 is rejected. Lastly, satisfaction showed the strongest effect on
loyalty (β = 0.67, p < 0.001), which provides support for hypothesis H5.
These above findings provide an answer to the second research question of this
study (How do non-academic service factors affect parent satisfaction and loyalty
in the context of secondary education in Vietnam?). Specifically, both PEQ and
ADQ directly affect satisfaction, while only PEQ directly affects loyalty. In the
same vein, satisfaction directly affects loyalty. In other words, satisfaction
partially mediates the relationship between PEQ and loyalty and fully mediates
the relationship between ADQ and loyalty. Overall, the non-academic service
quality explained 44.1% of the variance in satisfaction. The effect of satisfaction,
PEQ, and ADQ on loyalty explained 64.5% of the variance in loyalty.
Total Effect of Direct and Indirect Antecedents of Parent Loyalty

Table 7 shows calculations of the total effect of the antecedents on loyalty.
Amongst all, satisfaction has only a direct but also the highest effect (total effect =
0.674) on loyalty. This is followed by PEQ, which has both direct and indirect
effects (total effect = 0.416), and ADQ, which has only an indirect effect (total effect
= 0.353).
222
Table 7: Direct, indirect, and total effects of physical environment quality,

administrative quality, and satisfaction on loyalty
Antecedent Direct effect route and Indirect effect route and Total
magnitude magnitude effect
PEQ PEQ - loyalty: 0.28 PEQ - satisfaction - loyalty: 0.416

0.197 * 0.674 = 0.133
ADQ – ADQ - satisfaction - loyalty: 0.353

0.524 * 0.674 =
Satisfaction Satisfaction - loyalty: 0.674 – 0.674
5. Discussion and Conclusion

As far as we are aware, no research has been conducted on the effects of parents’
assessment of non-academic service quality on their satisfaction and loyalty in
secondary education. By surveying 230 Vietnamese parents, this study fills this
research gap by pointing out that besides increasing academic quality, secondary
education institutions should also focus on improving ADQ and PEQ to increase
parent satisfaction and loyalty. The results of this research make several
theoretical contributions to the literature on educational service quality.
Academic Implications
First, this research has re-established that satisfaction is still the main predictor of
loyalty (Brandano et al., 2019; Kandampully & Suhartanto, 2000; Kasiri et al., 2017;
Pham et al., 2019). The results also showed that the non-academic dimension is
composed of two factors, PEQ and ADQ. These factors have both direct and
indirect effects on parent satisfaction and loyalty. Furthermore, the total direct
and indirect effects of PEQ more significantly affect parent loyalty than the
indirect effect of ADQ. This result aligns with previous research on students’
assessment of ADQ and PEQ. According to Grönroos (2011), the evaluation of
value can be incrementally added throughout the service experience of the
customer by exposure to different indicators of quality. As the exposure to PEQ is
unavoidable when parents are present at the educational institution, the value
added incrementally may contribute to the positive relationship with satisfaction
and loyalty. This can also be used to explain the insignificant relationship between
ADQ and loyalty. According to lin and Tsai (2008), students perceived that ADQ
does not affect their loyalty due to the little interaction between them and the
administrative staff.
Similarly, this research has shown that positive ADQ experiences affect the only
the satisfaction of parents. Unlike ADQ, the exposure to PEQ is consistent and can
be accounted for by parent recommendation and loyalty to the school. PEQ has
been found to affect both parent satisfaction and loyalty through either direct or
indirect paths, which is also consistent with previous research on students’
assessment (Hemsley-Brown et al., 2010; Wiers-Jenssen et al., 2002).
223
Practical Implications
The findings of this research have two main direct practical implications for
secondary education institutions. First, secondary schools should focus on
increasing the quality of PEQ and ADQ to increase parent satisfaction and loyalty.
As previous research has mainly depended on the academic quality of educational
institutions, schools with equal academic quality and a stronger focus on
non-academic aspects can have a stronger competitive edge than their
counterparts (Kassim et al., 2010). Second, when distributing resources to increase
the competitive advantage over non-academic service quality, more focus should
be placed on PEQ than ADQ as PEQ has a direct effect on both satisfaction and
loyalty, while ADQ has a direct effect on satisfaction only but not loyalty.
Additionally, the total effect on loyalty stemming from PEQ is higher than that
from ADQ. Thus, school principals and governmental agencies should focus more
on PEQ constitutions such as libraries, cafeterias, recreational infrastructure, sport
infrastructure, and so forth.
6. Limitations and Suggestions for Future Research

The current research also had several limitations (Vuong, 2020). First, as this
research was one of the first attempts to examine the effect of non-academic
service quality from parents’ perspective, the survey developed to test our
hypotheses requires further investigation regarding its validity and reliability in
different contexts. We suggest future research to replicate these results by
including other measurements of non-academic service quality to predict parent
satisfaction and loyalty. Second, as Asian parents in general and Vietnamese
parents in specific have certain unique traits represented by cultural factors (Loh
& Teo, 2017), a more representative sample across the globe is required to further
confirm how non-academic service quality affects parent satisfaction and loyalty.
Third, the non-significant ADQ results may be the result of overlooking other
important determinants of loyalty, such as school image. We recommend that
further research use a more detailed model with relevant variables to test this
effect. Lastly, although we were able to collect a diverse and large sample, our
sampling method relied on educators and teachers to distribute the survey to the
parents. This may have triggered the demand effect, where participants try to
please the authoritative individual (teacher/researcher) by behaving in a
desirable manner (Nichols & Maner, 2008). Although the consent form of this
research assured the anonymity of the respondents, future research should utilize
a more randomized sampling method to replicate the results of this research.
Funding and Acknowledgements:

This work is part of the project “Model of K-12 school governance from the lens of
educational service” – number B2019 – VGK-02 (Mô hình quản trị cơ sở giáo dục phổ thông
theo tiếp cận quản trị dịch vụ giáo dục, số B2019 –VGK-02).
The authors would like to thank the funder for this valuable support.
224
7. References
Bloemer, J., De Ruyter, K., & Peeters, P. (1998). Investigating drivers of bank loyalty: The
complex relationship between image, service quality and satisfaction. International
Journal of Bank Marketing, 16(7), 276–286.
https://doi.org/10.1108/02652329810245984
Brandano, M. G., Osti, L., & Pulina, M. (2019). How motivations and satisfaction influence
wine tourists’ loyalty? An analysis of the Italian case. International Journal of
Culture, Tourism and Hospitality Research, 13(1), 55–69.
https://doi.org/10.1108/IJCTHR-04-2018-0054
Eid, M. (2011). Determinants of e-commerce customer satisfaction, trust, and loyalty in
Saudi Arabia. Journal of Electronic Commerce Research, 12(1), 78–93.
http://www.jecr.org/sites/default/files/12_1_p05.pdf
Eid, R. (2015). Integrating Muslim customer perceived value, satisfaction, loyalty and
retention in the tourism industry: An empirical study. International Journal of
Tourism Research, 17(3), 249–260. https://doi.org/10.1002/jtr.1982
Epstein, J., Osborne, R. H., Elsworth, G. R., Beaton, D. E., & Guillemin, F. (2015). Cross-
cultural adaptation of the Health Education Impact Questionnaire: Experimental
study showed expert committee, not back-translation, added value. Journal of
Clinical Epidemiology, 68(4), 360–369.
https://doi.org/10.1016/j.jclinepi.2013.07.013
Fantuzzo, J., Perry, M. A., & Childs, S. (2006). Parent Satisfaction with Educational
Experiences Scale: A multivariate examination of parent satisfaction with early
childhood education programs. Early Childhood Research Quarterly, 21(2), 142–152.
https://doi.org/10.1016/j.ecresq.2006.04.002
Grönroos, C. (2011). Value co-creation in service logic: A critical analysis. Marketing Theory,
11(3), 279–301. https://doi.org/10.1177/1470593111408177
Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (1998). Multivariate data analysis
(Vol. 5). Pearson. https://www.amazon.com/Multivariate-Data-Analysis-
Joseph-Hair/dp/0138132631
Hallowell, R. (1996). The relationships of customer satisfaction, customer loyalty, and
profitability: An empirical study. International Journal of Service Industry
Management, 7(4), 27–42. https://doi.org/10.1108/09564239610129931
Harvey, L., & Green, D. (1993). Defining quality. Assessment & Evaluation in Higher
Education, 18(1), 9–34. https://doi.org/10.1080/0260293930180102
Hemsley-Brown, J., Lowrie, A., Gruber, T., Fuß, S., Voss, R., & Gläser-Zikuda, M. (2010).
Examining student satisfaction with higher education services: Using a new
measurement tool. International Journal of Public Sector Management, 23(2), 105–123.
https://doi.org/10.1108/09513551011022474
Hoogland, J. J., & Boomsma, A. (1998). Robustness studies in covariance structure
modeling. Sociological Methods & Research, 26(3), 329–367.
https://doi.org/10.1177/0049124198026003003
Kamran-Disfani, O., Mantrala, M. K., Izquierdo-Yusta, A., & Martínez-Ruiz, M. P. (2017).
The impact of retail store format on the satisfaction-loyalty link: An empirical
investigation. Journal of Business Research, 77, 14–22.
https://doi.org/10.1016/j.jbusres.2017.04.004
Kandampully, J., & Suhartanto, D. (2000). Customer loyalty in the hotel industry: The role
of customer satisfaction and image. International Journal of Contemporary Hospitality
Management, 12(6), 346–351. https://doi.org/10.1108/09596110010342559
Kardoyo, K., Pitaloka, L. K., Rozman, R., & Hapsoro, B. B. (2019). Analyzing universities
service quality to student satisfaction; academic and non-academic analyses.
International Journal of Higher Education, 9(1), 126.
225
https://doi.org/10.5430/ijhe.v9n1p126
Kasiri, L. A., Guan Cheng, K. T., Sambasivan, M., & Sidin, S. M. (2017). Integration of
standardization and customization: Impact on service quality, customer
satisfaction, and loyalty. Journal of Retailing and Consumer Services, 35, 91–97.
https://doi.org/10.1016/j.jretconser.2016.11.007
Kassim, N. M., Najdawi, M., Al Azmeh, Z., & Sadiq, H. (2010). Effects of institutional
reform on corporate image and value in a developing country context. Measuring
Business Excellence, 14(2), 32–45. https://doi.org/10.1108/13683041011047849
Kaura, V., Durga Prasad, C. S., & Sharma, S. (2015). Service quality, service convenience,
price and fairness, customer loyalty, and the mediating role of customer
satisfaction. International Journal of Bank Marketing, 33(4), 404–422.
https://doi.org/10.1108/IJBM-04-2014-0048
Lan Anh. (2015). Giáo dục muốn phát triển phải cạnh tranh [Competition is required for
the development of education]. Vietnamnet.
https://vietnamnet.vn/vn/tuanvietnam/giao-duc-muon-phat-trien-phai-canh-
tranh-274106.html
Lee, S. Y. (2017). Service quality of sports centers and customer loyalty. Asia Pacific Journal
of Marketing and Logistics, 29(4), 870–879. https://doi.org/10.1108/APJML-10-
2016-0191
Li, C. K., & Hung, C. H. (2009). Marketing tactics and parents’ loyalty: The mediating role
of school image. Journal of Educational Administration, 47(4), 477–489.
https://doi.org/10.1108/09578230910967455
lin, C., & Tsai, Y. H. (2008). Modeling educational quality and student loyalty: A
quantitative approach based on the theory of information cascades. Quality &
Quantity, 42(3), 397–415. https://doi.org/10.1007/s11135-006-9051-5
Loh, C. Y. R., & Teo, T. C. (2017). Understanding Asian students learning styles, cultural
influence and learning strategies. Journal of Education & Social Policy, 7(1), 194–210.
https://www.jespnet.com/journals/Vol_4_No_1_March_2017/23.pdf
Mohammad Mosadeghrad, A. (2013). Healthcare service quality: Towards a broad
definition. International Journal of Health Care Quality Assurance, 26(3), 203–219.
https://doi.org/10.1108/09526861311311409
Mulyono, H., Hadian, A., Purba, N., & Pramono, R. (2020). Effect of service quality toward
student satisfaction and loyalty in higher education. The Journal of Asian Finance,
Economics and Business, 7(10), 929–938.
https://doi.org/10.13106/jafeb.2020.vol7.no10.929
Nguyen, P. (2019). Tự chủ trong giáo dục phổ thông là xu thế tất yếu [Autonomy in
secondary education is an inevitable trend]. Nhan Dan.
https://www.nhandan.com.vn/hangthang/item/41735702-“tu-chu-trong-giao-
duc-pho-thong-la-xu-the-tat-yeu”.html
Nichols, A. L., & Maner, J. K. (2008). The good subject effect: Investigating participant
demand characteristics. The Journal of General Psychology, 135(2), 151–166.
https://doi.org/10.3200/GENP.135.2.151-166
Pham, H.-H., & Lai, S. L. (2016). Higher education as an extended duration service. Journal
of Studies in International Education, 20(5), 454–471.
https://doi.org/10.1177/1028315316662978
Pham, H.-H., Lai, S. L., & Vuong, Q.-H. (2019). The role of subjective task value in forming
satisfaction and loyalty among Vietnamese international students: A structural
equation model. The Asia-Pacific Education Researcher, 28(5), 399–409.
https://doi.org/10.1007/s40299-019-00439-3
Rolfsman, E. (2020). Swedish students in the process of transition to upper secondary
education: Factors of importance for educational choice and for their future.
226
Education Inquiry, 11(4), 331–359. https://doi.org/10.1080/20004508.2020.1746480

Savolainen, H., Ahonen, T., Aro, M., Tolvanen, A., & Holopainen, L. (2008). Reading
comprehension, word reading and spelling as predictors of school achievement
and choice of secondary education. Learning and Instruction, 18(2), 201–210.
https://doi.org/10.1016/j.learninstruc.2007.09.017
Sharif, K., & Kassim, N. M. (2012). Non-academic service quality: Comparative analysis of
students and faculty as users. Journal of Marketing for Higher Education, 22(1), 35–
54. https://doi.org/10.1080/08841241.2012.705793
Sultan, P., & Yin Wong, H. (2013). Antecedents and consequences of service quality in a
higher education context. Quality Assurance in Education, 21(1), 70–95.
https://doi.org/10.1108/09684881311293070
Teeroovengadum, V., Kamalanabhan, T. J., & Seebaluck, A. K. (2016). Measuring service
quality in higher education: Development of a hierarchical model (HESQUAL).
Quality Assurance in Education, 24(2), 244–258. https://doi.org/10.1108/QAE-06-
2014-0028
Teeroovengadum, V., Nunkoo, R., Gronroos, C., Kamalanabhan, T. J., & Seebaluck, A. K.
(2019). Higher education service quality, student satisfaction and loyalty:
Validating the HESQUAL scale and testing an improved structural model. Quality
Assurance in Education, 27(4), 427–445. https://doi.org/10.1108/QAE-01-2019-
0003
Vuong, Q.-H. (2020). Reform retractions to make them more transparent. Nature,
582(7811), 149–149. https://doi.org/10.1038/d41586-020-01694-x
Wiers-Jenssen, J., Stensaker, B., & Gr⊘gaard, J. B. (2002). Student satisfaction: Towards an
empirical deconstruction of the concept. Quality in Higher Education, 8(2), 183–195.
https://doi.org/10.1080/1353832022000004377
227

Vol. 20, No. 8, pp. 227-239, August 2021
Developing a Framework Peace Education for

Primary School Teachers in Indonesia
Sahril Buchori, Sunaryo Kartadinata, Syamsu Yusuf and Ilfiandra

Universitas Pendidikan Indonesia, Indonesia
https://orcid.org/0000-0001-7184-8576
https://orcid.org/0000-0003-3767-4706
https://orcid.org/0000-0002-3775-8857
https://orcid.org/0000-0002-0886-8301
Nurfitriany Fakhri
Universitas Negeri Makasar, Indonesia
https://orcid.org/0000-0001-5323-8125
Sofwan Adiputra
Universitas Muhammadiyah Pringsewu, Indonesia
https://orcid.org/0000-0002-5100-1236
Abstract. Teachers as educators should have a sense of peace within

themselves so that a conducive classroom climate can be created. The
purpose of this study is to build a framework for peace education in
elementary schools. To develop this capacity for peace, the researcher
conducted three stages of study: the first, an empirical study; the
second, developing a framework; and the third, measuring its
effectiveness. This research was conducted in three schools with 12
teachers as participants. The data collection technique used an
instrument of teacher peace capacity in the form of a semantic
differential scale to determine the level of knowledge, skills and
attitudes of teachers' peace capacity. The results show that teachers do
not yet have a qualified peace capacity. This framework can be applied
to teachers in the form of training. The framework for developing
teacher peace capacity has proven to be effective.
Keywords: framework; teacher peace capacity; peace education
1. Introduction
In an era of development that is so complex, various challenges in the form of
personal and social problems become important to study. One of them is the
problem of living in peace and harmony. It takes a peaceful atmosphere to live
together in harmony. Educational institutions are places to foster the creation of
©Authors
228
peace; however, they can also be places that militate against peace in society.
Therefore settings and conditions are needed in educational institutions that
foster peace (Setiadi et al., 2017; UNESCO-APNIEVE, 2000).
Education is the means of developing students who are peaceful and
harmonious. However, the situation on the ground shows a different situation.
Violent behaviour by children is currently a social problem, including in schools.
This is a challenge for education in Indonesia. The results of research released in
2015 by the NGO Plan International and the International Center for Research on
Women (ICRW) showed that as many as 84% of children in Indonesia
experienced violence in schools (Eliasa, 2017). Violence is perpetrated by
students in the form of fights both at school and outside of school, brawls,
bullying, shouting at other students, skipping school, and graffiti on school
walls and the school environment (Chan et al., 2013; Fung et al., 2015).
Violent social behaviour results in the destruction of basic assumptions about
virtue, justice and security (Janoff-Bulman, 1992). The results of research
(Buchori, 2018) show that in elementary schools there is often a less peaceful
atmosphere in the classroom than outside the classroom. Students and teachers
still display anti-peace behaviour in the form of an inability to contain anger.
Students do not care and share, are rude, mock others and generally display
provocative behaviour until a fight breaks out. These forms of violent behaviour,
such as fighting at school, appear among both junior and senior high-school
students, while drug use and the possession of weapons are rife (Furlong &
Morrison, 2000).
The birth of the anti-violence movement is in line with the emergence of
Strengthening Character Education (SCE), which is carried out by applying the
values of pancasila in character education in Indonesia. This government
programme is a response to the rampant violence that occurs in schools. In
addition, other movements have emerged such as the Indonesian Anti-
Defamation Society (MAFINDO) organisation. The MAFINDO community is
concerned with educating the public, especially children, teenagers and students
to prevent the spread of slander, hate speech, hoaxes, radicalism, terrorism and
other anti-social behaviour. This indicates that violence has become a concern in
the school environment (Nurhadi & Muchtarom, 2020).
Violent behaviour by children at school is one of the aggressive behaviours that
originate from cognitive mechanisms. Children's behaviour reflects an imitation
of what is seen and heard around them, thus giving rise to normative beliefs in
children (Baron, 1992; Huesmann & Guerra, 1997; Tentama, 2012).
Children get justification for their violent behaviour in response to similar
behaviour because they think that the response has become a rule of the game
based on experience or general agreement that it is acceptable by those around
them. This includes both physical and verbal violence (Berkowitz, 1993; Buchori
& Fakhri, 2018; Henry et al., 2000; Werner & Hill, 2010).
Violent behaviour by students can also be influenced by teachers’ behaviour.
The teacher becomes a model for students to behave aggressively. This
behaviour can be in the form of yelling, hitting with a ruler, and giving students
derogatory nicknames (Harber, 2004; Hilarski, 2004; Meyer, 2006). Good and bad
229
imitated behaviour tend to be exhibited by elementary school students because

they do not have the ability to choose or differentiate.
Building peace in educational institutions has been carried out by various
countries, in both formal and non-formal institutions (Harber & Sakade, 2009).
Peace education programmes are conducted in primary schools in the United
Kingdom (UK) (Sakade, 2008). Such a programme, the Peacemaker Project, is
carried out by the West Midlands Quaker Peace Education Project (WMQPEP).
This peace education programme takes the form of conflict resolution that
promotes awareness of the nature of conflict and builds nonviolent conflict
management skills. The roles of researchers and teachers are those of controllers
and facilitators that encourage students to be able to control themselves, to keep
the rules set by themselves and be responsible for their own behaviour.
One of the studies on peace education in America was conducted by Hunter
(2008), a teacher at Redland's Adventist Academy, California. Hunter created a
classroom environment that fosters values such as inclusive compassion, social
justice, service, and active peace. These educators provide the theoretical basis
for an appropriate approach and curriculum for the classroom environment. The
curriculum is designed to develop students' intellectual, emotional and social
aspects. Peace education is carried out by referring to the curriculum comprising
four main skills that will encourage a culture of peace among students, namely
empathy training, respecting diversity of training, community awareness, and
conflict resolution. This concept is emphasised in extracurricular activities.
Classes are designed to provide a variety of books and tools that engage
students in learning about peace. Posters and student work are displayed on a
bulletin board as a reminder of the values of peace they are learning.
Peace education in Finland has been integrated into global education, which has
become an explicit part of the national education system (Kartadinata et al.,
2015). Peace education is carried out continuously and holistically, and aims to
develop the knowledge, skills, attitudes and values needed to create peace in the
form of avoiding and overcoming conflicts and violence in their environment.
The process involves teachers, students, parents and the community.
Peace education was also developed by UNESCO (UNESCO-APNIEVE, 2000)
under the name Peace Education, Human Rights and Democracy. Peace
education reconnects peace at all levels of education, develops various methods
and materials used by teachers and develops teacher education to enable them
to become peacemakers. The ultimate goal is developing everyone to have
universal values and the type of behaviour that form the basis for the creation of
a culture of peace. Peace education, human rights and democracy are included
in the curriculum in its implementation involving teachers, students, parents
and the community collaborating with each other. Teaching and learning
methods are directed at making peace, ensuring human rights and promoting
democracy, both in the form of behaviour in school and as something that needs
to be learned using active learning methods, group work, discussions on morals
and ethics, and individual learning.
A requirement for developing a peaceful classroom climate is a pedagogical
framework in schools. This is built through research relating to developing the
230
capacity of students and teachers. Building a classroom climate begins with the
teacher who has an important role in educating to build student capacity.
(Freire, 1998; O'Connor, 2013; Van Manen, 1991). Teachers as role models are
expected to be able to become examples and peacemakers in reinforcing
students' peaceful behaviour. The normative belief of students in committing
violence is caused by receiving justification through imitation and reinforcement
to do so (Taylor et al., 2009). Based on this, the peace capacity of teachers in the
form of knowledge, skills, and attitudes of peace needs to be developed.
The United Nations (UN) officially recognises schools as institutions that cannot
be separated from the prevalence of violent behaviour in them, whether
perpetrated by students, teachers or staff (Benbenishty & Astor, 2005; Dulmus &
Harber, 2009; Pinheiro, 2006; Sowers, 2004). The rampant violence that occurs in
schools has prompted the government to launch the Anti-Violence Movement in
Educational Environment programme. This movement is a form of prevention
and control of violence in schools.
The results of interviews with teachers revealed that elementary schools were
prepared to be known for a peaceful classroom climate. It is necessary to
strengthen the peace capacity of teachers in the form of knowledge, skills and
attitudes so that peace can be created in the classroom and at school. Teachers
need to be able to become peacemakers to build a peaceful classroom climate by
educating students to have good morals and mutual love for fellow students and
teachers as well as by preventing violent behaviour among students.
Peace Education
The importance of peace education was proclaimed at the 1994 International
Conference on Education and endorsed by the UNESCO General Conference in
1995 (UNESCO-APNIEVE, 2000). The phenomenon of physical and
psychological violence that occurs in almost all parts of the world is most
concerning. Peace education is needed for children to build understanding,
solidarity, compassion and tolerance among individuals or groups. Peace
education is carried out by developing knowledge, values, attitudes and skills
that lead to peace. It is the process and practice of developing nonviolent skills
and promoting peaceful attitudes (UNESCO, 2017).
The United Nations Children's Fund (UNICEF) defines peace education as the
process of promoting the knowledge, skills, attitudes and values necessary to
bring about behavioural change that will enable children, youth and adults to
prevent conflict and violence, both directly and indirectly, overt and structural;
to resolve conflicts peacefully; and to create conditions conducive to peace,
whether at intrapersonal, interpersonal, inter- group, national or international
levels (Saputra, 2016).
Kartadinata et al. (2015) define peace education as a conscious and planned
effort that responds to various kinds of conflicts and violence, whether on a
personal, local, national or international scale. This behaviour is an effort to
create a future that is peaceful and secure in a sustainable manner.
231
Peace education fosters a base of knowledge, skills, attitudes and values that
seek to change patterns of thought, attitudes and behaviour related to violence.
Then it transforms and builds awareness and understanding, and develops
personal and social concerns that enable individuals to live in peace. It aims to
create conditions and systems without violence, promote justice and care for the
environment and imbue other peaceful values (Castro & Galace, 2010).
Peace education has five main postulates, namely explaining the root causes of
violence; teaching alternative behaviour; and looking for different forms of
violence and eliminating them. In addition, peace is a process that varies
according to the context while conflict has the potential to occur everywhere.
The example of peace education in the family should be carried over by parents
to their children. The function of parents in the family is to teach children about
peaceful behaviour as well as being a model in instilling values in order to form
character in children (Saputra, 2016).
Teachers’ Peaceful Capacity

Educators should have the capacity for peace to be peacemakers. This capacity is
in the form of seven main competencies of critical peace that educators should
possess, namely critical thinking and analysis, empathy and solidarity, building
cooperation, participatory and democratic engagement, education and
communication strategy, conflict transformation skills and continuous reflection
practice (Bajaj, 2015).
The peaceful capacity of teachers is their ability in the form of knowledge, skills
and attitudes that reflect peace. This knowledge includes the regulation of
rights, protection, strengthening of student character education, effective
communication skills, conflict resolution, reflective ability, empathy, respect,
democracy, and fairness.
Developing professional teacher capacity can be in the form of developing
teacher knowledge, skills, attitudes and competencies in teaching (Barber, 2005;
Goldman & Grimbeek, 2008; Sinkinson, 2009). Teachers are expected to be able
to practise non-violent skills, promote peaceful attitudes and find ways to
achieve them (UNESCO, 2017).
The teacher's peace capacity is the teacher's ability to create peace in the
classroom. Creating peace is based on mutual respect for human rights, respect,
tolerance, love, democracy, and acceptance of one another (Castro & Galace,
2010; Galtung, 1996).
Peace capacity is characterised by the presence of intrapersonal peace in the
form of consistency of behaviour, emotions and attitudes towards peace; peace
within oneself and in all aspects of life; and having values, competencies and
cognitive dispositions that tend to promote peace. Tolerance, for example,
appears to be relevant to attitudes and behaviour across all domains and will
tend to influence a person to be at peace with oneself, others, and other groups
(Nelson, 2014).
232
3. Methods
Research Design
The research used a research and development approach (Borg & Gall, 2007).
The design comprised a pre-experimental study of the impact of teachers' peace
capacity. The research design used is one-group pretest-posttest design in which
the subject is given a pretest, then given treatment in the form of a framework
for developing the peace capacity of the teacher. This is followed by a posttest.
Data Collection
The instrument of teacher peace capacity was in the form of a semantic
differential scale with a range of 1 to 10. This scale was developed based on
indicators of the teachers’ level of knowledge, skills and attitudes of peace. This
instrument aimed to collect data on the peaceful capacity of teachers in primary
schools.
Population and Sample

This research involved three schools in Makassar City, namely Borong State
Elementary School, Borong Impres Elementary School and Farida Aryani
Madrasah Ibtidaiyah Makassar. The participants involved were 12 fourth and
fifth-grade teachers to determine the sample using purposive random sampling.
Research Stages
The research procedure was carried out through six stages, namely studying the
concept of teacher quality, an empirical study of the profile of teacher capacity in
three schools, the development of a hypothetical training structure, training
testing, intervention, and finally, evaluation of the training quality.
Analysis of Research Data

For testing the effectiveness of the training, an analysis was conducted using the
Wilcoxon signed-rank and Kruskal-Wallis test rank. This data analysis technique
was used to be able to determine the difference in values between the pretest
and posttest. The research data has a small N value so that in data analysis non-
parametric statistics are used.
4. Results
The research process to determine the level of the students' classroom climate
began by measuring the peaceful capacity of the teacher who interacts with
students as well as the learning process in the classroom. The results of the study
regarding the initial description of the peace capacity of teachers before
treatment in the form of teacher peace capacity development training can be
seen in Table 1.
Table 1: Initial profile of teachers’ peaceful capacity

Category formula Score F %
Teachers’ Peace Mean + 1SD ≤ X 94 ≤ X 1 8.3
Peaceful Less Peace Mean - 1SD ≤ X < Mean + 1SD 69 ≤ X < 94 9 75.0
Capacity Not Peace X < Mean - 1SD X < 69 2 16.7
233
Most of the teachers are in the ‘less peaceful’ category. Only a small number are
in the ‘good’ category. Based on these data, in general the initial description of
the peaceful capacity of teachers reflects the ‘inadequate’ category.
The data above shows that most teachers lack the capacity for peace in building
a classroom climate. The results of this study are in line with the findings of
Setiadi et al. (2017) which found that the peace capacity of teachers is still low.
According to the findings of Akhmad et al. (2016), the attitudes and behaviour of
teachers contribute 50% to the non-peaceful events experienced by students at
school. These behaviours include giving verbal and physical punishment to
students, language that is not educating or appropriate, and reacting
aggressively, both verbally and physically, to the negative behaviour of
students.
The initial description of the peaceful capacity of teachers is seen from three
aspects, namely aspects of knowledge, skills and attitudes.
Table 2: Aspect profile of teachers’ peaceful capacity

Aspect Category Formula Score F %
High Mean + 1SD ≤ X 38 ≤ X 2 16.7
Knowledge Medium Mean - 1SD ≤ X < Mean + 1SD 28 ≤ X < 38 8 66.7
Low X < Mean - 1SD X < 28 2 16.7
High Mean + 1SD ≤ X 27 ≤ X 3 25.0
Skills Medium Mean - 1SD ≤ X < Mean + 1SD 17 ≤ X < 27 8 66.7
Low X < Mean - 1SD X < 17 1 8.3
High Mean + 1SD ≤ X 30 ≤ X 2 16.7
Attitude Medium Mean - 1SD ≤ X < Mean + 1SD 23 ≤ X < 30 7 58.3
Low X < Mean - 1SD X < 23 3 25.0
The level of teacher capacity seen from all aspects of peace reflects the ‘least’
category. Moreover, judging from the indicators, there are 12 indicators out of 14
indicators in the medium category. Therefore teachers need to be trained in all
aspects of knowledge, skills and attitudes contained in the indicators.
The capacity for peace is an important quality, especially in a teacher as an
educator (UNESCO, 1996; UNESCO, 2017). However, the results of the
preliminary research indicate that most teachers do not have the capacity for
peace within themselves.
Table 3: Results of mean and standard deviation analysis

based on pre- & posttest training
Training N Mean Std. Minimum Maximum
Deviation
Pretest 12 81.33 12.565 61 96
Posttest 12 127.92 3.476 122 131
Table 4: Assumptions test for Wilcoxon signed-rank test

p α Conclusion
Posttest – Pretest KKG .002 0,05 Signifikan
234
The results of the analysis showed p = 0.002 <0.05, which means that there is a
significant difference in the peace capacity or ability of the teachers before and
after the training. With the difference in the mean value of pretest = 81.33 and
posttest = 127.92, it can be concluded that after being given training, there is an
increase in the peace capacity of teachers.
Based on the analysis of the Wilcoxon signed-rank test, it can be seen that p =
0.002 < 0.05. Therefore it can be concluded that there are differences in the peace
capacity of teachers after being given training when viewed in terms of each
indicator, with the biggest change occurring in the peace capacity of teachers
being the indicator of ‘Strengthening Character Education’.
Table 5: Results of the analysis of the mean and standard deviation based on the pre-
& posttest aspects of teachers’ peaceful capacity
Aspect Pre & N Minimu Maximu Mean Std.
Posttest m m Deviation
Knowledge Pretest 12 25 40 33.08 4.757
Posttest 12 53 59 56.08 1.730
Skills Pretest 12 12 29 21.92 5.299
Posttest 12 34 37 35.83 1.115
Attitude Pretest 12 21 31 26.33 3.257
Posttest 12 34 38 36.00 1.348
Valid N 12
(listwise)
Table 6: Aspect assumption test using Kruskal-Wallis analysis test rank

Aspect N Mean Rank p α Conclusion
Knowledge 12 26.67
Skills 12 16.63 .003 0.05 Conclusion
Attitude 12 12.21
Total 36
Based on the Kruskal Wallis' analysis, p = 0.003 < 0.05. Therefore it can be
concluded that there is a difference in the peace capacity of teachers after being
given training when viewed from each aspect, with the biggest change occurring
in the peaceful capacity of teachers in the knowledge aspect.
5. Discussion
After the provision of training, there was a change in the teachers’ peace
capacity. There is also an increase in the number of teachers who have the
capacity to be included the peaceful category. The results of this study indicate
that teachers as educators should develop the capacity for peace in order to have
the knowledge, skills and attitudes of peace within themselves (Arifudin, 2007;
Cavanagh et al., 2012; Invernizzi, 2005; Kirkwood-Tucker, 2004; Tilaar, 1999;
Widayati, 2002).
The application of peaceful values should be integrated into both educational
and extracurricular activities. This can be achieved by strengthening the values
of peace in supporting the learning process. In extracurricular activities,
reinforcing the values of peace is carried out in order to develop the potential,
235
interests, talents, abilities, personality, and cooperation of students in voicing

and building peace or being peacemakers.
The capacity for peace that teachers should have is an understanding of the
concept of peace in the form of the concept of diversity and relationships among
communities, accepting, respecting, working with people of different
backgrounds, having empathy, and exhibiting tolerance (Montgomery &
McGlynn, 2009). Teachers who have the capacity for peace are better able to
provide peaceful learning (Helu, 2010). Teachers who are able to teach by
applying the values of peace make students feel comfortable in receiving
learning. Students who feel comfortable in participating in learning tend to find
it easy to internalise the learning content. Such students should excel in class and
in the community (Akhyak et al., 2013).
Students who are taught with a sense of peace are inclined to have peaceful
personalities and be able to apply the values of peace in every situation and
behaviour. On the other hand, students who are always at the receiving end of
unreasonable behaviour from their teachers and friends will be negatively
affected by this, exhibiting negative behaviour in turn (Deutsch, 1993).
Students who are taught with a sense of peace will have a happy personality,
attitude and positive behaviour (Deutsch, 1993). Teachers who have a peaceful
attitude are happier and find life meaningful (Kyte, 2016). Peace creates comfort
and security which ultimately increases psychological well-being in individuals
(Cohrs et al., 2013) and economic well-being for society (Cairns & Lewis, 2003;
Swee et al., 2019). Parents' perceptions also affect students' peace of mind
(Adiputra et al., 2019).
Teachers who have the capacity for peace will teach with great compassion.
Students tend to be able to follow the lessons and practise what they have
learned (Naway, 2019). Moreover, teachers who have the capacity for peace are
more inclined to have students with a happy personality (Biggs et al., 2008;
Twemlow et al., 2005), and who can more easily deal with the various trials,
challenges and problems in life (Cacioppo et al., 2011; Cohrs et al., 2013;
Hetherington, 2003).
6. Conclusion
Trends in the peace capacity of teachers in schools contributed to the findings of
research on teacher peace capacity. Teachers who have knowledge of peace will
be better able to be peaceful and have peace-making skills. Then the teachers’
ability to implement the values of peace in learning will increase and will lead to
a peaceful atmosphere or climate in the classroom.
Based on theoretical studies and the results of empirical studies, this training is
effective in developing the peace capacity of teachers in the form of increasing
knowledge, skills and attitudes of teachers' peace. Teachers who have a good
peace capacity will create a peaceful classroom climate.
236
7. Acknowledgment
The writers would like to express their gratitude to the Indonesian government,
especially the Indonesian Directorate General of Higher Education (DIKTI), and
the Indonesia Endowment Fund for Education (LPDP) for funding this research.
8. References
Adiputra, S., Mujiyati, & Hendrowati, T. Y. (2019). Perceptions of inclusion education by
parents of elementary school-aged children in Lampung, Indonesia. International
Journal of Instruction, 12(1), 199-212. https://doi.org/10.29333/iji.2019.12113a
Akhmad, S. N., Kartadinata, S., & Ilfiandra. (2016). Perspektif Peserta Didik tentang
Kedamaian dan Resolusi Konflik di Sekolah[Learners Perspectives on Peace and
Conflict Resolution in Schools]. Pedagogia: Jurnal Ilmu Pendidikan, 14(2), 343–355.
https://doi.org/10.17509/pedagogia.v14i2.3884
Akhyak, M. A., Idrus, M., & Bakar, A. Y. (2013). Implementation of teachers’ pedagogy
competence to optimizing learners’ development in public primary schools in
Indonesia. Journal of Education and Research, 1(9), 113-122.
Arifudin, I. (2007). Urgensi Implementasi Pendidikan Multikultural di Sekolah [The
Urgency of Implementation of Multicultural Education in Schools]. Jurnal
Pemikiran Alternatif Pendidikan Insania, 12(2), 220-233.
https://doi.org/10.24090/insania.v12i2.252
Bajaj, M. (2015). Pedagogies of resistance and critical peace education praxis. Journal of
Peace Education, 12(2), 154-166. https://doi.org/10.1080/17400201.2014.991914
Barber, N. (2005). Practitioner adaptations of strengths-based practice in statutory child
protection work. The Australian Institute of Family Studies Newsletter, 13, 10–15.
Baron, J. (1992). The effect of normative belief on anticipated emotions. Journal of
Personality and Social Psychology, 63(2), 320-330.
https://psycnet.apa.org/doi/10.1037/0022-3514.63.2.320
Benbenishty, R., & Astor. S. (2005). School violence in context. Oxford University Press.
Berkowitz, L. (1993). Aggression: Its causes, consequences, and control. McGraw-Hill.
Biggs, B. K., Vernberg, E. M., Twemlow, S. W., Fonagy, P., & Dill, E. J. (2008) Teacher
adherence and its relation to teacher attitudes and student outcomes in an
elementary school-based violence prevention program. School Psychology Review,
37(4), 533-549. https://doi.org/10.1080/02796015.2008.12087866
Borg, W. R., & Gall, M. D. (2007). Educational research: An introduction (8th ed.). Pearson-
Allyn & Bacon.
Buchori, S. (2018). Profil Budaya Damai Siswa Sekolah Dasar di Cilandak Jakarta Selatan
[Profile of the Peaceful Culture of Elementary School Students in Cilandak, South
Jakarta]. Supriatna, M., & dan Wahidin (Eds.). Orientasi Pribadi Optimal Perspektif
Bimbingan dan Konseling. UPI Press.
Buchori, S., & Fakhri, N. (2018). Terapi Film dalam Mengembangkan Budaya Damai
Siswa [Movie Therapy in Building a Culture of Peace in Students]. Prosiding.
Kongres dan Konvensi Nasional Bimbingan dan Konseling ke XX. ABKIN
bekerja sama dengan Universitas Riau. Pekanbaru [Proceedings. XXth National
Congress and Convention on Guidance and Counseling. ABKIN cooperates with
the University of Riau. Pekanbaru].
Cacioppo, J. T., Reis, H. T., & Zautra, A. K. (2011). Social Resilience: The Value of Social
Fitness with an Application to the Military. American Psychologist, 66, 43–51.
https://psycnet.apa.org/doi /10.1037/a0021419
Cairns, E., & Lewis, C. A. (2003). Empowering peace. The Psychologist, 16, 142–143.
237
Castro, L. N., & Galace, J. N. (2010). Peace education: A pathway to a culture of peace. Center
for Peace Education, Miriam College.
Cavanagh, T., Macfarlane, A., Glynn, T., & Macfarlane, S. (2012). Creating peaceful and
effective schools through a culture of care. Discourse: Studies in the Cultural
Politics of Education, 33(3), 443–
455. https://doi.org/10.1080/01596306.2012.681902
Chan, J. Y., Fung, A. L., & Gerstein, L. H. (2013). Correlates of pure and co-occurring
proactive and reactive aggressors in Hong Kong. Psychology in the Schools, 50 (2),
181-192. https://doi.org/10.1002/ pits.21665
Cohrs, J. C., Christie, D. J., White, M. P., & Das, C. (2013). Contributions of positive
psychology to peace: Toward global well-being and resilience. American
Psychologist, 68(7), 590–600. https://psycnet.apa.org/doi/10.1037/a0032089
Deutsch, M. (1993). Educating for a peaceful world. American Psychologist, 48(2), 510-517.
Deutsch, M. (2015). Educating for a peaceful world. Springer briefs on pioneers in science
and practice, 89–103. https://doi.org/10.1007/978-3-319-15443-5_7
Dulmus, C., & K. Sowers. (Eds). (2004). Kids and violence: The invisible school experience.
Haworth Press.
Eliasa, E. I. (2017). Budaya Damai Mahasiswa di Yogyakarta. Journal of Multicultural
Studies in Guidance and Counseling, 1(2), 175-19.
https://doi.org/10.17509/jomsign.v1i2.8286
Freire, P. (1998). Pedagogy of freedom. Ethics, democracy and civic courage. Rowman &
Littlefield Publishers.
Fung, A. L., Gerstein, L. H., Chan, Y., & Engebretson, J. (2015). Relationship of
aggression to anxiety, depression, anger, and empathy in Hong Kong. Journal of
Child and Family Studies, 24(3), 821-831. http://dx.doi.org/10.1007/s10826-013-
9892-1
Furlong, M., & Morrison, G. (2000). The school in school violence: Definitions and facts.
Journal of Emotional and Behavioral Disorders, 8(2), 71–81.
https://doi.org/10.1177%2F106342660000800203
Galtung, J. (1996). Peace by peaceful means peace and conflict, development and civilization.
Sage.
Goldman, J. D. G., & Grimbeek, P. (2008). Student-teachers’ understanding of policy
behavioural directives concerning the reporting of child sexual abuse: Findings
from one Australian state. Educational Research, 50(3), 291–305.
https://doi.org/10.1080/ 00131880802309457
Harber, C. (2004). Schooling as violence. Routledge Falmer.
Harber, C. (2009). Forthcoming. Long time coming: Children as only occasional decision-
makers in schools. In S. Cox, S., A. Robinson-Pant, C. Dyer, & M. Schweisfurth
(Eds). Children as decision makers in education. Continuum.
Harber, C., & Sakade, N. (2009). Schooling for violence and peace: How does peace
education differ from ‘normal’ schooling? Journal of Peace Education, 6(2), 171–
187. https://doi.org/10.1080/ 17400200903086599
Helu Thaman, K. (2010). Teacher capacities for working towards peace and sustainable
development. International Journal of Sustainability in Higher Education, 11(4), 353–
364. https://doi.org/ 10.1108/14676371011077577
Henry, D., Guerra, N., Huesmann, L. R., Tolan, P., Van Acker, R., & Eron, L. (2000).
Normative influences on aggression in urban elementary school classroom.
American Journal of Community Psychology, 28(1), 59-81.
238
Hetherington, E. M. (2003). Social support and the adjustment of children in divorced

and remarried families. Childhood: A Global Journal of Child Research, 10, 217–236.
https://doi.org/10.1177/ 0907568203010002007
Hilarski, C. (2004). Corporal punishment: Another form of school violence. In C. Dulmus
and K. Sowers (Eds). Kids and violence: The invisible school experience. Haworth
Press.
Huesmann, L. R., Rowell, L., & Guerra, N. G. (1997). Children’s normative belief about
aggression and aggressive behavior. Journal of Personality and Social Psychology,
72(2), 408-419. https://psycnet.apa.org/doi/ 10.1037/0022-3514.72.2.408
Hunter, T. J. (2008). Creating a culture of peace in the elementary classroom. The
Education Digest, 74(1), 54-58.
Invernizzi, M. A., Landrum, T. J., Howell, J. L., & Warley, H. P. (2005). Toward the
peaceful coexistence of test developers, policymakers, and teachers in an era of
accountability. The Reading Teacher, 58(7), 610–
618. https://doi.org/10.1598/RT.58.7.2
Janoff-Bulman, R. (1992). Shattered assumptions: Towards a new psychology of trauma. The
Free Press.
Kartadinata, S., Affandi, I., Wahyudin, D., & Ruyadi, Y. (2015). Pendidikan kedamaian.
Remaja Rosdakarya.
Kirkwood-Tucker, T. F. (2004). Empowering teachers to create a more peaceful world
through global education: Simulating the United Nations. Theory & Research in
Social Education, 32(1), 56–74. https://doi.org/10.1080/00933104.2004.10473243
Kyte, D. (2016). Toward a sustainable sense of self in teaching and teacher education:
Sustainable happiness and well-being through mindfulness. McGill Journal of
Education / Revue des Sciences de l'Education de McGill, 51(3), 1143–1162
https://doi.org/10.7202/ 1039632ar
Meyer, E. (2006). Gendered harassment in North America: Recognising homophobia and
heterosexism among students. In F. Leach & C. Mitchell (Eds.) Combating gender
violence in and around schools. Trentham Books.
Montgomery, A., & McGlynn, C. (2009). New peace, new teachers: Student teachers’
perspectives of diversity and community relations in Northern Ireland. Teaching
and Teacher Education, 25(3), 391–399. https://doi.org/10.1016/j.tate.2008.11.002
Naway, F. A. (2019). Sang Guru: Panduan Guru Profesional Menuju Indonesia 4.0. PGRI
Gorontalo-Press.
Nelson, L. L. (2014). Peacefulness as a personality trait. In G. K. Sims, L. L. Nelson, &
M.R. Puopolo (Eds.). Personal peacefulness: Psychological perspectives. Springer.
Nurhadi, A. M., & Muchtarom, M. (2020). Developing civic responsibility in the
community through the social community “Masyarakat Anti Fitnah Indonesia
(Mafindo)”. In 3rd International Conference on Learning Innovation and Quality
Education (ICLIQE 2019) (pp. 1087-1094). Atlantis Press.
O’Connor, P. (2013). Pedagogy of love and care: Shaken schools respond. Disaster
Prevention and Management, 22(5), 425-433.
https://doi.org/10.1016/j.tate.2008.11.002
Pinheiro, P. (2006). World report on violence against children. United Nations.
Sakade, N. (2008). Peace education in practice? A case study of peace education in England.
(Doctoral thesis in preparation). University of Birmingham.
Saputra, W. N. A. (2016). Pendidikan kedamaian: Peluang penerapan pada pendidikan
tingkat dasar di Indonesia [Peace education: Opportunities for application to
primary education in Indonesia]. Jurnal CARE Edisi Khusus Temu Ilmiah, 03(3),
hlm. 88–94.
239
Setiadi, R., Kartadinata, S., Ilfiandra & Nakaya, A. (2017). A peace pedagogy model for
the development of peace culture in an education setting. The Open Psychology
Journal, 10(1), 182-189. http://dx.doi.org/10.2174/1874350101710010182
Sinkinson, M. (2009). “Sexuality isn’t just about sex”: Pre-service teachers’ shifting
constructs of sexuality education. Sex Education: Sexuality, Society and Learning,
9(4), 421–436. https://doi.org/ 10.1080/14681810903265352
Swee, E., Zhan, H., & Powdthavee, N. (2019). Do perceptions of economic well-being predict
the onset of war and peace? (No. 12650). IZA Discussion Papers.
Taylor, S. E., Peplau, L. A., & Sears, D. O. (2009). Psikologi social: edisi dua belas [Social
psychology: twelfth edition]. Kencana Prenada Media Grup.
Tentama, F. (2013). Perilaku anak agresif: Asesmen dan intervensinya [Aggressive child
behavior: Assessment and intervention]. Jurnal Kesehatan Masyarakat (Journal of
Public Health), 6(2), 162-232.
Tilaar, H. A. R. (1999). Beberapa Agenda Reformasi Pendidikan Nasional Dalam
Perspektif Abad 21 [Several Agendas for National Education Reform in 21st Century
Perspective]. Indonesia Tera.
Twemlow, S. W., Fonagy, P., & Sacco, F. C. (2005). A developmental approach to
mentalizing communities: II. The Peaceful Schools experiment. Bulletin of the
Menninger Clinic, 69(4), 282–304. https://doi.org/10.1521/bumc.2005.69.4.282
UNESCO. (1996). Learning: Treasure within. UNESCO Publishing.
UNESCO. (2017). Transformative pedagogy for peace-building: A guide for teachers. UNESCO-
International Institute for Capacity Building in Africa (IICBA).
UNESCO-APNIEVE. (2000). Belajar untuk Hidup Bersama dalam Damai dan Harmoni:
Pendidikan Nilai untuk Perdamaian, Hak-hak Asasi Manusia, Demokrasi dan
Pembangunan Berkelanjutan untuk Kawasan Asia-Pasifik [Learning to Live
Together in Peace and Harmony: Values Education for Peace, Human Rights,
Democracy and Sustainable Development for the Asia-Pacific Region]. Buku
Sumber UNESCO-APNIEVE untuk Pendidikan Guru dan Jenjang Pendidikan Tinggi.
Penerjemah W.P. Napitupulu. Bandung: Komisi Nasional Indonesia untuk
UNESCO dan UPI.
Van Manen, M. (1991). The tact of teaching: The meaning of pedagogical thoughtfulness.
Althouse Press.
Werner, N. E., & Hill, L. G. (2010). Individual and peer group normative beliefs about
relational aggression. Child Development, 81(3), 826-836.
https://doi.org/10.1111/j.1467-8624.2010.01436.x
Widayati, S. (2002). Reformasi pendidikan dasar [Primary school education reform]. Grasindo.
240

Vol. 20, No. 8, pp. 240-261, August 2021
The Measurement of Knowledge Construction in

A Course of Diagnostic Evaluation of Learning
Disorders in Psychology Students
Guadalupe Elizabeth Morales-Martinez, Michelle Garcia-Torres and

Maria del Carmen Castro-Gonzalez
National Autonomous University of Mexico, Mexico City, Mexico
https://orcid.org/0000-0002-4662-229X
https://orcid.org/0000-0003-1997-9274
https://orcid.org/0000-0001-5883-0857
Yanko Norberto Mezquita-Hoyos

Autonomous University of Yucatán, Yucatan, Mexico
https://orcid.org/0000-0001-6305-7440
Abstract. This study evaluated the effects of academic learning on the

organization and structure of a knowledge schema among psychology
students. The authors designed three studies based on the Chronometric
Constructive Cognitive Learning Evaluation Model. This article describes
the first method of evaluation, which included a conceptual definition
task based on the Natural Semantic Networks technique. The
participants' task was to define ten target concepts using verbs, nouns, or
adjectives as definers, and then rate the quality of each definer, taking into
account the degree of semantic relationship between it and the target
concept. The results suggest that the students' initial knowledge schema
underwent modifications due to the restructuration of the cognitive
structure of knowledge, the assimilation of new information nodes, and
the elimination or establishment of new relationships between the
conceptual nodes of the knowledge schema. The measurement of these
cognitive expressions of academic learning through mental
representation techniques can have relevant implications for cognitive
characterization in student learning and the design of new teaching
strategies that take account of the cognitive psychology principles of
information processing underlying academic learning.
Keywords: learning evaluation; knowledge schema; cognitive

construction; Natural Semantic Networks; psychology students
©Authors
241
1. Introduction
Twenty-first century society demands that students learn information as well as
skills in order to be able to select and generate new knowledge from this
information. The new educational models attempt to address these two types of
learning, declarative and procedural, in their study programs. Meanwhile, new
technologies have allowed innovation in teaching strategies for concepts and
procedures using tools such as virtual reality and digital teaching platforms.
However, there are few innovations in the field of learning assessment;
furthermore, there are currently no instruments which measure all aspects of
declarative and procedural learning (El-Yassin, 2015). To date, the predominant
instruments in evaluation have been those that measure academic performance
rather than learning process. These instruments are useful in terms of providing
numerical indicators of a student’s academic performance; however, these
instruments measure the learning through a product without considering the
context (Sadeghi & Rahmati, 2017), nor do they measure the personal
characteristics of students. In general, summative assessments do not take account
of the fact that each student assimilates the knowledge they review in class in a
very personal way, and therefore, there is a wide range of results from academic
learning even with the same teaching quality and teaching conditions (Wiliam,
2011).
Arieli-Attali (2013) points out that teaching exclusively for performance deprives
teachers of valuable information which would allow them to make decisions
about how to modify their instructional techniques. Therefore, the academic
community has begun to recognize that the cognitive needs and characteristics of
the student are central elements in the design of the teaching-learning-evaluation
cycle (Morales-Martinez et al., 2021). In this regard, the United Nations
Educational Scientific and Cultural Organization (UNESCO, 2015) emphasizes
that the design of learning evaluation tools demands a vision beyond knowledge
measurement. Creating effective evaluation tools requires the purpose, scope,
nature, impact, and level of the evaluation to be considered.
2. The Chronometric Constructive Cognitive Learning Evaluation Model

(C3-LEM)
The incorporation of technologies within education has brought about advances
in educational design and instruction. However, there is still a gap between these
developments and innovation in terms of the methods used for assessing learning.
Most of the contributions in this field are limited to a technological encapsulation
of already existing evaluation strategies, and there are few evaluation proposals
native to the digital age (Morales-Martinez, 2020). Additionally, Arieli-Attali
(2013) points out that the available evaluation instruments provide information
on specific moments. Thus, it is not possible to obtain a complete picture of the
continuous progress of a student’s learning process; therefore, it is difficult for the
results obtained from these assessments to help improve a wide spectrum of
aspects of teaching. Furthermore, given the nature of these instruments, it is
difficult to obtain information on the essential cognitive aspects of learning. For
example, there is scarce information about the ways in which students organize
information, represent problems, select, and use learning strategies, and make use
242
of self-control skills. All these aspects are of relevance if we take into account that
fact that students live in a society with an economy based on the processing of
information. One way to approach this educational challenge is to use cognitive
technology to design new methods for evaluating learning.
Cognitive psychology includes a wide range of techniques used to create

instruments that provide information about the state of a student’s knowledge,
before, during, and after a course. For example, Lopez-Ramirez and Morales-
Martinez (2019), Lopez et al. (2014), Morales & Lopez (2016), Morales-Martinez,
Lopez-Perez, et al. (2020), Morales-Martinez et al. (2021), Morales et al. (2017) and
Morales-Martinez et al. (2015) have proposed the Chronometric Constructive
Cognitive Learning Evaluation Model (C3-LEM), which suggests the use of
cognitive tools to measure the cognitive properties of knowledge schemas learned
in an academic environment. For example, this model measures what content is
in a student’s memory, how it is organized and structured, the temporal and
dynamic patterns of these knowledge structures, and other aspects of a student’s
knowledge state.
The C3-LEM obtains the learning indicators in two phases in the evaluation
(Figure 1). The first phase is the constructive cognitive evaluation, whose main
objective is to measure the changes in the organization, structure, and cognitive
dynamics of a student’s knowledge schema, which are assumed to be due to the
learning produced by the course. The second phase involves the chronometric
cognitive evaluation of the changes that occur in the temporal patterns of
schematic behaviour and that are a measure of the degree of consolidation of the
schema in each student's long-term memory.
Figure 1: The two phases and components of the C3-LEM

Note: From “Cognitive e-tools for diagnosing the state of medical knowledge in students
enrolled for a second time in an anatomy course”, by Morales-Martinez, Angeles-
Castellanos, et al. (2020), International Journal of Learning, Teaching and Educational
Research, 19(9), p. 346 (https://doi.org/10.26803/ijlter.19.9.18). Copyright 2020 by the
authors and IJLTER.ORG.
Overall, this evaluation model provides information about conceptual advances

throughout academic training. To this end, the C3-LEM takes into account the
243
principles and laws of human processing based on two approaches to cognitive

psychology: the theory of human information processing or HIP and the theory
of parallel distributed processing or PDP. Since, in this paper, the main objective
is to explore the cognitive construction of the knowledge schema, the following
section focuses on some of the cognitive principles of knowledge construction
from these two approaches.
2.1. The Cognitive Principles of Knowledge Construction

From HIP, students' minds build cognitive schemas with the motor, procedural
and declarative information that they learn and store in their memories
throughout a course or career. Here, declarative schemas are of particular interest
since, in an academic learning context, students create knowledge networks about
what they learn from the semantic and conceptual information on the course. HIP
posits that semantic knowledge networks are made up of information nodes
(concepts) and relational links among these concepts. The organization and
structure of these nodes and relational links enable the students to give a
psychological meaning to their knowledge. In this regard, Figueroa-Nazuno
(2007) mentions that each person creates a meaning of the world by constructing
or reconstructing the knowledge schemas in their memory beyond free
association.
The authors of this paper define meaning construction as a cognitive process

through which students obtain a personal vision of their knowledge. In an
academic environment, meaning results from interaction among the cognitive and
emotional characteristics of students and their context and learning experiences.
Then students can create meaning or modify the meaning of their knowledge by
constructing or reconstructing their cognitive structure throughout an academic
learning process. Thus, learning experiences produce changes in the students’
cognitive structure.
From the cognitive psychology point of view, the degree to which students can
modify their knowledge structures depends on the flexibility and stability of their
schemas. According to Lopez-Ramirez and Morales-Martinez (2019) schematic
flexibility refers to the malleability of cognitive structure. It means the degree to
which the cognitive structure can be reorganized or reconfigured without being
destroyed during the assimilation of new information. On the other hand, these
authors define schematic stability as the degree to which the cognitive structure
can hold its cognitive configuration and organization after assimilating new
information.
The academic development level of students influences the cognitive flexibility

and stability of knowledge schemas. Generally, students who begin their learning
process in a new field of knowledge have a pre-schema with a vague organization
and a structure that is not very clear (Morales-Martinez et al., 2017). The authors
of this paper have hypothesized that this kind of initial cognitive structure could
be modified more easily than schemas which are completely organized and
configured. Furthermore, grasping the level of organization and structure of
knowledge schemas provides information about the learning needs that a student
244
might have. According to Messick (1984), the academic objective for beginners
with incipient cognitive structures of knowledge should be the assimilation of
new knowledge. Thus, the learning evaluations must measure the recognition or
retrieval of information in order to be consistent with this first academic level. In
contrast, students at an advanced level of academic development require teaching
strategies and assessments that promote the restructuring of the schemas and the
use of these schemas in problem-solving.
Since the learning needs of beginners are quite different from those of advanced
students, it is necessary to diagnose their academic development level. To this
end, it is helpful to consider Marzano and Pickering’s learning model (1997)
because it proposes three learning dimensions which are directly related to the
development of cognitive structures of knowledge, namely: the acquisition and
integration of knowledge; extending and refining knowledge; and the meaningful
use of knowledge. The first two dimensions are especially linked to this research
work, and this was the focus of this study.
According to Marzano and Pickering (1997), the acquisition and integration of

declarative knowledge require that students store information (facts, concepts,
and principles) in their memory, deliberately and consciously. Additionally, this
learning dimension involves meaning construction by linking old and new
knowledge. Another component of this dimension is information organization. In
order to acquire declarative knowledge, students must look for relational patterns
within the information. In contrast, extending and refining knowledge refers to
discovering new perspectives about the knowledge or establishing new links
within information by comparing, classifying, analyzing, or reasoning.
In this work, the authors interpreted these two learning dimensions from the
perspective of the mental representation of knowledge. The acquisition of
knowledge implies incorporating new nodes of information in students'
memories regardless of the organization of these concepts. Knowledge integration
requires the formation of meaning based on the organization of information
stored in the memory (e.g., the priority of recall of new nodes, the general patterns
of recall). Extending and refining knowledge entails incorporating more
specialized and accurate concepts to theorize each knowledge domain,
establishing new relationships among conceptual nodes of the knowledge
schema, forming new knowledge structures, or creating innovative inferences
from the information stored in the memory.
All these aspects of the cognitive construction of knowledge can be measured

throughout the constructive cognitive evaluation, which provides evidence about
the type of information that students choose as relevant, the way they organize
and structure it to build their knowledge, and their academic development. This
is illustrated in the next section.
2.2. Advances in the Cognitive Assessment of Knowledge Construction

When students enroll on a course, teachers and the education system expect them
to learn by undergoing the experiences designed specifically for that purpose.
245
Constructive cognitive evaluation allows the changes in students’ knowledge

schema produced by the learning experiences to be measured. This kind of
assessment involves applying a mental representation technique and
computational simulations (Morales-Martinez et al., 2021; Morales-Martinez,
Angeles-Castellanos, et al., 2020; Morales-Martinez, Lopez-Perez, et al., 2020). In
this regard, the C3-LEM uses Natural Semantic Networks (NSNs) to explore the
mental representation of academic knowledge. The NSNs give information about
how persons construct their own meaning of the world during a process of
memory construction and reconstruction of the knowledge that is stored in their
memory (Figueroa-Nazuno, 2007; Figueroa et al., 1976).
From a C3-LEM perspective, the main purpose of NSNs is to discover the meaning
that students give to the knowledge that they learn on academic courses.
Furthermore, since this technique could be applied at different points in time
during the academic course, it is possible to obtain a fluid understanding of the
cognitive dynamic of construction of knowledge schemas. Students modify their
declarative knowledge schemas endlessly by assimilating the information they
consider relevant from the material they review throughout the course.
Consequently, the construction of meaning from knowledge is a continuous
process through time. The result of this learning process depends upon various
factors as the prior knowledge that a student has at the beginning of a course
(Morales-Martinez, Lopez-Perez, et al., 2020; Urdiales-Ibarra et al., 2018; Morales-
Martinez et al., 2021), the student motivation to learn, the cognitive functioning
level on which the course focuses (Morales, 2020), the distribution of topics during
the course and the importance assigned to each topic (Morales-Martinez, Angeles-
Castellanos, et al., 2020).
The NSNs are helpful in detecting the characteristics and cognitive changes that
the schemas undergo due to learning at different time points within an academic
year. In this regard, Morales-Martinez, Lopez-Perez, et al. (2020) observed that
psychology students seem to begin the courses with a pre-schema. However,
these pre-schemas do not show a clear organization among their information
nodes. After the course, students have learned new information nodes, eliminated
other nodes, and established new relationships between the nodes they
assimilated from the course. These changes are specific indicators that the student
has experienced a learning process due to the academic experiences during the
course (Morales-Martinez et al., 2021). In addition, the NSNs technique has made
it possible to identify difficulties in the integration of information in a unified
schema in students with low academic performance and in beginners, when
learning a topic (Morales-Martínez, Mezquita-Hoyos, et al., 2018; Morales-
Martinez, Angeles-Castellanos, et al., 2020; Morales-Martinez, Lopez-Perez, et al.,
2020; Urdiales-Ibarra et al., 2018). Furthermore, Morales-Martinez, Angeles-
Castellanos, et al. (2020) applied this technique to explore the formation of the
anatomy schema in medicine students and they discovered that other factors
could affect schema configuration in the minds of students, for example the
difficulties associated with academic performance, the level of academic
development of students, and the emphasis and distribution of the topics to be
reviewed.
246
The NSNs can provide information on the effectiveness of corrective learning

strategies. Using this tool, Morales-Martinez, Mezquita-Hoyos, et al. (2018)
observed that engineering students who did not achieve passing grades in a
computational usability course could integrate the knowledge schema in this
subject after attending a remedial course. Bearing in mind that NSNs are very
useful for assessing several aspects of learning, Morales-Martinez, Trejo-
Quintana, et al. (2021) used this technique to explore the cognitive properties of
knowledge acquisition on human cognition in psychology students. These
authors observed that students with previous information about the evaluated
topic have a cognitive pre-schema of knowledge. However, the pre-schemas of
advanced students could be different in terms of the content quality, organization,
and cognitive structure of knowledge from those of beginners, namely students
who are enrolled on a course for the first time (Morales-Martinez, Lopez-Perez, et
al., 2020). This result suggests that the changes in knowledge structure are
qualitatively different depending upon the expertise of the students.
Additionally, the cognitive properties of pre-schemas seem different between
beginners and advanced students. Since there are very few studies based on this
cognitive approach, it is necessary to provide empirical evidence on the learning
properties of knowledge schemas based on the academic development level of
different students.
Assuming that students with expertise on a topic demonstrate better organization

and structure in the knowledge schema than beginner students, the present
research work explored if the cognitive structure of the diagnostic evaluation for
learning disorders schema in participants in this study was well-organized and
structured since students had been enrolled on previous courses on this topic. If
the students had a pre-schema based on their expertise in the diagnosis and
learning disorders, then, it was expected, that their pre-schema would include
well-organized information and a schema configuration which was clearly
integrated as observed in the study by Morales-Martinez, Trejo-Quintana, et al.
(2021). In brief, this work aims to broaden our understanding of the scope of this
technique in a different domain of knowledge and provide more evidence about
the cognitive characteristics of knowledge schema in students with some expertise
in a topic.
3. Method
In this research work, the present authors explored the changes in the knowledge
schema that psychology students underwent during a course on the diagnostic
evaluation of learning disorders. In order to achieve this, the organizational and
structural properties of the students’ knowledge schema were measured
throughout the NSNs. This technique is based on a cognitive view of the mental
representation of knowledge. From this approach, students form a cognitive
structure of the information that they learn during academic courses. NSNs is a
tool for observing the cognitive characteristic of such knowledge. In this study,
the main objective was to observe the organization, structure, and dynamic
properties of the cognitive schemas of knowledge in a group of psychology
students.
247
3.1. Sample
The participants were 43 undergraduate students enrolled in the 5th semester of
psychology (91% female and 9% male). Their ages ranged between 18 and 25 years
old (M = 19.7, SD = 1.30). Participants were chosen based on purposive sampling.
Participation was voluntary, and the teacher awarded points to the students for
their participation. The participants belonged to two different groups; since these
groups were each made up of a small number of students, the teaching conditions
for each class (time exposure, readings, activities) were the same, and the same
teacher taught the topics within the same context, and academic period; thus, the
researchers integrated the participation of both groups within a single data set.
3.2. Instruments and Materials

The authors designed the NSNs instrument using the Protocol for the Collection
of Target Concepts and Central and Deferred Definers (Morales-Martinez, 2015).
This protocol provides directions to the researcher to help the teacher identify the
ten target concepts for a course. In this study, the target-identified concepts were
development, psychomotor, learning, dyslalia, dyslexia, dyscalculia, attention deficit,
evaluation, diagnosis, and reporting. The presentation of the target concepts of the
study required the use of EVCOG software. This software allows the design,
application, capture and analysis of the data for cognitive studies for the C3-LEM
(Morales-Martinez, Angeles-Castellanos, et al., 2020) (Morales-Martínez and
López-Ramirez, 2018 a, b, c, d).
3.3. Research Design

This research employed a quasi-experimental design. The authors applied the
NSNs technique before and after the course to explore the changes in the schema,
for the diagnostic evaluation of learning disorders, produced by the learning
process undertaken on the course. The study involved a definitional task to
recover the main conceptual nodes related to the evaluated knowledge schema.
3.4. Procedure
The study comprised two stages; during the first one, the authors informed the
participants about the objectives, the procedure, and the benefits of participating
in the study. In this informative stage, the students who decided to participate
voluntarily gave their informed consent. In the second stage, the students received
instructions, practiced the NSNs task and answered the final study. The task was
to define ten target concepts using verbs, nouns, adjectives, and pronouns as
definers. Each objective had to be defined within 60 seconds, and then the
participants rated the definers using a scale from 1 to 10. Low scores meant that
the quality of the word as a definer was low, and high scores indicated that the
definer was significantly related to the target. Three essential restrictions
delimited this task; the first was that the targets had to be defined based on the
course content; that is, free association was ruled out. The second one restriction
was that there was a pre-established time for defining each concept (60 seconds).
Finally, the targets were presented at random. The application time oscillated
between 15 and 20 minutes. The NSNs task was carried out twice, at the beginning
and the end of the course.
248
4. Data Analysis
The authors examined the data using two analysis approaches. The first one was
the conventional NSNs analysis of the participant data using the EVCOG. This
analysis implies the computation of organization and structure indicators
proposed by Figueroa et al. (1976) and modified by Lopez (1996) and Lopez and
Theios (1992). First, semantic richness (J value) was computed as the number of
different definers in each target concept. Semantic weight (M value) was
calculated as the quality degree estimated by participants considering the
semantic relationship between the definer and target. The SAM group (Semantic
analysis of M) was made up of the ten definers with the greatest semantic weight
for each target concept. The semantic density (G value) was computed by the
closeness score between the concepts of the network. In addition, the authors
calculated the inter-response time (IRT), which is the time that student needed to
recover and write a definer. Furthermore, the authors computed the percentage
of concepts appearing in the initial SAM groups as well as in the final ones (the
conceptual constancy or CC Value). Also, the conceptual valuation consistency
(CVC Value) was computed taking into account the similarity percentage between
the weights assigned to the definers that were constant at the beginning and the
end of the course in each SAM group. This indicator is a modification of the Q
value of Figueroa et al. (1976) that measures the percentage of similarity in the
hierarchies of the common definers between two semantic networks.
The second analysis was qualitative and was based on a visual inspection of the
organization and structure of the concepts through a GEPHI analysis (Bastian et
al., 2009). This analysis first required the SASO matrix (matrix of the semantic
analyzer of schemata organization) to be extracted; it is a matrix of association
weights among the concepts of the NSNs (Lopez, 1996; Lopez & Theios, 1992).
Then, the researchers extracted the SASO matrix by calculating the probability of
co-occurrence between the concepts with the following formula:
WIJ = -1n{[p(X=0 & Y = 1) p(X=1 & Y = 0)]*[p(X=1 & Y = 1) p(X=0 & Y = 0)]-1}[1]
The EVCOG system allows the automatic calculation of the association weight
(WIJ) between the concepts (X and Y). This software calculates the probability of
co-occurrence between the pairs of concepts. First, the program calculates the joint
probability that Y appears, but X does not appear in a SAM group p(X = 0 & Y =
1). The procedure is similar for each element of the formula. However, the
calculation of p(X = 1 & Y = 1) involved estimating the hierarchical modulation of
M-values in SAM groups. Finally, the authors fed the GEPHI software with the
SASO connectivity matrix to visualize the schema (see Figure 4).
5. Results
Three aspects were analyzed through the NSNs data. First, the authors
determined what kind of information fitted into the initial knowledge schema of
students and how this information had changed at the end of course in the
students’ memories. The second aspect was to explore the changes in the dynamic
of connection among the main conceptual nodes (targets). Finally, it was carried
on an inspection of the structure and organization of concepts and structure
schema was carried out, through a visual representation of NSNs.
249
5.1. Results from NSNs Analysis

The analysis considered the NSNs indicators described by Lopez and Theios
(1992). Tables 1 and 2 present the NSNs indicators (frequency, IRT, M, F, J, and G
value) that the participants obtained at the beginning and the end of the course.
Also, there were changes in the content and number of conceptual nodes through
the NSNs (Tables 1 and 2). In this respect, the J value increased for each target
towards the end of the course. This result means the students showed an increase
in the semantic richness of their knowledge structure which was linked to the
course. The increase was not homogeneous across the targets, however.
Dyscalculia was the target with the most significant increase in the number of
definers, followed by report. Moreover, diagnosis and learning were the targets with
the smallest increase in the number of definers.
Table 1. SAM groups at the beginning of the course

Development Psychomotor Learning
F Definidor M TIR F Definidor M TIR F Definidor M TIR
1 Growth 224 20 1 Movement 276 19 1 Knowledge 153 20
4 Learning 97 28 1 Motricity 116 16 1 Memory 97 26
1 Evolution 81 22 1 Body 97 21 2 School 88 22
1 Physical 77 39 2 Development 86 36 2 Attention 68 28
1 Maturation 73 25 3 Skills 76 32 2 Development 64 34
1 Stages 72 28 1 Brain 69 28 3 Skills 52 32
1 Go forward 57 29 1 Mind 54 24 1 Study 51 20
3 Skills 50 28 3 Kids 47 41 2 Information 50 32
1 Psychological 49 29 1 Motor 40 28 1 To learn 39 14
1 Process 40 19 1 Psychology 37 14 1 Education 38 22
Valor J: 220 Valor G: 18.40 Valor J: 201 Valor G: 23.90 Valor J: 235 Valor G: 11.50
Dyslalia Dyslexia Dyscalculia
F Definidor M TIR F Definidor 197 TIR F Definidor M TIR
2 Language 194 13 5 Disorder 136 14 2 Numbers 161 14
5 Disorder 148 24 2 Words 130 18 1 Math 132 26
1 Speech 139 23 2 To read 129 26 5 Problems 104 18
2 Words 116 22 5 Problems 120 16 5 Disorder 93 29
3 Difficulty 99 20 1 Letters 111 18 4 Learning 78 39
5 Problems 89 17 1 Writing 108 33 3 Difficulty 55 20
1 Pronunciation 71 35 1 Confusion 84 32 1 Addition 46 34
1 Phonemes 62 44 2 Language 76 27 1 Operations 44 39
2 To read 51 25 2 Numbers 51 29 1 Subtraction 40 33
2 Tongue 48 38 3 Kids 50 34 2 School 31 31
Attention-deficit Evaluation Diagnosis
1 Distraction 147 31 2 Tests 153 25 2 Evaluation 172 17
5 Problems 141 24 1 To rate 136 24 2 Tests 99 30
5 Disorder 120 21 1 Diagnosis 92 21 1 Result 58 20
3 Kids 101 27 1 Exam 85 21 5 Problems 51 38
1 Hyperactivity 96 21 1 Knowledge 70 22 5 Disorder 50 24
1 Concentration 61 32 4 Learning 59 31 1 Interview 49 29
4 Learning 54 35 1 Revision 44 43 1 Patient 46 32
1 Lack of attention 52 38 2 Questions 42 39 1 Treatment 46 43
2 Attention 46 37 3 Results 36 24 2 Questions 34 30
3 Difficulty 38 21 1 To know 29 49 3 Results 33 22
Report
F Definidor M TIR
1 A text 180 23
2 Information 113 22
1 Investigation 76 25
1 Reading 67 37
2 Evaluation 62 26
1 Summary 61 29
1 Data 57 22
1 Essay 45 37
1 Conclusion 43 40
3 Results 43 35
Valor J: 189 Valor G: 7.00
250
In contrast, the reader can observe in Table 2 that the participants eliminated
information nodes that appeared in the first phase, and they also included new
information nodes in each SAM group. For example, the target dyslalia presented
a CC value of 90% for the NSNs; nine of the ten initial concepts remained in the
final NSNs (language, disorder, speech, words, difficulty, problems, pronunciation,
phonemes, reading), while one of them was eliminated (language). Thus, a new
concept (articulation) was assimilated in the SAM group towards the end of the
course. On the other hand, evaluation had a CC value of 30% for the NSNs. Of the
ten initial concepts, only three concepts (tests, diagnosis, exam) appeared again in
the final SAM group, while seven concepts (qualify, knowledge, learning, review,
questions, results, know) were eliminated and in their place appeared seven new
concepts (treatment, interviews, observation, diagnosis, psychological, analysis,
evaluate).
Table 2. SAM groups at the end of course
Development Psychomotor Learning
1 Growth 138 15 1 Movement 246 22 1 Knowledge 163 18
1 Stages 120 20 1 Motricity 153 15 1 School 75 32
1 Physical 107 24 1 Development 148 23 1 To learn 71 20
1 Maturation 99 20 1 Body 108 27 1 Memory 56 17
1 Evolution 94 12 1 Fine motricity 76 33 3 To read 56 38
1 Cognitive 91 23 1 Gross motricity 68 33 1 Experiences 55 26
1 Social 88 28 1 Laterality 57 28 1 Skills 47 30
2 Process 78 18 5 Difficulty 54 38 2 Information 42 29
1 Psychomotor 61 22 2 Psychological 54 42 1 To know 40 48
4 Learning 58 29 1 Brain 36 36 2 Process 38 24
Dyslalia Dyslexia Dyscalculia
F Definidor M TIR F Definidor 197 TIR F Definidor M TIR
2 Language 159 18 3 To read 224 21 1 Numbers 236 17
1 Phonemes 157 17 1 Writing 178 19 1 Math 173 18
5 Difficulty 150 18 5 Difficulty 171 13 5 Disorder 173 16
5 Disorder 128 16 5 Disorder 157 14 5 Difficulty 142 17
1 Articulation 104 25 1 Letters 107 22 1 Subtraction 89 34
1 Speech 79 21 2 Language 88 22 1 Addition 83 33
2 Words 65 26 4 Learning 78 37 1 Arithmetic 82 31
2 Problems 53 26 2 Words 68 14 4 Learning 68 31
1 Pronunciation 46 49 1 Reading-writing 59 30 2 Problems 65 20
3 To read 45 17 1 Omissions 44 40 2 Kids 60 28
Attention-deficit Evaluation Diagnosis
5 Disorder 193 16 2 Diagnosis 155 26 2 Evaluation 215 18
1 Hyperactivity 175 21 3 Tests 151 21 3 Tests 153 23
1 Attention 136 26 3 Treatment 75 35 3 Treatment 89 25
5 Difficulty 134 18 3 Interviews 65 32 1 Functional 82 24
4 Learning 68 33 1 Observation 63 31 2 Results 67 22
1 Distraction 67 29 1 Diagnostic 52 17 2 To evaluate 63 17
1 Uneasy 63 37 1 Psychological 44 33 5 Disorder 56 30
2 Kids 60 34 2 Analysis 35 36 3 Interviews 45 40
1 Lack of attention 49 23 2 To evaluate 30 28 1 DSM-V 41 45
1 Impulsiveness 44 18 1 Exam 29 31 1 Therapy 36 45
Report
F Definidor M TIR
2 Diagnosis 180 31
2 Results 168 19
2 Evaluation 141 26
3 Tests 126 21
2 Psychological 124 17
3 Treatment 66 27
2 Information 58 36
1 A text 56 22
3 Interviews 51 17
2 Analysis 48 23
Valor J: 259 Valor G: 12.00
251
Furthermore, the constant nodes had changed their weight of relevance towards
the end of the course. This change was heterogeneous through the targets. For
example, development obtained a CVC of 41%. In contrast, psychomotor obtained a
CVC of 38%. On another note, concerning the structural changes, three of the ten
SAM groups (development, psychomotor, dyslalia) indicated a decrease in the G
value, which means the closeness of the definers in these three SAM groups had
increased at the end of the course. In addition, five SAM groups showed a notable
increase in the dispersion of their definers at the end of the course (dyslexia, report,
dyscalculia, attention deficit, diagnosis), and two SAM groups had a tiny increase in
the dispersion of the definers (learning, evaluation). In addition, there were changes
in connectivity between the targets. Figure 2 shows the distribution of connections
at the beginning and end of the course.
Figure 2: Changes in the number of connections among targets

Note: The number of connections for each conceptual node is reported with the number
inside the circle. Furthermore, the connectivity graph after the course gives the number of
constant connections (black number) and the new connections (blue number). Also, in the
graph for after the course, three aspects of conceptual connectivity are reported: the
fading of the initial target connections (red circles), the constancy in the number of target
connections (gray circles), and the emergence of new target connections (blue circles).
252
Dyscalculia is the target with the most connections and report is the target with
least connections before the course. Meanwhile, at the end of course, the target
with the highest number of connections was dyslexia, and evaluation had the lowest
number of connections. Dyslexia was the target with the greatest number of new
connections, and evaluation presented the lowest number of connections at the end
of the course. Diagnosis, attention deficit, dyslexia, and dyscalculia were the targets
that retained the number of connections presented in the initial NSNs. Evaluation
was the target with the highest number of lost initial connections.
5.2. Results from GEPHI Analysis

The GEPHI analysis (Bastian et al., 2009) revealed changes in the conceptual
organization of the NSNs at the end of the course (Figure 3). Before the course, the
students had information about the subject, and they presented an organized
pattern in terms of the relationships between the information nodes they had. At
the beginning of course, students organized their knowledge schema into seven
conceptual modules. The first one (purple colour) involved 31.43% of the NSNs
definers (interview, questions, treatment, patient, information, evaluation, tests, result,
results, distraction, concentration, hyperactivity, pronunciation, lack of attention,
phonemes, speech, tongue, speech, development, attention, school). The main nodes in
this module were problems and disorders. The second module (light green)
included 22.86% of the definers (psychological, maturation, evolution, go forward,
growth, physical, process, stages, exam, diagnosis, revision, to rate, knowledge). The
nodes with more relevance in this group were skills and learning. The third module
(light blue) included 15.71% of the definers (movement, body, motricity, mind, brain,
motor, psychology, writing, letters, confusion), and the node with more connections
was kids. The fourth module (orange) consisted of 10% of the definers (a text,
summary, investigation, conclusion, reading, essay, data), and it was disconnected
from the NSNs. The fifth module (coffee) contained 7.14% of the definers
(difficulty, language, numbers, to read, words). The sixth module (pink) incorporated
7.14% of the definers (to learn, memory, education, study, knowledge). The last module
(dark green) integrated 5.71% of the definers (addition, subtraction, operation, math).
At the end of the course, students grouped the NSNs definers into nine conceptual
modules. The first group (purple) included 21.74% of the definers (psychomotor,
physical, cognitive, maturation, social, stages, growth, to read, words, language, problems,
kids, evolution, problems), and the more relevant nodes were learning and process.
The second module (green) included 20.29% of the definers (lack of attention,
uneasy, election, impulsiveness, distraction, hyperactivity, disorder, functional,
articulation, DSM-IV, phonemes, speech, pronunciation), and the node with more
connections was disorder. The third module (light blue) comprised 14. 49% of the
definers (information, evaluation, results, psychological, treatment, analysis, interviews,
test, diagnosis, evaluate), and the main node was kids. The fourth module (coffee)
contained 13.04% of the definers (brain, motricity, laterality, gross motricity, body,
development, fine motricity, movement), and the most salient node was difficulty. The
fifth module (orange) included 10.14% of the definers (knowledge, skill, school,
memory, experiences, to learn, to know). The sixth group (pink) included 7.25% of the
definers (numbers, subtraction, math, addition, arithmetic). The seventh module (dark
green) included 5.8% of the definers (writing, letters, omissions, reading, writing).
The eighth group (pale pink) involved 5.8% of the definers (diagnosis, observation,
253
exam, psychological), and it was separated from the NSNs; finally, the last group
(gray) contained just 1.45% of the definers (a text).
Figure 3: Before and after of the course: GEPHI analysis

Note: Each conceptual module obtained by GEPHI from NSNs is represented by a colour.
254
Furthermore, the organization of concepts and the meaning of several conceptual

nodes had changed at the end of the course. For example, the definer difficulty
attained greater centrality during the second part of the course. This result may
indicate that the students had re-signified this concept based on the knowledge
they acquired. At the beginning of the course, difficulty was grouped with nodes
associated with general areas (language, numbers, words, reading) in which children
may present learning difficulties during their development. Meanwhile, at the
end of the course, difficulty was grouped with words of a more sophisticated level
related to the etiology of these difficulties (brain), the specific areas of
development (fine motor, gross motor, laterality) and was also connected to broader
spectrum concepts (development or body). Figure 5 illustrates the change in
connectivity for the definer difficulty.
Figure 5: Change in connectivity for the definer difficulty
6. Discussion
Learning assessment is one of the most significant twenty-first--century classroom
challenges. UNESCO (2015) noted that the effectiveness of evaluation tools
depends on taking account of the purpose, scope, nature, impact, and level of
evaluation in their design. In this regard, this work has demonstrated the
effectiveness of constructive cognitive assessment to obtain information about
students' state of knowledge before and after a course. For this, the authors
applied the NSNs technique to measure the changes in the organization and
structure of the knowledge schema for the diagnostic evaluation of learning
disorders. The authors explored three aspects of cognitive learning, the first
referring to the conceptual content that the students had in their memory about
the course, the second relating to the conceptual structure interpreted as the
arrangement between the conceptual nodes of the natural semantic network, and
the third concerning the general meaning that results from the combination of
conceptual content and its organization and structure.
The results indicated that students entered the course with prior knowledge about
the topic. These results are similar to those of the study by Urdiales-Ibarra et al.
(2018) in biology and Morales-Martinez, Lopez-Perez, et al. (2020), and Morales-
Martinez et al. (2021) in psychology. The role of these pre-schemas in academic
255
learning has not been widely discussed from the cognitive perspective of
knowledge mental representation. Figueroa-Nazuno (2007) mentioned that
people construct and reconstruct meanings based on what is in their memory.
Thus, the authors hypothesize that these previous cognitive structures are a
primary mechanism of knowledge construction. Students use pre-schemas to
have cognitive coherence and to create a more efficient learning path. Based on
HIP, the participants used their previous schemas to assimilate and accommodate
new knowledge; therefore, they probably used a top-down processing style
predominantly to manage the information in their memories. There is not enough
information to know if this kind of processing is apparent among beginners.
A comparison of the conceptual content in Tables 1 and 2 indicates a relevant

change in the degree of generality of the NSNs. The initial schema included
general concepts connected to the NSNs throughout a psychology macro-schema;
additionally, students used common sense to build this seminal schema.
However, these concepts were not associated in a strict sense with the content of
the course. For example, at the beginning of the course, students defined
development alongside concepts taken from a general psychology schema (e.g.,
psychological) or inferred by common sense (e.g., skills). Meanwhile, at the end of
the course, students had eliminated these definers and included more specific
concepts such as cognitive, social, and psychomotor. According to Marzano and
Pickering's learning model (1997), these results suggest that the course's learning
experiences influenced the dimension of the refinement of the information in the
memory of these students.
Another aspect that the reader can observe in Tables 1 and 2 is the absence of
examples to define the targets. The use of examples is common among students
who are beginners in the study of a specific topic. For example, Morales-Martinez
et al. (2021) reported that at the beginning of a course on human cognition,
psychology students used many examples to define the concept of cognition. This
result can be associated with the students' initial expertise level on the subject. So,
in the present study, the kinds of concepts used in the initial NSNs suggest that
the participants had at least some knowledge on the subject, although the
knowledge schema they had at the beginning of the course was quite general. In
fact, according to their career curriculum, students had received some information
on the subject from previous courses.
In addition, some concepts showed greater conceptual constancy (Tables 1 and 2).
For example, the target dyslalia retained 90% of its definers in the final SAM group,
while evaluation had retained only 30% of its definers by the end of the course.
Although there is no theoretical discussion about the meaning of conceptual
constancy, the authors hypothesize that the constancy of a definer in the NSNs
over time is related to the condition of necessity or sufficiency of the definer in
defining the target. If the definer is an indispensable or central property of the
target, this will be presented constantly overtime in the NSNs. On the other hand,
conceptual migration (elimination of conceptual nodes) may be related to
circumstances when the initial nodes are not indispensable attributes for the
concept; then the definer appeared incidentally in the initial NSNs. For example,
256
when the definers are part of a macro schema of knowledge and this schema is
the only conceptual resource that students have, these general definers will
appear in the NSNs while the schema evolves towards a more sophisticated one.
In addition, a high CC may suggest that the target presents invariance of meaning;
thus, the greater the invariance of meaning, the lower the level of flexibility in
terms of the schema being modified by new incoming information. In this study,
the objectives with the highest CC were dyslalia, attention deficit, dyscalculia,
dyslexia, and development; therefore, these targets underwent less migration or
conceptual change. The authors suggest that the conceptual stability of these
targets could be related to the type of schemata; if a target is clearly defined
(classic schemata) or there are clear guides to identify its conceptual properties
(probabilistic schemata), then the constancy of definers increases. There is
agreement about which characteristics are necessary and sufficient to define
learning disorder in psychology. If students learn the criteria to diagnosis these
conditions from standardized diagnostic manuals (e.g., Diagnostic and Statistical
Manual of Mental Disorders), they demonstrated major consensus in their
definitions across time. The second factor influencing the CC could be that
students were familiar with these five topics since they had reviewed this
knowledge domain throughout their entire academic training. The opposite
happened with evaluation, which was the target with the lowest CC, and which
therefore underwent greater modification in its final NSNs group. These results
support the authors' idea that when a student is a beginner in a certain subject,
knowledge schemas can be more flexible, and therefore, these structures can be
more easily modified through learning experiences. As students gain more
knowledge of a topic, their flexibility in terms of the schemas may decrease, and
the stability of the schema may increase.
On the other hand, the constructive cognitive evaluation explored the structural
characteristics of the knowledge schema that the students constructed during the
course. The G value indicates the dispersion between the concepts of the NSNs;
in this regard, the reader can observe in Tables 1 and 2 that the target development
showed a marked decrease in the dispersion of the defining concepts, while report
and dyslexia markedly increased their dispersion. Interpreting these results is not
an easy task, given that there is no detailed description of the meaning of network
dispersion or density from the field of mental representation of cognitive
psychology (e.g., Figueroa et al., 1976; Lopez et al., 2014; Morales-Martinez,
Angeles-Castellanos, et al., 2020; Morales-Martinez et al., 2021; Urdiales-Ibarra et
al., 2018).
The authors suggest that the G values may indicate the degree of similarity among
the definers concerning the conceptual belonging they have regarding the target.
Low G values could suggest a major homogeneity in definers in defining the
concept, while high values may indicate greater variability in the degree to which
each definer conceptually typifies a target. The reasons for the changes in G values
are unknown; the decrement in this NSNs dispersion could suggest that the
learning experiences increase the quality of the selection process for definers.
Therefore, the closeness of meaning among these definers increase. Another
possibility is that learning experiences influence the valuation process for the
257
grade of relatedness among definers and their target because students acquire
abilities to find relations among the definers and their targets.
Concerning the pattern of connections between the targets for the NSNs, the
results indicated that the targets related to disorders were the most stable in terms
of the number of connections and the kind of definers involved in these
connections. Since there are few reports about the cognitive nature of NSNs
connectivity, it is difficult to explain these results from the cognitive perspective
of mental representation. For example, Morales-Martinez et al. (2021) reported on
the NSNs connectivity of anatomy in students who did not achieve a passing
grade in this subject; however, they did not provide information about the
cognitive nature of the connection pattern. In this study, the authors suggest that
the persistence of the connection of the schema structure over time could relate to
the level of expertise demonstrated by a student. Since, participants in this study
had read on learning disorders, they could have formed a schema of this subject
very early in their academic development. Then, their NSNs on this topic had well
established connections.
The third dimension of analysis was related to the meaning of the NSNs. The
analysis of the CVC values suggested that the readings reviewed during the
course changed the perception that students have about the relevance of the
conceptual nodes for the NSNs. The CVC of the definers with CC has not been
discussed in any article on constructive cognitive assessment (e.g., Morales-
Martinez et al., 2021; Morales-Martinez, Angeles-Castellanos, et al., 2020; Morales-
Martinez, Lopez-Perez, et al., 2020; Urdiales-Ibarra et al., 2018). The authors
propose that the CVC value may reflect the cognitive flexibility of the schema in
terms of conceptual valuation. If the definers with CC are cognitively permeable
to the input of new information, they can change their relevance values through
a revaluation process given the new information entering the schema and the new
relationships established. Therefore, when CVC is high, the best chance of
increasing the students' learning rate is by presenting novel information in the
form of new conceptual nodes, new conceptual relationships, and new forms by
which to interpret and use information.
In addition, the GEPHI analysis indicated an essential change in the meaning

attributed to learning disorders; at the beginning of the course, the students had
learning, disorder, problems, and skills as central nodes, while at the end of the
course, the central nodes were learning, disorder and difficulty. The definer difficulty
replaced problems; this suggests that the students changed their vision towards a
more proactive view of diagnosing and treating learning difficulties. However,
since the centrality of skills disappeared, evaluation and diagnosis represented a
focus on what is missing or damaged (difficulties and disorders), and thus, the
conceptualization of diagnosis continues to be driven by the medical model of
disease.
Finally, the organization of initial definers suggests that the students had a general
idea about assessing and diagnosing learning disorders. They established
relationships between the definers in relation to their general knowledge about
258
development and learning. At the end of the course, the students had integrated
new information nodes into their knowledge schema, refined the organization of
the concepts with CC, and extended their schema with new relationships between
the definers (Figure 4). For example, the definer difficulty at the beginning of the
course was not central to the NSNs, and its relationships were very general. This
definer was connected with concepts of a vague conceptual spectrum for the
course (e.g., learning, children, school), and it was connected with only a handful of
the distinctive features of learning difficulties (e.g., language, numbers, words,
reading). At the end of the course, difficulty acquired greater richness in terms of
its conceptual definition and conceptual relationships with other definers (Figure
5).
In general, transformations in the participants’ knowledge schema in the three

levels analyzed –content, structure, and organization – were apparent. So, in this
work, the evidence supports the idea that constructive cognitive assessment
effectively diagnoses cognitive changes due to learning. The findings in this
research have implications at the theoretical level since there is currently little
information available about the cognitive mechanism underlying the
development of cognitive structures in academic environments. At a
methodological level, empirical evidence about the effectiveness of NSNs as a
cognitive approach to evaluate academic learning has been presented in this
study. Furthermore, at the level of application, this study provides an alternative
tool and cognitive indicators for learning that are useful for formative assessment
or assessment for learning.
However, since the sample in this study was very small, and there was not a
control in the instructional sequence, future research should include comparison
and control groups, to explore the effects of teaching strategies. Additionally, it
would be very interesting to contrast the quality of organization, structure, and
content of knowledge schema based on the academic levels of students. Many
other variables and manipulations could be introduced to enhance our
comprehension of declarative learning and the formation of cognitive knowledge
structures in an academic environment (e.g., induvial characteristics, type of
institution, nature of content).
7. Conclusion
In sum, this study has offered empirical evidence of the cognitive changes that
occur in cognitive structures of knowledge due to the academic learning process.
The results indicated that constructive cognitive assessment is helpful in terms of
measuring cognitive expressions of learning. For example, in this study, it was
possible to identify the changes in the configuration of cognitive schemas for
knowledge, changes in the conceptual content, and modifications in the
relationships among the conceptual nodes that students had assimilated in their
memories due to learning. Thus, cognitive assessment tools such as NSNs are an
effective means for diagnosing, monitoring, and evaluating the learning process.
Furthermore, the data obtained with this type of tool can be used to improve or
generate new teaching strategies adapted to the cognitive characteristics of
students.
259
Mapping the state of students’ knowledge gives valuable information about the
difficulties that students have in learning material whether due to theoretical
confusions, problems in semantic understanding, or difficulties with the structure
of information. This kind of cognitive diagnosis will allow teachers to design
learning materials and strategies that increase the clarity of the concepts presented
in classes. These actions will help students adequately discriminate the conceptual
categories presented and achieve significant clarity about their knowledge.
However, to implement this type of proposal, it is necessary to continue exploring
the benefits and limitations of using cognitive learning assessment tools and to
continue the search to establish cognitive measurement parameters that are useful
for improving teaching and learning processes.
8. References
Arieli-Attali, M. (2013, October 20−25). Formative assessment with cognition in mind: The
cognitively based assessment of, for and as learning [Paper presentation]. 39th Annual
Conference: Educational Assessment 2.0: Technology in Educational Assessment,
Tel Aviv, Israel.
Bastian, M., Heymann, S., & Jacomy, M. (2009, May 17−20). Gephi: an open-source software
for exploring and manipulating networks [Paper presentation]. Association for the
Advancement of Artificial Intelligence, Third International AAAI Conference on
Weblogs and Social Media, San Jose California, United States.
El-Yassin, H. D. (2015). Integrated assessment in medical education. Journal of
Contemporary Medical Sciences, 1(4), 36–38.
http://www.jocms.org/index.php/jcms/article/view/51
Figueroa, J. G., González, E. G., & Solís, V. M. (1976). An approach to the problem of
meaning: Semantic networks. Journal of Psycholinguistic Research, 5(2), 107−115.
https://doi.org/10.1007/BF01067252
Figueroa-Nazuno, J. G. (2007, October 24−25). El significado de las Redes Semánticas
Naturales: Y la tradición oral, 20 años después [The meaning of Natural Semantic
Networks: And oral tradition, 20 years later] [Paper presentation]. Primer
Simposium Internacional: Cognición y Representación del Conocimiento.
Monterrey, Nuevo León, México.
Lopez, E. O. (1996). Schematically Related Word Recognition (Publication No. 9613356)
[Doctoral dissertation, University of Wisconsin-Madison]. ProQuest Dissertations
& Theses Global.
Lopez-Ramirez, E. O., & Morales-Martinez, G. E. (2019). Learning´s constructive-
chronometric cognitive assessment model [Unpublished manuscript]. Institute of
Research on the University and Education, National Autonomous University of
Mexico.
Lopez, E. O., & Theios, J. (1992). Semantic analyzer of schemata organization (SASO).
Behavior Research Methods, Instruments, & Computers, 24(2), 277−285.
https://doi.org/10.3758/BF03203508
Lopez, E. O., Morales, G. E., Hedlefs, I., & Gonzalez, C. J. (2014). New empirical directions
to evaluate online learning. International Journal of Advances in Psychology, 3(2),
40−47. https://doi.org/10.14355&ijap.2014.0302.03
Marzano, R. J., & Pickering, D. J. (1997). Dimensions of learning: Teacher's manual (2nd ed.).
ASCD.
Messick, S. (1984). The psychology of educational measurement. Journal of Educational
Measurement, 21(3), 215–237. https://doi.org/10.1111/j.1745-3984.1984.tb01030.x
Morales-Martinez, G. E. (2015). Protocolo para la recolección de conceptos objetivo y definidores
centrales y diferidos (PRECODECD): Un sistema de codificación de conceptos extraídos
de las redes semánticas naturales [Protocol for the collection of objective concepts
260
and central and deferred definers (PRECODECD): A coding system for concepts
extracted from natural semantic networks] [Unpublished manuscript]. Institute of
Research on the University and Education, National Autonomous University of
Mexico.
Morales-Martinez, G. E. (2020). Sistema de evaluación cognitiva constructiva cronométrica del
aprendizaje en línea y presencial [Online and face-to-face learning´s constructive-
chronometric cognitive assessment system] [Manuscript submitted for
publication]. Institute of Research on the University and Education, National
Autonomous University of Mexico.
Morales-Martinez, G. E., Ángeles-Castellanos, A. M., Ibarra-Ramírez, V. H., & Mancera-
Rangel, M. I. (2020). Cognitive e-tools for diagnosing the state of medical
knowledge in students enrolled for a second time in an anatomy course.
International Journal of Learning, Teaching and Educational Research, 19(9), 341−362.
Morales-Martinez, G., & Lopez-Ramirez, E. (2016). Cognitive responsive e-assessment of
constructive e-learning. Journal of e-Learning and Knowledge Society (Je-LKS), 12(4),
39−49. http://www.je-lks.org/ojs/index.php/Je-LKS_EN/article/view/1187
Morales-Martinez, G. E., & Lopez-Ramirez, E. O. (2018a). EVCOG Modulo 1. Configurador
RSN [EVCOG Module 1. NSN Configurator] (Version 1.1.) [Computer software].
Registro Público del Derecho de Autor. 03-2018-111311554300-01
Morales-Martinez, G. E., & Lopez-Ramirez, E. O. (2018b). EVCOG Modulo 2. Capturador
RSN [EVCOG Module 2. NSN Data storage] (Version 1.1.) [Computer software].
Registro Público del Derecho de Autor. 03-2018-1113115235001
Morales-Martinez, G. E., & Lopez-Ramirez, E. O. (2018c). EVCOG Modulo 3. Editor RSN
[EVCOG Module 3. NSN Editor] (Version 1.1.) [Computer software]. Registro
Público del Derecho de Autor. 03-2018-1113115235001
Morales-Martinez, G. E., & Lopez-Ramirez, E. O. (2018d). EVCOG Modulo 4. Análisis RSN
[EVCOG Module 4. NSN Analysis] (Version 1.1.) [Computer software]. Registro
Público del Derecho de Autor. 03-2018-1113115235001
Morales-Martínez, G. E., López-Ramírez, E. O., Castro-Campos, C., Villarreal-Treviño, M.
G., & Gonzales-Trujillo, C. J. (2017). Cognitive analysis of meaning and acquired
mental representations as an alternative measurement method technique to
innovate e-assessment. European Journal of Educational Research, 6(4), 455-464.
https://www.eu-jer.com/EU-JER_6_4_455_Morales-Martinez_etal.pdf
Morales-Martinez, G. E., Lopez-Ramirez, E. O. & Lopez-Gonzalez, A. E. (2015). New
approaches to e-cognitive assessment of e-learning. International Journal for e-
Learning Security (IjeLS), 5(2), 449−453.
https://doi.org/10.20533/ijels.2046.4568.2015.0057
Morales-Martinez, G. E., Mezquita-Hoyos, Y. N., Gonzalez-Trujillo, C. J., Lopez-Ramirez,
E. O., & Garcia-Duran, P. J. (2018). Formative e-assessment of schema acquisition
in the human lexicon as a tool in adaptive online instruction. In R. Lopez-Ruiz
(Ed.), From natural to artificial intelligence: Algorithms and application (pp. 69−88).
IntechOpen. http://doi.org/10.5772/intechopen.81623
Morales-Martínez, G. E., López-Pérez, R. M., García-Collantes, A., & López-Ramírez, E. O.
(2020). Evaluación constructiva cronométrica para evaluar el aprendizaje en línea
y presencial [Constructive-chronometric evaluation as a tool to evaluate online
and face-to-face learning]. Tecnología, Ciencia y Educación, 15(1), 105−124.
https://www.tecnologia-ciencia
educacion.com/index.php/TCE/article/view/371
Morales-Martinez, G. E., Trejo-Quintana, J., Charles-Cavazos, D. J., Mezquita-Hoyos, Y.
N., & Sanchez-Monroy, M. (2021). Chronometric constructive cognitive learning
evaluation model: Measuring the construction of the human cognition schema of
261
psychology students. International Journal of Learning, Teaching and Educational

Research, 20(2), 1-21. https://doi.org/10.26803/ijlter.20.2.1
Sadeghi, K., & Rahmati, T. (2017). Integrating assessment as, for, and of learning in a large-
scale exam preparation course. Assessing Writing, 34, 50-61.
https://doi.org/10.1016/j.asw.2017.09.003
United Nations Educational Scientific and Cultural Organization (2015). Rethinking
education: Towards a global common good?
https://unesdoc.unesco.org/ark:/48223/pf0000232555
Urdiales-Ibarra, M. E., Lopez-Ramirez, E. O., Castro-Campos, C., Villarreal-Treviño, M.
G., & Carrillo-Colon, J. E. (2018). Biology schemata knowledge organization and
meaning formation due to learning: A constructive-chronometric approach to
concept mapping usability. Creative Education, 9(16), 2992−2706.
https://doi.org/10.4236/ce.2018.916203
Wiliam, D. (2011). What is assessment for learning?. Studies in Educational Evaluation, 37(1),
3–14. https://doi.org/doi:10.1016/j.stueduc.2011.03.001
262

Vol. 20, No. 8, pp. 262-277, August 2021
The Effect of Resource-Based Instructions on

Pre-service Biology Teachers' Attitudes towards
Learning Biology
Josiane Mukagihana
African Centre of Excellence for Innovative Teaching and Learning Mathematics
and Science (ACEITLMS),
University of Rwanda-College of Education (URCE), Rwanda
https://orcid.org/0000-0001-7334-331X
Catherine M. Aurah
Masinde Muliro University, of Science and Technology, Kenya
https://orcid.org/0000-0002-0781-3202
Florien Nsanganwimana
African Centre of Excellence for Innovative Teaching and Learning Mathematics
and Science (ACEITLMS),
University of Rwanda-College of Education (URCE), Rwanda
https://orcid.org/0000-0002-3152-9893
Abstract. A positive attitude correlates to successful learning, while a

negative attitude leads to poor learning. The present study aimed to
compare the pre-service biology teachers' attitudes towards learning
biology and assessing their attitude change before and after being taught
by resource-based instructions (RBI) in Rwanda. A survey research
design was used, in combination with a quasi-experimental control group
and the equivalent time-series designs. Pre-service biology teachers were
taught at a public university in three separate groups, with one control
group and two experimental groups. Each group received a pre-attitude
and a post-attitude assessment. In contrast, at a private university, they
were taught as a single group, in a series of instructions, starting with the
lecture method, followed by animation-based instruction and small-
group laboratory activities, in which at each stage, the pre-service biology
teachers received the same assessment. The biology Attitude Scale
showed a Cronbach alpha reliability of 0.625 before its use. The data were
analyzed quantitatively, and the results revealed no statistically
significant difference between the public and the private pre-service
teachers' attitudes before learning microbiology. Basically, the results
show that pre-service biology teachers at the private University had much
more positive attitudes than those in a public university. Contrary to the
private university, pre-service teachers’ attitudes towards biology
©Authors
This work is licensed under a Creative Common Attribution Non-Commercial No-Derivatives 4.0
263
changed after instruction in all groups across all the attitudinal statements
at the public university. Nevertheless, all methods used during the
intervention, namely the lecture methods, animation-based instructions,
and small-group laboratory activities, equally improved pre-service
biology teachers’ attitudes towards learning biology. The lack of
statistically significant difference among methods suggests that the
preservice teachers’ attitudes towards learning biology could also be
influenced by other factors including their maturity level as well as their
anticipated career commitment.
Keywords: attitude scale; biology; pre-service biology teachers; resource-

based instructions
1. Introduction
Students' attitude towards learning is an essential cognitive behavior that
contributes more to their learning outcomes, such as academic achievement,
engagement with subject learning, and concept understanding (Prokop, Tuncer,
et al., 2007). Hussein (2017) proved that attitude significantly improves students'
intention to use instructional tools, like technology-related instructions, such as e-
learning. Students' attitude determines their behavioral differences towards a
given subject or concepts (Haddock, 2017); therefore, understanding students'
attitudinal changes after teaching and learning is essential for improving the
quality of education. Studies done in Rwanda have shown that secondary-school
students feel bored with learning physics; since they think that it does not relate
to any real-world experiences (Mbonyiryivuze et al., 2021); moreover, some
studies found that students lose interest in learning mathematics due to the
instructional tools used by their teachers (Ukobizaba et al., 2020).
Although attitude has been defined by many studies differently, all are of the
same opinion that attitude is the expression of the inner feelings of an individual
that may be perceived as a behavior of liking or disliking a particular thing
(Shrigley et al., 1988, Lovelace & Brickman, 2013; Musengimana et al., 2021;
Prokop, Prokop, et al., 2007; Prokop, Tuncer, et al., 2007; Tikka et al., 2000).
Attitude is an individual mental state that characterizes his or her experiences of
consistently responding to environmental phenomena. In its dimensions, attitude
is conceptualized as being cognitive, which is the belief in an object as being
affective, which reflects the favorability or the availability of an object and the
behavioral intentions and the expression of intention towards the object (Kiecolt,
1988).
The word attitude may be used interchangeably to refer to one's belief, opinion,
or value. Furthermore, an individual's attitude can be perceived in two opposite
behaviors: positive and negative, and these can be influenced by different factors
(Shrigley et al., 1988). Several factors, such as the instructional methods
(Mukagihana et al., 2020, 2021a), gender, students' interest, and classroom
environment, were pointed out to affect students' attitudes towards scientific
subjects, including chemistry (Musengimana et al., 2021).
264
Various instructional methods have been used to test their influence on students'
attitude towards scientific subjects, and among others, co-operative learning
methods, computer-assisted instruction, problem-based active learning, media-
based instruction, simulation-assisted laboratory instruction, and instructional
technology and material-design course, which positively affected students,
including pre-service science teachers' attitude towards science learning
(Akinoǧlu & Tandoǧan, 2007; Çepni et al., 2006; McManus et al., 2003; Rabgay,
2018, Calik-Uzun, Selcen, et al., 2019, Selçuk, 2010a; Ulukök & Sari, 2016, Bektaşli,
2013). Many of these instructions are applied in the resource-supported learning
environment, and they are referred to as resource-based instructions, such as
computer-assisted instructions, simulation-assisted laboratory instruction,
instructional technology, and material-design courses. With resource-based
instructions, the teaching and learning process uses a single instruction or a
variety of instructional resources; and students learn by using these resources
(Butler, 2012). Resource-based instructions promote students' active participation
by learning in real and tangible learning environments, allow them to interact
with instructional resources through hands-on activities, and discovering
knowledge and facts themselves. Hence, they enhance students' learning
outcomes and their attitude changes towards learning science (Hill & Hannafin,
2001).
A good number of studies focused on the students' attitude towards learning

science in general (Rabgay, 2018, Calik-Uzun, Selcen, et al., 2019, Selçuk, 2010a;
Ulukök & Sari, 2016, Bektaşli, 2013); but a few of these studies emphasized the
pre-service biology teachers' attitude towards learning biology. Furthermore, only
a few of the current studies focused on the effect of instructional methods on pre-
service biology teachers' attitude towards learning biology; however, the effect of
resource-based instructions, especially animation-based instruction and
laboratory methods through small-group laboratory activities on pre-service
biology teachers' attitudes towards learning biology, was not highlighted. With
such observations made in the current literature, the present study aimed to find
out the pre-service biology teachers'attitudes towards learning biology in higher
learning institutions in Rwanda.
It also aimed to find out the effect of resource-based instructions on pre-service

biology teachers' attitudes towards learning biology in higher learning
institutions in Rwanda, with three main objectives: (1) to compare the pre-service
biology teachers' attitude towards learning biology at private and public science
teacher-training universities; (2) to assess the change in pre-service biology
teachers' attitudes towards learning biology after learning via animations and
small-group laboratory activities, as resource-based instructions; and (3) to test
the statistically significant effect in the pre-service biology teachers' attitudes after
being taught by animation-based instruction and small-group laboratory
activities at private and public universities in Rwanda.
The study was guided by the following research questions :

1) Do the pre-service biology teachers' attitudes towards learning biology differ
between private and public universities in Rwanda?
265
2) How did the pre-service biology teachers' attitudes change after learning by
animation-based instruction and small-group laboratory activities at private and
public universities in Rwanda?
3) Is there any statistically significant effect of animation-based instruction and
small-group laboratory activities as resource-based instructions on pre-service
biology teachers' attitudes towards learning biology at private and public
universities?
The significance of the study

The present study contributes significantly to the current literature by enriching
the information related to the status of pre-service biology teachers' attitudes
towards learning biology in higher learning institutions. It also emphasizes the
role of resource-based instructions in improving pre-service biology teachers'
attitudes towards learning biology. Further information would give pre-service
biology teacher trainers some insight into the attitude characterizing pre-service
biology teachers, thereby enabling them to decide which instructional method to
apply when there is a need to improve their attitude towards learning biology.
Furthermore, higher learning institutions, which train pre-service biology
teachers would be better informed on the attitudes of pre-service biology teachers
towards learning biology.
The theoretical framework

The study is grounded on reasoned action and planned behavior theories, as
described by Ajzen Icek (1985). These theories assume that human behavior is
sensibly based on the available information that might stimulate their feelings of
liking or disliking. The theories postulate that humans react on the basis of their
behavioral intention to perform, or not to perform, and then to act. The theories
add that individual intention may change in time; and that they are determined
by personal factors, which are the positive or negative decisions to perform a
particular behavior. These factors define a person's attitude towards behavior that
can be predicted from the knowledge of his or her attitude towards the object.
These theories are related to this study that aims to evaluate the attitudinal
changes of students towards learning biology. These attitudinal changes may be
perceived as students' changes in their feeling of like or dislike of learning biology
after being treated with different resource-based instructions that, in this case, act
as stimuli in the study.
2. The Methodology
This study used a survey design, in combination with an experimental design,
whereby quasi-experimental and equivalent time series were used to check the
pre-service biology teachers' attitude change after being taught by different
instructional methods. The Survey design is one of the procedures in quantitative
research, in which researchers may measure individuals' different aspects, such
as behavior, attitudes, and opinions (Creswell, 2015). The survey was conducted
before and after interventions given to groups of pre-service biology teachers. The
design was used to permit the researchers to observe the changes in groups of pre-
service biology teachers after receiving different treatments.
266
2.1 Data-collection instrument and reliability

A biology attitude scale developed by Russell and Hollander (1975) was adapted
in this study in order to measure the pre-service biology teachers' changes in
attitude towards learning biology. The scale was composed of fourteen (14) items
or statements that were intended to measure the same behavior at a correlation of
0.80 to each item. The scale was rated from strongly agree (1) to strongly disagree
(5); and it contained both negative and positive items. In this study, all items were
administered to pre-service biology teachers; however, six (6) negative items were
removed during the analysis after recognizing that pre-service biology teachers
responded in the right way by positively agreeing to a positive statement and by
negatively reacting to negative statements. Consequently, we recognized them as
being opposite to positive items. For instance, items 2 and 3 are opposite to item
1 (see Table 1); item 6 is contrary to item 5, which states that "Biology makes me
feel secure; while at the same time, it is stimulating." Items 8 and 9 are contrary to
item 7: "In general, I have a good feeling towards biology." Item 12 is contrary to
items 11 and 12 (see Table 1). Therefore, a total of 6 negative items were removed
from the analysis; and only eight positive items were considered (items 1, 4, 5, 7,
10, 11, 13, 14). Before using the scale, its reliability was checked in the Rwandan
context; and it was found to be 0.625 Cronbach alpha by the SPSS 23.0 version.
This positive correlation confirmed the removal of negative-constructed items;
and when considering all 14 items, the reliabilities tended to cancel each other out;
and they provided a coefficient of nearly 0 in the negative direction.
Table 1: Biology attitude scale (Russell & Hollander, 1975) [Note: 1 = strongly agree; 2
= Agree; 3 = Undecided; 4 = Disagree; and 5 = strongly disagree]
Statements 1 2 3 4 5
1. Biology is very interesting to me.
2. I don't like biology, and it scares me to have to take it.
3. I am always under a terrible strain in a biology class.
4. Biology is fascinating and fun.
5. Biology makes me feel secure, and at the same time, it is
stimulating.
6. Biology makes me feel uncomfortable, restless, irritable, and
impatient.
7. In general, I have a good feeling towards biology.
8. When I hear the word biology, I have a feeling of dislike.
9. I approach biology with a feeling of hesitation.
10. I really like biology.
11. I have always enjoyed studying biology in school.
12. It makes me nervous to even think about doing a biology
course.
13. I feel at ease in biology; and I like it very much.
14. I feel a definite positive reaction towards biology; it is
enjoyable.
267
2.2 Participants and data-collection procedures

This survey study was conducted at two intentionally selected Universities that
train pre-service biology teachers in Rwanda; a private university, (University of
Technology and Arts of Byumba (UTAB), and the University of Rwanda College
of Education (URCE), a solely public university that trains pre-service teachers.
At these Universities, the study was conducted after an ethical check-up, and
approval from the research unit at the University of Rwanda College of education,
and with both Universities' permission. Pre-service biology was introduced to the
study; and they signed the consent form voluntarily. The data were collected from
February to April 2021.
The participants consisted of 50 pre-service biology teachers enrolled in Year

(Level) two in the Faculty of Education, Department of Education in Sciences at
UTAB. They formed a single group that received a pre-assessment survey; and
after the group had been taught a course of biology (microbiology) by traditional
methods (Lecture method), by animation-based instruction, and finally by
laboratory methods through small-group laboratory activities. The teaching and
learning time interval was three weeks, and after each session, they were post-
surveyed by administering a Biology-Attitude Scale similar to that used in the pre-
assessment. Thirty-four pre-service biology teachers responded positively to all
the surveys, including pre-assessment, and their data were taken for analysis. At
URCE, one hundred and eighty pre-service biology teachers enrolled in biology
and chemistry education (BCE) Year-2; participated in the study. They were taken
from a population of 528 pre-service biology teachers enrolled in the module of
microbiology that was purposely selected for teaching by different resource-based
instructions. They were randomly assigned to three groups, and each group was
pre-assessed by a biology-attitude scale; and thereafter, it was treated at a
particular time by the specific instructional method.
The first group was taught by the traditional method (lecture method), the second
by animation-based instruction, while the third group was treated by laboratory
methods through small-group laboratory activities. After those teaching sessions,
each group responded to the similar post-Biology Attitude Scale.
2.3 The Data analysis

The data were entered into MS Excel 2016 and then filtered in order to track
participants' full attendance throughout the intervention. The "COUNT IF"
function was used to compute the number of answers, with either strongly agree,
agree, undecided, disagree, or strongly disagree with each statement (for more
details on how the data analysis was done, see our data deposited in Mendeley
repository: "https://data.mendeley.com/datasets/bpfb5sdmyg/1"). The formula
matches the answer provided by the respondent for each statement. Then, it
counted all those who replied "1" as strongly agree, "2" as agree, "3" undecided,
"4" Disagree, and "5" strongly disagree to each statement. After that, the software
summed up the total agreements and all the disagreements to each statement. For
the data collected from UR-CE, the analysis was done to both the pre-assessment
and the post-assessment data of each of the three groups. The generated pre-
268
assessment results of the groups (control group, animation group, and lab group)
and their post-assessments results were presented on a plotted cluster-column
graph, and they were then compared.
The same analysis was done to the data from UTAB, and the pre-assessment
results, the results of the pre-service biology teachers after being taught by the
lecture method, animation-based instruction, and laboratory methods through
small-group laboratory activities, were compared by presenting them on a plotted
cluster-column graph. To compare the private and the public pre-service biology
teachers' attitudes towards learning biology and with the interventions given, we
computed the averages and formed three scales by combining scale-1 and scale-2,
as well as scale-3 alone, and then scale-4 and scale-5 together. We then considered
the agreement rates among the pre-service teachers and the agreement rates
across the interventions. We used the Chi-square test to compute the significance
between the pre-service teachers and the intervention groups. This test was
appropriate, as the actual count of the individual respondents was in place,
instead of their scores.
3. The Results
Figure 1 compares the pre-service biology teachers' attitudes towards learning
biology, before implementing the resource-based instructions to pre-service
teachers at public and private universities. The quick insight is that the pre-service
teachers hold positive attitudes towards biology. However, those at a private
university (UTAB) hold considerably more attitude than those in a public
university (URCE) (see Figure 1, blue and yellow colour of the cluster-column
graph). Except on statement-7 and statement-14, public universities hold more
attitudes than those from private universities. In general, pre-service teachers at
URCE (81%) have a better feeling towards biology than do those at UTAB (79%)
[see Statement-7]. Likewise, they (81% of URCE pre-service biology teachers) "feel
a definite positive reaction to biology; since it is enjoyable" in contrast to those at
UTAB (74% of UTAB pre-service teachers) [see Statement-10].
269
100 97 94 94
88
90 85
81 79 80
84 82 85 81
80 74
68
70 63
% of respondents
61
60
50
40
30
20
10
0
STAT 1 STAT 4 STAT 5 STAT 7 STAT 10 STAT 11 STAT 13 STAT 14
UR-CE (Agreement) UR-CE (Neutral)

UR-CE (Disagreement) UTAB (Agreement)
UTAB (Neutral) UTAB (Disagreement)
Figure 1: Attitude of Pre-service Biology teachers at a public and private university

towards learning Microbiology. [Note: Cluster columns show the distribution of pre-
service biology teachers into 3 groups: an agreement, neutral, and disagreement scales.
The vertical axis displays their percentage rates, while the horizontal axis displays their
attitude statements.]
Biology is very interesting (97%, see Statement-1), fascinating and fun (68%, see
Statement-4), and this makes the UTAB pre-service teachers feel secure, while at
the same time stimulated (88%, see Statement-5) more than URCE pre-service
biology teachers. Similarly, UTAB pre-service teachers "really like biology" (94%,
see Statement-10), and they enjoy studying biology at school (94%, see Statement-
11), and they feel at ease in biology and like it very much (85%, see Statement-13)
more than do the URCE pre-service teachers. Despite these variations in attitude,
however, the Chi-square test shows no statistically significant difference between
URCE and UTAB pre-service teachers, as regards their attitudinal statements (X2
= 48, df = 47 (1), p=0.243).
Figure 2 shows the attitude results from URCE pre-service teachers before and
after learning biology. It is worth mentioning that we only considered the positive
statements in our analysis. Thus, eight out of 14 attitude statements are under
consideration (see the detailed explanation in the methodology section). The
URCE pre-service teachers' attitudes towards biology were computed in the three
groups of intervention of resource-based instruction (RBI) after being taught in
those modes. Figure 2 displays two sides. The side on the left is before the
assessment, while the one on the right side is after the intervention. The first group
is the control group or the lecture group, the second is the animation group, and
the third group is the lab-based group.
270
The figure is a cluster-staked graph. Thus, it accumulates both agreement, neutral,

and disagreement scales at the 100% rate. For instance, 59 pre-service teachers
participated via the control group in the pre-assessment stage; 40 were in
agreement, with one being neutral; while 18 fell into the disagreement category.
Thus, 68% agreed, 2% were neutral, and 30% disagreed with the first statement
[Biology is very interesting to me], and these taken together equate to 100% (see
Figure 2).
Pre assessment Post assessment
STAT 13
LAB GROUP
STAT 10
STAT 5
STAT 1
ANIMATION
STAT 13
GROUP
STAT 10
STAT 5
STAT 1
CONTROL GROUP
STAT13
STAT10
STAT5
STAT 1
0% 20% 40% 60% 80% 100%
Pre (Agreement) Pre (Neutral) Pre (Disagreement)

Post (Agreement) Post (Neutral) Post (Disagreement)
Figure 2: The Attitude of Pre-service Biology teachers at a public university before and
after learning Microbiology. [Note: N: 59 for the control group, N = 60 for the animation
group, and N = 52 for the lab group]
The attitude change did not significantly differ between the teaching interventions
(see Table 2). However, the pre-service teachers' attitudes towards biology
changed after instruction in all the groups across all the attitude statements,
irrespective of the use of the animation, laboratory, and lecture instructions. For
instance, statement-4 [Biology is fascinating and fun] shifted from 56%, who had
a positive attitude and reached 83% after being taught in the traditional mode. On
the same statement, 67% of the pre-assessment teachers had a positive attitude, or
they agreed with the statement; and this increased to 73% after learning through
animation-based instructions.
Similarly, the lab-based activities increased the positive attitudes towards biology
from 60% to 79% of pre-service biology teachers (see Figure 2).
271
Table 2: Statistical significance between intervention groups at URCE.

Pre-test Post-test X2 df p-value
1 Control group
Control group 34 30 0.281
Animation group 42 36 0.227
Laboratory group 28 24 0.260
2 Animation group Control group 34 30 0.281
3 Laboratory group Control group 35 25 0.094
Note: X2 is the Chi-square value, df is the degree of freedom, a p-value is computed, based
on a 0.05 level of significance or on a 95% confidence interval .
Figure 3 shows the results of the UTAB pre-service teachers. In this case, a time-
series design was used. Thus, attitudinal change was measured quince across four
stages of one group of pre-service teachers. The attitude does not seem to change
from pre-assessment to another stage of intervention, such as lecture, animation,
or lab-based instruction. Statement-4 and statement-5 seem to be persistent across
all the interventions. The understanding of the statements might be the reason for
this.
STAT 14
assessement)
Lab (Post-
STAT 11
STAT 7
STAT 4
assessement)
Animation
STAT 13
(Post-
STAT 10
STAT 5
STAT 1
STAT 14
Lecture (Post-
Pre-assessment assessement)
STAT 11
STAT 7
STAT 4
STAT 13
STAT 10
STAT 5
STAT 1
0% 20% 40% 60% 80% 100%
Agreement Neutral Disagreement
Figure 3: The Attitude of Pre-service Biology teachers at a private university before and
after learning Microbiology. [Note: N = 34 both at re-assessment and continuous
intervention]
272
Table 3 shows how the interventions made no statistically significant differences

in the attitude between the pre-assessment and the pre-service biology teachers at
UTAB. Neither did their attitude change from pre-assessment to lecture,
animation, laboratory instruction, lecture instruction to animation, or to
laboratory instruction. It changed, however, from animation to laboratory
instruction at a p<0.05 (see Table 3).
Table 3: Statistically significant different effects between the series of interventions at

UTAB.
Prior series Next series X2 df p-value
1 Pre-assessment Lecture instruction 26 24 0.353
Animation 27 24 0.320
instruction
Laboratory 35 30 0.255
instruction
2 Lecture instruction Animation 17 16 0.393
instruction
Laboratory 22 20 0.329
instruction
3 Animation instruction Laboratory 32 20 0.043
instruction
Note: X2 is the Chi-square value, df is the degree of freedom, the p-value is computed,
based on a 0.05 level of significance, or on a 97% confidence interval.
4. Discussion
With research question one, the study sought to identify the pre-service biology
teachers' attitudes towards learning biology at a private University (UTAB) and
at a public University (UR-CE) before any interventions. Although they both
presented a positive attitude, and that a statistically non-significant difference in
attitude towards the statements was revealed, pre-service biology teachers at the
private University (UTAB) show considerably more attitude than do those in a
public university (URCE).
The difference in pre-service biology teachers' attitudes at private and public

universities may be explained by many facts. From these results, one can consider
how private and public university pre-service biology teachers are enrolled in
biology subjects. Pre-service biology teachers select biology subjects within their
wide choice at the private University; whereas at a public teacher-training
university, students are enrolled in biology programs, based on the combination
they learned at secondary schools. Those at a Public University may be enrolled
in Mathematics, Biology Education, or Biology Chemistry education; while on the
basis of choice, they could decide to learn subjects other than biology, such as
Mathematics and Chemistry.If this happens, the pre-service biology teachers
would follow the program with much effort in one subject that is interesting to
him or her, but with a low attitude or negative attitude towards biology, where
the subject was just presented without any individual choice. Related studies that
compared students' attitudes at private and public schools or universities are still
very rare; however, looking into other domains, such as the motivation towards
learning biology, the study of Bernardo et al. (2015) advocated a difference in
273
motivational level towards learning science between public and private-school

students. Related studies that compared students' attitudes at private and public
schools or universities are still very rare.
Apart from the statistical results, in reality, the pre-service biology teachers at the
private university presented a higher positive attitude towards learning biology
than did their colleagues at a public university. This implies that they might also
perform differently in biology. The recommendation to the Government is to
ensure that all students are oriented towards Biology education at the public
University (UR-CE); and that they hold a positive attitude towards learning
biology. Additionally, the orientation on the basis of the student's subject choice
should be more emphasized.
Although the pre-service, biology teachers at public universities presented a low

level of attitude; they showed a good feeling and a more positive reaction towards
biology than did those in private universities. Furthermore, they perceived
biology as an enjoyable science subject. This may be explained by the fact that they
learned in different physical and classroom environments, where public pre-
service biology teachers learn in a more diverse environment, with different
biological stimuli than those in a private university.
A long time ago, it was noted that the classroom environment presented more
relation with the student's attitude towards a science that includes biology (Talton
& Simpson, 1987). The findings relate with those of other studies, such as those of
Nordqvist and Johansson (2020), Prokop, Prokop, et al. (2007), Prokop, Tuncer, et
al. (2007), and Sanders and Morrison-Shetlar (2001). They noted that, in general,
students have a positive attitude towards biology. Indeed, the present study's
findings did not show any change of attitude, despite the instructions used. The
theory of reasoned action and planned behaviour (Ajzen, 1985), which directed
this study confirms the obtained results; as it concludes the final students'
agreement or disagreement in terms of having a positive or negative attitude
towards learning biology as a subject.
We identified how the pre-service biology teachers' attitude changes after being
taught by different instructional methods, at both private (UTAB) and public (UR-
CE) universities. The data form the public university did not show any data that
revealed any statistically significant difference in attitudinal change between pre-
service biology teachers – taught by the traditional (lecture) method, animation–
based instruction, and taught by laboratory methods via small-group laboratory
activities. However, instructions improved the pre-service biology teachers'
attitudes differently across all the attitudinal statements.
The fact that both the instructional methods improved the pre-service biology
teachers' attitude to the positively considered statements may be explained by the
exciting qualities of microbiological content , used during the intervention. This
study agrees with our previously published results, where we found that all the
resource-based interventions improved pre-service biology teachers' academic
achievement in Rwanda (Mukagihana et al., 2021b). Consequently, attitude may
correlate with the performance of the students. The findings deviate from what
274
other studies identified, where traditional methods of teaching (lecture) did not
improve the students' attitude towards learning biology, such as did active
instructions, like computer-aided instruction, co-operative learning methods
(Rabgay, 2018; Yesilyurt et al., 2019).
Although the three groups were taught by using different instructional methods
at the public University (UR-CE), pre-service biology teachers learned the same
content. Consequently, in each instruction, new knowledge was gained; and this
added to their pre-existing attitude, thus changing the mind or beliefs of one who
responded negatively to the statement before teaching, to agree with the
statements positively. This makes pre-service biology teachers present a positive
attitude on some statements after receiving interventions. The implication is that
both lecture methods and animation-based instructions could improve pre-
service biology teachers' attitudes towards learning biology. However, animation-
based instructions and small-group laboratory activities may do more; since they
are resource-based instructions that create an active learning environment that
attracts students in the teaching and learning process.
The findings of the study are in line with those of other studies (Akinbadewa &
Sofowora, 2020; Akinoǧlu & Tandoǧan, 2007; Koksal & Berberoglu, 2014; Sen &
Oskay, 2016, (Akinbadewa & Sofowora, 2020; Akinoǧlu & Tandoǧan, 2007; Çepni
et al., 2006; Koksal & Berberoglu, 2014; Rabgay, 2018). These studies tested the
effect of instructional methods, like technology-based instructions and laboratory
methods on students' attitude towards biology, whereby such instructional
methods improved students' attitudes towards science, including biology.
In particular, our study showed that even the lecture method could improve the
pre-service biology teachers' attitudes. This is because pre-service teachers are
mature enough; and they have to study the subject; since they will probably be
teaching it after their program. Thus, the fact that they plan to teach that subject
in their future career, the attitude to learning improves, despite the instructional
method used to deliver it. The implication is that after learning, pre-service
biology teachers knew their attitude towards biology – whether it is very
interesting to them, or not. It was noted that students, when learning by the lecture
method, develop their listening capacity; and they can master and maintain
concepts (McManus et al., 2003); thus, it is easy for them to decide on their
opinion.
Conversely, the findings disagree with those of Koksal and Berberoglu (2014),
who noticed that there were no improvements in students' attitude after learning
via the traditional methods. Therefore animation-based instruction and small-
group laboratory activities are recommended for improving pre-service biology
teachers' attitudes towards learning biology.
At the private University (UTAB), the attitude change was measured in a single
group of pre-service biology teachers, who received a pre-assessment test by
responding to the biology-attitude scale, and a post-assessment test, after
receiving treatment through a series of instructions. Looking into the results, we
275
found that pre-service biology teachers maintain their positive attitude – both
before and after a series of interventions. There were no significant differences in
attitude change after a series of instructions, except animation to laboratory
instruction. However, each series of instructions changed their agreement on
statements.
Small-group laboratory activities statistically improved pre-service biology

teachers' attitude towards learning biology, thereby showing that those pre-
service biology teachers are eager to learn, understand the biological concepts,
and become familiar with laboratory experiments, which they are expected to
reflect during their teaching in their future career. Indeed, the very recent study
by Uwamahoro et al. (2021) conducted in Rwanda has shown that laboratory
activities improve students' understanding of concepts through appropriate
hands-on activities. Therefore, laboratory methods should be emphasized at
private universities to develop pre-service biology teachers' scientific skills and
improve their attitude towards learning biology, thereby improving their
performance.
5. Conclusion, limitations, and further research

The present study aimed to identify how the pre-service biology teachers' attitude
changed due to animation-based instruction and small group laboratory activities
as resource-based instructions at private and public universities in Rwanda. The
study compared their attitude towards learning biology before teaching sessions;
and it revealed that both groups of pre-service biology teachers present a positive
attitude towards learning biology. However, based on the differences in positive
agreement towards statements and in their feelings towards biology, pre-service
biology teachers at the private University (UTAB) were found to have much more
positive attitudes than those in a public university (URCE).
At a public university, pre-service teachers' attitudes towards biology changed

after instruction in all groups across all the attitudinal statements. However, in
private, the pre-service biology teachers' attitudes towards learning biology did
neither change from pre-assessment, nor after all the series of interventions. There
was no statistically significant difference in attitudinal change between the
treatment groups at both public and private universities. The attitudinal change
did not show any statistically significant differences between teaching
interventions at either public or private universities.
After teaching the interventions, the results revealed that both lecture methods,
animation-based instructions, and laboratory methods through small-group
laboratory activities equally improved pre-service biology teachers' attitudes
towards learning biology. Nevertheless, via time-series design, laboratory
activities showed a great attitudinal change. This unique finding reveals that pre-
service biology teachers are mature enough to study through any instructional
method. They develop a positive attitude, since they would also teach the same
subject in their future career.
276
Given that the present study used different designs during the interventions and
a non-equivalent quasi-experimental design and equivalent time series at a public
university (UR-CE) and at a private university (UTAB). we did not compare the
pre-service biology teachers' attitudinal changes after the interventions at both
universities. Therefore, further research should focus on the use of single and
similar designs at both Universities. Furthermore, the study did not consider the
gender difference in pre-service biology teachers' attitudes before and after the
interventions. This information may give insight to trainers about the status of
gender difference; consequently, they should be able to know which gender needs
more emphasis to improve their attitude towards biology. Future research is
required to highlight pre-service biology teachers' gender differences in attitude
towards biology and improve their attitude towards learning and teaching
biology. This is, therefore, recommended.
Acknowledgments
We would like to thank the Directorate of Research and Innovation at the
University of Rwanda College of education (URCE) and the University of
Technology and Arts of Byumba (UTAB) for granting ethical clearance and
permission to conduct this study. Also, we sincerely thank the African Center of
Excellence for Innovative Teaching and Learning Mathematics and Science
(ACEITLMS) for the financial support provided to successfully conduct this
study.
6. References
Akinbadewa, B. O., & Sofowora, O. A. (2020). The effectiveness of multimedia
instructional learning packages in enhancing secondary school students' attitudes
towards biology. International Journal on Studies in Education, 2(2), 119–133.
https://doi.org/10.46328/ijonse.19
Akinoǧlu, O., & Tandoǧan, R. Ö. (2007). The effects of problem-based active learning in
science education on students' academic achievement, attitude and concept
learning. Eurasia Journal of Mathematics, Science and Technology Education, 3(1), 71–
81. https://doi.org/10.12973/ejmste/75375
Bernardo, A. B. I., Ganotice, F. A., & King, R. B. (2015). Motivation gap and achievement
gap between public and private high schools in the Philippines. Asia-Pacific
Education Researcher, 24(4), 657–667. https://doi.org/10.1007/s40299-014-0213-2
Çepni, S., Taş, E., & Köse, S. (2006). The effects of computer-assisted material on students'
cognitive levels, misconceptions and attitudes towards science. Computers and
Education, 46(2), 192–205. https://doi.org/10.1016/j.compedu.2004.07.008
Hussein, Z. (2017). Leading to Intention: The role of attitude in relation to technology
acceptance model in e-Learning. Procedia Computer Science, 105(December 2016),
159–164. https://doi.org/10.1016/j.procs.2017.01.196
Koksal, E. A., & Berberoglu, G. (2014). The effect of guided-inquiry instruction on 6th
Grade Turkish students' achievement, science process skills, and attitudes toward
science. International Journal of Science Education, 36(1), 66–78.
https://doi.org/10.1080/09500693.2012.721942
Mbonyiryivuze, A., Yadav, L. L., & Amadalo, M. M. (2021). Students' attitudes towards
physics in Nine Years Basic Education in Rwanda. International Journal of
Evaluation and Research in Education (IJERE), 10(2), 648–659.
https://doi.org/10.11591/ijere.v10i2.21173
McManus, D. O., Dunn, R., & Denig, S. J. (2003). Effects of traditional lecture versus
277
teacher-constructed & student constructed self-teaching instructional resources

on short-term science achievement & attitudes. The American Biology Teacher, 65(2),
93–102. https://doi.org/10.2307/4451447
Nordqvist, O., & Johansson, S. (2020). Secondary school biology students' attitudes
towards modern biotechnology characterised using structural equation modeling.
Eurasia Journal of Mathematics, Science and Technology Education, 16(2).
Prokop, P., Prokop, M., & Tunnicliffe, S. D. (2007). Is biology boring? Student attitudes
toward biology. Journal of Biological Education, 42(1), 36–39.
https://doi.org/10.1080/00219266.2007.9656105
Prokop, P., Tuncer, G., & Chudá, J. (2007). Slovakian students' attitudes toward biology.
Eurasia Journal of Mathematics, Science and Technology Education, 3(4), 287–295.
https://doi.org/10.12973/ejmste/75409
Rabgay, T. (2018). The effect of using co-operative learning method on tenth grade
students' learning achievement and attitude towards biology. International Journal
of Instruction, 11(2), 265–280. https://doi.org/10.12973/iji.2018.11218a
Russell, J., & Hollander, S. (1975). A biology attitude scale. American Biology Teacher, 37(5),
270–273. https://doi.org/10.2307/4445229
Sanders, D. W., & Morrison-Shetlar, A. I. (2001). Student attitudes toward web-enhanced
instruction in an introductory biology course. Journal of Research on Computing in
Education, 33(3), 251–262. https://doi.org/10.1080/08886504.2001.10782313
Sen, S., & Oskay, O. O. (2016). The effects of 5E inquiry learning activities on achievement
and attitude toward chemistry. Journal of Education and Learning, 6(1), 1.
https://doi.org/10.5539/jel.v6n1p1
Talton, E. L., & Simpson, R. D. (1987). Relationships of attitude towards classroom
environment with attitude towards and achievement in science among tenth
grade biology students. Journal of Research in Science Teaching, 24(6), 507–525.
https://doi.org/10.1002/tea.3660240602
Ukobizaba, F., Ndihokubwayo, K., & Uworwabayeho, A. (2020). Teachers' behaviours
towards vital interactions that attract students' interest to learn mathematics and
career development. African Journal of Educational Studies in Mathematics and
Sciences, 16(1), 85–94.
Uwamahoro, J., Ndihokubwayo, K., Ralph, M., & Ndayambaje, I. (2021). Physics students'
conceptual understanding of geometric optics: Revisited analysis. Journal of
Science Education and Technology, 30(0123456789), 1–13.
https://doi.org/10.1007/s10956-021-09913-4
Yesilyurt, M., Dogan, M., Acar, M., Doğan, M., & Acar, S. M. (2019). The meta-analysis of
the effect of computer-aided instruction on students' attitudes in science and
mathematics. Acar, S Journal of Primary Education, 1(2), 57–69.
https://dergipark.org.tr/en/pub/temelegitim/issue/44423/542243
278

Vol. 20, No. 8, pp. 278-299, August 2021
The Effectiveness of a Self-Questioning Strategy

at Developing Academic Achievement and
Critical-Thinking Skills among Secondary-School
Students in Saudi Arabia
Ali Tared Aldossari*

Imam Abdulrahman Bin Faisal University, College of Education,
Curriculum & Instruction Department, Dammam, Saudi Arabia
https://orcid.org/0000-0001-6329-2387
Mohammed Msnhat Aldajani

Imam Abdulrahman Bin Faisal University, College of Education,
Curriculum & Instruction Department, Dammam, Saudi Arabia
https://orcid.org/0000-0002-8531-437X
Abstract. This study aims to examine how effective a self-questioning

strategy is at developing academic achievement and critical-thinking
skills in the jurisprudence curriculum (Fiqh), followed by secondary-
school students. The research sample is taken from secondary-school
students in Dammam, the Kingdom of Saudi Arabia (KSA); 64 students
from two different schools were selected: 32 as a control and 32 as an
experimental group. The research instruments consist of an achievement
test, a test to see how critical-thinking skills are applied to the Fiqh
curriculum, and a teachers’ guide to teaching the Fiqh curriculum, using
the self-questioning strategy. The study recommends the use of the self-
questioning strategy when teaching the Fiqh curriculum to secondary-
school students. It also highlights the need to raise teachers’ awareness of
the importance of teaching critical-thinking skills. Considering the
research results and recommendations, the researchers recommend
several further avenues of research related to the current research
variables.
Keywords: academic achievement; critical-thinking skills; Fiqh

curriculum; self-questioning strategy; secondary-school students
1. Introduction
Islamic studies are a central subject at all academic levels in the Kingdom of Saudi
Arabia (KSA); the curriculum guidelines state that “religious sciences are the basis
* Corresponding Author: Ali Tared Aldossari; Email: ataldossari@iau.edu.sa
©Authors
This work is licensed under a Creative Common Attribution Non-Commercial No-Derivatives 4.0
279
of all levels of study – primary, intermediate and secondary – and Islamic culture
is the basis of all years of higher education” (Education Document, 1970). Modern
methods of teaching are applied to such Islamic subjects. Abanmi (2016) states
that traditional teaching strategies (such as lectures and indoctrination) are no
longer accepted because they limit learner activity, passively impacting students’
cognitive achievement, limiting their skills gains, and weakening their attitudes.
As such, it is now advised that teaching strategies should be reconsidered. Al-
Mofda (2005) stresses this needs, while Abu Ajwa (2009) highlights the
importance of interaction between teachers and their students, especially when
teaching Islamic studies, which requires an interest in and inclination towards the
subject. Abdel-Fattah (2010) observes that a self-questioning strategy helps
students to develop a self-awareness of the learning process; this helps them
examine their own understanding, causing them to become aware of what they
are learning and the way in which they learn.
Arian's study (2003) supports the idea that self-questioning is a strategy that
benefits secondary-school students more than students at any other stage, because
it helps them to transfer the impact of learning and develops organisational skills
of the learning process. The self-questioning strategy is a modern educational
strategy that synergises well with critical thinking, as it requires students to
analyse and evaluate information in order to solve problems and make decisions,
rather than simply requiring them to narrate the issue at hand.
The development of critical-thinking skills is one of the most important

contemporary educational goals; scholars in the field of educational studies
believe that one of the primary aims of education is to equip every student with
the skills to reach the right decision at the right time. In this regard, Magdy (2003)
emphasises the contribution that training students in critical-thinking skills makes
to the development of individuals who are capable of taking responsibility and
making decisions, as well as expanding their mental processes.
2. Research Problem
Despite the importance of academic achievement in Islamic studies and their
impact on learners’ lives, many studies (Al-Balawi, 2006; Al-Matroudi, 2009; Al-
Mousa, 2001; Al-Qarni, 2013) have indicated that there has been a decline in
students’ achievements, due to the teaching methods used, as these neglect the
learner and disregard the various approaches to thinking, especially critical
thinking. As such, it has been noted, students’ scientific and practical skills must
be developed (Wahed, 2018).
Al-Surour (2005) notes that there is a strong relationship between critical-thinking

skills and self-questioning, as critical-thinking skills are based on questioning (i.e.,
analytical and evaluative questions), leading to knowledge and understanding.
Therefore, Abu Jadu and Nofal (2007) stress the idea that developing the habit of
questioning is a basic requirement when developing critical-thinking skills. The
current study will examine this issue by asking several pertinent research
questions.
280
2.1. Research questions

This study seeks to answer the following main question:
How effective is the self-questioning strategy at developing academic
achievement and critical-thinking skills studying the Fiqh curriculum for
secondary-school students?
The following sub-questions arise from this main question:

1. How effective is the self-questioning strategy at developing academic
achievement studying the Fiqh curriculum for secondary-school students?
2. How effective is the self-questioning strategy at developing critical-
thinking skills studying the Fiqh curriculum for secondary-school
students?
3. What is the nature of the correlation between critical-thinking skills and
academic achievement studying the Fiqh curriculum for secondary-school
students?
2.2. Research hypotheses

This study seeks to verify the following hypotheses:
1. There is no statistically significant difference at the level (α ≥ 0.05) between
the average test scores of the experimental group (which was studying
using the self-questioning strategy) and the control group (which was
studying using the traditional method).
2. There is no statistically significant difference at the level (α ≥ 0.05) between
the average test scores when testing the critical-thinking skills of the
experimental group and the control group.
3. There is no statistically significant correlation at the level (α<0.05) between
critical-thinking skills and academic achievement.
2.3. Research aims

This study aims to:
1. Determine the effectiveness of the self-questioning strategy at developing
students.
2. Determine the effectiveness of the self-questioning strategy at developing
critical-thinking skills studying the Fiqh curriculum for secondary-school
students.
3. Determine the nature of the correlation between critical-thinking skills and
students.
2.4. Research importance

2.4.1 Theoretical importance
This is the first study to investigate the effectiveness of the self-questioning
strategy studying the Fiqh curriculum in the KSA. It evaluates the use of modern
teaching trends, which emphasise the students’ role in the educational process
and focus on developing students’ critical thinking.
281
2.4.2. Practical importance

This study aims to develop the critical-thinking faculty of students through a self-
questioning strategy. It directs teachers to apply the self-questioning strategy in
the teaching of Islamic studies and introduces a guide to the self-questioning
strategy that can be applied when teaching the Fiqh curriculum.
3. Terminology used in this study

The following terms are used in this study:
• Self-questioning strategy:
In this strategy, the student asks themselves several questions before, during
and after a lesson. These questions may help develop academic achievement
and critical-thinking skills.
• The Fiqh curriculum:
A KSA curriculum that tackles the purposes of Sharia and offers a set of legal
rulings and limits (Ministry of Education, 2018).
• Academic achievement:
Grades that secondary-school students receive following the achievement test
prepared for this purpose.
• Critical thinking:
A pattern of thinking that gives a learner the ability to make correct judgments
(according to certain criteria) related to the issues, situations and problems
they face.
• Critical-thinking skills:
The ability to evaluate the validity and accuracy of information considering
the evidence and data available to the learner and to reach correct logical
conclusions.
4. Theoretical literature and related studies

Over the past century, educators have focused their research efforts on different
teaching methods, examining their ability to achieve desirable educational
outcomes for learners at different educational stages. On this topic, it has been
said that “a successful teacher is nothing but a successful method, and those in
charge of teacher training have trained their students to use different teaching
methods that help them achieve the goals of teaching easily and successfully”
(Bashir, 2017).
4.1. The self-questioning strategy

The self-questioning strategy is an important strategy used to develop awareness.
As noted, when following this strategy, a student will ask themselves several
questions before, during and after a lesson. These questions will facilitate
understanding and encourage the student to think about important elements,
such as their main ideas, the title of their work, and their previous experience (Al-
Atoum et al., 2008).
Nassar (2015) also observes that this strategy is one that combines cognitive and
metacognitive properties when used in different ways. This is because it helps the
student to focus on the details, to summarise their knowledge, and to recognise
282
the important elements of their study, as well as increasing an awareness of the

knowledge they have gained and the processes that have been performed on it.
Self-questioning is known under several different names, including self-planning,
self-assessment, and self-reflection (Hamid, 2000), as well as the strategy of self-
interrogation (Attia, 2009).
4.1.1. The importance of the self-questioning strategy

The self-questioning strategy is based on the student's positivity as applied to the
educational process. This is because the questions that students ask build
emotional and cognitive motives; thus, they become more responsible for their
own learning. The importance of the self-questioning strategy can be summarised
as follows:
1. It helps students to formulate any questions they have about the topic they are
studying, allowing them to engage in a dialogue concerning what they know
and what they would like to know.
2. It increases students’ understanding of the topic at hand, allowing more of
their energy to be directed towards teamwork; thus, it makes them more
efficient students.
3. Students are made to rely on themselves when constructing meaning; as such,
its impact is retained for a long time.
4. Students' questions reveal their thinking style, alternative concepts, their
perceptual understanding, and what they want to know.
5. Students become more sensitive to important parts of the lesson content. They
evaluate their understanding of the educational material (in the sense that
they become aware of what they previously did not understand) and they take
remedial action by asking themselves and/or their peers questions or by
engaging in additional reading (Ramadan, 2005).
4.1.2. Self-questioning strategies: steps

According to Attia (2009), the self-questioning strategy contains the following
steps:
The teacher introduces the topic to the students, provided that it is a topic that
arouses the students’ interest.
1. The students are encouraged to ask themselves questions in order to activate
their metacognitive processes before they start the learning process.
2. Students read around the subject carefully, asking themselves questions that
aid the planning and organisation of the learning process.
3. Students evaluate the results and take notes on their observations and
performance.
4. The teacher investigates and examines the students’ responses to self-
interrogation and shows the students how to benefit from them. This is the
final stage of the self-questioning process.
4.1.3. The difference between the self-questioning strategy and the “know, want, learn”
strategy
Due to the similarity of the steps involved, El-Ghahori (2012) believes that the self-
questioning strategy is the same as the “know, want, learn” (KWL) strategy.
However, some scholars – such as Bahloul (2004) – differentiate between the two
strategies, but without showing the differences. Having reviewed several studies
283
on this topic, the authors of this current paper have determined what these
differences are: namely, that metacognitive strategies have three main
dimensions: planning, organising and supervising.
We have also found that, while the control aspect is more visible in the self-
questioning strategy, the organising aspect is more visible in the KWL strategy
(the two strategies share the planning aspect). As such, we believe that questions
asked as part of the self-questioning strategy are more profound, due to the
development of the learners’ questions.
4.2. Critical thinking

Critical thinking is one of the most important topics studied here, because it
empowers learners with the basic skills needed as part of the learning and
teaching process.
4.2.1. The definition of critical thinking

Critical thinking is a careful judgment as to whether we should accept, reject or
postpone a decision regarding a particular request or issue. It also assumes that
we have a degree of confidence in what we accept or reject (Saadeh, 2011). Udall
and Daniels (1991) define critical thinking as the ability to verify and evaluate
something based on previously agreed criteria.
Critical thinking is also the ability to evaluate information and examine opinions,
taking into account the perspectives on the topic under study (Abu Jad & Nofal,
2007). Nabhan (2001) states that critical thinking is the apparent behaviour of an
individual in a particular situation and regarding a specific subject about which a
judgment or practical behaviour is required, considering the data and facts
available to the individual. Finally, Qatami (2004) states that critical thinking is
reasonable reflective thinking that focuses on what an individual believes or does.
In summary, critical thinking can be defined as thinking patterns that give a
learner the ability to make correct judgments (according to certain criteria)
regarding the issues, situations and problems they face.
4.2.2. Critical thinking and education

Education benefits from critical thinking, being an evaluative and judgmental
process that moves through a logical thought process to solve problems. Some
researchers believe that critical thinking corresponds to the higher levels of
Bloom's classification; this approach has been adopted by the KSA Ministry of
Education (Faqihi, 2016).
4.2.3. Critical-thinking skills

There are many elements to and classifications of critical-thinking skills. This is
due to the various different opinions on and definitions of the topic, the most
famous of which is Watson and Glaser’s classification, which divides critical-
thinking skills as follows (Al-Qahtani, 2010):
1. The skill of inference:
This is the ability to distinguish between the degrees of probability concerning
how right or wrong a result is, depending on the degree of its connection to
certain given facts.
284
2. The skill of forecasting assumptions:

This is the ability to examine events or facts; it is judged considering the
available data or evidence.
3. The skill of deduction:
This is the learner’s ability to derive the relationships between given facts,
such that they can judge the extent to which a result derived from said facts is
truly connected or not, regardless of the validity of the facts given or the
position of the learner.
4. The skill of interpretation:
This is the ability to justify or draw certain conclusions considering facts or
witnessed incidents that have been accepted by the human mind.
5. The skill of evaluating discussions:
This is learners’ ability to distinguish between strengths and weaknesses
when judging an issue or a fact considering the available evidence.
4.2.4. The relationship between self-questioning and critical thinking

Abu Jad and Nofal (2007) stress that developing a questioning ability is one of the
basic requirements needed for the development of critical thinking. As Al-Surour
(2005) observes, there is a strong and stable relationship between critical-thinking
skills and self-questioning; this is because critical-thinking skills are based on
questioning, which leads to knowledge and understanding through analytical
and evaluation questions.
Therefore, we should not hesitate to ask the questions that lead to knowledge of
what we must do in matters of religion, worship and dealings with people. At the
same time, when asking questions, we should maintain the necessary legal limits
and morals.
A student should be aware that knowledge is derived from questions and

answers; that a good question is half the way to possessing knowledge and that
the ability to ask the right question is one of the most important thinking skills.
4.3. Academic achievement

Academic achievement is a basic criterion affecting most decisions related to the
student, the curriculum and the educational process. Students’ progress in the
educational process and their achievement of goals are recognised through
academic achievement.
With its various knowledge aspects, academic achievement is one of the factors
shaping students’ personality; this is why educators are keen for students to
obtain the maximum possible amount of knowledge through the learning process.
4.3.1. The concept of academic achievement

For the purpose of this study, we define “academic achievement” as marks
obtained by secondary-school students in the achievement test prepared for this
research.
285
4.3.2. The importance of academic achievement

Academic achievement is one of the most important goals of education, as it
allows an awareness of the level at which students are performing, of teachers’
performance, and the appropriateness of the means and curricula used to achieve
society’s educational goals.
Academic achievement helps students to become aware of their abilities, as well

as informing parents of the level of their child’s performance. It is one of the
criteria that institutions rely on in order to move students from one educational
stage or level to another.
Finally, high academic achievement (in comparison to low achievement) inspires

confidence in students (Adam, 2015).
4.3.3. Factors affecting academic achievement

There are many factors that affect academic achievement, including the mental
abilities of the student, their motivation to learn, the educational, economic and
social level of the family, and teachers’ competence and sincerity.
4.4. The Fiqh curriculum and secondary-school students’ critical thinking

Critical thinking may help to enlighten students about legal rulings, offering them
the ability to elicit, distinguish and direct their questions and to anchor them to
reality. As such, we believe that the Fiqh curriculum has a major role to play in
developing critical-thinking skills, because it covers important subjects related to
students’ lives, environments, homeland and nation. It contributes to building a
generation whose members are useful to society; this is because it plays a major
role in teaching students about legal rulings, linking them to reality and
developing students’ abilities to solve problems.
Secondary-school students need to acquire the various aspects of thinking skills

that will enable them to generate mental evidence or clues, make conclusions,
recognise the large number of lessons learned from a religious perspective, and
evaluate all these with reference to appropriate criteria (Metwally, 2017).
4.5. Studies relating to the self-questioning strategy, critical thinking and

academic achievement
4.5.1. Studies on the self-questioning strategy
The self-questioning strategy is relatively recent; it conforms to the requirements
of the current era as it helps the learner control their study. Al-Anzi’s (2013) study
aims to identify the impact of the self-questioning strategy through its
development of second-grade intermediate KSA students’ metacognitive skills
and academic achievement in monotheism (Tawheed). Al-Tarhouni (2012)
examines the effect that the self-questioning strategy has on achievement and on
the development of critical-thinking skills among students of the Faculty of
Arts/Department of Education at Misurata University, while Gamal (2014)
evaluates the impact of the self-questioning strategy when teaching the Prophet’s
biography, analysing its effect on the achievement and critical thinking of first-
grade intermediate students at Al-Azhar in Egypt.
286
Fallatah (2015) considers the effectiveness of self-questioning and reciprocal

teaching strategies in terms of their ability to improve academic achievement and
develop reflective-thinking skills among female students attending the Islamic
Education Teaching Methods course at Umm Al-Qura University. Al-Shayji
(2018) evaluates how effectively the self-questioning strategy develops the
academic achievement and critical-thinking skills of ninth-grade intermediate
students in the Islamic Fiqh unit in the State of Kuwait.
Finally, Al-Qahtani (2010) examines the effectiveness of teaching science using the
self-questioning strategy in terms of academic achievement and the development
of reflective-thinking skills. His study uses the quasi-experimental approach, the
results of which show that there are statistically significant differences attributed
to the use of the self-questioning strategy (in terms of its effect on students’
achievement and the development of reflective-thinking skills) when compared
to the usual traditional method. Considering this, Al-Qahtani recommends both
using the self-questioning strategy at different educational stages and including
certain self-questioning activities in scientific curricula.
4.5.2. Studies on critical thinking

Ali (2009) considered the impact of the guided discovery and brainstorming
methods in terms of their effect on the development of critical-thinking skills and
the academic achievement of secondary-school students in Tabuk under the Fiqh
curriculum, while Al-Anzi (2013) demonstrates the effectiveness of using the
constructivist learning model to develop critical-thinking skills and academic
achievement under this curriculum. Al-Shamlaty (2015) identifies the effect using
probing questions has on achievement and on the development of critical-
thinking skills when teaching interpretation to second-grade intermediate
students. Al-Shammari (2019) examines how a learner autonomy model can be
used to develop critical-thinking skills and jurisprudential curiosity under the
Fiqh curriculum, considering its effect on third-grade intermediate students in
Dammam.
4.5.3. Studies on academic achievement under the Fiqh curriculum

Al-Ghamdi’s study (2009) evaluates the impact of the systemic approach when
teaching the Fiqh curriculum, looking at its effect on the academic achievement of
second-grade intermediate students in the Makkah region. Al-Qarawi (2010),
using a quasi-experimental approach, considers the effect that using the multiple
intelligence teaching strategy has on the achievement of third-grade intermediate
students undertaking a Fiqh course. Al-Khatib (2010) shows the effect of using the
V-shaped map on the acquisition of jurisprudential concepts and the formation of
an integrated conceptual structure among students at Al-Hussein Bin Talal
University in Jordan.
Abu Shareekh (2011) investigates the effect that the multiple intelligence and “six
hats” strategies have on the achievement and development of the critical-thinking
skills of tenth-grade students, as well as their effect on students’ attitudes towards
learning Islamic Fiqh. Al Boughbran (2013) examines the effect software has on
287
first-year secondary female students' achievements (at Khamis Mushait

Educational District, KSA) and their attitudes to this software.
Awajan (2015) considers the effectiveness of teaching according to the theory of

multiple intelligence and its effect on academic achievement and the development
of creative-thinking skills among fifth-grade primary female pupils studying the
Islamic Fiqh unit, comparing this effect to that of the usual teaching strategy. Ali
(2016) examines the effect using the cooperative learning strategy has on the
achievement of seventh-grade pupils studying for a basic education in Sana’a.
Faqihi (2016) investigates the effectiveness of using concept maps in terms of their
ability to improve the achievement and critical thinking of second-year
secondary-school students, while Al-Otaibi (2017) considers the use of a visual
strategy (based on knowledge-load theory) and its effect on the achievement of
second-grade secondary-school students studying the Fiqh curriculum. Finally,
Al-Omari (2019) has studied the effectiveness of the inverted class, in terms of
academic achievement, focusing on first-year secondary students.
5. Research procedures
Our research uses the quasi-experimental approach, which is a scientific method
of solving problems that is based on a real test of hypotheses concerning cause-
and-effect relationships. The experimental method has the highest degree of
scientific control.
In this study, we sought to learn and measure the impact of the independent
variable, which is the self-questioning strategy, on the two dependent variables
(the academic achievement and critical-thinking skills of secondary-school
students). Our research design is based on two groups (see Table 1).
Table 1
The academic The academic
achievement and achievement and
critical-thinking The experiment critical-thinking
skills tests (before skills tests (after
the experiment) the experiment)
Teaching the Prohibited
Experimental
 Sales Unit using the self- 
group
questioning strategy
Teaching the Prohibited
Control
 Sales Unit using the 
group
traditional method
As seen in Table 1, the design of the research experiment is based on the

experimental group that studied using the self-questioning strategy, and the
control group that studied in the usual way, and the research applied a test of
academic achievement and a test of critical-thinking skills in Fiqh curriculum
before and after the experiment on both groups.
288
5.1. The research community

The current research community consists of all secondary-school students in West
Dammam in the Eastern Province of the Kingdom of Saudi Arabia that are
enrolled in government schools during the first semester of the academic year
2020/2021 (8179 students).
5.2. The research sample

A random sample was selected from the study community, and the researchers
followed the following research procedures:
1. An inventory was made of all secondary public education schools in the city
of Dammam, based on the official lists of the General Administration of
Education in the Eastern Province (52 schools, including 23 schools in East
Dammam, and 30 schools in West Dammam).
2. An inventory was made of all regular secondary-school students studying
during the first semester of 2020/2021, according to the West Dammam
Supervision Office (8179 students).
3. Two secondary schools were randomly selected from the list of schools.
4. Two classes that were studying Fiqh 1 were randomly chosen from the
selected schools.
5. The Al-Qasim bin Muhammad school was chosen to represent the control
group, while the Abu Sufyan bin Al-Harith school was chosen as the
experimental group.
6. Six classes were studying Fiqh in the Al-Qasim bin Muhammad School, two
of which were randomly selected to represent the control group, while eight
classes were studying Fiqh 1 in the Abu Sufyan bin Al-Harith School, two of
which were randomly selected to represent the experimental group.
5.3. The research variables

5.3.1. First (independent) variable
The independent variable is the self-questioning strategy.
5.3.2. Second (dependent) variable

The dependent variable is academic achievement and critical-thinking skills.
5.3.3. Third (confounding) variables

• Type:
The two study groups were selected from male students only, in government
schools in Dammam, which are subject to the education system in the
Kingdom of Saudi Arabia.
• Chronological age:
The students’ ages were determined using the school records; it was found
that all the students studying the Fiqh 1 course are between 16–18 years of age
and are studying the course for the first time. Therefore, it can be said that the
two groups are equivalent in terms of the chronological age variable.
• Study content:
To ensure parity and equality between the two groups, in terms of the
academic content under study, all elements of the academic content were
taken from the Fiqh 1 book for the secondary stage: Prohibited Sales Unit.
289
Table 2 shows the specific topics considered as part of this study.

Table 2
Topic Subject matter Topic Subject matter
no. no.
1 Reason for prohibited earning 6 Selling the sample
2 Monopoly 7 Sale of Tawaruq
3 Usury 8 Gambling and ambiguous sale
4 Usury of sales 9 Insurance
5 Exchange contract
This unit was chosen because of its consistency and the interdependence of its
contents, which make the unit suitable for an analysis of the self-questioning
strategy and critical-thinking skills.
5.4. The research instruments

5.4.1. First instrument: an academic achievement test for secondary-school students
The construction of an achievement test requires the following steps:
1. Determining the objective of the achievement test.
2. Preparing a table for the specifications.
3. Determining the type of test questions to be asked.
4. Formulating and arranging the test questions.
5. Preparing the test instructions.
6. Forming the answer key.
5.4.2. Second instrument: a test of critical-thinking skills for secondary-school students

This test aims to determine the level of secondary-school students’ critical-
thinking skills before they are developed using the self-questioning strategy. The
purpose of this test is to establish a valid scale with which to measure the critical-
thinking skills of students undertaking the Fiqh 1 Prohibited Sales Unit.
The test consists of 40 questions, which are divided into five skills: inference;
deduction; discussion evaluation; knowledge of axioms or assumptions; and
interpretation. Each skill consists of 8 questions; the student gets one mark for
each correct answer.
Table 3: Correlation coefficients for each statement of the axis statements with the
total score of the axis to which it belongs
Correlation coefficient
Knowledge of
Serial Discussion
Inference Deduction axioms or Interpretation
evaluation
assumptions
1 0.505 ** 0.736 ** 0.803 ** 0.492 ** 0.465 **
2 0.804 ** 0.736 ** 0.859 ** 0.591 ** 0.757 **
3 0.449 ** 0.515 ** 0.491 ** 0.730 ** 0.662 **
4 0.769 ** 0.763 ** 0.701 ** 0.494 ** 0.409 **
5 0.429 ** 0.658 ** 0.635 ** 0.671 ** 0.784 **
6 0.498 ** 0.691 ** 0.536 ** 0.494 ** 0.4665 **
7 0.818 ** 0.406 ** 0.729 ** 0.433 ** 0.4977 **
8 0.559 ** 0.562 ** 0.446 ** 0.748 ** 0.458 **
** significant at 0.01.
290
The reliability of the instrument was calculated using Cronbach’s alpha.

Table 4 shows the value of the reliability coefficient for each axis of the
questionnaire.
Table 4: Reliability coefficient
Reliability
Axis
coefficient
Inference skill 0.760
Deduction skill 0.758
Discussion evaluation skill 0.807
Knowledge of axioms or assumptions skill 0.824
Interpretation skill 0.778
The questionnaire as a whole 0.917
It is clear from Table 4 that the reliability coefficients are high, indicating that the
questionnaire has a high degree of reliability.
6. Research results
This study’s aim was to learn how effective the self-questioning strategy is at
developing the academic achievement and critical-thinking skills of secondary-
school students studying the Fiqh curriculum for secondary-school students.
Therefore, a quasi-experimental design was used that tested the control group
both before and after the experiment.
The researchers’ aim is to present the results of the study and its interpretation,
and to discuss the results by answering the research questions and their associated
hypotheses.
6.1. The first hypothesis

There is no statistically significant difference at the level (α ≥ 0.05)
between the average test scores of the experimental group (which
was studying using the self-questioning strategy) and the control
group (which was studying using the traditional method).
To verify the validity of this hypothesis, the researchers used a t-test to show the
differences between the two independent groups, so as to identify the significance
of the differences between the average scores of the experimental group and the
scores of the control group in the post application of the total score of the
achievement test (Table 5).
Table 5: T-test for the significance of the differences between the average scores of the
experimental group and the scores of the control group in the post-application of the
achievement test
Arithmetic Standard T- Significance Eta
Group No Comment
average* deviation value level square
Experimental 32 63.52 19.44 Significant
2.90 0.005 0.12
Control 32 50.23 17.09 at 0.01
*The average has been converted to 100 marks.
291
Table 5 shows that the value of T is significant at a level less than 0.05, which
indicates that there are statistically significant differences between the average of
the experimental group and the control group in the scores of the post-application
of the achievement test. These differences were in the experimental group’s
favour.
It is also clear from the same table that the effect of ETA square on the total score
of the achievement test is 0.12, which means only 12% of the total variance of the
differences between the average scores of the post-application of the achievement
test for the two groups. This is considered an average effect, according to Cohen's
criteria (which explains the effect of the independent variable on the dependent
variable), because an effect that explains about 1% of the variance indicates a small
effect, about 6% indicates a medium effect and about 15% or more indicates a high
effect.
The results of this study agree with Omar (2008) and Al Shayji (2018), both of
whom examine how effective the self-questioning strategy is at developing
students’ academic achievement. The results also agree with those of Al-Qarawi
(2010), Awajan (2015) and Al-Otaibi (2017), which consider the development of
academic achievement under the Fiqh curriculum.
These results can be attributed to the fact that the self-questioning strategy made
students feel more positive during the learning process, as they are given the
opportunity to test their previous knowledge and link it with their new
knowledge, leading to corrected mistakes and questions generated during and
after studying, and thus leading to better academic achievement. In addition, self-
questioning provided an effective learning environment that made the student the
central focus of the educational process through positive participation.
6.2. The second hypothesis

There is no statistically significant difference at the level (α ≥ 0.05)
between the average test scores when testing the critical-thinking
skills of the experimental group and the control group.
To verify the validity of this hypothesis, the researchers used a t-test to show the
differences between the two independent groups, so as to identify the significance
of the differences between the average scores of the experimental group and the
scores of the control group in the post application of the sub-skills and the total
score of the critical-thinking test (Table 6).
Table 6: T-test for the significance of differences between the average scores of the
experimental group and the scores of the control group in the post application of the
critical-thinking test
Arithmetic Standard T Significance Eta
Skills Group No Comment
average* deviation value level square
Inference Experimental 32 71.48 19.37 Significant
2.74 0.008 0.11
skill Control 32 58.20 19.47 at 0.01
Deduction Experimental 32 58.20 21.44 No
1.85 0.069 0.05
skill Control 32 48.44 20.76 significance
Experimental 32 66.41 19.43 1.40 0.166 0.03
292
Discussion
No
evaluation Control 32 59.77 18.44
significance
skill
Knowledge Experimental 32 54.69 20.27
of axioms or Significant
2.24 0.029 0.08
assumptions Control 32 42.97 21.52 at 0.05
skill
Interpretation Experimental 32 61.72 24.99 No
0.97 0.338 0.01
skill Control 32 56.25 20.08 significance
Total mark Experimental 32 62.50 16.52
for the Significant
2.43 0.018 0.09
critical- Control 32 53.20 14.20 at 0.05
thinking test
* The average has been converted to 100 marks.
Table 6 shows that the T values are significant at the level of 0.05, and less for the
skills of inference and knowledge of axioms or assumptions, as well as for the total
score for the critical-thinking test. This indicates that there are statistically
significant differences between the averages of the experimental group and the
control group in the post-application scores for those skills. These differences
were in the experimental group’s favour.
Table 6 also shows that the values for T are not significant for the skills of
deduction, discussion evaluation and interpretation, indicating that there are no
statistically significant differences between the averages of the experimental
group and of the control group in terms of the marks given for the post application
of those skills in the critical-thinking test.
This table also indicates that the ETA square for the total score of the critical-
thinking test is 0.09, which means that only 9% of the total variance in the
differences between the average marks given for the post-application scores in the
critical-thinking test for the two groups is due to the effect of teaching using the
self-questioning strategy for the two groups. This is considered an average effect
according to Cohen's criteria.
These results agree with those of studies that have examined the effectiveness of
the self-questioning strategy at developing critical-thinking skills among students
(Al-Shayji, 2018).
The study of the inference and knowledge of axioms or assumptions skills shows
a large function in the experimental group’s favour. “Inference” means the ability
to draw conclusions based on premises, considering a set of facts and knowledge
before judging their validity; this result thereby illustrates the interrelationship
between the self-questioning strategy and critical-thinking skills. We can attribute
this to the fact that the students in the experimental group are practising different
mental activities and skills during the educational process, generating ideas and
training their inference and assumption abilities. This result agrees with Ramadan
(2005) and Al-Shammari (2019).
Accordingly, we reject the second hypothesis. This can be attributed to the use of
the self-questioning strategy to developing critical-thinking skills, because
learners asks questions and guesses the answer, leading to the stimulation of the
293
their critical-thinking ability and helping them to retrieve relevant previous

experiences.
6.3. The third hypothesis

There is no statistically significant correlation at the level (α < 0.05)
between critical-thinking skills and academic achievement.
To answer this question, the researchers used the Pearson correlation coefficient
to measure the relationship between the scores for the research sample (n = 64) in
the sub-skills and the total score for the critical-thinking test, and their scores in
the achievement test (Table 7).
Table 7: Results for both the experimental and control groups measuring the
relationship between the scores for sub-skills and the total score for the
critical-thinking skills test and their scores in the achievement test (n = 64)
Correlation Description of the
Skills Significance
coefficient correlation
Positive
Inference skill 0.4864 Significant at 0.01
correlation
Positive
Deduction skill 0.5605 Significant at 0.01
correlation
Discussion evaluation Positive
0.4899 Significant at 0.01
skill correlation
Knowing axioms or Positive
assumptions skill correlation
Positive
Interpretation skill 0.4430 Significant at 0.01
correlation
Total mark for the Positive
critical-thinking test correlation
Table 7 shows that there are direct (positive) correlations between the skills
covered by the critical-thinking test (inference, deduction, evaluation of
discussions, knowledge of axioms or assumptions and interpretation) and the
total score for the achievement test. This indicates that, potentially, the higher the
marks achieved for the critical-thinking test by the research sample (secondary-
school students), the higher their level of academic achievement under the Fiqh
curriculum. These correlations were statistically significant at a 0.01 level.
It is also clear from Table 7 that there is a direct (positive) correlation between the
total score for the critical-thinking test and the total score for the achievement test.
This indicates that, potentially, the higher the marks achieved for the critical-
thinking test by the research sample (secondary-school students), the higher their
level of academic achievement under the Fiqh curriculum. These correlations
were statistically significant at a 0.01 level.
294
Table 8: Results for the experimental group measuring the relationship between the
scores for sub-skills and the total score for the critical-thinking skills test and their
scores in the achievement test (n = 32)
Correlation Description of the
Skills Significance
coefficient correlation
Inference skill 0.5452 Significant at 0.01 Positive correlation
Deduction skill 0.4631 Significant at 0.01 Positive correlation

Discussion
0.3896 Significant at 0.01 Positive correlation
evaluation skill
Knowing axioms or
assumptions skill
Interpretation skill 0.4832 Significant at 0.01 Positive correlation
Total mark for the
critical-thinking test
curriculum. These correlations were statistically significant at a 0.05 level or less.
It is also clear that there is a direct (positive) correlation between the total score
for the critical-thinking test and the total score for the achievement test. This
indicates that, potentially, the higher the marks achieved for the critical-thinking
test by the research sample, the higher their level of academic achievement. These
correlations are statistically significant at a 0.01 level.
Table 9: Results for the control group measuring the relationship between the scores
for sub-skills and the total score for the critical-thinking skills test and their scores in
the achievement test (n = 32)
Description
Correlation
Skills Significance of the
coefficient
correlation
Positive
Inference skill 0.2818 No significance
correlation
Positive
Deduction skill 0.6034 Significant at 0.01
correlation
Discussion Positive
evaluation skill correlation
Knowing axioms
Positive
or assumptions 0.4706 Significant at 0.01
correlation
skill
Positive
Interpretation skill 0.3569 Significant at 0.01
correlation
Total mark of
Positive
critical thinking 0.6416 Significant at 0.01
correlation
test
295
curriculum. These correlations are statistically significant at a 0.05 level or less,
except for the inference skill, which has no statistical significance.
It is also clear that there is a direct (positive) correlation between the total score
for the critical-thinking test and the total score for the achievement test. This
indicates that, potentially, the higher the marks achieved for the critical-thinking
test by the research sample, the higher their level of academic achievement. These
correlations are statistically significant at a 0.01 level. This result is consistent with
Al-Anzi (2013).
The researchers attribute the above results to the fact that the development of
critical-thinking skills and academic achievement are highly correlated, that is,
developing one side leads to the development of the other. Accordingly, we reject
the third hypothesis.
7. Conclusions
The study offers useful findings, in that it indicates:
1. How effective the self-questioning strategy is at improving the academic
achievement of secondary-school students studying the Fiqh curriculum.
2. How effective the self-questioning strategy is at developing the critical-
thinking skills of secondary-school students studying the Fiqh curriculum.
3. There is a correlation between critical-thinking skills and the academic
achievement of secondary-school students studying the Fiqh curriculum.
We can explain this by saying that the self-questioning strategy makes students
more positive during the learning process. Additionally, students have the
opportunity to test their previous knowledge and link it to their new knowledge;
this leads to corrected mistakes and to the generation of new questions, leading in
turn to better academic achievement and improved critical-thinking skills. The
self-questioning strategy also provides an effective learning environment that
makes students the focus of the educational process, facilitating the assimilation
of information and making students capable of making their own decisions on
different aspects of life.
The researchers attribute the third finding to the fact that development of critical-
thinking skills and academic achievement is highly correlated, that is, developing
one side leads to the development of the other side.
8. Research recommendations and suggestions

Considering the research results, we recommend the following:
1. Attention should be given to modern teaching methods (including the
strategy of self-questioning) at all academic levels, especially the secondary
296
stage. These methods should be included in the curricula of teaching colleges,

thereby training student teachers in a manner that can keep up with today’s
needs.
2. The Ministry of Education, and those in charge of curricula, need to pay
attention to means of developing thinking skills. Additionally, they should
encourage teachers to use these approaches through continuous training
courses, informing them of the most important developments in this field and
thereby improving their teaching competencies.
3. Greater awareness among teachers is necessary; this could be achieved
through educational bulletins and guided reading concerning the importance
of teaching critical thinking to young people.
4. The training of thinking skills needs further attention, both in general
education and among Islamic education teachers and supervisors in
particular. This can be achieved by organising in-service training courses.
5. The completion of a course related to thinking skills in general and critical
thinking in particular should be a professional requirement for teachers.
6. Students should be made the main focus of the educational process. This can
be achieved by giving them a primary role in any discussion and analysis and
by teaching them to search for questions, derive ideas and use them correctly.
Based on our results, and considering the recommendations, we suggest that the
following future avenues of research would be worthwhile:
1. Building a programme based on self-questioning strategies that can develop
all types of thinking skills.
2. Conducting research into the effect of self-questioning strategy on the
development of jurisprudential skills.
3. Conducting an analytical study of the effect of critical-thinking skills on the
curricula overall.
Acknowledgement
The authors extend their thanks to the participating research sample and
everyone who facilitated the research procedures, as this work is part of
an unpublished master's thesis under the supervision of the first author in
this paper and the hard work of the second author.
9. References
Abanmi, F. bin A. (2016). The effect of using the flipped classroom strategy in teaching
interpretation on academic achievement and attitude toward the subject, among
second year secondary students. Journal of Reading and Knowledge, 172, 21–48.
Abdel Fattah, A. (2010). Teaching and Learning Strategies Models and Applications. University
Book House.
Abdel Wahed, A. A. (2018). The effectiveness of teaching critical-thinking skills in the life
skills of fourth-grade science students. Journal of Arts, Literature, Humanities and
Sociology, 25, 400–415.
Abdul Hamid, A. (2000). The effectiveness of certain cognitive strategies in developing
some higher reading comprehension skills among first-year secondary students.
Reading and Knowledge Journal, 2, 189–241.
297
Abu Ajwa, H. S. (2009). The effect of self-questioning strategy on developing solving chemical
problems for eleventh grade students [Unpublished master’s thesis]. Gaza Islamic
University.
Abu Jad, Saleh & Nofal, B. (2007). Teaching Thinking: Theory and Practice. Dar Al Masirah
for Publishing and Distribution.
Abu Shareekh, S. T. (2011). The effect of using strategies of multiple intelligences and the
six hats on the achievement and development of critical-thinking skills among the
tenth-grade students in Jordan and their attitudes towards teaching Islamic Fiqh.
Education Journal, 145(2), 341–375.
Adam, A. K. (2015). Problems facing secondary school principals and their impact on academic
achievement: An application on Khartoum State in the period from (2014–2015)
[Unpublished doctoral dissertation]. University of the Holy Quran and Islamic
Sciences.
Al Boughbran, M. (2013). The effect of using a computerized educational software on the
achievement of first-year secondary-school students in the field of Fiqh in the schools of
Khamis Mushait Educational District in the Kingdom of Saudi Arabia and their attitudes
towards it [Unpublished doctoral dissertation]. University of Jordan.
Al-Anzi, F. bin M. (2013). The effectiveness of using the constructivist learning model in
developing critical-thinking skills and academic achievement in Fiqh course for third-
grade intermediate students [Unpublished doctoral dissertation]. Umm Al Qura
University.
Al-Atoum, A. Y. et al. (2008). eilm alnafs altarbawii alnazariat waltatbiq alaistadh
alduktur shafiq falah ealawna alaistadh alduktur eadnan yusuf aleatuwm
alduktur eabd alnaasir dhiab aljaraah alduktur mueawiat mahmud abu
ghazal [Educational Psychology Theory and Application 11th ed.]. Dar Al Masirah.
Al-Balawi, H. M. (2006). The extent to which the second intermediate-grade students in Jeddah
acquire the prescribed Fiqh concepts [Unpublished master’s thesis]. College of
Education for Girls in Jeddah.
Al-Ghamdi, F. A. (2009). The effect of using the systemic approach in teaching Fiqh on the
academic achievement of students in Makkah Al-Mukarramah intermediate
schools. Journal of Studies in Curricula and Educational Supervision, 1(1), 162–243.
Ali, A. G. M. G. (2016). The effectiveness of the cooperative learning strategy in teaching
Fiqh on the achievement of seventh-grade students of fundamental education.
Specialized International Educational Journal, 50(7), 333–353.
Ali, Y. bin A. K. (2009). On the path of guided discovery and brainstorming methods in developing
critical-thinking skills and academic achievement in Fiqh of secondary-school students in
the city of Tabuk [Unpublished master’s thesis]. Umm Al Qura University.
Al-Khatib, O. S. (2010). The effect of using the V-shape map on the acquisition of
jurisprudential concepts and the formation of an integrated conceptual structure
among students of Al-Hussein Bin Talal University in Jordan. Journal of the
Association of Arab Universities, 55, 197–230.
Al-Matroudi, K. bin I. (2009). The extent to which sixth-grade primary pupils in Riyadh
have acquired the prescribed Fiqh concepts. Reading and Knowledge Journal, 98,
129–169.
Al-Mofda, S.S. (2005). The effect of applying cooperative learning on the achievement of
second-grade secondary students of Fiqh (experimental study). Reading and
Knowledge Journal, 48, 94–144.
Al-Mousa, Y. M. F. (2001). The Degree of Acquisition of Basic Concepts Contained in Islamic
Fiqh in the Sharia Schools Affiliated with the Palestinian Ministry of Awqaf and Religious
Affairs by Students of the Seventh, Eighth and Ninth Grades [Master’s thesis, An-Najah
National University].
298
Al-Otaibi, H. (2017). The effectiveness of the formality strategy based on the cognitive
burden on the achievement of secondary school female students in the subject of
Fiqh. Journal of Scientific Research in Education, 18(2), 425–443.
Al-Qahtani, N. bint S. (2010). Critical-thinking skills of female students at the College of
Education at the universities of King Saud University and the Emirate University.
A comparative study. Journal of the Modern Education Association, 3(7), 329–406.
Al-Qarawi, A. R. bin A. (2010). The effect of using the multiple intelligences strategy on the
achievement of the third intermediate grade students in Fiqh course [Unpublished
master’s thesis]. Imam Muhammad Bin Saud Islamic University.
Al-Qarni, A. bin M. (2013). The effect of using mental maps in teaching the developed course of
Fiqh on the achievement of first-grade intermediate students in Jeddah [Unpublished
master’s thesis]. Umm Al Qura University.
Al-Shamlaty, O. A.-Q. (2015). The effect of teaching interpretation using probing questions
on the achievement and development of critical-thinking skills among second-
grade intermediate students. Journal of Human and Administrative Sciences, 7, 64–
96.
Al-Shammari, T. (2019). The effectiveness of the learner independence model in developing
critical-thinking skills and jurisprudential curiosity in Fiqh curriculum for third-grade
intermediate students [Unpublished master’s thesis]. Imam Abdul Rahman Al-
Faisal University.
Al-Shayji, S. (2018). The impact of self-questioning strategy on developing academic
achievement and developing critical thinking among middle ninth-grade
students in the Islamic Fiqh unit in the State of Kuwait. Educational Journal,
127(32), 51–88.
Al-Surour, N. (2005). Teaching Thinking in the School Curriculum. Dar Wael for Publishing
and Distribution.
Al-Tarhouni, A. A. R. (2012). The effect of using self-questioning strategy on the achievement
and development of some critical-thinking skills among students of the Department of
Education, Faculty of Arts, Misurata University [Unpublished master's thesis].
University of Misurata.
Arian, S. (2003). The effectiveness of the metacognition strategy in the achievement of
philosophy among first-year secondary students and its impact on their attitudes
towards reflective philosophical thinking. Journal of Reading and Knowledge, 20,
113–139.
Attia, M. A. (2009) Metacognitive Strategies in Understanding Reading. Dar Al-Manahg for
Publishing and Distribution.
Awajan, W. S. M. (2015). The effectiveness of strategies based on multiple intelligences in
developing academic achievement and creative thinking skills among female
students of Islamic Fiqh in the fifth-grade primary. Journal of Educational and
Psychological Sciences, 16(1), 197–236.
Bahloul, I. A. (2004). Recent trends in metacognitive strategies in teaching reading. Journal
of Reading and Knowledge, 30, 148–280.
Bashir, M. T. (2017). Effective teaching strategies and ways of thinking. Researches of the
First International Conference: Reading the Arab and Islamic Heritage Between the Past
and the Present, 5, 459–472.
Education Document (1970). Education Policy in the Kingdom of Saudi Arabia, amended in
1995.
El-Ghahori, N. A. (2012). The effectiveness of self-scheduling strategy (K.W.L.H.) in
developing deep understanding of physical concepts and metacognition skills for
eighth-grade students in the Sultanate of Oman. Arab Studies in Education and
Psychology, 1(32), 11–58.
299
Fallatah, R. (2015). The effectiveness of self-questioning strategy and reciprocal teaching in

developing academic achievement and reflective thinking among female students of the
Islamic Education Teaching Methods course at Umm Al-Qura University [Unpublished
doctoral dissertation]. Umm Al-Qura University.
Faqihi, F. M. A. (2016). The effectiveness of using concept maps in developing achievement and
critical thinking among female students of the second-year secondary in the subject of Fiqh
[Unpublished master’s thesis]. King Khalid University.
Gamal, T. (2014). The effect of using self-questioning strategy in teaching the biography of
the Prophet on the achievement and critical thinking of Al-Azhar first-year
preparatory students. Education Journal, 158(3), 327–369.
Magdy, H. A. K. (2003). Teaching Thinking: Future Strategies for the New Millennium. Dar Al
Fiker Alarabi.
Metwally, E-S. S. E.-S. (2017). A proposed strategy based on active learning for teaching
Fiqh in developing some critical-thinking skills among Al-Azhar secondary-
school students. Arab Studies in Education and Psychology, 86, 151–181.
Ministry of Education (2018). Secondary Education Guide: Curricula System. Secondary
Education Development Project.
Nabhan, S. (2001). A proposed program for developing critical thinking in mathematics among
ninth grade students in Gaza [Unpublished doctoral dissertation]. Ain Shams
University.
Nassar, A. A. H. (2015). The effect of using thinking maps strategy on developing critical-thinking
skills and science processes for tenth-grade students [Unpublished master’s thesis].
Islamic University, Gaza.
Omar, S. M. (2008). The effectiveness of using students' self-questioning in developing
cognitive achievement and self-efficacy among students studying philosophy at
the secondary stage. Journal of the Educational Society for Social Studies, 17, 15.
Qatami, N. (2004). Effective Teaching Skills. Dar Al-Fikr.
Ramadan, H. A. M. (2005). The interaction between some metacognitive strategies and
information processing levels in developing scientific concepts and critical
thinking for first-year middle-school students in science. Journal of Scientific
Education, 8(1), 181–236.
Saadeh, J. A. (2011). maharat altafkir (mae 1997, altatbiqiatu) jawdat 'ahmad saeada
[Teaching Thinking Skills.] Dar Al-Shorouk for Publishing and Distribution.
Udall, A. J, & Daniels, J. E. (1991). Creating the Thoughtful Classroom: Strategies to Promote
Student Thinking. Zephyr Press.
300

Vol. 20, No. 8, pp. 300-320, August 2021
The Implementation of Formal Assessments in

Intermediate Phase Mathematics at Primary
Schools in South Africa
Senzeni Sibanda
Central University of Technology, Free State, South Africa
https://orcid.org/0000-0002-5937-3018
Awelani M. Rambuda*
Central University of Technology, Free State, South Africa
https://orcid.org/0000-0002-2518-0832
Abstract. The purpose of this research was to explore the

implementation of formal assessments in intermediate phase
mathematics at primary schools. The research was elicited by reports
that assessment methods and procedures for tackling learners’ needs
had been observed to be insufficient in South African schools. The
study is grounded in Piaget’s cognitive constructivism and Vygotsky’s
social constructivism. The researchers conducted document analysis of
teachers’ portfolios which were purposefully selected. Nine teachers–
three from each of Grades 4 to 6 were sampled. The portfolios were
analysed to establish whether the implementation of the formal
assessments was aligned with the Curriculum and Assessment Policy
Statement requirements. A checklist was used to determine teachers’
implementation of formal assessment. Measures of central tendency
were used to analyse data. The results revealed that teachers were not
developing the abilities of learners in handling complex mathematical
procedures as per the requirements of the policy. This implies that
learners lacked the ability to break down mathematical problems into
different factors or constituent parts. Learners were given a test instead
of a project or investigation. Hence teachers were not promoting
cooperative learning which is advocated by the policy. Therefore,
teachers should be assisted by knowledgeable colleagues and subject
advisors in their adoption and use of assessment. There must be a close
examination of the classroom observation tools that are currently being
utilised. Classroom observation assists teachers to improve their
assessment strategies. The Department of Basic Education should
supply tablets to primary schools to promote social constructive
*
Corresponding author: Awelani M. Rambuda, arambuda@cut.ac.za
©Authors
301
interaction amongst teachers and learners to enhance effective teaching

and learning.
Keywords: assessment; cooperative learning; formal assessment;

intermediate phase; mathematics
1. Introduction
Chigonga (2020) asserts that assessment is the procedure employed to collect
information with the intention to examine the learner’s previous understanding
of the content, identify the gaps and formulate instructional decisions thereafter.
In the context of this study, the intention of assessment is to enhance mathematics
teaching and learning. Assessment serves a dual purpose: first, as proof of
learning to indicate what the learner has achieved; second, to assist the teacher to
ascertain whether the learner is performing as specified in the curriculum. A
difference is made between informal assessments, such as daily monitoring of the
learner’s progress, and a formal assessment, which is the systematic evaluation of
learner’s progress. In this study, the main emphasis is on formal assessment
because Reddy et al. (2015) assert that South African mathematics teachers lack
knowledge in formal assessment. To reinforce their significance in teaching and
learning enhancement, formal assessments have been continuously implemented
on-line during the COVID-19 pandemic in South Africa. Teachers need to collect
and evaluate learners’ understanding to come up with reasonable decisions on the
way forward (Liberman et al., 2020). Whilst formal assessment is incorporated in
government policy documents, there is inadequate evidence based on research to
confirm whether teachers implement assessments properly and adequately.
Furthermore, research indicates that, although several studies have been carried
out on teacher assessment practices, there has been limited research on teachers’
assessment literacy in South Africa. Additionally, it is indicated that a greater
number of teachers have inadequate proficiency in the implementation of formal
assessments and that there is a lack of proper guidance and support regarding
this aspect. There is thus an inadequacy of assessment knowledge and practice
amongst teachers. Mathematics teaching and learning has been found to be highly
insufficient in South Africa schools (Jojo, 2019). Papadikis et al. (2017) contend that
the adoption of mathematics teaching theory such as Realistic Mathematics
Education (RME) develops connection between mathematics and reality in
kindergarten learners. Teachers and learners interact so that learners can
construct reality and under complex mathematical problems. Papadikis et al.
(2021) further suggest that teacher use of smart mobile devices such as tablets lead
to effective teaching and meaningful learning in mathematics. Mathematics
applications (apps) expose learners to real life problems which enable them to
understand fundamental mathematical concepts. The comprehension of numbers
is more evident in early childhood learners who use tablets (Papadikis et al., 2018).
However, to check if there is meaningful mathematics learning, teachers must
possess knowledge of formal assessment practices. As a result, a study on how
teachers carry out assessments for teaching and learning enhancement is essential
(Kanjee & Mthembu, 2015). Therefore, what is the nature of formal assessment in
intermediate phase mathematics in schools? What are the most tested cognitive
levels in formal assessments? Consequently, the objective of this study is to
302
suggest strategies mathematics teachers could adopt to improve their teaching to

implement effective formal assessment.
The South African Department of Basic Education (DBE) is relevant because one
of the key imperatives of the Curriculum and Assessment Policy Statement
(CAPS) is to implement a valuable and functional assessment program which give
pertinent information to all role-players to enhance teaching and learning
procedures (DBE, 2011). The preceding section provides the introduction to the
research. The following sections are organised as follows: section 2 provides a
literature review, section 3 presents the methodology, section 4 provides the
results, section 5 provides discussion of the results, section 6 presents the
conclusion and section 7 presents a list of references. The upcoming section
discusses the literature review.
2. Literature review
This research is framed within constructivism theory based on the constructivist
perspective, an aspect that views knowledge as repeatedly formulated by human
beings in everyday life situations, not just as stipulated (Donald et al., 2014).
Additionally, the principal assertion of constructivism is that knowledge is not
imposed by external forces; instead, it is internally constructed by an individual.
Furthermore, reference is made to the key to constructivism being learner activity.
The aim is to understand how teachers teach and assess learners, building from
their prior knowledge. This will, in turn, lead learners to purposefully formulate
new, worthwhile concepts (Lombard &Themane, 2015). Two constructivism
theories and concepts related to teaching and learning of mathematics are
clarified, which will inform the development of the theoretical framework of this
research. These theories are Piaget’s cognitive constructivism (1953) and
Vygotsky’s social constructivism (1978). Cognitive constructivism theory calls for
teachers to deviate from the common practice of direct teaching and be facilitators
in teaching and learning. It is a view of learning suggesting that instead of
imparting knowledge, that is formulated beforehand, learners must be given
opportunities to make use of their own capabilities and skills to create their own
understanding (Paulsen & Dednam, 2016). Social constructivism theory is a
theory concerned mainly with the way cognitive development happens from ‘the
outside in’. Social constructivism theory emphasises the conception that cognitive
development happens within social connections. Furthermore, reference is made
to the theory that all better intellectual systems are a result of social interaction
(Vygotsky, 1978). Vygotsky’s theory proposes the key role of teachers and other
members in society in children’s acquisition of a particular measure of cognitive
development. The theory stresses socialisation for sustained development. The
mediator must perform the role of providing instructional support for the learners
so that they can become independent (Donald et al., 2014). Similarly, Paulsen and
Dednam (2016) mention that learning must be extended to the home and other
out of school settings. This can be accomplished through learner use of tablets to
develop their mathematical competence (Papadakis et al., 2016). At the same time,
the teacher still has a significant role to decide to choose the most effective, well-
informed potential mediator for the learners. This can also be attained through
scaffolding and student-to-student discussions (Abrie et al., 2016). However,
mathematics is usually viewed as challenging to teach, even under normal
303
circumstances, and the situation is currently worsened by the COVID-19

pandemic (Fritz et al., 2019).
Literature selected for review was derived from the gap identified, guided by the
research questions. In other words, the nature of formal assessments in the
intermediate phase in South African schools was precisely used to guide the
literature review.
2.1. The nature of formal assessments in intermediate phase mathematics in

South African schools
Formal assessment consists of School Based Assessment (SBA) and end of year
examinations (DBE, 2011). It is the duty of concerned teachers to monitor and
control assessment tasks and record them for subsequent promotional purposes.
Before being administered to learners, all formal assessment tasks must be
moderated to ensure control of quality and retention of proper measures.
Although the SBA element may have various formats, regarding mathematics,
assignments, projects, investigations, tests, and examinations are suitable, as
outlined in Table 1 below.
Table 1: Minimum requirements for formal assessment in the Intermediate Phase
Form of Term Term Term Term Number Weighting

Assessments 1 2 3 4 of tasks
per year
Tests 1 1 1 3
Continuous, Examinations 1 1
SBA Assignments 1 1 2
Investigation 1 1 75%
Project 1 1
End of the Total 2 2 2 2 8
year 1 25%
Examination
(DBE, 2011, p. 294).
As indicated in Table 1, formal assessments, in the form of SBA, have a significant

role in learners passing their grades in mathematics. Formal assessments count
75% towards the final grade mark. The forms of assessment are tests and
examinations, assignments, projects, and investigations. These forms of
assessment did not change, even during the COVID-19 pandemic, and currently
guide the program of assessment (DBE, 2021). The forms of assessment, according
to requirements, are explained below.
2.1.1. Tests and examinations

Tests and examinations are written individually by the learners. These tasks must
be accurately set to enable a clear indication of content mastery in mathematics by
the learners (DBE, 2011). Additionally, the main point is not merely to get a mark
to record, but to discover what learners have learned, and on what concepts they
need more practice. Borich (2014) mentions that tests precisely measure the skills
which are expected to be acquired by the learners. Moreover, authentic tests must
ask learners questions which will enable them to display their skills in real-life
304
situations. This implies that mathematics teachers must ensure that their methods
of instruction in the classrooms will, in turn, enable learners to display what was
taught in the real world. Borich (2014) further suggests The Test Blueprint (TTB),
which must complement test objectives and guarantees that teachers include all
the information crucial to a good test. TTB ensures that the test will cater for
different cognitive levels of learners. The table below illustrates the blueprint for
mathematics.
Table 2: Test blueprint for a unit on subtraction without borrowing
Comprehension
Application
Knowledge
Per cent
Content Outline
Total
1. The student will discriminate the
subtraction sign from the addition sign. 1 1 4%
2. The student will discriminate addition 2 2 8%

problems.
3. The student will discriminate correctly

solved subtraction problems from 4 4 16%
incorrectly solved subtraction problems.
4. The student will solve correctly single-
digit subtraction problems. 6 6 24%
5. The student will solve correctly
subtraction problems with double-digit 6 6 24%
numerators and single-digit denominators.
6. The student will solve correctly double-
digit subtraction problems. 6 6 24%
Total 3 4 18 25
Percent 12% 16 72 100%
% %
(Borich, 2014: 382).
The information in Table 2 implies that teachers must ensure that the six cognitive
levels of Bloom’s taxonomy are applied when setting tests and examinations. The
next paragraph explains the assignment as a form of assessment.
2.1.2. Assignment
An assignment is given to the learners as an individual task. This may be obtained
from past examination papers; but whatever, it must centre on challenging
content, as there is a variety of resources to refer to. It can be done at home, not
under class supervision (DBE, 2011). Borich (2014) suggests that assignments must
be given immediately after the lessons or activities to which they relate.
Furthermore, teachers must display the assignment in their classrooms, so that
learners who have missed information can always refer to the displays. The
following paragraph explains the use of projects as another form of assessment.
305
2.1.3. Projects
A project is an activity which extends learning beyond the classroom and
positions it in the real world (Coombs, 1995). Moreover, projects are employed to
evaluate a variety of abilities and capabilities. Projects should enable learners to
implement their mathematical concepts in practical situations. Through projects,
learners are expected to gather the data, analyse it thereafter, and draw conclusions (DBE,
2011, p.294). Gawe et al. (2016) point out that the project method is learner-centred
learning and highly based on the constructivist principle. This gives learners an
opportunity to work on their own in collecting relevant information required for
the project and be able to present it thereafter. Furthermore, the project method
assesses a variety of skills, at the same time integrating various activities like
planning, research, data analysis, and reporting. This is essential when
intermediate phase mathematics learners have data handling as a topic.
Moreover, Mays et al. (2016) acknowledge that a project widens the kinds of skills
needed by learners as they are assessed. Additionally, Borich (2014) confirms that
project-based learning conveys to learners the significance of the learning process,
aids them to set goals, and affords them opportunities to work co-operatively.
This implies that teachers must ensure that skills like research and presentations
are taught effectively before handing out projects for assessments. The paragraph
below explains the investigation as a formal task.
2.1.4. Investigation
An investigation is a formal task which can be employed to determine rules or
concepts. It can include connections of patterns, arriving at conclusions, and
identification of patterns. Rubrics with specific marks to be given per skill are used
to assess investigations. The skills come in various forms, such as organising and
recording, communicating ideas, calculations and generalising, and drawing a
conclusion. Teachers must take note that all tasks must accommodate all the
cognitive levels of learners. The forms of assessment used should also take the
suitability of learners’ ages into consideration. Tasks should thus be designed in
a manner that allows the subject content to be achieved and ensures that all the
aims and objectives are attainable. Moreover, suitable marking tools, for instance,
rubrics and memoranda, must be utilised (DBE, 2011). Nieuwoudt and Reyneke
(2016) mention a rubric as a valuable tool in assessing learners’ responses. They
assert that this is so because of the main elements that a rubric contains. Two of
the three main elements are outlined below.
2.1.5. Evaluative criteria

Evaluative criteria are used to distinguish between acceptable and unacceptable
responses. The criteria will differ according to the skills being tested. This suggests
that teachers must assist learners in developing skills through teaching and
learning (Nieuwoudt &Reyneke, 2016).
2.1.6. Quality definitions

Quality definitions are level descriptors which are used to specify the number of
points to be earned per specific skill. Complementing the use of rubrics is the work
of Elrod and Strayer (2015), which reveals that a rubric is an essential tool for
teachers to monitor the learners’ work. Furthermore, it can be used as a
mechanism to indicate cultural behaviours and practices in the classroom, hence
306
becoming an essential part of teachers’ assessment practices for learners. This is

so because working with rubrics both helps teachers and learners understand
competencies required and can be used to provide feedback. Formal assessment
tasks must cater to different levels of cognitive development as seen in Table 3
below.
Table 3: A range of cognitive levels to be catered for intermediate phase mathematics

teaching and learning
Cognitive Description of skill to be Examples
levels demonstrated
Knowledge • Straight recall 1. Write down the next three

• Estimation and numbers in the sequence:
(=25%) rounding off 107; 109; 111… [Grade 4]
• Identification and
correct use of formula 2. Determine the factors of
• Use of mathematical 44 [Grade 5]
facts
3. Write down the prime
• Appropriate use of
numbers that are factors of
mathematical
36 [Grade 6].
vocabulary.
Routine • Perform well-known 1.Determine the value of
procedures procedures y+5=15 [Grade 4]
• Simple applications and
(=45%) calculations 2.Use three different
• Derivations from given techniques of calculating
information 59910 [Grade 5]
• Identification and use of
3. Calculate 12/4+3/12-1/3.
correct formula
[Grade 6]
Complex • Complex calculations 1. Mandy is 6 years old and

procedures and higher-order Betty is 12 years old.
reasoning Determine the ratio
(=20%) • Investigations to between their ages. Write
describe rules and the ratio in simplest form.
relationships [Grade 4]
• Problems not based in
real-world contexts 2. Investigate the properties
• Conceptual of rectangles and squares
understanding and identify similarities and
differences. [Grade 5]
3. There are 20 sweets in the

packet. William and his
friends ate 2 fifths of the
sweets. How many sweets
are left? [Grade 6]
307
Problem- • Unseen non-routine 1. The sum of three

solving problems consecutive numbers is 29.
• Higher-order Find the numbers. [Grade 4]
(=10%) processing and
understanding 2. John divides a certain
required number by 16. He found an
• May require answer of 246. What is the
breaking down into number? [Grade 5]
constituent parts to
3. Busi has a bag containing
solve
six coloured balls: 1 blue, 2
red balls and 3 yellow balls.
She puts her hand in the bag
and draws a ball. What is
the chance that she will
draw a red ball? Write the
answer in the simplest
fractional form [Grade 6]
(DBE, 2011: 296
Intermediate mathematics teachers should adhere to this continuum of cognitive

levels when setting tests and examinations so that the unevenness of cognitive
development of learners is addressed. This implies that there must be accurate
moderation of assessments to guarantee that different cognitive levels of learners
are catered for. The ensuing paragraph discusses the moderation of assessments.
Despite the significant role meant to be played by effective implementation of

formative assessment in classrooms, the literature indicates various challenges
faced by teachers in this regard. Assessing learners has many facets. Additionally,
the procedure of making sense of learners’ mathematical reasoning and
explanation of approaches is more convoluted than can be assumed and poses
challenges to teaching and learning (Suurtamm et al., 2016).
3. Methodology
The study follows a quantitative approach, using a survey design. Quantitative
research is an approach which utilises numerical data in a structured and
empirical scheme. It makes use of a particular subgroup to derive its data;
subsequently, findings are generalised to the sector that is currently under
consideration (Maree & Pietersen, 2016). The aim of quantitative research is to
clarify trends amongst given factors in a particular study (Ivankova et al., 2016).
The survey design was employed for this research because of its appropriateness
to describe current conditions in the implementation of assessments in
intermediate phase mathematics teaching and learning. The researchers
conducted the document analysis from the primary schools because they offer
intermediate phase mathematics. A checklist (Appendix 1) was used to determine
whether the requirements of formal assessment implementation were met by
intermediate phase mathematics teachers.
308
3.1. Format of the checklist

Killen (2015) points out that checklists are useful for assessment in situations
where a very specific set of objective judgements needs to be made about learner
performance – in this case, in the implementation of formal assessments in
intermediate phase mathematics teaching and learning. The checklist utilised in
this study was constructed from the information provided in the Curriculum and
Assessment Policy Statement Grades 4-6 mathematics (DBE, 2011, pp. 294-296).
The checklist comprised three sections, namely:
• Section A: Biographical data with three items.
• Section B: Minimum Requirements for Formal Assessment: Intermediate
Phase Mathematics with seven items.
• Section C: Tests Cognitive Levels with Description Skills to be demonstrated
with sixteen items.
Document analysis was conducted in the school environment. Ethical issues were
considered by the researchers. Gasa and Mafora (2015) emphasise that if the
information is not openly accessible, written consent for the utilisation of the data
ought to be obtained from the possessors. The researchers obtained permission
through the consent form which was annexed to the letter addressed to the
principals and intermediate phase teachers. Document analysis was explained to
the participants before it was conducted. Purposive sampling, which is classified
under a non-probability sampling method, was utilised for document analysis.
Maree and Pietersen (2016) point out that this method of sampling is applied with
a clearly defined aim in mind – in this instance, with the purpose of obtaining
information on how formal assessments are implemented in intermediate phase
mathematics teaching and learning. Nine teachers were sampled for document
analysis, three from each of Grades 4 to 6. Their portfolios, which included
learners’ formal tasks for the year 2018, were analysed. The checklist’s Cronbach’s
alpha coefficient was 0.83, which suggested a moderate reliability for the scale.
3.2. Biographical details of the respondents

The following table represents the biographical the data of the teachers whose
formal assessment records were analysed.
Table 4: Biographical details of the teachers whose formal assessment records were
analysed N=9
Personal The % Respondents % Total
data According to Category
A1. Gender Male 4 (44.4%)
Female 5 (55.6%) 100
4 3 (33.3%)
A2. Grade 5 3 (33.3%)
6 3 (33.3%) 100
1 2 (22.2%)
A3. School 2 1 (11.2%)

Quintile 3 3 (33.3%) 100
5 3 (33.3%)
309
Analysis of the data in Table 4 indicates that three (44.4%) formal assessment
records of male teachers were analysed, whilst five (55.6%) formal assessment
records of female teachers were examined. In terms of representation according
to the gender of the teachers who agreed that their formal assessment records be
analysed, there was no significant difference. Further analysis of the data in the
table reveals that formal assessment records of all the three grades in the
intermediate phase were analysed. However, representation according to schools’
quintile was not evenly spread. Analysis of the data in the table discloses that
22.2% of the analysed formal assessment records were from quintile 1 schools,
11.1% were from quintile 2 schools, 33.3% were from quintile 3 schools, and the
remaining 33.3% were from quintile 5 schools. The researcher did not analyse
documents from quintile 4 schools because all teachers from this quintile who
were approached were not willing for their formal assessment records to be
analysed. They volunteered to participate only in the interviews. The results are
sufficient, as 80% of the schools in quintile 5 in the Lejweleputswa district were
represented, which gives a clear indication of how formal assessments are
implemented. The ensuing table presents, and gives an analysis of, the data
obtained for the minimum requirements for formal assessment in intermediate
phase mathematics as stipulated in the CAPS document.
4. Results
4.1. The minimum requirements for formal assessment in intermediate phase
mathematics
Table 5: Minimum requirements for Formal Assessment N=9

Forms of Assessment Mean Median Standard
Deviation
School-based B1 Tests 2.11 2.00 0.33
Assessment
(SBA) B2 Midyear 2.00 2.00 0.00
Examination
75% B3 Assignment 1.67 2.00 0.50
B4 Investigation 1.11 1.00 0.33
B5 Project 1.44 1.00 0.53
B6 Total 1.11 1.00 0.33
End of the year B7 End of the year 2.00 2.00 0.00
Examination Examination
25%
Analysis of the data in Table 5 reveals that teachers fully administer examinations
(M=2.00, MD=2.00, SD=0.00). Statistics show that the data is symmetrical, since
the skewness measure is zero. Furthermore, all formal assessment records that
were analysed indicated that teachers fully administer examinations as stipulated
in the CAPS document. This is because the standard deviation is zero, which
implies that there is no deviation from the mean. The mean value of 2 indicates
that the implementation of examinations had been achieved as per the rubric of
the checklist. Another revelation is that tests were wholly administered (M=2.11,
MD=2.00, SD=0.33). Statistics show that the data is positively skewed, since the
310
mean is higher than the median. Resultantly, the mean value indicates that the
implementation of the test was fully achieved according to the CAPS
requirements. With reference to assignments (M=1.67, MD=2.00, SD=0.50),
statistics show that this form of assessment is moderately achieved as it is not fully
done according to the CAPS requirements. The data is negatively skewed, since
the mean is smaller than the median. Investigation as a form of formal assessment
is not done accurately as per the CAPS requirements (M=1.11, MD=1.00, SD=0.33).
The mean value of 1.11 indicates that the implementation of investigation was not
achieved as per the rubric of the checklist. This implies that some of the teachers
whose records were analysed are not giving their learners investigative tasks. This
implies that teachers are not promoting critical and creative thinking in their
learners as suggested in CAPS. Statistics show that the data is positively skewed,
since the mean is higher than the median.
Additionally, statistics show that the project as a form of assessment is not

achieved at all (M=1.44, MD=1.00, SD=0.53). The mean of 1.44 is an explanation
of underachievement as per the rubric of the checklist. This implies that teachers
are not providing learners with opportunities through which they can express
their competencies of solving complex issues in daily occurrences. By virtue of
assignment, investigation, and project not being achieved, the total tasks per year
is resultantly not achieved (M=1.11, MD=1.00, SD=0.33). The mean value reveals
that the total number of tasks per year was not met. Statistics show that the data
is positively skewed, since the mean is higher than the median. The following
table serves to present and analyse cognitive levels and abilities tested in learners.
4.2. Cognitive levels and abilities tested in learners

Table 6 indicates levels and description of skills which should be demonstrated
by Grades 4 to 6 learners in mathematics. Analysis of the data in Table 4.26 below
shows that there is greatest achievement in the testing of cognitive skills such as
straight recall (M=2.11, MD=2.00, SD=0.33). The mean value of 2.11 is a clear
indication of achievement as per the checklist rubric. Statistics show that the data
for this item is positively skewed, since the mean is higher than the median.
Furthermore, testing of cognitive skills such as performing well-known
procedures, simple applications and calculations, unseen-non routine procedures,
and breaking down problems into constituent parts are fully mastered as
stipulated in the CAPS document (M=2.00, MD=2.00, SD=0.00). This implies that
these skills are taught effectively, which in turn makes the broad aims of the
subject achievable (DBE, 2011: 295). The standard deviation is zero, which implies
that there is no deviation from the mean. The mean value of 2 indicates that the
testing of cognitive levels and abilities of learners has been achieved as per the
rubric of the checklist. Statistics stipulate that the data for these skills is
symmetrical, since their skewness measure is zero.
The following table presents and analyses the data on testing of cognitive levels.
Regarding the testing of cognitive skills, such as estimation and appropriate
rounding off and use of mathematical facts, the statistical results are similar,
which indicate that they were moderately achieved (M=1.89, MD=2.00, SD=0.33).
Their standard deviations are not very far from the mean. The implication,
311
therefore, is that knowledge as a cognitive level skill is tested as stipulated in the

CAPS document, hence making the aims and objectives of the subject achievable.
However, with reference to the testing of cognitive skills, such as problem solving
and investigations to describe rules and relationships, their statistical results are
similar, which indicates that there is high underachievement (M=1.11, MD=1.00,
SD=0.33). The mean of 1.11 for these skills reveals that their testing is not being
done according to the CAPS requirements as per checklist rubric. This implies that
teachers are not exposing learners to complex procedures which are designed to
improve their higher-order reasoning. Moreover, the learners do not have
opportunities to solve unseen non-routine procedures – as a result, this impedes
their conceptual understanding of the subject. The data for these statistics is
positively skewed, since the means are higher than their medians.
Table 6: Testing of cognitive levels with description of skills N=9

Checklist items Mean Median Standard
Deviation
C1 Estimation and appropriate rounding off of 1.89 2.00 0.33
numbers.
C2 Straight recall. 2.11 2.00 0.33
C3 Identification and direct use of correct 1.78 2.00 0.33
formula.
C4 Use of mathematical facts. 1.89 2.00 0.33
C5 Appropriate use of mathematical 1.33 1.00 0.50
vocabulary.
C6 Perform well known procedures. 2.00 2.00 0.00
C7 Simple applications and calculations, which 2.00 2.00 0.00
might involve many steps.
C8 Derivation from given information may be 1.44 1.00 0.52
involved.
C9 Identification and use after changing the 1.67 2.00 0.50
subject of correct formula, generally similar
to those encountered in class.
C10 Problems involving complex calculations 1.11 1.00 0.33
and/or higher order reasoning.
C11 Investigations to describe rules and 1.11 1.00 0.33
relationships; there is often not an obvious
route to the solution.
C12 Problems not based on real world context 1.33 1.00 0.50
could involve making significant
connections between different
representations.
C13 Conceptual understanding. 1.89 2.00 0.33
C14 Unseen, non-routine problems (which are 2.00 2.00 0.00
not necessarily difficult).
C15 Higher order understanding and processes 1.67 2.00 0.50
are often involved.
C16 Might require the ability to break the 2.00 2.00 0.00
problem down into its constituent parts.
312
The data shows that testing of the appropriate use of mathematical vocabulary is
not achieved (M=1.33, MD=1.00, SD=0.50). The mean of 1.33 indicates that the
testing of this skill does not meet the CAPS requirements as per the checklist
rubric. This denotes that teachers are not developing the correct use of the
language of mathematics, as stated under the specific skills which must be
developed in mathematics (DBE, 2011:8). This, in turn, affects the achievability of
aims and objectives of the subject. Ultimately, the data shows that derivation from
given information as a cognitive skill is not tested (M=1.44, MD=1.00, SD=0.52).
The mean of 1.44 indicates that this skill is not tested according to the CAPS
requirements as per the checklist rubric. This implies that teachers are not
exposing learners to different formulas applicable in mathematics, as stated under
routine procedures as a cognitive skill to be taught. This means that learners
cannot identify and use other mathematical formulas they may encounter – other
than those used or taught in their classrooms. The following table presents and
analyses the data on the most tested cognitive levels in formal assessments.
Table 7: The most tested cognitive levels in formal assessments N=9

Cognitive Levels Mean Median Standard
Deviation
D1 Knowledge 1.80 1.80 0.24
D2 Routine procedures 1.78 1.75 0.63
D3 Complex procedures 1.36 1.25 0.22
D4 Problem-solving 1.89 2.00 0.17
Analysis of the data in Table 7 shows that the cognitive level that is most
developed and tested in learners is problem-solving, although it is moderately
tested (M=1.89, MD=2.00, SD=0.17). The data is negatively skewed, because the
mean is lower than the median. The mean of 1.89 confirms a moderate
achievement according to CAPS requirements as per checklist rubric. This finding
means that learners can moderately solve non-routine problems which are not
necessarily difficult, which might lead to their understanding of word sums. The
standard deviation is nearer to the mean, indicating that the documents analysed
revealed similar information.
The second most developed and tested cognitive level is knowledge, which is also
moderately implemented in formal assessment (M=1.80, MD=1.80, SD=0.24). The
mean of 1.80 indicates that this cognitive level is moderately achieved according
to the stipulated requirements in the CAPS and as per checklist rubric. This
suggests that learners moderately round off and recall some of the mathematical
facts. Statistics indicate that the data is normally distributed, since the mean is
equal to the median. The third cognitive level which is promoted and tested in the
learners is routine procedures. This cognitive level is also moderately tested
(M=1.78, MD= 1.75, SD=0.63). The implication of this finding is that learners are
taught well-known procedures and other simpler calculations as stipulated in the
CAPS document. Statistics indicate that the data is positively skewed, since the
mean is higher than the median. However, the data reveals that development and
testing of complex procedures in learners is not being done (M=1.36, MD=1.25,
SD= 0.17). This mean indicates that teachers are neglecting the development of
313
learners in this cognitive level. Therefore, teachers are not developing the abilities
of learners in handling complex mathematical procedures as per the requirements
of CAPS. This implies that learners lack the ability to break down mathematical
problems into different factors or constituent parts. In summary, document
analysis shows that minimum requirements for formal assessments in
intermediate phase mathematics are not being met as stipulated in the CAPS
document.
5. Discussion
Startlingly, it has been proven that learners were not actively engaged in the
implementation of some of the assessment requirements in intermediate phase
mathematics teaching and learning. A quantitative analysis of the checklist shows
that investigations and projects are not implemented as forms of assessment in
mathematics – rather, learners were given a test instead of a project or
investigation. Resultantly, learners lack critical and creative thinking, which must
be triggered by active engagement and exploration. This finding implies that
teachers are not promoting cooperative learning, which is advocated by the CAPS.
Cooperative learning is a set of instructional strategies in which learners work in
mixed ability groups to reach specific cognitive and social development objectives
(Eggen & Kauchak, 2016). Additionally, co-operative learning provides learners
with an opportunity of working together and makes certain that every member of
the group has a chance to participate. Moreover, it encourages learners to act as
learning resources for one another (Gawe et al., 2016). Vygotsky (1978) also
underpins this idea through the role of social interaction. In mathematics teaching
and learning, learners can work together on projects – for example in data
handling projects – collecting, organising, representing, analysing, interpreting,
and reporting the data. They can also work together, regarding space and shape,
to construct 3-D shapes using mathematics apps as suggested by Papadakis et al.
(2018). Borich (2014) discusses some of the outcomes of co-operative learning,
which are attitudes and values, pro-social behaviour, alternative perspectives and
viewpoints, and higher thought and processes. People’s values and attitudes are
modelled by interacting with society through the exchange of information.
Learners can achieve this by working in groups or in pairs, sharing ideas and
exchanging information. This plays a crucial role in shaping their values and
attitudes, which, in turn, they need to deepen their independent thought (Borich,
2014). This idea is endorsed by Vygotsky (1978) who mentions the significance of
social interaction in cognitive development. Classrooms are now a significant
medium in which to reinforce pro-social behaviours, as a result of the high volume
of working parents or guardians. Therefore, teachers must plan for and
implement co-operative learning programmes to bring learners together. This
implies that teachers must come up with tasks and activities which will promote
working together on the part of learners (Borich, 2014). Similarly, teachers must
create learning experiences which give learners opportunities for working
cooperatively in interesting, challenging, and open-ended tasks such as projects
and investigations (Killen, 2015). Cooperative learning furnishes the context in
which several views and ideas can be exchanged (Borich, 2014). This is closely
linked to participative learning, in which learners are motivated to state their
views on the subject matter. It rests on the premise that learning takes place when
314
negative criticism does not exist in class (Vakalisa, 2016). This is applicable when
learners work together in projects and assignments which develop critical
thinking skills which improve mathematics competence. Furthermore,
cooperative learning is associated with outstanding academic achievement of
learners. It improves learners’ critical-thinking and problem-solving skills. Higher
thought process cannot occur without an amalgamation of attitudes and values,
prosocial behaviour, viewpoints, and integrated identity. This implies that
teachers must come up with tasks and activities which will stimulate learners’
higher thought processes – higher-order thinking must be stimulated by complex
thinking tasks. This, in turn, implies that teachers must engage learners in, for
example, research and problem-solving skills, which encourage high-order
thinking (Borich, 2014).
Acclaiming the idea of the aforementioned outcomes are the ideas of Gawe et al.
(2016), who emphasise the benefits of co-operative learning. They mention learner
achievement and social consequences as some of the noticeable benefits.
Outstanding achievement has been reported in the classrooms where co-operative
learning takes place. Furthermore, higher-order concepts can be taught effectively
through co-operative learning. As stated by Gawe et al. (2016), “The expression
that ‘two heads are better than one’ suggests the superiority of ideas that emerge
when more than one person is engaged in a project…some of the complex tasks
that learners are given to investigate” (p. 267). Additionally, co-operative learning
can contribute to integrating learners into networks of peer social relationships
which, in turn, assist them with constructive conflict resolution leading to
academic performance. Therefore, teachers must ensure that every learner has a
chance to participate in a group (Gawe et al., 2016). Sustaining this idea is
Vygotsyk’s (1978) work, through mediation and the Zone of Proximal
Development (ZPD), where cognitive mediation will take place. Consequently,
learners’ thinking competencies can be lessened if not directed to comprehend on
Bloom’s level of thinking (Jacobs, 2016). The emphasis of the level of
understanding is underpinned by one of mathematics’ specific aims – to establish
a profound understanding of concepts to have a logic of mathematics as a subject
(DBE, 2011). This aim is not achieved according to this study. Additionally, the
checklist data confirm the issue of language challenges, which affects testing of
cognitive skills. Teachers are not making use of appropriate mathematical
language when testing learners for formal assessments.
6. Conclusion
The research has contextual limitations because it was conducted in primary
schools with different circumstances of working conditions; therefore, the results
cannot be generalised because schools differ in contextual factors. The findings of
the study confirm that formal assessments are inadequately implemented.
Mathematics is usually viewed as a challenging subject to teach, even under
normal circumstances, and the situation is currently worsened by the COVID-19
pandemic. Therefore, teachers should be assisted by knowledgeable colleagues
and subject advisors in adopting and using assessment. Professional development
must be a significant element that seeks to establish the constructive use of
formative assessment. Three aspects which relate to assessments are the nature of
tasks and materials to support teachers’ use of formative assessments;
315
professional development that supports changes in teaching practice; and

classroom observations with a formative assessment focus. There must be a close
examination of the classroom observation tools that are utilised to enable teachers
to ascertain the capacity of the tool to give valid feedback on formal assessment
tasks. Furthermore, classroom observation assists teachers to improve their
assessment strategies when they are given feedback. Feedback from classroom
observations supports teachers in identifying areas of formal assessments in
which they need to improve to enhance the teaching and learning of mathematics.
Teachers need to promote practical learning, which can be achieved using
portfolios and oral presentations to evoke deeper understanding of mathematical
concepts. Concept mapping, linking connections involving related mathematical
ideas, must be utilised to achieve improvements in critical and creative thinking
in investigations and problem-solving tasks. Moreover, teachers are encouraged
to adapt the instructional method to promote the utilisation of assessment to
ascertain learners’ misconceptions. Subsequent teaching and learning alternatives
must be drawn from these misconceptions and, consequently, accord learners
who misinterpreted the concept(s) another chance of achievement (Chigonga,
2020). Although, South Africa is a developing country, with several socio-
economic problems, the Department of Basic Education should supply tablets to
schools to enhance the teaching and learning of mathematics. Research indicates
that the use of tablets in mathematics teaching and learning improves competence
in early childhood education.
7. References
Abrie, M., Blom, N., & Fraser, B. (2016). Theoretical foundations. In M. Jacobs, N. C. G.
Vakalisa, & N. Gawe (Eds.), Teaching-Learning Dynamics. (pp. 37). Pearson
Education.
Borich, G. D. (2014). Effective teaching methods: Research-based practice. Pearson.
Chigonga, B. (2020). Formative assessment in mathematics education in the twenty-first
century, theorizing STEM education in the 21st century. IntechOpen, 1-10.
http://dx.doi.org/10.5772/intechopen
Coombs, B. (1995). Successful teaching: A practical handbook. Heinemann.
Department of Basic Education, (2011). Curriculum and assessment policy statement, grades
4-6 mathematics. Government Printers. https://www.education.gov.za/
Portals/0/CD/National%20Curriculum%20Statements%20and%20Vocational/
CAPS%20IP%20%20MATHEMATICS%20GR%204-6%20web.pdf?ver=2015-01-
27-161430-553
Department of Basic Education, (2021). Intermediate phase annual teaching plans (ATPs),
mathematics. Government Printers. https://www.education.gov.za/
Curriculum/IntermediatePhaseATPs2021.aspx
Donald, D., Lazarus, S., & Moolla, N. (2014). Educational psychology in social context: Eco
systematic applications in South Africa. ABC Press.
Eggen, P. D, & Kauchak, D. P. (2016). Educational psychology: Windows on classrooms.
Pearson.
Elrod, M., & Strayer, J. F. (2015). Using an observational rubric to facilitate change in
undergraduate classroom norms. In C. Suurtamm & A. McDuffie (Eds.), Annual
Perspectives in Mathematics Education 2015: Assessment to Enhance Teaching and
Learning (pp. 87-96). National Council of Teachers of Mathematics.
316
Fritz, A., Haase, V. G., & Rasanen, P. (2019). International handbook of mathematical learning
difficulties: From the laboratory to the classroom. https://doi.org/10.1007/978-3-319-
97148-3
Gasa, V., & Marofa, P. (2015). Using secondary sources of data. In C. Okeke & M. van Wyk
(Eds.), Educational research: An African approach. (pp. 355-370). Oxford University
Press.
Gawe, N., Jacobs, M., & Vakalisa, N.C.G. (2016). Learner-centred methods. In M. Jacobs,
N. C. G. Vakalisa & N. Gawe N. (Eds.), Teaching-learning dynamics. (pp. 250-281).
Pearson Education.
Ivankova, N.V., Creswell, J.W., & Plano Clark, V.L. (2016). Foundations and approaches
to mixed-methods research. In K. Maree (Ed.), First steps in research. (pp. 306-
336). Van Schaik.
Jacobs, M. (2016). Aims and objectives. In M. Jacobs M., N. C. G. Vakalisa & N. Gawe
(Eds.), Teaching-learning dynamics. (pp. 105-137). Pearson Education.
Jojo, Z. (2019). Mathematics education system in South Africa. IntechOpen,
https//doi.org/10.5772/intechopen.85325
Kanjee, A., & Mthembu, J. (2015). Assessment literacy of foundation phase teachers: An
exploratory study. South African Journal of Childhood Education, 5(1), 142-
168. https://doi.org/10.4102/sajce.v5i1.354
Killen, R. (2015). Teaching strategies for quality teaching and learning. Juta.
Lombard, K., & Themane, M. (2015). Contextualising teaching practice as a component of
teacher education. In M. J. Taole (Ed.), Teaching practice, perspectives and frameworks.
(pp. 11-24). Van Schaik.
Liberman, J. Levinin, J., & Luna-Bazaldua, D. (2020). Are students still learning during
COVID-19? formative assessment can provide the Answer. World Bank Blogs.
https://blogs.worldbank.org/education/are-students-still-learning-during-
covid-19 formative-assessment-can-provide-answer
Maree, K., & Pietersen, J. (2016). The quantitative research process. In K. Maree (Ed.), First
steps in research. (pp. 162-172). Van Schaik.
Mays, T., Grosser, M., & de Jagger, L. (2016). Getting practical: A guide to teaching and
learning. Oxford University Press.
Nieuwoudt, S., & Reyneke, M. (2016). Assessment. In M. Jacobs, N. C. G. Vakalisa & N.
Gawe (Eds.), Teaching-learning dynamics. Pearson Education.
Papadakis, S., Kalogiannakis, M., & Zaranis, N. (2016). Comparing tablets
and PCs in teaching mathematics: An attempt to improve mathematics
competence in early childhood education. Preschool and Primary Education,
4(2), 241-253. https://doi.org/10.12681/ppej.8779
Papadakis, S., Kalogiannakis, M., & Zaranis, N. (2018). The effectiveness of
computer and tablet assisted intervention in early childhood students’
understanding of numbers. An empirical study conducted in Greece. Education
and Information Technologies, 23(5), 1849–1871.
https://www.learntechlib.org/p/191703/
Paulsen, R., & Dednam, A. (2016). Challenges in Mathematics: Mathematical Literacy and
Numeracy. In E. Landsberg, D. Kruger & E. Swart (Eds.), Addressing barriers to
learning: A South African perspective. (pp. 241-262). Van Schaik.
Piaget, J. (1953). The origin of intelligence in the child. Routledge and Kegan Paul.
Reddy, C., Le Grange, L, Beets, P., & Lundie, S. (2015). Quality assessment in South African
schools. Juta.
317
Suurtamm, C., Thompson, D., Kim, R., Moreno, L., Sayac, N., Schukajlow, S., Silver, E.,
Ufer, S., & Vos, P. (2016). Assessment in mathematics education: Large scale assessment
and classroom assessment. Springer.
Vakalisa, N. C. G. (2016). Participative teaching. In M. Jacobs, N. C. G. Vakalisa & N. Gawe
(Eds.), Teaching-learning dynamics. (pp. 39-71). Pearson Education.
Vygotsky, L. (1978). Mind in society: The development of higher mental processes. Harvard
University Press.
William, D. (2011). What is assessment for learning? Studies in Educational Evaluation,
37(1), 3-14. https://doi.org/10.1016/j.stueduc.2011.03.001
318
Appendix 1
Checklist for Formal Assessments
Checklist Number
SECTION A: Biographical Data
A1. Indicate gender of the teacher.

Male 1
Female 2
A2. Grade: 4 5 6 (choose only one grade per checklist)

1 2 3
A3. School quintile

1 2 3 4 5
SECTION B: Minimum Requirements for Formal Assessment: Intermediate Phase

Mathematics
Forms Minimum Number
Achievement 3
of requirements of tasks
Achieved 2
Outstanding
Achieved 1
assessment per term per year
Not
Weighting
T T T T
1 2 3 4
1 2 3
B1.Tests 1 1 1 3
SBA 1 2 3
B2.Examinations 1
1 2 3
B3.Assignment 1 1 2
75% 1 2 3
B4.Investigation 1 1
1 2 3
B5.Project 1 1
1 2 3
End of the B6.Total 2 2 2 2 8
year B7. Examination 1 25% 1 2 3
Examination
319
For office use only

Key
1 Not achieved
2 Achieved
3 Outstanding
achievement
SECTION C: Tests Cognitive Levels
Cognitive Description of skills Comments
Not achieved
levels to be demonstrated
Outstanding
achievement
Achieved
C1. Estimation and 1 2 3
appropriate rounding
Knowledge off of numbers
(=25%) C2. Straight recall 1 2 3
C3. Identification and 1 2 3

direct use of correct
formula
C4. Use of 1 2 3
mathematical facts
C5. Appropriate use 1 2 3
of mathematical
vocabulary
C6. Perform well- 1 2 3
Routine known procedures
procedures C7. Simple 1 2 3
(=45%) applications and
calculations, which
might involve many
steps
C8. Deriviation from 1 2 3
given information
may be involved
C9. identification and 1 2 3
use after changing the
subject) of correct
formula generally
similar to those
encountered in class
C10.Problems 1 2 3
involving complex
calculations and/or
higher order
Complex reasoning
procedures C11. Investigations to 1 2 3
(=20%) describe rules and
relationships- there is
320
often not an obvious

route to the solution
C12.Problems not 1 2 3
based on real world
context-could involve
making significant
connections between
different
representations
C13.Conceptual 1 2 3
understanding
C14.Unseen, non- 1 2 3
routine problems
(which are not
Problem necessarily difficult)
solving C15.Higher order 1 2 3
(=10%) understanding and
processes are often
involved
C16.Might require the 1 2 3
ability to break the
problem down into
its constituent parts
Consent Form
I, the undersigned hereby agree to participate in the research on the Implementation of

Formal Assessments in the Intermediate Phase Mathematics as foundation of teaching and
learning enhancement in Lejweleputswa district.
___________________________ ___________________
Signature of the Educator Date
PUBLISHER International Journal of Learning, Teaching and
Society for Research and Knowledge Management Educational Research
Port Louis
Republic of Mauritius
www.ijlter.org The International Journal of Learning, Teaching
and Educational Research is an open-access
Chief Editor journal which has been established for the dis-
Dr. Antonio Silva Sprock, Universidad Central de semination of state-of-the-art knowledge in the
Venezuela, Venezuela, Bolivarian Republic of field of education, learning and teaching. IJLTER
welcomes research articles from academics, ed-
Editorial Board
ucators, teachers, trainers and other practition-
Dr. Fatima Zohra Belkhir-Benmostefa
ers on all aspects of education to publish high
Dr. Giorgio Poletti
quality peer-reviewed papers. Papers for publi-
Dr. Chi Man Tsui
cation in the International Journal of Learning,
Dr. Fitri Suraya Mohamad
Teaching and Educational Research are selected
Dr. Hernando Lintag Berna
through precise peer-review to ensure quality,
Dr. Charanjit Kaur Swaran Singh
originality, appropriateness, significance and
Dr. Abu Bakar
readability. Authors are solicited to contribute
Dr. Eglantina Hysa
to this journal by submitting articles that illus-
Dr. Mo'en Salman Alnasraween
trate research results, projects, original surveys
Dr. Hermayawati Hermayawati
and case studies that describe significant ad-
Dr. Selma Kara
vances in the fields of education, training, e-
Dr. Michael B. Cahapay
learning, etc. Authors are invited to submit pa-
Dr. Bunmi Isaiah Omodan
pers to this journal through the ONLINE submis-
Dr. Vassiliki Pliogou
sion system. Submissions must be original and
Dr. Meera Subramanian
should not have been published previously or
Dr. Muhammad Kristiawan
be under consideration for publication while
Dr. Wahyu Widada
being evaluated by IJLTER.
Dr. Som Pal Baliyan
Dr. Reem Khalid Abu-Shawish
Dr. Froilan Delute Mobo
Dr. Mohamed Ali Elkot
Dr. Anabelie Villa Valdez
Mr. Teody Lester Verdeflor Panela

Vol 20 No 8 August 2021

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Vol 20 No 8 August 2021

Uploaded by

Copyright:

Available Formats

IJLTER.

Society for Research and Knowledge Management

The International Journal of Learning, Teaching and Educational

Aims and Objectives

The main objective of this journal is to provide a platform for educators,

Indexing and Abstracting

The International Journal of Learning, Teaching and Educational

We are very happy to publish this issue of the International Journal of

The International Journal of Learning, Teaching and Educational

We are grateful to the editor-in-chief, members of the Editorial Board

The Editorial Board is composed of renowned people from across the

We will endeavour to ensure the reputation and quality of this journal

Editors of the August 2021 Issue

Online Design Thinking Problems for Enhancing Motivation of Gifted Students....................................................... 91

Teaching-Learning Strategies to Production Planning and Control Concepts: Application of Scenarios to

The Measurement of Knowledge Construction in A Course of Diagnostic Evaluation of Learning Disorders in

The Effectiveness of a Self-Questioning Strategy at Developing Academic Achievement and Critical-Thinking

International Journal of Learning, Teaching and Educational Research

Washback Impact of the MUET: The Before and

Najihah Mahmud and Nor Hazwani Munirah Lateh

Ariezal Afzan Hassan, Amaal Fadhlini Mohamed

intended to add more insights to the less explored areas of washback,

Keywords: washback; language assessment; high-stake test; language

In a doctoral study by Mahmud (2018), it was hypothesised that the students’

This study investigated the Malaysian students’ perception of the MUET by

students’ perceptions using a questionnaire and through interviews. Two groups

4.1 Students’ Perception of the MUET

Table 3: The MUET in general

1.12 More speaking activities should be conducted in MUET 4.29 .629

4.1.2 Perception of Test Importance

Table 4: Test Importance

2.4 It is very important for my future undertakings that I do 4.38 .816

4.1.3 Perception of the Consequences of Exam Failure

Table 5: Consequences of Exam Failure

ranking across the grouping in an ascending manner being as follows: Low

4.2 Relationship between the students’ perceptions and washback impact on

Table 6: Language Learning Strategies

The findings revealed that the students reported a moderate-to-high frequency of

To determine the relationship between the students’ perception of the test

4.3 The before and after effect of the MUET

“Even when we enter the university, we still have to learn grammar

That way, we could memorize it better. Well actually we memorized the

“My teacher last semester conducted some sort of game. In my opinion,

5.2 Language Learning Strategies

5.3 The before and after effect of the MUET

reservation when speaking in English, it was encouraging to see a slight self-

5.4 Limitation of the study

5.5 Implications of the study

International Journal of Learning, Teaching and Educational Research

The Efficacy of Promoting Financial Literacy

Khoo Yin Yin

Abstract. It is important to educate financial literacy to pre-service

Keywords: financial literacy; Economics pre-service teachers; MOOCs;

Most of the MOOCs in Malaysia offer academic courses; nevertheless, a minority

Financial literacy has been the only cross-curricular element in Malaysia's

Many researchers only focused on pedagogical skills and on the content

In addition, prior research showed evidence that financial literacy can be

2. To investigate the correlation among financial-literacy variables, collaborative

3. The Literature review

These cognitive processes determine what information is to be drawn upon in the

Multimedia Sensory Working Memory Long-Term

Selecting Organizing Pictoral

Figure 1: Conceptual Framework: Mayer Multimedia Learning Theory

Pearson 1 0.118* 0.216 -0.240 0.227**