Proceedings of The 18th Latin American Conference On Learning Technologies (LACLO 2023)

Lecture Notes in Educational Technology
Santiago Berrezueta Editor
Proceedings of the
18th Latin American
Conference
on Learning
Technologies
(LACLO 2023)
Series Editors
Ronghuai Huang, Smart Learning Institute, Beijing Normal University, Beijing, China
Kinshuk, College of Information, University of North Texas, Denton, TX, USA
Mohamed Jemni, University of Tunis, Tunis, Tunisia
Nian-Shing Chen, National Yunlin University of Science and Technology, Taiwan,
China
J. Michael Spector, University of North Texas, Denton, TX, USA
The series Lecture Notes in Educational Technology (LNET), has established itself
as a medium for the publication of new developments in the research and practice of
educational policy, pedagogy, learning science, learning environment, learning resources
etc. in information and knowledge age, – quickly, informally, and at a high level.
Abstracted/Indexed in: Scopus, ACM Digital Library, ERIC, INSPEC, Norwegian
Register for Scientific Journals and Series, SCImago
Santiago Berrezueta
Editor
Proceedings of the 18th Latin

American Conference
on Learning Technologies
(LACLO 2023)
Editor
Santiago Berrezueta
Computer Science Education
Technical University of Munich
München, Germany
ISSN 2196-4963 ISSN 2196-4971 (electronic)

ISBN 978-981-99-7352-1 ISBN 978-981-99-7353-8 (eBook)
https://doi.org/10.1007/978-981-99-7353-8
© The Editor(s) (if applicable) and The Author(s), under exclusive license
to Springer Nature Singapore Pte Ltd. 2023
This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether
the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of
illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar
methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors, and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the
editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors
or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in
published maps and institutional affiliations.
This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd.
The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721, Singapore
Paper in this product is recyclable.

Foreword
The intersection of technology and education in Latin America has always been a
vibrant nexus of innovation and potential. LACLO (Conferencia Latinoamericana de
Tecnologías de Aprendizaje—Latin American Conference on Learning Technologies)
stands at the forefront of this movement. The commitment to advancing education in
the digital age becomes even more palpable as we gather for the 18th LACLO, which is
recognized as one of the most crucial events in Latin America for experts in education,
computing, and technology. The 35 papers accepted for LACLO 2023 and presented
in this Proceedings book embody the very essence of this commitment in the region
in a year in which major changes in education are being brought about by advances in
artificial intelligence (AI).
The scientific debate at this conference delves into the profound implications of geo-
metric road design education and the omnipresent influences of digital technologies in
daily life. The complexities surrounding cell phone addiction and digital identity man-
agement highlight the modern learner’s relationship with their digital tools (a connection
that is both intricate and impactful). The debate smoothly moves onto the realm of gam-
ification. As educational landscapes evolve, the appeal of play and competition grows
stronger. This sentiment resonates in the explorations of platforms like TikTok for lin-
guistic enhancement and Learning Analytics Dashboards for fostering engagement. A
central theme across this collection is the human-centric approach in education. This is
captured in gender inclusion through art, experiential learning in software engineering,
and the promise of personalization through virtual assistants. Furthermore, the presence
of ChatGPT with GPT-3.5 and GPT-4 in the debate emphasizes the pivotal role of AI
in reshaping the contours of modern pedagogy. As we reflect on global challenges, the
discourse inevitably turns to resilience in education. The discussions on the role of ped-
agogical practices during complex times, coupled with insights into analytics, offer a
roadmap for educators navigating the challenges of our era.
At its core, this Proceedings book, with LACLO 2023 as a backdrop, is a record of
the dedication of the academic community and the unyielding spirit of Latin American
innovators. To every contributor, thank you for enriching this narrative. To our readers
may these insights shape your educational journeys ahead.
Warm regards,
Carlos Alario-Hoyos
Preface
The 18th Latin American Conference on Learning Technologies (LACLO 2023) took
place in Cuenca, Ecuador, from October 18 to 20, 2023. It was hosted at the central
campus of the University of Cuenca (UCuenca).
When UCuenca was chosen to host LACLO 2023, we recognized it as a golden
opportunity to not only bring this prestigious conference closer to the educational com-
munity but also to curate a unique experience for its 18th anniversary. LACLO stands
as one of Latin America’s premier conferences, dedicated to innovation and research in
learning technologies. This event brings together researchers, educators, professionals,
and students from around the globe, fostering the exchange of research findings and
promoting academic endeavors in technology-enhanced learning.
The 18th edition of LACLO attracted submissions from 362 authors spanning 17
countries. Every manuscript underwent similarity checks against previous works and
was subject to a rigorous double-blind peer review. We ensured the quality of the review
process by assigning a minimum of three independent reviewers to each article. This
meticulous process was facilitated by the academic expertise of around 52 seasoned
researchers from various Latin American countries. Out of the submissions, 35 papers
made the cut, resulting in an acceptance rate of just below 32%. A notable achievement
for this edition was our collaboration with Springer, allowing us to reach a broader
audience and offering increased benefits to our contributing academic community.
The organizers of LACLO 2023 extend special recognition to UCuenca and CEDIA
for their generous Administrative and Financial Sponsorship and the LACLO commu-
nity for entrusting the UCuenca Dirección de Innovación Educativa and the Computer
Science Department (DCC) with the organization of the conference’s 18th edition.
The invaluable support from these organizations plays a pivotal role in advanc-
ing research, fostering collaboration, and disseminating results in technology-enhanced
learning and quality education.
Additionally, heartfelt gratitude goes to everyone who played a part in making this
conference a success:
• The authors of the manuscripts, keynote speakers, and workshop presenters.
• Members of the scientific committee
• Members of the local organization committee
• The program and diffusion chairs
Your collective contributions have been instrumental to the conference’s success.
October 2023 Jorge Maldonado Mahauad

Otto Parra Gonzalez
Ana Gabriela Núñez
Gabriela Parra Robles
Santiago Berrezueta
Organization
General Chair
Jorge Maldonado Universidad de Cuenca, Ecuador
Steering Committee
Otto Parra Gonzalez Universidad de Cuenca, Ecuador

Ana Gabriela Núñez Universidad de Cuenca, Ecuador
Gabriela Parra Universidad de Cuenca, Ecuador
María Fernanda Granda Universidad de Cuenca, Ecuador
Priscila Cedillo Universidad de Cuenca, Ecuador
Antonio Silva Universidad Central de Venezuela, Venezuela
Julio César Ponce Universidad Autónoma de Aguascalientes,
México
Ismar Frango Universidade Presbiteriana Mackenzie, Brazil
Cristian Cechinel Federal University of Pelotas, Brazil
Mario Chacón Instituto Tecnológico de Costa Rica, Costa Rica
Manuel Ibarra Universidad Nacional Micaela Bastidas de
Apurimac, Perú
Organizing Committee
Karina Abad Universidad de Cuenca, Ecuador

Doris Suquilanda Universidad de Cuenca, Ecuador
Lilian Tapia Universidad de Cuenca, Ecuador
Fabian Vélez Universidad de Cuenca, Ecuador
Gabriela Parra Universidad de Cuenca, Ecuador
Sonia Balcázar Universidad de Cuenca, Ecuador
Ivón Vela Universidad de Cuenca, Ecuador
María Teresa Arteaga Universidad de Cuenca, Ecuador
x Organization
Program Committee
Alejandro Artopoulos Universidad de San Andrés, Argentina

Alejandra B. Lliteras Universidad Nacional de La Plata, Argentina
Ana Gabriela Nuñez Avila Universidad de Cuenca, Ecuador
Andres Auquilla Universidad de Cuenca, Ecuador
Antonio Silva Sprock Universidad Central de Venezuela, Venezuela
Beatriz Angélica Toscano Universidad Autónoma de Nayarit, Mexico
Cesar A. Collazos Universidad de Cauca, Colombia
Cristian Cechinel Federal University of Pelotas, Brazil
Cristiane Hernandez Fundação Arnaldo Vieira de Carvalho, Brazil
Cristina Bender Universidad Nacional de Rosario, Argentina
Diego Matos Federal University of Alagoas, Brazil
Diana Torres Universidad Técnica Particular de Loja, Ecuador
Edgard Benítez-Guerrero Universidad Veracruzana, Mexico
Elina Maria Avila Ordóñez Universidad de Cuenca, Ecuador
Fernanda Campos Universidade Federal de Juiz de Fora, Brazil
Francisco Alvarez Universidad Autónoma de Aguascalientes,
Mexico
Irma Elizabeth Cadme Universidad Técnica Particular de Loja, Ecuador
Jaime Muñoz-Arteaga Universidad Autónoma de Aguascalientes,
Mexico
Joao Soares de Oliveira Neto Universidade Federal do Recôncavo da Bahia
Jorge Lopez-Vargas Universidad Técnica Particular de Loja, Ecuador
Jorge Maldonado-Mahauad Universidad de Cuenca, Ecuador
Juan Carlos Morocho Universidad Técnica Particular de Loja, Ecuador
Karina Abad Universidad de Cuenca, Ecuador
Karla Lobos Universidad de Concepción, Chile
Leandro Carlos Fernandes Universidade Presbiteriana Mackenzie, Brazil
Lilia Quituisaca Samaniego Universidad de Guayaquil, Ecuador
Luis Alvarez-Gonzalez Universidad Austral de Chile, Chile
Luis Chamba-Eras Universidad Nacional de Loja, Ecuador
Luis G. Montané Jiménez Universidad Veracruzana, Mexico
Mauro Pequeno Universidade Federal do Ceará, Brazil
Marcelo Pereira Instituto Federal de Educação, Ciência e
Tecnologia de São Paulo (IFSP), Brazil
Maria Elena García Diaz Universidad Nacional de Asunción, Paraguay
Maria Fernanda Granda Juca Universidad de Cuenca, Ecuador
Miguel Angel Zúñiga Prieto Universidad de Cuenca, Ecuador
Nelson Piedra Universidad Técnica Particular de Loja, Ecuador
Nicholas Beliz-Osorio Universidad Tecnológica de Panamá, Panama
Niels Martínez Universidad Veracruzana, Mexico
Organization xi
Otto Parra-González Universidad de Cuenca, Ecuador

Pablo Quezada Universidad de Especialidades Turísticas, Escuela
de Ciencias Sociales y Humanas, Ecuador
Pablo Torres-Carrion Universidad Técnica Particular de Loja, Ecuador
Patricia Quiroz Universidad Laica Eloy Alfaro de Manabí,
Ecuador
Priscila Cedillo Universidad de Cuenca, Ecuador
René Estrella Universidad de Cuenca, Ecuador
René Faruk Garzozi-Pincay Universidad Estatal Península de Santa Elena,
Ecuador
Tannia Cecilia Mayorga Universidad Tecnológica Israel, Ecuador
Teresita De Jesús Álvarez Universidad Veracruzana, Mexico
Villie Morocho Universidad de Cuenca, Ecuador
Viviana Cabrera Universidad de Cuenca, Ecuador
Viviane Sartori Universidad Federal de Santa Catarina, Brazil
Victor Saquicela Universidad de Cuenca, Ecuador
Yasmany García-Ramírez Universidad Técnica Particular de Loja, Ecuador
Yosly Caridad Hernández Universidad Central de Venezuela, Venezuela
Contents
Enhancing Geometric Road Design Education through Active Learning

Techniques: Student Perceptions and Implications for Academia . . . . . . . . . . . . . 1
Yasmany García-Ramírez
Cell Phone Addiction, Anxiety and Distraction and Their Relationship

to Learning Achievement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
O. Gutierrez-Aguilar, A. Duche Pérez, J. Leon-Lucano,
F. Cordova-Buiza, and S. Chicaña-Huanca
Factors that Influence Digital Identity Management in Personal Learning

Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
O. Gutierrez-Aguilar, A. Duche Pérez, J. Leon-Lucano,
F. Cordova-Buiza, and S. Chicaña-Huanca
Ethics, Big Data and Artificial Intelligence: Exploring Academic Works

in the Educational Landscape . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Jones Baroni Ferreira de Menezes, Cristian Cechinel, Roberto Muñoz,
Emanuel Marques Queiroga, Vinicius Ramos, Tiago Thompsen Primo,
and João Batista Carvalho Nunes
Gamification as a Learning Strategy in the Training of B1 English Learners . . . . 49

Aura-Liz Cevallos-Almeida and Miguel-Angel Quiroz-Martinez
Implementing Synchronous Hybrid Learning: Insights from Teachers

and Students’ Experiences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Katherine Chiluiza, Vanessa Echeverria, Adriano Pinargote,
Gladys Carrillo, and Margarita Ortiz-Rojas
Beyond Numbers: Exploring the Qualitative Dimension of a Learning

Analytics Dashboard’s Usefulness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
Vanessa Heredia Jimenez, Irving Valeriano, Danny Torres,
Alberto Jimenez-Macías, and Margarita Ortiz-Rojas
Computational Thinking as a Didactic Strategy for the Development

of Mathematical Competencies in Problem-Solving . . . . . . . . . . . . . . . . . . . . . . . . 86
Sonia Peñaloza-Ochoa and Patricia Ortega-Chasi
Identifying Master’s Program Web Pages with Machine Learning . . . . . . . . . . . . . 101

Sara Díaz Oporto, Sergio Dueñas Vera, Walker Manrique Chalco,
César Bragagnini Mendizábal, and Alvaro Fernández Del Carpio
xiv Contents
Methodological Actions for the Electronic Configuration of the Educa

Plus Platform: Promoting Interactive Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
Anabel Aguayza, Christian Chuquizala, Ibeth Rendón-Enriquez,
Andrés Tirado-Espín, and Diego Almeida-Galárraga
Problem-Based Learning Applied to Software Engineering: An Experience

Report of The Software Residence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
Maria Amelia Eliseo, Marcilyanne Moreira Gois, Fábio Silva Lopes,
and Ivan Carlos Alcantara de Oliveira
Gamification Strategies as Formative Assessment Methods. A Systematic

Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
Manuel Larrosa, Leandro Wives, and Virginia Rodés
Gender Inclusion from the Art for Children . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157

Krystle Peñaherrera and Gabriel Barros
Educational use of TikTok to Enhance Complex Grammatical Aspects

of French: A Pilot Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
David Martínez Cerqueda, Alma Eloisa Rodriguez Medina,
and Alonso Gómez Ortíz
Review of Smart Grid Failure Prediction and the Need for its Study
in STEM Careers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
Marcelo Fabian Guato Burgos, Jorge Luis Morato Lara,
and Fernanda Paulina Vizcaino Imacaña
LeSeVen: A Web Application for Learning Written Vocabulary Through

the Venezuelan Sign Language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
Jesús Pérez, Yajaira Nuñez, and Jesús Pérez-Pérez
Predicting Learners’ Performance in a Programming Massive Open Online

Course . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
Jorge Maldonado-Mahauad, Andrés Auquilla, Karina Abad,
Jaime Veintimilla-Reyes, and Miguel Zúñiga-Prieto
Project-Based Learning and its Impact on the Mathematical Knowledge

of 1st Year High School Students in Schools in Southern Ecuador . . . . . . . . . . . . 226
Ricardo Orellana and Marcos Chacón-Castro
Student Perception of Peer Review in the Digital Age . . . . . . . . . . . . . . . . . . . . . . . 242

Marcio Bigolin, Eliseo Reategui, Sofía García Cabeza, and Regina Motz
Contents xv
The CLG Framework: A Methodical Approach to Designing Educational

Video Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
Alberto Rojas-Salazar and Mads Haahr
Ontology for Open Digital Educational Credentials: A Step Towards

Decentralized Learning Paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
Diego Pinto, Francisco Vargas, and Nelson Piedra
Exploring the Perceived Satisfaction of Flipped Learning in E-Learning

Courses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286
Margarita Ortiz-Rojas, Cindy Bolaños-Mendoza,
and Maria Alejandra Ruano
Technology Scouting for Techno-Pedagogical Laboratories in Ecuador:

A Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301
Diego Lasso Lazo, Francisco Alvarez Arévalo, Juan Pablo Carvallo,
Javier Valdiviezo Ortiz, Javier Patiño Chuni, Jorge Bermeo Conto,
and Katherine Espinoza Sánchez
Applying Cognitive Walkthrough to Usability Assessment: A Case Study

of an Academic Management System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
Socorro Vânia Lourenço Alves, Celson Pantoja Lima,
and Enoque Calvino Melo Alves
On the Use of ChatGPT to Support Requirements Engineering Teaching

and Learning Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328
Juan Pablo Carvallo and Lenin Erazo-Garzón
Pedagogical Support Through the Implementation of a Virtual Assistant

to Personalize Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343
Victor Farinango and Santiago Castro Arias
The Use of Indicators in the Creation of Distance Learning Undergraduate

Courses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357
Gleice Louise Garcia Costa dos Santos and Letícia Lopes Leite
Building Resilient Educational Systems: The Power of Digital Technologies . . . 370

Regina Motz, Mariana Porta, and Eliseo Reategui
ChatGPT-4 as a Tool for Reviewing Academic Books in Spanish . . . . . . . . . . . . . 384

Jonnathan Berrezueta-Guzman, Laura Malache-Silva,
and Stephan Krusche
xvi Contents
Towards Enhancing Novice Learners’ Programming Skills: Lessons

Learned from a MOOC-Based Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398
Julio C. Caiza, Danny S. Guamán, and Franklin L. Sanchez
Criteria of Pedagogical Practices in Times of Pandemic: A Systematic

Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413
Asdrúbal Emilfo Ayala-Mendoza, Lizzie Verónica Pazmiño-Guevara,
Jorge Álvarez-Tello, and Marco Duque-Romero
Academic Performance During the Pandemic with ICT Was Better Before
and After It . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 429
Diego Salazar-Armijos, Héctor Revelo-Herrera,
Pablo Niquinga-Vargas, Carlos Segovia-Segovia,
and Juan Estrada-Arana
Investigating Variation in Learners’ Behavior Through the Lens

of Learning Design, Process Mining and Learning Analytics . . . . . . . . . . . . . . . . . 442
Karina Abad and Jorge Maldonado-Mahauad
Expectations of High School Teachers Regarding the Use of Learning

Analytics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 459
Geane Biancato, Emanuel Marques Queiroga, Roberto Muñoz,
Vinicius Ramos, Tiago Thompsen Primo, Virgínia Rodés,
and Cristian Cechinel
Teaching Ethics in HCI Courses: Guidelines for Teachers and Future

Designers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472
Ismar Frango Silveira
Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 489

About the Author
Carlos Alario-Hoyos is Associate Professor in the Department of Telematics Engineer-

ing, Universidad Carlos III de Madrid, Spain. He received the M.S. and Ph.D. degrees in
telecommunication engineering and information and communication technologies from
Universidad of Valladolid, Spain, in 2007 and 2012, respectively. He was Deputy Vice
President for Strategy and Digital Education at Universidad Carlos III de Madrid between
2017 and 2023. His skills and experience include research and development in MOOCs,
social networks, collaborative learning, and evaluation of learning experiences, among
others.
Enhancing Geometric Road Design Education
through Active Learning Techniques: Student
Perceptions and Implications for Academia
Yasmany García-Ramírez(B)
Universidad Técnica Particular de Loja, San Cayetano Alto, Loja, Ecuador

ydgarcia1@utpl.edu.ec
Abstract. Active learning techniques have been increasingly employed in civil

engineering education, enhancing student engagement and academic outcomes.
However, the use of active learning in geometric road design courses remains
limited, despite the complexity and importance of this field. Geometric design
courses in road engineering often rely on traditional teaching methods, which
may hinder students’ ability to develop the diverse competencies required for
considering various factors in road design projects. Therefore, this study aims to
investigate the effectiveness of various active learning techniques in a face-to-face
course on geometric highway design. Two groups of students enrolled in the ‘Road
Construction II’ subject during the April to August 2022 academic period partic-
ipated in the study. The selected active learning techniques were based on flipped
classes, problem-based and project-based learning, and workgroup learning, and
music was incorporated into face-to-face classes. A survey was administered, and
students’ final grades were recorded for analysis. The study revealed positive corre-
lations between students’ perception of self-learning and their overall assessment.
Students expressed positive feedback for various active learning moments, high-
lighting their value and positive reception. Active learning techniques not only
increase student interest in courses but also cultivate the essential skills required
for road engineers. Engaging students in real-world projects and promoting team-
work prepares them for future challenges. Ongoing exploration of these active
learning techniques is vital, as they not only benefit new students but also align
with the evolving demands of our society.
Keywords: Active Learning · Geometric Road Design · Student Engagement
1 Introduction
Active learning is a method of teaching that involves students being actively involved
in the learning process instead of just receiving information from a teacher or a book
[1]. This approach requires students to participate in different activities, such as group
discussions, problem-solving, and practical experiments to enhance their learning expe-
rience. The main objective of active learning is to promote critical thinking, questioning,
and a comprehensive understanding of the subject matter. Active learning is necessary
in all areas of knowledge, but it is especially important in Civil engineering, where there
may still be a tendency to use traditional lecture-based learning.
© The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2023
S. Berrezueta (Ed.): LACLO 2023, LNET, pp. 1–13, 2023.
https://doi.org/10.1007/978-981-99-7353-8_1
2 Y. García-Ramírez
Several engaged learning strategies were used in civil engineering courses include
flipped classroom, problem-based learning, project-based learning, computer games,
hands-on-practices, experiments, laboratory works, and active learning tasks [2–5].
These techniques aim to engage students in the learning process and help them acquire
knowledge in the right environment using up-to-date tools and technology. Studies have
shown that active learning tasks are preferred by students over traditional lectures and
can lead to improved student ratings of course coverage and confidence in demonstrat-
ing skills gained through the course [4, 6]. Today, active learning is more possible as
modern technology is preferred by students [7], and the majority of university campuses
and homes provide them with internet access [8].
In a study on enhancing civil engineering education, computer games, hands-on-
practices, experiments, and laboratory works were used as active teaching methods in
courses such as Statics, Dynamics, Strength of Materials, and Fundamentals of Rein-
forced Concrete [5]. This trend continues to increase; however, not all areas of civil
engineering frequently use active learning. For example, in the field of road engineering,
active learning techniques have been used in transportation courses, pavement design,
and road and geotechnical engineering. Some studies have used online simulations to
teach transportation planning [9], while some have used board games to teach trans-
portation planning [10]. Collaborative curriculum design has also been found to influ-
ence educational beliefs, communities of practitioners [11], and classroom practice in
transportation engineering education [12]. Some studies have investigated the use of
Belbin’s team roles in road design courses to improve team performance and enhance
the learning experience of students [13]. In a virtual road design course, six learning
moments were planned for each topic, including watching prerecorded lectures, creating
a comic image, attending synchronous virtual classes, and preparing an infographic or
presentation of the project’s progress [14]. The last two studies were carried out through
virtual and non-face-to-face modes of learning. Also, previous studies have used team-
based learning or problem/project-based learning to enhance the learning experience of
students [15, 16]. Overall, active learning techniques have been found to be effective in
enhancing the learning experience of students in the field, and further research is needed
to explore the use of these techniques in geometric road design projects, especially in
face-to-face mode.
Geometric design teachers are a bit reserved about using active learning and try to
keep it traditional. Geometric design is a critical element in road design, and students
must develop diverse competencies to consider various factors such as traffic, topography,
geology, drainage, waterworks, soil movement, environmental impact, pavement design,
and road safety. Using multiple active techniques could lead to better academic outcomes
and make it easier to handle complex subjects such as geometric road design projects.
In this context, the aim of this study is to explore the use of various active learning
techniques in a face-to-face course on geometric highway design. For this purpose, two
groups of students who enrolled in the ‘Road Construction II’ subject during the April
to August 2022 academic period participated. The techniques were chosen based on a
previous virtual study [14], and music was played during the face-to-face classes. The
main contribution of the paper is to explore the use of various active learning techniques
in a face-to-face course on geometric highway design, and to investigate how these
Enhancing Geometric Road Design Education through Active Learning Techniques 3
techniques could enhance the learning experience and improve the academic outcomes
of students. The study also aims to contribute to the development of diverse competencies
among students, and to facilitate the handling of complex subjects such as geometric
road design projects.
2 Methods
In this section, the study’s methodology is introduced, including information about the
participants, design, materials, procedure, and data analysis. The study involved two
groups of civil engineering students who were enrolled in the ‘Road Construction II’
subject. The groups were experimental and attended face-to-face classes for 16 weeks
during the academic term, where they engaged in five weekly activities. A survey was
designed to evaluate their impressions at the end of the semester, and their final grades
for the course were recorded. Data analysis was conducted by associating the students’
gender and quartile of final grades with the collected data and attempting to find a
relationship with the survey responses. Various statistical tools were used to analyze
these relationships.
2.1 Participants
The study involved two groups of students who were enrolled in the ‘Road Construction
II’ subject during the April to August 2022 academic period. The participants were civil
engineering students who had completed the prerequisite courses related to road design.
Group A had 16 students and Group B had 42.
2.2 Design
The two groups were experimental and attended face-to-face classes for 16 weeks during
the academic term. At the end of the semester, their impressions were analyzed through
a survey, and their final grades for the course were recorded.
2.3 Materials
Both groups had the same weekly activities (see Fig. 1), which can be summarized in
five steps:
1. Prior to class, students were required to watch a prerecorded lecture on the weekly
topic.
2. Using the meme template provided by the instructor, students created a meme related
to the weekly theme that had to be funny.
3. In a face-to-face class, examples related to the weekly theme were presented.
4. In person, the students either worked individually or in teams to apply the concepts
learned to a real road project.
5. After class, students presented the progress of the road project using either an
infographic or a pre-recorded presentation.
During face-to-face classes and the students’ autonomous work, a set of music chosen
by the students was played. The songs were related to their musical tastes and ranged
from classical to reggaeton.
Fig. 1. Selected learning moments applied to both study groups.
2.4 Procedure
The study was conducted during the regular academic period. The study lasted for
16 weeks, during which both groups received the same course content. The active learn-
ing techniques were applied for all of the total course hours. A survey was designed
where students could rate each activity from 1 to 10, with 1 being the lowest score and
10 being the highest. Additionally, an open-ended question was included for students to
comment on each learning moment or activity of the course. Questions were also asked
regarding the general rating of the subject and the perception of autonomous work, also
using scores from 1 to 10. Finally, the final grades of the students were collected.
2.5 Data Analysis

The gender of the students and their quartile of final grades obtained in the course were
associated with the collected data, and an attempt was made to find some relationship
with the answers that the students provided in the survey. Various statistical tools were
used to analyze these relationships.
3 Results
The survey was anonymous and optional, and very few students did not answer it.
Thirteen students from Group A and thirty-nine students from Group B responded,
which shows a high percentage of participation, that is, 81% and 93% respectively. The
calculation of the quartiles based on the final scores of the groups was also performed.
Regarding the gender of the students, no significant relationship was found in either
group, possibly due to the sample size.
3.1 Group A Results

The results of the grades and the answers of Group A are shown in Table 1. In this
table, the average values, along with a 95% confidence interval, have been provided.
Quartile 3 consisted of only one student. However, certain relationships can still be
observed in Table 1. Students in quartiles 1 and 4 exhibited a lower perception of self-

learning compared to the other quartiles. There is a correlation between the perception
of self-learning and the overall assessment of the subject. Students in quartiles 1 and
4 rated high the learning moments, including the presence of music during classes,
quite similarly. Interestingly, despite their grades, both groups valued active learning
techniques similarly. It is important to note that a majority of students in quartile 4 did
not pass the subject. Quartile 2 students assigned lower scores to the learning moments,
except for the presence of music during face-to-face classes. These trends, although
significant, involved a small number of students and should be interpreted with caution.
Table 1. Final grades and answers from students in Group A.
Quartile 1 2 3 4
Gender of the students M:3, F.:0 M:0, F.:2 M:1, F.:0 M:5, F.:2
Student grades 8.1±[1.4] 7.4±[0.7] 7.1 6.4±[0.5]
Students’ perception of self-learning 8.3±[2.9] 9.5±[6.4] 10.0 7.6±[0.5]
Students’ overall assessment of the subject 9.3±[2.9] 8.5±[6.4] 10.0 9.1±[1.1]
Prerecorded lectures 9.7±[1.4] 8.0±[12.7] 8.0 8.4±[0.9]
Creation of internet memes 9.7±[1.4] 9.0±[12.7] 10.0 9.1±[1.7]
Problem-solving sessions 10.0 10.0 10.0 10.0
Weekly road projects 10.0 7.5±[6.4] 10.0 9.0±[1.6]
Music during classes 9.3±[2.9] 9.5±[6.4] 10.0 8.7±[2.0]
Weekly presentations or infographics 10.0 7.5±[6.4] 10.0 9.1±[1.5]
M: Male, F: Female.
In order to analyze all the responses and scores of the students, a correlation matrix
was calculated (see Table 2). In this table, the most significant relationships have been
highlighted using the p-value. The problem-solving sessions did not yield any results
in this table because all the students assigned a value of 10, indicating no variability to
evaluate. All significant relationships between variables were positive. The creation of
internet memes is positively related to the grades of prerecorded lectures and the weekly
presentations or infographics, although to a lesser extent, it is also related to music
during classes. Furthermore, the prerecorded lectures and the weekly road projects are
positively related to the weekly presentations or infographics, while the students’ overall
assessment of the subject is related to the completion of the road project. Finally, the
prerecorded lectures are positively related to students’ grades. These results highlight the
flipped classroom and project-based education as key elements valued by the students.
Additionally, they also value the creative aspect through the creation of internet memes
and the collaborative aspect through presentations or infographics.
Regarding the open question of the survey, the comments from the students in Quar-
tile 1 show that they value the different learning moments proposed in the road design
course. They appreciate the methodology used, the opportunity to be creative, apply
Table 2. Matrix correlation between several variables in Group A.
1 2 3 4 5 6 7 8
2 0.415
3 0.184 0.324
4 0.499** −0.135 0.356
5 0.265 -0.155 0.075 0.586*
6 – – – – −
7 0.244 0.141 0.773* 0.34 -0.167 –
8 0.331 0.213 0.176 0.299 0.806** – -0.133
9 0.263 -0.134 0.316 0.555* 0.676* – 0.35 0.586*
*P-value < 0.05, **P-value < 0.10.
1. Student grades.
2. Students’ perception of self-learning.
3. Students’ overall assessment of the subject.
4. Prerecorded lectures.
5. Creation of internet memes.
6. Problem-solving sessions.
7. Weekly road projects.
8. Music during classes.
9. Weekly presentations or infographics.
knowledge to real projects, and communicate their progress. However, it is suggested

to improve the teaching time in Moments 3 and 4. There is also a positive comparison
made between this course and a previous one, indicating an improvement in the teaching
methodology.
On the other hand, the comments from the students in Quartile 2 suggest that they
find the five learning moments proposed in the road design course beneficial. However,
they highlight areas for improvement, such as the clarity of the videos, the time and
options available to address doubts and errors, the duration of the projects, and fairness
in the grading system. These suggestions could be considered to enhance the learning
experience in future editions of the course.
The responses from the students in Quartile 3 indicate that they find the learning
moments proposed in the highway design course beneficial. However, certain areas
for improvement stand out, such as the need for greater practical applicability of the
pre-recorded exhibition. There are also concerns about equity in the evaluation sys-
tem, particularly regarding the exhibition or infographic. These observations could be
considered to enhance the learning experience and assessment in future editions of the
course.
In general, the comments from the students in Quartile 4 reflect a positive reception of
the learning moments proposed in the road design course. Students appreciate the videos,
face-to-face classes, and the opportunity to apply their knowledge to real projects. There
is also a suggestion to include more content related to specific software used in highway
design. Overall, the comments indicate satisfaction with the methodology and activities
of the course.
In summary, the majority of students across quartiles find value in the different learn-
ing moments of the road design course. Their feedback highlights areas for improve-
ment, such as the clarity of videos, more support for addressing doubts and errors, longer
project durations, and fairness in grading. Incorporating these suggestions can lead to
an enhanced learning experience for future iterations of the course.
3.2 Group B Results

The grades and responses of Group B are presented in Table 3, including the average
values accompanied by a 95% confidence interval. In this group, there is a larger number
of students compared to Group A. However, the number of women is lower than men,
which is attributed to the demographics of the major. The perception of self-learning
among students is slightly higher in Quartiles 1 and 2 compared to Quartiles 3 and 4.
Students in Quartile 1 obtained the lowest scores in the overall assessment of the subject.
There is a positive correlation between the perception of self-learning and the general
evaluation of the subject. Almost all quartiles, except the first one, rated the prerecorded
lectures highly. All the groups gave high ratings to the different learning moments;
however, Quartile 1 gave slightly lower ratings compared to the other quartiles. It can
be inferred that the creation of memes and problem-solving sessions are enjoyed by
all students. Moreover, based on the scores, it can be concluded that active learning
activities are well-received by all students.
Table 3. Final grades and answers from students in Group B.
Quartile 1 2 3 4
Gender of the students M:7, F.:3 M:8, F.:2 M:5, F.:4 M:8, F.:2
Student grades 7.7±[0.1] 7.3±[0.1] 7.2±[0.0] 6.9±[0.2]
Students’ perception of self-learning 8.2±[0.8] 8.1±[0.9] 7.9±[1.2] 7.9±[1.1]
Students’ overall assessment of the subject 8.4±[0.8] 9.2±[0.7] 9.1±[0.5] 9.0±[0.8]
Prerecorded lectures 8.3±[0.9] 9.4±[0.6] 9.3±[0.8] 9.4±[0.4]
Creation of internet memes 9.8±[0.3] 9.6±[0.4] 9.7±[0.5] 9.9±[0.2]
Problem-solving sessions 9.3±[0.6] 9.8±[0.3] 9.6±[0.6] 9.6±[0.6]
Weekly road projects 8.9±[0.9] 9.5±[0.4] 8.9±[1.0] 9.0±[0.6]
Music during classes 8.8±[1.1] 8.9±[0.9] 9.1±[0.9] 9.2±[1.4]
Weekly presentations or infographics 8.9±[0.9] 9.0±[0.7] 8.8±[1.1] 9.3±[0.6]
M: Male, F: Female.
In order to delve into the students’ answers to the numerical questions of the survey
and their final grades, a correlation matrix was prepared as shown in Table 4. This
table highlights statistically significant relationships with a P-value of 0.05 and 0.10.
The presence of music during classes and the development of the road project show a
positive relationship with the other learning moments, as well as with the perception
of self-learning and the overall assessment of the subject. Additionally, the problem-
solving sessions are positively related to almost every moment, except for the weekly
presentations or infographics and the perception of self-learning. The perception of self-
learning is related to the general evaluation of the subject. When it comes to grades,
only one moment, the prerecorded lectures, shows a negative relationship. However, it is
important to note that this statistical relationship does not imply causality, as it is merely
a correlation.
This result also highlights the need to analyze grades and the learning experience
separately. Grades are not necessarily related to the student’s experience during the
course. In summary, students value the learning moments that have been implemented,
including the flipped class, problem- and project-based learning, and teamwork during
the project, presentations, and infographics. The positive assessment of music during
classes warrants further investigation in future studies.
Table 4. Matrix correlation between several variables in Group B.
1 2 3 4 5 6 7 8
2 0.155
3 −0.226 0.304**
4 −0.337* 0.056 0.663*
5 −0.034 0.249 0.066 0.208
6 −0.069 0.261 0.398* 0.529* 0.415*
7 0.068 0.462* 0.36* 0.280** 0.310** 0.459*
8 −0.035 0.369* 0.424* 0.284** 0.473* 0.379* 0.313**
9 −0.110 0.144 0.340* 0.164 0.336* 0.266 0.535* 0.363*
*P-value < 0.05, **P-value < 0.10
1. Student grades.
2. Students’ perception of self-learning.
3. Students’ overall assessment of the subject.
4. Prerecorded lectures.
5. Creation of internet memes.
6. Problem-solving sessions.
7. Weekly road projects.
8. Music during classes.
9. Weekly presentations or infographics.
Overall, students from Quartile 1 appreciate the different learning moments imple-
mented in the road design course. The asynchronous methodology, problem-solving
exercises, real project application, and presentations/infographics receive positive feed-
back. Areas for improvement include video quality, clarity of instructions, guidance in
practice exercises, fairness in group work, and sufficient time allocation. The feedback
provides valuable insights for enhancing future editions of the course. In general, students
in Quartile 2 show a positive assessment of the different learning moments of the course.
However, there are some suggestions for improvement, such as delving deeper into cer-
tain aspects, providing more guidance in the use of specific software, and ensuring equal
participation in group work. Students in Quartile 3 also show a positive assessment of
the different learning moments of the course. They find the activities dynamic, entertain-
ing, and helpful for their professional development. Some suggest small improvements,
such as whiteboard exercises during classes and adjustments to the duration of projects.
The comments from students in Quartile 4 also value the different learning moments of
the course positively. They appreciate the clarity of the videos, the practice of creating
memes, the resolution of exercises in class, the application of knowledge in a real project,
and the presentations/infographics. Some mention the need to improve aspects such as
the volume of videos and time management for the development of practices.
Based on the feedback provided it can be concluded that the implementation of
various learning moments in the road design course has been well-received overall.
The asynchronous methodology, problem-solving exercises, real project application,
and presentations/infographics have been positively appreciated by students. However,
there are areas for improvement identified, such as video quality, clarity of instructions,
guidance in practice exercises, fairness in group work, and time allocation. Suggestions
for improvement include delving deeper into certain aspects, providing more guidance
in software usage, and ensuring equal participation in group work. Despite these sug-
gestions, the feedback indicates that the learning moments have been beneficial for
the students’ professional development. Considering the students’ comments will be
valuable in enhancing future editions of the course.
Considering that Group B had a larger number of students compared to Group A,
a more detailed analysis was conducted on the grades, perception of self-learning, and
overall assessment of the subject. The data was divided into quartiles based on students’
grades, and the results are presented in Fig. 2. The responses from students in Quartile
4 show a positive trend for both cases, while Quartiles 2 and 3 exhibit relatively stable
trends. In Quartile 1, however, there is a positive trend in the perception of self-learning,
Fig. 2. Grades versus Students’ Perception of Self-Learning and Overall Assessment of the
Subject across Four Quartiles.
but a negative trend in the general assessment of the subject. This phenomenon observed
among students with higher grades should be further examined in future research for a
more comprehensive understanding.
4 Discussion
Perception of self-learning and its correlation with the overall assessment. The study
reveals that students in quartiles 1 and 4 exhibited a lower perception of self-learning
compared to the other quartiles. Additionally, there is a positive correlation between the
perception of self-learning and the general evaluation of the subject. In general, there
are no studies that have established a direct relationship between these two elements;
most studies have instead focused on self-learning and educational performance. A
previous study found moderate to high correlations among the various components of
self-regulated learning at the assessment level [17].
The feedback from students in all quartiles highlights the value and positive reception
of various learning moments, such as asynchronous methodology, problem-solving exer-
cises, real project application, and presentations/infographics. Asynchronous method-
ology has been shown to promote self-paced learning and flexibility, allowing students
to engage with course materials at their own convenience [18]. Internet memes can
enhance students’ understanding and retention of information through humor and visual
appeal [19]. Students benefit from the flexibility and control over their learning pace
and review opportunities [20]. Prerecorded lectures allow students to revisit complex
topics, pause, rewind, and review the content at their own convenience. Problem-solving
exercises stimulate critical thinking skills and help students apply theoretical concepts to
real-world scenarios [21]. The integration of real project applications fosters hands-on
learning experiences, enabling students to bridge the gap between theory and practice
[22]. Presentations and infographics encourage students to communicate their under-
standing of complex topics effectively, enhancing their communication and presentation
skills [23, 24].
The study identifies several areas for improvement based on students’ feedback,
including video quality, clarity of instructions, guidance in practice exercises, fairness
in group work, and time allocation. Firstly, high-quality videos can significantly con-
tribute to effective online instruction [25]. Clear visuals, good audio, and appropriate
resolution can enhance comprehension and engagement among students. Secondly, clar-
ity of instructions is crucial for students to understand the expectations and requirements
of assignments and activities. Well-defined instructions help students navigate through
the learning process and complete tasks more effectively [26]. Guidance in practice
exercises is another aspect highlighted for improvement. Providing clear instructions,
examples, and scaffolding can support students in applying their knowledge and devel-
oping practical skills [27]. Fairness in group work is an important consideration. Group
projects should be designed to ensure equal participation and contribution from all group
members [28]. Clear guidelines, regular monitoring, and opportunities for feedback can
help create a fair and collaborative learning environment. Time allocation is also a key
factor. Students appreciate a well-structured course with reasonable deadlines and a
manageable workload. According to a study [29], students procrastinated more when
completing high-complexity tasks than low-complexity tasks.
This study has several limitations. The study involved a relatively small number of
participants. The small sample size may limit the generalizability of the findings and
the ability to detect significant relationships or patterns accurately. The study focused
on civil engineering students enrolled in the ‘Road Construction II’ subject, which may
limit the representation of other disciplines or student populations. The data collection
relied on a survey designed to evaluate the students’ impressions and perceptions of the
learning moments and their final grades. Self-report measures are subjective and may be
influenced by various factors such as recall bias, social desirability bias, or interpretation
differences. The study utilized an experimental design with two groups attending face-
to-face classes and engaging in the same learning activities with no control group. The
absence of a control group makes it challenging to isolate the specific effects of the
learning moments and assess their impact accurately. The study was conducted over a
16-week academic term, which may be a relatively short timeframe to fully capture the
long-term effects or sustainability of the implemented learning moments.
Despite these limitations, the study provides several aspects that can be helpful
for academia. The majority of students across different quartiles expressed positive
feedback and valued the various learning moments implemented in the road design
course. This positive feedback indicates that these learning moments were beneficial and
well-received by the students. Also, the results highlight specific elements that students
valued in the course, such as the creation of internet memes, problem-solving sessions,
flipped classroom and project-based learning approaches, and the inclusion of music
during classes. Furthermore, the study identified positive correlations between certain
learning moments and students’ grades. Also, the study gathered valuable feedback from
students regarding areas for improvement in the course. Students provided suggestions to
enhance the learning experience, including improving video quality, providing clearer
instructions, offering more guidance in specific software usage, ensuring fairness in
group work, and allocating sufficient time for different activities. Incorporating these
suggestions can lead to an enhanced learning experience and address student concerns.
There are several future research that emerge from this study. It is necessary to
understand the relationship between the perception of self-learning and overall assess-
ment. The study found a correlation between students’ perception of self-learning and
their overall assessment of the subject, and this relationship should be explored fur-
ther in future research. Additionally, the relationship between music and other learning
moments should be analyzed in similar or different courses. Furthermore, the correla-
tion between academic performance and the perception of student experience should be
examined in depth.
5 Conclusions
The objective of this study was to explore the use of various active learning techniques in
a face-to-face course on geometric highway design. Two groups of students participated
in the study, enrolling in the subject ‘Road Construction II’ during the academic period
from April to August 2022. The use of diverse active learning techniques in the face-to-
face course was well-received by students. Active techniques not only make the courses
more appealing to students but also foster the development of skills necessary for road
engineers. Flipped classes, problem-based learning, and projects promote autonomous

learning, which is essential in road projects. The involvement of students in real projects
and teamwork prepares them for future endeavors. Positive student feedback, suggestions
for improvement, understanding the relationship between perceived self-learning and
overall assessment, and prospects for future iterations highlight the value and impact of
the study. It is crucial to continue exploring these active learning techniques as they are
not only beneficial for new students but also demanded by our ever-changing society.
References
1. Bonwell, C., Eison, J.A.: Active Learning: Creating Excitement in the Classroom. School of
Education and Human Development, Washington, D.C. (1991)
2. Sayed-Ahmed, E.: Reforming undergraduate structural engineering education: a leap of faith.
In: Proceedings of International Structural Engineering and Construction. ISEC Press (2018)
3. Smadi, O., Akili, W.: Infrastructure asset management education: active learning and
engagement-based practices. Transp. Res. Rec., 16–18 (2006). https://doi.org/10.3141/195
7-03
4. Abraham, Y.S.: Importance of active learning in an undergraduate course in construction
scheduling. In: ASEE Annual Conference and Exposition, Conference Proceedings. American
Society for Engineering Education (2020)
5. Çalışkan, N., Mertol, H.: Enhancing civil engineering education using active learning
techniques. Education (Chula Vista) (2013)
6. Warren, K., Padro, M.: Design and preliminary data from a partially flipped classroom (PFC)
study in a geotechnical engineering course. In: ASEE Annual Conference and Exposition,
Conference Proceedings. American Society for Engineering Education (2019)
7. Sohrabi, B., Iraj, H.: Implementing flipped classroom using digital media: a comparison of two
demographically different groups perceptions. Comput. Hum. Behav. 60, 514–524 (2016).
https://doi.org/10.1016/j.chb.2016.02.056
8. Bergmann, J., Sams, A.: Flip your classroom reach every student in every class every day.
Get Abstr. Compress. Knowl., 1–5 (2014). https://doi.org/10.1111/teth.12165
9. Zhu, S., Xie, F., Levinson, D.: Enhancing transportation education through online simulation
using an agent-based demand and assignment model. J. Prof. Issues Eng. Educ. Pract. 137,
38–45 (2011). https://doi.org/10.1061/(ASCE)EI.1943-5541.0000038
10. Huang, A., Levinson, D.: To game or not to game: teaching transportation planning with
board games. Transp. Res. Rec. 2307, 141–149 (2012). https://doi.org/10.3141/2307-15
11. Hurwitz, D.S., Swake, J., Brown, S., et al.: Influence of collaborative curriculum design on
educational beliefs, communities of practitioners, and classroom practice in transportation
engineering education. J. Prof. Issues Eng. Educ. Pract. 140 (2014). https://doi.org/10.1061/
(ASCE)EI.1943-5541.0000196
12. Hossain, Z.: Assessment of project-based effective learning in transportation engineering.
In: Proceedings of the International Conference on Industrial Engineering and Operations
Management. IEOM Society (2020)
13. García-Ramírez, Y.: Belbin’s team roles and their performance in road design courses: a study
with undergraduate and postgraduates students. Espacios 42, 176–188 (2021). https://doi.org/
10.48082/espacios-a21v42n01p15
14. Garcia-Ramirez, Y.: Active learning techniques in remote learning: a case study in road design
courses in Ecuador. In: 16th Latin American Conference on Learning Technologies, LACLO
2021, pp. 224–229. Institute of Electrical and Electronics Engineers Inc. (2021)
15. Chen, D., Chen, S.E., Gehrig, G.B.: A project-based integrated work/review cycle (PBIWR)
for design and learning of accelerated construction monitoring. In: ASEE Annual Conference
and Exposition, Conference Proceedings (2013)
16. Garrido de la Torre, M.E., Pérez-Zuriaga, A.M., Martínez-Ibáñez, V., et al.: PIME Aprendizaje
Basado en Proyectos: Ingeniería de Carreteras e Ingeniería Geotécnica. In: VI Congreso de
Innovación Educativa y Docencia en Red (2020)
17. Dörrenbächer-Ulrich, L., Weißenfels, M., Russer, L., Perels, F.: Multimethod assessment of
self-regulated learning in college students: different methods for different components? Instr.
Sci. 49, 137–163 (2021). https://doi.org/10.1007/s11251-020-09533-2
18. Karabulut-Ilgu, A., Yao, S., Tarmo, P., Jahren, C.: A flipped classroom approach to teaching
transportation engineering. In: ASSE’s 123rd Annual - Conference and Exposition. New
Orleans, LA (2016)
19. Reddy, R., Singh, R., Kapoor, V., Churi, P.P.: Joy of learning through internet memes. Int. J.
Eng. Pedagogy (iJEP) 10, 116–133 (2020)
20. Smith, G., Fidge, C.: On the efficacy of prerecorded lectures for teaching introductory pro-
gramming. In: Proceedings of the 2008 International Conference on Advances in Computer
Entertainment Technology (2008)
21. Wilson, J.A., Hiley, A., Collis, A.: The development of a module to equip students with real-
world problem-solving skills. In: EE 2008 - International Conference on Innovation, Good
Practice and Research in Engineering Education (2008)
22. Wagener, T., Zappe, S.: Introducing real-world hydrology case studies into an undergrad-
uate civil and environmental engineering curriculum. In: ASEE Annual Conference and
Exposition, Conference Proceedings. American Society for Engineering Education (2008)
23. Fuvesi, I.: Interactive boards and their function in education. Int. J. Adv. Corp. Learn. (iJAC)
2, 31–35 (2009)
24. Akhmad, R., Sugandi, D., Nandi, N., Rahmawati, R.: Infographic design as visualization of
geography learning media. In: IOP Conference Series: Earth and Environmental Science,
p. 012011. IOP Publishing (2018)
25. Lackmann, S., Léger, P.M., Charland, P., et al.: The influence of video format on engagement
and performance in online learning. Brain Sci. 11, 1–21 (2021). https://doi.org/10.3390/BRA
INSCI11020128
26. Kemma Abderrazak, E.L.: Giving effective instructions in EFL classrooms. Int. J. Innov.
Educ. Res. 7, 74–92 (2019). https://doi.org/10.31686/IJIER.VOL7.ISS1.1286
27. Usmeldi, U., Amini, R., Trisna, S.: The development of research-based learning model with
science, environment, technology, and society approaches to improve critical thinking of
students. Jurnal Pendidikan IPA Indones. 6, 318–325 (2017). https://doi.org/10.15294/JPII.
V6I2.10680
28. Curşeu, P.L., Pluut, H.: Student groups as learning entities: the effect of group diversity and
teamwork quality on groups’ cognitive complexity. Stud. High. Educ. 38, 87–103 (2013).
https://doi.org/10.1080/03075079.2011.565122
29. Sun, T., Kim, J.E.: The Effects of online learning and task complexity on students’ procrasti-
nation and academic performance. Int. J. Hum. Comput. Interact. (2022). https://doi.org/10.
1080/10447318.2022.2083462
Cell Phone Addiction, Anxiety and Distraction
and Their Relationship to Learning Achievement
O. Gutierrez-Aguilar1(B) , A. Duche Pérez1 , J. Leon-Lucano1 ,

F. Cordova-Buiza2,3 , and S. Chicaña-Huanca1
1 Universidad Católica de Santa María, Arequipa, Perú
{ogutierrez,aduche,jair.leon,s.chicana.h}@ucsm.edu.pe
2 Universidad Privada del Norte, Lima, Perú
franklin.cordova@upn.edu.pe
3 Universidad Continental, Huancayo, Perú
Abstract. This study sought to establish the causal relationship between cell
phone addiction, anxiety, and distraction and their influence on learning achieve-
ment among university students. The methodology applied is non-experimental at
the explanatory level of the cross-section. Data were collected from 210 students
aged 17 to 25 years old. The data was treated by modelling structured equations
based on the variance, which is a multivariate model. This study focused on crit-
ical factors: cell phone addiction, anxiety, distraction, and their relationship to
learning achievement. Each of these variables was subjected to meticulous anal-
ysis, the reliability of which was confirmed throughout the process. This detailed
examination revealed its importance and influence on the academic performance
of higher education students. The most salient finding suggests that cell phone
addiction and anxiety are positively correlated with learning achievement. In con-
trast, distraction was harmful. An increase in anxiety and cell phone addiction was
associated with better academic performance, while an increase in distractibility
was associated with decreased academic performance. These findings highlight
the complexity of the relationship between these variables and their impact on
academic performance.
Keywords: Anxiety · Learning achievement · Cell phone addiction ·

Distraction · PLS-SEM
1 Introduction
Depression and anxiety are two types of disorders associated with the student stage,
especially at the university level, due to the changes people experience, such as forming
new interpersonal relationships and adapting to new stresses. Academic [1], in the same
way, how addiction to cell phones and distraction [2], can be related as predictor variables
in learning achievement in university students. Thus, inappropriate use of mobile phones
(MMPU) is generally defined as excessive use of phones that interferes with work and
social interactions [3]; this can result in problems of social adaptation and will directly
influence academic performance in students.
https://doi.org/10.1007/978-981-99-7353-8_2
Cell Phone Addiction, Anxiety and Distraction and Their Relationship 15
Although numerous studies have examined the relationships between mobile phone
use and educational outcomes, many have yielded conflicting results [4], thus giving rise
to monophobia, which mediates the impact of mobile phone use. Smartphones at bedtime
affect academic performance [5]; thus, using smartphones and academic performance
maintain a negative association, based on a series of empirical studies [6]. On the other
hand, the excessive use of social networks results in mental health problems, that is, a
lack of real emotional connection, cyberbullying, and significantly a decrease in thoughts
and understanding [7–10], which will inevitably result in learning achievements.
Undoubtedly, there is an increasingly alarming trend of depression, isolation, and
suicide that has increased since the proliferation of smartphones, through a study with
a total of 200 university students found significant positive relationships between anx-
iety about attachment, loneliness, depression and addiction to smartphones. It is con-
cluded that addiction to smartphones affects several areas of people’s lives, including
interpersonal relationships, studies, emotions and health [11].
Anxiety in its mediating role between the problematic use of cell phones and per-
formance in the case of mathematics, according to the study developed by Da Zhoua
et al., about Relationships among problematic smartphone use, mathematics anxiety,
learning interest, and achievement [12], found that problematic smartphone use was
negatively correlated with math performance; Likewise, math anxiety and interest in
learning partially mediated the relationship between problematic smartphone use and
math performance, and the relationship between problematic smartphone use and math
performance was serially mediated by math anxiety. -Thematic and then by interest in
learning mathematics.
Anxiety in students generates mental distractions that reduce the load on their work-
ing memory so that learning achievement is at risk [13]. Likewise, anxiety could limit
critical activities for the achievement of learning, such as the search for information
related to key concepts such as subject anxiety, ICT anxiety, library anxiety, and quality
anxiety. Of resources [14]. On the other hand, test anxiety can lead to learning prob-
lems that contribute to reduced concentration and can lead to errors and unproductive
learning, especially in science, technology, engineering, and math STEM careers [15].
Distraction generates concentration problems in the educational context; such is the
case that today, speaking of digital distraction is a common phenomenon in electronic
reading contexts since screen reading has close links with multitasking. Moreover, a
possible distraction is any research on reading in the digital environment [16]. In the same
way, distractions from social networks have an impact on student evaluation systems of
teaching effectiveness [17].
As for the excessive use of cell phones, they distract from other incidents and actual
events. Being hypnotized and anything other than being in the moment with your smart-
phone has become the norm for many. It has shaped the current term ‘nomophobia’ [18],
which is a type of dependency or addiction to the phone, which can lead the college
student to sexting and mental health problems [19], as well as psychological distress
including anxiety [20], stress [21], depression [22] and loneliness [20]. Few empirical
investigations would demonstrate the impact of addiction to cell phones; however, the
following tips are essential: use smartphone applications to end addiction problems; go
on a digital detox; and change current behaviours [23].
16 O. Gutierrez-Aguilar et al.
In the university educational context, it is necessary to rethink teaching strategies so

that they are effective for students, that is, align strategies with the interests of digital
natives, considering the personality traits of natives. Phones with each component of cell
phone addiction syndrome (MAS) [24], some research has shown a positive correlation
between cell phone addiction and neuroticism [25]. Likewise, the relationship between
the use of smartphones has a strongly negative impact on the performance of university
students, as argued [26, 27].
On the other hand, the moderating function of gender in the Smartphone Addiction
Inventory (SPAI) [28], with the Big-Five-Inventory (BFI) in university students, suggest
positive and significant relationships between neuroticism (imbalance -emotional river)
and female-only smartphone addiction [29]. Based on the above considerations, the
following hypotheses are proposed:
H1 There is a significant relationship between anxiety and learning achievement.
H2 There is a significant relationship between cell phone addiction and learning
achievement.
H3 Distraction positively influences learning achievement.
2 Methodology
An adaptation by Adi Suryanto was used for the study: Study of working from home:
the impact of ICT anxiety and smartphone addiction on lecturers at NIPA School of
Administration on job performance [30]. A sample of 210 university students of both
sexes was used, 82.0% women and 18.0% men, ages between 17 and 25 years, with a
standard deviation of SD: 2.30. The instrument was applied in March 2023.
The variables observed are Cell phone addiction (CPA), Anxiety (ANX), Distraction
(DIS) and Relationship to learning achievement (RLA). Table 1 presents the reliability
indices for the observed variables, which are acceptable for both cases, using Cronbach’s
alpha (α) and McDonald’s (ω).
Table 1. Reliability indices.
Variables Cronbach’s alpha α McDonald’s ω

CPA 0.71 0.72
RLA 0.87 0.88
DIS 0.74 0.75
ANX 0.73 0.76
Subsequently, an Exploratory Factor Analysis (EFA) was performed. The result for
the Kaiser-Meyer-Olkin (KMO) test was 0.740, indicating that the data are adequate to
perform an EFA and Bartlett’s test with a p-value of <0.001, which means that there
are significant correlations between the variables and that it is advisable to perform the
EFA. The factors were loaded with an exit criterion of 0.35 and the minimum residue
extraction method using the Oblimin rotation method, whose results were satisfactory.
The development of the research followed at all times a systematic and rigorous process
of data classification and analysis with the statistical software Jamovi (v. 2.3.24).
3 Results
The results of “Model tests” in Table 2 present two models: the “User Model” and the
“Baseline Model”. The User Model is proposed for the research, while the Baseline
Model is a reference model. Three model fit statistics are being presented: X2 , df and p.
The X2 measures how much the User Model deviates from the Baseline Model, and the df
indicates the degrees of freedom associated with the X2 . The p indicates the probability
that the X2 is due to chance rather than genuine differences between the two models. In
this case, the User Model has an X2 of 91.26 with 71 df and a p of 0.053, which suggests
that the model fits the data reasonably well.
Table 2. Model tests
Label X2 df p
User Model 91.26 71 0.033
Baseline Model 571.23 91 <.001
Table 3 presents the results of the “Fit indices” for the confirmatory factor analysis,
which provide additional information on the model’s fit. Two model fit statistics are
presented: Standardized Root Mean Square Residual (SRMR) [31] and Root means the
square error of approximation (RMSEA) (RMSEA) [32]. The SRMR measures how
much the observed covariances deviate from the covariances estimated by the model.
The RMSEA measures how much the observed covariances deviate from the model-
estimated covariances, adjusted for the number of parameters in the model. In this case,
the SRMR is 0.07, and the RMSEA is 0.05, suggesting that the model fits the data
reasonably well.
Table 3. Fit indices of the confirmatory factor analysis
SRMR RMSEA 95% Confidence Intervals RMSEA p

Lower Upper
0.07 0.05 0.05 0.08 0.450
Table 4 presents in “User model versus baseline model”: Comparative Fit Index
(CFI), which is a fit index that compares the proposed model with a null model. A CFI
of 0.96 suggests that the proposed model fits the data well. Likewise, the Tucker-Lewis
Index (TLI) is a fit index similar to the CFI that also compares the proposed model with a
null model; in this case, the TLI is 0.95, which suggests that the proposed model fits well
with the results. Data. The Bentler-Bonett Non-normed Fit Index (NNFI) is another fit
index that compares the proposed model with a null model. The following table presents
an NNFI of 0.95, meaning the proposed model fits well with the data. The Bentler-Bonett
Normed Fit Index (NFI) is a fit index that measures the improvement in the fit of the
proposed model about a null model. According to the results, the NFI is 0.84, which
suggests that the proposed model fits better than a null model but not perfectly.
Parsimony Normed Fit Index (PNFI): The PNFI is a fit index that considers the
proposed model’s complexity about its fit; in this case, the PNFI is 0.66, which suggests
that the proposed model could be more parsimonious. Bollen’s Relative Fit Index (RFI):
The RFI is a fit index that compares the proposed model with a fitted null model;
according to the model results, the RFI is 0.80, which suggests that the proposed model
fits better than a null model fitted, but not perfectly. Bollen’s Incremental Fit Index (IFI)
is a fit index that measures the improvement in the fit of the proposed model concerning
a fitted null model; according to the results, the IFI is 0.96, which suggests that the
proposed model fits the data well compared to a fitted null model.
Relative Noncentrality Index (RNI) is a fit index that measures the amount of variance
explained by the proposed model about a null adjusted model; in this case, the RNI is
0.96, which suggests that the proposed model explains well the variance in the data
compared to a fitted null model. Consequently, the fit indices of the proposed model
have a good representation of the observed data.
Table 4. Fit indices
Model
Comparative Fit Index (CFI) 0.96
Tucker-Lewis Index (TLI) 0.95
Bentler-Bonett Non-normed Fit Index (NNFI) 0.95
Bentler-Bonett Normed Fit Index (NFI) 0.84
Parsimony Normed Fit Index (PNFI) 0.66
Bollen’s Relative Fit Index (RFI) 0.80
Bollen’s Incremental Fit Index (IFI) 0.96
Relative Noncentrality Index (RNI) 0.96
The ANOVA test (Analysis of Variance) is used to compare the means of different
groups to determine if there are significant differences between the analysis groups. In
the case of regression analyses, this test is used to assess whether the regression model
proposed in its entirety is significant, that is, that at least one of its independent variables
has a significant effect on the dependent variable. The results obtained in Table 5 indicate
that the proposed regression model is significant because the p-value (0.000) is less than
the significance level of 0.05. This suggests that at least one of the model’s independent
variables (CPA, ANX and DIS) significantly affects the dependent variable: Learning
Achievement (RLA). See Table 5.
Table 5. Summary ANOVA
Sum square df Mean square F P value

Total 3.122.807 108 0.000 0.000 0.000
Error 2.571.511 105 24.491 0.000 0.000
Regression 551.296 3 183.765 7.504 0.000
Note. Significance level is p < 0.05.
Table 6 presents the results of the multiple regression analysis, which tries to explain
the relationship of a dependent variable: Relationship to learning achievement (RLA)
with the independent variables: (CPA, ANX and DIS), so such that based on its non-
standardized and standardized coefficients, as well as the standard errors (SE), t-values,
p-values and confidence intervals, it can be explained that the variable: Distraction
(DIS), for each increase of one unit in the variable (DIS), the dependent variable (RLA)
is expected to decrease by 0.370 units, keeping the other independent variables constant.
According to the p-value (0.006), it is less than the significance level (p < 0.05), which
indicates that the relationship between I and JP is statistically significant.
Table 6. Summary coefficients
Unstandardized Standardized SE T value P value 2.5% 97.5%

coefficients coefficients
Cell phone 0.323 0.214 0.139 2.317 0.022 0.047 0.599
addiction
(CPA)
Distraction −0.370 −0.265 0.131 2.829 0.006 −0.629 −0.111
(DIS)
Anxiety 0.357 0.305 0.113 3.154 0.002 0.132 0.581
(ANX)
Intercept 8.606 0.000 1.897 4.536 0.000 4.844 12.368
Note. Significance level is p < 0.05.
Regarding Anxiety (ANX), the dependent variable: Relationship to learning achieve-

ment (RLA), is expected to increase by 0.357 units, keeping the other independent vari-
ables constant. On the other hand, the standardized coefficient tells us that an increase of
one standard deviation in (ANX) is associated with an increase of 0.305 standard devia-
tions in the dependent variable (RLA). The p-value (0.002) is less than 0.05, indicating
that the relationship between (ANX) and the variable (RLA) is statistically significant.
On the other hand, for the variable: Cell phone addiction (CPA), for each increase
it makes, the variable (RLA) is expected to increase by 0.323 units so that the other
independent variables remain constant. Likewise, the standardized coefficient indicates
that an increase of one standard deviation in (CPA) is associated with an increase of
0.214 standard deviations in the dependent variable. The p-value (0.022) is less than
0.05.
Figure 1 is a graphical representation Cronbach’s alpha is a coefficient used to mea-
sure the internal consistency of a set of scales or items. It’s commonly used to assess the
reliability of research measurement tools. The values range from 0 to 1, with a higher
value indicating greater internal consistency and reliability.
Fig. 1. Cronbach’s alpha – PLS- SEM
Cell phone addiction (CPA) is considered a formative construct; hence Cronbach’s

alpha is not calculated. Formative constructs are made up of items that together create
the concept; that is, the sum of the parts creates the whole. In these cases, a high corre-
lation between items is not necessarily expected, so Cronbach’s alpha, which precisely
measures these correlations, is not considered a relevant indicator.
On the contrary, the variables of anxiety (ANX), distraction (DIS), and the relation-
ship to learning achievement (RLA) are reflective constructs, and therefore, Cronbach’s
alpha has been calculated. The correlation among the items defines reflective constructs;
each item reflects the underlying construct.
The values obtained for ANX (0.726) and DIS (0.747) show moderate internal con-
sistency. This suggests that the items making up these variables are reasonably correlated,
providing evidence that they measure the same underlying construct.
The highest value is found in RLA (0.866), which indicates good internal consistency.
This suggests that the items chosen for this variable are strongly correlated, strengthening
the reliability of this construct’s measurement.
Cronbach’s alpha coefficients suggest that the anxiety, distraction, and relationship
to learning achievement scales are consistent and dependable. On the other hand, the
cell phone addiction measure accurately represents the concept without needing a solid
correlation among its items since it is a formative construct.
Figure 2 is a graphical representation of R2 (coefficient of determination) using
Partial Least Squares Structural Equation Modeling (PLS-SEM), which explains that
Anxiety (ANX) -> Relationship to learning achievement (RLA): Anxiety has a posi-
tive and significant relationship with learning achievement. For each unit that anxiety
increases, learning achievement averages 0.337 units. This relationship is statistically
significant (p < 0.001), which means that it is highly likely that this relationship is
not due to chance. Regarding Cell phone addiction (CPA) -> Relationship to learning
achievement (RLA): Mobile phone addiction has a positive and significant relationship
with learning achievement.
Fig. 2. Coefficient of determination R2 – PLS- SEM
For every unit that mobile phone addiction increases, learning achievement increases
by an average of 0.200 units. This relationship is statistically significant (p = 0.032),
indicating that there is a moderate probability that this relationship exists and is not due to
chance, and the Distraction (DIS) -> Relationship to learning achievement (RLA): Dis-
traction has a negative and significant relationship with learning achievement. Learning
achievement decreases on average by 0.243 units for each unit that distraction increases.
This relationship is statistically significant (p = 0.029), indicating a moderate prob-
ability that this relationship is not due to chance. The R2 of 0.223 suggests that 22.3% of
the variability in the “Relationship with learning achievement” (RLA) can be explained
by the variables “Anxiety” (ANX), “Mobile phone addiction” (CPA), and “Distraction”
(DIS). Therefore, it is inferred that there are other factors not included in the model that
are also influencing the RLA.
Table 7. Hypothesis testing - Bootstrapping
Original Sample Standard T statistics P values

sample (O) mean (M) deviation (|O/STDEV|)
(STDEV)
H1 Anxiety 0.337 0.331 0.085 3.969 0.000
(ANX) ->
Relationship to
learning
achievement
(RLA)
H2 Cell phone 0.200 0.230 0.108 1.853 0.032
addiction (CPA)
-> Relationship to
learning
achievement
(RLA)
H3 Distraction -0.243 -0.231 0.128 1.898 0.029
(DIS) ->
Relationship to
learning
achievement
(RLA)
Table 7 presents the p-values that have been adequately adjusted through bootstrap-
ping so that the p-values are more precise in analysing models of a certain complexity
with several predictors. Regarding the H1 Anxiety (ANX) -> Relationship to learning
achievement (RLA): Anxiety has a positive and significant impact on learning achieve-
ment (p < 0.001); The H2 Cell phone addiction (CPA) -> Relationship to learning
achievement (RLA): Mobile phone addiction has a positive and significant impact on
learning achievement (p = 0.032) and the H3 Distraction (DIS) -> Relationship to learn-
ing achievement (RLA): Distraction has a negative and significant impact on learning
achievement (p = 0.029); therefore the three hypotheses are accepted.
4 Conclusions
This study examines how cell phone addiction, anxiety, and distraction relate to learning
achievement. Through various statistical techniques, the results consistently show that
these three variables are significant predictors of learning achievement. It was found that
anxiety and cell phone addiction have a positive correlation with learning achievement.
These findings may seem counterintuitive at first glance. However, a deeper analysis
provides a more nuanced view. In the case of anxiety, it could be argued that a certain
level of anxiety can be beneficial for learning, as it can heighten students’ awareness
and focus. As for cell phone addiction, although it’s generally considered harmful, this
study suggests it can have a positive impact on learning. This could be because mobile
phones are increasingly being used as learning tools, providing access to a wide range
of educational resources.
On the other hand, distraction, which can also be related to mobile phone use, has a
negative impact on learning achievement. This suggests that while mobile phone use can
facilitate learning by providing access to resources, it can also divert students’ attention
away from learning tasks.
Despite these significant associations, these three variables only explain 22.3% of the
variability in learning achievement. This is a crucial finding as it signals that, although cell
phone addiction, anxiety, and distraction are influential factors, there are other factors
also impacting learning achievements. These overlooked factors may include aspects
like students’ study skills, their motivation, the quality of teaching, socioeconomic
conditions, among others.
Overall, the results of this study underscore the complexity of the factors affecting
learning achievement. Although cell phone addiction, anxiety, and distraction play a
role, it’s clear that there is a variety of other factors at play. This suggests that efforts to
improve learning achievement must adopt a holistic approach, taking into account not
only these factors but also other elements that can influence learning. Additionally, the
need to conduct further research to explore these additional factors and better understand
how they interact with cell phone addiction, anxiety, and distraction to influence learning
achievement becomes evident.
Consequently, the role that cell phone addiction, anxiety, and distraction play in
learning achievement is an area that warrants ongoing attention and more detailed study.
This research provides a valuable foundation for further exploration of these factors and
their interaction with other elements that may impact learning achievement.
References
1. Liu, Y., et al.: Predictors of depressive symptoms in college students: a systematic review and
meta-analysis of cohort studies. J. Affect. Disord. 244, 196–208 (2019). https://doi.org/10.
1016/j.jad.2018.10.084
2. Rahmillah, F.I., Tariq, A., King, M., Oviedo-Trespalacios, O.: Is distraction on the road
associated with maladaptive mobile phone use? A systematic review. Accid. Anal. Prev. 181,
106900 (2023). https://doi.org/10.1016/j.aap.2022.106900
3. Billieux, J., Maurage, P., Lopez-Fernandez, O., Kuss, D.J., Griffiths, M.D.: Can disordered
mobile phone use be considered a behavioral addiction? An update on current evidence and
a comprehensive model for future research. Curr. Addict. Rep. 2(2), 156–162 (2015). https://
doi.org/10.1007/s40429-015-0054-y
4. Sunday, O.J., Adesope, O.O., Maarhuis, P.L.: The effects of smartphone addiction on learning:
a meta-analysis. Comput. Hum. Behav. Rep. 4, 100114 (2021). https://doi.org/10.1016/j.chbr.
2021.100114
5. Lin, Y., Zhou, X.: Bedtime smartphone use and academic performance: a longitudinal analysis
from the stressor-strain-outcome perspective. Comput. Educ. Open 3, 100110 (2022). https://
doi.org/10.1016/j.caeo.2022.100110
6. Amez, S., Baert, S.: Smartphone use and academic performance: a literature review. Int. J.
Educ. Res. 103, 101618 (2020). https://doi.org/10.1016/j.ijer.2020.101618
7. Al-Ansi, A.M., Hazaimeh, M., Hendi, A., Al-hrinat, J., Adwan, G.: How do social media
influencers change adolescents’ behavior? An evidence from Middle East Countries. Heliyon
9(5), e15983 (2023). https://doi.org/10.1016/j.heliyon.2023.e15983
8. Delgado-Gutierrez, P., Rojas-Carrera, O., Gutierrez-Aguilar, A., Gutierrez-Aguilar, O.: Influ-
ence of masculinity and femininity, family dysfunction and extroversion and introversion on
the role of aggressor victim and spectator in Cyberbullying. In: Proceedings - 2021 4th
International Conference on Inclusive Technology and Education, CONTIE 2021, pp. 76–81
(2021)
9. Gutierrez-Aguilar, O., Escobedo-Maita, P., Calliñaupa-Quispe, G., Vargas-Gonzales, J.C.,
Torres-Huillca, A.: The use of social networks, usefulness and ease of use, enjoyment through
positive emotions and their influence on school satisfaction mediated by school achievement.
In: Iberian Conference on Information Systems and Technologies, CISTI 2022 (2022)
10. Huanca, S.C., Aguilar, A.G., Perez, A.B.D., Aguilar, O.G.: The internalization of ICT and its
influence on emotional states in university students. In: Iberian Conference on Information
Systems and Technologies, CISTI 2021 (2021)
11. Kim, E., Cho, I., Kim, E.J.: Structural equation model of smartphone addiction based on adult
attachment theory: mediating effects of loneliness and depression. Asian Nurs. Res. 11(2),
92–97 (2017). https://doi.org/10.1016/j.anr.2017.05.002
12. Zhou, D., Liu, J., Wang, T., Liu, J., Li, G.: Relationships among problematic smartphone
use, mathematics anxiety, learning interest, and achievement: a multiple mediation model.
Comput. Hum. Behav. 129, 107171 (2022). https://doi.org/10.1016/j.chb.2021.107171
13. Justicia-Galiano, M.J., Martín-Puga, M.E., Linares, R., Pelegrina, S.: Math anxiety and math
performance in children: the mediating roles of working memory and math self-concept. Br.
J. Educ. Psychol. 87(4), 573–589 (2017). https://doi.org/10.1111/bjep.12165
14. Yang, X., Khan, A.U.: Validity and reliability of factors causing information-seeking anx-
iety during information-seeking behaviors. Collect. Curation (2023). ahead-of-print(ahead-
of-print). https://doi.org/10.1108/CC-11-2022-0040
15. Smiley, P.C., Bedford, R.L., Smiley, E.D.: The impact of test anxiety and test bias on the
absence of African American males in STEM. In: Robins, A.G., Knibbs, L., Ingram, T.N.,
Weaver, M.N., Hilton, A.A. (eds.) Young, Gifted and Missing, vol. 25, pp. 87–97. Diversity
in Higher Education. Emerald Publishing Limited (2022)
16. Liu, Z.: Reading in the age of digital distraction. J. Doc. 78(6), 1201–1212 (2022). https://
doi.org/10.1108/JD-07-2021-0130
17. Nema, P., Srivastava, R., Bhalla, R., Chakarboty, A.: Impact of social media distraction on
student evaluation of teacher effectiveness. Int. J. Educ. Manag. 37(2), 300–313 (2023).
https://doi.org/10.1108/IJEM-10-2022-0389
18. Hynes, M.: The smartphone: a weapon of mass distraction. In: The Social, Cultural and
Environmental Costs of Hyper-Connectivity: Sleeping Through the Revolution, pp. 71–84.
Emerald Publishing Limited (2021)
19. Scott, D.A., McCormick, K., Smith, K., Budlong, E., Vitonis, A.: The prevalence, risks and
associations of problematic cell phone use and sexting on mental health. Ment. Health Soc.
Incl. 25(1), 52–62 (2021). https://doi.org/10.1108/MHSI-08-2020-0053
20. Enez Darcin, A., Kose, S., Noyan, C.O., Nurmedov, S., Yılmaz, O., Dilbaz, N.: Smartphone
addiction and its relationship with social anxiety and loneliness. Behav. Inf. Technol. 35(7),
520–525 (2016)
21. Lin, Y., Liu, Y., Fan, W.: How does mobile ICT affect psychological unease? A longitudinal
study (2020)
22. Alhassan, A.A., Alqadhib, E.M., Taha, N.W., Alahmari, R.A., Salam, M., Almutairi, A.F.:
The relationship between addiction to smartphone usage and depression among adults: a cross
sectional study. BMC Psychiatry 18(1), 1–8 (2018)
23. Kim, K., Milne, G.R., Bahl, S.: Smart phone addiction and mindfulness: an intergenera-
tional comparison. Int. J. Pharm. Healthc. Mark. 12(1), 25–43 (2018). https://doi.org/10.
1108/IJPHM-08-2016-0044
24. Chen, W.: Exploring digital natives’ mobile addiction syndrome in Taiwan: psychological
issues and beyond. Inf. Technol. People 36(3), 1326–1355 (2023). https://doi.org/10.1108/
ITP-08-2021-0633
25. Horwood, S., Anglim, J.: Self and other ratings of problematic smartphone use: the role of
personality and relationship type. Comput. Hum. Behav. 116, 106634 (2021). https://doi.org/
10.1016/j.chb.2020.106634
26. Han, S., Yi, Y.J.: How does the smartphone usage of college students affect academic
performance? J. Comput. Assist. Learn. 35(1), 13–22 (2019)
27. Lin, Y., Liu, Y., Fan, W., Tuunainen, V.K., Deng, S.: Revisiting the relationship between
smartphone use and academic performance: a large-scale study. Comput. Hum. Behav. 122
(2021). https://doi.org/10.1016/j.chb.2021.106835
28. Lin, Y.H., Chang, L.R., Lee, Y.H., Tseng, H.W., Kuo, T.B.J., Chen, S.H.: Development and
validation of the smartphone addiction inventory (SPAI). PLoS ONE 9(6) (2014). https://doi.
org/10.1371/journal.pone.0098312
29. Arpaci, I., Kocadag Unver, T.: Moderating role of gender in the relationship between big five
personality traits and smartphone addiction. Psychiatr. Q. 91(2), 577–585 (2020). https://doi.
org/10.1007/s11126-020-09718-5
30. Suryanto, A., et al.: Study of working from home: the impact of ICT anxiety and smartphone
addiction on lecturers at NIPA School of Administration on job performance. Heliyon 8(12),
e11980 (2022). https://doi.org/10.1016/j.heliyon.2022.e11980
31. Shi, D., Maydeu-Olivares, A., DiStefano, C.: The relationship between the standardized root
mean square residual and model misspecification in factor analysis models. Multivar. Behav.
Res. 53(5), 676–694 (2018)
32. Kenny, D.A., Kaniskan, B., McCoach, D.B.: The performance of RMSEA in models with
small degrees of freedom. Sociol. Methods Res. 44(3), 486–507 (2015)
Factors that Influence Digital Identity
Management in Personal Learning
Environments
O. Gutierrez-Aguilar1(B) , A. Duche Pérez1 , J. Leon-Lucano1 ,

F. Cordova-Buiza2,3 , and S. Chicaña-Huanca1
1 Universidad Católica de Santa María, Arequipa, Perú
{ogutierrez,aduche,jair.leon,schicanah}@ucsm.edu.pe
2 Universidad Privada del Norte, Lima, Perú
franklin.cordova@upn.edu.pe
3 Universidad Continental, Huancayo, Perú
Abstract. The main objective of this research is to determine the influential

factors in digital identity management in personal learning environments. The
methodology used for the study corresponds to nonexperimental research; the
questionnaire was applied to a sample of 292 randomly selected university stu-
dents, and validity and reliability tests and analysis were used to update the model.
For data analysis, modeling of structural equations of partial least squares was
used. The most prominent results indicate strong causal relationships between the
exchange of information and content and interaction through digital technologies.
Similarly, interaction through digital technologies and copyright and licensing
protects the environment and its influence on the management of digital identity.
Keywords: Personal learning environments · PLE · digital identity · PLS-SEM
1 Introduction
The personal learning environments (PLE) are tools, resources, services and personal
tools students use to manage their learning and achieve their educational objectives [1].
The management of PLEs is focused on a student-centered pedagogical approach, so
that students can personalize their learning experiences, connect with their peers and
teachers, and access educational resources [2]. Educational experiences based on social
constructivism recognize that using personal learning environments helps create collab-
orative environments [3]. In the same way, it promotes self-regulated learning (SRL) in
terms of research topics: adoption; integration; model, architecture and framework; and
recommendations [4].
The acquisition and updating of teachers’ digital skills is a fundamental require-
ment today in the knowledge society, in such a way that the various educational
resources available generate collaborative environments (blog, wiki), resources for learn-
ing management (ePortfolio, PLE, educational video, and various digital channels),
https://doi.org/10.1007/978-981-99-7353-8_3
Factors that Influence Digital Identity Management in Personal Learning Environments 27
such as ICT Competency Standards for Teachers (ICT-CST) (UNESCO, 2008), and ICT
Competencies’ Pentagon (MEN, 2013) [5].
There are studies on the relationship of PLEs and what is proposed in the Com-
mon Framework for Teacher Digital Competence (MCCDD), which addresses critical
aspects of the integration of technologies in education with the development of digital
skills, from a perspective of students and teachers, among which we have: Information
and information literacy, for the development of search capacities, evaluation and orga-
nization of digital information in an effective way. Communication and collaboration,
through the PLEs, encourage communication between students and teachers to promote
online collaboration in their educational practices—the creation of digital content and
its adaptation and personalization. Security, by using and managing a PLE, teachers can
acquire knowledge and develop skills in digital security, to ensure safe environments
for students. Problem-solving implies a good selection and use of digital tools and
resources. Through integration with the PLE, innovation and continuous improvement
allow teachers to participate in learning communities and keep current.
In a super computerized context, the use of personalized learning environments
(PLE), the selection of efficient and high-quality resources for implementing these virtual
environments, present some difficulties, due to the large amount of electronic content
(e-content) qualified. On the other hand, the generation of electronic content for these
systems is an expensive process that requires much time, apart from the style or type
of intelligence of each student, therefore, the need arises to use dynamic, reusable and
low-cost electronic content. Cost for the implementation of the PLE [6].
The autonomy of learning is promoted with the PLE this because students can make
their decisions about their learning process and adapt it to their needs and preferences
[7, 8], on the other hand, the PLE enable the integration of technologies to facilitate access
to information, communication and especially collaboration through digital channels [9,
10]. Social networks, for their part, encourage collaboration by sharing information
and content so that the integration and re-elaboration of digital content in collaborative
environments enrich their learning processes and promote the construction of knowledge
[11, 12].
With digital identity being a critical condition in elaborating their cognitive process
to achieve significant learning, personalized learning implies adapting the conditions
for learning to the needs, interests, and learning styles of each student [4, 13], This
implies programming; with it, educational project planning, development, and online
evaluation use programming languages. Also, the development of computational think-
ing skills, video games, creating algorithms and generalizing them, data processing,
robotics, among others.
The processes of reflection and metacognition assist students in becoming aware
of their learning processes and enhancing their ability to self-regulate, just as identity
management encourages levels of adaptation and the selection of necessary and appro-
priate tools in a PLE [14, 15]. The development of digital skills allows students in the
PLE to optimize their navigation, search, and filtering skills of information and digi-
tal content in the same way as the integration, creation, curation [16] and reprocessing
of digital content. Essential to navigate in today’s world [17, 18], expecting copyright
and environmental protection within a framework of responsibility and ethics. It is also
true that there are significant obstacles to its implementation, particularly in developing
nations, such as the existing digital divide caused by students’ lack of digital literacy
and educators’ resistance to change [19, 20].
On the other hand, evaluating and monitoring learning in unique environments allows
students and teachers to measure progress and adjust when necessary [21, 22]. Likewise,
the interconnection that makes access to learning resources and content from multiple
sources and formats more efficient enables students to make their academic work more
efficient [23, 24]. On the other hand, PLEs are highly flexible and adaptable, allowing
university students to modify and customize them according to their changing needs,
thus facilitating lifelong learning [25, 26].
With the personalization and autonomy in the PLE, without a doubt, the motivation
and commitment of the students in their learning processes increase, making them more
responsible and contributing to their academic development [14], For its part, the role of
the teacher changes by being a facilitator or mediator that guides the learning process,
contributing to the student’s ability to navigate and use the tools and resources available
more effectively [12, 19], in the same way, developing in fundamental skills such as
thinking critically, problem-solving, collaboration and communication to learn in an
autonomous and personalized way [11, 20]. Based on the above considerations, the
following hypotheses are proposed:
Collaboration through digital channels:
H1a There is a statistically significant effect between Collaboration through Digital
Channels and Integration and Reprocessing of Digital Content.
H1b Collaboration through Digital Channels positively influences Interaction through
Digital Technologies.
Sharing information and content:
H2a There is a statistically significant effect between Information and Content Sharing
and Integration and Reprocessing of Digital Content.
H2b Information and Content Sharing positively influences Interaction through Digital
Technologies.
Copyright and licenses:
H3a Copyright and Licensing positively influences Digital Identity Management.
H3b There is a statistically significant effect between Integration and Reprocessing of
Digital Content and Copyright and Licensing.
H3c Interaction through Digital Technologies positively influences Copyright and
Licensing.
Navigation, searching, and filtering information and digital content:
H4a Navigation, Search, and Filtering of Information and Digital Content positively
influences Digital Identity Management.
H4b There is a statistically significant effect between Navigation, Search, and Filtering
of Information and Digital Content and Integration and Reprocessing of Digital Content.
H4c Navigation, Search, and Filtering of Information and Digital Content positively
influences Interaction through Digital Technologies.
Programming:
H5a Programming positively influences Integration and Reprocessing of Digital Content.
H5b There is a statistically significant effect between Programming and Interaction
through Digital Technologies.
Environmental protection:
H6a Environmental Protection positively influences Digital Identity Management.
H6b There is a statistically significant effect between Environmental Protection and
Integration and Reprocessing of Digital Content.
H6c Environmental Protection positively influences Interaction through Digital Tech-
nologies.
2 Methodology
The study used an adaptation of what is contemplated in the Common Framework of
Teacher Digital Competence (MCCDD). Likewise, the background is the European
Framework of Digital Competences for Educators (DigCompE-du), based on the Com-
mon Framework of Digital Competence for Teachers (MCCDD). The DigCompEdu
framework provides a framework to assess and develop the digital skills of educators
and PLEs. A sample of 292 university students of both sexes from the professional
schools of advertising and psychology was used, with 66.4% women and 33.6% men,
ages between 18 and 27 years, with a standard deviation of SD: 1.84. The instrument
was applied in March 2023.
Table 1. Reliability indices.
Variables Cronbach’s alpha α McDonald’s ω

NSFIDC 0.708 0.718
IRDC 0.800 0.796
CAL 0.805 0.810
PROG 0.857 0.859
EP 0.836 0.844
DIM 0.810 0.815
CDC 0.814 0.812
ICS 0.820 0.833
IDT 0.716 0.728
The observed variables are Collaboration through Digital Channels (CDC), Copy-
right and Licensing (CAL), Digital Identity Management (DIM), Environmental Pro-
tection (EP), Information and Content Sharing (ICS), Integration and Reprocessing
of Digital Content (IRDC), Interaction through Digital Technologies (IDT), Naviga-
tion, Search, and Filtering of Information and Digital Content (NSFIDC), Programming
(PROG). Table 1 presents the reliability indices for the observed variables, which are
acceptable for both cases, using Cronbach’s alpha (α) and McDonald’s (ω).
Subsequently, an exploratory factor analysis (EFA) was performed, and the result for
the Kaiser-Meyer-Olkin (KMO) test was 0.900, indicating that the data are adequate to
perform an EFA and Bartlett’s test with a p-value of <0.001, which means that there are
significant correlations between the variables and that it is advisable to perform the EFA.
The factors were loaded with an exit criterion of 0.400 and the minimum residue extrac-
tion method using the Oblimin rotation method, whose results were satisfactory. The
research development followed a systematic and rigorous process of data classification
and analysis with the statistical software Jamovi (v. 2.3.24).
3 Results
A correlation matrix was made using the non-parametric Spearman coefficient (p, rho)
since it was verified that there is no normality in the data structure. Table 2 presents
the matrix finding the strongest correlations between IDT and IRDC (0.707), CAL
and IRDC (0.728), and CMDC and DIM (0.727), all highly significant (p < .001).
The weakest correlation between PROG and DIM (−0.066) indicates no significant
monotonic relationship between these two variables.
Table 2. Correlation matrix.
Variable NSFIDC IRDC CAL PROG EP DIM CDC ICS IDT

NSFIDC —
IRDC 0.505*** —
CAL 0.438*** 0.728*** —
PROG 0.112 0.617*** 0.630*** —
EP 0.408*** 0.642*** 0.702*** 0.642*** —
DIM 0.663*** 0.342*** 0.370*** -0.066 0.365*** —
CDC 0.632*** 0.614*** 0.521*** 0.292*** 0.561*** 0.727*** —
ICS 0.420*** 0.517*** 0.575*** 0.531*** 0.528*** 0.433*** 0.619*** —
IDT 0.570*** 0.707*** 0.649*** 0.465*** 0.654*** 0.524*** 0.666*** 0.709*** —
Note. * p < .05: significant correlation, ** p < .01: very significant correlation, *** p < .001:
highly significant correlation.
The variance inflation factor (VIF) test must be done because the analysis model
is for formative constructs. The VIF test is a way to measure multicollinearity in a
regression model, which is when two or more independent variables in the model are
highly correlated. The values obtained in the test are between 1.337 (DIM6) and 2.686
(PROG6), which indicates that all the VIFs are below 5 [27–29], therefore, there are no
multicollinearity problems in the regression model. Thus, the independent variables in
the model are not highly correlated, and the regression coefficients can be interpreted
with greater confidence.
Fig. 1. Model R2 - SmartPLS.
Figure 1 is a graphical representation of the reliability and validity analysis of the

model based on the R2 (coefficient of determination). The R2 for the variable Integration
and reprocessing of digital content (IRDC) is 0.510; that is, the predictor variables of the
model explain 51.0% of the variance, while for the variable Interaction through digital
technologies (IDT), whose R2 is 0.570, that is, 57.0% of the variance is explained by its
predictor variables of the proposed model. With an R2 of 0.368, the predictor variables
IRDC and IMTD can explain 36.8% of the variation in copyright and licenses (CAL).
The Digital Identity Management (DIM) variable has an R2 of 0.499; that is, 49.9% is
explained by the variables: EP; CAL and NSFIDC.
Table 3. R-square - overview
R-square (R2 ) R-square adjusted

Copyright and Licensing (CAL) 0.368 0.363
Digital Identity Management (DIM) 0.499 0.494
Integration and Reprocessing of Digital Content (IRDC) 0.510 0.501
Interaction through Digital Technologies (IDT) 0.570 0.563
The values R2 and R2 adjusted are statistical measures used to understand the good-
ness of fit of a model in regression analysis. They tell us the proportion of variance for
a dependent variable explained by an independent variable or variables in a regression
model. Here is a breakdown of its meaning: R-square (R2 ): This is also known as the
coefficient of determination. Represents the proportion of variance in the dependent vari-
able that independent variables can explain. For example, an R2 of 0.570 for interaction
through digital technologies (RTD) suggests that the model can account for 57% of RTD
variability.
The closer this value is to 1, the better the model’s fit. R-square adjuster: While the R-
square increases regardless of the effect of the predictors (it will never decrease), the R2
adjuster increases only when a significant predictor is added to the model and decreases
when a non-significant predictor is added. It is always less than or equal to square R. It
considers the number of predictors in the model and the number of observations. The
square R adjuster is a more robust measure of the explanatory power of the regression
model.
In Table 3, we see that the models for each of Copyright and Licensing (CAL), Digital
Identity Management (DIM), Digital Content Integration and Reprocessing (IRDC), and
Interaction through Digital Technologies (RTD) explain a significant proportion of the
variance in their respective dependent variables. For example, the RTD model explains
56.3% to 57.0% of the variance, indicating a robust model.
Table 4. Bootstrapping test results.
Hypothesis Original sample (O) Sample mean (M) Standard deviation T statistics P values
(STDEV) (|O/STDEV|)
H1a Collaboration 0.105 0.108 0.064 1.650 0.050
through Digital
Channels (CDC) ->
Integration and
Reprocessing of
Digital Content
(IRDC)
H1b Collaboration 0.051 0.065 0.071 0.714 0.237
through Digital
Channels (CDC) ->
Interaction through
Digital Technologies
(IDT)
H2a Information and 0.099 0.104 0.069 1.433 0.076
Content Sharing (ICS)
-> Integration and
Reprocessing of
Digital Content
(IRDC)
H2b Information and 0.471 0.457 0.066 7.082 0.000
Content Sharing (ICS)
-> Interaction through
(IDT)
(continued)
Table 4. (continued)
(STDEV) (|O/STDEV|)
H3a Copyright and 0.132 0.143 0.058 2.271 0.012
Licensing (CAL) ->
Digital Identity
Management (DIM)
H3b Integration and 0.361 0.369 0.064 5.604 0.000
Reprocessing of
Digital Content
(IRDC) -> Copyright
and Licensing (CAL)
H3c Interaction 0.327 0.338 0.063 5.162 0.000
through Digital
Technologies (IDT)
-> Copyright and
Licensing (CAL)
H4a Navigation, 0.169 0.172 0.059 2.873 0.002
Search, and Filtering
of Information and
Digital Content
(NSFIDC) -> Digital
Identity Management
(DIM)
H4b Navigation, 0.305 0.297 0.058 5.260 0.000
of Information and
Digital Content
(NSFIDC) ->
Integration and
Reprocessing of
Digital Content
(IRDC)
H4c Navigation, 0.153 0.151 0.056 2.739 0.003
of Information and
Digital Content
(NSFIDC) ->
Interaction through
(IDT)
H5a Programming 0.264 0.265 0.071 3.732 0.000
(PROG) ->
Integration and
Reprocessing of
Digital Content
(IRDC)
H5b Programming 0.096 0.101 0.064 1.496 0.067
(PROG) ->
Interaction through
(IDT)
(continued)
(STDEV) (|O/STDEV|)
H6a Environmental 0.517 0.515 0.063 8.262 0.000
Protection (EP) ->
Digital Identity
Management (DIM)
H6b Environmental 0.131 0.139 0.067 1.959 0.025
Protection (EP) ->
Integration and
Reprocessing of
Digital Content
(IRDC)
H6c Environmental 0.144 0.153 0.059 2.460 0.007
Protection (EP) ->
Interaction through
(IDT)
Note. Level of significance is p < 0.05.
Table 4 presents the bootstrapping test results that make it possible to know that
the relationships are significantly different from zero, which is carried out through a
process to extract many resamples (10,000) by replacing the original sample. And then
estimate the model parameters for each bootstrap resample. The standard error of an
estimate is inferred from the standard deviation of the bootstrap estimates. Considering
the significance level for the p-value (p < 0.05), hypotheses H1b, H2a and H5b are
rejected, and the other hypotheses are accepted.
4 Conclusions
According to the results, we reached the following conclusions: Collaboration through
Digital Channels: This concept refers to interaction and cooperation in online environ-
ments. In the context of our study, collaboration through digital channels impacts the
integration and reprocessing of digital content. However, the relationship between col-
laboration via digital channels and interaction via digital technologies could be more
transparent and require additional research to comprehend the dynamics fully.
Sharing Information and Content: The act of sharing information and content in
digital environments plays a role in integrating and reprocessing digital content, although
the relationship could be more robust. However, there is a strong influence of sharing
information and content on interactions through digital technologies, suggesting that
this behaviour may encourage greater participation in digital environments.
Copyright and Licences: The study shows that the norms and laws that regulate the
use of digital content and technologies, i.e., copyrights and licenses, are closely tied to
digital identity management. Moreover, these legal aspects are linked to integrating and
reprocessing digital content and interaction through digital technologies.
Navigation, Search, and Filtering of Digital Information and Content: How indi-
viduals seek, select, and navigate information in digital environments shows a strong
and significant relationship with digital identity management, integrating and repro-
cessing digital content, and interaction through digital technologies. This suggests that
other digital behaviours and practices can affect how individuals interact with online
information.
Programming: Programming skills, i.e., the ability to create and modify digital soft-
ware and applications, are related to the integration and reprocessing of digital content.
This suggests that programming skills facilitate the customization and manipulation of
digital content. The relationship with interaction through digital technologies must be
more evident and might require further exploration.
Environmental Protection: The study demonstrates that the consideration of envi-
ronmental impacts in the creation and use of digital technologies, or environmental
protection, has a close relationship with digital identity management, the integration
and reprocessing of digital content, and interaction via digital technologies. This finding
could suggest that attitudes and behaviours related to sustainability are reflected in how
individuals handle their digital identity and interact with digital content and technologies.
These findings suggest that various behaviours and practices in digital environments
are interrelated. Understanding these relationships may be vital to promoting effective
and responsible digital practices.
Some recommendations for future research are presented: Clarifying relationships: It
would be beneficial for future studies to focus on the less clear relationships identified in
this study. In particular, the relationship between collaboration through digital channels
and interaction through digital technologies could provide valuable information.
We recommend conducting additional research to determine how programming skills
impact digital behaviors and interactions. The impact of these skills on digital identity
management could have significant implications. Longitudinal approach: A longitudi-
nal research approach is recommended to observe how these behaviors and digital rela-
tionships evolve. This can give us a dynamic perspective on managing digital identity.
Intercultural Perspective: It is recommended that future research examine these relation-
ships in various cultural contexts. This can provide insight into the universality of these
findings and broaden our understanding of the global digital landscape. Environmen-
tal considerations: We recommend that future studies further investigate how attitudes
towards environmental protection shape digital behaviours. This could reveal interesting
layers of influence in managing digital identity. Development of interventions: Based
on the results of this research, we recommend designing and evaluating interventions
that promote responsible digital practices. This could lead to practical applications that
promote ethical and sustainable digital behaviours.
References
1. Parra-González, M.E., Parejo-Jiménez, N., Conde-Lacárcel, A., Olmedo-Moreno, E.M.: Rela-
tionship between the personal learning environment and the educational level of unaccom-
panied foreign minors. Int. J. Intercult. Relat. 80, 17–26 (2021). https://doi.org/10.1016/j.iji
ntrel.2020.10.009
2. Wei, W., Mejia, C., Qi, R.: A personal learning environment (PLE) approach to mobile
teaching and learning on a short-term study abroad. J. Hosp. Leis. Sport Tour. Educ. 29,
100296 (2021). https://doi.org/10.1016/j.jhlste.2020.100296
3. Budiastuti, P., Khairudin, M., Santosa, B., Rahmatullah, B.: The use of personal learning
environment to support an online collaborative strategy in vocational education pedagogy
course. Int. J. Interact. Mob. Technol. 17(2), 24–41 (2023). https://doi.org/10.3991/ijim.v17
i02.34565
4. Nan Cenka, B.A., Santoso, H.B., Junus, K.: Using the personal learning environment to sup-
port self-regulated learning strategies: a systematic literature review. Interact. Learn. Environ.
(2022). https://doi.org/10.1080/10494820.2022.2120019
5. Cacheiro-González, M.L., Sánchez-Romero, C., González-Lorenzo, J.M.: Digital teacher
competence. Turk. Online J. Educ. Technol. (Spec. Issue IETC), 180–190 (2017)
6. Yilmaz İnce, E., İnce, M.: Automatic e-content sequencing system for personalised learning
environments by using fuzzy AHP based on multiple intelligences. J. Inf. Sci. 47(6), 821–837
(2021). https://doi.org/10.1177/01655515211041333
7. Castañeda, L., Attwell, G., Dabbagh, N.: Personal learning environments as a framework for
flexible learning: exploring consensus, asking questions and tracing challenges. Edutec (79),
80–94 (2022). https://doi.org/10.21556/edutec.2022.79.2347
8. García-Martínez, J.A., Fallas-Vargas, M.A.: Self-directed learning and personal learning envi-
ronments in Costa Rica university students. Educar 58(2), 373–387 (2022). https://doi.org/
10.5565/rev/educar.1520
9. Dabbagh, N., Kwende, M.: Personal learning environments as digital spaces that are col-
laborative, adaptive, and autonomous. In: ACM International Conference Proceeding Series,
pp. 510–513 (2021)
10. García-Garnica, M., Martínez-Martínez, A., Tomé-Fernández, M., Olmedo-Moreno, E.M.:
The personal learning environment of unaccompanied foreign minors in Spain. J. Int. Migr.
Integr. 23(4), 1621–1644 (2022). https://doi.org/10.1007/s12134-021-00888-0
11. Perifanou, M., A. Economides, A.: Digital skills for building and using personal learning
environments. In: ACM International Conference Proceeding Series, pp. 527–532 (2021)
12. Sockett, G.: Input in the digital wild: online informal and non-formal learning and their
interactions with study abroad. Second Lang. Res. 39(1), 115–132 (2023). https://doi.org/10.
1177/02676583221122384
13. Tur, G., Ramírez-Mera, U., Marín, V.I.: Self-regulated learning and personal learning envi-
ronments in pre-service teacher education: students’ perceptions and proposals for tools and
resources. Revista Complutense de Educacion 33(1), 41–55 (2022). https://doi.org/10.5209/
RCED.71002
14. Martínez-Martínez, A., Tomé-Fernández, M., García-Garnica, M., Olmedo-Moreno, E.M.:
Explanatory model of the relationships between self-efficacy and personal learning envi-
ronments in immigrant minors without a legal Guardian. J. Refug. Stud. 34(4), 4225–4241
(2021). https://doi.org/10.1093/jrs/feab002
15. Serrano-Sánchez, J.L., López-Vicent, P., Gutiérrez-Porlán, I.: Personal learning environments:
strategies and technologies used by Spanish University students. Revista Electronica Educare
25(2) (2021). https://doi.org/10.15359/ree.25-2.22
16. Suárez-Perdomo, A., Byrne, S., Rodrigo, M.J.: Evaluation of “the positive parent”, a Span-
ish web-based program to promote positive parenting in a personal learning environment.
Electron. J. Res. Educ. Psychol. 20(56), 177–200 (2022)
17. Muchlas, Budiastuti, P., Khairudin, M., Santosa, B., Rahmatullah, B.: The use of personal
learning environment to support an online collaborative strategy in vocational education ped-
agogy course. Int. J. Interact. Mob. Technol. 17(2), 24–41 (2023). https://doi.org/10.3991/
ijim.v17i02.34565
18. Wei, W., Mejia, C., Qi, R.: A personal learning environment (PLE) approach to mobile
teaching and learning on a short-term study abroad. J. Hosp. Leis. Sport Tour. Educ. 29
(2021). https://doi.org/10.1016/j.jhlste.2020.100296
19. Ben Rebah, H., Barthes, D., Carnus, M.F.: Personal learning environment: instrument system
for learning beyond the boundaries of the university. Learn. Environ. Res. (2023). https://doi.
org/10.1007/s10984-023-09457-x
20. Tzavara, A., Lavidas, K., Komis, V., Misirli, A., Karalis, T., Papadakis, S.: Using personal
learning environments before, during and after the pandemic: the case of “e-Me”. Educ. Sci.
13(1) (2023). https://doi.org/10.3390/educsci13010087
21. Alserhan, S., Alqahtani, T.M., Yahaya, N., Al-Rahmi, W.M., Abuhassna, H.: Personal learning
environments: modeling students’ self-regulation enhancement through a learning manage-
ment system platform. IEEE Access 11, 5464–5482 (2023). https://doi.org/10.1109/ACCESS.
2023.3236504
22. Ramírez-Mera, U.N., Tur, G., Marin, V.I.: Personal learning environments in online and face-
to-face contexts in Mexican higher education. Open Educ. Stud. 4(1), 148–160 (2022). https://
doi.org/10.1515/edu-2022-0009
23. Carrasco-Sáez, J.L., Butter, M.C., Badilla-Quintana, M.G., Molina-Farfán, J.: Analysis of
psychometric properties and validation of the personal learning environments questionnaire
(B-ple) in higher education students. Sustainability 13(16) (2021). https://doi.org/10.3390/
su13168736
24. García-Martínez, J.A., González-Sanmamed, M., Muñoz-Carril, P.C.: Lifelong learning
and personal learning environments: a productive symbiosis in higher education. Revista
Complutense de Educacion 34(1), 167–177 (2023). https://doi.org/10.5209/rced.77232
25. Castañeda, L., Tur, G., Torres-Kompen, R., Attwell, G.: The influence of the personal learning
environment concept in the educational research field: a 2010–2020 systematized review. Int.
J. Virtual Pers. Learn. Environ. 12(1) (2022). https://doi.org/10.4018/IJVPLE.2022010102
26. Ramos, J.F.C., Álvarez, A.A., Naranjo, A.V.J.: The personal learning environment of pre-
service teachers in physical education. Study of preferences in Chile during the COVID-19
health crisis. Edutec (79), 166–181 (2022). https://doi.org/10.21556/edutec.2022.79.2443
27. Gujarati, D.N., Porter, D.C.: Basic Econometrics, 5th edn. McGraw-Hill Education (2009)
28. Stock, J.H., Watson, M.W.: Introduction to Econometrics, 3rd edn. Pearson (2015)
29. Hair, J.F., Black, W.C., Babin, B.J., Anderson, R.E., Tatham, R.L.: Multivariate Data Analysis,
6th edn. Pearson Prentice Hall (2006)
Ethics, Big Data and Artificial Intelligence:
Exploring Academic Works in the Educational
Landscape
Jones Baroni Ferreira de Menezes1(B) , Cristian Cechinel2 , Roberto Muñoz3 ,

Emanuel Marques Queiroga4 , Vinicius Ramos2 , Tiago Thompsen Primo5 ,
and João Batista Carvalho Nunes1
1 Universidade Estadual do Ceará, Fortaleza, Brazil
{jones.baroni,joao.nunes}@uece.br
2 Universidade Federal de Santa Catarina, Araranguá, Brazil
contato@cristiancechinel.pro.br, v.ramos@ufsc.br
3 Universidad de Valparaíso, Valparaiso, Chile
roberto.munoz@uv.cl
4 Instituto Federal Sul-rio-grandense, Pelotas, Brazil
emanuelmqueiroga@gmail.com
5 Universidade Federal de Pelotas, Pelotas, Brazil
tiagoprimo@gmail.com
Abstract. Big Data and Artificial Intelligence (AI) confer substantial advance-
ments across diverse sectors of society, education included. However, it is imper-
ative to extend the discourse surrounding these interventions to align with ethical
research principles, data usage, and prevailing legislation. Consequently, this study
endeavors to scrutinize scholarly literature spanning the interval between 2011 and
2022, concentrating on the ethical facets recommended for inquiries that meld big
data and artificial intelligence within educational contexts. Employing a biblio-
graphic research approach, the inquiry was conducted on the CAPES Periodicals
Portal, utilizing descriptors such as “ethics,” “big data,” “artificial intelligence,”
and “education.“ Out of a corpus of 84 articles, nine were incorporated into this
research subsequent to the application of inclusion and exclusion criteria. These
works encompass both empirical and theoretical contributions, interlinking big
data and AI with educational settings. Researchers employ documentary sources,
questionnaires, and their individual pedagogical experiences for data collection.
Methodologically, these studies lean towards techniques such as descriptive con-
tent analysis, descriptive statistical analysis, confirmatory factor analysis, and tex-
tual data mining for data analysis. With regard to ethical principles spotlighted in
the studies, salient themes include responsibility, transparency, reliability, and pri-
vacy. The outcomes suggest that crucial lacunae still exist, particularly concerning
aspects like informed consent and other pivotal ethical protocols.
Keywords: Educational Data Analysis · Data Protection · Machine

Intelligence · Ethics · Artificial Intelligence · Information Security
https://doi.org/10.1007/978-981-99-7353-8_4
Ethics, Big Data and Artificial Intelligence 39
1 Introduction
With the emergence and progression of digital technologies, coupled with the ever-
growing realm of virtual experiences, there has been a substantial surge in the flow
of data and information. This substantial influx of data is commonly referred to as Big
Data, encompassing a wealth of significant and valuable insights that can be harnessed for
effective management across diverse domains of expertise [1]. This analytical endeavor
finds its execution through the discipline of Data Science, employing statistical tech-
niques to delve into and dissect data, derive meaningful inferences, all facilitated by
computational methodologies, including the utilization of Artificial Intelligence (AI).
The initial and groundbreaking definition that encompasses Artificial Intelligence
characterizes it as the field dedicated to creating intelligent machines endowed with sub-
stantial processing capabilities, mirroring human-like capacities and functionalities [2].
Its inception and operation are inherently interdisciplinary, weaving together numerous
realms such as Philosophy, Psychology, Logic, Mathematics, Computer Science, Biol-
ogy, and Neurosciences, among others [3]. On a different note, ethics, in its simplified
essence, constitutes the study of morality and the ethical conduct of individuals within
society [4], exemplified by the Declaration of Helsinki in 1964, which stands as the first
set of ethical principles governing medical research involving human subjects [5].
In recent decades, AI has intertwined with education, propelling advancements in
both domains. This fusion offers technological benefits to teaching and learning, sup-
porting educators, students, and enriching essential human skills [6]. Integrating AI
in education opens avenues like promoting collaboration, monitoring virtual learning
engagement, enabling ongoing assessment, and enhancing student learning processes
[7].
Expanding investigations must be tightly linked with scientific and methodologi-
cal rigor, particularly to ensure adherence to ethical research principles and data usage
within the bounds of current legislation. To this end, various countries and regions are
formulating laws and regulations centered around data protection. As an illustration,
the United Nations Educational, Scientific and Cultural Organization (UNESCO) has
pinpointed six challenges essential for the sustainable progress of Artificial Intelligence
in education. Among these, ensuring ethical integrity and transparency in data collec-
tion, utilization, and dissemination holds significant prominence [8]. In a more recent
development, UNESCO has issued a document establishing worldwide benchmarks for
AI Ethics. All 193 member countries have unanimously agreed to and signed this treaty.
The document acknowledges the significance, functionality, and potential of AI, while
also shedding light on emerging hazards linked to the preservation of cultural, social,
and ecological diversity [9].
Considering the context previously presented, we propose the following research
question:
• RQ1: What ethical aspects are suggested to be considered in research related to the
use of big data and artificial intelligence in education?
To answer this question, this study aims to explore academic publications between
2011 and 2022, emphasizing ethical considerations suggested for research merging big
data and artificial intelligence in education.
40 J. B. Ferreira de Menezes et al.
Out of the 84 academic works analyzed, nine were included as they specifically
tackle ethical considerations within research involving big data and artificial intelligence
in education. These chosen studies are featured on the CAPES Journal Portal.
The article is structured in five sections. This first addresses the theoretical foun-
dations related to ethics, big data and artificial intelligence applied to the educational
context. In Sect. 2, the search methods and analysis criteria of the articles are described.
Sections 3 and 4 present the details and discussion of the results. Section 5 brings the
final considerations, in addition to limitations of the work and possibilities for future
investigations.
2 Search and Review Method
Bibliographical research was selected as an investigative method, in which one seeks to

carry out a survey and critical analysis of previously published documents on a given
topic to be researched, in order to deepen, update and generate new knowledge [10].
This survey was carried out by consulting the CAPES Periodicals Portal. This
database was officially created by the Coordination for the Improvement of Higher
Education Personnel (CAPES), foundation of the Ministry of Education (MEC), on
November 11, 2000. It brings together a variety of databases that bring together national
academic and scientific publications and international publications, in addition to patents,
theses and dissertations, among other types of material, covering all areas of knowledge
[11]. Search terms used included “ethics”, “big data”, “artificial intelligence”, and “ed-
ucation”. Only peer-reviewed articles published from January 2011 to December 2022
were included.
The path taken is adapted from the proposition of [12] and has the following structure:
1) Troubleshooting Question:
a) What ethical aspects are suggested to be considered in research related to the use
of big data and artificial intelligence in education?
2) Selection of Studies:
a) Descriptor choices;
b) Database: CAPES Periodicals Portal;
c) Exclusion Criteria: articles that are not in English, Spanish or Portuguese; surveys
that are a literature review; inaccessible text; and articles not directly related to
the object of study.
3) Selected Studies:
a) Quantitative analysis of studies;
b) Qualitative analysis based on the ethical aspects addressed.
In the initial search, considering the search for descriptors anywhere in the article and
using the Boolean AND operator, 84 papers were found. After applying the exclusion
criteria, nine articles remained that were analyzed.
Of the 75 excluded works, 33 (44.0%) were not peer-reviewed articles, two (2.7%) are
written in unselected languages (Italian and Chinese), 10 (13.3%) are literature reviews,
two (2.7%) were not available and 28 (37.3%) have no direct relationship between big
data, AI and the educational theme.
The excluded works had as object of investigation the improvement and development
related to computerized health systems, improvement of efficiency in agriculture, use
in the improvement of transport services and in the improvement of the computational
area.
The analysis of the nine resulting works verified the frequency of publications per
year, as well as the geographic distribution of the studies. A posteriori, the following
stood out: central objective; research typology; data collection and analysis methods;
dimensions and ethical aspects addressed.
3 Results
This section is divided into two parts. Subsection 3.1 focuses on the description of
selected studies, prioritizing the quantitative analysis of the frequency of publications
per year, geographical distribution of studies and methodological aspects of these pro-
ductions. In Subsect. 3.2, the main ethical aspects identified in the selected articles are
expressed.
3.1 Corpus of Studies

This research focuses on nine articles published in internationally qualified academic
journals. The productions are concentrated in the period from 2020 to 2022, with only
one article from 2015 (Table 1).
The selected productions are based in Europe (six studies), Asia (two studies) and
North America (one study). Table 1 indicates the country of the first author. This locality
extends to most of the other authors of each work, being observed the presence of
interinstitutional collaborations in different regions, but within the country or continent
itself. An exception is the work of [14], in which there is collaboration with a researcher
from Canada.
Regarding the central objective of the articles analyzed, they focus on: i) society’s
perception of the use of artificial intelligence and its connection with the sustainable
development goals (SDGs) [15]; ii) challenges and potentialities of AI research and its
applicability in different areas of knowledge, such as education [20], legal education [19],
data science [21] and engineering [14]; iii) practical applications of AI in the context of
the teaching and learning process [13, 16–18].
Given these objectives, we can characterize them as empirical (55.5%) and theoret-
ical (44.5%) studies that relate AI to the educational context. Empirical research used
documentary sources and questionnaires as data. The theoretical ones used documents
and reports from the teacher herself as a source of investigation. As for the methods
of data analysis, the descriptive content analysis present in all selected studies stands
out, with one of them specifying the use of textual data mining [13]. In two studies
[13, 15], the use of descriptive statistical analysis is detailed, while the article [21] uses
confirmatory factor analysis.
Table 1. Productions selected during the Systematic Literature Review
N Article title Authors Country of origin of the first author Year
1 Curriculum Mapping with Komenda et al. [13] Czech republic 2015

Academic Analytics in Medical
and Healthcare Education
2 Interdisciplinary Research in Kusters et al. [14] France 2020
Artificial Intelligence: Challenges
and Opportunities
3 Public Perception of Artificial Yeh et al. [15] Taiwan 2021
Intelligence and Its Connections
to the Sustainable Development
Goals
4 Ethical Issues in Democratizing Mulvenna et al. [16] UK 2021
Digital Phenotypes and Machine
Learning in the Next Generation
of Digital Health Technologies
5 Educational Warehouse: Modular, Amo; Gómez; Hernández-Ibáñez; Spain 2021
Private and Secure Cloudable Fonseca [17]
Architecture System for
Educational Data Storage,
Analysis and Access
6 Ethical Questions Raised by Köbis; Mehner [18] Germany 2021
AI-Supported Mentoring in
Higher Education
7 Artificial Intelligence Empowers Ma; Hou China 2021
the Integrated Development of [19]
Legal Education: Challenges and
responses
8 Artificial Intelligence Potential in Bucea-Manea-T, oniş et al. [20] Romania 2022
Higher Education Institutions
Enhanced Learning Environment
in Romania and Serbia
9 Teaching Responsible Data Lewis; Stoyanovich [21] USA 2022
Science: Charting New
Pedagogical Territory
Source: Research data (2023).
3.2 Ethical Aspects
In this subsection, we analyze the main ethical aspects present in the selected studies,
using the ethical principles pointed out by Cerratto Pargman & McGrath [22], and
the normative data protection documents. In this sense, below we indicate the seven
fundamental principles highlighted by the authors [22].
• Transparency - relates to the activities of describing the data sources, the purposes
and metrics of the analysis and who has access to them;
• Privacy - is related to access and de-identification of student data;
• Consent - refers to making students aware of the traceability actions, analysis and use
of their data, making them aware and vigilant of the process;
• Responsibility - comprises the responsibility of institutions/researchers for the legal,
ethical and effective use of data;
• Minimization of adverse impacts - is related to damage, non-maleficence and risks
involved in the administration of student data;
• Reliability - refers to the precision of the data and its forms of interpretation and
analysis;
• Facilitating interventions - ethical approach to the institutions’ forms of intervention
based on data analysis.
We identified that there is more than one of these principles in the selected works.
However, [13] was the only article in which none of them was mentioned. In reference
to ethical data protection documents, four articles (44.4%) categorically cite basing their
investigations on the European General Data Protection Regulations, Local Educational
Data Analytics (LEDA), Code of Ethics of the University of La Salle, The Ethics Guide-
lines for Trustworthy Artificial Intelligence and ACM Code of Ethics and Professional
Conduct. These intersections can be seen in Table 2.
Table 2. Ethical principles addressed in selected works and basic legislation
ID Transparency Privacy Consent Responsibility Minimization Reliability Facilitating Base

of adverse interventions Legislation
impacts
1 –
2 X X –
3 X X –
4 X X X X European
General Data
Protection
Regulations
5 X X X LEDA e
Código de
Ética La Salle
6 X X X X X X X The Ethics
Guidelines for
Trustworthy
Artificial
Intelligence
7 X X –
8 X –
9 X X X X ACM Code of
Ethics and
Professional
Conduct
Source: Research data (2023).
Among these principles, transparency and responsibility were the most cited in
the articles, being quantified in six of them (66.7%). While consent, minimization of
adverse impacts and facilitating interventions were mentioned in only one (11.1%). We
emphasize that the production did not address any of the highlighted principles (Fig. 1).
Fig. 1. Quantification of ethical principles, based on [22], highlighted in selected articles. Source:
Research data (2023).
The next section presents and discusses the nine selected works.
4 Discussion
Big Data and Artificial Intelligence are emerging technologies that offer great poten-
tial in different sectors of society, including education, the focus of this study. Their
combination generates advances in personalized and automated decision-making and
management [23].
We were able to visualize this articulation in the main objectives of the selected
articles, whether relating to the challenges, potentialities and applicability of big data
and AI, or through their connection with the sustainable development goals (SDGs) and,
also, in the possibilities and applications practices in the context of the teaching and
learning process, in which many of them relate different areas of knowledge.
Although binomial big data + AI provides a wide range of benefits, it can also pose
risks to users regarding the use of this data. Concerns about the collection, management,
data processing, opacity of algorithms and implementation of results are growing among
the population [24].
For this, it is necessary to implement policies for the use of ethical standards in
investigative and daily practices. It is in this context that guides, manuals, rules and
data protection regulations appear in several countries. In Brazil, we are supported by
the General Law for the Protection of Personal Data (LGPD), Law No. 13,709/2018,
enacted with the aim of protecting the fundamental rights of freedom and privacy, and
the free formation of the personality of each individual, in physical and/or digital [25].
In this work, regarding the ethical aspects, we punctuate the principles followed by
the studies and the official documents on which these productions were based. Initially
analyzing these ethical principles, the ones that stood out the most in the selected pub-
lications were: transparency and responsibility, privacy and reliability. These principles
are the most found in investigations related to the analysis of educational data [26, 27].
Similarly, [28] mapped, in addition to those found in this research, other princi-
ples such as: justice, equity, non-maleficence, beneficence, freedom, autonomy, trust,
auditability, security and protection, among others, which, in a certain way, way, they
are linked to one of the seven principles used as the basis of this research.
Transparency, accountability and privacy go hand in hand in the approach to big
data and AI. The right to privacy is to ensure that data collection, analysis and use
processes are carried out in a transparent and responsible manner, offering cybersecurity
to the population and a broader adoption of big data and AI in the educational context.
Complementarily, it was possible to observe in our results that the studies that are based
on some legislation, cover a greater amount of ethical principles. There is therefore a
need for regulatory public policies for better governance, accessibility and reliability of
my Big data and artificial intelligence [29].
Among some basic documents, there is a guide with ethical guidelines, the Ethics
Guidelines for Trustworthy AI, produced by a group of high-level experts in artificial
intelligence that make up the European Commission. The ethical reflections addressed
in it serve to reaffirm the need to protect individuals and encourage the promotion of
values in line with the 2030 Agenda and the SDGs of the United Nations (UN), namely
SDG 4 - Quality Education, which is based on ensure an inclusive, equitable and quality
education [29, 30].
From the above, it is possible to recognize ethics as a fertile area of concern in
investigations and interventions in the various areas of knowledge, especially in the area
of education and from the uses of educational data, the focus of this investigation [31]
and [32].
5 Final Remarks
In this study, we conducted an analysis of nine selected articles out of 84 gathered from
the CAPES Journal Portal, covering the years 2011 to 2022. These articles revolved
around the topics of big data, artificial intelligence, education, and ethics. The selection
process followed specific inclusion and exclusion criteria, resulting in five empirical
and four theoretical studies for evaluation. Notably, none of the selected works were
authored by Latin American scholars. It is worth mentioning that, apart from a singular
work from 2015, the remaining articles were published during the span of 2020 to 2022.
The works were analyzed based on the presence of the following ethical principles:
transparency, privacy, consent, responsibility, minimization of adverse impacts, reliabil-
ity, facilitating interventions and use of some basic legislation. After the analysis, some
contributions and results deserve to be highlighted here.
First, it should be noted that existing research within this context is quite recent,
which points to a possible growth of this ethical concern in the coming years. There
is also space that must be occupied by Latin American researchers involved with the
development of Big Data and Artificial Intelligence solutions in Education and the ethical
aspects related to the use of these solutions.
Regarding the ethical aspects evaluated, it was identified that the most cited prin-
ciples were transparency and responsibility (cited in six articles - 66.7%), followed by
reliability and privacy (cited in five articles - 55.5%). On the other hand, aspects such as
consent, minimization of adverse impacts and facilitating interventions were mentioned
only once each (11.1%). Such data reaffirm the four most identified principles that are
fundamental to be present in investigations. However, the lack of expressiveness of the
consent principle, for example, is an investigative weakness, since the legal documents
based on research ethics focus on it as one of the fundamentals in this approach. Like-
wise, the lack of forecasting and minimization of risks is also not in line with these basic
documents.
It was identified, moreover, that less than half of the articles (44.4%) are based on
existing ethical norms, and these articles are the ones that, in general, addressed the
greatest number of principles. This is an indication that the use of existing legal bases
should be prioritized or at least considered by future research involving the ethical aspects
of using big data and AI in education.
Future work will add new articles indexed in Scopus, SciELO and Web of Science
databases. Lastly, it is also intended to extend the classification of works by verifying
other ethical aspects and relating them to the different characteristics of the developed
big data and AI products (types of data used, user audience, techniques employed, etc.).
Acknowledgments. This work was funded by the Brazilian National Council for Scientific and
Technological Development - CNPq (process number 409633/2022-4, 305731/2021-1).
References
1. Sagiroglu, S., Duygu S.: Big data: a review. In: 2013 International Conference on Collabora-
tion Technologies and Systems (CTS). IEEE (2013)
2. Chen, L., Chen, P., Lin, Z.: Artificial intelligence in education: a review. IEEE Access 8,
75264–75278 (2020)
3. McCarthy, J.: What is Artificial Intelligence. Stanford University, Stanford (2007)
4. Vázquez, A.S.: Ética. Rio de Janeiro: Civilização Brasileira (2017)
5. Manzini, J.L.: Declaración de Helsinki: Principios éticos para la investigación médica sobre
sujetos humanos. Acta Bioethica 6(2), 321–334 (2000)
6. Bates, A.W.: Teaching in a Digital Age: Guidelines for Designing Teaching and Learning
Vancouver BC. Tony Bates Associates Ltd. (2015)
7. Holmes, W., Bialik, M., Fadel, Ch.: Artificial Intelligence in Education. Promises and
Implications for Teaching & Learning. Center for Curriculum Redesign, Boston (2019)
8. Unesco: CONSENSO DE BEIJING. Conferência Int. sobre Inteligência Artificial e Educação,
Planejamento da Educação na Era da IA: Lead the Leap, Pequim (2019)
9. Unesco: Recommendation on the Ethics of Artificial Intelligence. Organização das Nações
Unidas para a Educação, a Ciência e a Cultura, Paris (2022)
10. Lima, T.C.S.D., Mioto, R.C.T.: Procedimentos metodológicos na construção do conhecimento
científico: a pesquisa bibliográfica. Revista katálysis 10, 37–45 (2007)
11. Nunes, J.B.C.: Busca de literatura científica na área de educação. In: Nunes, J.B.C., Farias,
I.M.S., Nóbrega-Therrien, S.M. (orgs.) Pesquisa científica para iniciantes: retomando o
debate, Fortaleza, pp. 11–23. EdUECE (2021)
12. Almeida, A.C.A.: Analítica da aprendizagem e formação de professores a distância: um olhar
para a ética em pesquisa na área de educação, Fortaleza, pp. 1–170. UECE (2021)
13. Komenda, M., et al.: Curriculum mapping with academic analytics in medical and healthcare
education. PLoS ONE 10(12), e0143748 (2015)
14. Kusters, R., et al.: Interdisciplinary research in artificial intelligence: challenges and
opportunities. Front. Big Data 3, 577974 (2020)
15. Yeh, S.C., Wu, A.W., Yu, H.C., Wu, H.C., Kuo, Y.P., Chen, P.X.: Public perception of artificial
intelligence and its connections to the sustainable development goals. Sustainability 13(16),
9165 (2021)
16. Mulvenna, M.D., et al.: Ethical issues in democratizing digital phenotypes and machine
learning in the next generation of digital health technologies. Philos. Technol. 34(4), 1945–
1960 (2021)
17. Amo, D., Gómez, P., Hernández-Ibáñez, L., Fonseca, D.: Educational warehouse: modular,
private and secure cloudable architecture system for educational data storage, analysis and
access. Appl. Sci. 11(2), 806 (2021)
18. Köbis, L., Mehner, C.: Ethical questions raised by AI-supported mentoring in higher
education. Front. Artif. Intell. 4, 624050 (2021)
19. Ma, B., Hou, Y.: Artificial intelligence empowers the integrated development of legal
education: challenges and responses. Future Hum. Image 16, 43–54 (2021)
20. Bucea-Manea-T, oniş, R., et al.: Artificial intelligence potential in higher education institutions
enhanced learning environment in Romania and Serbia. Sustainability 14(10), 5842 (2022)
21. Lewis, A., Stoyanovich, J.: Teaching responsible data science: charting new pedagogical
territory. Int. J. Artif. Intell. Educ., 1–25 (2022)
22. Cerratto Pargman, T., McGrath, C.: Mapping the ethics of learning analytics in higher edu-
cation: a systematic literature review of empirical research. J. Learn. Anal. 8(2), 123–139
(2021)
23. Ryan, M., Antoniou, J., Brooks, L., Jiya, T., Macnish, K., Stahl, B.: Research and practice of
AI ethics: a case study approach juxtaposing academic discourse with organisational reality.
Sci. Eng. Ethics 27, 1–29 (2021)
24. Christoforaki, M., Beyan, O.: AI ethics — a bird’s eye view. Appl. Sci. 12(9), 4130 (2022)
25. Brasil. Lei nº 13.709, de 14 de agosto de 2018. Lei Geral de Proteção de Dados Pessoais
(LGPD), Brasília, DF (2018)
26. Pardo, A., Siemens, G.: Ethical and privacy principles for learning analytics. Br. J. Edu.
Technol. 45(3), 438–450 (2014)
27. Ferguson, R., Hoel, T., Scheffel, M., Drachsler, H.: Guest editorial: ethics and privacy in
learning analytics. J. Learn. Anal. 3(1), 5–15 (2016)
28. Fjeld, J., Achten, N., Hilligoss, H., Nagy, A., Srikumar, M.: Principled Artificial Intelligence:
Mapping Consensus in Ethical and Rights-Based Approaches to Principles for AI. Berkman
Klein Center for Internet & Society, Rochester (2020)
29. Flores-Vivar, J.M., García-Peñalvo, F.J.: Reflexiones sobre la ética, potencialidades y retos
de la Inteligencia Artificial en el marco de la Educación de Calidad (ODS4). Comunicar 74,
2023–2031 (2023)
30. Comissão Européia: Ethics Guidelines for Trustworthy AI. GPAN IA, Bruxelas (2019)
31. Rodríguez-Triana, M.J., Martínez-Monés, A., Villagrá-Sobrino, S.: Learning analytics in
small-scale teacher-led innovations: ethical and data privacy issues. J. Learn. Anal. 3(1),
43–65 (2016)
32. Menezes, J.B.F., de Almeida, A.C.A., de Lima, M.V.A., Sousa, T.M.: Ética e Learning
Analytics: uma combinação necessária para o rigor científico. Práxis Educativa (18), 2 (2023)
Gamification as a Learning Strategy
in the Training of B1 English Learners
Aura-Liz Cevallos-Almeida1 and Miguel-Angel Quiroz-Martinez2(B)

1 ICT for Education, Universidad Politécnica Salesiana, Guayaquil, Ecuador
acevallos@ups.edu.ec
2 Computer Science Department, Universidad Politécnica Salesiana, Guayaquil, Ecuador
mquiroz@ups.edu.ec
Abstract. Throughout the years, education has been exposed to constant changes
and it is imperative to implement new and modern pedagogical models. Traditional
teaching needs new innovative and technological elements to complement the
learning process. Education has the challenge of transforming the processes by
which we acquire knowledge and, in turn, be available to meet all current and
future needs of society. We have faced a new economic and scientific reality and a
new way of relating to each other as people. Technology has gained ground in all
areas of human beings and their way of interacting. In this way, gamification has
become a learning tool in the educational field as a support to traditional teaching.
Based on this problem, this study aims to examine this and other educational
techniques and compare their influence on the teaching-learning process of B1
level English learners. Its review is based on a quantitative and descriptive study.
As a result, gamification proves to be a versatile tool to create content based on
specific interests and needs that improve performance and motivation to learn a
new language.
Keywords: Gamification · foreign language · pedagogy · Game-based learning ·

Learning strategy · English learners · Educational technology
1 Introduction
Science, technology, and globalization are transformational phenomena that allow a

society to evolve impactfully. According to [1] all these phenomena transcend and break
cultural barriers and social strata, promoting changes in learning and communication.
For this reason, education is challenged to transform the processes by which we acquire
knowledge and, in turn, be available to meet all the current and future needs of society.
Additionally, the different changes our society has been exposed to have made
English the most studied language, to the point of being considered the language of
science. The contemporary demands and requirements of our century place the learning
of the English language as a reliable tool of maximum benefit for their future professional
development. [2].
https://doi.org/10.1007/978-981-99-7353-8_5
50 A.-L. Cevallos-Almeida and M.-A. Quiroz-Martinez
It is increasingly difficult to ignore the transformation in the educational field; during

the last few years, the focus has been concentrated on achieving equity and maximiz-
ing instructional levels; in addition to many other educational components, such as the
acquisition of a foreign language, which have also been reformed and incorporated, man-
aging to be determinant for its development. [3] highlighted the changes that education
has been going through, as well as the pedagogical models of relevance that have been
incorporated, not only innovative but also motivating; likewise, the implementation of
methodologies in which students may be able to experience the transcendence of what
they have learned during their academic development.
The main objective of this study is to examine different educational techniques that
constitute crucial strategies, such as gamification, that respond to the requirements of the
training of English learners at the B1 level. For this purpose, a review of several works
related to the subject was carried out.
Considering that the English language is conceived as an avant-garde language,
innovative learning strategies should be established that allow students to obtain mastery
of it effectively through meaningful and experiential learning. Determining that their
mastery of the language augurs a better preparation and development in their future
professional field and guarantees better adaptability to a globalized environment in which
the English language is catapulted as the language of science [4].
1.1 Preliminary
This section will provide a comprehensive overview of the most significant works and
research conducted in gamification as a learning strategy in education. Gamification
refers to applying game elements and mechanics in non-game contexts to promote
motivation, engagement, and learning.
Based on multiple studies and research, the deficiency of traditional English language
education has been demonstrated, where the teaching and learning process is based, for
the most part, on the teacher. According to a study conducted by [5] many institutions
still use the translation method and the grammar-based class, where students are not
exposed to the language nor its use. Therefore, it is urgent to implement strategies to
achieve optimal learning of the English foreign language and, at the same time, increase
the levels of motivation and exposure to it.
[6] Define that the content of this type of education is based on knowledge and
theories accumulated by science and society, the same that by themselves are insufficient
since they are divorced from the experience and the experience in which students are
immersed. In addition, all knowledge is based on the context and the representation of
the teacher who teaches the subject. On the other hand, the new generation of students
leads transcendentally that traditional teaching does not generate in them the interest to
acquire new knowledge; without leaving aside also the power exerted by their points
of view and their diverse way of learning. For this reason, it is impossible to consider
that only the traditional way of teaching a foreign language understands and satisfies
adequately and completely the academic demand that society requires for the era of
which we are a part.
Gamification as a Learning Strategy in the Training of B1 English Learners 51
How does gamification as a learning strategy assertively impact the learning of B1

English learners? Implementing various virtual platforms framed to the educational field
is vital as a tool to support traditional teaching. He explains it this way [7] stating, “Play is
part of the life of the human being and, especially, of the child”. The new educational era
requires establishing new bases on which traditional teaching can develop meaningfully,
allowing broad access to learning from all its forms and aiming to apply the acquired
knowledge from experience. The influence of new technological tools and their wide
variety of options offer to expand the knowledge and acquisition of a new language,
such as English while playing.
There is no single strategy that achieves the objective. Still, gamification is a contrib-
utor that proposes to increase and improve factors such as motivation, concentration, and
performance that are needed during the learning process of a foreign language. Within
the pedagogical models, the innovative pedagogical model includes different technolog-
ical tools that enhance learning, making this process more experiential and active inside
and outside the classroom [8] point out that the primary objective of the designers of
Scapes Rooms is to achieve a gamified learning environment, which helps to develop
knowledge and social and communicative skills, as well as to create the innate need for
collaboration among students.
New innovative virtual and interactive methodologies are created and perfected annu-
ally, most based on gamification. These educational methodologies must be developed
responsibly and consider adequate pedagogical theories that promote optimal learning
and are not only framed to the game. According to [5], it is essential that these modern
digital competencies involve the critical and safe use of information and communication
technology and thus achieve objectives framed to the correct learning.
Given the complexity involved in learning a foreign language, it is a challenge for
teachers to implement strategies that motivate and capture the attention of students, strate-
gies within which gamification has been growing and has been adopted in the academic
environment by teachers, who attribute to this strategy an admirable and impressive
opportunity to design environments that favor the development of language skills [9].
Factors related to gamification have been investigated from various perspectives. The
following are different studies and research carried out related to the study topic. [10]
mentions factors that favor learning through digital platforms, added to the personality
adopted by the teacher; among the most relevant concludes in the student’s performance,
being more autonomous and influential. [11] the student’s opinions about the program in
which the concepts and practice were framed with gamification increased the desire to
participate. Likewise, there was greater autonomy and ability to address their different
learning styles.
On the other hand, [12] concluded that the gamification of learning a new language
not only points towards obtaining benefits for teachers as they provide academic rein-
forcement resources. For [13], gamification creates dynamic environments; students can
know their progress rate, which gives them the experience of producing the language
they are learning.
The balance between the sense of challenge and challenge within a classroom pro-
vides meaningful experiences based on students’ abilities. As compared [14] between
the knowledge area and the technical, adapting gamification to lesson plans is an easy
process as it is supported as a flexible, dynamic, and engaging tool for students of all
ages.
From the socioemotional point of view, [15] notes that a game has a system of
rules, which linked to the assignment of tasks, guides the players/students through a
process that will help them in master of knowledge and that, in turn, will impact their
cognitive area. Another essential fact he points out is that psychologically human self-
determination is a fundamental key to success. When implementing innovative tools and
methods such as gamification, not only the elements of the game should be considered,
but also the development of interest and motivation.
[16] Conducted a study based on classroom experience in which they determined
that during the sanitary situation of Covid-19, in these last two years, teachers around
the world were forced to use new pedagogical strategies adapted to the virtuality, which
led to the increase of ICT tools that helped to develop linguistic competences in the
English language. In addition, it was evidenced that parents were actively involved in
the training process, helping them with factors such as the pace of study and the correct
pronunciation of English words.
This research, it is intended to highlight the benefits of gamification as part of class-
room resources and demonstrate the promising results that could be obtained based on
the improvement of pedagogical resources and strategies, as indicated by [17]. Further-
more, it promises to be a framework that allows teachers to create new, technological,
and playful student experiences according to their needs, interests, and well-being during
their learning path.
2 Materials and Methods
Diagnostic tests and the survey are considered techniques for this study and analysis.
The study was based on a quantitative approach, with the active participation of the
students and descriptive in scope. The measurement of the variable was the students’
performance, applying an initial and final evaluation.
The study population consisted of 105 B1-level students of the Universidad
Politécnica Salesiana, Guayaquil, Ecuador.
The sample was established in advance and voluntarily, resulting in a total of 83
students aged 19 to 34 years old.
The study was conducted in English, level B1, according to the Common European
Framework of Reference for Languages (CEFR), during the first semester of the year
2022. The assessments were taken synchronously inside the classroom.
As depicted in Fig. 1, a three-phase methodology was used to develop an English
language teaching strategy using gamification. Each of these phases played a crucial
role in the creation process.
Stage 1, called “Academic Diagnosis,” involved a comprehensive analysis of the stu-
dents’ learning needs and objectives concerning English. This stage included conduct-
ing assessments and gathering relevant information to identify the areas where students
needed improvement and how gamification could address those needs.
To begin with, an initial diagnostic assessment consisting of a total of 15 questions
was administered; the design of this type of test was conducted in the traditional paper
Fig. 1. Phases of implementation of a learning strategy.
base assessment. This test made it possible to evaluate each student’s previous knowledge
of English. Performance levels were established according to the results obtained: Low
(0 to 2), Basic (3 to 6), High (6.5 to 8), and Advanced (9 to 10) (Fig. 2).
Fig. 2. Phases of implementation of a learning strategy.
Subsequently, an evaluation was developed within the Quizizz gamified platform,

where the same topics evaluated in the initial diagnosis were included. The same eval-
uation parameters and performance levels were established. This academic diagnostic
phase provided a clear understanding of the students’ level of knowledge of English, lay-
ing the groundwork for designing and adapting the gamified teaching strategy according
to their individual needs (Fig. 3).
Fig. 3. Quizizz Test developed and appearance of the student’s smartphone during the period
given to select the correct answers. Image obtained from the Quizizz platform.
In the second stage, “Design and Development of Gamified Activities”, gamified

activities and resources were created based on the findings obtained in the previous
phase. Game elements, such as challenges, rewards, and competitions, were incorporated
to encourage students’ active participation and maintain their motivation throughout the
learning process.
In the measurement stage, a comparative analysis was performed to evaluate the
impact of gamification concerning the final performance score. A validation was carried
out that contrasted the results obtained through the gamified tool with those of the initial
diagnostic evaluation. This made it possible to establish the differences in the results
obtained and verify the gamified tool’s effect.
A survey was applied using Google classroom forms to obtain an additional per-
spective. This survey allowed it to collect the students’ opinions about their experience
and determine how they felt they had achieved more significant learning. In addition,
it was explored whether the gamified tool impacted their final academic performance.
The surveys were designed to obtain information from each student’s experience and
point of view; each question was posed and written in simple language and explained
the technical language.
In the third phase, “Implementation and Evaluation of the Gamified Strategy”, the
gamified strategy was implemented, and its impact on students’ English language learn-
ing was evaluated. Students participated in the activities specifically designed for this
approach, and relevant data were collected to evaluate the results obtained.
Comparative analyses were conducted to measure and compare the results obtained
with traditional teaching methods. These analyses identified the strengths and weak-
nesses of the gamified strategy compared to conventional approaches and provided a
solid basis for evaluating its effectiveness.
In the proposals and recommendations stage, data collected from both evaluations
were used to identify possible limitations and challenges in implementing gamification in
the classroom. Based on these data, proposals, and recommendations were developed to
highlight the benefits of gamification as part of educational resources and to demonstrate
the promising results that could be achieved by enhancing resources and pedagogical
strategies through a practical gamified learning approach.
In summary, the third phase involved implementing and evaluating the gamified
strategy, followed by presenting proposals and recommendations based on the data col-
lected. This highlighted the benefits of gamification in the classroom and showed the
potential for positive outcomes through improved resources and pedagogical strategies
through a practical gamified approach.
In summary, this three-phase methodology enabled the creation of an effective
English language teaching strategy using gamification, providing students with a more
engaging, participative, and motivating learning experience.
3 Results
According to the data obtained from both the initial diagnostic test and the scores result-
ing from the test developed on the Quizizz gamified platform, it is evident that students
performed better on the gamified test. Both tests included the same number of questions
and comprised the same grammar, vocabulary, and language use topics (Table 1).
Table 1. Comparison between the two types of tests.
Level Total of students Number of questions Test type Average over 10

points
B1 105 25 Traditional 6,37
Diagnostic test
B1 105 25 Quizizz Gamma 8,49
Test
a Analysis of test results.
Subsequently, we proceeded to conduct a survey of 38 questions, applied to a popu-

lation of approximately 105 students divided into the 3 B1 level courses, which allowed
us to identify the positive impact of the implementation of classroom resources, as well
as interactive platforms and the high degree of acceptance of gamified tools as a teaching
method, on the other hand, the survey showed that students conceive the use of these
strategies as partially self-sufficient during the teaching-learning process (Fig. 4).
The results indicate that the most significant limitation when implementing an inno-
vative tool does not have a device; on the other hand, they point out that there is a
pedagogical limitation on the part of the teacher and his or her way of teaching. Among
the students’ observations, 29.2% of them pointed out that there are several reasons why
the use of methodological strategies is pedagogically limiting since some teachers are
Fig. 4. Results on the limitations encountered when implementing innovative tools.
not trained; on the other hand, they indicated that individualism, the economic factor, the
attitude of teachers and policies of the educational system are reasons why such inno-
vative strategies are limited. Finally, they pointed out that innovation requires a gradual
plan, a process linked to voluntary and creative change (Fig. 5).
On the other hand, 81.5% of students consider that through virtual platforms, they
obtain autonomy and greater freedom to organize their time and study pace and adapt
their class hours to where they are (Fig. 6).
The fields in which students consider that gamification has a high academic impact
are when learning new subjects to reinforce and practice previously acquired knowl-
edge. It should be noted that 18% believe they have achieved better results through this
implementation (Fig. 7).
Fig. 5. Learning autonomy outcome.
Fig. 6. Result on fields with learning achievement
Fig. 7. Results on level of motivation
Students widely accept implementing digital tools and gamified platforms within
knowledge development, and motivation rates are high. Thus, they constitute this
methodology as a resource and an innovative option in the learning process.
4 Conclusion
Teaching English as a foreign language has traditionally been a challenge for educa-
tors. However, using technologies, virtual platforms, and gamification has provided new
opportunities to update and revitalize the educational environment. As a learning strat-
egy, gamification has proven to be a versatile and complementary tool to traditional
teaching.
Through the analysis and review of high-impact articles, it can be evidenced that
the methodological strategies of gamification applied to teaching the English language
provides an enriching virtual environment and offers countless advantages during knowl-
edge acquisition and reinforcement. Students experience tangible results and stay moti-
vated in their academic development while influencing their behavior and attitude toward
learning.
In addition, the implementation of gamification promotes more meaningful and up-
to-date learning, bringing students closer to content and methodologies adapted to their
interests and needs. These new methodologies improve performance and increase interest
in learning the English language, which facilitates developing and strengthening all the
language skills necessary to master it.
According to the survey results, the most innovative educational techniques that
respond effectively to the needs of English language learners are those that imple-
ment gamification-based models. Platforms such as Kahoot, Genially, Plickers, Edmodo,
Socrative, and Quizziz are some of the educational platforms that currently converge on
a single objective to guide students towards autonomy and the organization of their learn-
ing, in addition to providing comfort and support during the process. These platforms
that include educational content are among many other platforms, the most feasible, and
the ones pointed out by students as valuable tools with advantages that they have been
able to experience when learning English.
In short, gamification has become a powerful tool for creating a student-centered
learning environment that encourages active participation, motivation, and integral devel-
opment. Through gamification, a solid foundation is established for students to acquire
the necessary skills to master the English language effectively and successfully.
Acknowledgments. This work has been supported by the GIIAR research group and the
Universidad Politécnica Salesiana.
References
1. Fabricio Molina-García, P.I., Rosanna Molina-García, A.I., Gentry-Jones III, J.: La gami-
ficación como estrategia didáctica para el aprendizaje del idioma inglés. Dominio de Las
Ciencias 7, 722–730 (2021). https://doi.org/10.23857/dc.v7i1.1672
2. Azzouz Boudadi, N., Gutiérrez-Colón, M.: Effect of gamification on students’ motivation and
learning achievement in Second Language Acquisition within higher education: a literature
review 2011–2019. EuroCALL Rev. 28, 40 (2020). https://doi.org/10.4995/eurocall.2020.
12974
3. Kim, J., Castelli, D.M.: Effects of gamification on behavioral change in education: a meta-
analysis. Int. J. Environ. Res. Public Health 18, 3550 (2021). https://doi.org/10.3390/ijerph
18073550
4. Saleem, A.N., Noori, N.M., Ozdamli, F.: Gamification applications in e-learning: a literature
review. Technol. Knowl. Learn. 27, 139–159 (2022). https://doi.org/10.1007/s10758-020-094
87-x
5. Safapour, E., Kermanshachi, S., Taneja, P.: A review of nontraditional teaching methods:
flipped classroom, gamification, case study, self-learning, and social media. Educ. Sci. 9, 273
(2019). https://doi.org/10.3390/educsci9040273
6. Burlacu, M., Coman, C., Bularca, M.C.: Blogged into the system: a systematic review of
the gamification in e-learning before and during the COVID-19 pandemic. Sustainability 15,
6476 (2023). https://doi.org/10.3390/su15086476
7. Redondo-Rodríguez, C., Becerra-Mejías, J.A., Gil-Fernández, G., Rodríguez-Velasco, F.J.:
Influence of gamification and cooperative work in Peer, mixed and interdisciplinary teams on
emotional intelligence, learning strategies and life goals that motivate university students to
study. Int. J. Environ. Res. Public Health 20, 547 (2022). https://doi.org/10.3390/ijerph200
10547
8. Makri, A., Vlachopoulos, D., Martina, R.A.: Digital escape rooms as innovative pedagogical
tools in education: a systematic literature review. Sustainability 13, 4587 (2021). https://doi.
org/10.3390/su13084587
9. Barragán-Pulido, S., Barragán-Pulido, M.L., Alonso-Hernández, J.B., Castro-Sánchez, J.J.,
Rabazo-Méndez, M.J.: Development of students’ skills through gamification and serious
games: an exploratory study. Appl. Sci. 13, 5495 (2023). https://doi.org/10.3390/app13095495
10. Valantinaitė, I., Sederevičiūtė-Pačiauskienė, Ž: The change in students’ attitude towards
favourable and unfavourable factors of online learning environments. Sustainability 12, 7960
(2020). https://doi.org/10.3390/su12197960
11. Huang, B., Hew, K.F., Lo, C.K.: Investigating the effects of gamification-enhanced flipped
learning on undergraduate students’ behavioral and cognitive engagement. Interact. Learn.
Environ. 27, 1106–1126 (2019). https://doi.org/10.1080/10494820.2018.1495653
12. Jodoi, K., Takenaka, N., Uchida, S., Nakagawa, S., Inoue, N.: Developing an active-learning
app to improve critical thinking: item selection and gamification effects. Heliyon 7, e08256
(2021). https://doi.org/10.1016/j.heliyon.2021.e08256
13. Ortiz-Colón, A.-M., Jordán, J., Agredal, M.: Gamificación en educación: una panorámica
sobre el estado de la cuestión. Educação e Pesquisa 44 (2018). https://doi.org/10.1590/s1678-
4634201844173773
14. Krath, J., Schürmann, L., von Korflesch, H.F.O.: Revealing the theoretical basis of gamifica-
tion: a systematic review and analysis of theory in research on gamification, serious games
and game-based learning. Comput. Human. Behav. 125, 106963 (2021). https://doi.org/10.
1016/j.chb.2021.106963
15. Padilla Martín, R.G., Caldera Montes, J.F.: La gamificación como estrategia para la mejora de
la gramática del idioma inglés. Educateconciencia, 62–77 (2021). https://doi.org/10.58299/
edu.v29iEsp.402
16. Vergara-Pareja, C.M., Nielsen-Niño, J.B., Niño-Vega, J.A.: La gamificación y el fortalec-
imiento de la habilidad oral en inglés a niños de primera infancia. Revista de Investi-
gación, Desarrollo e Innovación 11, 569–578 (2021). https://doi.org/10.19053/20278306.v11.
n3.2021.13355
17. Cortés-Pérez, I., Zagalaz-Anula, N., del Carmen López-Ruiz, M., Díaz-Fernández, Á.,
Obrero-Gaitán, E., Osuna-Pérez, M.C.: Study based on gamification of tests through
Kahoot!TM and reward game cards as an innovative tool in physiotherapy students: a
preliminary study. Healthcare 11, 578 (2023). https://doi.org/10.3390/healthcare11040578
Implementing Synchronous Hybrid
Learning: Insights from Teachers
and Students’ Experiences
Katherine Chiluiza1 , Vanessa Echeverria1,2 , Adriano Pinargote1(B) ,

Gladys Carrillo1 , and Margarita Ortiz-Rojas1
1
Escuela Superior Politécnica del Litoral, ESPOL, Centro de Tecnologı́as de
Información, Campus Gustavo Galindo Km. 30.5 Vı́a Perimetral, 09-01-5863,
Guayaquil, Ecuador
adriano.pinargote@cti.espol.edu.ec
2
Monash University, Clayton, VIC, Australia
Abstract. The global shift to hybrid learning due to the COVID-19

pandemic has resulted in a research gap regarding teacher and student
experiences, especially in Latin America. To address this, this mixed-
methods study focuses on two areas: 1) the impact of technology on
teacher-student interactions, and 2) perceptions of hybrid classes. Draw-
ing data from seven Ecuadorian educators and 141 students in hybrid engi-
neering courses, quantitative analysis includes classroom location, inter-
actions, and student satisfaction, while qualitative insights come from
interviews and open-ended student responses. Students reacted positively
to hybrid learning, with higher satisfaction than pre-pandemic methods,
highlighting teacher dedication and flexibility. Students pointed out audio
and video challenges that require attention, and teachers noted technologi-
cal issues that hampered course flow, planning, and limited student inter-
action. Furthermore, teachers believed that such problems could hinder
the adoption of more engaging and active learning strategies. The study
concludes by proposing guidelines for implementing hybrid courses.
Keywords: Synchronous Hybrid Learning · Higher Education ·

Engineering-oriented education · Analytics
1 Introduction
The COVID-19 pandemic has catalyzed a significant transformation in the edu-
cational landscape, compelling universities globally to transition from traditional
to hybrid learning models. This shift, while necessary, has presented a unique
set of challenges and opportunities, particularly in regions like Latin America
where resources are limited and research on the subject remains scarce [7]. This
paper aims to contribute to the growing body of knowledge on hybrid learning by
examining the experiences of teachers and students in Ecuadorian universities,
with a focus on engineering courses. Hybrid learning, specifically Synchronous
c The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2023
https://doi.org/10.1007/978-981-99-7353-8_6
Implementing Synchronous Hybrid Learning 61
Hybrid Learning (SHL), is a learner-centered approach that combines in-person

and online learning in real-time, allowing students to participate in a shared learn-
ing space either in person or virtually [1]. This model offers several advantages,
including increased student enrollment due to expanded classroom capacity, stu-
dent flexibility, and the freedom for teachers to design their learning environment.
However, it also presents challenges such as the need for high pedagogical and
technological competence among teachers, potential mental exhaustion, cognitive
overload, and the risk of neglecting virtual students [2,3].
Previous studies have explored the benefits and challenges of SHL in dif-
ferent countries, including the United States [4], England [5], and Singapore
[6]. These studies have highlighted students’ preference for hybrid classrooms
over traditional ones, the importance of pedagogical and technological training
for teaching staff, and the benefits of incorporating interactive activities and
collaborative learning with the support of technical tools and equipment. How-
ever, there is a notable gap in the literature concerning Latin America [7], and,
more specifically, how the technological setup might affect the learning experi-
ence from both teachers’ and students’ perspectives. This mixed-methods study,
therefore, seeks to investigate the impact of the technological setup on teacher-
student interactions and the perceptions of teachers and students towards hybrid
classes in Ecuador. By doing so, we provide valuable insights that can guide the
implementation of hybrid courses in the region and beyond, particularly in cir-
cumstances that require remote learning.
2 Foundations on Synchronous Hybrid Learning

and Related Works
Synchronous hybrid learning (SHL) is a learning approach where students partic-
ipate in a shared learning space, with some attending in person and others vir-
tually [1]. This concept falls under the broader term of Hybrid learning, which
is a learner-centered modality that integrates digital and traditional in-person
class activities [2]. Implementing SHL offers some advantages, along with certain
challenges and design guidelines, as summarized by [1,3]. In terms of advantages,
from an institutional perspective, SHL could increase student enrollment due to
increased classroom capacity. From a student’s standpoint, the main advantage
is flexibility. Students can choose whether to attend virtual or in-person classes
by considering personal or external circumstances (e.g., students can attend vir-
tual classes due to health-related issues or to avoid commuting to the univer-
sity). Conversely, teachers benefit from the freedom to design their learning envi-
ronment, including the possibility of inviting external experts remotely. How-
ever, SHL demands high pedagogical and technological competence from teachers,
requiring active methodologies and management of various digital tools to engage
all students. This can cause mental exhaustion and cognitive overload for teachers,
potentially neglecting virtual students, and causing an attention imbalance.
Several studies from different countries have examined the benefits and chal-
lenges of Synchronous Hybrid Learning (SHL). For example, [4] compared tra-
ditional and hybrid classrooms in a US higher education institution. Analyzing
62 K. Chiluiza et al.
pre and post-test surveys from 20 students, the study found that students per-
ceived hybrid classrooms as more productive than traditional ones. A study from
England [5] reported challenges implementing synchronous virtual and in-person
seminars in hybrid flexible (HyFlex) teaching during Covid-19. From the analy-
sis of 104 hybrid learning sessions, the authors highlighted: a) the importance of
students choosing their study modality, b) the need for pedagogical and techno-
logical training for teaching staff, c) the high cognitive load due to pre-structured
classes, and d) the significance of the learning environment and equipment design.
[6] presented a case study on re-imagining an interactive programming course for
the synchronous hybrid classroom during the pandemic. Using 147 student sur-
vey responses from a Singapore institution, the study emphasized the benefits of
interactive activities and collaborative learning with technical tools.
In Latin America, [7] noted the scarcity of studies on this topic (30 studies
indexed between 2017–2022). One case from Mexico, where [8] analyzed 35,840
student perceptions over four semesters, showed strong acceptance of the hybrid
methodology. There was no negative impact on grades, and students’ perception
of the new methodologies and teacher evaluations improved. The studies above
showed two main gaps in the literature. First, there are not enough studies in
Latin America about the implementation of SHL [7]. Second, although the stud-
ies shed insights related to students’ satisfaction [4,8], pedagogical design [5],
technological equipment needed and logistical aspects [6], it has not been specif-
ically addressed how the technological set-up might affect the learning experi-
ence from teachers’ and students’ perspectives. Thus, this study will focus on
answering the following questions:
RQ1: To what extent does the technological setup for hybrid classes affect the
interactions between teachers and students?
RQ2: What are the challenges and advantages of hybrid classes perceived by
teachers in an Ecuadorian engineering-oriented university?
RQ3: What are the challenges and advantages of receiving hybrid classes per-
ceived by students in an Ecuadorian engineering-oriented university?
3 Methodology
3.1 Context
This study was conducted at an Ecuadorian engineering-oriented university dur-
ing the first academic semester of the year 2022 (May-September). The semester
is composed of 14 weeks of classes, divided into two blocks of seven weeks each.
There was a one-week interval between these blocks for mid-term examinations
and an additional week at the end of the semester for final examinations. The
university trained 201 instructors to teach hybrid courses, as suggested by [3],
to provide an alternative learning experience to students unable to attend on-
campus classes. This training consisted of 20 h: two hours to learn how to use
the equipment and technology installed in classrooms, eight hours about flipped
learning strategies, and a 10-h group preparation to teach a 15-min synchronous
hybrid learning session. Half of the teachers participated virtually, while the
others attended in person, providing them with a learning experience similar to
that of their students.
3.2 Participants
A total of 18 trained teachers (21 hybrid courses, Mean class size = 27, Min = 12,
Max = 42, N = 552 students) who were involved in teaching a hybrid course were
invited to participate in the study. However, only 7 teachers accepted the invi-
tation (6 male and 1 female). These courses were taught in a variety of under-
graduate programs from different fields and involved 188 students. The courses
were: Structural Analysis (30), Advanced Control Systems (29), Wireless Com-
munications (39), Prestressed Concrete (22), Time Series (21), Electrical Instal-
lations (31), and Design of Electrical Distributed Systems (16). All of them were
engineering-oriented courses from undergraduate programs except for the last
course, which is taught at the postgraduate level.
3.3 Hybrid Classroom Setup

To support the hybrid teaching, 67 classrooms were accommodated with the
following technological setup (see Fig. 1). A Video conference system 1 which
includes an audio and video unit connected to the computer via USB. This unit
includes acoustic echo suppression; cancellation of background noise; 6 micro-
phones and two speakers with 90 dB output; a camera with UHD 4K resolution,
5× zoom, 120◦ field of vision, and interlocutor tracking; connected to a Smart
UHD TV. Smart UHD TV to project virtual students. Tablet with a digital
pen is used as an extension of the whiteboard, and it is connected to the PC.
The high-precision electromagnetic pen enables precise handwriting (size 13.3”,
USB-C port, plug and play). HDMI Splitter which is a connection concen-
trator between the computer and the equipment. In addition, already existing
equipment (e.g. a PC, a screen, and a projector) in the classroom enabled the
teacher to project resources and integrate the new technological infrastructure
for hybrid classes.
3.4 Data Collection and Instruments

To address the research questions, the study used several ways of gathering data
at different time points.
– Teacher positions and student interactions: To examine how the techno-

logical setup impacts teacher-student interactions (RQ1), we tracked teachers’
positions and interactions with students throughout the semester. Teachers
wore a location tag, allowing the ultra-broadband indoor positioning system,
Pozyx [11], to track their (x,y) coordinates.
1
Polycom system.
Fig. 1. Classroom Setup for Hybrid Courses. (1) Teacher’s Desk area for PC and tablet;
(2) A whiteboard (left) and display area (right) to project slides on the screen; (3)
Polycom Video conference system including a Smart TV.
An independent observer used LookWhosTalking [13] to log the type of

teacher-student interaction manually. Before each class session, we defined
actions such as in-person and virtual interactions and the initiator of the
interaction (teacher or student). The observer recorded the timestamp and
type of interactions.
– Teachers’ expectations about hybrid courses: To explore the challenges
and advantages of hybrid classes (RQ2), interviews were conducted to deter-
mine the teachers’ expectations about the course and what actually happened.
Interviews were scheduled two weeks after the midterm, and the following
questions were asked:
• What were your expectations of the interactions with your students in
the hybrid course?
• How did you prepare your classes for the hybrid modality of this course?
• What challenges and advantages have you experienced when teaching a
hybrid course?
– Teachers’ reactions to an analytics report: Near the semester’s end,
we collected teachers’ responses to a report detailing their positions and
interactions with students during classes (RQ1). This report, generated using
Tableau [12], aimed to help teachers reflect on their practice using class data.
Teachers’ positions were represented in a heat map, and their interactions
with students were counted and displayed in a timeline (see Fig. 2). This
allowed teachers to see their interaction count with in-person and virtual
students, including timestamps. Teachers had access to all class reports for
the semester. Of the seven teachers in the study, only three (including one
female) agreed to data collection (position and student interactions), so only
these three answered report-related questions. After presenting the report,
we introduced the second set of questions as follows:
• What do you think about your location in the classroom? Are these what
you expected to see?
• What about your interactions with your students? Were they what you
expected them to be?
Fig. 2. Teacher’s report per class: (a) Teacher’s location; (b) distribution of teachers’
position; (c) In-person and virtual students attendance; (d) Number of teacher-student
interactions; (e) Speech time distribution, and (f) the Interactions’ Timestamp.
• As to the sections in the report, is there any that you consider relevant
for your teaching practice?
– Student feedback on hybrid courses: At the semester’s end, students
completed two questionnaires. The first, an institutional questionnaire, eval-
uated the pedagogical and organizational aspects of each university course.
It assessed course planning (3 questions), teaching-learning process (7 ques-
tions), learning assessment (2 questions), technology use (1 question), and
student satisfaction (1 question) on a Likert scale (1 for ’totally disagree’ to 5
for ’totally agree’). Results were presented as scores out of 100 per evaluated
aspect. The second questionnaire collected student perceptions of the hybrid
course they attended, asking: (1) What challenges did you face during this
course? and (2) How can we improve this hybrid course? We received 141
responses to the first questionnaire and 38 to the second. No postgraduate
students responded.
3.5 Analysis
We used a mixed-method approach to address our research questions. To par-
tially answer RQ1, we analyzed teachers’ positions to understand their coverage
of specific classroom areas, such as the desk and front display [9]. Descriptive
statistics were calculated for teacher-student interactions and classroom area
coverage.
The same analysis was applied to students’ questionnaires about the pedagog-
ical and organizational aspects of the hybrid courses, addressing RQ3. We used
a qualitative approach to complement RQ1, answer RQ2, and complete RQ3.
Teachers’ interviews were transcribed and segmented into quotes. We analyzed
teachers’ quotes and students’ open-ended responses using an inductive approach
and affinity diagramming to identify emerging themes [10]. Two researchers cat-
egorized the quotes, discussing themes and identifying recurrent patterns.
4 Results
In this section, we present the results regarding 1) the technological setup in
hybrid classes and how it affects the interaction between teachers and students
(RQ1), 2) the perceived challenges and advantages of teaching hybrid classes
(RQ2), and 3) the perceived challenges and advantages of receiving hybrid classes
(RQ3).
4.1 How the Technological Setup in Hybrid Classes Affects

the Teacher-Student Interactions
Teachers’ positions and classroom observations (teacher-student interactions)
throughout the semester were used to answer partially RQ1. An initial visual
analysis of the heatmaps that were generated using the teacher’s position helped
us understand how teachers used the space during hybrid classes and if there were
any constraints when interacting with students. For example, Fig. 3 illustrates
the position of one teacher in three specific moments: a) when given the lecture,
b) when interacting with in-person students, and c) when interacting with virtual
students. Remarkable concentrations of teachers’ positions in certain classroom
spots allowed us to identify two areas of interest (AoIs). These areas were: (1) the
teacher’s desk area and (2) the front display and screen area. From this visual
analysis, we found that when a teacher was given the lecture, her position was
mainly between (1) and (2) (see Fig. 3(a)). This pattern is also observed when the
teachers interact with in-person students (see Fig. 3(b)). However, when there
were interactions between teachers and virtual students (see Fig. 3(c)), it stands
out that teachers tried to locate themselves centrally in the room.
Table 1 shows the results of classifying teacher-student interactions and the
time spent in the AoIs. There is a slightly higher participation of in-person
students than those attending virtually. It is also noticeable that teachers 1 and
2 spent, on average, more time at the front display area than at the desk area,
and in the case of teacher 3, the time proportions between both areas differed
by approximately 5 points and again spent more time in the front display area.
These results suggest that the technological setup forced these three teachers to
stay at the front when they were teaching or when they were interacting with
in-person students.
Moreover, they were located at the center front when they interacted with
their virtual students. The interactions also showed that there were more inter-
actions from the in-person students than from those virtual ones. Besides, this
Table 1. Descriptive class statistics of the time spent on each AoIa and teacher-student
interactions
Teacher Teacher-Student interactions (#) Time spent (%)

(# of Sessions) In-person Virtual Desk Front display
Mean St. Dev Mean St. Dev Mean St. Dev Mean St. Dev.
1 (9) 8,00 6,19 3,00 2,20 21,08 8,21 70,80 8,77
2 (8) 4,00 2,70 3,00 2,07 15,52 9,00 81,42 10,00
3 (7) 4,00 3,92 2,00 0,82 43,35 35,00 49,70 34,00
a AoI:Area of Interest.
Fig. 3. Teachers’ heat map positions for (a) Lecturing, (b) In-person, and (c) Virtual
teacher-student interactions
group of teachers did not move beyond the front area of the classroom and were
centered on lecturing most of the time.
4.2 Teachers’ Perceived Challenges and Advantages
This section presents the responses from teachers’ interviews, serving as a com-
plement to answer research questions RQ1 and RQ2. Additionally, the section
includes reactions to reports on teachers’ positions and interactions. Tracked
teachers in the classrooms are denoted as T1-T3, while other teachers are refer-
enced as T4-T7. Several themes emerged from the teachers’ reflections:
– The experience for both virtual and in-person students should be

the same, but it was not. When teachers reflected on their expectations
of the course, they indicated that they sought: a) equitable and quality expe-
rience among both groups of students; and b) a balance in the interactions
of in-person and virtual students. In the words of T1: “I expected that a stu-
dent says: Well, going to the classroom or being in the virtual one (is the
same), I’m receiving the same quality”. However, as stated by T4, teach-
ers’ expectations differed from what actually happened: “I expected a higher
number of students’ interactions. (However) the virtual ones, in many cases,
did not answer ”. T1 coincided, indicating: “I asked questions in the sessions;
but, there was always more participation from the in-person students than
from the virtual ones”. This pattern was observed by 6 teachers. Some teach-
ers “blamed” technology for this lesser participation. For example, T6 said:
“More in-person students participated than virtual ones, maybe the virtuals
wanted to talk, but the technology didn’t help them”. Students’ internet qual-
ity was not good as it was supposed to be, as mentioned by T2: “I had a
conversation with my students and they asked me for understanding in case
the internet failed them. I also requested them the same in case an equipment
failed ”.
– More preparation to teach and readjustment of the content on the
run. Three teachers took additional courses to manage the technological tools
they used in the classroom. Others decided to include different tools. For
instance, T1 was not quite comfortable with the technological solution in the
classroom (the video quality was not good enough); thus, he solved the issue
by incorporating other software: “I used OBS, I always thought the image
projected on the board was my biggest challenge. My students preferred to see
the explanations on the physical board, and I surveyed them to find that 75%
agreed on that”. Regarding content adjustments, even though all of them
received the hybrid-course training and redesigned their courses, some teach-
ers struggled with this new method of teaching and planning. T2 expressed:
“It was hard to restructure the topics and make them more interactive for the
students”. Moreover, two teachers needed to modify their planning when the
courses were already in progress to cover all the content. According to T6: “I
already planned my classes, but it was not enough. Initially, I did not manage
my time well. With gained experience, I was able to do it and complete the
topics”.
– An assistant co-located in the classroom is needed. According to
these teachers, the “time management struggle” they experienced during
the semester was one consequence of lacking a technical assistant that could
solve quickly some issues related to equipment or the internet. As T2 men-
tioned: “I was always afraid about technological failures, during the session.
For instance, if I wanted to highlight something with the tablet, I couldn’t
because it didn’t work. I needed to use another app to do the same, but until
I figured it out, my class time was over ” and T6 indicated that the technical
problems could have been addressed with additional support: “If I have had
an assistant when the equipment failed, things would have been different”;
T5 agreed: “When a problem appeared, I would have wanted to have techni-
cal support by my side; but there was only one, that was supporting other 5
courses simultaneously. So, this person appeared one hour later ”. Some teach-
ers designated students to be their assistants with their virtual partners since
their attention was divided into teaching, checking virtual comments, raising
hands, and managing the equipment (projector configuration, camera, slide
presentations, etc.) As T1 said: “I am really tired at the end of the session
because I am like in a movie creation. I am the director, the cameraman, and
the actor. An assistant would be very helpful ”. The need for an assistant was
required by five teachers.
– Technical and communication issues challenge the hybrid format.
Most teachers (six) reported technical problems related to equipment mal-
function, internet (dis)connection, and misconfiguration of the hybrid setup
due to class sharing, among others. T4 mentioned: “One of my main problems
is not being well heard by the virtuals due to the mic misfunction, empty bat-
teries for controls or loss of internet connection”. In this sense, three teachers
indicated that technical problems originated communication challenges with
the virtual students, e.g. if a teacher wanted their student to be focused on
a specific explanation, he had to be sure that not only his in-person students
were attentive, but his virtual students were too, which was hard since all his
students always turned-off the cameras. Therefore, he had to keep checking
for the attention of his students. This was the case of T7, who said: “You
had to be aware that the connection has not been lost, and check if the virtual
ones were listening to you”. The issue of internet loss hindered the possibility
of getting mixed teams (in-person and virtual students). Some teachers tried
these mixed groups but as stated by T3: “Fluid communication was somewhat
complicated due to delays or choppy audio. So, teamwork was really compli-
cated ”. The abstention from group work in the classroom was also remarked
due to the time it demands, as T1 said: “I would have liked to have done
group activities, but I didn’t have enough time”. This testimony also sustains
the need to readjust the teaching planning.
– Flexibility for students is the advantage in hybrid courses. In these
classes, the student chooses when to attend in-person or virtually; moreover,
T3 indicated: “Most of my students work, so, they decided to attend virtually”.
Additionally, students opted for hybrid courses due to the ease of reproducing
recorded class sessions as needed. T1 emphasized: “They are able to play a
recorded session, then they can consult a specific class and can strengthen
their knowledge.”
The following themes were derived from the reactions of the teachers who
received reports of positions and interactions with students:
– The technological setup ’confined’ teachers at the frontal area of
the classroom. The three teachers who were tracked in the classroom agreed
regarding the report’s findings on their positions. They confirmed that they
had to maintain a position facing the classroom in order to be observed and
heard by both groups of students. T3 remarked: “I needed to be available for
the students who were virtual and that’s why I’m practically at the front all the
time. I notice that I’m in front of the video conference system (See Fig. 3(c))
because (if I don’t) they couldn’t listen to me”. T2 also shared: “That’s what
I expected to see (on the heatmap) because during class I walk in front of the
board. I point out what is projected and I even roll the screen up and use the
marker and the board. So, my location is mainly at the front with the screen
and on the desk area, I have to pass the slides”. She also indicated: “I main-
tained a student-teacher distance due to the sanitary emergency. However, if
the emergency had not occurred, I believe I would have interacted differently”.
T1 claimed that this is his natural position to teach: “Definitely these are the
sectors in which I move”. Again, it is important to underline that all the
teachers who were observed centered their teaching on almost lecturing.
– More teacher-student interactions are required. When reviewing the
report, teachers’ attention got captured by sections (d), (e), and (f) (Fig. 2).
They reflected on the data shown and mentioned that they should have capi-
talized on students’ willingness to participate in improving the class dynamics.
As T3 explained: “When I see the timeline, I realize that there is a long time
in which I am being ’selfish’ explaining the class and they are not participat-
ing”; moreover, T2 signaled: “If I had gotten closer to the students when they
were doing evaluations or workshops, I think that they would have dared to
ask more when reviewing in each activity”.
Fig. 4. Comparison between pre-pandemic scores and hybrid scores for six hybrid
courses.
4.3 Students’ Perceptions About the Hybrid Courses
Students’ responses (141) to the questionnaire yielded a positive view of the

hybrid courses. This is shown in Fig. 4 which indicates not only the scores
obtained for the six hybrid courses per evaluated aspect but the scores of these
same courses in the previous semesters (in-person) during the COVID-19 pan-
demic. All the aspects evaluated got higher scores than in the previous semesters.
Therefore, a favorable reaction was received from the students of the courses
under study. Additionally, the responses to the open questions suggested the
following two themes (S# represents students’ comments):
– Bad quality of the video and other technology-related issues. Even

though the students answered optimistically to the hybrid courses, they did
find some issues related to the quality of the video and its sharpness. Specifi-
cally, they occasionally could not see either the details of what was drawn on
the board or they received pixelated videos. Moreover, students mentioned
that seldom the camera projection did not relate to what the teachers pre-
sented on the projection area. S1 said: “When I was virtual, the quality (of
the course) was affected. The camera did not capture what it was supposed to
get. So the teacher had to do it manually”. The dissonance between what was
projected and what was supposed to be projected was reported by 37% of the
students. Other students noticed this type of problem when they were review-
ing the recorded videos, as mentioned by S2: “When I was playing a recorded
class, the camera did not follow the teacher ”. Besides, other students added
that there were some audio issues too; e.g. S3 underlined: “The camera was
unfocused and sometimes I couldn’t hear the interaction between the teacher
and the in-person students”. A student who attended mostly in-person real-
ized this problem when he had to do it virtually (S4): “For the students who
were virtual, it was complicated because the screen was not always shared to
see what it was signaled by the teacher or we couldn’t hear her well”. Only 8%
of the students mentioned audio issues. Finally, students recognized that cer-
tain challenges encountered during virtual attendance stemmed from the low
quality of their home internet connections. In this sense, S10 mentioned: My
internet signal could be unstable” and other students said (S23, S20): “There
is a bad internet connection at home”. However, the students also recognized
that the hybrid courses and the technology used provided flexibility that is
not offered otherwise. As S32 said: “So far I liked the course. There were
few times when I couldn’t attend the class and I could review the video of the
recorded session”.
– The methodology needs adjustments to involve all students. Students
were aware of their teachers’ efforts to handle technology and to adapt their
teaching. Nevertheless, they demanded to care about their peers attending
virtually, in the words of S7: “Maybe the methodology was new, so the teacher
had to adapt his teaching during the first sessions. This created a delay in
everything. But I think my virtual peers need to listen to what we all say during
the session. So the class could be more interactive, and they can participate”.
Some students’ suggestions were as S9 mentioned: “The teacher need to pay
attention to the interaction with my virtual peers. S1 also asked for more
feedback: “Teachers also have to remember to give feedback, even though this
is a hybrid course”. Interestingly, all the methodological suggestions were
made by those students that attended mainly in-person (16%).
5 Discussion
Regarding RQ1, the technological setup confined teachers to the classroom’s

front area, prioritizing visibility and audio for virtual students. However, this
may not be solely due to the setup; all observed teachers primarily lectured,
not requiring movement from that area. As [16] found, lecturing is a natural
teaching approach for hybrid formats, allowing teachers to use the whiteboard
and interact with students. The analytics reports revealed more in-person than
virtual student interactions, a finding also observed in [3], particularly when
lecturing is central to teaching, as in this study. To our knowledge, no other
study reports on teachers’ positions in a hybrid format. However, [9] underlines
the importance of studying these positions in in-person sessions to understand
teacher-student interactions in small groups. The lower virtual participation was
not solely due to the technological setup, as explained in the answers to RQ2.
As for RQ2, teachers identified challenges, including the need for on-site
support, technical issues, and internet connectivity problems affecting student
engagement. These same challenges were found in the study of [3], teachers
and students have to constantly be aware of the session’s audio and video, which
interrupts the natural flow of the class. Some teachers refrained from using group
work due to time constraints, perhaps these challenges were counter-effective to
adopting alternative teaching strategies since it demands planning. Teachers
spent half of their time in non-teaching activities [17]. Hybrid courses brought
flexibility for working students, but teachers also felt fatigued and prompted for
an on-site assistant, mirroring the sentiments expressed in the studies of [1,3]
where teachers struggle especially with large class groups and when faced with
the complexities of coordinating both virtual and in-person students. Addition-
ally, in the present study, teachers sought supplementary training and made
methodological and time-based adjustments to their teaching planning on the
run.
Regarding RQ3, students generally had positive experiences with the hybrid
learning reported in this study. Students rated hybrid courses higher than previ-
ous in-person formats; similar to the cases reported in [8]. They appreciated the
flexibility of attending classes virtually and the chance to review recorded lec-
tures, which aligns with the findings of [14]. However, students noted technolog-
ical issues impacting the learning process, such as poor video quality and audio
mismatch but they acknowledged their teachers’ effort in providing an alterna-
tive to those students who could not attend in person. These issues were also
observed in the study of [3]; moreover, their students noted that the hybrid for-
mat was the best solution due to the sanitary emergency. Additionally, students
in our study desired more balanced attention to virtual students and prompted
feedback. However, in the study of [15] their teachers reflected that they could
not guarantee equality of participation or attendance from their virtual students
due to connection, technical problems, or because students turned off their cam-
eras, which was also an issue in the reported courses of this study. Notably, most
of the methodological recommendations came from frequent in-person students
who rarely experienced virtual attendance. Besides, the students’ viewpoints
were only from undergraduate learners. A further review of postgraduate stu-
dents is required. Finally, we acknowledge that the results of the study regarding
teachers’ views should be cautiously examined due to its small sample size.
6 Lessons Learned
To overcome the above challenges and capitalize on the findings, we propose the
following guidelines:
– The physical and technological setup of classrooms should ensure a high-
quality experience for both teachers and students. Teachers should have the
freedom to implement strategies that best suit their class and students, while
students should have the opportunity to choose the format that best adapts
to their educational needs, regardless of their mode of attendance.
– The technological setup in the classroom should guarantee the nearest to an

in-person classroom experience, both for teachers and students.
– Teachers would benefit from having a technical assistant who can troubleshoot
issues in both virtual and in-person groups. This person can also monitor the
virtual group and alert the teacher when someone wants to participate.
– Institutions should incorporate active learning in teacher training and person-
alized tech feedback. Plan for interactive, equitable virtual-in-person engage-
ment. These strategies prevent virtual students from feeling neglected, pro-
moting active participation.
– Before courses officially begin, it is important to assess students’ internet qual-
ity and communicate technical requirements and troubleshooting protocols.
Additionally, teachers should incorporate alternative approaches to address
challenges such as low-quality internet connections, as complete avoidance
may not always be possible.
– Feedback remains a vital aspect of a successful learning experience, regardless
of the delivery format.
7 Conclusions and Further Research
This mixed-methods study explored synchronous hybrid learning, revealing that

technological setup significantly influences teaching and learning experiences.
While students valued course flexibility, challenges emerged due to internet con-
nectivity issues. Students emphasized equitable virtual engagement and prompt
feedback. Teachers requested classroom assistants to enhance teaching focus.
Future research includes postgraduate student integration and technological con-
figurations. These targeted strategies are crucial to optimize hybrid learning.
References
1. Bülow, M.: Designing synchronous hybrid learning spaces: challenges and oppor-
tunities. In: Hybrid Learning Spaces, pp. 135–163. Springer, Heidelberg (2022).
https://doi.org/10.1007/978-3-030-88520-5-9
2. Jamison, A., Kolmos, A., Holgaard, J.: Hybrid learning: an integrative approach to
engineering education. J. Eng. Educ. 103, 253–273 (2014). https://onlinelibrary.
wiley.com/doi/abs/10.1002/jee.20041. https://doi.org/10.1002/jee.20041
3. Raes, A., Detienne, L., Windey, I., Depaepe, F.: A systematic literature review
on synchronous hybrid learning: gaps identified. Learn. Environ. Res. 23, 269–290
(2020). https://doi.org/10.1007/s10984-019-09303-z
4. Xu, J., Yang, H., MacLeod, J.: Connected classroom climate in hybrid classroom:
model and comparison. In: International Conference On Blended Learning, pp.
187–195 (2016)
5. Detyna, M., Sanchez-Pizani, R., Giampietro, V., Dommett, E.J., Dyer, K.: Hybrid
flexible (HyFlex) teaching and learning: climbing the mountain of implementation
challenges for synchronous online and face-to-face seminars during a pandemic.
Learn. Environ. Res. 26(1), 145–159 (2022)
6. Poskitt, C.M., Shim, J.K., Lau, Y.M., Ong, H.S.: Mind the gap: reimagining an
interactive programming course for the synchronous hybrid classroom. In: Hawaii
International Conference on System Sciences (2021)
7. Chauca, L., et al.: Analysis of hybrid educational modalities in Latin America and
their impact on the quality of education (2021)
8. Campos, E., Daruich, S., de la O, J., Castaño, R., Escamilla, J., Hosseini, S.:
Educational model transition: student evaluation of teaching amid the COVID-19
pandemic. In: Frontiers in Education (2022)
9. Martinez-Maldonado, R., Echeverria, V., Schulte, J., Shibani, A., Mangaroska, K.,
Buckingham Shum, S.: Moodoo: indoor positioning analytics for characterising
classroom teaching. In: Bittencourt, I.I., Cukurova, M., Muldner, K., Luckin, R.,
Millán, E. (eds.) AIED 2020. LNCS (LNAI), vol. 12163, pp. 360–373. Springer,
Cham (2020). https://doi.org/10.1007/978-3-030-52237-7 29
10. Bella, M., Hanington, B.: Universal Methods of Design, p. 204. Rockport Publish-
ers, Beverly (2012)
11. Pozyx: unleash the power of real-time location solutions (2015). https://www.
pozyx.io/
12. Software Tableau (2003). https://public.tableau.com/app/discover
13. Holstein, K., McLaren, B., Aleven, V.: SPACLE: investigating learning across vir-
tual and physical spaces using spatial replays, pp. 358–367 (2017). https://doi.org/
10.1145/3027385.3027450
14. Alhusban, H.: A novel synchronous hybrid learning method: voices from Saudi
Arabia. Electron. J. E-Learn. 20, 400–418 (2022)
15. Pham, A., Tran, T.: Teachers’ perceptions of synchronous hybrid teaching during
the Covid-19 pandemic. In: Proceedings Of The 8th International Conference On
Frontiers Of Educational Technologies, pp. 92–96 (2022). https://doi.org/10.1145/
3545862.3545878
16. Raes, A.: Exploring student and teacher experiences in hybrid learning environ-
ments: does presence matter? Postdigital Sci. Educ. 4, 138–159 (2022). https://
doi.org/10.1007/s42438-021-00274-0
17. OECD How Much Time Do Teachers Spend on Teaching and Non-teaching Activ-
ities? (2015).https://www.oecd-ilibrary.org/content/paper/5js64kndz1f3-en
Beyond Numbers: Exploring
the Qualitative Dimension of a Learning
Analytics Dashboard’s Usefulness
Vanessa Heredia Jimenez1 , Irving Valeriano1 , Danny Torres1 ,

Alberto Jimenez-Macı́as2 , and Margarita Ortiz-Rojas1(B)
1
Información, Campus Gustavo Galindo Km. 30.5 Vı́a Perimetral, 09-01-5863,
Guayaquil, Ecuador
margarita.ortiz@cti.espol.edu.ec
2
Universidad Carlos III de Madrid, Avda de la Universidad, 30, 28911 Leganes,
Spain
Abstract. Learning Analytics dashboards provide reflection and

decision-making opportunities for teachers and students. When it comes
to evaluating dashboards, most studies use a mix of quantitative meth-
ods to understand behavior and qualitative methods such as interviews
and focus groups to understand needs and insights. One limitation of
the latter is that they usually imply a small sample size due to the large
amount of work it demands in transcribing, reading, coding, and analyz-
ing the information. This study challenges this limitation by presenting
an in-depth exploration of the qualitative aspect involved in evaluating
the perceived usefulness of an academic counseling system. Although
a mixed methods approach was adopted, this study, that involved 113
teachers, places particular emphasis on the qualitative component, shed-
ding light on unique insights that enrich the understanding of the user’s
experience. The quantitative information gathered through a question-
naire allowed us to find statistical differences before and after the system
implementation. The qualitative information in the same questionnaire,
provided feedback on the reason why they liked the new changes. The
approached used, allowed us to reach more users and analyze in a fastest
way, allowing us to have strong reasons too keep the proposed design.
These were mainly attributed to visualization and interaction aspects.
Keywords: learning analytics · dashboards · mixed methods
1 Introduction
Learning analytics dashboards have been discussed in the literature concern-
ing their potential benefits to provide reflection opportunities and supporting
learning[1]. When it comes to evaluating dashboards, different methods arise
(e.g., questionnaires and use tracking) [2]. These are usually aimed at under-
standing the user’s behavior. With a general resurgence of qualitative techniques
https://doi.org/10.1007/978-981-99-7353-8_7
76 V. Heredia Jimenez et al.
in the social sciences, evaluators likewise have started to blend qualitative and
quantitative methods [3]. Focus groups and interviews are the most common
methods that help researchers providing insights on what the user thinks about
the tool. These usually involve 1 to 50 users, depending on the type of study
[4]) given that he emphasis is on the meaning rather than quantifying a phe-
nomenon [5]. Nevertheless, what if the purpose is to enrich the analyses by
gathering users’ insights in a fastest way and involving more users?. This study,
built upon a previous comprehensive quantitative analysis [6], seeks to enrich
the results by gathering a qualitative view on users’ insights into an existing
academic counseling system involving 113 teacher counselors.
This paper is organized as follows. Section 2 provides a brief overview of
the relevant literature. Section 3 describes the methodology used to evaluate
the dashboard and the updates in the academic counseling system. Section 4
presents the results of the evaluation. Section 5 presents the discussion. Finally,
Sect. 6 describes the conclusion, acknowledges its limitations, and suggests pos-
sible directions for future work in this area.
2 Related Work
As mentioned in the previous section, studies that involve qualitative measure-
ments with LADs (Learning Analytic Dashboards) usually contain a small sam-
ple size. In [7], a LAD called LISSA, is presented to facilitate the dialogue
between study advisers and first-year students. The authors evaluated the tool’s
experience and perception through timed notes and a seven-question question-
naire(using a five-point Likert scale) with 101 students. Semi-structured inter-
views with 26 teacher counselors were also conducted to analyze the tool’s effect
on the support of dialog applied to teachers during the counseling sessions.
In [8], the authors proposed an innovative and interactive workshop designed
to test the effectiveness of LADs. They evaluated the acceptance and satisfac-
tion with 63 users through a paper-based survey (five-point Likert scale) using
Technology Acceptance Model [9] about Perceived ease of use and Perceived
usefulness. Additionally, two open questions were included about “What do you
like?” and “What could be improved?”.
In [10], a quality pre-study in several courses on LA was carried on in order
to know what dashboard design elements learners use when interpreting infor-
mation by interviews with open questions. The results revealed students use
multiple dashboard elements when making sense of the displayed information
and other findings revealed learner goals and self-regulated learning skills affect
the perceived relevance of dashboard elements.
The authors in [3], generated evidence about the effectiveness of a Predictive
Learning Analytic system. In this study, Technology Acceptance and Academic
Resistance Models are used in combination with semi-structured interviews.
Questions were open-ended and aimed to detail the experiences and perceptions,
such as ease of use, feelings, and future intentions. Thematic Analysis [11] was
used to evaluate the qualitative data of 6 teachers to identify emerging themes
related to the aims of the study.
Exploring the Qualitative Dimension of a Learning Analytics Dashboard 77
The above studies confirm that to gather qualitative data, most studies
involve from 6 to 63 users. Although an in-depth analysis can be obtained with
a few people, a qualitative understanding of how most people perceive a tool is
sometimes needed. Thus, this study presents a pre-post quantitative evaluation,
enriched with an open question to gather all the user’s insights about the tool’s
usefulness.
3 Methodology
3.1 Research Question
This study aims to answer the following research question: What is the perceived
usefulness of a Learning Analytics Dashboard?
3.2 Context
The study was applied in an engineering-oriented university, in Ecuador during
the academic year 2018–2019. After a needs analysis of the current academic
counseling system, and following a user-centered methodology for the design
and improvement of the dashboards [12], three visualizations were implemented:
two were updated versions, and one was new.
3.3 Academic Counseling System Description: Former

and Improved Version
Academic History Module. As shown in Fig. 1, the former version of the
academic history module was displayed in the system using a table showing all
the subjects taken by the student during all the semesters. It included the year,
semester, subject name, final grade per subject, how many times the subject
had been taken, three grades (midterm, final and make-up grade), subject sta-
tus (passed/failed), and the teacher’s name. If the student wanted to see the
rest of the subjects taken, he/she had to click on a “next” button to see the
other information. Unlinke the former version, with the new design, as shown in
Fig. 2, all the subjects taken during the student’s academic life can be observed
in a single window. Clicking on each subject, shows further information of the
student’s grades compared to the students in the same course, with the option
of also being able to visualize all the students who saw the subject in the same
semester. Additionally, the different colors mark the subject’s type (basic, pro-
fessional, elective, etc) and a mark on the upper left shows its status (e.g. a green
check means the subject was passed the first time).
Available Subjects Module. Figure 3 shows the former version of the avail-
able and suggested subjects module. The table shows the names of the subjects
available for the student’s next academic record. With the new updates, the
module became more interactive with the teacher counselor.
Fig. 1. Academic History module before the updates
Fig. 2. Academic History module after the updates
As seen in Fig. 4, there are two sections. The first one called “available sub-
jects”, shows all the subjects the student can take, grouped by the level in
his/her career. Once the teacher selects the subjects, they move to the next
section called “suggested subjects”. There, the teacher counselor and student
can see the workload and difficulty level of the suggested subjects.
Fig. 3. Available subjects module evolution - Before
Fig. 4. Available subjects module evolution - After
Statistics Module. The statistics window provides information that was not
previously shown in the counseling system. As can be seen in Fig. 5, it is divided
in four sections. The first one shows a bar chart comparing the student with the
students in the same cohort and all the students in the same career based on
the semester average. The second section below, shows a comparison between
the subjects suggested by the teacher counselor (left side) and the subjects cho-
sen by the student (right side) in each of the semesters. It also shows with a
color whether the student passed or failed the chosen subject. The third section
displays a career dropout module. Its shows information about the number of
subjects passed two or other times, passing rate, among others. When the teacher
clicks on “see more”. A pop up windows appears showing the academic dropout
risk. More information about this specific module can be found in [13]. The last
section at the bottom, shows a table that follows up on the attention a student
is receiving from the Student welfare department.
Fig. 5. New statistics module
3.4 Sample
The sample involved 187 teacher counselors who attended a two-hour workshop
on the new visualizations. Out of this number, only 113 (45 female and 68 male)
completed the pre and post measurements to be included in the study. The
participants were from STEM and Non-stem (e.g. business) majors.
3.5 Procedure and Instruments

At the start of the workshop, the teacher counselors were asked to fill out a
questionnaire with one closed question about the perceived usefulness they expe-
rienced from the existing dashboard. The question was: The information (e.g.,
tables, graphs) currently provided by the counseling system is sufficient to make
better decisions to guide the students. Participants had to rate their agreement,
based on a 5-point Likert scale (1: strongly disagree, 5: strongly agree). The
second question was open and asked participants to justify their answer in a
comment box. A month later, the teacher counselors were contacted again after
finishing the academic counseling sessions with their students to fill out the same
questionnaire.
3.6 Data Analysis Procedure
For quantitative analysis, a paired sample T-test was performed to determine if

there was a significant difference between the participants’ answers during the
pre and post-test about the perceived usefulness of the counseling system. As
for the open question, a thematic analysis approach was used [14].
The initial coding was done by two authors independently and focused on the
participant’s general perception of the implemented changes (positive, negative,
neutral). Additional topics emerged in subsequent meetings in which the cod-
ing scheme was revised once more by the two researchers until a unified coding
scheme was reached. These subcategories were related to aspects in the dash-
board that influenced the participant’s perception: visualizations, interaction,
academic data, available subjects, Student wellness information, the inconsis-
tency of information. Moreover, a new category emerged, focusing on additional
user requirements. These were associated with the same subcategories previously
described (e.g., visualizations, interaction).
4 Results
In this section, we present the findings from both quantitative and qualitative
data analysis. While the main emphasis of this study lies in the qualitative
aspect, a small portion of the results show some quantitative data to comple-
ment and reinforce the outcomes of the study. For a further exploration of the
quantitative results, see [6].
Based on our research question: the quantitative results indicate that the
perceived usefulness of the academic counseling system in the post test (μ =
4.416, σ = 0.059) was significantly higher than in the pre test (μ = 3.513,
σ = 0.102): t = –7.661 and p = 5.5014 ∗ 10−1 ) as can be see in Fig. 6.
Regarding qualitative analysis, Fig. 7 depicts the general perception category
about the academic counseling system. Most comments were positive after using
the counseling system with the new views: “the page is very friendly”, “the
information given by the system have helped me guide the student”. On a minor
degree, there were negative comments: “There is data that is not useful.”, “There
are parts that confuse me” and neutral comments: “No comments”, “There is
no perfect tool”.
Figure 8 and Fig. 9 show an in-depth analysis on the reasons for the partici-
pant’s positive and negative comments. As can be seen, the positive comments
are mostly related to interaction and visualizations with the system: “The charts
help enough to identify the student’s situation”, “With the incorporation of the
new tables, the student’s academic monitoring can be carried out and their risks
and abilities can be known”. As for the negative comments, these are mostly
related to visualization issues: “There are graphs in the students’ curricular path
Fig. 6. Perceived usefulness of the academic counseling system
Fig. 7. General perception gathered during pre and post-test
Fig. 8. Number of positive comments
that do not appear updated or do not fit well to the semester that the student is
studying”, “the abandonment risk graph can cause confusion”.
As for the user requirements, Fig. 10 shows that most requirements come
after the new implementation. These are mostly associated to visualization: “I
Fig. 9. Number of negative comments
Fig. 10. Requirements
consider that the class schedule should be added in suggests subjects section,
because sometimes we make suggestions without knowing if the suggested subjects
can be taken or not by the student, due to incompatibility of class and exams
schedules”.
5 Discussion
The results indicated a strong signal on acceptance of the improvements of the

academic counseling system presented. The teacher counselors support the idea
that better visualizations and interaction within the system allow them to have
efficient academic counseling. These results positively contribute to a limited
view that usefulness evaluation has. In a review carried out by [15], the authors
indicate that evaluating usefulness usually relies on small-scale user studies and
provided little evidence if LA was solving teacher’s real issues and needs. We
were able not only to involve a large sample size (113 teacher counselors) but
also to exploit through a single open box question, the reasons why the users
liked the tool or not and what they still need from the system. We are aware like
in our previous study [6] that qualitative instruments, such as interviews and
focus groups, provide more information than just an open question. Nevertheless,
our experience in this study has shown that open questions in questionnaires,
even if it is only one, help gather insights in a faster way and can reach more
users.
Another aspect of analyzing is that the number of negative comments and
requirements regarding visualization increased in the post-test, although the gen-
eral perception was positive. We could attribute these results to the fact that
before the visualizations, the teachers were unaware of the advantages LA pos-
sesses when visualizing the information. After using the updated version of the
counseling system with the students, they might have become more demanding
in requirements.
6 Conclusions, Limitations, and Future Work
In this study, we evaluated the perceived usefulness of updates in an educational

counseling system. The results positively confirm that the teacher counselors feel
that they can make better decisions when it comes to giving academic advising,
due to the visualizations. The added value of this study is that using only one
open question, we were able to analyze in depth what the users liked, disliked and
needed in the system. This method allowed us to obtain qualitative data of more
than 100 users in a faster way, which could have taken more time and effort to
do through interviews and focus groups. Although we obtained positive results,
we also acknowledge some limitations. First, evaluating usefulness using one
question is too limited. More questions should be included to analyze usefulness
from different perspectives. Second, only usefulness was evaluated. Including
other aspects, such as utility or usability, could provide a complete view of the
user’s system satisfaction. As for future work, the information gathered about the
system requirements will be considered for an updated version of the counseling
system, with an especial emphasis on visualization.
Acknowledgements. Work funded by the LALA project (grant no. 586120-EPP-1-

2017-1-ES-EPPKA2-CBHE-JP). This project has been funded with support from the
European Commission. This publication reflects only the views of the authors, and
the Commission cannot be held responsible for any use which may be made of the
information contained therein.
References
1. Klerkx, J., Verbert, K., Duval, E.: Learning analytics dashboards (2017)
2. Charleer, S., Klerkx, J., Duval, E., De Laet, T., Verbert, K.: Creating effec-
tive learning analytics dashboards: lessons learnt. In: Verbert, K., Sharples, M.,
Klobučar, T. (eds.) EC-TEL 2016. LNCS, vol. 9891, pp. 42–56. Springer, Cham
(2016). https://doi.org/10.1007/978-3-319-45153-4 4
3. Herodotou, C., Rienties, B., Boroowa, A., Zdrahal, Z., Hlosta, M.: A large-scale
implementation of predictive learning analytics in higher education: the teachers’
role and perspective. Educ. Tech. Res. Dev. 67(5), 1273–1306 (2019)
4. Hernández-Sampieri, R., Torres, C.P.M.: Metodologı́a de la investigación, vol. 4.
McGraw-Hill Interamericana Méxicoˆ eD. F DF (2018)
5. Schutt, R.K.: Investigating the Social World: The Process and Practice of Research.
Sage publications, Thousand Oaks (2018)
6. De Laet, T., Millecamp, M., Ortiz-Rojas, M., Jimenez, A., Maya, R., Verbert, K.:
Adoption and impact of a learning analytics dashboard supporting the advisor-
student dialogue in a higher education institutgovernment in latin america. Br. J.
Educ. Technol. 51(4), 1002–1018 (2020)
7. Millecamp, M., Gutiérrez, F., Charleer, S., Verbert, K., De Laet, T.: A qualitative
evaluation of a learning dashboard to support advisor-student dialogues. In: Pro-
ceedings of the 8th International Conference on Learning Analytics and Knowledge,
pp. 56–60 (2018)
8. Rienties, B., Herodotou, C., Olney, T., Schencks, M., Boroowa, A.: Making sense of
learning analytics dashboards: a technology acceptance perspective of 95 teachers.
Int. Rev. Res. Open Distrib. Learn. 19(5) (2018)
9. Granić, A., Marangunić, N.: Technology acceptance model in educational context:
a systematic literature review. Br. J. Educ. Technol. 50(5), 2572–2593 (2019)
10. Jivet, I., Scheffel, M., Schmitz, M., Robbers, S., Specht, M., Drachsler, H.: From
students with love: an empirical study on learner goals, self-regulated learning and
sense-making of learning analytics in higher education. Internet High. Educ. 47,
100758 (2020)
11. Braun, V., Clarke, V.: Thematic analysis (2012)
12. Ortiz-Rojas, M., Maya, R., Jimenez, A., Hilliger, I., Chiluiza, K.: A step by step
methodology for software design of a learning analytics tool in Latin America:
a case study in Ecuador. In: 2019 XIV Latin American Conference on Learning
Technologies (LACLO), pp. 116–122. IEEE (2019)
13. Heredia-Jimenez, V., Yaguana, J., Jiméenez-Macı́as, A., Ortiz-Rojas, M.: Using
design-based research for an academic dropout and retention dashboard. In: 2023
Ninth International Conference on eDemocracy & eGovernment (ICEDEG), pp.
1–5. IEEE (2023)
14. Mårell-Olsson, E.: Using gamification as an online teaching strategy to develop
students’ 21st century skills. In: IxD&A: Interaction Design and Architecture(s),
no. 47, pp. 69–93 (2021)
15. Verbert, K., et al.: Learning dashboards: an overview and future research oppor-
tunities. Pers. Ubiq. Comput. 18(6), 1499–1514 (2014)
Computational Thinking as a Didactic Strategy
for the Development of Mathematical
Competencies in Problem-Solving
Sonia Peñaloza-Ochoa(B) and Patricia Ortega-Chasi
Universidad del Azuay, Cuenca, Ecuador

smp.ochoa@gmail.com
Abstract. Background: Acquiring mathematical problem-solving skills often

poses challenges for teachers and students. Aim: This study aimed to design didac-
tic strategies that utilize computational thinking to enhance problem-solving abil-
ities in seven-year-old children. Methodology: Based on the characteristics and
elements of Computational Thinking (CT) found in the literature, six strategies
for developing Mathematical Competencies in Problem-solving were developed,
evaluated, validated and implemented in an eight-week classroom intervention to
enhance students’ problem-solving skills in mathematics. The sample comprised
27 students of both genders. Pre- and post-tests were administered using the Pro-
calculus problem-solving subtest. Results: The validation of the strategies was
conducted using Aiken’s V, with criteria such as sufficiency, clarity, coherence,
and relevance surpassing a threshold of 0.7. The application of these strategies
yielded a significant impact (t = 9.918, p = .000, CI [−4.158302, −2.730587];
Cohen = 1.909). Conclusion: Computational thinking and the designed strategies
proved to be effective in facilitating the acquisition of skills related to arithmetic
problem-solving.
Keywords: Mathematical Competencies · Didactic Strategies · Innovative

Methodologies · Computational Thinking
1 Introduction
1.1 Problem Statement and Current Study
The teaching, learning, and evaluation process within the educational system, as
described by Revelo-Rosero et al. [1], plays a crucial role in identifying issues related
to the mastery and conceptual understanding of fundamental mathematical procedures
such as addition, subtraction, multiplication, and division. Furthermore the potential
consequences of a child’s inability to acquire arithmetic skills, which can hinder the
development of relevant abilities necessary for everyday and academic contexts [2, 3].
Consequently, seven-year-old students often face challenges when tackling mathemati-
cal problems, negatively impacting their academic performance. As a result, there is a
growing interest in integrating Computational Thinking (CT) into the educational realm,
particularly at an early age, focusing on fostering conceptualization and problem-solving
abilities [4, 5].
https://doi.org/10.1007/978-981-99-7353-8_8
Computational Thinking as a Didactic Strategy 87
Mathematical Competencies (MC) play a crucial role within the educational com-
munity, encompassing fundamental knowledge individuals should employ in their daily
activities [6]. MCs enable the development of thinking skills necessary for address-
ing everyday problems through comprehension, reasoning, and investigation, with the
ultimate goal of timely solving posed exercises [7, 8]. In light of this, Shute et al. [9]
and Valverde-Berrocoso et al. [10] propose the integration of Computational Thinking
(CT) into the teaching of MCs, particularly in the context of problem-solving. This fusion
emerges as a response to the need for students to cultivate effective strategies and provides
teachers with the means to create innovative and conducive learning environments.
It is important that the teaching of MCs takes place in various contexts, not just
within the classroom, for the learning to be meaningful. Additionally, it is necessary to
include didactic strategies grounded in the real world. This approach ensures that the
relevant assessment demonstrates the extent of knowledge and practical skills acquired
by students [11–13]. Furthermore, when solving tasks that incorporate affective, cogni-
tive, and actionable aspects, it becomes possible to obtain a replica of activities centered
on student learning [10, 14–16].
The origin of the principles of CT, as proposed by Papert [17], aims to address
social needs by providing solutions to various problems with the support of commonly
used computer tools in our environment. These tools represent universal forms and skills
everyone should know and utilize. Additionally, Wing [18, 19] emphasizes that CT offers
interdisciplinary applications in multiple areas of education.
Various conceptions [20–24] have emerged based on these ideas, focusing on criti-
cal, logical, creative, and mathematical thinking characteristics. Additionally has been
considered as a methodology within the field of computer science [5, 9, 25–27]. Conse-
quently, some authors [28–30] argue that CT should be taught as it allows for problem
decomposition into smaller parts, leading to recognition and abstraction and guiding
step-by-step progression through algorithms.
Therefore, Information and Communication Technologies (ICT) pave the way for
global education, reducing learning inequalities, improving the effectiveness and time-
liness of teaching, facilitating the work of educators and educational management [31].
In recent years, there has been significant interest in teaching Computational Thinking
(CT), particularly within the educational domain. According to Adell et al. [32], CT sup-
ports individuals in problem-solving with the aid of computers. Thus, all its specificities
contribute to the construction of computer systems [33]. Consequently, programming
is not merely a cognitive ability used for code design but also a cultural and social
competence. It constitutes a form of “connected learning,” enabling an understanding of
how technology and design offer solutions to everyday problems [10], whether through
computer-based learning or traditional pen-and-paper methods, commonly referred to
as unplugged learning [28, 35].
Creating interactive activities through a computer is common practice for teach-
ers who guide learning in virtual or face-to-face settings. Some programs only require
micro-practices of scripts to modify some basic programming functionality [32, 36].
This implies that teachers must also be prepared in the knowledge and skills of Com-
putational Thinking (CT) and Mathematical Competencies (MC), understanding how
they are interrelated. This understanding is of great support and a priority within the
88 S. Peñaloza-Ochoa and P. Ortega-Chasi
educational system [9, 37]. Indeed, Balladares et al. [38] and Moya [39] emphasize the
incorporation of Information and Communication Technologies (ICT) in educational
trends, progressing towards Technologies of Learning and Knowledge (TLK) and sub-
sequently to Technologies for Empowerment and Participation (TEP). This progression
enables skills development in employing didactic strategies to tackle the challenge of
independent problem-solving. One must comprehend the problem, execute the action
plan, and evaluate the process to achieve this.
Computational Thinking (CT) and its branches in computing improve problem-
solving skills and development, starting from something as basic as a sheet of paper and
a pencil [40, 41]. The Siemens’ Connectivism is ideal for studying mathematics [42, 43].
The Connectivism is closely related to CT and problem-solving, based on principles such
as diversity of opinions, specialized nodes, non-human devices, and individual interest.
These principles foster learning enhanced by knowledge networks and technology [44,
45].
There is a need for educational agreements regarding the teaching of Computational
Thinking (CT) to promote the acquisition of essential computer-related knowledge across
various areas of the curriculum [46, 47] and also highlight the importance of students
gaining experiences and skills, as it enables their adaptation to different environments
[48, 49]. They advocate for approaching the world with a formal logical perspective and
striving to identify and resolve problems.
Similarly, assert that it is crucial to explore suitable technological resources for class-
room use that enable the development of didactic strategies [50]. These strategies should
contribute significantly to enhancing understanding of the solution of mathematical
problems, fostering direct interaction with programming languages [51, 52].
1.2 Research Question
Based on the previously discussed reasons, the research question guiding this study is
how can Computational Thinking be applied to develop mathematical competencies in
problem-solving among seven-year-old children in a school in the south of Ecuador?
1.3 Aims
This study aims to identify the possibility of applying Computational Thinking (CT)-
based strategies that improve skills and competencies in solving mathematical problems.
According to Basogain et al. [20], teaching CT and its incorporation into the classroom
through programming projects highlight the importance of such learning. Pérez et al.
[50] also emphasize the importance of exploring suitable technological resources for
classroom use, significantly strengthening comprehension issues in solving mathemati-
cal problems. This study contributes to the growing body of evidence on the effectiveness
of incorporating Computational Thinking within instructional design and assessment,
as it is an emerging field of study in Ecuador. This work was grounded in designing,
validating and evaluating didactic strategies based on applying Computational Thinking
to enhance problem-solving skills in seven-year-old children.
This section outlines the step-by-step procedure employed in this research [53, 54] to
develop, evaluate, and validate strategies to enhance students’ problem-solving skills in
mathematics through CT.
2.1 Sample and Participants
The sample for the implementation of the designed didactic strategy consisted of twenty-
seven seven-year-old children (n = 27) enrolled in the third grade of a school in Cuenca.
Inclusion criteria included full enrollment and the parents or guardians reviewing and
signing the informed consent form. Exclusion criteria encompassed students with men-
tal health disorders, neurodevelopmental disorders, or moderate, severe, or profound
hearing impairments.
To validate the strategies, a non-probabilistic convenience sample of ten experts [55]
was considered. These experts were teachers with at least five years of experience in
elementary education, specifically in teaching mathematics.
2.2 Instruments
Subtest number nine -Arithmetic Problem Solving- of the Pro-cálculo test was used to
assess problem-solving skill development. This subtest aims to evaluate the essential
aspects of various competencies involved in combining, transforming, and comparing
concepts and numbers to perform additions, subtractions, or combined operations orig-
inating from a problem [56]. The subtest consists of four arithmetic problems, and the
sum of the scores obtained determines mastery in problem-solving [57].
For the content validation of the strategies designed based on Computational Think-
ing, the expert judgment form proposed by Escobar-Pérez and Cuervo-Martínez [58]
was employed. This form consists of two main parts. The first part includes the study’s
purpose, the expert’s background information, and the invitation to act as an expert. In
the second part, the form explains the indicators used to evaluate the items. These indi-
cators reflect the level of agreement among the judges and include sufficiency, clarity,
coherence, and relevance, each rated on a scale of (1) does not meet the criterion, (2)
low level, (3) moderate level, and (4) high level. The form also includes a section for
additional observations.
2.3 Procedure
Problem Identification. An approach was made to the legal representatives of the study
population, and a lack of strategies among the students to master mathematical problem-
solving skills was observed. This observation served as the foundation for the research
guiding question. To highlight the issue, the mathematics problem-solving subtest of
Pro-cálculo was utilized [56].
Plan Development. Based on a literature review, studies were analyzed from academic
databases such as Scopus, Dialnet, Redalyc, and Scielo to determine the characteris-
tics and elements of Computational Thinking (CT) relevant to problem-solving. These
elements and characteristics were extracted to design the strategies. Consequently, six
strategies were initially developed, aligning with the features and elements of CT. These
strategies underwent analysis by a panel of experts composed of ten experienced teach-
ers. The expert panel evaluated the strategies based on sufficiency, clarity, coherence,
and relevance criteria, thereby validating their content. The expert judgment form [58]
was utilized for this purpose. Following validation, the strategies were implemented in
the classroom over eight sessions, with one session per week, to enhance the mastery of
problem-solving skills in mathematics.
Plan Implementation and Evaluation. The developed plan was implemented over
eight weeks corresponding to the second semester of the academic year 2021–2022.
The plan consisted of session objectives, resources, strategies, monitoring, and feed-
back. During and after each session, students were provided feedback through dis-
cussions, allowing for ongoing assessment of the strategies. Finally, to evaluate the
plan’s impact, a new administration of the mathematics problem-solving subtest of
Pro-cálculo [56] was conducted to identify any differences between the pre-and post-
intervention moments. The interested reader can review the implementation plan at the
following link: https://drive.google.com/file/d/11Ifd9o7kddoNARDdg6wuAMkN53txv
fTS/view?usp=sharing.
Feedback. After analyzing the plan’s impact results, the students were informed about
their achievements, and the outcomes were also shared with their legal representatives.
2.4 Bioethical Conditions
The informed consent form was shared with the legal representatives, who reviewed it
and provided their signatures. The students and their representatives received no finan-
cial benefits and were not obligated to participate in the study. The benefits generated
were collective, as they acquired new strategies for mathematical problem-solving. The
information gathered is fully protected and safeguarded by the author of this study,
ensuring confidentiality regarding personal identification data.
2.5 Data Processing
After the respective data collection (elements and characteristics of CT, expert judgment,
and test application), the following procedures were carried out: (1) For the design of
didactic strategies, articles related to CT were reviewed, and those providing informa-
tion on the elements and characteristics were selected. (2) The validation of strategies
by experts was conducted using the V Aiken technique, considering sufficiency, clar-
ity, coherence, and relevance as criteria. (3) Finally, for intervention data analysis, the
normality distribution of the data was checked using the Shapiro-Wilk test due to the
sample size of the students (<30). Descriptive statistics such as frequencies and percent-
ages were used to characterize the population. A pre-test and post-test were conducted
for the subtest number nine -Arithmetic Problem Solving- of the Pro-cálculo test to deter-
mine if there were significant differences in students’ performance before and after the
intervention. The pre-test and post-test results were compared using the paired samples
t-test, and the effect size was determined using Cohen’s d.
3 Results
Scopus, Dialnet, Redalyc, and Scielo databases were consulted to determine the elements
and characteristics of Computational Thinking. Forty articles were found, and after a
thorough analysis of the full text, seventeen were excluded as they did not contain
the variables of interest for the study. The remaining 23 articles were used to extract the
elements and characteristics of Computational Thinking, which are presented in Table 1.
Following identifying relevant characteristics, the strategies were thoughtfully devel-
oped, aligning with the elements of the Computational Thinking framework. A panel
of expert judges conducted a meticulous validation process to ensure their quality and
effectiveness. These judges carefully assessed the strategies based on predetermined cri-
teria, such as sufficiency, clarity, coherence, and relevance. This comprehensive analysis
Table 1. Results of the bibliographic analysis of the elements and characteristics of Computational
Thinking.
Authors Elements Characteristics

Adell et al. [32]; Álvarez, [51]; Logic reasoning Decision making based on analysis
Csizmadia et al. [59]; Lockwood y Algorithm Define and identify problems by
Mooney [29]; Ortega-Ruipérez [60]; applying a series of steps
Pérez-Angulo [33]); Pérez et al. [50];
Polanco et al. [25]; Roig-Vila y Decomposition Ability to organize and analyze data
Moreno-Isac [61]; Román-González into smaller parts
et al. [30]; Sánchez-Vera [5]; Abstraction Simplify the problem in parts,
Sengupta et al. [35]; Shute et al. [9]; discarding unnecessary things that
Terroba et al. [41] make it difficult to understand
Representation Generality and change to other areas
Assessment Methods of abstraction,
categorization, analysis, application
and design
Balladares-Burgos et al. [38]; Analysis of the effects of Establishes scope, advantages and
Blikstein [46]; Bohórquez- Chacón & computing limitations in the use of technology
Amaya-Torrado [36].; Caspersen
et al. [47]; del Rosario Ramírez [62];
Denning [63]; Liljedahl [64];
Téllez-Ramírez [65]; Valverde-
Berrocoso et al. [10];; Vergel et al.
[16]; Velázquez y Martín [49]
Produce computational They go from being buyers of
artifacts technology to being creators
Use of abstraction and Its real elements are abstracted and
models models are built that directly
analyze or modify conditions
(continued)
Authors Elements Characteristics

Analyze problems and Disintegration of the problem as a
artifacts technique to reach the result
Pattern recognition and Interact and develop repetitive
generalization procedures through the location and
establishment of characteristics
Algorithmization Organize sequential and logical
processes to solve problems
Communicate processes Share information that serves as a
and results basis for new knowledge
Practical group work Foundation of knowledge to solve
problems and achieve excellent
benefits
provided insightful feedback on the strengths of the constructed strategies and areas that
could benefit from further refinement.
After applying the expert criterion rubric constructed based on Escobar-Pérez and
Cuervo-Martínez [58], which analyzed criteria such as sufficiency, clarity, coherence,
and relevance for each of the six proposed strategies, the attribute measurement was
conducted using a Likert-type scale. The response options on the scale were: 1. Very
low, 2. Low, 3. Moderate, and 4. High.
After applying the experts’ criteria, statistical analysis was carried out using Aiken’s
V coefficient [66], which indicates a minimum value of .69 to accept an item as valid.
Additionally, Merino-Soto and Livia-Segovia [67] suggest a threshold of .70, which this
study used as a reference. The reported indices can be found in Table 2.
Table 2. Expert Criterion V Indices.
Criteria
Strategies Sufficiency Clarity Consistency Relevance Total
Mental model 0.93 0.9 0.93 0.93 0.92
Flowchart 0.9 0.9 0.86 0.9 0.89
Data representation 0.9 0.93 0.93 0.9 0.91
Logical thinking 0.83 0.83 0.86 0.86 0.85
Pattern recognition 0.86 0.86 0.86 0.86 0.86
Solution pathway 0.93 0.9 0.93 0.93 0.92
When establishing the gender frequencies of the participants, it was observed that
there was a nearly equal proportion between them (male = 55.6%; female = 44.4%).
Their ages ranged from 7 to 8 years, and they belonged to the third grade of primary
education at a school in Cuenca.
As part of the action research process, during the initial data collection phase, it
was found that most students exhibited a categorization of normal learning in arithmetic
problem-solving. Following the intervention implemented through strategies based on
computational thinking, it was observed that the percentage of students categorized as
low (7.4%) rose to the normal category (59.3%), and 40.7% were categorized as high.
The normal distribution of the data generated by the summation of scores for each
problem was asses by the Shapiro-Wilk test, given the small sample size (n < 30). It
was found that both the pre-test and post-test data exhibited a normal distribution (p >
.05), allowing for parametric statistics.
After implementing the intervention for eight weeks, the paired t-test was conducted
for a related sample, assuming no differences between the samples to evaluate the acquisi-
tion of mathematical problem-solving skills based on computational thinking. Following
the application, a 95% confidence interval was reported ranging from −4.158 to −2.731,
with a t-value of 9.918 and p-value of .000. The level of significance allows us to reject
the null hypothesis and accept the alternative hypothesis, indicating differences between
the means of the two samples. The effect size was also calculated, resulting in 1.909,
categorized as a very large effect. This demonstrates that the intervention has generated
changes in the learning of arithmetic problem-solving among the students, as shown in
Table 3.
Table 3. Effect of the implementation of strategies on the learning of arithmetic problem-solving
M SD t p d
Pro cálculo pre-test 3.52 1.673 9.918 0 1.909
Pro cálculo post-test 6.96 1.018
Note: N = 27 students. M = mean score of the responses given in the Pro-calculation test
4 Discussion
This study aimed to design didactic strategies incorporating computational thinking
to enhance arithmetic problem-solving skills in seven-year-old children. As a result,
the study identified the specific elements and characteristics of computational think-
ing (CT) relevant to this context and proposed six activities validated by experts. The
study incorporated various elements of CT, such as logic reasoning, algorithm develop-
ment, decomposition, abstraction, representation, assessment, analysis of the effects of
computing, production of computational artefacts, utilization of abstraction and models,
problem and artefact analysis, pattern recognition and generalization, algorithmizing,
communication of processes and results, as well as practical group work. By integrat-
ing these elements into the activities, the study aimed to foster computational thinking
skills among the participants, promoting their ability to approach problems analyti-
cally, think logically, break down complex tasks, create efficient algorithms, and effec-
tively communicate their processes and outcomes within a collaborative environment.
These elements served as fundamental building blocks in designing the activities to

enhance the participants’ computational thinking abilities and facilitate their engagement
in problem-solving tasks.
Furthermore, research by Román-González et al. [30] and Zapata-Ros [28] highlight
the importance of incorporating additional elements into computational thinking educa-
tion. These elements include recognizing repetitive patterns, breaking down problems
into smaller stages, utilizing written algorithms for problem-solving, and abstracting
relevant information for the problem at hand. Thus, teachers must establish connec-
tions with appropriate didactic resources that facilitate student engagement and provide
opportunities to demonstrate their learning within a computational thinking framework.
Additionally, Lockwood and Mooney [29] suggest that each element should be devel-
oped according to its associated characteristic, with activities defined and aligned with
the curriculum and appropriate materials for children to demonstrate mastery through
assessment. On the other hand, Téllez-Ramírez [65] emphasizes that CT is related to indi-
viduals’ concrete knowledge, including concrete mathematical thinking, among others,
applying various aspects of everyday life. This is also affirmed by Valverde-Berrocoso
et al. [10], who argue that these particularities enable students to confront situations and
demonstrate problem-solving abilities in a real-life context, thus fostering 21st-century
competency. This viewpoint is supported by other authors [36, 38, 46, 47, 63, 68].
Furthermore, the production of computational artefacts, communication of results, and
teamwork are also highlighted [49, 62, 65].
Similar methods, such as the Singapore method and the Polya method promote active
learning of mathematics. Espinoza [69] considers that the Singapore method promotes
specific skills to develop competencies in problem-solving, starting from concrete steps,
concrete representations with pictograms, and gradually progressing to the abstract or
symbolic level. According to Meneses and Peñaloza [70], the Polya method aims to guide
students in solving problems through steps, demanding thinking skills and knowledge
mastery. Thus, it can be observed that both the strategies developed based on CT and the
Singapore and Polya methods are grounded in processes that foster critical and reflective
learning of mathematics pragmatically in the classroom.
The acquisition of computational thinking-based mathematical problem-solving
skills evaluated after implementing the intervention for eight weeks indicate that stu-
dents have shown significant changes in acquiring skills that enable problem-solving. In a
García-García et al. [71] study, children aged 9 to 13 were evaluated, forming two groups.
The first group underwent an intervention process to improve mathematical learning,
followed by a knowledge test administered to both groups, revealing specific differ-
ences. Similarly, Parra-Vallejo [72], through a quasi-experimental design with pre-and
post-tests, demonstrates improvement in mathematics learning using blended learning
and computational thinking. Our results suggests that the strategies enable students to
recognize repetitive patterns, process algorithms presented in the problem, abstract rel-
evant information related to the proposed problem, and finally decompose a problem
into smaller parts. All of this is supported by the differences found through the pre-and
post-tests.
The main limitations that emerged during this research are related to the popula-
tion size (n = 27), which is justified by the restrictions imposed by COVID-19, where
educational centers conducted hybrid classes. It was necessary to wait for each turn of
face-to-face classes to administer the pre-test, implement the strategies, and conduct the
post-test. As a prospective suggestion, it is recommended to expand the sample size of
the population to ensure representativeness using a probabilistic sampling method. Fur-
thermore, comparative intergroup or intragroup studies can be conducted, considering
differences between parallel classes or groups of the same age and between rural and
urban populations.
The results presented in this document provide a baseline for other researchers to
become interested in applying computational thinking to problem-solving. Additionally,
they enable teachers to seek strategies or alternatives that can address the challenges
encountered [48, 49, 73]. Sánchez-Vera [5] posits that it is essential for all teachers to
consider incorporating computational thinking into their present and future instructional
practices, a viewpoint shared by Adell et al. [32], Bocconi et al. [74], and González et al.
[75]. On the other hand, this study contributed to the gaps in knowledge and evidence
in Ecuador with respect to this topic, since, after reviewing the literature, the absence of
research in the country was noted.
5 Conclusions
This study effectively achieved its objective of investigating the role of computational
thinking (CT) in developing arithmetic problem-solving skills among third-grade stu-
dents. Consequently, strategies were developed by leveraging the elements and character-
istics of CT, guided by a comprehensive literature review. The validity of these strategies
was confirmed through expert panel evaluation, affirming their appropriateness for the
intended educational context.
The identified elements and characteristics of CT in the learning-evaluation process
can be summarized as follows:
• Recognition of repetitive patterns
• Decomposition of problems into smaller phases
• Abstraction of relevant information pertinent to the given problem
• Utilization of written algorithms for effective problem-solving
Therefore, teachers must establish meaningful connections with instructional
resources that empower students to showcase their learning. Integrating CT strategies
into the curriculum makes the learning process more purposeful, manifested through rel-
evant activities, and effectively assessed when addressing problems. Equipping students
with these skills enhances their abilities to navigate the challenges of the 21st century
and enables them to apply computational thinking in their daily lives.
The strategies developed based on the characteristics of computational thinking
obtained an initial level of evidence through expert judgment, which indicated a positive
acceptance of the strategies. Therefore, there was agreement among the experts, sug-
gesting that the strategies were acceptable. Additionally, the differences between the pre
and post-tests indicate a gain in knowledge and the development of skills that enhance
and promote various competencies, including problem-solving abilities.
Upon evaluating the acquisition of skills that enable the solving of mathemati-
cal problems based on computational thinking, it can be concluded that the students,
after the intervention, have demonstrated significant improvements. This highlights the
importance of designing further intervention processes that foster the development of
competencies across various areas of mathematical learning.
Furthermore, the impact of the designed strategies was assessed through psycho-
metric instruments. The arithmetic problem-solving subtest of the Test for Calculation
Evaluation was administered as the assessment tool. The subtest was initially adminis-
tered, followed by the intervention based on six strategies, and finally, the subtest was
administered again to analyze the generated impact. The results indicated a positive
impact of the intervention.
Hence, this study provided evidence for including computational thinking in devel-
oping mathematical skills. Therefore, readers of this work are encouraged to develop
and validate their strategies, thereby generating further evidence that contributes to
highlighting the importance of computational thinking in Ecuador’s classrooms.
Declarations Conflict of Interest. The authors declare no competing interests relevant to the
content of this article and report that they have not received any funding to support the preparation
of this manuscript.
References
1. Revelo-Rosero, J.E., Vinicio-Lozano, E., Bastidas Romo, P.: La competencia digital do-cente
y su impacto en el proceso de enseñanza–aprendizaje de la matemática. Espirales Re-vista
Multidisciplinaria De investigación 3(28), 156–175 (2019). https://doi.org/10.31876/er.v3i
28.630
2. AlKhateeb, M.: Effect of mobile gaming on mathematical achievement among 4th graders.
Int. J. Emerg. Technol. Learn. 14(7), 4–17 (2019). https://doi.org/10.3991/ijet.v14i07.10315
3. Pradana, L., Sholikhah, O.: Mathematical literacy training through virtual based mathematics
kits for best mathematics performance. J. Phys. Conf. Ser. 1318, 1–6 (2019). https://doi.org/
10.1088/1742-6596/1318/1/012017
4. Sánchez-Vera, M.E.: Pensamiento computacional en contextos educativos: una aproximación
desde la Tecnología Educativa. Res. Educ. Learn. Innov. Arch. 23, 24–39 (2019). https://doi.
org/10.7203/realia.23.1563
5. Bers, M., González- G.C., Armas-Torres, U.: Coding as a playground: promoting positive
learning experiences in childhood classrooms. Comput. Educ. 138, 130–145 (2019). https://
doi.org/10.1016/j.compedu.2019.04.013
6. Aldana-Bermúdez, E., López-Mesa, J.: Matemáticas para la diversidad: un estudio histó-
rico, epistemológico, didáctico y cognitivo sobre perímetro y área. Revista de Investigación,
Desarrollo e Innovación 7(1), 77–92 (2016). https://doi.org/10.19053/20278306.v7.n1.2016.
5602
7. Mazzilli, D., Hernández, De La Hoz, S.: Procedimiento para Desarrollar la Competencia
Matemática Resolución de Problemas. Escenarios 14(2), 103–119 (2016). https://doi.org/10.
15665/esc.v14i2.935
8. Vilca-Paye, C.: Resolución de problemas como estrategia en el desarrollo de competencias
matemáticas en estudiantes de secundaria. Revista De Investigaciones 8(2), 1028 – 1036
(2019).. https://doi.org/10.26788/riepg.v8i2.887
9. Shute, V., Sun, C., Asbell-Clarke, J.: Demystifying computational thinking. Educ. Res. Rev.
22, 142–158 (2017). https://doi.org/10.1016/j.edurev.2017.09.003
10. Valverde Berrocoso, J., Fernández Sánchez, M.R., Garrido Arroyo, M., del, C.: El pensa-
miento computacional y las nuevas ecologías del aprendizaje. Revista de Educación a
Distancia (RED), (46), 1–18 (2015). https://revistas.um.es/red/article/view/240311
11. Alsina, Á., García, M., Torrrent, E.: La evaluación de la competencia matemática desde la
escuela y para la escuela. UNIÓN. Revista iberoamericana de Educación Matemática 15(55)
(2019). http://www.revistaunion.org/index.php/UNION/article/view/294
12. Fernández, M.J., Gil, M.G., Pérez, M.J.: Evaluación Diagnóstica en Andalucía: Una investi-
gación del área competencia matemática. Aula abierta 43(1), 47–53 (2015). https://doi.org/
10.1016/j.aula.2014.07.001
13. Jiménez, L.: La aplicación del conocimiento contextualizado en la resolución de problemas
matemáticos: un estudio sobre las dificultades de los niños en la resolución de problemas
no rutinarios. Cultura y Educación 24(3), 351–362 (2012). https://www.researchgate.net/
publi-cation/261618638_La_aplicacion_del_conocimiento_contextualizado_en_la_resolu-
cion_de_problemas_matematicos_un_estudio_sobre_las_dificu%20ltades_de_los_ni-nos_
en_la_resolucion_de_problemas_no_rutinarios
14. García, B., et al.: Competencias matemáticas y actividad matemática de aprendizaje. Artes
Gráficas del Valle (2013). https://www.researchgate.net/publication/329127436_Competen-
cias_matematicas_y_actividad_matematica_de_aprendizaje
15. Restrepo-Becerra, J.: Concepciones sobre competencias matemáticas en profesores de edu-
cación básica, media y superior. Revista Boletín Redipe 6(2), 104–118 (2017). https://re-
vista.redipe.org/index.php/1/article/view/199
16. Vergel, M., Duarte, H., Martínez, J.: Desarrollo del pensamiento matemático en estudiantes
de cálculo integral su relación con la planificación docente. Revista Científica 23, (2015).
https://doi.org/10.14483/udistrital.jour.RC.2015.23.a2
17. Papert, S.: Mindstorms: children, computers and powerful ideas. Math. Gaz. 65(434), 298–299
(1980). https://doi.org/10.2307/3616611
18. Wing, J.: Computational thinking. Commun. ACM 49(3), 33–35 (2006). https://www.cs.cmu
edu/~15110-s13/Wing06-ct.pdf. https://doi.org/10.1145/1118178.1118215
19. Wing, J.: Computational Thinking: What and Why? (2010). https://www.cs.cmu.edu/~Com
pThink/resources/TheLinkWing.pdf
20. Basogain, X., Olabe, M., Olabe, J.: Pensamiento computacional a través de la Programa-ción:
Paradigma de Aprendizaje. RED Revista de Educación a Distancia 46(6) (2015). http://www.
um.es/ead/red/46. https://doi.org/10.6018/red/46/6
21. García-Peñalvo, F.J.: What computational thinking is. J. Inf. Technol. Res. 9(3) (2016). https://
repositorio.grial.eu/bitstream/grial/679/1/CT.pdf
22. Hemmendinger, D.: A plea for modesty. ACM Inroads 1(2), 4–7 (2010). https://doi.org/10.
1145/1805724.1805725
23. Mannila, L., et al.: Computational thinking in K-9 education. In: Proceedings of the Working
Group Reports of the 2014 on Innovation & Technology in Computer Science Education
Conference, pp. 1–29. New York, ACM (2014). https://doi.org/10.1145/2713609.2713610
24. Pérez, H., Roig-Vila, R.: Entornos de programación no mediados simbólicamente para el
desarrollo del pensamiento computacional. Una experiencia en la formación de profesores de
Informática de la Universidad Central del Ecuador. RED Revista de Educación a Distancia
46, 1–22 (2015). https://dialnet.unirioja.es/servlet/articulo?codigo=6022698
25. Polanco, N., Planchart, S., Fernández, M.: Aproximación a una definición de pen-samiento
computacional. Revista Iberoamericana de Educación a Distancia 24(1), 55–76 (2021). https://
doi.org/10.5944/ried.24.1.27419
26. Snalune, P.: The benefits of computational thinking. ITNOW 57(4), 58–59 (2015). https://
doi.org/10.1093/itnow/bwv111
27. Selby, C., Woollard, J.: Computational thinking: the developing definition. UK: University
of Southampton E-prints (2013). https://eprints.soton.ac.uk/356481
28. Zapata-Ros, M.: Computational thinking unplugged. Educ. Knowl. Soc. 20(18) (2019).
https://doi.org/10.14201/eks2019_20_a18
29. Lockwood, J., Mooney, A.: Pensamiento Computacional en Educación Secundaria: ¿Dónde
encaja? Una revisión literaria sistemática. Revista internacional de educación en ciencias de
la computaciónen las escuelas 2(1), 41–60 (2018). https://doi.org/10.21585/ijcses.v2i1.26
30. Román-González, M., Pérez-González, J.C., Jiménez-Fernández, C.: Test de pensamiento
computacional: diseño y picometría genera. Madrid, España:III Congreso Internacional so-
bre Aprendizaje, Innovación y Competitividad (CINAIC ) (2015). https://doi.org/10.13140/
RG.2.1.3056.5521
31. UNESCO. Marco de competencia en materia de TIC para docentes (2019). https://unes-doc.
unesco.org/ark:/48223/pf0000371024
32. Adell, J., Llopis, M., Ángeles, Esteve, F., Valdeolivas, G.: El debate sobre el pensamiento
computacional en educación. RIED. Revista Iberoamericana De Educación a Distancia 22(1),
171–186 (2019). https://doi.org/10.5944/ried.22.1.22303
33. Pérez Angulo, J.A.: El pensamiento computacional en la vida cotidiana. Revista Sci. 4(13),
293–306 (2019). https://doi.org/10.29394/Scientific.issn.2542-2987.2019.4.13.15.293-306
34. Ortiz Granja, D.: El constructivismo como teoría y método de enseñanza. Sophia, Colección
de Filosofía de la Educación 19(2), 93–110 (2015). https://doi.org/10.17163/soph.n19.201
5.04
35. Sengupta, P., Kinnebrew, J.S., Basu, S., Biswas, G., Clark, D.: Integrating computational
thinking with K-12 science education using agent-based computation: A theoretical frame-
work. Educ. Inf. Technol. 18(2), 351–380 (2013). https://doi.org/10.1007/s10639-012-9240-x
36. Bohórquez-Chacón, L.F., Amaya-Torrado, Y.K.: Diseño de un modelo pedagógico para la
enseñanza de fundamentos de programación de computadores basado en el uso de la tec-
nología como mediación pedagógica, Respuestas 10(1), 30–37 (2016). https://doi.org/10.
22463/0122820X.676
37. Voogt, J., Fisser, P., Good, J., Mishra, P., Yadav, A.: Computational thinking in compulsory
education: towards an agenda for research and practice. Educ. Inf. Technol. 20, 715–728
(2015). https://doi.org/10.1007/s10639-015-9412-6
38. Balladares Burgos, J.A., Avilés Salvador, M.R., Pérez Narváez, H.O.: Del pensamiento com-
plejo al pensamiento computacional: retos para la educación contemporánea. Sophia, colec-
ción de Filosofía de la Educación 21(1), 143–159 (2016). https://doi.org/10.17163/soph.n21.
2016.06
39. Moya, L.M.: De las TICs a las TACs: la importancia de crear contenidos educativos digitales.
DIM: Didáctica, Innovación y Multimedia 27, 1–15 (2013). https://raco.cat/in-dex.php/DIM/
article/view/275963
40. Diago, P.D., Arnau, D., González-Calero, J.A.: La resolución de problemas matemáticos
en primeras edades escolares con Bee-bot. Matemáticas, educación y Sociedad 1(2), 36–50
(2018). https://www.uco.es/ucopress/ojs/index.php/mes/article/view/12835
41. Terroba, M., Ribera, J.M., Lapresa Ajamil, D., Anguera, M.T.: Análisis observacional del
desarrollo del pensamiento computacional en Educación Infantil-3 años mediante una pro-
puesta de resolución de problemas con un robot de suelo de direccionalidad programada.
Revista de Educación a Distancia (RED) 21(68) (2021). https://doi.org/10.6018/red.480411
42. Guamán Caguana, N.A.: Teoría conectivista en el aprendizaje de derivadas con estudiantes de
bachillerato.Prometeo Conocimiento Científico 3(1), 1–15 (2022). https://doi.org/10.55204/
pcc.v3i1.13
43. Polo Aronés, L.E.: La resolución de problemas: una mirada desde el constructivismo, el
aprendizaje significativo y el conectivismo. Acta Herediana 63(1), 55–60 (2020). https://doi.
org/10.20453/ah.v63i1.3702
44. Siemens, G.: Conectivismo: una teoría de aprendizaje para la era digital. Espacio para
aprender, 4–6 (2004). http://skat.ihmc.us/rid=1J134XMRS-1ZNMYT4-13CN/George%20S
iemens%20-%20Conectivismo-una%20teor%C3%ADa%20de%20aprendizaje%20para%
20la%20era%20digital.pdf
45. Siemens, G.: Todas las respuestas sobre conectivismo. Learn. Rev. 1–5 (2010)
46. Blikstein, P.: Pre-College Computer Science Education: A Survey of the Field. Mountain
View. CA: Google LLC (2018). https://services.google.com/fh/files/misc/pre-college-com-
puter-science-education-report.pdf
47. Caspersen, M., Gal-Ezer, J., Gettrick, A., Nardelli, E.: Informatics for All –The strategy.
ACM Europe e Informatics Europe (2018). https://www.acm.org/binaries/content/as-sets/pub
lic-policy/acm-europe-ie-i4all-strategy-2018.pdf
48. Salgado, A.: Psicogénesis y aprendizaje significativo. Tempus Psicológico 5(1), 50–64 (2021).
https://doi.org/10.30554/tempuspsi.5.1.3595.2022
49. Velázquez, Á., Martín, M.: Análisis del pensamiento computacional desde una perspec-tiva
educativa. Revista de Educación a Distancia (RED) 21(68), 2–18 (2021). https://doi.org/10.
6018/red.484811
50. Pérez, H., Álvarez-Zurita, A., Guevara-Herrera, C.: Dominio de habilidades del pensa-miento
computacional en los estudiantes del Instituto Tecnológico Superior Sucre de Quito - Ecuador.
RIITE Revista Interuniversitaria de Investigación en Tecnología Educativa, (7), 48–60 (2019).
https://doi.org/10.6018/riite.394221
51. Álvarez, M.: Desarrollo del pensamiento computacional en educación primaria: una experien-
cia educativa con Scratch. Castilla. Revista de Ciencias de la Educación. (2), 45–64 (2017).
https://doi.org/10.17345/ute.2017.2.1820
52. Namukasa, I.K., Patel, M., Miller, M.: Introductory CT and Math in the middle grades: Tools
and Resources. Math + Code ‘Zine, 2(2) (2017). http://researchideas.ca/mc/ct-and-math-in-
middle-grades/
53. Elliott, J.: La investigación-acción en educación. Ediciones Morata, S. L (2000)
54. Hernández-Sampieri, R., Mendoza Torres, C.P.: Metodología de la investigación: las rutas
cuantitativa, cualitativa y mixta. McGraw Hill Education (2018)
55. Hyrkäs, K., Appelqvist-Schmidlechner, K., Oksa, L.: Validating an instrument for clinical
supervision using an expert panel. Int. J. Nurs. Stud. 40(6), 619–625 (2003). https://doi.org/
10.1016/S0020-7489(03)00036-1
56. Feld, V., Taussik, I., Azzareto, C.: Test PRO-CALCULO. Test para la evaluación del pro-
cesamiento del número y el cálculo en niños. Paidós (2006)
57. Formoso, J., Barreyro, J.P., Injoque-Ricle, I., Jacubovich, S.: Evaluación de habilidades
matemáticas básicas en niños de 4 años de edad. Subjetividad y procesos cognitivos 21(2),
42–58 (2017). http://www.scielo.org.ar/pdf/spc/v21n2/v21n2a02.pdf
58. Escobar-Pérez, J., Cuervo-Martínez, Á.: Validez de contenido y juicio de expertos: una
aproximación a su utilización. Avances en medición 6(1), 27–36 (2008). https://www.resear-
chgate.net/publication/302438451_Validez_de_contenido_y_juicio_de_exper-tos_Una_apr
oximacion_a_su_utilizacion
59. Csizmadia, A., et al.: Computational thinking-A guide for teachers (2015). https://eprints.
soton.ac.uk/424545/
60. Ortega-Ruipérez, B., Asensio, M.: Evaluar el pensamiento computacional mediante resolu-
ción de problemas: validación de un instrumento de evaluación. Revista Iberoamericanade
Evaluación Educativa 14(1), 153–171 (2021). https://doi.org/10.15366/riee2021.14.1.009
61. Roig-Vila, R., Moreno-Isac, V.: El pensamiento computacional en Educación. Análisis bi-
bliométrico y temático. Revista de Educación a Distancia (RED) 20(63) (2020). https://doi.
org/10.6018/red.402621
62. del Rosario Ramírez, J.G., Díaz Valdés, T., Bastardo Mercedes, E., Cortegaza Ávila, L., Díaz
Iñiguez, M.A.: Estrategias metodológicas para el desarrollo del pensamiento ló-gico en los
alumnos de sexto grado de Primaria. UCE Ciencia. Revista de postgrado 9(2) (2021). http://
uceciencia.edu.do/index.php/OJS/article/view/234/220
63. Denning, P.: Remaining trouble spots with computational thinking. Commun. ACM 60(6),
33–39 (2017). https://doi.org/10.1145/2998438
64. Liljedahl, P., Santos-Trigo, M., Malaspina, U., Bruder, R.: Resolución de problemas en la
educación matemática. En: Resolución de Problemas en la Educación Matemática. Encuestas
temáticas ICME-13. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-40730-2_1
65. Téllez Ramírez, M.: Pensamiento computacional: una competencia del siglo XXI. Educación
Superior 6(1), 23–32 (2019). https://ojs.cepies.umsa.bo/in-dex.php/RCV/article/view/23/22
66. Aiken, L.: Test psicológicos y evaluación. Pearson Education (2003)
67. Merino-Soto, C.M., Segovia, J.L.: Intervalos de confianza asimétricos para el índice la validez
de contenido: Un programa Visual Basic para la V de Aiken. Anales de Psi-cología/Ann.
Psychol. 25(1), 169–171 (2009). https://revis-tas.um.es/analesps/article/view/71631
68. González González, C.S.: Estrategias para la enseñanza del pensamiento computacional y
uso efectivo de tecnologías en educación infantil: una propuesta inclusiva. RiiTE Revista
Interuniversitaria de Investigación en Tecnología Educativa (7) (2019). https://doi.org/10.
6018/riite.405171
69. Espinoza, L., Matus, C., Barbe, J., Fuentes, J., Márquez, F.: Qué y cuánto aprenden de
Matemáticas los estudiantes de básica con el Método Singapur: Evaluación de impacto y
de factores incidentes en el aprendizaje, enfatizando en la brecha de género. Calidad en la
edu-cación 45, 90–131 (2016). https://doi.org/10.4067/S0718-45652016000200004
70. Meneses, M., Peñaloza, D.: Método de Pólya como estrategia pedagógica para fortalecer la
competencia resolución de problemas matemáticos con operaciones básicas. Zona Próxima
31, 7–25 (2019). https://www.redalyc.org/journal/853/85362906002/html/
71. García-García, J., Rodríguez Vásquez, F., Navarro Sandoval, C.: Las estrategias utiliza-das
por los niños tee savi en la resolución de problemas aritméticos. Revista Latinoamericana de
Investigación en Matemática Educativa 18 (2), 213–244 (2015). https://doi.org/10.12802/rel
ime.13.1823
72. Parra-Vallejo, M.: Aplicación de las TIC, b-Learning y Pensamiento Computacional para el
Fortalecimiento de las Competencias Matemáticas. Revista Tecnológica- Educativa Docen-tes
2.0 14(2), 29–41 (2022). https://doi.org/10.37843/rted.v14i2.312
73. Domínguez-Morales, S., Pérez-Hernández, M., Pérez-Sánchez, E. Ambientes de aprendi-
zaje para favorecer competencias matemáticas en educación básica. Revista RedCA, 5(13),
144–162 (2022). https://revistaredca.uaemex.mx/article/view/18790/13905
74. Bocconi, S., Chioccariello, A., Dettori, G., Ferrari, A., Engelhardt, K. Developing computa-
tional thinking in compulsory education. JRC Sci. Hub (2016). https://publications.jrc.ec.eur
opa.eu/repository/bitstream/JRC104188/jrc104188_com-puthinkreport.pdf
75. González, J., Estebanell, M., Peracaula, M.: Robots o programación? El concepto de Pen-
samiento Computacional y los futuros maestros. EKS. Educ. Knowl. Soc. 19, 29–45 (2018).
https://doi.org/10.14201/eks20181922945
Identifying Master’s Program Web Pages
with Machine Learning
Sara Díaz Oporto1 , Sergio Dueñas Vera1 , Walker Manrique Chalco1 ,

César Bragagnini Mendizábal2 , and Alvaro Fernández Del Carpio1(B)
1 Universidad La Salle, Arequipa, Perú
{sdiazo,sduenasv,wmanriquec,alfernandez}@ulasalle.edu.pe
2 Instituto Latinoamericano de Investigación y Estudios Tecnológicos, Arequipa, Perú
mbragagnini@iliiet.com
Abstract. In the journey of education and professional growth, there is a wide

variety of postgraduate programs and specializations offered by different uni-
versities and training centers, many of which have a significant online presence.
Simultaneously, for the institutions offering these programs, it is of interest to
gather information about the competition in order to develop their study offering
strategies accordingly. Likewise, for most individuals interested in pursuing these
studies, identifying these programs on various published web pages proves to be
challenging. This work presents a tool for detecting relevant information from web
page content, allowing the identification of master’s programs. One of the primary
challenges addressed in this research is the technological issue of accurately and
efficiently extracting information from diverse web sources, given the variability
in webpage structures and formats. The tool is based on the analysis of various
Machine Learning and Deep Learning models, along with the application of web
scraping techniques. Tests and validations were conducted on different models,
with the best result being achieved by the deep recurrent neural networks model,
which reached a 91.95% F1-score metric. The use of automated tools for selec-
tion, gathering, and classification of information enhances learning by enabling a
greater focus on points of interest. Therefore, when searching for master’s degree
programs, people would avoid the tedious task of manual searching and sorting.
Instead, they can focus on deeply analyzing and comparing the options. This focus
on exploration and analysis would enhance their understanding of the options and
promote more informed decisions.
Keywords: Master’s Program · Machine Learning · Deep Learning · Web

Scrapping
1 Introduction
Currently, the academic offering of postgraduate study programs is quite broad and
diverse. Academic units that provide postgraduate programs, as part of their bench-
marking activities, periodically monitor the academic offerings provided by competing
institutions. On the other hand, for most individuals interested in pursuing postgraduate
https://doi.org/10.1007/978-981-99-7353-8_9
102 S. Díaz Oporto et al.
programs, identifying content on the internet with the required information proves to
be challenging. Considering the vast amount of information available on the internet,
it is a challenge to locate pages containing postgraduate program information accord-
ing to the individual’s requirements [1]. According to [2], 53.7% of internet users seek
information about products or services, 47.4% search for educational information, 39%
seek clinical and health data, 27.9% search for work-related information, and 23.9% are
looking for information related to government and legal organizations. New techniques
and technologies that can help facilitate access to information are Natural Language
Processing (NLP) and machine learning. These technologies enable more precise and
comprehensive searching, making it easier to find relevant information for the interested
party [2].
On the contrary, manual tracking would be very tedious and would consume a lot
of time and effort. While it is possible to do so, it clearly does not facilitate exten-
sive monitoring or continuous tracking, as the information about these offerings can
change periodically. Furthermore, a critical technological challenge arises in efficiently
and accurately extracting information from various web sources, owing to the diversity
in webpage layouts and formats. Having a tool available would be very useful, both
for academic units managing postgraduate programs and for individuals interested in
pursuing this type of study program.
Therefore, the purpose of this work is to develop a model based on Machine Learning
techniques that enables the identification of web links containing information about
master’s programs. This proposal is a first stage of a project aimed at subsequently
extracting data of interest, page content, and thereby conducting comparisons regarding
types of programs offered, duration, topics, costs, and modalities.
The rest of the document is structured as follows: in Sect. 2, we present the funda-
mental concepts; in Sect. 3, we describe the related works; in Sect. 4, we present the
construction and analysis of the models. Finally, the conclusions and future work are
presented in Sect. 5.
2 General Background
2.1 Machine Learning (ML)
It focuses on the development of algorithms for supervised, semi-supervised, and unsu-

pervised learning. In supervised learning, inputs and outputs are known beforehand
to train the model for classifications and predictions. In unsupervised learning, inputs
are unknown and outputs are discovered values. Some important techniques for data
processing include:
• Support Vector Machine (SVM): It is a supervised machine learning classifier that can
function as both regression and classification. SVM requires a training set with both
positive and negative examples to identify a hyperplane that maximally separates the
positive and negative data points in n-dimensional space [3]. Although this method
has the ability to handle high-dimensional data [4], it presents some drawbacks such
as complexity in training and algorithm categorization, high consumption of time and
memory during the training and classification stages [5].
Identifying Master’s Program Web Pages 103
• Decision Tree: It is a supervised learning technique that divides a dataset into groups
that are as homogeneous as possible based on the variable to be predicted. It takes a
set of classified data as input and generates a structure similar to a tree, resembling
a flowchart. Each leaf node represents a decision or class, while each non-terminal
(internal) node represents a test [6].
• Multinomial Naïve-Bayes: It is a variant of the Naïve-Bayes classifier designed
for data with multinomial distributions, specifically for text classification. It does
not strictly follow Bayesian principles as it doesn’t estimate a posterior distribu-
tion over parameters by taking training documents and then using it for predictive
inference [7].
2.2 Deep Learning (DL)

It is a branch of ML based on artificial neural networks, comprising multiple hidden layers
between input and output layers with neurons for feature transformation and extraction.
Its structure can be customized by configuring the so-called hyperparameters. As relevant
techniques for data processing, we have the following:
• Word-Embedding: They represent a token as a multidimensional numerical vector,
usually in a dimensionality higher than 50 features. Among these word embeddings,
we have: Word2Vec produces a vector space, typically of several hundred dimensions,
where each unique word in the corpus is assigned a corresponding vector in the space
[8]; FastText: By extracting information from subwords, it forms vectors for them
that can be reassembled to create vectors for unknown words [9].
• Recurrent Neural Network (RNN): It is primarily used for NLP, speech recognition,
and any form of sequential data. Its structure features a directional loop and uses
internal memory to handle more extensive historical records. It’s called recurrent due
to the utilization of previous calculation outputs for the current calculation of each
element in a sequence [10].
• Long-Short Term Memory (LSTM): It solves the problem of RNNs regarding network
degradation when long dependencies exist. LSTM uses gates and cells to preserve
information within the network. Through its short-term memory cell, information is
updated at each time step. The information that should be retained or discarded from
the memory cell is controlled by the forget gate. An input gate is used to control
new information that will be stored in memory. Lastly, an output gate determines the
memory’s output [11].
• Gated Recurrent Unit (GRU): It is a variant of RNN as it uses a gating mechanism to
control information within a unit, without relying on a dedicated memory cell vector,
but only utilizing a reset gate and an update gate, allowing for long-term information
storage. Both LSTM and GRU have demonstrated satisfactory performance for many
NLP tasks [12].
• Transformers Networks: These are a type of DL model trained with large datasets and
include attention mechanisms that select specific parts of the input as they process it.
This allows for much faster and more accurate predictions. In addition to the attention
mechanism, transformers use the technique of multi-head attention, which enables
the model to focus on multiple parts of the input simultaneously. One of the most
popular transformer models is BERT (Bidirectional Encoder Representations from
Transformers), which is pretrained on a large amount of text data and then fine-tuned
for specific NLP tasks [13]. Existing variants of BERT include RoBERTa, ALBERT,
DistilBERT, SmallBERT, and more.
2.3 Web Page Classification
Web pages are semi-structured documents written in HTML, used to display content on
the internet. They often contain links to other pages through hyperlinks and content such
as advertisements and other irrelevant information [5]. The Document Object Model
(DOM) structure is used to extract data from the structured parts of a web page.
The classification of web pages involves assigning one or more predefined cate-
gory labels [14], playing a significant role in enhancing search results by providing a
categorized view of search outcomes [15, 16]. Applications developed to classify web
pages focus on information retrieval, recommendation systems, targeted advertising,
content filtering for spam detection, phishing and fraud detection, malware detection,
and parental control [16]. Classification can be divided into multiclass and multi-label
categories [17, 18]. Input elements for classifiers include HTML tags, images, text,
URLs (Uniform Resource Locators), or hyperlink structures [19–21], with URLs being
the least costly and most informative.
2.4 Web Scrapping
Also known as Screen Scraping, Web Data Extraction, and Web Harvesting [2], it’s a
technique used to gather information from web pages using automated tools that visit
lists of websites to seek and store data, analyzing web pages through coded programs
[22, 23]. This technique finds application in various fields such as stock markets, weather
data monitoring, price comparison, research, detecting changes on web pages, etc. This
technique involves two stages: acquiring web sources and extracting relevant data from
these sources [2].
3 Related Works
With the growth of web information, dealing with such a volume of data can become
a tedious task. Therefore, nowadays, there is a need for various proposed methods and
tools that facilitate data exploitation. Web page classification has been divided into topic
and functionality by Choi and Yao [24] and Qi and Davison [25]. Various strategies found
in the literature for web page classification included the topic model [26], ontologies
with decision trees [16], classical machine learning models like k-NN and SVM [27,
28] using link weights, multiclass label models [29], CNN [30], transformer networks
[31, 32], among others.
The extraction and structuring of information from text for different styles of web
pages based on a multi-function extraction method was addressed by [33]. The method
combined features of the text from web pages to convert them into a DOM tree, following
these steps: erasing irrelevant tags, extracting tags from containers, segmenting title and
subtitle tags at each level to count the most frequent words, calculating distance support
and text support degree, and the path distance between container tags. The results showed
that the accuracy of web page extraction from single and multiple texts was higher than
conventional methods.
By employing a probabilistic topic model from a collection of web pages, [26]
applied the classic unsupervised probabilistic model known as Latent Dirichlet Allo-
cation (LDA). This allowed for dimensionality reduction and grouping of semantically
related terms into the same dimension. SVM was then used for classification and pre-
dicting new examples. Using SVM algorithms, some studies employed a combination
of content and context based on HTML tags, which aided SVM algorithms in achiev-
ing better results [34]. Others employed hyperlinks functions where the combination
improved classification performance [35]. Employing ontologies, improved C4.5 deci-
sion trees, and Naïve Bayes classifiers, [16] categorized web pages into groups according
to decision rules. This approach involved preprocessing, semantic extraction, and clas-
sification. In [27], latent semantic indexing was used to extract features from web pages,
followed by a combination of k-NN and SVM models for web page classification. To
utilize link weights concerning web pages, [28] introduced a link-based kernel inspired
by the Gaussian kernel for SVM web classification. The implicit link-based distance
was calculated using the textual context of a web page and user intuitive judgment. The
results indicated that the distance calculation technique yielded better results when the
parameter controlling the shape of the peak was set to 1.
To classify web pages into the categories of gaming or online video streaming,
[30] developed a web page classification model using a CNN. Data preprocessing was
conducted to remove noise information, and then the web page context was extracted
from the title and body content of the web page to obtain features. Through a word cloud
image, the most frequent words were displayed, and the classification model analyzed
the word pattern to predict the web page’s category.
In addition to metadata, internal links, and web page text, URL features were used for
web page classification. Keywords from URLs were utilized to label web pages, applying
all-grams for classification [36, 37]. To address the challenge of utilizing URLs, [14]
applied URL patterns and the UPCA algorithm. A binary web page classifier based
on URLs included a linear SVM and a method to learn a category-specific universal
dictionary of discriminatory URL features [38]. The web page classification proposed
by [29] encompassed a single-label multiclass model that involved extracting features
from a URL using Word Sense Disambiguation (WSD) along with NLP, employing
YAGO for data extraction from Wikipedia.
As most ML methods for text classification predominantly require substantial com-
putational resources, in recent years, NLP work has focused on the use of transformer
networks that are built upon attention mechanisms [31]. The utilization of multi-labels
based on a hybrid BERT-CNN model to classify web pages in various languages was
proposed by [16], utilizing text extracted from the body, title, and meta description tags
of the web page. Gupta and Bhatia [32] employed an ensemble model for web page
classification, applying a BERT model to contextual representations and using them as
inputs to a deep residual starting model for web page classification.
This work differs from the cited literature by conducting a broader analysis of
machine learning and deep learning techniques, with a total of 12 models. Furthermore,
it focuses on the classification of web pages related to master’s programs, a context that
has not been encountered in the literature.
This research contributes by providing a tool that enables the identification of web
links of master’s program pages, serving as support for actions carried out by academic
units managing postgraduate programs and for personal interest.
4 Build and Analysis of the Detection Model
This section presents the analysis of models and the selection of the most efficient one
for detecting web pages of master’s programs. Figure 1 illustrates the methodology used,
which outlines the steps that were followed.
Fig. 1. Methodology for model construction and analysis.
4.1 Manual Elaboration of Link Pool
Considering that the purpose of the research work is to build a tool that enables the
identification of web pages providing information about master’s programs, a list of links
to web pages from universities offering these studies was compiled. English-speaking
universities were chosen to be included, as they offer a broader range of fields of study.
A total of 100 representative universities were considered for the creation of the link
pool. Table 1 presents a sample of the university list.
Table 1. Sampling of the university list for the link pool.
University Country
Massachusetts Institute of Technology (MIT) United States
California Institute of Technology (Caltech) United States
University of Oxford United Kingdom
ETH Zurich (Swiss Federal Institute of Technology) Switzerland
University of Cambridge United Kingdom
4.2 Web Scrapping to Extract Labels from Tags of Interest

Once the link pool of master’s program web pages was built, the information collection
from these pages was conducted using the web scraping technique through Scrapy.
Scrapy is a high-level framework used for web crawling and extracting structured data
from web pages. Additionally, it provides a structure for writing spiders, which are
programs that navigate web pages and extract information.
A Scrapy spider was configured to follow links on the master’s program web pages,
collecting HTML hyperlinks. Once the hyperlinks were gathered, the HTML content of
each of them was collected, and Scrapy’s selection functions were utilized to seek specific
tags containing relevant information, such as the names of the master’s programs. The
Beautiful Soup library was used to analyze and extract information from the master’s
program web pages. Firstly, using Python’s requests library, the HTML content of the
master’s program web pages was obtained. Then, Beautiful Soup was applied to parse
the HTML content and search for specific tags containing relevant information like
the program name, duration, cost, and admission requirements. This information was
located using Beautiful Soup’s find_all() function to search for all tags with a certain CSS
class or ID. Beautiful Soup’s search functions allowed for nested tag searches within the
previously found tags. Once the tags with the relevant information were identified, the
content of the tags was obtained using Beautiful Soup’s access functions and stored in
variables. Subsequently, a database of master’s programs offered by different universities
was created using these variables. It was deemed necessary to use other reliable web
pages to balance the dataset between terms of interest and those of no interest.
4.3 Building a Raw Dataset with Manual Labeling

To achieve a proper evaluation of the model, we decided to create two types of training
datasets that were categorized into:
• General domain: The dataset included information related to both master’s program
web pages and web pages with different content. A total of 1568 sentences were
generated along with their respective labels.
• Master’s programs domain: This dataset was created using information obtained
solely from master’s program web pages, generating 1167 sentences along with their
respective labels to recognize academic keywords such as master’s program names
when applying the web scraping technique.
Henceforth, we will refer to the corpus as a training set, comprised of words of

interest and non-interest.
4.4 Preprocessing of the Dataset

The data obtained from the pages contained HTML tags within the texts; unknown char-
acters, often belonging to an operation; symbols like accents, exclamation and question
marks, and punctuation marks; white spaces and line breaks. Noise-producing data was
removed for a better understanding of the model. The mentioned characters and spaces
were erased, data like numbers and dates were normalized, accents and diacritics were
replaced, and all uppercase letters were converted to lowercase.
A subword-level segmentation was employed, encoding and splitting words into two
tokens, symbolizing the lemma and morpheme of the word. The idea is to identify which
words consist of lemmas and morphemes. To achieve this, the Byte Pair Encoding (BPE)
algorithm was applied, which is a data compressor that replaces the most frequent pairs
of consecutive bytes with a non-existent byte in the data. This was utilized in model
experimentation, but due to not achieving optimal results, it was opted to exclude it.
Table 2 displays the original, preprocessed, and labeled content of the dataset.
Table 2. Sampling of the dataset content.
Original content Preprocessed content Label

How to Watch how watch 0
Stories from around the world stories around world 0
Why China is sending seeds into space why china sending seeds space 0
Send an SMS or MMS to send sms mms 0
US & Canada us canada 0
requirements for undergraduate study requirements undergraduate study 1
philosophy physics ma philosophy physics 1
veterinary sciences phd msc by veterinary sciences phd msc research 1
research
ACS: Digital Biology Msc acs digital biology msc 1
Applied Mathematics with Industrial applied mathematics industrial modelling 1
Modelling Msc msc
4.5 Development and Training of the Models

The solution model is defined based on a comparison made among different ML and
DL models used for natural language processing, as well as the use of subword and
word embeddings techniques. Among the DL models, recurrent networks and trans-
formers were chosen as they are more optimal for natural language treatment. Various
combinations of pre-trained embeddings were used for the models, including:
• Embeddings word2vec: These are pre-trained embeddings based on a skip-gram and

bag of words neural network architecture.
• Embeddings FastText: These are pre-trained embeddings based on a network
architecture combined with statistical counting methods.
Different models were trained primarily based on an RNN architecture:
• Model-1. An LSTM cell with 128 hidden neurons.
• Model-2. Multilayer LSTM network with a total of 3 LSTM layers, where each layer
has 128 hidden neurons.
• Model-3. Bidirectional recurrent network based on an LSTM cell, where each LSTM
layer has 128 hidden neurons.
• Model-4. Artificial neural network with a multilayer architecture, where the initial
layer is a bidirectional recurrent network based on an LSTM cell, and each LSTM
layer has 128 hidden neurons. The second layer consists of a multilayer network
composed of two LSTM layers.
• Model-5. A GRU cell with 128 hidden neurons.
• Model-6. Multilayer GRU network with 3 GRU layers, where each layer consists of
128 hidden neurons.
• Model-7. Bidirectional recurrent network based on a GRU cell, where each GRU
layer contains 128 hidden neurons.
• Model-8. Small Bidirectional Encoder Representations from Transformers (Small
BERT). It is a pre-trained BERT model with 4 hidden layers, 512 hidden neurons per
layer, and a total of 8 attention heads.
• Model-9. A Lite Bidirectional Encoder Representations from Transformers
(ALBERT). It is a pre-trained BERT model with 12 hidden layers, where each layer
has 768 hidden neurons, and a total of 12 attention heads.
• Model-10. A linear kernel SVM with a parameter C = 1.0, where γ (gamma) is
inversely proportional to the variance of the training data multiplied by the number
of features.
• Model-11. An ensemble ML model based on Random Forest (RF), which uses Gini
Impurity as a criterion for constructing decision trees, with a minimum split of 2
samples and a maximum split criterion of the square root of the number of features.
• Model-12. Multinomial Naive Bayes model without smoothing parameter.
Models 1 to 7 applied a dropout normalization technique with a probability of 0.5,
using the ADAM optimizer as the training method, and binary cross-entropy as the loss
function.
In models 8 and 9, the first layers consist of the BERT model with a dropout proba-
bility of 0.1, and the classification layer is a feed-forward neural network. The ADAM
optimizer was used, along with binary cross-entropy as the loss function.
In models 10 to 12, each token was represented using the Term Frequency – Inverse
Document Frequency (TF-IDF) technique. Each of these models is a binary classifier,
designed to determine the likelihood that the label-content represents a page within the
domain of master’s programs or a general domain.
4.6 Selection of the Best Model
This stage involved passing the HTML content from the collected tags to each of the
models to identify whether the webpage was related to a master’s program or not. Addi-
tional tests and validations were performed to ensure the accuracy and quality of the
collected information, and measures were implemented to avoid gathering duplicate or
irrelevant information. These additional validations included assessing the performance
of the best models on a production dataset, which means a dataset independent of the
training set.
To select the best model, it was necessary to train each of the models presented in the
previous section with the 2 types of embeddings. Table 3 shows the training results of
the models using the general domain dataset. Models 1 to 9 were configured with a batch
size of 64. Models 1 to 7 were trained for 15 epochs, while models 8 and 9 were trained
for 2 epochs. For the models using the master’s program domain, models 1 to 5 were
trained for 15 epochs, model-8 for 30 epochs, and model-9 for 11 epochs, respectively.
It is clearly evident that the best model, based on the F1-score, was StackRNN LSTM
(model-2) using FastText embedding. It’s worth noting that only these two datasets were
considered for training, as using one that doesn’t contain complete master’s program
information would result in metrics scores of zero.
In addition, to give greater reliability to the selection of the models, the McNemar’s
test [39] statistical significance was developed. McNemar’s test is based over paired
nominal data using a contigency table to compare the distribution of counts expected
under the null hypothesis. The null hypothesis is rejected when the probability that this
statistic is less than 0.05. This McNemar test was applied over the most representative
models within groups of models. This groups of models are the models based on recurrent
networks (model 1, 2, 3, 4, 5, 6 and 7), models based on transformers (model 8 and 9),
and models based on in machine learning models and NLP techniques (model 10, 11 and
12). To choose the most representative model of the based on recurrent networks group,
the model with the larger the f1-score metric was chosen. Therefore, the model 2 is most
representative of the models based on recurrent networks. The model 8 was chosen of
the based on transformers models group, the model 11 was chosen of the based on NLP
techniques group. The pvalue obtained when comparing the models 2 vs 8, 8 vs 11, 2
vs 11 was close to zero, rejecting the null hypothesis that the models are statistically
similar and have the same error rate. Also considering the best f1-score we conclude
that model 2 is the best of all.
4.7 Evaluation of the Best Model
The conducted experiments were carried out with the best models from each dataset
and were performed on pages from various domains using a production dataset. Conse-
quently, in order to validate the performance of the selected model, StackRNN LSTM
with FastText embeddings, two external datasets were constructed that were separate
from the training sets. These were built from two different web pages related to post-
graduate programs. In this way, information was taken from the program degree page of
UNC (The University of North Carolina at Charlotte) and from KTH (Kungliga Tekniska
Högskolan). Table 4 presents the results using the UNC and KTH datasets. These datasets
Table 3. Results with the general and master’s program datasets.
General Domain Dataset Master’s Program Dataset

Model Embedding F1-score Precision Recall F1-score Precision Recall
Model-1 Normal 89.74% 89.43% 90.14% 91.58% 91.33% 91.86%
Word2Vec 89.32% 89.00% 89.72% 90.41% 88.02% 92.96%
FastText 88.54% 89.87% 87.36 88.75% 93.91% 84.15%
Model-2 Normal 89.11% 89.30% 88.98% 91.61% 93.78% 89.54%
Word2Vec 90.45% 88.95% 92.08% 90.67% 88.49% 92.96%
FastText 90.51% 89.43% 91.67% 91.95% 90.69% 93.25%
Model-3 Normal 89.05% 87.04% 91.30% 90.57% 88.11% 93.25%
Word2Vec 89.19% 87.65% 90.93% 90.44% 87.83% 93.25%
FastText 89.93% 87.34% 92.87% 91.40% 88.63% 94.36%
Model-4 Normal 89.51% 88.01% 91.25% 84.77% 77.30% 93.83%
Word2Vec 88.45% 87.78% 89.31% 90.31% 89.94% 90.70%
FastText 89.63% 88.65% 90.83% 90.38% 88.21% 92.68%
Model-5 Normal 89.01% 87.39% 90.88% 91.61% 91.11% 92.14%
Word2Vec 89.71% 88.06% 91.62% 90.38% 88.44% 92.43%
FastText 90.12% 90.94% 89.35% 90.85% 89.62% 92.14%
Model-6 Normal 89.19% 87.65% 90.93% 90.78% 88.99% 92.72%
Word2Vec 89.05% 87.04% 91.30% 90.53% 89.55% 91.57%
FastText 90.05% 88.96% 91.30% 90.53% 88.48% 92.72%
Model-7 Normal 88.66% 87.06% 90.51% 91.38% 90.42% 92.43%
Word2Vec 88.66% 86.73% 90.88% 90.62% 88.95% 92.43%
FastText 89.98% 89.58% 90.51% 91.49% 90.85% 92.14%
Model-8 Normal 82.06% 90.31% 76.27% 80.89% 68.39% 100.00%
Model-9 Normal 68.84% 52.56% 100.00% 81.07% 68.39% 100.00%
Model-10 TF-IDF 86.50% 85.50% 88.00% 87.50% 87.00% 87.00%
Model-11 TF-IDF 88.00% 88.00% 88.00% 74.90% 79.00% 83.50%
Model-12 TF-IDF 75.50% 83.50% 76.00% 60.00% 68.50% 60.00%
are datasets obtained with a lower data quality than those presented in Sect. 4.6. In addi-
tion, the domain of these datasets is different from the domain of the training data.
Therefore, is expected to obtain a lower performance, however, this performance is also
acceptable.
Based on the results and experiments, it is established that the best model is a 3-layer
stack LSTM with pre-trained word embeddings using FastText, while the worst model
for the master’s dataset is A Little BERT (ALBERT) with F1-score results lower than
Table 4. Results using UNC and KHT datasets.
UNC dataset KHT dataset

F1-score Precision Recall F1-score Precision Recall
Master’s 72.08% 73.38% 70.83% 64.58% 87.32% 51.24%
program
domain
General 62.50% 57.55% 68.38% 63.39% 81.69% 51.79%
domain
82%, and for the general dataset, the worst model is Naive Bayes with values below
60%. Furthermore, this transformer model is outperformed by a classical ML model
such as SVM, where both datasets achieved an F1-score of 85%. It is established that
this transformer model does not generate suitable results due to its complex architecture,
which is not well-suited for the small datasets presented, and the Naive Bayes model
does not achieve good results because it is too simplistic for the dataset. As a result, with
such limited training data, this model experiences underfitting, presenting high bias with
low variance.
As an example, Fig. 2 illustrates the output of identifying relevant information from
web pages of master’s programs. In one case, the majority of the links in the page’s
content fulfill this criteria, while in the other case, they do not.
Fig. 2. Interface displaying the result of identifying master’s program web pages.
On the other hand, classical models like SVM still serve as a suitable baseline
model for small datasets. It’s worth mentioning that recurrent neural network models
like LSTM, due to their effective processing capability for natural language data, are the
best fit for this type of dataset. This is achieved by employing a stack of LSTM layers
along with pre-trained word embeddings to gain an advantage over the scratch model.
The difference between pre-trained word embeddings from FastText and Word2vec is
not significant enough to indicate that one is better than the other. However, the use of
pre-trained embeddings is essential for achieving better results.
5 Conclusions and Future Works

A DL model is useful as a discriminator and automator for identifying pages of interest
with an accuracy of over 90%, as demonstrated by the StackRNN LSTM model (model-2)
using FastText embeddings. Furthermore, the model demonstrated acceptable values on a
different dataset with a domain similar to the datasets used for training. The information
contained within a web page serves as representative data for a discriminator model.
Scraping tools and a well-suited set of selection techniques enable the construction of
datasets with high variability and data quality. The use of automated tools for selection,
collection, and classification of information enhances a person’s learning by allowing
greater focus on the most relevant points. In turn, these automated tools empower the
acquisition of relevant information within a shorter working time.
As future work, we want to categorize the content of each page by topics and extract
master program prices and durations using artificial intelligence techniques, specifically
collaborative topic models using DL, and named entity recognition for numbers.
References
1. Chen, Z., Liu, J., Zhai, H., Jiang, L., Cao, B.: Web page recognition algorithm based on link
analysis in theme search engine. In: 2012 Second International Conference on Cloud and
Green Computing, pp. 405–409. Xiangtan, Hunan, China (2012)
2. Mehak, S., Zafar, R., Aslam, S., Bhatti, S.M.: Exploiting filtering approach with web scrapping
for smart online shopping: penny wise: a wise tool for online shopping. In: 2019 2nd Inter-
national Conference on Computing, Mathematics and Engineering Technologies (ICoMET),
pp. 1–5. United States (2019)
3. Cao, F., Su, L.: Internet addiction among Chinese adolescents: prevalence and psychological
features. Child: Care Health Dev. 33(3), 275–281 (2007)
4. Khan, A., Baharudin, B., Lee, L.H., Khan, K.: A review of machine learning algorithms for
text-documents classification. J. Adv. Inf. Technol. 1(1), 4–20 (2010)
5. Safae, L., Habib, B.El., Abderrahim, T.: A review of machine learning algorithms for web page
classification. In: Colloquium in Information Science and Technology, CIST, 2018-October,
pp. 220–226. Marrakech, Morocco (2018)
6. Hssina, B., Merbouha, A., Ezzikouri, H., Erritali, M.: A comparative study of decision tree
ID3 and C4.5. Int. J. Adv. Comput. Sci. Appl. 4(2), 13–19 (2014)
7. Xu, S., Li, Y., Wang, Z.: Bayesian multinomial Naïve Bayes classifier to text classification.
Lecture Notes Electr. Eng. 448(15), 347–352 (2017)
8. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in
vector space (2013)
9. Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword
information. Trans. Assoc. Comput. Linguist. 5, 135–146 (2017)
10. Guo, J., Cheng, J., Cleland-Huang, J.: Semantically enhanced software traceability using
deep learning techniques. In: 2017 IEEE/ACM 39th International Conference on Software
Engineering (ICSE), pp. 3–14. Buenos Aires, Argentina (2017)
11. Choetkiertikul, M., Dam, H.K., Tran, T., Pham, T., Ghose, A., Menzies, T.: A deep learning
model for estimating story points. IEEE Trans. Softw. Eng. 45(7), 637–656 (2018)
12. Cho, K., Van Merriënboer, B., Bahdanau, D., Bengio, Y.: On the properties of neural machine
translation: Encoder-decoder approaches (2014)
13. Giuliari, F., Hasan, I., Cristani, M., Galasso, F.: Transformer networks for trajectory forecast-
ing. In: Proceedings - International Conference on Pattern Recognition, pp. 10335–10342.
Milan, Italy (2020)
14. Yang, Y., Zhang, L., Liu, G., Chen, E.: UPCA: An efficient URL-Pattern based algorithm for
accurate web page classification. In: 2015 12th International Conference on Fuzzy Systems
and Knowledge Discovery, FSKD 2015, pp. 1475–1480. Zhangjiajie, China (2015)
15. Käki, M.: Findex: search result categories help users when document ranking fails. In: Pro-
ceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 131–140.
Portland Oregon USA (2005)
16. Artene, C.G., Tibeica, M.N., Leon, F.: Using BERT for multi-label multi-language web
page classification. In: Proceedings - 2021 IEEE 17th International Conference on Intel-
ligent Computer Communication and Processing, ICCP 2021, pp. 307–312. Cluj-Napoca,
Romania (2021)
17. Kag, A., Jenila Livingston, L.M., Livingston Merlin, L.M., Agnel Livingston, L.G.X.: Mul-
ticlass single label model for web page classification. In: 2019 International Conference
on Recent Advances in Energy-Efficient Computing and Communication, ICRAECC 2019.
Nagercoil, India (2019)
18. Huang, W., You, H.: Web page classification algorithm based on semi-supervised sup-
port vector machine. In: Proceedings of 2018 2nd IEEE Advanced Information Manage-
ment, Communicates, Electronic and Automation Control Conference, IMCEC 2018, Imcec,
pp. 2144–2148. Shaanxi China (2018)
19. Kan, M.-Y., Thi, H.O.N.: Fast webpage classification using URL features. In: Proceedings
of the 14th ACM International Conference on Information and Knowledge Management,
pp. 325–326. Bremen, Germany (2005)
20. Tragha, A., et al.: Machine learning for web page classification: a survey. Int. J. Inf. Sci.
Technol. 3(5), 38–50 (2019)
21. Hashemi, M.: Web page classification: a survey of perspectives, gaps, and future directions.
Multimedia Tools Appl. 79(17–18), 11921–11945 (2020). https://doi.org/10.1007/s11042-
019-08373-8
22. Pillai, P., Amin, D.: Understanding the requirements of the Indian IT industry using web
scrapping. Procedia Comput. Sci. 172, 308–313 (2020)
23. Malik, S.K., Rizvi, S.: Information extraction using web usage mining, web scrapping and
semantic annotation. In: Proceedings - 2011 International Conference on Computational
Intelligence and Communication Systems. CICN 2011, pp. 465–469. Gwalior, India (2011)
24. Choi, B., Yao, Z.: Web page classification. Found. Adv. Data Mining 221–274 (2005)
25. Qi, X., Davison, B.D.: Web page classification: features and algorithms. ACM Comput. Surv.
(CSUR) 41(2), 1–31 (2009)
26. Gu, M., Zhu, F., Guo, Q., Gu, Y., Zhou, J., Qu, W.: Towards effective web page classification.
In: IEEE/ACM BESC 2016. In: Proceedings of 2016 International Conference on Behavioral,
Economic, Socio - Cultural Computing. Durham, NC USA (2017)
27. Shi, X., Zhao, Y., Dong, X.: Web page categorization based on k-NN and SVM hybrid pattern
recognition algorithm. In: Proceedings - 5th International Conference on Fuzzy Systems and
Knowledge Discovery, FSKD 2008, vol. 2, pp. 523–527. Shandong, China (2008)
28. Belmouhcine, A., Benkhalifa, M.: Implicit links based kernel to enrich Support Vector
Machine for web page classification. In: 2015 10th International Conference on Intelligent
Systems: Theories and Applications, SITA 2015, pp. 15–18. Rabat, Morocco (2015)
29. Modi, S.S., Jagtap, S.B.: Web page classification using WSD and YAGO and ontology. In:
Proceedings of the 3rd International Conference on Communication and Electronics Systems,
ICCES 2018, ICCES, pp. 887–891. Tamil Nadu, India (2018)
30. Apandi, S.H., Sallim, J., Mohamed, R., Madbouly, A.: Web page classification using con-
volutional neural network (CNN) towards eliminating internet addiction. In: Proceedings
- 2021 International Conference on Software Engineering and Computer Systems and
4th International Conference on Computational Science and Information Management.
ICSECS-ICOCSIM 2021, pp. 149–154. Pekan, Malaysia (2021)
31. Vaswani, A., et al.: Attention is all you need. Adv. Neural Inf. Process. Syst. 30 (2017)
32. Gupta, A., Bhatia, R.: Ensemble approach for web page classification. Multimedia Tools
Appl. 80 (2021)
33. Wang, C., Wei, P.: A novel web page text information extraction method. In: Proceedings
of 2019 IEEE 3rd Information Technology, Networking, Electronic and Automation Control
Conference, ITNEC 2019, ITNEC, pp. 2213–2218. Chengdu, China (2019)
34. Sun, A., Lim, E.-P., Ng, W.-K.: Web classification using support vector machine. In: Proceed-
ings of the 4th International Workshop on Web Information and Data Management, pp. 96–99.
Virginia, USA (2002)
35. Fang, R., Mikroyannidis, A., Theodoulidis, B.: A voting method for the classification of
web pages. In: 2006 IEEE/WIC/ACM International Conference on Web Intelligence and
Intelligent Agent Technology Workshops, pp. 610–613. Hong Kong, China (2006)
36. Rajalakshmi, R., Aravindan, C.: Web page classification using n-gram based URL features. In:
2013 Fifth International Conference on Advanced Computing (ICoAC), pp. 15–21. Chennai,
India (2013)
37. Abdallah, T.A., de La Iglesia, B.: URL-based web page classification: with n-gram language
models. In: Knowledge Discovery, Knowledge Engineering and Knowledge Management:
6th International Joint Conference, IC3K 2014, pp. 21–24. Rome, Italy (2014)
38. Rajalakshmi, R., Aravindan, C.: An effective and discriminative feature learning for url based
web page classification. In: Proceedings - 2018 IEEE International Conference on Systems,
Man, and Cybernetics, SMC 2018, pp. 1374–1379. Miyazaki, Japan (2019)
39. Everitt, B.S.: The analysis of contingency tables. Chapman and Hall/CRC (1977)
Methodological Actions for the Electronic
Configuration of the Educa Plus Platform:
Promoting Interactive Learning
Anabel Aguayza1 , Christian Chuquizala1 , Ibeth Rendón-Enriquez1,2 ,

Andrés Tirado-Espín1,3,5 , and Diego Almeida-Galárraga1,4,5(B)
1 Education in Experimental Sciences, Universidad Yachay Tech, San Miguel de
Urcuquí 100119, Ecuador
dalmeida@yachaytech.edu.ec
2 School of Chemical Sciences and Engineering, Universidad Yachay Tech, San Miguel de
3 School of Mathematical and Computational Sciences, Universidad Yachay Tech, San Miguel
de Urcuquí 100119, Ecuador
4 School of Biological Sciences and Engineering, Universidad Yachay Tech, San Miguel de
5 Av. de los Sarances y Av. los Pendoneros, Universidad de Otavalo, Otavalo, Ecuador
Abstract. This work implements the use of the “Educa Plus” platform to achieve
interactive and meaningful learning in the field of chemistry. The research is
developed through an analysis that combines quantitative and qualitative methods,
using a conceptual framework and research variables within the educational field.
These variables are examined through a study that includes a statistical analysis
with the aim of analyzing the factors involved in the pedagogical methodology. The
main purpose of this article is to present a concrete proposal: a manual of activities
that improves the teaching-learning process through the use of technology and the
promotion of teamwork. The intention is to provide educators with practical tools
that allow them to make the most of available technological resources and create
a collaborative learning environment.
Keywords: interactive learning · virtual platform · learning and teaching

optimization
1 Introduction
The traditional educational process teaches in a unidirectional way where the teacher is
in charge of the transmission of content and the student, who has a passive attitude, learns
it. Over the years, the teaching-learning process has been changing and seeks to awaken
certain qualities or abilities in the student to obtain knowledge more efficiently. Students
should have an interest in what they seek to learn, they should use their imagination, cre-
ativity, experiences and prior knowledge for meaningful learning. “Learning has evolved
from acquisition/accumulation of knowledge to a more complex one, where knowledge
https://doi.org/10.1007/978-981-99-7353-8_10
Methodological Actions for the Electronic Configuration 117
is built mediated in a social and cultural environment. This implies modifications in the
cognitive structures of the student” [1]. Therefore, the introduction of innovative and
interactive educational approaches is required to address this new paradigm.
The digitalization of education in the twenty-first century has led to a meaningful
learning approach that transcends mere memorization, leaving behind the traditional
scheme of vertical and unidirectional teaching that seems to isolate both teacher and
student, in a dynamic, interactive, and attractive learning [2]. Consequently, teaching
must be adapted to promote skills that allow the application of the knowledge acquired
through the analysis of real situations that involve decision making, observation and the
use of technology. At the Baccalaureate level, the subject of chemistry is fundamental,
students must acquire basic knowledge in this area to avoid difficulties when starting a
university career related to science. In addition, it is important to encourage scientific
thinking in students that will help them in the construction of critical thinking before any
scientific or social phenomenon they face. Dynamic learning, which combines theory
and practice, is a tool that can help students to be attracted to science, develop skills in
the academic field, be self-taught and competent in their social life, thus promoting a
comprehensive approach to education, where one of the ways to evaluate the efficiency of
education will be through extracurricular activities based on the use of the implemented
platform, thus validating its value as an effective and holistic educational approach, [3].
In teaching chemistry, one of the great challenges is understanding mathematical
concepts and solutions that are often used. One of the challenges for the teacher is that
the student can relate theoretical concepts with empirical applications of the environment,
which will allow the student to assimilate more meaningfully the concepts dictated.
This understanding can be achieved through three interrelated levels of thinking:
the macroscopic level, accessible through the senses and allowing us to approach the
real world. The submicroscopic level, more imaginative and accessible only through
thought, allows us to formulate theoretical explanations of phenomena. The symbolic
level, which allows us to express and represent phenomena in written form, both at the
macroscopic and submicroscopic level [1].
Chemistry is an experimental science that is usually seen as a matter of memory;
however, many concepts can be related to simple practical experiences. Connections can
be made between practice and theory through linguistic tools, chemical equations and
mathematical calculations integrated into a virtual platform.
1.1 Context of Education with Respect to ICTs
At present, a series of virtual platforms have been created that offer opportunities for free,
personalized learning with permanent access to teaching materials. All these advantages
have allowed teachers to use these tools and integrate them into the education system
so that there is a more didactic approach to learning and not as an isolated support [4].
Teachers face constant challenges to adapt to new teaching-learning processes, some of
these challenges involve the reorganization of curricula and pedagogical resources to
meet the needs of students. Currently, a new challenge has arisen to incorporate migrant
populations, indigenous communities, and people with limited resources into the public
education system.
118 A. Aguayza et al.
It is important to emphasize that in Ecuador, despite the fact that the national cur-
riculum emphasizes the use of Information and Communication Technologies (ICTs)
[5] to facilitate the teaching-learning process, a large part of the population has little or
no access to technological tools due to socioeconomic factors or the lack of implemen-
tation of these tools by educational institutions that is a consequence of the little teacher
training in the field of innovative methodologies. These limitations prevent access and
use of both current information and available tools by teachers and students. However,
the implementation of new methodologies such as the incorporation of virtual platforms
generate significant learning [6]. Anyone can make progressive use of technology to learn
about any subject, including science and thus progressively and personally improve their
learning.
The methodologies in the teaching-learning processes undergo a constant evolution,
there is a growing interest in using education approaches applying virtual platforms
with the aim of improving learning processes, expanding educational coverage, and
providing flexible alternatives of high quality [6]. The teaching model based on ICTs is
an approach characterized by being interactive, adaptable, and collaborative. The model
centers learning on the student, allows personalized teaching, without limits of place or
time, exploration and exchange of knowledge and multisensory experiences that finally
allow a more meaningful learning. “This teaching approach is based on a constructivist
pedagogical model that focuses on the student, with the aim of fostering autonomy and
knowledge acquisition through collaboration and information sharing” [7].
In our work, various theoretical analyses aimed at evaluating virtual interaction to
promote social learning are presented. In addition, based on the aforementioned and with
the purpose of solving the needs detected in the teaching-learning process of chemistry,
the use of a technological tool is proposed. Specifically, in chemistry the subject of “Elec-
tronic Configuration” implies that students have knowledge in relation to the periodic
table, chemical symbols, distribution and organization of the elements. Unfortunately,
they have been unacquired skills and concepts that prevent them from advancing in the
curricular unit. Based on the above, we look for methodologies and resources that allow
us to develop skills not only in the field of chemical sciences, but also tools that allow us
to take advantage of both intrinsic and extrinsic motivation and the development of criti-
cal and analytical thinking by students. The use of the “Educa plus” platform is proposed
as a strategy for learning and the implementation of a guide for its management.
The central idea is that interaction becomes the tool for acquiring knowledge. In
order to face technological challenges during the educational process and to meet the
specific pedagogical needs that teachers may have, through the text particularities and
appreciations of the cited authors will be explained. In addition, the interactive processes
generated on the platform also enable social learning, the construction of knowledge and
the generation of dialogue environments.
2 Methodology
The research is developed with a mixed approach in two sequences: the first of qualitative
type where the observation data, interview with teacher and bibliographic review are
implemented. The second sequence is a quantitative design where tests were used to
assess students’ knowledge. The bibliographic review allows to identify the indicators
of educational quality with the implementation of the virtual platform “Educa pus”.
The bibliographic analysis will allow us to define the main components in the
teaching-learning process that are: observation, analysis of participation and knowl-
edge of students. It is important that, in addition to the use of platforms, the teacher has
a good planning of the subject, a clear and specific methodology and strengthens the
interaction and communication with their students. For the statistical analysis, different
factors were taken into account that allowed us to select an appropriate procedure. Each
process has different variables that are described in Table 1.
Table 1. Research variables
Variable Dimension Indicators Measuring

techniques and
instruments
Dependent: Student Learning and Knowledge acquired Pre – test
Teaching-learning Academic Interviews with the
chemistry Performance teacher and students
Activity in class Open-ended
interaction chemistry questions
about chemical
concepts
Mastery of electronic Post – test
configuration concepts Interview with the
teacher and students
Possibilities for Accessibility to digital Student interviews
students to access resources
technology and the
internet
Use of “Educa Plus” Knowledge and skills Use of virtual
virtual platforms of students and platform strategies
teachers on virtual Resource, means,
platforms tasks, evaluation,
activities
Independent: Class environment and Distribution of Virtual Platform
Teaching through the time arrangement schedules and Deployment
implementation of didactics Methods
virtual platform Support and resources Support for Activities with the
for parents and teaching-learning with implementation of
teachers for the proper technological virtual platforms
use of ICTs methodologies
This study focuses on a broad range of dimensions and utilizes diverse indicators
to deeply comprehend how the implementation of virtual platforms impacts the chem-
istry teaching and learning process. The dependent variables center around academic
performance and student learning, whereas the independent variables encompass the
implementation and support of teaching through virtual platforms, considering both
technical and pedagogical aspects.
The population for this research were the students of the Millennium Educational
Unit YACHAY. I worked with 34 first-year high school students who come from a virtual
education in times of pandemic.
For the quantitative evaluation of the students, a pre-test of 10 questions was used
before implementing the virtual platform “Educa Plus” and a post-test after the didactic
strategy. The answers were measured with Likert scale: nothing, little, regular, enough,
and much. The students were also analyzed through classroom observations and group
activities that allowed us to have an overview of the students’ abilities and their progress
in learning based on the knowledge acquired.
On the other hand, the interview will be very useful to gather information from the
teacher in relation to her criteria and predisposition to the use of virtual platforms for
the teaching-learning process, as well as the reinforcement of the contents in chemistry.
3 Results and Discussion

Based on the data obtained, a large part of the students experienced a considerable
improvement in their learning process. They went from getting grades that tended to be
low (blue) to achieving much higher grades (red), as shown in Fig. 1.
Fig. 1. Comparison between the results obtained in the pre-test and in the post-test (Gades).
The notable increase in student grades reflects the effectiveness of the method imple-
mented in the learning process of students. To obtain a more detailed analysis and accu-
rate visualization of the results, it is recommended to see Fig. 2, where the data are
presented graphically, and the improvements achieved stand out. This illustration shows
the change generated for each student in the case study. The results show an inferior
performance, in spite of having implemented the indicated tool. This phenomenon can
be attributed to several factors. Among these, we can highlight the possible demotiva-
tion of the students, perhaps influenced by lower previous grades. Likewise, the level of
interest that students give to the subject and the amount of time they dedicated to their
own learning can also be considered as an influential factor in this situation.
Fig. 2. Results obtained in the pre-test and in the post-test, for each of the students.
3.1 Interpretation of the Interview
The teacher’s answers during the interview showed, on the one hand, the lack of interest
of the students in learning, as well as the lack of availability of resources or tools that
can be used to improve the teaching-learning process in the classroom.
1. What strategies implemented in chemistry teaching were or were not effective
during the pandemic?
Strategies such as project-based learning, home experiments and the phet virtual
platform were implemented. However, they were not entirely effective as the students
showed no interest.
The lack of interest in the virtual platform used may be due to its high complexity,
which is not very attractive to students.
2. Do you consider that the use of virtual platforms is effective for the learning of
chemistry by students?
Yes, it would be useful, although some students do not have access to the internet and
it is complicated to use it for their learning. However, I believe that if it is implemented
in a coordinated manner during class hours, new knowledge could be generated.
According to the teacher’s comments, a greater approach of students to ICTs would
be useful, however, this has complications, due to the students’ own interest and the
economic lack to acquire technological devices or hire internet.
3.2 Impediments to Results
At the time of carrying out this study, specifically in the analysis of results we can
visualize some impediments that limit the research and are:
3.2.1 Sample Selection

We could highlight the study carried out by Asencio and Ibarra (2022) where they
highlight the importance of adequately evidencing the sample that will be used in an
investigation. It is important to note that the sample can be considered as a subset of

the entire study population, where the acceptance or rejection of the research depends
on the quality of the selected sample. In this way, it is essential to make a clear sample
selection that guarantees the confidence and validity of the study.
In our research, the sample selected was a specific group of students proposed for the
case study, it should be considered that there were two courses of students in which we
worked with only one of the groups as we indicated. We also believe that this selection
may affect and limit the overall results to a broader scale. Within the analysis, a random
or representative selection of the population had to be followed, as this may generate
biases and particularities in the results. Therefore, this limitation must be taken into
account when interpreting the results of the research.
3.2.2 Longitudinal Effects

The second aspect to consider is longitudinal effects, which refer to the time needed to
plan a problem and the changes that can be generated in each period, according to Avello
et al. (2019). Doing a study in a specific time can generate limitations to the case study,
since there may be long-term variations, and this can affect the validity of the results. In
our study it is important to mention that there was a specific period for conducting the
research.
3.3 Discussion
In the rapidly evolving landscape of education, the role of teachers has been undergo-
ing transformative shifts with the integration of innovative methodologies, particularly
in the context of teaching and learning chemistry. This paradigm shift has catalyzed a
reevaluation of pedagogical praxis and teacher training methodologies, ultimately lead-
ing to more effective educational outcomes. Embracing new teaching approaches that
draw on philosophical theorems of chemistry offers educators an enriched foundation for
crafting engaging and enlightening learning experiences [3]. Through well-structured
teacher training, educators can empower themselves with the necessary skills to har-
ness the potential of virtual platforms, creating interactive and collaborative classroom
environments that foster active learning.
The introduction of digital tools has ignited renewed motivation and interest in the
realm of chemistry among students. These tools have proven to be instrumental in culti-
vating essential cognitive and technological skills. By employing virtual platforms and
interactive resources, educators can tap into the digital prowess of today’s generation,
leveraging multimedia elements, simulations, and dynamic content to elucidate complex
chemical concepts [7]. This approach not only enhances understanding but also kin-
dles curiosity and enthusiasm for the subject matter. Moreover, the integration of these
tools into the learning process nurtures crucial skills like critical thinking, problem-
solving, and digital literacy, which are indispensable for success in both academic and
professional endeavors.
Technology’s pivotal role in education is evident as it reshapes learning with engage-
ment, collaboration, and personalization. Moving from one-way teaching to interactive
platforms marks a monumental shift towards student-centered learning. Quantitative
results highlight tech’s positive impact on academics and student motivation. Table 2’s
evidence of virtual platforms in chemistry education further strengthens this method’s
foundation: interactive learning and educational excellence through virtual tools.
To enhance the efficacy of such methodologies, it is imperative to meticulously
establish the fundamental characteristics of the chosen virtual platform. This approach
prioritizes interactivity, user-friendliness, and compatibility, ensuring that technological
complexities do not impede the learning experience. Moreover, identifying the pivotal
components that contribute to the quality and enrichment of teaching and learning is
paramount. An interactive platform must not only facilitate engagement but also encap-
sulate a high standard of virtual learning environment, supporting multifaceted activities
that cater to diverse learning styles and preferences. Scholarly reinforcement and empir-
ical research serve as cornerstones for substantiating the significance of virtual envi-
ronments in knowledge acquisition and construction. By advancing an evidence-based
approach, educators are better equipped to guide their pedagogical choices, resulting in
effective and meaningful learning experiences that extend beyond the confines of the
classroom.
In the triumvirate of knowledge, teacher, and student, the role of the teacher becomes
increasingly pivotal. As educators embrace digital tools and virtual platforms, they
become facilitators of knowledge exploration, guiding students through the intricate
tapestry of chemical concepts. This transition necessitates a shift from traditional didac-
tic instruction to a more dynamic and participatory model. Teachers morph into cura-
tors of learning experiences, orchestrating collaborative projects, interactive discus-
sions, and problem-solving exercises that transcend the boundaries of textbooks. This
transformation redefines the teacher-student dynamic, fostering an environment of co-
creation and shared exploration, where the teacher’s expertise interweaves with students’
inquisitiveness and technological adeptness.
Table 2. Authors who contribute to interactive learning and virtual platforms.
Authors Contribution in the line of research

[1] Reflections on the role of teacher with new methodologies improving pedagogical
praxis in the teaching and learning of chemistry
[3] Teacher training based on philosophical theorems of chemistry
[7] Educational strategies for active learning in the area of chemistry interrelating the
trilogy of knowledge, teacher and student
[8] The knowledge of teachers to implement virtual platforms generating an interactive
and group class
[9] Digital tools generating motivation and interest in chemistry by developing
cognitive skills and technological skills
In summary, the amalgamation of technology with pedagogical innovation presents

a compelling narrative of transformation in education. This evolution goes beyond the
mere transmission of knowledge, encompassing the holistic development of students’
cognitive faculties, digital skills, and motivation to learn. Through the strategic utilization
of virtual platforms, educators are poised to cultivate a generation of learners who are
not just proficient in chemistry but are also equipped with critical skills for success in an
increasingly technology-driven world. This transition is underpinned by comprehensive
teacher training, the application of philosophical theorems, and the strategic interplay
between digital tools and active learning strategies. As educational frontiers continue to
expand, the trilogy of knowledge, teacher, and student remains a guiding constellation,
charting a course towards pedagogical excellence in the domain of chemistry and beyond.
4 Process of Implementing “Educa plus”

At the global level, the establishment of standards that allow certifying the quality of
projects based on interactive learning tools is sought. The incorporation of technology in
education implies reconsidering teaching methods and training in virtual environments
should be seen as useful tools that can be accessed to improve the acquisition of knowl-
edge. The interaction of students and teachers with technology leads to the formation of
virtual communities that are finally also communication channels.
This proposal involves the participation of teachers and students in order to facilitate
their learning and mastery of the online platform. On the platform, information will be
provided on symbols, names of compounds and their electronic configuration according
to their atomic number. The students’ previous knowledge, combined with the new
knowledge acquired, will contribute to the creation of a meaningful learning process.
The proposal aims to serve as a manual of activities that allows improving the teaching-
learning process of students, using technology as an interactive tool and encouraging
teamwork. All this will contribute to achieving meaningful learning by combining theory
with practice on the virtual platform “Educa Plus”. The objective of introducing this
platform as an educational resource is to stimulate students’ interest in chemistry and
facilitate dynamic learning, while promoting the integration of technology in education
by the educational community.
4.1 Components that Guide the Development of Activities
The main objective of the proposed teaching strategy is to facilitate the understanding
of the structure of an atom and the electronic configuration, encouraging the connection
between theoretical concepts and reality. This allows students to strengthen the acquired
knowledge and encourage interactivity in the learning process. Since Experimental Sci-
ences are based on facts, laws and principles, especially in the context of transmitting
more complex knowledge, the following essential components have been identified for
this educational approach:
• Interactive learning: the generation of creative and innovative structures that allow the
flow of connections between ideas is promoted. This facilitates stable and meaningful
learning, where students can relate and apply concepts in a practical and dynamic
way. The interactive approach encourages active student participation, exchange of
ideas and peer collaboration.
• Motivation and active learning: The importance of keeping students motivated and
engaged in the learning process is recognized. Rather than being limited to theory
and passive observation, students are given the opportunity to “learn by doing”. This
implies that students can manipulate and experiment with the concepts, which in turn
contributes to the development of stronger mental skills and procedures.
With these components in mind, the following proposal is put forward for practical
application in the classroom:
• Design interactive activities: activities should be designed that promote interac-
tion and dialogue among students. This may include debates, group discussions,
presentations, hands-on demonstrations, simulations, experiments and collaborative
projects. These activities will allow students to connect theoretical concepts with real-
life situations, thus strengthening their understanding and application of the subject
matter.
• Provide adequate resources and tools: It is essential to have resources and tools that
support interactive learning. This may include visual materials, simulation software,
physical models, laboratory equipment, internet access and other relevant resources.
By using these resources effectively, students will be able to explore and experiment
with the concepts, which will increase their active participation in the learning process.
• Stimulate curiosity and creativity: Curiosity and creativity should be fostered in stu-
dents, encouraging them to ask questions, seek answers for themselves and propose
innovative solutions. This can be achieved through open-ended challenges and prob-
lems, research activities, individual or group projects, and exploration of practical
applications of the concepts learned.
• Provide timely feedback: It is important to provide constant and constructive feed-
back to students. This allows them to assess their progress, identify areas for improve-
ment, and strengthen their understanding of concepts. Feedback can be provided by
both the teacher and peers, through reviewing assignments, evaluating projects, or
participating in group discussions.
By implementing these components and considering these guidelines, an active, par-
ticipatory, and meaningful learning environment will be promoted for students, allow-
ing them to acquire a deeper and more enduring understanding of atom structure and
electronic configuration.
4.2 Criteria According to the Education Curriculum
Proficiency with Performance Criteria:

“CN. Q.5.1.5. Observe and apply the quantum-mechanical model of matter in writing the
electron configuration of atoms considering electron duality, quantum numbers, types
of orbitals and Hund’s rule” [10].
Evaluation Criteria:
“CE. CN. Q.5.2. Analyzes the structure of the atom based on the comparison of the atomic
theories of Bohr (explains the spectra of the chemical elements), Democritus, Dalton,
Thompson and Rutherford and performs exercises of the electronic configuration from
the quantum-mechanical model of matter” [10]. The evaluation indicators, techniques

and instruments to interactive learning and virtual platforms could see in Table 3.
Table 3. Evaluation indicators, techniques and instruments to interactive learning.
Parameter Contribution of virtual platforms

Evaluation indicators I.CN. Q.5.2.1 Analyzes the structure of the atom by comparing
the atomic theories of Bohr (explains the spectra of the chemical
elements), Democritus, Dalton, Thompson and Rutherford, and
performs exercises of the electronic configuration from the
quantum-mechanical model of matter. (I.2) [10]
Techniques and instruments Resources
Computer or cellular phone and internet
Educa Plus virtual platform
Techniques
Observation of the criteria and knowledge of the students on
topics related to Electronic Configuration
Interaction of constructivist learning
Instruments:
Questions about “electronic configuration”
Evaluation on “electronic configuration”
Learning Objectives
• Value the contributions of chemistry based on critical and reflective reasoning creat-
ing responsibility and using technological means providing the theoretical with the
practical.
• Demonstrate critical scientific skills that allow the investigation and curiosity of
atomic models and chemical elements by applying those learned on the subject of
“Electronic Configuration” through the Educa Plus platform.
• Allow students to learn and understand “Electronic Configuration”.
4.3 Learning Activities and Use of the Virtual Platform
Anticipation
• Research how to use the Educa Plus platform.
• General indications on the use of Educa Plus “Electronic configuration”.
General Indications of the Platform
• Enter the electronic configuration of the virtual platform for interactive resources
“Educaplus” (Fig. 3).
Fig. 3. Educa plus virtual platform startup
• Select element to develop the electronic configuration, once selected click on

“electronic configuration” (Fig. 4).
Fig. 4. Electronic configuration, Bohr model
• Being on this page you can see the symbol of the element, the electronic cloud and
the levels in a precise order.
• As the order electron spin stipulated according to the theory is placed, electrons will
appear in the cloud (Fig. 5).
• Through this interactive platform you can consolidate theory with practice where you
can learn by playing with the elements, and identifying their symbols, also allows
Fig. 5. Checking the electronic configuration of the element
you to observe the number of electrons in the electron cloud density, which allows
students to consolidate their teaching and learning.
4.4 General Recommendations Before the Implementation of the Proposal
• Socialize aspects and criteria of the class: It is essential to establish clear and open
communication with students about the aspects and criteria that will govern the
dynamics of the class. This includes the presentation of the learning objectives, the
contents to be covered and the performance expectations. By socializing this infor-
mation, students will have a clear understanding of what is expected of them and will
be able to engage more effectively in the educational process.
• Establish agreements and commitments between students and teachers: To foster a
harmonious environment conducive to learning, it is important to establish agreements
and commitments on the part of both students and teacher. These agreements may
include aspects such as mutual respect, punctuality, active participation in class, and
responsibility for assigned tasks. By establishing these guidelines from the beginning,
an atmosphere of trust and collaboration is created that favors the teaching-learning
process.
• Explain the evaluation: It is essential to provide a clear and detailed explanation of
the evaluation criteria to be used in the class so that students can plan and organize
their study time effectively.
By following these general recommendations, a solid foundation is established for

the implementation of the proposal, promoting a favorable learning environment and
facilitating the active participation and commitment of students.
5 Conclusion
In the results of the research on the use of virtual platforms in the interactive teaching
of chemistry, coincidences are identified that support the need to adopt an education
that takes advantage of technological resources to create motivating, dynamic and com-
municative environments. According to the analysis of the theoretical foundation, it is
highlighted that platforms are an effective resource in education; therefore, it is important

to improve the skills of both teachers and students in the management of digital tools and
understand their relevance to effectively implement interactive learning, thus enhancing
knowledge and generating satisfaction in the process. The proposal is directed to what
was identified in the study, to the dependent variable: teaching learning of chemistry
and independent: teaching through the implementation of virtual platform, this allows a
practical application allowing to increase the confidence of students in their preparation
with the virtual platform.
As mentioned previously, the analysis of results supports the idea that the implemen-
tation of didactic strategies favors the teaching-learning process. The study revealed a
high acceptance by students, who confirmed its effectiveness. In addition, it would be
advisable to replicate the study in a larger sample to corroborate the data obtained in
this particular case. Finally, it is important to keep in mind that not all strategies have
the same acceptance, since each student has a unique learning style.
It is necessary to recognize and adapt education to the digital environment, taking
advantage of the tools available to build individuals prepared for the global era. This
implies encouraging dialogue and interactive learning, breaking with traditional schemes
and overcoming cultural barriers. Evidence shows that technology is effective and nec-
essary in the educational field. On the contrary, if the new challenges are not accepted,
the new generations could be affected, experiencing apathy and demotivation. In order to
achieve this objective, this article offers criteria to characterize teaching-learning, ensur-
ing that its implementation is adequate and that the results are evaluated effectively. This
allows a quality training alternative that encourages student competence. Therefore, it is
imperative to increase efforts to promote the application of virtual platforms in the field
of education, as this improves the quality of the learning process.
References
1. Ordaz, J., Mostue, B.: Los caminos hacia una enseñanza no tradicional de la química Actu-
alidades Investigativas en Educació. Instituto de Investigación en Educ. 18(2), 1–20 (2018).
https://doi.org/10.15517/aie.v18i2.33164
2. Veytia, M., Flores, L., Tapia, M.: Clase invertida para el desarrollo de la competencia: uso de
la tecnología en estudiantes de preparatoria. Revista Educación 44(1), 1–30 (2019). https://
doi.org/10.15517/revedu.v44i1.36961
3. Chamizo, J., García, J.: Una experiencia en la formación de docentes a partir de la historia
y la filosofía de la química. Revista Eureka 17, 160101–160117 (2019). https://doi.org/10.
25267/Rev_Eureka_ensen_divulg_cienc.2020.v17.i1.1601
4. Véliz, M., Gutiérrez, V.: Modelos de enseñanza sobre buenas prácticas docentes en las aulas
virtuales. Apertura 13(1), 150–165 (2021). Redalyc. https://www.redalyc.org/articulo.oa?id=
68869704010
5. Mercado, W., Guarnieri, G., Rodríguez, G.: Análisis y evaluación de procesos de interactividad
en entornos virtuales de aprendizaje. Trilogía Ciencia Tecnología Sociedad 11(20), 63–99
(2019). https://doi.org/10.22430/21457778.1213
6. Covarrubias, L.: Educación a distancia: transformación de los aprendizajes Telos 23, 150–158
(2021). https://doi.org/10.36390/telos231.12
7. Sandoval, M., Mandolesi, M., Cura, R.: Estrategias didácticas para la enseñanza de la química
en la educación superior. EDUC 16(1), 126–138. Scielo (2013). http://www.scielo.org.co/pdf/
eded/v16n1/v16n1a08.pdf. ISSN 0123–1294
8. Agustín, F., Chan-Te-Nez, A., Sánchez, J.: La dimensión tecnológica en el conocimiento

profesional docente: reperfilando el conocimiento didáctico del contenido de profesores uni-
versitarios. REXE. Revista de Estudios y Experiencias en Educ. 20(44), 53–72 (2021). https://
doi.org/10.21703/0718-5162.v20.n43.2021.004
9. Tuárez, M., Loor, I.: Herramientas digitales para la enseñanza creativa de química en el
aprendizaje significativo de los estudiantes. Revista científica 7(2), 1048–1063 (2021). https://
doi.org/10.23857/dc.v7i6.2380
10. Ministerio de Educación.: Currículo de EGB y BGU CIENCIAS NATURALES (2016).
https://educacion.gob.ec/wp-content/uploads/downloads/2016/03/CCNN_COMPLETO.pdf
11. Beltrán, J.: Estrategias de aprendizaje. In: Beltrán, E.J.,. Genovard, C., (eds.), Psicología de
la instrucción I. Variables y procesos básicos. Madrid: Síntesis (1996)
12. Bravo, L., Torruco, U., Martínez, M., Varela M.: La entrevista, recurso flexible y dinámico.
Investigación en Educación Médica 2(7), 162–167 (2013). https://www.redalyc.org/pdf/3497/
349733228009.pdf
13. Briceño, J., et al.: El aprendizaje de fenómenos electromagnéticos mediante una herramienta
interactiva. Educere: Revista Venezolana de Educ. 13(45), 501–507 (2009). http://ve.scielo.
org/pdf/edu/v13n45/art24.pdf
14. Cárdenas, S.: Dificultades de aprendizaje en química: caracterización y búsqueda de alterna-
tivas para superarlas. Ciência Educação (Bauru) 12(3), 333–346 (2006). https://doi.org/10.
1590/s1516-73132006000300007
15. Carrión, P., Franklin, A., Rivera, R., Ronald, R.: Uso de los tics en el aprendizaje de química
(2019). Redalyc.org. http://repositorio.ug.edu.ec/handle/redug/39603
16. García, L.: Educación a distancia y virtual: calidad, disrupción, aprendizajes adaptativo y
móvil. RIED. Revista Iberoamericana de Educación a Distancia 20 (2), 9–25 (2017)
17. Asencio, E., Ibarra, N.: Limitaciones en la escritura de artículos de investigación educativa.
Estudio con fines didácticos para mejorar la preparación de autores. Información, Cultura Y
Sociedad (47), 65–78 (2022). https://doi.org/10.34096/ics.i47.11370
18. Avello, R., Rodríguez, M., Rodríguez, P., Sosa, D., Companioni, B., Leandro, R.: Por qué
enunciar las limitaciones del estudio? MediSur 17(1), 10–12 (2019)
19. Hernández, R., Fernández, C., Baptista, M.: Metodología de la investigación.
McGrawHill (2014). https://academia.utp.edu.co/grupobasicoclinicayaplicadas/files/2013/
06/Metodolog%C3%ADa-de-la-Investigaci%C3%B3n.pdf
20. Jarero, M., Aparicio, E., Sosa, L.: Pruebas escritas como estrategia de evaluación de apren-
dizajes matemáticos. Un estudio de caso a nivel superior. Revista Latinoamericana de Inves-
tigación en Matemática Educativa, RELIME, 16(2), 213–243 (2013). https://www.redalyc.
org/pdf/335/33527851004.pdf
21. Macías, A., López, A., Ramírez, M.: Recursos educativos abiertos para la enseñanza de las
ciencias en ambientes de educación básica enriquecidos con tecnología educativa. Revista
Iberoamericana de Educ. 58(3), 1–18 (2012). https://doi.org/10.35362/rie5831431
22. Ministerio de Telecomunicaciones y de la Sociedad de la Información. Logros de la Rev-
olución Tecnológica en Ecuador, se destacan por el Día Nacional de las Telecomunica-
cione”s (2020). https://www.telecomunicaciones.gob.ec/logros-de-la-revoluciontecnologica-
en-ecuador-se-destacan-por-el-dia-nacional-de-las-telecomunicaciones-2/
23. ONU.: Resolución aprobada por la Asamblea General sobre la base del informe de la Segunda
Comisión (A/63/414 y Corr.1). 63/209. Año Internacional de la Química (2008). http://www.
un.org/es/events/chemistry2011/resolution.shtml
Problem-Based Learning Applied to Software
Engineering: An Experience Report of The
Software Residence
Maria Amelia Eliseo , Marcilyanne Moreira Gois(B) , Fábio Silva Lopes ,

and Ivan Carlos Alcantara de Oliveira
Laboratório de Estudos em Ambientes de Produção de Software (MackLEAPS)-Faculdade de

computação e Informática, Universidade Presbiteriana Mackenzie, São Paulo, Brasil
{mariaamelia.eliseo,marcillyanne.gois,fabio.lopes,
ivan.oliveira}@mackenzie.br
Abstract. This paper reports on the use of the Problem-Based Learning (PBL)
approach in project execution within the Software Residency Program (SRP),
conducted at the MackLEAPS laboratory at Mackenzie Presbyterian University.
We describe the main challenges related to problem-solving in student learning
regarding software development as the utilization of approaches applied in many
companies. We report the challenges encountered by both teachers and students
when adopting an API-First development approach in the early cohorts of the Soft-
ware Residency Program. Moreover, we discuss the configuration of the SRP and
the impacts of the solution adopted, along with how the restructuring of the knowl-
edge topics sequence and the new directions provide to students have improved the
learning process through the PBL methodology. From adjustments in the sequen-
tial arrangement of approached topics in modules of SRP, we enhance the student
engagement, the quality of delivered artifacts, reduce dropout of students, and
contributed to the effective development of soft skills.
Keywords: Problem-based learning · Software development · Software

residence
1 Introduction
Problem-Based Learning (PBL) is a pedagogical approach in which students learn
through the resolution of real problems. It is an approach that emphasizes active stu-
dent participation in the learning process, rather than direct presentation of facts and
concepts. Additionally, PBL can promote the development of Critical Thinking skills,
problem-solving abilities, and communication skills [1].
Critical Thinking is an essential skill that should be developed by students and is
highly demanded in the job market. According to [2] teaching critical thinking is one
of the most educational challenges. In this regard, educational proposals have been
discussed to find ways to include and effectively promote this skill in the curriculum.
According to [3, 4], Critical Thinking is a planned action for the improvement of global
https://doi.org/10.1007/978-981-99-7353-8_11
132 M. A. Eliseo et al.
education in 2050 and it is a core skill for 2030 [5]. Implementing efficient techniques
to develop critical thinking can lead to improvements in problem-solving abilities [2].
A systematic review presented in [6] reported that PBL has gained recognition as an
effective methodology for enhancing critical thinking in students. The authors discuss
the necessary adaptations required in the PBL model to improve the development of
critical thinking skills.
Research conducted in [4] shows an analysis of the effects of PBL on the development
of critical thinking skills and found it to be effective for this purpose. However, certain
characteristics need to be considered and combined with the PBL approach. Among
these characteristics, the level of student maturity is a factor that influences the effects
of acquiring critical thinking through PBL. One of the conclusions of this study is that
students in advanced stages such as the final of undergraduate studies, they have greater
positive effects.
The PBL approach was employed in the Software Residency Program at Mackenzie
Presbyterian University [7], where students from computer science courses (both on-
campus and online) are trained in software development, considering the principles
of Software Engineering and Human-Computer Interaction. Students are exposed to a
real-world computing problem and, working in teams, are required to present a solution
over the course of a semester. A professor is assigned to each workgroup, providing
information, educational resources, online meetings to showcase results, clarifications
of doubts, identification of obstacles hindering progress, and even offering feedback
to students to promote improvements and new directions in the search for problem
solutions. These principles align with topics addressed in agile methods.
The Software Residency Program focuses on providing students with real market
experience to prepare them for the demands of the job market by developing the skills
and competencies required by companies. The proposal includes up-to-date techniques,
methodologies, and behavioral skills that have gained attention from employers.
The application of PBL in the Software Residency Program yielded relevant results
in student learning, developing both hard and soft skills. However, some difficulties and
challenges were encountered in the teaching and learning process, which were overcome
through adjustments in the sequence of modules in the Software Residency Program,
leading to an improvement in student learning.
In this context, this experiential report describes the difficulties and challenges that
teachers and students encountered when adopting an API-First development approach in
the early cohorts of the Software Residency Program (SRP). It discusses and examines
the changes implemented in the delivery of the latest cohort, which began their activities
in the first semester of 2023, aiming to enhance student learning and engagement in the
program, as well as mitigate the challenges experienced by students at the beginning of
their residency.
This paper is divided into the following sections, in addition to this Introduc-
tion: Sect. 2 presents the theoretical background that supported this work, along with
related studies. Section 3 presents the methodology adopted in the SR program and
the improvements made. Section 4 discusses the achieved results and the procedures
adopted to continue the SR program. Finally, in Sect. 5, the conclusion and future work
are presented.
Problem-Based Learning Applied to Software Engineering 133
2 Theoretical Background
2.1 PBL
PBL (Problem-Based Learning) is an instructional development methodology aimed at
engaging students in their learning through problem-solving in real-world situations.
The instructional focus is on assisting students in developing the ability to solve these
problems. Consequently, the student takes on the role of the protagonist in the process.
PBL is an active and interactive process that serves to identify both what students know
and, more importantly, what they do not know. Learning revolves around problems
perceived as significant or relevant to the students. The intention is to enhance students’
self-learning abilities, cooperative and communication skills, professional competence,
familiarity with academic writing standards, as well as to cultivate critical thinking [10,
11]. In this process, the role of teachers is to present students with a set of problems, after
which student workgroups analyze the problem, conduct research, engage in discussions,
and produce explanations, solutions, or recommendations [8, 9].
The selection of the problem is crucial for learning. Problems should be chosen in
a manner that motivates students to seek a deeper understanding of concepts, require
students to make informed decisions and defend them, incorporate content objectives
in a way that connects them to prior knowledge, and possess a level of complexity that
ensures students collaborate to solve them. If the problem can be divided into stages,
the initial stages should be open-ended and engaging to capture students’ interest [1].
Reference [8] proposes a 9-step method for PBL problem design that is suitable and
comprehensive to ensure student learning in all aspects of content acquisition, problem-
solving skills, and self-directed learning. The initial steps constitute a front-end anal-
ysis of PBL and involve the following steps: Step 1: Establish goals and objectives;
Step 2: Conduct content/task analysis; Step 3: Analyze context specification. Step 4,
Select/Generate PBL problem, and Step 5, Conduct PBL problem affordance analysis,
correspond to problem analysis. Analyses of problem affordance and adjustment are in
Step 6, Correspondence analysis, and Step 7, Calibration processes. Step 8, Construct
reflection component, describes the design of the reflecting component. Finally, Step
9 examines the inter-supporting relationships among content, context, and connection
components, as well as researching, reasoning, and reflecting components. PBL is a
complex instructional method that demands thorough analysis and meticulous planning
to ensure that different involved components work in harmony to achieve the planned
objectives [8].
In PBL, teachers take on the role of facilitators and collaborate with students, encour-
aging them to express their knowledge. Students work in groups to identify the prob-
lem and seek the necessary knowledge to propose a solution. This strategy promotes
active student participation, aids in exploring the subject matter, stimulates critical
thinking, enhances organizational skills for problem-solving, fosters group skills and
communication ability, and encourages self-directed learning (Fig. 1).
Studies conducted in [10] reveal that PBL is effective in promoting critical thinking.
However, factors such as students’ maturity, instruments used, nationality, discipline,
group size, and experiment duration may influence the effectiveness of PBL on criti-
cal thinking skills. This author also emphasizes that critical thinking promotes logical
problem-solving, contributing to improvements in education and the job market.
Fig. 1. Roles of Teachers and Students in the PBL Approach. Based on [11]
2.2 API Design and API-First Approach
APIs serve as an intermediary layer that defines how computer systems communicate
through predefined rules and protocols. APIs provide connection, integration, the sharing
of data, and information between software systems [12].
There are several advantages to focusing on API development such as enable manage-
ment of services, facility of integrating new services in applications already developed
and allows for the reuse the API in different application scenarios [12].
An API establishes the communication between backend and frontend, as illustrated
in Fig. 2. The front end is a software layer in which the user can see and interact with.
Frontend includes the development of the graphical interface considering visual aspects
used by the end users. In general, frontend contains the client device and an application
software, for example a browser enabling back-end access [14].
On the other hand, Backend consists of a software layer that provides the functionality
to users but cannot be viewed for them. It is responsible for implementation of the
business logic, storing data and running the algorithms [12]. In general, Backend includes
servers, data storage, and APIs [13]. It is important to highlight that the API is part of
the Backend, which is also not visible to the user, however, not every backend is an API.
APIs can be used to communicate the frontend and backend of the system.
API development is a challenge and has gained greater attention in companies and
among developers. API-first is an approach that has been explored in software engi-
neering and it is adopted by many enterprises, in which the development of a software
product starts from the API specification before writing any code for both the backend
and frontend. This approach is very important, and many studies have been conducted
on the development of APIs first.
However, it should be highlighted that the learning process of this approach in the
academic environment, particularly for students in the early stages with little or no
knowledge of programming languages, poses a challenge and may entail difficulties in
understanding the problem and seeking solutions. Several studies have been conducted
Fig. 2. Communication between Front-end and backend through APIs.
to demonstrate these difficulties and challenges for programmers. Although it is not easy
to understand and develop, especially for beginner developers.
Members of the development team should understand the underlying functionality
and how applications work behind a system interface. Another challenge is API devel-
opment, for which developers require some prior knowledge due to the prerequisites
involved in its implementation. Additionally, often learning resources and documenta-
tion do not provide sufficient information for a comprehensive understanding of how to
design and create a reusable API.
The study conducted in [15] presented some obstacles faced in learning how to use
and design APIs from the perspective of Microsoft developers. The author conducted a
survey to gather insights about the API experiences of 83 developers at different stages,
from novices to senior professionals. The study identified several factors that make
learning APIs difficult. Many respondents mentioned that one of the obstacles is the API
documentation and the lack of knowledge about the high-level resources behind API
design and which API is most suitable for a given scenario. Some respondents reported
that the main difficulties were their previous experience and background related to APIs,
as well as insufficient and inadequate examples available in the documentation, where
issues were not well specified.
According to [16], designing an API involves making many decisions at different
levels, such as the overall architecture of the API, design patterns, and even the names
of classes. These decisions can impact the usability of the API. Additionally, security
considerations should also be considered when designing APIs. A Systematic Literature
Review about the development and the validation of security in APIs recommendation
has been presented in [17]. This research presented the many approaches for the secure
developments of APIs.
In [18] insights are presented about the experience and knowledge required for API
design. The authors interviewed 24 professionals from 7 companies. The professional’s
reported that they learned API design in work, and it was lot approached in school. They
also highlighted the challenges in identifying the most important use cases.
An overview of API-first design in Academia and Industry is presented in [19]. The
study focuses on the concepts and importance of developing software with an API-first
approach. The authors discuss topics related to software architecture, design patterns, as
well as functionalities and security in APIs. Additionally, the study addresses the tools,
resources, and documentation available for API development. According to the author,
the purpose of this survey is to showcase the advancements related to API-first, as it is
still an emerging methodology, and to highlight the potential of the approach and the
importance of continuing studies and research in the field.
2.3 Software Residency
A software residency program is conceptually like medical residency programs, aiming

to train professionals in a specific area or activity, preparing them for the job market. In
this context, reference [20] presents experience in software residency, addressing both
software product line and version control processes. The experiment was conducted with
master’s students who were tasked with solving a problem for a real company, related
to software product line and version management. The objective was to determine if the
knowledge gained from the residency process was sufficient for a group of students to
solve a real-world problem. The residency implementation followed the following steps:
environment setup (corporate scenario; product features), training, residency execution
(students perform tasks), testing and validation (checking if residents could solve similar
problems). The results showed that the residency program was effective and enabled
the groups to solve the identified problems. Analyzing the results individually, it was
observed that no group failed to address the issue of creating a new distribution (or new
installation).
Reference [21] presents the implementation process of a Software Residency Pro-
gram for the undergraduate Computer Science course, which started in 2010. This pro-
gram provides an environment for undergraduate students to experience real software
development, following standards and rules established by the software industry. The
program consists of six stages: partnership with a Human Resources company with
operations in selecting and recruiting professionals for the software industry; selection
of a group of 20 Computer Science students for the residency program; definition of
the software “Pilot Project” to be developed by the students; training process; software
development; and student participation in the development of a real project for a soft-
ware development company. The results showed an increase in students’ maturity in
software development, improvement in their interpersonal relationships, and strength-
ening of their theoretical aspects. The authors also emphasize the importance of bridging
the gap between academia and the job market.
The reference [22] discusses the OKIoT project, which aims to provide project-
oriented education in both short-term and long-term modes, based on Software Engi-
neering methodology for Internet of Things (IoT) solutions. In relation to long-term
projects, a Software Residency Program was implemented for the MBA course. This
program consisted of the following steps: User-centric Value Proposition; Software Engi-
neering Modelling Method; iterative development; documentation of results and anal-
ysis. The authors report that one of the contributions of the software residency to the
OKIoT Project was the study of an open architecture for a smart speaker that combines
basic services from different vendors with minimized response time. They also suggest
Problem-Based Learning (PBL) as an alternative to the black-box approach for future
OKIoT implementations.
Unlike references [20] and [22], this article presents an experience report on a Soft-
ware Residency Program for undergraduate students. While reference [21] presents a
program for undergraduate students, it is limited to students in the Computer Science
course. The Software Residency Program presented in this article faces the challenge of
accommodating students from different undergraduate programs in Computing, such as
Data Science, Information Systems, Systems Analysis and Development, and Computer
Science, at different stages of learning and in different teaching modalities (face-to-
face and online). This peculiarity with different expertise and levels of knowledge leads
us to reflect on the adequacy of the program structure to provide a real experience to
students and promote the dissemination of concepts related to quality, production pro-
cess, and project management in software development, considering aspects of Software
Engineering and Human-Computer Interaction.
3 Methodology
The Software Residency Program has a duration of two semesters, with a new cohort of
twenty students from computer science and informatics courses starting each semester.
These students may be at different stages of learning and can include both on-campus and
online students. To encompass software development activities, the program is divided
into five modules that cover topics relevant to software engineering as well as those
addressed in the market. The left side of Fig. 3 illustrates the five modules proposed at
the beginning of the Software Residency Program [7]:
1. AMPG (2 months): This module focuses on project management skills development
and understanding the problem to be addressed through requirements analysis.
2. DEV (4 months): In this module, students develop the software architecture and API
based on the requirements.
3. T&Q (ongoing throughout modules): This module ensures software quality through
continuous testing, and it permeates the other modules.
4. UXD (4 months): The UXD module is dedicated to the development of user interfaces,
user experiences and usability evaluations.
5. DevOps: This module encompasses the implementation and deployment of the
necessary infrastructure for the functioning of the software under development.
Following the Problem-Based Learning (PBL) approach, students were immersed in
real-world problems, often in collaboration with business organizations. Upon entering
the Software Residency Program, students are organized into squads, which are groups
composed of up to five students, to work on specific or ongoing software projects. Each
squad has a Product Owner, who is a professor from the Faculty of Computing and
Informatics that guides the students in their activities in each module. The develop-
ment activities for each module are divided into sprints, and at the end of each sprint, the
squads are required to deliver a portion of the product, following agile software develop-
ment methods. From the modules structure, initially the students analyzed the problem,
defined system requirements and managed agile projects. Content topics were covered,
and digital educational resources were made available to guide students in utilizing sup-
port tools and techniques. The subsequent module focused on system architecture and
backend development. During the backend development phase, specific attention was
given to Application Programmable Interfaces (APIs) development. This was the initial
proposal of the Software Residency Program.
However, it was observed that this step is one of the biggest challenges, especially
for students in the early stages of their undergraduate studies, as it requires knowledge
of various technologies, and developers need to understand the problem at a higher level
of abstraction. To develop a suitable API with good usability and efficiency, some prior
knowledge is necessary. For example:
• Backend and frontend development
• RESTful (Representational State Transfer) principles
• Web services
• Client-server communication
• HTTP protocols
• Data formats (JSON or XML)
• Programming
Therefore, novice programmers may face many difficulties in understanding how
to develop a suitable API. In the context of the Software Residency Program, it is
important to highlight that starting software development from modules that involve
a higher level of abstraction makes the learning process more challenging, especially
when development teams consist of students at different stages of learning, with different
expertise and in different course modalities: face-to-face and online.
After recognizing the difficulties encountered in previous student cohorts of the
Software Residency Program, which resulted in disinterest and apathy, it was decided to
alter the module sequence, bringing the UXD module to be addressed before the DEV
module, as shown in Fig. 3. Thus, after defining the requirements, the squads in the new
cohort began analyzing user interaction for the development of system interfaces. This
inversion resulted in an increase in student engagement in the proposed activities, as
they were able to create prototypes of the interfaces and test them with users, promoting
confidence among the students regarding the proposed solution.
The new module structure proposal was implemented in the new cohort of students
who started their residency in the first semester of 2023, totaling 20 students. The students
were divided themselves into four squads, each consisting of five members. A specific
problem was assigned to each squad, and each problem was presented to the students
in the form of a real-life scenario. Table 1 presents the scenario for each problem along
with their respective specifications. Based on the discussion and understanding of the
scenario, each squad undertook activities from two training modules: Agile Methods,
Project Management (AMPG), and User Experience Design (UXD) Module.
Initial Proposal New sequence modules proposal

Fig. 3. Modules of the Software Residency Program.
Table 1. Description of the real scenario of each specific project [23].
Groups Scenarios Problem Description of the real scenario

Squad A Weather Station A system to collect data from a meteorological
station located in (omitted for double-blind
reviewing), which includes sensors to measure
temperature, humidity, barometric pressure, wind
direction, speed, and cameras to show the sky
Squad B Evaluation Website - Oriente Assessment system for Final Term Paper, which
involves poster and paper submission; indication of
members for Review Committee by the advisor;
description of reviews; and evaluation of works on
topics such as introduction, methodology, results, and
conclusions. In addition, indication of date, time, and
available rooms for face-to-face assessments
Squad C People Counter System System that analyzes the flow of people in open
spaces and which times of greater and lesser flow of
people. In addition, situations of evacuation of spaces
must be analyzed to ensure that they are unoccupied
Squad D Laboratory website Development of a website for the research laboratory,
dynamic generated, and a system for its content
register with restricted access, containing all the
research, projects, publications, partnerships, and
members of the laboratory
The squads aimed to gain work experience by seeking solutions to real problems.
The activities were carried out in mixed groups (regardless of the academic semester or
teaching modality) to develop teamwork skills. A professor, referred to as the Product
Owner (PO), was assigned to each group to assist and guide the students in their activities
[23].
4 Results and Discussions
Despite the API-first practice often adopted by companies, for many developers it is a
challenge since it is a change of software development culture. In this context, such dif-
ficulties are also real for teaching in the academy, mainly due to the levels of abstraction
and for students who start learning programming.
It is observed that there are many challenges in API-first software development,
despite it being a trend. Many research papers describe difficulties and the number of
requirements and concepts necessary for developing a proper and secure API. In this
sense, to improve the experience and engagement of students, there has been a reversal
in the sequence of software residency modules. The UXD module, in which prototypes
are created and interface evaluations are subsequently conducted.
The adjustments implemented in the Software Residency Program, particularly the
inversion of the DEV and UXD modules, starting with project management practices,
requirements, and UXD, allowed mentors (professors) to observe improvements in three
aspects of the program: Dropout Reduction, Engagement, and Quality of the produced
artifacts. Regarding the reduction in dropout rates, in the last sprint of the program,
we started with 20 students, and by the end, the group consisted of 15 students. This
remaining number was higher compared to previous sprints, with a dropout rate exceed-
ing 50%. When we asked the students about their continued participation in the program,
they reported a strong alignment between the implemented practices and their expecta-
tions for the program. Furthermore, the requirements practices complemented the content
covered in their undergraduate courses, while the UX practices were additional, partic-
ularly for the Computer Science curriculum. As for the students in Information Systems
and Analysis and System Development, they were more familiar with the theory, but
they had not previously had the opportunity to execute an end-to-end project like the
one conducted in the program.
Among the four squads that started, dropouts were more prevalent in only one of
the groups. Our understanding of this point was that the project seemed to lack the
expected relevance for the group, meaning it was not as interesting as the others. Since
the project selection was random, this may have contributed to the dropouts. However,
all four projects achieved the expected outcomes.
We observed that the responses regarding dropout also supported and reinforced
the perception of engagement. The fact that participants were more involved in the
project promoted attendance at the program’s checkpoint meetings, as well as teamwork.
Initially, all members participated together in the activities. However, the proposed tasks
required task division. The groups had to divide the tasks and consider individual skills,
especially those of the leaders who naturally emerged among the members of each squad.
Quality was the third highlighted aspect. As the squads had time to produce UXD
artifacts, the development process was enhanced with the creation of wireframes, naviga-
tion, definition of visualization patterns, and ultimately, heuristic analysis and usability
testing. These activities resulted in a significant improvement in the quality of the pro-
duced artifacts. Additionally, the students recognized the importance of understanding
user interaction needs for interface development and realized the potential of heuristic
evaluations and usability testing to identify design issues in the interface. Although the
time spent was greater than initially planned, we understand that these activities con-
tributed to the solidification of applied concepts, as well as increased engagement and
reduced dropout rates.
(a) (b)
(c)
(d)
Fig. 4. Mosaic of prototypes developed by each squad for each of the scenarios using Figma [23].
The reports presented in the sprint review raised concerns about the quality of the
developed software products being compromised due to a lack of project time needed to
mature the understanding of stakeholders’ needs. There was also concern about the time
required to generate and validate interfaces that better meet users’ interaction needs and
ensure good usability. The results of the interfaces prototypes are shown in Fig. 4. The
prototypes mosaic illustrates the prototypes created by the Squads presented in Table 1.
5 Conclusions and Future Works

This article presented an experiential report from the Software Residency Program at
the (omitted for double-blind reviewing), using the Problem-Based Learning (PBL)
approach to train students in software development based on the concepts of Software
Engineering and Human-Computer Interaction. The experience with the first cohorts of
students made us reconsider the structure of the Software Residency Program to align
with the learning needs of the diverse students involved, enhance their engagement in
activities, reduce dropout rates, and improve the quality of the produced artifacts.
The inversion of modules, starting with project management practices, require-
ments, and UXD, provided a stronger alignment of practices with student expectations.
Although dropout occurred in one of the groups, all four projects achieved the expected
outcomes. Engagement was driven by participant involvement in the project, promoting
attendance and teamwork. The quality of artifacts improved due to the dedicated time
for wireframe production, visualization pattern definition, and usability testing. While it
required more time, these activities contributed to the consolidation of applied concepts
and reduced dropout rates. Final reports highlighted the concern for software product
quality, indicating the need for more time to understand stakeholder needs and develop
interfaces that align with user interaction and expectations.
With the UXD module in the early stages of the Software Residency Program, it is
expected that students will have a more consistent and confident understanding of the
problem, better preparing them for the next DEV module. As future work, it is proposed
to revise the API-First approach to align with the diverse learning needs of students in
the development of upcoming modules.
Acknowledgements. This work was supported by MackPesquisa – Fundo Mackenzie de Pesquisa

e Inovação (Mackenzie Reasearch and Innovation Fund). Project nº 221053 MACK – 0008916.
References
1. Duch, B.J., Groh, S.E., Allen, D.E.: The Power of Problem-Based Learning: a Practical How
to for Teaching Undergraduate Courses in any Discipline. Stylus Publishing, LLC (2001)
2. Dwyer, C., Hogan, M.J., Stewart, I.: The promotion of critical thinking skills through argu-
ment mapping. In: Horvath, C.P., Forte, J.M., (Eds.) Critical Thinking (1 ed.). Nova Science
Publishers (2011). https://www.researchgate.net/publication/259286333
3. International Commission on the Futures of Education Commission. Reimagining our futures
together: a new social contract for education. UNESCO (2021). https://unesdoc.unesco.org/
ark:/48223/pf0000379707
4. Liu, Y., Pásztor, A.: Effects of problem-based learning instructional intervention on critical
thinking in higher education: a meta-analysis. Think. Skills Creativity 45, 101069 (2022)
5. OECD. The future of education and skills: Education 2030 (2018). https://www.oecd.org/edu
cation/2030/E2030%20Position%20Paper%20(05.04.2018).pdf
6. Yu, L., Zin, Z.M.: The critical thinking-oriented adaptations of problem-based learning
models: a systematic review. Front. Educ. 8, 1139987 (2023)
7. Lopes, F.S., Eliseo, M.A.: Software residency practices as a complement to the teaching-
learning process in software engineering: an experience report. In: 2022 XVII Latin American
Conference on Learning Technologies (LACLO), Armenia, Colombia, pp. 1–6 (2022). https://
doi.org/10.1109/LACLO56648.2022.10013402
8. Hung, W.: All PBL starts here: the problem. Interdiscipl. J. Prob. Based Learn. 10(2) (2016).
https://doi.org/10.7771/1541-5015.1604
9. Farinazzo Martins, V., De la Higuera Amato, C.A., Frango Silveira, I., Amelia Eliseo, M.:
Problem-based learning applied to the development of accessible serious games. In: 2019 14th
Iberian Conference on Information Systems and Technologies (CISTI), Coimbra, Portugal,
pp. 1–6 (2019). https://doi.org/10.23919/CISTI.2019.8760652
10. UDEN, Lorna. Technology and problem-based learning. IGI Globa(2006)
11. Li, F., Wang, M., Nan, X., Liu, X., Zhou, C., Gong, D.: Exploration of problem-based learning
at inter-course level. In: Proceedings of the 13th International Conference on Education Tech-
nology and Computers (ICETC ‘21), pp. 309–313. Association for Computing Machinery,
New York, NY (2022). https://doi.org/10.1145/3498765.3498813
12. Biehl, M.: RESTful Api Design, vol. 3. API-University Press (2016)
13. Aguado, D.: A Practical Approach to Cloud IaaS with IBM SoftLayer: Presentations Guide.
IBM Redbooks, Estados Unidos (2016)
14. Roger, S.P., Bruce, R.M.: Software Engineering: a Practitioner’s Approach. McGraw-Hill
Education. 8th edition (2014)
15. Robillard, M.P.: What makes APIs hard to learn? answers from developers. IEEE Softw.
26(6), 27–34 (2009). https://doi.org/10.1109/MS.2009.193
16. Myers, B.A., Stylos, J.: Improving API usability. Commun. ACM 59(6), 62–69 (2016). https://
doi.org/10.1145/2896587
17. Patnaik, N., Dwyer, A., Hallett, J., Rashid, A.: 2023. SLR: from saltzer and schroeder to
2021…47 years of research on the development and validation of security API recommen-
dations. ACM Trans. Softw. Eng. Methodol. 32(3), Article 60, 31 (2023). https://doi.org/10.
1145/3561383
18. Murphy, L., Kery, M.B., Alliyu, O., Macvean, A., Myers, B.A.: API designers in the field:
design practices and challenges for creating usable APIs. In: 2018 IEEE Symposium on
Visual Languages and Human-Centric Computing (VL/HCC), Lisbon, Portugal, pp. 249–258
(2018).https://doi.org/10.1109/VLHCC.2018.8506523
19. Beaulieu, N., Dascalu, S.M., Hand, E.: API-first design: a survey of the state of academia
and industry. In: Latifi, S. (eds.) ITNG 2022 19th International Conference on Information
Technology-New Generations. Advances in Intelligent Systems and Computing, vol. 1421.
Springer, Cham (2022). https://doi.org/10.1007/978-3-030-97652-1_10
20. L’Erario, A., Fabri, J.A., Gonçalves, J.A., Duarte, A.S.: Software residence application in the
versions of a software product line. J. Inf. Syst. Eng. Manag. 2(2), 11 (2017). https://doi.org/
10.20897/jisem.201711
21. Begosso, L.R., Begosso, L.C., de Souza Poletto, A.S.R., de Lima, F.C., da Cunha, D.S.: Work
in progress — software residency: a contribution for professional maturity. In: 2011 Frontiers
in Education Conference (FIE), Rapid City, SD, USA, pp. S4B-1-S4B-2 (2011). https://doi.
org/10.1109/FIE.2011.6142852
22. Hayashi, V.T., Garcia, V., de Andrade, R.M., Arakaki, R.: OKIoT open knowledge IoT project:
smart home case studies of short-term course and software residency capstone project. In
Proceedings of the 5th International Conference on Internet of Things, Big Data and Security
(IoTBDS2020), pp. 235–242. SCITEPRES (2020). https://doi.org/10.5220/000936600235
0242
23. Gois, M.M., Eliseo, M.A., Mascarenhas, R., de Oliveira, I.C.A., Lopes, F.S.: Evaluation rubric
based on Bloom taxonomy for assessment of students learning through educational resources.
In: EDULEARN23 Proceedings, pp. 7765–7774. IATED, Palma (2023). https://doi.org/10.
21125/edulearn.2023.2021
Gamification Strategies as Formative
Assessment Methods. A Systematic Review
Manuel Larrosa1(B) , Leandro Wives2 , and Virginia Rodés3

1 DINTER - Universidad de la República, Montevideo, Uruguay
manuel.larrosa@gmail.com
2 PGIE - Universidade Federal do Rio Grande do Sul, Porto Alegre, Brasil
3 Instituto del Futuro de la Educación - Tecnológico de Monterrey, Monterrey, México
Abstract. This article aims to deepen the analysis of gamification strategies as

a method to offer feedback in the form of formative assessment for elementary
school students (K-12). Gamification is defined as the set of strategies that involve
dynamics that recreate game experiences, with the purpose of transmitting spe-
cific content, and not only for the pleasure of playing itself. It is considered that
the psychological effects that playing generates are very beneficial, since playful
environments have great motivating potential and allow fluid exchanges through
active roles on the part of their participants. Despite the fact that academic produc-
tion regarding gamification strategies has been very important in the last decade,
studies do not discriminate between the different game elements that are part
of gamification, such as leaderboards, levels, rankings, etc. For this reason, this
review focuses specifically on one of them: the awarding of medals, to learn more
about its effects. Beyond its benefits, the incorporation of these practices also sup-
poses a great challenge, which is the need for evaluation that education systems
demand. In response to these topics, a systematic review of the literature cover-
ing the period 2011–2020 is proposed, specifically delving into the link between:
gamification strategies in formal education environments and the implementation
of formative assessment methods through gamification by badges.
Keywords: gamification · badges · formative assessment · motivation
1 Introduction
The constant interest from educators and researchers in the field of innovation in educa-
tion has led in recent years to a growing focus on game dynamics as a motivating mech-
anism appropriate to deliver concepts framed in diverse disciplines. Several strategies,
known as game-based learning, serious games, and gamification, have been developed
as practical tools for educational content [1]. This interest is accentuated by the place
that mobile devices, digital applications, and video games occupy in audiences of all
ages. Digital environments permeate all manifestations of recent human activity and,
therefore, also permeate gaming logic. Likewise, education is no stranger to digitization
and finds in it as much potential as challenges. At this intersection between educational
systems that must adapt on the move and the overwhelming ubiquity of digitization,
questions arise, while some of their answers can be found in gamification.
https://doi.org/10.1007/978-981-99-7353-8_12
146 M. Larrosa et al.
Definitions agree that it is the application of game logics to other contexts to invoke
the motivating and relaxation experience that playing provides [2, 3]. Different defini-
tions link the term gamification with digital environments, tying it to the development
and proliferation of video games and gaming [4].
The advantages it introduces seem evident, about which there is a remarkable con-
sensus and extensive bibliography. Among them, it can be mentioned the motivation and
involvement that generates amid participants; the wealth of practices that it offers for
the design of activities by teachers; the horizontal exchanges that it promotes between
teachers and students, which also encourage an active role from of all those immersed
in its dynamics. Despite this, when its application is analysed within formal educational
systems, a question emerges of how to evaluate the development of activities carried
out through gamified strategies. Even deeper, when it comes to delivering educational
content, it is important to know how could be assessed students’ appropriation. As [5]
argue, many studies consider gamification as a uniform concept when in practice it takes
many different forms, with heterogeneous designs and environments, since gamification
can manifest itself in various ways and combine multiple game elements. Therefore, its
results are also very diverse and are directly tied to the elements used in each of the
strategies designed.
The objective of this article is to carry out a systematic review of the literature to
identify the game elements that can be part of a gamified environment. In turn, it seeks to
deepen the possibilities that gamification offers as a formative assessment mechanism,
as feedback to the activities carried out inside and outside the classroom. Two large
areas and their possible link are then considered. On the one hand, gamification strate-
gies themselves, which are made up of various game elements such as badges, missions,
points, levels, leaderboards, and within these, specifically the badges´ earning as a moti-
vating mechanism to encourage student involvement or engagement. On the other hand,
what is considered formative assessment, i.e., the feedback process students receive
about their school performance. At the intersection of these two fields, the awarding of
badges can be seen as a method of formative assessment for primary school students
(K-12), as an element of gamification with multiple purposes: promoting motivation,
facilitating game dynamics for teachers, offering attractive and enriching returns for
students which channel educational content and focus on curricular content. Reference
[6] defines feedback as the information provided by an agent (teacher, peer, book, father,
mother, experience, oneself), regarding individual performance or understanding.
2 Systematic Study Procedure

To evaluate the potential of this approach, a systematic review is proposed, aimed at
finding gaps in the literature. The two large areas mentioned (gamification and formative
assessment) have been widely studied but independently. The hypothesis that this study
raises is the relatively low presence of research that links the two areas and the potential
that they can offer together as a pedagogical practice.
Systematic studies are an effective research method to identify the state of the art in a
given topic by mapping and classifying the studies available in the specific literature [8].
In this way, it is possible to identify research opportunities based on objectives that may
Gamification Strategies as Formative Assessment Methods 147
still be generic, but whose purpose is precisely to find a focus in which to investigate
more deeply [9]. Therefore, the most important contribution of systematic studies is
the reliability in which they support conclusions by applying a method with exhaustive
rigor, capable of reducing the biases that may influence the investigation [10, 11] define
a series of steps to follow, which are considered for this study. They are described below:
A. Planning - at this stage, is recommended conducting a first exploratory investigation
to obtain a general skimming of the field of study. This overview allows evaluating the
real need to carry out the secondary study if extensive previous literature justifies the
systematic study. As a central point, at this point are defined the research questions
that guide all subsequent development.
B. Protocol - in this step are determined the methods that will be adopted throughout
the procedure to reduce the possibility of biases.
C. Conduction - finally, at this stage is applied the search protocol. In this last step, it is
suggested to use a series of selection criteria to systematize the process. Reference
[10] describes this stage as an iterative process in which search filters are applied to
refine the results.
3 Objective and Research Questions
The central objective guiding this systematic review is to map the academic production
in the fields of gamification strategies in educational settings and its link with formative
assessment mechanisms. The following research questions are posed:
• Within the studies on gamification: is there evidence that links reward systems through
badges with formative assessment?
• Are these studies directed towards formal education settings for students under 12?
• What are the main findings regarding formative assessment through gamification
strategies by the awarding of badges?
4 Selection Criteria
The literature about gamification has been very profuse in recent years, so it is necessary
to limit the period in which the analysis will focus. The same happens regarding formative
assessment, which has gained significant focus due to the development of skills that
cannot be evaluated on a summative basis. Therefore, according to the general panorama
obtained from a first exploratory mapping, the selection criteria are the following:
• studies published between 2011 and 2020;
• written in English;
• finished articles, no previews or short articles (6 or more pages), nor complete books,
theses or conferences;
• primary studies, not systematic reviews or mappings;
• studies that show the link between gamification through reward systems based on
badges and formative assessment;
• studies that focus on formal educational settings, at elementary education level (K-12).
5 Search Engines and Keywords
From a general survey of the topic and other systematic mappings [8, 9], it appears
that the most appropriate search engines for this study are ACM Digital Library, IEEE
Xplore, Scopus, and Science Direct.
For each of the two major topics on which the search will focus, the keywords and
their most common synonyms are:
Gamification, including its possible variants and terms linked to specific gamification
strategies - gamified [gamif*], game-based, badges, open recognition, rewards.
Formative assessment [assess*], evaluation [eval*], feedback.
Therefore, the search argument in each case must obey to the rules of operation of
search engines, since each one has its own specificities. The logical operator OR is used
within each group, while the logical operator AND is used to link both. In addition,
the NOT logical operator was used to eliminate unwanted themes or topics specific to
repository’s profile, which appear very frequently. Thus, the search argument for each
site is configured as follows:
ACM: (gamif* OR “game-based” OR “badges” OR “rewards”) AND (“feedback”
OR “K-12”) NOT (“higher education” OR “high school”) Other filters applied in ACM:
between 2011 and 2020, only journals, only research articles, PDF, published by ACM.
IEEE: gamif* OR “game-based” OR “badges” OR “rewards” AND assess* OR
“feedback” OR “K-12” NOT “higher education” NOT “high school” NOT “software
engineering” NOT “computer science” NOT CS NOT engineering.
Scopus: (gamification AND feedback) OR (badges AND assessment) OR (rewards
AND open PRE/recognition) AND NOT (high PRE/ school OR higher PRE/ education)
[PRE/ - term to indicate that said word precedes the following].
Other filters used: only articles, no conferences, no short papers; by subject area -
social sciences, computer science, psychology, arts, and humanities; by language - only
in English.
Science Direct: (gamification OR “badges” OR “open recognition” OR “rewards”)
AND (assessment OR “feedback” OR “K-12” OR education).
In addition, other filters were established by the search engine to refine the search:
containing the keywords only in the title and abstract, only articles published in journals,
and only in PDF format.
6 Results
From the application of the search arguments in each repository, were obtained the results
shown in Table 1. After the results are returned by each search engine, an approximation
is carried out that analyses first the title; if it is related to the topic, we move on to the
analysis of the abstract. If there are doubts about its relevance to the topic, in-depth
reading is carried out to include it or discard it from the study.
It should be noted that gamification studies cover very varied areas of interest, since
the game logic approach is potentially applicable to any discipline and not only to the
area of education as proposed by this study. This is the reason why, when extracting
the results of the search argument, values present a significant difference compared to
Table 1. Results obtained from the application of search engine arguments.
Search engine Abstract Title Introduction Complete

ACM 479 14 5 3
IEEE 574 21 15 12
SCOPUS 280 28 11 9
Science Direct 236 10 5 4
when the title of each one is analysed. An initial conclusion is that the specific area of
interest of each one of the repositories profoundly permeates the content it publishes. For
example: in the case of ACM, a large part of its publications is dedicated to computing
studies, so most of its results are related with this topic. In the case of Scopus as well
as Science Direct, they include a greater number of publications linked to the social
sciences and education, fields more closely related to this review.
One aspect that immediately emerges is the evolution that gamification as a subject
of study has shown in the studied period. Beyond this systematic review, which due
to its scope, is not representative of the entire field of gamification, multiple authors
highlight the exponential increase in the topic in recent years [2, 12, 13]. The reasons
suggested by these authors link this increase to the psychological experience that playing
arouses through gamified practices, which through playful stimuli, seek to motivate those
involved to achieve specific objectives, modify their behaviour or develop new skills [14].
7 Discussion
Studies on gamification have been approached in very different ways, in some cases
analysing its potential as means to deliver other knowledge, in other cases emphasizing
its own characteristics as a pedagogical tool to promote student involvement, motivation
and provide specific achievement criteria. Therefore, it is necessary to clarify the field of
study since not all experiences that integrate game dynamics can be considered within
the field of gamification. In this sense, [15] distinguishes between what they consider
learning supported by games and gamification. In the first case, it involves environments
formally designed to channel specific concepts in a particular area and with a function
not directed by entertainment but by the content they transmit, known as serious games.
While in the second case, gamification focuses on using game elements in a combined
or isolated way, framed in educational experiences incorporated in real-life contexts [5].
Regarding the game elements considered within what is defined as gamification, the
literature has had an exponential production in the last decade, but in general, they have
been analysed jointly. Aligned with the core of studies on the subject, [15] mentions
among the main game elements that participate in gamified experiences: points, levels,
badges, leaderboards, rewards, progress bars, and narratives. However, as they point
out, not all experiences include all these elements but emphasis is placed on some of
them to make them more effective. In general, they affirm that, although the experiences
may integrate several of these, most use only 2 or 3, enough to improve the students’
learning results. Nevertheless, there is a real need to analyse some of them in isolation
to delve into their effects and psychological mechanisms. Specially in the phenomenon
of rewards based on badges, which have received particular attention and investment in
online education in recent years. Using the research questions posed as a guide, they are
answered one by one below:
Is There Research Linking Badge Reward Systems with
Formative Assessment? There is a broad consensus in defining gamification in educa-
tional environments as a series of practices that seek to adapt elements of video game
design to other areas of daily life [2, 4, 5], to encourage motivation and participation,
recognize and validate skills, and offer credentials indicating the knowledge acquired
[18]. Among the most commonly mentioned elements are the delivery of points, badges,
rewards, leaderboards, rankings, levels, campaigns, and progress bars, among others
[19]. However, as [17] mention, each of these elements separately has a different degree
of significance in the effects it generates, determining a specific motivational function.
For example, personalization promotes autonomy; badges aim to establish clear goals to
achieve; leaderboards encourage competition. Therefore, when making a more detailed
study of the game design elements grouped within what are considered gamified strate-
gies, it is necessary to analyse each separately to quantify their effects. In this case, the
focus on badges is because it is an element that has gained relevance as a subject of study
in recent years, due in part to its widespread presence in many studies, but above all due
to initiatives such as Mozilla Open Badges or Khan Academy, which positions them as
an alternative credential system to be used in the educational or work environment.
Reference [20] define badges as the representation of an achievement obtained in a
digital, visual format and available online, containing metadata that helps to contextual-
ize its meaning, the process, or the activity by which it was obtained. For the approach
pursued by this systematic review, the definition introduced by the author coincides with
the hypothesis that links badges in digital format, with the possible return that they trans-
mit. Another symbolically similar example, previous in time but that follows the same
purpose, are the badges used by Scouts, which in physical format have the same charac-
teristics described by the author: they offer members of the organization the possibility to
certify certain knowledge, which are generally not part of formal education; it makes this
recognition visible, which within the internal structure of the organization functions like a
CV, as a sample of their skills. One of the common features that both examples share (dig-
ital badges and physical badges) is the motivating characteristic, since they act as mile-
stones in a learning path, establishing specific goals to aim for, tracing a path in which
there are certain obstacles to go through, while at the same time making those achieve-
ments visible once they have been completed. Another critical feature they share is that
both examples are developed outside formal education settings, allowing for the recogni-
tion of skills that would not be certified otherwise. To summarise, reward systems based
on badges have the advantage of validating abilities not contemplated in traditional school
curricula and providing external links that extend the information the badge certifies. As
a weakness, in most cases, the validation of the skills developed is only recognized by the
organizations that grant them, but there is still no fully shared standard.
What is Referred to When Talking About Formative Assessment? Perhaps the best
way to explain it is in comparison to summative assessment. The first thing to mention
is that any form of evaluation is a way of contrasting student performance through

categories that allow progressively segmenting levels. While the summative evaluation
does it from a numerical scale or through standardized judgments, which seek to measure
the learning developed by a student and place it on that scale, formative assessment
emphasizes the conceptual content, focusing on generating feedback about the activity
carried out. Reference [6] mention the qualities an effective evaluation must have. First,
they mention that the evaluation must contain specific information about the task or
learning process. Its function is to be a frame of reference that the student can confirm,
contrast, and restructure. Therefore, formative assessment is a form of return (feedback)
that cannot be sustained in a vacuum, by the contrary, its potential is given by the learning
context on which it is built. According to the authors, formative assessment must answer
three questions: where is the student going? How is his process going? What are his
next steps? In this way, to be truly effective, it must be accompanied with the conceptual
content intended to be transmitted, trace objectives achieved, and future goals towards
which to go. On the contrary, summative assessment lacks conceptual or superficial
content since it only transmits the student’s location on the measurement scale through
a numerical value or a standardized judgment.
A widely cited study linking the psychological qualities that develop motivation with
formative assessment is that of [7]. The authors mention that one of the strengths of a
system that promotes formative assessment through badges is that it allows much more
freedom when facilitating student feedback. Especially if one takes into account that the
focus is on those skills not contemplated within the traditional curricular framework,
this is the case of the so-called 21st-century skills, social skills such as teamwork,
communication skills, skills to overcome frustration, entrepreneurship, creativity, among
others; that in general are not part of academic or professional certifications and therefore
are not easily validated, but that nonetheless are increasingly valued as elements to be
developed for the growth of the individual. As the authors suggests, the institutions
that award educational badges can validate through these any type of skill, knowledge
acquired, or completed achievement as a complementary recognition to the summative
evaluations that make up the official certifications. This way, badges can be viewed and
shared with others as evidence of the competence developed. For this reason, they can
be an element that influences and encourages involvement in educational activities and
therefore learning.
Are These Studies Directed Toward Formal Education Settings for K-12
Students? This is one of the main findings of this systematic review. Given the nature
of badge systems, which make it possible to recognize and validate skills that are not
necessarily part of formal curricula, they have been used mainly in non-formal educa-
tion settings. With the development of Virtual Learning Environments and educational
platforms that schools adopt due to increasing digitalization, their use has intensified.
However, since they are still considered as a complementary playful element, more as
a motivator than an evaluative tool or as feedback, their use is still marginal. There are
exceptions, such as Khan Academy, a widely used platform that does not escape the
logic of being an accessory tool [17].
Moreover, given that badges allow the skills developed to be added to a digital
portfolio, and that these become part of a sort of CV of a candidate, they find signifi-
cant potential among advanced students of secondary, technical, or tertiary education in
general, besides workers seeking to validate skills or be certified in fields that add pro-
fessional value. Therefore, most studies that focus on gamified environments that award
badges do so by analysing these qualities, mainly oriented towards adults, advanced
students, or professionally active people; but to a lesser extent, they are directed towards
students at elementary school level.
Even though its use is more widespread among young people and adults, studies
show that its potential as a solid tool for formative assessment makes it viable to be
adopted at any level. This fact is mainly reflected in the playful and visual qualities
that badges integrate, especially among school-age children. But their potential is not
only given by their visual characteristics, much further by the content they can deliver.
That is the case of badges that integrate information about the activity for which they
were awarded, skills obtained, number of hours dedicated, date, and institution. In the
same way, badges can channel for younger students, information related to the concepts
sought to be transmitted, linked to the school curriculum.
What are the Main Findings Regarding Formative Assessment Through Badges? It
should not be ignored that a reward system based on delivering educational badges is not
exempt from possible negative connotations. In this sense, the literature about intrinsic
and extrinsic motivation is forceful in its conclusions, and its results also apply to the case
of a badge system. As [17] argue, gamification in educational contexts is conceived as
an instrument to enhance motivation, understanding that this element is one of the most
determining factors in school performance. Therefore, the effect that a badge system has
on motivation is central to discussing its scope.
First, it is necessary to differentiate between intrinsic and extrinsic motivation. The
first refers to the inherent tendency of human beings to seek novelty and challenge,
to extend and exercise their own abilities, to explore and learn. The second refers to
the performance of an activity to obtain some result and therefore contrasts with an
intrinsic motivation that suggests doing an activity for the inherent satisfaction that the
activity itself causes [21]. To analyse the scope of motivation, [22] introduce the Self-
Determination Theory (SDT), with which there is a broad consensus on its usefulness in
explaining the phenomenon of motivation. The SDT exposes that the activities involving
feedback, communications, and rewards, which lead to feelings of competence as part
of their development, can amplify the intrinsic motivation in the process. Likewise,
appropriate challenges to its participants and positive feedback on performance also
enhance intrinsic motivation, while negative feedback has the opposite effect. In the
same way, what the authors call functional significance explains that a stimulus by itself
is not capable of determining motivation, but rather its motivational impact is mediated
by the functional significance that the individual gives to it [17]. Therefore, the way in
which a game element is capable of motivating is determined by how the user applies
it. In other words, there is no direct relationship between the game elements and the
motivational effect they generate. Each of these can trigger a different effect on the user,
depending on its meaning and how it is put into practice. This means that a particular game
element can have a certain effect (or none), and it depends directly on the individual’s
intrinsic motivation, which functions according to his psychological needs.
In this way, intrinsic motivation is manifested when an individual, for example,
practices a sport or plays a musical instrument, simply because he enjoys it and is satis-
fied. This is how he regulates himself by his own choice and interest. This intrinsically
motivated behaviour does not require any kind of reinforcement and is a prototypical
example of self-determination. The internalization of external motivators gives the oppo-
site extreme of self-regulation. These manifest when the motivation is directed towards a
benefit that does not come from the individual himself, but is external, such as receiving a
congratulation, a reward, or a gift for reaching a goal. The internalization process occurs
when a functional significance is given to an activity that is incapable of developing
intrinsic motivation, which is driven by external motivators that give it value, and in the
ideal case, can even promote self-regulation and self-determination [22].
In the same way, [23] reinforce the idea that, in educational contexts, different stu-
dents can react differently to the same gamification method. Even gamification can
produce a result contrary to what is expected when it reduces the internal motivation
towards the activity, replacing it with an external motivation. This happens when exter-
nal rewards are perceived as controlling and not informational or reinforcing something
that is intended to be emphasized. In this sense, the so-called achievement badges have
a function that occurs parallel to the game’s goals or activity. As [24] mention, they
introduce an optional goal independent of the game objectives. However, they have an
essential function, which is to visualize certain stages in progress in the activity and
offer feedback on the performance of the participant. The social function they introduce
is even as important as their feedback function since they allow these achievements to
be exposed to other participants or spectators.
Certain game elements are more effective than others when it comes to providing
feedback, such as badges, as the recognized element with the most weight in this regard
[5]. Badges are defined as visual representations of an achievement that can be obtained
or collected within the gamified environment. Its function is to make the achievement
visible, confirm and clarify it for the user, but more importantly, show the scope of this
goal to the other participants. In this sense, badges fulfil a double function: on the one
hand, they officiate as a return for the user that indicates that they are in the right direc-
tion, but at the same time, they socialize the objective achieved. It is worth asking, what
psychological need does each of the elements satisfy? In the case of badges, they offer
a visual return, show achievement, and are a way of evaluating the user’s performance.
This assessment in the form of feedback evokes the feeling of competence, by directly
communicating the success in the development of the activities by the player [5]. But we
must not lose sight of the fact that all the elements mentioned are external motivators,
and if they do not acquire a functional significance that positively internalizes them, they
can generate a negative motivating effect. As [17] mention, a classroom environment
that promotes autonomy can functionalize the teacher’s feedback as informational and
enriching, and therefore encourage the development of intrinsic motivation. But, on the
contrary, a controlling environment can frustrate autonomy and therefore foster a feel-
ing of extrinsic motivation, which is consistent with negative consequences. The authors
state a direct link between highly self-determined environments driven by intrinsic moti-
vation and positive educational outcomes, whereas, in the opposite case, environments
extrinsically motivated through control correlate with negative outcomes (Fig. 1).
Badge charach- User charach- Context chara-

teristics teristics chteristics
Learning
Motivation Behaviour
outcome
Fig. 1. Theoretical framework from Van Roy et al. 2019.
Among the characteristics inherent to the medals, beyond the environment to which
they are attached, the following are listed:
• They operate as a return of success and therefore feed the competition.

• They make the status of the participant visible and therefore feed the desire for
progress.
• Communicate the integration of a group or activity, belonging.
Reference [24] define badges as optional rewards and goals outside the objectives
that make up the core of an activity. This definition suggests that the optional character
accompanying the badge does not involve a controlling attitude, and therefore a negative
extrinsic motivation, which, although external, can be positively internalized. This aspect
is highlighted in the fact that obtaining or not medals does not affect the participant’s
progress in the activity, but rather, in such a case, they enhance or visualize it. Still, they
do not suppose a goal by itself. Instead, the goal is the activity or challenge that awards
the badge.
One of the most important conclusions derived from the study by [7] is that different
types of badges affect student motivation in different ways. In the case of low-performing
students, badges work as an incentive to participate in educational activities, but do not
significantly affect skills acquisition. Even more importantly, they can have a negative
effect on learning, since the study confirms that extrinsic motivators result in a certain
rejection of the activities they promote. Therefore, one of the recommendations for
instructional designers is to consider the capabilities of the public to which they are
directed, since, if badges do not channel appropriate content for a specific public, there
is a possibility of generating a motivational effect contrary to what is desired.
8 Conclusions
From the research questions, a clear gap in the literature regarding the link between
both topics emerges in the first instance. Although gamification strategies have been
addressed from different academic fields, in general, the focus of analysis and interest
in its application is given by its motivational qualities or by introducing methodologies

that promote active dynamics between teachers and students. However, the focus on
formative assessment has been outlined more superficially, and it is here where a future
study can emphasize this technic’s advantages.
A recurring criticism of reward systems is the questioning of their behaviourism
nature. In this sense, the studies by [7, 17, 24] agree that there is no direct determinism
between the function of the badge and its effect. Because many other elements come into
play, such as each player’s characteristics, context, and motivation toward the activity,
it is challenging to establish causality. Mainly due to the SDT, the motivating scope is
directly determined by how each one functionalize the stimuli [17] mentions it when
stating they “question the existence of a one-to-one relationship between some game
element and its motivational function”. It suggests that any game element can trigger
a motivational function (or not), depending on the meaning a user gives it and the
relationship established between it and their psychological needs.
In conclusion, it can be deduced that if there is no direct relationship between
the influence of the badges and the effect they generate, it is difficult to sustain the
behaviourist hypothesis, by which it is understood that the delivery of badges is an
external reinforcer and therefore its function is to change behaviour. Based on the argu-
ments presented, it is understood that the elements that come into play in motivation are
multiple, very complex, and that their effects do not always have a direct and unilateral
relationship with the function given to them. Furthermore, a strong two-way link between
the user and the different game elements draws up diverse relationships. Game elements
are linked to each other since they rarely exist in isolation; at the same time, they are
nourished by the psychological needs of the users and how each one functionalizes their
presence according to their motivation, obtaining a different effect on the game in each
case.
References
1. Terras, M.M., Boyle, E.A.: Integrating games as a means to develop e-learning: Insights from
a psychological perspective. Br. J. Edu. Technol. 50(3), 1049–1059 (2019)
2. Hamari, J., Koivisto, J., Sarsa, H.: Does gamification work? a literature review of empirical
studies on gamification. In: 2014 47th Hawaii international conference on system sciences,
pp. 3025–3034. IEEE (2014)
3. Garcia-Iruela, M., Hijón-Neira, R.: What perception do students have about the gamification
elements? IEEE Access 8, 134386–134392 (2020)
4. Deterding, S.: Situated motivational affordances of game elements: A conceptual model. In:
Gamification: using game design elements in non-gaming contexts, a workshop at CHI, vol.
10, no. 1979742.1979575 (2011)
5. Sailer, M., Hense, J.U., Mayr, S.K., Mandl, H.: How gamification motivates: an experimental
study of the effects of specific game design elements on psychological need satisfaction. Com-
put. Hum. Behavior 69, 371–380 (2017). https://doi.org/10.1016/j.chb.2016.12.033. Accessed
21 Aug 2023
6. Hattie, J., Timperley, H.: The power of feedback. Rev. Educ. Res. 77(1), 81–112 (2007).
Accessed 21 Aug 2023. http://www.jstor.org/stable/4624888
7. Abramovich, S., Schunn, C., Higashi, R.M.: Are badges useful in education? It depends upon
the type of badge and expertise of learner. Educ. Technol. Res. Dev. 61(2), 217–232 (2013).
https://www-jstor-org.proxy.timbo.org.uy/stable/24546520. Accessed 21 Aug 2023
8. Klock, A.C.T.: Mapeamentos e revisões sistemáticos da literatura: um guia teórico e prático.

Cadernos de Informática 10(1), 1–9 (2018)
9. Petersen, K., Vakkalanka, S., Kuzniarz, L.: Guidelines for conducting systematic mapping
studies in software engineering: an update. Inf. Softw. Technol. 64, 1–18 (2015)
10. Conforto, E.C., Amaral, D.C., da Silva, S.L.: Roteiro para revisão bibliográfica sistemática:
aplicação no desenvolvimento de produtos e gerenciamento de projetos. In: Resumos. Porto
Alegre, RS: Escola de Engenharia de São Carlos, Universidade de São Paulo (2011)
11. Klock, A.C.T., Gasparini, I., Pimenta, M.S., Hamari, J.: Tailored gamification: a review of
literature. Int. J. Hum. Comput. Stud. 144, 102495 (2020)
12. Hamari, J.: Do badges increase user activity? a field experiment on the effects of gamification.
Comput. Hum. Behav. 71, 469–478 (2017)
13. Hamari, J., Koivisto, J.: Social motivations to use gamification: an empirical study of gami-
fying exercise. an empirical study of gamifying exercise. In: ECIS 2013 - Proceedings of the
21st European Conference on Information Systems, vol. 105, no. 5, pp. 18–19 (2013)
14. Klock, A.C.T., da Cunha, L.F., Gasparini, I.: Um modelo conceitual para a gamificação de
Ambientes Virtuais de Aprendizagem. Renote 13(1) (2015)
15. Chu, M.-W., Fowler, T.A.: Gamification of formative feedback in language arts and mathe-
matics classrooms: application of the learning error and formative feedback (LEAFF) model.
Int. J. Game-Based Learn. (IJGBL) 10(1), 1–18 (2020)
16. Giessen, H.W.: Serious games effects: an overview. Procedia – Soc. Behav. Sci. 174, 2240–
2244 (2015). Accessed 21 Aug 2023. https://doi.org/10.1016/j.sbspro.2015.01.881
17. Van Roy, R., Deterding, S., Zaman, B.: Collecting Pokémon or receiving rewards? how people
functionalise badges in gamified online learning environments in the wild. Int. J. Hum.-
Comput. Stud. 127, 62–80 (2019). Accessed 21 Aug 2023. https://doi.org/10.1016/j.ijhcs.
2018.09.003
18. Glover, I.: Play as you learn: gamification as a technique for motivating learners. Edmedia+
Innovate Learning, 1999–2008 (2013)
19. Werbach, K., Hunter, D.: For the Win: How Game Thinking can Revolutionize your Business.
Wharton Digital Press (2012)
20. Gibson, D., Ostashewski, N., Flintoff, K., Grant, S., Knight, E.: Digital badges in education.
Educ. Inf. Technol. 20(2), 403–410 (2015)
21. Ryan, R., Deci, E.L.: La teoría de la autodeterminación y la facilitación de la motivación
intrínseca, el desarrollo social, y el bienestar. Am. Psychol. 55(1), 68–78 (2000)
22. Deci, E.L., Ryan, R.M.: Self-determination. Corsini Encycl. Psychol. 1–2 (2010)
23. Auvinen, T., Hakulinen, L., Malmi, L.: Increasing students’ awareness of their behavior in
online learning environments with visualizations and achievement badges. IEEE Trans. Learn.
Technol. 8(3), 261–273 (2015)
24. Hamari, J., Eranti, V.: Framework for designing and evaluating game achievements. In: Digra
Conference, vol. 10, p. 9966 (2011)
Gender Inclusion from the Art
for Children
Krystle Peñaherrera1 and Gabriel Barros2(B)

1
Universidad de Cuenca, Campus Yanuncay, 12 de Octubre,
010205 Cuenca, Ecuador
krystle.penaherrera@ucuenca.edu.ec
2
Tivo.ec Research Institute, Don Bosco 2-07, 010205 Cuenca, Ecuador
gbarrosg@tivo.ec
Abstract. Art presents a technological solution to talk about gender,

providing educational innovation through open educational resources.
This work presents an Instructional Methodological Class for teachers
of Basic Education, where they reflect on gender identity, from art, for
students of nine years of age, which promotes inclusion in the classroom.
The applied methodology is inspired by educational references such as
Paolo Freire and Luis Camnitzer. Three classes were designed based on
international artistic references, who discursively and technically expose
gender deconstruction stereotypes such as Sarah Lucas, Cindy Sherman,
and Yasumasa Morimura. In this case, educational innovation consists on
the creation of three hand-made prototypes, one in each session. Addi-
tionally, these resources are available for teachers on open repositories
through QR codes. By applying the pilot classes in two schools: one
public and one private, the results showed that this age group does not
use gender stereotypes negatively. This is an initial step to validate the
assessments through survey and self-assessment, to obtain real data for
which triangulation may be used in the future.
Keywords: Gender · Art · Children Education · Equality rights ·

prototyping
1 Introduction
It is known that there is gender inequality throughout the world, we can say that it
is due to social differences. According to ECLAC reports [20] one of the problems
is the overload of care given to women, which does not allow them to carry out
their activities with autonomy, which is understood by the capacity of people to
make free and informed decisions about their lives, to be and act according to their
wishes within the historical context that allows them [12]. In Latin America and
the Caribbean, there are 132 women living in extreme poverty for every 100 men
[13]. Inequality generates a great gap making women below men, all this according
to the UN sustainable development indicators. To improve this condition, in recent
years progress has been made in education; however, these have not been enough
https://doi.org/10.1007/978-981-99-7353-8_13
158 K. Peñaherrera and G. Barros
to narrow the gap, there are 15 million girls who will never learn to read and write
because they do not have access to education, they are out of school, compared
to 10 million boys who have managed to join school classrooms [13]. In Ecuador,
for example, there is also a large rate of gender inequality, especially at the school
level. According to official data from the study of violence between peers in the
educational system, there are victims of bullying [17] due, among other things,
to the increase in cases of gender violence among young people. Therefore, it is
essential to focus on incorporating the issue into the national education system to
ensure fairer and more impartial learning.
According to the study on violence called “An in-depth look at school bullying
in Ecuador”, presented by the Ministry of Education, 1 in 5 students between
the ages of 11 and 18 have been victims of violence or bullying [11]. Currently,
in Ecuador all citizens have the right to study in equality, inclusion and equity
within the classroom. It is established that the school classroom is considered a
space for the creation of identity, but what happens inside reflects what happens
outside, be it in family relationships, on the street, in the community, in the
country. Therefore, situations are produced and repeated among students related
to each of their previous experiences with the outside world, as well as to the
subjectivity of the group[15].
There are several laws that support rights, such as the Constitution of the
Republic based on principles and values such as freedom, equity, peace, equal-
ity, human dignity and that must be applied by law [2](Gender Equality in the
Ecuadorian Constitution, 2011). According to the National Institute of Statistics
and Censuses; In the province of Azuay, 68.8% of women have experienced some
form of violence; in Azuay, 4 out of 10 women have suffered physical violence and
3 out of 10 sexual violence (INEC, 2019). Considering that the problem of gender
violence is very high, and affects the development of girls and boys in the short
and long term, it is urgent to incorporate the issue at the level of public and
private education, since there are no institutions where children and girls learn
about gender equity. Therefore, the theme of this research designs an instructive
methodological class called: An approach to learning gender inclusion from cur-
rent art for students of Basic Media, which provides a pedagogical roadmap for
teachers within the subject of Cultural Education and Artistic, for 9-year-old
girls and boys, that promote inclusion from the classroom. It is important to
provide students with inputs to unlearn stereotypes and normalized gender roles
in society, which produce a social imbalance from childhood, and therefore it
is necessary to reflect on hegemonic violence, from the binary social consensus,
from the pedagogical possibilities to educate boys and girls, so that they learn
to live free, healthy and egalitarian childhoods and adolescences.
2 Related Works
2.1 Gender Roles
Gender roles are transmitted in various ways unconsciously and involuntarily,
since often the people with whom we have a relationship have acquired them in
Gender Inclusion from the Art for Children 159
the same way. For this reason, having an education that teaches critical thinking
is a very powerful element that can and should play a decisive role in avoiding
the inequalities caused by these stereotypes. Students learn about gender roles
through games, assignments, tasks from the classroom; and it is there where
there is a gap that assigns a social separation, the same one that affects the ade-
quate development of individuals from their early childhood, adolescence, adult
life, family and work development. Ramirez [14] indicate that generally when
asking male Educating about professions, men develop their studies and work in
scientific roles such as astronauts, doctors, scientists, physicists, etc. and women
in care roles such as nurses, teachers, physiotherapists, cooks, hairdressers, etc.
In the article, this divergence between the choices is attributed to the fact that
there are differences in self-perception of intelligence, the same ones that begin
to appear among students as early as five years of age, that is, before there is
no significant difference between children and girls who call themselves “very
smart.” However, from the age of six, girls were more reluctant to get involved
in games that were for the “highly intelligent”.
This is a very important finding for this study because it shows the scope
and impact of gender stereotypes.
2.2 Theoretical Pillars from Paulo Freire

In his book Pedagogy of the Oppressed, Freire emphasizes the recognition of the
state of oppression as a situation of violence and emphasizes the general strategy
of revolutionary work with the oppressed, making a new educational proposal
where” The educator is no longer just the one who educates, but the one who,
while educating, is educated through dialogue with the student, who, by being
educated, also educates ” [6].
Freire emphasizes the relationship between dialogue and transformation,
which expresses a type of communication to build a truth that no one pos-
sesses in its entirety, but whose benefits are to open a door to learning, thus
transforming the mental structure of the individual in order to generate critical
thinking, as a leading action in the classroom. Education makes sense because
people are projects, and at the same time we can have projects for the world.
Education makes sense because women and men should be considered as equal
beings. If women and men simply left, there would be no reason to talk about
education.
2.3 Theoretical Pillars from Luis Camnitzer

When defining the word “art” from traditional education, it is done with the
objective of how the student will function in the labor market, but not to develop
their creativity.
That educational system is designed to facilitate the work of the businessman,
who is based on the grades of the applicants and their achievements to determine
who is the most suitable to take the position, exalting the relations with the
elaboration of works of art from the technical part, instead of understanding
“art” as a means of communication and cultural transformation[9]. We are taught

to equate the word “education” with “learning” and, therefore, we are trained
for work, this, obviously, is a filter that ends up predisposing the student.
As a result, pedagogy should not enter into that game, but rather attack
this attempt to objectify the student. Both the production of “art” and the
production of “education” subject us to the labor market. Understanding the
role of “art” and “education” is to understand and change society.
Complex activity is integrated into this intersection, where we learn to con-
stantly learn, to question the established system, looking for alternative systems
of order, understanding the world as a system of complex configurations. That
is why art is a form of knowledge, and not a form of production. Under this
rule, arts education becomes essential to train active citizens capable of devel-
oping artistic skills[4]. In his book Art and Education[3], Camnitzer questions
that when teaching how to do things, the most important question of what to
do with them is often neglected.
He tells us that this approach failed to first identify the need for a message;
communication develops, without first establishing its need. He affirms that you
cannot first identify and explore the need to communicate, and then find an
appropriate way to carry out that communication. Knowledge becomes prede-
termined and closed when both definition and identification take place within
this restricted functional field, while a more open field would stimulate question-
ing and creation. But since institutional education in other fields is organized
solely for the transmission of known knowledge and the preservation of conven-
tional practices, he points to fundamentally opposed pedagogies.
This will be achieved when art is an instrument of knowledge, and education
is responsible for freeing students as unique individuals who develop their own
ideas.
3 Methods: Design and Organization

of the Methodological Class
The instructive methodological class deals with the teaching of the arts through
the subject Cultural and Artistic Education, which triggers the students to
become sensitized in many aspects, such as: sight, taste and critical sense [16].
To make it possible, the sensory artistic technique will be used, which is used
empirically with manual techniques that are used to sculpt, paint, draw, etc. [1].
The methodological class uses didactics as a teaching method, it includes teach-
ing and learning strategies. Which refers to the design of strategies that teachers
implement through the organization of classes [5]. This class will apply the three
transversal axes of the curriculum, the most important being the teacher’s guide
for the development of the class. Instructive is understood as all material that
is intended for teachers, and not for students directly. Therefore, it has two
objective beneficiaries: the first to receive the information will be the teachers in
charge of Cultural and Artistic Education, who will be in charge of applying the
class to the final beneficiaries, students from 9 to 10 years of age. Likewise, it is
important that the class be collaborative, it is intended to generate opportunities

to establish contacts between the students, that is, that they support each other
if necessary; these being the main axes of the educational and teaching process.
This class proposes from deconstruction, through the creation of new artistic
products without an established canon, using international artistic references,
which work on issues of gender inclusion.
The class name is “Be as I want to be”, the content to be taught requires
three sessions, each one will last 2 h, a method that is socialized in six parts:
survey, adaptation of the class, artistic references, practical exercises, evaluation
and self-assessment.
3.1 Survey
A survey for the students is proposed to find out their level of acceptance of the
topic to be discussed. They must answer the Yes - No questions’ below:
1. Can men take care of their children?

2. Can women be presidents?
3. Can boys play with dolls?
4. Can girls play with cars or planes?
5. Can girls be firefighters?
6. Can the boys help with the housework?
7. Can boys study dance?
3.2 Adaptation of the Class
The students are put in context, a space of collective security will be created
where the students are comfortable with the subject, to be treated to provide
security and stimulate learning. This setting will be carried out through audiovi-
sual resources (videos from the YouTube platform) with similar content in each
session, which will be described below for each class session:
Session 1-Knowing Our Bodies. Video 1 Title: I want to know, and

you? - Chapter. 2: “Be as I want to be” Argentina country Author: Pakapaka.
YouTube channel Year: 2022 Duration: 03m15s Description: Animated series
that addresses the contents of comprehensive sexual education for boys and girls
at the primary level. In this chapter, a student (man) paints his fingernails with
nail polish. Within his school, this action leads to debates and ridicule, because
this activity has been assigned to women according to their gender role. The
participation of an adult through dialogue with the group of students invites
them to reflect that the gender division must be deconstructed and respect indi-
vidual decisions [10]. Video 2 Title: Pink candy - children’s stories about gender
equality - international day of the woman. Country: Italy Author: Imagiland
kids. YouTube channel Year: 2019 Duration: 06m01s Description: In this story,
women are represented by female elephant calves, which are pink, because they
eat flowers. Only the females are kept inside a fence, where they are educated
from birth, telling them to be pink is the only way to be beautiful and get mar-
ried. On the other hand, men are represented by small gray elephants, who have
complete freedom to eat, walk and sleep wherever they want without worrying
about anything. The protagonist Margarita will not follow these guidelines and
will open the path of freedom and equality for her and the other elephants.
The story tells us about gender equality. This project visualizes gender violence
from the allocation of color psychology, which has naturalized in the divisions
of gender roles. The colors of the visual language must be deconstructed [18].
Session 2: Diverse Portrait. Video 3 Title: The UN video that wants to end
gender stereotypes. Country: England Author: The Huff Post. YouTube channel
Year: 2017 Duration: 01 m28 s Description: The video shows two newborn babies,
where the social expectation of each sex is presented. It proposes role changes
within the family, educational, work systems, etc. The “He For She” foundation,
which is part of the UN, fights for a world free of gender stereotypes. Allowing us
to reflect on assigned masculinities and femininities, letting students form their
own criteria as to what is correct and what is not [19].
Session 3-Gender Stereotypes. Video 4 Title: He for She - UN Women Cam-

paign Country: MEXICO Author: Women’s Institute for the State of Morelos.
YouTube channel Year: 2016 Duration: 01m11s Description: This video shows
how normalized activities that are often stereotyped as feminine or masculine,
seeks to show that people are capable regardless of gender assignment or stereo-
types, these only cause inequalities. Audiovisual project of the “He For She”
movement, who work in favor of gender equality, developed by UN Women to
involve men and boys as defenders [8].
Video 5 Title: Loeches Town Hall Country Spain Author: Loeches Town Hall
Year: 2017 Duration: 00m58s Description: The video presents the use of language
and how it harms both men and women. This example deals with sexist violence
defined as discriminatory against men and at the same time makes women lose
value by talking about them in a discriminatory way, which decreases their
confidence and self-esteem. Referring to people adequately is very important,
since the upbringing of students and their listening normalize violent, regularized
languages that affect their childhood and later their adulthood. Therefore, the
video promotes prevention and awareness against gender violence [7].
3.3 Artistic References

As a didactic resource, three international artistic references were chosen, who
raise stereotypes of gender deconstruction through photography, and performa-
tivity, representing other gender identities than those socially assigned by their
sexuality. With these referents, the contents of the class are designed, through
artistic discourse, it will seek to reflect on diversity as a natural act, as presented
in the following descriptions.
Sara Lucas. English artist (1962), over the last thirty years she has created a
distinctive and provocative body of work which subverts traditional notions of
gender, sexuality and identity. Since the late 1980 s, Lucas has transformed found
objects and everyday materials like cigarettes, vegetables, and socks into absurd
and conflicting paintings that boldly challenge social norms. The human body
and anthropomorphic forms are repeated, often appearing deformed, humorous,
fragmented or reconfigured. “Au Naturel” is the work that we present as a ref-
erence, the same one that replaces domestic furniture with parts of the human
body, evoking the possibility of a natural state without the limitations estab-
lished by society. Lucas, who has developed a deconstruction of the body in her
work, and from her the component of session 1 called Knowing Our Body is
inspired.
Cindy Sherman. American artist (1954), she studied at the University of Buf-
falo, there she began painting, but soon realized that she was not her thing; and
she acknowledges that she was copying, hers, she acknowledges, was not paint-
ing. That’s where she realizes that she was meticulously imitating another art,
and so she decides to use the camera and develop her ideas. And certainly, it
was in photography where she found her means of expression, becoming one of
the most recognized photographers in the United States. She has been named as
the woman of a thousand identities, she sees herself as a blank canvas on which
Table 1. Organization of exercises in sessions, with materials and description of pro-

totypes.
Didactic unit Resources Description

Knowing our -Nylon stocking -1 Pedagogical proposal that presents and
bodies: The self; pound of raw rice deconstructs, recycled, through plastic and
the identity -Threads -Scissors creativity, beauty stereotypes, such as gender role
for men and women from the reflection and artistic
production of the English Sarah Lucas. Prototype 1
Diverse portrait: -Photograph type Pedagogical proposal to modify and reconstruct the
The encounter with passport enlarged face of each of the students in a face collage. The
others in an A4 format in students question the western parameters of beauty,
black and white for that they will be shown the work of the artists
-Scissors -glue Cindy Sherman and Yasumasa Morimura. The
-Cardboard process will be to divide the photos into 5 equal
parts, 4 parts will be exchanged with colleagues to
finally build the final product on cardboard.
prototype 2
Gender stereotypes: -Paints -Markers Pedagogical proposal to argue about sessions 1 and
The environment 2. Within this activity it is exemplified through
linear sheets that will be delivered to the students
to color. These sheets are deconstructions of
stereotypes. That is, changes of roles within the
home. Prototype 3
she captures different iconographies of women. Her photographs are not autobi-
ographical, but sometimes, through artifice, they seem to forge harsh reality.
Yasumasa Morimura. Japanese artist (1951) known for being a representative

of the staged photography, as well as the artistic current known as appropria-
tionism, is an artistic movement in which the artist uses elements of others to
elaborate his work. For more than 30 years, he has worked as a conceptual pho-
tographer and thanks to this he has become a great figure of international art.
Morimura is known for being part of his works in an unimaginable way, since
thanks to the makeup, costumes and accessories that he uses, he becomes a cen-
tral part of the paintings that other great artists created. Reinventing the iconic
works of artists such as Frida Kahlo, Vincent van Gogh and Diego Velázquez is
the work of this artist who has repeatedly stated that his primary interest is to
be seen.
3.4 Practical Exercises

Table 1 presents different exercises for each session with required materials. Much
attention was paid to generate activities that do not require expensive or difficult-
to-obtain materials. This with the aim of lowering the access barrier for a broad
public. Figure 1 shows three prototypes declared for each session in Table 1.
3.5 Evaluation
To evaluate, several factors must be considered and thus allow a general evalua-
tion to be carried out, which measures the procedural development of the activ-
ities for their qualification. It is complex to generate a numerical measurement
process when free expression is raised, the teacher must evaluate the student in
this way, not by mimetic quality, taste, or aesthetic excellence.
Fig. 1. Results for prototypes of each session. From left to right: the body, me and the
others, and the environment.
This class invites the development of critical thinking from the arts, from
the reconfiguration of what exists in society as normal. Any rubric consid-
ered by teachers must be thought and rethought for disruptive education, since
inevitably, quantifying students does not reach the distance from behavioral con-
cepts of education. A rubric has to be designed for participation, collaboration,
performance, respect and initiative in the classroom, which can demonstrate
learning based on the concepts of non-violence, tolerance, democracy, solidar-
ity and justice. This rubric should move away from the aesthetic perspectives of
what has been assigned as “beautiful art”, since this concept and perspectives are
cultural relativisms. Thus, the design of the instructional class itself has exposed
examples of contemporary art, which has that have meant a disruptive act to
the aesthetics of traditional taste. It should be considered that the teacher must
identify, not evaluate each and every student, in the same way, students with
different abilities present characteristics that lead to providing special support in
their training process. They can be the following: physical, intellectual disabil-
ity, mental disorders, emotional difficulties, socio-familial, cultural or linguistic
dysfunction and intellectual giftedness. Whatever the condition of students with
different abilities is, it will always be necessary to make curricular adaptations
to satisfy their special learning needs.
3.6 Student Self-assessment

The purpose of the self-assessment is to assess the knowledge of the students,
their opinion about the classroom experience, what they learned from it and if
they felt in a comfortable environment on the topics covered. The self-evaluation
format is a list of Yes or No questions, as follows:
1. I feel like I learned something new in this class.
2. Did I like the class?
3. Did I feel comfortable in this class?
4. I feel that I can be creatively free with these artistic exercises carried out.
5. I feel that I can freely express my criteria and ideas in class.
3.7 Teacher Self-assessment

It is important to have indicators for pedagogical and didactic improvements
in the school, as well as to have information on the perspective of the teachers
participating in this research. The form for the teachers of the institutions where
the instructive methodological class was given seeks to know their criteria as
specialists and is done through a Google form, the questions are as follows:
1. Do you think that students can identify the differences between sex and gen-
der? Yes or not.
2. Do you consider that the contents taught are relevant for the age chosen for
this research? Very pertinent, Little pertinent, Not pertinent at all.
3. This questionnaire meets the requirements for a Cultural and Artistic Edu-
cation (ECA) class? Yes or not
4. How do you think the class could be improved? Write your suggestions
Each class has a QR code, with the development of each instructive method-
ological class session to facilitate the work of teachers, see Fig. 2
Fig. 2. QR codes to access materials described in the three sessions.
4 Results
The instructive methodological class was applied in two educational institutions.

Namely, School 1 is a Private educational institution called “Cuenca’s French
School”, a practice carried out with 22 fifth grade children. And School 2 is a
public school “Manuela Cañizares School”, practice carried out with 25 children
of sixth grade.
As stated before, activities are organized in three sessions. Session 1: Know-
ing our bodies, which include a survey additional to the adaptation of the class,
review of artistic references, and perform practical exercises. Session 2: Diverse
portrait, with the adaptation of the class, review of artistic references, and per-
form practical exercises. Session 3: Gender stereotypes, which includes a Self
appraisal and Teacher self-assessment additional to the adaptation of the class,
review of artistic references, and perform practical exercises.
4.1 Survey
As Fig. 3 shows, the majority of the students answered affirmatively in the two
schools. The students have a high reception with the topics covered. Based on
the structure of the survey, the results indicate that these groups still do not
clearly mark gender stereotypes, identified as negative in the introduction of this
research based on biopolitics, law and recent studies on gender. Identifying that
in the private school some do address conversations about gender diversity, the
students participated actively and in the theoretical sharing they asked about
their doubts and gave their opinions with examples. In the public school, it was
not possible to raise a questioning dialogue, the students debated little about
the activities, and it was difficult to carry out actions of free creativity. The
group constantly waited for instructions and there was no participation for the
dialogue, indicating a receptivity of a behavioral nature, it was perceived the

fear of participating freely in the three work sessions.
Survey results in schools 1 y 2.
School 1 School 2
22 25
Can men take care of their children? 100% 100%
Can women be presidents? 100% 100%
Can boys play with dolls? 86,3% 80%
Can girls play with cars or planes? 95.4% 100%
Can girls be firefighters? 95.4% 96%
Can the boys help with the

100% 100%
housework?
Can boys study dance? 100% 80%
Fig. 3. Results for both schools, public and private, to session 1’s survey.
School 1: The students of this school were more receptive to dialogue. They
believed that each human being can do with his body what he sees fit as long
as it does not harm anyone. Respectfully, they showed interest and acceptance.
School 2: The students in this school were much less receptive to dialogue. They
limited themselves to watching the videos, without making further comments,
despite the fact that an attempt was made to allow them to speak from flipped
classroom strategies.
4.2 Session 1: Knowing Our Bodies - Practical Exercises
In both schools, the practical exercises began with an explanation of the steps to
follow, through photographs using slides. Then, the students they took out the
materials requested in advance to work (recycled nylon stocking, rice, thread and
scissors). All students performed the steps explained above, when they needed
Technical help was requested. To complete the first activity, details are drawn
with markers, for example, a face or clothing. For some students it was difficult
to separate the parts of the body, apparently that was the first time they did it,
but with a little help they all made it successfully.
Everyone generated a doll with their own ability, the students obtained very
diverse results in the construction of the stick figures, some including non-human
figures, including zoomorphic ones.
Recommendations. During this first section, it should be considered that mate-

rials such as the rice is spilled, it is suggested to work on newspaper to collect
the surplus. Do not predispose the student to cleanliness as a punishment, but
rather emphasize the cleanliness of their activity as part of the good process of
creative construction.
4.3 Session 2: Diverse Portrait, Practical Exercises

Before starting the practice, the students observed through slides the steps to
follow, that is, the procedural design of the object, then they used the requested
materials (black and white photograph, enlarged A4, ruler, scissors and pencil).
The photograph was divided into five equal parts, as indicated in Fig. 1. Subse-
quently, the parts are exchanged among the classmates, and each student must
keep one part and exchange four sections with her group, obtaining five parts of
different faces. The parts of the faces must be joined in a collage on a support.
Recommendations. As a good practice for the teaching portfolio and the pro-
cesses of the subject, it is suggested that the teacher record the results, either
with a cell phone or a good resolution digital camera, verifying the distance
between the objective and the lens to guarantee the homogeneity of the collage.
4.4 Session 3: Gender Stereotypes - Practical Exercises

As part of this activity, students were given coloring sheets with line drawings,
where deconstructions of role stereotypes about people’s work activities can be
observed. For this activity, the student should be asked to bring paints, markers,
crayons, watercolors, or any material they have at home and feel comfortable
working with in classroom.
While the students color, it is important to invite them to use a diversity of
colors, to reinforce that the chromatic palette has no gender and that this social
construct must be broken. During the two experiments of this exercise in school
1 and 2, the girls and boys used various colors, showing that with the necessary
instruction they are not influenced by the social stereotypes with which they are
educated.
4.5 Assessment
For the evaluation, it is suggested to use the table developed for the rubric
explained in Sect. 3.2.5. Consider the freedom of expression of each student and
that this is not a reason for assessment based on the subjectivity of perspectives
between teacher and student.
Self Evaluation. Finally, Fig. 4 indicates the students responded to the self-
evaluation presented in Sect. 3.6, which seeks to find out how the students felt
in the class. In these exercises, most of the students responded positively to the
class.
Student self-assessment
School 1 School 2
22 25
I feel like I learned something 95.4% 92%

new in this class.
Did I like the class? 100% 100%
Did I feel comfortable in this class? 86,3% 88%
I feel that I can be creatively free with

these artistic exercises carried out. 86,3% 96%
I feel that I can freely express

100% 72%
my criteria and ideas in class.
Fig. 4. Results for both schools at the end of the three sessions.
Teacher Self-assessment. The teachers participating in the instructive

methodological class filled a Google’s Forms with questions presented in Sect. 3.7.
All teachers from both schools answered positively to all questions, indicating
that contents are appropriate and contributing for the students.
In addition, the teachers provided the following recommendations for future
tools that strengthen the themes of the class:
– There must be direct communication with families.

– Insert role plays on gender roles and extra activities for students who do not
wish/can participate.
These suggestions are outside the scope of this research, but could be useful
in another class.
General Recommendations.
– It is recommended to respect the time of two hours for the construction of

the final products, since it is not an activity that they have carried out in
advance, and they must work with a time that gives them comfort in the
development of their procedural knowledge.
– Teachers must be aware that they are the ones who must generate spaces
for dialogue with students. As it is a matter of inequality for the rights to
diversity, it is recommended to narrate an experience, case or example (inves-
tigated by the teacher), which contributes to empathy in the classroom.
– It is important to insist during the classroom dialogue on the use and assign-
ment of colors and the deconstruction of gender imaginaries and their assign-
ment.
– Making a students’ work exhibition at the end is valuable, so more people
can see the final products and thus also the whole school, their socialization
can generate questions and critical thinking in the users.
5 Conclusions
This instructive methodological class explores gender identity and seeks an inclu-
sive education, with references from Freire and Camnitzer. The activities and
tools developed align with the Cultural and Artistic Education curriculum,
through international artistic references of contemporary art such as: Sarah
Lucas, Cindy Sherman and Yasumasa Morimura.
A pilot test was implemented in two schools in the city of Cuenca in Ecuador.
When comparing the results, no significant difference was observed between
School 1 (private) and School 2 (public). In conclusion, students do not use
gender stereotypes negatively.
Future works include the implementation of the instructive methodological
class in a large-scale plan in the city, including rural areas, to obtain a better
understanding of inequality between genders.
References
1. Acha, J.: Educación artı́stica: escolar y profesional. Trillas (2001). https://www.
worldcat.org/title/1097719359
2. Cajas Córdova, A.K.: Igualdad de género en la constitución ecuatoriana de 2008.
Aportes Andinos (29) (2011). http://hdl.handle.net/10644/2804
3. Caminitzer, L., Helguera, P., Marı́n, B.: Art and Education (2014)
4. Camnitzer, L.: Didatica de la liberacion: arte conceptualista latinoamericano. Casa
Editorial HUM (2008)
5. Casasola Rivera, W.: El papel de la didáctica en los procesos de enseñanza y apren-
dizaje universitarios. Comunicación 29, 38–51 (2020). https://doi.org/10.18845/
RC.V29I1-2020.5258, https://revistas.tec.ac.cr/index.php/comunicacion/article/
view/5258
6. Freire, P.: Capı́tulo I, Pedagogı́a del Oprimido (2005)
7. Ayuntamiento:, L.: Sexist violence in childhood (2017). https://www.youtube.
com/watch?v=kSreklAi34w, (Accessed 16 July 2023)
8. Nations, U.: He for she - campana onu mujeres - youtube (2016). https://www.
youtube.com/watch?v=3TVKnLbwW M, (Accessed 16 July 2023)
9. Olmedo Carrasco, C.: Didacts of liberation: the latin American artist’s coloniality
and emancipatory project in the Luis Camnitzer thought (2017)
10. Pakapaka: Yo quiero saber, y vos? - cap. 2: Ser como quiero ser (2022). https://
www.youtube.com/watch?v=acbFgRpdqhE, (Accessed 16 July 2023)
11. Peñafiel, F., Herrera, M., Andres, C., Ormaza, P.: ”una mirada en profundidad al
acoso escolar en el ecuador”. Ministerio de Educación (2015)
12. Ponte, N.B.: La agenda 2030 y la agenda regional de género sinergias para la
igualdad en América Latina y el Caribe (2017)
13. Quiñones, L., Duarte, S.: Las mujeres están por debajo de los hombres en todos los
indicadores de desarrollo sostenible — noticias onu (2018). https://news.un.org/
es/story/2018/02/1427081
14. Ramirez, R.F., Manosalvas, M., Cardenas, O.: Gender stereotypes and their impact
on the women of Latin-America and Ecuador. Espacios (41) (2019). https://www.
revistaespacios.com/a19v40n41/19404129.html
15. del Rosario Ayala-Carrillo, M.: Violencia escolar: Un problema complejo. Ra

Ximhai 11, 493–509 (2015). http://www.raximhai.com.mx/Portal/index.php/
ejemplares/7-ejemplares/39-volumen-11-num-4
16. Sotelo, G., Domı́nguez, M.: Methodological approach to teach the understanding
of art: a discussed reflection. Revista Ciencia Administrativa 2017(1), 8–18 (2017).
https://www.uv.mx/iiesca/files/2017/10/02CA201701.pdf
17. Trujillo, Y.: La reforma a la ley educativa pone freno a la violencia en las
aulas - el comercio (2021). https://www.elcomercio.com/tendencias/sociedad/
acoso-escolar-violencia-ley-educacion.html
18. Turı́n, A., Bosnia, N.: Rosa caramelo - cuentos infantiles de igualdad de género -
dı́a internacional de la mujer - youtube (2019). https://www.youtube.com/watch?
v=eqLfmacRU0k, (Accessed 16 July 2023)
19. United Nations: El vı́deo de la onu que quiere acabar con los estereotipos de género
- youtube (2017). https://www.youtube.com/watch?v=C1bI0UkhmQA, (Accessed
16 July 2023)
20. United Nations: La sociedad del cuidado horizonte para una recuperación sostenible
con igualdad de género (2022). www.issuu.com/publicacionescepal/stacks
Educational use of TikTok to Enhance Complex
Grammatical Aspects of French: A Pilot Study
David Martínez Cerqueda , Alma Eloisa Rodriguez Medina(B) ,

and Alonso Gómez Ortíz
Universidad Veracruzana, Lomas del Estadio S/N, 91000 Xalapa, Veracruz, Mexico
{davimartinez,almrodriguez}@uv.mx
Abstract. In this pilot study, we present a preliminary evaluation of the impact

of incorporating a microlearning path through educational videos on TikTok for
pre-intermediate French students at a Mexican vocational training institute that
offers French as a foreign language. The aim is to establish a solid foundation
for further research involving pre-intermediate French students pursuing a French
Language Bachelor at a Mexican university. These students need to reinforce
recurring complex grammatical aspects in their learning process, as identified in
a previous survey where the past compound tense emerged as the most difficult
grammatical aspect. In this pilot study, a pre-experimental design of a single
group was proposed, using a pretest – posttest design to observe and compare the
participant responses, focusing on the use of the past tense, spelling correction and
other general aspects. Preliminary findings revealed a positive effect on students’
performance in relation to the addressed grammatical aspects after TikTok usage.
As a second stage of our research, we plan to continue evaluate the impact of
TikTok on one or more student groups at the mentioned university. In addition, we
intend to establish a repository of educational pills as complementary resources to
traditional classroom teaching, specifically designed to reinforce FLE grammatical
aspects. These educational pills will be integrated into microlearning paths hosted
on TikTok.
Keywords: TikTok · French as a foreign language · grammar · social media
1 Introduction
1.1 Context and Significance of the Research
In a world currently influenced by social media and artificial intelligence (AI), language
teaching-learning process is an area that is gaining more interest in the field of higher
education. The accelerated advancement of technology has brought new possibilities to
the acquisition to a foreign language.
In this context, this research focuses on the use of the social media TikTok to
strengthen hard grammar aspects from French as foreign language (FLE) since was
detected by a simple survey the need to provide additional support in these topics to
https://doi.org/10.1007/978-981-99-7353-8_14
Educational use of TikTok to Enhance Complex Grammatical Aspects of French 173
students within a higher level of a BA degree in French language in a Mexican pub-

lic university, Universidad Veracruzana. This survey asked whether the current student
enrolled in the course of Francés Avanzado had experienced difficulties with the gram-
matical topics studied at the beginning stages. This way, it was identified the hardest
topics were complément dóbject direct (COD), passé composé and the subjonctif .
Our interest in approaching FLE grammar in this research lies in the importance
of this linguistic component in the language acquisition process, since effective com-
munication is facilitated when students know the elements of the target language that
contribute to assemble meaningful dialogue [1].
This pilot study was conducted in pre- intermediate students of French of a state
training institute in Mexico. First, this research seeks to establish a solid foundation that
allows make a deeper study in pre- intermediate or advanced students of the BA in French
language of the mentioned university. Second, this research pursues to design a bank
of educational microlearning pills to complement the traditional classroom teaching
to strength grammar aspects of the FLE through microlearning routes enriched with
videos hosted on the social media platform of TikTok.
It should be noted that social media is getting an important role as a resource of
distance education [2]. In this respect, we consider that the use of TikTok may benefit
the students since it is a tool that allows students to watch the videos as many times as
they need it and also it allows them to have a better understanding of a topic studied
in class. Therefore, TikTok might be used as a suitable techno-pedagogical tool for the
FLE learning.
Objective
The purpose of this pilot study was to present a preview of the evaluation of the
impact of the use of a microlearning path of educational videos on TikTok in students
from Francés pre-intermedio subject of a state institute for job training that teaches
FLE. This pilot stage allowed us to recognize strengths and areas of opportunity to be
addresses in the second stage, which will consist of the LLF case study.
The development of this pilot study was guided by the following research questions:
– What is the impact of the consumption of a microlearning path of educational videos
on TikTok on the academic performance of pre-intermediate French students at a
state job training institute?
– To what extent does the consumption of TikTok videos support pre-intermediate
French learners in reinforcing complex grammatical aspects of French as a foreign
language?
– What are the strengths and areas of opportunity identified in this pilot stage of TikTok
video consumption during the pre-intermediate French teaching learning process?
2 Theoretical Framework
2.1 Technology Mediated Language Teaching and Learning
Historically, technology has played an important role in language teaching. According
to Ghanizadeh et al. [3], technology has several fundamental roles in language teaching;
as a resource, delivery system or productivity. The Real Academia Española [4] defines
174 D. Martínez Cerqueda et al.
technology as the set of theories and techniques that allows the practical use of scientific
knowledge. In view of the above, in the context of this study, technology is understood
as the set of theories, resources and techniques that enable language learning.
In this sense, technology is referring to the use of devices, online platforms, inter-
active resources, immersive games, simulations, virtual and augmented reality, social
media, chatbots, automatic translators and voice recognition supported by AI as well as
other rising technologies that may be useful to enhance and complement the traditional
foreign language teaching. Moreover, emerging technologies such as cloud computing,
computational thinking, and natural language processing are growing, and they are very
promising for their use in language learning and teaching [5]. Thus, it is considered of
particular importance to understand how the use of these technologies is impacting the
process of foreign language teaching and learning and how to take advantage of these
new possibilities to improve the linguistic and intercultural development of the students.
Regarding the effectiveness of technologies in language learning, several studies
have shown that the use of technologies provides almost unlimited access to quality
linguistic and cultural materials [3]. For instance, Traore and Blankson [6] used audio-
visual technologies in English language teaching and found that it helped students learn
English more than other groups using a single technology. In addition, in the literature it is
possible to find reviews of studies related to the technologies used for language learning,
Duman et al. [7] presented research trends in mobile-assisted language learning from
2000 to 2012; Ghanizadeh et al. [3] and Golonka et al. [8] reviewed studies published
before 2014; Shadiev and Yang [5] reviewed which and how technologies were used in
studies published from 2014 to 2019, as well as which social media highlight. According
to Borromeo [9], social media is a group of applications and collaborative spaces where
there are social connections and information exchanges in a network environment.
In accordance with Shadiev and Yang [5], language learning through social net-
working sites (SNS) has attracted millions of users worldwide. Some authors cited have
pointed out that SNS are services that help learners create profiles and connect with
other language learners [10]. Language learners can build a linguistic identity, build a
relationship to the target culture [11, 12] or join a closed language learning organiza-
tion that uses social networks to improve communication and knowledge sharing among
learners [10].
The use of audiovisual resources is another strategy that teachers have used for years
to support language teaching and learning. First, in the early years of television, films in
the classroom, documentaries, or recordings of television programs were projected in the
target language using videocassettes, then on DVD and Blue-Ray. In recent decades, with
the advent of the internet, platforms such as YouTube or Vimeo have emerged, offering
visual content through online videos for educational or entertainment purposes, used to
reinforce linguistic and cultural aspects. Online videos refer to video content delivered
or available via internet. Learners can access language learning video resources via the
internet and watch them to learn the target language [5].
Like the aforementioned video hosting platforms, social networks also offer the pos-
sibility of exploring online videos and allow both teachers and students to discover and
share a good variety of visual content to address specific aspects of foreign language
acquisition. Several studies show the effectiveness of using online videos, for example,
the results obtained in a study by Lin and Wang [13] showed that videos were partic-
ularly useful for students to use communicative strategies and to carefully prepare for
multicultural communication.
2.2 Use of Tiktok in Language Teaching and Learning

TikTok, known in Asia as DouYin, is a social network application that allows users to
create and share short videos in a quick and easy way [14]. TikTok is an application that
is getting special attention in the educational field despite the negative aspects attributed
to it, since it might lead to misinformation and concerning due to the access of certain
contents and the excessive time that some users invest to consume content in the social
network [2].
Nevertheless, after the COVID-19 pandemic, TikTok has become one of the most
used applications among Centennials or Generation Z aged between 19 and 29 years
old. Currently, the platform has approximately 1500 million active users in 154 countries
making it one of the most popular social networks in the world [2]. These figures refer
to the cultural phenomenon and are enough to support the relevance of its analysis to
think about school, since users of this social network are mostly school age [15].
The learning that is produced in the context of TikTok may be compared to the one
of the virtual communities which promote social interaction among students as well as
the construction of knowledge. TikTokers learn by doing. That means that they do not
approach what they intend to master from theory, they approach it from the beginning
in its practical implementation taking as a reference the members of their community
whom they begin to imitate [15].
In the case of language learning, the students can have access to videos created by oth-
ers and observe authentic styles of communication, cultural and linguistic aspects which
allows them to build their own knowledge about the target language, either imitating,
adapting, or creating their own linguistic content, providing feedback or participating
with other similar users. In this sense, we agree with Tobeña [15] who states that learning
by doing in the context of digital communities there is an implicit theory of the learn-
ing that involves both the importance of the experience (as a source of learning) and
social constructivism complemented by connectivism that makes TikTok a community
of practice with an immense potential learning.
Regarding the studies related to the use of TikTok videos in language learning, it is
important to highlight that the amount of research that has explored this topic is increas-
ing. Several researchers have studied TikTok for the English language learning mainly
as a source of motivation for the students [16]; as an interacting learning to improve stu-
dents speaking skills [17, 18] as well as grammatical skills of higher education students
[19], In addition to analyze the effectiveness of using this social network to improve the
speaking skill of the students [20] and know the perspective of the students regarding to
this issue [21]. Furthermore, key features of TikTok have been identified that might be
useful to improve pedagogical strategies to keep the teaching of linguistic skills in the
ESL classrooms [22]. It is noted in an exploratory study about TikTok in mobile-assisted
English language learning (MALL) with contradictory findings between the students’
perceptions and the additional quantitative stage with MALL in which TikTok was used
to support teaching in the classroom [23]. The cited studies provide a solid base that
allows to understand TikTok educational potential and show that the social network may
be a worthy platform for language learning.
2.3 TikTok in the Context of Learning French as a Foreign Language

According to Tobeña, social networks present a series of characteristics that make them
a paradigm form of communication and, therefore, susceptible to be used in the learning
process, especially in the case of languages and, more specifically, French. [15]. In this
regard the learning needs of a group of FLE students have been analyzed based on the
use in class of a social network in combination with video [24]. It has also studied the
attitudes and evaluation of the use of a social network for FLE learning [25].
In this context, it is important to note that, the specific case of the use of TikTok for
FLE learning is an under-explored topic compared to the studies that demonstrate its
use for English learning. This evidences the need for further research that analyzes the
potential and benefits of TikTok for learning FLE. However, it is possible to find in the
literature some previous studies such as the one by Flores Borjabad [26] Which propose
a method of teaching of French and other languages through the transmedia experience
and social networks. In this research, it was found that, the videos made in TikTok, the
students developed oral production in the target language without feeling pressure from
the teacher.
3 Methodology
First, it was necessary to obtain the grammatical points that presented the greatest
difficulty among French students, for which an initial survey was conducted.
3.1 Initial Survey

In order to identify the current grammatical points among French learners, students who
already had previous knowledge of French were selected in order to obtain a retrospective
look at the main grammatical points that presented them with difficulties during their
learning.
In this way, 20 students were surveyed, presented with 10 grammatical items, namely
definite and indefinite articles, partitive articles, direct and indirect complements pro-
nouns, double pronominalization, participles, compound past, imperfect, subjunctive
and passive voice.
A Likert-type scale, adapted from the original 1932 proposal, was used to identify the
degree of difficulty of the subjects. Bertram [27] explains that the so-called Likert scales
are psychometric instruments where the interviewer must indicate his/her agreement
on a statement, item, or reagent, which is done through an ordered and unidimensional
scale. In the scale used in the initial survey, 5 levels of appreciation are detailed, where
1 is the one that presented the last difficulty and 5, the greatest difficulty. The result
was as follows: the most difficult item was the compound past, followed by the double
pronominalization, direct and indirect complement pronouns and the subjunctive. For
this preliminary study, the compound past was used.
3.2 Experimental Phase

This pilot study shows the impact of the consumption of a microlearning path composed
of educational videos in TikTok applied in a state institute for job training in Mexico, as
a first part of the research. It is important to mention that the study will have a second
phase of application in a Mexican public university that trains professionals in language
teaching.
3.3 Type of Research

In order to evaluate the scope of the exposure of the microlearning route composed of
educational videos on TikTok to a group of participants and to identify possible changes
in their use, we sought a design that would allow us to measure whether or not there
were changes after exposure to an independent variable, so it was decided to follow an
experimental type of research.
Experimental research, as explained by Tamayo and Tamayo [28] Experimental
research, as explained by Tamayo and Tamayo is presented through the manipulation
of an untested experimental variable in rigorously controlled situations to describe how
or why a particular situation or event occurs. Pardinas, in [28] adds the following: the
experiment is a situation provoked by the investigator to introduce certain variables
introduced by him, to control the increase or decrease of those variables and their effect
on the observed behaviors.
In concordance with the above, a pre-experimental design of a single group was
proposed in response to the identified problem of complex grammatical aspects that are
recurrent in French students.
In this way, the researchers were able to observe the behavior of the participants
‘responses and record possible changes following exposure to the educational video
microlearning path of educational videos on TikTok. The dependent variable consists of
explaining and exemplifying the information, use and exceptional cases of the French
indicative past tense (passé composé) through videos posted on TikTok.
3.4 Participants
The participants were pre-intermediate French students, so they had completed two
previous courses. It is necessary to mention that the invitation was open to the group of
7 students, of which 5 answered the pretest exercise, but only 4 concluded both pretest
and posttest, so only these last participants will be considered in the discussion. As a
particular characteristic of the group, we would like to add that these students had online
courses distributed in five synchronous hours with a teacher and three hours in autonomy
in which they could solve online exercises or oral and written comprehension a written
production; orality was addressed in the hours with the teacher.
3.5 Educational Videos on TikTok

To create the videos, we previously resorted to an instructional design approach based
on the needs of the FLE students and on reliable references that would guarantee the
reliability of what would be exposed in front of the camera. After the review of the
references, the parts to be addressed in the videos were established. Then, the topics
were classified, and finally, it was considered to design 5 videos that, on the one hand,
would have independent contents and, on the other hand, would avoid the saturation of
information for the consumers.
The scripts were reviewed and rehearsed among the researchers to obtain scripts
that contained all the necessary information, that were not long and that followed the
technical details proposed by the authors in a complementary guide for the creation of
videos. Thus, the first videos show one of the researchers who introduces the channel
and briefly explains what type of information will be found in it.
The second video explains and exemplifies the formation and use of the past tense
with it two auxiliars. The third video shows the verbs that use the auxiliary avoir. The
fourth video show the use of the auxiliary être with a list of verbs called “movement
verbs” and pronominal verbs. The last video explains the verbs that can use both aux-
iliaries with their particularities, examples, and characteristics. The five videos were
hosted on an account created for this purpose by the teacher in charge; a screenshot of
this account is shown in Fig. 1.
Fig. 1. Screenshot from the TikTok account used to shows microlearning path of educational
videos on TikTok.
3.6 Experimental Design

In experimental research it is possible to see three very recurrent designs: pre-
experimental, experimental, and quasi experimental. According to the phase of the
research, we chose the pre-experimental design with a pretest-posttest design of a single
group. In this way, an online exercise was designed with 37 items including general
uses and particular cases in the use of passé compose. It should be remembered that all
the cases were explained in the videos with the intention of covering all the needs of
the participants. The exercise was hosted on Google Forms and the link was sent via
WhatsApp to the group created for the participants. The instruction was as follows:
“We invite you to answer the exercise in the attached link. You are going to answer
the exercise the first time without any help, just with your previous knowledge,
finish the exercise and send it. After that we invite you to watch the TikTok videos
that I share to you right here, as many times as you consider necessary and, when
you feel ready, solve the exercise once more and send it a second time using the
same email address you used the first time.”
In this way we integrated the three moments: pretest (first response), training phase
(explanatory videos) and posttest (second submission of responses), five of the seven
invites’ students responded to the first exercises and, of the five, only four sent the second
exercise.
3.7 Data Analysis
Once the responses were received in their two phases, they were reviewed individually
by means of an answers key prepared by the researchers and, subsequently, a comparison
was made between the two products in order to identify if there were differences in the
responses. The aspects identified from the biggening for review were the exceptional
cases in the use of the past tense, spelling correction and general aspects.
4 Results
4.1 Sample Characteristics
A total of 4 students participated and voluntarily responded to the online questionnaire.

Of the 4 participants, 1 was male and 3 were female. The ages of the participants are
diverse because it is a job training institute and range from 29 to 68 years old.
4.2 Comparison of Responses Between Pretest and Posttest
The charts in this section show comparisons of some of the answers given by the partic-
ipants at the two moments of experimental design (pretest and posttest). The responses
are indicated as written by the participants.
Table 1 shows the participants ‘responses to the exercise on conjugation of the
verb écouter. Some of the participants made errors using the infinitive form instead of
conjugating the verb in the past tense. It is therefore necessary to emphasize the correct
use of the verb in question by reviewing the proposed video again or by proposing new
videos and/or reinforcement exercises specifically for this verb. We consider, after this
exercise, that it should not be taken for granted that first group verbs are simple and
known by all students, the performance of the participants shows this and indicates that
regular verbs also need their own space in the comprehension and acquisition of the
language.
Table 2 shows changing answers. While some participants spelled the verb cor-
rectly (the correction is understood here through attention to the different possibilities
of integrating the cases of gender and number agreement between the subject and the
Table 1. Comparison of responses on the verb écouter
Participant Pretest Posttest

F: Est-ce que tu __________ (écouter) F: Est-ce que tu __________ (écouter)
la nouvelle chanson de ton groupe la nouvelle chanson de ton groupe
préféré ? préféré ?
1 R: Est-ce que tu as écouté la nouvelle R: Est-ce que tu as écouté la nouvelle
chanson de ton gripe préféré ? chanson de ton groupe préféré ?
2 R: ecouté R: Est-ce que tu as écoute la nouvelle
chanson de ton groupe préfére?
3 R: écouter R: a écouter
4 R: Est-ce que tú as écouté la nouvelle R: Est-ce que tú as écoute la nouvelle
chanson de ton groupe préféré ? chanson de ton groupe préféré ?
participle) in some of the moments of the experimental phase, others made error when
trying to conjugate it. As in the previous example, it would be necessary to visualize the
reinforcement video again or to create more videos specifically focused on the use of
subject-participial agreement in the case of verbs with the auxiliary être.
Table 2. Comparison and responses on verb aller

F: Hier, nous __________ (aller) au F: Hier, nous __________ (aller) au
cinéma cinéma
1 R: Hier, nous sommes allées au R: Hier, nous sommes allées au
cinéma cinéma
2 R: allé R: Hier, nous est allées au cinéma
3 R: aller R: Hier, nous allons au cinéma
4 R: Hier, nous sommes allions au R: Hier, nous sommes allés au cinéma
cinéma
In the example on the Table 3, at least two of the participants made errors when
conjugating the verb. It is worth noting the extreme cases between participant 1, who
answered correctly from the pretest, and participant 4, who had a remarkable improve-
ment in the correct application of the verb in the posttest, after watching the videos on
TikTok. It also seems evident that is not only necessary to strengthen the differences in
the use of auxiliaries, but also the orthographic correctness of the auxiliaries.
Table 3. Comparison and responses on verb naître

F: Madame Dubois, vous __________ F: Madame Dubois, vous __________
(naître) en France, n’est-ce pas? (naître) en France, n’est-ce pas?
1 R: Madame Dubois, vous êtes née en R: Madame Dubois, vous est neé en
France, n’est-ce pas? France, n’est-ce pas?
2 R: naitré R: Madame Dubois, vous est née en
France, nést-ce pas?
3 R: Madame Dubois, vous naître en R: Madame Dubois, vous aves naître
France, n’est-ce pas? en France, n’est-ce pas?
4 R: Madame Dubois, vous ētés né en R: Madame Dubois, vous êtes née en
France, n’ est-ce pas? France, n’est-ce pas?
4.3 Overall Performance of Participants
The data presented in Tables 4 and 5 below point to an improvement in the participants’
performance in relation to the complex grammatical aspects addressed which can be
seen especially in the posttest.
Pretest
Posttest
Table 4. Overall performance of the participants in the pretest
Participant Correct answers Incorrect answers Formation Spelling Observations
33 4 Ok Ok Choice of participle or spell-checker by the

cellphone
1 0 37 No No The student shows knowledge of participles

but does not structure as expected
2 0 37 No No The student shows a total lack of knowledge

in formation and structure
3 19 18 Yes Yes Her error is related to the choice of the

auxiliary in three cases and the agreement of
participles
4 33 4 Ok Ok Choice of participle or spell-checker by the

cellphone
Table 5. Overall performance of participants in posttest.
Participant Correct answers Incorrect answers Formation Spelling Observations

33 4 Ok Ok Choice of participle or
spell-checker by the
cellphone
1 32 5 No No The student’s
performance shows a
very evident positive
change in her posttest
responses
2 8 29 No No The student show
progress at the level of
training by answering
the elemental questions
3 29 8 Yes Yes A clear correction of
aspects suggested by
the explanatory videos
can be observed
4 33 4 Ok Ok Choice of participle or
spell-checker by the
cellphone
5 Discussion
In order to start the discussion, contributions of each participant will be highlighted
individually, considering those aspects that may motivate a reflection by the researchers.
We are interested to highlight two aspects: On the one hand, this study is the result of
a pilot test, which bring us clues to the creation of the final study, possible changes,
as well as the necessary adjustments for the design; of course, we have considered the
experimental and control groups to a second stage prior to the final experiment. On the
other hand, the validation of the experiment does not lie on the number of participants, but
in the observations that the researchers determined to continue with the experimental
situation. Thus, we will start with participant one, whose performance stood out by
showing almost all the correct answers in both tests; Secondly, we have two participants
whose performance was very similar and that shows the opposite extreme of the first by
not having any correct answers. Finally, participant four stands up for presenting a great
before and after in the execution of the assigned task.
Participant 1. The case of one of the participants is distinctive since it had almost all
the correct answers in the pretest. In the posttest she improves and answers correctly one
of the three exercises that she had answered with an error in choosing an auxiliary. The
following two errors are due to an incorrect choosing of a participle. It is assumed that in
this case it is about a more advanced student of French, which has permitted her a better
performance in relation to their classmates. The fact of making an incorrect choice of
the participles may suggest the fossilization of the error or could invite the researchers
to create other learning paths aimed at the practice and use of the participles. On the
other hand, the correction of one of her errors suggests that the videos were useful for
the participant and helped her to correct herself.
Participant 2 and 3. Both participants had an honest and positive participation. They
solved the exercises without their notes as instructed. It is important to mention that they
did not have one hundred percent correct answers. In other words, one of the participants
made occasional use of some correct participles, despite she did not use auxiliaries,
while the other participant did not show the use of any auxiliary in the pretest, which
draws attention since every student of French is aware of the relevance of the use of the
auxiliaries since the teaching of this tense.
In most cases, the participant transcribes the verb in the filling the blanks exercise,
which might show inconsistencies in their learning process, lack of study habits, learning
or teaching deficits; or human factors that could range from basic health or circumstantial
issued derived from daily life to motivational factors or different conditions linked to
personal learning process perhaps related to socialization, stress, or specificities of each
participant.
Participant 2. During the posttest, participant 2 answered using the auxiliaries in all
cases even though they were not the correct ones. It is striking that it is possible to
observe that corrects almost all the expressions by always using auxiliaries and also
showing a positive use of all the particularities of the auxiliary être as indicated in the
composition of the tense, which could suggest a positive effect after using the learning
routes through TikTok videos. The past tense (passé composé) of French has a series of
specificities related to exceptional cases but in daily use are very frequent, that is the
reason why the need to revise them and understand them. The student in question does
not show a control of the particularities related to the use of the auxiliaries, although it
is evident that she adds them and uses the spelling correction, when necessary, despite
not doing it with a domain of the topic.
Participant 3. The male participant 3 shows an improvement at a basic level, apparently,
an aware use of the auxiliary is made to the formation of the past in French, and, in
the remaining cases, she also uses them but still showing a lack of knowledge of the
participles.
Participant 4. The last case to report is very interesting since it seems to show a very
positive use of the learning paths. During the pretest, the participant makes errors in the
formation of the past tense that are related to spelling issues in the use of the auxiliaries,
for example, she seems to have a very consistent use of her participles in almost all cases.
Nevertheless, during the posttest, she systematically corrects the situation in each phrase.
It is evident that she could clearly observe her mistakes and correct them punctually in
each case. It seems evident that the intervention of the videos may contribute or remember
previous knowledge and transform it into elements in which students could enrich their
assets and, we hope, to apply them in their daily life.
6 Conclusions
As a conclusion, we would like to highlight two main topics found. On the one hand,
the view towards challenging topics such as the approach to grammar instruction in
a time when perspectives are changing continuously and privilege other aspects like
teaching through cultural contexts over the communicative approach. We believe that
it is necessary not to forget that the student is the central axis of our practice; it is the
student who must guide and provide the ideas to find the most appropriate strategies for
learning and, at the same time, present some challenges to the teaching practice.
On the other hand, the use of new technologies for language learning. Language
acquisition has historically leaned towards innovation in different ways, not only ped-
agogically, but also incorporating digital resources. The use of online platforms in lan-
guage teaching and learning is relatively new and it is in full development, not only
proving that technology in this area is positive, but that the use of the new platforms
such as TikTok refreshes skills as hard as grammar in French learning. Exploring how this
integration impacts the understanding and development of linguistics skills in students
enables new aspects of science specifically in the area of languages and their relationship
with technologies.
In conclusion, this research allowed to identify strengths in the pilot stage showing
positive results in the performance of the pre- intermediate French students on the gram-
matical aspects addresses after using TikTok. This suggest that the use of TikTok as an
educational platform can benefit in terms of better understanding and correct applica-
tions of complex aspects of French grammar. We consider that the positive results of this
study provide valuable input for the second stage of the research to be conducted in a
university context that responds not only to the evaluation of the impact of using TikTok
but also their perceptions of the experience of using TikTok for educational purposes. For
further research, it is necessary to expand the number of microlearning path to continue
addressing grammatical aspects and to consider other linguistic components and other
languages, as well as to continue improving the instructional design and the scripts to
produce the videos.
References
1. López Sosa, C.D., Fonseca Ayala, M.: La grammaire: sa place dans l’enseignement-
apprentissage des langues et sa demarche. Cuadernos de Lingüística Hispánica (2018). https://
doi.org/10.19053/0121053x.n31.2018.7762
2. Rodriguez Medina, A.E., Martínez Cerqueda, D., Balbuena Ortega, M. del P.: TikTok para
la enseñanza y aprendizaje de lenguas en educación superior: percepciones de profesores
mexicanos. Revista Paraguaya de Educación a Distancia (REPED) 4, 46–59 (2023)
3. Ghanizadeh, A., Razavi, A., Jahedizadeh, S.: Technology-enhanced language learning
(TELL): a review of resourses and upshots. Int. Lett. Chem., Phys. Astron. 54, 73–78 (2015).
https://doi.org/10.18052/www.scipress.com/ILCPA.54.73
4. Real Academia Española: Diccionario de la lengua española. https://dle.rae.es/tecnolog%
C3%ADa?m=form
5. Shadiev, R., Yang, M.: Review of studies on technology-enhanced language learning and
teaching. Sustainability 12(2), 524 (2020)
6. Traore, M., Kyei-Blankson, L.: Using literature and multiple technologies in ESL instruction.
J. Lang. Teach. Res. 2 (2011). https://doi.org/10.4304/jltr.2.3.561-568
7. Duman, G., Orhon, G., Gedik, N.: Research trends in mobile assisted language learning from
2000 to 2012. ReCALL 27(2), 197–216 (2014). https://doi.org/10.1017/S0958344014000287
8. Golonka, E.M., Bowles, A.R., Frank, V.M., Richardson, D.L., Freynik, S.: Technologies for
foreign language learning: a review of technology types and their effectiveness (2014)
9. Borromeo., C.A.: Redes sociales para la enseñanza de idiomas: el caso de los profesores.
Pixel-Bit. Revista de Medios y Educación 48, 41–50 (2016)
10. Maier, C., Laumer, S., Eckhardt, A., Weitzel, T.: Giving too much social support: social
overload on social networking sites. In: European Journal of Information Systems (2015)
11. Blattner, G., Fiori, M.: Virtual social network communities: an investigation of language
learners’ development of sociopragmatic awareness and multiliteracy skills. CALICO J. 29(1),
24–43 (2011). https://doi.org/10.11139/cj.29.1.24-43
12. Klimanova, L., Dembovskaya, S.: L2 identity, discourse, and social networking in Russian.
Lang. Learn. Technol. 17, 69–88 (2013)
13. Lin, Y.J., Wang, H.C.: Using enhanced OER videos to facilitate English L2 learners’ multicul-
tural competence. Comput. Educ. 125, 74–85 (2018). https://doi.org/10.1016/J.COMPEDU.
2018.06.005
14. Escamilla-Fajardo, P., Alguacil, M., López-Carril, S.: Incorporating TikTok in higher edu-
cation: pedagogical perspectives from a corporal expression sport sciences course. J. Hosp.
Leis. Sport Tour. Educ. 28, 100302 (2021)
15. Tobeña, V.: Pensar el futuro de la escuela desde comunidades de prácticas. Claves desde
TikTok. Dilemata. 12, 221–233 (2020)
16. Yélamos-Guerra, M.S., García-Gámez, M.: The use of TikTok in higher education as a
motivating source for students. Porta Linguarum 2022, 83–98 (2022)
17. Zaitun, Z., Hadi, M.S., Harjudanti, P.: The impact of online learning on the learning motivation
of junior high school students. Jurnal Studi Guru dan Pembelajaran 4(2), 263–271 (2021).
https://doi.org/10.30605/jsgp.4.2.2021.569
18. Xiuwen, Z., Razali, A.B.: An overview of the utilization of TikTok to improve oral english
communication competence among EFL undergraduate students. Univers. J. Educ. Res. 9(7),
1439–1451 (2021). https://doi.org/10.13189/ujer.2021.090710
19. Cagas, R.L.: The use of Tiktok videos in enhancing the speaking and grammar skills of higher
education students. Lang. Educ. Forum 3, 1–3 (2022)
20. Diko Putri, S.M.: The effectiveness of using TikTok to improve EFL learners’ speaking ability.
MIMESIS 3, 101–110 (2022). https://doi.org/10.12928/mms.v3i2.6096
21. Komariyah, T., Sulistiowati, W., Fajri, L.A., Allatif, N.: The implementation of TikTok appli-
cation to learn speaking skill in english language teaching (ELT). Conf. Engl. Langu. Teach.
2, 142–154 (2022). https://doi.org/10.24090/celti.v2.43
22. Tan, K.H., Rajendran, A., Muslim, N., Alias, J., Yusof, N.A.: The potential of TikTok’s key
features as a pedagogical strategy for ESL classrooms. Sustainability 14(24), 16876 (2022).
https://doi.org/10.3390/su142416876
23. Hu, H., Du, K.: TikTok in mobile-assisted english language learning: an exploratory study.
Int. J. Inf. Educ. Technol. 12, 1311–1320 (2022). https://doi.org/10.18178/ijiet.2022.12.12.
1755
24. Alvarez Alvarez, A.: Detección de necesidades de aprendizaje mediante el uso de una red
social y del vídeo en una clase de francés lengua extranjera / Detection of learning needs by
means of a social network and video in a class of french as a foreign language. Revista de
Comunicación de la SEECI (2016). https://doi.org/10.15198/seeci.2016.39.1-16
25. Álvarez Álvarez, A.: Actitudes y valoración en el uso de una red social para el aprendizaje
del francés lengua extranjera. Vivat Academia. Revista de Comunicación (2014). https://doi.
org/10.15178/va.2014.128.92-106
26. Flores Borjabad, S.A.: Innovación docente en la enseñanza del inglés, el francés y el árabe a
través de la experiencia transmedia y las redes sociales. In: Transformación digital docente.
La gestión sostenible de las organizaciones educativas, pp. 74–88. Dykinson, Madrid (2022)
27. Bertram, D.: Likert Scales…are the meaning of life (2008)
28. Tamayo y Tamayo, M.: El proceso de la investigación científica. Limusa Noriega Editores,
México (2015)
Review of Smart Grid Failure Prediction
and the Need for its Study in STEM Careers
Marcelo Fabian Guato Burgos1(B) , Jorge Luis Morato Lara2 ,

and Fernanda Paulina Vizcaino Imacaña3
1 Universidad Carlos III de Madrid, Getafe, Spain
100390410@alumnos.uc3m.es
2 Universidad Carlos III de Madrid, Madrid, Spain
jmorato@inf.uc3m.es
3 Universidad Internacional del Ecuador, Quito, Ecuador
pvizcaino@uide.edu.ec
Summary. Smart grids (SGs) are cyber-physical systems (CPSs) essential for
human activities. This document shows the relevance of the SGs and identifies
solutions for the prediction of failures, from which a reflection is made on the
importance of considering these results in the STEM career curricula. The method-
ology used is a systematic review of publications related to the prediction of fail-
ures in the SGs. Two research questions are defined: What is currently known
about the failure prediction in smart grids? And what are the methods most widely
used in this field? In the search process, keywords are defined, and successive
refinements are applied. The selection of papers eliminates duplicities, is based
on five criteria, and applies the consensus of two out of three researchers, consid-
ering the rigor, credibility, and relevance of each publication. The answers to the
questions raised show the trend towards artificial intelligence and highlight cross-
cutting study topics. New engineering and technology professionals, especially
electrical, software and computer engineers, must be prepared to contribute to this
sector of industry and to know the different trends around artificial intelligence
methods applied to prediction. The main beneficiaries of fault prediction solutions
are energy consumers or network customers. The generators of solutions are the
agencies responsible for operating and providing energy services and academia
through applied research. In the last decade, the Internet of Things and the tools,
methods, and technologies of big data have become cross-sectional components
of the SGs and are considered in the failure prediction solutions. Combinations
of machine learning algorithms, neural networks, data mining, and deep learning
are part of the solutions. This is an opportunity for new lines of research in STEM
careers.
Keywords: smart grids · cyber physical systems · failure prediction
1 Introduction
A smart grid (SG) is an electric grid that uses advanced information and communications
technologies to monitor and manage the transmission and/or distribution of electricity
from all generating sources to meet the different energy demands of end-users [1].
https://doi.org/10.1007/978-981-99-7353-8_15
Review of Smart Grid Failure Prediction 187
The SGs aim to coordinate the needs and capabilities of generators, network opera-
tors, end users, and power market stakeholders to operate all parts of the system in the
most efficient way possible, minimizing costs and environmental impacts while max-
imizing profitability, reliability, resilience, and stability [2], and including intelligent
operation concepts combining wide area control, computing, and communications [3].
The trend towards decentralized information technology has changed the rules for
electric services [1]. This shift in industry leads to a more dynamic and accelerated devel-
opment of intelligent power supply networks, whose architectures must also consider
the environmental impact [4].
Electrical networks are the backbone of electrical systems; they need to be expanded
and modernized to support energy transitions. The size of smart grids will increase by
approximately 90% between 2021 and 2050 in the established policy scenario (STEPS)
and another 30% in the announced promise scenario (APS). Annual investment in STEPS
increases from around USD 300 billion in recent years to USD 550 billion in 2030 and
averages USD 580 billion per year until 2050. In the APS, investment increases further
to USD 630 billion by 2030 and USD 830 billion in 2050. Taking a decade or more to
execute is not uncommon for complex projects in this field. In the long run, planning is
vital and considers, among other things, the growth of demand, the increase in quantities
of variable renewable energies, and the opportunities for digitization [5].
The largest consumers of electricity today are buildings, homes, and industry, which
together account for more than 90% of global consumption and have contributed more
than 90% to the growth of demand for electricity since 2010 [5].
Technological developments related to SGs include process automation and solutions
for generation, transportation, and distribution of electricity incorporating physical cyber
systems [6].
Given the observed rise in investment and the extended period of projects, it is
anticipated that there would be a corresponding increase in the total demand for skilled
experts across many domains associated with SG development initiatives.
The integration of electrical infrastructure, supported by an information and com-
munications technology infrastructure, has led to the development of smart grids (see
Fig. 1). These smart grids incorporate advanced automated procedures to enhance their
functioning [1, 7].
Therefore, the multidisciplinary teams to provide support, maintenance, develop new
technological solutions, or implement projects related to SG should incorporate profes-
sionals in computer science, software, telecommunications, electricity, and electronics.
Specialties that are part of the STEM careers and that will face the technological chal-
lenges [8], considering the trajectory established for the development of Smart Grids,
with a forward-looking perspective towards the year 2050.
Figure 2 shows the conceptual model of the complex system that represents an
intelligent power grid [9]. The operation of current power grids is based on four levels
resulting from their structure [4]:
Power generation, large power units positioned strategically in relation to the power
grid, typically close to primary energy sources, produce most of the world’s electricity.
188 M. F. Guato Burgos et al.
Fig. 1. Borlase, S. (2013). Smart grids: Infrastructure, technology, and solutions.
Transmission system, the section of the network that transfers electricity from a
large power plant to large consumption centers or other energy sub transmission and dis-
tribution systems. It is the backbone of the entire electrical system, contains sophisticated
equipment, and has highly centralized management.
Power Distribution Networks, the interface between transmission and end users,
are connected to transmission networks through substations and power transformers and,
for economic reasons and operating practices, are generally operated in radial structures.
End users or consumers, mostly passive consumers characterized by no controllable
loads and who do not contribute to the management and operation of the electrical system.
Each level involves multidisciplinary and specialized work. According to the
National Energy Technology Laboratory (NETL), an intelligent power grid tends to
auto-repair energy disruption events as well as operate resiliently against cyber-attacks.
The primary requirement of power grid planners and operators is to ensure the safety
and reliability of the power network to minimize interruptions [10].
Today, power grid intelligence is usually limited to asset management processes with
the aim of optimizing the operation to make it more efficient. According to NETL, the
trend is towards a deep integration of network intelligence with assets and management
applications, not focusing solely on asset protection but on generating the ability to
prevent service disruptions or minimize the impact of incidents.
An occurrence of electrical system failure take place when the characteristics of the
electrical energy provided to the consumer diverge from the specifications established
in the service agreement. The power outage is considered the most serious failure [11].
Power outages occur due to various factors, including physical component breakdown,
network overloads [12], and even cyber-attacks.
Detection, prevention, and prediction of power grid failures are of utmost signifi-
cance. The current economic prosperity and growth of a nation will depend in many
ways on the reliable operation of energy systems [13].
Early detection of failures also considers whether these occur due to aging or damage
to the network infrastructure; it also covers physical threats such as weather conditions,
fires, road accidents, and cyber threats, which can cause widespread and lasting outages
[10].
Fig. 2. Framework and Roadmap for Smart Grid Interoperability Standards, Release 3.0. Fuente:
NIST
The following are the actors interested in the development and application of
predictive methods:
• Organizations responsible for the operation of electricity networks or their generation,
transmission, and distribution components.
• The regulatory bodies of the electricity sector, understanding that each country has
its own regulation.
• Providers of electricity services, within the framework of the standards established
by the regulator.
• The academy, with the generation of knowledge from research and the preparation
of professionals with skills to face the new technological challenges of the industry.
Detection, localization, and predictability of power grid failure events has the
potential to reduce the impact on consumers:
• Domestic, because of the impact on the quality of life, for example, in climatic
conditions where heating or cooling is required, in addition to connectivity-related
services.
• Critical infrastructures, where the lack of electricity supply implies a lack of some
priority service.
• Industrial or commercial, due to the impact on the economy resulting from the
cessation of manufacturing, distribution, sales, and logistics processes.
2 Methodology
A systematic review of the work related to the detection of failures in smart grids was
carried out. The following steps were considered: definition of research questions, use
of tools and standards, planning, search for studies, definition of a search strategy, estab-
lishment of criteria and procedures for the selection of papers, evaluation of the quality
of the studies, data extraction, and analysis of results.
2.1 Definition of Research Questions

The objective of the review is to identify research work related to the predict of failures in
SG. For this, it is necessary to answer the following questions: What is currently known
about the failure prediction in smart grids? And what are the methods most widely used
in this field?
The bibliographic management tool Mendeley and access to electronic resources,
whose search tools allow the localization of documents relevant to this study, were used:
ACM Digital Library, Compendex/ScienceDirect (Elsevier), IEEE Xplore, ISI Web of
Science, SpringerLink, and Wiley Interscience Journal Finder.
The criterion used in this initial search was to explore in search engines with the
keywords: smart networks, bug prediction, cyber physical systems, and fault/failure
prediction.
2.2 Search Strategy

The aim is to identify the largest number of related documents by searching databases,
journals, and publications obtained from the digital repositories identified in Sect. 2.1.
Test searches allowed us to identify the number of jobs related to the prediction of failures
in smart power grids. Three iterative refinements were carried out to include papers to
answer research questions (see Fig. 3).
START
SELECTION OF RESOURCES AND TOOLS
SELECT KEYWORDS
TEST SEARCH
REFINE SEARCH
CHECK
UNTIL IDENTIFY
PRIMARY STUDIES
< 90%
MATCH
STORE INITIAL LIST OF PAPERS
STOP
Fig. 3. Publication Selection Process Flow (adapted from [14]).
Searches were carried out by filtering the titles of papers or publications that have the
following keyword combinations: smart grids with fail/failure prediction, cyber physical
systems with fail/failure prediction. Finally, it is considered the date of publication since
2010, the starting year for recent studies of global energy perspectives [5].
2.3 Criteria and Procedure for the Selection of Papers

The search process consists of four stages. A first list of papers was identified in the
first phase with automated searches; duplicate documents were manually discarded; and
finally, documents that did not meet the criteria shown in Table 1 were excluded. The
information was contrasted between three researchers by selecting the publications on
which there was consensus by two of three.
Table 1. Criteria for selection of studies or publications found.
Criteria
The study aims to investigate the issue that has been raised in connection with smart grids
It is a primary study that presents empirical results
There is a presentation of results obtained
The study presents methods and their applications
The study presents future lines of research
To assess the quality of the selected studies or articles in addition to the already
defined criteria, three cross-cutting principles are considered: rigor, is there a compre-
hensive approach, and are research methods applied to the study? Credibility: are the
findings well-presented and significant? Relevance: how useful are the results in the
context of smart grids?
3 Results and Analysis

A distinction should be made between fault identification and fault prediction. In the
first case, the task is to quickly determine whether a failure occurred, where it occurred,
and what kind of failure has occurred. This information must then be communicated to
an operator or system that takes appropriate action. In the second case of prediction,
it is about knowing the event before it occurs. Here, machine learning algorithms have
demonstrated very fast processing times once they have been properly trained [15].
Predicting failure is one of the most important challenges facing smart networks
today. The intrinsic complexity of the real-world energy distribution lines, which are
highly nonlinear and time-variable systems, makes this a highly complex task.
The studies are mainly focused on the prevention of power cuts originating from
damage to some physical components of the network and network overloads [12]. It
is also evident that there are external factors, such as weather conditions or accidents,
that cause damage to network components, which also affect the electrical operation.
Nowadays, with the architecture of modern intelligent networks, rapid failure diagnosis
and identification are essential for locating the faulty elements and their position in the
power grid [16].
With the increasing complexity of power grids, the need for a shift from a reac-
tive to a proactive network management paradigm is increasing and is being achieved
with advanced monitoring tools, data analysis, and prediction methods, including those
associated with artificial intelligence [11].
Progressively, smart grids are playing a central role in critical infrastructures, espe-
cially with the industry 4.0 revolution. They have become more dependent on connec-
tivity by supporting new communication and remote-control features [17]. However, it
is inevitable that this consequently expands the risks of cyber-attacks resulting from the
appearance of accidental or intentional network failures.
It becomes clear that SG are commonly conceptualized as industrial cyber-physical
systems (ICPS) comprising diverse interoperable components. These systems are recog-
nized for their potential to significantly affect society, the economy, and other essential
infrastructure in the event of failure.
The authors of the publications covered three areas (Table 2): bug identification and
recognition (IRF), bug prediction (PF), big data, and optimization, which discuss various
techniques useful in the prediction of failures in smart grids. Big Data concepts and tools
are already considered a cross-sectional element in smart networks, with an application
perspective in physical cyber systems for pattern recognition, event prediction, and
network operations optimization.
Table 2. Topics in revised documents
Topic Publication/Paper
IRF [18, 16, 19, 20, 12, 21, 22, 23, 24 13, 25, 26]
PF [27, 11, 28, 29, 30, 15, 31]
PF (Application with Big Data) [32, 33]
PF (Prediction and optimization) [34–37]
3.1 Failure Identification and Recognition (IRF)
Regarding energy distribution networks, some models of fault identification and loca-
tion are studied [16], mainly focused on protection systems such as relays (circuit break-
ers) and generally start from a SCADA system that, through a remote terminal unit (RTU)
and communication channels, receives and evaluates data. Thus, the proposed fault diag-
nosis method is usually: 1) the development of an appropriate PN model (Petri Net) for
fault detection; 2) the detection of data transmission failures; and 3) the identification of
faults in the electrical system.
Most of the existing methods of failure diagnosis in intelligent distribution networks
are mainly focused on the information of the relays and protection switches, but there is
incomplete or uncertain information in the process of receiving the data [20], so relying
only on that information would lead to erroneous conclusions. An improvement in the
accuracy of the fault diagnosis methods is to use approximate sets combined with genetic
algorithms such as the global optimization process (GA) and Tabu search (TS), which
have a strong local search capacity, with the aim of reducing the complexity of the data.
Information about the fault location, load type, protection device configuration, and
previously recorded faults is also essential. With the availability of modern technologies
such as optical transducers instead of conventional current transformers and current
voltage transformers, wave-based methods [22] are considered for future intelligent
networks. This requires the use of advanced sensors with high sampling rates for greater
accuracy and reliability.
In the study of the identification of failures, the location of the failure in the network
is crucial; this location may be a geographical location in urban or rural areas. High
impedance failures (HIF) in rural areas may not be detected by conventional protection
schemes [19]. For this reason, the proposals are oriented to consider the overlapping of
signals in electric lines in a frequency band that can be used by electricity companies
for the transmission of data. The idea is to detect significant deviations between normal
entry impedance and failure impedance for the selected frequency range.
Regarding power transmission network failures, the proposal that can be made
visible is the use of failure classifiers, using the Discrete Wavelet Transform (DWT) to
extract key characteristics of the voltage and current wave forms captured by a failure
recorder by multi-resolution decomposition. This feature extraction helps to decrease
the volume of data and, thus, the computational load. Finally, the extracted data is
classified by an intelligent decision tree-based classifier (DT) [24] [13]. DWT can be
combined with an Extreme Learning Machine (ELM), which classifies the data set for
high detection performance.
About high-voltage networks, the failure diagnostic study for HVCB (high-voltage
circuit breaker) devices is identified [23]. The idea is to structure a hybrid fault diagnosis
model using KPCA (kernel main component analysis).
The SPE (square prediction error) monitor detects HVCB failures, and then KFCM
(kernel fuzzy C-means) is used to judge whether the test sample is a new failure or not.
In the meantime, the VXB index (Xie-Beni index) is adopted to estimate the validity
of the optimal grouping results. Finally, support vector machines (SVMs) combined
with KFCM are used to establish a self-learning and recognition mechanism. If the test
sample is a new fault, it will be considered the new knowledge used to update the fault
recognition module.
The combination of the k-means algorithm, sensitive to the cluster initialization
strategy in the learning phase, combined with SVM to perform fault recognition proved
to be useful tools for the modeling of failure recognition in smart grid [21, 25].
Preventing network shutdowns or collapses is addressed by considering, for exam-
ple, the increase in power demand and directing efforts towards the detection and local-
ization of possible voltage collapse points [18]. An example in this field is the use of
techniques based on Ant Colony Optimization (ACO) developed to calculate the location
of the nearest voltage crash point.
Table 3 summarizes the methods identified. It should be considered that it is neces-
sary to evaluate the limitations of each method used; these limitations may originate in
the architecture or network topology on which failures are wanted to be identified or pre-
vented. Designing experiments to compare and evaluate methods with real or synthetic
data can be very useful.
Table 3. Summary on Failure Identification and Recognition
Purpose Model /Method /Tool

Detect and prevent SG failures Overlapping of signals on electric lines, PN, GA, Tabu search,
WT / DWT, Symlet wavelet, DT, ELM, ACO, SVM, KPCA,
KFCM, VXB, K-means
3.2 Failure Prediction (PF)
The trend is an advance towards a proactive model of predicting general failures or

shutdowns to form smart network early warning systems. Among the revised studies,
approaches vary due to the heterogeneity of the networks being studied.
They highlight models that evaluate the performance of the system in a probabilistic,
stable, and dynamic state, preparing a historical database for normal and cascading failure
states. For this, a SVM is trained with the historic database, and the SVM output is used
to initiate emergency control systems in the prevention of shutdowns [28].
Another approach is algorithms that analyze the worst heuristic configurations of
the electrical network; for example, the instanton-amoeba scheme [27] was originally
developed to study low-probability events and error correction failures. One finding is
that if the normal operating mode of the network is healthy enough, failure modes, also
called instantons, are rare enough; that is, failures are caused by load fluctuations on
only a few buses. The technique is useful for discovering weak links that are saturated at
instantons. It can also identify behaviors related to the work of generators, thus providing
predictive capacity to improve the reliability of any electrical network.
Other models relate the parts of the system with the measurements of phasor mea-
surement units (PMU) based on knowledge of the network topology to identify the parts
of the system where, according to historical records, disturbances are the most frequent
and thus be able to establish a relationship between the parts of the system in terms
of disturbance propagation. The main use of this model is the preliminary selection
of the most indicative characteristics of a fault (line trips, generator faults, cybernetic
infrastructure faults, and interdependent faults). In the first stage, these events must be
adequately identified, classified, and described with a range of related feature values.
In a second stage, this data set is analyzed, and a predictor based on machine learning
(neural networks, SVM) allows determining the probability of a network failure [11].
Intelligent electrical meters are an object of great interest. It is sought to identify
the most common attributes that affect these devices: environmental conditions, state
of electricity consumption, reliability, operating time, abnormal frequency, and mainte-
nance. The prediction model in this case is based on historical failure information with
the C5.0 algorithm for data mining of the training set, generating a failure decision tree
with a set of prediction rules [29].
The mean time to failure (MTTF) of physical elements of the network that suffer
from the passage of electricity due to the unidirectional passage of current is identified as
a factor of potential effect on the network. The proposal in the prediction is an automatic
learning model using neural network (NN) regression [31].
Another mechanism to predict failures is to learn the model of the states of an elec-
trical network, classifying them as failures or normal operating states, from a historical
data set of the real world that contains labeled states of the electrical network. Supervised
versions of evolutionary agent-based clustering (E-ABC) [30] work in this approach.
Table 4. Failure Prediction Summary
Purpose Model /Method /Tool

Failure Prediction Machine Learning, Instanton-amoeba, SVM, NN, C5.0, data mining, DT,
Regression of Neural Networks, Deep Learning, Boltzman Machine
3.3 Cross-Cutting Themes: Big Data; Prediction and Optimization; Education

The new generation of artificial intelligence (AI), called AI 2.0, includes ML (machine
learning), big data, cloud computing, the IoT (Internet of Things), and other technologies.
This field of computer science has recently become a focus of research. Data-based
AI will accelerate the development of intelligent power and electricity systems (Smart
EEPS). In AI 2.0, machine learning (ML) forms a typical category of representative
algorithm used for predictions and judgments, analyzing, and learning from massive
amounts of historical and synthetic data to help people make optimal decisions. ML
has been applied in the fields of SG and energy internet (EI), which are important
representatives of smart EEPS. AI 2.0, especially ML, is experiencing a critical period
of rapid development worldwide and will play an essential role in smart EEPS [37].
Failure prediction is leveraging on big data technology and methods. The reason
is that smart grids are constantly growing and incorporating more and more intelligent
meters in highly populated areas. It is a real challenge to store this data and perform an
efficient analysis of the records generated by smart meters [33]. There is talk of billions
of records to be stored and analyzed daily.
Dynamic energy management (DEM) in smart networks is a big data problem that,
faced with the extreme size of the data, requires high computational performance,
the adoption of advanced data analysis, powerful data set processing techniques, and
monitoring [32]. We will see more and more applications aimed at providing greater
intelligence to power grids supported by big data mechanisms.
Failure prediction has evolved from methods based on neural networks; deep learning
techniques are expected to increase prediction accuracy by being stochastic and allow-
ing for bidirectional connections between neurons [35]. Some research results related
to smart grids confirm that deep neural networks (DNNs) make accurate and robust
predictions compared to other prediction models [37].
Knowing and participating in the advances of the modern power grid is a global trend,
where physical infrastructure and computational cyber infrastructure must coordinate
to ensure an efficient and reliable electric grid [34]. It is imperative that engineering
and technology schools participate and develop strategies that enable students to take
advantage of the challenges and opportunities offered by globalization [38], part of
which is the electricity sector and applications related to smart grids. The SGs require
the participation of multidisciplinary teams with specialists in computer science, com-
puting, mathematics, electrical engineering, and electronic engineering. This is aligned
with the aspect of STEM studies in which the construction of the curriculum includes
interdisciplinarian integration around big ideas and longitudinal connections [39].
4 Discussion
The technology infrastructure of smart grids [1, 9] is the basis for visualizing the special-
ties coming from STEM careers that contribute to the development and implementation
of solutions and the execution of related projects.
The field of action of the SG where the multidisciplinary working teams intervene is
wide [4]. There are three differentiated levels that are usually integrated: the generation,
transmission, and distribution of energy, each with its own specific problem that is not
exempt from incorporating failure management in its operation.
There is a fourth level, that of network users or consumers, in which no special
interest is identified in the study of failure prediction; here, the tendency to investigate
the detection of anomalies is visible. This subject can be considered a new line of
research.
Smart grids, in addition to investment and related projects, will grow in the next
30 years [5], so a greater demand is expected for professions related to science and
technology applied to electrical energy, including artificial intelligence [11].
One of the limitations inherent in this study is its narrow scope, as it primarily con-
centrates on a highly particular implementation of artificial intelligence. Simultaneously,
the nature, amalgamation, and wide range of the suggested solutions posed challenges
in ascertaining the specific areas that ought to be explored in STEM careers within this
context.
This review demonstrates a notable propensity for failure prediction in the energy
sector. One recommendation is for educational institutions to provide a comprehensive
curriculum that incorporates applied artificial intelligence, exploring the development
of failure prediction models that transcend specific industries.
5 Conclusions
This document shows the path that the electricity industry is indicating regarding the
identification and prediction of failures in intelligent electrical networks. Regardless of
the country, the new electrical engineers, software and computer engineers, electronics,
telecommunications, and mathematicians who work in the field of intelligent networks
must be able to collaborate in multidisciplinary and interdisciplinary teams and to know
the different trends around the methods and tools of artificial intelligence and their
practical application, for example, around predicting failure events.
Over the past decade, the integration of the Internet of Things, along with the associ-
ated tools, methodologies, and technologies for big data, has become an integral element
within smart grid systems. These advances have been included in solutions designed
to predict failures within such systems. Several techniques are also used in combi-
nation, such as machine learning algorithms, neural networks, data mining, and deep
learning, among others. This is a potential new research line for further exploration
in STEM professions, as the diverse nature of communication components, network
designs, software providers, information management technologies, and geographical
locations where smart networks are deployed require complex solutions. In this context,
the highlighted efforts in this field exhibit a multidisciplinary and interdisciplinarity
approach, which requires the involvement of specialists from different countries and
cultural backgrounds.
Cybersecurity in industrial environments is a relevant issue in smart grids. The study
of the detection of anomalies from cyber-attacks or from the same conditions of the
operation of the electrical network that can lead to failures over time is an opportunity to
contribute to the development of specialists who can create prediction models appropriate
to each reality in which the smart grid is operating.
References
1. Borlase, S.: Smart Grids: Infrastructure, Technology, and Solutions. CRC Press, Boca Raton
(2013)
2. IEA, SMart Grids, París (2011)
3. Wietfeld, C., Cardenas, A., Chen, H., Popovski, P., Wong, Y.V.: Smart Grids. IEEE Wirel.
Commun. 24(2), 8–9 (2017)
4. Belu, R.: Smart Grid Fundamentals Energy Generation, Transmission and Distribution. CRC
Press, Taylor & Francis Group, Boca Raton (2022)
5. Agency International Energy. World Energy Outlook 2022. IEA Publications, Paris (2022)
6. Schätz, B., et al.: Cyphers. [En línea]. Available: http://cyphers.eu/sites/default/files/d6.1+2-
report.pdf. (2013). [Último acceso: 2022]
7. Borlase, S.: Smart Grids Advanced Technologies and Solutions. CRC Press, Taylor & Francis
Group, Boca Raton (2018)
8. Deming, D.J., Noray, K.L.: STEM Careers and the Changing Skill Requirements of Work,
National Bureau Of Economic Research, J24 (2019)
9. NIST. The National Institute of Standards and Technology, U.S. Despartment of Commerce,
[En línea]. Available: https://www.nist.gov/. [Último acceso: 03 2023]
10. United States Department of Energy, Smart Grid System Report 2020, U.S. Department of
Energy, Washington, DC (2022)
11. Kaitovic, I., Lukovic, S., Malek, M.: Proactive failure management in smart grids for improved
resilience: a methodology for failure prediction and mitigation. In: 2015 IEEE Globecom
Workshops, GC Wkshps 2015 - Proceedings, pp. 1–6 (2015)
12. Santis, E.D., Livi, L., Mascioli, F.M.F., Sadeghian, A., Rizzi, A.: Fault recognition in
smart grids by a one-class classification approach. In: Proceedings of the International Joint
Conference on Neural Networks, pp. 1949–1956 (2014)
13. Jana, S.A. De: Transmission line fault pattern recognition using decision tree based smart
fault classifier in a large power network. In: 2017 IEEE Calcutta Conference, CALCON 2017
- Proceedings, pp. 387–391 (2018)
14. Unterkalmsteiner, M., Gorschek, T., Islam, A.K.M.M., Cheng, Chow Kian, Permadi, R.B.,
Feldt, R.: Evaluation and measurement of software process improvement—a systematic
literature review. IEEE Trans. Software Eng. 38(2), 398–424 (2012)
15. Andresen, C.A., Torsaeter, B.N., Haugdal, H., Uhlen, K.: Fault detection and prediction
in smart grids. In: 9th IEEE International Workshop on Applied Measurements for Power
Systems, AMPS (2018)
16. Calderaro, V., Hadjicostis, C.N., Piccolo, A., Siano, P.: Failure identification in smart grids
based on Petri net modeling. IEEE Trans. Ind. Electron. 58(10), 4613–4623 (2011)
17. Khaled, A., Ouchani, S., Tari, Z., Drira, K.: Assessing the severity of smart attacks in industrial
cyber-physical systems. ACM Trans. Cyber-Phys. Syst. 5(1), 1–28 (2020)
18. Church, C., Morsi, W.G., El-Hawary, M.E., Diduch, C.P., Chang, L.C.: Voltage collapse
detection using ant colony optimization for smart grid applications. Electr. Power Syst. Res.
8(81), 1723–1730 (2011)
19. Milioudis, A.N., Andreou, G.T., Labridis, D.P.: Enhanced protection scheme for smart grids
using power line communications techniques – part I: detection of high impedance fault
occurrence. IEEE Trans. Smart Grid 3(4), 1621–1630 (2012)
20. Sun, Q., Wang, C., Wang, Z.: A fault diagnosis method of smart grid based on rough sets
combined with genetic algorithm and tabu search. Neural Comput. Applic. (Springer) 23,
2023–2029 (2013)
21. De Santis, E., Livi, L., Sadeghian, A., Rizzi, A.: Modeling and recognition of smart
grid faults by a combined approach of dissimilarity learning and one-class classification.
Neurocomputing 170, 368–383 (2015)
22. Dhend, M.H., Chile, R.H.: Fault diagnosis of smart grid distribution system by using smart
sensors and symlet wavelet function. J. Electron. Test.: Theor. Appl. (JETTA) 3(33), 329–338
(2017)
23. Fei, M., Yi, P., Kedong, Z., Jianyong, Z.: On-line hybrid fault diagnosis method for high
voltage circuit breaker. J. Intell. Fuzzy Syst. 5(33), 2763–2774 (2017)
24. Ucar, F., Alcin, O.F., Dandil, B., Ata, F.: Power quality event detection using a fast extreme
learning machine. Energies 1(11), 1–14 (2018)
25. De Santis, E., Rizzi, A., Sadeghian, A.: A cluster-based dissimilarity learning approach for
localized fault classification in Smart Grids. Swarm Evol. Comput. 39, 267–278 (2017)
26. Liu, B.I.N., Mao, Y., Wang, J., Fan, Y.: A dependable time series analytic framework for
cyber-physical systems of IoT-based smart grid. ACM Trans. Cyber-Phys. Syst. 1(3), 1–18
(2018)
27. Chertkov, M., Pan, F., Stepanov, M.G.: Predicting failures in power grids: the case of static
overloads. IEEE Trans. Smart Grid 1(2), 150–160 (2011)
28. Gupta, S., Kambli, R., Wagh, S., Kazi, F.: Support-vector-machine-based proactive cascade
prediction in smart grid using probabilistic framework. IEEE Trans. Ind. Electron. 62(4),
2478–2486 (2015)
29. Jincheng, Y., Ping, J., Guangyu, C., Tiejiang, Y., Fei, X.: Application of C5.0 algorithm
in failure prediction of smart meters. In: 2016 13th International Computer Conference on
Wavelet Active Media Technology and Information Processing, ICCWAMTIP 2017, pp. 328–
333 (2017)
30. Giampieri, M., De Santis, E., Rizzi, A., Mascioli, F.M.F.: A supervised classification system
based on evolutive multi-agent clustering for smart grids faults prediction. In: Proceedings of
the International Joint Conference on Neural Networks, pp. 1–8 (2018)
31. Dey, S., Nandi, S., Trivedi, G.: Machine learning approach for fast electromigration aware
aging prediction in incremental design of large scale on-chip power grid network. ACM Trans.
Des. Autom. Electron. Syst. 25(5), 1–29 (2020)
32. Diamantoulakis, P.D., Kapinas, V.M., Karagiannidis, G.K.: Big data analytics for dynamic
energy management in smart grids. Big Data Res. 2(3), 94–101 (2015)
33. Liu, Y., Hu, S., Rabl, T., Liu, W.: DGFIndex for smart grid: enhancing hive with a cost-effective
multidimensional range index. VLDB Endowment 7, 1496–1507 (2014)
34. Simmhan, Y., et al.: Cloud-based software platform for big data analytics in smart grids.
Comput. Sci. Eng. 15(4), 38–47 (2013)
35. Mocanu, E.: Machine learning applied to smart grids. [En línea]. Available: https://research.
tue.nl/en/publications/machine-learning-applied-to-smart-grids. (2017). [Último acceso: 01
05 2021]
36. Shi, L., Dai, Q., Ni, Y.: Cyber–physical interactions in power systems: a review of models,
methods, and applications. Electric Power Syst. Res. 163, 396–412 (2018)
37. Cheng, L., Yu, T.: A new generation of AI: a review and perspective on machine learning
technologies applied to smart energy and electric power systems. Int. J. Energy Res. 6(43),
1928–1973 (2019)
38. Ayokanmbi, F.M.: Competencies for global engineers and technologists. Ind. Technol. 27(1),
1–6 (2011)
39. Hu, W., Guo, X.: Toward the development of key competencies: a conceptual framework for
the STEM curriculum design and a case study. Front. Educ. 6, 684265 (2021)
LeSeVen: A Web Application for Learning
Written Vocabulary Through the Venezuelan
Sign Language
Jesús Pérez1(B) , Yajaira Nuñez2 , and Jesús Pérez-Pérez1

1 LaSDAI, Universidad de Los Andes, Mérida, Venezuela
jesuspangulo@ula.ve
2 Unidad Educativa Especial Bolivariana “Ofelia Tancredi de Corredor”, Mérida, Venezuela
Abstract. In special education, people with hearing impairment are taught written
language as well as sign language to ensure inclusion in society with equal oppor-
tunities as hearing people. Some applications facilitate the learning of both lan-
guages, however, in the educational context of primary schools, an application that
meets the following requirements does not exist or is not available: organization
of vocabulary by school grade, demonstrations of Venezuelan signs, evaluation of
students’ knowledge, and possibility of supervision by teachers. To cover these
requirements, this paper proposes a web application called LeSeVen. The ADDIE
(Analysis, Design, Development, Implementation, and Evaluation) instructional
design model served as a methodology to build and evaluate the impact of LeSeVen
on both students and teachers. To evaluate the impact, an acceptance questionnaire
was applied based on the UTAUT model (Unified Theory of Acceptance and Use
of Technology) and a quasi-experimental design to measure learning gain in stu-
dents from a primary school of special education. The results indicate that both
teachers and students accepted the web application given their favorable responses
about the constructs: attitude, intention to use, adaptability, enjoyment, ease of use,
and trust. In addition, a gain was observed in the learning of the students of all the
evaluated grades.
Keywords: Web Application · Educational Tool · Special Requirements ·

Hearing Impairment · Sign Language
1 Introduction
Due to education should prepare hearing-impaired children for an inclusive environment
in which they should not be disadvantaged by their hearing peers [1], it is common to use
the Logogenia method to teach them to be literate [2]. This method claims that hearing-
impaired children can learn a written language simply by being exposed to it through the
visual channel. For that reason, sign language and written language are taught together.
Given that ICTs (Information and Communication Technologies) have demonstrated
their positive impact on education [3], and that children with different disabilities also
respond positively to the use of ICTs in the classroom [4], several proposals have been
https://doi.org/10.1007/978-981-99-7353-8_16
LeSeVen: A Web Application 201
made based on ICTs. In a review of the literature on the use of ICT in children with hearing
impairment, the authors [5] found that the majority (35%) of ICT-based proposals pursue
vocabulary teaching. The other proposals are related to mathematics (18%), grammar
(12%), music (8%), sounds (6%), and others (21%).
The interest in our work is the same as in most, the learning of vocabulary. In this
case, the main challenge is that sign language is not a universal language, and therefore
the sign that represents a word can vary between countries and even between deaf
communities in the same country. In this context, a well-known proposal to promote
the organization and learning of vocabulary is the Spread Signs application [6], which
is considered the largest sign language dictionary in the world with more than 200,000
signs from different countries. However, Spread Signs has limitations. On the one hand,
the application offers a free version that includes words with their corresponding videos
in sign language but requires the paid version to show the list of complete words by
country. On the other hand, the application does not cover all countries, for example, in
Latin America it only includes Mexico.
The variability of signs to express words and the lack of vocabulary have moti-
vated the development of various ICT-based proposals in Latin America. For exam-
ple, in Colombia, ATPLIS [7]; in Ecuador, Software de Lengua de Señas [8], and
Glosario Básico Virtual de Lengua de Señas [9]; in Mexico, Dilo en Señas [10], and
Aprende Señas: LSM [11]; in Nicaragua, Señas Nicas [12]; in Panama, Lengua de
Señas Panameña [13], IPHE Inclusivo [14], and ENSENIAS [15]; and in Venezuela,
Mis Primeras Señas [16].
Although Mis Primeras Señas exists in Venezuela, it presents at least four limitations
for the school context. First, the words are not organized by school grade, second, it does
not allow evaluations, third, it does not allow supervision of students, and fourth, there
are not all the signs that are used in different cities. To cover these limitations, we
propose the LeSeVen web application as an alternative that better fits the requirements
of the school context.
The rest of the paper is organized as follows: Sect. 2 details the methodology used to
build LeSeVen; in Sect. 3, the results of the impact of LeSeVen on students and teachers
of a primary school for children with hearing impairment are presented; in Sect. 4, a
comparison is made with related works in Latin America; and finally, in Sect. 5, the
conclusions and future work are presented.
2 Methodology
The ADDIE (Analysis, Design, Development, Implementation, and Evaluation) instruc-

tional design model served as the methodology to build and evaluate the LeSeVen web
application. According to [17], ADDIE is a methodology that can be used to integrate and
implement some software in the teaching-learning process, because it allows defining the
target audience and selecting the study sample to perform measurements before and after
their integration into the learning environment. The most relevant aspects considered at
each stage are described below.
202 J. Pérez et al.
2.1 Analysis
A sign is associated with seven basic parameters [15]: configuration of the hand (palm
up, down, or towards the signer), place of articulation (trunk, left arm, right arm, head),
plane (contact with the body, moderately away from the body, away from the body,
very far away), hand movement (rotary, straight, inverted), orientation (up, down, back,
forward), point of contact (fingers, palm, the other hand, other part of the body), and
non-manual component (facial expression, trunk movement, shoulders). For that reason,
the signs must be recorded on videos that show, at least, the upper part of the body. In
addition, in this work, it is considered that an image allusive to the word is very useful
for the learning process by association so that each word will have its associated video
of the execution of the sign and an image allusive to it.
The objective of LeSeVen is to complement the teaching of vocabulary to children
with hearing impairments in primary education. The vocabulary to be taught is organized
by school grades (preschool, first, second, third, fourth, fifth, and sixth grade) and each
grade by categories (food, clothing, etc.). LeSeVen should contain all the vocabulary,
allow students to browse the vocabulary of their choice and evaluate their knowledge of
a school grade by category, and provide teachers with statistics on student activity in the
web application. In Table 1, the functional and non-functional requirements are presented
with their respective code. On the one hand, LeSeVen must be a web application (R1)
so that it is accessible from any web browser and allows the use of devices such as
personal computers, tablets, or smartphones. On the other hand, it should have a simple
user interface (R2) so that students are not distracted. To differentiate the statistics of
each student (R7) and the students associated with each teacher (R8), it must be possible
to log in (R3) with two user roles: student and teacher. For allowing the student to look
at the vocabulary of their choice, lists of categories must be presented, and then lists
of words by category (R4), and show the three elements (R5) when the student selects
a word. To evaluate the student’s knowledge (R6), the image must be shown, and the
student must select the correct word from three options. Finally, the teacher must be able
to observe statistics (R9) about the student’s activity and the results of the evaluations.
2.2 Design
LeSeVen was divided into three modules: learning, evaluation, and supervision. The
purpose of the learning module is to offer children the possibility of looking at the
vocabulary of their choice organized by categories belonging to a school grade. The
child selects a category, and then, from a list of words associated with that category,
selects one of them to display its three elements. The evaluation module implements a
system for children to evaluate the vocabulary they have learned by categories. For each
evaluation, the child selects a category belonging to a grade, and the system will choose
10 words at random. For each word, the image and three possible words in text format
are presented for the child to select the correct word. For each answer, the system will
show the child a congratulatory message when the answer is correct or display the three
elements of the word in a window when the answer is wrong. The supervision module
provides teachers with statistics on the use that each student makes of the learning and
evaluation modules. In this way, for each child, the teacher will have data that will allow
him to determine strategies to improve the quality of learning.
Table 1. Functional and non-functional requirements.
Code Requirement Type

R1 Web Application Not functional
R2 Distraction-free interface Not functional
R3 User registration and login Functional
R4 Visualization of lists of words Functional
R5 Visualization of a single word Functional
R6 Evaluation by school grade Functional
R7 Student interaction log Functional
R8 Assignment of students to teachers Functional
R9 Visualization of student statistics Functional
2.3 Development
We selected the PERN stack to develop the web application because it is a good com-
bination for reliable, scalable, and performant full-stack solutions. PERN stack consists
of four components: PostgreSQL, Express, React, and Node.js. In this stack, data flows
from the PostgreSQL database to the Express back-end, where it is processed and sent
to the React front-end through Node.js, and finally, the front-end displays the data to
the user. The development was divided into weekly iterations to present advances to
the coordinator of the educational institution to validate the operation of the software
modules developed and determine if any changes were necessary. Upon completion,
the application was deployed on a web server so that it could be accessed through web
browsers.
2.4 Implementation
As an exploratory study, a quasi-experimental design was applied to the students at
a special school that is dedicated to the primary education of children with hearing
impairment. The objective is to explore whether children can learn new vocabulary
by using the LeSeVen web application. In this stage, all the teachers (15) and all the
students (27) of the special school were involved. Figure 1 shows the configuration of
the workspace that allowed three students to attend simultaneously under the permanent
supervision of at least 2 teachers located behind them.
2.5 Evaluation
Learning Gain. To determine if children can learn new vocabulary by using the
LeSeVen web application, a pre-test was applied to know their initial mastery of a
set of words, then a session of use of LeSeVen was allowed, and finally, a post-test was
applied. The pre-test consisted of 10 random words from 3 categories belonging to the
corresponding school grade to answer in 15 min. For each word, the child was shown an
Fig. 1. Workspace configuration.
image and three words in text format for the child to select the correct word. The session
of use allowed the exploration of 15 words from these 3 categories for 30 min so that the
child could see the relationship between words, images, and signs. The post-test again
evaluated 10 random words from the same categories. The children were allowed to use
the application with complete autonomy, except for preschool children who received per-
manent assistance from the teacher, that is, each child told the answers and the teacher
marked in LeSeVen. It is important to mention that first-grade children, in addition to
their hearing impairment, have very strong cognitive compromises, therefore, they were
not able to follow the instructions to use LeSeVen, even with assistance.
Technology Acceptance. To determine if the users accepted LeSeVen and to know the
expectations of use, a questionnaire was designed based on the UTAUT model (Unified
Theory of Acceptance and Use of Technology), which predicts how well a system will
be accepted by a certain user group [18]. Possible responses are: Strongly Agree, Agree,
Neutral, Disagree, and Strongly Disagree. As shown in Table 2, the first 6 questions of the
questionnaire are aimed at knowing the expectations of the students, and the last 6, are
towards the expectations of the teachers. Since children cannot answer the questionnaire
autonomously, it was decided to collect the opinions of the teachers to determine the
expectations of the students. Then, as the tests were applied to the children of the different
grades, the teachers of the grade to which the child belonged were asked to observe how
the child behaved. Afterward, the teachers used the application for at least 5 min and then
answered a questionnaire with their opinion about the use of the application from their
perspective as teachers. The questions about the students were answered by 13 teachers,
and 2 teachers were added to the questions about their perception of themselves.
Table 2. Technology acceptance questionnaire.
No. Construct Item

1 Attitude Children like the web application for learning sign language
2 Intention to use Children would use the web application throughout the school period
3 Adaptability The web application adapts to the interests of children as learners
4 Enjoyment Children enjoy using the web application
5 Ease of use The web application is easy to use for children as learners
6 Trust Children are confident while using the web application
7 Attitude I like the web application for teaching sign language
8 Intention to use I would use the web application throughout the school period
9 Adaptability The web application suits my interests as a teacher
10 Enjoyment I enjoy using the web application
11 Ease of use The web application is easy to use for me as a teacher
12 Trust I trust the web application as a means of teaching sign language
3 Results
3.1 LeSeVen
After logging in, a user with the student role has access to both the learning and evalu-
ation module. The learning module allows studying the words organized by grades and
categories. Figure 2 shows an example of a word to study, where the user is given three
options to view: sign, word, or image. This module tries to implement the Logogenia
method, that is, children learn written language simply by seeing the word, but relating it
to the corresponding allusive image and sign. The user can change the element as many
times as he deems necessary and can mark the word as learned when he deems it to be
the case.
On the other hand, the evaluation module allows a user to assess his knowledge of
the vocabulary belonging to a category of a school grade. As mentioned before, each
evaluation is composed of 10 random words, where an image is shown to the user, and
the user must select the word that is related to the image. Figure 3 shows an example
where the image is alluding to avocados, and the options are bullring, avocado, and
metro.
Fig. 2. Learning module.
Fig. 3. Evaluation module.
Finally, when logging in with the role of teacher, the user has access to the supervision
module, which allows him to monitor the use that students make of the application. The
teacher can see a list of all the students he is following. Then, by selecting one of them,
it is possible to see their statistics on the number of words studied (viewed), the number
of correct ones (words evaluated and answered correctly), and the hit rate (proportion
of correct words as a function of total words evaluated), as shown in Fig. 4.
Fig. 4. Supervision module.
3.2 Learning Gain
Figure 5 presents a comparison with box-and-whisker plots of the number of hits per
school grade. In all grades, a higher average was obtained in the post-test (see the
average of each test marked with an X in Fig. 5), indicating a learning gain in all grades.
Considering the averages, the highest gain was obtained in sixth grade, from 4.67 in
the pre-test to 7.67 in the post-test. On the other hand, the minimum value of correct
answers was always the same or higher in the post-test than in the pre-test in all grades,
showing that students maintain or improve their correct rate as expected in an educational
intervention. Particularly, in preschool and sixth grade, it was found that the minimum
value of the post-test was equal to the maximum value of the pre-test, revealing a greater
impact in these grades than in the others. The least impact was observed in second grade
because, although the average increased from 4.25 to 4.5, the minimum and maximum
values of the pre-test were maintained in the post-test.
Although the vocabulary presented to each student was appropriate for their grade, a
greater number of correct answers was observed in the last grades (fourth, fifth, and sixth),
indicating greater cognitive ability due to mastering more words of their corresponding
grade. The maximum possible number of 10 correct answers was only reached in fifth
grade for both the pre-test and the post-test. On the other hand, the minimum amount
of 1 hit occurred only in preschool. The greatest dispersion in the number of correct
answers was observed in fourth and fifth grade, indicating that they are the grades with
the greatest difference between students. On the contrary, the greatest similarity was
found in the number of correct answers in the second grade.
Fig. 5. Number of correct answers in the pre-test and post-test.
3.3 Technology Acceptance

Table 3 presents the results of the questions where the teachers provided their opinion
on the acceptance of the students. In general, the answers to all the constructs were
favorable because most of the answers were in Totally Agree (TA) and Agree (A).
The constructs attitude and enjoyment obtained the most favorable responses because
they were all Totally Agree or Agree. Regarding intention to use and adaptability, only
7.7% and 15.4%, respectively, provided a neutral response (N). Regarding ease of use
and trust, unfavorable Disagree (D) responses of 15.4% and 7.7%, respectively, were
obtained, indicating that teachers perceived some students as having difficulties in the
use of LeSeVen.
Table 3. Results of the student’s acceptance questionnaire.
Construct Item TA A N D
Attitude 1 46.2% 53.8%
Intention to use 2 61.5% 30.8% 7.7%
Adaptability 3 38.4% 46.2% 15.4%
Enjoyment 4 76.9% 23.1%
Ease of use 5 38.4% 30.8% 15.4% 15.4%
Trust 6 23.1% 53.8% 15.4% 7.7%
Table 4 presents the results of the questions where the teachers provided their opinion
about their acceptance. Like their perception of the students, the responses to all the
constructs were favorable because most of the responses were Totally Agree and Agree.
Only the constructs intention to use and adaptability received 6.7% of neutral responses,
the percentage corresponding to the response of only one teacher. On the other hand,
except for the enjoyment construct, all the constructs received the majority of Totally
Agree responses. Particularly, the enjoyment construct received 46.7% in Totally Agree.
Table 4. Results of the teachers’ acceptance questionnaire.
Construct Item TA A N D
Attitude 7 73.3% 26.7%
Intention to use 8 60.0% 33.3% 6.7%
Adaptability 9 60.0% 33.3% 6.7%
Enjoyment 10 46.7% 53.3%
Ease of use 11 66.7% 33.3%
Trust 12 66.7% 33.3%
4 Comparison with Related Works

Table 5 presents a comparison of applications for learning words and sign language
in Latin America. The comparison is made according to the categories in which the
words are organized, the elements that are associated with each word, the possibility
of applying evaluations (Eval.), and the option of supervision of student activities by
teachers (Sup.). Among the most common categories are the alphabet, colors, numbers,
animals, family, and food. Some uncommon categories can be found in Glosario Básico
Virtual de Lengua de Señas [9], where they incorporate nouns, adjectives, and verbs. In
contrast, our application groups the signs by school grade, making it easier for students
to use because it allows them to quickly find the words they need to learn.
In general, applications for learning sign language use relationships between the
written word, images alluding to the word, and videos of the execution of the sign.
The most common combination includes the written word and video of the sign. The
applications that use this combination are ATPLIS [7], Glosario Básico Virtual de Lengua
de Señas [9], IPHE Inclusivo [14], Aprende Señas: LSM [11], and ENSENIAS [15].
The applications that combine written words, images, and videos are Lengua de Señas
Panameña [13], Señas Nicas [12], Dilo en Señas [10], Mis Primeras Señas [16]. Our
application is in this category too. Another less common combination, such as the written
word and cartoon sign, can be seen in Software de Lengua de Señas [8].
Table 5. Comparison with related works.
Ref. Categories Elements Eval. Sup.

[8] Food, family, numbers, animals, Image, word, and cartoon sign No. No.
colors, alphabet
[13] Alphabet, numbers, colors, social Word, image, and video No. No.
relationships
[7] Letters, numbers, colors, days, Word and video Yes. No.
months, family, expressions,
clothing
[9] Nouns, adjectives, verbs Word and video Yes. No.
[12] Alphabet, anatomy, colors, Word, image, and video No. No.
directions, family, numbers,
greetings, jobs, verbs, animals,
food, emergency, geography,
clothing, weather, transportation
[10] Family, food, toys, animals, Word, image, and video No. No.
colors, numbers, alphabet
[14] Alphabet, numbers, months, days Word and video No. No.
[11] Alphabet, numbers, colors, Word and video Yes. No.
animals, professions, sports,
greetings, places, dates, clothing,
family, food
[16] Food, education, family, health, Word, image, and video No. No.
city, sports, music, religion,
transportation, time, space,
animals, plants, Venezuela, world
[15] Alphabet, math, location, Word and video No. No.
clothing, verbs, religion, animals,
association, institution, national,
agency, time, calendar,
computing
LeSeVen Preschool, first, second, third, Word, image, and video Yes. Yes.
fourth, fifth, and sixth grade
Some applications present the possibility of evaluating knowledge, such as ATPLIS

[7], Glosario Básico Virtual de Lengua de Señas [9], and Aprende Señas: LSM [11]. For
example, in [7], for a total of 4 signs, the video of the sign is shown, and the student must
select the correct option from a list of words. In contrast, in [9], for a total of 10 signs,
the image is shown, and the student must select the correct word among four options.
In our case, the image and three options are shown for the student to select the correct
word. Finally, none of the related applications allow you to monitor student activity. In
contrast, our application counts per student the words studied, the words evaluated, and
the hit rate.
5 Conclusions
LeSeVen successfully fulfilled its purpose of offering students vocabulary organized by

school grade. In addition, the exploratory results were encouraging. On the one hand,
it received a very favorable acceptance by teachers and students, being evidenced in
the answers obtained from the questionnaire applied to teachers where they responded
mostly Totally Agree and Agree to the questions related to the constructs attitude, inten-
tion to use, adaptability, enjoyment, ease of use, and trust. On the other hand, it showed
a positive impact on students of the different grades of primary education because the
results of the post-test were better than those in the pre-test, indicating learning gains
in all grades. Particularly, a greater positive impact was observed in students in the last
grades (fourth, fifth, and sixth) because the learning gain was greater.
In this study, limitations indicate some possible future work. First, all the students
who participated were diagnosed with hearing impairment, however, some of them have
additional conditions such as autism spectrum disorders. Future work should determine
the impact of LeSeVen according to the conditions of each participant to predict the user
profile required to obtain some significant measure of learning gain. In addition, it can
be very useful in this context to incorporate a user knowledge model to determine the
specific skills of each student, for example, the model based on reinforcement learning
proposed in [19]. Second, the exploratory test was conducted using three laptops in a
classroom under controlled conditions under teacher supervision. However, LeSeVen
is a web application that can be accessed from any device with an Internet connection,
such as phones or tablets. Furthermore, LeSeVen has a supervision module that would
facilitate, in future work, measuring the impact of LeSeVen when freely used by the
participants and studying the long-term learning gain.
Other future work could be aimed at extending the scope of LeSeVen. First, LeSeVen
allows consulting vocabulary organized by school grade, however, it does not implement
any pedagogical strategy that maximizes the learning gain. For example, flow theory [20]
is a strategy that would determine which words or categories a student should practice
staying in a state of interest and concentration. Therefore, future work should be oriented
to investigate which pedagogical strategies can maximize the learning gain according
to the student’s profile. Second, some AI-based features could increase the impact of
LeSeVen on students. For example, applications such as SignQuiz [21] use deep neural
networks to automatically recognize signs captured by a camera. Third, LeSeVen is
originally aimed at elementary school children, but they are not the only ones who need
to learn sign language since there are hearing people who also need to communicate with
them such as family members (mother, father, siblings) or the very hearing teachers who
teach them, therefore, future work should validate the impact of LeSeVen on hearing
people, and perhaps consider aspects of human-machine interaction [22] to optimize
interactions with LeSeVen, such as the incorporation of a user interface based on natural
language or the adaptation of the application based on the user’s emotions.
References
1. Flórez, L., Cano, S., Collazos, C., Solano, A., Slegers, K.: Collaborative learning as educa-
tional strategy for deaf children: a systematic literature review. In: Proceedings of the XVIII
International Conference on Human Computer Interaction, New York, USA (2017)
2. Radelli, B.: Una nueva aplicación de la lingüística: la logogenia. Dimensión Antropol. 23,
51–72 (2001)
3. Colás, M., De Pablos, J., Ballesta, J.: Incidencia de las TIC en la enseñanza en el sistema
educativo español: una revisión de la investigación. Rev. Educ. Distancia 18(56), 1–23 (2018)
4. Pérez, J., Azuaje, M., León, C., Pedroza, O.: Effects of social robotics on episodic memory
in children with intellectual disabilities. IEEE R. Iberoamericana Tecnologías Aprendizaje
16(5) (2021)
5. Izaguirre, E., Abásolo, M., Collazos, C.: Educational methodologies for hearing impaired chil-
dren supported by mobile technology and extended reality: systematic analysis of literature.
IEEE R. Iberoamericana Tecnologías Aprendizaje 16(4), 410–418 (2021)
6. Spread Signs, Google Play Apps. https://play.google.com/store/apps/details?id=com.spread
thesign.androidapp_paid&hl=en. Last accessed 15 July 2023
7. Hernández, C., Sánchez, J., Sánchez, G.: Herramienta tecnológica para el aprendizaje
autónomo de la lengua de señas. Revista Espacios 41(6) (2020)
8. Maiquiza, P.: Software educativo y la enseñanza del lenguaje de señas. Bachelor’s thesis,
Universidad Técnica de Ambato, Ecuador (2019)
9. Guacho, B.: Diseño e implementación de un software educativo para el aprendizaje de lengua
de señas ecuatorianas para docentes de educación general básica de la Escuela de Educación
General Básica Cnel. Oswaldo Vaca Lara. Bachelor’s thesis, Universidad Central de Ecuador,
Quito (2017)
10. Dilo en Señas, Google Play Apps. https://play.google.com/store/apps/details?id=com.diloen
senasexp.ecs&hl=en. Last accessed 15 July 2023
11. Aprende señas: LSM, Google Play Apps, https://play.google.com/store/apps/details?id=
rodolfo.com.systemsolutionanddevelopment.aprendiendolsm&hl=en. Last accessed 15 July
2023
12. Señas Nicas, Google Play Apps. https://play.google.com/store/apps/details?id=com.marcos
miranda.seniasnicas&hl=en. Last accessed 15 July 2023
13. Pimentel, M.: Sitio web para el aprendizaje de Lengua de Señas Panameñas. Master’s thesis,
Universidad de Panamá, Panamá (2018)
14. IPHE Inclusivo, Google Play Apps. https://play.google.com/store/apps/details?id=ca.costari.
apps.ipheinclusivo&hl=en. Last accessed 15 July 2023
15. Alaín, L.: Desarrollo y validación de una aplicación web y cd educativa inclusiva para el
aprendizaje de la lengua de señas panameña. Doctoral thesis, Universidad de Granada, Panamá
(2019).
16. Mis Primeras Señas: https://funvapedigital.alwaysdata.net/DLSV/index.php. Last accessed
2023/07/15.
17. Castellanos, H., Rocha, E.: Aplicación de ADDIE en el proceso de construcción de una
herramienta educativa distribuida b-learning. Revista Iberoamericana de Tecnología en
Educación y Educación en Tecnología 26, 10–19 (2020)
18. Heerink, M., Kröse, B., Evers, V., et al.: Assessing acceptance of assistive social agent
technology by older adults: the Almere Model. Int. J. Soc. Robot. 2, 361–375 (2010)
19. Pérez, J., Dapena, E., Aguilar, J., Carrillo, G.: Reinforcement learning for estimating student
proficiency in math word problems. In: 2022 XVII Latin American Conference on Learning
Technologies (LACLO), pp. 1–6, Armenia, Colombia (2022)
20. Shernoff, D., Csikszentmihalyi, M., Shneider, B., Shernoff, E.: Student engagement in high
school classrooms from the perspective of flow theory. School Psychol. Quart. 18(2), 158–176
(2003)
21. Joy, J., Balakrishnan, K., Sreeraj, M.: SignQuiz: a quiz based tool for learning fingerspelled
signs in Indian sign language using ASLR. IEEE Access 7, 28363–28371 (2019)
22. Pérez, J., Aguilar, J., Dapena, E.: MIHR: a human-robot interaction model. IEEE Latin Amer.
Trans. 18(9), 1521–1529 (2020)
Predicting Learners’ Performance
in a Programming Massive Open Online Course
Jorge Maldonado-Mahauad1,2(B) , Andrés Auquilla1 , Karina Abad2 ,

Jaime Veintimilla-Reyes1 , and Miguel Zúñiga-Prieto1
1 Departamento de Ciencias de la Computación, Facultad de Ingeniería, Universidad de Cuenca,
Cuenca, Ecuador
jorge.maldonado@ucuenca.edu.ec
2 Dirección de Innovación Educativa, Universidad de Cuenca, Cuenca, Ecuador
Abstract. In recent years, predictive models in Massive Open Online Courses

(MOOs) have mostly focused on predicting student success in cohort MOOC envi-
ronments which are designed with structured timing and planned content release.
However, in self-taught courses, which are characterized by their flexibility in
timing and release of content, predictions can be more critical because students’
success depends on their behavior during learning. Where, student behavior is
defined by the combination of complex variables that describe their interactions
with course resources. Therefore, existing models must be adapted in such a way as
to consider heterogeneity in student behavior. To address this need, this paper stud-
ies how student interactions with self-taught MOOC resources can be included
in predictive models. Twelve types of interactions with video-readings, assess-
ments and supplements are analyzed to measure their effect on predicting success
in a population of 38,838 students enrolled in a course. Additionally, this work
contributes to a methodology that aims to improve predicative models of stu-
dent performance in a course by identifying student profiles and their probability
of success. Results of this work show that the interactions of students with a
course have a high predictive power, among them the most relevant are complet-
ing video-readings, completing evaluations, and reviewing previously completed
supplements.
Keywords: Prediction · MOOCs · Learning Analytics · Predictive Methods
1 Introduction
The massive and open nature of Massive Open Online Courses (MOOCs) contributes to
attracting a great diversity of students, who have seen MOOCs as an opportunity for their
personal growth. Most students who enroll in a MOOC choose which course content to
interact with, and eventually only a small proportion of enrolled students complete the
course (typically less than 10%) [1]. This has sparked interest in studying the causes
why students complete or drop out of a MOOC.
Those enrolled in a MOOC present a diversity of behaviors depending on: knowledge,
previous experiences, intentions and motivations [2, 3]. On a MOOC platform, this
https://doi.org/10.1007/978-981-99-7353-8_17
Predicting Learners’ Performance 215
behavior is recorded as learners’ interactions with course content, generating a wealth

of data that offers an opportunity to identify patterns and predict trends [4]. In fact,
using all this data to run predictions about success in a MOOC is of special relevance.
Understanding the learning behavior of enrolled students can help detect students who
are “likely” not to pass the course [5]. In addition, this analysis could be used to better
understand how students work in the course and what kind of support they may need,
anticipating problems that may lead to dropout.
Several studies have tried to predict attrition, retention, and completion in MOOCs.
Most of these studies have been conducted in cohort MOOC settings (e.g., where time is
usually structured, students follow a fixed schedule, and course materials are released at
specific times). However, in self-taught MOOCs, prediction models can be more critical.
On the one hand, success in self-paced courses, without the support of an instructor,
depends on students’ ability to self-regulate their behavior [6]. On the other hand, student
behavior could be more variable, as they do not follow a strict schedule, all materials
are released when the course starts, and dates for assessments are flexible [7]. As a
consequence, detecting and predicting trends in MOOCs remains a challenge that has
been addressed in previous work with different approaches. For example, authors in
[8] developed a predictive grade method that uses characteristics of student activity to
forecast whether or not a student can obtain a certificate. The authors in [9] developed
a predictive model to understand when students will correctly answer a question.
In [10], the authors analyzed the relationship between interactions and the number
of days students interact with the content. Despite the predictive power of the proposed
models, these models raised some discussions in the community. On the one hand, some
researchers argue that frequency and event count are not the best metrics for getting prac-
tical indicators to explain differences in online learning [11]. On the other hand, existing
models rely on the use of low-level indicators of learners’ interaction with the course,
but this makes it difficult to obtain meaningful patterns of more complex behaviors,
such as the use of learning patterns [11]. Therefore, there is an opportunity to improve
these predictive models by considering data that reports both student heterogeneity (e.g.,
learning strategies), and more complex behaviors represented by sequences of activities
rather than individual events.
As a first proposal in this line, a methodology is defined for the determination of
student profiles and their probability of passing a course. The phases of the methodology
are then executed to present an exploratory study that uses learning interactions related to
student success as coarse-grained data to predict their behavior in a MOOC. Specifically,
we investigate whether or not students pass the course based on these patterns (or profiles)
along with iteration variables that define student behavior. As a result, we identified new
factors to improve predictive models of student success in MOOCs.
216 J. Maldonado-Mahauad et al.
2 Methodology
2.1 Context: Participants and Course
This exploratory study was conducted in the context of a MOOC called “Introduction
to Python Programming”. Offered in Spanish on the Coursera platform by the Pontificia
Universidad Católica de Chile between July 2017 and January 2018. The course is
structured through 6 modules. Each module is made up of a set of lessons and each
lesson is composed of a set of video-readings, theoretical and practical quizzes and
readings. In total the course has 35 video-readings (VL), 11 readings (L), 13 practical
questionnaires (QP) and 11 theoretical questionnaires (QT). All modules were available
during the 6 weeks of the course.
In the course, 38,838 (N = 38,838) students were registered, of which 10,916 did not
register any type of activity on the course. For the analysis, all students were considered,
except those who did not report any activity, leaving a cohort of 27,922 students who
made about 3.5 million interactions in the course. 15% of registered students were
between 18 and 24 years old, 50% were between 25 and 34 years old, and 22% were
between 35 and 44 years old. 64% of students reported being male and 34% female.
2.2 Variables
In this study, variables related to the student’s interactions with the MOOC resources
are used. Variables were obtained that reflect frequency counts, means, and others that
are detailed below. The concept of a study session was defined. A study session is a
period of time in which students interact with course resources and record continuous
activity, with inactivity intervals no longer than 45 min. I mean; that if the student did
not perform any activity or interaction with the course, the platform will terminate the
session and consider it as a new one. Additionally, the concept of interaction was defined.
An interaction is an action that is stored in the data traces recorded by Coursera and that
reflects the interaction of a student with any of the digital resources of the MOOC. For
this work, 12 types of interactions that the student can perform when interacting with
video-readings, readings and evaluations were defined. Table 1 presents the interactions
and their meanings.
Additionally, other variables were obtained such as: number of study sessions, fre-
quency, mean and standard deviation of formative evaluations, summative, supplements
and video-readings carried out for each student. The following were also obtained: the
average time of each student’s sessions, the total interaction time on the platform, the
number of active days on the platform, the percentage of video-readings, formative eval-
uations, summative and supplements completed. Additionally, the frequency, mean and
standard deviation of each of the interactions in Table 1 were obtained. Finally, the grade
obtained by the student and their status were added. For the state, a student’s success in
a MOOC was defined based on the grade the student achieves in the course. Therefore,
successful students (those who pass the course) include any student who gets at least the
minimum score to pass the course (80%) regardless of whether they address most course
materials (most common form of success). Figure 1 shows the distribution of students
who pass and fail the course. A total of 27,922 students have enrolled in this course of
which, 27,448 have not passed, while a total of 473 have.
Table 1. Defined interactions for MOOC resources.
Resource Interaction Meaning

Video-Lecture (VL) Begin Start a video-lecture
Complete Complete a video-lecture
Review Return to a VL you’ve already completed
Formative assessment (AF) Try Start a formative assessment
Pass Pass a formative assessment
Review Return to an AF you’ve already passed
Summative assessment (AS) Try Start a summative assessment
Pass Pass a summative assessment
Review Go back to an AS you’ve already passed
Supplement (SP) Begin Start a supplement
Complete Complete a supplement
Review Returns to an SP that has already been completed
Fig. 1. Distribution of the results of the students who have taken the course.
2.3 Proposed Approach

For the process of characterizing types of students and determining the probability that
they will pass the MOOC, this work proposes a methodology composed of interconnected
phases (Fig. 2).
The first phase in the proposed methodology involves the use of a technique called
“Feature Extraction”, where the original dataset is transformed into one with smaller
dimensions. The objective of feature extraction techniques is to discover the intrinsic
dimensionality of a dataset, in order to simplify the clustering and classification process
that is subsequently carried out. The dataset used in this work contains 54 numerical
Fig. 2. Proposed methodology for determining student profiles and their probability of passing a
course.
variables, where several of them have high correlation with each other and little cor-
relation with the variable that indicates whether a student passes or not. Due to their
performance, the following techniques were used for this phase: 1) Principal Compo-
nent Analysis (PCA), 2) Isomap, and 3) T-distributed Stochastic Neighbor Embedding
(TSNE). Subsequently, through a visual analysis, the technique that offered the greatest
clarity for the representation of the students’ data was selected.
Subsequently, with the dataset transformed to a lower dimensionality (two dimen-
sions), the Clustering phase is executed. For this work, the DBSCAN technique was
selected because it does not need to define the number of clusters as a parameter. In
addition, this technique is based on densities within the dataset and can generate clusters
where their number of elements is different. This feature is important, given the nature of
the problem, it is expected to have diverse groups of students of different sizes. Although
this technique requires a hyper-parameter tuning process, this does not represent greater
complexity.
With the clusters generated, we proceed with the process of Interpretation of them. In
this phase, the characteristics of each of the clusters obtained are identified. With this, the
conditions that a student must meet in order to be classified or placed within the cluster
in which most of the students who pass the course are located. In the Selection phase,
with the clusters generated, those that contain students who have passed the course are
selected. In this way, in the Classification phase, classification models are built trained
with the data of the students belonging to each cluster. For example, if there are two
clusters with students who pass the course, two classification models are created; each
trained with data from their respective clusters. The ideal case at this point is to find pure
clusters, i.e., clusters where there are only students who pass the subject. In this way, it
is possible to characterize the behavior of the students who are more likely to pass the
course. Finally, several classification models are created: Logistic Regression, Support
Vector Machine, Random Forest, and XGBoost to, with the data from the clusters,
determine the probability that a student will pass the course.
3 Results
As a result of the execution of the methodology proposed in this work, the following
results were obtained.
3.1 Transforming and Reducing Dimensions of the Original Dataset
The PCA, Isomap, and TSNE techniques were executed on the original dataset, which
was previously normalized for better results (Fig. 3).
Fig. 3. Two-dimensional distribution of the original dataset using three feature extraction tech-
niques. The only technique that produces a clear divide between students who pass and do not is
TSNE.
The goal of using feature selection techniques in the original dataset was to achieve a
transformation where the instances are in well-defined high-density regions; In addition,
where students who passed the course appear in clearly identifiable regions. Based on
these characteristics, the technique that allowed these objectives to be met to a greater
extent is TSNE (Fig. 3C). The transformation produced by PCA (Fig. 3A) failed to create
areas with instances of different density. Additionally, this transformation generated an
important dispersion in the group of students who passed the subject. In the case of the
transformation using Isomap (Fig. 3B), although the group of students who passed the
course is in a well-defined and high-density section, the students who did not pass are in
nearby regions in two-dimensional space. This could result in the clustering algorithm
failing to find the right clusters. TSNE was able to correctly generate dispersed regions
with high density (Fig. 3C). These high-density regions are easily recognized by a
clustering technique such as DBSCAN, thus representing the types of students present
in the dataset. In addition, students who passed the course are not only clearly identified
in a region with high density but are conveniently separated from the other regions.
3.2 Cluster Analysis
Based on the dataset transformed into a two-dimensional space, the DBSCAN algorithm
was able to identify 7 clusters of different sizes and densities (Fig. 4).
Fig. 4. Clustering resulting from applying DBSCAN
Among the resulting clusters, only cluster 7 (located to the right of Fig. 4, mustard
color) contains all the students who passed, except 2 who are in cluster 0 (orange). In
cluster 7 there are 572 students: 471 students who passed the course, and 101 who did
not. This result is a clear indication that the behavior of students who pass the course is
different from students who do not. Although there are 101 students who did not pass in
this cluster, the fact that a student belongs to this is a clear indication that he has more
chances of passing the course (82%).
Intuitively it can be thought that the behavior of the students who belong to this
cluster makes the chances of passing the course increase markedly.
In this dataset, the vast majority of students did not pass the course. In total there
were 27,922, of which only 473 approved. In percentage terms, only 1.7% of students
pass. However, as shown in Fig. 4, there are several types of students who reveal different
behaviors. These behaviors make a student more or less likely to pass a subject.
An analysis of the different clusters was performed. In Fig. 5, a comparative graph
of the number of students who have passed the course by clusters has been included.
Thus, it is clear to see that the majority of approved students have been classified in
cluster 7. A total of 471 students have been placed in cluster 7, giving a total of 99.58%.
Similarly, it can be seen that cluster 1 is located the largest number of students (22,634)
corresponding to 81.06%.
From Fig. 6A, it can be seen that the students who have been classified in cluster
7 are those who have spent the longest average time on the platform. In Fig. 6B you
can see that they have completed more video readings, they have also been classified
in cluster 7. In addition, those who have completed longer sessions, Fig. 6D, have been
placed in the cluster that has the most approved students. Looking at Fig. 6C, it can be
clearly seen that students who take the most assessments are the most likely to pass the
course.
Fig. 5. Distribution of students who have passed the course in each of the clusters.
3.3 Performance Prediction Analysis
Although the cluster where the students who passed the course are located has a high
purity, i.e. relationship between students who passed and did not, there is a small percent-
age of students who did not pass. Therefore, different classification models were created
and tested to have greater certainty about whether a student belonging to this cluster
passes the course or not. The models chosen for this task were: Logistic Regression,
Support Vector Machine, Random Forest, and XGBoost. The cluster instances were
divided into a training and test set, where 75% of the data was used for training and
the rest for testing. Being an unbalanced dataset, stratified sampling was used to main-
tain relationships between classes. The hyper-parameters of the models were selected
through a manual process of trial and error. The classification models were tested on the
test test and the balanced accuracy and F1 metrics were calculated (Table 2). This was
because the dataset was unbalanced. A linear model such as Logistic Regression did not
obtain satisfactory results; however, nonlinear models obtained optimal results. SVM
failed only one classification instance. Meanwhile, the Random Forest and XGBoost
models did not produce incorrect classifications. This is an indication that, even having
students with similar behaviors, Random Forest and XGBoost can correctly separate
those students who pass from those who do not.
Additionally, the most informative variables for the different models were identified;
in this way, those variables that help to better characterize the behavior of students who
pass the course can be determined.
To do this, the following feature selection algorithms were executed: Pearson Correla-
tion, Chi Squared, Recursive Feature Elimination (RFE), Random Forest (RF), XGBoost,
and Boruta. Based on this analysis, only 24 variables appear, at least once, as informative
variables for any of the methods previously described (Fig. 7). There are variables that
appear several times listed as informative by more than one method. In other cases, they
are listed as informative by only one method.
A B
C D
Fig. 6. Comparative tables of the variables and their distribution in each cluster.
Table 2. Grading results to determine whether a student passes the course.
Model Balanced Accuracy F1

Logistic Regression 59.5% 0.81
SVM 98% 0.99
Random Forest 100% 1.0
XGBoost 100% 1.0
To simplify the analysis process, the most informative variables that have been
selected by at least two feature selection methods are shown below (Table 3).
Fig. 7. More informative variables for the problem. Variables are represented in columns and
feature selection methods are represented in rows. A variable is important for a method if it has a
light color in the corresponding cell.
Table 3. Variables that most often appear as listed by feature selection methods as more
informative.
Variable Number of times it is selected

evaluaciones_Complete 5
modulos_Complete 5
frec_AS_Pass 4
frec_SP_Review 2
frec_VL_Begin 2
frec_AF_Pass 2
VL_Complete 2
4 Conclusions
This paper presents an exploratory analysis of the variables that define the learning
behavior of students enrolled in a self-taught MOOC course. Where, the behavior is
defined by the interactions (actions that are saved in the data traces registered by Cours-
era) of a student with any of the digital resources of the MOOC. Different predictive
models and algorithms were analyzed. The results show that not only variables related
to interactions with exercises are the best predictors as indicated in [12, 13]; but, other
interaction variables related to video readings, evaluations and supplementary resources
have great predictive power. These results suggest that other variables that define behav-
ior may be included if variables related to interaction with exercises are not available
or are scarce (for example, in a MOOC where interactions with exercises are limited to
certain modules of the course).
Additionally, this work contributes to a methodology that aims to improve predictive

models of student success in a course by identifying student profiles and their probabil-
ity of success. To discover the intrinsic dimensionality of the dataset and simplify the
process of clustering and classification, among all the applied techniques (PCA, Isomap,
and TSNE) the TSNE feature extraction technique was the only one that identified a clear
division between students who pass and not the course. As a result of applying DBSCAN
during clustering, 7 clusters were identified; where 99.58% of the students who passed
the course were in one of them (only 473 of 27,922 students who registered some inter-
action with the course approved it). The behavior of cluster students most likely to finish
the course was: longer average time on the platform, completed more video-readings,
perform more evaluations, have done longer sessions. With respect to the prediction
analysis among the applied classification models (Logistic Regression, Support Vec-
tor Machine, Random Forest, and XGBoost), the Random Forest and XGBoost models
achieved 100% accuracy (Balanced Accuracy). However, it is important to mention that
the accuracy of the Support Vector Machine classification model was 98%, while the
accuracy of the Logistic Regression model was 59.5%. Finally, the variables that help to
better characterize the behavior of the students who pass the course were identified (com-
pleted evaluations, completed modules, frequency of summative evaluations approved,
frequency of supplement resources reviewed after they have been completed, frequency
of video-readings initiated, among others).
The results of this study are subject to some limitations due to the nature of the
data, and methodological choices. First, the study is based on student behavior data
automatically collected by the platform. Secondly, study sessions are calculated taking
into account the inactivity threshold of 45 min. Future work will expand the study
by considering (1) week-by-week rather than sessional analysis, and (2) considering
interactions with other MOOC resources such as forum posts, access to external resources
outside the MOOC.
Acknowledgements. This work has been co-funded by the University of Cuenca (DIUC),
Cuenca-Ecuador, under the project “Learning analytics for the study of self-regulated learning
strategies in a hybrid learning context (DIUC_XVIII_2019_54)”, by the Pontificia Universidad
Católica de Chile and the Direction of Engineering Education – DEI. We also thank the “Fun-
dación Carolina” for awarding to the authors the 2022 short-term postdoctoral scholarship, which
contributed to the development of this research work.
References
1. Chuang, I.: HarvardX and MITx: four years of open online courses – Fall 2012-Summer 2016.
SSRN Electron. J. (2017)
2. Littlejohn, A., Hood, N., Milligan, C., Mustain, P.: Learning in MOOCs: motivations and
self-regulated learning in MOOCs. Internet High. Educ. 29, 40–48 (2016)
3. Reich, J.: Rebooting MOOC research. Science 347(6217), 34–35 (2015)
4. Grainger, B.: Massive Open Online Course (MOOC) Report 2013. University of London,
London (2013)
5. Zhao, C., Yang, J., Liang, J., Li, C.: Discover learning behavior patterns to predict certification.
In: ICCSE 2016 – 11th Int. Conf. Comput. Sci. Educ., pp. 69–73 (2016)
6. Maldonado, J.J., Palta, R., Vazquez, J., Bermeo, J.L., Perez-Sanagustin, M., Munoz-Gama, J.:
Exploring differences in how learners navigate in MOOCs based on self-regulated learning
and learning styles: a process mining approach. In: Proc. 2016 42nd Lat. Am. Comput. Conf.
CLEI 2016 (2017)
7. Kocdar, S., Karadeniz, A., Bozkurt, A., Buyuk, K.: Measuring self-regulation in self-paced
open and distance learning environments. Int. Rev. Res. Open Distance Learn. 19(1), 25–43
(2018)
8. Xu, B., Yang, D.: Motivation classification and grade prediction for MOOCs learners. Comput.
Intell. Neurosci. 2016, 1–7 (2016)
9. Brinton, C.G., Buccapatnam, S., Chiang, M., Poor, H.V.: Mining MOOC clickstreams: video-
watching behavior vs. in-video quiz performance. IEEE Trans. Signal Process. 64(14), 3677–
3692 (2016)
10. Sinha, T., Jermann, P., Li, N., Dillenbourg, P.: Your click decides your fate: Inferring Informa-
tion Processing and Attrition Behavior from MOOC Video Clickstream Interactions, pp. 3–14
(2015)
11. You, J.W.: Identifying significant indicators using LMS data to predict course achievement
in online learning. Internet High. Educ. 29, 23–30 (2016)
12. Ruipérez-Valiente, J.A., Cobos, R., Muñoz-Merino, P.J., Andujar, Á., Kloos, C.D.: Early
prediction and variable importance of certificate accomplishment in a MOOC. In: Lect. Notes
Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol.
10254, LNCS, pp. 263–272 (2017)
13. Moreno-Marcos, P.M., Muñoz-Merino, P.J., Alario-Hoyos, C., Estévez-Ayres, I., Delgado
Kloos, C.: Analysing the predictive power for anticipating assignment grades in a massive
open online course. Behav. Inf. Technol. 37(10–11), 1021–1036 (2018)
Project-Based Learning and its Impact
on the Mathematical Knowledge of 1st Year High
School Students in Schools in Southern Ecuador
Ricardo Orellana1 and Marcos Chacón-Castro2,3(B)

1 Maestría en Innovación y Liderazgo Educativo, Facultad Ciencias de La Educación - Faced,
Universidad Indoamericana, 180103 Ambato, Ecuador
rorellano@indoamerica.edu.ec
2 Maestría en Entornos Digitales, Facultad Ciencias de la Educación - FACED,
Universidad Indoamérica, 180103 Ambato, Ecuador
mhcacon10@indoamerica.edu.ec
3 Grupo Investigación GIECI, Fundación Universitaria Internacional de La Rioja,
Bogotá, Colombia
Abstract. The main principle of this research is the implementation of a didactic

proposal for the teaching of mathematics, through the Project Based Learning
(PBL) methodology, which is developed as a response to the challenges presented
by the current curricular designs. The PBL approach is defined as a pedagogical
strategy that seeks to integrate and verify the construction of individual and group
knowledge for the creation, products and proposed solutions to environmental
problems. The aim of this research was to intervene pedagogically in the subject
of mathematics with PBL and to describe its positive effect on students with the
application of a PISA-D type test. The methodology used was descriptive with a
positivist approach, applied to students in the first year of secondary education in
three public educational institutions in the south of Ecuador. Various evaluation
instruments were used, including pre-test and post-tests, evaluating the impact
of the implementation of PBL at the educational level in the mathematics class.
Finally, the findings of this study allowed us to determine that PBL is an effective
strategy in the learning and teaching of mathematics in secondary education.
Keywords: PBL · meaningful learning · mathematics · PISA-D · Mathematics
1 Introduction
The results of the PISA-D tests in science, mathematics and reading for 15-year-old
students [1], presented by the Ministry of Education of Ecuador at the end of 2018,
showed that only 30% of students managed to achieve a basic level in mathematics [2].
Although it is difficult to identify all the causes of this problem, some factors that may
help to understand it are mentioned below. In urban areas, students represent 53.58%
and in rural areas 46.42%, and there are 208,000 teachers present to attend to them.
Government investment in education is 3.7% of GDP. Literacy levels are 94.5%, the
school dropout rate is 2.11%, and the school dropout rate is 2.4% [3].
https://doi.org/10.1007/978-981-99-7353-8_18
Project-Based Learning and its Impact on the Mathematical Knowledge 227
According to the National Institute for Educational Evaluation (INEVAL), the fol-
lowing historical figures are presented for the 2016 mathematics level in the fourth,
seventh, tenth and eleventh grades. Students in grade 4 obtained an insufficient achieve-
ment of 21.6%, ranking second last after the reading section. Students in grade 7 had a
35.8% insufficient achievement, being comparatively the most deficient percentage of
all. The evolution of these low results covers more than 50% of the student population.
Consequently, low performance in mathematics is the result of a deficient education that
has been endured for several years (see Fig. 1).
Fig. 1. Mathematics performance in grades 4, 7, 10 and 3. Source: National Institute for

Educational Evaluation (INEVAL, 2016).
This frame of reference allows us to deduce the need to introduce improvements in

teaching, particularly in mathematics. The transformation of education involves apply-
ing active methodologies, in which the student manages to acquire new knowledge, from
previous ones, apply them in concrete situations and help improve their environment,
in articulation with the development of participatory and cooperative skills [4], which
enhances the performance of group work [5]. On the other hand, these methodologies
are approached as those that allow a better learning from reading to practice, imple-
menting in a better way the knowledge learned from reality [6]. There is a wide range
228 R. Orellana and M. Chacón-Castro
of active teaching methodologies, such as the collaborative, problem-solving, challenge

and project-based learning strategy shown in Fig. 2 [4].
Fig. 2. Various active methodologies. Source: Own elaboration.
Based on this information on the main active methodologies that can be brought
into the classroom, we chose to apply “Project Based Learning” (PBL) in this research,
a strategy, mediated by the teacher, which integrates the active role of the student in
collaboration with others to build knowledge with the intention of producing a prod-
uct that provides high satisfaction in the beneficiaries, in this case for the teaching of
mathematics. The group of students selected corresponds to the first year of baccalau-
reate or middle school, which corresponds to young people between 15 and 16 years
of age. It also highlights the importance of allowing teachers to teach with this type of
methodology in order to increase the range of mathematical skills and knowledge.
The PBL methodology is based on an instructional and learning process centred on
students that encourages personal and group interaction with the object of study, achiev-
ing an approach to the approach of conjectures, predictions and analysis of information
to reach inferences, deductions and logical-mathematical reasoning that allow the cre-
ation of a product in which knowledge is applied [7]. This strategy makes it possible
to approach everyday reality from mathematics and with the generation of an input,
optimising the student’s learning. In turn, the final product or project will require a good
design in the organisation of concepts, their structure, exploration, understanding, anal-
ysis and reflection. In this order of ideas, it is hypothesised that there may be significant
differences between the experimental group and the control group. The elements that
make up the PBL methodology can be adapted to other areas of knowledge, including
interdisciplinarity. In this research, with the aim of proposing an education that improves
the mathematical skills to be achieved by the student, the question is established: What
impact does this methodology have when used in the teaching of mathematics in the 1st
level of high school in Ecuador?
2 Related Work
The following Table 1 shows the related works that address the same problem regarding
the implementation of PBL methodology in the area of mathematics in Ecuadorian
schools.
Table 1. Experiences of ABP in the subject of mathematics in Ecuador 2018 - 2022
Place and date of Age Type of schooling Research approach Contribution to the study
implementation OBA of PBL in Mathematics
Pujilí, 2018 17–18 years Secondary, Public Mixed Significantly positive
results were obtained in
mathematical
understanding, improved
academic performance
and increased
mathematical skills [8]
El Carmen, 2019 9–10 years Primary, Public Mixed The effectiveness of PBL
in guiding student
cognitive development
and application of
mathematical knowledge
was determined [9]
Guayaquil, 2021 16–17 years Secondary, Public Quantitative The incidence of PBL as
a motivating agent for
students’ mathematical
knowledge was
established [10]
El Empalme, 2021 10–11 years Primary, Public Mixed A PBL training plan was
drawn up and the
proposal was
implemented with
satisfactory results in
terms of students’ levels
of understanding and
cognitive development
[11]
3 Methodology
The proposed research had a quantitative approach, using descriptive, exploratory and
correlational analysis techniques between the research methodology and the develop-
ment of mathematical skills in first-year students. Likewise, the BGU is taken in accor-
dance with the standards specified by a PISA-D type test in mathematical knowledge
[12]. A non-probabilistic sample was used, where a control group was employed and
pre- and post-tests were conducted to evaluate the results.
3.1 Participants
During this research, a sample focused on first year high school students in three edu-
cational institutions located in the south of Ecuador was used. They were classified as:
Institution 1, Institution 2 and Institution 3; Institution 1 had two groups with 25 and 27
students respectively, while Institution 2 had four groups of 38, 39, 41 and 43 students
each. Finally, Institution 3 had two groups with 35 and 31 students. The experimental
group consisted of those students who obtained a lower average score in the diagnostic
test, while the control group consisted of those students who obtained a higher score in
the diagnostic test.
3.2 Instruments
For this study, the measurement instrument was a PISA-D type test in mathematical
reasoning and skills. It is expected that the students who took part in the implementation
of the methodology will show an increase in their ability in the subject competences
compared to the students in the control team. In relation to the experimental team, the
PBL methodology was applied in the subject, while in the other team (control group)
the conventional teaching approach was maintained. Then, the second evaluation was
applied to both groups. To contrast the results found, the Mann-Whitney U test and the
Wilcoxon W test were used.
The “ABP-PPD20” instrument was used to carry out the tests. After applying the
first test and obtaining the results, 4 weekly mathematics sessions were established with
the students using the ABP strategy, during a period of 5 months, which was the time of
application of the study. On the other hand, the control team received the mathematics
subject according to the plan designed by the Ministry of Education of Ecuador; it is
worth mentioning that in both groups, the mathematics class was taught by the same
teacher. At the end of the first period of the academic year, the second test was carried out
for the two groups involved in this study. In this way, the scores obtained were analysed
using SPSS statistical software, in order to test the reliability of the equipment, the Kuder
Richardson statistic (KR20) was used.
3.3 Procedure
A quantitative approach was adopted during this research, focusing on the analysis of
data collected on the variables. Descriptive, exploratory and inferential analysis tech-
niques were used, as well as correlation tests to examine the conformity or inconformity
between the methodology applied in the study and the development of mathematical
skills in students. Thus, the ABP-PPD20 instrument was used for the application of the
project, which consisted of two tests to assess the students’ performance in mathematical
competences. Firstly, the first test was applied and the results were recorded. In relation
to these results, four mathematics sessions were arranged in which the PBL was applied
over a period of five months. These sessions were held four times a week. Meanwhile, the
control group followed the standard curriculum established by the Ministry of Education
of Ecuador for the same period of time.
At the end of the first term of the academic year, the second test was administered
to both groups. The scores obtained in both tests were analysed using SPSS statistical
software for data analysis. To assess the reliability of the instrument used in the tests, the
Kuder Richardson statistic (KR20) was applied, on the other hand, the ABP-PPD20 in the
study, a process of validation of the instrument was carried out. First of all, the opinions of
mathematics teachers and principals in the participating schools were gathered regarding
the suitability of the instrument. Subsequently, corrections and adjustments were made
in terms of format and linguistic approaches in the presentation of the exercises. The
revised instrument was submitted to experts in PBL and mathematics pedagogy, who
offered comments and suggestions. Finally, these contributions were taken into account
to arrive at the final version of the ABP-PPD20 used in the study.
3.4 Design and Implementation
Mathematics teachers were invited to participate in this study in the selected educational
institutions. The research design was explained, a timetable was established and the
specific objectives for teachers, management and students were presented. In the first
stage, teachers received initial training in PBL, and a plan was established for support
and advice during implementation. In this way, two projects - Mate Chef Ecuador and
Conociendo la Belleza de mi Ciudad - were set up, discussed, fine-tuned and implemented
during the first term of the school year.
During the planning phase, the documentation of the first project “Mate Chef
Ecuador” was presented, the material was revised and corrected, the partial products
and the final product were designed, the products were consolidated according to the
curricular contents to be taught to the students and, due to the COVID-19 crisis, teachers
were trained in technological resources and remote teaching. In addition, one month
before the end of the first project “Mate Chef Ecuador”, a second project “Knowing the
beauty of my city” was organised in the same way as the previous one. The process
established in the “AIRE” model (Activation, Investigation, Reanalysis and Evaluation)
developed by the University of Alcalá de Henares [13, 14] was chosen for the application
of PBL. This model provided structure and pedagogical guidance for PBL during the
study period (see Fig. 3).
Fig. 3. Diagram of the “AIRE” model and its phases for the application of PBL, based on Prieto
(2006) Arpí et al. (2013).
The “Mate Chef Ecuador” programme was inspired by a popular television pro-
gramme, which aims to discover the presence of mathematics in food preparation. The
aim was to make students aware of the importance of mathematics in the kitchen, to learn
how to classify, to make choices based on statistics and to improve the taste of food. The
general strategy used by “Mate Chef Ecuador” is shown in the following Fig. 4.
Fig. 4. Outline of the development of the “Mate Chef Ecuador” project using PBA methodology
As mentioned above, the pandemic meant that this research had to take on the chal-
lenge of addressing teaching and learning from the students’ homes. Therefore, once
the “Mate Chef Ecuador” project was launched, a weekly follow-up meeting was estab-
lished with the teachers to make adjustments to the schedule, correct by-products and
evaluate the teaching processes with the students in the experimental group. The crisis
caused by COVID-19 led to the adoption of changes in accordance with the objectives,
in terms of content, delivery of material and duration of the stages according to the AIRE
model. At the end of the implementation of the “Mate Chef Ecuador” project schedule,
the second project was planned. The teachers presented some proposals, some related to
COVID-19, the architecture of the building where the school operates and the presenta-
tion of the city where the students lived. After some reflection on each proposal, the last
option was chosen and given the name “Calculating the Beauty of my City” (CBC). The
objective guiding the selection of the second project was to convey a positive message
amidst the negative consequences of the pandemic. The most touristic or emblematic
locations in each city were chosen. The measurement exercises were carried out using
Google Maps and other applications. Finally, given the motivation of the students, it
was decided to hold a special session to showcase the best projects from each school,
allowing students from the three schools to showcase the cities of Paute, Azogues and
Loja through mathematics. As with the previous project, instructions were drawn up for
both the by-products and the final product. The final result was a video showing the
presence of mathematics in the most important places of the city.
Based on the PISA-D results before the test, it was found that the results did not follow a
normal pattern in the control and experimental team. - Smyrno Tane and Shapiro-Wilk.
Therefore, a non-parametric test is needed to determine whether what was found in the
test differs before and after the application of PBL (Table 2 and Table 3).
Table 2. Analysis of normality using the Kolmogorov-Smirnov and Shapiro-Wilk tests in the
control group.
Pretest Control Group

Kolmogorov-Smirnov Shapiro-Wilk
Statistic ,171 ,913
Gl 74 74
Sig ,000 ,000
Table 3. Analysis of normality using the Kolmogorov-Smirnov and Shapiro-Wilk tests in the
experimental group.
Pretest Experimental Group

Kolmogorov-Smirnov Shapiro-Wilk
Statistic ,162 ,949
Gl 74 74
Sig ,000 ,005
4.1 Comparison of Pretest Results Between the Control Team

and the Experimental Team
When comparing in the 1st quarter of the year, the normal performance of the control
group with the performance of the experimental group, it is found (Table 4):
Table 4. Statistical analysis of the experimental group.
Average Standard deviation Coefficient of variation Mín Máx

Experimental Group 9,28 0,91 10% 4,24 10
Control Group 9,38 0,50 5% 8 1
0
It should be noted that the performance of both groups is similar, and this statement
was validated by statistical tests carried out afterwards. In the experimental group, there
were only 2 students who failed, while the control group obtained a 100% pass rate. In
terms of grades, it was found that the control team had an average of 9.38 points out of
10, slightly higher than the average of the experimental group, which was 9.28 points.
In addition, the standard deviation of the control group was lower, indicating that this
group was more homogeneous in terms of grades compared to the experimental team,
which showed a higher deviation.
4.2 Control and Experimental Group for Post Test
Control Group
When analysing the descriptive data of the control group in the post-test and their
compliance with the Kolmogorov-Smirnov and Shapiro-Wilk normality test (Table 5):
Table 5. Kolmogorov-Smirnov and Shapiro-Wilk normality tests
Kolmogorov-Smirnova Shapiro-Wilk
Stadistic gl Sig Stadistic gl Sig
Pos-Test ,159 74 <,001 ,948 74 ,004
Control
Group
Because the p-value (Sig) is lower than the stated significance level of 0.05 (0.004
< 0.05), the decision is made that there is evidence not to pursue the null hypothesis.
This indicates that the distribution of the posttest scores in the control group does not
follow a normal distribution (Table 6 and Table 7).
Table 6. Comparison between pre-test and post-test results of the control group.
Ranks
N Mean Rank Sum of Ranks
Post-Test Control - Pre-Test Control Negative Ranks 9a 46,94 422,50
Positive Ranks 64b 35,60 2278,50
Ties 1c
Total 74
a. PostTestControl < PreTestControl
b. PostTestControl > PreTestControl
c. PostTestControl = PreTestControl
Table 7. Application of the Wilcoxon signed-rank test to compare the posttest of the control
group with the pretest of the same group.
Test Statisticsb
PostTestControl - PreTestControl
Z -5,102a
Asymp. Sig. (2-tailed) ,000
a. Based on negative ranks
b. Wilcoxon Signed Ranks Test
The results showed significant differences. In fact, even the students in the compari-
son group showed a difference in PRESEST results compared to the traditional teaching
applied by the facilitators during the semester. As explained in the next chapter, there
were also significant changes in the experimental group. However, it should be mentioned
that the difference between the mean PRESEST and POSTEST of the experimental team
is higher than the difference of the comparison team.
Experimental Group
According to a statistical analysis of the POSTEST test of the experimental group, in
comparison with the PRETEST test, the two tests obtained results very different from
the normal distribution, therefore the PBA methodology is carried out with the following
results (Table 8):
Table 8. Test statistics with Experimental Post-Test and Experimental Pre-Test
Test Statisticsb
PostTestExperimental - PreTestExperimental
Z −2,662a
Asymp. Sig. (2-tailed) ,008
a Based on negative ranks
b Wilcoxon Signed Ranks Test
Since the p-value is less than significant, it is statistically certain that there is a signifi-
cant difference compared to what was found in the POSTEST and PRETEST results. The
statistical analysis supports the hypothesis that the implementation of PBL in the disci-
plines studied has a positive impact on the performance of students’ academic skills and
attitudes. Significant differences in PISA-D test scores before and after PBL implemen-
tation suggested that this approach had a favourable impact on students’ performance.
These findings support the idea that PBL contributes to improved mathematics learning
and academic achievement.
The results indicated that students who engaged in the approach performed better
than those who received instruction in the traditional method of teaching. The use of this
methodology proved to be essential in fostering the development of mathematical skills
and supporting active and meaningful learning.These results underline the importance
of implementing new pedagogical approaches that allow students to participate more
actively in their own learning process.
In the same way, the results of this study are consistent with previous studies showing
that the application of PBL methods in mathematics education has significant benefits
on students’ academic performance and learning development [15–17]. These studies
complement and reinforce the importance of adopting new approaches to promote active
and meaningful learning in mathematics.
Comparison Between Control and Experimental Group

Figure 5 shows the data related to the performance of the students in the experimental
team in terms of their individual results in the PISA-D test. The averages of the scores
obtained are presented, as well as the lowest and highest values achieved by the students
in each school. In addition, the mathematical competence that was assessed during the test
is indicated. These data provide a more detailed and specific picture of the performance
of students in each school after the implementation of PBL, allowing for analysis and
comparison of the results obtained in relation to the subject tested.
Fig. 5. Results of the PISA-D type test (Post-test)
Figure 5 shows the mathematical ability levels and descriptions of the test groups
after using PBL. Therefore, Fig. 6 shows the mean values of the PISA-D tests after using
the PBL method. In addition, the mean results using the standardised PISA-D test after
using the PBL method are shown.
Fig. 6. Average results using the PISA-D standardised test after using the PBL method.
Fig. 7. Distribution of PISA-D POST-TEST scores by 100-point range in the three experimental
groups.
Figures 6 and 7 show the evolution of students to higher levels after implementing
PBL. In the test, 4 students were selected in the 100–200 section, but there was only one
student in this section of the test. Seven of the initial 14 students at the 200–300 level
progressed to other higher level courses after ABP. Similarly, with 300–400 as a group,
it reduces from 16 students in the first test period to 9 students in the next period. It is
worth mentioning that Institution 3 went from 5 students to 7 students on this scale, and
Institution 1 went from 9 students in the test to 0 students, because it makes everyone
better. Between 400–500 points, only three students were selected for the first test, but
the number of students in the test increased to 7 and the learning area of Institution 1
increased significantly. On the other hand, students with 500–600 points dropped from
32 to 24. Institution 2 reduced the number of students from 18 to 13 this year. However,
there was a substantial increase from 600–700 points, from 6 candidates to 15. Finally,
for the 700–800 units, initially there was only one student per exam, and with the use
of the ABP, the number increased to three. These results demonstrate the progression
of students towards higher scores and reflect the positive impact of PBL on academic
performance in mathematics.
5 Conclusions
Given the educational problems that the COVID-19 pandemic generated, particularly in
mathematics, the three participating educational institutions have appropriately adopted
active learning methods such as PBL to apply and strengthen the mathematical skills
of secondary school students. The results obtained after the use of PBL in the PISA-D
test were significant, with better performance in two of the three study centres, with an
improvement of 18.16% and 31.62%. The results in learning mathematical competence
of Institution 1 rose from level 1a to level 3, Institution 3 from level 1b to level 2 and
Institution 2 has remained at level 3 although it achieved an increase of 6.24 points. On
average, in the three educational centres there was an increase in mathematical compe-
tence of 17%. These results obtained with the application of the PBL strategy demonstrate
the relevance and effectiveness of the methodology implemented in mathematics educa-
tion, therefore, it is necessary to deepen this line of research through similar studies to
contrast these results with each other and in the same way by applying other evaluation
instruments that cover other approaches not established by the PISA-D tests. The phases
of the “AIRE” process (activation of knowledge, research and analysis, re-analysis and
elaboration of the final product) can be proposed for other areas of knowledge, as well
as for new forms of interdisciplinary teaching and learning.
The level of involvement of managers, teachers and students in the execution of the
projects allows the configuration of a proposal for innovation in the development of
the curriculum and mathematical competence. The projects “Mate Chef Ecuador” and
“Calculating the Beauty of my City” presented in this study are not only proof that it is
possible to develop innovative ways of planning and executing instructional processes
in the subject of mathematics, but they can also be part of other processes involving
other areas of knowledge. Based on the results obtained in the research, it is possible
to initiate a process in which, through the implementation of Problem-Based Learning
(PBL) according to the “AIRE” model, additional inputs are provided to reconsider in
High School education. The final projects or products related to the student’s context,
current events, and the stimulation of personal inquiry - even after the completion of
the final project - present a viable alternative to the traditional instructional method. It
is recommended that PBL adopts an interdisciplinary approach that involves not only
a single subject but also encompasses the majority or entirety of those constituting the
curriculum body.
Finally, it is proposed to continue opening lines of research in this area, such as the
pedagogical attitude of teachers in the application of PBL, the comparison of PBL with
other active methodologies, and the results obtained by Ecuadorian high school students
in comparison with other tests that assess competency levels in different knowledge
areas.
References
1. Organización para la Cooperación y el Desarrollo Económico (OCDE): Marco de Evaluación
y de Análisis de PISA para el Desarrollo: Lectura, matemáticas y ciencias, Versión preliminar.
OECD Publishing, Paris (2017)
2. INEVAL: Instituto Nacional de Evaluación Educativa, “Educación en Ecuador: Resultados
de PISA para el desarrollo humano,” Quito, Ecuador, INEVAL, 2018. [Online]. Disponible:
http://evaluaciones.evaluacion.gob.ec/BI/informe-general-pisa-2018
3. Ministerio de Educación del Ecuador: Datos abiertos del Ministerio de Educación del Ecuador.
Estadísticas Educativas. Quito, Ecuador, Ministerio de Educación del Ecuador, 20 de mayo
2022. [Online]. Disponible: https://educacion.gob.ec/datos-abiertos/
4. Puga, L., Jaramillo, L.: Metodología activa en la construcción del conocimiento matemático.
Sophía 19, 291–314 (2015)
5. Labrador, M.J., Andreu, M.A. (eds.): Metodologías Activas. UPV (2008)
6. Muntaner, J., Pinya, C., Mut, B.: El impacto de las metodologías activas en los resultados
académicos (2020)
7. Blumenfeld, P.C., Soloway, E., Marx, R.W., Krajcik, J.S., Guzdial, M., Palincsar, A.: Moti-
vating project-based learning: sustaining the doing, supporting the learning. Educational
Psychologist, pp. 369–398 (1991)
8. Cadena-Zambrano, V., Nuñez-Naranjo, A.: ABP: Estrategia didáctica en las matemáticas.
593 Digital Publisher CEIT, vol. 1, no. 5, pp. 69–77 (2020). https://doi.org/10.33386/593dp.
2020.1.184
9. Párraga Valdez, W., López Cedeño, J., Alcívar Chávez, A., Elizalde Cordero, C.: La influencia
del ABP en el desarrollo cognitivo del área de matemáticas. Suplemento CICA Multidisci-
plinario ISSN: 2631–2832, vol. 6, no. 013, pp. 62–80 (2022). Recuperado a partir de https://
suplementocica.uleam.edu.ec/index.php/SuplementoCICA/article/view/99
10. Vera, J., Maricela, A.: Estrategia aprendizaje basado en proyectos influye en el aprendizaje
de matemáticas en estudiantes, bachillerato, guayaquil, ecuador. 2021. Universidad César
Vallejo (2022). [Online]. Disponible: https://hdl.handle.net/20.500.12692/77253
11. Sabando, J., Laz, J., García, L., Cando, M.: Aprendizaje Basado en Proyectos y su influencia
en el desarrollo cognitivo de estudiantes del subnivel medio. Universidad Técnica Estatal de
Quevedo (2021). [Online]. Disponible: https://maestroysociedad.uo.edu.cu/index.php/MyS/
article/view/5759/5577
12. Organisation for Economic Co-operation and Development (OECD): PISA 2018 Assessment
and Analytical Framework. PISA, OECD Publishing, Paris (2019). [Online]. Disponible:
https://doi.org/10.1787/b25efab8-en
13. Arpí, C., et al.: El ABP: origen, modelos y técnicas afines. Aula de Innovación Educativa
216, 1–18 (2012)
14. Prieto, A., et al.: Un nuevo modelo de aprendizaje basado en problemas, el ABP 4x4, es eficaz
para desarrollar competencias profesionales valiosas en asignaturas con más de 100 alumnos.
Aula Abierta 87, 171–194 (2006)
15. Flores-Fuentes, G., Juárez-Ruiz, E. de L.: Aprendizaje basado en proyectos para el desarrollo
de competencias matemáticas en bachillerato. Revista Electrónica de Investigación Educativa
19(3), 71–91 (2017). https://doi.org/10.24320/redie.2017.19.3.721
16. Perico-Granados, N., Umba, M., Tovar-Torres, C., Reyes, C.: Proyectos educativos para estu-
diantes de educación básica en Colombia: Estrategia de aprendizaje en matemáticas. Revista
Venezolana de Gerencia, pp. 1741–1757 (2020). https://doi.org/10.37960/rvg.v25i92.34292
17. Vargas, N., Niño, J., Fernández, F.: Aprendizaje basado en proyectos mediados por tic para
superar dificultades en el aprendizaje de operaciones básicas matemáticas. REDIPE 9(3),
167–180 (2020). https://doi.org/10.36260/rbr.v9i3.943
Student Perception of Peer Review in the Digital
Age
Marcio Bigolin1,2 , Eliseo Reategui1(B) , Sofía García Cabeza3 ,

and Regina Motz4
1 PPGIE, Universidade Federal do Rio Grande do Sul, Porto Alegre, Brazil
marcio.bigolin@canoas.ifrs.edu.br, eliseo.reategui@ufrgs.br
2 Instituto Federal de Educação, Ciência e Tecnologia do Rio Grande do Sul, Canoas, Brazil
3 Facultad Latinoamericana de Ciencias Sociales, Montevideo, Uruguay
sgarcia@flacso.edu.uy
4 Facultad de Ingeniería, Universidad de la República, Montevideo, Uruguay
rmotz@fing.edu.uy
Abstract. This study investigates students’ perception of peer review as an online

learning activity specifically focused on essay writing in the context of upper
secondary education. The research aims to provide insights into students’ under-
standing, experiences, and attitudes towards peer review. The study was conducted
among first-year students in writing classes at a public institution in Brazil, in
which students used RevisãoOnline, a specific tool to write and review essays.
Participants were students who had written at least 2 essays and completed 4
reviews using the aforementioned tool, totaling 47 students. After engaging in
writing and reviewing activities, these students were administered a questionnaire
about their perception of the peer review process. Results showed that overall, stu-
dents expressed positive perceptions regarding the value of giving and receiving
feedback from their peers in essay writing activities. This may have been influ-
enced by the use of RevisãoOnline, which provided step-by-step guidance during
the review process, offering support and identifying areas for improvement in the
reviewed essays. Students also reported a strong sense of comfort in both pro-
viding and receiving feedback. Regarding the incorporation of peer feedback by
teachers in the evaluation process, a significant majority of students agreed with
the approach. However, a notable percentage of students indicated the need for
further exploration of the teacher’s role in integrating peer feedback into their
evaluation process.
Keywords: Peer review · online learning · essay writing · student perception
1 Introduction
It is a known fact that assessment may influence learning in a positive or negative way
(Falchikov, 2007). Peer review is an instructional strategy that may be used for evaluation
purposes, besides engaging the students in an active learning activity that gives them
the possibility to develop writing and critical thinking skills (Orjuela-Laverde and Chen,
https://doi.org/10.1007/978-981-99-7353-8_19
Student Perception of Peer Review in the Digital Age 243
2014). Although the idea of getting students to review and score each other’s work may
seem appealing, educators often struggle with the difficulty of using these reviews in the
evaluation process. Furthermore, student’s perceptions of assessment methods should
also be taken into account, as this may influence their attitude towards studying and
learning (Struyven et al., 2005).
In this article we analyze key aspects related to the use of peer-review as a learning
and evaluation activity in higher education. Although automated scoring is becoming
more and more popular in situations in which multiple-choice questions can be used,
in some subject fields text production is essential to give students a thorough under-
standing of a topic (Bali, 2014) and get them acquainted with the standards of formal
written expression (Ellis et al., 2005). Thus, considering assignments that involve text
production, providing students with feedback is crucial to promote learning, a goal that
may be achieved through the use of peer review activities (Mulder et al., 2014). Get-
ting feedback on their work before submitting final drafts for assessment is crucial for
students in writing assignments (Kasch et al., 2020). This feedback helps them actively
participate in editing and revising their work, leading to improvement in their writing
skills. In a MOOC setting, peer feedback offers a secure environment for students to
exchange ideas and receive constructive criticism while also having the opportunity to
provide feedback to their peers.
Studies report that giving and receiving feedback has similar effects on the develop-
ment of writing (Huisman et al., 2018), although it can also generate rejection in students
at the beginning (Luo, 2014).
While the literature highlights the significant benefits of peer feedback, it is essential
to consider the perspectives of participants involved in both receiving and providing
feedback. In this particular context, the present study sought to address the following
research questions:
• What is the perception of students regarding the overall effectiveness and value of
peer review feedback?
• What are their perceived difficulties in engaging in peer review activities?
• How do students perceive the incorporation of their peers’ feedback by teachers in
the evaluation process?
These questions are closely aligned with a previous instrument that explores students’
perception of peer feedback in a MOOC context (Kasch et al., 2020). However, in our
study, our focus has been on students’ perception about peer review activities specifically
designed for writing and revising university entrance examination essays. This article is
organized as follows: Sect. 2 provides an overview of peer review, discussing the benefits
and challenges documented in the existing literature; Sect. 3 describes the research
methods employed in our study, outlining its scope and highlighting the key aspects
of peer review considered during the development of the proposed activities; Sect. 4
presents the results obtained from the study, while Sect. 5 discusses their implications
for further research; Sect. 6 concludes the article by summarizing the findings and
suggesting potential avenues for future work.
244 M. Bigolin et al.
2 Peer Review in the Digital Age

In the shift from traditional classroom settings to online learning environments, the
dynamics of peer feedback have undergone significant changes from an educational
perspective. In traditional classrooms, peer feedback often took place through face-
to-face interactions, such as group discussions or paper exchanges. Students would
provide feedback to their peers based on their immediate observations and discussions
during class activities. However, in online environments, peer feedback has become
more structured and facilitated through the use of digital tools and platforms. This shift
has brought both advantages and challenges.
On one hand, online platforms provide opportunities for synchronous or asyn-
chronous peer feedback, allowing students to provide and receive feedback at their own
pace and convenience (Chen and Tsuiping, 2014). Additionally, digital tools often offer
features such as highlighting, commenting, and tracking changes, enabling students to
provide more precise and detailed feedback on specific areas of improvement. On the
other hand, the absence of face-to-face interactions in online environments can impact
the quality and depth of peer feedback. Nonverbal cues and immediate clarifications
may be lost, leading to potential misinterpretations or incomplete feedback. To over-
come these challenges, educators are exploring strategies to enhance the effectiveness
of peer feedback in online environments. This includes providing clear guidelines and
rubrics, and implementing structured peer-review processes.
Peer review offers students an opportunity to revise their peers’ works, fostering a
collaborative learning environment and promoting the improvement of the overall quality
of students’ writing (Topping, 2009). However, it is important to recognize that not all
learners possess the skill of providing effective feedback (Carless and Boud, 2018).
Nevertheless, while there may be variations in the quality of feedback between students
and experts, research suggests that the majority of students are capable of providing
valuable feedback (Hovardas et al., 2014). Additionally, both trained peer feedback and
teacher feedback have shown similar positive effects on enhancing writing competence
and writing self-efficacy (Cui et al., 2021).
The literature highlights several key benefits of peer review, including:
• Contribution to critical thinking, comprehension, and writing skill development
(Garcia-Loro, 2020).
• Promotion of autonomy, motivation, social belonging, and higher-order thinking skills
(Luo, 2014).
• Facilitation of reflection, transforming the concept of learning from teaching to
learning through assessment (Topping, 1998).
• Enhancement of engagement, fostering interest, interactivity, and empathy (Topping,
1998).
• Improvement in writing quality by developing independence, autonomy, self-
regulation, and critical thinking (Baker, 2016).
It has also been found that students who receive feedback from multiple peers tend to
improve their writing more than those who receive feedback from a single expert (Cho
and MacArthur, 2010). However, peer review also has some reported disadvantages,
which include:
• Teachers’ time is not necessarily saved (Topping, 1998).

• Potential initial resistance or rejection from students (Luo, 2014).
In this regard, it is crucial to highlight that feedback plays a vital role in the learning
process (Pearce et al., 2009). However, its impact may be diminished for individuals
who hold a negative perception of the feedback process (Atwater et al., 2000).
3 RevisãoOnline1 - A Writing Tool to Support Peer-Reviewing

RevisãoOnline is a writing tool that facilitates the peer feedback process for students
working on writing assignments, particularly within the context of the Brazilian ENEM
writing entrance examination. The tool operates based on a pedagogical intervention
model where the user (student) assumes dual roles as both a writer and a reviewer.
Although these roles can be concurrent, a standardized flow is followed, as depicted in
Fig. 1:
Fig. 1. RevisãoOnline writing and reviewing cycle
a. The student begins by writing an essay on a topic suggested by the teacher or of their
own choice, which is then submitted to the system.
1 https://revisaoonline.com.br/
b. The system randomly assigns the essay to two peers, who can be fellow students
from the same group or from the entire system, to serve as reviewers. These reviewers
receive email notifications regarding the available essays for review.
c. To receive reviews on their own essay, the student is required to evaluate a minimum
of two essays written by other students. Additional reviews for an essay are released
on a 1-to-1 basis. For instance, if the student has evaluated only one essay, only one
review will be made available for their own essay.
d. During the review process, the student is requested to provide comments and rate the
feedback received, based on specific criteria. This allows for the establishment of the
reviewer’s reputation within the system.
When the student accesses the writing module of RevisãoOnline, the system min-
imizes intervention during the student’s writing process. This approach is designed to
prioritize intervention during the revision process. For instance, the tool disables the
browser’s spell-checker and does not provide writing assistance. The only available fea-
tures are a word counter that limits the student’s essay to 650 words. It is also important
to highlight that the writing interface does not allow the student to copy and paste texts;
they can only type directly.
Regarding the revision features, they were structured to guide students through the
revision process, addressing both local aspects such as writing mistakes, syntactic and
grammatical errors, as well as global elements like cohesion and argumentation.After
reviewing two essays, the system offers the student peer feedback on their own essay.
This ensures that each student receives a minimum of two revisions for their work.
4 Methods
The study presented in this work was conducted as part of peer review activities imple-
mented among students who wrote essays and provided feedback on their peers’ work
using RevisãoOnline.
4.1 The Participants

This study was conducted among first-year students in the context of writing classes at
a public institution in Brazil. Data was collected from three different classes in distinct
technical courses. These classes were selected because they had a longer experience
using RevisãoOnline and engaging in peer-review activities. Students who had written
at least 2 essays and completed 4 reviews using RevisãoOnline were invited to participate
in the study. Out of the 405 invited students, 47 consented to participate and completed
a questionnaire that assessed their perceptions of peer-review activities, including their
utility, difficulties, and the potential challenges of incorporating peer feedback in the
evaluation process by teachers.
4.2 The Activity

The assignments in our study were structured around the National Secondary Education
Exam (ENEM) in Brazil, an annual assessment whose main objective is to evaluate the
performance of high school students in various areas of knowledge, including essay

writing. The ENEM essay is an important part of the test and consists of producing a
persuasive argumentative text, in which students must develop a given topic by present-
ing arguments and opinions based on information and knowledge acquired throughout
their schooling. The essay is evaluated considering criteria such as writing proficiency,
text structure, coherence of arguments, and appropriate use of the standard Portuguese
language.
The participants in this research were required to write a minimum of two essays
using RevisãoOnline. They had the flexibility to choose their essay topics from a wide
range of previously given topics from past ENEM essay tests. The peer-review process
followed the steps outlined in Sect. 3 and depicted in Fig. 1. The design of the peer review
model for this study incorporated various suggestions from the literature considering
participants’ perceptions of the process. These suggestions included:
• Providing a clear rubric to guide the production and review process (Ashton and
Davies, 2015)
• Conducting training for the peer review process (Luo et al., 2014)
• Involving multiple evaluators to provide diverse perspectives on the work. (Cho and
Schunn, 2018)
• Allowing sufficient time for each stage of the peer evaluation process. (Baker, 2016).
The students utilized RevisãoOnline to both write their essays and provide revisions
for their peers in a double-blind revision process.
4.3 The Questionnaire

The questionnaire used in this research was adapted from a previous study that examined
the perceptions of secondary teachers regarding peer review within the context of an in-
service training course (García et al., 2022). The questionnaire consisted of 17 questions
structured on a 5-point Likert scale, primarily addressing three research questions related
to students’ perceptions of the usefulness of peer review, their difficulties in engaging in
such activities, and their views on the incorporation of peer feedback into the assessment
of their work. An additional open-ended question allowed students to provide their own
comments on their perception of the peer-review process.
While previous research has explored students’ perceptions of peer feedback in vari-
ous contexts, such as learning English as a Foreign Language (Altstaedter and Doolittle,
2014), psychology courses (Hanrahan and Isaacs, 2001), and MOOCs (Kasch et al.,
2021), our study specifically focused on students’ perceptions of peer review activities
designed for writing and revising university entrance examination essays. Furthermore,
we were aware that in our particular context, the revision process was guided by a system
that provided instructional support throughout each step, which likely influenced stu-
dents’ perception of the peer-review process, particularly in terms of providing feedback
to their peers.

Our study addresses three research questions related to students’ perception of peer
review feedback. First, we examine the overall effectiveness and value of peer review
feedback as perceived by students. Second, we explore the difficulties students encounter
when engaging in peer review activities. Finally, we investigate how students perceive
the incorporation of their peers’ feedback by teachers in the evaluation process. By
answering these research questions, we gain insights into students’ perspectives and
experiences with peer review and the use of RevisãoOnline, shedding light on its benefits
and challenges from their point of view.
5.1 Student Perception About the Overall Value of Peer Review Feedback
Overall, students conveyed positive perceptions regarding the value of providing and
receiving feedback from their peers in the context of essay writing activities, as indicated
by their responses to the questionnaire. These positive perceptions are depicted in Fig. 2.
Fig. 2. The Benefits of Receiving and Providing Feedback in Peer-Review Activities
Figure 2 shows that a large percentage of students found giving and receiving feed-
back very useful, with little difference between them. In the context of second language
learning, Chang (2016) reported contrasting findings, suggesting that reviewing was
perceived as more beneficial than receiving feedback from peers. However, the litera-
ture does not yet reach a consensus on this matter (Huisman et al., 2018). In our study,
we also explored a related question: whether reviewing other students’ essays enabled
participants to identify areas for improvement in their own writing. Of the 47 students
surveyed, 43 (91.5%) agreed or strongly agreed with this statement. This finding is
consistent with prior research, which has demonstrated that students perceive both pro-
viding and receiving feedback from peers in writing activities as highly beneficial (Wu
and Schunn, 2021). Some of the reasons to support this argument are:
• Different Perspective: Peers can provide a fresh and unique perspective on one’s work.
They bring diverse backgrounds, experiences, and insights that can offer alternative
viewpoints and suggestions for improvement.
• Relatability: Peers are fellow students who share a similar learning journey. They
can relate to the challenges and struggles faced in writing assignments, making their
feedback more relatable and practical.
• Empowerment and Ownership: Receiving feedback from peers empowers students
to take ownership of their learning. It encourages self-reflection, self-assessment, and
active engagement in the improvement of their writing skills.
However, it is important to note that, in our study, the students’ appreciation for
the utility of providing feedback is specific to their experience with peer review using
RevisãoOnline. The system provided step-by-step guidance to students during the review
process, offering helpful tips and identifying issues in the essays being reviewed, such
as grammar mistakes and deviations from the assigned topic.
To gain a deeper understanding of their perception of the peer-review process and
the RevisãoOnline system, a few students were randomly selected for interviews. One
student expressed, “I found RevisãoOnline to be highly beneficial because it analyzed
word repetition. Additionally, the table of connectors provided by the tool was quite
helpful. It clearly highlighted repeated occurrences of words, which was valuable.“
Another student shared their surprise when the system displayed all instances of the
relative pronoun ‘that’ in the text being reviewed, a frequent problem of learners writing
in Portuguese. In these circumstances, it is significant to observe that the automated
functions of the system that assist students in evaluating their peers’ work also influenced
their perception of the usefulness of the peer review process.
5.2 Perceived Difficulties in Engaging in Peer Review Activities
In general, students expressed a strong sense of comfort in both providing feedback to

their peers and receiving feedback from them, as depicted in Fig. 3.
Fig. 3. Sense of comfort of Receiving feedback and giving feedback in peer-review activities
According to these results, students felt at ease in participating in the peer-review

activity. In the interviews, students mentioned that the blind review process contributed
to this matter. However, there were also occasional remarks about a certain level of
discomfort with the feedback they received: “The feedback I usually received was either
too vague or overly critical. Sometimes, it felt like the person barely marked anything,
possibly just to indicate that they reviewed it. Other times, they would highlight things
that, in my opinion, didn’t need to be addressed. Furthermore, 10 out of the 47 students
(21.27%) reported that they found the actual task of reviewing difficult. Some reasons
for this could be the lack of confidence in their own writing abilities: Students who are
not confident in their own writing abilities may find it challenging to provide feedback
on their peers’ essays. Students in a study by Hanrahan and Isaacs (2001), for example,
encountered challenges when evaluating poor assignments, as they often lacked clear
parameters to differentiate between good and bad work. One student explicitly expressed
his struggle, stating that he “had no idea what he was doing” because of the lack of
external benchmarks to compare his work to.
However, it is noteworthy that in our study, 37 out of 47 participants (78.73%)
expressed either a neutral stance or did not perceive the task of providing feedback to their
peers as difficult. This result may be attributed to the use of RevisãoOnline, as the system
guided students through the review process and highlighted various issues in their peers’
work. This support likely contributed to improving students’ confidence in providing
feedback, as mentioned by some students in their interviews. One student expressed
appreciation for the tool’s ability to identify informal words, stating: “For instance,
certain words that are recommended for us not to use because they are more informal…
When they appeared marked in the tool, it was great!”. Another student highlighted the
impact of RevisãoOnline on their understanding of essay structure, stating: “It was while
doing my first review [with RevisãoOnline]… That I realized that my essays never had
an intervention proposal with all the required elements… Then, by reviewing, I was able
to start identifying and seeing how each little part of the conclusion functions.“
5.3 Incorporating Peers’ Feedback in the Evaluation Process

When considering students’ perception of incorporating peer feedback into the evalua-
tion process by teachers, it is worth noting that while a significant majority of students
agreed or strongly agreed that it is appropriate for teachers to utilize the revisions made
by their peers in assessing their essays (Fig. 4), there remains a notable percentage
(25.5%) who disagreed with this approach. The primary concern of students revolves
around the teacher’s lack of verification regarding the appropriateness of the peer feed-
back received, as expressed in a specific comment: “I believe that the teacher should
evaluate the revisions of the students who provided evaluations as well, in case any errors
or similar issues occurred”. While there is no definitive consensus in previous research
on this matter, Loretto et al. (2016) highlight that some students express a preference
for receiving feedback from multiple sources instead of relying solely on their peers
or their teacher. Therefore, adopting a balanced approach that integrates feedback from
both peers and teachers can provide valuable benefits for students, enabling them to gain
insights from diverse perspectives, as highlighted in one of the interviews: “Peer-review
broadens our knowledge by exposing us to different perspectives, and by analyzing and
correcting other revisions, we can incorporate key points of argumentation into our
future essays.“
Fig. 4. Students’ perception of the use of their peers’ feedback in the assessment of their work
6 Conclusion
The study presented in this article provides insights into students’ perception of peer
review feedback in the context of essay writing activities. The research focused on the
use of a specific online tool, which serves as a guide for students during the reviewing
process. By examining students’ experiences and perspectives, the study seeks to shed
light on the effectiveness and value of peer review feedback in enhancing their writing
skills and overall learning outcomes.
The findings indicate that students generally view peer feedback as valuable and
beneficial, both in terms of providing feedback to their peers and receiving feedback from
them. This aligns with previous research in the field, which suggests that peer feedback
can contribute to improved writing skills and overall learning outcomes. However, it
is important to acknowledge that the students’ positive perception of peer feedback
has been influenced by their experience with RevisãoOnline, which provided guidance
and support throughout the revision process. Moreover, students expressed a sense of
comfort in engaging in peer review activities, with a majority not considering the task
of providing feedback to their peers as difficult. This highlights their willingness and
ability to actively participate in the peer review process.
Regarding the incorporation of peer feedback by teachers in the evaluation process,
there was a mixed response from students. While a significant majority agreed that
teachers should consider peer revisions when assessing their essays, a notable percentage
disagreed with this approach. This suggests that students value the diverse perspectives
offered by both their peers and teachers in the feedback process.
These findings stress the importance of incorporating peer review activities in writing
pedagogy, as they provide students with opportunities to enhance their writing skills, and
engage in meaningful interactions with their peers. Furthermore, our study highlights
the need for further exploring the role of teachers in integrating peer feedback into
the evaluation process, taking into account students’ preferences and perceptions. In
this context, it’s also relevant to consider the potential of employing Natural Language
Processing (NLP) in text assessment, an approach that has garnered significant attention
in recent years (Liu et al., 2021). Additionally, new research tracks are now emerging
due to the availability of large language models, like GPT, aimed at supporting students
in writing and reviewing activities.
Acknowledgement. This research received funding from the National Agency for Research and
Innovation (ANII) in Uruguay, under the code FSED_2_2021_1_169701.
References
Altstaedter, L.L., Doolittle, P.: Student’s perceptions of peer feedback. Argentinian J. Appl.
Linguis. 2(2), 60–76 (2014)
Ashton, S., Davies, R.S.: Using scaffolded rubrics to improve peer assessment in a MOOC writing
course. Dista. Educ. 36(3), 312–334 (2015)
Atwater, L.A., Waldman, D., Atwater, D., Cartier, P.: An upward feedback field experiment:
supervisor’s cynicism, follow-up and commitment to subordinates. Pers. Psychol. 53, 275–297
(2000)
Baker, K.M.: Peer review as a strategy for improving students’ writing process. Act. Learn. High.
Educ. 17(3), 179–192 (2016)
Bali, M.: MOOC pedagogy: gleaning good practice from existing MOOCs. Merlot J. Online Learn.
Teach. 10(1), 13 (2014)
Carless, D., Boud, D.: The development of student feedback literacy: enabling uptake of feedback.
Assess. Eval. High. Educ. 43(8), 1315–1325 (2018). https://doi.org/10.1080/02602938.2018.
1463354
Chang, C.Y.: Two decades of research in L2 peer review. J. Writ. Res. 8(1), 81–117 (2016)
Cho, K., MacArthur, C.: Student revision with peer and expert reviewing. Learn. Instr. 20(4),
328–338 (2010)
Cho, K., Schunn, C.D.: Finding an optimal balance between agreement and performance in an
online reciprocal peer evaluation system. Stud. Educ. Eval. 56, 94–101 (2018)
Cui, Y., Schunn, C.D., Gai, X., Jiang, Y., Wang, Z.: Effects of Trained Peer vs. Teacher Feedback
on EFL Students’ Writing Performance, Self-Efficacy, and Internalization of Motivation. Front
Psychol. 12, 788474 (2021 Nov 24). https://doi.org/10.3389/fpsyg.2021.788474
Ellis, R.A., Taylor, C.E., Drury, H.: Evaluating writing instruction through an investigation of
students’ experiences of learning through writing. Instr. Sci. 33, 49–71 (2005). https://doi.org/
10.1007/s11251-004-7686-y
García, S., Motz, R., Reategui, E.: Retroalimentación entre pares como estrategia de evaluación
formativa desde la perspectiva de las personas participantes. In: V Congreso Latinoamericano
y Caribeño de Ciencias Sociales, pp. 270–288. Montevideo, Uruguay (2022)
Garcia-Loro, F., Martin, S., Ruipérez-Valiente, J.A., Sancristobal, E., Castro, M.: Reviewing and
analyzing peer review Inter-Rater Reliability in a MOOC platform. Comput. Educ. 154, 103894
(2020)
Hanrahan, S.J., Isaacs, G.: Assessing Self- and Peerassessment: The students’ views. High. Educ.
Res. Dev. 20(1), 53–70 (2001). https://doi.org/10.1080/07294360123776
Hovardas, T., Tsivitanidou, O.E., Zacharia, Z.C.: Peer versus expert feedback: An investigation
of the quality of peer feedback among secondary school students. Comput. Educ. 71, 133–152
(2014). https://doi.org/10.1016/j.compedu.2013.09.019
Huisman, B., Saab, N., van Driel, J., van den Broek, P.: Peer feedback on academic writing:
Undergraduate students’ peer feedback role, peer feedback perceptions and essay performance.
Assess. Eval. High. Educ. 43(6), 955–968 (2018)
Kasch, J., van Rosmalen, P., Löhr, A., Klemke, R., Antonaci, A., Kalz, M.: Students’ perceptions
of the peer-feedback experience in MOOCs. Distance Educ. 42(1), 145–163 (2020). https://
doi.org/10.1080/01587919.2020.1869522
Liu, C., Hou, J., Tu, Y-F., Wang, Y., Hwang, G-J.: Incorporating a reflective thinking promoting
mechanism into artificial intelligence-supported English writing environments. Interac. Learn.
Environ. (Dec 2021). https://doi.org/10.1080/10494820.2021.2012812
Loretto, A., DeMartino, S., Godley, A.: Secondary students’ perceptions of peer review of writing.
Res. Teach. Engl. 51(2), 134–161 (2016). http://www.jstor.org/stable/24889912
Luo, H., Robinson, A.C., Park, J.-Y.: Peer Grading in a MOOC: Reliability, Validity, and Perceived
Effects. Online Learning 18(2) (2014)
Mulder, R.A., Pearce, J.M., Baik, C.: Peer review in higher education: Student perceptions before
and after participation. Act. Learn. High. Educ. 15(2), 157–171 (2014)
Falchikov, N.: The place of peers in learning and assessment. In: Boud, D., Falchikov, N.
(eds.) Rethinking assessment in higher education: Learning for the longer term, pp. 128–143.
Routledge, London (2007)
Orjuela-Laverde, M., Chen, L.R.: Peer Review as an Active Learning Strategy in a Large FirstYear
Course. In: Proceedings of the 2014 Canadian Engineering Education Association Conference
(CEEA14). Canmore, AB; June 8–11, 1–5 (2014)
Pearce, J., Mulder, R., Baik, C.: Involving students in peer review: Case studies and practical
strategies for university teaching. Centre for the Study of Higher Education, University of Mel-
bourne (2009). https://melbourne-cshe.unimelb.edu.au/__data/assets/pdf_file/0006/3590943/
Involving-students-in-peer-review.pdf
Struyven, K., Dochy, F., Janssens, S.: Students’ perceptions about evaluation and assessment in
higher education: a review. Assess. Eval. High. Educ. 30(4), 325–341 (2005). https://doi.org/
10.1080/02602930500099102
Topping, K.: Peer assessment between students in colleges and universities. Rev. Educ. Res. 68(3),
249–276 (1998)
Topping, K.: Peer Assessment. Theory Into Practice 48(1), 20–27 (2009)
Wu, Y., Schunn, D.: The effects of providing and receiving peer feedback on writing performance
and learning of secondary school students. Am. Educ. Res. J. 58(3), 492–526 (2021). https://
doi.org/10.3102/0002831220945266
The CLG Framework: A Methodical Approach
to Designing Educational Video Games
Alberto Rojas-Salazar1(B) and Mads Haahr2

1 Universidad de Costa Rica, San José, Costa Rica
alberto.rojassalazar@ucr.ac.cr
2 Trinity College Dublin, Dublin, Ireland
Abstract. This paper introduces the Context Learning Game (CLG) framework,
a game design framework developed for educational video games in formal educa-
tion settings. The CLG framework is structured into three stages: context, learning,
and game. The context stage establishes general game aspects such as the objec-
tives, target audience, and available resources. The learning stage defines formal
learning components, including learning theory, content, learning objectives, and
learning activities. Finally, the game stage encompasses the formal game elements,
development, and testing. The stages are hierarchical, with the context forming the
foundation for the learning stage, and both serving as the basis for the game stage.
The framework aims to integrate concepts from educational video game design,
entertainment game design, and educational design while providing a simple iter-
ative workflow. To illustrate the framework?s potential, a successful case study of
an educational game?s design, development, and testing process is presented.
Keywords: educational video game · design framework · design model
1 Introduction
Creating educational video games is a challenging activity that requires careful con-
sideration of both the formal aspects of a video game and the educational elements it
encompasses. Finding the right balance between these two dimensions can be a diffi-
cult task. Consequently, the utilization of comprehensive yet user-friendly frameworks
or methodologies becomes essential in guiding the design, development, and testing
processes of educational video games.
This paper introduces the Context Learning Game (CLG) framework, a comprehen-
sive design framework for educational video games. The CLG framework specifically
focuses on creating engaging and enjoyable educational video games for formal edu-
cational settings. Criticism has often been directed towards educational video games
for their failure to deliver an enjoyable experience [30]. Additionally, it is known that
disguising educational content can lead to negative learning outcomes [30]. To address
these concerns, the CLG framework combines concepts and principles from educational
game design frameworks and entertainment video game methodologies. The application
of entertainment game design principles brings several benefits: (1) it aims to create an
https://doi.org/10.1007/978-981-99-7353-8_20
The CLG Framework: A Methodical Approach to Designing Educational Video Games 255
enjoyable experience that meets the player?s expectations; (2) it provides mechanisms
to systematize the design process; and (3) it incorporates testing as an integral part of
the design process.
Additionally, the CLG framework utilizes the alignment principle, which suggests
for the alignment of all learning components with a learning theory (as explained in
Sect. 2.4), to guide the design process of the educational aspects in video games. The
alignment principle offers several benefits during the design process: (1) it enables the
creation of theory-based foundations for the learning process; (2) it helps the designer
integrate the learning aspects with the game?s ludic elements; and (3) it provides a means
to assess the players? learning progress and the effectiveness of the video game. The
alignment principle ensures that the video game meets the rigorous requirements for use
in a formal educational environment.
The CLG framework offers several contributions: (1) it integrates concepts and prin-
ciples from educational game design, entertainment game design, and educational design
into a framework with a simplified structure and workflow; (2) thanks to the learning
alignment principle, educational video games developed using the framework will pos-
sess the necessary theoretical rigor for implementation in formal educational environ-
ments; and (3) it provides guidance on the artifacts (e.g., game documents, reports,
prototypes, etc.) that may be developed at each stage of the workflow.
The objective of this paper is to make the CLG framework widely accessible and
available for independent use and validation by other researchers. Furthermore, we
present a compelling case study of the CLG framework by applying it to a specific game
design process discussed in Sect. 4. While we acknowledge that this case study does
not constitute formal validation of the framework, it serves to demonstrate the frame-
work?s promising potential. We hope that this example will inspire other researchers
and designers to adopt the CLG framework in their own projects.
This paper is organized as follows: Sect. 2 provides a literature review related to
design frameworks for educational video games, design methodologies for entertain-
ment video games, and the alignment principle. In Sect. 3, we present the CLG frame-
work. Section 4 showcases a successful case study of the CLG framework by presenting
a successful educational video game that was designed using it. Section 5 discusses
the features and limitations of the framework. Finally, Sect. 6 concludes the paper,
summarizing the key findings and contributions.
2 Related Work
2.1 Design and Evaluation Frameworks for Educational Video Games
This subsection provides a review of six design frameworks for educational video games,
forming the foundation of the CLG framework. Each review highlights the features and
limitations of the respective framework. Our review focuses exclusively on frameworks
specifically designed for serious educational games, excluding those intended for her-
itage, marketing, or rehabilitation purposes. Additionally, we have omitted general-
purpose serious game design frameworks as they do not emphasize the theoretical
foundations of learning.
256 A. Rojas-Salazar and M. Haahr
The Four-Dimensional (4D) framework [12] is an assessment tool designed to eval-

uate the learning potential of educational video games. Its objective is to assist educators
in selecting appropriate games and learning strategies that align with their specific learn-
ing objectives (LOs) and environment. The framework encompasses four dimensions:
context, learners, internal representational world, and processes of learning. The con-
text dimension considers the learning environment, including historical, political, and
economic factors, as well as the availability of resources and the instructor?s technical
understanding. The learner dimension assesses factors such as age, knowledge level,
learning background, styles, and preferences. The third dimension focuses on the inter-
nal representational world of the game, including aspects like mode of representation,
interactivity, immersion levels, and fidelity. Lastly, the processes of learning dimen-
sion evaluates the methods, theories, models, and frameworks that support the learning
experience. The main strength of the 4D framework lies in its emphasis on learning
aspects and the contextual factors that influence game design. By considering learning
theory, content, activities, learner characteristics, and available facilities, the framework
enables the creation of games that meet specific learning requirements. However, it
does not provide extensive information regarding the design and development of game
elements.
The framework introduced by Aleven et al. [2] serves as a tool for analysing and
designing educational video games. It consists of three components: (1) learning objec-
tives, (2) the MDA framework (Mechanics, Dynamics, and Aesthetics), and (3) Instruc-
tional Design Principles. The first component, learning objectives, focuses on defining
specific goals to enhance the effectiveness of the learning process. The MDA framework,
which originally analyses entertainment video games [15], examines games through
three layers: mechanics (game components such as materials, rules, goals, basic moves,
and control options), dynamics (player interactions and resulting behaviours), and aes-
thetics (player experiences and emotional responses). The third component, Instructional
Design Principles, incorporates research-based principles for instructional design. For
instance, the framework suggests utilizing principles like the Multi-Media Principles
[21], the Cognitive Tutor principles [3], and Gee?s 36 principles of game-based learning
[13]. The primary strength of the framework lies in its emphasis on learning aspects, as it
emphasizes the importance of defining LOs and incorporating relevant learning theories
or principles to create effective educational games. However, the framework does not
account for the specific contextual factors that influence the learning experience.
The Serious Game Design Assessment (SGDA) framework, designed for evaluating
educational video games [24], focuses on six key components: purpose, contents and
information, game mechanics, fiction and narrative, aesthetics and graphics, and fram-
ing. The purpose component assesses the game?s objectives and the intended impact
on players. The content and information component evaluates the information and facts
presented within the game. The game mechanics component describes the actions and
interactions available based on the game?s rules and algorithms. The fiction and narra-
tive component analyses the fictional elements, such as settings, story, characters, and
problem presented in the game. Aesthetics and graphics encompass the audio-visual
aspects that materialize the game?s content, fiction, framing, setting, and mechanics.
The framing component considers the target group and their characteristics. The SGDA
framework excels in its emphasis on the playful aspects of educational game design and
their role in facilitating learning. However, the framework primarily focuses on analysis,
neglecting aspects like prototyping and testing in the design process.
The Game for Learning Institute?s (G4LI) framework for Game-Based and Play-
ful Learning examines and designs educational video games from a pedagogical and
psychological perspective [25]. It emphasizes four foundations: affective, behavioural,
cognitive, and socio-cultural. The cognitive foundation focuses on processing learn-
ing content and cognitive load during gameplay. The framework suggests incorporating
learning principles such as situated learning, transfer of learning, scaffolding and feed-
back, and dynamic assessment. The motivational foundation addresses game aspects
that promote engagement and continued participation, including intrinsic and extrinsic
motivation, individual interests, and goal orientation. The affective foundation considers
emotions, attitudes, and how game elements can foster positive engagement for learning.
The socio-cultural foundation focuses on facilitating social interactions and knowledge
construction in games. G4LI?s framework highlights educational, affective, and social
aspects but does not consider contextual factors such as aspects of game design, available
resources, and development context.
The GAGE (Goal, Audience, Game, Environment) framework [18] emphasizes four
essential categories (goal, audience, game, and environment) along with two optional
elements (stakeholders and experience) in educational video game design. The goal cat-
egory defines LOs, content, and instructional methods. The audience category considers
player characteristics, including demographics and academic information. The game cat-
egory focuses on the logical, functional, and technological aspects of the game. The envi-
ronment category addresses social, physical, and technological factors that may impact
gameplay. The stakeholder category involves actors involved in design, development,
and distribution. The experience category encompasses elements added by the designer
to enhance player motivation. Deep learning is emphasized, requiring the selection of
an appropriate learning theory. The GAGE framework is comprehensive, considering
design context, learning aspects, and game elements. However, it lacks specific guidance
for the design and development process, such as a workflow or inventory of necessary
artifacts.
Medina-Medina et al. [23] propose four key axes that guide the design process of
an effective educational video game: (1) integrating the educational team throughout
the design and development stages, (2) incorporating educational aspects into the game
elements, (3) utilizing an engaging narrative for learning, and (4) implementing an assess-
ment system within the game. The framework places particular emphasis on learning
and assessment aspects, increasing the likelihood of meeting educational requirements.
However, it does not extensively address the ludic elements of the educational video
game, such as game mechanics, levels, and challenges, nor does it provide specific
guidelines for their design.
2.2 Game Design Methodologies for Entertainment Video Games

The Iterative Design methodology for games emphasizes playtesting and prototyping
[29]. It advocates making game design decisions based on previous experiences gath-
ered through game testing. As players? experiences are hard to predict, early testing
with players is vital to increase the chance of success. Thus, game designers focus on
developing an early prototype, primarily addressing key aspects like the game system,
mechanics, and player interactions. Aesthetics (visual art, sound, music, etc.) are added
in later iterations. The designer conducts playtesting, evaluates the results, modifies the
game system based on insights, and repeats the process multiple times until the game is
considered ready for delivery.
The Player-Centric framework proposes that game designers should focus on a rep-
resentative player when designing their games [1]. According to this framework, a suc-
cessful game should (1) entertain the player and (2) fulfil the player?s expectations and
desires. To implement this methodology, designers need to conduct audience research
or envision an ideal player. The representative player archetype should include infor-
mation about the player?s gaming preferences, likes, dislikes, and motivations. Using
this player profile, designers can make informed design decisions that align with the
player?s expectations, increasing the chances of matching the players? capabilities and
limitations [14]. The Player-Centric framework follows a three-stage approach for video
game design: (1) concept, (2) elaboration, and (3) tuning [1]. However, we believe that
the Iterative Design methodology offers a more effective development approach.
2.3 Game Design Documents

A game design document is a proposal, plan, or record of a game, consisting of a descrip-
tion of its concept and main aspects. It serves two fundamental purposes: communication
and guidance. In essence, it informs others about the game?s content while ensuring con-
sistency among all team members involved in the video game?s development concerning
its main concept [27]. During its early stages, the game design document is subject to
continuous criticism and feedback, allowing many design concerns to be resolved at this
stage.
The structure of a game design document can vary from one designer to another;
however, the aims are similar. According to T. Ryan [27, 28], there are four types of
game design documents: (1) game concept; (2) game proposal; (3) game functional
specifications; and (4) game technical specifications. Each document serves a distinct
purpose and is built upon the foundation of the previous one.
A different classification is proposed by Adams [1], who defines nine types of game
design documents: (1) high concept document; (2) game treatment document; (3) char-
acter design document; (4) world design document; (5) user interface design document;
(6) story and level progression document; (7) on-screen text and audio dialog scripts; and
(8) game script. It is not necessary to create all of them, and some can be combined into
a single document. Adams suggests that game designers should write these documents
for five reasons: (1) to record design decisions; (2) to transform ideas into specific plans;
(3) to communicate the game concepts and guide the development stages of the game;
(4) to set contractual obligations; and (5) to find funding.
2.4 Alignment Principle

When designing a learning experience (e.g., lessons, simulations, learning tools, etc.), it
is necessary to define the LOs, learning activities, and assessment methods [7]. However,
according to the alignment principle, these learning components must be aligned with
a learning theory that defines how people learn [22]. Adopting a learning theory helps
in selecting pedagogical methods that support the learning assumptions. An excellent
example of the alignment principle is constructive alignment [7], which involves fram-
ing all learning elements within the constructivist paradigm. Constructive alignment is
based on the idea that individuals construct their knowledge. As a result, to achieve the
LOs, learning experiences must employ pedagogical methods and activities that promote
active learning. Additionally, assessment methods should evaluate the construction of
knowledge through active learning.
The adoption of alignment principles offers three main advantages: (1) it systematizes
the design of learning experiences, which is essential in formal education; (2) it guides the
consistent definition of pedagogical methods, learning activities, and assessment meth-
ods aligned with theory-based learning assumptions; and (3) it provides the theoretical
foundations to critically evaluate the success or failure of the designed experience.
3 The Context Learning Game Framework

The CLG framework suggests dividing the design process for educational video games
into three stages: (1) context, (2) learning, and (3) game (see Fig. 1). Each stage addresses
a critical aspect of the video game design process. The framework follows a hierarchi-
cal structure, where each stage serves as the foundation for the next, becoming more
specialized and complex. Specifically, the context stage forms the basis for the learning
stage, and together, they form the foundation for the game stage. Additionally, the CLG
framework emphasizes an iterative design process, represented graphically as a circu-
lar pattern of arrows around the pyramid. The following subsections provide detailed
explanations of each stage in the framework.
3.1 The Context Stage
The context stage forms the foundation of the pyramid and must be prioritized. Effective
design necessitates a thorough comprehension of the problem at hand. To properly
understand the problem, designers must analyse and define key contextual elements
relevant to the development and utilization of the designed object. For example, designers
should consider the individuals responsible for designing and developing the object, the
intended users of the tool, available design and development resources, and the intended
usage environment. Through a comprehensive analysis and definition of the context,
designers can identify strengths and constraints that may impact the design process and
subsequent design decisions. By understanding the context, designers can enhance the
likelihood of finding an optimal solution to the problem at hand.
During the context stage, the game designer analyzes the problem at hand, which
involves designing an educational video game, as well as the available resources to
address the problem. The following list describes some context aspects that designers
may contemplate:
Fig. 1. The CLG framework and its stages.
1. The purpose and scope set the primary objectives and scope of the educational video
game. This section offers a concise overview of the intended functions, content cover-
age, project scale, and desired final presentation of the game. Its purpose is to provide
guidance for future design expectations and decisions. Defining this aspect requires
close collaboration between the designer and relevant stakeholders involved in the
game?s development. A clear definition of the purpose and scope of the video game
facilitates the establishment of essential elements within the context, learning, and
game stages.
2. The educational environment encompasses the specific type of learning setting in
which the educational experiences will occur. Designers should also consider internal
regulations, organizational culture, and political factors that may impact subsequent
design decisions. Furthermore, the educational environment considers whether it is a
formal environment, such as a high school or university, or an informal setting, such
as a student?s home.
3. The target audience should cover key characteristics of the players/learners who
will engage with the game. Essential aspects include demographic data, academic
level, and gaming experience. This information proves valuable for defining both
the learning aspects (e.g., learning theory, covered content, LOs, etc.) and the game
aspects (e.g., game genre, aesthetics, narrative, challenge difficulty, etc.). An effective
approach to gaining a deeper understanding of the target audience and guiding design
decisions is the creation of personas as suggested in [9, 10]. Personas are profiles that
represent anticipated players/learners.
4. The facilities and equipment section defines the specific location where the educa-
tional video game will be played, such as a classroom, laboratory, home, outdoor
space, etc. It also summaries the available equipment for playing the game, such
as high-end computers, low-end computers, tablets, smartphones, etc. This informa-
tion proves valuable for making appropriate decisions regarding the learning aspects,
game aspects (e.g., user interface, aesthetics, input mechanisms, etc.), and the game?s
technological requirements.
5. Deadlines are a crucial consideration. Time restrictions have a significant impact on
various design decisions, including the complexity of the game, the number of levels
and challenges, and the quality of the graphics. It is essential to account for deadlines
as they influence the overall design process and determine the achievable scope within
the given timeframe.
6. Design and development team. This aspect addresses the human resources available
for designing and developing the solution. The composition of the team will determine
its strengths and limitations, which should be carefully considered before commenc-
ing a project, as they will influence subsequent decisions. Moreover, designers should
identify potential collaborators who are not part of the core team but can contribute
to the project, such as illustrators, graphic designers, game developers, subject matter
experts, teachers, researchers, etc.
7. Budget and development tools. This section considers the financial resources and
development tools available for designing and developing the educational video game.
Development tools encompass hardware, software, information sources, assets (such
as 3D models, animations, scripts, etc.), and equipment required for producing mul-
timedia content. It is essential to consider the budgetary constraints and the avail-
ability of development tools to make informed decisions throughout the design and
development process.
All the elements described above should be documented in a concise report and
reviewed by the relevant stakeholders for potential adjustments. This report serves to
provide the designer with a comprehensive understanding of the project context aligned
with the stakeholders? perspectives. Such knowledge of the game context will facili-
tate informed decision-making in subsequent stages. By considering these factors, the
chances of designing a successful educational video game are significantly enhanced.
3.2 The Learning Stage
The learning stage, situated in the middle of the pyramid, builds upon the foundation
established in the context stage and serves as the basis for the game stage. Its objective
is to define essential elements such as the learning theory, content, LOs, pedagogi-
cal approaches, learning activities, and assessment methodologies. The learning aspects
should be carefully aligned with the elements established in the context stage. For exam-
ple, the designer must choose a suitable learning theory that resonates with the target
population, select appropriate content and LOs based on the background and knowledge
level of the audience, and design learning activities that are suitable for the academic
environment and facilities outlined in the context stage.
The learning stage must warrant the alignment principle. It requires a careful align-
ment and integration of the learning objective, learning activities, and assessment meth-
ods with the chosen learning theory. The following list outlines key aspects that need to
be defined within the learning stage:
1. The learning theory serves as the foundation for understanding how the target popula-
tion learns. It provides a systematic and validated approach to comprehending human
learning processes, and it should guide the selection of LOs, pedagogical approaches,
and learning activities within the educational video game. Moreover, a learning the-
ory offers a framework for discussing and evaluating the effectiveness of the game
as an educational tool.
2. Content and learning objectives. The content encompasses the skills or knowledge
that the learner is intended to acquire. The LOs specify the scope and desired level of
mastery that the learner should attain upon successful completion of the educational
video game. It is crucial for the LOs to be clearly stated and measurable. Established
taxonomies of LOs, such as Bloom?s taxonomy [4] or the SOLO taxonomy [7], can
provide valuable guidance in formulating the LOs.
3. The pedagogical approaches refer to the methods employed to teach knowledge or
skills, such as scaffolding or problem-based learning. These approaches are typically
aligned with the underlying learning assumptions defined by the learning theory.
4. The learning activities includes the tasks and exercises that learners are required
to undertake to achieve the LOs. These activities should align with the learning
assumptions, address the LOs, and effectively implement the chosen pedagogical
approaches. Furthermore, it is crucial for the designer to consider that these learning
activities will be translated into ludic activities.
5. Assessment refers to a systematic approach for measuring the extent to which students
have achieved the LOs. Typically, this involves developing an assessment tool or
protocol specifically designed for the purpose. The assessment should align with the
chosen learning theory and the LOs. Additionally, it should possess a comparable level
of difficulty to the learning activities and demonstrate the practical applicability of the
acquired skills or knowledge beyond the context of the video game. Additionally, the
assessment plays a significant role in evaluating the overall efficacy of the educational
video game.
At the conclusion of this stage, the game designer should have prepared a compre-
hensive document encompassing all the learning components. It is important for all the
aspects to undergo a thorough review by stakeholders and validation by subject matter
experts. This document will serve as a guiding resource for subsequent decisions made
during the game stage, particularly in relation to game mechanics, rules, and narrative
development.
3.3 The Game Stage
The game stage, situated at the pinnacle of the pyramid, is constrained by the contextual
and learning requirements defined in the preceding stages. The primary objective of this
stage is to design and develop an engaging and enjoyable educational video game. It is
divided into three iterative and intertwined substages: (1) game design, (2) development,
and (3) testing, each producing its own corresponding artifact. In the game design sub-
stage, the game designer defines and describes the key elements of the educational video
game in a game design document. As soon as possible, the designer should jump into the
development substage that entails the creation of an initial prototype for initial evaluation
and validation. The testing substage involves evaluating the prototypes, encompassing
aspects such as engagement, enjoyment, user experience, and learning efficacy. Fol-
lowing game testing, the designer may revisit previous substages of the game stage or
prior stages of the framework to modify sections that do not meet the video game?s
requirements or return to the development subsection to increase the game prototype?s
features.
The game stage, positioned at the pinnacle of the pyramid, is influenced by the con-
textual and learning requirements defined in the preceding stages. The primary objective
of this stage is to design and develop an engaging and enjoyable educational video game.
It comprises three iterative and intertwined substages: (1) game design, (2) development,
and (3) testing, each generating its own corresponding artifact. In the game design sub-
stage, the game designer defines and describes the essential elements of the educational
video game in a game design document. Rapidly, the game designer should transition to
the development substage to create an initial prototype for preliminary evaluation and
validation. The testing substage involves evaluating the prototypes, considering aspects
such as engagement, enjoyment, user experience, and learning efficacy. After game test-
ing, the designer may revisit previous substages of the game stage or prior stages of the
framework to modify sections that fail to meet the video game?s requirements or return
to the development subsection to enhance the features of the game prototype.
The Game Design Substage. The primary objective of the game design substage is
to define various game elements, including game mechanics, challenges, story, aes-
thetics, user interface, and game world. This substage necessitates close collaboration
with subject matter experts and stakeholders. To ensure effective management and com-
munication of design decisions, designers can utilize game design documents. These
documents serve as a cost-effective tool for defining and visualizing the core concepts
of the game, enabling designers to identify any flaws and make swift corrections. During
the process of creating the game design document, designers may also conduct minor
research and prototyping tasks to explore concepts that could be incorporated into the
document.
The game design substage is constrained by the context and learning aspects. The
designer must align the game elements with the target audience, available resources,
learning theories, content, pedagogical approaches, LOs, and learning activities. Col-
laborating with subject matter experts and stakeholders is crucial to ensure the accurate
definition of these aspects. They should review and validate the design decisions made
by the design team.
When considering the learning aspects, the designer must effectively transform
the learning activities into engaging ludic activities using game elements such as
game mechanics, challenges, narrative, game world, and rules. It is essential for the
game design document to clearly demonstrate the alignment between the LOs, learn-
ing activities, and ludic activities. To facilitate this mapping between the learning and
game aspects, the designer can utilize frameworks like the Game Mechanics Learning
Mechanics framework [5].
Concerning the entertainment aspects, the designer should consider the factors that
contribute to making a video game enjoyable, including challenges, fantasy, curiosity,
control, competition, and social features [8, 19, 20]. These elements enhance intrinsic
motivation and player engagement, both of which are desirable qualities for facilitating
learning [16].
The game design substage should be efficient to allow for rapid prototyping, as
many design decisions may evolve throughout the process. However, it remains crucial
to maintain a set of game design documents that captures and tracks all design decisions,
enabling effective communication within the design and development team. This docu-
ment serves as a reference point and facilitates collaboration, ensuring that everyone is
aligned and informed throughout the iterative design process.
The Game Development Substage. Meeting all the requirements of an educational
video game is a challenging task that needs continuous testing. Hence, early prototyping
plays a fundamental role. By gradually building and testing initial versions of game pro-
totypes, designers can gather valuable feedback from players and subject matter experts
to validate the game?s effectiveness. If the prototype meets the desired requirements,
the development team can proceed with implementing more advanced versions of the
game, refining it further until it meets all specifications. In cases where the prototype
falls short of the requirements, the designer should revisit the game design substage or
previous stages of the framework for necessary adjustments and improvements.
The initial prototype should be a small, playable version of the game. It does not
need to be a fully polished but should effectively demonstrate the core concept, game
mechanics, and the main learning aspects. The primary objective of the game prototype
is to conduct testing. Consequently, a cost-effective and efficient development tool is
essential before proceeding to create the final version of the game. Depending on the
team?s capabilities, the development team can opt for paper-based prototypes or digital
prototypes. The choice of prototype medium should align with the team?s resources and
expertise.
At the end of the first iteration of the development substage, the design and develop-
ment team should have a functional prototype of the game. This prototype will be utilized
for subsequent testing and will serve as a foundation for refining future iterations of the
game prototype as well as the final version of the game.
The Testing Substage. The validation and testing process is essential to determine
whether the educational video game meets the required standards. As soon as the first
prototype is implemented, the testing substage begins.
The validation process is essential to ensure that the video game effectively delivers
the intended contents and teaching methods. Subject matter experts, educators, and
stakeholders play crucial roles in this stage. Validation can be formal or informal. Also,
various qualitative or quantitative research methods can be employed for validation.
The development team should carefully choose the most suitable option based on their
specific context and skills.
Game testing should evaluate the learning efficacy and factors related to the use
of the video game, including usability, user experience, usefulness, and instructional
aspects. It should be performed on a sample with the same characteristics as the target
population. Like the validation process, testing can be formal or informal, employing
qualitative or quantitative methods.
At the end of the testing stage, the designer should have prepared a report describing
the methods used to collect data, the results of the evaluation, an analysis of the data,
and a discussion of the findings. The report should highlight the strengths of the game
as well as any gaming and learning concerns that need to be addressed. Additionally, it
may identify unexpected player behaviours.
4 Case Study
This section provides an example of the CLG framework?s application, showing its
potential. It showcases the design, development, and testing process of DS-Hacker, an
educational video game aimed at teaching binary search tree (BST) data structures to
higher education students. The focus is on the CLG?s workflow used to successfully
design and develop the game.
The context stage covered aspects such as purpose, objectives, learning environment,
target population, resources, and location. Additionally, a review of game engines and
assets was conducted, including sound effects, particle effects, scripts, and 3D models.
The objective was to find a development environment suitable for rapid prototyping and
offering high asset quality. Three game engines, Godot, Unreal Engine, and Unity were
tested and reviewed. Unity was ultimately chosen for several reasons: (1) it offers a free
licensing option suitable for academic purposes, (2) it provides a comprehensive and
well-documented library for 3D development, (3) Unity?s Asset Store contains quality
assets and tools for fast prototyping, and (4) the developer was proficient in Unity.
In the learning stage, all the pedagogical aspects were defined. Firstly, Kolb?s expe-
riential learning theory (KELT) [17] was chosen due to its resemblance to the game
cycle of video games [31]. KELT suggests that learning is a holistic process where indi-
viduals construct and reconstruct knowledge based on their experiences. After selecting
the learning theory, the contents and LOs were determined, following the guidelines for
undergraduate degree programs developed by the Association for Computing Machinery
[6]. The LOs covered BST introductory concepts and algorithms, considering the knowl-
edge level of the target audience. An analogy-based pedagogical approach was chosen to
facilitate learning, as it allows the construction of new knowledge using familiar knowl-
edge [11]. Then, learning activities were drafted to be translated into ludic activities and
aligned with the defined learning aspects. For assessment, a BST conceptual knowledge
test was developed. All these learning components were included in the game design
document.
The design and development process of DS-Hacker suffered three major iterations
with several modifications. In the game design substage, elements such as game genre,
theme, aesthetics, mechanics, levels, challenges, goals, narrative, and game world were
defined. The action-adventure genre was selected for its inclusion of physical and cogni-
tive challenges [1], making it suitable for learning conceptual and procedural knowledge.
Additionally, its reliance on narrative elements facilitates conveying conceptual knowl-
edge. As for the theme and visual aesthetics, the science fiction (cyberpunk) theme
was chosen for its relevance to computer science elements and the superior quality of
available assets compared to other themes.
Regarding the narrative and game world, they were designed to convey the learning
content. The game story revolves around a robot created to hack a corrupt corporation?s
data centre, structured as a BST, and reflecting the BST data structure. To overcome
challenges, players must learn about BST concepts presented through the game?s story
and dialogue system.
Regarding levels and challenges, each level focuses on one or two LOs and includes
an appropriate number of game challenges or activities matching the learning activities.
The challenges serve to practice BST concepts taught in the game story, comprising
cognitive puzzles and navigation tasks through the game world.
After completing the game design document, the development substage commenced.
The first digital prototype consisted of six levels and covered five LOs, with an in-game
tutorial explaining game mechanics and controllers using text. A complete description
of the game can be found in [26].
During the testing substage, the first prototype underwent informal validation by two
Algorithms and Data Structures professors at Trinity College Dublin. They provided
feedback on content, data structure representation, user interface (UI), and playability.
Based on the results, minor changes were made to the UI (navigation map, dialogue
system, and menus) and the organization of the game world to better reflect the BST
model. The second prototype was also made available in English and Spanish.
To assess the second prototype?s effectiveness, a pilot experiment was conducted
with thirty-two engineering students from Universidad de Costa Rica. The pre-test post-
test experiment evaluated learning gains, perceived learning, clarity of contents, intrin-
sic motivation, usability, user experience, usefulness, and presentation of instructions.
The results showed increased learning gains and motivation among students, but some
pedagogical and playability problems hindered the learning process.
To address the observed obstacles, the framework stages were revisited and modified.
Changes were made in the context, where the target audience and technological resources
were adjusted. Initially, it was assumed students had high video game proficiency, but
the pilot experiment revealed some struggled with game controls. Also, the available
computer power in laboratories fell short of expectations, leading to performance and
user experience issues during the pilot experiment.
In terms of learning aspects, content and LOs were reduced, and a clearer explanation
of the Binary Search Tree data structure was added, accompanied by explanatory images
for better understanding.
Game components suffered four modifications: (1) a tutorial level was included to
teach game mechanics; (2) the dialogue system and content were updated; (3) level goals
were divided into smaller sets of challenges; and (4) in-game signages were added for
player guidance. Additionally, graphic quality was lowered to accommodate computers
with lower specifications.
The final version of the video game was evaluated with students from Universidad
de Costa Rica and Universidad de Colombia. Results demonstrated its effectiveness, as
players increased their learning gains and intrinsic motivation. A full description of the
evaluation is available in [26].
5 Discussion
The CLG framework unifies educational game design, entertainment video game design,
and pedagogical design into a simple, easy-to-use workflow, systematically addressing
learning and gaming aspects.
The framework is structured into three stages, providing an iterative design process.
Each stage focuses on specific aspects that serve as the foundation for subsequent stages.
Design decisions are documented using artifacts like design documents, reports, or pro-
totypes, supporting communication, criticism, and modifications. The iterative nature of
the framework allows for problem-solving and validation.
In terms of learning, the CLG framework employs the alignment principle, sup-
porting learning and formal education environments. It helps define LOs, assessment
methods, and evaluation criteria for the video game?s efficacy. Furthermore, the frame-
work recommends documenting the learning aspects to guide future design decisions
and ensure the video game meets learning requirements.
For the game stage, the framework enables the balance between learning and enjoy-
able game elements. The game design substage facilitates systematic definition, criti-
cism, and modification of video game components through design documents, aligning
them with learning requirements. Rapid prototyping and multiple iterations are rec-
ommended to achieve a balance between learning, fun, and engagement. The testing
substage assesses the video game?s performance against requirements, identifying and
correcting issues in subsequent iterations.
Additionally, Table 1 lists the design aspects addressed by the CLG framework
and indicates which frameworks or methodologies (reviewed in Sect. 2) consider
those aspects. Through this, we aim to demonstrate our commitment to integrating
the educational and entertainment design concepts and the comprehensiveness of our
framework.
Concerning the limitations, the framework has not been validated extensively. As
shown in the previous section, we have shown a successful case study by applying it
in one educational video game, proving the framework potential. However, it is nec-
essary to develop more educational video games to test the efficacy of the framework.
These educational video games should have different purposes, target audiences, learn-
ing goals, genres, etc. to validate the flexibility of the framework. Additionally, the CLG
framework?s process and workflow have not been compared with other frameworks?
workflow. This makes it difficult for designers and development teams to visualize the
strengths and limitations of our framework. However, as mentioned in the introduction,
the aim of this paper is to make widely available, such that it can be used and vali-
dated independently by other researchers. We expect that other researchers find the CLG
framework valuable and use it for their further designs.
Table 1. List of aspects considered by the CLG framework, and the design frameworks and
methodologies reviewed in Sect. 2.1 and 2.2.
CLG 4D Aleven et al. SGDA G4LI GAGE Medina Iterative Player-Centric

Framework Framework Framework [24] Framework [18] et al. Design Design [1]
[12] [2] [25] Framework [29]
[23]
Purpose & X X X X X X
scope
Educational X X X
environment
Target X X X X X X
audience
Facilities & X X X
equipment
Deadlines X X
Team X X X
Budget & X X
tools
Learning X X X X X X
theory
Content X X X X X X
Learning X X X X X
objectives
Pedagogical X X
approach
Learning X X
activities
Assessment X X
Game design* X X X X X X X X X
Development X X X
Testing X X X
* Game design includes the definition of game elements such as rules, game mechanics, game world, levels, challenges, narrative elements,
characters, user interface, etc.
6 Conclusion
This paper has presented the CLG framework, which focuses on creating fun and engag-
ing educational video games while ensuring the fulfillment of learning requirements.
The case study presented in this study showcased its potential. Its straightforward struc-
ture and workflow make it easy to follow. We believe that game designers will find the
CLG framework valuable, as it facilitates the design and development of high-quality,
effective educational video games.
References
1. Adams, E.: Fundamentals of Game Design. New Riders Publishing, Thousand Oaks (2014)
2. Aleven, V., et al.: Toward a framework for the analysis and design of educational games. In:
2010 Third IEEE International Conference on Digital Game and Intelligent Toy Enhanced
Learning, pp. 69?76 (2010). https://doi.org/10.1109/DIGITEL.2010.55
3. Anderson, J.R., et al.: Cognitive tutors: lessons learned. J. Learn. Sci. 4(2), 167?207 (1995).
https://doi.org/10.1207/s15327809jls0402_2
4. Anderson, L.W., Krathwohl, D.R.: A Taxonomy for Learning, Teaching, and Assessing: A
Revision of Bloom?s Taxonomy of Educational Objectives. Longman, New York (2001)
5. Arnab, S., et al.: Mapping learning and game mechanics for serious games analysis. Br. J.
Edu. Technol. 46(2), 391?411 (2015)
6. Association for Computing Machinery (ACM) Joint Task Force on Computing Curricula,
IEEE Computer Society: Computer Science Curricula 2013: Curriculum Guidelines for
Undergraduate Degree Programs in Computer Science. ACM, New York, NY, USA (2013)
7. Biggs, J., Tang, C.: Teaching for Quality Learning at University. Open University Press, New
York (2007)
8. Boyle, E.A., et al.: Engagement in digital entertainment games: a systematic review. Comput.
Hum. Behav. 28(3), 771?780 (2012). https://doi.org/10.1016/j.chb.2011.11.020
9. Canossa, A., Drachen, A.: Patterns of play: play-personas in user-centred game development.
In: DiGRA &#3909 - Proceedings of the 2009 DiGRA International Conference: Breaking
New Ground: Innovation in Games, Play, Practice and Theory. Brunel University (2009)
10. Cooper, A., et al.: About Face: The Essentials of Interaction Design. Wiley Publishing (2014)
11. Duit, R.: On the role of analogies and metaphors in learning science. Sci. Educ. 75(6), 649?672
(1991)
12. de Freitas, S.I., Oliver, M.: How can exploratory learning with games and simulations within
the curriculum be most effectively evaluated? Virtual Learn. 46(3), 249?264 (2006)
13. Gee, J.P.: What Video Games Have to Teach Us about Learning and Literacy. Palgrave
Macmillan, New York (2007)
14. Hodent, C.: The Gamer?s Brain: How Neuroscience and UX Can Impact Video Game Design.
CRC Press, Boca Raton (2017)
15. Hunicke, R., et al.: MDA: A Formal Approach to Game Design and Game Research (2004)
16. Kapp, K.M.: The gamification of learning and instruction: game-based methods and strategies
for training and education. Pfeiffer, United States of America (2012)
17. Kolb, D.A.: Experiential Learning: Experience as the source of learning and development.
Pearson, New Jersey (2014)
18. Lepe-Salazar, F.: A model to analyze and design educational games with pedagogical founda-
tions. In: Proceedings of the 12th International Conference on Advances in Computer Enter-
tainment Technology. Association for Computing Machinery, New York, NY, USA (2015).
https://doi.org/10.1145/2832932.2832951
19. Malone, T.W.: What makes things fun to learn? heuristics for designing instructional com-
puter games. In: Proceedings of the 3rd ACM SIGSMALL symposium and the first SIGPC
symposium on small systems, pp. 162?169. ACM, Palo Alto, California, USA (1980)
20. Malone, T.W., Lepper, M.R.: Making learning fun: a taxonomy of intrinsic motivations for
learning. In: Snow, R.E., Farr, M.J. (eds.) Aptitude, Learning and Instruction III: Conative
and Affective Process Analyses. Erlbaum, Hillsdale, New Jersey (1987)
21. Mayer, R.E., Moreno, R.: Nine ways to reduce cognitive load in multimedia learn-
ing. Educational Psychologist. 38(1), 43?52 (2003). https://doi.org/10.1207/S15326985EP3
801_6
22. Mayes, T., de Freitas, S.: Review of e-learning theories, frameworks and models. Joint
Information Systems Committee, London (2004)
23. Medina Medina, N., et al.: Video games for learning: design considerations and its application
in the video game ?The Time Invaders?: video games for learning: design considerations and
application. In: Ninth International Conference on Technological Ecosystems for Enhancing
Multiculturality (TEEM?21), pp. 150?156. Association for Computing Machinery, New York,
NY, USA (2021). https://doi.org/10.1145/3486011.3486437
24. Mitgutsch, K., Alvarado, N.: Purposeful by design? a serious game design assessment frame-
work. In: Proceedings of the International Conference on the Foundations of Digital Games,
pp. 121?128. Association for Computing Machinery, New York, NY, USA (2012). https://
doi.org/10.1145/2282338.2282364
25. Plass, J.L., et al.: Foundations of game-based learning. Educational Psychologist. 50(4),
258?283 (2015). https://doi.org/10.1080/00461520.2015.1122533
26. Rojas-Salazar, A.: Game-Based Learning of Data Structures Based on Analogies: Learning
Gains and Intrinsic Motivation in Higher Education Environments. Trinity College Dublin
(2022)
27. Ryan, T.: The Anatomy of a Design Document, Part 1: Documentation Guidelines for
the Game Concept and Proposal, http://www.gamasutra.com/view/feature/131791/the_ana
tomy_of_a_design_document_.php (1999)
28. Ryan, T.: The Anatomy of a Design Document, Part 2: Documentation Guidelines for the
Functional and Technical Specifications, https://www.gamasutra.com/view/feature/131818/
the_anatomy_of_a_design_document_.php (1999)
29. Salen, K., Zimmerman, E.: Rule of play: game design fundamentals. The MIT Press,
Cambridge, Massachusetts (2004)
30. Sanchez, E.: Game-based learning. In: Tatnall, A. (ed.) Encyclopedia of Education and Infor-
mation Technologies, pp. 791?798. Springer International Publishing, Cham (2020). https://
doi.org/10.1007/978-3-030-10576-1_39
31. Whitton, N.: Digital Games and Learning: Research and Theory. Routledge, New York (2014)
Ontology for Open Digital Educational
Credentials: A Step Towards Decentralized
Learning Paths
Diego Pinto , Francisco Vargas(B) , and Nelson Piedra
Universidad Técnica Particular de Loja, San Cayetano Alto,

Loja 11 01 608, Loja, Ecuador
{dfpinto1,fovargas,nopiedra}@utpl.edu.ec
Abstract. This manuscript presents the conceptualization and design of

a decentralized ontology tailored specifically for educational credentials.
Designed to act as a tool for structuring personalized learning paths, the
ontology recognizes and responds to the current demand for a more inte-
grated and holistic education system. In designing the ontology, we seek
to ensure the interoperability of educational credentials from different
systems and platforms.
This integration is vital in an increasingly diverse and fragmented
educational world. In addition, it seeks to enhance efficiency and trust
by incorporating secure issuance and verification mechanisms in the pro-
posal, making use of the inherent advantages of decentralized technolo-
gies, particularly Blockchain technology.
Keywords: learning path · credential · ontology · descentralization ·

blockchain
1 Introduction
Education, as one of the fundamental pillars of our society, has undergone a
significant evolution in recent years. For example, the emergence of the con-
cept of open digital educational credentials has introduced a new mechanism
for recognizing and validating learning, allowing for greater flexibility and per-
sonalization in learning paths [1]. However, the decentralization of these routes
poses significant challenges, especially in terms of interoperability, validation
and recognition. In this context, the need for an ontological framework that can
provide a common structure and language for describing, sharing and validating
credentials is highly relevant.
Learning paths are an essential component of educational models. They allow
students to define a personalized path through their education, choosing the
courses and learning experiences that best align with their goals, individual
aspirations and labor market demands. Personalizing these learning paths can
be difficult, considering that educational credentials can come from a variety of
sources. The proposed ontology can serve as a starting point for recommender
https://doi.org/10.1007/978-981-99-7353-8_21
272 D. Pinto et al.
systems, making use of the latest trends in Artificial Intelligence, to generate

truly personalized and relevant pathways for professionals [2].
This manuscript proposes an ontology for educational credentials as a step
towards the decentralization of learning paths. The paper is structured as follows:
In Sect. 1, a brief introduction of the manuscript is provided, highlighting
the need for the ontology and the challenges of decentralization. Section 2 pro-
vides a literature review, exploring key concepts and previous work related to
educational credentials and ontology in the context of education.
In Sect. 3, the proposed ontology framework for educational credentials is
presented, detailing its structure and functions. Section 4 describes a use case
for a university department offering continuing education courses, explaining
the methodology and how the proposed ontology framework is applied.
Section 5 discusses the benefits of applying the ontology, as well as possible
directions for future work. Finally, Sect. 6 concludes the article, summarizing the
findings and identifying the implications of the proposal.
2 Literature Review
Through the literature review, the objective is to explore the concepts related
to the definition of the Open Digital Education Credential Ontology. With the
increasing adoption of digital technologies in education, the issuance and vali-
dation of educational credentials has become an area of interest and discussion.
Open digital educational credentials represent an innovative approach to the
issuance and recognition of educational achievements, allowing digital verifica-
tion of acquired skills and knowledge. There are some proposals for ontologies
in this area, but concepts such as interoperability between different systems and
organizations have not yet been achieved.
In this sense, decentralization emerges as a promising solution, where tech-
nologies such as Blockchain, decentralized digital identities and distributed file
systems can play a key role in credential management and validation, overcom-
ing existing barriers and providing professionals with greater control over their
learning paths.
2.1 Open Digital Educational Credentials

Open digital educational credentials are digital certifications that represent edu-
cational achievements and acquired skills. Unlike traditional formats, such as
paper diplomas, open digital credentials are interoperable, portable and digi-
tally verifiable. These credentials use open standards to enable the exchange of
information between different systems and organizations, which facilitates the
validation and recognition of educational achievements [3,4].
2.2 Learning Paths

Learning paths refer to the set of courses, activities and learning experiences that
an individual follows to acquire knowledge and skills in a specific field. Learning
Ontology for Open Digital Educational Credentials 273
pathways can be structured, such as formal academic programs, or more flexible,

allowing for customization of the educational process according to the student’s
needs and preferences. These paths may include a combination of face-to-face
and online courses, internships, projects and other learning activities [5].
2.3 Decentralization
Decentralization involves distributing authority, control and resources through-
out a network or system, as opposed to a centralized structure. In the context
of educational credentials and learning pathways, decentralization seeks to elim-
inate reliance on centralized institutions or entities for issuing, validating and
storing credentials and defining educational pathways. Decentralization can offer
benefits such as greater autonomy for students, greater transparency, interop-
erability, and the ability to validate credentials without recourse to a central
authority [6].
Blockchain. Blockchain is a distributed registry technology that enables the

secure storage and verification of transactions in a decentralized network. In
the context of digital educational credentials, Blockchain technology provides a
secure and trusted framework for storing and managing credentials, avoiding the
need for a central authority to verify their authenticity. The immutability and
transparency of Blockchain technology can help prevent fraud and improve trust
in credentials [7].
2.4 Decentralized Digital Identities
Decentralized digital identities (DIDs) are based on the idea that individuals
should be in control of their own identity information online. These identities
use cryptographic and distributed registry technologies to allow users to maintain
control over their personal data and selectively share information needed to verify
their identity in different contexts, such as issuing and validating educational
credentials. This provides greater privacy, security and autonomy for individuals
in managing their digital identities [8].
2.5 Distributed File Systems
Distributed file systems allow storage and access to files in a decentralized net-
work. These systems are based on file distribution and replication across multiple
nodes in the network, which improves availability, redundancy and resilience to
failures. In the context of open digital educational credentials, distributed file
systems can provide a secure and reliable infrastructure for decentralized stor-
age and sharing of credentials, enabling easy and verifiable access across different
systems and applications [9].
274 D. Pinto et al.
2.6 Related Work
The literature review of [10], highlights several fundamental elements (Fig. 1) to

consider in the definition of learning paths. Among these are the parameters,
methods, and techniques used to establish the characteristics of these path-
ways. In addition, the approaches and measures needed to assess the quality and
effectiveness of personalization are discussed. This study represents a relevant
contribution, as it points out several challenges to be addressed in this area, such
as the lack of standards, data scarcity and computational complexity.
Fig. 1. Components of customized learning paths
Por supuesto, aquí tienes una versión mejorada y enriquecida del texto:
In the preceding review, the structure for organizing learning paths is com-
prehensively delineated, allowing for a systematic categorization within specific
levels. This content hierarchy, as illustrated in Fig. 2, serves as a foundational
framework that informs the design and sequencing of educational modules or
courses. It is typically constituted by several distinct levels.
1. Course (C): covers a number of related learning objectives. An example

course could be “Introduction to Python programming”.
2. Lessons (L): The content of a course is divided into lessons. Each lesson
focuses on one aspect of the overall course topic. In the example course above,
the lessons might be “Basic Python syntax”, “Python control structures” and
“Object-oriented programming in Python”.
3. Topics (T): the content of the lessons is divided into topics. Topics are
subsections of a lesson that focus on an even more specific aspect of the
lesson topic. For example, within the lesson “Basic Python syntax”, it can have
topics such as “Variables and data types in Python”, “Operators in Python”
and “Strings in Python”.
4. Learning Objects (OA): is a self-contained unit of learning content that
focuses on a single learning objective. Learning objects can be of several types:
readings, videos, interactive exercises, quizzes, etc. For example, within the
thematic “Variables and data types in Python”, you can have an explanatory
video about variables in Python, a reading about data types in Python, and
an interactive exercise to practice the creation and use of variables and data
types in Python.
Fig. 2. Content hierarchy
A study relevant to ontology design is presented in [11], where the KEPLAIR

project is discussed. This project focuses on the development of an online plat-
form intended to facilitate personalized educational experiences. The logical
architecture of KEPLAIR is detailed in the study, including a description of
the system components. Of particular importance is the role of the ontology,
which is identified as a crucial element to ensure homogeneity and interoperabil-
ity among the system components.
Similarly, some works where Blockchain is used for the decentralization of
educational credentials were reviewed. For example, [12] in its paper describes a
Blockchain-based educational certificate verification system that uses smart con-
tracts to ensure the authenticity of documents. This system provides a decen-
tralized certificate verification solution, thus minimizing the risks of fraud or
document tampering. By leveraging Blockchain technology and smart contracts,
this proposed system offers a secure and reliable way to verify the validity of
educational certificates, eliminating the need for a centralized authority and
providing greater transparency in the verification process.
276 D. Pinto et al.
[13] addresses academic credential verification using Blockchain and decen-

tralized identity technologies with the goal of improving security and reduc-
ing fraud. The proposal presents a system that enables real-time verification of
credentials while ensuring the privacy and security of student data. These fea-
tures are particularly relevant to the data model being considered, especially if
a scalable and secure solution for digital credential verification at a university
is sought. By considering this approach, significant benefits can be realized in
terms of reliability, transparency and integrity protection of academic credentials
in a university environment.
[14] highlights the ability to share and verify educational and professional
data through digital credentials, which is highly beneficial for the management
of such credentials. In this regard, the proposed system is based on a private
blockchain platform that ensures the security and privacy of user data, while
allowing interoperability between various data management systems. These fea-
tures are particularly relevant for the proposed data model, especially if a solu-
tion that allows the integration of different credential management systems is
sought.
In this study, the primary focus is to establish a robust framework that
guarantees both confidentiality and user autonomy over their educational and
professional information. This goal is not merely about protecting privacy but
also about empowering individuals to control their data actively. Furthermore,
it is imperative to ensure advanced interoperability capabilities, which will facil-
itate seamless collaboration and data exchange between various institutions,
organizations, and sectors. By doing so, this fosters a more dynamic, efficient,
and integrated ecosystem in the management of digital credentials. Such an
approach has far-reaching implications, enhancing trust and engagement, and
enabling more personalized and flexible learning and career pathways.
3 Proposal
Given the need for a secure and reliable method to verify their authenticity, it
is essential to develop a data model that allows decentralized storage, access
and verification of digital credentials. In this regard, Blockchain technology has
emerged as a promising tool.
The main objective of this proposed data model prototype is to establish
an efficient and secure system for the verification of educational digital creden-
tials. Entities along with their corresponding attributes have been designed to
represent the relevant information in this process, as shown in Fig. 3.
Interoperability is a key aspect of this model, so emphasis has been placed
on ensuring that data from different systems can be adapted to this structure.
This would allow for a smoother flow of information throughout the credential
generation environment. In addition, it seeks to leverage Blockchain technology
to achieve greater transparency in the validation and access to credentials.
The proposal has been developed considering the integration with existing
systems in the educational institutions. Options such as the use of APIs and
Fig. 3. Proposed ontology
web services to facilitate communication and data exchange with existing data
management systems have been considered. This would allow for easier adoption
and greater interoperability between different credential management systems.
278 D. Pinto et al.
Table 1. Proposed ontology classes
Class Subclass of Prefix Description

Organization foaf This class represents an organization
Consortium Organization vivo This class represents a consortium. In the
context of the proposed ontology, a consortium is
a group of educational institutions participating
in an ecosystem for issuing and verifying
educational credentials
University Organization creduni This class represents a university
Organization Unit foaf This class represents a university academic
department
LearningPath creduni This class represents a learning path
Course vivo This class represents a course
Skill vivo This class represents an acquired skill or
knowledge
LearningOutcome cs2013 This class represents a learning outcome
LevelOfMastery cs2013 This class represents a level of mastery of
acquired skills
Person vivo This class represents a person
Student Person vivo This class represents a student
Instructor Person vivo This class represents a course instructor
Issuer DID creduni This class represents an issuer of educational
credentials
Verifier DID creduni This class represents a verifier or validator of
educational credentials
Digital Credential vivo This class represents an educational credential
Microcredential Digital Credential creduni This class represents a microcredential
Digital Badge Digital Credential creduni This class represents an open digital badge
Manage Records creduni This class represents a file manager
The main classes involved in the proposed ontology are described below. For
a complete list of the classes involved in the ontology, see Table 1.
– University: The concept of “university” represents higher education insti-

tutions. Properties such as “name” (name of the university), location“ (geo-
graphic location) and programs” (academic programs offered by the univer-
sity) make it possible to characterize and relate universities within the ecosys-
tem, facilitating the management of educational information and making it
possible to define the rules and standards of participation.
– Course: The concept of “course” represents the collection of skills and knowl-
edge that can be acquired by professionals. Properties such as “hasCour-
seInstructor”, “hasPrerequisite”, “hasCourseOffering”, “hasLearningOutcome”,
“completed”, “holdsAccount” and “teaches” allow characterizing and relating
courses to entities such as educational institution, teacher and student.
– Person: The ontology also considers the general concept of “person” to rep-
resent any individual, whether a student or not. Properties such as “name”
(name of the person), “dateOfBirth” (date of birth), “email” (email address)

and “affiliations” (institutional affiliations) allow to characterize persons in
general terms and to establish relationships with other entities in the system.
– Student: The term “student” is used to represent individuals enrolled in a
university. Key properties such as “name” (student’s name), “enrollmentYear”
(year of enrollment) and “skills” (skills acquired by the student) allow to
describe and relate students, facilitating the management of their academic
information and associated skills.
– Skill: The “skill” concept provides a detailed description of the capabilities
or competencies that an individual may possess. Properties such as “name”
(name of the skill), “description” (description of the skill) and “level” (level of
mastery of the skill) allow to characterize and differentiate the skills within
the system.
– Decentralized Digital Identifiers (DIDs): The term “DID” refers to the
decentralized digital identification used to uniquely represent a digital entity,
such as a student or a university. Properties such as “Id” (decentralized digital
identifier), “publicKey” (public key associated with the entity) and “authenti-
cation” (authentication methods) help establish a trusted and secure digital
identity for entities within the system.
– Manage Records: This class represents the manager of records of issued
educational credentials. In the ontology design, it is proposed that the records
management process will be carried out through a decentralized file manage-
ment system, such as IPFS. This type of system ensures security, privacy and
implements mechanisms that allow the systems to be resilient to failures.
The prototype of the proposed data model offers a solution for the verification
of educational digital credentials. In addition, through the use of international
standards, Blockchain technology and smart contracts, it seeks to streamline and
improve the verification process, providing efficiency, transparency and reliability
in the management of credentials in the educational field.
4 Use Case
The Continuing Education Center of the Universidad Técnica Particular de Loja
(UTPL) serves as a relevant use case for the application of the proposed ontol-
ogy. This center is dedicated to providing continuing and permanent education,
designed to foster the integral development of individuals and meet the compet-
itive demands of today’s world and the labor market.
280 D. Pinto et al.
Table 2. Main properties of the proposed ontology
Property Prefix of Domain Range

has Course Offering creduni Organization Unit Course
has Learning Outcome vivo Course Learning Outcome
has Level Of Mastery cs2013 Learning Outcome Level Of Mastery
has Course Instructor vivo Course Instructor
Is Part Of creduni Course Learning Path
issues creduni Issuer Credential
holds Account vivo Student Course
has Credential vivo Student Credential
Through its training units, it manages and executes high quality training pro-
grams, designed by teachers and specialists in various areas. The implementation
of the proposed ontology can facilitate the issuance and verification of educa-
tional credentials for these training programs, allowing students and employers
to easily verify the skills and competencies acquired.
At its current stage, this use case is under development, and the main tech-
nical requirements have been established. The following is a description of the
improvements anticipated by integrating the ontology described in the docu-
ment:
4.1 Current Status
At the Lifelong Learning Center, students are now allowed to freely choose their
courses, without the need to follow a specific learning path. Upon completion
of a course, the student is recognized with a certificate that validates his or her
competencies in the area of knowledge studied. This model, although flexible,
lacks a structure to guide the student in an educational path that fits his or her
personal and professional objectives (Fig. 4).
4.2 Ontology Implementation and Relevance
The proposed ontology constitutes a useful tool for the abstraction of the current
problematic. The ontology, in general terms, represents the structure of the set
of concepts and the relationships between them. For the case of the Lifelong
Learning Center, this representation can be summarized as follows:
– Learning path modeling: Through the ontology, it is possible to define

and structure different learning paths that can be sequential or personalized.
– Course Interconnection: The ontology also makes it possible to identify
and represent the relationships and dependencies (prerequisites) between dif-
ferent courses, facilitating the creation of relevant learning paths.
Fig. 4. Previous situation
– Adaptability: Since the ontology allows a dynamic representation of knowl-

edge, it can adapt and evolve, allowing the definition of new educational
models.
In order to decentralize and personalize the learning paths, we propose the

implementation of an ontology that defines a model for these paths. This ontol-
ogy seeks to offer two main types of paths:
– Sequential Routes: Are predetermined and follow a logical sequence of

courses that take the student from a basic level to an advanced level in a
specific discipline.
282 D. Pinto et al.
Fig. 5. Ontology applied in use case
– Adjusted Routes: Are tailored to the individual student’s needs, allowing

them to select and combine different courses based on their interests, previous
skills and learning objectives.
Educational credentials and learning paths are closely related, where cre-
dentials act as evidence of knowledge acquired at specific stages of a learning
path. These credentials validate progress and completion of certain milestones,

facilitating the creation of personalized learning paths. The accumulation of cre-
dentials reflects progress along the pathway, providing incentives and a tangible
representation of the learner’s progress.
4.3 Decentralized Technologies Participating in the Model

Decentralization is essential to this new model, and is achieved through the use
of emerging technologies:
– Blockchain: This technology will enable the issuance and verification of cer-
tificates. With Blockchain, each certificate issued will be immutable and ver-
ifiable, thus guaranteeing its authenticity and validity.
– IPFS (InterPlanetary File System): It is used to store certificates, offer-
ing a decentralized form of access and data retrieval, which increases the
efficiency and security of issuance and consultation.
– DID (Decentralized Identifiers): It is used to ensure the decentraliza-
tion and security of the identities of students, teachers, issuers and verifiers.
Through DID, for example, each student will have a unique and verifiable
digital identity that guarantees their privacy and the authenticity of their
educational records.
5 Discussion
The implementation of the ontology at the Lifelong Learning Center brings mul-
tiple benefits that reinforce the essence of the center: to offer learning adapted
to each stage of students’ lives. One of the most significant advantages is the
structuring and modeling of learning paths, which not only provide coherence in
the educational trajectory, but also better align with the aspirations of students
and professionals. This structured organization of courses allows the center to
more effectively evaluate its current offerings, identifying gaps or redundancies
in pathway elements.
With the creation of these pathways, the identification and relationship
between components of the curriculum has been facilitated, providing a clear
vision of the proposed educational pathway. This approach has also increased
transparency, allowing students to understand how each course aligns with their
profile and aspirations.
Going forward, the ontology presented can be further enriched with the inte-
gration of other components. For example, it would be useful to consider includ-
ing indicators of acquired competencies, feedback from previous participants and
integration with labor platforms to highlight how certain courses or pathways
can increase employability.
On the other hand, this ontology can serve as a starting point for the devel-
opment of broader solutions, such as course recommendation systems based on
Artificial Intelligence algorithms and integration with adaptive learning plat-
forms.
284 D. Pinto et al.
The distinctive value of the presented work, compared to the studies ref-
erenced in the literature review, lies in its holistic approach. Unlike previous
approaches, which often dealt with these issues in isolation, this study succeeds
in systematically integrating the concepts of learning paths, micro-credentials,
Blockchain technology and decentralized identifiers into a single model.
6 Conclusions
The proposal of an ontology for university credentials represents a useful tool

for the standardization and definition of the interactions between the different
entities involved in the issuance and verification process. This ontology frame-
work has the potential to serve as a reference point for future adaptations in
other educational institutions.
The ontology is a proposal that facilitates the efficient management of decen-
tralized identities in the credential issuance and verification process. In this sense,
the ontology establishes the concepts and relationships necessary for the proper
representation and manipulation of DIDs.
One of the main benefits of the ontology is its emphasis on interoperability,
which facilitates the integration of educational credentials from different systems
and organizations. This promotes a holistic and integrated educational system
and allows students to carry their credentials across diverse educational environ-
ments.
By adopting a decentralized approach to educational credentialing, students
and professionals gain greater control over their learning and professional devel-
opment paths. In this way, they have the ability to efficiently access, share and
verify their credentials, increasing transparency and reducing reliance on cen-
tralized authorities.
The use of Blockchain technology guarantees secure credential issuance and
verification. Thanks to the immutable nature of Blockchain, the authenticity of
credentials is ensured and the risk of fraud is minimized. All this contributes
to strengthening trust among stakeholders, including employers and educational
institutions.
Education is undergoing an accelerated transformation, driven by the intro-
duction of Open Digital Educational Credentials that pave the way for more per-
sonalized and adaptive learning. However, this transformation poses challenges
related to the compatibility and acceptance of these credentials. The proposed
ontology is intended to be a solution to address these challenges and provide a
common framework for authenticating these credentials.
Learning paths are crucial in current educational models, as they allow tailor-
ing education to specific goals and needs. The proposed ontology can be a basis
for recommender systems in charge of creating personalized paths that enhance
educational experiences.
References
1. Van de Laar, M., West, R. E., Cosma, P., Katwal, D., Mancigotti, C: The value of
educational microcredentials in open access online education: a doctoral education
case. Open Learn. J. Open Distance e-Learning, 1–14 (2022)
2. Smaili, E.M., Khoudda, C., Sraidi, S., Azzouzi, S., Charaf, M.E.H.: An innovative
approach to prevent learners’ dropout from MOOCs using optimal personalized
learning paths: an online learning case study. Statistics, Optimiz Inform. Comput.
10(1), 45–58 (2022)
3. Jirgensons, M., Kapenieks, J.: Blockchain and the future of digital learning cre-
dential assessment and management. J. Teacher Educ. Sustain. (2018). https://
doi.org/10.1007/s001090000086
4. IMS Global Learning Consor: Understanding Digital Credentials. https://www.
imsglobal.org/understanding-digital-credentials. (Cited 14 Jul 2023)
5. N/D: Use Learning Paths to Create Structured Training Programs. https://www.
learnupon.com/blog/learning-paths-walkthrough. (Cited 14 Jul 2023)
6. PeopleHum: What is Decentralization?. https://www.peoplehum.com/glossary/
decentralisation. (Cited 14 Jul 2023)
7. IBM Blockchain: What is Blockchain Technology?. https://www.ibm.com/topics/
blockchain. (Cited 14 Jul 2023)
8. Dock: Decentralized Identifiers (DIDs): The Ultimate Beginner’s Guide 2023.
https://www.dock.io/post/decentralized-identifiers. (Cited 14 Jul 2023)
9. Athanere, S., Thakur, R.: Blockchain based hierarchical semi-decentralized app-
roach using IPFS for secure and efficient data sharing. J. King Saud Univ.-Comput.
Inform. Sci. 34(4), 1523–1534 (2022)
10. Hossein, A., Leal, J., Hamed, N., Rajiv, R.: Learning path personalization and
recommendation methods: a survey of the state-of-the-art. J. Expert Syst. Appli.
159, 0957–4174 (2020)
11. Ferilli, S., Redavid, D., Di Pierro, D., Loop, L.: An ontology-driven architecture for
intelligent tutoring systems with an application to learning object recommendation.
Inter. J. Comput. Inform. Syst. Indust. Manag. Appli. 14, 297–312 (2022)
12. H. Gaikwad, N. D’Souza, R. Gupta, A. K.: A blockchain-based verification system
for academic certificates. In: International Conference on System, Computation,
Automation and Networking, ICSCAN 2021 (Jul 2021). https://doi.org/10.1109/
ICSCAN53069.2021.9526377
13. Ambast, S. K., Sumesh, T. A.: A blockchain based credential verification system
using IPFS. In: 2022 IEEE 19th India Council International Conference (INDI-
CON), pp. 1–5. IEEE (2022)
14. Wang, Y., Sun, Q., Bie, R.: Blockchain-based secure sharing mechanism of online
education data. Proc. Comput. Sci. 202, 283–288 (2022)
Exploring the Perceived Satisfaction
of Flipped Learning in E-Learning
Courses
Margarita Ortiz-Rojas1(B) , Cindy Bolaños-Mendoza2 ,

and Maria Alejandra Ruano3
1
Información, Campus Gustavo Galindo Km. 30.5 Vı́a Perimetral,
P.O. Box 09-01-5863, Guayaquil, Ecuador
margarita.ortiz@cti.espol.edu.ec
2
Escuela Superior Politécnica del Litoral, ESPOL, Centro de Investigaciones y
Servicios Educativos, Campus Gustavo Galindo Km. 30.5 Vı́a Perimetral,
3
Escuela Superior Politécnica del Litoral, ESPOL, Facultad de Ciencias Sociales y
Humanı́sticas, Campus Gustavo Galindo Km. 30.5 Vı́a Perimetral,
Abstract. The Covid-19 pandemic pushed the introduction of e-

learning courses in Higher Education. Nevertheless, e-learning alone is
not sufficient to guarantee good academic performance. Active learn-
ing methodologies are necessary to be included. In this context, flipped
learning enters as an approach where students study the course materials
independently before attending synchronous sessions with instructors for
interactive discussions and clarification. This study is the first attempt to
analyze the perceived satisfaction of an e-learning course using a flipped
learning approach involving 67 teachers and 768 students (46 STEM
courses and Forty non-STEM courses). The mixed methods approach
using a survey with open- and-closed questions showed that teachers
and students perceived the modality as satisfactory regardless of the
study area (STEM or non-STEM). The qualitative information provided
insights about the reasons why they liked the modality, such as flexibil-
ity and preparation before class. However, it also showed what aspects
needed improvement: more student involvement in pre-class assignments
and teacher workload. The study concludes by providing future lines of
research.
Keywords: flipped learning · e-learning · online learning
1 Introduction
1.1 Defining E-Learning
E-learning, also known as online learning, refers to the use of the Internet and
technologies along with instructional design principles to offer educational con-
tent and facilitate learning. It includes the use of online courses (synchronous
https://doi.org/10.1007/978-981-99-7353-8_22
Perceived Satisfaction of Flipped Learning in E-Learning Courses 287
or asynchronous) along with digital resources, such as multimedia materials,

forums, and virtual classrooms, to provide instruction and support to learners.
E-learning can take various forms, including self-paced courses, live classes or
webinars, and blended learning approaches that combine online and in-person
instruction [1]. However, its definition can vary according to the context and the
educational institution [2].
E-learning can be synchronous, asynchronous, or a mix of both. In a syn-
chronous class, teachers and students attend the class simultaneously via a video-
conference online platform such as Zoom, Teams, or Google Meet, where they
work on content and activities imitating a regular classroom. In an asynchronous
class, all the content from the classroom is readily available, usually in a Learn-
ing Management System (LMS), which they can access any time, whenever they
want. Still, there is, as a rule, a time frame where they need to complete the
activities. Students and teachers can still interact with each other but at different
times [3].
E-learning has several advantages. Authors such as [4] highlight the ease of
access from anywhere around the world and the flexibility of time and location
in synchronous and asynchronous e-learning. This also means it is less time-
consuming since it eliminates transportation to the classroom. Asynchronous
e-learning also offers access to previously curated materials and sometimes even
the ability to personalize learning experiences [5]. In comparison, synchronous
e-learning has the advantage of immediate feedback and real-time collaboration
[6]. [7] emphasized that asynchronous e-learning is efficient and cost-effective
since even if creating e-learning courses is expensive due to the infrastructure
needed, the per-student costs are reduced when repeating the courses several
times.
E-learning has grown in popularity in recent decades, especially during and
even after Covid-19. During Covid-19, most educational institutions turned to
emergency e-learning, meaning that due to Covid restrictions, they had to use
video conference platforms for their classes. Most institutions applied the same
traditional teaching methodologies as in-person classrooms without any peda-
gogical change. Furthermore, since neither teachers nor students were prepared
for e-learning, most found the fast technological change challenging [8]. The lack
of preparation was not only technical but also pedagogical. In some institutions,
teachers were thrust into e-learning without previous preparation, and the only
difference in their teaching was lecturing in a classroom versus lecturing in front
of a camera, causing students to lack engagement [9].
1.2 Flipped Learning as an Active Methodology

Flipped learning is one active engaging methodology where the traditional model
of first-in-class lectures and then-home assignments is reversed [10]. In a flipped
classroom, students attempt to gain foundational knowledge before class to apply
it through group work. The foundational knowledge is provided by the teacher,
who either creates video lessons or materials from scratch or uses information
from trusted sites [11]. As students review these materials before class, students
288 M. Ortiz-Rojas et al.
are more productive and engaged in the in-class activities, and teachers can pro-
vide better feedback and support [12]. Since most of the materials that students
need to review before in-person class is uploaded in an LMS, some authors may
explain that flipped learning uses e-learning as a vehicle [13]. However, com-
bining active methodologies such as flipped learning and synchronous e-learning
(via videoconference) is quite a challenge.
1.3 Integrating Flipped Learning with E-Learning

Previous research has analyzed the perceived satisfaction of synchronous e-
learning and flipped learning worldwide separately, finding mixed results. For
instance, [14] analyzed the student’s perception of a flipped classroom in a
research methods university course. The author found that most students, from
a sample of 240, had a positive attitude towards flipped classrooms since they
considered it increased their motivation, engagement, and learning. Contrarily,
[15] found that first-year undergraduate students strongly opposed the flipped
learning methodology, while [16] indicated that most students preferred new
material to be introduced during class and not before. Both authors considered
that the main reason for the negative perception was pre-class homework and
not understanding the instructions.
Furthermore, they advised that pre-work should have frequent and consis-
tent deadlines with sufficient time to complete the task. [17] determined in their
meta-analysis that a reason for the mixed results in the student’s satisfaction
with flipped learning could be the different ways that flipped learning is applied,
especially the different approaches taken by instructors in the pre-class home-
work. Some instructors assigned passive activities (watching a video, reading),
while others included activities such as quizzes, problem questions, or others.
The difference in satisfaction also depends on the nature of the courses. [18]
reviewed around 50 studies on the reactions to the flipped classroom in STEM
and not STEM courses in higher education. She found that even though, in gen-
eral, students from different fields preferred flipped learning over the traditional
methodology, the STEM fields benefited more from flipping the classroom. Stu-
dents highlighted benefits such as better learning foundational knowledge, devel-
oping higher-order thinking skills, improved skills-based abilities, engagement,
better working with others, and learning.
Other authors, such as [19], highlight that most studies address students’
perspectives on flipped learning, not teachers’. They found that teachers believe
that flipping the classroom at the beginning takes much time; however, those
who experienced using flipped classrooms on different semesters specified that
overtime takes less time and energy. Also, teachers emphasized that motivat-
ing students to do their pre-class homework is an essential component of the
methodology and their biggest challenge. The authors explained the importance
of training programs and peer assistance among teachers to ease into flipped
learning.
On another note, authors such as [20] determined and explored the main fac-
tors influencing students’ level of satisfaction with e-learning. Their results show
that the main satisfaction factors are course design and contents, the facility for
accessing and visualizing information on the LMS, and the interaction opportu-
nity. [21] adds that part of the student’s satisfaction with synchronous e-learning
is the ability to get live feedback from the teacher, seeing their classmates and
teacher live, and more engagement.
In Latin America, few studies analyze students’ satisfaction based on the
application of the flipped learning or e-learning methodology. [22] show that
students in Latin America positively perceive flipped learning since they feel
more motivated and engaged with the content as they interact with it pre-class.
Other authors do systematic reviews, such as the one from [23], who reviewed
several Ecuadorian studies on flipped learning and found that the methodology
improves the in-class time for participation, interaction, and questions.
1.4 Addressing the Gap
No studies, as far as we know, analyze the satisfaction of synchronous e-learning

and flipped learning. There is a need to investigate teachers’ and students’ satis-
faction with synchronous e-learning courses that use the flipped learning method-
ology. Thus, this paper aims to answer the following research questions:
– RQ1: What are the teacher’s and students’ perceived satisfaction of an e-

learning course using a flipped learning approach?
– RQ2: How different is the perceived satisfaction of an e-learning course using
a flipped learning approach in teachers and students of STEM and non-STEM
courses?
In this paper, we conducted a mixed-methods analysis using a survey that

combined a Likert scale plus open-ended questions. The sample included 67
teachers and 769 students from STEM and non-STEM courses that followed
a flipped learning and synchronous e-learning methodology altogether. The
teachers and students are from an engineering-oriented university in Guayaquil,
Ecuador.
The rest of the paper is organized as follows. Section 2 describes the method-
ology used to analyze students’ satisfaction with the methodology. Section 3
presents the results of the survey and the discussion. Finally, Sect. 5 describes
the conclusion, acknowledges its limitations, and suggests possible future work
directions.
2 Methodology
2.1 Context and Participants
The study was conducted at an engineering-oriented university in Ecuador in

2022. A total of 81 teachers (99 courses, Min = 2, Max = 45, N = 1514 students)
who were assigned to teach an e-learning course were invited to participate in the
study. Nevertheless, only 67 teachers (36 male and 31 female) teachers accepted
the invitation (86 courses, Min = 3, Max = 45, N = 768 students). The courses
studied were 46 STEM (e.g., vector calculus, structural geology), while 40 were
non-STEM (marketing research, sociology) from the different faculties at the
university. Overall, a response rate of 50.66% was obtained from students, while
the average response rate per course was 52.75%.
Before implementation, from April to September, teachers received a 75 h
online Professional development program on how to design an e-learning course
using flipped learning. The program comprised five modules: what is flipped
learning, how to plan e-learning courses, how to assess online, how to design
educational digital materials, and how to integrate all planning elements in the
learning management system. One particularity of this program is that it was
designed in such a way that it followed a flipped learning methodology. As such,
teachers would experience the same methodology as expected for them to apply.
This program was an institutional prerequisite for teaching an e-learning course
the following semester.
The implementation took place during the second semester (October to
February). Teachers were asked to redesign their face-to-face classes into e-
learning classes using what they learned in the training program (a flipped
learning methodology). This transformation was designed to be gradual because
it involved considerable planning and material design. Thus, the first semester
would start with 30% of the class material being redesigned, followed by 50%,
80%, and 100% per semester. This means that for a full redesign, it would take
four academic semesters. This study analyzed the implementation of the first
30% of the class material, which was equivalent to two course units (approxi-
mately 1,5 months).
2.2 Procedure and Data Collection Instruments

One month before finishing the semester, teachers and students were asked to
fill out an online questionnaire with one closed question about their perceived
satisfaction with the e-learning course they were about to finish. The question
was: How would you rate your experience in the E-learning modality? Partic-
ipants rated their satisfaction on a 5-point Likert scale (1 = unsatisfactory, 5
= very satisfactory). The second question was open and asked participants to
justify their answers in a comment box.
2.3 Data Analysis Procedure

A mixed-methods approach was used to address these research questions. To
answer RQ1 and RQ2, a quantitative analysis was performed to describe the
perceived satisfaction that teachers and students had with the e-learning course.
To further understand RQ1, a qualitative analysis was performed with the infor-
mation of the open question using a thematic analysis approach [24]. The initial
coding was done by one author and focused on the general participant’s perceived
satisfaction with the online course using the flipped learning approach (posi-
tive, negative, neutral). In addition, during the general review, macro-themes
emerged, which allowed the comments to be grouped into categories. These cat-
egories were reviewed and refined by two authors until only seven were obtained.
– Perceived perception of the teaching-learning process: in the case of

the student, it describes the perception of learning if it was better or not; and
the perception of the development of autonomy associated with learning. In
the case of teachers, it details how teachers perceive modality and how they
deal with change.
– Digital Material: for students, it explains the availability, quality and ben-
efits of the material that the teacher shares. As for teachers, this refers to the
challenges involved in developing and implementing didactic materials.
– Teacher/Student preparation: for students, it includes the organization
of the course, the adequate preparation of pre- and post-class activities, the
feedback they receive from the teacher, issues related to group work (e.g.
organization), and motivation of the teacher in class. As for teachers, it means
perceptions of students’ willingness to participate in class, their performance
of pre- and post-class activities, and student motivation to improve their
academic performance.
– Workload: For teachers, it refers to spending too much time designing the
activities. As for students, it means, spending too much time doing prework
activities. It also encompasses the opportunity to work and organize time.
– Face-to-face component: In both teachers and students, it covers topics
such as the preference for face-to-face modality and the perceived need for
face-to-face practice for certain topics or subjects.
– Subject-modality integration: it contains comments in which teachers or
students express that the subject, or certain topics of the subject, have a
higher degree of complexity than cannot be covered by the modality.
– Others: it includes comments dealing with the form of communication and
teacher-students interaction, the presence of technological issues, and those
participants who chose not to express any opinion.
3 Results
In this section, we present the results regarding the perceived satisfaction of
teachers and students in an e-learning course using the flipped learning approach
(RQ1) and the perceived satisfaction of teachers and students in an e-learning
course using a flipped learning approach, according to the knowledge area (RQ2).
3.1 Perceived Satisfaction of an Online Course Using a Flipped

Learning Approach
The quantitative analysis displayed in Fig. 1 shows that, for the most part, both
teachers and students perceive the experience of an e-learning course using the
flipped learning approach satisfactorily, while about 30 % of teachers perceive it
in a neutral and unsatisfactory way.
Fig. 1. Perceived satisfaction of teachers and students
The qualitative analysis shown in Fig. 2, confirms the students’ satisfaction

with more positive than negative comments about the course: “The e-learning
modality is useful and the autonomous activities helps to have a knowledge prior
to class where feedback is made, making the knowledge more concrete and clearer.”
(S16) and “This modality gives you the facility to prepare yourself theoretically
before the class, thus allowing you to have knowledge of what will be done in
the class, where the teacher explains and clarifies doubts of what was previously
learned.” (S85). The percentage of negative comments are mostly related more to
the time dedicated to the course activities, and to the lack of integration of the
modality and the subject in more complex topics such: “The amount of work was
excessive in comparison with the study time.” (S664), “I do not like it very much
because in subjects with high complexity there can be gaps, besides, one is left with
doubts and does not know if what he/she is investigating is reality.” (S681).
However, the percentage of teachers’ comments was divided almost equally
between positive and negative, although they quantitatively expressed satisfac-
tion with the course. Positive comments are directed towards the perception of
improvement in the teaching-learning process and the preparation of the student
with pre- and post-class activities “It allows the teacher to incorporate new cre-
ative materials for the transfer of learning. Additionally, it encourages students
to manage their time and work autonomously.” (T45), “I see that students feel
more involved and value flexibility in performing tasks and activities. I think it
helps a lot to have previous activities that prepare students about the topics that
will be covered in class and also the subsequent activities that reinforce the top-
ics covered.” (T52). While the negative comments focus more on the overload
of time for the adequate development of the class, such as “It was carried out
according to plan. However, it took a long time to develop material since it was
a subject I was teaching for the first time.” (T42), “It takes excessive time to
prepare each week of work.” (T26).
Fig. 2. Qualitative analysis by type of comment
An in-depth analysis of comments by category was conducted separately.

Figure 3 shows that teachers focus most of their comments on two categories:
perceived perception of teaching-learning and student preparation. First, teach-
ers recognize more student involvement: “Although it is the first time I do in
this modality the result perceived by me is of a more meaningful participation
and learning, I felt that the student took over the knowledge.” (T50). In the
second category mentioned, teachers perceive the lack of preparation of the
student to face this new modality: “Students do not participate responsibly in
doing the asynchronous activities developed...” (T17), “Specifically, students in
this semester came accustomed to online classes, but they were not active in
forums...” (T7).
Fig. 3. In depth analysis of categories by teachers

Figure 4 presents the results obtained from the qualitative analysis by cate-
gories for students, in which it is observed that the greatest number of comments
are oriented to perceived perception of learning a category that coincides with
the teachers. The comments collected in Perceived perception of learning express
that students strengthened their learning autonomy with the help of the modal-
ity “The experience allowed me to learn on my own certain aspects of the theory,
which resulted in my being able to remember this information better than what
came through other means.” (S597), “It is a modality that motivates teamwork
and self-preparation prior to the class.” (S543).
Fig. 4. In depth analysis of categories by students
3.2 Perceived Satisfaction of an E-Learning Course Using a Flipped

Learning Approach by Knowledge Area
A quantitative analysis of teachers’ perceived satisfaction according to their area

of knowledge is shown in Fig. 5. Teachers in both areas value the course satis-
factorily. Only 10.71% of STEM teachers and 5.13% of non-STEM teachers had
unsatisfactory perceptions of the methodology.
Students have a similar perception of teachers, as observed in Fig. 6, regard-
less of the area of study. It is perceived satisfactorily and very satisfactorily.
Only 1.82% of STEM students and 2.09% of non-STEM students valued their
experience unsatisfactorily.
The qualitative analysis of teachers’ comments according to knowledge area
in Fig. 7 shows that both STEM and non-STEM teachers had divided perceptions
about the course. The positive comments from the STEM area focus on the fact
that the methodology used can improve students’ performance and their moti-
vation to be more participative: “The motivation and academic performance of
students attached to the methodology are differentiating with respect to the tradi-
tional methodology.” (T53), The negative comments are linked to the additional
Fig. 5. Perceived Satisfaction in STEM and non-STEM courses by teachers
Fig. 6. Perceived Satisfaction in STEM and non-STEM courses by students
work time involved in the change of methodology and the lack of linkage between
certain topics of the course and the modality: “The dedication of the work was
doubled.” (T29), “Some topics of the course and the modality are not related to
each other.” (T34).
In the case of non-STEM teachers, positive comments related to the chal-
lenge associated with the modality and how students benefit from it were ana-
lyzed: “The planning and continuous activities for the students allowed to have
an active and dynamic course.” (T25), while negative comments are directed
towards the lack of preparation on the part of the student as evidenced by
“Students are still not adapted to the modality.” (T61). In the case of neutral
comments, they indicate that the modality needs adjustments to adapt to all the
topics that may be covered in the course: “I think it is a process to adjust the
strategies and planning to teach an e-learning course.” (T33), “I only consider
it for certain classes, not for the whole subject.” (T57).
Fig. 7. Qualitative analysis by knowledge area by teachers
The students’ comments were also analyzed qualitatively based on their area
of knowledge. Figure 8 shows that the highest percentage of comments from both
groups of students indicated that they perceived the experience in the modality
as a positive regardless of the knowledge area. The positive comments of the
students in the STEM area focus on the fact that this methodology allows them
to have dynamic classes, they find support in the material and the predisposition
of the teacher to solve doubts, as shown by the following comments “Very sat-
isfactory, since it was possible to take advantage of the support materials for a
better understanding, in addition to the fact that the classes were developed in a
very dynamic and quite understandable way.” (S669), “Very satisfactory because
the professor was always open to doubts, and I could study from the department
where I reside.” (S276). The negative comments of the students in this area are
related to the lack of integration between the modality and the subject, due to
the complexity of the topics covered: “I thought the subject is very complicated
and should be seen in a face to face modality.” (S273), “Personally I do not like
the E-learning modality, there are professors who explain absolutely nothing and
do not bother to give feedback.” (S311). Regarding the neutral comments, 6.01%
of the comments indicate that the modality is like the classroom or virtual one:
“I consider that for this subject, the modality did not present a great change in
comparison if the subject had been virtual or classroom.” (S292).
On the other hand, the qualitative analysis of the non-STEM students’ com-
ments indicates that the positive comments value the knowledge prior to the
class to solve specific doubts in the time of contact with the teacher and the
flexibility to work that the modality provides: “In this subject it helps to prac-
tice the method already to focus on clear questions during the class.” (S740), “It
is a very useful method for students who want to start working.” (S734). Like
their peers in the STEM area, the negative comments are related to the dis-
connection of the modality and the complexity of the topics covered in classes:
“In my opinion, this modality should be for subjects such as sociology, not for
subjects that are more time consuming, I think some subject should not follow
this modality, such as Microeconomics 3 for example.”. Some of the neutral com-
ments are directed towards the need for more contact time with the teacher and
others towards not noticing a difference in the modality, such as “It is necessary
to have more contact with the teacher, to clarify doubts, which would be better if
the course were face-to-face.” (S47), “I did not notice a difference in the amount
of content learned or improvement of skills.” (S46).
Fig. 8. Qualitative analysis by knowledge area by students
4 Discussion
Regarding RQ1, both teachers and students perceived the modality as satisfac-
tory. Nevertheless, the qualitative analysis shows differences between teachers
and students. The students showed a marked difference between positive and
negative comments. Teachers’ negative comments were slightly higher. It seems
that teachers liked implementing the modality, but there were still some issues
that affected the learning process. Categorical analysis showed that one of these
issues was student preparation. Teachers mentioned that the students did not
perform the previous activities, which slowed down their progress in the sub-
ject because they were not prepared. This coincides with previous reviews of
flipped learning, which mention that students are reluctant to develop previous
activities because they prefer to review new materials in class rather than before
[16]. To overcome this challenge [25], recommends to provide access to pre-class
material at least one week before classes.
In terms of RQ2, the results show that the satisfaction of the e-learning
course is equally accepted in both STEM and non-STEM courses by teachers
and students. However, the qualitative analysis of teachers showed that the opin-
ions of both knowledge areas were almost equally divided between positive and
negative perceptions of the modality. Teachers positively valued the modality
because they perceived and improvement in the students’ academic develop-
ment by stimulating autonomous work. This was confirmed in the study of [26],
where the authors asked students about the usefulness of the flipped classroom
methodology. On the other hand, teachers perceive that the workload is exces-
sive compared to other subjects with face-to-face modality, which was confirmed
by [27] where teachers also negatively valued a flipped learning class because of
the workload. To counteract this negative assessment, more institutional support
is necessary in terms of acknowledging more time to prepare classes, or as [28]
mentions, having a team that can help develop digital material or organize the
class in the Learning Management System.
Due to the uniqueness of the innovation, it is not possible to compare these
results to similar studies since they do not combine e-learning and flipped class-
rooms in the same scenario.
5 Conclusions, Limitations, and Future Work
This study explored the perceived satisfaction of teachers and students in e-

learning courses using a flipped learning approach. The results showed that both
teachers and students liked this new modality, regardless of the area (STEM or
non-STEM). Students liked the fact that they could check the material before-
hand and had the flexibility to do it in their own time, while teachers perceived
that there was more student involvement since they came prepare to class. Nev-
ertheless, some negative comments mention that some students do not come to
prepare, which causes problems in class flow.
Despite the compelling conclusions drawn from the study, it is important to
acknowledge the limitations of this study to provide a balanced and comprehen-
sive understanding of its findings. First, it is possible that students answered
the survey by thinking about the other 70% of the class. Although the instruc-
tions specifically mentioned that they should assess the first two units, we do not
know for sure whether the students actually did so. Second, it is also possible that
the teachers did not implement e-learning using the flipped learning approach
because the class itself was not analyzed. This could cause students’ perceptions
to be biased, confusing e-learning with emergency remote teaching. Third, we
understand that perception is not sufficient to affirm that the combination of e-
learning and flipped learning works. Since the implementation of the classes was
only 30%, it was not possible to analyze students’ final grades to determine their
impact on learning performance. Fourth, more questions should be included in
the survey to analyze aspects such as flipped learning implementation.
The future work of this study calls for a new line of research aiming to go
beyond the analysis of perception to evaluate its effect on academic performance.
Additionally, it would be important to assess the transfer of learning from the

teacher development program to ensure that all teachers are actually applying
flipped learning and not just an e-learning course with synchronous lectures and
little or no interaction with students. Moreover, the complete redesign of the
course should be assessed to obtain a complete view of the implementation and
its impact.
References
1. Corbeil, J.R., Corbeil, M.E.: E-learning: past, present, and future. In: International
Handbook of E-Learning Volume 1, pp. 51–64. Routledge (2015)
2. Laurillard, D.: E-learning in higher education. In: Changing Higher Education,
pp. 87–100. Routledge (2005)
3. Amiti, F.: Synchronous and asynchronous e-learning. Eur. J. Open Educ. E-Learn.
Stud. 5(2) (2020)
4. Behera, S.K.: E-and m-learning: a comparative study. Int. J. New Trends Educ.
Implications 4(3), 65–78 (2013)
5. Smyrnova-Trybulska, E., Morze, N., Varchenko-Trotsenko, L.: Adaptive learning
in university students’ opinions: cross-border research. Educ. Inf. Technol. 27(5),
6787–6818 (2022)
6. Arkorful, V., Abaidoo, N., et al.: The role of e-learning, advantages and disadvan-
tages of its adoption in higher education. Int. J. Inst. Technol. Dist. Learn. 12(1),
29–42 (2015)
7. Frehywot, S., et al.: E-learning in medical education in resource constrained low-
and middle-income countries. Hum. Resour. Health 11, 1–15 (2013)
8. Qurotul Aini, Q.A., Mukti Budiarto, M.B., Poh Putra, P., Untung Rahardja, U.R.:
“Exploring e-learning challenges during the global COVID-19 pandemic: a review.
Jurnal Sistem Informasi (J. Inf. Syst.) 16(2), 47–65 (2020)
9. Maatuk, A.M., Elberkawi, E.K., Aljawarneh, S., Rashaideh, H., Alharbi, H.: The
covid-19 pandemic and e-learning: challenges and opportunities from the perspec-
tive of students and instructors. J. Comput. High. Educ. 34(1), 21–38 (2022)
10. Bergmann, J., Sams, A.: Flip your classroom: reach every student in every class
every day. In: International Society for Technology in Education (2012)
11. Brewer, R., Movahedazarhouligh, S.: Successful stories and conflicts: a literature
review on the effectiveness of flipped learning in higher education. J. Comput.
Assist. Learn. 34(4), 409–416 (2018)
12. Santhanasamy, C., Yunus, M.M.: A systematic review of flipped learning approach
in improving speaking skills. Eur. J. Educ. Res. 11(1), 127–139 (2022)
13. Soliman, N.A.: Teaching English for academic purposes via the flipped learning
approach. Procedia. Soc. Behav. Sci. 232, 122–129 (2016)
14. Nouri, J.: The flipped classroom: for active, effective and increased learning-
especially for low achievers. Int. J. Educ. Technol. High. Educ. 13(1), 1–10 (2016)
15. He, W., Holton, A., Farkas, G., Warschauer, M.: The effects of flipped instruction
on out-of-class study time, exam performance, and student perceptions. Learn.
Instr. 45, 61–71 (2016)
16. Yong, S.T., Tiong, K.M., Chan, A., Khiew, P.S.: A flipped classroom: learning
experiences in programming. Int. J. Virtual Personal Learn. Environ. (IJVPLE)
11(1), 23–37 (2021)
17. Kapur, M., Hattie, J., Grossman, I., Sinha, T.: Fail, flip, fix, and feed-rethinking
flipped learning: a review of meta-analyses and a subsequent meta-analysis. Front.
Educ. 7, 956416 (2022)
18. Roehling, P.V., Roehling, P.V.: Student reactions to and the effectiveness of the
flipped classroom across learning domains. In: Flipping the College Classroom: An
Evidence-Based Guide, pp. 15–43 (2018)
19. Long, T., Cummins, J., Waugh, M.: Use of the flipped classroom instructional
model in higher education: instructors’ perspectives. J. Comput. High. Educ. 29,
179–200 (2017)
20. Martı́n, R., Fernández-Molina, O.: Montero-alonso, & gonzález-gómez, f the main
components of satisfaction with e-learning. Technol. Pedagog. Educ. 24(2), 267–
277 (2015)
21. Strang, K.: Cooperative learning in graduate student projects: comparing syn-
chronous versus asynchronous collaboration. J. Interact. Learn. Res. 24(4), 447–
464 (2013)
22. Rivero, C., Chávez, A., Vásquez, A., Blumen, S.: Las tic en la formación univer-
sitaria: logros y desafı́os para la formación en psicologı́a y educación. Revista de
Psicologı́a (PUCP) 34(1), 185–199 (2016)
23. Cantuña Avila, A.A., Cañar Tapia, C.E.: Revisión sistemática del aula invertida
en el ecuador: aproximación al estado del arte. Estudios pedagógicos (Valdivia)
46(3), 45–58 (2020)
24. Boyatzis, R.E.: Transforming Qualitative Information: Thematic Analysis and
Code Development. Sage, Thousand Oaks (1998)
25. Han, E., Klein, K.C.: Pre-class learning methods for flipped classrooms. Am. J.
Pharm. Educ. 83(1), 6922 (2019)
26. Colomo-Magaña, E., Soto-Varela, R., Ruiz-Palmero, J., Gómez-Garcı́a, M.: Uni-
versity students’ perception of the usefulness of the flipped classroom methodology.
Educ. Sci. 10(10), 275 (2020)
27. Wilson, K.: What does it mean to do teaching? A qualitative study of resistance to
flipped learning in a higher education context. Teach. High. Educ. 28(3), 473–486
(2023)
28. Lo, C.K., Hew, K.F.: A critical review of flipped classroom challenges in k-12
education: possible solutions and recommendations for future research. Res. Pract.
Technol. Enhanc. Learn. 12(1), 1–22 (2017)
Technology Scouting for Techno-Pedagogical
Laboratories in Ecuador: A Case Study
Diego Lasso Lazo(B) , Francisco Alvarez Arévalo , Juan Pablo Carvallo ,

Javier Valdiviezo Ortiz, Javier Patiño Chuni, Jorge Bermeo Conto ,
and Katherine Espinoza Sánchez
CEDIA, Gonzalo Cordero y J Fajardo, Cuenca, Ecuador

{diego.lasso,francisco.alvarez,jpcarvallo,javier.valdiviezo,
javier.patino,jorge.bermeo,katherine.espinoza}@cedia.org.ec
Abstract. Utilizing the UNE (A Spanish Standard) 166000:2018 technology

scouting methodology, this research undertakes an in-depth exploration of the
status of techno-pedagogical laboratories within Ecuador’s higher education land-
scape. The methodological framework, encompassing a sequence of strategic
steps, enables the delineation of Key Scouting Factors (KSFs) and the formulation
of bespoke information gathering techniques, specifically designed to cater to the
unique requirements of higher education stakeholders. The study incorporates the
participation of twenty-one higher education institutions, each exhibiting distinct
levels of development in their techno-pedagogical laboratories. The systematic
analysis of the amassed data provides critical insights into the prevailing condi-
tions of these laboratories, in addition to identifying the challenges and oppor-
tunities presented within the context of Ecuador’s technological and economic
environment. The results underscore the efficacy of applying a standardized tech-
nology scouting methodology in generating valuable, actionable information for
pertinent stakeholders.
Keywords: Technology scouting · Techno-pedagogical laboratories · Strategic

intelligence · education
1 Introduction
Technology scouting, also referred to as technology monitoring or surveillance, is the

strategic process of retrieving information tailored to an organization’s specific needs [1].
The specific information collected enhances organizational competence by facilitating
informed, strategic decision-making. The primary objective of technology scouting is the
early identification of challenges and opportunities within technological, commercial,
and innovative sectors [2], hence its importance as the best mechanism for gathering,
filtering and providing strategic information [3]. As a foundational step in technology
development projects, technology scouting paves the way for technology mapping [2].
Over the years, this practice has evolved beyond its traditional confines within dynamic
technological enterprises and has been adopted by organizations engaged in research,
https://doi.org/10.1007/978-981-99-7353-8_23
302 D. Lasso Lazo et al.
development, and innovation. Within the educational context, technology scouting can
boost innovation, particularly within universities engaged in advanced scientific research
[4], thereby playing a pivotal role in technology transfer processes.
The present paper aims to implement a technology scouting system based on the
UNE 166006:2018 standard, to collect and analyze relevant data regarding the status
of techno-pedagogical laboratories in Ecuador. A techno-pedagogical laboratory is a
venue equipped with technology to enhance teaching and learning within a pedagogi-
cal framework, which nowadays has become of high relevance for Ecuadorian higher
education institutions [5, 6]. Through this process, we aim to validate the notion that a
standardized technology scouting methodology can provide valuable insights for higher
education institutions, in this case, regarding techno-pedagogical laboratories. This anal-
ysis is relevant to our parent institution, the Ecuadorian Corporation for the Advancement
of Research and Academia (CEDIA), formed by most of the higher education institu-
tions in Ecuador. As such, this study highlights how CEDIA used technology scouting
based on the UNE 166006:2018 standard to obtain valuable information on the status of
the techno-laboratories in the country. This standard provides a more specific framework
for technology scouting compared to others standards such as ISO 56002:2019, which
focusses on innovation management in general.
This paper is organized as follows: Sect. 2 present a literature review of related works.
Section 3 outlines the methodology, detailing the application of the UNE 166006:2018
standard for technology scouting in the context of techno-pedagogical laboratories in
Ecuador. In Sect. 4, we present and discuss the data collected from the participating
higher education institutions. Section 5 provides a discussion that contextualizes these
findings within the broader Ecuadorian technological and economic landscape. Finally,
Sect. 6 concludes the paper with a summary of the key findings, the benefits of applying
a standardized technology scouting methodology, and potential directions for future
research.
2 Related Work
Technology scouting has emerged as a valuable tool in the planning and management
of educational ecosystems. Marquéz and Galano [7] present a process for implementing
technology scouting in educational institutions, adhering to the UNE 166006, which is
the standard that delineates the process of technology scouting. It was first proposed in
2006 and then further refined in 2011 and 2018. As of now it encompasses the entire
process of extracting valuable technological evolution information and producing valu-
able insights from it. Their study concludes that technology scouting not only meets the
information requirements of the institution but also augments scientific research. Thus,
highlighting the value of a systematized methodology for the monitoring of technology
trends.
In their work, Cruz-Rojas et al. [8] describe the implementation of the technology
scouting process at Universidad el Valle in Colombia, also developed in accordance
with the UNE 160006:2018 standard. They identified twenty university needs. Of these,
“Access to academic sources” and “Virtual campus for education” were selected as the
most pressing pain points to be addressed to meet the stakeholders’ requirements. In
Technology Scouting for Techno-Pedagogical Laboratories 303
a similar work, Berges et al. [9] implemented a support platform based on strategic
intelligence for educational management in the community of Madrid, Spain. This plat-
form integrates diverse information sources and employs dynamic analysis aligned with
the framework of technology surveillance. The results demonstrated that applying a sys-
tematic methodology for technology scouting improved decision-making agility through
reduced interaction requirements for analysis, resulting in widespread user acceptance.
Wang et al. [10] examine the role of external technology scouting in generating inbound
open innovation in high-technology industries. They propose that firms with strong
absorptive capacity are more likely to generate innovations and capture external technol-
ogy. The study, conducted in Taiwan’s high-tech industry, finds that external technology
scouting, absorptive capacity, and technological turbulence all play important roles in
generating inbound open innovation. The authors suggest that firms should focus on
developing effective external technology scouting mechanisms and maintaining absorp-
tive capacity in uncertain technological environments to enhance innovation generation.
Moyares and Infante [11] studied technological observatories in higher education, aim-
ing to identify common aspects and relevant information for universities in developing
decision-making platforms. They analyzed four observatories using documentary anal-
ysis and induction-deduction methods. The observatories studied had heterogeneous
characteristics, depending on their surveillance objectives. The article emphasizes the
need for further research in this area. The authors highlight the importance of technolog-
ical observatories as platforms for technology scouting, as they facilitate the collection,
processing, and dissemination of information. Delgado-Fernandez et al. [12] studied the
importance of technological scouting in universities, specifically focusing on a Cuban
university of technical sciences. They emphasized the need for systematic implemen-
tation of technology scouting and the use of ICT tools to support this process. The
article highlighted the usefulness of technological observatories in identifying strategic
research programs and trends in engineering and architecture education. The authors
also discussed the role of data mining techniques in identifying technological trends and
opportunities for research. Marulanda et al. [13] studied the perception and utilization
of technological surveillance among undergraduate students at the National University
of Colombia, Manizales Campus. The study found that students were not effectively
using surveillance tools provided by the university. They emphasized the importance of
technological surveillance in decision-making and innovation, and recommended strate-
gies to enhance its use in learning processes. The study also highlighted the need for
improvement in the university’s management of technological surveillance.
Finally, Yulianto and Kasahara [14] developed a data warehouse system to con-
solidate and analyze information related to tuition fee management, utilizing a business
intelligence approach. This system provides valuable information to institutional admin-
istrators for making strategic decisions within the educational context. All these instances
display the pertinence and usefulness of technology scouting as a methodology for gath-
ering technological development trends and extracting valuable insights to be used in
the decision-making process in the educational context.
3 Methodology
The methodology employed in this study is guided by the UNE 166006:2018 standard
[15], which defines the process of technology scouting. This standard provides a frame-
work for data collection, analysis, and diffusion, as well as leveraging technology mon-
itoring into producing strategic intelligence to support the data-driven decision-making
process. Figure 1 depicts the steps of the technological monitoring process implemented
in this study.
Fig. 1. Technological monitoring and intelligence workflow described in UNE 166006:2018.

Translated form [8].
3.1 Identification of Needs, Information Sources and Means of Access
The initial step of this methodology involves identifying the needs of technology scout-
ing. To define the information necessities, we utilize the mechanism “Expected demands
of internal or external stakeholders” described in the UNE 166006:2018 standard. In line
with CEDIA’s objective of reducing the gap between its members and access to innovative
technology, particularly academic tools and methodologies, technology monitoring for
techno-pedagogical laboratories in Ecuador holds significant importance. This process
supports the continuous development of solutions for the academic sector and facilitates
the identification and assessment of strategic segments where CEDIA has emerged as
an innovation pioneer.
3.2 Planning of the Technology Scouting
The second step encompasses the planning phase of the technology scouting process.
Adhering to the UNE 166006:2018 standard, we selected the “Research and scouting of
new unknown areas” approach for this study. Table 1 outlines the allocation of resources
and the timetable for the technology scouting activities.
Table 1. Technology scouting planning (AT = Academic Team; SIT = Strategic Intelligence
Team)
Activity Assigned resources Timetable

Selection of primary sources AT One week
Selection, construction and validation of the means of AT/SIT One week
information access
Data collection SIT Three weeks
Extraction, Transform and Load (ETL) AT/SIT Two weeks
Analysis SIT One week
3.3 Information Search and Treatment
The third step entails defining the strategy for information search and treatment. This
stage adheres to the UNE 166006:2018 standard and comprises three key components.
First, the definition of Key Scouting Factors (KSFs) addresses the expressions rep-
resenting the knowledge or intelligence needs of the organization. In response to the
identified needs in step 2.1, the KSFs encompasses:
– Services and stakeholders of the techno-pedagogical laboratories.
– Main issues and challenges faced during the use of educational tools in the techno-
pedagogical laboratories.
– Main educational methodologies and tools used in the Ecuadorian context.
– Economic context of the techno-pedagogical laboratories in Ecuador.
Second comes the definition and validation of means of information access, which
involves selecting the primary sources of information, namely the individuals respon-
sible for managing the techno-pedagogical laboratories at the Universities and higher
education institutes. To collect valuable information related to the KSFs, an online sur-
vey was chosen as the principal means of information access. Table 2 provides a detailed
description of the survey components.
Lastly, we applied an initial analysis of structured information, recommended by
the UNE 166006:2018 standard. This entails performing quantitative and/or qualitative
analysis of the information retrieved to obtain valuable insights. To uncover hidden
structures or correlations, the response distribution for each question was graphed and
analyzed.
Table 2. Survey construction (GI = General Information, SS = Services and stakeholders, MIC =
Main issues and challenges, MEMT = Main educational methodologies and tools, EC = Economic
context)
KSF Assigned resources Timetable

GI Write your full first name and last name: Open
GI Describe your position within the institution: Open
GI Indicate the name of the institution to which you Multiple choice
belong:
GI Your institution is classified as: Multiple choice
GI Does your institution have techno-pedagogical Yes/No/To be implemented
laboratories?
GI How many techno-pedagogical laboratories does your 1 to 5
institution have?
GI Enter the name of each laboratory Open
SS Who makes or will make use of your Likert
techno-pedagogical laboratory?
SS What is the scope of the people who use or will use Likert
the techno-pedagogical laboratory geographically?
SS In general, what is the level of use of the Likert
techno-pedagogical laboratory by users?
MIC What do you think are the main difficulties or barriers Likert
that users have faced when using the
techno-pedagogical laboratory in their educational
activities?
MEMT What is the thematic area of the laboratory? Likert
MEMT What teaching methodologies do you use in the Likert
techno-pedagogical laboratory?
MEMT What lines of teaching does your techno-pedagogical Likert
laboratory cover?
SS Which of the following services does your Likert
techno-pedagogical laboratory provide?
EC What is the estimated value of the investment made in Likert
your techno-pedagogical laboratory?
EC What was the main source of investment for the Likert
creation of the techno-pedagogical laboratory?
(continued)
KSF Assigned resources Timetable

EC What is the main source of financing for the Likert
maintenance and operation of the techno-pedagogical
laboratory?
SS What is the main value proposition of the Open
techno-pedagogical laboratory of your educational
institution, in terms of the capabilities, resources and
services it offers to improve teaching and learning?
3.4 Added Value of Information
This step focuses on adding value to the information collected through technology scout-
ing. Following the UNE 166000:2018 standard, we implemented a five-step process to
construct information value-added products. These steps are data integration, informa-
tion interpretation, graphic representation, extraction of analyzed facts and implications,
and recommendations of actions.
3.5 Storage and Distribution of Information

The last step involves defining a strategy for the storage and distribution of the added-
value products. As the case analyzed in this paper is an internal process within the organi-
zation, the information was stored locally and was delivered as a report to the stakeholders
of CEDIA, particularly those involved in the data-driven decision-making process. More
specific decisions related to storage and distribution are considered confidential and are
not within the scope of this study.
4 Results
4.1 Characteristics of the Group who Answered the Survey
First, we present the characteristics of the respondents belonging to the institutions

that answered the survey. The survey was created in SurveyMonkey and sent to all
higher education institutions members of CEDIA, 56 in total. We received 21 valid
responses, which represents a response rate of 37.5%. With the support of personnel
from CEDIA’s team, we identified that while the response rate may seem low, the most
relevant institutions which possess techno-pedagogical laboratories did respond to the
survey. Furthermore, the response rate is well above the typical response rate of web
surveys, hence is acceptable for the purpose of this study [16]. Table 3 shows distribution
of the institutions which participated in the survey.
Table 3. Distribution of the existence of Techno-laboratories by type of Institution
Type To be implemented soon YES NO

Technological Institute 1 2
University 4 4 10
Total 4 5 12
The collected data was initially in an unstructured format, which was not useful
for conducting statistical analysis. For the process of extraction, transformation, and
loading (ETL) of the data, a python script was used. The script was developed in a
Jupyter Notebook and its function was to read the raw data in csv format, perform all
the pertinent transformations so that the distribution graphs could be constructed. The
cleaned data was saved as an excel file. The graphs were then constructed in Excel.
Of the 21 responses, 12 participants possess at least one techno-pedagogical labo-
ratory, while four plan to implement techno-laboratories soon. Five institutions do not
possess any techno-pedagogical laboratories. Hence, 76.19% of the participants possess
or plan to implement techno-pedagogical laboratories.
Table 4. Number of techno-pedagogical laboratories per Institution
Type Average number of techno-pedagogical laboratories

Technological Institute 3.5
University 2.9
Total 3
Table 4 shows the average number of techno-pedagogical laboratories per institution.

On average, each institution possesses three facilities classified as techno-pedagogical
laboratories. Regarding the position of the person who is participating in the survey,
the majority fell into the category of “Director of Research or education”. Having pre-
sented the characteristics of the institutions surveyed, we now present the results of the
technology scouting and intelligence process based on the key scouting factors:
4.2 Services and Stakeholders of the Techno-Pedagogical Laboratories

This first KSF include data regarding type of users, geographical scope of the services,
level of use, and type of services.
Users. Figure 2 shows the percentage of each type of user of the techno-pedagogical
laboratories. Seventy five percent of users are either students or teachers (continuous
training or research). Hence, the main objective of the techno-pedagogical laboratories
is to support teaching activities and research.
5% Students
8% Teachers (continuous training)
30%
Teachers (research)
10% 2%
Public in general
Companies
20% Does not apply
25% Directives
Fig. 2. Distribution of type of users of the techno-pedagogical laboratories
Geographical Scope. Figure 3 shows the percentage distribution of the techno-

pedagogical laboratories based on the geographical scope. More than half of the
laboratories concentrate their services within the province.
5% For the institution only

13%
31% Province
City
15%
Country
Does not apply
16% 20% International
Fig. 3. Distribution of the geographical scope of the techno-pedagogical laboratories
Level of Use. Figure 4 shows the percentage distribution of the techno-pedagogical

laboratories per level of use. Almost half of the laboratories reported a high level of use.
Twenty three percent chose the option “Does not apply”, which could indicate the lack
of metrics to measure the level of use.
16% High: Several times per week
Does not apply

44%
17%
Moderate: Once per week
Very high: Daily

23%
Fig. 4. Distribution of level of use of techno-pedagogical laboratories
Type of Services. Figure 5 shows the percentage distribution of the techno-pedagogical

laboratories based on the service they provide. More than half of the laboratories focus
on services for improving the teaching process such as: Training and teaching educa-
tion, Design of educational resources, Design, and implementation of virtual learning
environments.
Training and Teacher Education
Design and Development of Digital Educational Resources

5% 5%
8% 30% Design and Implementation of virtual learning environments
Research and Evaluation of Educational Technologies

11%
Evaluation and Selection of Technological Tools and Resources
11% Advice on integration in Technology

18%
12% Technical Support and Maintenance of Technological Infrastructure
DOES NOT APPLY
Fig. 5. Distribution of services of techno-pedagogical laboratories
4.3 Main Issues and Challenges
This KSF focuses on gathering information regarding the difficulties faced while using
the facilities of the techno-pedagogical laboratories. Figure 6 shows the distribution of
difficulties reported by the techno-laboratories. The main issue is “Lack of training or
education in the use of technologies”. Around 15% of the laboratories chose “Does not
apply”, which could indicate the lack of metrics to identify the difficulties.
Lack of training or education in the use of technology

8%
11% DOES NOT APPLY
36%
Lack of access to equipment or technological tools
13%
Technical difficulties or problems with software/hardware
16% Limitations on internet connectivity

16%
Limitations on the availability of digital educational resources
Fig. 6. Distribution of difficulties reported in techno-pedagogical laboratories.
4.4 Main Educational Methodologies and Tools in the Ecuadorian Context

This third KSF includes the methodologies adopted, the technological tools imple-
mented, and the fields of knowledge covered by the techno-laboratories.
Technological Tools Implemented. Figure 7 shows the distribution of the technologies

implemented in the techno-pedagogical laboratories. There is not a significant differ-
ence between the percentages of the technological tools implemented in the techno-
pedagogical laboratories. However, forms of Augmented reality, Cloud computing, and
Learning analytics are the main technologies available.
Virtual/Mixed/Augmented/ Extended reality

Cloud Computing
3%
7% Learning analytics
22%
Other
7% Smart Classroom
1% 1% 1%
Artificial intelligence
7% Hybrid Learning Spaces
13% Robotics
Virtual Worlds – Metaverse
7%
IoT
8% 11% Home automation
Big data
12%
Blockchain
Fig. 7. Distribution of technologies implemented in techno-pedagogical laboratories.
Methodologies Adopted. Figure 8 shows the distribution of the methodologies adopted

in the techno-pedagogical laboratories. There is not a significant difference in the per-
centages of the different methodologies adopted by the techno-pedagogical laboratories;
however, Collaborative learning, Project based learning, and problem-based learning
together account for nearly 50%.
Collaborative learning
3% Project Based Learning
5% 18%
Problem-based learning
8%
Gamification
1% 1%
Flipped Classroom
8%
Case studies
16% Blended learning (b-learning)
9% DOES NOT APPLY
Peer or team learning
10% 11% Adaptive Learning

10% HyFlex Learning
Immersive Learning
Fig. 8. Distribution of methodologies adapted in techno-pedagogical laboratories.

Fields of Knowledge. Figure 9 shows the distribution of academic fields the techno-
pedagogical laboratories cover. Education, Science, and Engineering are the main fields
in which techno-pedagogical laboratories focus their services.
Education
2% Sciences
9%
5% 24% Engineering, industry and construction
8% Health and social services
Humanities and arts
9%
Social sciences, business education and law
18%
9% Services
16% Other
Agriculture
Fig. 9. Distribution of fields of knowledge covered by techno-pedagogical laboratories.
4.5 Economic context

This last KSF contemplates the investment funding for implementing techno-
pedagogical laboratories and their operational and maintenance funding.
Investment Funding Sources. Figure 10 shows the percentage of distribution of the
source of investment funding for techno-pedagogical laboratories. The main investment-
funding source for techno-pedagogical laboratories come from the institution itself. This
reality reflects the economic burden that institutions have to face to implement these
educational solutions.
Institution's own resources

4%
Government or public agency funds
13%
1% DOES NOT APPLY
2%
2% Research funds or specific projects
17% 61% Other types of financing, depending on the institution and its context
Donations or sponsorships from private companies or foundations
Capital investments by entrepreneurs or investors
Fig. 10. Source of investment funding for the techno-pedagogical laboratories.
Operational and Maintenance Funding Sources. Figure 11 shows the distribution

of the sources of the operational and maintenance funding for the techno-pedagogical
laboratories. In this case, even a higher percentage of institutions rely on their own
funding for the operation and maintenance, which again reflects the economic burden
that institutions face.
2% Institution's own resources

4% 2%
5% Government or public agency funds
11% Research funds or specific projects
Donations or sponsorships from private companies or

foundations
DOES NOT APPLY
76%
Capital investments by entrepreneurs or investors
Fig. 11. Source of funding for operation and maintenance.
5 Analysis
The analysis of the obtained results provides valuable insights into the status of techno-
pedagogical laboratories in Ecuador. The data clearly indicate a strong and growing
interest among Higher Education Institutions in the country to adopt innovative educa-
tional technologies. This enthusiasm for transformation encompasses the implementa-
tion of novel technological tools and a dedication to refining teaching methodologies,
all with the aim of enhancing the educational process. These institutions are making
significant efforts to equip both students and faculty members with the necessary tools
for a dynamic and robust learning environment. The provision of resources stimulates
a culture of innovation and exploration, leading to wider usage and adoption of these
technologies. Notably, a considerable portion of these resources is directed towards
enhancing teacher training.
While the thematic focus of the techno-pedagogical laboratories varies extensively,
there is a clear inclination towards technologies related to computation and information.
Digital technologies are perceived as instrumental in bolstering the learning process.
Interestingly, despite the current availability of these technologies, a radical shift in ped-
agogical methodologies has not yet taken place. The data from this study suggests a cau-
tious approach, with institutions preferring to balance traditional educational approaches
with the opportunities presented by modern technology. Collaborative learning, project-
based learning, and problem-based learning continue to be the most frequently employed
pedagogical methods, even as newer methodologies are acknowledged.
Certain fields of knowledge benefit more from these resources than others. Accord-
ing to the respondents, techno-pedagogical laboratories are predominantly deployed in
the areas of Engineering, Education, and the Sciences. This observation indicates the
intrinsic compatibility of these subjects with the available technology.
Implementing and operating these innovative spaces come with various challenges.
Stakeholders often face complex issues in fully utilizing these resources. Notably, the
lack of appropriate training in the use of these technologies emerges as a prominent

obstacle. The gap between having the technology and effectively utilizing it poses a
significant challenge. Additionally, the financial aspect plays a crucial role, with the
main sources of funding for implementation and ongoing maintenance relying on the
institutions themselves. Despite limited budgets and prohibitive costs, higher educa-
tion institutions demonstrate commendable perseverance in their endeavor to integrate
modern technologies into their learning spaces.
Therefore, the current state of techno-pedagogical laboratories in Ecuador represents
a notable stride towards a technologically enriched educational landscape. While chal-
lenges exist, the commitment to continual learning and adaptation offers a promising
outlook for the future of education in the country.
6 Conclusion
This research serves as a valuable case study illustrating the effective application of
technology scouting as a strategy to gain insights on the status and progression of a
technological area, specifically, innovations in educational tools. The intelligence gath-
ered through this systematic approach could prove useful to stakeholders involved in
the higher education sector in Ecuador. By adhering to a standardized method of data
collection and processing, this study has successfully produced pertinent information
that holds potential for incorporation into further analyses and strategic planning. Signif-
icantly, the methodology of technology scouting exhibits versatility, as it can be utilized
across any subject matter where technological advancements are of interest.
The insights derived from this process can be harnessed by stakeholders and key
players in the relevant sectors, considering that this study is exploratory in nature. A more
comprehensive understanding of the current landscape and future trajectories can guide
strategic decision-making, investment directions, and policy formulations. Thus, the
power of technology scouting extends beyond mere information collection, providing a
robust basis for informed actions in the complex and dynamic world of higher education.
While the results obtained in this study are not yet meant to be considered for the
decision-making process, the main contribution of this study is displaying the benefits
of implementing a technology scouting methodology.
This study focuses specifically on techno-pedagogical laboratories in Ecuador, hence
it is important to acknowledge the unique socio-economic and technological landscape of
the country. To mitigate the limitation of generalizability, future research could consider
conducting comparative studies across different geographical contexts, including coun-
tries with diverse socio-economic and technological backgrounds. This would provide
a broader understanding of the impact of technology scouting on techno-pedagogical
laboratories in various settings.
Furthermore, while the methodology proposed in UNE 166006:2018 provides a guid-
ance for the development of technology scouting, it does not provide specific solutions
for common problems of information recovery process such as stagnation points and
information bias. The last one especially represents a major challenge given that collec-
tion methods as surveys rely heavily on self-reported data, which is subject to various
forms of bias, including social desirability bias and recall bias. To address this issue,
additional measures can be implemented to minimize potential biases. For instance, a

mixed-methods approach may offer solutions by combining survey data with qualitative
interviews or observations. This triangulation of data sources would allow for a more
comprehensive understanding of the phenomena under investigation, reducing reliance
solely on self-reported information. The study of mixed methods however is out of the
scope of this study, which focus only on proving the advantages of using a standardized
method for the planning and conducting of technology scouting process.
In conclusion, the status, and prospects of techno-pedagogical laboratories in
Ecuador demonstrate a commendable commitment to a technologically enriched edu-
cational environment. Despite the challenges, the consistent efforts towards integrating
modern technologies into learning spaces are testimony to Ecuador’s proactive app-
roach towards embracing the digital era in education. The intelligence garnered from
technology scouting, as demonstrated in this research, provides a promising foundation
to inform and guide this continuous journey of educational transformation.
Acknowledgments. This study was financed by CEDIA through the projects “Laboratorios de
tendencias tecnopedagógicas” and “Conformación de una Unidad de Vigilancia tecnológica y Red
de Expertos”.
References
1. Gudanowska, A.E.: Technology mapping as a tool for technology analysis in fore sight studies.
In: 2014 IEEE International Technology Management Conference, pp. 1–4 (2014)
2. Stute, M., Sardesai, S., Parlings, M., Senna, P.P., Fornasiero, R., Balech, S.: Technology
scouting to accelerate innovation in supply chain. In: Fornasiero, R., Sardesai, S., Barros,
A.C., Matopoulos, A. (eds.) Next Generation Supply Chains. LNMIE, pp. 129–145. Springer,
Cham (2021). https://doi.org/10.1007/978-3-030-63505-3_6
3. Phaal, R., Farrukh, C.J., Probert, D.R.: Technology roadmapping—A planning framework
for evolution and revolution. Technol. Forecast. Soc. Change 71(1–2), 5–26 (2004)
4. Schultz, C.: Does technology scouting impact spin-out generation? An action research study
in the context of an entrepreneurial university. In: Mietzner, D., Schultz, C. (eds.) New Per-
spectives in Technology Transfer. FSSBE, pp. 107–128. Springer, Cham (2021). https://doi.
org/10.1007/978-3-030-61477-5_7
5. Díez, Ó.: Formación tecnopedagógica: DIY para tecnófobos. Apertura: Rev. de Innovación
Educativa 2(2), 108–121 (2010)
6. R. Perspectivas: Laboratorios para una metodología pedagógica y tecnológica más partici-
pativa. Cultura Científica – UTPL. https://culturacientifica.utpl.edu.ec/laboratorios-para-una-
metodologia-pedagogica-y-tecnologica-mas-participativa/ (2019). Accedido 18 de agosto de
2023
7. Márquez, J.V., Galano, A.U.: La vigilancia tecnológica: Un reto para el proceso enseñanza
aprendizaje en función de la física general usando las TIC. Latin-Am. J. Phys. Educ. 13(1),
3 (2019)
8. Cruz-Rojas, G.A., Molina-Blandón, M.A., Valdiri-Vinasco, V.: Vigilancia tecnológica para la
innovación educativa en el uso de bases de datos y plataformas de gestión de aprendizaje en
la universidad del Valle, Colombia. Revista de Investigación, Desarrollo e Innovación 9(2),
303–317 (2019)
9. Berges, A., Ramírez, P., Pau, I., Tejero, A., Crespo, Á.G.: A framework for strategic intelli-
gence systems applied to education management: a pilot study in the community of Madrid.
IEEE Access 9, 75313–75323 (2021)
10. Wang, C.H., Quan, X.I.: The role of external technology scouting in inbound open innovation
generation: evidence from high-technology industries. IEEE Trans. Eng. Manage. 68(6),
1558–1569 (2019)
11. Moyares, Y., Infante Abreu, M.B.: Caracterización de los observatorios como plataformas
para la gestión de la vigilancia tecnológica en el sector de la Educación Superior. Revista
Venezolana de Información, Tecnología y Conocimiento 13(1), 11–27 (2016)
12. Delgado-Fernández, M., Infante-Abreu, M., Abreu-Lee, Y., Infante-Pérez, O., DíazBatista,
A., Martínez-Moreno, J.: Vigilancia tecnológica en una universidad de ciencias técnicas.
Ingeniería Industrial XXXII(1), 69–75 (2011)
13. Marulanda, C.E., Hernández, A., López, M.: Vigilancia tecnológica para estudiantes uni-
versitarios: El caso de la universidad nacional de colombia, sede manizales. Formación
Universitaria 9(2), 17–28 (2016)
14. Yulianto, A.A., Kasahara, Y.: Data warehouse system for multidimensional analysis of tuition
fee level in higher education institutions in Indonesia. Int. J. Adv. Comput. Sci. Appl. 11(6),
541–550 (2020)
15. UNE: Management of R&D+i: System of surveillance and intelligence. https://www.une.org/
encuentra-tu-norma/busca-tunorma/norma?c=N0059973 (2018)
16. Fan, W., Yan, Z.: Factors affecting response rates of the web survey: a systematic review.
Comput. Hum. Behav. 26(2), 132–139 (2010)
Applying Cognitive Walkthrough to Usability
Assessment: A Case Study of an Academic
Management System
Socorro Vânia Lourenço Alves(B) , Celson Pantoja Lima ,

and Enoque Calvino Melo Alves
Postgraduate Program in Society, Nature and Development, Federal University of Western Pará
(Ufopa), Santarém, Brazil
socorro.alves@ufopa.edu.br
Abstract. Cognitive walkthroughs are used to examine the usability of a prod-

uct. They are designed to determine whether a new user can easily perform tasks
within a given system. The aim of this study was to describe the application of the
Cognitive Walkthrough inspection method in evaluating the ease of learning of
an Academic Management System. The evaluators’ steps, the selected evaluation
tasks, and the adopted analysis protocol are presented, providing a theoretical foun-
dation for the application of the method. The data collected from the evaluation is
qualitative in nature, derived from the evaluators’ records of their experiences and
behaviors while using the system. The results show that the method is relatively
simple to apply and does not require a significant amount of resources. It also
provides quick feedback to enable decision-making in the design process.
Keywords: Cognitive Walkthrough · Academic Management System · Usability
1 Introduction
The constant advancement of information and communication technologies has pro-

vided educational institutions with various advantages, especially in terms of academic
management and communication. In this sense, many of these institutions have allocated
resources and invested in technology to computerize systems designed to record, collect,
and generate data [1]. In particular, the use of Academic Management Systems (AMS)
has facilitated and improved the internal process flows of these organizations.
Academic management, in its broadest sense, can be understood as the set of pro-
cesses designed to oversee and enhance the quality of teaching and learning in an edu-
cational institution. An AMS is software, typically accessed through a web portal and
application, that primarily aims to enhance the management of institutional processes,
making them more efficient, convenient, and impactful [2].
In recent years, many AMS have also started to incorporate modules from Virtual
Learning Environments (VLE) into their platforms. Thus, in addition to the administra-
tive modules, they have started integrating functionalities that enhance the relationship
https://doi.org/10.1007/978-981-99-7353-8_24
318 S. V. Lourenço Alves et al.
between teachers and students in the teaching-learning process. The creation of vir-
tual rooms and/or classes in VLE, for example, allows teachers to share instructional
materials, post notices, assign tasks, monitor activity submissions, schedule classes and
assessments, record attendance, among other features [3]. Students, on the other hand,
typically find resources in these environments that provide them with access to course
materials, grades, deadlines, absences, substitutions, discussion forums, among other
things. In short, these systems are platforms that facilitate the cultivation of collaborative
learning, whether in person or remotely, by encouraging interaction and experimentation
through the utilization of technological resources.
Studies have shown that many users of AMS have difficulties in using and learning
their functionalities, primarily due to the presence of usability problems. In these systems,
ease of learning is a crucial feature as it reduces the complexity of user interaction and
assists educational institutions in evaluating the quality and adoption of software products
in this category. For students in particular, these systems can offer various benefits as they
provide different functionalities that facilitate and expedite access to academic services.
The literature presents efforts to develop approaches and techniques aimed at enhanc-
ing the usability of applications through various methods of usability evaluation. Scholars
in the field of Human-Computer Interaction (HCI) have demonstrated the significance
of evaluating the usability of a system’s interface. Users expect it to be user-friendly and
provide a pleasant and engaging experience [4]. The Cognitive Walkthrough is one of
these methods, which aims to assess the ease of learning by exploring the interface [5].
With this in mind, this article describes how to use the Cognitive Walkthrough
inspection method in a practical manner. It evaluates the ease of learning for users when
working with an AMS, emphasizing the importance of interaction design in supporting
and facilitating user learning during system use. One of the most widely used AMS
in Brazilian educational institutions was selected as a case study. The guidelines and
analysis protocol adopted in this study can assist other researchers who wish to utilize
the Cognitive Walkthrough as a methodology for assessing the ease of learning of an
educational system. In addition, it is important to highlight the strengths of this technique.
The approach is focused on addressing user problems, and it does not require a system
to be fully developed before it can be applied. This opens up the possibility of using it
in the early stages of software development.
The rest of the article is divided as follows: Sect. 2 presents a brief theoretical
background on the Cognitive Walkthrough method. Section 3 discusses the applied
methodology. Section 4 presents the results obtained from the evaluation of the selected
AMS as the case study. And finally, in Sect. 5, there are the final considerations.
2 Cognitive Walkthrough
ISO 9241 defines usability as the ability of an interactive system to allow its users to
effectively, efficiently, and enjoyably perform tasks within a specific operating context
[6]. Nielsen [7] states that usability is divided into five attributes: learnability, efficiency,
memorability, errors, and subjective satisfaction. Usability is a combination of objective
factors related to productivity and subjective factors related to the user’s satisfaction
with their interaction with the system.
Applying Cognitive Walkthrough to Usability Assessment 319
The aim of usability is to design interfaces that facilitate easy, enjoyable, effective,
and efficient interaction. This characteristic enables the creation of transparent interfaces,
which do not hinder the process of operating a computer system. It offers the end user
full control of the environment without becoming an obstacle during interaction.
In the field of Human-Computer Interaction (HCI), methods have been developed
to evaluate usability in interfaces. Among the evaluation methods are user tests (or
observation methods), which involve the direct participation of users to carry out the
evaluation, and inspection methods (or analytical methods), carried out by evaluators
or experts, without the presence of users, such as Heuristic Evaluation and Cognitive
Walkthrough [8].
The Cognitive Walkthrough inspection method was used to evaluate the AMS
selected in this study because it is relatively simple to apply and does not require many
resources. Applying this method can reveal numerous inconsistencies and usability flaws,
particularly for users who are new to using a system, commonly referred to as beginners
or newcomers.
The Cognitive Walkthrough aims to assess the ease of learning, making it possible to
identify usability problems in the effects that the interface will have on the user through
exploratory interaction with it. In other words, how the system will influence the user’s
ability and expectations when performing a given task [9]. The method was motivated by
the preference of many people to learn through hands-on experience, rather than relying
on reading and following manuals [10]. With this method, it is possible to evaluate not
only the interface but also the cognitive processes required for a user who is new to the
system.
This method aims to examine the learning advantages for users in relation to various
factors, including: the alignment between users’ and designers’ understanding of a task;
the appropriate or inappropriate selection of signs; and the provision of adequate feed-
back on the outcomes of an action [11]. For example, if a user is attempting to add new
information to a system, it is highly probable that their initial instinct will be to search
for a button labeled “new” or “add”. If the action of adding something new is hidden
inside an “edit” menu, for example, the process of learning to use this new interface can
be more difficult.
The user interface is the visible part of the software that allows users to interact
with the system and perform their tasks. It is important for the user interface to be user-
friendly, meaning it should be easy to use and meet the expectations and needs of its
users.
The Cognitive Walkthrough does not involve end users. It can be carried out by one
or more evaluators (also called inspectors) who must empathize with potential users
of the system and simulate the actual context of performing the proposed tasks. They
should answer a set of predefined questions (Fig. 1) that are essential to the Cognitive
Walkthrough [5].
The set of questions shown in Fig. 1 is an essential tool during the inspection. It
must, therefore, be clarified and properly understood by the assessors. The answers to
the first three questions are linked to the mental models of the audience of the system
being evaluated. These models may vary depending on the users’ profiles. For instance,
an icon may hold more significance for one profile, while descriptive buttons may be
Fig. 1. Questions from the Cognitive Walkthrough method.
more helpful for another. The fourth question should be asked at the end of each step,
i.e. when the user anticipates receiving a response from the system.
With the results of the evaluation, it is possible to identify aspects that hinder the
execution (flow) of the task. Based on these findings, suggestions can be made to improve
the system’s interface, making it easier for the user to learn how to use it [12].
According to Dix et al. [13], in order to conduct a Cognitive Walkthrough, it is
necessary to have information about the system, the tasks that will be evaluated, the
steps required to complete each task, and the profile of the potential user. If the task is
too complex to write in a list format, an action flow diagram can be used, like the one
shown in Fig. 2.
Fig. 2. Task flow diagram.
The next section presents the methodology used in this study to apply the Cognitive
Walkthrough inspection.
3 Methodology
To conduct the Cognitive Walkthrough inspection, two evaluators were recruited to
analyze the same sequence of tasks. This measure was implemented to prevent bias in
the results. By having multiple individuals analyze the same flow, it was possible to
minimize this issue.
Both evaluators were graduates of the Computer Science course, where they had
studied Human-Computer Interaction (HCI) with excellent results. However, they were
unfamiliar with the Academic Management System used as a case study. Together, they
went through the sequence of actions for the tasks that needed to be inspected in the
system.
The tasks selected in this study were adapted from Carvalho [14], who provides
an overview of the primary complaints from users of the student profile in a well-
known Academic Management System. These complaints are also representative and
commonly found in various systems of this kind (Fig. 3). As depicted in the figure,
three tasks that were identified as the most challenging were divided into actions and
subsequently executed in a step-by-step manner. The evaluators followed the script of
the tasks, simulating the behavior of the user profile being analyzed (in this case, an
incoming university student), and checked for any potential problems. Their goal was
to predict interaction difficulties that could affect the user’s learning when performing
tasks in the system. In other words, they carefully analyzed the sequence of actions,
critiquing the system’s use.
Fig. 3. Tasks to be carried out by the evaluators.
Throughout the evaluation, the evaluators utilized a registration form to answer the
four essential questions that direct the implementation of the Cognitive Walkthrough
method: Will the user attempt to achieve the desired outcome? Will the user notice
that the correct action is available? Will the user correctly associate the action with the
desired effect they are trying to achieve? If the correct action is performed, will the user
see progress being made towards solving the task?
To conduct the Cognitive Walkthrough test, we considered a typical student persona

who has recently enrolled in university and has no prior experience with any other
academic management system. Personas are an important tool in interaction design and
consist of descriptions of the types of users for whom a product is designed [15]. They
allow us to understand how people behave, what their frustrations are, their desires, and
how they use the product or service being studied. Figure 4 shows a concise description
of the persona utilized in the inspection.
Fig. 4. Persona’s User Profile.
When starting the Cognitive Walkthrough, it is important for the evaluator to have
pens, post-its, paper, notepads, or any other materials at hand to help them take notes.
There are also several different ways to document the analysis of the Cognitive Walk-
through. In this study, the results were organized in descriptive tables, along with the
evaluators’ comments and observations, as depicted in the figures in the following
section.
To enhance the documentation further, the evaluator can also address key questions
at the conclusion of each task, such as:
• What was the most challenging step to execute? Why?
• What basic knowledge does the user need to quickly learn and use this system?
• What actions/steps can be improved?
• What feedback can be improved?
• What is functioning effectively in the system?
4 Results
The data collected from the evaluation using the Cognitive Walkthrough method is
qualitative in nature. It results from records of the evaluators’ experiences and behavior
when using the Academic Management System as a case study. The method revealed
that the system encounters difficulties in executing the tasks outlined in this study.
The evaluations, described below, have been summarized in the form of tables that
show the actions taken to complete the tasks and how they relate to the issues investigated
by the method. The Fig. 5 shows the results obtained from the evaluation of the first two
actions of task 1, which involve consulting the course curriculum. For the sake of space,
we are only presenting some of the actions in which the evaluators detected problems
in the interaction (highlighted in the table with pink).
Fig. 5. Cognitive Walkthrough of Task 1.
By analyzing the evaluators’ comments, as described in Fig. 5, it is possible to

identify difficulties in carrying out the first task. It can be observed that at the start of
the first two actions, the presence of certain queries in the main menu may prompt a
novice student user to initially search for the desired query in the displayed list. It is
unclear where to find the desired option, necessitating prior knowledge to search for the
curriculum structure of the course. The excessive number of services available under
“Teaching” also makes it difficult. The arrangement of the elements should adhere to a
consistent pattern and logical sequence, ensuring coherence with the user’s mental model.
Following this recommendation helps minimize the workload and makes it easier for
novice users to learn. Choosing the appropriate icons for an interface can significantly
impact the ease of understanding the information. The Fig. 6 shows the results obtained
after evaluating the task involving the request for student assistance (Task 2) using the
Cognitive Walkthrough.
In Task 2, there were also issues with interacting with the Academic Management
System. Once again, there was difficulty in connecting the immediate action with the
completion of the task. There was a breakdown in communication between the designer
and the user. The assessors repeatedly asked themselves, “Where is it?” when they
couldn’t find the service option. They also expressed that it was not possible to carry out
the requested action. It is important to clarify the available repertoire of actions because
users will select from the actions that are visible to them. An action hidden in a menu,
for example, is less likely to be discovered.
Also in Task 2, the word “Other” generated doubts as it is vague and provides little
indication of its actual function in the system. It is important to make an appropriate
choice of terms, i.e. the vocabulary used in the system. This difficulty was evident in the
amount of time it took to complete the task. The evaluators spent several minutes trying
to locate the information they needed in the system because the interface did not clearly
indicate the available options. For novice users, this affects the ease of learning.
While exploring and learning an interface, users may make mistakes. These mistakes
can hinder the learning process. Therefore, carefully considering how to present errors
is fundamental to teaching people how to use a system. At the same time, the system
should also provide the user with an emergency exit from the state they entered to the
safe state they were in before the last action was performed.
Finally, Fig. 7 shows the results obtained from the evaluation of the “Consult subject
grade” task. This task is quite common in the Academic Management System and is
also one of the most frequently used by students when interacting with the system.
As can be seen above, all the actions in task 3 have been successfully completed.
The objects, actions, and options are visible, making it easier for novice users to learn.
Feedback messages throughout the interaction inform the user about the consequences
of each task action. There are only a few steps to perform the tasks, which also makes
learning easier. The more choices a user has in a flow, the higher the likelihood that they
will not successfully complete the task.
It is worth noting that when users are using an interface for the first time, they
unconsciously employ a strategy known as label following [16]. In this strategy, users
search for clues or keywords in the interface that can be associated with the task they
are trying to complete. This behavior was observed in the evaluators during the analysis
of the Academic Management System.
Finally, the evaluators also noted that the system’s aesthetics and design did not
adhere to a minimalist pattern, which is an essential characteristic of modern interfaces.
The more information there is in an interface, the more analysis and decision-making the
user will have to do. Therefore, it is crucial to include only the necessary information,
while secondary information can be kept in the background.
5 Final Considerations
A good interface must satisfy the users’ needs. However, in order for this to happen,
the development process must be user-centered. This means that the interface must be
designed with the aim of satisfying the needs of the end user. The Cognitive Walkthrough
is relatively simple to apply and does not require a lot of resources. With this evalua-
tion method, it is possible to identify numerous inconsistencies and usability flaws,
particularly for users who are unfamiliar with the system.
After inspecting the tasks in the Academic Management System selected for evalu-
ation, we observed some elements that could be improved in its design to enhance the
user learning experience. No significant usability problems were detected. However, it
is worth noting that the method only focuses the evaluator’s attention on specific actions
that are predefined in the test script. This limitation can potentially conceal existing prob-
lems, which can result in biased analysis. It is also a method that requires significant
attention and time from evaluators.
The four questions that guide the inspection of the method, and which seek to high-
light the cognitive process used to perform the task, provided indications of how easily
users were able to learn how to use the interface. However, it is important to note that
the difficulty in understanding the questions can limit the interpretation of the success
or failure of task execution.
If the task is performed by different user profiles, it is recommended to conduct the
Cognitive Walkthrough at least once for each user profile. In this way, it will be possible
to detect more specific problems for that user profile. In the case of the selected Academic
Management System, this system is accessed by various user profiles, including older
students, indigenous people, quilombolas, and other individuals who enroll in Brazilian
universities every year. Thus, driven by the increasing number of users of this system,
there is a growing concern to make it more user-friendly.
Like any inspection method, the Cognitive Walkthrough also has some limitations.
As this test is conducted with the user in a controlled environment, this configuration
does not simulate a real-life usage scenario. In practice, this can mean that some of
the encountered problems are very specific and have a lesser impact on the system in
a real-life situation compared to what is observed. It is important to bear in mind that
the purpose of this analysis is to identify potential issues that users may encounter when
learning to interact with a system, rather than simulating everyday usage. Some of the
problems found in interfaces cannot be discovered simply by applying the cognitive
walkthrough questions. The evaluator should complement the application of the method
with a comprehensive study of interface construction guidelines, as outlined in [17].
References
1. Amber Sistemas: As vantagens de um sistema web para gestão acadêmica e comuni-
cação – Amber Sistemas. [online]. Disponível em: https://www.ambersistemas.com.br/as-van
tagens-de-um-sistema-web-para-gestao-academica-e-comunicacao/ (2019). Acessado em 12
jun 2023
2. Ziulkoski, L.C.C.: Integração do Moodle com o banco de dados institucional na UFRGS
(2010)
3. Assunção, A.B.A., Pimenta, I.L., Mol, A.L.R.: O uso do ambiente virtual de aprendizagem e
sua relação com a avaliação da docência: um estudo na Universidade Federal do Rio Grande
do Norte. Revista Gestão & Tecnologia 16(2), 127–152 (2016)
4. Barbosa, S.D.J., da Silva, B.S.: Interação Humano-Computador. Elsevier Editora Ltda. (2010)
5. Wharton, C., Rieman, J., Lewis, C.: Polson, P: The cognitive walkthrough method: a practi-
tioner’s guide. In: Nielsen, J., Mack, R.L. (eds.) Usability Inspection Methods. John Wiley &
Sons, New York, NY (1994)
6. ISO 9241, Parte 11: Orientações sobre Usabilidade. Associação Brasileira de Normas
Técnicas. ABNT (2011)
7. Nielsen, J.: Usability engineering. San Francisco: Morgan Kaufmann, 340 p. (1993). Wharton,
C., Rieman, J., Lewis, C., Polson, P.: The cognitive walkthrough method: a practitioner’s
guide. In: Nielsen, J., Mack, R.L. (eds.) Usability Inspection Methods. John Wiley & Sons,
New York, NY (1994)
8. Dias, C.:Usabilidade na Web: Criando portais mais acessíveis. 2ª edição. Rio de Janeiro: Alta
Books, 296 p. (2007).
9. Bonifácio, B., Vieira, S., Viana, D., Conte, T.: Aplicando técnicas de inspeção de usabilidade
para ava liar aplicações móveis. In: Proceedings of the IX Symposium on Human Factors in
Computing Systems, pp. 189–192 (2010)
10. Santos, F.S.: Um método de Percurso Cognitivo especializado para avaliar jogos digitais.
Dissertação de Mestrado do Programa de Pós-Graduação em Ciências de Computação e
Matemática Computacional (PPG-CCMC), Universidade de São Paulo, São Carlos – SP, 018
(2018)
11. Prates, R.O., Barbosa, S.D.J.: Avaliação de interfaces de usuário: conceitos e métodos. In:
Anais da Jornada de Atualização em Informática, XIX Congresso da Sociedade Brasileira de
Computação, Campinas (2003)
12. Pinto, L.S., Torres, E.F., Moura, J.M., Sousa, E.S., Pinto, L.A., Nóbrega, C.V.: Avaliação
da aceitação das ferramentas tecnológicas no ambiente do trabalho docente. Revista Gestão
Universitária na América Latina – GUAL. Florianópolis 12(2), 118–138 (2019)
13. Dix, A., Finlay, J., Abowd, G.D., Beale, R.: Human Computer Interaction, 3rd edn. Pearson
Education Limited, Harlow, England (2003)
14. Carvalho, N.O.: Avaliação de Usabilidade do Módulo de Alunos do Sistema Integrado de
Gestão de Atividades Acadêmicas – SIGAA/UFC: Estudo de Caso. Trabalho de Conclusão de
Curso (graduação) – Universidade Federal do Ceará, Campus de Quixadá, Curso de Sistemas
de Informação, Quixadá (2016)
15. Benyon, D.: Interação humano-computador/David Benyon; tradução Heloísa Coimbra de
Souza; revisão técnica Illana de Almeida Souza Concílio, 2nd edn. Pearson Prentice Hall,
São Paulo (2011)
16. Terwilliger, R.B., Poison, P.G.: Task elaboration or label following: an empirical study of
representation in human-computer interaction. In: Conference Computer – CHI 96 Human
Factors in Computing Systems, pp. 201–202. ACM (1996)
17. National Cancer Institute (NCI) – Research-Based Web Design & Usability Guidelines. http://
usability.gov/guidelines
On the Use of ChatGPT to Support
Requirements Engineering Teaching
and Learning Process
Juan Pablo Carvallo(B) and Lenin Erazo-Garzón
Universidad del Azuay, Cuenca, Ecuador

{jpcarvallo,lerazo}@uazuay.edu.ec
Abstract. The use of artificial intelligence (AI) in education is an emerging trend

that offers multiple opportunities and challenges. In this article, we present an
exploratory study on the use of ChatGPT, an AI-based tool that allows generating
responses to questions posed by the user, as support for project-based learning
in the Requirements Engineering course. The objective of the study is to identify
the advantages and disadvantages of using ChatGPT in this context, as well as to
provide recommendations for its implementation and improvement. To this end,
a survey was conducted with 27 students who used ChatGPT in the course, and
their responses were analyzed using statistical and qualitative techniques. The
study results show that students perceived ChatGPT as a useful and novel tool
that facilitated the generation and understanding of concepts, models and tools
related to requirements engineering, as well as the simulation of interviews with
value chain actors. However, some limitations and aspects to improve using Chat-
GPT were also identified, such as the accuracy and consistency of the generated
responses, the adequate contextualization of information, and the ability to cus-
tomize and adjust the tool. The study concludes that ChatGPT has great potential
to support project-based learning in the Requirements Engineering course, but also
requires technical and pedagogical improvements to optimize its performance and
effectiveness.
Keywords: Artificial intelligence · ChatGPT · Project-based learning ·

Requirements engineering
1 Introduction
Requirements engineering is a discipline that deals with identifying, analyzing, specify-

ing, and validating the requirements or needs that a system or software must meet [1]. It
is essential for the success of any software development project as it allows for defining
the scope and objectives, as well as establishing quality and customer satisfaction cri-
teria [2]. However, learning requirements engineering is not an easy task as it involves
mastering complex, abstract theoretical concepts, as well as developing practical skills
to apply them in real contexts [3].
https://doi.org/10.1007/978-981-99-7353-8_25
On the Use of ChatGPT 329
One methodology used to facilitate learning in requirements engineering is project-

based learning (PBL), which involves designing and developing a product or service that
addresses a real need or problem [4]. PBL has multiple benefits for students, such as
developing cognitive, social, and professional skills, fostering creativity and innovation,
and increasing motivation and commitment, among others [5–7]. However, PBL also
presents some challenges, such as difficulty accessing reliable and up-to-date information
sources, lack of guidance and feedback during the process, and scarcity of resources and
time to carry out activities, among others [8].
In this context, an opportunity arises to use an artificial intelligence (AI) tool that
can support and improve PBL in the Requirements Engineering course (REC). The
field of AI has experienced significant development and advancement in recent years
thanks to increased computational capacity and the availability of large amounts of
data [9]. AI has multiple applications and benefits in various fields, including education
[10]. In this sense, various AI-based tools have been developed to support and improve
teaching-learning processes for both teachers and students [11].
In this article, we present an exploratory study on the use of ChatGPT, an AI-based
tool that allows generating responses to questions posed by users, to support PBL in a
REC. ChatGPT is an application based on GPT-3 (Generative Pre-Trained Transformer
3), a natural language model based on deep neural networks, which has demonstrated
a remarkable ability to generate coherent and diverse text on a wide variety of topics
and domains [12]. This study aims to identify the advantages and disadvantages of
using ChatGPT in this context and provide recommendations for its implementation and
improvement. To this end, we conducted a survey with 25 students who used ChatGPT
in the course and analyzed their responses using statistical and qualitative techniques.
The paper is structured with six sections, in addition to the introduction: Sect. 2
presents the related work, Sect. 3 introduces the case study and methodology, Sect. 4
shows the results of the study, Sect. 5 discusses the results, Sect. 6 discusses the threats
to validity, and finally, Sect. 7 presents some conclusions and future work.
2 Related Work
The application of Artificial Intelligence in Education (AIEd) has garnered significant
research attention in recent years, leading to a growing body of literature on the topic.
These studies have explored the use of AIEd tools in various educational contexts,
investigating their advantages, challenges, and limitations.
In the field of natural language processing (NLP), Jurafsky and Martin (2009) intro-
duced NLP and its practical applications, including its relevance to education [13].
McTear et al. (2016) discussed the design and implementation of AI-based conversa-
tional interfaces, such as chatbots and virtual assistants, in different fields, including
education [14]. Johnson and Valente (2008) developed Tactical Language and Culture
Training Systems (TLCTS), an AI-based system designed to teach foreign languages
and cultures to military students [15]. These studies demonstrate the broad applications
of AI and NLP in educational contexts.
Furthermore, Crompton and Burke (2023) conducted a study examining the applica-
tion of AI in Higher Education (HE) from 2016 to 2022 [16]. The research revealed a rapid
330 J. P. Carvallo and L. Erazo-Garzón
increase in AIEd studies published in HE, spanning across six continents. The most com-
mon department affiliation was Education (28%), followed by Computer Science (20%).
The study identified five primary usage codes for AIEd in HE: Assessment/Evaluation,
Predicting, AI Assistant, Intelligent Tutoring System (ITS), and Managing Student.
The study also emphasized the need for empirical exploration of new tools like
ChatGPT, which became publicly available in late 2022. The authors suggested that
ChatGPT could align with several use codes found in the study, providing support to
students in writing papers, while also helping instructors assess students’ work and
assisting with writing tasks [16].
Hulick (2023) highlighted the potential impact of ChatGPT and similar AI technolo-
gies on education. While these tools can benefit students by offering assistance with
assignments, concerns regarding cheating and the potential for misinformation have
been raised [17]. To address these challenges, the UCLA Center for the Advancement
of Teaching (2023) developed a comprehensive guidance document for instructors on
effectively incorporating generative AI, including ChatGPT, into their teaching practices,
with an emphasis on ethical considerations and pedagogical strategies [18].
Although several studies [19–21] have explored the use of IA, including ChatGPT,
to support foreign language learning and highlighted the significant impact of the use of
these technologies on learning outcomes, research has also examined the specific appli-
cation of AI-based tools in engineering education. Goldberg (2012) provided insights
into the design of competent genetic algorithms, an AI technique applicable to solving
complex problems across various domains, including engineering [22]. Qadir (2022)
reflected on the promises and challenges of integrating ChatGPT and other generative
AI tools in engineering education, highlighting the opportunities for enhancing learn-
ing, creativity, and collaboration while acknowledging the risks associated with misuse
and improper application [23]. Nikolic et al. (2023) conducted a multidisciplinary and
multi-institutional study to assess the impact of ChatGPT on the integrity of engineering
education assessment [24]. They investigated how ChatGPT could both enhance and
compromise the assessment process, depending on the context and task design.
These studies lay a strong foundation for exploring the use of ChatGPT in a require-
ment engineering course. They demonstrate the potential of AIEd tools like ChatGPT in
enhancing educational practices and supporting students and instructors in various ways.
However, they also highlight the need for further research to fully comprehend and lever-
age the capabilities of these tools in specific educational contexts, such as requirements
engineering.
While the aforementioned works provide valuable insights, it is important to
acknowledge their limitations. Although studies have examined the benefits of Chat-
GPT and AI-based tools in education, further research is needed to fully understand
their pedagogical impact and address potential ethical concerns. These studies often
focus on specific domains, such as language education or engineering, and may not
encompass the full range of educational contexts in which ChatGPT can be applied.
Additionally, it is crucial to evaluate the long-term effects of using AI tools in education
and develop robust frameworks for assessing their effectiveness and impact on student
learning outcomes.
Building on the existing body of research, our study aims to contribute to the field of
engineering education in several ways. Firstly, our research evaluates the use of ChatGPT
as a support tool in a university-level engineering requirements course in Ecuador. By
examining students’ perceptions, experiences, and learning outcomes when using Chat-
GPT to simulate a fictitious company and its components, we aim to provide insights
into the practical application of AI in the specific context of engineering requirements.
Moreover, our study aims to identify the advantages, disadvantages, limitations, and
areas for improvement in using ChatGPT in this educational setting. By gaining a deeper
understanding of the potential benefits and challenges, we can develop strategies to
enhance the integration of AI tools in engineering education while mitigating associated
risks. This includes considering how to foster critical thinking and academic integrity
among students while harnessing the creative and collaborative potential of these tools.
Overall, our research expands upon the existing knowledge base by focusing on the
application of ChatGPT in engineering education and examining its specific impact in
the context of a requirements course. By addressing the limitations of previous studies
and providing new insights into the use of AI tools, we aim to contribute to the broader
discussions on the effective integration of AI in higher education, with implications for
pedagogy, student engagement, and the future of engineering education.
3 Case Study
3.1 Context
Universidad del Azuay, located in Cuenca, Ecuador, was established in 1970 and has
since become a recognized institution of higher education. The university is known
for its commitment to providing quality education and fostering academic excellence
in various disciplines. The Computer Science program at Universidad del Azuay is
designed to equip students with a comprehensive understanding of computer science
principles and practical application of technology.
Within the Computer Science curriculum, the REC holds significant importance.
This course focuses on the essential process of gathering, specifying, and validating
software requirements. Students learn the necessary skills to effectively communicate
with stakeholders and translate their needs into actionable requirements. By emphasizing
this critical aspect of software development, the university prepares students to excel in
the field.
Traditionally, the course required students to contact real companies and conduct
interviews to obtain firsthand information for requirements elicitation. This approach
aimed to bridge the gap between theoretical concepts and real-world applications, allow-
ing students to experience the challenges and complexities of working with stakeholders
in industry settings.
However, with recent curriculum changes, the requirement’s engineering course was
moved to the second semester, posing unique challenges. Second-semester students typ-
ically have limited technical knowledge compared to those in higher semesters, making
it even more difficult to engage with external companies and their employees. Acquiring
agreements with companies and securing their employees’ time and cooperation became
increasingly challenging, impacting the quality of the students’ learning experience.
To overcome these challenges and maintain the practical and experiential nature
of the REC, an innovative approach was introduced: The integration of ChatGPT as a
support tool aimed to enhance the teaching-learning process and provide students with
an experience close to industrial reality.
The introduction of ChatGPT as an aid in the teaching-learning process of require-
ments engineering is expected to yield several benefits. Firstly, it addresses the challenges
of engaging with real companies and their employees, especially for students in the sec-
ond semester with limited technical knowledge. Secondly, to provide a simulated but
realistic experience that allows students to develop their understanding of requirements
engineering concepts and enhance their practical application skills. Finally, by using
ChatGPT as a support tool, students are expected to create comprehensive models that
accurately depict the organization’s context, functional decomposition, and descriptions
of business and system use cases.
3.2 The Requirements Engineering Course in Universidad del Azuay
The REC in Universidad del Azuay, is based in the farmwork proposed by Klaus Pohl
[25], which fundaments the requirements engineering process in a well-established Sys-
tem Context. System context defines the scope and limits of the system, whose require-
ments are refined with three core activities, Elicitation, Negotiation and Documentation.
Also, three kinds of artefacts are used to model requirements: Goals, Scenarios and
Solution-oriented requirement models. Management and Validation are cross-sectional
activities to be performed along the entire requirements lifecycle.
To materialize the framework and guide the process in a systematic way, the
DHARMA method [26] is applied. This method aims to the definition of enterprise
architectures using the i* notation. The theoretical bases to support the method in the
analysis of enterprise context, structure and strategy, are two concepts defined by Porter
[27]: 1) the model of the market forces, used to reason about potential available strate-
gies and how to make them profitable, by analyzing existing dependencies with external
actors within five market forces, and 2) Value chain, which includes primary and sup-
port activities helpful to identify internal actors and dependencies in the scope of the
organization. The DHARMA method consists in four activities, as shown in Fig. 1:
• Activity 1. Modelling the enterprise context. The organization and its strategy are
carefully analyzed to identify its role inside the context, allowing the definition of
Context Actors (CA) and Organizational Areas (OA). At the end of this activity, i*
SD models are built and used to support reasoning and represent results.
• Activity 2. Modelling the environment of the system. In this activity, a system-
to-be is placed into the organization and its impact over the elements in the CM is
analyzed. The strategic dependencies of OAs and CAs are inspected to determine
which of them may be totally or partially satisfied by the system. The result of this
activity is also an i* SD model representing the dependencies that the system can
satisfy in relation to the different CAs or OAs.
• Activity 3. Decomposition of system goals. Dependencies included in the CM are
analyzed and decomposed into a hierarchy of intentional elements required to satisfy
them. These elements depict the services that the system must provide (functional
requirements) as well as restrictions on them (non-functional requirements). An i*

SR diagram for the system is built.
• Activity 4. Identification of system architecture. This activity includes the identi-
fication of System Actors (SA), which play a role in the system and represent atomic
software domains. Intentional elements identified in Activity 3 are analyzed and
semantically grouped. Each aggrupation reveals the services that are expected to be
covered by SA.
During the course, students are required to perform several activities with the support
of the AI, namely, to describe a fictional company, define is organizational structure and
Porter´s models of five forces and value chain, identify CA and OA base on them and to
interview the AI in the role of representatives of OA, to identify interactions (intentional
elements) with CAs and also with other OAs. At the end, students are required to compose
Use Cases to describe the behavior of intentional elements in the i* SR model (seen as
scenarios). Finally, UML Use Case models are built to show the relation among them.
In order to evaluate the use of ChatGPT in the course, a survey was designed. The
survey encompassed seven questions aimed to assess students’ perceptions, advantages,
disadvantages, lessons learned, problems, limitations, and suggestions regarding the use
of ChatGPT in the course. We obtained 27 responses from students in the course, which
was taught during the first semester of 2023. The questions were answered using different
formats, such as multiple choice, Likert scale, and open-ended.
Fig. 1. The DHARMA Method

4 Analysis of the Results

The statistical analysis of the survey results is presented in this section. Invitations to
answer the questionnaire were sent to all students of the two parallels of the course. A
total of 27 students participated and sent back their answers. The demographic data of
the participants are shown in Fig. 2.
Fig. 2. Demographics of participants: ages (X-axis) and genders.
Question 1: What was your general perception about using ChatGPT as a support tool
in the Requirements Engineering course?
This question was answered using a five-level Likert scale, where 1 means very
negative, 2 means negative, 3 means neutral, 4 means positive, and 5 means very positive.
The frequency and percentage of each option are shown in Fig. 3. The results indicate
that most students had a positive or very positive perception of using ChatGPT in the
course (88.8%), with no negative responses.
Question 2: What do you consider were the main advantages of using ChatGPT in the
course?
This question was answered using a multiple-choice format, where the students could
select more than one option from a list of six possible advantages. The frequency and
percentage of each option are shown in Table 1. The results indicate that almost all of the
students (96.3%) agreed that “ChatGPT facilitated the generation of the simulation of
a fictitious company”, which was the main advantage of using the tool in the course. In
addition, “Allowing to perform simulated interviews with the AI”, was another advantage
pointed out by almost three-fourths of the students (70.4%). Other advantages that were
selected by half or more of the students were: “Helping create the organizational chart”
and “the model of Porter’s five forces”.
Fig. 3. Distribution of responses regarding the perception of using ChatGPT in the course.
Table 1. Frequency and percentage of each option in question 2.
Option Frequency Percentage

It facilitated the generation of the simulation of a fictitious company 26 96.3%
It helped in the creation of the organizational chart of the fictitious 15 55.6%
company
It facilitated the generation of the value chain of the fictitious 12 44.4%
company
It helped in the creation of the model of Porter’s five forces for the 12 44.4%
fictitious company
It allowed to perform simulated interviews with the AI in the role of 19 70.4%
each actor in the value chain
Question 3: Did you identify any disadvantages or limitations in using ChatGPT as a

support tool?
This question was also answered using a multiple-choice format. The frequency and
percentage of each option are shown in Table 2. The results indicate that the most common
disadvantages or limitations identified by the students were “Limitations in understand-
ing and answering specific questions precisely”, with 17 participants (63.0%), followed
by “Difficulty in adequately contextualizing the information generated by ChatGPT”,
with 14 participants (51.9%). Four students did not identify any disadvantages or lim-
itations. In turn, five participants wrote additional responses, such as “Sometimes it
forgot what we were talking about”, “Sometimes it forgot about the type of company
and changed the actors”, “Too vague answers”, or “It cannot generate graphs”.
Question 4: What lessons or relevant knowledge did you obtain in the use of ChatGPT?
This question was answered using an open-ended format, where the students could
write their own responses. The lessons learned can be grouped into three main themes:
Firstly, the importance of pre-training ChatGPT and providing it with contextual
knowledge is highlighted to obtain accurate and contextualized responses. Furthermore,
the need to properly apply ChatGPT is emphasized, including asking specific questions,

Limitations in understanding and answering specific questions 17 63.0%
precisely
Difficulty in adequately contextualizing the information generated by 14 51.9%
ChatGPT
Not identify any disadvantages or limitations 4 14.8%
Other: 5 18.5%
• Sometimes the chat would forget that we were talking about a
company and generate other responses and had to remind it of the
main topic
• Limitation of knowledge: ChatGPT3 only has knowledge up to
2021
• Initially, it was not possible to share the same chat with two people
(we had to share the same email to access the fictitious company’s
chat)
• Sometimes it forgets about the type of company and changes the
actors
• Responses are too ambiguous
• Problems with generating coherent and logical responses
• It cannot generate graphs
repeating previous responses, and verifying the quality of the obtained answers. Lastly,
the benefits of using ChatGPT in different contexts, such as interview simulations, creat-
ing fictional companies, and problem-solving support, are acknowledged. By following
these lessons, one can fully harness the potential of ChatGPT in the REC.
Question 5: According to your experience, what are the main benefits of using ChatGPT
in a requirement engineering course?
The responses were given as multiple choices, with six possible options and one for
others. The results showed that the most selected option was “Better ability to simulate
a fictitious company and its components”, with 22 participants (81.5%), followed by
“A greater understanding of the concepts related to requirements engineering”, with
19 participants (70.4%). The least selected option was “Simulation of documents and
contents”, with 5 participants (18.5%). No participant wrote an additional response. The
frequency and percentage of each option are shown in Table 3.
Question 6: Did you face any problems or difficulties when using ChatGPT in the
course?
The responses were given as multiple choices, with four possible options and one
option for others. The results showed that the most selected option was “Limitations in
generating desired results”, with 16 participants (59.3%), followed by “Difficulties in
obtaining clear and precise answers”, with 15 participants (55.6%). The least selected
option was “No problem or difficulty faced”, with 3 participants (11.1%). Two partici-
pants wrote additional responses: “Sometimes it did not generate answers and I had to
refresh the page” and “It does not generate graphs”. The frequency and percentage of
each option are shown in Table 4.
Question 7: Do you have any suggestions or recommendations for improving the use
of ChatGPT in the course?
Responses were given as multiple choices, with four possible options and one option
for others. Results show that the most selected option was “Expand the knowledge
base of ChatGPT to address a greater variety of topics”, with 19 participants (70.4%),
followed by “Provide a greater capacity for customization and adjustment of ChatGPT”,
with 13 participants (48.1%). The least selected option was “No limitation or aspect
to improve identified”, with 1 participant (3.7%). No participant wrote an additional
response. Frequency and percentages of options are shown in Table 5.

A greater understanding of the concepts related to requirements 19 70.4%
engineering
Better ability to simulate a fictitious company and its components 22 81.5%
Acquisition of skills in the practical application of models and 9 33.3%
tools used
Ease to simulate interviews with managers of areas 11 40.7%
Detailed answers at an appropriate level 6 22.2%
Simulation of documents and contents 5 18.5%

Difficulties in obtaining clear and precise answers 15 55.6%
Problems maintaining a coherent and logical conversation with 7 25.9%
ChatGPT
Limitations in generating desired results 16 59.3%
No problem or difficulty faced 3 11.1%
Other: 2 7.41%
• Sometimes it did not generate answers and I had to refresh the
page
• It does not generate graphs
5 Discussion
The study conducted at Universidad del Azuay in Ecuador evaluated the use of ChatGPT
as a support tool in a university-level engineering requirements course. The research
aimed to provide insights into the practical application of AI in the specific context
of engineering requirements. The study also sought to identify the advantages, disad-
vantages, limitations, and areas for improvement in using ChatGPT in this educational
setting.

Improve the accuracy and consistency of the responses generated 10 37.0%
Expand the knowledge base of ChatGPT to address a greater variety 19 70.4%
of topics
Provide a greater capacity for customization and adjustment of 13 48.1%
ChatGPT
No limitation or aspect to improve identified 1 3.7%
The results of the study were largely positive. Most students had a positive or very
positive perception of using ChatGPT in the course, with no negative responses. The
main advantage identified by the students was that ChatGPT facilitated the generation
of the simulation of a fictitious company. This was seen as a significant benefit, as it
allowed students to gain practical experience without the challenges of engaging with
real companies.
However, the study also identified some limitations and areas for improvement. Some
students noted that ChatGPT sometimes had difficulty providing precise and contextu-
alized responses to specific questions. Additionally, there were instances where the AI
would forget the context of the conversation and generate unrelated responses. The main
themes that emerged from the analysis of the responses are:
• The need for more training and guidance on using ChatGPT effectively, including
best practices, tips, and examples.
• The importance of providing more context and background information to ChatGPT
before asking questions or generating information related to the fictitious company
and the requirements engineering concepts.
• The suggestion to use ChatGPT in combination with other tools or resources, such as
textbooks, articles, videos, etc., to complement and verify the information generated
by ChatGPT.
• The recommendation to update ChatGPT’s knowledge base regularly to include more
recent information and developments in the field of requirements engineering.
• The possibility of integrating ChatGPT with other tools or platforms used in the
course, such as learning management systems, collaboration tools, etc., to facilitate
its use and improve its effectiveness.
In light of these findings, the following recommendations can be made to improve

future courses:
1. Enhance AI Contextual Understanding: Future iterations of the course could include
more structured guidelines on how to interact with the AI. For instance, students
could be taught to provide more context in their questions or to use specific keywords
that the AI is more likely to understand. For example, instead of asking “What is
this?”, students could be encouraged to ask “What is this concept in the context of
software engineering?”. This would help the AI provide more accurate and relevant
responses. On the AI development side, we expect that improvements will be made to
the AI’s ability to understand and maintain context throughout a conversation. This
could involve refining the AI’s algorithms or training the AI on more diverse and
context-rich datasets.
2. Improve Result Generation: This could be addressed by providing more specific
instructions or parameters to the AI. For example, if students are looking for a specific
type of information, they could be taught to request this more effectively from the AI.
For instance, if a student wants to know about usability requirements for a mobile app
for a specific type of users, they could ask “Can you provide the usability requirements
for a mobile app to be used by special or different abilities people?”, instead of a
more general question like “Tell me about the usability requirements for a mobile
app”, This would help the AI generate more relevant and useful responses.
3. Address Contextual Forgetting: This could be mitigated by incorporating reminders
or cues into the conversation to help the AI maintain context. For example, students
could be taught to refer back to the main topic of the conversation regularly or to
reiterate key points. On the AI development side, improvements could be made to the
AI’s ability to remember and refer back to previous parts of the conversation. This
could involve enhancing the AI’s memory capabilities or refining its algorithms to
better handle the context.
4. Enhance Graphical Capabilities: If the graphical representation of data is an impor-
tant part of the course, alternative tools or methods could be introduced to supplement
the AI’s capabilities in this area. For example, students could be taught to use data
visualization tools like Tableau or Power BI to create graphs and charts. Alterna-
tively, the AI could be integrated with a tool that can generate graphs based on the
data provided. However, when it comes to the use of requirements modeling frame-
works, some attention has to be taken, since seamless integration with goal or object
based graphical modeling tools may introduce ethical risks due to students’ lack of
dedication to more manual modeling tasks.
5. Provide More Guidance: This could be addressed by providing more guidance on
how to ask specific and detailed questions to get more precise responses. For example,
students could be taught to break down complex questions into simpler, more specific
questions. This would make it easier for the AI to understand and answer each part
of the question.
6. Develop guidelines for specific activities: Based on the lessons learned in this study,
it is recommended to propose comprehensive guidelines to teach students to collect,
specify and validate software requirements with the support of AI-based tools (e.g.,
ChatGPT), simulating scenarios very close to reality. In this specific course, guidelines
may include procedures to: i) elaborate the mission, vision, policies, and strategic
objectives of a fictitious company; i) identify the main stakeholders in the market
(Porter’s competitive forces) and strategic partners to deliver its value proposition; ii)
develop its functional organic structure; iii) design structured interviews with chatbot
representatives from various roles within the company; and, iv) elicit, document and
validate business and system requirements with the support of high-level models
(context, goals, use cases).
After this experience, we are convinced that integrating AI tools like ChatGPT
in engineering education has shown promising results. However, it is crucial to con-
tinue exploring strategies to enhance their integration while mitigating associated risks.
This includes fostering critical thinking and academic integrity among students while
harnessing the creative and collaborative potential of these tools.
6 Threats to Validity
When interpreting the results of the study, it is essential to consider several threats to
validity that may impact the reliability and generalizability of the findings. One imme-
diate threat to validity is related to the students’ prior knowledge and experience with
ChatGPT. Since students did not receive formal training on using this tool, they had to
learn alongside the tasks required to complete the course. While more formal training
might have yielded more relevant results and improved task efficiency, this initial expe-
rience also aimed at exploring students’ ability to investigate and learn about new tool
paradigms.
However, despite the lack of formal training, the results of the course exceeded
expectations in terms of the level of detail and quality of deliverables produced by
students, surpassing previous editions of the course where students interacted with real
companies. This improvement may be attributed to the limited time and information
provided by real companies and their employees compared to the support and detailed
information available through ChatGPT.
It is important to acknowledge that the survey conducted in the study only included
27 students who used ChatGPT in a specific REC. Consequently, the generalizability of
the study’s results to other courses and educational contexts may be limited. However,
it is important to note that the students who took part in the survey belonged to two
distinct groups, both simultaneously attending the same subject during the semester but
with different schedules. Both groups included 18 students; 17 from the first group and
10 from the second answered the survey. While this may enhance the reliability of the
results to some extent, we acknowledge that it is required to conduct further studies with
additional cohorts to validate and extend the promising findings observed in this study.
7 Concluding Remarks
By incorporating ChatGPT as a tool, students were able to overcome the challenges of

engaging with real companies and their employees. This approach provided a simulated
but realistic experience, allowing students to develop their understanding of requirements
engineering concepts and enhance their practical application skills. The use of ChatGPT
facilitated the creation of a fictional company, analysis based on industry models, and
interviews with simulated employees, enabling students to construct comprehensive
models that accurately depicted the organization’s context, functional decomposition,
and descriptions of business and system use cases. This case study demonstrates the suc-
cessful integration of ChatGPT as a valuable support tool in the requirements engineering
education process at Universidad del Azuay.
However, the study also highlights some limitations of using AI in this context,
including difficulties in understanding and answering specific questions, generating
desired results, maintaining context, and producing graphical outputs. These limitations
present opportunities for both pedagogical and technological improvements.
The recommendations provided aim to address these limitations. However, further
studies are required in order to make them more comprehensive for students.
The integration of AI tools in higher education, particularly in engineering courses,
is a promising avenue that can significantly enhance the learning experience. However, it
is crucial to continuously evaluate and refine these integrations to ensure they effectively
meet the learning objectives and cater to the needs of the students. The insights from
this study contribute to the broader discussions on the effective integration of AI in
higher education, with implications for pedagogy, student engagement, and the future
of engineering education.
As future work, it is proposed to develop a guide to accompany the project-based
learning process in requirements engineering with the support of AI tools. This guide
will be used in future RECs, together with the incorporation of other AI tools.
References
1. Sommerville, I., Sawyer P.: Requirements Engineering: A Good Practice Guide. John Wiley &
Sons (1997)
2. Kotonya G., Sommerville I.: Requirements Engineering: Processes and Techniques. John
Wiley & Sons (1998)
3. Nuseibeh, B., Easterbrook, S.: Requirements engineering: a roadmap. In: Proceedings of the
Conference on the Future of Software Engineering, pp. 35–46 (2000)
4. Blumenfeld, P.C., Soloway, E., Marx, R.W., Krajcik, J.S., Guzdial, M., Palincsar, A.: Moti-
vating project-based learning: sustaining the doing, supporting the learning. Educ. Psychol.
26(3–4), 369–398 (1991)
5. Thomas, J.W.: A Review of Research on Project-Based Learning. Autodesk Foundation, San
Rafael (2000)
6. Krajcik, J.S., Blumenfeld, P.C., Marx, R.W., Bass, K.M., Fredricks, J., Soloway, E.: Inquiry
in project-based science classrooms: initial attempts by middle school students. J. Learn. Sci.
7(3–4), 313–350 (1998)
7. Hmelo-Silver, C.E.: Problem-based learning: what and how do students learn? Educ. Psychol.
Rev. 16(3), 235–266 (2004)
8. Russell, S., Norvig, P.: Artificial Intelligence: A Modern Approach, 4th edn. Pearson (2020)
9. Luckin, R., Holmes, W., Griffiths, M., Forcier, L.B.: Intelligence Unleashed: An Argument
for AI in Education. Pearson, London (2016)
10. Nkambou, R., Bourdeau, J., Mizoguchi, R. (eds.): Advances in Intelligent Tutoring Systems.
Springer Berlin Heidelberg, Berlin, Heidelberg (2010)
11. Tom, B., et al.: Language models are few-shot learners. Adv. Neural. Inf. Process. Syst. 33,
1877–1901 (2020)
12. Roumeliotis, K.I., Tselikas, N.D.: ChatGPT and open-AI models: a preliminary review. Future
Internet 15, 192 (2023). https://doi.org/10.3390/fi15060192
13. Jurafsky, D., Martin, J.H.: Speech and Language Processing: An Introduction to Natural
Language Processing, Computational Linguistics, and Speech Recognition. Pearson/Prentice
Hall Upper Saddle River (2009)
14. McTear, M., Callejas, Z., Griol, D.: The Conversational Interface. Springer International
Publishing, Cham (2016). https://doi.org/10.1007/978-3-319-32967-3
15. Johnson, W.L., Valente, A.: Tactical language and culture training systems: Using artificial
intelligence to teach foreign languages and cultures. In: Proceedings of the 20th national
conference on Innovative applications of artificial intelligence (IAAI’08), vol. 2, pp. 1632–
1639 (2008)
16. Crompton, H., Burke, D.: Artificial intelligence in higher education: the state of the field. Int.
J. Educ. Technol. High Educ. 20(1), 22 (2023). https://doi.org/10.1186/s41239-023-00392-8
17. Hulick, K.: How ChatGPT and similar AI will disrupt education. Science News. https://
www.sciencenews.org/article/chatgpt-ai-artificial-intelligence-education-cheating-accuracy
(2023)
18. UCLA Center for the Advancement of Teaching: Guidance for the Use of Generative AI.
https://teaching.ucla.edu/resources/ai_guidance/ (2023)
19. Han, J., Yoo, H., Kim, Y., Oh, A.: RECIPE: How to Integrate ChatGPT into EFL Writing
Education. https://arxiv.org/abs/2105.09938 (2023)
20. Yeh, S.-Y., et al.: A study of EFL students’ participation patterns and learning performance
in an English chatbot system. Comput. Educ. 151, 103862 (2020)
21. Bibauw, S., François, T., Desmet, P.: Dialogue-based CALL: a multilevel meta-analysis. Lang.
Learn. Technol. 24(1), 110–139 (2020)
22. Goldberg, D.E.: The design of innovation: Lessons from and for competent genetic algorithms.
Springer Science & Business Media (2012)
23. Qadir, J.: Engineering Education in the Era of ChatGPT: Promise and Pitfalls of Generative
AI for Education. https://www.researchgate.net/publication/366712815_Engineering_Edu
cation_in_the_Era_of_ChatGPT_Promise_and_Pitfalls_of_Generative_AI_for_Education
(2022)
24. Nikolic, S., et al.: ChatGPT versus engineering education assessment: a multidisciplinary and
multi-institutional benchmarking and analysis of this generative artificial intelligence tool to
investigate assessment integrity. Eur. J. Eng. Educ. 48(4), 559–614 (2023). https://doi.org/10.
1080/03043797.2023.2213169
25. Pohl, K.: Requirements Engineering. Springer-Verlag, Berlin Heidelberg (2010)
26. Carvallo, J.P., Franch, X.: On the use of i* for architecting hybrid systems: a method and
an evaluation report. In: Persson, A., Stirna, J. (eds.) The Practice of Enterprise Modeling,
pp. 38–53. Springer Berlin Heidelberg, Berlin, Heidelberg (2009). https://doi.org/10.1007/
978-3-642-05352-8_5
27. Porter, M.: Competitive Strategy. Free Press. New York, NY, USA (1980)
Pedagogical Support Through
the Implementation of a Virtual Assistant
to Personalize Education
Victor Farinango1(B) and Santiago Castro Arias2

1 Universidad Politécnica Salesiana, Quito, Ecuador
vfarinangoc@est.ups.edu.ec
2 Universidad Politécnica Salesiana, Cuenca, Ecuador
scastroa1@ups.edu.ec
Abstract. Virtual assistants in education have become popular quickly due to

their advantages, such as the ability to answer students’ questions 24 h a day,
seven days a week, improving education’s personalization. In this study, a virtual
assistant trained in Dialogflow and executed on the Facebook Messenger platform
was developed to deliver information and resolve student queries about grades,
educational materials, homework, and other related topics, personalizing educa-
tion. A survey was conducted with students to evaluate their perception of the use
of the assistant, and the results obtained show that the assistant had a high degree
of acceptance (100%) in the ninth and tenth year of high school for the science
subject. 98.7% of the students had some degree of satisfaction in interacting with
Ciencia Bot Kitu Kara; they used it frequently and considered that the assistant
helped them in their learning process, which ratifies that its use was frequent. On
the other hand, most students would recommend using the assistant in the applied
subject and suggest its implementation in different subjects. These results indi-
cate that virtual assistants, such as Ciencia Bot Kitu Kara, provide added value
to the personalization of education, in which students receive support in different
educational areas.
Keywords: Virtual assistant · Personalization of education · Learning Analytics
1 Introduction
The importance of using artificial intelligence based tools to support the personalization
of education to optimize educational processes is becoming increasingly evident. Virtual
assistants offer many benefits, such as providing feedback, setting goals, personalized
study plans and materials, recommending extracurricular activities, and monitoring stu-
dents’ progress. In addition, these assistants can identify and solve problems or chal-
lenges that may result in educational processes. Therefore, using artificial intelligence
tools such as virtual assistants is becoming a necessary element that needs to be repli-
cated, as stated by [1] in his study, in which he concludes that artificial intelligence has
great potential to address problems by providing constructivist activities.
https://doi.org/10.1007/978-981-99-7353-8_26
344 V. Farinango and S. Castro Arias
There is a growing need to improve pedagogical support and personalize education

in educational institutions. In particular, the Kitu Kara Educational Unit, which belongs
to the intercultural bilingual system, has a curriculum different from the national one
and promotes the preservation of the Kichwa language and the knowledge of Ecuado-
rian peoples; a relevant problem has been identified in the natural sciences Since the
application of technology in the classroom has been scarce, students lack the necessary
educational materials for learning, little interaction between students and teachers, low
pedagogical support to students, and no type of personalization of education. Therefore,
[2] indicates that traditional learning should be rethought, and new scenarios, including
collaborative, participatory, and interactive virtual environments, should be built.
Pedagogical support can be defined as a set of services and systems that advise
teachers to improve the quality of student learning and the work of the teaching staff and
educational management (Brigg, cited by Oblitas, 2017). In addition, personalization
considers the student’s participation, allowing them to decide on their responsibility for
their learning [4].
According to [1], Information and Communication Technologies (ICT) have two
essential characteristics: the elimination of time barriers and the promotion of indepen-
dent learning, both individually and in groups, and the use of multimedia materials can
be very beneficial to achieve educational objectives. In this same sense, [5] in his study
determined that 80% of teachers agree that ICTs should be used to improve the learning
of adults and young people. On the other hand, personalized learning has become a vital
learning paradigm for educational technology researchers [6].
According to [7], technological mediation introduces diversified pedagogical
resources for students and teachers, leading to greater direct participation, structuring of
processes, and implementation of follow up and feedback mechanisms. This technolog-
ical mediation allows the introduction of new pedagogical tools, diversifies the activities
of students and teachers, actively participates in educational processes, structures col-
laborative processes, and establishes follow up and feedback mechanisms to achieve
educational objectives. In this same sense, [8] mentions that at least 74.8% of students
in a school use digital technologies at least once a week.
The purpose of this study was to improve the quality of student learning by apply-
ing the personalization of education through which students could consult a variety of
resources such as videos, texts, images, questionnaires, study guides, infographics, inter-
active online games, relevant information about grades, homework, including guidelines
and due dates of assignments, by the virtual assistant created and called Kitu Kara Bot
Science, which was integrated into the Facebook Messenger platform and was devel-
oped on the Dialogflow platform; as also pointed out by [9] in his study, which indicates
that better performance can be obtained if an educational intervention based on learning
styles is adopted.
Students could select the type of educational material that best suited their needs,
promoting more efficient and personalized learning. All these functionalities allowed a
more personalized and accessible learning experience for students. This same line of
study [10] allowed students to select content and resources and add to a collection of a
topic of interest; similarly [11], a survey of personalization will enable students to choose
Pedagogical Support Through the Implementation 345
their favorite learning materials. On the other hand, [12] relied on artificial intelligence
to make recommendation systems and mentions that they are handy tools for learning.
This study will contribute to broadening research in the educational field by address-
ing the problem of poor interaction between teachers and students to improve the flow
of scholarly content through a virtual assistant. The aim is to innovate how to reach stu-
dents and enhance their learning process. In addition, the student’s perception of using
this type of virtual assistant will be evaluated to identify their potential implementation
in other subjects or educational institutions. Other researchers, such as [13], conducted
their study of the perception of learning on a standard course and a personalized online
course, in which they obtained results showing that personalization positively affects
student autonomy and favors their feeling of competence.
2 State of the Art

Virtual assistants, also known as bots, employ artificial intelligence and natural language
processing to maintain conversations with people and can be adapted to various digital
channels [14]. Moreover, using these assistants on the Facebook Messenger platform
does not require installing a particular application, and the advantage is that the interface
is very familiar and uses an existing mobile structure [15].
Virtual assistants are increasingly used in education to improve student’s experience
in their learning activities [16]. Since each learner has different needs and learning
styles, personalization of learning is critical to achieving better academic performance.
However, not all educational environments can provide specialized tutors for each student
[17]. In this sense, virtual assistants can effectively offer personalized attention to each
student and improve their learning process.
According to [18], the personalization approach in education aims to provide learners
with educational materials tailored to their profiles and allow them to choose the option
that best suits their needs. According to the researchers, personalization implies that
learners can choose rather than being given a predefined learning or content list. In
this type of personalization, teachers must provide students with items they can select at
their discretion. The virtual assistant integration, in this case, the virtual assistant Ciencia
Bot Kitu Kara, works as a tool to decrease teacher overwork and make courses more
accessible to students.
Nowadays, personalization has come to exceed their expectations because it allows
sharing content through platforms, augmenting educational content, and with the inclu-
sion of virtual reality and artificial intelligence, they mention making learning more
engaging, interactive, and not least motivating for students [19]. In addition, the per-
sonalization of learning involves revising, reorienting, and reconstructing a unified cur-
riculum. Personalization can help overcome signed students’ learning difficulties and
improve the teaching learning process’s productivity (Yonezawa et al., 2012, cited by
Bhutoria, 2022).
On the other hand, virtual assistants can provide personalized guidance to each
learner according to their individual needs is a solution to address the lack of communi-
cation, as in the case of the use of online learning systems [20]. In the study conducted
by [21], they describe their experience in personalizing learning through the MOODLE
platform, taking into account three variables: the level of knowledge, learning styles,
and the goals and interests of the students. In their conclusions, they determined that the
elements selected in the repository can be adapted to the student’s tastes and can also be
offered guides according to different methods and objectives. According to the authors,
this customization would allow the user to have at their disposal all the materials of a
training class of their liking.
In the United States of America, specifically at the University of Arizona, a study
was conducted that used personalization in education and developed a computer lab.
This lab collects information from student interaction and can predict their performance
in the following semester based on the data collected. In addition, it provides different
lab materials for each student in future labs and sends feedback to the instructor when
it detects at risk students who need help [22].
In a study conducted by [23], online learning personalization was used, in which
a personalized study plan and content were provided according to each individual’s
preferences. The result of this study indicates that students who received online learning
personalization were more successful in the academic part, contrasting with those who
did not have personalization. The authors of this study define that the quality of online
learning improves with personalization. In addition, they evaluated student satisfaction
and found that student satisfaction increased if the learning style was effective.
According to [16], there are multiple uses in different sectors, such as banking, edu-
cation, health, engineering, business, psychology, and astronomy. Of these, the industry
with the most significant progress is health. However, in recent years, in education, there
has been a substantial increase in its application due, among other things, to the advance-
ment of assistants by certain technology companies and also due to the increased use of
mobile applications and messaging. Although natural language processing is still in its
infancy, the help of a virtual assistant in education is promising [24].
The use of virtual assistants in education is very diverse and ranges from primary
education to university, with several prototypes having been developed that provide
pedagogical support in science teaching and language learning [14]. In this sense, the
evolution of educational technology calls for a more personalized learning experience
that can adapt to individual learning patterns and meet the diverse needs of each student
in both online interactions and face to face environments [24].
According to [14], studies indicate the usefulness of virtual assistants in education
and students’ preference to use them. In addition, it is mentioned that secondary school
students are an appropriate population to use these agents since they are familiar with
using different applications.
It is essential to highlight that, in educational research, more attention should be paid
to students’ understanding of their learning. It is necessary to know their perspective on
the use of new technologies. For technology to contribute to bringing education and
society into a new era of learning, students must also assume a proactive attitude toward
technology [25].
On the other hand, a bibliographic analysis was performed using the Vosviewer soft-
ware, taking the information from the Scopus data, in which the most recent documents
were analyzed in order. The keywords "chatbot," "conversational agent," and "education"
were used for the search.
Fig. 1. Countries with the highest incidence of virtual assistant use in the study.
Figure 1 shows that the countries with the highest incidence and most studies and
influence in the study of the use of conversational agents in education are: the United
States in the first place, followed by India in second place, Germany in third place, and
the United Kingdom in fourth place, followed by China, Italy, Australia, Canada, Spain,
Switzerland, and Brazil.
3 Methodology
The research was conducted in a bilingual secondary school in Ecuador, Quito, and the
parish of Calderón. The learning model in this educational unit is based on a traditional
approach, where teachers present their classes without technology or virtual assistants.
The activities and pedagogical tools used are the same for all students, without consid-
ering the personalization of learning. In addition, students do not have sufficient educa-
tional resources, which makes their academic process even more difficult. Consequently,
students require tutoring to improve their academic performance.
With the above highlights, a solution was implemented to address the problem by
creating a virtual assistant called Kitu Kara Bot Science, which was implemented in
Facebook Messenger using Dialogflow. Before its implementation, its operation and the
benefits of its use were explained to the students. Subsequently, the students used the
virtual assistant, and a survey was conducted to evaluate their acceptance and obtain
feedback on the experience of using the virtual assistant.
During the fourth quarter of the 2021-2022 academic year, students had the oppor-
tunity to interact with the virtual assistant to request information about their grades,
assignments, guides, and educational material, such as videos, quizzes, study guides,
texts, and infographics. Most conversations with the virtual assistant were successful
and received satisfactory responses.
Finally, a descriptive analysis was conducted to evaluate the students’ experience

using the virtual assistant in the natural sciences subject. This analysis was performed
with a probabilistic sample applied to students in the ninth and tenth years, equivalent
to the first and second years of high school.
The total population of the educational unit was 300 students, and we worked with a
significant sample of 82 students, with a confidence level of 95% and a margin of error
of 3%. In addition, the model considered the participation of males and females, with a
distribution of 51.2% and 48.8%, respectively, to ensure equal participation.
3.1 Surrey
The following are the questions asked to the students.

1. Select the course and parallel you are in.
2. What gender do you belong to?
3. Before using the Kitu Kara Science Bot virtual assistant, did you know what chatbots
or virtual assistants are and how they work?
4. How often do you use the Kitu Kara Bot Science virtual assistant?
5. Do you consider the Kitu Kara Bot Science virtual assistant providing support during
your learning process?
6. In what areas did the Kitu Kara Science Bot virtual assistant support you in your
learning process?
7. Do you consider that the use and interaction with the Kitu Kara Science Bot virtual
assistant was friendly?
8. What aspects of the Kitu Kara Science Bot virtual assistant could be improved?
9. What would you like the Kitu Kara Science Bot virtual assistant to be able to respond?
10. Would you be satisfied interacting with the Kitu Kara Science Bot virtual assistant?
11. What other platform would you like the Kitu Kara Science Bot virtual assistant to
work on?
12. Would you recommend using the virtual assistant Science Bot Kitu Kara for the
next year to natural sciences students?
13. From your experience with the use of the virtual assistant in the natural science
subject, would you recommend that Science Bot Kitu Kara should be able to support
more assignments?
4 Results
Results are presented on the perception of students regarding the use of virtual assistants.
Students of both genders participated equally in the survey, representing 51.2% of men
and 48.8% of women. It was exciting to inquire about the student’s knowledge of the
functioning of virtual assistants, and based on the answers obtained, it was determined
that 37.8% of the students claimed to know about the use of virtual assistants or chat-
bots. In comparison, the remaining 62.2% did not. These data show students’ reduced
technological and internet access in rural Ecuador.
In addition, we inquired about the frequency of use of the virtual assistant, and as
shown in Figure 2, it could be established that the students used it frequently since 98.8%
of the respondents stated that they used it at least once a week. These results indicate
the students’ high acceptance of the virtual assistant. To further analyze the frequency
of use of the chatbot, the frequencies obtained by analyzing sessions and interactions
for all requests on the Dialogflow platform were examined in the analytics section. This
analysis confirmed that the chatbot was widely used, further reaffirming students’ high
use of the virtual assistant.
Fig. 2. Kitu Kara chatbot sessions and interactions
The survey results indicate that the respondents positively valued the virtual assistant
since 100% of the students stated that the Kitu Kara chatbot supported them regularly.
Notably, no student reported not receiving help from the virtual assistant, indicating that
the chatbot has fulfilled its purpose of giving attention and support to students. The study
by [26] suggests that if the teacher allows students to determine the content, procedures,
and learning processes using technology, they may feel better supported individually.
Consequently, their motivation for learning may be enhanced.
Figure 3 describes the foundations of virtual assistance created in Dialogflow.
Through Facebook Messenger platform, interaction input training involving educational
material such as grades and homework was supplied. Students can request their educa-
tion materials based on unique criteria which includes the following: infographics, study
guides, quizzes, videos, texts, interactive games, grades and homework.
Figure 4 shows the students’ perception of the areas in which the virtual assistant
has supported them, with the first place going to the organization of tasks, followed by
studying for exams and obtaining educational material; in addition, it is necessary to
indicate that no student suggests that the virtual assistant has not supported them.
Based on Figure 5, it was determined that 96.3% indicated that the use of the vir-
tual assistant was friendly. Of this value, 52.4% considered that they agreed that the
interaction was close, 43.9% mentioned that they agreed, on the other hand, 3.7% were
indifferent, and 0% disagreed. In this sense, it can be defined that the virtual assistant
was friendly, as shown by the survey and the Dialogflow analysis section in the following
figures.
Fig. 3. Virtual Assistant Design
Fig. 4. Areas in which the virtual assistant supports students in the learning process.
We also inquired about possible improvements that could be implemented in the

virtual assistant. According to the student’s responses, Fig. 6 identified the following
areas for improvement in the order of priority, from highest to lowest: delivery of grades,
explanations of assignments, delivery of study guides, questionnaires, and delivery of
educational resources. These results provide valuable information to improve the virtual
assistant further and adapt it to the needs of the students.
Fig. 5. Students’ perception of the virtual assistant’s friendliness.
Fig. 6. Aspects of the virtual assistant that can be improved.
We wanted other information requirements students would like the virtual assistant
Ciencia Bot Kitu Kara to answer. As shown in Figure 7, the most mentioned by the
students are final averages, no information, information about supplements, information
about changes in the institution, homework advice, and schedules; these are among the
most relevant.
We asked about the students’ experience using the virtual assistant in the natural
science subject to determine if they would recommend that Science Bot Kitu Kara
provide support in other matters. The results indicate that 96.1% of the students would
recommend using Science Bot Kitu Kara in other subjects, which suggests that it was a
satisfactory experience for the vast majority since only 4.9% indicated that they would
not recommend it. These findings suggest that students are happy with the tool.
It is worth noting that it was exciting to know the subjects in which students would
like to have a chatbot. The results indicate that the most requested topics are Mathematics,
Kichwa, English, Language and Literature, Social Sciences, Arts, Computers, etc., as
shown in Figure 8.
Fig. 7. Information requirements that the student needs from the virtual assistant.
Fig. 8. Other subjects that students would like Ciencia Bot Kitu Kara to support.
Figure 9 shows an analysis of the interaction between the questions on frequency of

use and the areas in which the virtual assistant is used, from which it can be determined
that students who sometimes used the virtual assistant limited its service to the organiza-
tion and delivery of assignments; on the other hand, those who used it almost constantly
and continually diversified its use and requested more resources than the former.
Fig. 9. Frequency of use of the virtual assistant and the areas where I support them.
These results are essential to continue to improve the virtual assistant and adapt it to
the needs of the students.
5 Conclusions
The evaluation carried out by the students at the end of the application of the virtual
assistant Ciencia Bot Kitu Kara determined that there was a high degree of acceptance
(100%) in the use of the virtual assistant for the subject of Natural Sciences in the ninth
and tenth levels of primary, secondary school.
This high degree of acceptance is because 98.8% of the students had a certain degree
of satisfaction in the interaction with Ciencia Bot Kitu Kara, 98.8% used it frequently (at
least once a week), 100% of the students consider that the virtual assistant helped them
in the learning process in the Natural Sciences subject, Last but not least, 98.8% of the
students would recommend the use of the virtual assistant in the Natural Sciences subject
in the following years and 95.1% would recommend its application in other matters.
The interaction of the students with Ciencia Bot Kitu Kara was successful since 100%
of the student’s questions were answered, given that it has been possible to personalize
the learning process by responding to each student’s specific concerns. The operation
and use of the virtual assistant were successful for all the reasons mentioned above; all
the students used it, stated that it was easy to use, resolved the conversations, fulfilled
the work required by delivering material requested by the students, and helped all the
students. During the surveys, the students mentioned their satisfaction (98.8%) with
the use of the virtual assistant, requested that it be replicated again in future years
(98.8%), and indicated that it is an application that helped them to improve their learning
experience (100%).
According to the bibliographic review, the analysis of the state of the art indicates
that there needs to be more information related to pedagogical support with virtual assis-
tants for the personalization of education because they are technologies that have been
emerging recently. In addition, in most studies, it is highlighted that personalization was
performed by models that determine what students require for their learning; however,
certain studies allow students to select the type of information they need for their knowl-
edge. In this field, there has yet to be a study on improving performance with virtual
assistants implementing the personalization of education.
Overall, it can be concluded that virtual assistants can be a valuable tool to enhance
student learning if technological access is taken into account and continuous work is
done to improve and adapt these tools to the needs of students. Discussion can focus on
how virtual assistants can be improved and how they can be effectively implemented in
different learning environments.
6 Recommendations
The results found in the recent study have found satisfactory findings. The purpose of
this study for the foreseen future is to amplify Chatbox, where it is able to respond to
more scientific inputs and be more adaptable to other multicultural educational institutes.
Acknowledgments. The GIIAR research group and the Universidad Politécnica Salesiana have
supported this work.
References
1. Chen, Y.L., Hsu, C.C., Lin, C.Y., Hsu, H.H.: Robot-assisted language learning: integrating
artificial intelligence and virtual reality into English tour guide practice. Educ. Sci. (Basel)
12(7) (2022). https://doi.org/10.3390/educsci12070437
2. Flores, A., García, A.: Revista cubana educación superior. Scielo 36(2), 27–40 (2017). http://
scielo.sld.cu/scielo.php?script=sci_arttext&pid=S0257-43142017000200003
3. Oblitas, E.: Soporte pedagógico y desempeño docente en la institución educativa N°0180
‘Señor de los Milagros’ del distrito de Bellavista, región San Martín - 2017. J. Chem. Inf.
Model. 53(9), 1689–1699 (2017)
4. Engel Rocamora, A., Coll Salvador, C.: Entornos híbridos de enseñanza y aprendizaje para
promover la personalización del aprendizaje. RIED. Revista Iberoamericana de Educación a
Distancia 25(1) (2021). https://doi.org/10.5944/ried.25.1.31489
5. Ramirez, A., Inga, E.: Educational innovation in adult learning considering digital transfor-
mation for social inclusion. Educ. Sci. (Basel) 12(12) (2022). https://doi.org/10.3390/educsc
i12120882
6. Xie, H., Chu, H.C., Hwang, G.J., Wang, C.C.: Trends and development in technology-
enhanced adaptive/personalized learning: a systematic review of journal publications from
2007 to 2017. Comput. Educ. 140, 103599 (2019). https://doi.org/10.1016/j.compedu.2019.
103599
7. Villalta, M., Guzman, A., Nussbaum, M.: Teaching processes and technology use in the
classroom—procesos pedagógicos y uso de tecnología en el aula. Revista Complutense de
Educacion 26, 405–424 (2015). https://doi.org/10.5209/rev_RCED.2015.v26.n2.43303
8. Schmid, R., Petko, D.: Does the use of educational technology in personalized learning envi-
ronments correlate with self-reported digital skills and beliefs of secondary-school students?
Comput. Educ. 136, 75–86 (2019). https://doi.org/10.1016/j.compedu.2019.03.006
9. Nazempour, R.: Education sciences personalized learning in virtual learning environments
using students. Behav. Anal. (1) (2023)
10. Tsybulsky, D.: Digital curation for promoting personalized learning: a study of secondary-
school science students’ learning experiences. J. Res. Technol. Educ. 52(3), 429–440 (2020).
https://doi.org/10.1080/15391523.2020.1728447
11. Peng, H., Ma, S., Spector, J.M.: Personalized adaptive learning: an emerging pedagogical app-
roach enabled by a smart learning environment. In: Lecture Notes in Educational Technology,
pp. 171–176 (2019). https://doi.org/10.1007/978-981-13-6908-7_24
12. Huang, A.Y.Q., Lu, O.H.T., Yang, S.J.H.: Effects of artificial intelligence– enabled person-
alized recommendations on learners’ learning engagement, motivation, and outcomes in a
flipped classroom. Comput. Educ. 194, 104684 (2023). https://doi.org/10.1016/j.compedu.
2022.104684
13. Alamri, H., Lowell, V., Watson, W., Watson, S.L.: Using personalized learning as an instruc-
tional approach to motivate learners in online higher education: learner self-determination
and intrinsic motivation. J. Res. Technol. Educ. 52(3), 322–352 (2020). https://doi.org/10.
1080/15391523.2020.1728449
14. Paredes, C.: Chatbots en Educación Secundaria : Retos y propuestas para su aplicación
en el aula (2021). https://uvadoc.uva.es/bitstream/handle/10324/50989/TFMG1530.pdf?seq
uence=1&isAllowed=y
15. Smutny, P., Schreiberova, P.: Chatbots for learning: a review of educational chatbots for the
Facebook Messenger. Comput. Educ. 151, 103862 (2020). https://doi.org/10.1016/j.compedu.
2020.103862
16. Auqui, J.: Chatbot del proceso de aprendizaje universitario: Una revisión sistemática. Alpha
Centaury 2(2709–4502), 16 (2021). https://doi.org/10.47422/ac.v2i2.22
17. Baha, T.A.I.T., Hajji, M.E.L., Es-Saady, Y., Fadili, H.: Towards highly adaptive Edu-Chatbot.
Procedia Comput. Sci. 198(2018), 397–403 (2021). https://doi.org/10.1016/j.procs.2021.
12.260
18. Zalavra, E., Papanikolaou, K., Dimitriadis, Y., Sgouropoulou, C.: Personalising learning:
towards a coherent learning design framework. In: Proceedings - 2022 International Confer-
ence on Advanced Learning Technologies, ICALT 2022, pp. 77–79 (2022). https://doi.org/
10.1109/ICALT55010.2022.00030
19. Bhutoria, A.: Personalized education and Artificial Intelligence in the United States, China,
and India: a systematic review using a Human-In-The-Loop model. Comput. Educ. Artif.
Intell. 3, 100068 (2022). https://doi.org/10.1016/j.caeai.2022.100068
20. Shkarlet, S.: Mathematical Modeling and Simulation of Systems (MODS 2020) (2020)
21. Lerís, D., Sein-Echaluce, M.L.: La personalización del aprendizaje: un objetivo del paradigma
educativo centrado en el aprendizaje. Arbor 187(EXTRA 3), 123–134 (2011). https://doi.org/
10.3989/arbor.2011.Extra-3n3135
22. Deng, Y., Lu, D., Chung, C.J., Huang, D., Zeng, Z.: Personalized learning in a virtual hands-
on lab platform for computer science education. In: Proceedings - Frontiers in Education
Conference, FIE, vol. 2018, pp. 1–8 (2019). https://doi.org/10.1109/FIE.2018.8659291
23. Sanjabi, T., Montazer, G.A.: Personalization of E-learning environment using the Kolb’s
learning style model. In: 2020 6th International Conference on Web Research, ICWR 2020,
pp. 89–92 (2020). https://doi.org/10.1109/ICWR49608.2020.9122314
24. Gonda, D.E., Kong, H., Chu, B., Kong, H.: Chatbot as a learning resource? Creating conversa-
tional bots as a supplement for teaching assistant training course. In: International Conference
on Engineering, Technology and Education (TALE), pp. 1–5. IEEE (2019). https://bibliotecas.
ups.edu.ec:2095/stamp/stamp.jsp?tp=&arnumber=9225974
25. Masullo, G., Addeo, F., Paoli, A.D., Ruopolo, A.: Learning with ICTS at primary level:
teachers’ and pupils’ perceptions. Ital. J. Sociol. Educ. 13(3), 21–44 (2021). https://doi.org/
10.14658/pupj-ijse-2021-3-2
26. Schmid, R., Pauli, C., Stebler, R., Reusser, K., Petko, D.: Implementation of technology-
supported personalized learning—its impact on instructional quality. J. Educ. Res. 115(3),
187–198 (2022). https://doi.org/10.1080/00220671.2022.2089086
The Use of Indicators in the Creation of Distance
Learning Undergraduate Courses
Gleice Louise Garcia Costa dos Santos(B) and Letícia Lopes Leite
Universidade de Brasília Campus Universitário Darcy Ribeiro, Brasília, DF 70910-900, Brazil

gleicelouise@gmail.com, llleite@unb.br
Abstract. A substantial number of educational problems are complex and there

are no quick solutions with immediate answers to solve them. Evasion is one of
these problems and deserves special attention, which justifies the need for studies,
research and critical reflections upon nuances that underlie, permeate and consti-
tute approaches that can contribute to understanding and fighting this phenomenon.
In distance learning, this phenomenon is even more prevalent, as evasion can be
influenced by the use of educational platforms, according to more recent studies.
The resources of educational technologies that distance learning uses allow the
construction of knowledge to take place in different spaces, with teachers and
students developing activities in different places or times. This study, based on
a current theoretical framework, analysed the educational resources available in
Virtual Learning Environments and their impact on evasion in distance learning
Higher Education courses. Based on a qualitative approach, the case study was
conducted using data from a Brazilian public university that identified the most
used educational technology resources and related them to dropout rates. The
results obtained suggest that the diversity of resources can impact those rates. The
data analysis comprised eight courses, five academic semesters, 257 virtual class-
rooms, 19 educational resources and 1.023 students. Thus, the use of indicators in
the creation of distance learning undergraduate courses is recommended to reduce
dropout rates.
Keywords: Dropout · Educational Resources · Distance Learning
1 Introduction
The subject of evasion arouses great interest in the academic environment, being stud-
ied worldwide in all types of education, in all educational levels and by all types of
institutions. Several factors are pointed out by specialists: the student’s economic situa-
tion, impact on social life, use of technologies, among others, and in Distance Learning
(DL) this phenomenon occurs even more [1]. According to the Ministry of Education
(MEC) [2], distance learning is a modality that uses information and communication
technologies for the construction of knowledge, allowing students and teachers to carry
out activities in different places and times.
In the period from 2009 to 2019, distance learning Higher Education courses showed
an exponential growth of almost five times the number of incoming students, representing
https://doi.org/10.1007/978-981-99-7353-8_27
358 G. L. Garcia Costa dos Santos and L. Lopes Leite
an increase of 378.9% in relation to face-to-face courses, which had a growth of 17.8%,

according to the National Institute of Educational Studies and Research Anísio Teixeira
(Inep) [1]. However, the 2018 Higher Education Census showed that, among students
who started a distance learning course in 2010, 56.8% had dropped out, 37.9% completed
the course and 5.3% remained in the undergraduate course for six years after admission
[3].
The evasion in distance learning modality is caused by various reasons such as
difficulty in reconciling studies and work, lack of educational monitoring and use of
educational technology resources [17]. In this way, this last factor is addressed in this
work from an exploratory case study based on a qualitative approach, in which data
analysis identified the most used resources in undergraduate courses and related them
to dropout rates. The results obtained suggest that the diversity of resources can impact
the dropout rates, for this, it is appropriate to adopt the use of indicators in the creation
of distance learning undergraduate courses.
2 Theoretical Framework
This work addresses education under the context of distance learning, an educational
modality in which teachers and students are physically separated, being, in this way,
necessary to use Information and Communication Technology (ICTs) resources so that
communication and knowledge construction can happen [7].
Higher education contributes significantly to the personal development of individu-
als, as it impacts on social development, argues the Institute of Research and Applied
Economics (Ipea) [4]. Thus, with regard to the modalities of education, MEC classi-
fies them as face-to-face and distance [5]. The distance modality can be implemented
in Basic Education, comprising the education of young people and adults, in special
education, in technical professional education and in Higher Education, according to
Decree 9,057, of May 25, 2017 [6].
Distance Learning (DL) has as its main feature the establishment of a two-way
communication, in which teacher and student are not together, in the same physical
space, requiring therefore means that enable communication between both parts [18].
Currently, this communication takes place by means of educational technology resources.
In general, it is necessary to verify and identify certain factors that interfere with the
evasion regarding distance learning courses and how this phenomenon is inserted in all
schooling levels [19], being predominant in the distance modality. Hence, through this
research, it is expected to identify new perceptions about this theme.
2.1 Distance Learning

Over the years, distance learning has been labelled in several ways, from distance edu-
cation to non-traditional education, through open teaching, experimental study, among
other terms [18]. This can be attributed to the possibility of building knowledge out-
side conventional classrooms, some authors divide the history of distance learning into
stages or based on the technology adopted [20]. The first model of DL recorded dates
from 1856, when language classes were sent and received by means of letters. There are
The Use of Indicators in the Creation of Distance Learning 359
reports that distance learning can be traced back to the first century, through the epis-
tles of Saint Paul, serving as the basis for Christian theology, in which these teachings
continue to be transmitted throughout the centuries [21].
Some studies bring an interesting fact about the history of distance learning in Brazil:
in the middle of the 19th century, records of distance typing courses were found, which
proves the presence of this educational modality since that time [8]. However, the authors
point out a crucial difference towards other countries: for a long time, Brazilian distance
learning focused only on basic technical training or supplementary education. Then,
from the 1990s, with the popularization of the internet and ICTs, some more robust
distance learning initiatives emerged, such as the Brazilian Open University System,
initially aimed at the continued training of teachers in the public school system.
2.2 Dropout
Dropout is the moment when students interrupt their academic path before completing
it [16]. However, this broad definition can generate disagreements among researchers,
compromising the universality and accuracy of the studies that have been conducted.
Thus, evasion is a complex phenomenon that requires greater depth and rigour in all
contexts. This need is emphasized by experts who reinforce the importance of under-
standing all the nuances and implications of evasion to promote a more efficient and
inclusive education [22].
There is a two-pronged approach to addressing the problem of school evasion [1].
First, there is the case in which the student leaves the educational institution without the
intention of returning to it. Secondly, the student leaves school for one year, but returns in
the following year. Although they are different concepts, both situations require specific
and equally important approaches. Evasion still means interrupting a study cycle without
achieving the main objective of obtaining a degree and this may occur due to dropout,
failure, change of course or loss of the enrolment deadline [16]. Therefore, the author
considers evasion and abandonment as equivalent terms. School evasion is also defined
as the definitive exit of the student from the course without completing it [23], concept
which is adopted in this work.
2.3 Brazilian Open University System
In Brazil, there are several government actions underway whose main aim is to increase
the number of free places in higher education. Among which, the Brazilian Open Univer-
sity Program stands out – Decree No. 5,800, on June 8, 2006 [9]. Through partnerships
set up with federal universities, this program offers undergraduate and postgraduate
courses in the distance modality, in order to offer training opportunities and enhance-
ment to teachers, managers as well as leaders of Basic Education. It is undeniable that
the Brazilian Open University System plays a key role in the reduction of educational
inequalities in the country. By developing and expanding the national system of distance
learning towards higher education, this program not only does contribute to the training
of qualified professionals, but also to the construction of a more just and equal society
[10].
2.4 Dropout in the Brazilian Distance Learning Modality and in the Open
University System
According to the Distance Learning Census conducted, by the Brazilian Association of

Distance Education (ABED), concerning the years 2019/2020, institutions are aware of
the factors that lead to evasion in distance learning including other types of education.
Among these factors, we highlight the pedagogical aspects, such as student performance,
attendance and participation, as well as the way Higher Education Institutions (HEIs)
assist student, especially in distance learning courses [11].
In this research, it was noticed that the most important information about distance
learning in Brazil comes from the National Census of Distance Education, coordinated
and executed by ABED. When it comes to face-to-face modality, it is worth mentioning
the annual survey conducted by Inep [3]. In 2018, ABED revealed that dropout rates of
22% related to distance learning regulated courses was between 26% and 50%.
Among the face-to-face courses, 18.5% of them rated dropouts between 21% and
25%. The data reinforces that the phenomenon of evasion remains a problem in dis-
tance learning [11]. In 2022, Inep released data from the Higher Education Census that
demonstrate that the distance learning modality is expanding, with an increase of 474%
in the number of incoming students of between 2011 and 2021; and, in the same period,
the ones in face-to-face courses decreased by 23.4% [3].
For the MEC, in 2021, the rates confirm that evasion in distance learning Higher Edu-
cation courses remain high, since in the Bachelor’s, Licentiate’s and Technical courses,
the completion rates were 40%, 43% and 38%, and the dropout rates were 59%, 56%
and 62%, respectively [12]. In Brazil between 2005 and 2020, 825,031 vacancies were
offered in undergraduate, bachelor’s, extension, technical, upskilling, reskilling and lato
sensu specialization courses by 109 public institutions of Higher Education. The average
dropout rate of the Brazilian Open University System, in this stratification, was 44.51%,
in a total of 367,201 vacancies [36].
2.5 Dropout in Open University Courses at a Public University
As the offer of vacancies for the Brazilian Open University System increases, by conse-
quence, the number of evaded students also increases. In 2012, in the initial analysis and
from the perspective of the course, it was found that the courses of Music and Theater
have the highest percentage of dropouts, both above 50%; and the lowest percentage was
within Pedagogy with 35.3%. In 2009, the Biology course had the highest rate of dropout
with 66.3%, which represents a very relevant number. The Physical Education course
had 35.8%, the lowest percentage of the year. For 2011, the rates from most courses
analysed did not exceed 20%, with emphasis on the Pedagogy course, with 4.6% [24].
In the scope of the Brazilian Open University, the data presented denote the difficulty
of the students who opted for distance learning. It is noticeable that they face difficulties
to stay in the course and made, in some cases, more than half of the class dropout of the
course. This phenomenon is related to the distance learning student’s profile and may
indicate some factors, as it is mostly related to a student who works and supports their
family, making it difficult to reconcile their reality with their studies. Other difficulties,
such as the use of technologies, are pointed out by some authors [36].
2.6 Educational Technologies Used in Courses of the Brazilian Open University

System
To support distance learning, several platforms are used and, among them, Moodle stands
out. It is a tool that was designed to allow teachers, managers and students to use a tailor-
made learning environment, as it allows them to keep detailed records of the activities that
students carry out. There are few studies that evaluate these tools as to their importance.
They show that several resources are not used in the Virtual Learning Environment
(VLE), so the authors realized that the relationship regarding these resources can reduce
students’ performance [25].
Thus, investigating, from the interactions of students, the use of educational resources
used in VLEs, in the distance learning higher education courses, can show whether these
same resources contribute to the conclusion of the course of these students.
3 Analysis of Data on the Use of Technologies in Distance Learning

Undergraduate Courses
The Brazilian Open University courses offered at a public university are made avail-
able from the Moodle virtual environment that gathers information from the disciplines
registered and taught at a distance, through resources that can be used by teachers and
students to carry out activities, favouring, in this way, interaction. There are difficulties
in analysing the large amount of data stored in VLEs and their implications; and this is
a necessary demand in distance learning, as it is necessary to know and identify certain
elements that can contribute to the regulation of learning [26]. The educational resources
made available in VLEs, when used in a reductive and inadequate way, cannot meet the
expected results [27].
The data selection comprised a total of eight courses: Visual Arts, Physical Education,
Physics, Geography, Languages, Music, Pedagogy and Theater; five academic semesters:
2020.1, 2020.2, 2021.1, 2021.2 and 2022.11 ; 257 virtual classrooms, in which each
virtual class corresponds to a subject; 19 educational resources and 1,023 students.
For the survey of resources made available in the VLE and, due to the specificity
of this work, the extraction of Moodle data occurred manually. Thus, each classroom
was visited, and the information was noted in a spreadsheet, in the OpenDocument
Spreadsheet (ODS) file format. The file stored data of the courses grouped by semesters,
resources used, quantity of interactions and respective disciplines related to the classes.
The version of AVA Moodle was (3.11.3) and the Postgres database was Postgres (12.3
(Ubuntu 12.3–1.pgdg18.04+1)).
For the extraction of evasion data, the Integrated Academic Management System
(SIGAA)2 was used. The data was made available in a Report in the format Document
Format (PDF). This data was transported to the Excel software, in which the resource
information used, and the evasion data were processed, manipulated and interpreted.
1 The interpretation and analysis of the Open University data from this research began in July
2022. Therefore, the data provided covered the semesters completed by this period.
2 Integrated Management System for Academic Activities, the main management tool for the
academic life of students at public universities.
The number of disciplines offered per semester and the total number per course are:
Visual Arts 31; Physical Education 32, Physics 29, Geography 41, Languages and Music
28, Pedagogy 38 and Theater 30. Regarding the resources made available, we have: in
the Visual Arts course, Forum, Resources and H5P resources were made available in all
academic semesters; and the semester that most used resources was 2021.2, 9 in total.
The Physical Education course made use of Forum, Questionnaires and Resources in all
semesters; most resources were used in 2020.2, 10 in total. In the Physics course, it was
found that Forum, Questionnaires, Resources, Tasks and H5P resources were made avail-
able in all semesters; and 2020.1, 2021.1 and 2022.1 semesters used 9 resources each. In
the Geography course, the resources used in all semesters were Forum, Questionnaires,
Resources, Tasks and Glossary; and 2020.2, 2021.2 and 2021.1 were the semesters in
which more resources were used, 7 in each one. In the Languages course, it was identified
that the educational resources such as Forum, Questionnaires, Resources and Tasks were
used in all academic semesters; 10 resources were used in 2021.2. Music course made
use of Forum, Questionnaires, Resources; and Tasks. 2020.1 was the semester in which
most resources were used, 11 in total. The following educational resources, Forum,
Questionnaires, Resources and Tasks were made available in all semesters in the Peda-
gogy course; 12 resources were used in 2020.1. The resources used in all semesters in the
Theater course were Forum, Resources, Tasks and Glossary; 2021.1 was the semester
in which most resources were used, 9 in total.
Based on the surveys conducted, it was observed that the resources available in
the Brazilian Open University System at the public university were: BigBlueButton,
Forum, Questionnaires, Resources, Tasks, Survey, Games, Glossary, Choice, H5P, Wiki,
Chat, Dialogues, Diaries, Evaluation Laboratory, Database, Evaluation Research, PDF
Annotations and Hot Potatoes - 19 resources in total. Among the resources identified,
those most available were: Forum, Questionnaires, Resources, Tasks, Glossary, H5P,
BigBlueButton, Games, Survey and Diaries. It can be seen that all courses provided
educational resources: Forum, Questionnaires, Resources, Tasks, Glossary and H5P.
Regarding the number of resources that each course used, we noticed that the Music
course, with 14 resources, provided the largest number of educational resources, followed
by the Languages course, with 13 resources. The Physical Education, Physics, Pedagogy
and Theater courses used 12 resources each. The Visual Arts course used 11 educational
resources; and the Geography course had the least number of resources available, 9 in
total.
The resource that promoted most interactions was Forum, followed by Tasks, Ques-
tionnares, Resources, Glossary and H5P; these resources were used by all courses. Still
regarding the analysis of interactions, it was observed that the courses of Pedagogy,
Music and Physical Education are those with the highest number of interactions in
comments on the Forum resource.
3.1 Dropout Data

Regarding the analysis of data referring to dropout in Fig. 1, it is noted that the most
significant dropout rate occurred during the second semester of 2020, considering the
number of active students who did not choose to withdraw from the course. During this
period, 264 students who stopped attending university classes were registered. On the
other hand, the number of evasions has been showing a gradual drop, as verified in the
subsequent semesters: 27 in the first half of 2021, 11 in the second half of 2021 and,
finally, 6 in the first half of 2022.
Fig. 1. Survey of students dropped out per semester
However, when considering the number of students who dropped out, in all academic
semesters, those who never attended classes, and those who started the courses, it was
noticed that, proportionally, the Languages course obtained the highest percentage of
evasion. Regarding the largest number of students who withdrew from Physics, 83 in
total, it is clear that although it is a traditional course, with a more rigorous approach, it
is understood that this can generate difficulties for teachers to identify which resources
should be used.
There are several methodologies associated with the same resource that can sig-
nificantly impact the participation of students in the teaching and learning process.
According to the data analysis carried out and comparing with the educational resources
made available in the courses with the lowest and highest dropout rates, Geography and
Languages, respectively, it was noticed that the latter used more resources, 13 in total,
while the former one used 9. Of these 9 resources used by Geography, 8 were also used
by the other course: BigBlueButtons, Forums, Questionnaires, Resources, Tasks, Game,
Glossaries and H5Ps.
When comparing the number of resources made available in each course and the
respective dropout rate, Fig. 2, it is possible to notice a pattern: courses that used more
resources usually had a higher dropout rate. This can be caused by the excess of informa-
tion, because it is thought that the effort to find, understand and apply so many contents
can influence the student’s dropout.
It is considered that the use of a great number of educational resources in VLE can
be an obstacle in the learning process, and this contributes to dropout rates. Although
the intention in diversifying resources is to enrich knowledge, probably the excess of
information is perhaps not the correct strategy and, in this way, a smaller number of
resources would provide greater efficiency in learning.
Thus, it is understood that the number of resources must be sufficient for the student
to understand the content, but not to the point of raising doubts and insecurity. Another
important observation is that resources are chosen carefully and strategically, so that the
teacher can monitor the progress of learning with more integrity and the student with
greater effectiveness. In summary, the relationship between the number of educational
Fig. 2. Resources made available and dropout rates
resources used, and dropout rates is a complex one and must be analysed carefully, since
the availability of resources is important to ensure learning, and the excess of them can
have an impact on student learning, contributing to the increase in dropout.
4 Indicators Used in this Research

After analysing and interpreting the data from Sect. (3), indicators, which impact stu-
dents’ behaviour and influence dropout, were explored [28]. The selected features were
collected from the data used in this research: the Course Area, the Student’s Profile,
the Number of Interactions and the Resource Classification. The research conducted
suggests that the indicators should be taken into account, from the creation of distance
learning courses, since it is been estimated that the area of knowledge, the student’s
profile, the educational resources and the type of resource used may impact the selection
of educational resources used in the VLE.
It is also important to consider the technical, pedagogical and administrative aspects,
ensuring the quality of the teaching offered to students. It is noted that it is necessary
to have a balance between the use of educational resources and the basis in the area of
knowledge, the course and the student’s profile, because these factors may reflect on the
students’ experience. Therefore, it is recommended that Higher Education institutions
promote the evaluation and use of educational resources in distance education courses,
so that they find an ideal combination of resources that in a way do not stand out from
learning, which can cause an increase in dropout rates.
4.1 Course Area

Inep classifies higher and sequential courses, by area, based on the International Standard
Classification of Education (Isced), in Portuguese, International Standard Classification
of Education (Cine) [13]. The Institute addresses that this classification allows to com-
pare, internationally, the Brazilian statistics of Higher Education, in 11 major areas: 00
Basic Programmes; 01 Education; 02 Arts and Humanities; 03 Social Sciences, Commu-

nication and Information; 04 Business, Administration and Law; 05 Natural Sciences,
Mathematics and Statistics; 06 Computing and Information Technologies and Commu-
nication (ICT); 07 Engineering, Production and Construction; 08 Agriculture, Forestry,
Fisheries and Veterinary; 09 Health and Welfare and 10 Services. Thus, the first indicator
used in this research is the course area. In general, the Brazilian Open University Degree
courses are related to the great area of ‘01 Education’ and for Capes these great areas, in
turn, encompasses basic areas, or areas of knowledge that are grouped into other areas
and specialities [14].
By grouping the courses by area of knowledge, it was found that the area of Lin-
guistics, Language and Arts presented the highest dropout rate, reaching 49.07%, which
includes the courses of Literature, Music and Theater. The area of Exact Sciences and the
Earth, encompassing the Physics course, presented the largest number of students who
evaded, 83 in total, however, the rate was 48.82%, because according to the calculation
methodology used, the number of students who entered the course was considered, so,
this number was divided by the number of students who dropped out. In the Health Sci-
ences area, which covers the Physical Education course, the dropout rate was 28.99%.
Finally, the Human Sciences area, which is composed of the Visual Arts, Geography
and Pedagogy courses, presented the lowest percentage of dropout, 26.24%. As a way to
avoid increasing dropout rates and achieve the objectives proposed in Higher Education,
it is important to know the characteristics of each course because these features can
influence the decision to withdraw from the course.
4.2 Student’s Profile

The student’s profile in the Human Sciences area is heterogeneous in relation to several
aspects such as: social class, personal and cultural preferences and political orientation.
In this area of knowledge, the student needs to develop skills in written and spoken
language, organise their thoughts, expose and reproduce opinions and ideas, so it is
necessary that the student of Human Sciences appreciates reading, speaking and writing.
The Secretariat points out, in this way, the second indicator used which is the student’s
profile based on the area of knowledge [29].
The area of Human Sciences, in this research, encompasses the courses of Visual
Arts, Geography and Pedagogy and the resources used in the VLE were: Database, Big-
BlueButton, Diaries, Forum, Glossary, H5P, Questionnaires, Resources and Tasks; 9 in
total. In the area of knowledge of Exact Sciences and the Earth, the student needs to be
analytical and good at numbers. The student of this area must have the ability to create
and logical reasoning; skilled at calculations, solving problems and carrying out research
based on Mathematics, Chemistry and Physics [30]. The Physics course, in the area of
knowledge known as Exact and Earth Sciences, used the following resources: BigBlue-
Button, Choice, Forum, H5P, Evaluation Research, Survey, Questionnaires, Resources,
Tasks; 9 resources altogether. In the Area of Health Sciences, the student needs to be
skilled in dealing with people, empathetic to others and have initiative to help others. The
student needs to have a critical and reflective world view and that, in this area, it is not
only sought to train professionals for the work market, but someone who assists people
and who is concerned with improving the services provided. Hence, human relationships
are fundamental, because being in contact with, taking care of and helping other people,
is part of the routine of professionals in the Health Sciences [30]. The Physical Education
course, in the Health Sciences, used 8 resources: Chat, Diaries, Forum, Glossary, H5P,
Evaluation Laboratory, Questionnaires and Resources.
The area of Linguistics, Languages and Arts consists of the following courses: Lan-
guages, Music and Theater. They used 12 resources: Database, BigBlueButton, Chat,
Diaries, Choices, Forum, Glossary, PDF Annotations, Questionnaires, Resources, Tasks
and Wiki. ´The resources made available in the VLE are relevant, in particular, the Wiki,
a bibliographic collection and an important resource for students [31].
4.3 Number of Interactions

The third indicator of this research is related to the number of interactions, by resource, in
the virtual environment. Interactions are the ways in which students communicate with
each other and with their teachers in a VLE, and can happen through chats, discussion
sessions, videoconferences and other asynchronous communication tools [32]. In the
analysis conducted, the resources that provided the highest number of interactions were:
Forum, Tasks, Questionnaires and Survey. Pedagogy courses, in the area of knowledge
of Human Sciences; Music and Geography, area of Linguistics, Languages and Arts;
and the the Physical Education course, in the area of Health Sciences, had the highest
number of interactions. Students who interact with each other and with their teachers
become more committed and interested [33], and commitment may reflect on academic
performance, reduction of social isolation and the development of interpersonal skills
[34].
4.4 Resource Type

The fourth and last indicator is related to the type of educational resources available in
the VLE. The classification adopted was the same used by Moodle [15] based on the
documentation from: https://moodle.org/, which classifies educational resources into
activities or resources. The effective learning of educational resources implies the inter-
nal understanding of the content presented and the integration of different contexts,
therefore, it is necessary to use other educational resources in VLE – the use of commu-
nication resources, which stimulate the interaction between students and their teachers
and contribute to the reflection and interpretation of the content, fundamental for the
learning process. However, it is clear that practical activities are a resource that should
be mobilized, since learning should not only be theoretical, but should be developed in
practice. In this way, the student can understand the context presented and mange to
apply it in real situations, because the combination of educational resources is essential
for the success of learning in a VLE. In this way, it is important to use different resources
in a virtual environment for full and effective learning [35].
5 Conclusion
The objective of this research was to investigate the educational resources most fre-
quently used in the virtual rooms of distance learning undergraduate courses and their
relationship with the dropout rates recorded in these courses. To achieve this objective,
an exploratory case study approach was adopted, using data from the Brazilian Open
University System at a public university. The collected data was selected, analysed and
interpreted, comprising the number of disciplines, the resources used, the interactions
established, the number of students who dropped out and the dropout index in each
course. The analysis found a recurring pattern: courses that employed a greater number
of educational resources had higher dropout rates.
It appears that it is essential to monitor the development process of the course and it is
noticed that making use of indicators, since its creation, can guide the use of educational
resources in VLE. Different training areas and student profiles can affect the choice of
these resources. Therefore, a certain resource used in the Physics course would not be
suitable, for example, for the Language course. Thus, the choice of these resources can
impact student performance, so the adoption of indicators is necessary for the course to
be efficient, attractive and able to collaborate with the professional and personal training
of students effectively, contributing, in this way, to the reduction of dropout rates.
For future research, it is suggested that innovative indicators be explored with regard
to evasion in distance learning undergraduate courses, in order to implement more effec-
tive strategies identifying the educational resources to be made available in the VLE that
may contribute to the dropout reduction. To act in reducing dropout, it is necessary to
consider not only the traditional aspects such as age, gender and socio-economic profile
of students, but also the specific variables related to the use of these resources. It is
believed that the understanding of the impact and the active participation of students
using these resources could contribute to the dropout rate reduction in distance learning
courses. Therefore, investing in research that explores these issues is fundamental for
the improvement and success of the distance learning undergraduate education.
References
1. Inep, Instituto Nacional de Estudos e Pesquisas Educacionais An´ısio Teix-
iera: Informe estatístico do MEC revela melhoria do rendimento escolar. Min-
istério da Educação do Brasil (2021). http://portal.inep.gov.br/artigo/-/asset
publisher/B4AQV9zFY7Bv/content/informeestatistico-do-mec-revela-melhoria-do-
rendimento-escolar/21206. Acesso em: 25 de Out 2021
2. Brasil, Ministério da Educação: Informe estatístico do MEC revela melhoria do rendimento
escolar Panorama da EAD no Brasil. Centro de Gestão de Estudos Estratégicos (2022). https://
www.gov.br/mec/pt-br/reunidigital/pdf/reuni-digital-volume 02 17–05–2022.pdf Acesso em:
05 de Jun 2022
3. Inep, Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixiera: Ensino a Dis-
tância Cresce 474 vezes em uma Década (2023). https://www.gov.br/inep/pt-br/assuntos/not
icias/censo-da-educacao-superior/ensino-adistancia-cresce-474-em-uma-decada Acesso em:
23 de Nov 2022
4. Ipea, Instituto de Pesquisa Econômica Aplicada: Objetivos de desenvolvimento sustentável:
Assegurar a educação inclusiva e equitativa e de qualidade, e promover oportunidades de
aprendizagem ao longo da vida para todos (2019). https://www.ipea.gov.br/ods/ Acesso em:
10 de Out 2021
5. Brasil, Ministério da Educação: Objetivos de desenvolvimento sustentáavel: Cursos: quanto
à formação. Cadastro Nacional de Cursos e Instituições de Educação ao Superior Cadastro
e-MEC ˜ (2021). https://emec.mec.gov.br/emec/educacao-superior/cursos Acesso em: 23 de
Mar 2021
6. Brasil, Ministério da Educação ao: Decreto º 9.057, de 25 de maio de 2017: Regulamenta o

art. 80 da Lei nº 9.394, de 20 de dezembro de 1996, que estabelece as diretrizes e bases da
educação nacional. Presidência da República do Brasil (2017). https://www.planalto.gov.br/
ccivil 03/ ato2015–2018/2017/decreto/d9057.htm Acesso em: 30 de Nov 2021
7. Brasil, Ministério da Educação: O que é educação a distância? (2022). http://portal.mec.gov.
br/escola-de-gestores-da-educacao-basica/355-perguntas-frequentes911936531/educacao-
a-distancia-1651636927/12823-o-que-e-educacao-a-distancia Acesso em: 05 de Nov 2021
8. de Oliveira, A.F.P., et al.: Educação a Distância no mundo e no Brasil. Revista Educação
Pública (2019). https://educacaopublica.cecierj.edu.br/artigos/19/17/ead-educacao-a-distan
cia-no-mundo-eno-brasil Acesso em: 03 de Fev 2021
9. da Educação Brasil, M.: Decreto nº 5.800: de 08 junho de 2006: Dispõe sobre o Sistema
Universidade Aberta do Brasil - UAB. Brasília. Presidência da República do Brasil. (2006).
https://www.planalto.gov.br/ccivil 03/ ato2004-2006/2006/decreto/d5800.htm Acesso em: 30
de Nov 2021
10. CAPES, Coordenação de Aperfeiçoamento de Pessoal de Nível Superior: de 08 junho de 2006:
Universidade Aberta do Brasil (2006). https://www.gov.br/capes/pt-br/acesso-a-informacao/
acoes-e-programas/educacao-adistancia Acesso em: 13 de Ago 2021
11. ABED, Associação Brasileira de Educação a Distância: BR: Relatório analítico da aprendiza-
gem a distância no Brasil 2019/2020 [Org.] ABED – Associação Brasileira de Educação a Dis-
tância. (2020). http://www.abed.org.br/site/pt/midiateca/censo ead/1986/2021/03/censoeadbr
- 2019/2020 Acesso em: 02 de Nov 2021
12. Inep, Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixiera: Novo Painel de
Estatísticas do Censo da Educação Superior (2023). https://www.gov.br/inep/pt-br/acesso-a-
informacao/dados-abertos/inep-data/estatisticascenso-da-educacao-superior Acesso em: 13
de Jan 2023
13. Inep, Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixiera: Manual para
Classificação dos Cursos de Graduação e Sequenciais — VERSÃO VIGENTE (Publicada em
12 de maio de 2021). Áreas de Atuação, Pesquisas Estatísticas e Indicadores Educacionais,
Cine Brasil, Classificação (2021). https://www.gov.br/inep/pt-br/areas-de-atuacao/pesquisas-
estatisticas-e-indicadores/cinebrasil/classificacao Acesso em: 27 Abr 2023
14. CAPES, Coordenação de Aperfeiçoamento de Pessoal de Nível Superior: Tabela de Áreas
de Conhecimento/Avaliação (2021). https://www.gov.br/capes/pt-br/acesso-a-informacao/
acoes-eprogramas/avaliacao/instrumentos/documentos-de-apoio-1/tabela-de-areas-deconh
ecimento-avaliacao Acesso em: 27 Abr 2023
15. MOODLE: TWelcome to the Moodle Community. Moodle, [S.I.] (2021). https://moodle.org/
Acesso em: 20 Dez 2022
16. de L. Gaioso, N. P.: Evasão discente na educação superior: a perspectiva dos dirigentes e dos
alunos. 2005. 99 f. Dissertação (Mestrado)-Programa de Pós-Graduação “Stricto Sensu” em
Educação (2005)
17. Xenos, M., Pierrakeas, C., Pintelas, P.: A survey on student dropout rates and dropout causes
concerning the students in the Course of Informatics of the Hellenic Open University. Comput.
Educ. 39, 361–377 (2002)
18. Pery, W., Rumble, G.: A short guide to distance education. International Extension College,
Office D, Dales Brewery, Gwydir Street, Cambridge CB1 2LJ, England 1986, pp. 361–377
(2002)
19. de Oliveira, C.V.S.B., Bezerra, D.H.D., de Souza Torres, G.V.: Revisão sistemática da lit-
eratura sobre as causas de evasão da Educação a Distâancia no Brasil. EmRede-Revista de
Educação a Distância [S.I.] 2021, pp. 1–15 (2021)
20. Vidal, E.M., Bessa, J.E.: Introdução a educação a distância. Fortaleza: Editora RDS 2010
(2010)
21. Gouvea, G., Oliveira, C.I.: Educação a distância na formação de professores: viabilidades,
potencialidades e limites. [S.I.]. Vieira & Lent 2006 (2006)
22. Cunha, E.R., Morosini, M.C.: Evasao na educação superior: uma tematica em discussão.
Revista Cocar. 2006 (2013)
23. Andifes, A.A., Abruem, S.: SESU/MEC: Avaliação: Revista da Rede de Avaliação Institu-
cional da Educação Superior. Campinas; Sorocaba, SP, 55–65 (1996)
24. Fernandes, M.L.B.: Educação a Distância no Ensino Superior. Editora Universidade de
Brasília (2012)
25. Magalhaes, E., Gomes, V., Rodrigues, A., Santos, L., Conte, T.: Impacto da usabilidade
na educação a distância: um estudo de caso no Moodle IFAM. In: Proceedings of the IX
Symposium on Human Factors in Computing Systems, pp. 231–236 (2010)
26. Barbosa, A.A.S., Andrade, F.S., de Carvalho, R.N.: Mineração de dados em ambientes virtuais
de aprendizagem: Aportes para a pesquisa em Educação a Distância. Interfaces Científicas –
Educação, 125–135 (2017)
27. da Silva Reis, R.M., Leite, B.S., Leao, M.B.C.: Estratégias didáticas envolvidas no uso das
TIC: o que os professores dizem sobre seu uso em sala de aula? ETD - Educação Temática
Digital, 551–571 (2021)
28. da Silva, E.C., de Freitas, S.A.A., Ramos, C.S., de Menezes, A.E.M., de Araújo, L.K.S.R.: A
systematic review of the factors that impact the prediction of retention and dropout in higher
education. Proceedings of the 56th Hawaii International Conference on System Sciences.
HICSS, 1271–1280 (2023)
29. Bento, E.G., do Nascimento, D.L.: O perfil socioeconômico e profissional do trabalhador-
estudante do curso de Pos-graduação lato sensu em docência (EaD) do Instituto Federal de
Minas Gerais-Campus Arcos, 251–262 (2023)
30. Tavares, F.J.P., Costa, A.R., da Silva Ilha, F.R., Cardozo, P.L., Rigo, L.C.: Evasão no Ensino
Superior: em pauta os cursos de Licenciatura em Educação Física da UFPEL. Avaliação:
Revista da Avaliação da Educação Superior (Campinas). SciELO Brasil, 571–590 (2023)
31. de S. Jatobá, A.L.H., Duarte, F.R., Batista, K.M.: O ambiente Virtual Moodle no Processo
de Ensino e Aprendizagem do Curso de Formação Pedagogica em Artes Visuais. Revista de
Educação da Universidade Federal do Vale do São Francisco (2014)
32. Mülbert, A.L., Girondi, A., Pereira, A.T.C., Nakayama, M.K.: A interação em ambientes
virtuais de aprendizagem: motivações e interesses dos alunos. CINTED-UFRGS. Revista
Novas Tecnologias na Educação (2011)
33. de Oliveira, W.P., Bittencourt, W.J.M.: A evasão na EaD: uma análise sobre os dados e
relatórios, ano base 2017, apresentados pelo Inep, UAB e ABED. Educação Pública, [S.I.]
(2020)
34. de Martins, L.M., Ribeiro, J.L.D.: Os fatores de engajamento do estudante na modalidade de
ensino a distância. Revista GUAL – Florianópolis 11, 249–273 (2018)
35. de Jesus Silva Silva, G., Ramos, W.: O ambiente virtual de aprendizagem (AVA) como poten-
cializador da autonomia do estudante: Estudo de caso na UAB-UNB. Revista Iberoamericana
de Evaluacion Educativa (2021)
36. Nogueira, D.X.P., Ferreira, M., de Lira, L.A.R.: A evasão no sistema Universidade Aberta
do Brasil: uma análise das justificativas das instituições integrantes. Educação e Fronteiras.
Dourados (2020). https://doi.org/10.30612/eduf.v10i29.14169
Building Resilient Educational Systems:
The Power of Digital Technologies
Regina Motz1(B) , Mariana Porta2 , and Eliseo Reategui3

1
Facultad de Ingenierı́a, Universidad de la República, Montevideo, Uruguay
rmotz@fing.edu.uy
2
Facultad de Ciencias Sociales, Universidad de la República, Montevideo, Uruguay
mariana.porta@cucel.edu.uy
3
PPGIE, Universidade Federal do Rio Grande do Sul, Porto Alegre, Brazil
eliseo.reategui@ufrgs.br
Abstract. Recently, the COVID-19 pandemic has posed unprecedented

challenges to the education sector, requiring the adoption of digital sys-
tems to ensure continuity of learning. This paper aims to examine how
digital education systems contribute to enhancing educational resilience.
It highlights the role of digital technologies in strengthening academic
resilience by enabling adaptive and supportive learning environments.
The discussion emphasizes the intricate interplay between diversity,
inclusive practices, resilient responses, and the utilization of digital tech-
nologies. By exploring the potential of digital systems to enhance edu-
cational resilience, this paper contributes to the broader goal of achiev-
ing inclusive and equitable quality education, aligned with the United
Nations’ Sustainable Development Goal 4.
1 Introduction
Resilience is an inspiring construct that has long been associated with child
development in the field of education [2,29]. It is interrelated with studies on
vulnerability, protection, survival, and the response to adversity [15].
The term resilience in its Latin etymology comes from resilio which means
to rebound or to go back. The Royal Spanish Academy (RAE) defines resilience
as the ability of a material, mechanism or system to recover its original state
after experiencing disturbance.
In the context of physics, resilience refers to the ability of a material or system
to endure stress, deformation, or damage while retaining its structural integrity
and intended functionality. This means that resilient physical entities can with-
stand external forces and rebound to their initial state or level of performance.
Examples of resilient materials include rubber bands, bungee cords, and springs,
all capable of stretching and returning to their original shape.
The concept of resilience has been applied in various areas of knowledge.
The recurrence of the concept, both in documents and communications from
international organizations [21,23,33,34] and recent academic articles examining
https://doi.org/10.1007/978-981-99-7353-8_28
Building Resilient Educational Systems: The Power of Digital Technologies 371
educational experiences during the COVID-19 pandemic [1,6,19,30] highlight its

importance in research.
Unesco proposes that education actors should develop actions to increase
prevention, preparation and response to crises, such as alternative educational
continuity plans using technologies [34]. Additionally, research in education sug-
gests that resilience can be a product of intentional design [22], and from systemic
thinking, resilience has been understood as a capacity that can be developed [7].
Addressing resilience in education is crucial for achieving inclusive and equi-
table quality education, as outlined in the United Nations’ fourth Sustainable
Development Goal (SDG) within its 2030 Agenda for Sustainable Development
[32]. Inclusion involves creating environments where every individual or group
feels welcome, respected, supported, and valued, allowing them to participate
fully. Equitable education acknowledges diversity by considering the unique char-
acteristics that differentiate one individual or group from another.
The intricate interplay between diversity, inclusive practices, resilient
responses, and the utilization of digital technologies requires a thorough exam-
ination. Therefore, this article aims to examine how digital education systems
contribute to enhancing the resilience of education.
The rest of this work is organized as follows. Section 2 introduces the concept
of resilience across different disciplines. Section 3 presents the components of the
concept of resilience in education, analyzing the concept of academic resilience
and the resilience of students and teachers. Section 4 delves into the role of digital
systems in strengthening academic resilience. Lastly, Sect. 5 provides concluding
remarks and outlines potential areas for future research.
2 Resilience Across Different Disciplines
The concept of resilience originates from the field of Physics and was coined
by Michael Rutter in 1972 and 1979 [29] for the study of child development.
Subsequently, it was adopted in the field of social sciences, particularly in social
work research and socio-educational policies. However, its application extends
across numerous disciplines.
In psychology, resilience has long been associated with the understanding of
child development. It explores how individuals can overcome traumatic experi-
ences and setbacks, emerging more robust and more resilient on the other side.
The study of resilience delves into the protective factors, coping mechanisms, and
support systems that enable people to navigate difficulties and achieve positive
outcomes.
In the field of engineering and materials science, resilience takes on a physical
dimension. It refers to the ability of materials, mechanisms, and systems to with-
stand stress, deformation, or damage while retaining their structural integrity
and functionality.
Ecology unveils yet another facet of resilience, focusing on the ability of
ecosystems to absorb disturbances, adapt, and maintain their essential func-
tions. In the face of environmental changes and disruptions, resilient ecosystems
372 R. Motz et al.
demonstrate the capacity to regenerate, sustain biodiversity, and provide critical

ecosystem services.
Despite the differences in how resilience is applied in these diverse disci-
plines, the core concept remains strikingly similar. Across structural engineering,
social sciences, and ecology, resilience shares the common essence of withstand-
ing challenges, adapting to changes, and preserving crucial functions. Analyzing
resilience in these varied fields provides valuable insights that can enrich our
understanding and application of resilience in educational systems.
2.1 Resilience in Structural Engineering

Within the field of structural engineering, the study of resilience primarily
focuses on constructing anti-seismic houses [5,9]. These studies highlight the
examination of four crucial properties known as the “4Rs” for resilience analy-
sis:
• Robustness: refers to the strength and ability of elements, systems, and other
units of analysis to withstand a given level of stress or demand without experi-
encing degradation or loss of function. In anti-seismic construction, robustness
is of paramount importance to ensure the integrity and stability of buildings
during seismic events.
• Redundancy: addresses the presence and extent of substitutable elements, sys-
tems, or other units of analysis within a structure. It involves having backup
components or alternative mechanisms that can fulfil functional requirements
in case of disruption, degradation, or loss of functionality. Structures can
maintain their essential functions by incorporating redundancy even if cer-
tain elements become compromised during a seismic event.
• Resourcefulness: reflects the capacity of structural engineers and stakeholders
to identify potential problems, establish priorities, and mobilize resources in
the face of conditions that may threaten to disrupt elements, systems, or
other units of analysis. This involves strategic planning, coordination, and
decision-making to mitigate risks and effectively ensure structures’ resilience.
• Rapidity: refers to the capacity to meet priorities and achieve goals in a
timely manner. Rapid response and recovery are crucial in minimizing the
impact of the event, ensuring the safety of occupants, and restoring function-
ality to affected structures. It involves efficient communication, well-defined
emergency protocols, and the availability of necessary resources.
Additionally, the study of resilience explores integrating smart technologies,
sensors, and real-time monitoring systems to enhance structures’ adaptive capa-
bilities and response mechanisms.
2.2 Resilience in Social Systems

Resilience is the ability of a social system to absorb disruptions and reorganize
itself while undergoing changes in such a way that it still maintains essentially
the same function, structure, identity, and feedback [36].
Furthermore, social systems exhibit diversity and redundancy. They encom-

pass a variety of components, social actors, and institutional arrangements. This
diversity provides alternative pathways for adaptation and enables the system
to function effectively even when certain components or relationships are dis-
rupted. Redundancy ensures that multiple actors or resources are available to
fulfil critical functions, reducing the vulnerability of the system to single-point
failures.
Connectivity also plays a vital role in the resilience of social systems. Inter-
actions and linkages between different components within the system enable the
flow of information, resources, and feedback loops. These connections allow for
the diffusion of knowledge, cooperation, and the exchange of resources, which
enhances the system’s capacity to respond and adapt to changes.
The emphasis in studying resilience in social systems lies in understanding the
dynamics of the system. The focus is on comprehending how the system operates,
adapts, and responds to internal and external influences. By examining social
components’ interactions, feedback loops, and interconnectedness, researchers
aim to uncover the underlying mechanisms contributing to the system’s ability
to withstand disruptions, recover from challenges, and maintain its essential
functions. Works such as Fath et al. [14] and Cote and Nightingale [10] integrate
the study of resilience in social systems with the concept of resilience in ecological
systems. They highlight the transformative processes that occur during times of
crisis, chaos, confusion, and innovation as the phases that foster resilience.
Social resilience in the face of disasters and crises involves a cyclical pro-
cess that encompasses the following four key phases: mitigation, preparation,
response, and recovery [12].
• Mitigation: focuses on strengthening the community’s capacity to manage
future disasters effectively. This involves taking proactive measures, such
as constructing climate-responsive housing, implementing land-use planning
strategies, and promoting sustainable practices.
• Preparation: involves anticipating emergencies and developing a robust emer-
gency response system. This includes establishing early warning systems, cre-
ating response plans, stockpiling necessary resources, and conducting drills
and training to enhance preparedness.
• Response: entails immediate actions taken during and immediately after a dis-
aster. It involves activating emergency protocols, mobilizing resources, and
providing emergency relief and assistance to affected individuals and com-
munities. Effective response measures include search and rescue operations,
medical aid, and the establishment of temporary shelters and distribution
centers.
• Recovery: focuses on assessing the damage, initiating repairs, and facilitat-
ing the recovery of the community. This phase involves conducting damage
assessments, securing reconstruction funding, supporting affected individuals
and businesses, and restoring critical infrastructure and services.
This resilience cycle at the social and community level involves three distinct
levels that can be addressed to enhance resilience [12]. Firstly, the individual level
374 R. Motz et al.
focuses on building personal resilience among community members. This involves

promoting psychological well-being, providing access to resources and support
systems, and enhancing coping skills to navigate challenges and adversities effec-
tively. By empowering individuals to develop their own resilience, communities
can establish a strong foundation for overall resilience. Secondly, the commu-
nity level fosters collective resilience by establishing social networks, community
organizations, and collaborative decision-making processes. Building social cohe-
sion and fostering a sense of belonging within the community encourages mutual
support, cooperation, and the sharing of resources. Lastly, the systemic level rec-
ognizes the importance of addressing structural and systemic factors influencing
community resilience. This involves analyzing and addressing social inequali-
ties, economic disparities, and environmental vulnerabilities that can hinder a
community’s ability to withstand and recover from shocks and stressors.
As an example case, we analyze the city of Montevideo, the capital of Uruguay
during the COVID-19 pandemic. Like many countries, the government had to
implement containment measures to prevent the spread of the virus. The eco-
nomic situation of a large part of the population was affected. Between March
and July 2020, more than 25 per cent of workers in the formal private sector was
sent to unemployment insurance. As a result of this situation, the solidarity of
the Uruguayans emerged to face the crisis and together with it, a large number
of soup kitchens and picnic areas arose. In a study carried out by a team from
the Faculty of Social Sciences of the University of the Republic, more than 60
per cent do not have any state support [27]. Since most of the popular pots and
picnic areas in Uruguay do not have state aid but solidarity, it was considered
a social resilience act. However, as [27] mentions, these seemingly spontaneous
initiatives build on existing networks and recreate them. Neighbourhood social
militants of many years and many struggles retake leadership. Collectives or com-
munities of different types of affinity (political, sports, religious) with decades of
neighbourhood work take action, although not necessarily structured and per-
manent. Other elements that stand out in the work of [27] is the capacity for
collaboration between these community experiences with more structured and
traditional social organizations, with more significant resources, such as worker
unions and the existence of meeting spaces that allow collaboration. However,
collaborating in these meeting spaces is not easy since it supposes the existence
of different “modes” can only be strengthened if they are reciprocally respected.
Olssen et al. [24] highlights the problems of using resilience as a universal
and unifying concept in the social sciences. The conclusions drawn are that
there are differences in resilience definitions, a lack of compatibility between
natural and social sciences, and the ambition to unify resilience beyond nat-
ural sciences is counterproductive. Additionally, caution is advised regarding
the risk of resilience being used in a depoliticizing or naturalizing manner by
political actors. For fruitful interdisciplinary collaboration, it is suggested to
prioritize pluralism over radical unification. The article finishes by stating that
in order to foster productive interdisciplinary collaboration, such as problem-
solving approaches, embracing pluralism is not only sufficient but also potentially
more favourable.
3 Resilience in Education
Resilience in education refers to the ability of individuals and educational orga-

nizations to face and overcome challenges, adversities, and changes in the educa-
tional context. Its study is usually approached from two perspectives: individual
resilience and organizational or system resilience. In this section, we briefly intro-
duce these two perspectives.
3.1 Student and Teacher Resilience
Individual resilience in education refers to the capacity of students and teachers

to adapt, persist, and succeed academically despite difficulties. It involves devel-
oping socio-emotional skills such as self-confidence, self-regulation, perseverance,
and the ability to cope with academic stress and pressure.
Resilient students have the ability to bounce back from failures and setbacks,
learn from them, and move forward. They can also leverage available resources,
seek support, and maintain motivation and commitment to learning over time.
Students’ resilience, also known as”academic resilience,” has been extensively
studied [4,11,16,17,20,25]. These works mainly examine the academic resilience
of students from psychological and social perspectives. They highlight the lead-
ership capacity of resilient students and their ability to work in a network, build
valuable interpersonal relationships, and establish social networks. Additionally,
their ability to adhere to beliefs, principles, or values is observed as a determin-
ing factor necessary to persevere and succeed in academic goals. However, since
the Covid-19 pandemic, studies in this area have embraced a broader concep-
tualization of academic resilience that recognizes significant concerns regarding
physical well-being and mental health in student populations.
In the same way, as with the study of student resilience, the work on teacher
resilience increased due to the Covid-19 pandemic [19,28,31,35,37].
Teacher resilience encompasses various dimensions that contribute to their
overall well-being and ability to navigate challenges. Similar to student resilience,
four possible dimensions of teacher resilience can be identified: profession-related,
emotional, motivational, and social.
The profession-related dimension of teacher resilience acknowledges the spe-
cific demands and responsibilities that teachers face within their educational
roles. It encompasses aspects such as their professional identity, sense of effi-
cacy, and belief in their ability to achieve desired outcomes for their students.
This dimension recognizes the importance of teachers’ self-perception and their
confidence in positively impacting student learning. The emotional dimension of
teacher resilience focuses on the emotional well-being of educators. It involves
their capacity to regulate and manage their emotions in the face of stressors
and challenges. This dimension acknowledges that teaching can be emotionally
376 R. Motz et al.
demanding and highlights the significance of emotional intelligence and self-

care practices in maintaining overall well-being. The motivational dimension of
teacher resilience pertains to teachers’ intrinsic motivation and passion for their
profession. It encompasses their drive to continually improve and adapt their
instructional practices and their ability to find purpose and fulfilment in their
work. This dimension recognizes the importance of fostering a sense of auton-
omy, mastery, and purpose in teaching to sustain long-term resilience. Lastly,
the social dimension of teacher resilience emphasizes the significance of social
support and positive relationships within the educational context. It acknowl-
edges that teachers thrive in environments where collaboration, collegiality, and
effective communication are fostered. This dimension highlights the importance
of building supportive networks, seeking mentorship opportunities, and engaging
in collaborative professional learning communities.
In measuring teacher well-being, Folk et al. [13] highlight four essential
concepts: self-efficacy, work and emotional stress, job satisfaction, and social-
emotional competence. Self-efficacy refers to teachers’ belief in their ability to
achieve desired outcomes for their students, which is closely linked to their pro-
fessional resilience. Work stress and wear recognize the potential challenges and
demands that teachers may face, underscoring the importance of managing stres-
sors and maintaining a healthy work-life balance. Job satisfaction reflects the
level of fulfilment and contentment teachers experience in their roles, as it sig-
nificantly influences their overall well-being. Lastly, social-emotional competence
recognizes the importance of teachers’ ability to effectively navigate and man-
age their emotions and build positive relationships with students and colleagues.
Promoting teacher resilience is crucial not only for their own well-being but
also for creating optimal learning environments and promoting the resilience of
students.
Moreover, according to Wang [37], one of the major challenges that teachers
encountered during the COVID-19 pandemic was the sudden need to transition
their courses online without having sufficient mastery of the necessary technol-
ogy. This situation resulted in teachers experiencing technostress, which added
to the difficulties they faced. Consequently, it is crucial to ensure that faculty
members are equipped with contemporary digital skills to address such situa-
tions effectively. By providing teachers with the necessary training and resources,
they can develop the competence and confidence required to navigate digital plat-
forms and tools seamlessly. This preparation will not only enable them to adapt
to unexpected circumstances but also enhance their overall teaching effectiveness
in an increasingly digital era.
3.2 Educational Systems Resilience

According to UNESCO, education resilience refers to ’the capacity of children,
families, communities and systems to resist, adapt and recover from shocks and
stresses’ [33]. As Jacobson et al. [18] and Pickernell [26] pointed out, educational
systems are complex systems because they are made up of a set of actors (stu-
dents, teachers, directors of institutions, parents, society, the economic system,
the health system) where all these actors interact together and can create new
and sometimes uncertain results. Since they all affect everyone, it is considered
a complex system. In complex systems, different actors hold diverse perspectives
on what defines an ideal solution, leading to a complex challenge. Achieving
a viable solution requires the active participation of all stakeholders, fostering
open dialogue, and collectively arriving at a solution. It is crucial to acknowl-
edge and recognize the diverse capabilities that each group of actors brings to
the table. The process begins by creating a collaborative environment where
all voices are heard and valued. This involves establishing effective communica-
tion channels and platforms that facilitate meaningful interactions among the
actors. By encouraging active participation and fostering a culture of respect
and inclusivity, the barriers to collaboration can be overcome. Furthermore, it is
essential to understand and appreciate the different perspectives and expertise
that each actor contributes. Each group may bring unique insights, experiences,
and knowledge that can enrich the problem-solving process. By recognizing and
leveraging these diverse capacities, a more comprehensive understanding of the
complex system can be achieved.
Building resilience in complex systems involves navigating the dynamic
nature of the system and progressing through several key phases [8]. These
phases are essential for understanding and addressing the challenges presented
by complex systems while fostering resilience: Detection, Prioritization, and
Reconfiguration.
Detection involves recognizing and identifying potential threats, vulnerabil-
ities, or disruptions that could impact the system’s stability and functionality.
This phase entails implementing robust monitoring mechanisms, data analy-
sis, and early warning systems to identify signs of stress or disturbances. By
promptly detecting potential risks, stakeholders can initiate timely responses and
interventions. Once potential risks or disruptions are detected, the next phase
is prioritization. This involves assessing and ranking the identified risks based
on their severity and potential impact on the system. Prioritization requires
careful analysis, considering the consequences of each risk on different aspects
of the system, such as its operations, resources, or stakeholders. By prioritiz-
ing risks, resources and efforts can be directed towards addressing the most
significant threats first, ensuring a more efficient allocation of resources. The
reconfiguration phase focuses on adapting and adjusting the system in response
to identified risks and priorities. It involves evaluating changes in the system’s
structure, processes, and strategies to enhance its resilience. This may include
modifying operational procedures, updating policies, investing in new technolo-
gies, or developing alternative pathways. Reconfiguration aims to strengthen the
system’s ability to withstand and recover from disruptions, improving its adap-
tive capacity and ensuring its continued functionality.
These three phases are interconnected and iterative, forming a continuous
cycle of resilience-building within the system. Detection allows stakeholders to
identify potential risks, which are then prioritized to guide decision-making and
resource allocation. Based on the priorities, the system undergoes reconfigura-
378 R. Motz et al.
tion to enhance its resilience. However, as the system evolves and new risks
emerge, the cycle restarts with detection, followed by revised prioritization and
reconfiguration.
By systematically progressing through these phases, stakeholders can proac-
tively identify, address, and mitigate risks, enhancing the system’s resilience and
its ability to withstand and recover from disturbances.
In this process, a crucial moment arises when determining the appropriate
time to halt these processes and declare, “Let’s consolidate this new process
and secure the necessary resources.” This decision-making entails leveraging the
available resources and those accessible through networks, identifying the timely
moment to acquire additional resources, and obtaining them to successfully con-
solidate the change. Implementing this process effectively involves reorganizing
and integrating new resources that may not currently be in existence, such as
new teachers or advanced technologies [26].
However, as highlighted by the recent study conducted by Borazon and
Chuang [3], further research in this field is imperative. Current studies on edu-
cational resilience predominantly concentrate on the resilience of students and
educators, overlooking other important aspects. Therefore, there is a need for
additional research to obtain a comprehensive understanding of resilience within
the broader educational context. Expanding the scope of research to encompass
various dimensions, such as institutional, systemic, and community resilience,
will enable a more nuanced and holistic perspective. By considering these addi-
tional facets, policymakers and stakeholders can develop more effective strategies
and interventions to address the multifaceted challenges faced by educational
systems.
4 The Role of Digital Systems in Education Resilience

We have identified various perspectives in the literature regarding the discus-
sion of the role of digital systems in education resilience. We have organized
this discussion into five axes, each of which is presented below. But initially,
it seems fair to acknowledge the fact that the integration of technology into
education has revolutionized learning processes, offering adaptability and conti-
nuity in the face of unforeseen events, such as the COVID-19 pandemic. Digital
tools and platforms have proven instrumental in enabling seamless transitions to
remote and online learning, ensuring educational activities persist despite phys-
ical constraints. However, with these advantages came challenges, particularly
in the realm of digital safety and literacy. To address these issues, at present, it
becomes essential to focus on fostering digital resilience in students, educators,
and the education system as a whole. This new discussion explores the dark side
of technology, highlighting the need for digital resilience and how it can empower
individuals to navigate and overcome the potential pitfalls of the digital world.
The Dark Side of Technology: Unveiling the Digital Threats. As digital
technology infiltrates educational systems, it brings along an array of challenges
that impact students and educators alike. Digital safety, a key component of
teachers’ digital literacy, encompasses issues ranging from mental and physical
health to social aspects and technical concerns associated with the use of informa-
tion and communication technologies. The rise of electronic threats (e-threats)
poses serious problems, including cyberbullying, hate speech, image manipula-
tion, and the spread of misinformation through echo chambers. These threats
jeopardize the well-being of individuals and the dynamics of digital communities,
demanding a proactive approach to address them effectively.
The Role of Digital Literacy in Building Resilience. Addressing the chal-

lenges posed by digital threats necessitates a comprehensive understanding of
digital literacy. Modern schools must equip students and educators to navigate
the digital world safely and responsibly. Digital literacy, therefore, should be
considered an integrating construct that prepares individuals to deal with the
risks and uncertainties of the digital landscape. It empowers learners to critically
evaluate information, recognize biases, and discern legitimate sources from mis-
information. Furthermore, digital literacy enables educators to support students
in developing the necessary skills and mindset to protect their well-being and
foster positive online relationships.
Fostering Digital Resilience: A Vital Educational Imperative. Digital
resilience emerges as a solution to combat the negative consequences of tech-
nology integration in education. It goes beyond mere technical proficiency; it
entails a proactive approach to address digital risks and challenges effectively.
Building digital resilience involves cultivating essential skills, such as media lit-
eracy, critical thinking, and emotional intelligence. By encouraging students and
educators to confront online negativity, misinformation, and manipulation, edu-
cational institutions empower them to maintain a positive and informed online
presence.
Integrating Digital Resilience in Educational Curricula. To truly pro-
mote digital resilience, educational institutions must prioritize it within their cur-
ricula and pedagogical approaches. Infusing digital resilience education through-
out various subjects and age levels can help instil a deep understanding of digital
safety and ethics. Interactive workshops, real-life case studies, and open discus-
sions on digital dilemmas can equip students with practical skills to handle digital
challenges effectively. Moreover, teachers must be trained to become role mod-
els in digital resilience, demonstrating appropriate online behaviour and guiding
their students through the complexities of the digital world.
Collaboration and Community Engagement: Strengthening Digital
Resilience. Building digital resilience requires a collaborative effort between
educational institutions, families, and the broader community. Parents and care-
givers play a critical role in supporting students’ digital learning journeys, ensur-
ing they comprehend the importance of digital safety and ethical behaviour.
Additionally, partnerships with industry experts, non-profit organizations, and
governmental bodies can enrich digital resilience initiatives with up-to-date
knowledge and resources. Moreover, the use of analytics and insights from digital
380 R. Motz et al.
systems can guide the development of targeted support mechanisms, ensuring

every learner receives the necessary assistance to thrive academically.
Embracing digital technology in education is essential for fostering adaptabil-
ity and continuity. Nevertheless, it comes with its share of challenges, including
digital threats that require proactive action. Cultivating digital resilience is cru-
cial for empowering students and educators to navigate the darker side of tech-
nology effectively. By integrating digital resilience education into curricula and
promoting collaboration with families and the community, educational institu-
tions can create a safer, inclusive, and empowered digital learning environment
for all.
5 Conclusions and Future Work
After reviewing various disciplinary perspectives and striving to embrace a plu-

ralistic viewpoint, two main approaches come to light: structural and dynamic
(Table 1). When considering decision-making moments, these approaches seem to
offer distinct goals: design and management. Design would involve contemplat-
ing the characteristics of the system, including the territory it operates within,
the networks it is connected to, and the academic communities it engages. Con-
versely, management could entail on-the-spot decision-making, resource mobi-
lization, and the activation of assets at different levels of governance. This pro-
cess is informed by accessing, processing, and interpreting current data. Design
would emphasize robustness and redundancy, and management would prioritize
rapidity and resourcefulness. In essence, designing resilient structures should
target mitigation, preparedness, and formative actions, whereas effective man-
agement requires the ability to detect, prioritize, and reconfigure in order to
respond and recover swiftly. In conclusion, by understanding and incorporat-
ing both structural and dynamic perspectives, organizations can better navigate
complex challenges and foster a more adaptable and resilient environment.
Table 1. Two perspectives of the resiliency of educational systems.

As for educational resilience, the focus has been mainly on the resilience of
students and educators, overlooking other crucial aspects. Therefore, there is a
pressing need for further research to gain a fuller understanding of resilience
within the educational context as a whole. By broadening the scope of the
research, which encompasses various dimensions such as institutional, systemic
and community resilience, a more nuanced and holistic perspective can be
obtained. This research should embrace a pluralistic view, which would require
not only methodological rigour but also a predisposition to engage in interdis-
ciplinary dialogue. This expanded research would enable policymakers, educa-
tors, and stakeholders to develop more effective strategies and interventions that
address the multifaceted challenges facing education systems. Therefore, future
research efforts should aim to explore and unravel the intricate dynamics of
resilience within the broader educational landscape, ensuring a comprehensive
approach encompassing all relevant factors.
By understanding and leveraging the opportunities presented by digital tech-
nologies, educational institutions can enhance their capacity to respond effec-
tively to challenges, promote equitable educational experiences, and facilitate
sustainable development in the education sector.
Acknowledgements. This research received funding from the National Agency for
Research and Innovation (ANII) in Uruguay under the code FSED 2 2021 1 169701.
References
1. Abdolrezapour, P., Jahanbakhsh Ganjeh, S., Ghanbari, N.: Self-efficacy and
resilience as predictors of students’ academic motivation in online education. PLoS
ONE 18(5), e0285984 (2023)
2. Becoña, E.: Resiliencia: definición, caracterı́sticas y utilidad del concepto. Revista
de Psicopatologı́a y Psicologı́a Clı́nica 11, 125–146 (2006)
3. Borazon, E.Q., Chuang, H.H.: Resilience in educational systems: a systematic
review and directions for future research. Int. J. Educ. Dev. 99, 102761 (2023).
https://doi.org/10.1016/j.ijedudev.2022.102761
4. Brewer, M.L., et al.: Resilience in higher education students: a scoping review.
Higher Educ. Res. Dev. 38(6), 1105–1120 (2019)
5. Bruneau, M., Reinhorn, A.: Overview of the resilience concept. In: Proceedings of
the 8th US National Conference on Earthquake Engineering, vol. 2040, pp. 18–22
(2006)
6. Camacho, N., Cortez, C., Carrillo, A.: La docencia universitaria ante la educacion
confinada: oportunidades para la resiliencia. Revista 21, 85–98 (2020)
7. Castello, O.R.: Resiliencia: un enfoque sistémico. GestioPolis, pp. 1–8 (2003).
http://www.gestiopolis.com/recursos6/Docs/Rrhh/resilenciaunenfoquesistemico.
htm
8. Chowdhury, M.M.H., Quaddus, M.: Supply chain resilience: conceptualization and
scale development using dynamic capability theory. Int. J. Prod. Econ. 188, 185–
204 (2017). https://doi.org/10.1016/j.ijpe.2017.04.015
9. Cimellaro, G.P., Reinhorn, A.M., Bruneau, M.: Framework for analytical quantifi-
cation of disaster resilience. Eng. Struct. 32(11), 3639–3649 (2010)
382 R. Motz et al.
10. Cote, M., Nightingale, A.J.: Resilience thinking meets social theory: situating social
change in socio-ecological systems (SES) research. Prog. Hum. Geogr. 36(4), 475–
489 (2012)
11. Dueñas Herrera, X., Godoy Mateus, S., Duarte Rodrı́guez, J.L., López Vera, D.C.:
La resiliencia en el logro educativo de los estudiantes colombianos. Revista Colom-
biana de Educación 76, 69–90 (2019)
12. Edwards, C.: Resilient Nation (2009)
13. Falk, D., Frisoli, P., Varni, E.: La importancia del bienestar docente para la salud
mental de los alumnos y los sistemas educativos resilientes. Revista Migraciones
Forzadas 66, 17–21 (2021). http://hdl.handle.net/10045/114365
14. Fath, B.D., Dean, C.A., Katzmair, H.: Navigating the adaptive cycle: an approach
to managing the resilience of social systems. Ecol. Soc. 20(2) (2015)
15. Forés, A., Grané, J.: La resiliencia. Crecer desde la adversidad. Plataforma Edito-
rial, Barcelona (2008)
16. Garcı́a-Crespo, F.J., Fernández-Alonso, R., Muñiz, J.: Academic resilience in Euro-
pean countries: the role of teachers, families, and student profiles. PLoS ONE 16(7),
e0253409 (2021). https://doi.org/10.1371/journal.pone.0253409
17. Gómez, G., Rivas, M.: Resiliencia académica, nuevas perspectivas de interpretación
del aprendizaje en contextos de vulnerabilidad social. Calidad en la educación 47,
215–233 (2017). https://doi.org/10.4067/S0718-45652017000200215
18. Jacobson, M.J., Levin, J.A., Kapur, M.: Education as a complex system: conceptual
and methodological implications. Educ. Res. 48(2), 112–119 (2019). https://doi.
org/10.3102/0013189X19826958
19. Llopis Orrego, M.d.M., Volakh Sokolova, E., Pérez Llopis, Á.: Resiliencia en
docentes universitarios: Afrontando retos en tiempos de pandemia. Actualidades
investigativas en educación 22(3), 130–164 (2022)
20. Martin, A.J., Marsh, H.W.: Academic resilience and academic buoyancy: Mul-
tidimensional and hierarchical conceptual framing of causes, correlates and cog-
nate constructs. Oxf. Rev. Educ. 35(3), 353–370 (2009). https://doi.org/10.1080/
03054980902934639
21. de las Naciones Unidas, O.: Informe de polı́ticas: La educación durante la covid-19
y después de ella. Technical report (2020)
22. Naidu, S.: Building resilience in education systems post-covid-19. Distance Educ.
42(1), 1–4 (2021)
23. OEA: Comunicado de prensa d-016/20 (2020)
24. Olsson, L., Jerneck, A., Thoren, H., Persson, J.: Why resilience is unappealing
to social science: Theoretical and empirical investigations of the scientific use of
resilience. Sci. Adv. (2015). https://doi.org/10.1126/sciadv.1400217
25. Organisation for Economic Co-operation and Development: PISA 2015 Results:
Excellence and Equity in Education (Vol. 1). OECD Publishing (2016)
26. Pickernell, D.G.: El desafı́o de construir resiliencia en sistemas complejos. Escuela
de Invierno. Centro de Estudios Fundación Ceibal. YOUTUBE (2021). https://
www.youtube.com/watch?v=OplD9BFEGkI
27. Rieiro, A., Castro, D., Pena, D., Veas, R., Zino, C.: Entramados comunitarios y sol-
idarios para sostener la vida frente a la pandemia: ollas y merenderos populares en
uruguay 2020. informe final. Udelar (2021). https://hdl.handle.net/20.500.12008/
34243
28. Román, F., et al.: Resiliencia de docentes en distanciamiento social preventivo
obligatorio durante la pandemia de covid-19. J. Neuroeduc. 1(1), 76–87 (2020)
29. Rutter, M.: Privación materna, 1972–1978: Nuevos hallazgos, nuevos conceptos,
nuevos enfoques. Vol. 50, No. 2 (junio de 1979), pp. 283–305 (1979). https://www.
jstor.org/stable/1129404
30. Stack, M.: Responding to the covid-19 pandemic: university rankings or co-
operatives as a strategy for developing an equitable and resilient post-secondary
education sector? Int. Rev. Educ. 67(1), 127–144 (2021)
31. Tenorio-Vilchez, C., Sucari, W.: Entender la resiliencia docente. una mirada sis-
temática. Revista Innova Educación 3(3), 187–197 (2021)
32. UNESCO: Artificial intelligence in education: challenges and opportunities for sus-
tainable development. Working papers in education policy 07, United Nations Edu-
cational, Scientific and Cultural Organization (2019)
33. UNESCO: Construir sistemas educativos resilientes durante el periodo
de la covid-19 (2021). https://www.unesco.org/es/articles/construir-sistemas-
educativos-resilientes-durante-el-periodo-de-la-covid-19
34. UNESCO-ILO Task Force on Teachers for Education 2030: Supporting teachers in
back-to-school efforts: Guidance for policy makers. Technical report (2020)
35. Ávila Valdiviezo, B.Y., Carbonell-Garcı́a, C.E., Salas-Sánchez, R.M.: Resiliencia y
desempeño docente en tiempo de pandemia en instituciones educativas de secun-
daria peruana. Revista Arbitrada Interdisciplinaria Koinonı́a 6(12) (2021). https://
dialnet.unirioja.es/servlet/articulo?codigo=8019931
36. Walker, B., Holling, C.S., Carpenter, S.R., Kinzig, A.: Resilience, adaptability and
transformability in social-ecological systems. Ecol. Soc. 9, 5 (2004)
37. Wang, Y.: Building teachers’ resilience: practical applications for teacher education
of china. Frontiers in Psychology 12 (2021). https://doi.org/10.3389/fpsyg.2021.
738606, https://www.frontiersin.org/articles/10.3389/fpsyg.2021.738606
ChatGPT-4 as a Tool for Reviewing
Academic Books in Spanish
Jonnathan Berrezueta-Guzman1 , Laura Malache-Silva2(B) ,

and Stephan Krusche1
1
Technical University of Munich, Munich, Germany
{s.berrezueta,krusche}@tum.de
2
CEDIA, Cuenca, Ecuador
laura.malache@cedia.org.ec
Abstract. This study evaluates the potential of ChatGPT-4, an arti-

ficial intelligence language model developed by OpenAI, as an editing
tool for Spanish literary and academic books. The need for efficient and
accessible reviewing and editing processes in the publishing industry has
driven the search for automated solutions. ChatGPT-4, being one of the
most advanced language models, offers notable capabilities in text com-
prehension and generation. In this study, the features and capabilities
of ChatGPT-4 are analyzed in terms of grammatical correction, stylistic
coherence, and linguistic enrichment of texts in Spanish. Tests were con-
ducted with 100 literary and academic texts, where the edits made by
ChatGPT-4 were compared to those made by expert human reviewers
and editors. The results show that while ChatGPT-4 is capable of making
grammatical and orthographic corrections with high accuracy and in a
very short time, it still faces challenges in areas such as context sensitiv-
ity, bibliometric analysis, deep contextual understanding, and interaction
with visual content like graphs and tables. However, it is observed that
collaboration between ChatGPT-4 and human reviewers and editors can
be a promising strategy for improving efficiency without compromising
quality. Furthermore, the authors consider that ChatGPT-4 represents a
valuable tool in the editing process, but its use should be complementary
to the work of human editors to ensure high-caliber editing in Spanish
literary and academic books.
Keywords: Artificial intelligence in editing · Language models ·

Automated editing tools · Natural Language Processing · Text revision
with AI · Language technologies for editing · Intelligent writing
assistants · AI in the publishing industry · human-AI interaction
1 Introduction
ChatGPT-4, developed by OpenAI, is an advanced iteration of artificial intelli-
gence language models. It utilizes extensive datasets and deep learning algorithms
to comprehend and generate coherent text [1]. In the publishing industry, text
https://doi.org/10.1007/978-981-99-7353-8_29
ChatGPT-4 as a Tool for Reviewing Academic Books in Spanish 385
review and editing is a critical process that involves the exploration and correction
of manuscripts to enhance clarity, coherence, and grammatical accuracy [2]. The
implementation of ChatGPT-4 in the editing of Spanish academic books could rev-
olutionize the industry in several ways. Firstly, ChatGPT-4 can automate the cor-
rection of grammatical and spelling errors, increasing the efficiency of the editing
process. Furthermore, its capability to analyze large volumes of text could facili-
tate the detection of inconsistencies and enhance coherence in lengthy texts [3].
However, language models like ChatGPT-4 still face challenges in terms
of contextual sensitivity and tone, necessitating a collaborative approach with
human reviewers and editors to maintain quality [4,5]. When combined with
human editors, ChatGPT-4 can enable a more agile and rigorous editing pro-
cess, especially important in the academic context where accuracy and clarity
are crucial. Additionally, ChatGPT-4 and similar models must be adequately
trained and adapted to the particularities of the Spanish language, to ensure
their effectiveness in editing texts in this language [6].
This paper documents a research investigation that examined the perfor-
mance of ChatGPT-4 in revising 100 academic books and compares the derived
results with those produced by human reviewers (two for each book). The struc-
ture of this paper is organized as follows: Sect. 2 introduces studies similar to this
research and provides a comparison. The applied methodology for this study is
elaborated in Sect. 3. The results derived from this study are presented in Sect. 4,
and a discussion regarding these results is conducted in Sect. 5. Lastly, Chap. 6
brings forth the conclusions drawn from this study and proposes potential future
work in the same vein.
2 Related Work
In the landscape of using AI for academic purposes, various studies have been
conducted. One pertinent study in the realm of AI application in academic edit-
ing is the one conducted by Wang et al. [9]. They probed into the efficacy of Chat-
GPT for generating effective boolean queries in systematic literature reviews
through multiple experiments. The results echoed the value of ChatGPT as a
potent tool for conducting systematic reviews. However, they simultaneously
highlighted challenges associated with striking a balance between automation
and the critical analysis intrinsic to human-verified, documented work.
Checco et al., [10], explore the possibility of employing artificial intelligence
(AI) as a tool to aid or automate the process of peer-review. The authors con-
struct a machine-learning mechanism, which is trained using 3300 conference
papers, and the results exhibit that the system can aptly foresee the results of
the peer-review procedure based on merely the manuscript’s superficial traits.
The study underscores the advantages of such AI applications, such as improved
efficiency and enhanced understanding of the reviewing procedure. Nevertheless,
it also emphasizes the requirement for addressing any prospective prejudices and
ethical dilemmas related to these instruments.
In an examination of scientific manuscripts submitted to various artificial
intelligence conferences, a comprehensive analysis was conducted. It comprised
386 J. Berrezueta-Guzman et al.
semantic, lexical, and psycholinguistic dissections of the manuscripts’ full text,

juxtaposed with the outcomes of the peer review process. The study determined
that manuscripts that were accepted had lower readability scores and utilized
more scientific and AI-related jargon than the rejected ones. Additionally, the
accepted manuscripts employed less common, more abstract vocabulary, usually
acquired at an older age. When analyzing the references within the manuscripts,
it was observed that accepted submissions were more likely to cite identical
publications-a fact reinforced by pairwise comparisons of manuscript word con-
tent, indicating greater semantic similarity among accepted works. Lastly, a
prediction of the manuscripts’ peer review outcomes was attempted based on
their word content. The study concluded that usage of terms related to machine
learning and neural networks was positively correlated with acceptance, whereas
words tied to logic, symbolic processing, and knowledge-based systems were neg-
atively correlated [11].
Another recent work explored ChatGPT’s potential role in peer review pro-
cesses in academia, examining aspects such as the roles of reviewers and editors,
review quality, reproducibility, and the social and epistemic implications of peer
reviews. It was found that while LLMs like ChatGPT could assist in creating con-
structive reports and addressing review shortages, they also introduced potential
biases, confidentiality issues, and concerns about reproducibility. Moreover, there
are potential unforeseen social and epistemic consequences within academia due
to partially delegating editorial work to LLMs. Despite the promise, the study
calls for caution in using LLMs in academic contexts due to the possibility of
amplifying existing biases and access inequalities [15].
The study [16] investigated ChatGPT’s competence in performing tasks
related to bibliometric analysis, comparing the chatbot’s outputs with those
from a recent bibliometric study on a similar topic. Results demonstrated signif-
icant disparities and raised questions about the tool’s reliability in this specific
domain, thereby suggesting that researchers should proceed with caution when
employing ChatGPT for bibliometric studies.
3 Methodology
In contrast to related work, this study focuses on analyzing the similarities

and differences between the content revisions of academic books conducted by
ChatGPT-4 and human reviewers. This will help us determine the role this tool
might play in the future of universities and literary publishing houses. To do
this, we have selected a sample of 100 Spanish academic books that have been
previously reviewed (some are already published) from January 2021 to June
2023 across 15 topics.
3.1 Books Selection
Table 1 displays a thematic classification of the books to be analyzed in this

study. It is also noticeable that the demand for life science books surpasses that
of other subjects. These areas of study are considered highly sensitive. Therefore,
in these books, we have decided to place greater emphasis on the analysis of
ChatGPT, but from a more ethical perspective.
Table 1. Distribution of books analyzed by topic, average page count, and percentage
of graphical content.
Topics Number of Average number Percentage of graphical

books of pages content
Advanced Mathematics 5 150 22 % (mostly equations)
Architecture 3 80 42 %
Botany 2 75 12 %
Computer Sciences 7 60 5 % (code snipes)
Culture 7 80 12 % (mostly photographies)
Economics 5 90 6%
Education 7 100 5%
History 4 150 9%
Innovation 3 40 14 %
Law and Legislation 8 65 4%
Medicine 15 100 13 %
Odontology 10 70 42 %
Pharmacology 9 110 37% (mostly formulas)
Psychology 10 130 5%
Sociology 5 120 3%
3.2 Preparation of the Texts

The text from these books will be prepared for processing. In certain instances,
this involves digitizing the text if it is not already in a digital format, and
ensuring that the text is free from any Optical Character Recognition (OCR)
errors or any extraneous elements (such as watermarks) that might interfere with
the text analysis [7].
3.3 Process of Pair Review and Edition with Reviewers and Editors
Each of these steps requires a careful, meticulous approach, and each plays a
crucial role in ensuring that the final published book is accurate, engaging, and
valuable to its intended audience. Note that depending on the size and structure
of the publishing house, some or all of these tasks might be carried out by
different individuals or even different departments.
Manuscript Assessment: In this first stage, the editor assesses the manuscript
for its content, structure, and alignment with the intended audience. They pro-
vide an overall evaluation and make suggestions for improvement. This step often
involves a lot of collaboration with the author.
Substantive or Developmental Editing: This is a deep, intensive editing

phase focused on the content, structure, organization, and presentation of the
manuscript. The editor may suggest reordering sections, rewriting passages,
expanding or condensing content, and more. This stage requires a deep under-
standing of the subject matter.
Copy Editing: This is where the editor reviews the manuscript for grammar,
punctuation, syntax, and consistency in style and voice. They will also check for
consistency in facts and may cross-verify references and citations. The editor will
ensure that the manuscript adheres to a specific style guide (like APA, Chicago,
MLA, etc.).
Proofreading: This is the final stage of editing, which involves checking for
typographical errors, spelling mistakes, and any inconsistencies missed during
the copy-editing stage. The text is polished to be print-ready at this stage.
Indexing: While not editing per se, creating an index is an important part
of preparing academic books. It involves listing the topics, names, and places
mentioned in the book, along with page numbers, to help readers navigate the
content.
Layout and Design: Although this is typically a designer’s job, an editor
might have some involvement in deciding the overall look and layout of the
book, including the typography, cover design, and the arrangement of visual
elements (like charts, graphs, or images).
Review: In academic publishing, the manuscript typically goes through a
peer review process, where other experts in the field review the work for accu-
racy, relevance, and originality. The editor manages this process, communicates
feedback to the authors, and oversees any revisions.
Approval and Publication: Once the manuscript has been thoroughly edited
and reviewed, the editor gives their final approval for publication.
3.4 Process of Review with ChatGPT-4
Each book was processed through ChatGPT-4 and the workflow is explained
in Fig. 1. The AI conducted a comprehensive review and revision of the text,
correcting any grammatical or spelling errors, and endeavored to improve the
coherence and clarity of the text. Furthermore, the AI also analyzed the syntactic
structure of the sentences to ensure they conformed to grammatical rules. It
assessed the use of tenses, punctuation, and other grammatical components for
correctness and consistency throughout the text.
In addition to this, ChatGPT-4 analyzed the semantic aspects of the text.
This included checking for the appropriate use of vocabulary and terminology,
particularly important for subject-specific jargon in academic texts [8]. The AI
also aimed to evaluate the logical flow and structure of the content, checking
for any inconsistencies or contradictions in the argument or presentation of the
material.
Another critical aspect of this process was the evaluation of stylistic ele-
ments such as tone, voice, and overall readability of the text. While maintaining
the original author’s voice, the AI attempted to enhance the text’s readability,
ensuring that it was accessible and engaging for the intended audience.
In the next step, ChatGPT-4 performed an intertextual analysis, checking for
potential plagiarism issues by comparing the text with a large corpus of academic
literature. It also ensured correct citation and referencing as per the academic
style guide relevant to each book. Finally, a human editor will randomly select
a book from each topic, analyze it with the assistance of ChatGPT, and then
compare the time taken against another editor who will not use ChatGPT. The
results are presented in Table 2.
Fig. 1. Workflow to conduct the review of the selected books by ChatGPT before the
comparison of results with the review made by the human reviewers and editors.
4 Results
In this study, we applied ChatGPT-4 to the task of editing 100 academic books
in the Spanish language, covering a variety of academic fields. The results present
a compelling case for the implementation of AI in the editorial process. Over-
all, ChatGPT-4 showed a high proficiency in identifying and correcting basic
grammatical and spelling errors, demonstrating its value as a first-pass editing
tool.
Table 2 displays the average time that ChatGPT required to analyze the text
of the selected book collection. It’s evident that topics primarily composed of
text require less time for ChatGPT to interpret, and the total number of errors
is lower compared to books with more graphical content like tables, figures,
formulas, and equations.
Table 2. Average time spent by ChatGPT to analyze the book collection, total errors
recorded in the books by topic, and the time saved for a human editor when he collab-
orated with ChatGPT to analyze one random book of each topic.
Book Topics ChatGPT Misunderstandings and Saved time

invested time errors by ChatGPT
Advanced Mathematics 88 min 142 (5 books) 9%
Architecture 56 min 35 (3 books) 8%
Botany 49 min 12 (2 books) 32 %
Computer Sciences 328 min 28 (7 books) 14 %
Culture 252 min 27 (7 books) 25 %
Economics 194 min 28 (5 books) 38 %
Education 132 min 29 (7 books) 45 %
History 115 min 31 (4 books) 51 %
Innovation 95 min 54 (3 books) 26 %
Law and Legislation 74 min 65 (8 books) 47 %
Medicine 977 min 58 (15 books) 37 %
Odontology 630 min 43 (10 books) 12 %
Pharmacology 742 min 32 (9 books) 29 %
Psychology 257 min 23 (10 books) 47 %
Sociology 139 min 21 (5 books) 55 %
We split the results by analyzing limitations and the qualitative comparison

with a human editor explaining advantages and disadvantages and finally, we
pointed out in which sense ChatGPT and a human editor are similar.
4.1 Limitations
Although ChatGPT is a powerful tool, there are still several limitations to be

taken into account when using it for book reviewing [13,14,18]. These limitations
have been considered in some approaches to avoid academic misconduct [19].
1. Token Limitation: ChatGPT has a limit to the number of tokens (units

of text) it can process at one time. For GPT-4, this limit is 2048 tokens.
Therefore, it may not be able to analyze very long texts in their entirety in
a single pass. This was time-consuming afterward because we had to extract
only the text from all the collections of books.
2. Inability to Interpret Visual Elements: ChatGPT is a natural language
processing model and cannot analyze or interpret visual elements such as
graphs, tables, images, or diagrams. This means it cannot verify the accuracy
or relevance of such elements within the text, which results in several misun-
derstandings in books that had a considerable amount of this content (e.g.
Advanced Mathematics, Architecture, Odontology).
3. Inability to Access External Databases in Real-Time: ChatGPT can-
not access external databases or browse the internet in real-time to verify
facts or references. Its knowledge is based solely on the data it was trained
with and does not update with new or real-time information.
4. No File System Interactivity: ChatGPT cannot interact with a computer
or network’s file system, meaning it cannot open, close, read, or write files
directly. This must be done by a human to correct the text that was reviewed
by ChatGPT.
5. Inability to Interpret Markup Languages: ChatGPT may struggle to
interpret and work with markup languages such as HTML, LaTeX, or XML
which are often used in the production of academic documents and books.
This must be done by a human and correct the text that was reviewed by
ChatGPT on the source files.
4.2 Qualitative Comparison of the Results of the Review

with ChatGPT-4 with the Review by Human Reviewers
ChatGPT, as an artificial intelligence language model, can offer certain advan-
tages compared to human reviewers of academic books. However, it’s important
to note that it does not entirely replace human editing and reviewing but can
complement it. The key advantages of ChatGPT compared to a human editor
include:
1. Efficiency: ChatGPT can process and analyze large volumes of text in a
short period, which can enhance efficiency in reviewing lengthy documents.
Even though books with many graphical elements combined with text took
longer for ChatGPT to analyze and subsequently required correction by the
human editor, there is still a noticeable time-saving benefit (e.g. Advanced
Mathematics - 9% time saved).
2. Availability: As an artificial intelligence program, ChatGPT is available
24/7, does not need breaks or time off, and can work at the same constant
speed for extended periods.
3. Basic error correction: ChatGPT can swiftly detect and correct grammat-
ical, orthographic, and punctuation errors. However, its effectiveness depends
on the language and the specific characteristics of the text.
4. Consistency: ChatGPT can maintain consistent coherence in the applica-
tion of grammatical and stylistic rules, which is particularly useful for lengthy
documents where maintaining consistency can be a challenge for human
reviewers.
While artificial intelligence models like ChatGPT have made remarkable
strides in language processing tasks [12], there are still several areas where human
reviewers and editors outperform AI when it comes to reviewing academic books:
1. Contextual Understanding: Human reviewers can better understand

complex contextual information that AI might miss. They can interpret
nuances, subtleties, and complexities within the text that an AI might not
fully comprehend.
2. Style and Tone: Humans have a deeper understanding of the author’s style
and tone, which is critical for preserving the author’s unique voice in academic
writing. Humans can also detect and correct inappropriate shifts in tone more
effectively than AI.
3. Logic and Argumentation: Human reviewers can evaluate the logical flow
of arguments and the presentation of ideas, which is vital in academic books.
They can recognize whether the author’s argumentation is solid, consistent,
and properly substantiated.
4. Subjective Judgement: Humans can make subjective judgments that AI
can’t. This includes evaluating whether the text is engaging, interesting, or
appropriate for the intended audience.
5. Ethical Considerations: Human reviewers are better at detecting ethical
issues in the text, such as cultural insensitivity, bias, or potential plagiarism,
that may not be readily apparent to AI.
6. Adherence to Guidelines: Human reviewers can ensure adherence to spe-
cific academic or publishing guidelines, which may involve complex or sub-
jective elements that AI could struggle to interpret.
4.3 Quantitative Comparison of the Results of the Review

with ChatGPT-4 with the Review by Human Reviewers
In future versions, two areas where ChatGPT should excel would be:
1. Processing of multiple documents: If ChatGPT could handle multiple

documents simultaneously, it would have a significant advantage over a human
editor. At the moment both have this limitation.
2. Style and voice: The human editor acknowledges that identifying and pre-
serving an author’s unique style and voice can be challenging. ChatGPT also
faces this challenge in certain cases.
5 Discussion
Table 2 illustrates that based on the content percentage (text, figures, tables,
equations, and formulas) and the length of the book, ChatGPT can be immensely
valuable for various stages of a traditional editorial process.
A book containing a mix of text with images, equations, tables, or formu-
las can be confusing for ChatGPT, especially when the text references content
that ChatGPT cannot read. Therefore, for primarily text-based books, Chat-
GPT is faster and makes fewer errors (e.g. Law and Legislation, Psychology and
Sociology). This means a human editor would spend less time addressing these
misunderstandings that ChatGPT encounters.
Based on the results Table 2, Fig. 2 provides a summarization of the spe-

cific stages in the editorial process where ChatGPT can lend support to human
editors, along with the extent of its potential contribution.
The calculation of how beneficial ChatGPT can be in subsequent stages is
based on the average time it took a human editor to review a randomly selected
book from each topic using ChatGPT, compared to the time it took another
human editor to perform the same task without ChatGPT.
– 1. Manuscript Assessment: ChatGPT could potentially assist in con-
ducting an initial assessment of a manuscript by providing a general analysis
of the text, including measures of readability, stylistic metrics, and possible
inconsistencies in tense, style writing, or voice (in the best case). It could also
help in identifying overtly complex sections that may need to be rewritten for
clarity, which could give editors a high-level view of the manuscript’s quality
and make a decision about its possible publication. However, ChatGPT can-
not replace an editorial committee. Depending on their policies, collections,
and audience, the committee will decide on the relevance and potential for
publication.
– 3. Copy Editing: This is perhaps the area where ChatGPT could offer the
most assistance to editors. The AI could be used to spot and correct issues
related to grammar, punctuation, and syntax. It could also help ensure con-
sistency in terms of the use of terminologies across the manuscript. However,
this varies depending on the book. In multi-author books, it’s challenging for
both ChatGPT and a human editor to adjust the style and voice to a single
standard.
– 4. Proofreading: ChatGPT could serve as an effective tool for spotting
typographical errors and minor mistakes that may have been overlooked dur-
ing the previous editing stages. Given its ability to process large amounts of
text quickly, it could be especially useful for the final proofreading of lengthy
academic texts. However, at this stage, it’s crucial to provide specific instruc-
tions to only analyze these types of errors. Otherwise, ChatGPT begins to
overemphasize its observations regarding style and voice, which can lead to a
cycle of corrections upon corrections.
While ChatGPT can significantly aid some parts of the editorial process
(Fig. 2), it doesn’t replace the need for a human editor or reviewer. The AI
tool lacks a deep understanding of context, cannot appreciate the nuances of
language in the same way a human can, and does not possess the ability to
fact-check content or verify the validity of the arguments presented with the
references.
– 2. Double-blind review: The peer review process involves managing the
exchange of complex, specialized feedback between experts in a specific field.
While ChatGPT can generate text and summaries, it lacks the deep, field-
specific knowledge and human judgment required to understand and manage
this feedback. It also can’t ensure that the feedback has been adequately
addressed in revisions.
Fig. 2. Summary of the editorial process stages where ChatGPT can assist human
editors and the degree of its potential contribution.
– 5. Indexing: Creating an index for a book involves understanding the core

themes, concepts, and key figures within a text and associating them with
specific page numbers. While ChatGPT can identify keywords and themes, it
lacks an understanding of the relative importance of these elements within the
broader context of the book. Moreover, it is currently incapable of interacting
with pagination information, which is crucial for this task.
– 6. Layout and Design: This stage involves aesthetic judgments and an
understanding of visual design principles, which are outside the capabilities
of a text-based AI model like ChatGPT. It can’t process or make decisions
about visual elements such as typography, cover design, or the arrangement
of charts, graphs, or images. And even this is a problem when the text refers
to a figure or table. ChatGPT just makes some assumptions and this leads
to bad analysis.
– 7. Approval and Publication: The final approval for publication involves
making judgments on the manuscript as a whole. This requires a holistic
understanding of the content, context, and relevance of the work, which is
beyond the capabilities of current AI models. This stage may also involve tasks
such as communicating with publishing services and making decisions about
publication timelines, tasks that AI models like ChatGPT are not currently
equipped to handle.
Based on the evidence gathered throughout this study, coupled with insights
derived from an extensive examination of relevant literature, it is evident that
ChatGPT-4, albeit a highly advanced AI language model, should be optimally
deployed as an augmentative instrument to enhance the capabilities of human
editors, rather than serve as a complete replacement. Its efficacy primarily lies
in its ability to support and streamline certain aspects of the editing process,
while crucial tasks that require nuanced understanding, holistic judgment, and
sophisticated design aesthetics continue to necessitate the unique expertise of
human professionals.
One of the editors involved in this study highlighted a prevalent issue where
ChatGPT may not offer much assistance. “Spanish, like many languages in Latin
America, is replete with regionalisms. Often, it is necessary to engage with the
author directly to clarify their intended meaning without altering the essence of
their idea, thereby preserving the context of their research”.
“At times, it’s more advantageous to first analyze texts from the perspective
of their region of origin before neutralizing them and enhancing clarity. This
scenario occurs especially in social sciences or art books, where authors take
more liberty in describing situations, artworks, or investigations”.
“Another aspect that often slows down the editing process is the presence
of formulas and symbols, common in mathematical, physical, or chemical texts.
Since an editor may not specialize in these precise scientific branches, they often
need to return this portion of the text to the author for verification of data
before finalizing the editing process”.
This complex interplay between contextual understanding, content authen-
ticity, and technical validation, further underscores the limitations of AI tools
like ChatGPT and the indispensable role of human reviewers and editors.
6 Conclusions
ChatGPT undoubtedly offers the potential for automating aspects of the aca-
demic book-reviewing process, but its technical limitations underscore the con-
tinuing necessity for human involvement. The model excels at preliminary editing
and proofreading tasks, such as correcting grammar and spelling mistakes, and
making basic stylistic improvements. However, the indispensable role of human
oversight in the process can’t be overlooked.
Despite ChatGPT’s powerful capabilities, it faces substantial challenges such
as limited deep contextual understanding, interpretation of subtle nuances, and
the absence of human-like editorial judgment. Consequently, an AI-edited text
might not match the standards of one edited by a human. Moreover, the model’s
inability to interact with visual content like graphs and tables, process real-
time data, or understand markup languages further cements the need for human
editors.
Another area of concern is the model’s ability to handle lengthy texts con-
sistently, treat sensitive content appropriately, and prevent the propagation of
errors. These issues are difficult for AI to manage and they underline the require-
ment for a human touch in the editing process.
AI tools such as ChatGPT have further limitations when used in bibliometric
analysis, as evidenced by comparisons with traditional bibliometric studies. Dis-
crepancies between results indicate that trustworthiness and reliability in such
a context are still not at the desired levels.
These limitations are also shared by other Large Language Models (LLMs)
such as Bard. However, Bard could potentially assist in verifying the truthfulness
of a book’s content [17]. This is a promising area of investigation that we plan to
explore in future research. A possible approach would be to compare how ’Bard’
or ’Llama 2’ respond to this same experiment and highlight the comparison of
the results between these LLMs.
In conclusion, while AI exhibits substantial potential in aiding the editorial
process, it is not yet poised to fully replace human editors. The optimal app-
roach may involve a synergistic relationship between AI and human reviewers,
integrating the strengths of both to ensure high-quality, contextually nuanced,
and ethically sound editing in academic books. As we move forward, the evolv-
ing landscape of AI in academia will necessitate careful consideration of how to
effectively balance human and AI contributions to ensure the highest standards
of academic integrity.
Acknowledgements. The authors of this work would like to thank the CEDIA edi-
torial team for allowing them to conduct this research and for welcoming the analysis
of this Artificial Intelligence tool with the works that have been reviewed since the
inception of the editorial.
References
1. Brown, T., et al.: Language models are few-shot learners. Adv. Neural. Inf. Process.
Syst. 33, 1877–1901 (2020)
2. Einsohn, A., Schwartz, M.: The Copyeditor’s Handbook: A Guide For Book Pub-
lishing And Corporate Communications, 4th edn. University of California Press,
USA (2019)
3. Indurkhya, N., Damerau, F.J. (Eds.): Handbook of natural language processing.
Vol. 2. CRC Press, (2010)
4. Bender, E.M., Koller, A.: Climbing towards NLU: On meaning, form, and under-
standing in the age of data. In: Proceedings of the 58th Annual Meeting of the
Association for Computational Linguistics, pp. 5185–5198. (2020)
5. Kasneci, E., et al.: ChatGPT for good? On opportunities and challenges of large
language models for education. Learn. Individ. Diff. 103, 102274 (2023)
6. Tiedemann, J., Agić, Z.: Synthetic treebanking for cross-lingual dependency pars-
ing. J. Artif. Intell. Res. 55, 209 (2016)
7. Wu, J., et al.: Recursively summarizing books with human feedback.
arXiv:2109.10862 (2021)
8. Fang, T., et al.: Is chatGPT a highly fluent grammatical error correction system?
A comprehensive evaluation. arXiv:2304.01746 (2023)
9. Wang, S., Scells, H., Koopman, B., Zuccon, G.: Can ChatGPT write a good boolean
query for systematic review literature search?. arXiv:2302.03495 (2023)
10. Hyland-Wood, B., Gardner, J., Leask, J., Ecker, U.K.: Toward effective government
communication strategies in the era of COVID-19. Hum. Social Sci. Commun. 8(1)
(2021)
11. Vincent-Lamarre, P., Larivière, V.: Textual analysis of artificial intelligence
manuscripts reveals features associated with peer review outcome. Quant. Sci.
Stud. 2(2), 662–677 (2021)
12. Bukar, U., Sayeed, M.S., Razak, S.F.A., Yogarayan, S., Amodu, O.A.: Text anal-
ysis of chatGPT as a tool for academic progress or exploitation. http://ssrn.com/
abstract=4381394 Accessed July 2023
13. OpenAI: GPT-4 Technical Report. arXiv:2303.08774 (2023)
14. Tamkin, A., Brundage, M., Clark, J., Ganguli, D.: Understanding the capabilities,
limitations, and societal impact of large language models. arXiv:2102.02503 (2021)
15. Hosseini, M., Horbach, S.P.: Fighting reviewer fatigue or amplifying bias? Con-
siderations and recommendations for use of ChatGPT and other Large Language
Models in scholarly peer review. Res. Integr. Peer Rev. 8(1), 4 (2023)
16. Farhat, F., Sohail, S.S., Madsen, D.Ø.: How trustworthy is ChatGPT? The case of
bibliometric analyses. Cogent Eng. 10(1), 2222988 (2023)
17. Siad, S.M.: The Promise and Perils of Google’s Bard for Scientific Research (2023)
18. Hill-Yardin, E.L., Hutchinson, M.R., Laycock, R., Spencer, S.J.: A Chat (GPT)
about the future of scientific publishing. Brain Behav. Immun. 110, 152–154 (2023)
19. Berrezueta-Guzman, J., Krusche, S.: Recommendations to create programming
exercises to overcome ChatGPT. In: 2023 IEEE 35th International Conference
on Software Engineering Education and Training (CSEE&T), Tokyo, Japan, pp.
147–151 (2023). https://doi.org/10.1109/CSEET58097.2023.00031
Towards Enhancing Novice Learners’
Programming Skills: Lessons Learned
from a MOOC-Based Approach
Julio C. Caiza(B) , Danny S. Guamán , and Franklin L. Sanchez
Escuela Politécnica Nacional, Quito 17-01-2759, Ecuador

{julio.caiza,danny.guaman,Franklin.sanchez}@epn.edu.ec
Abstract. Nowadays, programming is one of the essential competencies that cit-

izens must have. This skill, together with other digitals, could help to carry out the
urgent digital transformation in Latin-America. Ecuador is not the exception, and
it has an even more accentuated problem because it has a shortage of programming
skills. To bridge this gap, Massive Open Online Courses (MOOCs) may provide
quickly and effectively with such skills. This paper presents the development
of the MOOC “De Principiante a Avanzado en C: Primeros Pasos”, in Spanish,
and the evaluation of its impact on the learning process of 293 novice students.
It incorporates various state-of-the-art resources, including video explanations,
online programming environments, automated grading, and gaming-based activ-
ities. We collected students’ perceptions about skill improvement and the impact
of the resources on their learning process. The results highlight the effective-
ness of the MOOC. The students report significant improvement in their pro-
gramming skills, with 232 of them passing the course, and consider that most of
the resources are helpful. However, challenges including the need for personal-
ized feedback, improved communication between students and instructors, and
enhanced gamification-based were identified. Additionally, this study provides
insights into enhancing the design and delivery of future programming courses.
Keywords: Programming · MOOC · teaching · education · feedback
1 Introduction
In today’s digital society, it is imperative to develop competences that enable citizens to
actively participate in it. This is recognized by the European Union, which has included
Programming in the DigComp 2.2 framework in 2022 [1]. Similarly, the Economic Com-
mission for Latin America and the Caribbean (ECLAC) recognizes the need for digital
skills for organizational transformation and modernization. According to ECLAC [2],
three levels of skills are required: basic, intermediate, and advanced, with Programming
being an advanced skill.
In the Ecuadorian context, the modernization of business processes in diverse produc-
tion areas requires professionals with programming skills, as recognized by the Prospec-
tive Agenda for the Knowledge Sector and Human Talent [3]. Yet, these skills are still
https://doi.org/10.1007/978-981-99-7353-8_30
Towards Enhancing Novice Learners’ Programming Skills 399
scarce at the local level and do not meet the demands of the organizations. For exam-
ple, the executive director of the Ecuadorian Chamber of Innovation and Technology
(CITEC) has estimated that there is a deficit of 9,000 programmers per year in Ecuador
[4].
In addition, Casacuberta et al. [5] further state that programming skills not only help
with job training but also help to develop creativity, teamwork and to work independently
of geographical boundaries. Locally, the Project “Ecuador 2030” [6] further states that
programming is a basic and necessary skill, as it fosters other important cognitive skills
for organizations, such as logical reasoning (the ability to deduce unknown information
based on those known) and problem-solving (the ability to detect failures and propose
effective solutions).
In this scenario, efforts are required to develop Programming skills massively, reach-
ing the largest number of people, and effectively, establishing learning resources and
strategies to ensure the development of these skills. Massive Open Online Courses
(MOOCs) are a potential means to address this gap. MOOCs are online courses that
allow for widespread access to learning resources and materials. They have been used in
multiple domains (e.g., [7–9]), as people can access resources from reputable institutions
and experts globally, overcoming geographical boundaries.
In the Programming discipline, certain learning resources and strategies common
to other disciplines are required in MOOCs, such as video explanations of concepts or
multiple-choice quiz assessments. However, in this domain, additional resources and
strategies are also required. Learning resources, such as online programming environ-
ments [10], support students to edit, compile and test programs without leaving the
learning platform. Also, some strategies, such as automated testing and grading [11]
and gaming-based activities [12] help students, particularly novices, to understand and
develop their skills quickly and to keep them motivated in their learning process.
To contribute to this effort, in this paper, we present the development of the MOOC
“De Principiante a Avanzado en C: Primeros Pasos”, which has been developed in Span-
ish, due to the necessity of the Latin-American region, including Ecuador. It integrates
several state-of-the-art resources and strategies that aim to facilitate the learning of C
programming to mainly novice students. This MOOC was uploaded on the Open edX
platform and is aimed to be open access1 throughout different editions. We also present
the evaluation of its impact on the learning process of 293 high school students. We
have conducted a voluntary survey to these students to know both their perception of
improving their skills in the covered programming topics and their perception of the
contribution of the resources and strategies in their learning process. Finally, based on
these results and on the MOOC development process itself, we identify and show to the
community the lessons learned and where research efforts could be directed.
The remainder of the paper is organized as follows: Section 2 briefly presents some
resources and strategies for teaching programming and related work that has been
reported. Section 3 presents the methodology followed in this work. Section 4 presents
the results of a perception survey regarding both the improvement of programming skills
1 MOOC “De Principiante a Avanzado en C: Primeros Pasos” https://educacionvirtual.cedia.edu.

ec/courses/course-v1:CEDIA+EPN-INNOVA22-001+2023_T1/course/. Last access: June 11,
2023.
400 J. C. Caiza et al.
and the contribution of the resources and strategies in their learning process. Section 5
presents a discussion of lessons learned. Finally, Section 6 presents conclusions and
future work.
2 Background and Related Work

Programming training, like other disciplines, encompasses teaching concepts as well as
the design and implementation of algorithms using a programming language to solve
problems. Evaluating students’ achievement of learning outcomes is also an essential
activity when teaching Programming. In the state of the art, various learning resources
and strategies have been proposed to support these tasks.
Learning resources, which consist of tools or materials used for teaching and evalu-
ating programming knowledge and skills, have received research efforts for some years
now. For example, works as [13, 14] describe useful studies in the field of feedback gen-
eration previously to 2018. To explore recent advancements, a search was conducted in
Scopus in April 2023 using the following search string: (programming OR coding) AND
(learning OR teaching) AND tools AND (teenagers OR novice). Analysis of papers pub-
lished since 2018 revealed the prevalence of programming environments, such as Alice
[10], App Inventor [10], CodeBoot [15], ModKit [16], LadeBook [17], and others. These
environments provide user interfaces for program implementation using code or blocks,
as well as debugging capabilities. Additionally, efforts have led to improving automatic
grading tools, including Soploon [18] and WebTA [11]. Finally, in a previous study [19],
we also found that multimedia resources such as text, videos, and images have been used
to enrich the programming concepts and exercises explanation.
In terms of learning strategies, which refer to approaches used to reinforce learning
resources and the overall learning process, two notable strategies have gained much
attention recently: automated feedback [14, 20] and gamification-based activities [12,
21]. The automatic feedback is still mainly oriented to provide a grade to the students.
That is, the fundamental purpose is to help teachers evaluate a large number of exercises,
which is an essential requirement in a MOOC due to the high number of students. Regard-
ing gamification-based activities, the effectiveness of including game elements, such as
challenges and rewards, to capture students’ attention and motivation is highlighted.
Some of the mentioned resources and strategies have been used in some recent
studies. The most recent one, proposed by Moreno et al. [22], presents the results of
a case study using a flipped-classroom approach to teach programming online, which
was used during COVID. This study employed videos, programming exercises, and
a WhatsApp communication channel for first-year university students to autonomously
learn Programming Fundamentals. We have also used videos and programming exercises
in the MOOC developed. However, we have enhanced those resources by applying
learning strategies, e.g., game-based programming exercises and concept explanation
videos that use analogies from a domain familiar to the students. In addition, we have
identified a set of lessons learned based on student perceptions that we hope can be used
by practitioners and researchers to guide them towards potential improvements.
3 Methodology
This section describes the methodology followed in this study. As shown in Fig. 1, it
consists of four main steps that are described below.
Fig. 1. Methodology followed in this study.
3.1 Design, Implementation, and Deployment of the MOOC

The first step consisted of the design, implementation, and deployment of the MOOC.
A team of three academics with expertise in programming topics performed the MOOC
instructional following a standard procedure. This course, whose specific name is “ De
Principiante a Avanzado en C: Primeros Pasos “, consisted of 6 modules, as presented
in Table 1. A module was broken down into a group of topics, each supported by a set
of learning resources. Some of these resources (those marked with *) were reinforced
using some strategies.
For each topic, a single programming concept was explained using both video and
text resources. The creation of videos considered criteria related to content, interactivity,
and duration. Regarding content, each concept was explained using analogies or exam-
ples within a domain that is widely familiar to students: burger-making, thus avoiding
unnecessary complexity. For instance, since we selected the burger-making domain, the
concept of if-else conditionals was illustrated through an example of calculating the
cost of a burger order for dine-in (no discount) or takeout (15% discount). The selected
domain was used consistently across all topics to maintain a cohesive narrative and facil-
itate familiarity as students progress throughout the course. Interactivity was strongly
emphasized through dynamic and animated presentations, fully and meticulously syn-
chronized with the teacher’s explanation, rather than relying solely on traditional plain
text. The duration of the videos was relatively short (an average of 7 min) to keep the
learners’ attention and to be able to watch them again if necessary.
Additionally, alongside the video explanations, students have access to text- and
image-based explanations on the same topic. This served the dual purpose of ensuring
accessibility and allowing students to quickly skim through the text-based explanation,
especially if they already had prior knowledge of such topic. Finally, we also provided
the students with key points as takeaways.
Example exercises were used to guide students step-by-step in the application of
the covered concepts to solve problems by developing programs with the C language.
Table 1. Programming MOOC modules, topics, and resources. (*) These resources have been
enhanced using some strategies.
Module Topics covered Programming Example Solved Proposed Questionnaire-based Synchronous

concept exercises challenge challenges test tutoring
explanation -video, exercises exercises session
– video and text & – text, – text &
text (*) code (*) video & code (*)
code (*)
1 Motivation and 3 2 0 0 2 1
online
development
environment
2 C program 4 4 1 1 2 1
structure
(pre-processor
directives, data
types,
variables and
operators)
3 Data input & 3 3 1 1 2 1
output
4 Conditionals 4 3 1 1 2 1
5 FOR loop 2 1 1 1 2 1
6 FOR nested 2 2 1 1 2 1
loop
These exercises were supported by video and text resources that provide explanations and
comprehensive guidance on the implementation of a program. Additionally, students had
access to an online development environment, Codeboard2 , embedded in the MOOC,
which allowed them to easily edit, run and debug their program code.
A single program was developed incrementally throughout the course. In addition, the
problem that was addressed belongs to the same domain used to explain programming
concepts, thus maintaining a cohesive narrative as discussed above. The incremental
development approach allows students to reuse previously implemented code and focus
solely on implementing the new concept they are learning. For example, in the initial
topic on preprocessor directives, students applied those concepts to define constants for
hamburger prices. In a later topic, such as input/output (I/O), students used those concepts
to implement a menu screen for customers, incrementally improving the functionality of
the program. At the end of the course, the students were able to develop a fully functional
application allowing a customer to order hamburgers, track his order status, and get the
order’s total price.
The average duration of the exercise videos is 12 min since they offer a detailed
step-by-step explanation. Moreover, as in the previous section, a textual explanation of
the coded example is also provided for the reasons already discussed.
Challenge exercises were designed for students to apply their knowledge to solve
game-based programming challenges. They seek to create an exciting scenario to involve
2 A web-based IDE which can work alone or interact with some MOOCs. https://codeboard.io/
students in an experience that favours their learning. To this end, a clear objective was
defined, since their mission, throughout the course, was to save multiversal life by pro-
tecting relics distributed on planets in various galaxies. Challenges (getting the relics),
rewards (one relic allows you to get the next one), risks and consequences (the student’s
failure would kill multiversal life) were set to achieve a feeling of urgency and defi-
ance while solving them and, also, generating a rewarding feeling as the student solves
exercises.
A solved challenge and a proposed challenge were set up at the end of each mod-
ule. Regarding the solved challenge, the student was first required to try to solve a
challenge on his own, although, if necessary, he could access a step-by-step video expla-
nation to solve the challenge. Regarding the proposed challenge, it had to be developed
autonomously by the student. At the end, he could submit the challenge code and receive
his grade and feedback. The grading was based on the successes or failures of a set of
test cases. Feedback was also provided with information about the failed test cases and
tips to help the student correct his mistakes, if any. The student had an unlimited number
of attempts to solve the exercises.
The questionnaire-based tests included standard multiple-choice questions. Ques-
tionnaires were included for formative and summative evaluations.
A synchronous tutoring session was also held three days before the end of a module,
using Zoom, in order to solve any doubts. As an additional element to answer doubts, a
discussion forum was also used.
These resources were created and enriched with the aforementioned strategies by
three university professors with the support of students of Information Technology
Engineering and Telecommunication Engineering. Finally, the resulting resources were
deployed on the Open edX platform, guaranteeing a convenient layout and navigation to
facilitate students’ learning. The MOOC is available and freely accessible to any user.
3.2 Student Diagnostic Survey
Before the students started the MOOC, a diagnostic survey was conducted. This survey
had the objective of obtaining information about the student’s previous experience in
three aspects: in programming using the C language, in the topics covered in the MOOC,
and in the use of the learning resources and strategies relevant to this study. The survey
consisted of a set of 16 questions with response options aligned to the Likert scale. In
total, 2793 students who enrolled in the course completed the survey.
3.3 MOOC Programming Teaching
The course was available from April 27 to July 1, 2023. The course enrolled 293 students
coming, equally, from the last four years of secondary school.
The seven modules that make up the course were taken autonomously and asyn-
chronously by the enrolled students. Each module had to be completed within a speci-
fied period (between 1 and 2 weeks per module). After this period, all resources could
3 Actually, 293 students were enrolled in the course. Yet, 279 answered the diagnostic survey.
be reviewed, but the summative evaluations (questionnaires and proposed programming

challenges) were disabled.
Also, three days before the end of each module, a synchronous tutoring session
was carried out through Zoom to answer any doubts the students might have. For this
purpose, groups of up to 20 students were formed, which were tutored by 2 instructors.
The instructors are sixth and seventh-semester students of Information Technology and
Telecommunications Engineering. As an additional element to answer doubts, discussion
forums were also used.
3.4 Student Perception Survey
Once the MOOC was finished, the final step was to conduct a satisfaction survey. This
survey had the purpose of obtaining information about the students’ perception of two
aspects: the appreciation and contribution of the resources and strategies in their learning
process of C programming fundamentals, and the improvement of their knowledge and
skills in the programming topics covered in the MOOC. The survey consisted of a set of
38 questions with both Likert-scale aligned response options and open-ended questions
to receive feedback from the students. In total, 131 students responded to this survey.
4 Results
This section presents the perception results regarding the improvement of the skills of the
students and regarding the contribution of the resources and strategies in their learning
process of C programming fundamentals.
To understand these results, it is important to consider that 293 students started the
course, while 232 (79%) approved. This group of students obtained an average of 8.01
out of 10 points. Additionally, the surveys were voluntarily filled; thus, we obtained 279
responses for the diagnostic survey and 131 responses for the final survey. We present
the next result as percentages to make the comparison. Raw diagnostic and perception
survey responses, as well as final grades, can be found at http://dx.doi.org/10.17632/fz6
f2prz8y.1.
4.1 Previous Experience and Familiarity of Students
A significant majority of the responders to the diagnostic survey stated that they had no
prior knowledge of C programming. According to the results of the diagnostic survey
conducted before starting the MOOC, 89% stated that they had no prior knowledge of
programming with the C language. This indicates that most of the course participants
were novices in this language.
However, it is important to note that 78% of the students indicated having had some
experience with block-based languages, such as PSeInt. This suggests that, although
they did not have specific programming experience with the C language, some students
might have some familiarity with programming concepts and logic through other visual
or block-based languages. This prior experience might have provided them with a general
base of understanding of programming logic, which could have facilitated their learning
in the MOOC.
Students were asked about their level of prior knowledge in the following topics:
program structure; variables, data types and operators; input/output (I/O); conditionals;
and loops. As shown in Fig. 2, the results show a varied distribution in students’ prior
knowledge levels in each topic, with most students placed in the medium and low levels.
Only a small percentage of students (between 3% and 7%) reported having a very high
level of knowledge in some specific topics. This data will be used later to determine if
a student has improved his or her knowledge.
On the other hand, 58% of the students indicated that they had never used a web
platform for autonomous learning. This information was relevant to understand students’
familiarity and comfort with the MOOC virtual environment and their ability to take
advantage of the learning resources and strategies provided on the platform. During
the teaching of the MOOC, we noticed the lack of experience and had to elaborate
additional tutorials, e.g., on how to retrieve the password of the service provided by the
online development environment.
Fig. 2. Level of knowledge of the topics covered before starting the MOOC
Before starting the MOOC, students were also asked about their familiarity with dif-
ferent learning resources (Fig. 3-a) and strategies (Fig. 3-b). The results show a varied
distribution of familiarity with learning resources and strategies, with almost half of the
students having rarely or never used them. Videos both to explain concepts and to explain
solved exercises, automated code evaluation, and gamification-based activities are the
least frequently (rarely and never) used resources and strategies, perhaps because their
regular education is based on a face-to-face modality and because gamification in pro-
gramming has just begun to be introduced. On the other hand, most of the students have
had more frequent previous experience with multiple-choice questionnaires, perhaps
because it is a standard resource widely used to evaluate students in many disciplines.
(a)
(b)
Fig. 3. Students’ familiarity with the (a) resources and (b) strategies of interest prior to starting
the course.
4.2 Level of Improvement of Students’ Knowledge and Skills upon Completion

of the Course
Furthermore, as can be seen in Fig. 4, the results of the satisfaction survey conducted after
completing the MOOC showed significant improvements in all the topics covered in the
MOOC. If we compare with the results in Fig. 2, the students experienced remarkable
progress, to the extent that there were no students with a very low level of knowledge in
any of the topics covered and the low level decreased from values around 20% to values
shorter than 7%. This suggests that the MOOC was generally effective in raising the
knowledge level of the students.
We observe that the C Program structure topic had the highest increase in knowledge,
it increased the high and very high level from 16% to 71%. However, it should also be
noted that the Loops topic had the lowest increase in knowledge. The high and very
high levels went from 22% to 44%. One possible reason for this lower increase could be
related to the inherent complexity of this concept in programming. Unlike I/O statements,
loops are statements that require a solid understanding of control logic and the correct
application of other concepts, such as conditionals. Additionally, it is possible that the
last part of this course coincided with the last month of the academic year, which could
have limited the time available to students. Some students found it difficult to understand
and properly apply loops in their exercises and challenges.
Fig. 4. Level of knowledge of the topics covered at the end of the course, according to the students.
4.3 Perception of the Contribution of Resources and Strategies in the Learning

Process of C Programming Fundamentals
Through the final satisfaction survey, students were also asked about their perception
of how much the resources and strategies contributed to their learning process. As can
be seen in Fig. 5, apart from the discussion forums and the Zoom synchronous tutoring
sessions, 75% of the students, on average, expressed that the resources and strategies
used and described in detail in Section III contributed “very much” and “strongly” to
their learning process.
Zoom synchronous tutoring sessions were optional, hence 45% of students did not
attend. The lack of flexibility in tutoring schedules and the preference of some students
who feel more comfortable learning autonomously could be some factors for this high
rate of non-attendance. Still, 24% of students, who did attend, stated that the synchronous
tutorial sessions contributed “very much” and “strongly” to their learning process, which
is a significant percentage that calls for continued use of this resource.
The discussion forums were also considered to contribute the least; 37% indicated
that they contributed little or nothing, although we did not receive concrete feedback on
this.
Finally, more than 35% of the students indicated that the strategies of gaming-based
activities and the challenges proposed through correction and code completion con-
tributed between moderately (mainly), little and nothing. Students stated that the gaming-
based activities were very engaging but preferred a shorter context description to avoid
confusion. This may be due to the lack of clarity in the instructions or the need for
a better connection between the game challenge and the programming concepts being
learned. Regarding the correction and completion strategy, some students pointed out
that before these challenges it would be useful to include other challenges that have to
be solved from scratch.
Fig. 5. Perception of how the resources and (*) strategies contributed to their learning process.
5 Lesson Learned and Challenges

The high pass rate of 79% indicates that most students successfully acquired the neces-
sary knowledge and skills to pass the evaluation criteria. This suggests that the MOOC
was effective in delivering the course content and helping students grasp the fundamen-
tals of C programming. Additionally, the resources and strategies used in the MOOC
were also perceived positively by most students. Yet, we have learned some lessons and
identified some challenges that need to be addressed:
– Videos highly supporting the learning process. The role of videos is crucial in
online or hybrid education, as it can be noted, students have perceived the videos
explaining concepts (80%) and explaining examples (75%) as very useful. We want
to emphasize that these were the elements that took the longest time to design and
build. Precisely some attributes of these resources are as follows: they were script-
based, used a narrative of incremental examples in difficulty, and used elements such
as images that appeared dynamically and were synchronized to the teacher’s narrative,
among others. We plan to apply this same approach in other student support videos
for the explanation of challenge resolution and the video tutorials on the use of the
platform.
– Online development environment. Students (77%) consider that these types of ele-
ments were very useful in their learning process; therefore, they would be a key
element in online or hybrid education. It is worth noting that the Codeboard tool pro-
vides great versatility when developing this type of course; and thanks to its alignment
with the LTI (Learning Tools Interoperability) standard, it can be easily connected to
OpenEdx. Some of the main advantages we have noticed are the following: the ease of
the student to have all the learning elements encapsulated in a single environment, and
the avoidance of requiring additional software installations on the computer. Due to
Codeboard’s flexibility in terms of the languages it supports and to interface through
LTI with different platforms, we plan to use it within other platforms such as Moodle
and for more programming subjects.
– Automated code evaluation. Most students (74%) found the automated code eval-
uation to be very useful, which is in line with previous works as [23]. In this sense,
we would like to emphasize that, in line with the state of the art, we considered the
perspective of usefulness for the student and to go beyond the mere identification of
an error. With this, we sought to dispense with the need for a teacher to correct, but at
the same time provide more descriptive or useful feedback in addition to the identifi-
cation of an error that can be provided by a compiler, or a test based on the approval
of the number of test cases. This was achieved by considering the objective of each
topic and analyzing the possible errors that students might make. For example, in the
anatomy unit of my first program, the student was expected to work properly with
arithmetic operators. To this end, it was detected that the student made an error in the
formula and was asked to review the formula, and not just be told the line of the error.
To the best of our knowledge, Codeboard does not provide any facility for evaluating
code in C and so we had to program our own mechanism. We are planning to make
improvements that provide descriptive and useful information to the student, but also
easy to use or manage by the teacher designing the problems. Finally, it is necessary
to mention that the automated evaluation is possible because of LTI support for the
interaction between OpenEdx and Codeboard.
– AI-based feedback for formative evaluation. While conducting this course, one
event caught our attention: the emergence of commercial AI-based tools that are
being used to support programmers. Initially, we have seen the versatility of Code-
board to support several programming languages and their corresponding evaluation
libraries. This would help in carrying out summative evaluation processes. Also,
we noticed that some artificial intelligence, such as ChatGPT, could be useful for
explaining code, providing alternatives, answering questions, explaining concepts,
and providing personalized recommendations, regardless of the programming lan-
guage, which would help to carry out formative evaluation. Thus, we envision future
work in which we can take the virtues of both platforms to automate and improve both
types of assessments, fundamental in the learning process of programming students.
– Enhanced gamification-based activities. While gamification can enhance student

engagement and motivation [24], it is crucial to strike a balance between game ele-
ments and educational content. Overemphasis on game aspects might distract stu-
dents from the core learning objectives, leading to a superficial understanding of
programming concepts. Further research on gamification-based activities in teaching
programming to novices is still needed.
– Most effective communication between students and instructors. While the syn-
chronous tutoring sessions conducted via Zoom and the use of discussion forums
were implemented in the current study, they were not well received or utilized by the
students. Understanding the reasons behind this lack of engagement and exploring
alternative communication channels that better align with students’ preferences and
needs is crucial.
Finally, operational actions are also essential to facilitate MOOC learning, particu-
larly for inexperienced students who may encounter challenges in navigating the online
learning platform. For instance, in our study, it was observed that some students faced
difficulties in recovering their passwords for the service providing the online execution
environment. This issue hindered their progress and may cause frustration. To address
such operational hurdles, it is crucial to provide comprehensive tutorials. It will empower
students to navigate the platform more effectively and minimize obstacles that may hinder
their learning experience.
6 Conclusions
In this study, we developed and evaluated a MOOC aimed at teaching C program-

ming to novice students. The MOOC incorporated a range of resources and strategies,
including video explanations, online programming environments, automated evaluation,
and gaming-based activities. The results indicated a high approval rate and significant
improvement in programming skills among the participating students. These findings
demonstrate the potential of MOOCs in addressing the demand for programming skills,
especially in contexts where such skills are scarce.
However, the study also highlighted certain challenges that need to be addressed to
enhance the effectiveness of MOOCs in programming teaching. One key aspect is the
need for improved communication between students and instructors. While synchronous
tutoring sessions and discussion forums were available, their limited utilization suggests
a need for further research to determine the most effective communication channels and
support mechanisms. Furthermore, lessons learned revealed the importance of personal-
ized feedback, the integration of various forms of feedback beyond error identification,
and enhanced gamification-based activities. Our future research leads to exploring inno-
vative approaches, such as the use of artificial intelligence, to provide personalized and
detailed feedback and enhanced gamification-based activities to students, supporting
their individual learning journeys.
Overall, the findings from this study contribute to the ongoing efforts to address
the demand for programming skills through MOOCs. By leveraging the strengths of
MOOCs, such as widespread accessibility and flexible learning resources, while address-
ing the identified challenges, we can enhance the design and delivery of programming
courses, ultimately empowering learners with the skills needed for today’s digital society.
References
1. Rina, V., Kluzer, S., Punie, Y.: DigComp 2.2: The Digital Competence Framework for
Citizens-With new examples of knowledge, skills and attitudes. Joint Research Centre
(Seville) (2022)
2. CEPAL: Un camino digital para el desarrollo sostenible de América Latina y el Caribe (2022).
https://repositorio.cepal.org/handle/11362/48460. Accessed 13 Jul 2023
3. Ministerio Coordinador de Conocimiento y Talento Humano: Ecuador hacia el 2035: agenda
prospectiva para el Sector del Conocimiento y Talento Humano (2017)
4. Dávelos, N.: Estos son los profesionales tecnológicos más demandados por las empre-
sas (2021). https://www.primicias.ec/noticias/tecnologia/profesionales-sector-tecnologico-
empesas/. Accessed 13 Jul 2023
5. Casabuerta, D.: Programar: una nueva competencia digital básica. In: Praxis y políticas de
capacitación digital. http://alfabetitzacio-digital.recursos.uoc.edu/praxis-politiques/es/1-4-3-
programar-una-nueva-competencia-digital-basica/. Accessed 13 Jul 2023
6. Padulla, M.: Informe de Empleabilidad en Ecuador 2021 (2021)
7. Grasel Da Silva, P., Carvalho, M.J.S., Teixeira, A.C.: Study on MOOC for social learning
In: Proceedings - 13th Latin American Conference on Learning Technologies, LACLO 2018,
pp. 444–449 (2018). https://doi.org/10.1109/LACLO.2018.00080
8. Carrión-Martínez, M.A., Jara-Roa, D.I., Sarango-Lapo, C.P., Samaniego-Franco, J.B.: Use
of interactive multimedia resources in the MOOC of the Universidad Técnica Particular de
Loja (UTPL). In: Proceedings - 14th Latin American Conference on Learning Technologies,
LACLO 2019, pp. 134–137 (2019). https://doi.org/10.1109/LACLO49268.2019.00032
9. Maldonado, J.J., Pérez-Sanagustín, M., Bermeo, J.L., Muñoz, L., Pacheco, G., Espinoza,
I.: Flipping the classroom with MOOCs. A pilot study exploring differences between
self-regulated learners. In: 12th Latin American Conference on Learning Objects and
Technologies, LACLO 2017, pp. 1–8 (2017). https://doi.org/10.1109/LACLO.2017.8120934
10. Papadakis, S., Orfanakis, V.: Comparing novice programing environments for use in secondary
education: app Inventor for Android vs Alice. Int. J. Technol. Enhanced Learn. 10(12), 44–72
(2018)
11. Ureel II, L.C., Wallace, C.: Automated critique of early programming antipatterns. In:
Proceedings of the 50th ACM Technical Symposium on Computer Science Education,
pp. 738–744 (2019)
12. Suh, S., Lee, M., Xia, G.: Coding strip: a pedagogical tool for teaching and learning pro-
gramming concepts through comics. In: 2020 IEEE Symposium on Visual Languages and
Human-Centric Computing (VL/HCC), pp. 1–10. IEEE (2020)
13. Caiza, J.C., Del Alamo, J.M.: Programming assignments automatic grading: review of tools
and implementations. In: INTED2013 Proceedings, pp. 5691–5700 (2013)
14. Keuning, H., Jeuring, J., Heeren, B.: Towards a systematic review of automated feedback
generation for programming exercises. In: Proceedings of the 2016 ACM Conference on
Innovation and Technology in Computer Science Education, 2016, pp. 41–46 (2016)
15. Feeley, M., Melançon, O.: Teaching Programming to Novices Using the codeBoot Online
Environment, arXiv preprint arXiv:2207.12702 (2022)
16. DesPortes, K., DiSalvo, B.: Trials and tribulations of novices working with the Arduino. In:
Proceedings of the 2019 ACM Conference on International Computing Education Research,
pp. 219–227 (2019)
17. Luxton-Reilly, A., McMillan, E., Stevenson, E., Tempero, E., Denny, P.: Ladebug: an online
tool to help novice programmers improve their debugging skills. In: Proceedings of the 23rd
Annual ACM Conference on Innovation and Technology in Computer Science Education,
pp. 159–164 (2018)
18. Vallejos, S., Berdun, L.S., Armentano, M.G., Soria, Á., Teyseyre, A.R.: Soploon: a virtual
assistant to help teachers to detect object-oriented errors in students’ source codes. Comput.
Appl. Eng. Educ. 26(5), 1279–1292 (2018)
19. Guerrero, M., Guamán, D.S., Caiza, J.C.: Revisión de herramientas de apoyo en el proceso
de enseñanza-aprendizaje de programación. Revista Politécnica 35(1), 84 (2015)
20. Deeva, G., Bogdanova, D., Serral, E., Snoeck, M., De Weerdt, J.: A review of automated feed-
back systems for learners: classification framework, challenges and opportunities. Comput.
Educ. 162, 104094 (2021)
21. Dichev, C., Dicheva, D.: Gamifying education: what is known, what is believed and what
remains uncertain: a critical review. Int. J. Educ. Technol. High. Educ. 14(1), 1–36 (2017)
22. Moreno Cadavid, J., Gómez Jaramillo, S., Piedrahíta Ospina, A.A.: Teaching through flipped
classroom and learning communities: analisys of a computer programming course during
pandemic (2023)
23. Souza, D.M.D., Isotani, S., Barbosa, E.F.: Teaching novice programmers using ProgTest. Int.
J. Knowl. Learn. 10(1), 60–77 (2015). https://www.inderscienceonline.com/doi/abs/https://
doi.org/10.1504/IJKL.2015.071054. Accessed 13 Jul 2023
24. Beltrán, J., Sánchez, H., Rico, M.: E-learning y gamificación como apoyo al aprendizaje de
programación, DISIT – Tesis doctorales (2017). http://hdl.handle.net/10662/6429. Accessed
13 Jul 2023
Criteria of Pedagogical Practices in Times
of Pandemic: A Systematic Review
Asdrúbal Emilfo Ayala-Mendoza1(B) , Lizzie Verónica Pazmiño-Guevara1 ,

Jorge Álvarez-Tello2 , and Marco Duque-Romero3
1 Facultad de Ciencias de la Educación, Universidad Indoamérica, Quito, Ecuador
{asdrubalayala,lizziepazmino}@indoamerica.edu.ec
2 Centro de Innovación Social y Desarrollo (CISDE), Quito, Ecuador
jorge.alvarez@cisde-ec.com
3 Centro Universitario (CIFE), Quito, Ecuador
duquem@rumipamba.edu.ec
Abstract. The COVID-19 pandemic has posed significant challenges to the field
of education, requiring pedagogical practices to swiftly evolve from technological
and socioformative perspectives, as well as the incorporation of other pedagogi-
cal practices that dominated the teacher-student interaction during the pandemic
to ensure the continuity of ongoing academic cycles. The objective is to char-
acterize the criteria that can be derived from pedagogical practices during the
2020–2022 pandemic period. A qualitative and quantitative evaluation of authors’
contributions was conducted through a systematic review based on the PRISMA
2020 guidelines using SCOPUS and Web of Science (WoS) databases. Boolean
equations based on keywords were employed for publication filtering. Cohen’s
Kappa index was calculated to assess review concordance. The analysis reveals
that pedagogical practices influence the deficient technological inclusion of teach-
ers’ skills. Socioformation shows a shift from traditional characteristics in teach-
ing processes. Additionally, there is openness to the inclusion of new pedagogical
practices associated with digital education and virtual environments. Technology
and methodology integrate co-evaluation, feedback, and interactive methodolo-
gies as alternatives for enhancing acquired and inherent technological skills in
virtual classrooms using technological platforms.
Keywords: Education · Educational Experiences · Pandemic · Pedagogical

Practice · Educational Technology
1 Introduction
The World Health Organization (WHO) declared the onset of the COVID-19 coron-
avirus pandemic in March 2020, prompting educational shifts through telematic means
[1, 2]. This led educators to acquire digital competencies [3, 4]. Institutions adopted
distance and virtual education, utilizing digital platforms and tools [5]. The COVID-19
pandemic prompted significant shifts in education, leading to rapid evolution of peda-
gogical practices through technology, socioformation, and other educational approaches
https://doi.org/10.1007/978-981-99-7353-8_31
414 A. E. Ayala-Mendoza et al.
that dominated interactions between teachers and learners. This accelerated virtualiza-
tion necessitated digital competency acquisition among educators. Institutions adopted
remote and virtual learning using digital tools and platforms. Companies in this field
capitalized on the crisis, increasing technological demand and costs, impacting families’
access and rural education [6, 7]. Educators’ skills strengthened with technology use,
reshaping their profiles. This learning phase validates good practices and fosters new
digital competencies [5], encouraging creativity and innovation [8]. COVID-19 trans-
formed global education, urging self-evaluation [4], teaching practices need reflection
and self-assessment [9]. Teacher efficacy in some countries is measured by adaptation to
student needs [7]. Technology enhances pedagogy and adaptability [10]. Crisis acceler-
ated curriculum changes [11], demanding critical thinking and adaptability, reinforcing
connections and adaptation to changing contexts. Interdisciplinary practices fortified
educational resilience [2], where Governments adjusted policies for accessibility, focus-
ing on learning over teaching styles. Teacher training requires updated skills [12]. This
systematic review aims to contrast traditional and innovative practices during pandemic-
induced virtual education, unveiling new pedagogical criteria, thought the following
research question: What are the criteria that can be extracted from pedagogical practices
during the 2020–2022 pandemic period?
2 Method
A systematic and characterization review of the current state of the art is conducted. The
research was exploratory in nature, with a mixed approach to understand how criteria
can be characterized and described based on the characterization of author contributions
(see Fig. 1).
Fig. 1. Process for conducting the review and characterization of the criteria.
The research had a qualitative approach. For the characterization of categories, a

filter of relevant articles was applied using a Boolean search equation from the SCO-
PUS and Web of Science (WoS) databases. Subsequently, the systematic review was
initiated following PRISMA 2020 [13], focusing on identifying new studies through
database searches and records (see Fig. 2) to determine categories and author contribu-
tions for designing both the criteria matrix and author contribution density. Information
is then processed to establish the matrix structure as a foundational model for evaluat-
ing pedagogical practices across three dimensions: Technology and Pedagogical Prac-
tice, Socioformative Pedagogical Practice, and Other Pedagogical Practices. The review
Criteria of Pedagogical Practices in Times of Pandemic 415
encompasses the period from January 2020 to March 2022, aiming to cover a substantial
number of articles by identifying them from both databases during the PRISMA 2020
process.
In SCOPUS, the following English language search configuration is used:
TITLE-ABS-KEY (pedagogical AND practices) and (limit-to (pubyear, 2022) or
limit-to (pubyear, 2021) or limit-to (pubyear, 2020)) and (limit-to (doctype, article))
and (limit-to (subjarea, “soci”)) and (limit-to (pubstage, “final”)) and (limit-to (srctype,
“journal”)).
In WoS, the following English language search configuration is used::
“pedagogical practices” (All Fields) and Articles (Document Types) and Education
Educational Research or Education Scientific Disciplines or Social Sciences Interdis-
ciplinary (Web of Science Categories) and 2022 or 2021 or 2020 (Publication Years)
(Table 2).
Fig. 2. Process for the Systematic Review

Table 1. Exclusion Criteria for Screening
Reason Criteria for Screening

1 Articles cited fewer than 2 times are removed
2 Articles related to foreign language development are excluded
3 Articles about early childhood education are not considered
4 Articles regarding physical education are not included
5 Articles related to learning difficulties or disabilities are excluded
6 Duplicate articles are removed
Source: Prepared by the authors
Table 2. Cohen’s Kappa Evaluation Scale
Range Level of Agreement

0.00–0.20 Slight
0.21–0.40 Fair
0.41–0.60 Moderate
0.61–0.80 Substantial
0.81–1.00 Almost perfect
For organizing the information, Table 1 presents the following exclusion criteria that
will be used in identification and screening. Ultimately, articles meeting these criteria
are those cited at least 2 times. This selection yields articles for review. From the search,
the systematic information extraction process is conducted following PRISMA 2020. To
showcase reviewer agreement, the concordance is assessed using Cohen’s Kappa coef-
ficient (1) (for 2 reviewers)[14]. Calculation requires the independent characterization
results from both reviewers.
(Po − Pe)
index Kappa Cohen = (1)
(1 − Pe)
where:
Po: Overall observed agreement proportion.
Pe: Proportion of chance agreement expected.
Cohen’s Kappa concordance can be assessed using the following table:
3 Results
3.1 Characterization
To develop information about pedagogical practices, criteria summarizing the techno-
logical, socioformative, and other practice dimensions were established based on the
results analysis. Table 1 displays the criteria for pedagogical practices in the technolog-
ical and socioformative realms, as well as the intervention of other practices, spanning
from the years 2020 to 2022 (Fig. 3).
Fig. 3. Process for the Systematic Review with results
Based on the review, criteria for evaluating authors’ contributions were determined
using descriptions constructed from the comparison of characteristics found in the arti-
cles, as well as criteria developed based on existing definitions. The characterization
of the contribution made by the authors in Table 4 defines the description of the cri-
teria in Table 3, where the densities of criterion usage are presented, showing trends,
complementarities, and new fields of exploration. The established relationships involve
descriptive calculations that provide an idea of the information density of the criteria
frequently used to better understand pedagogical practices.
Table 3. Criteria (C) on pedagogical practice for author contribution assessment.
Axe C Criteria name Description

Technology and A Inadequate technological Digital divide in internet
pedagogical practice inclusion in education access; incorporating everyone
into the information society
democratizes accessibility,
maximizing time and
possibilities [15]
B Technology for presenting Technology enables access,
information processing, and production of
essential information to
enhance learning capacity. The
effective use of technology
during the pandemic has been
restricted to information
transmission [11]
C Mobile devices are not used Technological devices offer an
in the classroom opportunity to integrate new
technologies into educational
processes. However, their use
in the classroom has been
limited. On the other hand,
mobile learning serves as a
motivating tool for learning
during classes [16]
D Deficient technological The expectation of teachers to
skills in teachers enhance their digital skills is
not a priority, even after the
pandemic-induced shift to
virtual learning. Many
educators have reverted to
their traditional teaching
practices in the educational
process [17]
E Technologically native New contradictions arise in the
students, digitally migrant teaching-learning processes;
teachers teachers were not adequately
prepared for the novel
technological changes brought
about by the pandemic [18]
(continued)

F Deficient configuration in Learning management systems
virtual classrooms have sometimes caused
confusion and discomfort
among students and families,
as locating resources, and
navigating these platforms to
meet educational objectives
has proven challenging [17]
G Institutions need to improve Educational institutions must
technology platform ensure the seamless
functionality of technological
platforms to enhance
productivity, enabling access
from any location and at any
time [19]
Pedagogical practice of H Collaboration It involves support for
Socio-training addressing coordinated,
complementary activities,
agreements, and the
assignment of roles to tackle
the challenge of the context,
achieving learning outcomes,
and developing a relevant
product with problem-solving
in line with the competence of
the training project [20]
I Problem solving It encompasses addressing a
contextual challenge while
searching for a strategy of real
or simulated solution that
enables the achievement and
socialization of the expected
product and the obtained
impact [20]
J Interdisciplinarity By applying complex
Trans-disciplinarity thinking, various areas, fields,
or disciplines that address
contextual challenges in
training, engagement, research
projects, and administrative
management are
interconnected [20]
(continued)

Other Pedagogical K Learning environments Optimization of training
Practices spaces involves adapting
educational activities to the
unique dynamics of virtuality
during the pandemic [21]
L Feedback Evaluate the achievement of
an outcome for continuous
improvement and the
generation of learning within
the educational process [22]
M Gamification Game-based learning
motivates dynamic student
interaction, aligning activities
with learning objectives [23]
N Learning based on projects, Regular education has
challenges, and problems underestimated the potential
of creativity and innovation.
Teachers should harness
students’ innate skills in idea
generation, autonomy, and
collaborative problem-solving
through Problem-Based
Learning, Project-Based
Learning, and
Challenge-Based Learning
[24]
O Self-regulation, Metacognition goes beyond
metacognition, and limits to comprehend and
motivation integrate new learning,
achieving goals through
self-regulation in interactions
with family, society, and the
environment [25]
P Social network Structured virtual tools for
connectivity based on
common interests or values
improve communication skills
in students and teachers in the
educational context, despite
some learning challenges [26]
(continued)

Q Flipped classroom A tool that fosters values and
creates a personalized, shared,
and participatory learning
environment, encompassing
learning, doing, and learning
from mistakes [27]
R Graphic organizers A valuable tool utilized by
teachers to motivate, evaluate
learning criteria, correct
errors, and provide feedback in
initial student education [21]
S Collaborative learning Team-based learning promotes
leadership, communication,
positive attitudes, and social
skills, encouraging
problem-solving within an
educational setting [28]
T Hybrid education The education system is
responsible for bridging the
digital divide, ensuring
students receive adequate
digital education and ICT
skills [28]
U Mobile devices Teachers must proactively and
collaboratively determine the
importance, interest, and
didactic use of mobile devices
to ensure the reliability and
educational impact [29]
V Robotics Robotic intelligence is an
innovative element in
pedagogical practice, fostering
interest and creativity from an
early age [8]
From this characterization, criteria for pedagogical practices during the pandemic
period in 2020–2022 can be extracted (Tables 5, 6 and 7).
For assessing the coherence of information extraction, an initial Cohen’s Kappa

index of 0.5781 (moderate) was obtained, considering the total characterization of each
criterion by evaluator 1 and evaluator 2. Ultimately, with the assistance of Cohen’s Kappa
Table 4. Axes and criteria of pedagogical practice during the pandemic.
Authors/Axes Technology Socioformación Other Practices

Criteria?s A B C D E F G H I J K L M N O P Q R S T U V
[30] 2
[31] 2
[32] 1
[10] 3
[11] 3
[33] 5
[34] 1
[35] 1
[36] 1
[22] 1
[37] 1
[38] 1
[24] 1
[25, 39] 1
[39] 1
[5] 1
[40] 3
[27] 2
[9] 2
[41] 1
[42] 1
[43] 1
[21] 1
[44] 1
[45] 1
[28] 1
[29] 1
[46] 4
[47] 1
[20] 3
[48] 1
[8] 1
Total 8 3 3 7 4 1 2 2 2 2 1 1 1 2 2 2 2 1 2 1 1 1 51
% 16 6 6 14 8 2 4 4 4 4 2 2 2 4 4 4 4 2 4 2 2 2 100
Source: Prepared by the authors

Table 5. Assessment of Cohen’s Kappa observed concordance.
Observed Observer subjectivity 1 Margin

coincident no coincident
Observer coincident 18 3 21 50%
subjectivity 2
no coincident 3 18 21 50%
21 21 42
50% 50%
Margin
Po: coincidence observed 0.86
Table 6. Evaluation of Cohen’s Kappa, theoretically expected agreement by chance.
Theoretical = expected by chance

10.50 10.50 21.00
0.25 10.50 10.75
10.75 21.00 31.75
Pe: agreement by chance 0.66
Table 7. Characterization data by observer.
C a b C d e f g h i j k l M n o p q r s t u v K-Cohen
O1 8 3 3 7 4 1 2 2 2 2 1 1 1 2 2 2 2 1 2 1 1 1 0.5781
O2 8 4 3 6 4 2 1 2 2 2 1 1 1 2 2 2 2 1 2 1 1 1
OF 8 3 3 7 4 1 2 2 2 2 1 1 1 2 2 2 2 1 2 1 1 1 1.00
index, results were aligned between evaluators, criteria were harmonized, and a Cohen’s
Kappa index of 1 was achieved, ensuring a 100% agreement in the evaluated criteria.
Kappa Cohen 0.5781 Moderate

After reconciling all evaluations with the reviewers, the following are obtained:
Kappa Cohen 1.00 Almost perfect
Where,
C: Evaluated criteria from the papers.
O1: Criteria from observer 1.
O2: Criteria from observer 2.
OF: Final observation after consensus is reached among the evaluators.
4 Discussion
Pedagogical practice has given way to establishing axes for its characterization in relation
to technology, socioformación, and other practices addressed below.
Technology and Pedagogical Practice.
The deficient technological inclusion in education demands that teachers standard-
ize the use of ICT, enhancing skills to optimize results. However, the integration of
educational technology is complex, lacks key foundations, and its use is often superfi-
cial. Resistance to mobile device use persists in some educational settings. Well-trained
educators can provide meaningful and quality learning, promoting educational empow-
erment. Technological inclusion requires adapting pedagogical practices to fully harness
the advantages of ICT. Educating teachers about ethical and effective use is essential for
transformative integration.
Pedagogical Practices in socioformación.
Pedagogical practices evolve with the integration of ICT and virtual environments,
highlighting the socioformación approach for digital education and holistic development.
It involves methods for complex thinking, metacognition, and collaboration, enhancing
the environment and problem-solving. Traditional practices neglect context, leading
to shallow learning. Socioformación integrates inquiry and problem-solving, fostering
creativity, innovation, and values. Assessment involves detailed instruments with criteria
and indicators to provide feedback on values. Rooted in complex thinking, it seeks
collaboration and problem resolution through essential formative actions.
Other Pedagogical Practices.
New challenges arise for educators, who must not only develop technological skills
but also update their pedagogical practices with new active methodologies where stu-
dents participate more actively in sessions, becoming active learners rather than passive
recipients. The reviewed studies emphasize that these new teaching practices should
include the following actions:
1. Strengthening the Learning Environment and Educational Spaces [36]
2. Ongoing Feedback [22]
3. Gamification [38]
4. Project-Based, Challenge-Based, and Problem-Based Learning [24]
5. Self-regulation, Metacognition, and Motivation [25]
6. Influence of social media in the Classroom [39]
7. Collaborative Learning [5]

8. Flipped Classroom [27, 42]
9. Graphic Organizers [21]
10. Hybrid Education [28]
11. Adoption of Mobile Devices and Robotics [29]
Each of the above proposals corresponds to a suitable theoretical and practical depth,
subject to further research in the future, and underscores the commitment of educators
to continuous professional development.
5 Conclusion
Regarding pedagogical practices in relation to technology, it is concluded that there
is indeed a worldwide deficiency in the technological skills of many educators, which
became more evident during the pandemic-induced virtualization of education and con-
tinues to be a challenge for many educators [9, 30, 33, 34, 40, 41, 46]. This aspect,
which is not new, has remained an unresolved issue in teacher training. Both personal
and institutional commitment are crucial, both in training and in the practical application
of these skills in the evolution of pedagogical practices.
In the cases presented that demonstrate the use of technology in educational set-
tings, there are still some difficulties, such as the traditional teaching practices applied
through technological means, often resulting in one-way communication of information
without meaningful interaction [33, 37], This deficiency leads to a poor integration of
technology into pedagogical practices, where technology is present in teacher activities
but does not significantly impact the learning process, and in some cases even hin-
ders it. Complex and confusing virtual spaces intended to facilitate learning sometimes
exacerbate this situation [9, 11, 30, 32, 33, 35, 40, 46]. Generational gaps in teach-
ing contribute to the lack of alignment between students and teachers in the use of
mobile devices due to limited educational infrastructure [11, 31, 33, 40, 46]. This study
underscores the importance of delving into pedagogical practices. Contributions to the
socioformative approach are identified, emphasizing collaboration, transdisciplinarity,
and context-based problem-based learning. Essential actions are highlighted, such as
student motivation, case-based and values-driven approaches, effective communication,
creativity, innovation, and continuous assessment [10, 20].
Finally, the rapid shift towards virtual education demands a reinvention of peda-
gogical practices. It is suggested that educators begin with self-assessment to recognize
their strengths and weaknesses in relation to new practices associated with technology
and innovation. The current society expects more relevant education. The global cri-
sis has altered the context, rendering previous practices inadequate. It is essential to
acknowledge the progress made and to continue evolving pedagogically, not only due to
the pandemic but also to meet the needs of new generations. Adapting to this constant
change is a pressing necessity in our dynamic reality. It is recommended to expand on
this topic or conduct further analyses on its impact, involving additional reviewers to
enrich pedagogical development strategies.
References
1. CEPAL, OREALC y UNESCO La educación en tiempos de la pandemia de COVID-
19. https://www.iesalc.unesco.org/2020/08/25/informe-cepal-y-unesco-la-educacion-en-tie
mpos-de-la-pandemia-de-covid-19/
2. Sequeira, L., Dacey, C.M.: The COVID-19 diaries: identity, teaching, and learning at a
crossroads. Front. Educ. (Lausanne) 5 (2020). https://doi.org/10.3389/feduc.2020.586123
3. Lillis, F., Bravenboer, D.: The best practice in work-integrated pedagogy for degree appren-
ticeships in a post-viral future. High. Educ. Skills Work-Based Learn. 10(5), 727–739 (2020).
https://doi.org/10.1108/HESWBL-04-2020-0071
4. Feyzi Behnagh, R., Yasrebi, S.: An examination of constructivist educational technologies:
key affordances and conditions. Br. J. Educ. Technol. 51(6), 1907–1919 (2020). https://doi.
org/10.1111/bjet.13036
5. Gemmel, P.M., Goetz, M.K., James, N.M., Jesse, K.A., Ratliff, B.J.: Collaborative learning
in chemistry: impact of COVID-19. J. Chem. Educ. 97(9), 2899–2904 (2020). https://doi.org/
10.1021/acs.jchemed.0c00713
6. Apak, J., Taat, M., Suki, N.: Measuring teacher creativity-nurturing behavior and readiness
for 21st century classroom management. Int. J. Inf. Commun. Technol. Educ., 52–67 (2021).
https://doi.org/10.4018/IJICTE.20210701.oa4
7. Guillén-Gámez, F., Mayorga-Fernández, M., Álvarez-García, F.: A study on the actual use of
digital competence in the practicum of education degree. Technol. Knowl. Learn., 667–684
(2020). https://doi.org/10.1007/S10758-018-9390-Z/METRICS
8. Yang, Y., Long, Y., Sun, D., Van-Aalst, J., Cheng, S.: Fostering students creativity via educa-
tional robotics: an investigation of teachers, pedagogical practices based on teacher interviews.
Br. J. Educ. Technol. 51, 1826–1842 (2020). https://doi.org/10.1111/bjet.12985
9. Ifinedo, E., Rikala, J., Hämäläinen, T.: Factors affecting Nigerian teacher educators, technol-
ogy integration: considering characteristics, knowledge constructs. ICT practices and beliefs.
Comput. Educ. 146 (2020). https://doi.org/10.1016/j.compedu.2019.103760
10. Begazo, C.M.C.: Prácticas pedagógicas desde el enfoque socioformativo: Una autoevaluación
Docente en Perú (2020). https://www.redalyc.org/articulo.oa?
11. Betancourt-Odio, M.A., Sartor-Harada, A., Ulloa-Guerra, O., Azevedo-Gomes, J.: Self-
perceptions on digital competences for M-learning and education sustainability: a study with
teachers from different countries. Sustainability (Switz.) 13(1), 1–12 (2021). https://doi.org/
10.3390/su13010343
12. Dhurumraj, T., Ramaila, S., Raban, F., Ashruf, A.: Broadening educational pathways to stem
education through online teaching and learning during covid-19: teachers perspectives. J.
Baltic Sci. Educ. 19, 1055–1067 (2020). https://doi.org/10.33225/JBSE/20.19.1055
13. Page, M.J., et al.: The PRISMA 2020 statement: an updated guideline for reporting systematic
reviews. The BMJ (2021). https://doi.org/10.1136/bmj.n71
14. Kruse, C.S., Molina-Nava, A., Kapoor, Y., Anerobi, C., Maddukuri, H.: Analyzing the effect
of telemedicine on domains of quality through facilitators and barriers to adoption: systematic
review. J. Med. Internet Res. 25 (2023). https://doi.org/10.2196/43601
15. Akram, H. Aslam, S., Saleem, A., Parveen, K.: The challenges of online teaching in covid-19
pandemic: a case study of public universities. J. Inf. Technol. Educ. Res. 20, 263–282 (2021).
https://doi.org/10.28945/4784
16. Abdullah, A., Abirami, R., Gitwina, A., Varthana, C.: Assessment of academic performance
with the e-mental health interventions in virtual learning environment using machine learning
techniques: a hybrid approach. J. Eng. Educ. Transformations 34, 79–85 (2021)
17. Blikstad-Balas, M., Klette, K.: Still a long way to go narrow and transmissive use of technology
in the classroom. Nord. J. Digital Literacy 15, 55–68 (2020). https://doi.org/10.18261/ISSN.
1891-943X-2020-01-05
18. Annamalai, N., Mazeikienè, V., Tangiisuran, B., Oleskevicience, G.: How do students really
interact? An investigation of Lithuanian students. Interactions via smartphone apps. Malays.
J. Learn. Instr. 18, 65–83 (2021). https://doi.org/10.32890/MJLI2021.18.1.3
19. Goh, E., Sigala, M.: Integrating information & communication technologies (ICT) into class-
room instruction: teaching tips for hospitality educators from a diffusion of innovation app-
roach. J. Teach. Travel Tourism 20, 156–165 (2020). https://doi.org/10.1080/15313220.2020.
1740636
20. Tobón, S., Juarez-Hernandez, G., Herrera-Meza, S., Nunez, C.: Assessing school principal
leadership practices. Validity and reliability of a rubric. Educación XX1 23, 187–210 (2020).
doi: https://doi.org/10.5944/educxx1.23894
21. Machado, C., Carvalho, A.: Concept mapping: benefits and challenges in higher education. J.
Continuing High. Educ. 68, 38–53 (2020). https://doi.org/10.1080/07377363.2020.1712579
22. Chan, C., Luo, J.: Exploring teacher perceptions of different types of “feedback practices”
in higher education: implications for teacher feedback literacy. Assess. Eval. High. Educ. 47,
61–76 (2022). https://doi.org/10.1080/02602938.2021.1888074
23. Chen, J., Zhou, Z., Chen, L. and Ding, T.: Optimization of regional water-energy-food systems
based on interval number multi-objective programming: a case study of ordos, China. Int. J.
Environ. Res. Public Health 17(20), 1–18 (2020). https://doi.org/10.3390/IJERPH17207508
24. Cremin, T., Chappell, K.: Creative pedagogies: a systematic review. Res Pap Educ. 36, 299–
331 (2021). https://doi.org/10.1080/02671522.2019.1677757
25. Cuzzolino, M.P.: The awe is in the process: the nature and impact of professional scientists
experiences of awe. Sci. Educ. 105, 681–706 (2021). https://doi.org/10.1002/sce.21625
26. García, M., Alfonso C., Morera, T., Girona, E.: The identity of the social entreprise in Spain:
analysis from four different socioeconomic realities–La identidad de la empresa social en
España: Análisis desde cuatro realidades socioeconómicas. CIRIEC, 155–162 (2018). https://
doi.org/10.7203/CIRIEC-E.92.9236
27. González, M., Abad, E., Belmonte, L.: Aprendizaje significativo en el desarrollo de com-
petencias digitales. Análisis de tendencias. IJERI, 91–110 (2020). https://doi.org/10.46661/
ijeri.4741
28. Metscher, S., Tramantano, J., Wong, K.: Digital instructional practices to promote pedagogical
content knowledge during COVID-19, 121–124 (2021). https://doi.org/10.1080/02607476.
2020.1842135
29. Nikolopoulou, K., Gialamas, V., Lavidas, K., Komis, V.: Teachers’ readiness to adopt mobile
learning in classrooms: a study in greece, technology, knowledge and learning, 53–77 (2021).
https://doi.org/10.1007/s10758-020-09453-7
30. Akram, H., Aslam, S., Saleem, A., Parveen, K.: The challenges of online teaching in COVID-
19 pandemic: a case study of public universities in, 263–282 (2021). 10.2894 5/4784
31. Annamalai, N., Mazeikienè, V., Tangiisuran, B., Oleskevicience, G.: How do students really
interact? An investigation of lithuanian students’ interactions via smartphone apps, 65–83
(2021). https://doi.org/10.32890/MJLI2021.18.1.3
32. Attard, C., Holmes, K.: An exploration of teacher and student perceptions of blended learning
in four secondary mathematics classrooms. Math. Educ. Res. J. 34(4), 719–740 (2022). https://
doi.org/10.1007/s13394-020-00359-2
33. Blikstad, M., Klette, K.: Still a long way to go narrow and transmissive use of technology in
the classroom, 55–68 (2020). https://doi.org/10.18261/ISSN.1891-943X-2020-01-05
34. Boltz, L., Yadav, A., Dillman, B., Robertson, C.: Transitioning to remote learning: lessons
from supporting K-12 teachers through a MOOC, 1377–1393 (2021). https://doi.org/10.1111/
bjet.13075
35. Boyte, L., Minadeo, D.B., Bailey, S.: Age, gender, and race as predictors of opting for a
midterm retest: a statistical analysis of online economics students (2018). https://doi.org/10.
33423/JBD.V18I1.515
36. Carvalho, L., Nicholson, T., Yeoman, P., Thibaut, P.: ‘Space matters: framing the New Zealand
learning landscape. Learn. Environ. Res. 23, 307–329 (2020). https://doi.org/10.1007/s10984-
020-09311-4
37. Chávez, D., Cladellas, R., Castelló, A.: Habit and social influence as determinants of Pow-
erPoint use in higher education: a study from a technology acceptance approach, 497–513
(2023). https://doi.org/10.1080/10494820.2020.1799021
38. Chen, S., Zhang, S., Qi, G., Yang J.: Games literacy for teacher education: towards the
implementation of game-based learning (2020)
39. García, A.C., Gil, M., Álvarez, I., Casares, M.: The influence of social networks within
educational and social fields: a comparative study between two generations of online students,
1–12 (2020). https://doi.org/10.3390/su12239941
40. Goh, E., Sigala, M.: Integrating information & communication technologies (ICT) into
classroom instruction: teaching tips for hospitality educators from a diffusion of innovation
approach, 156–165 (2020). https://doi.org/10.1080/15313220.2020.1740636
41. Itow R-C: Fostering valuable learning experiences by transforming current teaching practices:
practical pedagogical approaches from online practitioners 121(5–6), 433–442 (2020). https://
doi.org/10.1108/ILS-04-2020-0106
42. Jacques, S., Lequeu, T.: The attractiveness of reversing teaching forms feedback on an
electrical engineering course, 21–34 (2020). https://doi.org/10.3991/IJEP.V10I3.12361
43. Lavrenova, M., Lalak, N.V., Molnar, T.I.: Preparation of future teachers for use of ICT in
primary school, 185–195 (2020). https://doi.org/10.18662/rrem/12.1sup1/230
44. Madanipour, P., Cohrssen, C.: Augmented reality as a form of digital technology in early
childhood education, 5–13 (2020). https://doi.org/10.1177/1836939119885311
45. Melissa, A., Lee, N.: The influence of self-regulation processes on metacognition in a virtual
learning environment, 1–17 (2020). https://doi.org/10.1080/03055698.2018.1516628
46. Abdullah, A.S., Abirami, R.M., Gitwina, A., Varthana, C.: Assessment of academic perfor-
mance with the e-mental health interventions in virtual learning environment using machine
learning techniques: a hybrid approach (2021)
47. Silva, E., Lino-Neto, T., Ribeiro, E., Rocha, M., Costa, M.: Going virtual and going wide: com-
paring team-based learning in-class versus online and across disciplines. Educ. Inf. Technol.
(Dordr.) 27, 2311–2329 (2022). https://doi.org/10.1007/s10639-021-10683-0
48. Vandeyar, T.: The academic turn: social media in higher education. Educ Inf Technol (Dordr.)
25, 5617–5635 (2020). https://doi.org/10.1007/s10639-020-10240-1
Academic Performance During
the Pandemic with ICT Was Better
Before and After It
Diego Salazar-Armijos1,3(B) , Héctor Revelo-Herrera1 ,

Pablo Niquinga-Vargas2 , Carlos Segovia-Segovia3 ,
and Juan Estrada-Arana3
1
Departamento de Ciencias de la Computación, Universidad de las Fuerzas
Armadas ESPE. Sede Santo Domingo, Vı́a Santo Domingo -Quevedo Km 24,
Sangolquı́, Santo Domingo de los Tsáchilas, Ecuador
{drsalazar,hmrevelo}@espe.edu.ec
2
Dirección de Posgrado, Pontificia Universidad Católica del Ecuador PUCE, Quito,
Ecuador
pjniquinga@puce.edu.ec
3
Dirección de Posgrado, Universidad Nacional de Chimborazo, Av. Eloy Alfaro y 10
de Agosto - Campus “La Dolorosa”, Riobamba, Ecuador
{diego.salazar,carlos.segovia,juan.estrada}@unach.edu.ec
Abstract. The research carried out at the Armed Forces University

ESPE, Santo Domingo Campus, aimed to explore the impact of chang-
ing teaching methodologies using ICT on the academic performance of
Information Technology Engineering students before, during, and after
the COVID-19 pandemic. For the analysis, a random sample of academic
performance from three student cohorts during different periods, with a
one-year difference between them, was taken. In accordance with various
authors, a one-way analysis of variance (ANOVA) test was employed due
to the characteristics of the sample, with the purpose of assessing whether
significant differences in performance means existed between the groups;
additionally, other tests were used for the same purpose. The sample was
validated for normality, homoscedasticity, and independence of samples
before conducting the ANOVA test. The results unveiled significant dif-
ferences in the average academic performance among the groups. Further
analysis, utilizing the Tukey test, demonstrated that the academic per-
formance of students during the pandemic differed significantly from the
other groups. However, there were no significant differences between the
groups before and after the pandemic. This led to the conclusion that
the post-pandemic teaching methodology might not have capitalized on
the positive aspects of employing ICT to enhance students’ academic
performance.
Keywords: Academic performance · statistics · ANOVA · Tukey ·

ICT in COVID19
https://doi.org/10.1007/978-981-99-7353-8_32
430 D. Salazar-Armijos et al.
1 Introduction
It is evident that the COVID-19 pandemic forced a change in teaching method-
ologies and techniques at all levels of education. The shift was from predomi-
nantly face-to-face teaching with limited use of technological tools for communi-
cation, collaborative activities with instructors, substitution of equipment with
simulators or emulators, limited access to computer equipment due to economic
factors, and the scarcity of preparedness among teaching staff who transitioned
from in-person to virtual synchronous classes. According to Vidal, the COVID-19
pandemic has brought significant changes in the educational realm, necessitating
the widespread adoption of distance learning modalities and causing disruptions
to the academic routine [14]. Rea, suggests that these sudden changes have posed
challenges for both educators and students, and it is crucial to understand how
they have impacted academic performance in this context [12]. Loor, argues
that the COVID-19 pandemic has caused an unprecedented disruption in all
aspects of life, including education. School closures and the massive adoption
of distance learning have generated significant challenges for students, teachers,
and educational systems worldwide [10]. In this context, following the progres-
sive return to face-to-face education established by the Armed Forces University
ESPE, the question arose as to whether the teaching methodology used dur-
ing COVID-19 affected academic performance and whether the post-pandemic
return to in-person learning, coupled with the new technological methodologies
learned by teachers, has strengthened competencies for improved student per-
formance. Based on the above, the objective of this research was to demonstrate
that the influence of changing teaching methodologies before, during, and after
the COVID-19 pandemic had an impact on the academic performance of stu-
dents in the Information Technology Engineering (ITIN) program at the Armed
Forces University ESPE, Santo Domingo Campus. To verify the mentioned dif-
ferences in the academic performance of the students in the said program, a
comparative analysis of academic performance means was proposed, consider-
ing a random sample of academic performance from three student cohorts at
the same level during the periods before the pandemic (Group 1, G1), during
the pandemic (Group 2, G2), and post-pandemic (Group 3, G3). The sample
consists of a total of 60 participants and is within the appropriate ranges to be
considered representative of the population.
To define a suitable test that allows obtaining results with the highest objec-
tivity regarding the stated problem, the changes in academic averages were ana-
lyzed using the statistical technique ANOVA.
According to Terrádez, one-way ANOVA (Analysis of Variance) is a statisti-
cal technique used to compare the means of two or more groups and determine
if there are significant differences among them. It is a widely used tool in vari-
ous fields, such as scientific research and industry, for making comparisons and
drawing important statistical conclusions [13].
Hurtado, asserts that the main objective of ANOVA is to analyze the variabil-
ity of the data both between and within groups. It is based on the premise that if
the differences between the groups are significantly greater than the differences
Academic Performance During the Pandemic with ICT 431
within the groups, then there is at least one group whose mean is significantly
different from the others, and it uses the following formula [7]:
ni
a a
ni
a

(Yij − Ȳ )2 = ni (Ȳi − Ȳ )2 + (Yij − Ȳ )2 (1)
i=1 j=1 i=1 i=1 j=1
That is to say:
SC(total) = SC(F actor) + SC(Error) (2)
Where:
ni = Number of observations for treatment i

Yij = The j-th observation of treatment i
Ȳi = Sample mean of treatment i
Ȳ = Overall sample mean (global mean)
While the mean squares:

Total Mean Squares (CMT):
SCT
CM T = (3)
n−1
Factor Mean Squares (CMF):
SCF
CM F = (4)
k−1
Error Mean Squares (CME):
SCE
CM E = (5)
n−k
Finally, the F statistic:
CM F
F = (6)
CM E
When comparing the obtained value of F with the critical values table of the F-
distribution for a given level of significance, it can be determined whether there
is a significant difference between the groups.
Cayuela, argues that ANOVA can only determine if there are significant dif-
ferences between the groups but cannot identify which specific groups are differ-
ent. To make comparisons between specific pairs of groups and detect significant
differences between them, post hoc tests can be conducted, such as the Tukey
test or multiple comparisons test [4].
The information on the average academic performance of the samples was
obtained from the student grading database of the cohorts 201950 (Pre-Pandemic
G1), 202051 (During Pandemic G2), and 202250 (Post-Pandemic G3) of the
mentioned career in Information Technology Engineering.
To determine the validity of applying one-way ANOVA, the three aspects

validating the test were checked: Normality of the sample, Homoscedasticity of
variances and Independence These aspects were validated using the Shapiro test
(Normality), Bartlett’s test (Homoscedasticity), and Durbin-Watson test (Inde-
pendence). After verifying the fulfillment of these requirements, the aforemen-
tioned ANOVA test was applied, and its results showed a p-value much less than
0.05, indicating the existence of significant differences in the averages between
at least one of the observed groups. Moreover, the F-statistic suggests that the
variation between the groups is approximately 19.45 times greater than the vari-
ation within the groups. Based on the ANOVA test, the Tukey test (Tukey’s
multiple comparisons of means) was performed to compare all possible com-
binations of means from various groups and determine if there are significant
differences between them. The results of this test determined that the academic
performance of the student group during the pandemic exhibits significant differ-
ences compared to the other groups. Finally, it was found that the groups before
and after the pandemic do not have marked differences, suggesting that the
post-pandemic teaching methodology did not incorporate the positive aspects of
using technological tools to improve students’ academic performance.
The intention behind identifying whether significant differences exist in aca-
demic averages before, during, and after the pandemic is to understand how these
results can be interpreted and applied to improve education in a post-pandemic
context.
A research study shows that a well-structured and organized sequential teach-
ing methodology, along with appropriate technology for the development of vir-
tual classrooms, leads to the construction of knowledge with new content during
workspaces [15].
The study of the impact of the pandemic on education and academic perfor-
mance is crucial to understand current challenges and devise effective teaching
strategies and student support. The application of the ANOVA technique will
provide a strong statistical basis for analyzing and comparing academic averages
across different time periods.
Statistical tests revealed that significant differences in student performance
existed during the pandemic. However, these differences do not necessarily deter-
mine the suitability of the teaching and learning mechanism. Therefore, it would
be important to focus on research that identifies the cause of these performance
differences. It is possible that they may be attributed, at first glance, to the use
of Information and Communication Technologies (ICT) or deficiencies in the
evaluation process that might have allowed copying.
Tukey’s tests demonstrated that before and after the pandemic, the results
are practically the same, indicating that teachers likely reverted to using only
their traditional teaching methods and did not consider the positive aspects of
the technological tools employed during the pandemic period.
Finally, to achieve the proposed objectives, a quantitative, longitudinal app-
roach will be used, utilizing data on academic averages before, during, and after
the pandemic. This approach aims to provide a clear insight into how the pan-
demic has affected student performance.

To establish the statistical model, we started with the problem statement whose
objective is to demonstrate that the influence of changing the teaching method-
ology before, during, and after the COVID-19 pandemic affected the academic
performance of students in the Information Technology Engineering program at
the University of the Armed Forces ESPE, Santo Domingo Campus. Translated
into statistical terms, this implies showing whether there are significant differ-
ences in the mean academic performances of students in the mentioned program
for the periods: before, during, and after the pandemic. In other words, the
statistical model is based on the following hypotheses:
X = Average grades obtained by students from three groups (Sample space)

Ho : Null Hypothesis: The performance of the 3 groups is the same.
Ha : Alternative Hypothesis: The performance of at least 1 group is different.
What can be expressed mathematically:
Ho: µg1 = µg2 = µg3

Ha: µg1 = µg2 = µg3
Regarding the materials, the following was considered:

Study Population: It corresponds to the academic performance of students
in the Information Technology Engineering program at the University of the
Armed Forces ESPE, Santo Domingo Campus, during the periods before the
pandemic, during the pandemic, and post-pandemic.
Sample Used: It consists of three groups of students from the same level,
selected randomly from different cohorts of the Information Technology Engi-
neering program at the University of the Armed Forces ESPE, Santo Domingo
Campus, with respect to the grades they obtained before, during, and after the
pandemic. Furthermore, the R software was used to obtain the results of the
analyses.
The methodology is based on the following: To estimate whether there is
a significant difference between the means of academic performance of differ-
ent groups, it was proposed to apply one-way analysis of variance (ANOVA).
However, before validating the mentioned test, it was checked that the sam-
ple meets the requirements of normality, homoscedasticity, and independence of
samples. Once the suitability of the ANOVA test was determined, the said test
was applied using R software to obtain the F-statistic and associated p-value.
Finally, the Tukey test was used to compare all possible combinations of means
from various groups and determine if there are significant differences between
them. The analytical and graphical methods are summarized below:
Analytical Method: To determine the validity of applying one-way ANOVA,

the three aspects validating the test were checked, namely:
1. Normality of the sample
2. Homoscedasticity of variances
3. Independence
These aspects were validated using Shapiro’s test (Normality), Bartlett’s test
(Homoscedasticity), and Durbin-Watson test (Independence). Once the relevant
requirements for applying one-way ANOVA were established, R code was used
to perform the hypothesis test and determine the acceptance or rejection of the
Null Hypothesis Ho. Then the Tukey test was applied to compare all possible
combinations of means from various groups and determine if there are significant
differences between them.
The Shapiro test is a statistical test used to assess whether a data sam-
ple comes from a normal distribution. It was developed by statisticians Samuel
Shapiro and Martin Wilk. The Shapiro test is based on the calculation of the
test statistic W and its comparison with a table of critical values [3].
Bartlett’s test is a statistical test used to assess the homogeneity of variances
in multiple groups or samples. It is used to determine if the variances of the
dependent variable are approximately equal across different groups. The test is
particularly useful when comparing the means of three or more groups using
analysis of variance (ANOVA) [9].
The Durbin-Watson test statistic (DW) ranges from 0 to 4, with a value
close to 2 indicating no autocorrelation (independence). A value of DW around 2
suggests that the residuals are not correlated with each other. Values of DW close
to 0 indicate positive autocorrelation (residuals tend to be positively correlated),
while values close to 4 indicate negative autocorrelation (residuals tend to be
negatively correlated) [1].
The Tukey test, also known as Tukey’s multiple comparisons of means or
Tukey’s honestly significant difference (HSD) test, is a statistical test used to
compare all possible pairs of group means in an analysis of variance (ANOVA)
setting. It is commonly used to determine which specific groups have statistically
significant differences in their means when there are three or more groups being
compared [2].
Graphical Method: To establish normality, the Normal Q-Q plot was used.
To establish homoscedasticity, the Boxplot was used. To determine the indepen-
dence of the samples, the Residuals vs Time (Sample) plot was used.
The Normal Q-Q (Quantile-Quantile) plot is a graphical tool used to assess
whether a data sample follows a normal distribution. It is a common diagnos-
tic plot used in statistics and data analysis to visually inspect the normality
assumption of a dataset.
The Q-Q plot compares the quantiles of the data sample to the quantiles of
a theoretical normal distribution. If the data points fall approximately along a
straight line in the Q-Q plot, it suggests that the data are normally distributed.
On the other hand, if the points deviate from the straight line, it indicates a
departure from normality [5].
To establish homoscedasticity, the Boxplot (also known as the Box-and-

Whisker plot) is a graphical tool that can be used to visually inspect the vari-
ability or spread of data across different groups or categories. It is particularly
useful in comparing the distribution of data in multiple groups or conditions to
assess if their variances are approximately equal.
The Boxplot consists of a rectangular “box” that represents the interquartile
range (IQR) of the data, with a horizontal line inside the box representing the
median. The “whiskers” extend from the box to indicate the minimum and
maximum values within a certain range. Data points that fall outside the range
of the whiskers are often displayed as individual dots or “outliers” [11]
The Residuals vs Time (Sample) plot is created by plotting the residuals on
the vertical axis and the time or sample number on the horizontal axis. If the
residuals appear to fluctuate randomly around the horizontal zero line with no
discernible pattern, it suggests that the residuals are independent and meet the
assumption of independence. On the other hand, if the residuals exhibit patterns,
such as cycles, trends, or clusters, it indicates the presence of autocorrelation and
suggests that the independence assumption may be violated[8].
3 Results and Discussions

The statistical tests revealed the following results: The point estimators of the
sample are established in Table 1.
Table 1. Descriptive statistics of the sample
Groups Mean µ Variance s2

Students before pandemic G1 14.6525 0.439767
Students during pandemic G2 15.796 0.408318
Students post-pandemic G3 14.66 0.406904
Statistics of all groups obtained with R 15.03583 0.7243501
Source: ITIN ESPE Student Database
The results of the descriptive statistics show, in the first instance, that the
group during the pandemic had better academic performance than the other
groups. The groups before and after the pandemic show similar performances;
however, there is a greater variability in the performance of the pre-pandemic
group compared to the post-pandemic group. For this reason, to determine the
possible differences between the groups with greater accuracy, the application
of one-way ANOVA was considered. However, before applying the mentioned
methodology, the three aspects that validate the application of the test were
checked, namely:
1. Normality of the sample

2. Homoscedasticity of variances
3. Independence
From the aforementioned tests, the following analytical results were obtained
and are shown in Table 2:
Table 2. Results of normality, homoscedasticity, and independence tests
Test / Prueba Parameters / Parámetros p-value

Normality / Shapiro W = 0.96605 0.09334
Homoscedasticity / Bartlett’s K-squared = 0.16318, df = 2 0.9216
Independence / Durbin-Watson DW = 2.8102 0.9988
Source: Own elaboration with R software
The results of the analytical tests allow us to infer the following: The Shapiro
test shows a high W value close to 1, and the p-value is also high. Assuming a
significance level of alpha = 0.05, we have p-value > alpha, so the null hypothesis
of the Shapiro test is not rejected, which states the following: Null Hypothesis
(H0): The sample follows a normal distribution. Thus, the first parameter for
applying the ANOVA test is met, which can also be demonstrated with the
graphical Quantile plot detailed in Fig. 1.
In Fig. 1, it can be observed that the data of the theoretical quantiles can prac-
tically fit the reference line (blue line), which leads to the conclusion that the
sample follows a nearly normal distribution.
Regarding the homoscedasticity test of the sample variances, the results of
Bartlett’s analytical test establish that the value of the statistic K-squared =
0.16318 is small, with two degrees of freedom df = 2, and a high p-value of 0.9216.
This implies that for a significance level of 0.05, p-value > alpha, and therefore,
the null hypothesis of Bartlett’s test is not rejected. The null hypothesis (H0)
states that the variances of the samples are equal (homoscedasticity), fulfilling
the second requirement to apply the one-way ANOVA test. Likewise, the quasi-
equality of the variances can be verified with the box plot as shown in Fig. 2.
According to the geometry of the boxplots, in terms of the size of the boxes
and the length of the whiskers, it can be inferred that the variances of the three
groups are practically equal, with a more pronounced variability in Group 3 of
post-pandemic students.
Finally, the independence test was conducted using the Durbin-Watson test,
and the results indicate that the value of DW is 2.8102, suggesting a lack of
autocorrelation in the residuals (as it is close to 2). Additionally, the p-value
is 0.9988, which is much larger than 0.05. Therefore, the null hypothesis of
no autocorrelation cannot be rejected, and it can be concluded that there is
not enough evidence to claim the presence of autocorrelation in the residuals,
confirming the independence of the samples. This is reasonable for the given
problem, as the students belong to three distinct cohorts.
Regarding the graphical method in Fig. 3, a scatter plot between the fitted
values of the model and the residuals was applied. The graph shows no discernible
Fig. 1. Quantile plot to check the normality of the sample. (Source: Self-made with R
software)
pattern in the dispersion around the line at 0, supporting the assumption of

independence of the samples in the model.
Once the three conditions are met, the one-way ANOVA test is applied to deter-
mine if there are differences between the means of the three observed groups.
The results of the one-way ANOVA test are summarized in the following
Table 3:
Table 3. Summary of one-way ANOVA results
Source of Variation Degrees of Freedom Sum of Squares Mean Square F-Test Pr(>F)
Factor 2 17.34 8.668 19.45 3.63e-07
Error 57 25.40 0.446
Total 59 42.74
Source: Self-made with R software
Given that the p-value is very small for any significance level, for example, 0.05,
we have alpha > p-value. Therefore, the null hypothesis is rejected, and there are
statistical grounds to infer that the academic performance of at least one group
of students before the pandemic, during the pandemic, and after the pandemic is
Fig. 2. Boxplot to check the homoscedasticity of the sample (Source: Self-made with
R software)
different. Following the ANOVA test, the Tukey test (Tukey’s multiple compar-
isons of means) was performed to compare all possible combinations of means
from various groups and determine if there are significant differences between
them. The results of the mentioned test are shown in Table 4.
Table 4. Tukey’s multiple comparisons of means results
Group Difference Confidence Interval

Lower Limit Upper Limit Adjusted p-value
G2-G1 1.1435 0.6355091 1.6514909 0.0000038
G3-G1 0.0065 −0.5014909 0.5144909 0.9994774
G3-G2 −1.1370 −1.6449909 −0.6290091 0.0000042
The interpretation of the results is as follows: G2 (Students during the pan-

demic) - G1 (Students before the pandemic): The average difference between
group G2 and group G1 is 1.1435. The 95% confidence interval for this difference
ranges from 0.6355091 to 1.6514909. The adjusted p-value for this comparison
is 0.0000038, indicating a statistically significant difference between the means
of groups G2 and G1.
Fig. 3. Scatter plot to check the independence of the sample. (Source: Self-made with
R software)
G3 (Students after the pandemic) - G1 (Students before the pandemic): The

average difference between group G3 and group G1 is 0.0065. The 95% confidence
interval for this difference ranges from −0.5014909 to 0.5144909. The adjusted
p-value for this comparison is 0.9994774, indicating that there is no statistically
significant difference between the means of groups G3 and G1.
G3 (Students after the pandemic) - G2 (Students during the pandemic):
The average difference between group G3 and group G2 is −1.1370. The 95%
confidence interval for this difference ranges from −1.6449909 to −0.6290091.
The adjusted p-value for this comparison is 0.0000042, indicating a statistically
significant difference between the means of groups G3 and G2.
In case the normality assumptions were not met, an alternative method for
group comparison is the Kruskal-Wallis and Dunn method. Authors such as [6]
conduct a comparison of these two methods. Therefore, to further solidify the
present study, this method is also employed, which is based on comparing the
medians. The obtained results are as follows: chi-squared = 23.44 and p-value
= 8.129e-06. Considering the p-value, it can also be inferred that significant
differences exist between the groups.
In other aspects, when applying the Dunn test, the following comparative
results are obtained (Table 5):
Table 5. Dunn (1964) Kruskal-Wallis multiple comparison
CompariSon Z P.unadj P.adj

G1-G2 −4.1017822 4.099801e-05 6.149702e-05
G1-G3 0.1765668 8.598487e-01 8.598487e-01
G2-G3 4.2783490 1.882847e-05 5.648540e-05
The results are adjusted using the Benjamini-Hochberg method to control

the Type I error when performing multiple comparisons. The interpretation of
the results is as follows:
The comparison between Group 1 (G1) and Group 2 (G2) has a Z value
of approximately -4.10. The unadjusted p-value (P.unadj) is extremely low
(4.099801e-05), indicating a significant difference between these two groups. The
adjusted p-value (P.adj) is also very low (6.149702e-05), suggesting that the dif-
ference remains significant after Benjamini-Hochberg adjustment.
The comparison between Group 1 (G1) and Group 3 (G3) has a Z value close
to 0.18. Both the unadjusted p-value (P.unadj) and the adjusted p-value (P.adj)
are high (around 0.86), indicating no significant evidence of difference between
these groups.
The comparison between Group 2 (G2) and Group 3 (G3) has a Z value of
approximately 4.28. The unadjusted p-value (P.unadj) is very low (1.882847e-
05), suggesting a significant difference between these two groups. The adjusted
p-value (P.adj) is also low (5.648540e-05) after Benjamini-Hochberg adjustment.
It can be observed that the results of both methodologies coincide to a great
extent.
4 Conclusions
The statistical tests showed that there were significant differences in student
performance during the pandemic, but these differences do not determine the
adequacy of the teaching-learning mechanism. Therefore, it would be important
to focus on research to determine the cause of these performance differences. It
is possible that these differences could have been due to the use of Information
and Communication Technologies (ICT) or deficiencies in the evaluation process,
which may have allowed for academic dishonesty such as copying.
The Tukey tests demonstrated that the results before and after the pandemic
are practically the same, suggesting that the teachers likely reverted to their
traditional teaching methods and did not consider the positive aspects of the
technological tools used during the pandemic period.
It can be concluded that using one-way ANOVA, a parametric test, and
using the non-parametric Kruskal-Wallis test led to similar results. Therefore,
it is indifferent whether the evaluation was carried out using either of the two
methods. It can also be observed that the p-value in ANOVA, which is based on
the comparison of means, is 3.63e-07, whereas in the Kruskal-Wallis test, based

on the comparison of medians, it is 8.129e-06. This demonstrates that ANOVA is
more accurate for conditions of normality, homoscedasticity, and independence
of the samples. In this context, it is suggested to update the study on these
groups to determine the evolution of academic performance and establish the
causes through which their performance decreased after the pandemic.
References
1. ALAM, M.J., Majumder, A.: The statistical analysis of rainfall trend and its vari-
ability (1901–2020) in Kolkata, India. Bull. Geogr. Phys. Geogr. Series 23, 5–16
(2022)
2. Azizi, F., Ghasemi, R., Ardalan, M.: Two common mistakes in applying Anova
test: Guide for biological researchers (2022)
3. Castaño Erazo, J.A.: Evaluación técnico-financiera del proceso de fitorremediación
con plantas gramı́neas para mitigar derrames de crudo en el departamento del
Meta, Colombia. B.S. thesis, Fundación Universidad de América (2018)
4. Cayuela, L.: Modelos lineales: Regresión, anova y ancova. Eco Lab, Centro Andaluz
de Medio Ambiente, Universidad de Granada. Notas de clase, pp. 1–57 (2010)
5. Hawkins, D.M.: Quantile-quantile methodology–detailed results. arXiv preprint
arXiv:2303.03215 (2023)
6. Hecke, T.V.: Power study of Anova versus Kruskal-Wallis test. J. Stat. Manag.
Syst. 15(2–3), 241–247 (2012)
7. Hurtado, M.J.R., Silvente, V.B.: Cómo aplicar las pruebas paramétricas bivariadas
t de student y anova en spss. caso práctico. Reire 5(2), 83–100 (2012)
8. Kellner, K.F., Fowler, N.L., Petroelje, T.R., Kautz, T.M., Beyer, D.E., Jr., Belant,
J.L.: ubms: an R package for fitting hierarchical occupancy and N-mixture abun-
dance models in a Bayesian framework. Methods Ecol. Evol. 13(3), 577–584 (2022)
9. Kozak, M., Piepho, H.P.: What’s normal anyway? residual plots are more telling
than significance tests when checking Anova assumptions. J. Agron. Crop Sci.
204(1), 86–98 (2018)
10. Loor, C.E.S., Loor, J.M.V., Meza, C.K.A., Rivera, A.C.B.: La educación ecuatori-
ana vs la pandemia del covid-19. Dominio de las Ciencias 7(2), 105–124 (2021)
11. Mahsin, M.: Data analysis techniques for quantitative study. In: Principles of Social
Research Methodology, pp. 235–255. Springer (2022). https://doi.org/10.1007/978-
981-19-5441-2 16
12. Rea, D.H.A., Palacios, L.A.Z., Yuquilema, J.C.P.: Covid-19 y la educación virtual
ecuatoriana. Investigacion academica 1(2), 53–63 (2020)
13. Terrádez, M., Juan, A.A.: Análisis de la varianza (anova). Universidad Oberta de
Catalunya, Catalunya (2003)
14. Vidal Ledo, M.J., Barciela González Longoria, M.d.l.C., Armenteros Vera, I.:
Impacto de la covid-19 en la educación superior. Educación Médica Superior 35(1)
(2021)
15. Villar Maguiño, J.P.: Enseñanza virtual y desempeño académico en estudiantes de
arquitectura de una universidad pública, huaraz, 2022 (2022)
Investigating Variation in Learners’ Behavior
Through the Lens of Learning Design, Process
Mining and Learning Analytics
Karina Abad1 and Jorge Maldonado-Mahauad1,2(B)

1 Dirección de Innovación Educativa, Universidad de Cuenca, Cuenca, Ecuador
jorge.maldonado@ucuenca.edu.ec
2 Departamento de Ciencias de la Computación, Universidad de Cuenca, Cuenca, Ecuador
Abstract. The analysis of data generated by Massive Open Online Courses

(MOOCs) platforms has been of great relevance in recent years, since it has made
possible to identify patterns of student behavior, success factors associated with
course completion, as well as opportunities for improvement in course design.
However, data collected without an adequate context does not allow to understand
and improve the learning sequences planned in a MOOC. This requires the use of
appropriate frameworks to understand and explain the behavior of students in an
online course. In this sense, the objective of this work is to address the analysis of
student behavior in MOOCs, using Learning Analytics (LA) and Process Mining
(PM) techniques to examine the impact of Learning Design (LD) on the partici-
pation and progress of students. Specifically, we seek to investigate variations in
student behavior in a 6-week programming MOOC. For this, using PM and LA
techniques under the umbrella of LD analysis, data from N = 38,838 students
enrolled in a MOOC was analyzed. The results revealed that students who passed
the course generally spent more time in their study sessions throughout the course,
routinely exceeding one hour. They also demonstrated strong engagement with the
summative assessments. In contrast, students who did not pass spent less time per
session and showed a decrease in the number of weekly sessions, especially from
the third week onwards. This emphasizes the importance of designing course
content in a way that maintains student engagement and motivation beyond the
first few weeks, highlighting the importance of course design in terms of learning
sequences that influence course completion.
Keywords: Learning Analytics · Process Mining · Learning Behaviour · MOOC
1 Introduction
Massive Open Online Courses attract people with different motivations, interests and
learning objectives [1]. However, participation levels and passing scores alone do not
constitute sufficient evidence about the learning process of students, nor do they help to
explain the desired academic success in these types of courses [2]. Regardless of whether
or not a student completes a MOOC, several efforts have been made in recent years to
https://doi.org/10.1007/978-981-99-7353-8_33
Investigating Variation in Learners’ Behavior 443
unveil students’ hidden trajectories. These trajectories are characterized as the result
of students’ interactions with MOOC elements (video lectures, readings, assessments,
etc.) [3, 4]. As a result of these works, it has been possible to identify the variables
that influence the successful completion of a course [5], MOOC dropout by students
[6], variations in the learning process [7], detect plagiarism or cheating in the exams of
online platforms [8]. Most of these efforts have focused on the use of educational data
mining (EDM), process mining (PM) and learning analytics (LA).
To study learner behavior in MOOCs, the aforementioned techniques have been
used in different ways. LA are strongly used to collect, analyze and generate reports of
learning data and their context, aiming to understand and optimize learning and the envi-
ronments in which they happen [9]. EDM employs statistical methods, machine learning
and data mining to analyze educational data generated by online course platforms [9].
PM combines data mining, machine learning and process modeling analysis, to extract
knowledge to improve processes [10].
While the above techniques have provided information on how different groups of
learners participate in MOOCs, some researchers have argued that these approaches
should be embedded in a solid science of learning [11]. Indeed, the design of the online
learning environment is known to influence students’ progress in different types of
learning activities [12]. Success in online learning has been found to be closely related to
the design of learning sequences. However, research on the design of learning sequences
from the pedagogical in MOOCs is at an early stage [13, 14].
This work, based on the studies conducted by [15] and [16] and through the use of
PM and LA, proposes a new approach to study students’ behavior in MOOCs. This study
is an attempt to understand how the Learning Design (LD) of MOOCs might affect the
way students participate and progress in the course.
The document is structured as follows: Sect. 2 introduces related work and research
questions guiding this study, Sect. 3 outlines the methodology, Sect. 4 presents key
results, and Sect. 5 covers conclusions, limitations and future work.
2 Related Work
2.1 Taxonomy of Learning Design
Nowadays, MOOC creators (e.g., Coursera, edX) seek to optimize the design of their
courses in order to attract more students, increase completion rates and improve aca-
demic success [14, 17]. Authors such as Mayer states that learning encompasses a series
of active processes associated with filtering, selecting, organizing and integrating new
information. Organizing these processes in course design, accounting for student char-
acteristics or preferences [18], can lower cognitive load, adjust topic difficulty and infor-
mation presentation, ultimately enhancing knowledge retention [19]. Furthermore, it is
common for learners to distribute their time in different learning activities to obtain
the maximum (subjective) benefit within a limited time frame [15]. Therefore, learning
activities in MOOCs should be aligned with the learning objectives. Thus, the temporal
dynamics of the designed learning activities is of special interest for researchers and
developers of MOOCs.
444 K. Abad and J. Maldonado-Mahauad
Learning design (LD) is the process of designing pedagogically informed learning

activities. In MOOCs, LD can provide a theoretical framework that facilitates the study
and mapping of learner interactions with MOOC resources in learning activities. The
OULDI conceptual framework developed by Conole [20] is used for a better under-
standing about learning design. This proposed taxonomy (see Table 1) provides a way
to abstract different learning activities in a meaningful way, suggesting that all learning
tasks can be categorized as one of the following seven types of activities: 1) assimila-
tive, 2) information management, 3) communication, 4) productive, 5) experiential, 6)
interactive/adaptive, 7) evaluative.
Table 1. OULDI taxonomy for learning design.
Type of activity Example

Assimilative Attending to information Read, write, listen, think about,
access
Finding and handling Searching for and processing List, analyze, collate, plot, find,
information information discover, access, use, gather
Communication Discussing module related Communicate, debate, discuss,
content with at least one other argue, share, report, collaborate,
person (student or tutor) present, describe
Productive Actively constructing an artifact Create, build, make, design,
construct, contribute, complete
Experiential Applying learning in a real-world Practice, apply, mimic,
setting experience, explore, investigate
Interactive/ adaptative Applying learning in a simulated Explore, experiment, trial,
setting improve, model, simulate
Assessment All forms of assessment Write, present, report,
(summative, formative and demonstrate, critique
self-assessment)
LD has been described as an adaptable template aiming to “make the intended teach-
ing and learning structures, the pedagogy, more visible and explicit, thus fostering under-
standing and reflection” [21]. In formal online learning contexts, a link has been found
between course design impact considering LD and online student engagement, where
LD could explain up to 60% of the variance in time spent by students in the Virtual
Teaching and Learning Environment (VTLE). This study employs OULDI to explore
student behavior in a programming MOOC and its relationship to connection to course
resources interactions.
2.2 Learning Sessions in MOOCs

Recent research has shown that study sessions remain the approach used by students to
organize their learning in an online context [22]. A study session can be defined as a
period of time during which the student interacts with the course resources and where
the intervals of inactivity are no longer than 30 min [23, 24]. According to Tough [25],
a session is a period of time devoted to a sequence of similar or related activities that
are not interrupted by other types of activities. In [26], authors identified the learning
strategies that students use in a study session for both a hybrid and an online context
using LA, as result, students were classified based on the identified strategies, resulting in
a characterization of students for the online context into “Samplers”, “Comprehensive”,
and “Targeting” students; as well as for the hybrid context into “intensive”, “strategic”,
“highly strategic”, “selective”, and “highly selective” students [26]. In these works,
authors used the instructional design of the course to try to relate and interpret the
patterns found. That is, they tried to connect the course design with the LA obtained.
However, there is still a lack of works that consider the analysis of the study sessions
with different levels of granularity of the data (macro, meso, micro) and with different
levels of activities in which the course is structured (activities, weeks, modules).
2.3 Approaches for the Study of Student Learning Sequences in MOOCs

In online learning, learner engagement is considered a prerequisite for learning [27]. In a
MOOC, learner engagement can take many forms, where the learner establishes different
levels of interaction and engagement with course activities [28, 29]. For example, Xiong
and colleagues [30] found that the level of learning engagement could be measured by
the number of video lectures viewed, forum postings, quizzes taken, and assignments
completed. Kizilcec and colleagues [31] classified students based on their participation
in the MOOC, considering the number of student interactions with video lectures and
assessments, they found four patterns of participation: 1) samplers - those who only
explore the course, 2) inspectors - those who only watch video lectures, 3) dropouts -
those who drop out of the course as time passes, and 4) completers - those who complete
the entire course. However, in a MOOC, each learner sets his or her own goals and
engages in different ways. A recent study [15] found that students in three MOOCs,
regardless of the course topic, set their own goals and selectively engaged with the
course content. In addition, for each type of learner, research was conducted on which
learning sequences they performed.
There are several studies on the learning sequences performed in a MOOC and how
different types of learners perform their own pathways. Hadwin and colleagues [33]
reported the relationship between interaction sequences and learning outcomes using
transition graphs. Biswas and colleagues [34] employed probabilistic Markov models
to determine the probability of performing MOOC activities in sequence. However, the
use of PM techniques have been found to be appropriate for understanding end-to-end
transitions or other temporal dynamics of learning trajectories within a MOOC. PM
methods allow for a comprehensive understanding of end-to-end learning paths in a
study session followed by large subgroups of learners [32]. In this study, to investigate
learners’ temporal navigation sequences, methods typically associated with Educational
Process Mining will be used [3].
2.4 Research Questions
This paper proposes an exploratory study to extract the learning sequences that students
perform when interacting with the course activities; understand how students organize
the different learning sequences; and figure out how their behavior is related to the
learning design of the MOOC. Specifically, the research questions are:
R.Q.1. ¿What are the most frequent interactions of students during a study session in a
MOOC?
R.Q.2. ¿What behavior patterns are identified?
R.Q.3. ¿How do different groups of students organize the time of their study sessions?
R.Q.4. ¿How does student engagement in a MOOC differ depending on the learning
design?
3 Methodology
This section presents the development of the exploratory study conducted in the con-
text of a Programming MOOC, including a description of the course structure and the
characteristics of the students participating. Then the method used to extract student
interactions with MOOC resources that model student behavior is presented.
3.1 Context: MOOC and Participants
This exploratory study was conducted in the context of a MOOC called “Aprendiendo a
Programar con Python”, offered on the Coursera platform by the Pontificia Universidad
Católica de Chile between July 2017 and January 2018. The course is structured in 6
modules. Each module is made up of a set of lessons. Each lesson is composed of a
set of video-lectures, theoretical quizzes, practical quizzes and readings. The course
has 35 video lectures, 11 readings, 13 practical quizzes and 11 theoretical quizzes. All
modules were available during the 6 weeks of the course. There were 38,838 (N =
38,838) students registered for the course, of which 10,916 did not register any type
of activity. For the analysis, all students were considered, except those who did not
report any activity, leaving a cohort of 27,922 students who registered about 3.5 million
interactions in the course. Fifteen percent of the registered students were between 18 and
24 years old, 50% between 25 and 34 years old, and 22% between 35 and 44 years old.
Sixty-four percent of the students reported being male and 34% were female. Sixty-eight
percent of the students reported having a college education and 19% reported holding a
master’s degree or PhD. Thirty-four percent of the students come from Chile, 20% from
Mexico, 13% from Peru, 12% from Colombia, 7% from Spain and 5% from Ecuador.
3.2 Procedure
For the development of this work, the methodology defined in [15] is used, which is
based on the PM2 methodology presented in [35]. The methodology comprises 4 stages:
1) extraction, 2) log generation, 3) discovery, and 4) analysis.
Extraction Stage. The activity data or log files recorded by the Coursera platform were
extracted. The platform generates a set of CSV files. This set consists of 86 files, only
8 of them were selected (Users, Course grades, Course progress, Course items, Course
item grades, Course item types, Course lesson, and Course module) to analyze student
behavior based on their interactions with the course resources. This was followed by
data cleaning and transformation using Jupyter Notebooks with Python.
Event Log Generation Stage. The event log is defined, this is a file that stores informa-
tion about the interactions of the students with the digital resources of the course and will
be used in the following steps to discover the process model that allows the interpretation
of student behavior in the MOOC. The first step in generating the event log involved
defining two concepts to interpret traces of interactions with the MOOC; specifically,
the concepts ‘study session’ and ‘interaction’ were defined. A study session is a period
of time in which students interact with the course resources and register continuous
activity, with intervals of inactivity no longer than 30 min, after that time, the platform
will terminate the session and consider it as a new one [36]. An interaction is an action
that is saved in the data traces registered by Coursera and that reflects the interaction of a
student with any of the MOOC digital resources. For this work, 17 types of interactions
were defined, in which the student can record when interacting with Video-readings,
formative assessments, summative assessments, forums and readings. Table 2 presents
the interactions and their meanings.
The resulting event log contains: user id, time stamp, the interaction performed, the
session number in which the interaction with the MOOC resources occurs (Table 3).
Model Discovery. The event log file is processed in Disco and Celonis software, the
data of each student is grouped and a count of the frequency of interactions with the
different resources of the course is performed. At this stage, a PM discovery algorithm is
applied, which allows extracting a process model that summarizes the students’ behavior
graphically [37].
Model Analysis. As a result of the previous step, a process model is obtained as shown
in Fig. 1 (Partial Model). This model represents the behavior of students who perform
only video-readings in the MOOC. Each box represents the activity performed and the
number inside represents the number of times that activity is repeated by the students
in the MOOC, in this case the activity VL-Start has been performed 287,033 times. The
arcs or arrows linking two activities represent the transitions from one activity to another
and the number above the arc represents the number of times they went from an activity
“A” to an activity “B”. For example, there is a path that goes from VL-Start to VL-End
and that path was taken 186,713 times.
Table 2. Interactions.
Resource Interaction Meaning

Video - lecture (VL) VL-Start A video-lecture starts
VL-StartAgain Re-start a video-lecture that has not
been completed
VL-End Video-lecture completed
VL-Repeat Return to a video lecture completed
in the past to review it
Formative Assessments (FA) FA-Start A formative assessment starts
FA-StartAgain Re-start a formative assessment that
has not been completed
FA-End Formative assessment completed
FA-Repeat Return to a formative assessment
completed in the past to review it
Summative Assessments (SA) SA-Start Summative assessment starts
SA-StartAgain Re-start a summative assessment that
has not been completed
SA-End Summative assessment completed
SA-Repeat Return to a summative assessment
completed in the past to review it
Forums Open-Question Start a question in the forum
Update-Question Update a question in the forum
Answer-Question Answer a forum question
Update-Answer Update a forum answer
Readings Reading Interact with a reading
Table 3. Event log example
Id Usuario Time stamp Interacción #Sesión

28 1434522567 Assessment-Formative 1
28 1434522567 Video-Lecture-Complete 1
161 1430520885 Assessment-Formative 1
161 1430520885 Navigation 1
161 1430520885 Navigation 1
Fig. 1. Example of the Process Model VL-Start to VL-End
4 Results
This section presents the results obtained from the analysis of the Event Logs. The results
have been organized according to the research questions.
R.Q.1. ¿What are the Most Frequent Interactions of Students During a Study Ses-
sion in a MOOC?
Based on the defined Event Log, the data for each student was grouped, and a count of
the frequency of interactions with different course resources was carried out throughout
the 6 weeks of the course. In total, 1 ’060.198 interactions were identified, distributed
among 188,050 sessions. Sessions have a duration of 20.1 min on average. Figure 2
shows the resulting process model.
It can be seen that the most frequent interactions in the study sessions are related
to video lectures (VL-Start, VL-End, VL-Repeat, VL-StartAgain), which have a stronger
blue color in relation to the rest of the interactions. This is an expected result since most
of the MOOC contents are video lecture type resources. It can be seen that the activity
VL-Start is the most repeated (287,033 repetitions), representing 27.07% of the total
number of interactions. This activity has a strong transition with VL-End, which is the
second common activity. This means that the students have a strong learning sequence
based on the video lectures. As can be seen in Fig. 2, it is quite common for students
to start some activity, not complete it, and not return until the next session. It is also a
common variant for students to dedicate sessions to doing one or more video lectures.
From the models generated by the Celonis tool, the data on interaction frequencies
per week of the MOOC can be seen in Table 4, where the highest number of interactions
are associated with Video Lectures. Conversely, the lowest number of interactions occur
with formative assessments and forums, where interactions do not exceed 435 per week.
R.Q.2. ¿What Behavior Patterns are Identified?

To group students, the sequences of activities that students perform in their study ses-
sions in the MOOC were considered. The following interaction frequencies were used
as grouping variables: 1) formative assessments, 2) summative assessments, 3) supple-
ments, 4) Forum, 5) Video-lectures, 6) the duration of the sessions. The above indicators
were considered regardless of whether they passed or failed the course, since it is desired
to identify the behavioral patterns that students exhibit when engaging with the MOOC.
The clustering of the students in the MOOC course was carried out similarly to [15]
and [16]. After creating the dendrogram, applying the elbow and silhouette method, and
using the k-means algorithm, 2 clusters were generated with the following characteristics:
Fig. 2. Process model of student interactions with MOOC resources
Table 4. Count of the interactions with resources of the course per week
Week VL Summative Assessment Formative Assessment Reading Forum

1 420268 30161 35207 67219 435
2 176592 31118 0 19130 426
3 68582 23191 0 7290 148
4 53319 24581 0 10931 143
5 37180 14624 0 5624 72
6 21267 9949 0 2605 127
Cluster 0: Groups students who take the course without a specific order, they mostly
start working with evaluations and then perform readings or video lectures. They have
an exploratory behavior and complete the assessments, mostly the summative ones;
formative assessments are taken only once, i.e., they do not start again or repeat. Seventy-
eight percent students of Cluster 0 have completed the course. The behavioral patterns
found are shown in Fig. 3 and listed next:
• SA-StartAgain → SA-End → SA-Repeat → Reading → SA-Start → SA-End →

SA-StartAgain
• SA-StartAgain → SA-End → SA-Repeat → Reading → FA-Start → FA-End →
VL-Start → VL-End → SA-Start → SA-End
• SA-StartAgain → SA-End → SA-StartAgain → Answer-Question → Update-
Question
• SA-StartAgain → SA-End → SA-StartAgain → OpenQuestion → Reading → SA-
Start → SA-End
Fig. 3. Behavioral patterns in Cluster 0
Cluster 1: They are characterized by reviewing the course material first, i.e., video
reading and reading activities. They minimally make attempts to solve assessments and
present low activity. No student in cluster 1 completed the course. The behavioral patterns
found are shown in Fig. 4 and listed next:
• VL-Start → VL-StartAgain → VL-End → VL-Start

• VL-Start → VL-StartAgain → VL-End → VL-Start → VL-End → VL-Start
• VL-Start → VL-StartAgain → VL-End → Reading → SA-Start → VL-Start
• VL-Start → VL-StartAgain → VL-End → Reading → SA-Start → VL-Start →
VL-End → VL-Start
• VL-Start → VL-StartAgain → VL-End → Reading → SA-Start → SA-Repeat →

SA-Start → VL-Start
• VL-Inicia → VL-IniciaNuevamente → VL-Termina → Lectura → Eva-Forma-Inicia
→ Eva-Forma- Termina → VL-Inicia
• VL-Start → VL-StartAgain → VL-End → Reading → FA-End → FA-StartAgain →
FA-End → VL-Start
R.Q.3. ¿How Do Different Groups of Students Organize the Time of Their Study
Sessions?
To answer this question, the time students spend on each session of the course was
analyzed. Students in cluster 0 who pass the course generally dedicate more time to
their work sessions than students in the same cluster who did not pass; this behavior is
maintained during the six weeks. Students in cluster 1 have shorter study sessions than
the students in cluster 0; likewise, the number of sessions per week is lower in cluster 1
from the third week onwards. Figures 5 and 6 show the evolution of the time spent by
students in each cluster per session during the six weeks of the MOOC.
Fig. 4. Behavioral patterns in Cluster 1
Furthermore, an analysis of connection frequency was conducted based on a session

duration definition, categorizing sessions as short if they lasted less than 15 min, medium
if they last between 15 and 45 min, and long if they exceeded 45 min. This approach
provides a clearer view of how students structure their study sessions. Figure 7 illustrates
the evolution of connections by students within each cluster throughout the course. It’s
Fig. 5. Dedication time per session for Cluster 0 over the six-week MOOC.
noteworthy that starting from week 3, the number of connections in cluster 1 significantly
drops, which could suggest that this is when the majority of students disengage from the
MOOC course.
Fig. 6. Dedication time per session for Cluster 1 over the six-week MOOC.
R.Q.4. ¿How Does Student Engagement in a MOOC Differ Depending on the Learn-
ing Design?
In order to analyze students’ engagement with different elements of the learning design,
the information associating each learning element with primary and secondary activi-
ties based on the OULDI taxonomy is included, classifying each learning element as
assimilative, experiential, or evaluative as a primary type, and experiential or interactive
as a secondary type. Subsequently, the relative frequency of interactions between each
Fig. 7. Connections of students from clusters 0 and 1 during the six weeks of the course.
group and the learning elements is calculated. Table 5 presents the results. It can be
observed that students in cluster 0 exhibit higher engagement (71.9%) with summative
assessments and assimilative type video lectures. Students in cluster 1 engage to a greater
extent (66.01%) with formative assessments and with readings and experiential video
lectures.
The same analysis was conducted to determine the engagement among students in
Cluster 0 who passed and those who did not pass the course. The results are shown in
Table 6. It can be observed that those who do not pass demonstrate engagement in both
formative and summative assessments (both of the assessment type). Students who pass
the course show higher engagement with summative assessments, followed by primary
type assimilative video lectures and secondary type experiential video lectures.
Table 5. Engagement of student groups with learning elements of the OULDI Taxonomy.
Cluster Primary Activity Secondary activity Learning Relative Frequency

element (%)
0 Assessments Interactive Summative 71.09
assessment
1 Assessments Interactive Summative 28.91
assessment
0 Assessments Interactive Formative 33.99
Assessment
1 Assessments Interactive Formative 66.01
Assessment
0 Experiential Interactive Video-lecture 26.51
1 Experiential Interactive Video-lecture 73.49
0 Experiential Interactive Reading 27.03
1 Experiential Interactive Reading 72.97
0 Assimilative Experiential Video-lecture 100
Table 6. Engagement of student groups with learning elements according to the OULDI
Taxonomy - Cluster 0.
State Primary activity Secondary activity Learning Relative Frequency

element (%)
Pass Assessments Interactive Summative 41.22
assessment
Not pass Assessments Interactive Summative 58.78
assessment
Pass Assessments Interactive Formative 12.54
Assessment
Not pass Assessments Interactive Formative 87.46
Assessment
Pass Experiential Interactive Video-lecture 11.90
Not pass Experiential Interactive Video-lecture 88.10
Pass Experiential Interactive Reading 11.31
Not pass Experiential Interactive Reading 88.69
Pass Assimilative Experiential Video-lecture 17.89
Not pass Assimilative Experiential Video-lecture 82.11
5 Conclusions and Limitations of the Study

This study has comprehensively addressed the analysis of student behavior in Mas-
sive Open Online Courses (MOOCs), using Learning Analytics (LA) and Process Min-
ing (PM) to examine the impact of Learning Design (LD) on student engagement and
progress in these courses. Through the conducted analyses, two main participation groups
in MOOCs have been identified, offering a deeper insight into the diverse trajectories of
students.
After analyzing the groups, it was possible to identify that students who pass the
course generally dedicate more time to their sessions during the entire course (more
than 1 h), while students who did not pass and belong to cluster 1 not only spend less
time per session but also engage in fewer weekly sessions.
Likewise, it was observed that starting from the third week of the course, the number
of sessions significantly dropped in cluster 1, indicating that until the second week,
students show interest in the course, but by the third week, most of them do not reconnect.
This could be an indicator for course design, where content of weeks 2 and 3 should
capture student interest, keep them engaged, and motivate them to continue.
Regarding students’ commitment to design elements, it was found that there is a
strong commitment to summative assessments by students in cluster 0, which encom-
passes all students who passed the MOOC and represents 78.4% of the total students in
that cluster. Furthermore, it was observed that students in cluster 1 were engaged with a
larger number of elements, despite not passing the course. This could be due to the com-
mitment metric being relative frequency and considering that around 80% of enrolled
students did not pass. When analyzing the behavior only of those who did pass, it was
identified that they have higher engagement with summative assessments (interactive,
assessment), followed by video lectures (assimilative, experiential).
The findings of this study have demonstrated that the design of the online learning
environment, particularly learning sequences, plays a crucial role in the success of learn-
ing. It is acknowledged that research on the pedagogical design of learning sequences in
MOOCs is still in its early stages, suggesting the need for future research and develop-
ments in this field. This comprehensive approach that combines LA and PM techniques
has enriched our understanding of student behavior in MOOCs and has opened new
perspectives towards advancing a robust science of online learning.
Despite the valuable results obtained, this study also presents certain limitations that
should be considered when interpreting its conclusions. First, due to the retrospective
nature of the analysis, the collected information was based solely on data available on the
MOOC platforms, potentially excluding some relevant variables that were not recorded.
Additionally, the focus on Learning Design (LD) in MOOCs might have overlooked
other possible influences on student engagement and progress, such as external factors
or individual motivations. Furthermore, the generalizability of the findings could be
limited by the specific selection of MOOCs and participants used in this study. On the
other hand, although Learning Analytics (LA) and Process Mining (PM) techniques
have been employed, these tools may have their own limitations and inherent biases.
Lastly, the data analysis mainly centered on quantitative patterns, which might not fully
capture the richness and complexity of students’ learning experiences. Despite these
limitations, this work lays the groundwork for future research that more comprehensively
and contextually addresses the role of learning design in MOOCs and its influence on
student behavior.
As future work, we plan to extend this analysis to other courses to validate the
findings and determine whether behavior regarding engagement remains consistent.
Acknowledgment. We are grateful to the “Fundación Carolina” for awarding to the authors the
2022 short-term postdoctoral scholarship, which contributed to the development of this research
work. This work has been co-funded by the University of Cuenca (VIUC), Cuenca-Ecuador,
under the project “Learning analytics for the study of self-regulated learning strategies in a hybrid
learning context (DIUC_XVIII_2019_54)”, by the Pontificia Universidad Católica de Chile and
the Direction of Engineering Education–DEI.
References
1. Kizilcec, R.F., Pérez-Sanagustín, M., Maldonado, J.J.: Self-regulated learning strategies pre-
dict learner behavior and goal attainment in Massive Open Online Courses. Comput. Educ.
104, 18–33 (2017)
2. Henderikx, M., Kreijns, K., Kalz, M.: An alternative approach for measuring MOOC success
based on participant’s intentions (2017)
3. Maldonado, J.J., Palta, R., Vázquez, J., Bermeo, J.L., Pérez-Sanagustín, M., Munoz-Gama, J.:
Exploring differences in how learners navigate in MOOCs based on self-regulated learning
and learning styles: a process mining approach. In: 2016 XLII Latin American Computing
Conference (CLEI), pp. 1–12. IEEE, October 2016
4. Matcha, W., et al.: Analytics of learning strategies: role of course design and delivery modality.
J. Learn. Anal. 7(2), 45–71 (2020)
5. Maldonado-Mahauad, J., Pérez-Sanagustín, M., Moreno-Marcos, P.M., Alario-Hoyos, C.,
Muñoz-Merino, P.J., Delgado-Kloos, C.: Predicting learners’ success in a self-paced MOOC
through sequence patterns of self-regulated learning. In: Pammer-Schindler, V., Pérez-
Sanagustín, M., Drachsler, H., Elferink, R., Scheffel, M. (eds.) EC-TEL 2018. LNCS, vol.
11082, pp. 355–369. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-98572-5_27
6. Moreno-Marcos, P.M., Muñoz-Merino, P.J., Maldonado-Mahauad, J., Pérez-Sanagustín, M.,
Alario-Hoyos, C., Kloos, C.D.: Temporal analysis for dropout prediction using self-regulated
learning strategies in self-paced MOOCs. Comput. Educ. 145, 103728 (2020)
7. Rizvi, S., Rienties, B., Rogaten, J., Kizilcec, R.F.: Investigating variation in learning processes
in a FutureLearn MOOC. J. Comput. High. Educ. 32, 162–181 (2020)
8. Alexandron, G., Lee, S.: Detecting Cheaters in MOOCs Using Item Response Theory and
Learning Analytics (n.d.)
9. Calvet Liñán, L., Juan Pérez, Á.A.: Educational Data Mining and Learning Analytics: differ-
ences, similarities, and time evolution. RUSC Univ. Knowl. Soc. J. 12(3), 98 (2015). https://
doi.org/10.7238/rusc.v12i3.2515
10. van der Aalst, W.: Process mining: the missing link. In: Process Mining. Springer, Heidelberg
(2016). https://doi.org/10.1007/978-3-662-49851-4_2
11. Mangaroska, K., Giannakos, M.N.: Learning analytics for learning design: a systematic lit-
erature review of analytics-driven design to enhance learning. IEEE Trans. Learn. Technol.
(2018). https://doi.org/10.1109/TLT.2018.2868673
12. Nguyen, Q., Rienties, B., Toetenel, L.: Mixing and matching learning design and learning
analytics. In: Zaphiris, P., Ioannou, A. (eds.) LCT 2017, Part II. LNCS, vol. 10296, pp. 302–
316. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-58515-4_24
13. Davis, D., Chen, G., Hauff, C., Houben, G.-J.: Gauging MOOC learners’ adherence to the
designed learning path. In: 9th International Conference on EDM (2016)
14. Sergis, S., Sampson, D.G., Pelliccione, L.: Educational design for MOOCs: design consid-
erations for technology-supported learning at large scale. In: Jemni, M., Kinshuk, Khribi,
M. (eds.) Open Education: From OERs to MOOCs. LNCS, pp. 39–71. Springer, Heidelberg
(2017). https://doi.org/10.1007/978-3-662-52925-6_3
15. Maldonado-Mahauad, J., Pérez-Sanagustín, M., Kizilcec, R.F., Morales, N., Munoz-Gam, J.:
Mining theory-based patterns from Big data: Identifying self-regulated learning strategies in
Massive Open Online Courses. Comput. Hum. Behav. (2018). https://doi.org/10.1016/j.chb.
2017.11.011
16. Maldonado-Mahauad, J., Alario-Hoyos, C., Delgado Kloos, C., Perez-Sanagustin, M.: Adap-
tation of a process mining methodology to analyse learning strategies in a synchronous mas-
sive open online course. In: Herrera-Tapia, J., Rodriguez-Morales, G., Fonseca C., E.R.,
Berrezueta-Guzman, S. (eds.) TICEC 2022. CCIS, vol. 1648, pp. 117–136. Springer, Cham
(2022). https://doi.org/10.1007/978-3-031-18272-3_9
17. Rai, L., Chunrao, D.: Influencing factors of success and failure in MOOC and general analysis
of learner behavior. Int. J. Inf. Educ. Technol. 6(4), 262 (2016)
18. Maldonado, J.J., Pérez-Sanagustín, M., Bermeo, J.L., Muñoz, L., Pacheco, G., Espinoza,
I.: Flipping the classroom with MOOCs. A pilot study exploring differences between self-
regulated learners. In: 2017 Twelfth Latin American Conference on Learning Technologies
(LACLO), pp. 1–8. IEEE, October 2017
19. Mayer, R.E.: The Cambridge Handbook of Multimedia Learning. Cambridge University
Press, Cambridge (2005)
20. Conole, G.: Designing for Learning in An Open World, vol. 4. Springer, Berlin (2012)
21. Cross, S., Conole, G.: Learn about learning design. Part of the OU Learn about series of
guides. The Open University: Milton Keynes (2009). http://www.open.ac.uk/blogs/OULDI/
wp-content/uploads/2010/11/Learn-about-learning-design_v7.doc
22. de Barba, P.G., Malekian, D., Oliveira, E.A., Bailey, J., Ryan, T., Kennedy, G.: The importance
and meaning of session behaviour in a MOOC. Comput. Educ. 146 (2019), 103772 (2020).
https://doi.org/10.1016/j.compedu.2019.103772
23. Romero, M., Usart, M.: The time factor in MOOCS: time-on-task, interaction temporal
patterns, and time perspectives in a MOOC. In: CSEDU 2014 - Proceedings of the 6th
International Conference on Computer Supported Education (2014)
24. Sapunar-Opazo, D., Pérez-Álvarez, R., Maldonado-Mahauad, J., Alario-Hoyos, C., Pérez-
Sanagustín, M.: Analyzing learners’ activity beyond the MOOC. In: CEUR Workshop
Proceedings (2018)
25. Tough, A.: The adult’s learning projects. A fresh approach to theory and practice in adult
learning (1971)
26. Fincham, O.E., Gasevic, D.V., Jovanovic, J.M., Pardo, A.: From study tactics to learning strate-
gies: an analytical method for extracting interpretable representations. IEEE Trans. Learn.
Technol. 12, 59–72 (2018)
27. Guo, P.J., Kim, J., Rubin, R.: How video production affects student engagement: an empirical
study of MOOC videos. In: Proceedings of the First ACM Conference on Learning@ Scale
Conference, pp. 41–50 (2014)
28. Hew, K.F., Cheung, W.S.: Students’ and instructors’ use of massive open online courses
(MOOCs): motivations and challenges. Educ. Res. Rev. 12, 45–58 (2014). https://doi.org/10.
1016/j.edurev.2014.05.001
29. Milligan, C., Littlejohn, A., Margaryan, A.: Patterns of engagement in connectivist MOOCs.
J. Online Learn. Teach. 9(2), 149–159 (2013)
30. Xiong, Y., Li, H., Kornhaber, M.L., Suen, H.K., Pursel, B., Goins, D.D.: Examining the
relations among student motivation, engagement, and retention in a MOOC: a structural
equation modeling approach. Glob. Educ. Rev. 2(3), 23–33 (2015)
31. Kizilcec, R.F., Piech, C., Schneider, E.: Deconstructing disengagement: analyzing learner
subpopulations in massive open online courses. In: Proceedings of the Third International
Conference on Learning Analytics and Knowledge, pp. 170–179 (2013)
32. Sonnenberg, C., Bannert, M.: Discovering the effects of metacognitive prompts on the sequen-
tial structure of SRL-processes using process mining techniques. J. Learn. Anal. 2(1), 72–100
(2015)
33. Hadwin, A., Nesbit, J., Jamieson-Noel, D., Code, J., Winne, P.: Examining trace data to
explore self-regulated learning. Metacognition Learn. 2(2–3), 107–124 (2007)
34. Biswas, G., Jeong, H., Kinnebrew, J.S., Sulcer, B., Roscoe, R.O.D.: Measuring self-regulated
learning skills through social interactions in a teachable agent environment. Res. Pract.
Technol. Enhanc. Learn. 5(02), 123–152 (2010)
35. van Eck, M.L., Lu, X., Leemans, S.J.J., van der Aalst, W.M.P.: PM$^2$: a process mining
project methodology. In: Zdravkovic, J., Kirikova, M., Johannesson, P. (eds.) CAiSE. LNCS,
vol. 9097, pp. 297–313. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-19069-
3_19
36. Kovanović, V., Gašević, D., Dawson, S., Joksimović, S., Baker, R.S., Hatala, M.: Penetrating
the black box of time-on-task estimation. In: ACM International Conference Proceeding
Series (2015). https://doi.org/10.1145/2723576.2723623
37. Günther, C.W., Rozinat, A.: Disco: discover your processes. BPM 940, 40–44 (2012)
Expectations of High School Teachers
Regarding the Use of Learning Analytics
Geane Biancato1 , Emanuel Marques Queiroga2 , Roberto Muñoz3 ,

Vinicius Ramos1 , Tiago Thompsen Primo4 , Virgı́nia Rodés5 ,
and Cristian Cechinel1(B)
1
Universidade Federal de Santa Catarina, Rod. Gov. Jorge Lacerda,
3201 Jardim das Avenidas, Araranguá, SC 88906-072, Brazil
v.ramos@ufsc.br, contato@cristiancechinel.pro.br
2
Instituto Federal de Educação, Ciência e Tecnologia Sul-rio-grandense,
Rua Gonçalves Chaves, 3218 - Centro, Pelotas, RS, Brazil
3
Universidad de Valparaı́so, 2362735 Valparaı́so, Chile
roberto.munoz@uv.cl
4
Universidade Federal de Pelotas, Rua Gomes Carneiro, 1 – Centro,
96010-610 Pelotas, RS, Brazil
5
Tecnológico de Monterrey, Av. Eugenio Garza Sada 2501 Sur,
Tecnológico, 64849 Monterrey, Mexico
Abstract. The search for improved teaching methods and a more per-
sonalized education has been a constant challenge in the educational
field. Learning Analytics (LA), the measurement, collection, analysis,
and reporting of data about students and their contexts, has emerged as
a promising approach to understand and optimize learning environments.
This paper focuses on identifying the expectations of high school teach-
ers in the southern region of Santa Catarina, Brazil, regarding the use
of LA. Through a questionnaire distributed to high school teachers, the
study investigated their perceptions and opinions about the application
of LA in their pedagogical practices. The results indicate that teach-
ers have positive expectations regarding the potential impact of LA on
improving teaching and learning. They understand the concept of LA and
believe that the feedback provided by the system should be presented in
a clear and accessible format. Furthermore, the study reveals that teach-
ers expect support and guidance from educational institutions in access-
ing and interpreting analytical results. They emphasize the importance
of proper training for all stakeholders involved in the implementation of
LA. Overall, the study highlights the teachers’ positive attitudes towards
utilizing LA as a powerful tool to enhance education and promote stu-
dents’ academic and professional development in the southern region of
Santa Catarina.
Keywords: Teachers · Learning Analytics · Survey · Brazil · School
https://doi.org/10.1007/978-981-99-7353-8_34
460 G. Biancato et al.
1 Introduction
The pursuit of enhancing teaching methods and providing a more effective and
personalized education has been a constant challenge in the field of education.
In this context, the use of technologies and data analysis has gained prominence
as tools capable of providing valuable insights into the students’ learning pro-
cess [1]. A promising approach that has stood out is Learning Analytics (LA),
which involves the measurement, collection, analysis, and reporting of data about
students and their contexts, with the aim of understanding and optimizing learn-
ing and the environments in which it takes place [2].
In this regard, investigating the practices adopted in teaching, as well as
analyzing teachers’ expectations and how student data is treated and collected,
becomes fundamental for the development and enhancement of pedagogical
strategies [3,4]. In this context, the present article aims to introduce an app-
roach for identifying the expectations of high school teachers in the Southern
Santa Catarina region regarding the use of Learning Analytics.
The state of Santa Catarina, renowned for its cultural and economic diver-
sity, also faces the common challenges of the Brazilian educational system. Thus,
comprehending teachers’ expectations regarding the use of educational technolo-
gies like Learning Analytics, as well as how this data is being collected and uti-
lized, is crucial for gaining a broader overview of the needs and improvement
opportunities within the local educational landscape.
This work was carried out through the adaptation of a questionnaire model
derived from the SHEILA project (Supporting Higher Education to Integrate
Learning Analytics), which aims to assist European universities in becoming
more mature and responsible users of digital data concerning their students [5].
As such, the questionnaire was translated, adapted, validated, and provided to
high school teachers in Southern Santa Catarina. Our aim was to investigate
educators’ perceptions and opinions regarding the implementation of Learning
Analytics in their pedagogical practices.
The significance of this research lies in its potential to identify gaps and
potential improvements in the use of Learning Analytics, enabling educational
managers and teachers to make data-driven decisions with the aim of optimizing
the teaching experience and, consequently, student performance. Furthermore,
understanding teachers’ expectations regarding the adoption of educational tech-
nologies can support the development of teacher training strategies and encour-
age more effective integration of these innovative tools in the school environment.
Considering that, we propose the following research questions:
• RQ 1: What are the expectations of high school teachers in Southern Santa
Catarina regarding the use of Learning Analytics (LA) as an educational tool?
• RQ 2: What are the teachers’ perceptions regarding the impact and effective-
ness of Learning Analytics in optimizing learning and enhancing the educa-
tional environment?
• RQ 3: How can teachers’ expectations and perceptions influence the adoption
and integration of Learning Analytics in the educational scenario of Southern
Santa Catarina?
Expectations of High School Teachers 461
Therefore, this article aims to contribute to the advancement of knowledge in

the field of Learning Analytics applied to education, emphasizing the significance
of teachers’ perspectives and the impact of this approach within the educational
context of Southern Santa Catarina. Through this study, it is expected to stimu-
late discussions and initiatives aimed at promoting a more personalized, efficient,
and education aligned with the needs and expectations of students and educators
in the region.
2 Review of the Literature

With the increasing use of information systems, it has become more and more
common for educational institutions to seek automation of activities and collect
diverse data in order to enhance students’ learning experiences and school effec-
tiveness [6]. In this context, Learning Analytics (LA) stands out as a field focused
on utilizing data analysis techniques and methods to gain relevant insights into
the learning process and student performance [7].
However, assessing the success and effectiveness of LA tools goes beyond
technical criteria, as it also involves human challenges when integrating innova-
tive technology into real contexts [8]. Therefore, studies have indicated that LA
can positively support student performance, but it’s important to understand
users’ perceptions of LA and its use in different educational settings [9].
Hence, comprehending the perceptions of both teachers and students regard-
ing the use of LA is essential for understanding how these tools can enhance
learning and educational practices [5]. In this aspect, focusing on the human fac-
tor is crucial for the successful implementation of LA initiatives. This is mainly
due to the sense of arbitrariness in changes to work practices imposed by LA
and the empowerment of users in decision-making [10].
The increasing “datafication” in education, which transforms various aspects
of life into data, has significantly impacted the educational sector [11]. In this
regard, the field of learning analytics still lacks a common understanding among
education stakeholders [4,5]. Research involving stakeholders has emphasized
the importance of students’ autonomy, critical reflection, and analytical skills [4,
5]. Therefore, understanding teachers’ expectations about learning analytics is
essential to inform decisions and enhance educational practices [12,13].
On the other hand, the utilization of learning analytics can enable the iden-
tification of subgroups of students with specific expectations and characteristics,
allowing for a more individualized approach [14]. The benefits also extend to
school administrators, who can make data-driven decisions at various levels of
the institution [2,13,15].
Based on these perspectives, numerous studies seek to measure the impact
and expectations and visions at different levels of educational understanding. In
this context, [16] points out that the diversity of expectations and perceptions
among stakeholders is a key element in the successful implementation of learning
analytics. Meanwhile, [17] define that understanding students’ needs regarding
learning analytics services and segmenting them into distinct groups can lead to
more effective solutions.
Furthermore, according to [18], proper data collection is essential to develop

a LA model that caters to the specific needs of educational institutions and users.
In this regard, [7] demonstrates that considering the expectations of higher levels
of educational management is crucial to ensuring the satisfaction and effective-
ness of LA services. These higher-level stakeholders often decide on the institu-
tionalization of LA use and define public policies based on insights generated
from the data.
However, as indicated by [19], to promote changes in teaching practices and
enhance student progress, teachers’ perceptions of LA are a crucial factor since
they are the ones who carry out the defined practices. Thus, the benefits of
LA can be maximized by understanding the expectations and perceptions of
key stakeholders and by developing a comprehensive LA model that supports
pedagogical practice [20].
In summary, Learning Analytics has emerged as a promising research area in
the educational field, aiming to automate activities and gather diverse data to
enhance students’ learning experiences and the effectiveness of educational insti-
tutions. However, evaluating the success and effectiveness of LA tools involves
both technical and, importantly, human challenges.
In this context, users’ perceptions, especially those of teachers and students,
regarding the use of LA are crucial to understanding how these tools can pos-
itively impact learning and educational practice. Meanwhile, the focus on the
human factor and understanding stakeholders’ expectations are essential ele-
ments for the successful implementation of LA tools and achieving significant
outcomes in the educational process. Therefore, research in this area has empha-
sized the importance of listening to and considering the needs and expectations
of key actors involved in the educational process in order to develop more effec-
tive solutions and improve the quality of teaching and learning.
3 Methodology
3.1 Context of the Research
The research context was the southern region of the state of Santa Catarina,
Brazil, composed of 45 municipalities that encompass a diverse area, including
significant cities such as Criciúma, Tubarão, Araranguá, Içara, among others,
with a total population of approximately 1,344,779 inhabitants.
This region is recognized for its significant activity in the industrial, agri-
cultural, and commercial sectors, which directly impacts the local economy and
the demand for quality education. In light of this, the southern region has been
directing efforts towards investing in educational infrastructure, teacher train-
ing, and implementing pedagogical projects aligned with the needs of the local
community. It is home to various centers for technical and higher education.
In the development of the research procedure, we initially sought out the
schools where potential participants work. To achieve this, we utilized the web-
site of the National Institute for Educational Studies and Research Anı́sio Teix-
eira (INEP), which allows searches based on cities and educational modalities,
providing agility and efficiency in the process. After applying specific filters,
focusing on schools offering high school education, 146 schools were identified
in the catalog, with 37 of them being categorized as private and 109 as public.
Armed with this data, contact was established with the schools to proceed with
the research.
3.2 Data Collection Instrument

The data collection instrument involved the distribution of a questionnaire based
on the SHEILA project, which was translated and adapted by the research group.
The primary objective of this questionnaire was to investigate teachers’ under-
standing of basic Learning Analytics concepts, their inquiries and expectations
regarding the process, and their initial perceptions about the use of automated
techniques for analyzing educational data.
In this regard, the questionnaire initially consists of a basic explanation of
the Learning Analytics concept, based on the definition presented in [21] as
referenced in Table 1. Following this, there are 31 questions covering the following
themes:
• Demographic Information: Gender, years of teaching experience, institution
and institutional model, city, and format of classes during the pandemic.
• Knowledge about Learning Analytics (LA): Understanding of the concept of
LA on a scale of 1 to 5.
• Guidance and Support: Opinions about the importance of receiving guidance
on data access and professional development opportunities in LA.
• Sharing and Feedback: Opinions about open discussions for sharing experi-
ences of using analyses and accessing data on student progress.
• Data Quality: Opinions about the importance of accurate data and compre-
hensible presentation of analysis feedback.
• Scope of Analysis: Opinions about including all students and disciplines in
the analysis.
• Action and Support: Opinions about the obligation to take action in cases
of at-risk students and feedback for the development of academic and profes-
sional skills.
• Knowledge about Collected Data: Identification of the types of data collected
at the institution.
• Implementation of Educational Data Analysis: Implementation status and
knowledge about tools/software used.
• Adequate Training: Opinions about the necessity of training for different
stakeholders involved in LA implementation.
• Suggestions and Comments: Space for additional suggestions and comments.
In this regard, the divisions of the questions followed the following perspec-
tive. The initial questions from 1 to 8 had a demographic informative charac-
ter and aimed to familiarize with the respondent group, their institutions of
operation (public or private), the mode of education, and whether they faced
difficulties in obtaining student data during the pandemic.
Question 9 asked participants to indicate, on a scale of 1 to 5 based on the

Likert scale, how well they understood the concept of Learning Analytics as
presented in the questionnaire.
Questions 10 to 29 were directed to assess teachers’ expectations regarding
learning analytics. Topics such as the need for guidance to access students’ edu-
cational data, professional development in using learning analytics for teaching,
the importance of open discussions to share experiences, among other aspects,
were addressed. These questions were also answered using a Likert rating scale
(1-5).
The final two questions (30 and 31) allowed participants to share their sug-
gestions for the implementation of learning analytics and provide additional
comments and suggestions about the applied questionnaire.
The questionnaire addressed relevant topics related to teachers’ understand-
ing, expectations, and suggestions regarding the use of Learning Analytics in
high schools in Southern Santa Catarina. The obtained responses provide criti-
cal insights into teachers’ perspectives on this topic, contributing to the under-
standing and improvement of the teaching and learning process.
The questionnaire was constructed using Google Forms and sent to the emails
provided by the schools. Following this process, we received responses from 69
educators in 19 different cities who completed the questionnaire.
3.2.1 Instrument Validation

The topic of Learning Analytics and instruments for acceptance assessment,
particularly when focused on high school education, is still relatively unexplored
in academic literature. Given this context, it was necessary to adapt and validate
the original questionnaire developed by the SHEILA project to cater to the
specificities of teachers at this educational stage. The alterations aimed to make
the questionnaire more comprehensible, especially for those teachers who might
not have prior familiarity with the technical terms of learning analytics.
The research instrument developed by the SHEILA project was initially vali-
dated with 200 participants from various regions around the world, as mentioned
in the study by [5]. However, for application to high school teachers, significant
adaptations were necessary to ensure that the questionnaire was suitable for the
context and level of knowledge of these professionals.
The adaptation of the questionnaire was conducted through a validation step
involving a small group of high school teachers. These teachers provided valuable
input, suggesting adjustments in terms and including additional explanations to
enhance the understanding of the questions. The proposed modifications were
thoughtfully incorporated into the questionnaire to ensure its clarity and rele-
vance for the specific research at hand.
Table 1. Basic Concept of Learning Analytics Presented in the Collection Instrument
Expectations of Teachers Regarding the Use of Automated Techniques for

Educational
Data Analysis in High Schools in Southern Santa Catarina
The use of quantitative analyses supported by computational techniques has
become increasingly common in education, particularly through the method of
Learning
Analytics (LA). Learning Analytics is the measurement, collection, analysis, and
reporting of data about students and their contexts for the purpose of
understanding and optimizing learning and the environments in which it takes
place [21]
The present study aims to investigate Learning Analytics (LA) in the context of
high schools in Southern Santa Catarina and their potential to assist in teaching
practices, potentially aiding in the understanding and improvement of the
teaching and learning process
Learning analysis involves the collection of educational data, such as grades, class
attendance, or the number of accesses to online resources from various learning
environments, to better inform how students learn and engage in their studies
Educational data is used to implement support services that assist student
learning, such as developing early warning systems for those at risk of failing or
dropping out of a course, personalized learning environments, and improving
student feedback processes
Among these approaches, Learning Analytics (LA) proposes the collection,
analysis, and reporting of educational data, aiding in the understanding of
student context and needs, with the goal of empowering teachers, administrators,
and students themselves to make decisions (at different levels) that will enhance
the teaching and learning experience for all
As teachers will be the primary beneficiaries of using this technique, it’s
important that their opinions and expectations are taken into consideration. We
would appreciate your participation in this 10-minute survey conducted through
an electronic questionnaire to investigate your expectations regarding the
creation of learning analytics services and the use of students’ educational data.
Your participation is voluntary. These questions have been designed to
understand what you would ideally desire from an automated educational data
analysis service. By completing this survey, you will provide critical insights into
teachers’ expectations regarding LA.”
4 Results
Demographic questions provided valuable information about the group of teach-

ers who participated in the research. With the majority of participants having
over twenty years of experience in high school education, it can be inferred
that the sample is composed of experienced educators who have accumulated
knowledge and pedagogical practices throughout their careers. Furthermore, the
predominance of teachers working in state public schools highlights the rele-
vance of the research for the educational context of the southern region of Santa
Catarina, especially considering the significance of this type of institution in the
educational landscape.
The high level of understanding of the concept of Learning Analytics, evi-
denced by the majority of teachers giving scores of 4 and 5 on the comprehension
scale, is a promising indicator for the application of this innovative approach in
high school education. This understanding is essential for the effective adop-
tion of educational analyses and the strategic use of collected data, aiming to
optimize teaching and enhance student learning.
When exploring teachers’ expectations regarding Learning Analytics, the
results pointed to the need for support and guidance from educational insti-
tutions. The majority of teachers consider it important for the school to provide
clear guidance on how to access student learning outcomes through the analyses.
This finding emphasizes the relevance of the institution’s role in the Learning
Analytics implementation process and in providing resources and support for
educators.
The feedback provided by the Learning Analytics service was a well-evaluated
theme by the participating teachers. The majority agree that this feedback
should be presented in an understandable and easily interpretable format, allow-
ing educators to use the information practically and guide their pedagogical
practices based on the obtained data. This aspect highlights the importance of
usability and accessibility of collected information, ensuring that it is effectively
applied for educational improvement.
Regarding access to educational data, the research revealed that teachers are
aware of the types of data collected by the administrative systems of institu-
tions. Most participants stated that they are aware of the collection of socio-
demographic data, historical student grade data, and student attendance data.
This perception is crucial to ensure the quality of data used in educational
analyses, as well as to provide a comprehensive and detailed view of student
performance.
Teachers’ expectations regarding adequate training for the implementation
of Learning Analytics were another highlight of the results. The majority of
teachers consider it essential to provide training to all parties involved in the
teaching process, including educators, students, and administrative staff. This
approach reflects the teachers’ commitment to ensuring that everyone involved
in education is prepared to use Learning Analytics efficiently and effectively.
Proper training is fundamental to empower educators to interpret and apply the
results of analyses in an effective manner that aligns with pedagogical objectives.
Therefore, it can be concluded that the research results indicate a strong
interest among teachers in the southern region of Santa Catarina in using Learn-
ing Analytics as a powerful tool to enhance teaching, provide personalized feed-
back to students, and promote their academic and professional development. The
positive expectations of the participants demonstrate receptivity and readiness
to adopt this innovative approach in the educational context.
Additionally, the research highlights the significance of educational institu-

tions’ role in promoting and guiding the use of Learning Analytics. The support
provided by schools is essential for the success of implementing this approach,
ensuring that teachers and other stakeholders are properly equipped and sup-
ported in this process of educational change.
Ultimately, the research underscores the importance of accurate and reliable
educational data for the effectiveness of educational analyses. Understanding the
types of data collected and utilizing this information for a comprehensive view
of student performance are key elements for a more personalized and efficient
education.
These results provide valuable insights for the successful implementation of
Learning Analytics in high schools in the southern region of Santa Catarina.
The adoption of this innovative approach can significantly contribute to the
enhancement of teaching and learning, offering education that aligns more closely
with students’ needs and potential. It also fosters the continuous development of
quality education focused on the academic and professional success of students.
5 Discussion
RQ 1: What are the expectations of high school teachers in the south-
ern region of Santa Catarina regarding the use of Learning Analytics
(LA) as an educational tool?
The expectations of high school teachers in the southern region of Santa
Catarina regarding the use of Learning Analytics (LA) as an educational tool
were explored in this research through questionnaires administered to the edu-
cators. The results revealed that teachers have high expectations for the use of
LA to enhance the teaching and learning process.
Firstly, the teachers demonstrated a positive understanding of the concept of
Learning Analytics, with the majority giving high scores on the comprehension
scale (ranging from 1 to 5). This understanding is essential for the effective use
of educational analyses, as it allows educators to comprehend how data can be
applied to improve teaching and student learning.
Furthermore, the results showed that teachers have a positive expectation
regarding the feedback provided by the Learning Analytics service. The major-
ity agreed that this feedback should be presented in an understandable and
easily interpretable manner, enabling teachers to practically apply the informa-
tion in their pedagogical practices. This highlights the importance of usability
and accessibility of collected data, ensuring that they are effectively utilized to
enhance education.
Teachers also indicated the importance of receiving clear guidance on how to
access student learning outcomes through the analyses. This need for support
and guidance reflects the relevance of the institution’s role in the implementation
process of Learning Analytics and in providing resources for educators.
Another significant aspect is the teachers’ expectation of proper training for
the implementation of Learning Analytics. The majority of participants con-
sidered it essential to provide training to all parties involved in the teaching
process, including teachers, students, and administrative staff. This data indi-
cates the commitment of teachers to ensure that everyone engaged in education
is prepared to use Learning Analytics efficiently and effectively.
Therefore, the results suggest that high school teachers in the southern region
of Santa Catarina have positive expectations regarding the use of Learning Ana-
lytics as a tool to enhance teaching, provide personalized feedback to students,
and promote their academic and professional development.
RQ 2: What are the teachers’ perceptions regarding the impact
and effectiveness of Learning Analytics in optimizing learning and
enhancing the educational environment?
The teachers’ perceptions regarding the impact and effectiveness of Learn-
ing Analytics in optimizing learning and enhancing the educational environment
were investigated in this research based on the teachers’ responses to the ques-
tionnaires.
The results showed that teachers have a positive outlook on the potential
impact of Learning Analytics in improving the teaching and learning process.
They believe that educational analyses can provide valuable insights into student
performance and needs, enabling educators to adopt a more individualized app-
roach to teaching. This means that through Learning Analytics, teachers can
identify subgroups of students with specific expectations and characteristics,
allowing for a more personalized and efficient education.
Furthermore, teachers recognize the importance of using educational data to
enhance the feedback process and academic guidance for students. They believe
that data analysis can assist in identifying students at risk of failure or dropping
out, enabling preventive measures to be taken to ensure students’ academic
success.
Another significant point is the teachers’ perception of the importance of
students’ autonomy and critical reflection in the educational process. Learning
Analytics can contribute to developing students’ analytical skills and enable
them to have greater participation and responsibility in their own learning
process.
Regarding the educational environment, teachers acknowledge that the use of
Learning Analytics can positively influence decision-making by school adminis-
trators. The collected and analyzed data can support decisions at various levels of
the institution, from the classroom to educational management. This underscores
the importance of considering the expectations of higher levels of management
to ensure the satisfaction and effectiveness of Learning Analytics services.
Therefore, teachers’ perceptions regarding the impact and effectiveness of
Learning Analytics in optimizing learning and enhancing the educational envi-
ronment were generally positive, highlighting the potential of this approach to
comprehensively improve education.
RQ 3: How can the teachers’ expectations and perceptions influ-
ence the adoption and integration of Learning Analytics in the edu-
cational landscape of the southern region of Santa Catarina?
The expectations and perceptions of teachers can influence the adoption and
integration of Learning Analytics in the educational landscape of the southern
region of Santa Catarina in several ways.
Firstly, the fact that teachers show a positive understanding of the concept of
Learning Analytics indicates that they are open and receptive to adopting this
innovative approach. This understanding is an important initial step for educa-
tors to effectively explore and utilize educational analyses in their pedagogical
practices.
Furthermore, teachers’ expectations regarding feedback provided by educa-
tional analyses demonstrate that they recognize the value of this information in
enhancing teaching and student learning. This perception can encourage edu-
cators to actively use the collected data to make informed decisions in their
teaching activities, which can significantly contribute to the effectiveness of edu-
cational practices.
Teachers’ expectations also emphasize the importance of support and guid-
ance from educational institutions in adopting Learning Analytics. The results
indicated that teachers desire clear guidance on how to access and interpret the
results of analyses. This aspect is crucial to ensure that educators feel empowered
and confident in using the analyses in their daily work.
Moreover, the need for proper training for all stakeholders involved in the
implementation of Learning Analytics was highlighted by teachers’ expectations.
This demand for training underscores the importance of investing in training and
professional development programs for educators, students, and administrative
staff, to ensure that everyone is prepared to use Learning Analytics efficiently
and effectively.
Therefore, the positive expectations and perceptions of teachers can posi-
tively influence the adoption and integration of Learning Analytics in the edu-
cational landscape of the southern region of Santa Catarina, promoting more
personalized, efficient education aligned with the needs and expectations of stu-
dents and educators in the region.
6 Final Remarks
The present study investigated the expectations of high school teachers in the
southern region of Santa Catarina regarding Learning Analytics (LA) as an
educational tool, as well as their perceptions about the impact and effectiveness
of this approach. The majority of teachers demonstrated an understanding of the
importance of analyzing educational data to enhance the teaching and learning
experience, showing receptivity towards adopting LA.
The primary expectations of teachers regarding LA included support and
guidance from educational institutions to access and interpret the results of
analyses, as well as understandable feedback to apply in their pedagogical prac-
tices. The quality of collected data was also highlighted as relevant to ensure the
accuracy and usefulness of the analyses.
The research emphasized the need for proper training for all involved in LA
implementation, including students and administrative staff. This training is
essential to interpret and effectively apply the results of analyses in the educa-
tional context.
The obtained results contribute to the advancement of knowledge in Learn-
ing Analytics applied to education, underscoring the importance of teachers’
perspectives and the impact of this approach on the educational context of the
region. They indicate that the use of LA can be a powerful tool to enhance the
quality of education, promoting a more personalized education aligned with the
needs and expectations of students and educators.
The assessment of teachers’ expectations and perceptions is crucial to guide
educational managers in the implementation of LA. Understanding the needs of
key stakeholders involved in the educational process enables the development of
more effective strategies aligned with the demands of the school community.
Based on the results, it is suggested to conduct comparative studies in differ-
ent regions or types of educational institutions, investigate students’ perceptions
regarding the use of LA, and explore the effectiveness of the approach in other
stages of education. Furthermore, investigating the impact of LA on student
retention, academic performance, and teacher satisfaction would be relevant to
measure the concrete benefits of this approach in education.
Acknowledgements. This work was funded by the Brazilian National Council for
Scientific and Technological Development - CNPq (process number 409633/2022-4,
305731/2021-1).
References
1. Romero, C., Ventura, S.: Educational data mining and learning analytics: an
updated survey. Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 10(3), e1355
(2020)
2. Baker, R.S., Inventado, P.S.: Educational data mining and learning analytics. In:
Larusson, J.A., White, B. (eds.) Learn. Anal., pp. 61–75. Springer, New York
(2014). https://doi.org/10.1007/978-1-4614-3305-7 4
3. Hilliger, I., et al.: Identifying needs for learning analytics adoption in Latin Amer-
ican universities: a mixed-methods approach. Internet High. Educ. 45, 100726
(2020)
4. Herodotou, C., Rienties, B., Boroowa, A., Zdrahal, Z., Hlosta, M., Naydenova, G.:
Implementing predictive learning analytics on a large scale: the teacher’s perspec-
tive. In: Proceedings of the Seventh International Learning Analytics & Knowledge
Conference, pp. 267–271 (2017)
5. Tsai, Y.-S., Moreno-Marcos, P.M., Tammets, K., Kollom, K., Gašević, D.: Sheila
policy framework: informing institutional strategies and policy processes of learn-
ing analytics. In: Proceedings of the 8th International Conference on Learning
Analytics And Knowledge, pp. 320–329 (2018)
6. Gaftandzhieva, S., Doneva, R., Pashev, G.: Learning analytics from the teacher’s
perspective: a mobile app. In: INTED2019 Proceedings. IATED, pp. 8133–8143
(2019)
7. Whitelock-Wainwright, A., Gašević, D., Tejeiro, R., Tsai, Y.-S., Bennett, K.: The
student expectations of learning analytics questionnaire. J. Comput. Assist. Learn.
35(5), 633–666 (2019)
8. Shum, S.B., Ferguson, R.: Social learning analytics. J. Educ. Technol. Soc. 15(3),
3–26 (2012)
9. Herodotou, C., Rienties, B., Verdin, B., Boroowa, A.: Predictive learning analyt-
ics ‘at scale’: towards guidelines to successful implementation in higher education
based on the case of the open university UK. J. Learn. Analyt. 6, 85–95 (2019)
10. Shum, S.B., Ferguson, R., Martinez-Maldonado, R.: Human-centred learning ana-
lytics. J. Learn. Analyt. 6(2), 1–9 (2019)
11. Jarke, J., Breiter, A.: The datafication of education. Learn. Media Technol. 44(1),
1–6 (2019)
12. Cechinel, C., Ochoa, X., dos Santos, H.L., Nunes, J.B.C., Rodés, V., Queiroga, E.:
Mapping learning analytics initiatives in Latin America. British J. Educ. Technol.
51(4), 892–914 (2020)
13. Queiroga, E.M., et al.: Using virtual learning environment data for the development
of institutional educational policies. Appl. Sci. 11(15), 6811 (2021)
14. Queiroga, E.M., et al.: A learning analytics approach to identify students at risk
of dropout: a case study with a technical distance education course. Appl. Sci.
10(11), 3998 (2020)
15. Gaftandzhieva, S., Docheva, M., Doneva, R.: A comprehensive approach to learning
analytics in Bulgarian school education. Educ. Inf. Technol. 26, 145–163 (2021)
16. Mahmoud, M., Dafoulas, G., El Aziz, R.A., Saleeb, N.: Learning analytics stake-
holders’ expectations in higher education institutions: a literature review. Int. J.
Inf. Learn. Technol. 38(1), 33–48 (2020)
17. Falcão, T.P., et al.: A penny for your thoughts: students and instructors’ expecta-
tions about learning analytics in Brazil. In: LAK22: 12th International Learning
Analytics and Knowledge Conference, pp. 186–196 (2022)
18. Queiroga, E.M., et al.: Experimenting learning analytics and educational data min-
ing in different educational contexts and levels. In: 2022 XVII Latin American
Conference on Learning Technologies (LACLO), pp. 1–9. IEEE (2022)
19. Saar, M., Rodrı́guez-Triana, M.J., Santos, L.P.P.: Towards data-informed teaching
practice: a model for integrating analytics with teacher inquiry. J. Learn. Analyt.
9(3), 88–103 (2022)
20. Kurilovas, E.: On data-driven decision-making for quality education. Comput.
Hum. Behav. 107, 105774 (2020)
21. Siemens, G., et al.: Open learning analytics: an integrated & modularized platform,
Ph. D. dissertation, Open University Press Maidenhead (2011)
Teaching Ethics in HCI Courses: Guidelines
for Teachers and Future Designers
Ismar Frango Silveira(B)
Mackenzie Presbyterian University, São Paulo, Brazil

ismar.silveira@mackenzie.br
Abstract. Ethics concerns have been present in the early stages of Computer
Science (CS) history, which straightforwardly reflects in the educational process
of all subjacent areas. With the popularization of software applications and its
pervasiveness in modern society, the worries about ethical risks associated with
software products have grown and became an integral part of the Software Engi-
neering process, which includes the Human-Computer Interaction (HCI) issues.
HCI deals with the most visible software’s layers and with higher potential impact
on final users, but often relegated to a second plane in syllabi of Computer Science-
related courses. In this context, the present paper brings a discussion on some of
ethical risks related to HCI, based on a narrative literature review about different
pitfalls in the interaction design that could have ethical impacts, aiming to help
teachers to plan HCI-related curricula, syllabi, and lesson plans.
Keywords: Ethics · HCI teaching · Education in Computer Science
1 Introduction
Ethical concerns when planning, designing, and implementing computer systems go be-
yond privacy and data protection, cybersecurity, intellectual property or simply avoid-
ing – or minimizing - algorithmic, AI-based bias: they involve a wide number of concepts,
attention points, guidelines, risks, and pitfalls that crosscut the entire software engineer-
ing process [1]. Increased public interest and dependency of software systems – for both
people and organizations – and the ubiquitous – sometimes invisible, often impercepti-
ble, and maybe deviant – presence of software applications in virtually all – or almost
all – aspects of everyday life brought to light the need of discussing Ethics as an integral
part of all phases of Software Engineering pipeline. In this sense, HCI plays an important
role in this pipeline, considering that it represents the most visible and noticeable aspects
of software. In this sense, ethical concerns should permeate all aspects of interaction
design of a software, mainly the UI/UX design (here standing for the classical definitions
for User Interface and User Experience, as in [2]).
UI and UX design, as the key components of Human-Computer Interaction (as a part
of Software Engineering process) that focus on the design and interaction aspects of a
software product, represents the most visible layer of every interactive software. While
Ui and UX refer to distinct approaches, they are closely related and often work together
https://doi.org/10.1007/978-981-99-7353-8_35
Teaching Ethics in HCI Courses 473
to create usable and utile interfaces, leading to a seamless user experience. Since they
represent the tier that is closer to the final user than any other software layer, the ethical
aspects should arise in the process of UI/UX design and must be present in all design
decisions. However, this requires an effort in the basic formation of every designer,
which passe through re-concepting the way HCI issues are approached in CS-related
undergraduate courses.
The landscape of ethics education in universities is evolving, with an increasing
number of standalone ethics classes being offered, which includes CS-related courses
[3]. In this sense, the demand for a more integrated approach to ethics within CS-
related curricula is on the rise. As a result, a growing number of CS instructors are now
incorporating ethics as an integral component of their courses [4], in spite of having a
lack of resources on this specific topic available to teachers.
But what is Ethics? Oxford’s dictionary standard definitions bring “moral principles
that govern a person’s behavior or the conducting of an activity” or “the branch of
knowledge that deals with moral principles” – in a shallow definition, it could be de-
fined simply as the set of values and principles that guide human behavior, guiding
individual and collective actions for the common good and social justice. However, this
broad and somehow abstract definition could hide some important aspects of human
activities that are not clear or understood by people – for instance, how to deal with
ethical aspects of software, which is considered for many people as abstract, non-tangible
elements incorporated – often in involuntarily or imperceptibly manners – in their lives?
Software benefits from a status of abstraction that is not comparable to other tangible
technologies, like machines, devices or tools. Sensors, given their discrete presence in
physical equipment, maybe perhaps they may be in a similar situation, or something in
between - however, that subject is beyond the scope of this article.
Fact is that the general desire for increasingly more advanced software solutions,
mixed with the absence of perceived accountability, a quasi-infinite potential for evo-
lution and the general perception that the increasingly massive adoption of computing
technologies is a trend for the near future brings to the software industry the increased
responsibility for designing software pieces with an unprecedent preoccupation with
ethical issues, given the impact and pervasiveness of their products [5].
In the Software Engineering process, often the interests of stakeholders are priori-
tized over the final users’ needs and expectation. Nonetheless, the first are meant to be
taken into account while managing ethical issues and risks to delivering high-quality
outcomes, whilst not ignoring the last - on the contrary, logic tells us that the more users
are satisfied with the software product, the greater the chances of achieving the owners’
goals. However, since the borders among “satisfaction”, “engagement”, “persuasion”,
“motivation” and more than a dozen of similar or correlate terms are not clearly speci-
fied, the search for potentializing every one of these aspects could bring important ethical
risks that are not often perceived by software designers – however, it must be noticed
that sometimes, stakeholders and software engineers are partial or totally aware of these
ethical risks, not wishing to employ sufficient efforts to mitigate them or even purpose-
fully inserting them – a good discussion can be seen on [6], focused on the requirements
stage, for instance.
474 I. Frango Silveira
In this sense, this paper will bring a discussion about ethical aspects related to HCI
education, focusing on UI and UX design processes, pointing out some commonly found
ethical problems, some good practices and it intends to warn about the presence of traps
and pitfalls in this process, aiming to bring some guidelines to teachers, synthetized in
few mind maps and “do’s” and “don’ts” tables spread through the paper.
2 Ethical Risks in HCI: A Discussion for Teachers

In general, many ethical issues found in software design process are perceived by users
through their interface. However, a new risky layer of unethical elements could arise
from the UI or UX design itself, and this session will present the most common cases,
according to literature, and bring some hints and suggestions of good practices that
could be incorporated in HCI teaching to help future UI or UX designers to make proper
choices. Thus, considering a narrative literature review, and freely based on [1, 5, 7, 12,
13], Fig. 1 shows a mind map of frequent ethical risks in the UI/UX design processes.
Fig. 1. Mind map of frequent ethical risks in HCI projects (Source: authors).
Next subsections will provide a deeper discussion on each one of these risks, bringing
some literature review on each topic and using tables to synthetize good practices (“do’s”)
and risks (“don’ts”), which could be useful as educational resources for HCI teachers.
2.1 Invasion of Privacy

Considered probably the most frequent ethical risk on software, invasion of privacy
occurs when unauthorized, unlawful, or inappropriate collection, use and disclosure of
personal data or private information to any person without their consent or knowledge.
While comprehensive discussion on ethics and privacy in Software Engineering can
be seen in [14], some important and risky ethics pitfalls when doing HCI projects are
listed as follows:
• Collecting excessive data: UI could not ask for data and information that will not
be strictly necessary for system’s working. According to [17], it may be considered
an invasion of privacy for software to collect personal data without obtaining explicit
consent or to collect more information than is necessary for the purposes for which
it is intended. Sensitive data, as defined by many data protection laws, requires a
specific treatment.
• Unconsented data storage, sharing, tracking and profiling: there is an important
privacy violation when, without the proper and clear users’ disclosure and agreement,
users’ data are stored, or misleadingly used for tracking (geolocation, biometrics and
so on) or profiling purposes. Data sharing with third entities, as well as cross-app
data sharing is also a preoccupation that have gained attention mainly with the advent
of social media and Internet of Things (IoT) applications – there is an interesting
discussion on this last topic in [16], proposing the use of blockchain to guarantee
privacy in vehicular applications. Thus, UI design must find ways to clearly inform
users about all uses that software provider plan to make with all their data, at the
same time avoiding excessively long legal terms that induce users to simply click on
“next” button chain without really gaining awareness about the process.
• Hidden privacy policies: To not infringing users’ right to information, HCI designers
must provide access to clear and easily understandable privacy policies, without
intentionally hiding relevant information that could affect their privacy. These pieces
of information should be easily reachable at any point of the software use.
Synthetically, Table 1 shows some good practices (“do’s”) and risks (“don’ts”)
regarding privacy.
Table 1. Do’s and don’ts for invasion of privacy
Do’s Don’ts
UI should collect only the strictly necessary Collect as many information as possible from
data, with special attention to sensitive data users, even if they are not necessary now
Inform users about all intended uses for their Do not bring details about data storage, sharing
data, but not as an information avalanche and processing
Always bring clear and ubiquitous Hide as much as possible the privacy policies
information in the UI about all privacy or show them only once
policies
2.2 Information Overload

Users’ confusion and frustration often occur when UI overwhelms users with huge
amounts of information, which is a clear usability problem, since this overload exceed
their cognitive capacity, being a challenge to them to find pieces of information and
detect the most important ones – the situation became worst and pervasive problem due
to the abundance of information through various sources. A broader discussion on this
topic can be seen in [17].
Future HCI designers should be taught to find alternatives to deal with massive
amounts of information on each time smaller screens and output devices, this condi-
tion becomes an ethical problem when it comes to interfere on how users are able to
autonomously make informed decisions. The users’ cognitive well-being could also be
affected by information overload, which can lead to stress and other symptoms. On the
other hand, users with developmental disorders or some other impairments could find
difficult to process huge amounts of information. Some common flaws on this aspect
are listed below.
• Irrelevant and/or non-meaningful information: An excess of information that is
not relevant to the user or the context could prevent users to take right actions and make
proper decisions, which poses serious ethical risks, especially when this state of men-
tal overwhelmingness is intentionally caused by design. Good HCI projects should
prioritize relevant information over optional, non-impacting terms. Long texts could
be arranged in small, organized chunks with consistent formatting that allows users
to skim the text and better understand its context. Messy icon-based or link-based
UIs can pass by a categorization process, proposing visual hierarchies and contex-
tual contract-expand or hide-and-show techniques. A user-centered design approach,
based in tests and iterations – and probably in codesign or similar techniques [38] – is
always desirable over designer-centered strategies for design.
• Not allowing user control and customization: different users have different inter-
action needs. A good HCI project would allow users to, by themselves, define the
amount of information or visual elements that are bearable by them, even if these con-
figurations are suggested by nudges. Empowering users means making the UI more
usable and inclusive, bringing potentially higher levels of satisfaction, and avoiding
falling into ethical risks.
Table 2 summarizes the do’s and don’ts about information overloading.
Table 2. Do’s and don’ts for information overloading
Do’s Don’ts
Prioritize relevant and meaningful information, Put all possible information at once to be
giving users the option to access second order consumed at one glimpse
information
Give users control over the amount of Work with fixed menus and toolbars, always
information and visual items that they want to presenting textual information as is
be exposed to
2.3 Data and Information bias/Lack of Transparency

Bias in data and information can give rise to a wide range of ethical problems and
dangerous impacts on users. Such impacts, potentialized by recent AI advances, have
been discussed by [18, 19] and they vary from reinforcement of stereotypes, affecting
marginalized or vulnerable groups, to limiting the access to opportunities for some

social groups. However, biased data and information is not only a problem for AI and
ML models, but it is also an issue to be covered in many disciplines in a CS-related
course, which includes HCI.
The immediate threats arise from the fact that, if UI is informed with biased data and
information, the information shown could lead to unfair treatment and discrimination
towards certain user groups, which could reach to undesirable scenarios of exclusion,
discrimination, and prejudice. However, it must be considered the special role played
by UI and the dangers that visual elements could reinforce the bias already existent in
data. Besides, it must be considered that even unbiased data and information could suffer
of a malicious design, which could proposedly inject bias on the data and information
being shown – the data source itself being unbiased, but the bias dwelling over data
selection algorithms or bad interaction choices. Together with fake news and deep fake
techniques, this ethical risk is a point of attention for expressive research on social media,
like [20] that studied racial discrimination on Twitter due to biased information; [21]
which focused on gender, also on Twitter; and [22, 23] that studied the deviant impact
of biased information about politics in Facebook.
When dealing specifically with HCI, some aspects must be observed to avoid falling
into the following ethical risks:
• Benefit or prejudice specific groups or users: Considering that UI, by its sensorial
features, could potentialize some undesirable behaviors and distort the interpretation
of the world, we could conclude that a serious ethical risk is that inequalities in society
can be exacerbated by biased UI designs that favor or disadvantage specific users,
even if data is not skewed. In this sense, good practices of Universal Design [23] are
highly recommended.
• Prioritize the owner’s, not the user’s interests and needs: Showing the most expen-
sive items (or those in stockpile) first in e-commerce websites; favoring clients that
pay higher taxes to highlight their products or services to final users; forcing the inte-
gration with third-part partners or conditioning some services to those whose accept
cross-platform data sharing are ways to insert bias in the interaction project, often by
using dark patterns [10, 11].
• Keeping users unaware of the exposition to biased of fake data and information:
While social media internal policies are struggling with a consistent number of trials
accusing their enterprises of promoting biased information, is also a task of HCI
designer to provide interaction mechanisms with tools and elements that clearly avoid
users to be exposed to suspicious, biased content, or at least keep them warned. This
lack of transparency appears in many scientific works, like two discussions about
Facebook policies about biased content which are found in [24], while the bias of
YouTube’s recommender system that are discussed by [25], to name a few.
Some good practices and ethical risks regarding skewed data and information can
be found in Table 3.
Table 3. Do’s and don’ts for data and information bias
Do’s Don’ts
Treat equally all users, trying to promote Prioritize specific groups or people by skewing
inclusive practices the interaction design
Always prioritize the users’ needs, not Always fulfill the owner’s expectancies, even
ignoring the owner’s requirements, but though bad practices like dark patterns are to be
implementing them in an ethical way used
Keep users aware of the quality and Do not take any action to prevent or warn users
liability of data and information being about being exposed to malicious, biased content
shown
2.4 Discrimination and Bad Support for Diversity
By encouraging inclusivity, representation, and equitable access for all users, HCI design
plays a vital part in the fight against discrimination and lack of diversity. Organizational
culture has a big impact on how software teams address human values in their practices,
as defended by [26], and models for fostering inclusive values in these teams are being
tested, as appears in [27]. These discussions are strongly appearing on recent proposals
on CS Education, as seen in [37].
In [28] there is a comprehensive discussion about how inclusion should shape
design – in a broader sense of the term – and inclusive design principles, as men-
tioned by [28, 29] could bring some lights to this discussion. Authors of [29] introduce
a conceptual framework for inclusive design – also in a broader sense – called DARE
(Design, Appraisal, Response, Experience) that encompasses three distinct levels, each
correlated with the decreasing extent of discrepancy between the user and the design
artifact: the provision of accessibility (Level 1), the cultivation of equitable experiences
through engaging user participation (Level 2), and the facilitation of empowered success
through the promotion of flow experiences (Level 3).
Based on this literature review, some dangerous pitfalls that could enhance
discrimination and do not contribute for diversity promotion in IHC follows:
• Not representing diversity in UI elements: HCI designers, when planning the visual,
audio, or textual elements of any interface might incorporate a variety of imagery,
illustrations, and icons – as well as spoken audios with culturally and geographically
different accents and texts that embrace diversity and inclusivity, thereby encompass-
ing a broad spectrum of individuals and cultures. Consider cultural differences help
to avoid perpetuating stereotypes in the design.
• Not moving towards an inclusive design: While Universal Design poses consid-
erable challenges for HCI, the design process should embrace users with different
needs and disabilities, even though the completeness of target audience seems to be
an unreachable goal – many times, it is, but it is better to try to fulfill the minimal
expectancies of the maximum amount of users that is feasible than to limit design
choices to a very narrow group of users – often occidental, white, urban, cisgender,
straight and neurotypical men.
• Forgetting gender gap: Gender inclusiveness, as mentioned by [30], sometimes

tends to be subtle when designing interaction, since the term “inclusion” not always
is understood by designers and often is related only to neuroatypical persons or users
with disabilities. Gender issues hold significant relevance within interfaces, as visual
features and textual elements often neglect to address feminine gender explicitly, or
when they do, they tend to perpetuate stereotypes and preconceived social roles that
fail to accurately depict the contemporary society.
Some do’s and don’ts regarding to how HCI projects could avoid discrimination, at
the same time that it could foster diversity, can be seen in Table 4 below.
Table 4. Do’s and don’ts for discrimination and bad support to diversity
Do’s Don’ts
Try to embrace different cultural and social Design interfaces for just one kind of persona,
groups when designing interfaces with represent better owner’s expected target
audience
By using Universal Design, try to design Prioritize specific groups or people, designing
interaction that fits the maximum number of interaction projects for the majority of your
users’ needs and expectancies potential users
Have genuine worries about gender Design the interaction having in mind only the
inclusiveness at every aspect of HCI projects common average man as a user model
2.5 Lack of Accountability
A significant area of concern in Software Engineering is Accountability, as defined in a

wider sense by [31] – the term stands for the set of practices employed by stakeholders
and professionals to define responsibility for the actions. This encompasses processes
of transparency, reporting, control, and oversight.
Given how deeply ingrained software is in our daily lives, a lack of accountability
can result in a wide range of ethical and practical problems and difficult obstacles to be
overthrown. Since software nowadays have a great potential to have important impacts
on users’ everyday lives, lack of accountability on software can make rise of important
ethical risks that could be hard to identify and be rectified, without accountability.
Some ethical risks that can arise from the lack of accountability in in HCI-related
aspects are listed below, followed by Table 5:
• Ignoring HCI negative social impacts: When overlooking potential risks and
adverse consequences that bad choices on HCI could have in society, like reinforcing
stereotypes and discriminatory discourses, the lack of accountability could prevent
designers to identify and correct these flaws. Even the legal aspects, regarding respon-
sibilities to be assigned to all participants of the decision-making process, are blurred
and not well-defined in local regulations.
• Users cannot blame anyone: The sensation of “being lost” and having no one to
complain with in frequent in systems with low accountability, or in bad HCI designs,
when this information is intentionally hidden or hard to be found. Even considering
the availability of customer support on commercial software, they each time more
powered by expert systems or generative AI-based textual chats, or almost infinite
automated phone assistances, they do not provide – yet – liable and complete solutions
that could help users to build a trustful relationship with software owners.
Table 5. Do’s and don’ts for lack of accountability
Do’s Don’ts
Ty to avoid them, but always keep track of Let the owner’s legal department the task of
potential negative impacts of interaction dealing with judicial process, hoping that
choices and clearly define the responsibilities harmed users won’t get to some bad point
for the important decisions
HCI Design should give clear means to users Client assistance information should be
to complain about software problems and hidden as possible in HCI design, to diminish
malfunctions costs and responsibilities, putting over the
client the task of finding alternative ways to
deal with software problems
The integration of ethical considerations and accountability measures across the

whole UI design process, encompassing the stages of conception, deployment, and ongo-
ing evaluation could facilitate the dissemination of best practices to foster accountability
in HCI design.
2.6 Bad Accessibility
The urge for accessibility in software is an extremely important issue to be addressed, as

mentioned by [32]. However, even when accessibility aspects are to be considered in HCI
design, poor implementation of accessibility features could lead to users’ frustration, but
also to ethical problems that should be addressed by the entire Software Engineering
team. Some frequent flaws on it can be seen as follows:
• Regular tests are enough: Accessibility often means to provide alternative means to
users with physical or cognitive disorders to have access to the main functionalities
and elements of software. This is done following accessibility standards and guide-
lines, such as WCAG, and by promoting awareness on software team and stakeholders
about understanding accessibility as an integral part of the whole software process
and a core component of the design. However, these actions are not enough since
specific and periodic accessibility testing must be performed to identify and address
accessibility barriers.
• No need for codesign: Beyond adopting design strategies like Universal and Inclu-
sive Design, the Codesign should also be considered by software team. By actively
involving individuals with disabilities in the design process, their unique perspectives,
expectations, and needs can be readily identified and effectively addressed. Accord-
ing to [38], teaching Codesign strategies in HCI courses could help the building of
effective ethics curriculum, since it empowers practitioners to engage in reflective
practices and contemplate both the intended and unintended consequences of the
technologies they develop right from the inception phase, rather than treating ethical
considerations as mere remedies or afterthoughts.
Some do’s and don’ts regarding bad accessibility are pointed out in Table 6.
Table 6. Do’s and don’ts for bad accessibility
Do’s Don’ts
Perform comprehensive accessibility tests, Just perform the regular tests and audits with
preferably with target audiences. These should typical users, just once at the end of process
be incremental and must follow the evolution
of HCI aspects on the process
Plan codesign sessions, involving as many Keep all design decisions concentrated on the
people with disabilities as possible interaction team
2.7 Manipulating Users’ Behavior/Tendency to Addiction

A very well-known design practice in UI/UX is the Persuasive Design [33, 34]. With
many uses on the design of games and gamified environments – but not limited to them,
this strategy by itself is not inherently unethical, but its unproper application could put the
software application in the border between simple persuasion and manipulating users’
emotions, feelings, and moods. This issue is addressed by [35], which mentions dark
patterns as a frequent bad habit on HCI projects aimed to manipulate users. The same
authors defend that trust, transparency, and user autonomy should serve as fundamental
principles to guide future HCI professionals.
Manipulating users’ behaviors is an extremely risky ethical dilemma, since in this
case users won’t be able to perform mindful decision-making processes, with could harm
them by forcing them to make choices that eventually go against their will, values, and
principles. Besides, some design techniques aimed to persuade users sometimes foster
existing habits and behaviors that could be converted into addictions. Whilst they are
more perceived in social media UIs, they could occur in virtually any software.
Some common problems found on interfaces regarding this topic:
• Intentionally using addictive elements on UI: For instance, Tiktok’s infinite
scrolling and autoplay of videos; social validation strategies applied on Facebook
or Instagram; and the abusive use of Twitter, Telegram or Whatsapp that causes the
fear of missing out (FOMO), which can trigger compulsive behaviors and create
an addiction for users to constantly checking their devices. These practices must be
avoided in HCI projects.
• Manipulating users’ emotions and molding their cosmovision: The omnipresence

of biased information pr fake news targeted towards the user’s profile or the interests
of the software owner, combined with spurious use of personalization techniques,
recommender systems and gamified elements composes a scenario that brings serious
ethical risks that could lead users to change the sense of reality and their vision
of the world, making them to take unwanted actions, reducing their autonomy and
reinforcing the development of addictive behaviors.
In spite of this topic being extremely complex, with different variations on the types
of design being proposed, Table 7 tries to summarize some do’s and don’ts on the subject.
Table 7. Do’s and don’ts for manipulating users and creating a tendency for addiction
Do’s Don’ts
Avoid or mitigate addictive elements and excess Time is money and the maximum of users’
of persuasive strategies in HCI projects time and attention that could be grasped by
software, the better
Users should not be manipulated to address All manipulating strategies could be applied
software owners’ expectative to accomplish the software’s goals
By adopting user-centric approaches that promotes responsible and mindful usage

of software, HCI designers could avoid or mitigate the flaws that deals with user
manipulation, diminishing the development of addicting behavior.
2.8 Low Sustainability/Risky Design
Both in global and personal levels, users’ safety and wellbeing cannot be compromised
by poor HCI designs. This might have serious ethical repercussions. Software that is not
designed having in mind the sustainability requirements pose important ethical prob-
lems, regarding to the environment and societies. There is a comprehensive review on the
subject in [36], but this a study that needs to be updated, considering nowadays’ reality
and environmental, social requirements and the concrete threat of a huge environmen-
tal crisis on the near future. Thus, sustainable design practices should be embraced by
designers to alleviate the environmental impacts that bad decisions on HCI could pro-
voke. In this sense, actions like extending hardware and software’s lifespan, not fostering
premature product replacement and accumulation of electronic waste, promoting ethical
consumption and being aware of the digital gap that could perpetuate digital inequality
on society.
On the other hand, interfaces that lack obvious and understandable controls run the
risk of causing accidents or other potentially fatal circumstances, which could negatively
impact users and society. People’s safety should come first in ethical UI design, and
dangers should be kept to a minimum. Thus, some common flaws regarding to low
sustainability and risky design on HCI projects are listed below:
• Excessive use of resource-intensive solutions: Some hardware and software solu-

tions are known by their intense dependency on resources, like blockchain archi-
tectures, huge TV displays and so forth. Responsible design principles should be
put in place, not prioritizing only the commercial gain, but considering society’s
well-being. In this context, interfaces that heavily rely on energy consumption, such
as those incorporating power-hungry visual effects, huge server load and constant
connection necessitate reconsideration and redesign.
• Abandonware caused by planned obsolescence: HCI projects that encourage delib-
erate obsolescence or lack long-term support can result in early product replacement
and an accumulation of electronic trash. The ideals of responsible design are incom-
patible with encouraging excessive consumption or encouraging people to engage
in risky activities. Thus, the promotion of sustainable and responsible consumption
habits should be in line with ethical design.
• Not keeping track of risk assessment and mitigation: Software maintenance team
must continuously detect any dangers, weaknesses, or unforeseen effects that may be
caused by poor HCI design. To reduce risks, proper mitigation techniques should be
implemented.
Table 8 summarizes some do’s and don’ts about this topic.
Table 8. Do’s and don’ts for low sustainability and risky design
Do’s Don’ts
Minimize resource-consuming elements on Do not take into account the consumption of
HCI design. A special attention must be paid resources when taking design decisions
to energy, either coming from portable
batteries or wiring
Keep away from planned obsolescence: plan Interfaces should be regarded as having a
your UI or UX with no previously defined limited time span. New technologies will
expiration date require brand new HCI design
Prevent risks that could arise from the usage of Do not worry about potential risks inherent in
the software product. They can be relatively using the software, they are always the
simple, like causing fatigue or boredom to responsibility of inexperienced or clumsy
users, or extremely serious, including fatal users
incidents
2.9 Dark Patterns

As defined by [10, 11], dark patterns are the umbrella term for spurious tricks used in
poor, malicious design that make users’ inadvertently behaving against their will, like
buying or sign up for things that them didn’t mean to. Figure 2 brings some types of
these tricks.
The objective of this paper is not to extensively dig into the various types of dark
patterns – also referred to as “deceptive patterns”. Ample resources exist in the literature
Fig. 2. Mindmap of some common dark patterns (Source: authors, based in [10, 11]).
and on the internet, such as [10], that provide thorough explanations of these patterns.
While some names are self-explanatory, others require clarification. For instance, “Con-
firmshaming” involves eliciting uncomfortable emotions, such as guilt or shame, to
influence users’ decision-making. Another example is “Roach Motel”, which refers to
intentionally poor design that makes it challenging for users to navigate away once they
are into a region of software that is interesting for owners, not necessarily for users.
As a rule of thumb, it must be taught in every HCI-related course that an ethical HCI
project would never contain a dark pattern.
3 Conclusions
Earlier in this paper, it was mentioned that ethical problems in software design could
be intentional or not. In this sense, it is important to define the intentional ethical risks
referring to circumstances in which people or organizations deliberately choose to act
in ways that could damage others, violate moral standards, or jeopardize the welfare of
users, stakeholders, or society in a broader sense. These dangers may be brought on by
several aspects, including external corporate pressures, a lack of ethical awareness of
designers and developers, competing interests, or simply a deliberate misbehavior from
stakeholders, designers and/or developers.
However, some ethical problems could arise from the design process in an unin-
tentional, unperceived way, when software – and by extension, HCI – designers are
unaware of committing or causing situations of potential ethical risks. Since these risks
are not often straightforwardly perceived, they must be pointed out and their mitigation
must be taught in CS-related careers, probably in HCI courses. Thus, aiming to give a
support to this audience, composed by HCI teachers and in-course future professionals,
the paper brought an overview of the most frequent ethical risks on HCI design and how
to avoid – or, at least, mitigate – them. To guarantee that interfaces prioritize user well-
being, fairness, and responsible use of technology, ethical HCI projects should entail
adhering to ethical principles, industry standards, and engaging in constant evaluation
and development of practice.
Finally, the main objective of this paper was, through a literature review, categorize
and present, in an organized way, the most commons ethical threats in HCI projects,
serving as a set of guidelines to be used as educational resources for HCI teachers and
future professionals. Further works will involve the validation of the categorization,
through surveys with teachers that used it and tests with focal groups and professionals
to improve this catalog.
References
1. Ozkaya, I.: Ethics is a software design concern. IEEE Softw. 36(3), 4–8 (2019)
2. Joo, H.: A study on understanding of UI and UX, and understanding of design ac-cording to
user interface change. Int. J. Appl. Eng. Res. 12(20), 9931–9935 (2017)
3. Fiesler, C., Garrett, N., Beard, N.: What do we teach when we teach tech ethics? A syl-
labi analysis. In: Proceedings of the 51st ACM Technical Symposium on Comptuer Science
Education, pp. 289–295 (2020)
4. Eriksson, E., Nilsson, E.M., Hansen, A.M., Bekker, T.: Teaching for values in human-
computer interaction. Front. Comput. Sci. 4, 830736 (2022)
5. Mitchell, A., Balasubramaniam, D., Fletcher, J.: incorporating ethics in software engineering:
challenges and opportunities. In: 2022 29th Asia-Pacific Software Engineering Conference
(APSEC) (pp. 90–98). IEEE (2022)
6. Biable, S.E., Garcia, N.M., Midekso, D., Pombo, N.: Ethical issues in software requirements
engineering. Software 1(1), 31–52 (2022)
7. Gray, C.M., Chivukula, S.S.: Ethical mediation in UX practice. In: Proceedings of the 2019
CHI Conference on Human Factors in Computing Systems, pp. 1–11 (2019)
8. Parrilli, D. M., Hernández-Ramírez, R.: Building a privacy oriented UI and UX design:
an introduction to its foundations and potential developments. In: Advances in Design and
Digital Communication II: Proceedings of the 5th International Conference on Design and
Digital Communication, Digicom (2021), 4–6, 2021, Barcelos, Portugal (pp. 16–30). Springer
International Publishing (2022). https://doi.org/10.1007/978-3-030-89735-2
9. Aydemir, F.B., Dalpiaz, F.: A roadmap for ethics-aware software engineering. In: Proceedings
of the International Workshop on Software Fairness, pp. 15–21 (2018)
10. Gray, C.M., Kou, Y., Battles, B., Hoggatt, J., Toombs, A.L.: The dark (patterns) side of UX
design. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems,
pp. 1–14 (2018)
11. Fansher, M., Chivukula, S.S., Gray, C.M.: #Darkpatterns: UX practitioner conversations about
ethical design. In: Extended Abstracts of the 2018 CHI Conference on Human Factors in
Computing Systems, pp. 1–6 (2018)
12. Rose, E.J., et al.: Social justice in UX: centering marginalized users. In: Proceedings of the
36th ACM International Conference on the Design of Communication, pp. 1–2 (2018)
13. Thomson, A.J., Schmoldt, D.L.: Ethics in computer software design and development.
Comput. Electron. Agric. 30(1–3), 85–102 (2001)
14. O’Regan, G.: Ethics and Privacy. In: Concise Guide to Software Engineering: From Funda-
mentals to Application Methods, pp. 155–180. Springer Int. Publishing, Cham (2022). https://
doi.org/10.1007/978-3-031-07816-3_9
15. Choi, J.P., Jeon, D.S., Kim, B.C.: Privacy and personal data collection with information
externalities. J. Public Econ. 173, 113–124 (2019)
16. Fan, K., et al.: A secure and verifiable data sharing scheme based on blockchain in vehicular
social networks. IEEE Trans. Veh. Technol. 69(6), 5826–5835 (2020)
17. Bawden, D., Robinson, L.: Information overload: an overview. In: Oxford Encyclopedia of
Political Decision Making. Oxford University Press, Oxford (2020)
18. Ntoutsi, E., et al.: Bias in data-driven artificial intelligence systems—an introductory survey.
Wiley Interdisc. Rev.: Data Min. Knowl. Discov. 10(3), e1356 (2020)
19. Tsamados, A., et al.: The ethics of algorithms: key problems and solutions. In: Floridi, L. (ed.)
Ethics, Governance, and Policies in Artificial Intelligence, pp. 97–123. Springer International
Publishing, Cham (2021). https://doi.org/10.1007/978-3-030-81907-1_8
20. Gunarathne, P., Rui, H., Seidmann, A.: Racial bias in customer service: evidence from Twitter.
Inf. Syst. Res. 33(1), 43–54 (2022). https://doi.org/10.1287/isre.2021.1058
21. Grace, A.N., Mueller, T.S.: Gender bias in sport media: a critical analysis of Twitter content
and the National Football League’s Carolina Panthers. J. Gend. Stud. 28(3), 363–370 (2019)
22. Luo, M., Hancock, J.T., Markowitz, D.M.: Credibility perceptions and detection accuracy of
fake news headlines on social media: effects of truth-bias and endorsement cues. Commun.
Res. 49(2), 171–195 (2022)
23. Mamedova, I.Y., Dryukova, A.E., Milchakova, N.E.: The concept of usability in terms of
universal design. Russ., 111 (2022)
24. Hitlin, P., Rainie, L.: Facebook algorithms and personal data. Pew Res. Cent., 16 (2019)
25. Kirdemir, B., Kready, J., Mead, E., Hussain, M.N., Agarwal, N.: Examining video recommen-
dation bias on YouTube. In: Boratto, L., Faralli, S., Marras, M., Stilo, G. (eds.) Advances in
Bias and Fairness in Information Retrieval: Second International Workshop on Algorithmic
Bias in Search and Recommendation, BIAS 2021, Lucca, Italy, April 1, 2021, Proceedings,
pp. 106–116. Springer International Publishing, Cham (2021). https://doi.org/10.1007/978-
3-030-78818-6_10
26. Hussain, W., et al.: Human values in software engineering: contrasting case studies of practice.
IEEE Trans. Softw. Eng. 48(5), 1818–1833 (2020)
27. Miranda, M., Prikladnicki, R.: Towards a model for managing diversity and inclusion in soft-
ware development teams. In: Proceedings of the XXXIV Brazilian Symposium on Software
Engineering, pp. 325–331 (2020)
28. Holmes, K.: Mismatch: how inclusion shapes design. MIT Press (2020)
29. Patrick, V.M., Hollenbeck, C.R.: Designing for all: consumer response to inclusive design. J.
Consum. Psychol. 31(2), 360–381 (2021)
30. Vorvoreanu, M., Zhang, L., Huang, Y.H., Hilderbrand, C., Steine-Hanson, Z., Burnett, M.:
From gender biases to gender-inclusive design: an empirical investigation. In: Proceedings
of the 2019 CHI Conference on human factors in computing systems, pp. 1–14 (2019)
31. Kroll, J.A.: Accountability in Computer Systems. The Oxford Handbook of Ethics of AI,
pp. 181–196 (2020)
32. Lindbäck, F.: Navigation impossible: connecting factors when evaluating accessibility
practices (2021)
33. Redström, J.: Persuasive design: fringes and foundations. In: Persuasive Technology: First
International Conference on Persuasive Technology for Human Well-Being, persuasive 2006,
Eindhoven, The Netherlands, May 18–19, 2006. Proceedings 1, pp. 112–122. Springer, Berlin,
Heidelberg (2006)
34. Torning, K., Oinas-Kukkonen, H.: Persuasive system design: state of the art and future direc-
tions. In: Proceedings of the 4th International Conference on Persuasive Technology, pp. 1–8
(2009)
35. Chamorro, L.S., Bongard-Blanchy, K., Koenig, V.: Ethical Tensions in UX Design Practice:
Exploring the Fine Line Between Persuasion and Manipulation in Online Interfaces (2023)
36. Penzenstadler, B., Bauer, V., Calero, C., Franch, X.: Sustainability in software engineering:
a systematic literature review (2012)
37. Garcia-Holgado, A., Vázquez-Ingelmo, A., Verdugo-Castro, S., González, C., Gómez,
M.C.S., Garcia-Peñalvo, F.J.: Actions to promote diversity in engineering studies: a case
study in a computer science degree. In: 2019 IEEE Global Engineering Education Conference
(EDUCON), pp. 793–800. IEEE (2019)
38. Pillai, G., et al.: Co-Designing resources for ethics education in HCI. In: Extended Abstracts
of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–5 (2021)
Author Index
A E
Abad, Karina 214, 442 Echeverria, Vanessa 60
Aguayza, Anabel 116 Eliseo, Maria Amelia 131
Alcantara de Oliveira, Ivan Carlos 131 Erazo-Garzón, Lenin 328
Almeida-Galárraga, Diego 116 Espinoza Sánchez, Katherine 301
Álvarez-Tello, Jorge 413 Estrada-Arana, Juan 429
Alvarez Arévalo, Francisco 301
Auquilla, Andrés 214
F
Ayala-Mendoza, Asdrúbal Emilfo 413
Farinango, Victor 343
Fernández Del Carpio, Alvaro 101
Ferreira de Menezes, Jones Baroni 38
B
Frango Silveira, Ismar 472
Barros, Gabriel 157
Bermeo Conto, Jorge 301
Berrezueta-Guzman, Jonnathan 384 G
Biancato, Geane 459 García Cabeza, Sofía 242
Bigolin, Marcio 242 Garcia Costa dos Santos, Gleice Louise 357
Bolaños-Mendoza, Cindy 286 García-Ramírez, Yasmany 1
Bragagnini Mendizábal, César 101 Gómez Ortíz, Alonso 172
Guamán, Danny S. 398
Guato Burgos, Marcelo Fabian 186
C Gutierrez-Aguilar, O. 14, 26
Caiza, Julio C. 398
Carrillo, Gladys 60
H
Carvalho Nunes, João Batista 38
Haahr, Mads 254
Carvallo, Juan Pablo 301, 328
Heredia Jimenez, Vanessa 75
Castro Arias, Santiago 343
Cechinel, Cristian 38, 459
Cevallos-Almeida, Aura-Liz 49 J
Chacón-Castro, Marcos 226 Jimenez-Macías, Alberto 75
Chicaña-Huanca, S. 14, 26
Chiluiza, Katherine 60
K
Chuquizala, Christian 116
Krusche, Stephan 384
Cordova-Buiza, F. 14, 26
L
D Larrosa, Manuel 145
Díaz Oporto, Sara 101 Lasso Lazo, Diego 301
Duche Pérez, A. 14, 26 Leon-Lucano, J. 14, 26
Dueñas Vera, Sergio 101 Lopes Leite, Letícia 357
Duque-Romero, Marco 413 Lourenço Alves, Socorro Vânia 317
© The Editor(s) (if applicable) and The Author(s), under exclusive license
to Springer Nature Singapore Pte Ltd. 2023
https://doi.org/10.1007/978-981-99-7353-8
490 Author Index
M Q
Malache-Silva, Laura 384 Queiroga, Emanuel Marques 38
Maldonado-Mahauad, Jorge 214, 442 Quiroz-Martinez, Miguel-Angel 49
Manrique Chalco, Walker 101
Marques Queiroga, Emanuel 38, 459 R
Martínez Cerqueda, David 172 Ramos, Vinicius 38, 459
Melo Alves, Enoque Calvino 317 Reategui, Eliseo 242, 370
Morato Lara, Jorge Luis 186 Rendón-Enriquez, Ibeth 116
Moreira Gois, Marcilyanne 131 Revelo-Herrera, Héctor 429
Motz, Regina 242, 370 Rodés, Virginia 145
Muñoz, Roberto 38, 459 Rodés, Virgínia 459
Rodriguez Medina, Alma Eloisa 172
Rojas-Salazar, Alberto 254
Ruano, Maria Alejandra 286
N
Niquinga-Vargas, Pablo 429 S
Nuñez, Yajaira 200 Salazar-Armijos, Diego 429
Sanchez, Franklin L. 398
Segovia-Segovia, Carlos 429
O Silva Lopes, Fábio 131
Orellana, Ricardo 226
Ortega-Chasi, Patricia 86 T
Ortiz-Rojas, Margarita 60, 75, 286 Thompsen Primo, Tiago 38, 459
Tirado-Espín, Andrés 116
Torres, Danny 75
P V
Pantoja Lima, Celson 317 Valdiviezo Ortiz, Javier 301
Patiño Chuni, Javier 301 Valeriano, Irving 75
Pazmiño-Guevara, Lizzie Verónica 413 Vargas, Francisco 271
Peñaherrera, Krystle 157 Veintimilla-Reyes, Jaime 214
Peñaloza-Ochoa, Sonia 86 Vizcaino Imacaña, Fernanda Paulina 186
Pérez, Jesús 200
Pérez-Pérez, Jesús 200 W
Piedra, Nelson 271 Wives, Leandro 145
Pinargote, Adriano 60
Pinto, Diego 271 Z
Porta, Mariana 370 Zúñiga-Prieto, Miguel 214

Proceedings of The 18th Latin American Conference On Learning Technologies (LACLO 2023)

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Proceedings of The 18th Latin American Conference On Learning Technologies (LACLO 2023)

Uploaded by

Copyright:

Available Formats

Lecture Notes in Educational Technology

Santiago Berrezueta Editor

Proceedings of the 18th Latin

ISSN 2196-4963 ISSN 2196-4971 (electronic)

Paper in this product is recyclable.

October 2023 Jorge Maldonado Mahauad

Jorge Maldonado Universidad de Cuenca, Ecuador

Otto Parra Gonzalez Universidad de Cuenca, Ecuador

Karina Abad Universidad de Cuenca, Ecuador

Alejandro Artopoulos Universidad de San Andrés, Argentina

Otto Parra-González Universidad de Cuenca, Ecuador

Enhancing Geometric Road Design Education through Active Learning

Cell Phone Addiction, Anxiety and Distraction and Their Relationship

Factors that Influence Digital Identity Management in Personal Learning

Ethics, Big Data and Artificial Intelligence: Exploring Academic Works

Gamification as a Learning Strategy in the Training of B1 English Learners . . . . 49

Implementing Synchronous Hybrid Learning: Insights from Teachers

Beyond Numbers: Exploring the Qualitative Dimension of a Learning

Computational Thinking as a Didactic Strategy for the Development

Identifying Master’s Program Web Pages with Machine Learning . . . . . . . . . . . . . 101

Methodological Actions for the Electronic Configuration of the Educa

Problem-Based Learning Applied to Software Engineering: An Experience

Gamification Strategies as Formative Assessment Methods. A Systematic

Gender Inclusion from the Art for Children . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157

Educational use of TikTok to Enhance Complex Grammatical Aspects

LeSeVen: A Web Application for Learning Written Vocabulary Through

Predicting Learners’ Performance in a Programming Massive Open Online

Project-Based Learning and its Impact on the Mathematical Knowledge

Student Perception of Peer Review in the Digital Age . . . . . . . . . . . . . . . . . . . . . . . 242

The CLG Framework: A Methodical Approach to Designing Educational

Ontology for Open Digital Educational Credentials: A Step Towards

Exploring the Perceived Satisfaction of Flipped Learning in E-Learning

Technology Scouting for Techno-Pedagogical Laboratories in Ecuador:

Applying Cognitive Walkthrough to Usability Assessment: A Case Study

On the Use of ChatGPT to Support Requirements Engineering Teaching

Pedagogical Support Through the Implementation of a Virtual Assistant

The Use of Indicators in the Creation of Distance Learning Undergraduate

Building Resilient Educational Systems: The Power of Digital Technologies . . . 370

ChatGPT-4 as a Tool for Reviewing Academic Books in Spanish . . . . . . . . . . . . . 384

Towards Enhancing Novice Learners’ Programming Skills: Lessons

Criteria of Pedagogical Practices in Times of Pandemic: A Systematic

Investigating Variation in Learners’ Behavior Through the Lens

Expectations of High School Teachers Regarding the Use of Learning

Teaching Ethics in HCI Courses: Guidelines for Teachers and Future

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 489

Carlos Alario-Hoyos is Associate Professor in the Department of Telematics Engineer-

Universidad Técnica Particular de Loja, San Cayetano Alto, Loja, Ecuador

Abstract. Active learning techniques have been increasingly employed in civil

Keywords: Active Learning · Geometric Road Design · Student Engagement

Fig. 1. Selected learning moments applied to both study groups.

2.5 Data Analysis

3.1 Group A Results

observed in Table 1. Students in quartiles 1 and 4 exhibited a lower perception of self-

Table 1. Final grades and answers from students in Group A.

Table 2. Matrix correlation between several variables in Group A.

knowledge to real projects, and communicate their progress. However, it is suggested

3.2 Group B Results

Table 3. Final grades and answers from students in Group B.

Table 4. Matrix correlation between several variables in Group B.

engineers. Flipped classes, problem-based learning, and projects promote autonomous

O. Gutierrez-Aguilar1(B) , A. Duche Pérez1 , J. Leon-Lucano1 ,

Keywords: Anxiety · Learning achievement · Cell phone addiction ·

In the university educational context, it is necessary to rethink teaching strategies so