Computers & Graphics 112 (2023) 1–12

Contents lists available at ScienceDirect

Computers & Graphics

journal homepage: www.elsevier.com/locate/cag

Special Section on EG2022 Edu Best Papers

Teaching the basics of computer graphics in virtual reality

∗
Birte Heinemann , Sergej Görzen, Ulrik Schroeder
Learning Technologies Research Group, RWTH Aachen University, Germany

article info a b s t r a c t

Article history: New technology such as virtual reality can help computer graphics education, for example, by providing
Received 1 November 2022 the opportunity to illustrate challenging 3D procedures. RePiX VR is a virtual reality tool for computer
Received in revised form 18 February 2023 graphics education that focuses on teaching the core ideas of the rendering pipeline. This paper
Accepted 6 March 2023
describes the development and two initial evaluations, which aimed to strengthen the usability, review
Available online 11 March 2023
requirements for different stakeholders, and build infrastructure for learning analytics and research.
Keywords: The integration of learning analytics raises the question of appropriate indicators to be approached
Computer graphics education through exploratory data analysis. In addition to learning analytics, the evaluation includes quantitative
Virtual reality techniques to get insights about usability, and didactical feedback. This paper discusses advanced
Multimodal learning analytics aspects of learning in VR and looks specifically at movement behavior. According to the evaluations,
Rendering pipeline
even learners without prior experience can utilize the VR tool to pick up the fundamentals of computer
Technology-enhanced learning
graphics.
© 2023 Elsevier Ltd. All rights reserved.

1. Introduction research, as well as the challenges, whose overall verdict is favor-

Learning technologies have been evolving for years. In the
first half of 2016, the first consumer-grade virtual reality head-
set marked a milestone for business and science. Recently, this
technology has also been entering in the field of education and
training. Teachers and researchers are discovering this technol-
ogy’s possibilities and challenges for teaching and learning. This
trend of general nature in e-learning can also be observed in
computer graphics (CG) teaching [1].
The first efforts to technically support CG teaching were made
early on [1]. Looking at the process since then, it is noticeable
that comparatively many different tools are used, and a large
part of these was not explicitly designed for use in education [1].
To date, no standard way has been established regarding what
principles are used to teach computer graphics content. On the
other hand, many approaches and ideas are showing up in small
applications, and we observe more and more web-based solu-
tions, e.g. [2]’s application Rayground. The future can perhaps
unite both trends, as WebXR (former WebVR) can offer positive
aspects from both worlds [3,4], although developments are not
yet production-ready [5].
The use of VR in teaching has already been investigated with
different questions and overall shows multiple positive effects
on the learning process, and motivation, e.g. [6–11]. [8] discuss
various benefits and provide an overview of the state of the
∗ Corresponding author.
E-mail address: heinemann@cs.rwth-aachen.de (B. Heinemann).
able. For example, they conclude that VR (and AR) contribute to
‘‘Develop[ing] students’ higher order thinking skills encouraging
learning by design’’. Furthermore, they summarize positive ef-
fects on engagement, self-learning, multi-sensory learning, spatial
ability, and others.
A challenge not directly related to the technical part of VR
is researching VR learning applications compared to other (tra-
ditional) learning situations, see [12]. Systematic processes to
integrate VR into teaching and design learning applications are
still at the beginning [13,14]. Learning analytics (LA) is a way to
study and quantify learning effectiveness [15], which offers the
chance to scale research extensively and not require additional
time for learners (as test subjects). Additionally, VR offers to col-
lect multimodal data, which can be used to understand, predict,
and quantify learning [16]. Apart from learning, other complex
constructs such as engagement, collaboration quality, and ex-
pertise are also researched using multimodal learning data [17].
Multimodal learning analytics (MMLA) is one approach to sys-
tematically analyze and compare the success of different peda-
gogical strategies and tools in CG education since this is a gap in
previous research [1].
In this extended journal paper (original [18]), we present the
results of the analysis of a tool specifically created for computer
science education, teaching the basics of introductory computer
graphics — the rendering pipeline. The rendering pipeline is a fun-
damental part of introductory courses [19]. Using the presented
theory-based VR application, learners can interactively experi-
ence the different steps of the graphics pipeline and see how
images are generated from models. The open-source application

https://doi.org/10.1016/j.cag.2023.03.001
0097-8493/© 2023 Elsevier Ltd. All rights reserved.
B. Heinemann, S. Görzen and U. Schroeder
offers the possibility to investigate the learning process of CG con-
cepts and learning in virtual reality. Furthermore, it could be used
to research influencing factors like motivation, spatial ability, and
the effects of feedback. The integration of multimodal learning
analytics provides opportunities for researchers, learners, and
teachers alike. This contribution adds theoretical derivation in the
development by further arguments, such as the factor of visual–
spatial ability, and by the deeper analysis of the collected data
with a focus on the motion data.
This article is divided into eight sections. This introduction,
motivation and overview of the contents are followed by the
second section, in which a framing in the current state of research
from different perspectives (computer graphics didactics, virtual
reality in educational research and learning analytics) is made.
Afterwards, the approach and methodology of the development,
which builds on the aforementioned preliminary work and theo-
retical foundations, is briefly described. In the fourth section, the
technical foundations are presented with a focus on the learning
analytics process. Based on these considerations, the developed
application RePiX VR is presented (in the fourth section). The fifth
section then follows with the evaluation methodology that was
used in two studies. The results of the studies are presented in
section six and discussed in section seven. Additionally the dis-
cussion contains the limitations and recommends future research.
The last section summarizes the paper’s finding giving a synthesis
of key points.
2. Related work & foundations
First, a summary of learning analytics and the benefits of VR is
given in this part, followed by an explanation of the role of com-
puter graphics in computer science curricula. Then, the current
state of research on computer graphics education technologies,
especially in connection with XR, is presented.
Chatti et al. define Learning Analytics (LA) as ‘‘a TEL
[technology-enhanced learning] research area that focuses on the
development of methods for analyzing and detecting patterns
within data collected from educational settings and leverages
those methods to support the learning experience’’ [20]. The
integration of LA can provide information about the learning
process, which offers interesting opportunities for researchers,
teachers, and learners [15,21]. To get a deeper understanding
of (complex) processes like learning, researchers gather data on
learners’ behaviors and learning outcomes to generate insights
and improve the learner’s experience regarding methodology and
didactics. The resulting areas of educational data mining and
learning analytics converged in various fields like multi-modal
learning analytics, acknowledging that a more holistic view is
required when technology-enhanced learning emerges beyond
traditional desktop setups. Terms like self-regulated learning,
adaptive experiences, and learner activation are often discussed
in that field and influence the work described here.
VR encourages student engagement with the learning mate-
rial [22,23], especially when used in conjunction with learning
analytics [24]. In VR, game-based learning, simulations, and ex-
ergaming are possible. VR could be used for more action-oriented
applications, such as teacher training or hard disk updates [10];
however, in this case, VR provides the opportunity to make ab-
stract domain knowledge interactive. VR could make fundamental
computer graphics concepts, including the rendering pipeline,
more accessible to everyone. Another aspect of connecting VR
with computer graphics is how VR provides a unique perspective
on three-dimensional learning subjects. Other research showed
indicators that spatial ability moderates control and active learn-
ing, which both have positive effects on learning outcomes [25].
For this, students were compared between two groups — one
2
Computers & Graphics 112 (2023) 1–12
Fig. 1. Relation between learning outcomes and selected factors particularly
connected to virtual reality. More factors in original research: [25].
with a high measurement in high spatial ability characteristics
and one with low values in spatial ability. When investigating dif-
ferent factors that have an impact on learning outcomes, e.g. mo-
tivation, presence and others, it could be shown that ‘‘control
and active learning’’ – i.e. giving learners control over learning
and actively involving them – is influenced by the learner char-
acteristic spatial ability (illustrated in Fig. 1). This could only
be found in the 3D variant, whereas in the desktop variant the
spatial characteristic has no modifying influence on the mediators
control and active learning.
In 2020, virtual reality and three-dimensional graphics were
identified as emerging topics in computer curricula [26]. One of
the components of the software fundamentals – which is one of
six categories, collected by different computing communities –
is ‘‘Graphics and Visualization’’ [26]. Basic rendering and funda-
mental graphics techniques are, for example, referred to as ‘‘affine
and coordinate system transformations’’, ‘‘polygonal representa-
tions’’, and ‘‘the graphics pipeline’’ [26]. In an examination of 20
curricula, Balreira found that 95% of CG courses cover these topics.
The visual–spatial ability is an example of a significant influ-
encing factor in teaching and learning in virtual reality as well
as with computer graphics. The visual–spatial ability (VSA) is
defined as ‘‘the ability to comprehend and conceptualize visual
representations and spatial relationships in learning and the per-
formance of tasks such as [. . . ] reading maps, navigating mazes,
conceptualizing objects in space from different perspectives, and
executing various geometric operations’’ [27]. As it relates to the
learning objectives in CG courses, the connection from VSA may
be a determinant of academic achievement. For instance, another
empirical study in the fields of mathematics, engineering, and
science demonstrates the link between VSA and success [28].
In addition to the link to the learning content, VSA is yet
another element important for VR learning. In the context of
desktop VR applications, spatial ability is discussed and discov-
ered as an influencing factor on the learning results in desktop
VR applications [25]. Given that the learning material was not
directly related to VSA, this remark should be appraised in terms
of the influence of VSA generally (the task was dissecting the
anatomy of a virtual frog). Besides this, research showed that
there are signs that the influence of control and active learning
on learning outcomes was moderated by spatial ability, while
active learning and control themselves had a positive impact
on learning outcomes [25]. Furthermore, Dalgarno and Lee have
shown that three-dimensional virtual learning environments can
support learning tasks that lead to increased spatial knowledge
representations [29].
There are still few educational computer graphics applications
for (VR), augmented reality (AR), and mixed reality (MR), which
mark different points on the reality–virtuality continuum [30].
The virtual platform called ‘‘Mental Vision’’ by Peternier et al. [31]
fuses virtual reality and computer graphics. To make it simpler for
learners to master the material, they use modules and tutorials.
They use a CAVE or mobile devices and teach programming. In
contrast to this approach, we do not wish to concentrate on
B. Heinemann, S. Görzen and U. Schroeder
Fig. 2. Short overview of steps created for RePiX VR (Version Jan. 2022): (1) Application, (2) 3D Geometry, (3) 3D Transformation, (4) Clipping, (5) Rasterization, (6)
(Local) Lighting, (7) Texturing, (8) Visibility Test and (9) Image.
imparting programming knowledge and do not use a CAVE and
mobile devices. ‘‘GetiT - Gamified Training Environment for Affine
Transformations’’ is another project investigating immersive VR.
Within the environment you can discover affine transformations
(ATs). According to evidence from Oberdörfer and Latoschick, VR
learning is preferable to control groups learning with desktop-
3D learning. Regardless of the specific learning material, using VR
to teach can motivate students and change them from passive to
active students [7]. The recent, (but only web-based) application
‘‘Rayground’’, by the Computer Graphics Group of Athens Uni-
versity, is useful for teaching ray tracing paradigms. It intends
to instruct students in a WebGL-based programmable graphics
pipeline. This is another analogous strategy [2,32].
In summary, from the above examples and the discussions,
it can be concluded that VR is suitable for computer graphics
education. It supports a constructivist approach and learning
analytics is, as well as traditional research methods in education,
a way to evaluate the environments.
3. Didactic conceptualization & development
The systematic and human-centered procedure we used to
develop the open-source tool is described in another publica-
tion [13]. Summarized, we followed an agile approach with pe-
riodic tests, so-called Design-Make-Learn cycles [33], which are
embedded in a Design-Based-Research (DBR) process [34].
We followed recommendations for best practices, such as [35].
These recommendations aid in creating efficient VR applications
that make the most of VR’s capabilities while maximizing the
potential of constructivist learning theories, situated, active, and
creative learning. The learning analytics approach is based on
investigating the needs of various stakeholders, namely develop-
ers, researchers, students, and teachers, as described in [13]. The
intended audience includes both high school students who are
interested and introductory courses at universities.
3
Computers & Graphics 112 (2023) 1–12
The theoretical and systematic planning ended in creating a
guided tour for the computer graphic basics. Learners are accom-
panied by a (virtual) host as they explore the various components
and steps of a simplified version of the rendering pipeline. Further
steps in development were prioritization of the individual stages
and the depth of the content to be learned, which is oriented to
the lecture at the RWTH Aachen University. A brief overview is
depicted in Fig. 2, followed by a detailed explanation in Section 4.
The result is a mixture of interactive and non-interactive stages,
especially simplifying the Rasterizer step with only four simpler
stages: Rasterization, (Local) Lighting, Texturing and Visibility Test.
4. Technical implementation & learning analytics
The reference model for learning analytics by Chatti et al. [20]
served as the foundation for the planning of the quantitative
evaluation and feedback process for learners and educators since
it offers an orientation framework for developing from various
viewpoints and aids in allocating weights to various factors. The
dimensions What? (Data and Environments), How? (Methods),
Why? (Objectives) and Who? (Stakeholders) help to address chal-
lenges and plan sustainable learning analytics structures.
In the dimension of What? the decision for the ‘‘Experience
API’’ (xAPI) as data format was made with ‘‘Learning Locker’’ as
a compatible Learning Record Store for the data collection [36].
Learning Locker is an open-source Learning Record Store, a data
warehouse application that can receive, store and return xAPI
statements [37] via HTTP requests. The xAPI specification permits
defining user activities in human-readable statements formatted
as a triplet: {actor, verb (action), and object (activity)}, such as
‘‘Emma fixated pointable object’’, which, together with the stored
extension, comprises the details that Emma was engaged in, like
‘‘Emma spent 500 ms examining the tiger’s mesh’’. Eye tracking
was one of the learning analytics modalities we used. Along with
tracking eye movements, we monitored controller actions, the
B. Heinemann, S. Görzen and U. Schroeder
user’s position, and head movements (nodding and shaking), in
order to get a diverse multimodal picture of our learners and
their approach. The characteristic of xAPI data, which is carefully
specified and allows for the unique identification of verbs and
objects, is essential to meet the FAIR principles for scientific data;
see [38] for more information. In a nutshell the FAIR principles
help to comprehensively characterize the data and to optimize
the reuse and exchange of it by following principles to make the
data Findable, Accessible, Interoperable, and Reusable. Goal of the
FAIRification is to allow us to combine data and meta data to
get a deeper understanding of learners, learning and technology.
Using xAPI is also interesting for comparability to a desktop
version (using mouse and keyboard instead of VR) or a future
extension with other learning opportunities, e.g. a learning videos
or quizzes in Moodle, collecting data cross platforms in the same
format, as suggested in [39].
A web application described in more detail in [40] was built
that controls connections to the Learning Record Store, Unity and
settings. It was created for researchers and interested educators
to configure the learning experience and the learning analytics,
for example, to choose which objects should trigger a statement
to be written when the user looks at it. This is important as
to not collect more data than necessary (privacy by design —
collecting no data is the default setting). Web socket connec-
tions link instances of our VR learning application to the web
application (ReCoPa — Researcher Companion Panel) and supply
their trackable scene objects. The user could choose pertinent
scene elements for tracking and start the calibration for the
(optional) eye-tracking with different privacy options. Tracking
can be started and stopped with the panel.
Initially created for the HTC Vive Pro Eye, the learning applica-
tion described in this paper was developed using Unity 2019.4.8f1
in conjunction with Microsoft Visual Studio 2019 for C# scripts
and the SteamVR plugin. An Intel Core i7-8750H processor run-
ning at 2.20 GHz, 64 GB of RAM, an NVIDIA GeForce RTX 2070
graphics card, and Windows 10 Education 64-bit were used in the
studies. The LA tools, the Researcher Configuration Panel, and its
server were developed using JetBrains WebStorm 2020.2, NodeJS
v12.13.1, and Nuxt.js v2.14.4.
Repix VR: The guided tour
RePiX VR – short for Rendering Pipeline eXperience in VR –
is a guided tour that takes a student through each step of a
simplified rendering process in a massive neutral setting while
being accompanied by an avatar as host (brief demo: https://
youtu.be/U77hR7udyak). The host is a robot, apparent in some
figures: Subfigures (2)–(9) of Figs. 2 and 3. The tour starts with a
brief lesson, as shown in Fig. 3, in which the learner is introduced
to the buttons, teleport function, locomotion technique, and the
laser pointer for selecting scene items. Then, the nine steps of the
condensed rendering pipeline are given as previews in Fig. 2. We
describe here the version from January 2022. Innovations not yet
present in the evaluations are marked with this symbol: ✬.
1. Application: A demonstration of the computations of the
translation, rotation, and scale matrix of a 3D object. For
example, a red cube transforms step by step (translation,
rotation, scale) and results in its TRS matrix (Fig. 2.1). ✬ Af-
ter this demonstration, the learner can practice this knowl-
edge by combining the matrices of a small solar system.
The matrices are colored (e.g., the rotation matrix is green)
to identify its responsibility.
2. 3D Geometry: An introduction to the basics of a 3D mesh
is given. A Stanford Bunny mesh (without surfaces) rotates
in front of the learner. Afterward, learners must build their
first triangle by placing its colored vertices (visualized as
4
Computers & Graphics 112 (2023) 1–12
Fig. 3. The first step is a tutorial explaining the controller, here teleportation.
small spheres) in the VR space using the VR controller and
a gradient color picker stuck on its trackpad. Finally, the
three vertices have to be connected to a triangle and can
be extended to a more complex object (Fig. 2.2). Thus, the
learners can observe the color interpolation of a triangle
and see its index by looking at a text label stuck above each
vertex.
3. 3D Transformation: In this step, the frustum-based projec-
tion is introduced. The learners can adjust its’ influencing
factors (far plane, near plane, the field of view, and aspect
ratio) by interacting with the frustum via laser-pointer
casting from the VR controller. A semi-transparent, blue
pyramid-shaped mesh represents the frustum. For demon-
stration, a few 3D objects are placed inside this pyramid
to represent a 3D scene. The learners can move the far
and near plane of the frustum mesh along its z-axis by
grabbing/selecting them with the laser-pointer (Fig. 2.3).
The field of view and aspect ratio of the frustum can be
adjusted with 3D sliders. Those also define the width and
height of pyramid-shaped mesh (the frustum).
4. Clipping: In this step, the normalized device coordinates
(NDC) concept is explained by using a unit-sized semi-
transparent cube combined with transformed mesh copies
of the 3D objects placed inside the frustum mesh from
the previous stage. Mesh primitives outside this unit-sized
cube are invisible because they are clipped (discarded).
However, to demonstrate them visually, they are tinted
cyan by enabling this preview via pressing a specific button
on the users’ VR controller (Fig. 2.4). Like in a real-world
rendering pipeline, the mesh transformation of the 3D
scene depends on the frustum and its values. Thus, in this
stage, learners can observe the results of the frustum-based
transformation by interacting with the pyramid-shaped
mesh again. Clipped fragments are discarded (or tinted)
in real-time. This way, the learners can observe which
fragments are discarded.
5. Rasterization: A textual explanation combined with a 2D
illustration of the rasterization principle introduces the
learners to the rasterization stage. Algorithms, like the
Digital Differential Analyzer algorithm, are only mentioned
but not explained in more detail. The illustration shows
the detection of fragments (potential pixels) of a 3D scene
(Fig. 2.5). ✬ An interactive version of this stage with more
details is currently under development.
6. (Local) Lighting: The idea of Local-Lighting is explained in
this stage. The directional light source (represented by the
sun) is turned on and off to demonstrate this idea. Then,
the learners are invited to seek the light source in the sky.
✬ Further, the learners are invited to place flashlights and
light bulbs with different colors and beam sizes to discover
B. Heinemann, S. Görzen and U. Schroeder
Fig. 4. The user can trigger different states of the clipping mechanism for the final image. Changes from left to right: Tint clipped fragments, made image transparent,
hidden clipped fragments, show final image.
the influence of different light sources (Fig. 2.6). With the
help of a user interface, the learners can choose between
phong and blinn shading, as well as vertex and fragment
shader. In addition, they can toggle the ambient, diffuse,
and specular modes.
7. Texturing: In the texturing stage, a ‘‘naked’’ (without tex-
ture) copy of the tour guide is placed next to it. The ac-
cording texture is displayed on a UV coordinate system at a
board on the other side next to the tour guide (Fig. 2.7). The
learners can go closer to the UV map and see an overlaying
2D mesh grid (UV map) on the texture. ✬ The newest
version allows the learner to paint on the UV map or the
‘‘naked’’ version of the robot (like spray cans) to explore
the connection between mesh and texture.
8. Visibility Test: To explain the concept of the rendering
pipeline’s z-buffer, textual explanations are displayed.
These aim to transfer the principle of deciding between
visible fragments (pixels on screen) and non-visible (dis-
carded) fragments. The final chosen fragments are illus-
trated by displaying the resulting image. It is located where
the NDC representation stood in the stage before (Fig. 2.8).
9. Image: In the final stage, the learners can interact with
the frustum-based mesh again. This time, they can see not
only the result of the clipping stage, but also the final
rendered image displayed in front of NDC (represented
again by the unit-sized cube) (Fig. 2.9). By using the laser-
pointer on this image plane, the opacity of image plane gets
reduced, so that the learners can observe the transformed
mesh inside the unit-sized cube. This way, the learners can
directly compare the resulting image with its transformed
3D geometry laid behind. In addition, clipped primitives
can be shown by tinting them cyan. This way, learners can
reflect directly on how it fits with the final image. The
different variants of possible user perspectives can be seen
in Fig. 4.
Generally speaking, each step includes explanations, anima-
tions, and some steps include tasks and VR interactions. A panel
with an overview of each step is set up in front of the tour
host for orienting. Each stage features a button that can be used
to launch the associated learning activity. Only the previously
visited stages or the next one on the tour schedule are available
to the learners. By doing this, we can guarantee linear navigation
while yet allowing for the free study of actual stages.
Especially the steps with interactive tasks should invite the
learner to explore and to try things. It refers to constructivist
learning theories in their composition. The student is given expla-
nations of the potential interactions and brief interaction tasks to
help them become accustomed to them. After this, the tour guide
extends an invitation to explore the stage. Finally, the students
can practice the things they have learned.
Additional control panels are used for a task if needed, such
as at the 3D Geometry stage. This panel lets the controller but-
tons switch between functions like ‘‘put vertices’’, ‘‘construct
5
Computers & Graphics 112 (2023) 1–12
triangles’’ (connecting vertices), ‘‘move vertices’’, and ‘‘remove
vertices’’. Another small task, which invites the user to interact
and experiment with the tool, is the modification of the frustum
values taught in the 3D Transformation stage. Later the changes
are directly shown as a change in the 2D view. The shape of the
frustum depends on the values of the near plane, far plane, aspect
ratio, and field of view. The user can also use these functions
in the clipping and the final stage and choose between different
styles, which visualize the results of this stage.
5. Evaluation methodology
The application has gone through different design-based re-
search (DBR) cycles; the process is described in [13]. We will
discuss the results of two evaluations at different stages of devel-
opment. Smaller experiments were carried out while the appli-
cation was being developed. Two larger-scale experiments were
done to enhance the usability, the learning process, and the
learning analytics components in 2020 and 2021. The enhanced
interaction in the second evaluation could be important for the
interpretation. Another vital piece of information concerns the
circumstances under which the data collection took place be-
cause careful safety procedures were needed due to the unique
circumstances in the academic years of 2020 and 2021. In addi-
tion to the novelty effect of VR, this face-to-face session was a
great exception to other learning opportunities at that time as
our regulations did not include presence meetings. These special
circumstances may have had an impact on students’ perceptions
and feelings, as reported by Knight et al. who found for example
missing social experiences caused by the pandemic [41].
In total, we evaluated 32 persons; in the first evaluation, eight
people participated. Of these, all tested the VR variant. In the
second evaluation, we varied between a desktop version (mouse
and keyboard instead of VR) with 14 and the VR version with ten
people.
The length of each session varied from 35 to 75 min. The ages
of participants ranged from 25 to 40. In all the tests, the first
assignment was to (nearly) independently examine the rendering
pipeline as part of the guided tour. When people had concerns
about how the controllers worked, we offered assistance. The
participants were told to share their ideas while they were ex-
ploring. A free interview was conducted after the instruction,
which was delivered using the think-aloud technique [42,43].
As described in Section 4 we collected eye fixations on various
objects, all controller actions, the user’s position, and nodding and
shaking of the head, which combines to a high-resolution data
collection compared to other educational research and provides a
high range of potential targets for exploratory analysis in learning
analytics; similar to the procedure in [44].
This inductive approach aims to derive models and theories
underlying the data. Moreover, an inductive approach allows for
aggregating the raw data. The goals of this approach correspond
to those of the ‘‘general inductive approach’’, whose approaches
B. Heinemann, S. Görzen and U. Schroeder
Fig. 5. A test person creates an object with vertices and triangles in the Geometry step. The task was to recreate the small polygon behind the self-made turquoise
object.
can be applied to the mix of qualitative and quantitative data we
collected [45]. Fig. 5 displays a photo obtained during the first
evaluation.
5.1. First evaluation
The initial evaluation in 2020 was intended to uncover tech-
nical issues, test usability, and validate the content’s difficulty.
Four subjects with little prior knowledge were explicitly tested
for usability and interaction. These four persons also had no ex-
perience with VR. They also have no prior knowledge of computer
graphics, and generally, the knowledge of computer science is
at a early undergraduate level. To ease the pressure due to the
limited pre-knowledge, we instructed the people that they did
not have to go through all the details. To get more information,
we asked the novices, but also four experts to evaluate RePiX
VR to get information concerning the content. Additional content
feedback – and not part of this evaluation – was obtained during
development in collaboration with computer graphics experts.
The four expert participants had experience in user research,
learning and VR, their task was to provide feedback from differ-
ent perspectives. They provided feedback on learning materials,
usability, lessons learned, and learning and user experience.
Overall, four novices and four experts with different perspec-
tives gave feedback, checked development progress, and offered
hints for the next iteration. Two experts evaluated the first ver-
sion of the Researcher Companion Panel [40] as it should only
be used by someone with experience in learning analytics and
prospective research design.
5.2. Second evaluation
In the subsequent more considerable evaluation, we asked
students enrolled in a computer science course at the RWTH
Aachen University to test the environment. Everyone took part
in the master’s course in learning technologies. Due to pandemic
restrictions, we had to develop a second version of the pipeline
quickly, which can be used from a desktop computer at home.
This way, it was possible for all of the students to participate
and collect data for the lecture projects. In order to compare the
interactions between the two platforms, we also gathered inter-
action data in the desktop version. Four of the ten VR students
previously learned about computer graphics or virtual reality.
Therefore we will specifically refer to them in the results section.
The students assigned themselves to the groups (VR and desk-
top). Particularly with the desktop version, we were unable to
directly control usage, as these were conducted unsupervised. The
6
Computers & Graphics 112 (2023) 1–12
data in the lab, on the other hand, were collected in a standard-
ized manner. After a welcome, methodological introduction, and
clarification about the experiment, the students were let into the
VR application and could learn there without having a time limit.
They were instructed to express their thoughts, and we helped
with questions about how to use the application. Without being
asked we only helped during the tutorial. We collected the set of
log and sensor data without gaze, which was not possible due to
technical issues. After the experience, students were asked about
their impressions and were asked about specific observations, if
any. They also had time to reflect on usability. Content-related
questions were discussed after the study. Feedback from students
who used the desktop version was obtained in a lecture exer-
cise. An investigator guide can be found in the supplementary
material.
6. Results
We conducted an exploratory and descriptive analysis of the
logged interaction, gaze, and motion data without considering
statistical reliability due to the small number of participants.
Feedback from the interviews and the evaluation of the think-
aloud methodology provide hints for the ongoing advancements
in this early development phase.
First, a few general results will be presented. Subsequently,
the two test runs will be explained individually. In both studies,
consistently favorable user impressions showed the potential of
a guided tour, especially for learners unfamiliar with computer
graphics. Also, persons without a computer science background
or ‘‘young’’ students achieved basic learning objectives and could
describe the overall process (understand) and could illustrate
(apply) steps of the pipeline, especially the interactive tasks. For
the six taxonomy levels (remember, understand, apply, analyze,
evaluate and create) see [46].
All test persons, independent of prior knowledge, agreed that
the virtual tour host was helpful and the application’s visu-
alizations are quite detailed. In both versions, we used small
videos showing interactions from a first-person perspective to
help learners with the interactions and to demonstrate usage.
This feature is based on other findings and suggestions, e.g. by
Sutcliffe and Kaur [47]. They found a usability issue with ap-
proaching and orienting the objects. They proposed that hurdles
in interactions could be avoided with small in-time video demon-
strations. We had not planned to systematically investigate this
feature but noticed mixed feedback for the animation videos.
Some learners understood the function immediately, while others
only understood the videos because they asked about it while
being in VR or noticing reflectively in the follow-up interviews.
B. Heinemann, S. Görzen and U. Schroeder
Lastly, before going into details of both evaluations, a finding
from an early pilot trial is presented, that had an impact on
the evaluations presented here because it formed the basis for
some decisions concerning usability. Auto-play for instructions
and texts was a feature in the very first development iteration but
was found to be not user-friendly. The dialogue system had to be
adjusted. However, differing prior knowledge makes it difficult to
establish a standard pace for the dialogue with the robot. Users
want to manage the speed individually. During these early tests,
we found out, that we need to check usability heuristics, like
the one from Nielsen [48], for the design of the user interfaces,
because with the earliest prototypes, we could interpret some
users’ expressions as an overload. They were unsure about the
‘‘visibility of system status’’. As a result, it was decided to display
the panel with the current step permanently and to use a bar to
visualize the progress in each step of the rendering pipeline.
Besides the foundations for usability in VR, Sutcliffe and Kaur
proposed walkthrough questions to evaluate VR user interfaces,
e.g. using questions like, ‘‘Can the user form or remember the
task goal?’’ [47]. Moreover, Sutcliffe et al. produced HCI design
heuristics for evaluating VR applications [49]. The questions from
the proposed walk-through could almost without exception be
answered with yes, e.g. by the beforehand tutorial and explicit
instructions of the robot. Only the question: ‘‘Can the user decide
what to do next?’’ was not quite clear for the very first interac-
tion with the panel (change to the second step of the pipeline),
therefore attention was paid to this in the evaluation.
The feedback we received from the users is consistent with
findings from Wang et al. [50]: ‘‘77% of usability feedbacks can
be mapped to Nielsen’s heuristics’’. Wang et al. mapped dis-
cussions and reviews to HCI design heuristics. Besides the find-
ings for Nielsen’s heuristics, only few themes could be linked
to other heuristic evaluations, presumed because of the goals of
the concrete VR application. We could not create a meaningful
direct mapping between the statements of the users and other
heuristics either. Still, heuristic 11 (‘‘Clear turn-taking, between
user and system initiative’’.) is interesting for our approach to
designing a guided tour and will be discussed in the relevant
section.
In conclusion, the pilot studies showed that usability is crucial
for VR learning applications and heuristics like Nielsen’s could
help to design user-friendly applications right from the beginning,
which is in line with Wang et al. [50].
Results of the first evaluation
An average of 4.500 xAPI statements were gathered for each
test learner during a test session. Even though this data is insuf-
ficient to make predictions about the learning behavior and the
fit of material and content, we were nevertheless able to identify
trends and test some basic statistical techniques, e.g. distribution
of the different types of the xAPI statements. The results could
be seen in [18] and the supplementary material. For example, the
triangle construction appears to be the most engaging interaction.
Besides vertices and triangles all other objects were selected at
least once.
We saw that complex interactions and visualizations, like also
the frustum transformation, are the most exciting and frequently
used, which is in line with the vocal expressions, while the learn-
ers used the application and tried to verbalize their thoughts. The
more interactive the content, the more engaging it was; the users
stayed longer beyond the completion of the intended task, most
visible at the geometry stage.
With regard to the two specific groups, the test revealed
that prior knowledge is essential. Both, VR experience level and
background had an impact on the user’s movement and usage.
The novices did not move that much and were not experimental
7
Computers & Graphics 112 (2023) 1–12
Fig. 6. Comparison of the nodding and shaking behavior of VR users.
in terms of the use of controllers. A graphic and an example of
interaction with the frustum showing three typical users from
different groups is shown in [18].
Lastly, five (out of eight) learners remained in the application
after accomplishing all of the tasks to experiment with the tool
and to observe the scene from different positions. This raised the
question about the movement for the second evaluation.
Results of the second evaluation
We collected about 147.000 statements from 24 persons in the
Learning Record Store. The evaluations demonstrate that VR users
spend more time overall using the application (29 min on aver-
age). Compared to VR users, desktop users spent nine minutes
less and had less variation in specific application steps. VR users
nod substantially more often than they shake their heads (see
Fig. 6), according to a preliminary analysis of head movements.
Additional visualizations and calculations can be found in the
supplementary material (third link).
In line with expectations, the data also reveals that depending
on the level of interactivity, the stages are used for varying
amounts of time. However, the data also shows that VR learn-
ers frequently engage with tasks (specifically three-dimensional
ones). Therefore, VR users spent six times as long in the Geometry
stage compared to desktop users. In Application is the animation
of matrices and the visualization of a red cube manipulated by
the matrices (as already shown in Fig. 2.1): VR users have been
looking at these three times, following how the values change
and the cube is adjusted. There is also a longer time spent in
the final stage, where VR users check the final 2D image from
different angles and re-perform the familiar interactions from
the previous steps. This is especially remarkable regarding how
little new information is provided in the final stage (Image stage:
Fig. 2.9).
There are only two steps in which the desktop users spent
more time than their fellow students in the VR group: Tutorial
and Rasterization. The tutorial took desktop users much longer.
From the quantitative data, we can also see that users do not have
significant differences in movement, with the exception that the
transformation step has more user movement than other steps.
Learners quickly discovered they could look at the scene from
every angle, which was done by most VR learners (at least in the
final step). Figs. 7 and 8 show heatmaps of the movement and
teleportation behavior of the users subject 13 and subject 22.
These visualizations show the top view of the virtual learning
environment. It also contains the objects visible in the largest
number of steps, the unit-sized cube, and the frustum with the
example objects, which can be seen in Fig. 9. As these visu-
alizations show the learner’s overall movement progress, some
common spots from the tutorial guidance (as already shown in
Fig. 3) can be identified on the right of the picture. Considering
these, the following differences between these two prototypical
examples can be seen: Subject 13 has moved more than subject
22. Subject 13 not only shows more different spots but also a
B. Heinemann, S. Görzen and U. Schroeder
Fig. 7. Heatmap of the teleportation and movement behavior of subject 13.
Fig. 8. Heatmap of the teleportation and movement behavior of subject 22.
longer stay, which is indicated by the darker color. Subject 13
used the teleporting function 19 times; subject 22 used it eleven
times. Values range from four to 23 teleports with a median of 19.
The connected spots show that there was also a major physical
movement (which could also be very small teleports if only the
visualization is considered, but could be excluded based on the
data). Subject 13 also teleported inside the frustum and observed
the scene from different positions. Subject 22 has moved less;
only the spot on the start position has long residence time.
The visualizations from other learners can be explored in the
supplementary material. The data is openly accessible and it is
also possible to filter by individual stages.
The Think-Aloud protocol revealed that learners with visual
impairments might find it challenging to read the text panels,
two persons have directly or subsequently remarked on this issue.
In addition, they stated that reading may be challenging due to
the robot’s modest motions behind the semi-transparent panel.
Usability notes were largely collected through the Think-Aloud
protocol to create a mapping like done by Wang et al. [50]. They
mainly concern the heuristics ‘‘Visibility of system status’’, ‘‘User
control and freedom’’ and ‘‘Recognition rather than recall’’ [48].
8
Computers & Graphics 112 (2023) 1–12
Fig. 9. The final result of the rendering pipeline (Image stage). On the left is the
interactable frustum, in the middle is the tour guide Kyle behind the navigation
menu, and on the right is the unit-sized cube representing the NDC. In front of
it, the resulting image and the clipped (tinted cyan) fragments of the Stanford
bunny.
One example is the follow-up of the already explained usabil-
ity issues found with the ‘‘Visibility of system status’’ heuristic:
While the persistent display of the application’s steps was intu-
itively understood, a few learners indicated that they overlooked
the progress bar.
Details and a resulting issue list can be traced in Gitlab, see
supplementary material. After exploring the content and stage
animations the learners offered different ideas for further de-
velopment. In particular, the four students who already knew
the rendering pipeline provided feedback. No urgent need for
changes was identified, instead, enhancements were suggested.
The overall expressed desire for interactive tasks for the not-yet-
interactive stages will be addressed in Section 7.2.
Mixed reviews were given by the fourteen persons using the
desktop version. While specific interactions are simpler to use,
others make the benefits of the VR application evident. For in-
stance, because of the difficulty of placing triangles in the 3D
space, desktop users generally constructed less in the 3D Geome-
try stage (four vs. ten triangles).
Finally, we made an unintended observation at the end of
the summer semester of 2021. As already reported in [18], the
user tests were at the beginning of the semester. The exam took
place in fall. The study was not part of the computer graphics
introduction, many students were still able to recall informa-
tion from the application for the oral exam. They used it to
explain general concepts of learning technologies, which suggests
persistent positive learning outcomes even after a longer period.
7. Discussion
This section is divided into three parts, the first discusses the
results from the previous section and some general VR in educa-
tion topics. The second one deals with the evaluations’ findings
that have already been included in the newest versions of RePiX
VR. Then third, additional adjustments, open research questions,
and limitations of this work are explained and discussed.
7.1. Interpretation of the results
Different sorts of feedback and analytics data were gathered.
The learners had varying perceptions of the learning material and
content. The learning analytics data shows indicators of different
types of users, not only interacting with objects but also moving
around. The different learner movement profiles could be related
to the cognitive load of the novel teaching environment and/or
to learning types, or personal characteristics like spatial ability.
B. Heinemann, S. Görzen and U. Schroeder
However, for statistical evidence, we collected too few data sets
in these evaluations.
Interpreting the collected gestures is challenging. Under some
circumstances, such as after the learners read the text displayed
on a panel or if they completed a task, we may interpret nodding
as an indication of understanding. Nods/shakes are defined by
the amplitude and the number of detected movements. Personal
calibration could optimize the detection. Furthermore, the inter-
pretation of the data is difficult, because these gestures can also
have different meanings and this can lead to different execu-
tion [51], which is currently not mapped. Another difficulty is that
some stages lead to shaking the head repeatedly from right to
left, such as stage Application. Because the learners could observe
how the values of matrices change, and how the TRS matrix is
calculated, the shaking of the users’ heads as a ’failure to under-
stand’ metric is currently not a reliable indicator. Nevertheless,
there is a noticeable discrepancy between nodding and shaking
that could be attributed to the interaction with the robot and the
overall feeling of understanding, especially in combination with
the personal statements collected in the qualitative data.
The last stage is intended to connect the knowledge of all
stages. This seems to invite users to experiment and perform their
tasks, such as reshaping the frustum to keep the bunny out of
the visible area and to test the result of cutting it up. From a
didactic perspective, it is interesting to see, that learners perform
repetitions of previously taught concepts, such as interactions
with the frustum.
An important note from the qualitative interviews concerns
the readability of the texts on the panel. We are actively re-
working the scene to take this requirement into account. One
additional person reported that reading was taxing, and in keep-
ing with the concepts of multimedia learning, voice could be
added in the upcoming version to support reading [52].
For the very first interaction with the panel showing the
pipeline overview, we could not answer the question ‘‘Can the
user decide what to do next?’’ positively [47]. The interaction to
navigate to the second step of the pipeline was difficult for some
learners, therefore attention was paid to this in the evaluation.
A more powerful cue like a flashing button could reinforce the
textual hint of the robot.
Another usability issue connects the two previous paragraphs.
Heuristic 11 [49], which proposes clear turn-taking, between user
and system initiative, is natural to achieve in the guided tour with
a robot as host. Nevertheless, improvement can be achieved by
having learners clearly notice when they have completed a task,
when the robot then takes over the turn and starts talking. This
can also help clarify what to do next.
When interpreting the results collected with the desktop ver-
sion, it is important to note that we do not aim for media
comparison studies that dominate research on VR and AR [53].
The desktop version is an addition and should be viewed with a
critical eye due to the current optimization for the VR version.
Learners report, that the setting is more engaging for users of
the VR application, which should be considered for data inter-
pretation in addition to usability issues in the desktop version.
Furthermore, only students in the VR condition were instructed
to Think-Aloud. On the one hand, the environment itself might
be more motivating to explore the triangles. All in all the general
learning context should be considered different.
Compared to other steps, the non-interactive stages Raster-
ization may have been less engaging, especially for VR users.
Nevertheless, desktop learners spent more time there, which is
in contrast to the other steps. It may be easier to take the time
to read content on the computer. In stages like Geometry, the
environment itself might be more motivating to experiment with
the triangles and the learning context is different.
9
Computers & Graphics 112 (2023) 1–12
In this study, we had no learners complaining of symptoms
of cybersickness [54]. Since learners are familiar with this or
similar concepts, it was addressed in the introduction before the
VR experience and interviews. Sickness is a typical limitation
and drawback discussed in the field of education and virtual
reality [10]. Research from multiple directions aims to minimize
cybersickness; a recent systematic survey aggregated guidelines
that could help to design VR applications [54]. For example, the
locomotion technique in RePiX VR (walking and teleportation)
was selected to reduce movement-induced sickness. In contrast,
movements that do not match the sensory expectations should
be avoided [54].
7.2. Already processed feedback
Feedback is presented with the consequent implications,
which were part of the DBR cycles that this research is based on.
A brief overview of the new features of the stages has already
been shown in Section 4.
The biggest request from the learners was to make all steps in-
teractive. Current versions of the application provide more inter-
active tasks, e.g., experimenting with light bulbs and spotlights to
create shadow, drawing on the UV map or the texture-less robot
to observe the connection, building a solar system to discover the
mathematics, and investigating rasterization to experience the
conversion from vectors to a raster image.
The experts’ wish to broaden the 3D Geometry and Transforma-
tion steps and provide students with more information about the
computation of the TRS matrix was integrated with further itera-
tions. The desire to experiment with matrices corroborates with
Oberdörfer et al. [55] and Huang et al. [35]. Huang et al. calls for
extensive interaction in VR, while Oberdörfer et al. created an ap-
plication (GetiT) to teach the transformations (already described
in Section 2). This request’s outcome is already evident in the
learning application’s description in this article (see Section 4).
The most recent version has a solar system where students can
change the matrices on their own.
Currently in the Learn phase of the iterative design cycle
of [33], are new versions of the steps: Lighting, Rasterization, and
Texturing. Additionally, these steps are requested to be interactive
in accordance with the findings of our evaluations. It is noticeable
that the didactic reduction in light and shadow, as well as textur-
ing, is easier than in the rasterization step. Reducing the content
to heterogeneous prior knowledge and groups of people remains
an open challenge.
7.3. Limitations and directions for future research
Open questions are to what extent an improvement of the
recognition of gestures can lead to a better understanding of
the learning processes and their optimization. Our study shows
that it is worth examining nodding (and shaking) as indicators
of comprehension in subsequent research and correlating them
systematically with other factors, such as the focused elements. In
particular, the reading interactions with the robot are interesting
learning moments.
Another open question arises from the unintentional observa-
tion described in Section 6. Which effect have interactive tasks
on learners with various learning motivations? Another open
question concerns the described animations demonstrating the
interactions. Some learners understood them immediately, which
could be related to the preknowledge in VR and/or gaming. The
suspected connection can possibly be traced back to the conver-
sations with the student.
An open question for future research is if the VR applica-
tion takes longer compared to the Desktop version because the
B. Heinemann, S. Görzen and U. Schroeder
students interact with the content more intensely and if this
increases the intensity of their learning process. How much might
the increased time in the application be related to zeal, which (at
best) moderates the overall course learning outcomes? Another
question is: Do we have to provide more assistance to certain
groups of students than to others, such as those who have weak
visual–spatial skills, as indicated by [28]?
Future plans include expanding the application to participa-
tory VR, where multiple visitors interact to enjoy the tour in
a shared virtual space. For more information, see [56]. Huang
et al. indicate that it is asserted that virtual reality has enormous
promise for cooperative and collaborative learning [35]. However,
participatory VR requires more didactic considerations. How can
the tasks be modified to allow for collaborative action? How can
we ensure that everyone has equal opportunities and that the
group work is distributed fairly?
Another open point of discussion is whether more telepor-
tation points could encourage learners to observe the 3D en-
vironment more closely and possibly achieve a better learning
effect. The number of participants needs to be larger to prove
this statistically. However, one assumption for the different user
behavior is that at least some people forgot that they could
teleport during the session. So the idea was generated to offer
more teleportation spots. Furthermore, the positions of desktop
users will be recorded in the future to measure differences and
conduct a more thorough investigation.
Although usability has been enhanced incrementally, there are
still certain areas that require improvement. For instance, the
controller’s trackpad can modify the vertices’ color. Only some
people initially found this feature. The pointing pattern we em-
ployed as an interaction method is another problem. Although it
seems to be intuitive, various interactions, such as altering the far
and near plane, need to work more accurately. Short development
cycles and rapid tests will be employed to enhance these usability
issues (as well as the usability of the desktop version).
One limitation of the two studies presented is that the stu-
dents we tested are a special group in special conditions due to
the pandemic, the latter was already reported in Section 5. Our
students chose a lecture about learning technologies; some were
familiar with the content and gave good feedback concerning the
material, and some were novices. In addition, computer graphics
is a non-mandatory area in our computer science program. There-
fore, it is not completely certain that particular interests will lead
to different results for those choosing computer graphics as an
elective area in their studies.
8. Conclusion
This study demonstrated the outcomes of a guided tour that
interactively explores the rendering pipeline and computer
graphic basics. Both analyses demonstrate that an iterative de-
velopment approach, as described in [33], results in user-friendly
VR environments. Furthermore, the students who tested our ap-
plication have reacted positively across the board. Together with
the results from [18], in which the interaction with particular
objects was highlighted, the evaluation of the learning analytics
data shows different usage patterns. These are partly attributable
to different prior experiences with VR.
The next step is to do further evaluations with a larger sample
size. The relatively high acquisition expenses and, for instance,
cybersickness are limiting factors for implementation in large-
scale educational applications [12,57]. Stojšić et al. [12] explicitly
mention the research gap: many studies on immersive VR in
educational contexts are conducted under laboratory conditions
and not in authentic teaching/learning situations. Furthermore, to
effectively assess the effects of the VR learning environment, we
10
Computers & Graphics 112 (2023) 1–12
must also address the challenge of creating genuinely equivalent
control groups for some questions. Future research could investi-
gate the findings of Janßen et al. which showed that personality
traits, like extraversion or agreeableness, influence the user expe-
rience (UX) in VR learning scenarios and that UX impacts learning
outcomes [58].
Diverse audiences are a challenge in introductory graphics
courses [59]. One strategy is to pay close attention to assessment,
learning activities and the learning outcomes, see [60]. Taking key
points from both sources, teaching material should be designed
to be modular to maintain the greatest possible flexibility. There-
fore, RePiX VR is open-source; we invite teachers and researchers
to contribute with ideas or to participate actively. VR can promote
self-regulated learning; with the data we collect, we can produce
adaptive learning scenarios that help deal with heterogeneous
preknowledge.
CRediT authorship contribution statement
Birte Heinemann: Conceptualization, Methodology, Valida-
tion, Formal analysis, Investigation, Data curation, Writing – origi-
nal draft, Writing – review & editing, Visualization, Project admin-
istration. Sergej Görzen: Conceptualization, Methodology, Valida-
tion, Formal analysis, Investigation, Data curation, Software, Writ-
ing – review & editing, Visualization. Ulrik Schroeder: Resources,
Supervision.
Declaration of competing interest
The authors declare that they have no known competing finan-
cial interests or personal relationships that could have appeared
to influence the work reported in this paper.
Data availability
It is shared in the paper (supplementary material)
Appendix A. Supplementary material
1. Link to Repository of the application RePiX VR: https://
gitlab.com/learntech-rwth/repix-vr/repix-vr-app
2. Link to Demo Video: https://youtu.be/U77hR7udyak
3. Link to Data & Learning Analytics (created using Python) &
Experimental Guide for the Second Study: https://doi.org/
10.18154/RWTH-2022-10631
References
[1] Suselo T, Wünsche BC, Luxton-Reilly A. Technologies and tools to support
teaching and learning computer graphics: A literature review. In: Proceed-
ings of the twenty-first Australasian computing education conference. New
York, NY, USA: Association for Computing Machinery; 2019, p. 96–105.
http://dx.doi.org/10.1145/3286960.3286972.
[2] Vasilakis AA, Papaioannou G, Vitsas N, Gkaravelis A. Remote teaching ad-
vanced rendering topics using the rayground platform. IEEE Comput Graph
Appl 2021;41(5):99–103. http://dx.doi.org/10.1109/MCG.2021.3093734.
[3] Matahari T. WebXR asset management in developing virtual reality learn-
ing media. Indonesian J Comput, Eng Des (IJoCED) 2022;4(1):38. http:
//dx.doi.org/10.35806/ijoced.v4i1.253.
[4] Pathak R, Simiscuka AA, Muntean GM. An adaptive resolution scheme for
performance enhancement of a web-based multi-user VR application. In:
2021 IEEE international symposium on broadband multimedia systems
and broadcasting. 2021, p. 1–6. http://dx.doi.org/10.1109/BMSB53066.2021.
9547069.
[5] Li K, Wang S. Development and application of VR course resources
based on embedded system in open education. Microprocess Microsyst
2021;83:103989. http://dx.doi.org/10.1016/j.micpro.2021.103989.
B. Heinemann, S. Görzen and U. Schroeder
[6] Martín-Gutiérrez J, Mora CE, Añorbe-Díaz B, González-Marrero A. Vir-
tual technologies trends in education. EURASIA J Math, Sci Technol
Educ 2017;13(2):469–86. http://dx.doi.org/10.12973/eurasia.2017.00626a,
Publisher: EURASIA.
[7] Pantelidis VS. Reasons to use virtual reality in education and training
courses and a model to determine when to use virtual reality. Themes
Sci Technol Educ 2009;12.
[8] Papanastasiou G, Drigas A, Skianis C, Lytras M, Papanastasiou E. Virtual
and augmented reality effects on K-12, higher and tertiary education
students’ twenty-first century skills. Virtual Real 2019;23(4):425–36. http:
//dx.doi.org/10.1007/s10055-018-0363-2.
[9] Scrivner O, Madewell J, Buckley C, Perez N. Best practices in the use of
augmented and virtual reality technologies for SLA: Design, implementa-
tion, and feedback. In: Carrió-Pastor ML, editor. Teaching language and
teaching literature in virtual environments. Singapore: Springer; 2019, p.
55–72. http://dx.doi.org/10.1007/978-981-13-1358-5_4.
[10] Zender R, Knoth AH, Fischer MH, Lucke U. Potentials of virtual reality
as an instrument for research and education. I-Com 2019;18(1):3–15.
http://dx.doi.org/10.1515/icom-2018-0042, Publisher: De Gruyter Section:
i-com.
[11] Asad MM, Naz A, Churi P, Tahanzadeh MM. Virtual reality as pedagogical
tool to enhance experiential learning: A systematic literature review. Educ
Res Int 2021;2021:1–17. http://dx.doi.org/10.1155/2021/7061623.
[12] Stojšić I, Ivkov-Džigurski A, Maričić O. Virtual reality as a learning tool:
How and where to start with immersive teaching. In: Daniela L, editor.
Didactics of smart pedagogy. Cham: Springer International Publishing;
2019, p. 353–69. http://dx.doi.org/10.1007/978-3-030-01551-0_18.
[13] Heinemann B, Görzen S, Schroeder U. Systematic design of effective
learning scenarios for virtual reality. In: 2022 International conference on
advanced learning technologies. Bucharest, Romania; 2022, http://dx.doi.
org/10.1109/ICALT55010.2022.00107.
[14] Fowler C. Virtual reality and learning: Where is the pedagogy?: Learning
activities in 3-D virtual worlds. Br J Educ Technol 2015;46(2):412–22.
http://dx.doi.org/10.1111/bjet.12135.
[15] Gedrimiene E, Silvola A, Pursiainen J, Rusanen J, Muukkonen H. Learning
analytics in education: Literature review and case examples from voca-
tional education. Scand J Educ Res 2020;64(7):1105–19. http://dx.doi.org/
10.1080/00313831.2019.1649718.
[16] Siemens G. What are learning analytics?. 2010, URL http://web.archive.org/
web/20180803152224/http://www.elearnspace.org/blog/2010/08/25/what-
are-learning-analytics/.
[17] Worsley M. Multimodal learning analytics’ past, present, and potential
futures. In: CrossMMLA@LAK. 2018.
[18] Heinemann B, Görzen S, Schroeder U. RePiX VR - learning environment
for the rendering pipeline in virtual reality. In: Bourdin J-J, Paquette E,
editors. Eurographics 2022 - Education papers. Reims, France; 2022, p. 8.
http://dx.doi.org/10.2312/eged.20221040.
[19] Balreira DG, Walter M, Fellner DW. What we are teaching in introduction
to computer graphics. In: Bourdin JJ, Shesh A, editors. EG 2017 - Educa-
tion papers. The Eurographics Association; 2017, http://dx.doi.org/10.2312/
eged.20171019, ISSN: 1017-4656.
[20] Chatti MA, Dyckhoff AL, Schroeder U, Thüs H. A reference model for
learning analytics. Int J Technol Enhanced Learn 2012;4(5/6):318. http:
//dx.doi.org/10.1504/IJTEL.2012.051815.
[21] Ihantola P, Vihavainen A, Ahadi A, Butler M, Börstler J, Edwards SH, et al.
Educational data mining and learning analytics in programming: Literature
review and case studies. In: Proceedings of the 2015 ITiCSE on working
group reports. ITICSE-WGR ’15, New York, NY, USA: Association for
Computing Machinery; 2015, p. 41–63. http://dx.doi.org/10.1145/2858796.
2858798.
[22] Allcoat D, von Mühlenen A. Learning in virtual reality: Effects on per-
formance, emotion and engagement. Res Learn Technol 2018;26. http:
//dx.doi.org/10.25304/rlt.v26.2140.
[23] Nesenbergs K, Abolins V, Ormanis J, Mednis A. Use of augmented and
virtual reality in remote higher education: A systematic umbrella review.
Educ Sci 2021;11(1):8. http://dx.doi.org/10.3390/educsci11010008.
[24] Banihashem SK, Aliabadi K, Pourroostaei Ardakani S, Delaver A, Nili Ah-
madabadi M. Learning analytics: A systematic literature review. Interdiscipl
J Virtual Learn Med Sci 2018;9(2). http://dx.doi.org/10.5812/ijvlms.63024.
[25] Ai-Lim Lee E, Wong KW, Fung CC. How does desktop virtual reality
enhance learning outcomes? A structural equation modeling approach.
Comput Educ 2010;55(4):1424–42. http://dx.doi.org/10.1016/j.compedu.
2010.06.006.
[26] ACM Association for Computing Machinery, IEEE Computer Society, editors.
Computing curricula 2020: Paradigms for global computing education.
ACM; 2020, http://dx.doi.org/10.1145/3467967.
[27] Visual–spatial ability – APA dictionary of psychology. 2018, URL https:
//dictionary.apa.org/visual-spatial-ability. (visited on 30 July 2021).
11
Computers & Graphics 112 (2023) 1–12
[28] González Campos JS, Sánchez-Navarro J, Arnedo-Moreno J. An empirical
study of the effect that a computer graphics course has on visual-spatial
abilities. Int J Educ Technol Higher Educ 2019;16(1):41. http://dx.doi.org/
10.1186/s41239-019-0169-7.
[29] Dalgarno B, Lee MJW. What are the learning affordances of 3-D
virtual environments? Br J Educ Technol 2010;41(1):10–32. http://dx.
doi.org/10.1111/j.1467-8535.2009.01038.x, _eprint: https://bera-journals.
onlinelibrary.wiley.com/doi/pdf/10.1111/j.1467-8535.2009.01038.x.
[30] Milgram P, Takemura H, Utsumi A, Kishino F. Augmented reality: A class
of displays on the reality-virtuality continuum. In: Das H, editor. Tele-
manipulator and telepresence technologies. SPIE; 1995, p. 282–92. http:
//dx.doi.org/10.1117/12.197321, URL http://proceedings.spiedigitallibrary.
org/proceeding.aspx?articleid=981543.
[31] Peternier A, Vexo F, Thalmann D. The mental vision framework - A
platform for teaching, practicing and researching with computer graphics
and virtual reality. In: Pan Z, Cheok AD, Müller W, El Rhalibi A, edi-
tors. Transactions on edutainment I. Lecture notes in computer science,
Springer; 2008, p. 242–60. http://dx.doi.org/10.1007/978-3-540-69744-2_
19.
[32] Vitsas N, Gkaravelis A, Vasilakis AA, Vardis K, Papaioannou G. Ray-
ground: An online educational tool for ray tracing. In: Romero M,
Sousa Santos B, editors. Eurographics 2020 - Education papers. The Euro-
graphics Association; 2020, http://dx.doi.org/10.2312/eged.20201027, ISSN:
1017-4656.
[33] Jerald J. The VR book: Human-centered design for virtual reality.
Association for Computing Machinery and Morgan & Claypool; 2015.
[34] Anderson T, Shattuck J. Design-based research: A decade of progress
in education research? Educ Res 2012;41(1):16–25. http://dx.doi.org/
10.3102/0013189X11428813, Publisher: American Educational Research
Association.
[35] Huang HM, Liaw SS. An analysis of learners’ intentions toward virtual real-
ity learning based on constructivist and technology acceptance approaches.
Int Rev Res Open Distrib Learn 2018;19(1):91–115.
[36] Documentation of experience API (or xAPI): A specification for learning
technology. 2018, URL https://xapi.com/. (visited on 18 January 2022).
[37] Documentation of learning locker: A conformant open source learning
record store (LRS). 2015, URL https://docs.learninglocker.net/welcome/.
(visited on 30 July 2021).
[38] Wilkinson MD, Dumontier M, Aalbersberg IJ, Appleton G, Axton M, Baak A,
et al. The FAIR guiding principles for scientific data management and
stewardship. Sci Data 2016;3(1):160018. http://dx.doi.org/10.1038/sdata.
2016.18, URL https://www.nature.com/articles/sdata201618. Number: 1
Publisher: Nature Publishing Group.
[39] Judel S, Schroeder U. EXCALIBUR LA - An extendable and scalable infras-
tructure build for learning analytics. In: 2022 International conference on
advanced learning technologies. 2022, p. 155–7. http://dx.doi.org/10.1109/
ICALT55010.2022.00053.
[40] Heinemann B, Ehlenz M, Görzen S, Schroeder U. xAPI made easy: A
learning analytics infrastructure for interdisciplinary projects. Int J On-
line Biomed Eng (IJOE) 2022;18(14):99–113. http://dx.doi.org/10.3991/ijoe.
v18i14.35079.
[41] Knight H, Carlisle S, O’Connor M, Briggs L, Fothergill L, Al-Oraibi A, et al.
Impacts of the COVID-19 pandemic and self-isolation on students and staff
in higher education: A qualitative study. Int J Environ Res Public Health
2021;18(20):10675. http://dx.doi.org/10.3390/ijerph182010675, URL https:
//www.mdpi.com/1660-4601/18/20/10675. Number: 20 Publisher: Multi-
disciplinary Digital Publishing Institute.
[42] Ericsson KA, Simon HA. Protocol analysis: Verbal reports as data. Protocol
analysis: Verbal reports as data, Cambridge, MA, US: The MIT Press; 1984,
p. 426, Pages.
[43] Grossman T, Fitzmaurice G, Attar R. A survey of software learnability:
Metrics, methodologies and guidelines. In: Proceedings of the SIGCHI
conference on human factors in computing systems. CHI ’09, New York, NY,
USA: ACM; 2009, p. 649–58. http://dx.doi.org/10.1145/1518701.1518803.
[44] Gibson D, de Freitas S. Exploratory analysis in learning analytics. Tech-
nol, Knowl Learn 2016;21(1):5–19. http://dx.doi.org/10.1007/s10758-015-
9249-5.
[45] Thomas DR. A general inductive approach for analyzing qualitative
evaluation data. Am J Eval 2006;27(2):237–46. http://dx.doi.org/10.1177/
1098214005283748, Publisher: SAGE Publications Inc.
[46] Krathwohl DR. A revision of bloom’s taxonomy: An overview. Theory
Into Pract 2002;41(4):212–8. http://dx.doi.org/10.1207/s15430421tip4104_
2, Publisher: Routledge.
[47] Sutcliffe AG, Kaur KD. Evaluating the usability of virtual reality user in-
terfaces. Behav Inf Technol 2000;19(6):415–26. http://dx.doi.org/10.1080/
014492900750052679.
[48] Nielsen J. Enhancing the explanatory power of usability heuristics. In: Pro-
ceedings of the sigchi conference on human factors in computing systems.
CHI ’94, New York, NY, USA: Association for Computing Machinery; 1994,
p. 152–8. http://dx.doi.org/10.1145/191666.191729.
B. Heinemann, S. Görzen and U. Schroeder
[49] Sutcliffe AG, Poullis C, Gregoriades A, Katsouri I, Tzanavari A, Herak-
leous K. Reflecting on the design process for virtual reality applications. Int
J Hum–Comput Interaction 2019;35(2):168–79. http://dx.doi.org/10.1080/
10447318.2018.1443898, Publisher: Taylor & Francis.
[50] Wang W, Cheng J, Guo JL. Usability of virtual reality application through
the lens of the user community: A case study. In: Extended abstracts of
the 2019 CHI conference on human factors in computing systems. CHI EA
’19, New York, NY, USA: Association for Computing Machinery; 2019, p.
1–6. http://dx.doi.org/10.1145/3290607.3312816.
[51] Wagner P, Malisz Z, Kopp S. Gesture and speech in interaction: An
overview. Speech Commun 2014;57:209–32. http://dx.doi.org/10.1016/
j.specom.2013.09.008, URL https://www.sciencedirect.com/science/article/
pii/S0167639313001295.
[52] Mayer RE. Multimedia learning. Cambridge University Press; 2001,
http://dx.doi.org/10.1017/CBO9781139164603, URL https://www.cambridg
e.org/core/books/multimedia-learning/E9595926786F5DEA326A3774D2F23
DB2.
[53] Buchner J, Kerres M. Media comparison studies dominate comparative re-
search on augmented reality in education. Comput Educ 2023;195:104711.
http://dx.doi.org/10.1016/j.compedu.2022.104711.
[54] Ramaseri Chandra AN, El Jamiy F, Reza H. A systematic survey on cyber-
sickness in virtual environments. Computers 2022;11(4). http://dx.doi.org/
10.3390/computers11040051, URL https://www.mdpi.com/2073-431X/11/
4/51.
12
Computers & Graphics 112 (2023) 1–12
[55] Oberdörfer S, Latoschik ME. Knowledge encoding in game mechanics:
Transfer-oriented knowledge learning in desktop-3D and VR. Int J Com-
put Games Technol 2019;2019:7626349. http://dx.doi.org/10.1155/2019/
7626349, Publisher: Hindawi.
[56] Velho L, Giannella J, Carvalho L, Lucio D. VR tour: Guided par-
ticipatory meta-narrative for virtual reality exploration. Rev GEMInIS
2019;10(2):122–40.
[57] LaViola JJ. A discussion of cybersickness in virtual environments. ACM
SIGCHI Bull 2000;32(1):47–56. http://dx.doi.org/10.1145/333329.333344.
[58] Janßen D, Tummel C, Richert A, Isenhardt I. Towards measuring user
experience, activation and task performance in immersive virtual learning
environments for students. In: Allison C, Morgado L, Pirker J, Beck D,
Richter J, Gütl C, editors. Immersive learning research network. Com-
munications in computer and information science, Springer International
Publishing; 2016, p. 45–58. http://dx.doi.org/10.1007/978-3-319-41769-
1_4.
[59] Fairén M, Pelechano N. Introductory graphics for very diverse audiences.
In: Bourdin J-J, Cerezo E, Cunningham S, editors. Eurographics 2013 -
Education papers. The Eurographics Association; 2013, http://dx.doi.org/
10.2312/conf/EG2013/education/009-010, ISSN: 1017-4656.
[60] Biggs JB, Tang CS-k. Teaching for quality learning at university: What
the student does. 4th ed.. SRHE and open university press imprint,
Maidenhead, England New York, NY: McGraw-Hill, Society for Research
into Higher Education & Open University Press; 2011.