Interactive Learning Environments

ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/nile20

An empirical framework for developing and

evaluating a Virtual Assembly Training System in
learning factories

Xolani Nelson Skosana, Khumbulani Mpofu, John Trimble & Etienne A. van
Wyk

To cite this article: Xolani Nelson Skosana, Khumbulani Mpofu, John Trimble & Etienne A.
van Wyk (2023) An empirical framework for developing and evaluating a Virtual Assembly
Training System in learning factories, Interactive Learning Environments, 31:10, 6428-6444,
DOI: 10.1080/10494820.2022.2039946

To link to this article: https://doi.org/10.1080/10494820.2022.2039946

© 2022 The Author(s). Published by Informa

UK Limited, trading as Taylor & Francis
Group

Published online: 01 Mar 2022.

Submit your article to this journal

Article views: 1374

View related articles

View Crossmark data

Citing articles: 2 View citing articles

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=nile20
INTERACTIVE LEARNING ENVIRONMENTS
2023, VOL. 31, NO. 10, 6428–6444
https://doi.org/10.1080/10494820.2022.2039946

An empirical framework for developing and evaluating a Virtual

Assembly Training System in learning factories
a a
Xolani Nelson Skosana , Khumbulani Mpofu , John Trimblea and Etienne A. van Wykb
a
Industrial Engineering Department, Tshwane University of Technology, Pretoria, South Africa; bFaculty of
Information and Communication Technology, Tshwane University of Technology, Pretoria, South Africa

ABSTRACT ARTICLE HISTORY

Higher Education Institutions are introducing Virtual-Reality (VR) Received 19 November 2021
development courses and divisions focussing on the development of Accepted 2 February 2022
Virtual Environments which are in high demand. Students without a
KEYWORDS
programming background are not familiar with the processes of Virtual Assembly Training
developing Virtual Assembly Training Systems (VATS), consequently, Systems; Virtual-Reality;
they learn through trial and error. To address this challenge the project-based learning
development of two similar VATS was observed to propose a
framework detailing the workflow, roles, tasks and deliverables
undertaken during development. Both VATS are based on the assembly
of a top bracket, an elementary railcar component rendered via the HTC
Vive Pro. The first VATS was developed by expert developers and the
other through project-based learning with five engineering students.
The case study was conducted at the Railcar Learning Factory wherein
SolidWorks, Sketchup, Blender and Unreal Game Engine were used for
CAD modelling and visual scripting. The encountered challenges and
best practices evolved into the proposed framework. Findings suggest
that engineers without a programming background can develop VATS
when capacitated with a blend of asynchronous and synchronous
learning. Importantly, Learning Factory Managers should consider the
lifecycle of VR prototypes to ensure their usage in the industry. This
approach can contribute to generating industry-relevant research output.

Introduction
Learning Factories (LFs) have become conducive learning environments. We can distinguish
between Physical Learning Factories (PLF) and Virtual Learning Factories (VLF). The PLF provides a
real value chain enabling an on-site learning approach based on a physical product (Abele et al.,
2015). On the other hand, the VLF lacks the physical element but has the advantages of scalability,
location independence and a widened scope of problems to be addressed (Sackey & Bester, 2016).
To avoid high investment costs and the lack of flexibility in physical settings (PLF), Virtual Environ-
ments (VEs) constituting the VLF are proposed (Hoedt et al., 2017) hence their widespread adoption
across industries such as aviation (Chittaro & Buttussi, 2015; Eschen et al., 2018), medicine (Gallo & De
Pietro, 2009; Riva, 2003) and manufacturing (Abidi et al., 2019; Hirt et al., 2019).
In manufacturing, research concerns are on proving the efficacy of Virtual-Reality (VR) with
respect to task time and error rate reduction in use cases such as assembly (Boonmee et al., 2020;
Shao et al., 2020; Stork et al., 2012) and maintenance (Borsci et al., 2016; Gavish et al., 2015; Liu

CONTACT Xolani Nelson Skosana Skosanaxn@outlook.com

© 2022 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group
This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives License (http://
creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the
original work is properly cited, and is not altered, transformed, or built upon in any way.
 INTERACTIVE LEARNING ENVIRONMENTS 6429

et al., 2014) training. Findings suggest that VR-based training can reduce training time and assembly
errors (Abidi et al., 2019). Realising these benefits is especially important for mass customization
which places a high cognitive load on manual assembly operators who need to be highly skilled
(Hoedt et al., 2017).
Manufacturing-related VR studies focus on assembly task complexity (Hoedt et al., 2017), the
influence of interaction hardware (Oren et al., 2012; Shao et al., 2020), feedback cues (Abidi et al.,
2019; Oren et al., 2012; Rojas-Murillo & Pennathur, 2019), evaluation methods (Hoedt et al., 2016;
Hoedt et al., 2017; Xia et al., 2012) and the integration of features such as authoring (Stork et al.,
2012), autonomy (Ho et al., 2018; Pan et al., 2019), gaming (Ho et al., 2018; Stork et al., 2012) and
adaptability (Ho et al., 2018). Fewer studies report on the design and building work with respect
to instructional design (Boonmee et al., 2020) and other development processes as done by Liu
et al., (2013) and Nelson et al., (2020).
Most studies reporting on development passively mention the hardware and software utilized
with the assumption that the workflow is defined and understood. This could be due to the compe-
tency of developers who have built multiple VEs before. Consequently during Project-Based Learn-
ing (PBL), novice developers undergo trials and errors which could have been avoided.
Hence, this study aims to propose a framework for developing and evaluating a Virtual Assembly
Training System (VATS). A case study was conducted at the Railcar Learning Factory (RLF) initially pro-
posed by Maheso et al. (2019). The outcome of the study is a workflow detailing the roles, activities
and deliverables observed during the development of two similar VATS. The first VATS was developed
by outsourced expert developers and the other by in-house novice developers. The encountered
challenges and best practices are presented to provide guidance to LFs in their exploration of VATS.

Literature review
The primary goal of VATS is to increase the operator’s skill level without placing them in the physical
assembly setting (Hoedt et al., 2017). This can be done on assembly products of varying sizes ranging
from toys (Hoedt et al., 2017; Oren et al., 2012), micro-devices (Cecil et al., 2016) and larger industrial
products (Hirt et al., 2019; Shao et al., 2020). Koumaditis et al. (2020) proposed task complexity as an
indicator of assembly errors in actual assembly and a criterion for designing effective VATS. In Virtual
Assembly Training (VAT), the assembly product is what differs, otherwise the industry and setting
have minimal impact on VR effectiveness.
Various evaluation methods can be employed to assess the extent to which a particular VATS
fulfils the primary goal. User-based studies objectively measure the learning rate (Etemadpour
et al., 2019) and training transfer (Oren et al., 2012; Shen et al., 2019). The latter measures how
much the user applies what was learnt and the former focusses on how much the user has learnt.
Alternatively, analytic evaluation methods can be employed to evaluate usability without invol-
ving end-users. Van Wyk (2015) defines these methods, i.e. model-based, walkthroughs and heuristic
evaluations done by expert evaluators. Boonmee et al. (2020) conducted a heuristic evaluation of an
HMD-based VA application. Their evaluation was composed of 10 questions not substantiated by the
literature and administered to trainees instead of VR experts. This study employed walkthroughs to
evaluate system performance or usability and reserved empirical evaluations for future work.
The performance of the user and system is both affected by the hardware used. For instance, Gal-
legos-Nieto et al. (2017) state that maximum VA effectiveness depends on the use of haptic-enabled
devices. Shao et al. (2020) further integrated multi-sensory input channels to improve the immersion
and intuitiveness of the VA experience. These vast configurations of VR hardware make the compre-
hension of VR capabilities elusive. The taxonomy by Anthes et al. (2016) addresses this concern by
identifying two categories of VR hardware namely (a) output devices consisting of visual, haptic and
multi-sensory feedback and (b) input devices with sub-categories of the controller, navigation and
body tracking. Although force-feedback was not mentioned in their taxonomy it could have been
implied under multi-sensory feedback.
6430 X. N. SKOSANA ET AL.

Mangina (2017) addressed the need to integrate available learning objects in formats accessible
through VR and identified ISO/IEC 19788 MLR (Metadata for Learning Objects) as one of the relevant
Learning Objects (LO) standards. The metadata of 3D LO can affect the performance of both the user
and VATS hence the technical, pedagogical and Intellectual Property (IP) aspects of metadata should
be regarded earlier on in development. For instance, annotations and animations may increase the
learners’ remembrance of the experience but they can slow down the application especially if
accessed via web-based platforms.
Organizing the information rendered via different output configurations requires thorough
instructional and interaction designs. Especially because multimedia can be distracting or entertain-
ing rather than educational if haphazardly added into the VE (Rackaway, 2012). Instructional
Designers (ID) have a myriad of features available for integration, namely; no feedback, animation,
video, text, visual highlights and audio. Abidi et al. (2019) established that feedback mechanisms
reduce the error rate. However, this effect only applies for complex products and is reduced
through practice. Integrating multiple feedback mechanisms can be time-consuming and costly.
Accordingly, Oren et al. (2012) suggested the need for a robust solution that enables importing of
parts from CAD software to reduce the time spent on developing training animations, videos or
guided training via haptics.
Other complex design concepts that lengthen development time include but are not limited to
gamification, autonomy and adaptability. Palmas et al. (2019) hypothesized that the use of gamifica-
tion can enhance the efficiency of VR training. Their findings show that VR users trained through
gamification completed training 12.2% faster than the non-gamified group and committed 30.2%
less errors. Most VATS require a facilitator and become futile without human intervention. To
avoid this, scholars are building VATS for learners to use on their own without any human assistance
(Ho et al., 2018; Pan et al., 2019). This saves time, similarly with self-adaptive VATS which render scen-
arios without redundant information to enhance the learners’ experience as done by Ho et al. (2018).
The wide spectrum of assembly tasks, hardware devices, feedback mechanisms and design con-
cepts all require intensive planning to be appropriately integrated into the VATS. This can complicate
and elongate the development process. However, advanced game engines are simplifying develop-
ment and shortening its duration thereof (Mangina, 2017). Most VATS are developed using Open
Source Game Engines (OSGE) such as Unity (Hirt et al., 2019; Liagkou & Stylios, 2019; Pan et al.,
2019; Shao et al., 2020) and Unreal (Boonmee et al., 2020). Liagkou and Stylios (2019) state that devel-
oping with Unity generally requires more coding and therefore necessitates the collaboration of pro-
fessional designers and programmers. The alternatives are Closed Source Game Engines (CSGE)
which require less coding but are less preferred due to their cost implications, sizeable forums
and inability to integrate with other platforms.
Professional developers are scarce and the demand for VEs is increasing. As a result, Higher Edu-
cation Institutions (HEI) are introducing VR development courses (Takala et al., 2016). However, dis-
ciplines without a programming background have no guidelines for developing and evaluating VEs
especially when using OSGE such as Unreal Game Engine (UE).

Methodology
The narrative literature review revealed that common development methods only describe pro-
cesses at the systems level and do not outline the sub-system detail that is exclusive to VR
systems. The most common of these methods is Human-Centred Design (HCD) which was also
applied by expert VR developers in building the first VATS (outsourced VR development). The gath-
ered literature and knowledge derived from observations of outsourced VR development were
applied in a case study entailing the development of the second VATS by Engineering students
(in-house VR development). The Project Manager was a B.Tech graduate together with the two
CAD Designers. Two of the developers were enrolled for B.Eng and B.Tech degrees. Their respective
courses are listed in Table 1 and they were not required to have completed any specific modules.
 INTERACTIVE LEARNING ENVIRONMENTS 6431

Table 1. In-house team members’ roles and responsibilities.

Project stakeholder and
background
. Project Manager (Full-
time)
. Industrial Engineering
Duration of participation and deliverables
02/2019-10/2021 (3 years)
1. Research: Literature review and benchmarking of similar applications
2. Selecting and inducting developers: Granting access to cloud management platforms to
access online tutorials, download CAD files and upload deliverables
3. User requirements specifications: Using an Operational Sequence Diagram (OSD) to express
user requirements and reduce the need to read as shown in Figure 4
4. Agile management: In SCRUM, all features are noted on the product backlog and drawn from
there according to their priority (Lozada-Martinez et al., 2019). This was not effectively
done in this study
5. Handover facilitation: Ensuring that the new developer spends a week shadowing the
resigning developer to be familiarized with the project
6. Prototype testing and evaluation: Since the second prototype was incomplete, walkthroughs
were not done and the only evaluation conducted was a pick and place test

. Developer A (Part-time) 10/2020-06/2021 (8 months)

. Electrical Engineering
1. Importing and texturing
2. Fixing snap to slot
3. Login and menu interfaces

. Developer B (Full-time) 02/2021-03/2021 (2 months)

. Mechanical Engineering
1. Importing and texturing
2. Scaling formula: Useful for sizing imported objects
3. Teleporting and grabbing
4. Snap to slot: Allows part to be place in the desired position and orientation, although not
perfectly integrated
5. Project migration: Useful during handover and saves time since visual scripts are written
differently

. CAD Designer A (Full-time) 10/2020-04/2021 (6 months) 3D modelled all Digital Media Objects (DMOs)/ CAD files except
. Mechanical Engineering assembly products and WJs

. CAD Designer B (Full-time) 10/2020-06/2021 (8 months)

. Industrial Engineering
1. Rigging clamps on Blender: Process of creating bone structures/joints to enable animation
2. Testing animation on UE.

Those who participated full time worked on the project eight (8) hours during weekdays and those
who were part-time only contributed four (4) hours.
Both VATS are based on the assembly of a Top-bracket, an elementary railcar component consist-
ing of four (4) parts assembled with the aid of a Welding Jig (WJ). They were developed using visual
scripting on Unreal Engine (UE) and deployed on the HTC Vive Pro. 3D models were developed using
SolidWorks, Blender and Sketchup whilst some were downloaded from GrabCAD. The next section
presents both VATS, their look and feel and technical specifications before explaining the proposed
framework which describes the sub-level detail of developing the VATS shown in Figure 2.

Findings
Figure 1 shows three screenshots of a fully functional prototype resulting from outsourced develop-
ment. Figure 2 shows screenshots of the incomplete prototype developed in-house. Figure 3 depicts
6432 X. N. SKOSANA ET AL.

Figure 1. (a) Training mode interface, (b) parts on table and (c) snap to slot.

Figure 2. (a) Three welding workstations and (b) snap to slot for support exhaust.

the VATS architecture of both VATSs. Elements of the architecture are explained under technical spe-
cifications together with more detailed user requirements depicted in Figure 4 of the Operational
Sequence Diagram (OSD).

Technical specifications
The technical specifications described below apply to both prototypes of the VATS and were refined
to this level of detail throughout the development cycle. The VATS is built for the HTC Vive Pro con-
sisting of Head-Mounted Display (HMD), controllers and base stations. The device is linked to a com-
puter in which the VATS is installed. The VATS consists of a control system that renders information
about the virtual world and interfaces. Figure 3 shows how the three elements of the VATS interact as
elaborated in the next two paragraphs.
The user has a first-person view of the virtual world and other interfaces shown as pop-ups or
windows. Interaction is possible through controllers enabling teleportation and part manipulation
in the virtual world. Some interfaces and metadata of interactive 3D LO are activated by changes in
the user’s virtual location. The metadata of some interactive 3D objects differs from that of fixed
objects with respect to: (i) animations, (ii) enabled physics; (iii) audio during part collisions; (iv)
colour cues showing part to pick and where to place it and (v) the ability to merge with other 3D
objects. In this study, the terms, 3D LO and Digital Media Objects (DMOs) are used interchangeably.
The control system also renders information about the time spent on assembly. This data is not
yet recorded since the database is not integrated, this database will also store the learners’ creden-
tials to monitor their progress and personalize the training experience. Since this study focusses on
the management of the development process, technical descriptions of how these specifications
were integrated are not discussed at length. Similarly, there are no descriptions of how the virtual
training will transpire since empirical evaluations are herein regarded as future work. More sub-
level specifications are illustrated in Figure 4 of the Operational Sequence Diagram (OSD).
Interactions that can result in more than one outcome are shown by the outlined blocks. Dotted
lines represent changes effected onto the interfaces(s), either by the user and/or database. The four
 INTERACTIVE LEARNING ENVIRONMENTS 6433

Figure 3. (a) Head-Mounted Display Software architecture.

interfaces are colour coded as shown on the legend and activated at different time intervals. Once
the database is integrated, the assembly time and errors will be stored therein. The four (4) user inter-
faces render information typically about (a) the appropriate buttons to press, (b) the position of the
user and objects in the virtual world, (c) the part to pick, where to place it and its proper orientation,
as well as (d) the total time spent on the task counted from when the user grabs the first part.

The VATS development and evaluation framework

The process of developing in-house was observed by the Researcher who documented challenges
and best practices to derive the proposed framework shown in Figure 5. The framework is a
workflow of activities undertaken by the Researcher who was also the User-Experience (UX) Designer
and Project Manager, working with the CAD Designer and Software Engineer. The section below
explains each task, decision and challenges encountered during the prototype development
process illustrated in Figure 5.

A. Preparatory work
As indicated in the Literature review, VR has proved to be effective across multiple domains. Hence
the question of VR effectiveness has now evolved to the identification of areas where VR can have a
sizeable impact. Given the time invested in development and the public appetite for VR, the fore-
sight of where a proposed VATS will be used is critical, especially for fundraising. This detail must
be captured in the proposal before the collection of user requirements.

B. Design
In this study, user requirements were elicited using a set of questions derived from the literature. The
questions are: which parts are assembled?, how do they look?, where will they be placed?, which
tools are required?, the specific conditions and sequences that must be followed? (Stork et al.
2012). This information is expressed in words and can easily be misunderstood by developers.
6434 X. N. SKOSANA ET AL.

Figure 4. Operational sequence diagram (OSD).

The first design iteration occurred during requirements review wherein the developer bench-
marked other VR applications and provided feedback on the possible features, their alternatives
and the overall sequence of interactions. This review induced minor alterations to the Operational
 INTERACTIVE LEARNING ENVIRONMENTS 6435

Figure 4 Continued

Sequence Diagram (OSD) shown in Figure 4. In documenting these changes, items of the initial
design should not be removed from the diagrams since what seems impossible to one developer
might not be to another. Moreover, a product backlog can assist in setting priorities to avoid building
features that add little value to the system.

C. Building
The SCRUM software development framework was applied in building the prototype. This fra-
mework assists people, teams and organizations generate value through adaptive solutions
6436 X. N. SKOSANA ET AL.

Figure 5. Framework for developing and evaluating a VATS.

for complex problems (Schwaber & Sutherland, 2021). During daily SCRUM meetings, developers
reported little to no progress and reiterated the need for training. More productivity was
observed when developers worked in pairs rather than alone. Activities undertaken during
this phase include:
 INTERACTIVE LEARNING ENVIRONMENTS 6437

Figure 5 Continued

. Scene creation: A VR template was used to create a VR-based environment, the template already
has the teleporting and grabbing feature.
. Importing and texturing: To enable interaction with individual meshes on UE, the Datasmith
plugin was used to import a.STEP file. Some meshes were merged to easily translate the
object, however, meshes that the user needs to interact with should not be grouped. Best practice
is to save merged meshes to easily access, import and scale them in any VR map. DMOs awaiting
rigging might need to be reimported after being rigged, therein the developer will spend time
scaling, positioning and testing. Since the VR simulation work ran in parallel with modelling, it
meant that when prioritizing DMOs to import, those that did not need rigging were ranked
first priority to avoid working backwards.
. Scaling and positioning: A dilemma here is that when objects are imported they appear small on
the world map, but massive during immersion. Thus the scale of imported objects needs to be
adjusted for them to retain their real-life sizes. To achieve this, the scaling formula shown in
Equation (1) was applied on all DMOs. For instance, the workbench with a length of 1.6 m trans-
lated to 1.3 m when imported into UE, x is the scaling factor for the x-axis and the same was done
for the y- and z-axes:

x × 1.3 = 1.6
1.6 (1)
x= = 1.23
1.3
6438 X. N. SKOSANA ET AL.

Teleporting to further located objects is easier thus the distance between objects was widened, ulti-
mately altering the layout. Additionally, scaling through trial and error is easier in the presence of a
test subject for the VR walkabout.

. Snap to slot: This feature is implemented to focus the users’ attention using highlights. A strict
slot orientation complicates part placement and increases part repositioning time, although this
experience can be unsettling, an assumption of this study is that it increases remembrance of the
procedure as opposed to a simple drag and drop.
. Animation: Animation can only be applied on skeletal meshes created through rigging, but there
is a challenge in that rigging is time-consuming. Clamping and jig rotation are the two instances
that require animation. Considering the effort required, the question was raised that since the
user only views an animation, would a video not suffice in showing motions that would otherwise
be animated? If videos can be used it will shorten development time.

D. 3D modelling and rigging

The CAD output file types were.STEP,.STP and.SLDPRT and were uploaded on a cloud platform for
ease of accessibility. Some were imported as one mesh making it impossible to separate the assem-
bly product from the jig and animate the clamps. Challenges encountered in this phase necessitate
the need for the solution by Wang et al. (2010) which allows live modification of parametric designs
in a virtual environment.
Modelling work began in October 2020 on Sketchup to construct the training center, raw parts
trolley, table with raw parts, workbench and products trolley shown in Figure 2 (a). The tig
welding tool and PPE rack were downloaded from GrabCAD. The Researcher gave specifications
on the functionality of each equipment and the CAD Designer constructed them after referring to
industry norms using videos and pictures available online. The main requirement by the Researcher
was that all DMOs should be sized using the scale of 1:1, no DMO was rejected during CAD Design
review.
Rigging is the process of creating a bone structure (armature) to a static mesh to create a skeletal
mesh for animation purposes. After rigging using Blender, the skeletal mesh can either be animated
on Blender or UE. To achieve this, two plugins were installed on Blender: SendToUnreal and Rigify.
Python Editor Script, Editor Scripting Utilities and Script are the three plugins activated on UE to
enable animation.

E. Evaluations
The VR walkabout, pick and place tests are explained under evaluation below, both these tests were
discovered during the course of this study. The cognitive walkthrough, think-aloud, protocol analysis
and heuristic evaluations were defined by Van Wyk (2015). These tests are proposed in the sequence
illustrated by the proposed framework. The first two tests provide immediate user feedback. The
third records the users’ performance for further analysis and the last allows for new requirements
to be benchmarked against existing VATS before concluding on whether they are implementable
or not.

In-house VR development
The proposed framework was derived from in-house development which began in May 2020 and
ended in June 2021 having attained deliverables listed in Table 1. All project stakeholders had no
experience in VR software development and capacitated themselves using asynchronous e-learning.
Therein various keywords were entered into a search engine to access and filter the relevant infor-
mation in a form of videos and comments from various sources such as YouTube, StackOverflow and
Unreal Engine Forums.
 INTERACTIVE LEARNING ENVIRONMENTS 6439

Outsourced VR development
The team of expert VR developers consisted of one Project Manager, a CAD Designer and a VR devel-
oper. The only specifications provided to expert developers are shown as learning outcomes in Appen-
dix A. Consequently more detailed user requirements were overlooked thus defying the significance of
Human-Centred Systems Design (HCSD). Ultimately, only three (3) of the six (6) specified learning out-
comes were integrated into the VATS in a year beginning August 2020 until August 2021.
The developers submitted the proposal, design document and three versions of the prototype
evaluated by the Researcher. The first evaluation checks whether the information required in suc-
ceeding phases is captured in documents and the latter confirms whether the system fulfils the
specified requirements (Zimmermann & Grötzbach, 2007). A notable outcome of document
reviews suggests that worded design documents are susceptible to misinterpretations and should
be supplemented by diagrams.

Evaluation
A cognitive walkthrough evaluates the usability and user-friendliness of a system without the actual
end-user (Van Wyk, 2015). The functional prototype underwent three (3) iterations of a cognitive walk-
through for about 1 week in June, July and August. Therein the Researcher and one developer took
turns in immersing themselves and executed tasks as end-users whilst noting errors and bugs. The out-
sourced developers provided feedback and identified each comment in one of four ways: (a) bug: an
error that needs fixing in the VATS, (b) change: change request that can be integrated although not in
the design document, (c) major change: change request that cannot be integrated due to the effort
required in implementing it and (d) non-applicable (N/A): a comment on an issue not experienced
by the developer during their testing. Results of this analytic evaluation are shown in Table 2.
The non-functional prototype was not subjected to any analytical evaluation as suggested by Van
Wyk (2015). Instead only documents reviews and the VR walkabout, pick and place tests were conducted.
The first evaluates whether the documents resulting from each development phase capture all infor-
mation needed in succeeding phases (Zimmermann & Grötzbach, 2007). The latter tests whether
virtual objects are correctly positioned for ease of movement, grabbing and placing in highlighted slots.

Discussion and recommendations

Of the 25 tested items, the cognitive walkthrough indicated only one Major change, 12 Changes (48%),
6 Bugs (24%) and 6 other concerns not experienced by the developer. The high number of Changes
indicated that the design document did not capture all the evolving requirements, especially notable
in this case where the end-users were not familiar with VR. The Major change represents the dilemma
of balancing realism with ease of learning as experienced users might prefer virtual hands whilst new
users need to see the controllers to remember the buttons to press. Bugs are inevitable and can be
reduced through multiple gameplays as some do not occur at every gameplay. Similarly, some con-
cerns were not identified by the developers, this emphasizes the benefit of keeping developers
close to the users for continuous feedback. Although this can be done via cloud-based deployments,
regular discussions between the user and developer provide more insights.

Comparison of development alternatives

The team size and duration of development were the same for both prototypes. However, only the
outsourced VATS is usable with the learning and assessment modes enabling the user to pick and
place parts whilst showing the assembly time. The planned features are shown in Figure 4 and
those attained in the in-house VATS are listed in Table 1. The gap is enormous due to two reasons:
(a) learning VR development from open access platforms requires the skill of effectively searching
6440 X. N. SKOSANA ET AL.

Table 2. Cognitive walkthrough feedback.

Change
Comment Feedback type
1. Remind user to press teleport button before making a There should be no need for pressing the teleport Bug
selection between discovery and assessment button before making selection, this bug was
fixed
2. Can there be an option of changing between the controller The current scope cannot cover this change as it is Major
and virtual hand? difficult to retrofit change
3. Objects move through the jig, this is not realistic Collision box added Change
4. Consider having only two modes: learning and assessment Done Change
The learning mode can be the same as the current
assessment mode but without the other irrelevant parts
5. The assessment mode will have the irrelevant parts and no Added a timer to limit the user to 60 s after which Change
blue highlight for correct positioning then when the user he/she can restart or quit
incorrectly positions the part on the jig the following can
happen: (i) score reduction when the user drops the parts or
picks up the wrong parts. (ii) if that is complex, there will be
a popup saying “try again” and provide the options of repeat
level or quit game
6. ESC key not working Fixed Bug
7. Add Voice Over (VO) saying “Proceed to the welding jigs The VO was reworded Change
bench”
8. Show an arrow towards the workbench Used a glow mesh to show arrow Change
9. No sound when dropping some objects Fixed Bug
10. Please ensure that the menu button works to allow the Done Bug
user the option of exiting the level
11. Time and score board to measure performance Effected using countdown timer Change
12. Picking up the wrong part should affect the score Replaced by countdown timer N/A
13. Some parts disappear after falling Solved by adding collisions Bug
14. There is no congratulations message after task completion Voice Over was already there N/A
15. Arrow for teleporting is not accurate Works perfectly when tested N/A
16. In discovery mode, position the workbench in the same Done by rotating the welding jig Change
orientation as in assessment mode
17. Add button for repeating audio instructions Done by adding a 30 s time delay before repeat Change
18. Highlight the part to pick first Done Change
19. Highlight where to place the part in learning mode Done Change
20. Controller should vibrate when passing through the table Beyond project scope and is difficult to retrofit N/A
21. Consider the orientation of parts when positioning the Done Change
parts
22. Record time spent on assessment mode Done by adding timer displaying elapsed time Change
23. Parts jump to outside the workstation Cannot emulate error N/A
24. The second part cannot fit onto the jig Third part not second, works well when tested, Bug
adjust error tolerance if problem persists
25. Grab button does not release sometimes Happens when passing an object from one hand N/A
to the other. Repress grab button to correct

for solutions as done by both expert and novice developers; (b) developing on a full-time basis allows
the developer to focus from start to finish unlike when there are other commitments.
Outsourcing development has the potential for skills and technology transfer, however, no time
and resources were contractually allotted for this course hence the need for a Service Level Agree-
ment (SLA). The SLA would explain the process of skills transfer and source code sharing to enable in-
house upgrades by the LF team.

Conclusion and future work

VR has proved to be an effective training tool in the manufacturing industry and academia alike. The
challenge and opportunity for academia are to equip students with VR development skills to expand
access to educational VEs and create new career pathways. This study investigated two approaches
of enhancing VR development skills selected based on time and resource constraints. The first
approach intended to expose novice developers to expert developers, but this did not fully materi-
alize due to the lack of an SLA detailing what is expected from the expert developers. The second
 INTERACTIVE LEARNING ENVIRONMENTS 6441

approach employed project-based learning with developers without any programming background.
The outcome of this approach suggests that asynchronous e-learning needs to be blended with syn-
chronous learning to enable live student enquiries with expert trainers.
The framework proposed in this study can assist developers in understanding the sequence of
tasks and decisions necessary in developing and evaluating a VATS. Future work will investigate
how the framework changes when: (a) used by experienced developers who could be capable of
fulfilling more than one role, (b) using different 3D modelling and game engine software, (c) integrat-
ing features such as gamification and authoring tools. In this study, only a narrative literature review
was conducted, there is a need for a systematic literature review to identify metrics and parameters
of comparing varying development methods at systems and sub-system level.
The contribution by interdisciplinary engineers proves that you do not have to be an ICT graduate
or trained developer to contribute towards developing VATS, however, an individual’s aspirations,
attitude and priorities do affect development time. As Brook (2021) rightfully said, the motivation
to act and learn is both personal and political and centres around an understanding of the
current situation and an urge to improve for a better future. If the urge is lacking, there will be
minute progress in development. Moreover, LF managers should consider the lifecycle of prototypes
they investigate to ensure that students work on sustainable products that will end up in the indus-
try’s value chain. This approach can contribute to industry-relevant research output.

Notes on contributors

Xolani Skosana is an M.Eng Industrial Engineering candidate with a B.Tech

degree in Industrial Engineering. He is the technical leader of the X-Reality
(XR) Lab at the Rail Manufacturing Centre for Entrepreneurship Rapid Incubator
(RMCERI). He is currently working as a Research Assistant managing the devel-
opment of the Digital Automotive Manufacturing Factory (DAMF) for use-cases
of training, virtual commissioning and product development.

Khumbulani Mpofu is a professor of Industrial Engineering, he serves as the

National Research Foundation South African Research Chair Initiative: Chair
In Future Transport Manufacturing Technologies as well as the Gibela Research
Chair in manufacturing and Skills Development. He is a national rated
researcher with an interest in the application of various technologies in manu-
facturing and their associated commercialisation.

John Trimble is the flagship professor in Industrial Engineering and supervises

Masters and Doctoral students. He has taught engineering and computer
science in USA, Zimbabwe, Rwanda and South Africa. He is also the president
of the International Network on Appropriate Technology (INAT).

Etienne van Wyk is the executive dean of the ICT Faculty at TUT and has been
teaching computer science for more than 30 years. His current research inter-
ests are in extended reality (XR) technologies and immersive learning. His pio-
neering research on the use of virtual reality for safety training in the mining
industry has led to the introduction of VR training systems at 15 mines.
6442 X. N. SKOSANA ET AL.

Acknowledgements
The authors wish to thank the developers who committed themselves to building the proposed VATS and SACGRA for
the MERSETA funding. They also acknowledge the support and assistance of the Industrial Engineering Department of
the Tshwane University of Technology, Gibela Rail and the National Research Foundation (123575) of South Africa for
their assistance towards executing this research project. The opinions presented in this paper are those of the authors
and not the funders.

Disclosure statement
No potential conflict of interest was reported by the author(s).

Funding
This work was supported by National Research Foundation of South Africa: [Grant Number 123575].

ORCID
Xolani Nelson Skosana http://orcid.org/0000-0001-5152-9447
Khumbulani Mpofu http://orcid.org/0000-0003-3429-7677

References
Abele, E., Metternich, J., Tisch, M., Chryssolouris, G., Sihn, W., Elmaraghy, H., Hummel, V., & Ranz, F. (2015). Learning fac-
tories for research, education, and training. Hochschule Reutlingen.
Abidi, M. H., Al-Ahmari, A., Ahmad, A., Ameen, W., & Alkhalefah, H. (2019). Assessment of virtual reality-based manufac-
turing assembly training system. The International Journal of Advanced Manufacturing Technology, 105(9), 3743–3759.
https://doi.org/10.1007/s00170-019-03801-3
Anthes, C., García-Hernández, R. J., Wiedemann, M., & Kranzlmüller, D. (2016). State of the art of virtual reality technology.
IEEE aerospace conference. IEEE, 1-19.
Boonmee, P., Jantarakongkul, B., & Jitngernmadan, P. (2020). VR training environment for electrical vehicle assembly
training in EEC. 5th International conference on Information technology (InCIT) (pp. 238–242). IEEE.
Borsci, S., Lawson, G., Jha, B., Burges, M., & Salanitri, D. (2016). Effectiveness of a multidevice 3D virtual environment
application to train car service maintenance procedures. Virtual Reality, 20(1), 41–55. https://doi.org/10.1007/
s10055-015-0281-5
Brook, C. (2021). What does action learning look like today? Taylor & Francis.
Cecil, J., Kumar, M. B. R., Lu, Y., & Basallali, V. (2016). A review of micro-devices assembly techniques and technology. The
International Journal of Advanced Manufacturing Technology, 83(9–12), 1569–1581. https://doi.org/10.1007/s00170-
015-7698-6
Chittaro, L., & Buttussi, F. (2015). Assessing knowledge retention of an immersive serious game vs. a traditional edu-
cation method in aviation safety. IEEE Transactions on Visualization and Computer Graphics, 21(4), 529–538.
https://doi.org/10.1109/TVCG.2015.2391853
Eschen, H., Kötter, T., Rodeck, R., Harnisch, M., & Schüppstuhl, T. (2018). Augmented and virtual reality for inspection and
maintenance processes in the aviation industry. Procedia Manufacturing, 19, 156–163. https://doi.org/10.1016/j.
promfg.2018.01.022
Etemadpour, R., Kozlenko, O., & Dwivedi, P. (2019). A visualization tool for analyzing an assembly training in VR. IEEE 19th
international conference on advanced learning technologies (ICALT) (pp. 150–152). IEEE.
Gallegos-Nieto, E., Medellín-Castillo, H. I., González-Badillo, G., Lim, T., & Ritchie, J. (2017). The analysis and evaluation of
the influence of haptic-enabled virtual assembly training on real assembly performance. The International Journal of
Advanced Manufacturing Technology, 89(1–4), 581–598. https://doi.org/10.1007/s00170-016-9120-4
Gallo, L., & De Pietro, G. (2009). Input devices and interaction techniques for VR-enhanced medicine. Multimedia techniques
for device and ambient intelligence. Springer.
Gavish, N., Gutiérrez, T., Webel, S., Rodríguez, J., Peveri, M., Bockholt, U., & Tecchia, F. (2015). Evaluating virtual reality and
augmented reality training for industrial maintenance and assembly tasks. Interactive Learning Environments, 23(6),
778–798. https://doi.org/10.1080/10494820.2013.815221
Hirt, C., Holzwarth, V., Gisler, J., Schneider, J., & Kunz, A. (2019). Virtual learning environment for an industrial assembly
task. IEEE 9th international conference on consumer electronics (ICCE-Berlin) (pp. 337–342). IEEE.
Ho, N., Wong, P.-M., Chua, M., & Chui, C.-K. (2018). Virtual reality training for assembly of hybrid medical devices.
Multimedia Tools and Applications, 77(23), 30651–30682. https://doi.org/10.1007/s11042-018-6216-x
 INTERACTIVE LEARNING ENVIRONMENTS 6443

Hoedt, S., Claeys, A., Van Landeghem, H., & Cottyn, J. (2016). Evaluation framework for virtual training within mixed-mo
del manual assembly. IFAC-PapersOnLine, 49(12), 261–266. https://doi.org/10.1016/j.ifacol.2016.07.614
Hoedt, S., Claeys, A., Van Landeghem, H., & Cottyn, J. (2017). The evaluation of an elementary virtual training system for
manual assembly. International Journal of Production Research, 55(24), 7496–7508. https://doi.org/10.1080/00207543.
2017.1374572
Koumaditis, K., Chinello, F., Mitkidis, P., & Karg, S. (2020). Effectiveness of virtual versus physical training: The case of
assembly tasks, trainer’s verbal assistance, and task complexity. IEEE Computer Graphics and Applications, 40(5),
41–56. https://doi.org/10.1109/MCG.2020.3006330
Liagkou, V., & Stylios, C. (2019). Introducing VR technology for increasing the digitalization of SMEs. IFAC-PapersOnLine,
52(13), 451–456. https://doi.org/10.1016/j.ifacol.2019.11.101
Liu, X., Cui, X., Song, G., & Xu, B. (2014). Development of a virtual maintenance system with virtual hand. The International
Journal of Advanced Manufacturing Technology, 70(9–12), 2241–2247. https://doi.org/10.1007/s00170-013-5473-0
Liu, Z., Zhu, X., Wang, Y., Wu, Y., & Sun, X. (2013). Design and implementation of virtual disassembly and assembly train-
ing system of plunger pump. International conference on virtual reality and visualization (pp. 289–292). IEEE.
Lozada-Martinez, E., Naranjo, J. E., Garcia, C. A., Soria, D. M., Toscano, O. R., & Garcia, M. V. (2019). SCRUM and extreme
programming agile model approach for virtual training environment design. IEEE fourth ecuador technical chapters
meeting (ETCM) (pp. 1–5). IEEE.
Maheso, N., Mpofu, K., & Ramatsetse, B. (2019). A Learning Factory concept for skills enhancement in rail car manufac-
turing industries. Procedia Manufacturing, 31, 187–193. https://doi.org/10.1016/j.promfg.2019.03.030
Mangina, E. (2017). 3D learning objects for augmented/virtual reality educational ecosystems. 23rd International confer-
ence on virtual system & multimedia (VSMM) (pp. 1–6). IEEE.
Nelson, S. X., Khumbulani, M., Innocent, R. B., Aziz, C., Thabiso, M., Sinenhlanhla, N., & Thierry, Y. (2020). Development of a
3D interactive training platform for assembly of bogie unit in the Railcar learning factory. Procedia Manufacturing, 45,
386–391. https://doi.org/10.1016/j.promfg.2020.04.041
Oren, M., Carlson, P., Gilbert, S., & Vance, J. M. (2012). Puzzle assembly training: Real world vs. virtual environment. IEEE
Virtual Reality Workshops (VRW) (pp. 27–30). IEEE.
Palmas, F., Labode, D., Plecher, D. A., & Klinker, G. (2019). Comparison of a gamified and non-gamified virtual reality
training assembly task. 11th International conference on Virtual Worlds and games for Serious applications (VS-
Games) (pp. 1–8). IEEE.
Pan, X., Cui, X., Huo, H., Jiang, Y., Zhao, H., & Li, D. (2019). Virtual Assembly of educational robot parts based on VR tech-
nology. IEEE 11th international conference on engineering Education (ICEED) (pp. 1–5). IEEE.
Rackaway, C. (2012). Video killed the textbook star?: Use of multimedia supplements to enhance student learning.
Journal of Political Science Education, 8(2), 189–200. https://doi.org/10.1080/15512169.2012.667684
Riva, G. (2003). Applications of virtual environments in medicine. Methods of Information in Medicine, 42((05|5)), 524–534.
https://doi.org/10.1055/s-0038-1634379
Rojas-Murillo, S., & Pennathur, P. R. (2019). Selection of key visual cues in real and virtual environments for assembly
tasks. International Journal of Industrial Ergonomics, 74, 102871. https://doi.org/10.1016/j.ergon.2019.102871
Sackey, S. M., & Bester, A. (2016). Industrial engineering curriculum in industry 4.0 in a South African context. South
African Journal of Industrial Engineering, 27(4), 101–114. https://doi.org/10.7166/27-4-1579
Schwaber, K., & Sutherland, J. (2021). The Scrum Guide. 2020. Accessed April.
Shao, X., Feng, X., Yu, Y., Wu, Z., & Mei, P. (2020). A natural interaction method of multi-sensory channels for virtual
assembly system of power transformer control cabinet. IEEE Access, 8, 54699–54709. https://doi.org/10.1109/
ACCESS.2020.2981539
Shen, S., Chen, H.-T., & Leong, T. W. (2019). Training transfer of bimanual assembly tasks in cost-differentiated virtual
reality systems. IEEE Conference on virtual reality and 3d user interfaces (VR) (pp. 1152–1153). IEEE.
Stork, A., Sevilmis, N., Weber, D., Gorecky, D., Stahl, C., Loskyll, M., & Michel, F. (2012). Enabling virtual assembly training
in and beyond the automotive industry. 18th International conference on virtual systems and multimedia (pp. 347–
352). IEEE.
Takala, T. M., Malmi, L., Pugliese, R., & Takala, T. (2016). Empowering students to create better virtual reality applications:
A longitudinal study of a VR capstone course. Informatics in Education, 15(2), 287–317. https://doi.org/10.15388/
infedu.2016.15
Van Wyk, E. A. (2015). An evaluation framework for virtual reality safety training systems in the South African mining indus-
try. University of South Africa.
Wang, Q.-H., Li, J.-R., Wu, B.-L., & Zhang, X.-M. (2010). Live parametric design modifications in CAD-linked virtual environ-
ment. The International Journal of Advanced Manufacturing Technology, 50(9–12), 859–869. https://doi.org/10.1007/
s00170-010-2575-9
Xia, P., Lopes, A. M., Restivo, M. T., & Yao, Y. (2012). A new type haptics-based virtual environment system for assembly
training of complex products. The International Journal of Advanced Manufacturing Technology, 58(1-4), 379–396.
https://doi.org/10.1007/s00170-011-3381-8
Zimmermann, D., & Grötzbach, L. (2007). A requirement engineering approach to user centered design. International
conference on human–computer interaction (pp. 360–369). Springer.
Appendix A

6444
X. N. SKOSANA ET AL.
1. Accurate selection of 5. Awareness on quality
parts to weld/Bill of 2. Commissioning parts problems and appropriate 6. Awareness on
Learning outcome Material (BOM) onto welding jig 3. Clamping 4. Welding in sequence tools to solve them safety precautions
Assessment criteria Multiple choice Timed operation Timed operation Timed operation Analyzing problem effects Multiple choice
questions and causes questions
Visual-Audio (AV) cues Display of part list in Green colour on part Green colour on clamps Colouring parts differently to Display of defective parts List of PPE and safety
3DError sound signals signals correct signals readiness to signal readiness to and associated measures that
selection of irrelevant positioning clampError sound weldIncorrect welding benefitsISHIKAWA and must be followed
part signals selection of sequence and pace is FMECA sheets before, during and
incorrect clamp signalled by error sound and after welding
message
Facilitator intervention Records all attempts to Records time till correct Records attempts of Records time from first weld Annotates over virtual Creates hazardous
select the relevant part positioning is selecting correct clamp until removal of welded operations that can scenarios and
parts signalled. assembly productRecords result in defects observes the
welding sequence and pace trainee’s behaviour
Facilitator feedback Calculates the trainee’s Provides time records to Calculates the trainee’s Calculates the trainee’s Evaluates and grades the Evaluates and grades
ability to select the indicate the trainee’s ability to select the welding ability using the ISHIKAWA and FMCECA the trainee’s
right part using the ability to commission right clamp using the scoreScore = correct welding sheets reactions to
scoreScore = correct part in correct sequence scoreScore = correct sequence and pace/sum of hazards
part selection/sum of clamp selection/sum of all welds
all selections all selections