Applying design-based research for developing virtual
reality training in the South African mining industry
Etienne van Wyk
Tshwane University of Technology
Pretoria, South Africa
vanwykea@tut.ac.za
ABSTRACT
Design research is a maturing research paradigm in a number of
disciplines. Design-based research is the educational technology
form of design research. This paper explains design-based
research (DBR) and its features and principles. After a brief
overview of other research designs of the design-and-
development genre, we present a new synthesised DBR model,
which was applied in the context of developing and evaluating
virtual reality safety training for the South African mining
industry. The cycles and iterative steps of the model functioned
effectively in supporting and representing the process flow of the
research, which moved from a complex real-world problem to
dual outcomes, namely a practical real-world solution in the form
of two virtual reality training systems and a contribution to
documented theory in the form of an evaluation framework.
Categories and Subject Descriptors
H.5.2 [Information Interfaces and Presentation]: User
Interfaces – Evaluation / Methodology;
H.5.1 [Information Interfaces and Presentation]: Multimedia
Information Systems – Artificial, augmented, and virtual
realities;
K.3.1 [Computers and Education]: Computer Uses in
Education.
General Terms
Measurement, Performance, Design, Theory
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. Copyrights
for components of this work owned by others than ACM must be
honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from
Permissions@acm.org.
SAICSIT2014, September 29 - October 01 2014, Centurion, South Africa
Copyright 2014 ACM 978-1-4503-3246-0/14/09…$15.00
http://dx.doi.org/10.1145/2664591.2664627
M.R. (Ruth) de Villiers
University of South Africa
Pretoria, South Africa
ruth.devilliers1@gmail.com,
dvillmr1@unisa.ac.za
Keywords
Design-based research, development, e-training, evaluation,
industry-related research, mine safety training, virtual reality.
1. INTRODUCTION
Design science was originated by Simon [39], who distinguished
between the natural sciences, such as anatomy, astronomy and
physics, and sciences of the artificial, or design sciences, such as
engineering, product design, information technology and
instruction. In natural sciences, theories and formulas explain how
natural phenomena occur, while the design sciences relate to man-
made phenomena, where theories and models outline goals to be
achieved and procedures to accomplish them, which are set out by
prescriptive laws. Design science is characterised by the
construction and evaluation of artefacts in authentic settings.
Applied design science led to design research, which is called
design science research in the discipline of information systems
[22, 27] and design-based research in educational technology
and e-learning [4, 46]. The present research applies design-based
research (DBR) to develop innovative virtual reality (VR) e-
training programmes in the South African mining industry.
Despite stringent safety legislation, accidents in the mining
industry cause numerous fatalities and injuries. Conventional
safety training has not reduced the incidence of accidents
significantly. VR training can provide visual, interactive
simulations to extend and improve the current training. A DBR
approach was followed to develop and evaluate two safety
training prototypes, and to generate an evaluation framework for
the evaluation of desktop VR training systems, which also
provides design principles for such systems.
Section 2 outlines the research design of the study. Section 3
introduces design research and design science research, while
Section 4 explains DBR in detail. In Section 5 we propose a new
design-based research process model, which is applied in Section
6 in real-world research. The findings are discussed in Section 7
and Section 8 concludes the paper.
2. RESEARCH DESIGN
The purpose of this paper is to:
 outline the features and processes of design-based research;
 present a new DBR model, synthesised by the authors to
generate artefacts in complex training contexts; and
 illustrate the model by outlining its application in generating
VR training systems for the South African mining industry.
Hence the main thrust of this study is to present a conceptually
new DBR model and an example of how it was applied. The DBR
70
processes in the model involve iterations of design, evaluation
and redesign. Using this model, a prototype system was
generated. It was evaluated twice and, in a third cycle, was
extended from a generic system to one with a specific focus. This
culminated in two versions of the prototype and a framework for
evaluating desktop VR training systems.
The study of design research, design-science research and design-
based research was conducted by literature reviews. The ensuing
generation and evaluation of the prototypes and evaluation
framework by a DBR process included various data collection
methods: interviews, user-satisfaction questionnaires among end
users, heuristic evaluations by experts and a case study. Section 6
illustrates use of the new model in an authentic research study.
3. DESIGN RESEARCH AND DESIGN
SCIENCE RESEARCH
Design research investigates man-made phenomena relating to the
development and evaluation of innovative products and
interventions to solve real-world problems. In a seminal paper,
March and Smith [27] proposed applying design research in the
disciplines of information systems and information technology
under the name design science research (DSR). They define DSR
artefacts as constructs, models, methods and instantiations, where
constructs are the basic concepts of a domain. When multiple
constructs are combined to indicate relationships, the resulting
artefacts are called models. Steps and processes that perform goal-
directed activities are called methods. An instantiation involves
actual implementation of a working artefact to demonstrate its
feasibility and effectiveness [27]. The main focus of DSR is to
create and evaluate new and improved IT artefacts as solutions for
organisational problems and to generate new knowledge for the
body of scientific evidence [1].
DSR can lead to theory building, resulting in elaboration of
previously theorised relationships, but there is no general
agreement on whether theory is a required output. Gregor and
Hevner [20] suggest that partial attempts at generating theory via
the development of artefacts are also acceptable. Evaluating an
artefact using quantitative measurements can be adequate
evidence of theory development [20]. Gonzalez and Sol [18]
stress the importance of artefact evaluation and validation of the
research process as elements that contribute to increased rigour,
clarity and structure in design science research.
4. DESIGN-BASED RESEARCH
Design research is increasingly used for studies in educational
technology, especially for studies on the development of e-
learning and e-training [15]. In this context, the term used is
design-based research (DBR). It was selected as the research
design for this study because of its cyclic nature of design,
evaluation and redesign, and its mandatory production of theory
and actual solutions in real-life contexts. Action research [14, 29]
was considered as a possible alternative paradigm, since it can
apply to inventions, interventions and products. However, DBR
was deemed the most appropriate choice for this research due to
its focus on: (i) solving complex problems; (ii) producing
authentic artefacts; and (iii) generating dual outcomes.
A broad definition of e-learning applications includes forms such
as tutorials, multimedia productions, simulations, educational
games, interactive learning/practice environments, immersive
virtual reality technology, educational software, Web-based
learning applications, and learning management systems. The
design-based educational technology research approach can be
applied to the delivery, content and architecture of such systems
[14]. Wang and Hannafin [46] state that design-based research has
an important role to play in the development of technology-
enhanced learning environments, which incorporate teaching for
the acquisition of skills and knowledge, a variety of tools, and
technological resources.
4.1 Origin of design-based research
DBR has its roots in the design sciences such as engineering and
product design, where iterative and context-based processes are
followed to create usable products [48]. It evolved as a research
methodology based on the initial work of Brown [7] and Collins
[9]. Brown used design experiments to bridge studies of learning
with instructional interventions in complex and changing
environments. Collins proposed a systematic methodology for
conducting design experiments that would assist in developing
design theory to guide implementation of innovations. The
terminology development research is also used [33, 42] but the
Design-based Research Collective [13] chose to use the term
design-based research to avoid confusion with experimental
design or experimentation with methods of teaching.
4.2 Design-based research as a research
methodology
DBR is thus a recent research methodology which is systematic
and flexible, and aimed at improving educational practices [26].
DBR methods bridge theory and practice as they uncover
relationships between educational theory, designed artefacts, and
practice [13]. DBR is an approach that can improve educational
research to yield discernable benefits and impact on practitioners,
while also being socially responsible [34].
According to Barab and Squire [4] and Amiel and Reeves [2],
DBR is fundamentally a problem-solving paradigm, also
described as “complex interventionist research” [5:243].
Furthermore, it can employ quantitative or qualitative research, as
researchers adjust data collection methods in response to
emerging research questions and to address research goals [10,
24]. “Design-based research integrates the development of
solutions to practical problems in learning environments with the
identification of reusable design principles” [23:4089]. The
design-based researcher aims at making both practical and
scientific contributions by designing and creating effective
interventions or authentic artefacts to solve substantial and
complex real-world problems, which direct application of theory
is not sufficient to solve [15]. Hay, Kim and Roy [21] propose
that DBR holds potential for developing emergent learning
technologies in situations where new technological affordances
can be systematically explored and developed. It supports
understanding of why, how and when educational innovations
work in practice.
The DBR process differs from experimental research. It it is based
in authentic environments with complexities, and dynamics,
whereas laboratory experiments are conducted in controlled
environments. Unlike experiments, DBR is characterised by
iterative and flexible revisions of the research design [24].
Furthermore, DBR can be differentiated from pure design in that
it is based on theory, has clear research goals and produces
empirical results [17]. Iterative formative evaluation can identify
gaps between the current design and the ideal design goals,
supporting researchers in tweaking the design to meet the goals.
71
4.3 Characteristics of design-based research
Table 1 is a synthesis of DBR features that are pertinent to this
study. Each is described, along with references to the literature.
Table 1. A summary of DBR features
DBR Features
1. Appropriate for complex environments
DBR deals with important issues, sizable effects and significant
results in complex environments [5, 11, 38, 46].
2. Problem-solving
DBR is fundamentally a problem-solving paradigm. It explores
research issues related to real-world practical problems. New
solutions must be developed that are relevant to authentic
organisational issues [2, 15, 20, 22].
3. Collaborative and participative
Practitioners and researchers are engaged in long-term
collaborations. Their expertise impacts decisions in the different
phases of the research process [4, 10, 12, 38].
4. Innovative
DBR addresses complex unsolved problems in unique or
innovative ways. The development of solutions is informed by
existing design principles and technological innovations.
Methods, technologies and innovative tools are generated,
whereby new design principles can be put into practice [2, 14,
22, 35, 46].
5. Iterative
Researchers conduct systematic and iterative processes of
analysis, design, development and implementation for
continuous refinement of design interventions [2, 3, 15, 16, 46].
6. Dual outcomes
The outcomes of DBR are: (i) a practical contribution in the
form of an innovative product or intervention.
(ii) a set of design principles or guidelines, with the objective of
a theoretical contribution [2, 4, 15, 46].
7. Pragmatic, yet theoretical
DBR aims to resolve actual challenges, as it extends theories and
refines design principles, i.e. it impacts directly on practice,
leading to substantial change, while simultaneously advancing
transferable theory [2, 14, 26, 35, 46].
8. Artefacts
Authentic artefacts are produced as purposeful practical
interventions in functioning environments [6, 20, 43].
5. A NEW DESIGN-BASED RESEARCH
PROCESS MODEL
In a meta-research venture, a customised DBR process model was
constructed after examining precedents in the design- and
development research genre. Wang and Hannafin [46] compared
approaches that contributed to DBR and determined that, despite
varying foci, the underlying goals and approaches were similar:
 design experiments [7, 9],
 developmental research [37],
 design research [8, 10, 17],
 formative research [36], and
 development research [33, 42].
Considering these and others, we propose a new DBR model. The
model is also influenced by design science research, as described
by Vaishnavi and Kuechler [41] and the DBR cycle explained by
Amiel and Reeves [2]. The custom-built model is presented in
Figure 2, while the three most relevant predecessors are addressed
in Subsections 5.1, 5.2 and 5.3, and portrayed in the sub-diagrams
of Figure 1.
5.1. Development research
One of the variants from which DBR evolved is the classic
development research (DR) methodology [14, 33, 42]. DR was
used when non-standard learning content was required for
complex real-world contexts and has been used to build effective
training and performance support tools. Such situations required
research designs that assessed both the process and the outcome
of the intervention. DR is problem-oriented and interdisciplinary,
and aims to optimise development activities implemented by
educational technology applications. Prototypical products are
developed and methodological directions generated for the design
and evaluation of such products [44]. DR thus has a dual focus,
being applied to develop practical and innovative ways of solving
problems, and to propose general principles to inform future
decisions. DR searches for innovative solutions, while also
seeking findings that are transferable and practical [14]. Plomp
[32] and Van den Akker [42] describe the two outcomes as:
immediate outcomes relating to the results of an intervention or
product produced by the cyclic process, and distant outcomes in
the form of generalisable principles.
The DR process is based on the dynamic and flexible ADDIE
design model of instructional technology [30]. The five phases—
Analysis, Design, Development, Implementation, and
Evaluation— are applied in an iterative manner, using the results
of each formative evaluation for analysis and further development
in the next cycle [15]. According to Plomp [31], DR aims to
support design decisions, test design principles, improve product
quality, and stimulate professional development in educators.
5.2. The design science research methodology
Another variant is the general methodology of design science
research in the information systems (IS) discipline (see Section
3). DSR involves “analysis of the use and performance of
designed artefacts to understand, explain and very frequently to
improve on the behavior of aspects of information systems”
[41:2]. The steps of a typical DSR project are:
1. Problem awareness: emerging from literature or from new
developments in an industry or discipline. This leads to a
proposal.
2. Suggestion: new functionality is envisioned and a tentative
design generated.
3. Development: the tentative design is implemented,
producing an artefact.
4. Evaluation: the artefact is evaluated according to criteria
specified in the proposal.
5. Conclusion: the project is considered complete when results
are adjudged as satisfactory.
As described in Section 3, the outcomes of DSR may be one or
more of the following artefacts: construct, model, method or
instantiation [19, 27, 28].
5.3. The design-based research cycle
72
The standard DBR cycle is similar to DR and has, in fact, evolved
from DR. It comprises the following steps [2]:
1. Analysis of practical problems by researchers in
collaboration with practitioners.
2. Development of solutions informed by existing design
principles and technological innovations.
3. Iterative cycles of testing and refinement of solutions in
practice.
4. Reflection to produce design principles and to enhance the
implementation of the solution.
Each step can result in refinement of previous steps and
processes. The outcomes are design principles/guidelines that can
be implemented by other researchers in similar contexts, with the
ultimate objective being theory development and a practical
contribution in the form of an innovative product or intervention.
This may require long- term engagement and several cycles of
design.
5.4. Consolidation of the foregoing models
Figure 1. Composite diagram depicting the Development Research model, the General Methodology of Design Science Research,
and the Design-based Research Cycle
Figure 2 depicts the new generic process model of DBR,
synthesised by the primary researcher and influenced by the
procedures in the subdiagrams of Figure 1.This new DBR model
differs from the DBR model of Amiel and Reeves as follows:
 It extends the classic DBR model by including the design of
solutions which are not necessarily based on “existing design
principles and technological innovations” [2:34] nor “within a
theoretical framework” [25:78] nor “drawn from the existing
knowledge / theory base for the problem area” [41:10]. In the
Figure 1 is a composite diagram illustrating the development
research model, the general methodology of design science
research, as implemented in IS, and the design-based research
cycle. There are close similarities in their phases. Each procedure
starts with a problem analysis phase. In DR and DBR, there are
searches for innovative ways of solving problems, while a DSR
project also analyses problems related to new developments. The
next phase in all three is the tentative design of a potential
solution to be developed in the subsequent phase. The solutions
are then implemented and evaluated in real-world practice. All
three procedures are iterative in that feedback can be input into
previous phases, resulting in redesign, refinement and
improvement. A DSR project concludes when the evaluated
results are acceptable when measured against criteria specified in
the project proposal. The DR process produces a solution
implemented in a real context, e.g. industry, as well as generic
design principles. This is similar to the output of the Reflection
phase of DBR. DBR emphasises solutions that can be adopted
elsewhere and newly generated theory to guide similar research
and development.
domain of virtual reality systems, where this work resides,
design theory is relatively new and established design
principles are not available. The new model allows for
conceptualisation of solutions beyond existing mature
theoretical frameworks, due to the innovative nature of the
technology being applied.
 The new model applies a feature of DSR by advocating a
proposal as an output of the first phase. Amiel and Reeves
[2:5] describe the first phase of DBR (problem analysis) as
73
 the “negotiation of research goals between practitioners and
researchers”, but Vaishnavi and Kuechler [41:7] describe the
output of the first phase of DSR (problem awareness) as a
“proposal, formal or informal, for a new research project”.
Their formal proposal includes a tentative design and
performance criteria to evaluate the prototype. Similarly, the
new DBR model requires a proposal that includes research
goals.
 The new model adapted the DBR model to include a
theoretical outcome that is not merely a set of design
principles. Amiel and Reeves [2:35] describe the outcomes of
DBR as “a set of design principles or guidelines derived
empirically and richly described, which can be implemented
by others interested in studying similar settings”. Being a
design-based research methodology, the importance of design
principles as an output is indeed acknowledged, but provision
is also made for new theoretical contributions that extend
even further. Such contributions, importantly, should inform
future design and evaluation in similar environments in
practice.
PROCESS OUTCOMES
Problem Analysis of
complex problem within Research goals
real-world context and proposal
Design solution Initial design
Refine
The new DBR model integrates the phases and processes of the
precedents, as well as explicitly indicating the outcomes
associated with each phase (Figure 4):
1. Problem analysis within context: A practical, authentic
problem is identified in a complex environment and literature is
reviewed to determine the significance of the problem and to
identify current theory on the topic. Researchers and practitioners
collaborate in analysing the problem and establishing research
goals. The outcome of this step is an explicit research proposal
containing goals and evaluation criteria, items not mandatorily
specified in most DBR studies, though it is done in some DSR
work.
2. Design solution: An initial design is proposed to address the
problem. Its appropriateness is influenced by contextual
limitations and the complexity of interactions that occur in real-
world settings.
3. Develop solution: Develop a prototype that serves the
research purpose. Development is informed by existing design
principles and technological innovations. The outcome is an
innovative, functional artefact (construct, model, method or
instantiation), aspects imported from the rigour of design science
research.
4. Evaluate in practice: The artefact is tested in the real-world
setting. Data is collected and analysed to answer the research
questions and to construct design principles.
5. Reflection, leading to dual outcomes:
 Practical:
Reflection enhances the implementation of the solution. As
reflection occurs upon the data, new designs can be developed
and implemented, which leads to an ongoing sub-cycle of design-
reflection-design.
 Theoretical:

Design principles should be continually and cumulatively

documented to be transferable and utilised by others in similar
settings. In this way, new theory can be developed, but may
require multiple DBR cycles over the long term.
Develop solution Functional
This new model has certain distinct attributes as well as standard
artefact
features. The outcomes of each step are specified, notably an
explicit research proposal as an outcome of problem analysis. The
arrows, some uni-directional and others bi-directional, emphasise
the iterative nature of the process. Feedback to previous steps is
labelled Refine, since the focus is not only on evaluation, but also
Evaluate in practice Research findings on systematic refinement. Our view of DBR includes evolution of
the innovation or product that is the designed solution, but also
refinement of the problem, the methods, the tools used in design
and evaluation, and the design principles. The process culminates
in dual outputs:
Reflection (i) an implemented solution that addresses the original problem
in its real-world setting, and
(ii) documented design principles and/or other theory that can
Implement guide similar research and development efforts.
Real-world solution
6. APPLICATION OF THE DBR MODEL
This section illustrates the new synthesised DBR model in action
Document by outlining an application of its use in an authentic research
Theory /
Design principles process, where virtual reality was used to improve safety training
in mines. To demonstrate application of the model, Figure 3
Figure 2. A synthesised generic model for design-based shows the process flow from problem to solution. Each DBR
research cycle, shown as a blue circle, is an instance of the model in Figure
2. Cycle 1 is shown in full, while Cycles 2 to 4 are compressed.

74
The red blocks represent actions and the green blocks indicate
artefacts or theory deliverables as outcomes of the process.

Develop Case study:

Define DBR DBR causes of DBR
problem evaluation
cycle 1 cycle 2 incidents cycle 3
framework

Problem Analysis Improve

prototypes

The colour-coded logo

Design solution above and to the right
represents the process
undertaken in each cycle.
DBR
The process is fully
cycle 4
presented in one instance,
Develop solution namely Cycle 1 to the
left.

Implement Document
Evaluate in practice real-world the theory
solution

Reflect
VR training Evaluation
systems framework

Figure 3. Process flow diagram of the research study

The research process commenced with a definition of the real-
world practical problem. Every year more than a hundred workers
die in the South African mining industry and thousands are
injured [47]. The main purpose of this paper is to present our new
DBR model, but a secondary aim is to apply the model to
propose, model, prototype and evaluate two novel electronic
training interventions to improve the safety of mine workers.
These e-training systems, implemented by virtual reality (VR)
technology, to simulate underground conditions and potential
hazards, will supplement conventional classroom training.
During Cycle 1, a desktop VR training prototype, Look, Stop and
Fix (LSF), was designed and developed based on the problem
analysis and preceding literature reviews. This prototype
simulates the underground working areas, incorporating potential
hazards that mine workers need to identify, as well as indicating
the appropriate actions to be followed in response to each hazard.
Trainees have to spot potentially hazardous conditions, identify
the hazards, and indicate which action/s should be taken to
address the situation. Failure to correctly identify a hazard or to
specify the correct action in dealing with such a hazard causes an
animation to play out, displaying the possible disastrous
consequences of ignoring or incorrectly responding to such a
hazard. The LSF prototype was evaluated in practice by user
surveys at a large platinum mine in Rustenburg, South Africa,
which will hereafter be referred to as the Mine. Details of the
evaluation are given later in this section. After the evaluation, the
reflection step identified several problems that should be
addressed.
Following DBR Cycle 1, a formal evaluation framework and a set
of criteria were developed for evaluating desktop VR training
applications in the next round of evaluation, which involved
heuristic evaluation by experts. During DBR Cycle 2 the
prototype design was refined and then evaluated using this
framework. Several internal subcycles of the DBR steps resulted
in an improved version of LSF.
Upon reflection, trainers at the Mine indicated that the prototype,
which focuses on generic hazards, should be expanded to focus on
the major causes of incidents. This led to a case study which
investigated and identified the causes of incidents, culminating in
the design and development of a geological conditions prototype
in DBR Cycle 3. This prototype, called Interactive Simulated
Geological Conditions (ISGC), focused on major threats, namely
falls of ground. In ISGC, trainees must identify 21 different
geological conditions and specify the associated risks and controls
for each condition. Animations of the possible results of ignoring
or not correctly addressing the geological conditions are shown.
The evaluation framework was again used for evaluating this
follow-up prototype and problematic issues were identified in
ISGC. Details of all the evaluations are given later in this section.
Inadequacies also emerged in the framework. After implementing
improvements to both prototypes, namely LSF and ISGC, a

75
further DBR cycle (Cycle 4) was applied to improve the ignoring them. Figure 4 is an LSF screenshot, showing a
evaluation framework itself, so as to strengthen future possible result of ignoring a winch hazard.
evaluations. A meta-evaluation questionnaire was designed and a 4. Evaluate in practice:
meta-evaluation of the evaluation framework identified possible  The artefact is tested in the real-world setting: LSF was
improvements. installed at a computer-based training facility at No 10 shaft at
Figure 3 also shows the solution emanating from the processes. In the Mine, where all the workers were trained with LSF.
line with DBR, the solution has dual outcomes: a practical real-
world contribution in the form of desktop VR training systems
and a theoretical contribution which is an evaluation framework
for evaluating VR training systems.
The next subsections detail the steps in the four cycles of the
DBR model. To assist recall of the information in Section 5.4,
each step is defined in blue font in Cycle 1, followed by the
associated application in this study. For Cycles 2, 3 and 4,
definitions of steps are not repeated, but explanations are
provided. In each case, the research methods are summarised in a
table.
DBR Cycle 1:
1. Problem analysis within context:
 A practical, authentic problem is identified and literature is
reviewed to determine the significance of the problem and
current theory on the topic: The serious problem of safety
training in the mining industry was pinpointed; relevant
literature and documents were reviewed to identify current
theory and determine the significance of the problem.
Figure 4. Miner being pulled into the winch as a possible
 Researchers and practitioners collaborate in analysing the
consequence of operating the winch without drum guards
problem and establishing research goals: See Table 2:
Research methods. Participants were senior mine officials
who confirmed the need for improved safety training.  Data is collected and analysed to answer the research
 The outcome of this step is a proposal containing research questions and to construct design principles: More than 800
goals: Done, along with establishment of research strategy. trainees did the training in the initial runs of LSF.
2. Design solution: An initial design is proposed to address the 5. Reflection: Reflection enhances solution implementation. As
problem. The appropriateness of a design is influenced by the the researcher reflects on the data, new designs can be
contextual limitations and the complexity of interactions that developed and implemented, leading to a continuous design-
occur in real-world settings: Literature studies and a usability reflection-design cycle. Design principles should be
context analysis indicating contextual requirements and documented to be transferable and utilised in similar settings.
constraints [45], informed the initial design of a prototype. New theory may be developed: Based on the data collected,
3. Develop solution: improvements were made to the prototype, leading to a new
 Develop a prototype that serves the research purpose. version of LSF. Following the general success of the training
Development is informed by existing design principles and done at No. 10 shaft, mine management decided that all
technological innovations: A taxonomy of interactive employees who worked underground should be trained with
computer-based visualisation systems for the mining industry the improved version of LSF. The trade union wished to be
was developed [40]. Based on the design principles derived in involved in the decision and requested an opportunity to view
Step 2 and the technological options for applying VR in the LSF. After four union officials underwent training, they
defined context, a prototype, Look, Stop and Fix (LSF), was endorsed the program strongly.
built. LSF simulates the underground working areas and These research methods for Cycle 1 are summarised in Table 2.
focuses on general hazard recognition and remedial actions.
The environment presents potential hazards; mine workers DBR Cycle 2:
must identify them and indicate responsive actions. The
1. Problem analysis within context:
primary researcher was assisted in the development of LSF by
Informal observation of trainees using the prototype at No. 10
a team of students from the Centre for Creative Technologies
shaft unveiled certain problems, due mainly to the trainees’
at the Tshwane University of Technology (TUT).
inexperience in using computers. They struggled to interact with
 The outcome is an innovative, viable artefact: The LSF
the system, and language was a barrier, with many being unable
prototype is a functional system presenting a 3D virtual
to read English.
underground environment, where haulage and stope areas are
2. Design solution:
realistically simulated, along with 27 generic hazards.
An improved version of the LSF system was designed to simplify
Trainees interact with the system to spot potentially hazardous
user interaction. Due to their lack of exposure to computers, many
conditions, identify them correctly, and indicate which
trainees struggled with the computer mouse. To provide pre-
action/s should be taken to address the situation. LSF
training intervention, a PC literacy course was designed, which
graphically depicts hazardous conditions and consequences of
focused mainly on mouse skills.

76
3. Develop solution: contributing to falls of ground. This system was called Interactive
The improved version incorporated four languages (English, Simulated Geological Conditions (ISGC) and 21 geological
Setswana, isiXhosa and Sepedi) with mouse voiceovers and conditions were simulated in a virtual environment of the
individualised sound via headsets. Interaction was simplified by underground workplace. The initial target group for ISGC was 52
having a camera pan through the virtual environment on a fixed employees from the higher ranks of underground workers, i.e.
path, eliminating the requirement for trainees to manipulate shift supervisors and mine overseers.
themselves through the simulated underground environment.
Table 2. Research Design: Cycle 1 research methods Table 3. Research Design: Cycle 2 research methods.
Cycle 1 Outputs Cycle 2 Outputs
3 Mine Managers  Taxonomy of 4 Union Officials  PC Literacy system
interactive  Prototype: Improved
4 Safety Officers visualisation systems LSF (camera on fixed
2 Usability Experts
3 Mine Training  Prototype: LSF path, 4 languages,
(generic hazards Participant mouse voiceover),
Participant Managers s 2 Mine Training
identification and still using Torque
s >800 mine workers remedial actions), Experts game engine
completed prototype using the Torque
training game engine as the 2 VR Development
main development Experts
environment
>16000 mine workers
Data Literature studies, Data completed prototype
Collection questionnaires and training
interviews  Questionnaires
Methods completed by 221 Data Interviews and Data
trainees Collection Survey research
Quantitative: Methods
 Transcripts of Heuristic evaluation  Semi-structured
statistical analysis interviews with 4
Data structured interviews
Analysis Qualitative: noting with 23 randomly- Quantitative: Union Officials
patterns and themes, selected trainees statistical analysis  Heuristic evaluation
Data Qualitative: thematic
seeing plausibility done by 6 experts
Analysis analysis, calculating  User satisfaction
frequencies of questionnaire
4. Evaluate in practice: occurrences and completed by 195
The Mine acquired more computers and the improved version of responses, clustering participants
LSF was installed at the training centre. The system was evaluated
using heuristic evaluation by experts, as well as by a user 3. Develop solution:
satisfaction questionnaire survey completed by end users. The In order to simulate geological conditions accurately, the graphics
heuristic evaluation was done by six experts: two usability were required to have a high level of realism. This could not be
experts, two mining training experts and two VR development achieved with the Torque game engine environment used for the
experts. The usability experts had master’s degrees in usability, as two versions of LSF and the models and animation in ISGC was
well as experience in usability evaluation. Both were enrolled for subsequently developed in 3D Studio Max, which is a current
PhD's. The mining training experts had many years of mine industry standard for animation movies. Once again, the
training experience and were heads of their respective researcher was assisted in the development and programming of
departments. The VR developers had been involved in such the ISGC system by students from the Centre for Creative
developments for the last seven years. The user satisfaction Technologies at TUT. Figure 5 is a screen shot from ISGC,
questionnaire was completed by 195 trainees after completion of indicating a geological hazard which the trainee should identify.
their LSF training. Failure to correctly identify the hazard would trigger a graphic
5. Reflection: and fatal fall-of-ground animation.
Due to the success of LSF, further extensions were requested. A
requirement was that the next system should focus not only on 4. Evaluate in practice:
generic hazards, but specifically on the main causes of accidents. The ISGC system was installed at the training centre and
evaluated through heuristic evaluation and a user satisfaction
Table 3 summarises the research methods applied in Cycle 2. questionnaire. The heuristic evaluation was done by the same six
experts who evaluated the LSF prototype in Cycle 2. The user
satisfaction questionnaire was completed by 52 trainees after
DBR Cycle 3: completion of the ISGC training.
1. Problem analysis within context:
Due to falls of ground being the greatest contributor to mining
injuries, a case study was done to analyse the circumstances
relating to falls-of-ground incidents at the Mine.
2. Design solution:
A prototype of a new system was designed with the particular aim
of focusing on the major geological hazardous conditions

77
 improve safety training, thereby addressing the original problem
in a real-world setting, which is in line with the spirit of DBR.
DBR Cycle 4:
Problem analysis in context and evaluation in practice:
The evaluations of the LSF and ISGC prototypes during Cycles 2
and 3 not only provided valuable information regarding the
prototypes, but inadequacies in the evaluation framework also
indicated. It was the appropriate time to determine the
effectiveness of the framework itself by performing a meta-
evaluation on the instrument and criteria that had been used as the
basis of the heuristic evaluations. The meta-evaluation
questionnaire was administered to the same six expert evaluators
who had conducted the heuristic evaluation of the two prototype
VR training programs. Findings from the evaluations of the LSF
and ISGC prototypes, plus problems and gaps pinpointed in the
meta-evaluation, led to an improved version of the evaluation
framework.

This final DBR cycle was a demonstration of the versatility of the

DBR process, in that we evaluated not only the VR training
Figure 5. A miner in unsupported blocky ground in the stope
products but also refined the instrumentation and increased its
face area
sensitivity so as to strengthen future VR training artefacts.
However, a detailed discussion of this final cycle is outside the
5. Reflection: scope of this paper.
Several cycles of design-reflection-design led to many
improvements to ISGC, including orientation labels, additional 7. DISCUSSION
visual learning material and challenging scenes containing
combinations of more than one geological hazard, as is practically The four DBR cycles presented in the previous section detailed
experienced underground. the application of the DBR model. To indicate how this research
conforms to the DBR features presented in Table 1, these features
The methods applied in Cycle 3 are summarised in Table 4. are now revisited with an explanation of how each was applied.
Appropriate for complex environments:
Table 4. Research Design: Cycle 3 research methods
Even though mines have “zero harm” polices, more than a
Cycle 3 Outputs hundred miners die annually in work-related incidents and
thousands are injured. With more than 200 000 miners working
2 Usability Experts  Prototype: ISGC
(geological underground in South Africa, improving safety is an important
2 Mine Training hazards national issue that can lead to significant results.
Participant Experts recognition and Problem-solving:
s remedial actions), Inadequate or insufficient training is often cited as a root cause for
2 VR Development
Experts using 3D Studio many mining fatalities and serious incidents. However, training
Max outside of the direct working environment provides only limited
>200 mine workers  High levels of real-life opportunities. As a result, such training may fail to make
completed prototype realism required a significant impact in the tense underground working
training
environment itself. Virtual reality-based training tools, however,
Data Case Study and Survey Data can provide a basis for workers to simulate presence in their
Collection research working conditions without the associated risks. This research
Methods Heuristic evaluation  Findings of case explores issues related to solving a real-world practical problem.
study done to Collaborative and participative:
Quantitative: determine major In defining the problem and determining the context of use,
elementary statistical causes of accidents interviews were held with three mine managers, two safety
analysis  Heuristic officers and three mine training managers. During the planning
Data evaluation done by
Analysis Qualitative: thematic and design of the systems the primary researcher collaborated
analysis, calculating 6 experts
 User satisfaction round-table with several mine training practitioners in the various
frequencies of cycles of design and evaluation of the systems. Furthermore, six
occurrences and questionnaire
completed by 52 industry and academic experts conducted heuristic evaluations of
responses, clustering
participants the prototypes and participated in the meta-evaluation of the
evaluation framework. All these stakeholders made helpful
Subsequent refinements resulted in ISGC being installed as an recommendations, which were used to enhance the systems.
official training system at the Mine. All underground mine Innovative:
workers on the mine do this training on return from their annual This study investigated innovative ways, previously unused in the
leave. Every trainee has to achieve a pass score before being South African mining sector, of applying virtual reality
allowed to resume work underground. ISGC is currently being technology to improve safety training. Such interactive systems
implemented at a number of mine training centres as a solution to

78
have the advantage of exposing employees to numerous hazards
in a safe environment, and simulate the possible consequences of
unsafe acts in a "forgiving" environment.
Iterative:
As indicated in the previous section, four cycles of a systematic
and iterative process of analysis, design, development, evaluation
and reflection was followed by the researchers. The findings of
each cycle continuously refined the design input to the next cycle
to ensure greater relevance and enhanced performance in practice.
Dual outcomes:
This study has dual outcomes in that it contributes theory in the
form of an evaluation framework with implicit design principles,
as well as a practical contribution in the form of innovative,
interactive VR training systems.
Pragmatic, yet theoretical:
Implementing the VR training systems was a substantial change
in mine training practice, but the theoretical contribution is also
sharable and transferable. The evaluation framework comprises
four categories of criteria that can be applied in other contexts of
VR training application areas and do not apply only to mining.
Artefacts:
Two authentic, tangible products were developed by following the
DBR process as explained. Both these systems, LSF and ISGC,
are currently in use at a number of mine training centres.
Application of distinct features of the new DBR Model:
In Section 5.4 the differences between the DBR model of Amiel
and Reeves [2] and the new DBR model are explicated. In the
application of the new DBR model in this research, these distinct
features were applied as follows:
 Due to the innovative nature of the solution, existing design
principles did not guide the initial development of the first
prototype. Various cycles of the DBR model led to improved
design of the prototypes and these, in turn, provided the
designers and evaluators with sharable theoretical concepts.
 The first phase of the DBR study included not only the
formulation of research goals, but also a formal research
proposal, as is done in DSR. Typically such a proposal should
include the criteria for evaluation of the prototype, but in this
case no comprehensive set of evaluation criteria was
identified at proposal phase. Hence such criteria were
developed in this research after the design but prior to the
empirical work, resulting in the benefit outlined in the next
point.
 The theoretical contribution of the study extended the usual
design principles by contributing a comprehensive evaluation
framework for evaluating desktop VR training systems. This
framework could implicitly also serve as a set of design
principles.
8. CONCLUSION
This paper described the evolution of DBR from design science,
design research and development research. Its characteristics were
overviewed and a consolidated summary of DBR features was
provided. The researchers presented their own synthesised cyclic
DBR model and demonstrated how it could be applied within the
process flow of a research study involving the iterative design,
development, evaluation and refinement of prototype virtual
reality systems for e-training in the mining industry.
Limitations are the high cost of immersive VR facilities, which
restricts current application to desktop VR solutions, and the
computer skills of the mining workforce. It is anticipated that
continued exposure to this technology will raise the computer
skills of the workforce as they interact with the VR simulations.
The scope of this research relates to the use of VR in recognising
hazards within the general work areas. It excludes malfunctions in
safety-critical computer systems at mines. Neither does the
research extend to the operation of equipment or systems, i.e. the
domain of simulators, but focuses on VR simulation of generic
and geological hazards encountered in the underground working
areas.
The findings of this study suggest that the proposed DBR model
is appropriate for designing and evaluating VR training artefacts
to enhance mine safety training. As a practical contribution, the
two systems are in use at several mine training centres. As a
theoretical contribution, the evaluation framework is transferable
and customisable to other industry contexts. Furthermore, the
criteria in the framework also serve as design principles.
9. REFERENCES
[1] Adikari, S., McDonald, C. and Campbell, J. 2011. A Design
Science Framework for Designing and Assessing User
Experience. In J.A. Jacko (Ed.) Human-Computer
Interaction, Part 1. HCI1. LNCS 6761 , pp 25-34. Springer-
Verlag. Berlin Heidelberg 2011.
[2] Amiel, T. & Reeves, T. C. 2008. Design-Based Research and
Educational Technology: Rethinking Technology and the
Research Agenda. Educational Technology & Society, 11
(4): 29–40.
[3] Bannan-Ritland, B. 2003. The role of design in research: The
integrative learning design framework. Educational
Researcher, 32(1): 21-24.
[4] Barab, S. & Squire, K. 2004. Design-based research: Putting
a stake in the ground. The Journal of the Learning Sciences,
13(1): 1-14.
[5] Bell, P. 2004. On the theoretical breadth of design-based
research in education. Educational Psychologist, 39(4):
243−252.
[6] Bowler, L. & Large, A. 2008. Design-based research for LIS.
Library & Information Science Research, 30 (2008): 39–46.
[7] Brown, A. L. 1992. Design experiments: Theoretical and
methodological challenges in creating complex interventions.
Journal of Learning Sciences, 2(2): 141-178.
[8] Cobb, P. 2001. Supporting the improvement of learning and
teaching in social and institutional context. In S. Carver & D.
Klahr (Eds.), Cognition and instruction: Twenty-five years of
progress (pp. 455–478). Cambridge,MA: Lawrence Erlbaum.
[9] Collins, A. 1992. Toward a design science of education. In
E. Scanlon & T. O’Shea (Eds.), New directions in
educational technology (pp. 15-22). Springer-Verlag.
[10] Collins, A., Joseph, D. & Bielaczyc, K. 2004. Design
research: Theoretical and methodological issues. The Journal
of the Learning Sciences, 13(1): 15-42.
[11] Dede, C. 2005. Why Design-Based Research is Both
Important and Difficult. Educational Technology. Jan-Feb
2005.
[12] Design-based Research SIG, 2006. What is Design-based
Research? [Online]. Available from:
http://cider.athabascau.ca/CIDERSIGs/DesignBasedSIG/.
[Accessed: 29/03/2013].
[13] Design-Based Research Collective, 2003. Design-Based
Research: An Emerging Paradigm for Educational Inquiry.
Educational Researcher, 32(1): 5-8.
79
[14] De Villiers, M.R. 2005. Interpretive research models for
Informatics: action research, grounded theory, and the family
of design- and development research. Alternation 12(2):10-
52.
[15] De Villiers, M.R. 2012. Models for Interpretive Information
Systems Research, Part 2: Design research, development
research, design-based research, design-science research – a
meta-study and examples. In: Mora, M., Gelman, O.,
Steenkamp, A. & Raisinghani, M. S. (eds.) Research
Methodologies, Innovations and Philosophies in Software
Systems Engineering and Information Systems. IGI Global,
pp. 238-255.
[16] De Villiers, M.R. & Harpur, P.A. 2013. Design-based
research – the educational technology variant of design
research: Illustrated by the design of an m-learning
environment. Proceedings of SAICSIT ‘13, October 2013,
East London, South Africa.
[17] Edelson, D. C. 2002. Design Research: What we learn when
we engage in design. Journal of the Learning Sciences,
11(1): 105-121.
[18] Gonzalez, R.A. & Sol H.G. 2012. Validation and Design
Science Research in Information Systems. In: Mora, M.,
Gelman, O., Steenkamp, A. & Raisinghani, M. S. (eds.)
Research Methodologies, Innovations and Philosophies in
Software Systems Engineering and Information Systems. IGI
Global, pp. 403 - 426.
[19] Gregor, S. 2002, Design Theory in Information Systems,
Australasian Journal of Information Systems, Special
Issue, pp. 14-22.
[20] Gregor, S. & Hevner, A.R. 2013. Positioning and Presenting
Design Science Research for Maximum Impact. MIS
Quarterly 37(2):337–355.
[21] Hay, K.E., Kim, B. & Roy, T.C. 2005. Design-based
Research: More than Formative Assessment? An Account of
the Virtual Solar System Project. Educational Technology.
Jan-Feb 2005.
[22] Hevner, A.R., March, S.T., Park, J. & Ram, S. 2004. Design
science in Information Systems Research. MIS Quarterly
28(1):75-105.
[23] Herrington, J., McKenney, S., Reeves, T. & Oliver, R. 2007.
Design-based research and doctoral students: Guidelines for
preparing a dissertation proposal. In C. Montgomerie & J.
Seale (Eds.), Proceedings of World Conference on
Educational Multimedia, Hypermedia and
Telecommunications 2007, 4089-4097. Chesapeake, VA:
AACE.
[24] Hoadley, C. M. 2004. Methodological Alignment in Design-
Based Research, Educational Psychologist, 39(4): 203–212.
[25] Ma, Y. & Harmon, S.W. 2009. A Case Study of Design-
Based Research for Creating a Vision Prototype of a
Technology-Based Innovative Learning Environment.
Journal of Interactive Learning Research. Charlottesville.
20(1): 75 -93.
[26] MacDonald, R.J. 2008. Professional Development for
Information Communication Technology Integration:
Identifying and Supporting a Community of Practice through
Design-Based Research. Journal of Research on Technology
in Education; Summer 2008; 40(4): 429-445.
[27] March, S.T. & Smith, G. 1995. Design and Natural Science
Research on Information Technology, Decision Support
Systems. 15(4): 251-266.
[28] March, S.T. & Storey, V.C. 2008. Design Science in the
Information Systems Discipline: An introduction to the
special issue on design science research, MIS Quarterly.
32(4):725–730.
[29] McNiff, J. 2013. Action Research: Principles and Practice.
3rd Edition. New York: Routledge.
[30] Molenda, M. 2003. In search of the elusive ADDIE model.
Performance Improvement , 42 (5), 34-37.
[31] Plomp, T. 2000. Development Research in/on Educational
Development [Online]. Available from:
http://www.cite.hku.hk/events/doc/2000/DevResHKGsumm0
B20.doc. [Accessed: 12/04/2007].
[32] Plomp, T. 2002. Some reflections on development research
(DR). Seminar at the Faculty of Education, University of
Pretoria. August 2002.
Reeves, T.C. 2000. Enhancing the worth of instructional
technology research through 'design experiments' and other
development research strategies. Symposium on International
perspectives on instructional technology research for the
21st century. New Orleans, LA, USA.
[33] Reeves, T. C., Herrington, J. & Oliver, R. 2005. Design
Research: A Socially Responsible Approach to Instructional
Technology Research in Higher Education. Journal of
Computing in Higher Education, 16(2): 97-116.
[34] Reeves, T., West, R. & Orrill, C. 2006. EDIT 9990 online
discussion on Design-based Research, [Online] Available
from: http://it.coe.uga.edu/~treeves/EDIT9990/index.html.
[Accessed: 12/02/2011].
[35] Reigeluth, C. M. & Frick, T. W. 1999. Formative research: A
methodology for creating and improving design theories. In
C. M. Reigeluth (Ed.), Instructional-design theories and
models (Vol. II, pp. 633–651). Mahwah, NJ: Lawrence
Erlbaum.
[36] Richey, R. C. & Nelson, W. A. 1996. Developmental
research. In D. Jonassen (Ed.), Handbook of research for
educational communications and technology (pp. 1213–
1245). London: Macmillan.
[37] Shavelson, R. J., Phillips, D. C., Towne, L. & Feuer, M.J.
2003. On the science of education design studies.
Educational Researcher, 32(1): 25–28.
[38] Simon, H.A. 1981. The Sciences of the Artificial. Second
Edition. Cambridge, MA. MIT Press.
[39] Stothard, P.M., Squelch, A.P., Van Wyk, E.A.,
Schofield,D.,Fowle, K., Caris, C., Kizil, M., and Schmid, M.
2008. Taxonomy of Interactive Computer-based
Visualisation Systems and Content for the Mining Industry -
Part 1, Proceedings of the AUSIMM Future Mining
Conference 2008, Sydney.
[40] Vaishnavi, V. & Kuechler, W. 2009. Design Research in
Information Systems, Association for Information Systems.
[Online]. Available from: http://desrist.org/design-research-
in-information-systems. [Accessed 08/04/2011].
[41] Van den Akker, J. 1999. Principles and methods of
development research. In J. van den Akker, N. Nieveen, R.
M. Branch, K. L. Gustafson & T. Plomp (Eds.), Design
methodology and developmental research in education and
training (pp. 1–14). The Netherlands: Kluwer Academic.
[42] Van den Akker, J., Gravemeijer, K., McKenney, S. &
Nieveen, N. 2006. Introducing education design research.
Educational design research. New York: Routledge.
[43] Van den Akker, J. & Plomp, T. 1993. Development research
in curriculum: Propositions and experiences. Paper
presented at AERA meeting, April 12-16. Atlanta.
80
[44] Van Wyk, E.A., and De Villiers, M.R., 2008. Usability
Context Analysis for Virtual Reality Training in South
African Mines. SAICSIT Conference 2008, George.
[45] Wang, F. & Hannafin, M.J. 2005. Design-based research and
technology-enhanced learning environments. Educational
Technology Research and Development, 53(4): 5-23.
[46] Webber-Youngman, R.C.W. & Van Wyk, E.A. 2013.
Incident reconstruction simulations-potential impact on the
prevention of future mine incidents. Journal of the Southern
African Institute of Mining and Metallurgy, vol.113.
[47] Zaritsky, R., Kelly, A.E., Flowers, W., Rogers, E. & O'Neill,
P. 2003. Clinical design sciences: A view from sister design
efforts. [Special issue]. Educational Researcher, 32(1): 32−3
81