See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/314838396

Potential Augmented Reality Applications in the Mining Industry

Working Paper · January 2016

DOI: 10.13140/RG.2.2.27751.44961

CITATIONS READS

0 3,244

3 authors:

Jónatan Jacobs R.C.W. Webber-Youngman

University of Pretoria University of Pretoria
7 PUBLICATIONS   5 CITATIONS    25 PUBLICATIONS   37 CITATIONS   

SEE PROFILE SEE PROFILE

Etienne van Wyk

Tshwane University of Technology
12 PUBLICATIONS   124 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

"Developing testing specifications for netting and the dynamic testing of tendons; and assess the feasibility of establishing an independent accredited support testing
facility" View project

International Mining VR Group View project

All content following this page was uploaded by Jónatan Jacobs on 13 March 2017.

The user has requested enhancement of the downloaded file.

 Potential Augmented Reality Applications in the Mining Industry
+J. Jacobs, +R.C.W. Webber-Youngman,*E.A. van Wyk
+University of Pretoria, Department of Mining Engineering

*Tshwane University of Technology, Department of Computer Science

Synopsis
Visual based technologies have been available for decades, including Augmented Reality (AR),
but it is only recently that AR technology has experienced a massive drive in research and
development. Following this, a lack of R&D was identified in the South African Mining Industry
in finding applications for this technology. In order to support the innovation drive in the
mining industry, a critical investigation was conducted into finding potential applications for AR
that would benefit the industry. The investigation was primarily concerned with identifying
possible uses for AR that would reduce costs, improve efficiencies, increase productivity and
enhance safety. For these criteria, or for various combinations thereof, multiple potential AR
applications were identified. From these applications, the ones that were deemed to have the
highest rate of practicality and implementability were further expanded into conceptual
applications. These conceptual applications can broadly be classified as AR applications for
drilling work, navigational aid and operator assistance, maintenance and repair tasks, and lastly,
the provision of real-time information. SWOT analyses were done on the conceptual
applications and the general conclusions were that AR could bring multiple benefits to the
mining industry and greatly add value and efficiency to the mining process.
Keywords: Augmented Reality, AR Applications, Mining Industry, New Technology, Visual
Technology, Innovative Technology, Futuristic Technology

Introduction
With the recent economic downturn affecting the mining industry, mining companies
experience greater calls for optimisation than ever before. This drive for innovation, in the
search for ever increasing efficiencies and lower operating costs, has extended to service
providing companies and researching bodies as well. It is through this need that critical
investigations into the technological field were conducted; in order to determine what
technologies are available that could benefit the mining industry.
AR was identified as such a technology, along with other visual-based technologies within the
Reality-Virtuality continuum, such as Augmented Virtuality and Virtual Reality. In a landmark
paper published in 1994, Milgram and Kishino introduced the concept of a virtuality continuum.
It relates to the mixture of objects presented in any particular display situation, where real
environments are shown at one end of the continuum and virtual environments at the opposite
end (Milgram & Kishino, 1994).
The merging of real and virtual worlds is generically referred to as Mixed Reality (MR). Mixed
reality is a term that is all encompassing and spans much of the virtual continuum, shown in
Figure 1. All real and synthetic aspects of real and virtual environments fall somewhere along
the virtual continuum and many of the terms presented by Milgram and Kishino (1994) are
currently in regular use.
The term reality, in its widest sense, includes everything that is, whether or not it is observable
or comprehensible. Augmented reality (AR) refers to the overlaying of computer-generated
imagery or data onto real-world imagery. AR technology makes it possible to project data,
diagrams, animation or video onto transparent glasses, which the user can then see while
viewing the real world (Van Krevelen & Poelman, 2010). AR research focuses on the use of
video imagery which is digitally processed and ‘augmented’ by the addition of computer-
generated graphics.
Augmented virtuality (AV) refers to the merging of real-world objects into virtual worlds.
Physical elements, such as physical objects or people, are dynamically integrated into, and can
interact with, the virtual world in real time.
By investigating existing applications for AR, conceptual applications were designed for the
mining environment. Focus was placed on designing applications that would optimise the
mining process by reducing costs, improving efficiencies, increasing productivity and/or
enhancing safety. From these conceptual applications a few, that were deemed to have the
highest possibility to be implemented, were further evaluated. Implementability was based on
applications that could be created and adopted into the working environment with the most
ease, least technological constraints and the least amount of human resistance to a potentially
mind-set changing technology.
The conceptual applications were evaluated by means of SWOT analyses in order to measure
potential benefits compared to associated challenges with implementation. General findings
concluded that AR has the potential to greatly benefit the mining process if the design of an AR
application is done effectively.

Augmented Reality (AR)

The term Augmented Reality (AR) refers to the augmentation, or enhancement, of the perceived
reality of a user of this technology. This is accomplished by influencing the senses of the user
through various sensory inputs, e.g. audio inputs, visual inputs, or even smell or touch
simulations. Although AR is a sensory based technology that, through its core definition, utilises
any or all of the human senses to augment your perceived reality, it is predominantly only
known as a visual technology. This is due to the visual aspect having found the most
applications in a wide variety of environments. For the visual aspect, elements such as any
object or environment viewed in the real world, are augmented (supplemented) by a computer-
generated sensory input, including video, graphical displays, sound or Global Positioning
System (GPS) data (Graham, 2012).
The most appropriate description of AR can then be given as a technology which overlays a
virtual, computer-generated, element into the real world view of the user. This virtual element
can be in the form of an image, video, 2D drawing, or 3D object or hologram. The composite
view that is then created can be called the augmented reality space. Within this space the user is
often able to interact with and manipulate the virtual elements to achieve a certain goal. An
example would be interacting with a 3D hologram of a vehicle, displayed on your desk or
kitchen table, where the user is able to dissect the vehicle and view its individual parts
separately (Chen, 2009).
Other existing AR examples include various forms of task assistance, with notable efficiency
improvements when AR guidance was used in vehicles maintenance and repair (47% and 56%
faster completion times compared to other methods of information provision in order to
complete the task) (Henderson & Feiner, 2009). Major companies, such as BMW and Audi, have
created applications where mechanics receive on-the-job visual instructions and assistance
while they are working on vehicles, or which provides overlain information on parts that you
view through your device’s camera (Woollaston, 2014). This can be achieved through smart
devices (e.g. smartphones or tablets) or through the use of specifically created AR-glasses, refer
to Figure 2 for an example of BMW’s AR glasses.
The aviation industry has also been at the forefront of AR adoption. Some applications allow
pilots to see hidden information that are fundamental to their decision making process.
Examples include the visualisation of runway boundaries which are otherwise hidden in misty
weather conditions (wordlesstech.com, 2014), as depicted in Figure 3.
The strong visual nature of the technology, combined with live information feeds at the working
site, allows for clear-cut visual guidance and assistance. This in turn reduces the risk of human
error and inaccurate perceptions formed from verbal, picture or video clip inputs. This formed
part of the reason why the Safety and Health in Mine Research Advisory Board (SHMRAB) in the
UK had identified the need for AR research in 2009. From this, the “Enhanced miner-
information interaction to improve maintenance and safety with augmented reality
technologies and new sensors” (EMIMSAR) was created for mining specific AR applications
(SHMRAB, 2009). An example of a result from the EMIMSAR drive is RAG coal mining company
in Germany, which has already implemented AR maintenance applications (Catalina et al.,
2014).

Results of critical investigation

Investigations were conducted into AR applications that are either already available or in
development in various industries and fields of specialisation. Some examples included
architecture, construction, education, emergency management, gaming, industrial design,
maintenance and repair, medical, military and navigation. This spurred ideas for the mining
industry from which multiple potential applications were drawn up. The applications that were
deemed to be the most applicable and ready for implementation were then categorised and are
discussed in the following subsections. Unless otherwise stated, the figures presented in this
section are from conceptual designs of the potential applications.
Drilling Applications
When applying AR to assist with drilling practices, through visual guidance on the usage of the
drilling equipment or machine, the accuracy and efficiency of drilling could be enhanced. This
application could further be expanded to display the real-time location and orientation of the
drill bit below surface or within the rock face, as measured through various tracking
technologies (such as the low-frequency electro-magnetic location transmitting systems used in
the oil and gas industry, dating back as far as 1972 (Elwood, 1974)). The visualisation of such
positioning information of the drill bit and rod would then allow for faster reaction and
adjustments by operators to improve drilling accuracy (see Figure 4).
When coupled with drilling technologies from directional drilling, that allows control over the
drill head while drilling a hole, the drill head could potentially be steered in real-time in order to
improve drilling accuracy to near 100%. Improving drilling accuracy leads to major down-the-
line benefits for production, as well as blasting practices and has corresponding impact on
loading and hauling efficiencies.
AR could also be applied when creating the blast pattern. The pattern along with the intended
burden and spacing measurements could be displayed virtually, which would eliminate the time
required for the marking of holes. Refer to Figure 5 as a concept for virtually overlain blast
patterns (note that this aerial view does not represent what the operator would see).
Navigational Aid and Operator Assistance
Surface mining operations with poor visibility, due to high amounts of dust or rain, could greatly
benefit from AR visuals for safety purposes. An AR application could be developed that allows
operators to see hazardous scenarios and objects, such as the road boundaries, approaching
vehicles, or the distance to reverse to the dumping site or crusher. The virtual displays over the
real world view of the operator could provide various forms of information, as well as a live
video feed on “blind-spots”. Figure 6 shows an example of applying AR to indicate the position
of an approaching haul truck which is not actually visible due to dust.
Other useful virtual overlays could include information on remaining fuel capacity, current load,
estimated duration until next required fuel refill, vehicle traveling speed, a GPS guidance system
or dispatch instructions. Operator efficiency and average loads per hours could also be
monitored and displayed to encourage operators to improve their personal efficiency (see
Figure 7).
Real-time information sharing between different mining equipment will improve fleet
efficiencies. Information could include the amount of loads left to fill a truck and the current fill
factor, which can be adapted per additional load in order to strive towards optimising the final
percentage fill during the loading process, as indicated in Figure 8.
Maintenance and Repair
General maintenance and repair tasks on equipment, machinery and entire systems (e.g.
conveyor belts, entire hoisting systems, pipelines etc.) can be conducted with greater efficiency
through the utilisation of AR.
Essential information can be recalled at the working site, in real-time, and displayed by using an
AR device. This could include information on various parts and devices, as well as where they
are stored or instructions on how to remove or replace them. Other equipment or machinery
information could also be displayed such as current air or pneumatic pressures, fluid levels or
the required torque to fasten or loosen a nut. Figure 9 provides a screenshot from an example
AR truck maintenance app.
Such an AR application could go further to provide real-time guidance and assistance with step-
by-step instructions on how to complete a specified task. Along with the ability to recall virtual
equipment manuals or any information required to perform the work, such an application could
bolster task efficiencies. In the same manner, assistance can be provided to perform effective
inspections in order to reduce the risk of unscheduled maintenance requirements. An example
of such an AR application is shown in Figure 10.
Live video chats can be held with distant or off-site experts for consultation, while working on a
certain task. Furthermore, the expert could also be able to see what the AR user is seeing,
through a camera installed on an AR device, as he is doing his work. Virtual 3D holograms or 2D
inputs, in the field of view of the user, could then be manipulated by the said expert to point out
specific areas, items or parts or to indicate required manoeuvres. Figure 11 shows an example
of applying AR for maintenance in the military industry.
Improved efficiencies in maintenance and repair activities lead to reduced time spent working
on a piece of equipment, machinery or another system. This is turn increases availability which
translates into valuable improvements in production.
Provision of Real-Time Information
AR could provide helpful and potentially lifesaving information. An AR system could be
combined with proximity detection technologies to detect dangerous equipment and warn
personnel to maintain a safe following distance. Another potential application is to combine AR
with other software systems to form a new integrated technological system. An example of this
would be where an AR-based system detects an operational error on a piece of equipment and
then provides the user with step-by-step guidance on how to troubleshoot and/or solve the
problem. Figures 12 and 13 provide examples of the application of such technology.
Information could also be embedded in individual clockcards, or other employee identification
systems. Higher-ranking employees could view this information when looking at an individual
in order to determine his/her name, qualifications, section of work, medical history, time
clocked in for work, etc. The information that can be retrieved could be dependent on the
hierarchical difference between personnel job levels. Similarly the historical location data on an
employee could be retrieved and retraced for security and efficiency monitoring purposes.
Real-time rescue operation coordination could also become more effective when utilising AR
technology. Instant commands and the provision of information on a dynamic situation could be
transferred between a rescue command station and a rescue team on surface or even
underground. The rescue team could wear AR enabled gear that allows them to see information
otherwise hidden in smoke, as well as navigation instructions from the rescue command station.
The command station would be able to track the locations of the rescue team members and
retrieve visuals on various parts of the environment to plan the optimum routes for extraction
and escape.

Analysis of Results
A SWOT analysis was completed on the various identified potential AR applications for the
mining industry. A summary can be seen in Table 1.

From Table 1 it is clear that various opportunities and challenges exist that are associated with
the implementation of AR applications. It is therefore vital to follow a structured and scientific
research and development program or process when designing specific applications prior to
attempting final implementation or development.

Conclusions
It was identified that multiple applications for AR technology in mining have already been
developed in other countries (EMIMSAR being a leader in Europe for finding innovative AR
applications for mining). Thereafter it was found that the South African mining industry has
been lagging behind in finding suitable uses for this technology. Various potential application
concepts were then designed and analysed by means of SWOT analyses.

From the investigation it was further concluded that AR could bring many benefits to the mining
industry if it is applied correctly and if applications are designed with great attention to detail;
in order to exploit the identified opportunities and negate the challenges associated with
implementation and effective usage.

Suggestions for Further Work

It is suggested that further investigative work be done on these potential conceptual

applications to take them from concept to practical implementation. In working towards this
progression a scientific research and development process is advised in order to ensure that
design and development of applications are in line with business strategy and desired
outcomes.
Further research into the psychological effects of technologies such as AR on the human mind is
also advised. Very little is currently understood of the impact of using AR for extensive (e.g.
eight hours per day) periods. As was seen in the SWOT analysis of AR applications in the mining
industry, it may be possible that extensive use of this reality-shaping technology could affect the
human mind in one of two (perhaps more) ways. One possibility is a detrimental effect, where
people become too reliant on technology. Their own sense of judgement and ability to make
accurate calculations or predictions could suffer. The other possibility is that the human mind
could reach a higher state of functionality as an AR application is capable of providing different
forms of aid in numerous environments. The last mentioned involves the human brain
“synchronising” with the technology. This could ultimately lead to increased processing capacity
due to an AR system taking over certain processing requirements (e.g. mundane calculations or
reminders). For safety reasons it is recommended that extensive research be done on the
psychological effects to determine how the threats mentioned above can be avoided and the
potential opportunities exploited.

References
Agis, F. 2016. Augmented reality vision to Pilots | wordlessTech. [ONLINE] Available at:
http://wordlesstech.com/augmented-reality-vision-pilots/. [Accessed 14 January 2016].
ARMAR. 2016. Augmented Reality for Maintenance and Repair in the U.S. military. [ONLINE]
Available at: http://monet.cs.columbia.edu/projects/armar/. [Accessed 20 April 2015].
Catalina, J. C. et al. 2012. EMIMSAR, Brussels: European Commission.
Chen, B. X. 2009. If You're Not Seeing Data, You're Not Seeing | WIRED. [ONLINE] Available at:
http://www.wired.com/2009/08/augmented-reality/. [Accessed 12 April 2015].
Continental-Corporation. 2016. Continental’s portfolio of diagnostic tools is keyed to the needs
of authorized workshops. [ONLINE] Available at: http://www.continental-
corporation.com/www/pressportal_com_en/themes/press_releases/3_automotive_group/inter
ior/press_releases/pr_2015_04_15_diagnoseportolfio_en.html. [Accessed 13 January 2016].
Driscoll, S. 2013. Imponderable Things (Scott Driscoll's Blog): Comparison of Augmented Reality
Glasses, Google Glass, Meta, castAR. [ONLINE] Available at:
http://www.imponderablethings.com/2013/09/minority-report-and-terminator-vision.html.
[Accessed 7 June 2015].
Elwood, A. 1974. Patent US3828867 - Low frequency drill bit apparatus and method of locating
the position of the of the drill head below the surface of the earth - Google Patents. [ONLINE]
Available at: https://www.google.com/patents/US3828867. [Accessed 20 September 2015].
Graham, M. et al. 2012. Augmented reality in urban places: contested content and the duplicity
of code. Transactions of the Institute of British Geographers, vol.38, no.3, pp. 464-479.
Henderson, S. & Feiner, S. 2009. Evaluating the Benefits of Augmented Reality for Task
Localization in Maintenance of an Armored Personnel Carrier Turret.Maintenance of an
Armored Personnel Carrier Turret. [ONLINE] Available at:
http://monet.cs.columbia.edu/projects/armar/pubs/henderson_feiner_ismar2009.pdf.
[Accessed 21 April 2015].
Milgram, P., Takemura, H., Utsumi, A., Kishino, F. 1994. Augmented Reality: A class of displays on
the reality-virtuality continuum [ONLINE] Available at:
http://web.cs.wpi.edu/~gogo/hive/papers/Milgram_Takemura_SPIE_1994.pdf. [Accessed 12
March 2015].
SHMRAB, 2009. Safety and Health in Mines Research Advisory Board - Annual Review 2009,
Sheffield: Health and Safety Executive.
Stothard, P.M., Squelch, A.P., Van Wyk, E.A., Schofield, D., Fowle, K., Caris, C., Kizil, M., & Schmid,
M. 2008. Taxonomy of Interactive Computer-based Visualisation Systems and Content for the
Mining Industry - Part 1. Proceedings of the AUSIMM Future Mining Conference 2008, Sydney.
Van Krevelen, D.W.F. & Poelman, R. 2010. A Survey of Augmented Reality Technologies,
Applications and Limitations. The International Journal of Virtual Reality, 9(2):1-20.
Woollaston, V. 2014. End of the mechanic? BMW smart glasses make it possible for ANYONE to
spot and fix a car engine fault just by looking at it. [ONLINE] Available at:
http://www.dailymail.co.uk/sciencetech/article-2543395/The-end-mechanic-Smart-glasses-
make-possible-fix-car-engine-just-looking-it.html. [Accessed 11 May 2015].
 Mixed Reality (MR)

Real Augmented Augmented Virtual

Environment Reality (AR) Virtuality (AV) Environment

Figure 1: Simplified representation of the virtual continuum (Stothard et al., 2008:11, adapted
from Milgram & Kishino, 1994).

Figure 2: BMW Augmented Reality Maintenance (imponderablethings.com, 2013)

 Figure 3: Augmented Reality vision to pilots (wordlesstech.com, 2014)

Figure 4: Augmented drill hole with indicated deflection on the drill head
 Figure 5: Conceptual design for AR overlain blast patterns

Figure 6: An augmented outline of an approaching haul truck

 Figure 7: Augmented overlays for operators

Figure 8: AR overlain information during loading process

Figure 9: Example of an AR app for maintenance of a haul truck (simfusionar, 2015)

Figure 10: Augmented maintenance on an engine (Continental, 2015)

Figure 11: Augmented maintenance in the military (Henderson & Feiner, 2011)

Figure 1: Real-time AR information in vehicles

Figure 2: An AR display of information on mining equipment

 Table 1: Summary of SWOT analysis of potential AR applications for the mining industry
Strengths Weaknesses
 Colour blindness factor.
 Fewer mishaps when completing tasks.
 Job loss/reduction.
 Expandable new technology.
 Dependent on other technological
 Consists of several technologies and/or
systems, devices, knowledge/ information
systems.
source or patents.
 Faster task completion times.
 Moderately high costs when various
 Can enhance communication and bridge long
technologies need to be acquired and
distances through collaboration.
combined.
 Can bridge language barriers.
 Internet or local server access is required.
 Increased awareness.
 Additional hardware and software
 Instant access to information, right at the requirements.
workplace.
 Limited visual space in human field of
 Information is in real time. vision.
 Can work with and connect numerous other  System is only as good as the software
systems and technologies. coding that runs it.
 Adds value to the other systems/  Reduces human skill improvement.
technologies that it is linked to.
 Computer system is unable to improve
 Interactive technological applications. and grow like a person.
 Better memory retention.  Retraining or additional training
 Reduced risk of inaccurate or wrong requirements.
perceptions.  Size/storage limitations on mobile display
 Elimination or reduction of mundane tasks. devices.
 Increased task efficiency.  Hardware limitations.
 Reduced operating costs.  Dependent on human interaction.
 Reduced labour requirements
 Reduced reliance on specialist
personnel/contractor skills
Opportunities Threats
 Higher task efficiency.  New/foreign concept.
 Higher brain functionality.  New ground/unknown territory.
 Increase safety to zero harm.  AR often means radical change.
 Technological incompetence.
 Too much reliance on technology.
 Poor synergy between components.

View publication stats