An integrated geotechnical risk management approach using

cloud-based risk assessments, artificial intelligence, satellite

monitoring and drone technology

J. Strydom
University of the Witwatersrand, Parktown, South Africa

ABSTRACT: Open-cast slope monitoring consists of a combination of visual, drone, satellite,

and radar monitoring. The size and complexity of open-cast strip mines and the amount of data
generated complicates hazard identification and communication on mines. In the South African
coal mining industry, geotechnical data is manually processed by the rock engineering depart-
ments. To ensure that the geotechnical data are optimally used, automated systems are required
to process and display the data. An improved geotechnical risk management approach, which uses
an integration of the latest technology, was proposed. This includes improvements in the geotech-
nical monitoring process by using artificial intelligence for crack detection, elevation differences
of drone data to track slope movement, radar monitoring, and satellite monitoring. A cloud-based
mobile application was developed to capture and display geotechnical risks to improve commu-
nication. This paper will focus on how these technologies have been utilized at Mine X.

1 INTRODUCTION

The arrival of the 4th Industrial Revolution (IR) introduced technology into the mining industry
that can significantly improve the geotechnical hazard management process on mines. The 4th IR
is recognised by the simultaneous advances in artificial intelligence (AI), robotics, the internet of
things (IoT), autonomous vehicles, 3D printing, nanotechnology, biotechnology, materials sci-
ence, energy storage, virtual reality, big data, and quantum computing (Skilton & Hovsepian,
2018). Due to the increased economic pressure and lower profit margins within the mining indus-
try, innovation and the adoption of this technology is essential to reduce cost, improve safety and
optimize productivity.
Mine X is situated in the Vaal Basin Coalfield, in the Free State Province of South Africa.
Historic underground workings are present in all three economical coal seams. The remaining
reserves are being extracted by means of an opencast dragline operation. Due to the complexity
of the geology and the presence of old underground workings, slope failures and sinkholes is a
major risk. Geotechnical risks are managed by a four-tier process. The four steps include the
evaluation of in situ geological information, cut risk assessments, monitoring, and reconciliations.
Monitoring is currently done using ground-based radars and visual inspections. The monitoring
sites for radars are prioritized based on a consequence vs likelihood matrix. The advantage of
ground-based radars is their broad slope coverage area, near real-time slope deformation tracking
and remote monitoring ability (Dick, Eberhardt, Cebrejo-Lievano, Stead, & Rose, 2014). There
is however a limitation in the ground-based radar’s ability to detect displacements above the slope
crest (Carla, Farina, Intrieri, Ketzimen, & Casagli, 2018). In strip mines, the radar is also limited
in the area of the slope that it can effectively monitor. While ground-based radars have proven to
be highly successful as an early warning system for the detection of slope failures, it is not feasible
to scan every slope on the mine. Further drawbacks in the current four tier risk management pro-
cess, is that it is managed with Excel spreadsheets, which is time-consuming, labour intensive,
and susceptible to human error. Due to the massive amount of geotechnical data that are generated
daily on a mine that need to be collected, analyzed, and communicated, a degree of automation is
required.
Until recently, it was considered that spaceborne monitoring systems could not yield infor-
mation in the required systematic fashion to act as an early warning system for slope failures.
However, Carla et al. (2019) reported that Interferometric Synthetic Aperture Radar (InSAR) data
of various slope failures indicated that there was an increase in the rate of the slope movement in
the time leading up to the failure, which indicates that failure is imminent. A combination of
ground-based and satellite radar interferometry proved to be highly useful in monitoring slopes
prone to failure (Carla, Farina, Intrieri, Ketzimen, & Casagli, 2018). Furthermore, drones have
been successfully utilized in the mining industry for the mapping of geological structures and
lithological contacts on high walls (Katuruza & Birch, 2019) (Kerle, Nex, Gerke, Duarte, &
Vetrivel, 2019). A recent trend in the civil and mining industry is to make use of artificial intelli-
gence and machine learning to automatically detect and highlight fractures or planes of weakness
on slopes (Dorafshan, Thomas, & Maguire, 2018).
The use of mobile apps is becoming increasingly common in the mining industry. Mobile apps
improve data collection and sharing, which leads to better data utilization, benefitting both pro-
duction and safety (Alpha Software , 2020).
This paper will focus on how the latest technology can be utilized to improve geotechnical
hazard management at Mine X. A slope failure at Mine X was used as a case study to evaluate
information from satellite monitoring and drone flights to determine their efficiency as early
warning systems for slope failures. To improve data management, a cloud-based risk assessment
is in the process of being developed. The aim is to eliminate human error, and automate data
capturing and processing to improve hazard communication by having a single, simplified dash-
board to display information generated by various sources.

2 METHODS

A multi bench high wall failure occurred at Mine X on 22 October 2020. The slope height was
62 m, and consisted out of sand, siltstone and shale. A pinnacle and fault plane were visible on
the high wall. It is important to note that prior to the failure and during the failure, the slope was
not scanned with radars. A back analysis of the failure was conducted to evaluate the effectiveness
of potential early warning systems.

2.1 Satellite Monitoring

SkyGeo assisted with the back analysis of the failure using InSAR to measure millimeter scale
displacements. The data acquired over Mine X originated from Sentinel-1 ascending orbital track
T116 and descending orbital track T50. The satellite acquires new data over the same area, once
every 12 days during each orbital repeat. This data is co-registered such that the data from differ-
ent acquisitions (but from the same orbital track) can be compared. Co-registration is the process
performed to spatially align all acquisitions. The pixels contain both amplitude and phase infor-
mation of the backscattered microwaves. Temporal changes in amplitude can be caused by
changes in the scattering characteristics of an object. Temporal changes in phase can be consid-
ered a combination of multiple factors, one of which being the physical change in distance be-
tween the satellite and the object. Persistent Scatterer Interferometry is the process used to correct
the phase difference between acquisitions for orbital changes, flat-Earth phase, topography and
atmosphere to attribute the remaining phase difference to a change in line of sight distance.

2.2 Drone Monitoring

Routine drone surveys are done twice per week at Mine X. This drone data was used to perform
a back analysis on the slope failure using aerial photogrammetry to create terrain elevation mod-
els. The data was processed in Agisoft Metashape. Elevation changes between two terrain models
of the same area, was illustrated by generating contours of elevation differences. The contour
threshold was set at 150 mm. This was done by comparing two tin files of the respective models.
Tin files were created in MicroStation Geopack.
 In addition, machine learning, and artificial intelligence was used during a proof of concept
study by RocketMine to automatically detect cracks that developed on the high wall crest. After
the failure occurred, the slope area was flown by RocketMine, and their crack detection algorithm
was applied.

2.3 Cloud-based Risk Assessments

To digitalize and streamline geotechnical inspections and the Cut Risk Assessments (permit to
mine), a mobile application (further referred to as the App) was conceptualized by the author and
developed by HCL Technologies Ltd. PowerApps and PowerBi was used for capturing data and
displaying reports, respectively.

3 RESULTS

3.1 Satellite Monitoring: Slope Movement Rates

The slope movement detected by InSAR can be seen in Figure 1. A stable slope directly north
of the failure was also monitored and plotted as a control. At 53 days before the failure, a clear
increase in the rate of movement was observed. In comparison, the rate of movement on the stable
slope remained consistent. Hence, the increase in the rate of movement on the failure slope could
indicate the onset of failure (Carla, et al., 2019). This indicates that even with a monitoring fre-
quency of 12 days, InSAR could have been utilized to identify potential failure areas and act as
an early warning system.

Figure 1: The total slope displacement prior to the failure was determined on stable (grey) and unstable (blue)
slopes. The red line indicates the increase in the displacement rate at 52 days prior to the failure.

3.2 Drone Monitoring: Elevation differences and crack detection

Elevation differences from aerial photogrammetry was also used to identify early sign of fail-
ure (Figure 2). In Figure 2A, 42 days before the failure, no slope movement was detected. In
Figure 2B, 21 days before the failure, downward movement on the sand crest (red contours) and
outward movement on the toe of the slope (blue contours) can be seen. Figure 2C indicates the
overall slope movement after the failure occurred. It was noted that the contours formed by the
initial movement recorded on the sand crest (see arrows, Figure 2B), have the same outline as
that of the failure (see arrows, Figure 2C). Hence, the failure could be detected with drone moni-
toring at least 21 days prior to the failure.
Furthermore, the cracks on the high wall were automatically highlighted in light blue using the
machine learning and artificial intelligence algorithms from Rocketmine (Figure 3). While the
slope had already failed in this proof-of-concept study, the automatic detection of cracks could be
a powerful tool in the detection of early crack formation prior to slope failures. The process did
not include the detection of fault planes on the high walls. Also, the high walls at Mine X Colliery
is buffer blasted, resulting in a highly fractured high wall. It should however be noted that the
processing time for the automatic crack detection amounted to 20 hours for an area of 1000 m by
120 m.

Figure 2: Elevation differences at the failure site is indicated by red and blue contours at 42 days (A), 21 days (B)
and 0 days (C) prior to the failure. The black arrows point to the outline of the sand crest where the failure occurred.

Figure 3: The automatically detected cracks are indicated in light blue. (A) 3D model of the bench, (B) zoomed in
image indicating the cracks detected on the bench.

3.3 Improved data management and hazard communication with cloud-based risk assessments
The concept of the App is summarized in Figure 4, where the green tick boxes indicate the
current completed phase of the App. Figure 5 indicates the home screen of the App and the rating
forms that can be used to capture data. Detailed reports can be shared via PowerBi and a weekly
or monthly summary of the residual risk of specific failure modes is also immediately available.
The proposed format for the reports can be seen in Figure 5. The App has been developed in such
a way to guide the user to assess specific failure modes in all the different working areas, while
also verifying compliance with operational standards. In this current phase of the App, paper-
based data capturing and the manual processing of this data into numerous Excel Sheets has been
eliminated. The cloud-based Cut Risk Assessment is still in the development phase.

Figure 4: The flow diagram summarizes the conceptualized input and outputs of the App. The green tick boxes
indicate which phases of the app development are completed.
 Figure 5 The App’s home screen (A), the rating form used for data capturing (B), the summary report (C), and the
detailed report shows the integration of the capturing and reporting of risk ratings on a single platform.

4 DISCUSSION

4.1 An integrated geotechnical risk-management approach

It was demonstrated that early signs of the slope failure from the case-study could be detected
from the accelerated slope movement from satellite data, and the elevation differences from drone
data. Due to the threshold settings used for calculating the elevation differences, the slope ap-
peared stable at 42 days prior to the failure, while the accelerated slope movement from satellite
data already detected the onset of the failure 53 days prior. The automatic crack detection algo-
rithms were only applied to drone footage after the slope had already failed and therefore served
as a proof-of-concept. The benefit in the use of InSAR is its ability to monitor large areas, acting
as a mine-wide, regional monitoring system. The InSAR data can also be imported into
Rocscience Slide 3 or RS3 for a stability analysis (Gervasi, 2020). The safety factor and proba-
bility of failure of the actual mined slopes can then be assessed routinely. It is realized that more
case-studies need to be investigated to establish trends in terms of slope accelerations on failed
and stable slopes. The author is currently working with SkyGeo on a further 10 case studies of
slope failures at Mine X. This work will include using the inverse velocity technique to the InSAR
data to predict time to failure. Current challenges of the crack detection algorithm and elevation
differences from aerial photogrammetry are the processing and interpretation time required when
analyzing large areas. The fact that the cracks and contours are highlighted in the model does
assist with the identification of unstable areas, but the end-user still needs to manually work
through the entire model. This is time-consuming, and the same outcome can be achieved by
scrutinizing drone videos, which is available immediately.
As part of the integrated geotechnical risk management approach, a georeferenced model will
be required, that can automatically display the cracks detected with AI, elevation differences de-
tected by drones, and accelerated slope movement detected with satellites on the same model.
This will indicate high risk areas to the Rock Engineer to deploy ground-based radars. The effec-
tiveness of ground-based radars as early warning systems has proven itself over time. The chal-
lenge with ground-based radars on a strip mine is to proactively identify the high-risk areas where
they must be deployed. While drones and satellites act as regional monitoring systems, their short-
comings in real time monitoring is complemented by the deployment of ground-based radars.
At the current developmental phase, the use of the mobile application to capture data during
geotechnical inspections will improve hazard communication on a weekly basis. The application
managed to simplify the capturing and displaying of geotechnical risks based on the currently
used monitoring processes, without compromising on the quality of reporting. The ideal would
be for information from the cut risk assessment, drone, radar and satellite data to be available live
on the App while a risk rating is being conducted in the pit. This could be achieved by georefer-
encing the user in the pit and ensuring only the information relevant to the user’s exact location
is displayed. Ultimately, the App should provide full integration between each step of the current
four tier risk management process.

5 CONCLUSION

The integration of radar, drone and satellite monitoring promised to be highly effective as an
early warning slope failure system. To fully utilize the data generated by various sources, an App
was conceptualized and is currently under development. The current phase of the App allows the
capturing and automatic displaying of geotechnical risks on a cloud-based platform.

6 ACKNOWLEDGEMENTS

I would like to thank my employer for allowing me to share this information. This project
would not have been successful without the help and support from SkyGeo, HCL Technologies,
RocketMine and the mine Survey Department.

7 REFERENCES

Alpha Software . (2020). Retrieved from How Mobile Apps Benefit Mining Companies:
alphasoftware.com/mining-apps
Carla, T., Farina, P., Intrieri, E., Ketzimen, H., & Casagli, N. (2018). Integration of ground-based
radar and satellite InSAR data for the analysis of an unexpected slope failure in an open-
pit mine. Engineering Geology, 39-52.
Carla, T., Intrieri, E., Raspini, F., Bardi, F., Farina, P., Ferretti, A., . . . Casagli, N. (2019).
Perspectives on the prediction of catastrophic slope failures from satellite InSAR. Nature,
1-9.
Dick, G., Eberhardt, E., Cebrejo-Lievano, A., Stead, D., & Rose, N. (2014). Developemnt of an
early-warning time-of-failure analysis methodology for open-pit mine slopes utilizing
ground-based slope stability radar monitoring data. Canadian geotechnical Journal, 515-
529.
Dorafshan, S., Thomas, R., & Maguire, M. (2018). SDNET2018:Anannotated image data set for
non-contact concrete crack detection using deep convolutionalneuralnetworks.
ELSEVIER, 1664-1668.
Gervasi, C. (2020, February 12). TRE ALTAMIRA. Retrieved from TRE ALTAMIRA:
https://site.tre-altamira.com/rocscience-3d-slope-stability-programs-now-integrated-
with-our-satellite-insar-data/
Hermanus, M. (2017). Mining Redesigned- innovation and technology needs for the future-a
South African perspective. The Journal of the South African Institute of Mining and
Metalurgy, 811-818.
Katuruza, M., & Birch, C. (2019). The use of unmanned aircraft system technology for highwall
mapping at Isibonelo Colliery, South Africa. The Journal of the Southern African Institute
of Mining and Metallurgy, 291-295.
Kerle, N., Nex, F., Gerke, M., Duarte, D., & Vetrivel, A. (2019). Drone Geologic Mapping of an
Active Sand and Gravel Quarry, Desoto County, Mississippi. International Journal of
Geo-Information, 1-14.
Skilton, M., & Hovsepian, F. (2018). The 4th Industrial Revolution Responding to the impact of
Artificial Intelegence on Business. Cham, Switzerland: Springer Nature.