Int. J. Human Factors and Ergonomics, Vol. 6, No.

2, 2019 103

The effect of human-robot interaction on trust,

situational awareness, and performance in drone
clearing operations

Thomas M. Schnieders* and Zhonglun Wang

Industrial and Manufacturing Systems Engineering Department,
Iowa State University,
Ames, Iowa, 50010, USA
Email: tom.m.schnieders@gmail.com
Email: zhonglun@iastate.edu
*Corresponding author

Richard T. Stone
Industrial and Manufacturing Systems Engineering Department,
Iowa State University,
Ames, Iowa, 50010, USA
and
Mechanical Engineering Department,
Iowa State University,
Ames, Iowa, USA
and
Story County Sheriff’s Office,
Story County, Nevada, Iowa, 50201, USA
Email: rstone@iastate.edu

Gary Backous
Story County Sheriff’s Office,
Story County, Nevada, Iowa, 50201, USA
Email: backous@storycountyiowa.gov

Erik Danford-Klein
Industrial and Manufacturing Systems Engineering Department,
Iowa State University,
Ames, Iowa, 50010, USA
Email: erikd@iastate.edu

Abstract: With advances in microcomputers, microprocessors, and battery

form factor, small drones are seeing a growing trend of deployment. Building
clearing operations, especially in active shooter scenarios, can be high risk
when officers need to clear a building on their own. This study analysed the use
of a small drone in a building clearing operation with a County Sheriff’s
Department to help mitigate the danger of single officer clearing operations.

Copyright © 2019 Inderscience Enterprises Ltd.

104 T.M. Schnieders et al.

Aspects of trust (human-robotic trust survey), situational awareness (SART),

mental demand (NASA-TLX), and performance (completion time and target
miss rate) were measured. Fourteen officers, age 22–63 with an average of
5.4 years law enforcement experience and three years of building sweeping
experience participated in the study. The results of the study indicate that the
use of a single drone during clearing operation can slow down the operation,
but accuracy and safety of clearing is enhanced. The use of the drone saw
non-significant changes in mental workload other than temporal demand,
increased situational awareness, a decrease in the number of targets missed, and
a moderate level of trust.

Keywords: drone; unmanned aerial vehicles; clearing operations; levels of

trust; situational awareness; mental demand; human-robotic interaction.

Reference to this paper should be made as follows: Schnieders, T.M.,

Wang, Z., Stone, R.T., Backous, G. and Danford-Klein, E. (2019) ‘The effect
of human-robot interaction on trust, situational awareness, and performance in
drone clearing operations’, Int. J. Human Factors and Ergonomics, Vol. 6,
No. 2, pp.103–123.

Biographical notes: Thomas M. Schnieders is a PhD candidate in the

Department of Industrial and Manufacturing, Systems Engineering at the Iowa
State University and Co-Founder of The ATHENA Lab. He received his MS in
Industrial Engineering and Human-Computer Interaction from the Iowa State
University in 2016 and BS in Mechanical Engineering in 2014. His current
research focuses on human performance engineering as it relates to exoskeleton
design, manufacturing, and testing. He has backgrounds in tool design, classic
ergonomic evaluation, tele-robotic, technology, biomechanical engineering,
drone technology, and virtual reality.

Zhonglun Wang is a PhD student in the Department of Industrial and

Manufacturing Systems Engineering at the Iowa State University. He received
his MS in Industrial and Manufacturing System Engineering in 2017 and BS in
Industrial and Manufacturing System Engineering in 2015 both from
the Iowa State University. His current research includes ergonomics hand tool
evaluation, ergonomics hand tool selection, and ergonomics evaluation. He has
backgrounds in tool evaluation, ergonomics evaluation, weight perception,
packaging design and virtual reality skill transfer.

Richard T. Stone is an Associate Professor in the Department of Industrial and

Manufacturing Systems Engineering at the Iowa State University. He received
his PhD in Industrial Engineering from the State University of New York at
Buffalo in 2008. He also has an MS in Information Technology, BS in
Management Information Systems as well as university certificates in Robotics
and Environmental Management Science. His current research focuses
primarily in the area of human performance engineering, particularly applied
biomedical, biomechanical and cognitive engineering. He focuses on the
human aspect of work across a wide range of domains (from welding to
surgical operations and many things in between). He works extensively in tool,
exoskeleton and telerobotics technologies, as well as classic ergonomic
evaluation.

Gary Backous has been with the Story County Sheriff’s Office since 2002
where he has served as a detention officer in 2002–2005, Deputy Sheriff in
2005–2012, tactical team member in 2008–2017, Policy Writing Specialist in
2012, Dive Team Supervisor in 2012–2017 and Support Services Sergeant in
2012–2017. He has a BA and MS in Criminal Justice and member of the
 The effect of human-robot interaction on trust, situational awareness 105

InterAgency Board where he serves as the National Sheriffs’ Association

representative on the Standards Coordination SubGroup. He also serves as a
subject matter expert and assistant to the Chair of the Department of Homeland
Security Emergency Services Sector Coordinating Council.

Erik Danford-Klein is an undergraduate student at the Iowa State University in

Industrial and Manufacturing Systems Engineering. He is an Undergraduate
Research Assistant in The ATHENA Lab, Human Factors Laboratory.
He researches ergonomics, biomedical engineering, paramilitary elements,
human factors and drones. Currently, he is working on projects with Story
County Sheriffs Department on several studies. He has background in
programming, experimental design, electronics, data analysis, creation of
custom sensors and process design.

1 Introduction

The Federal Aviation Administration defines drones as unmanned aircraft systems (UAS)
[also known as drones, unmanned aerial vehicle (UAV)], or ‘an aircraft without a human
pilot onboard’ (FAA, 2018). The earliest UAV can date back as early as WWI where the
US military used radio-controlled airplanes to attack German submarines (Keane and
Carr, 2013). Now, UAVs are classified into three primary categories:
1 tactical UAVs
2 strategic UAVs
3 micro/mini UAVs (Cavoukian, 2012).
Drones are performing various types of task such as location exploration (Sujit and
Beard, 2008), waterbody exploration (Sujit et al., 2009; Rathinam et al., 2007), mapping
(Tomoiagă and Morris, 2016; Remondino et al., 2011; Hirokawa et al., 2007), traffic
monitoring (Puri et al., 2007; Kanistras et al., 2013; Ro et al., 2007), reconnaissance
(Luotsinen et al., 2004; Miller, 1988), search and rescue (Tomic et al., 2012; Goodrich
et al., 2008; Doherty and Rudol, 2007), tactical situations (Sparrow, 2007; Hambling,
2010), and pipeline monitoring (Hausamann et al., 2005), among others.
With the advancement of microcomputers, drones are integrated with more functions
and getting lighter and smaller. Scientific communities are starting to add more features
to improve indoor flying ability (Mustafah et al., 2012; Mac et al., 2016; Ferrick et al.,
2012), explore applications for indoor flying, and perform tasks such as object tracking
(Chakrabarty et al., 2016), indoor search and rescue (Tomic et al., 2012), and surveillance
(Saska et al., 2012).
Room clearing operation is one of many tasks police officers need to perform on a
regular basis. Police officers need to physically rush into rooms to check for suspicious
activity and try to apprehend suspects. Close quarters combat room clearing operations
are considered a highly dangerous and complicated task (Greenwald, 2002). In
conventional military operations, it takes a small team of at least two soldiers to perform
the room clearing operation (Rose and Brandon, 2007) and the ‘point-man’ position of
the room clearing operation is considered the most dangerous since the doorway is the
focal point of the occupants in the room (Burton, 1998). However, due to the personnel
106 T.M. Schnieders et al.

shortage in the United States police force (Ali, 2017), especially in rural areas, police
officers are often forced to handle their case alone making them more vulnerable in room
clearing operations.
Since the early 2010s, militaries from multiple nations have been testing micro drones
for taking pictures and video in enemy territory (Whitwam, 2016). Studies regarding
using UAVs aiding police and military activities have been conducted in areas of
technology and feasibility (Daniel and Wietfeld, 2011; Ermacora et al., 2014), traffic
surveillance (Puri, 2005; Murphy and Cycon, 1999; Srinivasan et al., 2004), manhunting,
and scouting (Wright, 2005). However, none of the police related studies used the drone
for indoor close quarter combat situation.
With its small size, decent manoeuvrability, video streaming capability, and low
price, mini UAVs could be a good fit for police room clearing operations since it can
potentially replace the ‘point-man’ position in room clearing operations and help police
officers ‘see’ inside the room before entering. This can prevent police officers being
ambushed by perpetrators inside.
The aim of this study was to investigate if the use of a drone during room clearing
operations can:
1 increase situational awareness (measured by a modified SART-10D)
2 decrease the cognitive load of police officers (measured by a modified NASA-TLX)
3 increase efficiency of the room clearing operation (the ratio of targets found to the
number of targets presented)
4 measure the level of trust between police officers and the UAV used in the study
(human-robotic trust survey).

2 Materials and methods

The quadcopter chosen weights approximately 80 grams and measures 98 × 92.5

× 41 mm. This quadcopter has a maximum horizontal speed of 8 m/s. It is outfitted with a
5-megapixel camera with an 82.6-degree field of view and is capable of streaming 720p
videos at 30 frames per second.

2.1 Experimental procedure overview

Participants were required to fill out a pre-study survey and sign an informed consent
document before beginning the study. The pre-study survey asked participants their
agency experience, their level of training in special operations, their experience as a
law enforcement member, their experience with building clearing operations, their
experience operating a quadcopter, and their experience participating in tele-operational
experiments. Participants were comprised of law enforcement personnel above the age of
18 who could legally give consent and do not have conditions which would inhibit their
ability to perform a clearing operation. A total of 14 officers with experience in law
enforcement (mean = 5.4 years) and building sweeping (mean = 3 years) participated in
the study.
The 14 participants completed both the control and experimental conditions for a total
of 28 trials. The order of starting with the control or experimental condition was
 The effect of human-robot interaction on trust, situational awareness 107

randomised. Each participant started with one of the control scenarios (1, 2, 3, or 4) or
one of the experimental scenarios (5, 6, 7, or 8). The scenarios the participants completed
was also randomised.
Upon completion of a control condition and an experimental condition, each
participant filled out a post-study survey, a SART-10D rating sheet, a modified
NASA-TLX, and a trust in human-robotic interaction survey. Participants were then
debriefed and released from the study.

2.2 Experimental conditions and scenarios

The control condition had participants performing clearing operations on their own
and followed scenarios 1, 2, 3, and 4. The experimental condition consisted of one
on-the-ground officer and one drone operator. The drone operator was a trained member
of the research team who had extensive drone flying training and was an active member
of the County Sheriff’s Department. This group performed the clearing operations acting
with information provided by the drone operator. The drone would act as the initial point
of entry correspondent, flying and clearing each room prior to the officer’s entry. This
group followed scenarios 5, 6, 7, and 8.
Scenario 1 Single officer, no drone, no target
In this scenario, a single officer performed a clearing operation of a
one-story building. This operation had the officer enter a building followed
by the research team. They cleared the building and no target was present.
Scenario 2 Single officer, no drone, one active shooter
In this scenario, a single officer performed a clearing operation of a
one-story building. This operation had the officer enter a building followed
by the research team. They cleared the building and an active shooter will
be present in the fourth room.
Scenario 3 Single officer, no drone, one civilian
In this scenario, a single officer performed a clearing operation of a
one-story building. This operation had the officer enter a building followed
by the research team. They cleared the building and a civilian was present in
the sixth room.
Scenario 4 Single officer, no drone, one civilian
In this scenario, a single officer performed a clearing operation of a
one-story building. This operation had the officer enter a building followed
by the research team. They cleared the building and a civilian was present in
the third room.
Scenario 5 Single officer, with drone, no target
In this scenario, a single officer performed a clearing operation of a
one-story building. This operation had the officer enter a building followed
by the research team. The drone was required to clear each room prior to the
officer entering. They cleared the building and no target was present.
108 T.M. Schnieders et al.

Scenario 6 Single officer, with drone, one active shooter

In this scenario, a single officer performed performing a clearing operation
of a one-story building. This operation had the officer enter a building
followed by the research team. The drone was required to clear each room
prior to the officer entering. They cleared the building and an active shooter
was present in the fourth room.
Scenario 7 Single officer, with drone, one civilian
In this scenario, a single officer performed a clearing operation of a
one-story building. This operation had the officer enter a building followed
by the research team. The drone was required to clear each room prior to the
officer entering. They cleared the building and a civilian was present in the
fifth room.
Scenario 8 Single officers, with drone, one active shooter
In this scenario, a single officer performed a clearing operation of a
one-story building. This operation had the officer enter a building followed
by the research team. They cleared the building. The drone was required to
clear each room prior to the officer entering. A civilian was present in the
first room on the right.
For each of the scenarios, the officers completed a single floor circuit of a building with
seven rooms to be checked (see Figure 1).

Figure 1 Map of building

2.3 Drone operator script

The drone followed directions from the on-the-ground officer. When the officer
commanded the drone to move forward, the drone operator responded with ‘drone
proceeding forward’ and entered the room being cleared. The drone flies forward and
scans the room from left to right spending no more than 40 seconds in each room unless a
target has been located.
 The effect of human-robot interaction on trust, situational awareness 109

If a target is identified in the room, the drone remained in the room and the drone
operator reported to the officer with “Target spotted in room. Target on left/right side of
room.”
Officers then proceeded with regular room clearing operation. After target has been
identified as civilian or combatant by the officers, the drone left the room and the
operator reported ‘drone proceeding backward’ and awaited further commands. Building
clearing resumed until operation had been fully completed.
Following the experiment, participants were asked to complete a post-experiment
survey as well as an informal interview. The post-experiment survey included qualitative
questions such as “how much involvement did you make in the decision making?”,
“What was the processes you followed to complete the task?”, “Did you experience
frustration during the task?”, etc. Following this section of the post-experiment survey,
participants completed a SART 10D rating sheet, a NASA-TLX, and a trust in
human-robotic interaction survey.

2.4 SART-10D rating sheet

Officers had their situational awareness measured using the situational awareness
rating technique known as SART (Taylor, 1989). This questionnaire proposed by
Endsley (1988) measures three levels of situational awareness and is administered
post-experiment. SART looks at three domains:
1 attentional demand
2 attentional supply
3 understanding.
In the attentional demand domain, the questionnaire asks questions in three aspects:
the perceived instability, variability, and complexity of the situation. The response shows
the perceived likelihood of sudden change, the number of variables they need to pay
attention to, and how complicated the situation is perceived to be for the participants.
Four questions were asked in the attentional supply domain: participants’ readiness in the
situation, the amount of extra mental resources available for possible new variables, how
much attention the participants paid in the situation, and how divided their attention was
in the situation. In the understanding domain of the questionnaire, it asks about how
much information participants learned from the situation and what is their experience
level in a similar situation. Each question in the survey was on a 1 to 7 scale.

2.5 NASA-TLX
Workload is defined in the context of the NASA-TLX as the cost of the user to finish a
task. That is, how much mental, physical, or temporal demand was there to complete the
building clearing operation; how successful did the participant feel after completing the
task; how much effort or frustration did they fell during the task? The NASA-TLX
questionnaire gives an estimate for operators of their workload while they are performing
a task or immediately afterward (Hart, 2006). The questionnaire evaluates workload in
the aspects of mental demand, physical demand, temporal demand, frustration, effort, and
performance.
110 T.M. Schnieders et al.

The mental demand rating reflects how much the participants invested mentally
to finish the task such as thinking, searching, and calculating. The physical demand
rating shows how much physical effort participants spent to finish the task,
such as pushing, lifting, and/or controlling. The temporal demand score shows
the perceived pace of the task. Effort score of the NASA-TLX shows how much
mental and physical effort is being perceived to be devoted to finish the task.
Performance score reflect how participants perceive their performance, and frustration
level show how stressed, discouraged participant felt during the task. Unlike the weighted
scores used by Hart and Straveland (1988), a raw TLX score was used (Hart, 2006).
The survey has been used to evaluate a wide range of tasks over the years
(Cao et al., 2009), from simple tasks like driving screws into wood board to a complex
operation like piloting a spaceship.
Table 1 Trust in human-robotic interaction survey

Questions
The system is deceptive
The system behaves in an
underhanded manner
I am suspicious of the system’s
intent action, or output
I am wary of the system
The system’s action will have a
harmful or injurious outcome
I am confident in the system
The system provide security
The system has integrity
The system is dependable
The system is reliable
I can trust the system
I am familiar with the system
Purpose
If the participants believe the UAV is providing deceiving
information in the room clearing operation
If the participants think the UAV is working in secretive
manner in the room clearing operation
If the participants believe the UAV showing suspicious
intention in the room clearing operation
If the participant feels altered about the UAV in the room
clearing operation
If the participants perceive the UAV is a danger to
themselves or others in the room clearing operation
How much confidence does the participants have in the UAV
If the participant think the UAV provide security to
themselves in the room clearing
It evaluates how much integrity does the UAV perceived to
have in the room clearing operation
How dependable the UAV is being perceived in the room
clearing operation
How reliable the UAV is being perceived in the room
clearing operation
How much trust participants have been put in the UAV
during the room clearing
How familiar does the participants with the UAV used in the
room clearing

In this study, a modified NASA-TLX questionnaire were used to evaluate the

workload of the police officers in the room clearing operation with or without
the assistance of the drone. The raw TLX score was used to eliminate
pairwise comparison between the six subjective metrics. Research has been done to
indicate this shortened version of the NASA-TLX could increase experimental validity
(Bustamante, 2008).
 The effect of human-robot interaction on trust, situational awareness 111

2.6 Trust in human-robotic interaction

Trust is important in any partnership, without trust, police officers may never attempt to
use drones in their tasks, and the drone can become a liability in room clearing operation.
Officers responded to the questionnaire according to their feeling of trust to the drone
during the operation. In this questionnaire, the police officers answered 12 trust related
questions on a 1 to 7 scale. The question and its purpose can be found in Table 1.

2.7 Performance
Performance was measured by two factors: completion time and target miss rate.
Completion time is of interest because building clearing tasks are often time-sensitive.
Officers need to be able to efficiently move through a building and identify potential
threats. Completion time of clearing a single floor was timed to the nearest tenth of a
second. Target miss rate is another valuable indicator of performance. Ideally, all
potential threats should be identified in a building clearing operation. Missed potential
threats, especially if armed, can have detrimental effects to not only the officers, but
others in the vicinity. Target miss rate was measured as binary target identified or target
missed.

3 Results

Fourteen officers, age 22–63 (median = 26.5, SD = 10.8, 2 females, 12 males) with an
average of 5.4 years law enforcement experience and 3 years of building sweeping
experience participated in the study.
A paired-samples t-test was conducted to compare time to clear one floor in drone
assistance and no drone assistance conditions. There was a significant difference in time
taken for drone assistance (M = 215.6 s, SD = 50.79 s) and no drone assistance
(M = 109.5 s, SD = 26.05 s) conditions; t(13) = 11.713, p = 2.794e-08.

31 Hit/miss during scenarios

The control group and experimental group each ran a total of 14 scenarios. Of those
14 scenarios, it was possible for each group to encounter a civilian or active shooter nine
times. The control group ultimately missed finding the target a total of three times out of
nine times, yielding an average miss rate of 33.33%. The experimental group only missed
finding the target once out of nine times yielding an average miss rate of 11.11%.

3.2 SART-10D rating sheet

A paired-samples t-test was conducted to compare the results of each metric of the
SART. A detailed breakdown of the non-significant results from the SART-10D rating
sheet can be found in Table 2. There was no significant difference in the instability of the
situation between the control and experimental (p = 0.058). There was no significant
difference in the complexity of the situation between the control and experimental
(p = 0.45). There was no significant difference in the variability of the situation between
112 T.M. Schnieders et al.

the control and experimental (p = 0.27). There was no significant difference in alertness
between the control and experimental (p = 0.34). There was no significant difference in
concentration of attention between the control and experimental (p = 0.50). There was no
significant difference in division of attention between the control and experimental
(p = 0.055). There was no significant difference in spare mental capacity between the
control and experimental (p = 0.25). There was no significant difference in familiarity
with situation between the control and experimental (p = 0.17).

Figure 2 Time taken to complete floor (see online version for colours)

Figure 3 SART – information quality (see online version for colours)

A significant difference was found in information quality between the control (M = 5.29,
SD = 1.98) and experimental (M = 6.36, SD = 0.63); t(13) = 2.11; p = 0.027. The results
from each participant can be seen in Figure 3.
The SART-10D survey is an ordinal 1 to 7 scale. Information quality is described as
“how good is the information you have gained about the situation? Is the knowledge
communicated very useful (high) or is it a new situation (low)?”
 The effect of human-robot interaction on trust, situational awareness 113

Table 2 SART-10D non-significant results

SART-10D results
Instability of the situation
Control Experimental
P-value
Mean Standard deviation Mean Standard deviation
4.9 2.3 5.6 1.6 0.058
Complexity of the situation
Control Experimental
P-value
Mean Standard deviation Mean Standard deviation
4.4 2.4 4.36 2.3 0.45
Variability of the situation
Control Experimental
P-value
Mean Standard deviation Mean Standard deviation
4.6 2.3 4.36 2.3 0.27
Alertness
Control Experimental
P-value
Mean Standard deviation Mean Standard deviation
6.5 0.65 6.4 0.85 0.34
Concentration of attention
Control Experimental
P-value
Mean Standard deviation Mean Standard deviation
6.5 0.65 6.5 0.85 0.5
Division of attention
Control Experimental
P-value
Mean Standard deviation Mean Standard deviation
5.4 2.03 5.9 1.6 0.055
Spare mental capacity
Control Experimental
P-value
Mean Standard deviation Mean Standard deviation
5.2 1.97 4.9 1.99 0.25
Familiarity with situation
Control Experimental
P-value
Mean Standard deviation Mean Standard deviation
5.9 1.59 6.07 1.14 0.17
114 T.M. Schnieders et al.

3.3 NASA-TLX
A paired-samples t-test was conducted to compare the results of each metric
of the NASA-TLX. A detailed breakdown of the non-significant results from the
NASA-TLX survey can be found in Table 2. There was no significant difference
in mental demand between the control and experimental (p = 0.13). There was
no significant difference in physical demand between the control and experimental
(p = 0.195). There was no significant difference in performance between the control
and experimental (p = 0.48). There was no significant difference in effort between the
control and experimental (p = 0.08). There was no significant difference in frustration
between the control and experimental (p = 0.34).
Table 3 NASA-TLX non-significant results

NASA-TLX results
Mental demand
Control Experimental
P-value
Mean Standard deviation Mean Standard deviation
12.6 5.80 13.2 5.62 0.13
Physical demand
Control Experimental
P-value
Mean Standard deviation Mean Standard deviation
7.79 5.13 8.36 4.94 0.195
Performance
Control Experimental
P-value
Mean Standard deviation Mean Standard deviation
9.00 7.15 9.07 5.17 0.48
Effort
Control Experimental
P-value
Mean Standard deviation Mean Standard deviation
11.0 5.17 9.29 5.85 0.08
Frustration
Control Experimental
P-value
Mean Standard deviation Mean Standard deviation
4.57 4.22 4.79 3.96 0.34

A significant difference was found in temporal demand between the control (M = 10.14,
SD = 5.53) and experimental (M = 8.14, SD = 5.20); p = 0.04. The results from each
participant can be seen in Figure 4.
 The effect of human-robot interaction on trust, situational awareness 115

Figure 4 NASA-TLX – temporal demand (see online version for colours)

3.4 Trust in human-robotic interaction

The results of the post-study trust in human-robotic interaction survey for the
experimental group is summarised in Table 4. This survey was on a scale from 1 to 7
where a score of 1 indicates ‘not at all’ and a score of 7 indicates ‘extremely’.
For example, a score of 1.64 for the metric ‘the system is deceptive’ indicates the
participant believes the system (in this case, the drone) is not at all deceptive.
Table 4 Trust in human-robotic interaction survey results

Mean Standard deviation

The system is deceptive 1.64 1.08
The system behaves in an underhanded manner 1.50 0.94
I am suspicious of the system’s intent, action, or outputs 1.46 0.84
I am wary of the system 2.86 1.51
The system’s actions will have a harmful or injurious outcome 1.75 1.01
I am confident in the system 4.75 1.42
The system provides security 5.39 1.60
The system has integrity 4.79 1.67
The system is dependable 4.57 1.34
The system is reliable 4.29 1.27
I can trust the system 4.68 1.23
I am familiar with the system 2.50 1.29

4 Discussion

The aim of the study was to investigate if the use of a drone during room clearing
operations can increase the officer’s situational awareness, decrease cognitive load on
officers, increase their efficiency during room clearing operation, as well as accurately
116 T.M. Schnieders et al.

measure the level of trust between officers and the UAV used in the study. Efficiency for
this study is the ratio of targets found to the number of targets presented.
As the results section of this document suggest, the use of a drone during room
clearing exercises increased the information quality, increased cognitive demand in the
area of time pressure during the task, and the efficiency of room clearing was increased
with the use of drones. It is also very important to note that none of the areas assessed
were negatively impacted using drones.
Task performance is always a critical factor to consider when conducting studies in
the workplace, even more so when the task involves potentially life-threatening
scenarios. The results section of this paper shows that every officer took significantly
longer to complete their clearing task when utilising the drone. This was impacted by
several factors. Firstly, there is and delay between when the officer completes the room to
when they communicate the room is clear to when that information is processed by the
drone operator and reacted to. In addition, the drone itself slowed progress because
movement through doorways is a somewhat difficult operation due to the narrow size of
the doorframes. This is a common issue for both airborne robots as well as land-based
robots that has been well-documented in the literature (Stone et al., 2017).
The SART-10D survey, an indicator of situational awareness, indicated little
difference between the control and experimental groups. The only significantly different
metric was information quality. Many participants indicated the same score for the
SART-10D or significantly higher score for using the drones. Of the metrics surveyed,
information quality is arguably the most important in this type of scenario. Instability,
complexity, variability, alertness, concentration, division of attention, spare mental
capacity, and familiarity with situation are unlikely to drastically change due to the high
level of risk in building clearing operations. The only aspect that is different between the
control and experimental groups is the addition of the ‘second set of eyes’ via the drone.
The drone allowed for critical information to be relayed allowing for a higher success rate
in spotting potential targets. This is reflected qualitatively by the participants’ subjective
view of higher levels in information quality when using the drone, as well as
quantitatively with the higher ratio of found targets by the experimental group.
Performance in this study was also measured by the ratio of potential targets found.
During the study there were fewer instances of people and/or weapons missed while
searching with a drone than without one. In two instances, the control group (no drone
assistance) cleared rooms and did not see the armed individual inside, leaving the room
without acknowledging the subject. In one instance, the control group (no drone
assistance) did not see the unarmed individual inside and left the room without
acknowledging the subject. The experimental group (with drone assistance) did not miss
any armed individuals but did miss one unarmed individual. Under some conditions such
as searching for armed, suicidal, or barricaded subjects the increased time it takes to
conduct the operation is less problematic. As police tactics change from dynamic and
fast-pace room clearing operations to a more controlled and deliberate pace, technology
such as this could mitigate some of the risks officers have historically faced when
clearing rooms (Fenici et al., 2011).
The results of the trust in human-robotic interaction survey also illustrated a
somewhat high level of trust in the drone. The results indicated that while working with
the drone is unfamiliar (mean of 2.5/7), they are not wary of the system (mean of 2.85/7),
they believe it is not deceptive (mean of 1.64/7), and they do not believe the drone acts in
an underhanded manner (mean of 1.5/7). However, they believe the drone has moderate
 The effect of human-robot interaction on trust, situational awareness 117

levels of integrity (mean of 4.79/7), moderate confidence in the drone (mean of 4.75/7),
believe the drone offers security (mean of 5.39/7), and they can trust the drone (mean of
4.68/7). These are aspects that are critical for cooperative interaction to occur. It can be
easy to distrust technology when the officer has been trained in solo-engagement and
accepts the responsibility of putting their life in their own hands each time they enter a
room under those conditions. With training, experience, and continued improvement in
drone technology the level of trust in the interaction between officers and drones can be
easily improved as well.
What these findings begin to do for law enforcement practitioners is open the door to
identifying safer and more effective solutions for conducting high-risk search operations
in a way that increases their situational awareness and efficiency, while decreasing
cognitive load.
The current primary limitation to using drones in room clearing operations is the pace
at which the work is conducted. In active-killer situations a fast and effective response is
imperative and under the conditions of this study, drones may not be effective during the
initial response. This should not imply that drones will never be effective in such
circumstances, but currently this is a limitation that needs to be overcome.

5 Limitations

One limitation of this work could be using officers from only one county sheriff’s office
from the Midwest. Replication of this study could look at multiple sheriff’s offices from
multiple geographic distributions. Officers from other sheriff’s offices may have more or
less experience with room clearing operations and active shooter scenarios. The research
team does not think this limitation is large. All sheriff’s offices undergo extensive
training for room clearing operations as well as active shooter scenarios.
An additional limitation lies in the available drone technology. In preparing for this
study, the research team used multiple types and sizes of drones to determine the
expected success of each before ultimately choosing a relatively small drone that could
be flown indoors with a battery life long enough to complete the operation. In addition,
the drone was okayed for use by the county sheriff’s office as a viable solution for actual
use.
Another potential limitation was having a member of the research team be the drone
pilot and be in communication with the participant. The drone pilot and communicator
was also a member of the county sheriff’s office and has worked with the participants in
active duty as well as during training. The research team member also had extensive
training flying the chosen drone. The research team wanted to limit the effect of having a
new pilot fly the drone for each scenario. A future study could look at having participants
act as both drone pilot as well as the room clearing officer.

6 Future work

Future studies could reduce the amount of time the drones have to scan each room prior
to entry while assessing for the same factors as this study. This could help identify
reasonable operating parameters for balancing the need for trust, situational awareness,
118 T.M. Schnieders et al.

and performance in a variety of operational conditions, not just slow, deliberate searches.
Future work could also include force-on-force room clearing exercises where officers risk
being shot by Airsoft or Simunition rounds to better evaluate their trust in human-robotic
interaction. It can be assumed that it is easier to trust a technology when its failure has no
immediate consequences, but as soon as an officer must rely on it for their immediate
well-being in terms of whether they will experience a temporary painful stimulus could
change the level of trust officers have in the technology. Searching areas with staircases
has always been a danger for officers and tactical teams. Testing trust, situational
awareness, and performance on multilevel dwellings would also be valuable.
Future work on this topic should include the role of multiple and/or autonomous
drones. Having an autonomous drone digitally (remotely) tethered to an officer or tactical
team to provide ‘rear security’ or to hold a hallway while officers are searching rooms
would also be beneficial. The use of multiple drones may also decrease the amount of
time taken and temporal demand used during a building clearing operation. Additional
work with drone swarms will need to investigate orientation awareness, heading
direction, and relative location to be deployed successfully (Stone et al., 2017).
The authors of this work are currently finalising a paper on the use of multiple drones
compared to one drone in a similar room clearing operation.

Acknowledgements

Thank you to Story County Sheriff’s Office for their time and support in running this
study.

References
Ali, S.S. (2017) Police Shortage Hits Cities and Small Towns across the Country [online]
https://www.nbcnews.com/news/us-news/police-shortage-hits-cities-small-towns-across-
country-n734721 (accessed 6 February 2018).
Burton, M.P.S. (1998) Urban Operations, Untrained on Terrain, Pickle Partners Publishing.
Bustamante, E.A. (2008) ‘Measurement invariance of the NASA-TLX’, Proceedings of the
Human Factors and Ergonomics Society Annual Meeting, Vol. 52, No. 19, pp.1522–1526,
DOI: 10.1177/154193120805201946.
Cao, A., Chintamani, K.K., Pandya, A.K. and Ellis, R.D. (2009) ‘NASA TLX: software for
assessing subjective mental workload’, Behavior Research Methods, Vol. 41, No. 1,
pp.113–117.
Cavoukian, A. (2012) Privacy and Drones: Unmanned Aerial Vehicles, pp.1–30, Information and
Privacy Commissioner of Ontario, Ontario, Canada.
Chakrabarty, A., Morris, R., Bouyssounouse, X. and Hunt, R. (2016) ‘Autonomous indoor object
tracking with the Parrot AR. Drone’, in 2016 International Conference on Unmanned Aircraft
Systems (ICUAS), IEEE, pp.25–30.
Daniel, K. and Wietfeld, C. (2011) Using Public Network Infrastructures for UAV Remote Sensing
in Civilian Security Operations, Dortmund Univ., Germany FR.
Doherty, P. and Rudol, P. (2007) ‘A UAV search and rescue scenario with human body detection
and geolocalization’, in Australasian Joint Conference on Artificial Intelligence, Springer,
Berlin, Heidelberg, pp.1–13.
 The effect of human-robot interaction on trust, situational awareness 119

Endsley, M.R. (1988) ‘Design and evaluation for situational awareness enhancement’, Proceedings
of the Human Factors Society 32nd Annual Meeting, Human Factors and Ergonomics Society,
Santa Monica, CA, pp.97–101.
Ermacora, G., Toma, A., Rosa, S., Bona, B., Chiaberge, M., Silvagni, M. and Antonini, R. (2014)
‘A cloud based service for management and planning of autonomous UAV missions in smart
city scenarios’, in International Workshop on Modelling and Simulation for Autonomous
Systems, Springer, Cham, pp.20–26.
FAA (2018) Unmanned Aircraft Systems [online] https://www.faa.gov/uas/ (accessed 9 February
2018).
Fenici, R., Brisinda, D. and Sorbo, A.R. (2001) ‘Methods for real-time assessment of operational
stress during realistic police tactical training’, in Kitaeff, J. (Ed.): Handbook of Police
Psychology, pp.295–319, Routledge, New York.
Ferrick, A., Fish, J., Venator, E. and Lee, G.S. (2012) ‘UAV obstacle avoidance using image
processing techniques’, in 2012 IEEE International Conference on Technologies for Practical
Robot Applications (TePRA), IEEE, pp.73–78.
Goodrich, M.A., Morse, B.S., Gerhardt, D., Cooper, J.L., Quigley, M., Adams, J.A. and
Humphrey, C. (2008) ‘Supporting wilderness search and rescue using a camera-equipped mini
UAV’, Journal of Field Robotics, Vol. 25, Nos. 1–2, pp.89–110.
Greenwald, T.W. (2002) An Analysis of Auditory Cues for Inclusion in a Virtual Close Quarters
Combat Room Clearing Operation, Naval Postgraduate School Monterey CA.
Hambling, D. (2010) ‘Air force completes killer micro-drone project’, Wired Danger Room.
Hart, S.G. (2006) ‘NASA-task load index (NASA-TLX); 20 years later’, in Proceedings of the
Human Factors and Ergonomics Society Annual Meeting, Sage Publications, Los Angeles,
CA, Vol. 50, No. 9, pp.904–908.
Hart, S.G. and Staveland, L.E. (1988) ‘Development of NASA-TLX (task load index): results of
empirical and theoretical research’, Advances in Psychology Human Mental Workload,
pp.139–183, DOI: 10.1016/s0166-4115(08)62386-9.
Hausamann, D., Zirnig, W., Schreier, G. and Strobl, P. (2005) ‘Monitoring of gas pipelines – a civil
UAV application’, Aircraft Engineering and Aerospace Technology, Vol. 77, No. 5,
pp.352–360.
Hirokawa, R., Kubo, D., Suzuki, S., Meguro, J.I. and Suzuki, T. (2007) ‘A small UAV for
immediate hazard map generation’, in AIAA Infotech@ Aerospace 2007 Conference and
Exhibit, p.2725.
Kanistras, K., Martins, G., Rutherford, M.J. and Valavanis, K.P. (2013) ‘A survey of unmanned
aerial vehicles (UAVs) for traffic monitoring’, in 2013 International Conference on
Unmanned Aircraft Systems (ICUAS), IEEE, pp.221–234.
Keane, J.F. and Carr, S.S. (2013) ‘A brief history of early unmanned aircraft’, Johns Hopkins APL
Technical Digest, Vol. 32, No. 3, pp.558–571.
Luotsinen, L.J., Gonzalez, A.J. and Bölöni, L. (2004) Collaborative UAV Exploration of Hostile
Environments, University of Central Florida Orlando, Dept. of Electrical Engineering.
Mac, T.T., Copot, C., Hernandez, A. and De Keyser, R. (2016) ‘Improved potential field method
for unknown obstacle avoidance using UAV in indoor environment’, in 2016 IEEE 14th
International Symposium on Applied Machine Intelligence and Informatics (SAMI), IEEE,
pp.345–350.
Miller, J.B. (1988) Unmanned Air Vehicles-Real Time Intelligence without the Risk, Naval
Postgraduate School Monterey CA.
Murphy, D.W. and Cycon, J. (1999) ‘Applications for mini VTOL UAV for law enforcement’, in
Sensors, C3I, Information, and Training Technologies for Law Enforcement, International
Society for Optics and Photonics, Vol. 3577, pp.35–44.
Mustafah, Y.M., Azman, A.W. and Akbar, F. (2012) ‘Indoor UAV positioning using stereo vision
sensor’, Procedia Engineering, Vol. 41, pp.575–579.
120 T.M. Schnieders et al.

Puri, A. (2005) A Survey of Unmanned Aerial Vehicles (UAV) for Traffic Surveillance, Department
of Computer Science and Engineering, University of South Florida, pp.1–29.
Puri, A., Valavanis, K.P. and Kontitsis, M. (2007) ‘Statistical profile generation for traffic
monitoring using real-time UAV based video data’, in Mediterranean Conference on Control
& Automation, MED’07, IEEE, pp.1–6.
Rathinam, S., Almeida, P., Kim, Z., Jackson, S., Tinka, A., Grossman, W. and Sengupta, R. (2007)
‘Autonomous searching and tracking of a river using an UAV’, in American Control
Conference, ACC’07, IEEE, pp.359–364.
Remondino, F., Barazzetti, L., Nex, F., Scaioni, M. and Sarazzi, D. (2011) ‘UAV photogrammetry
for mapping and 3D modelling – current status and future perspectives’, International
Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol. 38,
No. 1, p.C22.
Ro, K., Oh, J.S. and Dong, L. (2007) ‘Lessons learned: application of small UAV for urban
highway traffic monitoring’, in 45th AIAA Aerospace Sciences Meeting and Exhibit, p.596.
Rose, A. and Brandon, T. (2007) ‘Training for military operations or urbanized terrain’, Armor,
Vol. 116, No. 3, p.26.
Saska, M., Krajnik, T. and Pfeucil, L. (2012) ‘Cooperative μUAV-UGV autonomous indoor
surveillance’, in 2012 9th International Multi-Conference on Systems, Signals and Devices
(SSD), IEEE, pp.1–6.
Sparrow, R. (2007) ‘Killer robots’, Journal of Applied Philosophy, Vol. 24, No. 1, pp.62–77.
Srinivasan, S., Latchman, H., Shea, J., Wong, T. and McNair, J. (2004) ‘Airborne traffic
surveillance systems: video surveillance of highway traffic’, in Proceedings of the ACM 2nd
International Workshop on Video Surveillance & Sensor Networks, ACM, pp.131–135.
Stone, R.T., Schnieders, T.M. and Zhong, P. (2017) ‘Way finding in human-robot collaborated
exploration: orientation awareness and its enhancement’, Proceedings of the Human Factors
and Ergonomics Society Annual Meeting, Austin, TX, 9–13 October 2017, Vol. 61,
pp.1257–1261, DOI: 10.1177/1541931213601795.
Sujit, P.B. and Beard, R. (2008) ‘Multiple UAV exploration of an unknown region’, Annals of
Mathematics and Artificial Intelligence, Vol. 52, Nos. 2–4, pp.335–366.
Sujit, P.B., Sousa, J. and Pereira, F.L. (2009) ‘UAV and AUVs coordination for ocean exploration’,
in Oceans 2009-Europe, IEEE, pp.1–7.
Taylor, R.M. (1989) ‘Situational awareness rating technique (SART): the development of a tool for
aircrew systems design’, Proceedings of the NATO Advisory Group for Aerospace Research
and Development (AGARD) Situational Awareness in Aerospace Operations Symposium
(AGARD-CP-478), October, p.17.
Tomic, T., Schmid, K., Lutz, P., Domel, A., Kassecker, M., Mair, E. and Burschka, D. (2012)
‘Toward a fully autonomous UAV: research platform for indoor and outdoor urban search and
rescue’, IEEE Robotics & Automation Magazine, Vol. 19, No. 3, pp.46–56.
Tomoiagă, C. and Morris, T. (2016) A Study for Autonomous Indoor Mapping with AR Drone
[online] https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=9&cad=rja&
uact=8&ved=2ahUKEwijmsTmmungAhUDZawKHam1A7gQFjAIegQICRAC&url=http%
3A%2F%2Fstudentnet.cs.manchester.ac.uk%2Fresources%2Flibrary%2F3rd-year-
projects%2F2016%2Fciprian.tomoiaga.pdf&usg=AOvVaw3JmlC6FDFf_K10MG5iRwy6.
Whitwam, R. (2016) US Army will put Nano-Drones in Soldiers’ Pockets by 2018 [online]
https://www.geek.com/geek-cetera/us-army-will-put-nano-dronesin-soldiers-pockets-by-2018-
1651622/ (accessed 25 January 2018).
Wright, S.E. (2005) ‘UAVs in community police work’, AIAA Infotechs., Aerospace, Arlington.
 The effect of human-robot interaction on trust, situational awareness 121

Appendix A

Table A1 SART 10-D rating sheet

Instability of situation
How changeable is the situation? Is the situation highly unstable and likely to
change suddenly (high), or is it very stable and straightforward (low)?
Low High
Complexity of situation
How complicated is the situation? Is it complex with many interrelated
components (high) or is it simple and straightforward (low)?
Low High
Variability of situation
How many variables are changing in the situation? Are there a large number of
factors varying (high) or are there very few variables changing (low)?
Low High
Alertness
How alert are you in the situation? Are you alert and ready for activity (high) or
do you have a low degree of alertness (low)?
Low High
Concentration of attention
How much are you concentrating on the situation? Are you bringing all your
thoughts to bear (high) or is your attention elsewhere (low)?
Low High
Division of attention
How much is your attention divided in the situation? Are you concentrating on
many aspects of the situation (high) or focused on only one (low)?
Low High
Spare mental capacity
How much mental capacity do you have to spare in the situation? Do you have
sufficient mental capacity to attend to many variables (high) or nothing to spare
at all (low)
Low High
Information quality
How good is the information you have gained about the situation? Is the
knowledge communicated very useful (high) or is it a new situation (low)?
Low High
Familiarity with situation
How familiar are you with the situation? Do you have a great deal of relevant
experience (high) or is it a new situation (low)?
Low High
122 T.M. Schnieders et al.

Appendix B

Hart and Staveland’s NASA task load index (TLX) method assesses work load on five
seven-point scales. Increments of high, medium, and low estimates for each point result
in 21 gradations on the scale.
Table A2 NASA task load index

Mental demand How mentally demanding was the task?

Very low Very high

Physical demand How physically demanding was the task?

Very low Very high

Temporal demand How hurried or rushed was the pace of the
task

Very low Very high

Performance How successful were you in
accomplishing what you were asked to do?

Perfect Failure
Effort How hard did you have to work to
accomplish your level of performance

Very low Very high

Frustration How insecure, discouraged, irritated,
stressed, and annoyed were you?

Very low Very high

 The effect of human-robot interaction on trust, situational awareness 123

Appendix C

Below is a list of statements for evaluating trust between people and automation. There
are several scales for you to rate intensity of your feeling of trust, or your impression of
the quadcopter during the operation.
Please mark an ‘x’ on each line at the point which best describes your feeling or your
impression.
Table A3 Trust in human-robotic interaction

1 The system is deceptive

2 The system behaves in an underhanded manner

3 I am suspicious of the system’s intent, action, or outputs

4 I am wary of the system

5 The system’s actions will have a harmful or injurious outcome

6 I am confident in the system

7 The system provides security

8 The system has integrity

9 The system is dependable

10 The system is reliable

11 I can trust the system

12 I am familiar with the system

Notes: ‘Not at all’ = 1 and ‘extremely’ = 7.