J Intell Robot Syst (2011) 64:7795

DOI 10.1007/s10846-010-9514-8
Robot-Assisted Bridge Inspection
Robin R. Murphy Eric Steimle Michael Hall
Michael Lindemuth David Trejo Stefan Hurlebaus
Zenon Medina-Cetina Daryl Slocum
Received: 18 May 2010 / Accepted: 29 November 2010 / Published online: 18 January 2011
Springer Science+Business Media B.V. 2011
Abstract The Center for Robot-Assisted Search and Rescue (CRASAR) de-
ployed a customized AEOS man-portable unmanned surface vehicle and two
commercially available underwater vehicles (the autonomous YSI EcoMapper and
This work was supported in part by grants from the Texas Transportation Institute, ONR
(N000140610775 Coordinated Operation of Humans, Agents, and Unmanned Vehicles for
Littoral Warfare), and NSF (EIA-022440 R4: Robots for Rescue, Research and Response).
R. R. Murphy (B) S. Hurlebaus Z. Medina-Cetina
Texas A&M, College Station, TX 77843-3112, USA
e-mail: murphy@cse.tamu.edu
S. Hurlebaus
e-mail: shurlebaus@cse.tamu.edu
Z. Medina-Cetina
e-mail: zmedina@cse.tamu.edu
E. Steimle M. Hall
AEOS, LLC, St. Petersburg, FL 33711, USA
E. Steimle
e-mail: esteimle@aeosciences.com
M. Hall
e-mail: mhall@aeosciences.com
M. Lindemuth
University of South Florida, St. Petersburg, FL 33701, USA
e-mail: lindemuth@gmail.com
D. Trejo
Oregon State University, Corvallis, OR 97331, USA
e-mail: david.trejo@oregonstate.edu
D. Slocum
YSI Inc., San Diego, CA 92121, USA
e-mail: dslocum@ysi.com
78 J Intell Robot Syst (2011) 64:7795
the tethered VideoRay) for inspection of the Rollover Pass bridge in the Bolivar
peninsula of Texas in the aftermath of Hurricane Ike. A preliminary domain analysis
with the vehicles identified key tasks in subsurface bridge inspection (mapping of
the debris field and inspecting the bridge footings for scour), control challenges
(navigation under loss of GPS, underwater obstacle avoidance, and stable positioning
in high currents without GPS), possible improvements to human-robot interaction
(having additional display units so that mission specialists can view and operate on
imagery independently of the operator control unit, incorporating 2-way audio to
allow operator and field personnel to communicate while launching or recovering the
vehicle, and increased state sensing for reliability), and discussed the cooperative use
of surface, underwater, and aerial vehicles. The article posits seven milestones in the
development of a fully functional UMV for bridge inspection: standardize mission
payloads, add health monitoring, improve teleoperation through better human-robot
interaction, add 3D obstacle avoidance, improve station-keeping, handle large data
sets, and support cooperative sensing.
Keywords Security and rescue robots Underwater unmanned vehicles
1 Introduction
Unmanned marine vehicles (UMV) are being used and considered for post-disaster
bridge inspection and other activities in littoral regions. While hurricanes are as-
sociated with large scale search and rescue activities on land, inspection of coastal
littoral structures are also important. Bridges must be inspected, as they are needed
for responders and recovery workers to have access to affected areas, and because
they influence the general recovery of the area. Seawalls, levees, or dikes may
be compromised and create a secondary disaster such as seen in New Orleans at
Hurricane Katrina. Channels must be restored and docks repaired as part of the
economic recovery and restoration of shipping. It the disaster is man-made, either
terrorism, an accident, or structural failure, forensic structural data is needed to
help agencies such as the Federal Bureau of Investigation (FBI) and the National
Transportation Safety Board (NTSB). Likewise, bodies may need to be recovered.
The term UMV encompasses three types of vehicles. Unmanned surface ve-
hicles (USV), often called autonomous surface vehicles (ASV), are boat or jet-
ski variants that work on the water surface or are semi-submersible. Unmanned
underwater vehicles (UUV) generally refer to tether-less platforms which either
execute automated paths or communicate with the surface through low baud-
rate acoustic modems. The term remotely-operated vehicle (ROV) designates a
UUV that is controlled through a tether, enabling real-time control and perception
normally prevented by acoustic modems.
This article, an extension of [1], describes the use of all three types of UMVs
for post-disaster bridge inspection in the aftermath of Hurricane Ike. Hurricane Ike
was a Category four storm that struck Galveston, Texas on September 13, 2008. The
Rollover Pass bridge on the adjacent Bolivar Peninsula, a major artery to the area,
was severely damaged. In December, 2008, the Texas Transportation Institute (TTI)
and the Texas Department of Transportation permitted CRASAR to test UMVs
for bridge inspection. In particular, the agencies desired an evaluation of how well
J Intell Robot Syst (2011) 64:7795 79
the Sea-RAI unmanned surface vehicle (shown in Fig. 1) could inspect the bridge
footings for scour (erosion and separation from the bottom) and map the debris field
around the bridge (i.e., is there debris likely to be pushed into the substructure and
damage it?).
As described in [2], the inspection of the underwater portion of a bridge, called the
substructure, is currently done manually with divers who must work in high currents,
low visibility, and debris to physically see and touch damage or scour (erosion under
the foot of the bridge). Manual inspection puts the divers at risk and is not efficient,
for example due to the tidal currents at the Rollover Pass bridge, divers could
only work for 15 min at the change of each tide. At the 2007 I-35 bridge collapse
in Minneapolis, high currents and cramped, confined spaces were cited as the
justification for bringing in a team of 17 Navy divers [3]. Those cluttered conditions
were so difficult that the FBI was unable to successfully use a unspecified miniature
UMV [4].
Post-disaster inspection is an excellent domain for UMVs, as it scores high on
Leibholz criteria for deploying a UMV [5], paraphrased and re-ordered below:
Of fers economic advantages. Post-disaster inspection has two major economics
impacts. First, it reduces the need for expensive manual inspection and staffing,
and second, it speeds economic recovery of an affected region.
Reduces human risk. As seen at Rollover Pass [2] and Minneapolis [3], a UMV
would reduce risk to dive teams.
Requires humanlike abilities. In this case, the maneuverability and perceptual
abilities of a diver are needed to adapt to conditions and reach all areas of the
structure; bridge inspection is not something easily done with a large vessel at a
standoff distance.
Is a repetitive or highly boring task requiring continued alertness. Inspection is
an intrinsically boring task, making it easy for a manual diver to miss important
details, even if not distracted by personal safety concerns. A UMV can provide
provable coverage of the entire area and structure.
Fig. 1 The sea-RAI operating
around the Rollover Pass
bridge
80 J Intell Robot Syst (2011) 64:7795
While post-disaster bridge inspection by UMVs is clearly desirable, it is more
demanding than routine bridge inspection. Ageev et al. [6] offer perhaps the earliest
description of the routine, unfavorable conditions around structures, most notably
the difficulties in controlling changes velocity and direction to support object in-
vestigation. However, experiences after Hurricane Wilma [7], as well as the effort
reported in this article, add three additional challenges:
Extreme environmental conditions, where tides, surge, flooding, currents, wind,
and water/wind disturbances near structures exacerbate control challenges;
Presence of debris and changes in topology, where a priori maps of channels
and structures may be rendered obsolete by damage, debris from upstream
devastation distributed through out the area of interest, and new channels and
shallows created by flooding and wave action, thus necessitating shallow water
USVs, UUVs with obstacle avoidance, and ROVs with tether management; and
Unfavorable deployment conditions, where boat ramps for launching may be
damaged or unavailable and it may not be safe to stand on the bridge to lower or
control devices, suggesting the need for lightweight, easily portable vehicles.
This paper is an extension of [1] and [8], providing a deeper review of the related
work on UMVs for littoral inspection, more details of the CRASAR deployment,
and contrasts challenges in vehicle control, multi-robot coordination, human-robot
interaction, and sensing with the state of the art. The article is expected to contribute
a domain theory for post-disaster bridge inspection for use by the larger robotics and
response communities and findings for the research community on control challenges,
human-robot interaction, multi-robot cooperation, and uncertain sensor data. The
domain theory and findings are expected to be applicable to UMV surveys of damage
from natural events (e.g., hurricanes and flooding) but also to man-made incidents
such as the 2007 I-35W bridge collapse in Minneapolis and general operations in
cluttered littoral regions.
The remainder of the article is organized as follows. Section 2 summarizes the
related work in UMVs for disaster response, summarizing the deployments after
Hurricane Wilma (2005) and the I-35 bridge collapse in Minneapolis (2007) and the
relationship to other littoral applications. Section 3 describes the deployments, team,
and conditions, the three unmanned marine vehicles used, and the results of the
inspection showing no damage, scour, or debris. Section 4 compares the suitability
and performance of the three UMVs, while Section 5 reports findings in control
challenges, human-robot interaction, multi-robot cooperation, and uncertain data.
The article concludes with a roadmap for improving UMVs for bridge inspection.
2 Related Work
Unmanned marine vehicles have not been widely used for disaster response [9],
though search and rescue has been cited as a motivation for UMV research in [10].
CRASAR is a center at Texas A&M that promotes robots for emergency response
and sends out volunteer teams of scientists and robot manufacturers to disasters
to assist and collect data, similar to tornado storm chasers and Doctors Without
Borders. It has experimented twice with post-disaster uses at the Marco Island bridge
after Hurricane Wilma in 2005 [7] and the Rollover Pass bridge after Hurricane Ike
J Intell Robot Syst (2011) 64:7795 81
in 2008 (this article). The FBI deployed an unspecified UUV and ROV to the 2007
I-35 bridge collapse in Minneapolis, Minnesota.
2.1 Post Hurricane Wilma
CRASAR conducted a survey of the Marco Island Yacht Club docks and seawalls
and near-by SR951 Marco Island bridge in the wake of Hurricane Wilma with an
AEOS man-portable surface vehicle; the reader is directed to [7] for a comprehensive
discussion. The survey was conducted on Oct. 27, 2005, after the Wilmas landfall
14 km away in Cape Romano, Florida. Despite being a Category five hurricane,
Wilma generated very little damage in the area. The USV used a DIDSON acoustic
camera, an imaging sonar that does not require the vessel to be moving, and found
scour under the pilings of the docks and railings from the damaged fishing pier
adjacent to the bridge.
The Wilma deployment identified USV localization and control and human-robot
interaction as major research issues. Findings from the Wilma deployment led to the
Sea-RAI project which refined the AEOS platform, added navigational autonomy,
and improved human-robot interfaces [11]. The Wilma effort also explored USV-
UAV cooperation [7, 12].
The deployment to Rollover Pass differed in that it was conducted on behalf of
the Texas Department of Transportation and the team included two civil engineers;
it was specifically tasked to inspect the substructure for scour, map the upstream
debris field, and to compare USV, UUV, and ROV suitability; it did not involve
UAVs; and an improved USV and interface was used.
2.2 I-35 Bridge Collapse
The FBI Underwater Search and Evidence Recovery Team (USERT) reportedly
sent two UUVs to assist with the recovery and forensic documentation of the Aug. 1,
2007, collapse of the I-35 bridge in Minneapolis, Minnesota. The news report [4] did
not specify the models, describing the larger one as having a manipulator (possibly a
SeaBotix or similar platform) and a smaller one being the size of shoe box (possibly a
VideoRay). No indication was given as to whether the UUVs were tethered, though
it is likely. The larger UUV could not be successfully used because it was too big
to maneuver amid the unstable, twisted bridge wreckage and vehicles in the murky
water. No follow up reports on whether the smaller UUV was successful could be
found, though the report implied some doubt over the utility of the smaller UUV
due to its smaller thrusters and high currents.
The I-35 bridge collapse deployment confirms the impact of extreme environ-
mental conditions the presence of debris and changes in topology on UMVs. It
also suggests that development of micro-UMVs such as [13] may not be sufficiently
powerful to be of use.
2.3 Other Littoral Uses
UMVs have been used in littoral regions for commercial, security, and research
applications. They have been used by the oil and gas industry for decades [14].
Castelin and Bernstein give an overview of military operations in littoral regions
82 J Intell Robot Syst (2011) 64:7795
in [15]. Discussion of specific security applications include general littoral mapping
and demining [16, 17], homeland security for port protection [18], and amphibious
troop deployment [19]. They have also be proposed for chemical plume tracking [20]
and inspection of underwater wrecks [21]. UMVs have begun assisting with research
efforts, including mapping the Great Barrier Reef [22], environmental mapping [23],
sampling sea-air interfaces [24] and sediments [25]. With the notable exception of
marina mapping [26, 27], these littoral operations do not take place in close proximity
to structures or constricted areas and thus provide little insight for post-disaster
inspection. Unlike port and littoral security [28], the submerged threats in post-
disaster inspection are debris, not other vehicles.
3 Mission
With permission from the Texas Department of Transportation (TxDOT) and
support from the Texas Transportation Institute (TTI), the CRASAR team was in
the field from Dec. 1719, 2008, approximately three months after Hurricane Ike to
inspect the Rollover Pass bridge (Fig. 2). The Rollover Pass Bridge is a two-lane with
median concrete span on Texas Highway 87 connecting a 0.06 km man-made channel
between the Gulf of Mexico, Galveston Bay, and the intercoastal waterway. The pass
is subject to intense tides and turbidity. The bridge, which had been heavily damaged,
had been inspected with divers and one of the three lanes had been repaired and
major debris removed to prevent further damage. However, the bridge provided
a good test case for evaluating UMVs for post-disaster inspection because of the
extreme environmental conditions (tides, currents), difficult deployment conditions
(UMVs had to be hand carried to the water), and there was a possibility that
manual methods had missed scour or debris. In addition, Rollover Pass was far more
cluttered than the Marco Island bridge; the Marco Island bridge has a span of 1.4 km
while Rollover Pass spans only 0.06 km.
The deployment had a primary and secondary mission for the agencies as well as
a general science mission for the robotics community. The primary mission was to
evaluate the utility and performance of the Sea-RAI USV for two tasks: inspection
of the bridge substructure and mapping of the debris f ield. The metric for success
Fig. 2 Rollover pass bridge
after Hurricane Ike (from
Wikimedia Commons, public
domain under PD-USGOV)
J Intell Robot Syst (2011) 64:7795 83
was the acquisition of comprehensible underwater imagery deemed of use to the
structural community by the civil engineers on the team. he Sea-RAI was able to
meet both objectives as described in Section 3.3. The secondary mission for the
agencies to compare the types of UMVs for post-disaster bridge inspection and
those results are reported in Section 4. The general science mission was to make
ethnographic observations to determine how a USV might be used in the future and
experimentation with the two UUVs; these findings are reported in the next section.
3.1 Deployments, Team, and Conditions
The team was comprised of roboticists, civil engineers, and responders and three un-
manned marine vehicles. The six person team consisted of two roboticists (Murphy,
Steimle), both of whom had participated in the Hurricane Wilma deployment, two
civil engineering professors (Hurlebaus, Medina-Cetina), a responder fromthe Texas
Engineering Extension Service (May), which is the state agency for urban search and
rescue, and a graduate student (Lindemuth). In addition, YSI sent two experts (Hall,
now with AEOS, and Slocum) to use the Ecomapper UUV on Dec. 18. The primary
vehicle was the Sea-RAI unmanned surface vehicle, with a VideoRay tethered ROV
and YSI UUV Ecomapper as secondary vehicles for comparision. The team focused
on the USV because as noted in [7], surface vehicles have important advantages over
underwater vehicles: they can be more accurately controlled and localized through
GPS, they can carry a larger payload, and they can continuously broadcast imagery
data to observers in real-time.
The team arrived on Dec. 17, 2008, and surveyed the site, which was still in
disarray, for access, possible problems, etc. The site was foggy and cloudy for most
of the duration, with bright sunshine only as the team departed on the afternoon of
Dec. 19. Temperatures were 1013

C. A launch point was selected on a sandy spit

on the interior side of the bridge about 0.5 km away and all operations we conducted
within line of sight. The spit had better access than lowering the vehicles over the
metal seawall. The cars parked about 10 m from the water and the transport van got
stuck in the sand and had to be pulled out, reinforcing the unfavorable conditions for
larger UMVs requiring trailers. A boat ramp was available about 1.5 km away and a
chase boat was launched from there but could not navigate the shallows in the foggy
conditions. No structures or electricity was available. There were some fishermen
present but generally this did not pose a problem. The team worked for two days,
spending two nights in a nearby beach house that had been rebuilt.
Deployments had to be scheduled for the change of tides to minimize the impact of
the currents. The USV was deployed twice on Dec. 18, initially with a safety line until
it was established that the motors were sufficient for the current. The YSI Ecomapper
was deployed five times concurrently with the USV and the VideoRay was deployed
once. The USV was deployed twice on Dec. 19th to ensure repeatability and to get
practice deploying the vehicle.
3.2 Unmanned Marine Vehicles
The three vehicles used at Rollover Pass are shown in Fig. 3 and represent each of
the three types of UMV. Each are commercially available.
84 J Intell Robot Syst (2011) 64:7795
a b
c
Fig. 3 Three UMVs used: a Sea-RAI USV, b VideoRay tethered ROV, and c YSI Ecomapper
autonomous UUV
The Sea-RAI unmanned surface vehicle shown in more in Fig. 4 is a custom
platform built by AEOS based on two 6ft catamaran hulls. It is similar to Charlie [29]
and smaller than ACES [30] but more stable than the kayak-based SCOUT [31]. It
is capable of autonomous waypoint navigation and supports teleoperation through
a wireless F4W 802.11 mobile ad hoc network. It is an adaptation of the AEOS
platform built for environmental mapping [23]. The Sea-RAI carries a DIDSON
acoustic camera for subsurface inspection and a three video cameras (forward, rear,
hemispherical) for viewing above the waterline. The robot can carry additional
sensors. The Sea-RAI is battery powered (lead-acid) and can operate for 46 h
depending on the currents. The Sea-RAI uses a team of three operators, one to pilot,
one to collect imagery, and one to serve as a safety officer. In addition, one or more
J Intell Robot Syst (2011) 64:7795 85
Fig. 4 Sea-RAI with key
components labeled
other engineers were engaged in observing and directing data collection. A notable
feature of the Sea-RAI is that it stores sensor data and the internal state of the robot,
creating a database that can be displayed in a unique Google Earth interface. Data
at any point in time or location can be retrieved with a click.
The VideoRay ROV and a YSI Ecomapper were also deployed. The VideoRay is
a tethered ROV already in use by some departments of transportation for visual
inspection of bridges and debris, either lowered off a bridge or from a boat. It
generally carries a video camera and a larger version can be modified to carry a
scanning acoustic camera such as a Blueview imaging sonar. The VideoRay ROV
required three operators, one to pilot, one to manage the tether and a safety officer.
The YSI Ecomapper is used for environmental research. It does not use a tether
and navigates with waypoints based off of GPS coordinates while above water and
inertial navigation when submerged. It carries a side-scan sonar suitable for mapping
debris fields. The Ecomapper was autonomous but required a two-person team to
set up and use.
3.3 Inspection Results
The Sea-RAI was successful in meeting its primary mission objectives. The robot
was deployed tghree times, twice on Dec. 18 and once on Dec. 19. It used waypoint
navigation to either conduct mapping of the debris field or to travel to the bridge. The
Sea-RAI was manually controlled near the bridge. Missions lasted approximately
2 h, with about 1 h of active investigation of the bridge and debris field in between
changes in tides. The VideoRay and Ecomapper were also deployed.
The USV found no sign of scour or washout of the bridge pilings, as can be seen
in Fig. 5. The upper image shows the healthy pilings at the Rollover Pass Bridge.
Pilings give off a bright, sharp line where the foot meets the ground. This is contrast
to dark holes in the image below showing the scour at pilings for a dock at Marco
Island inspected after Hurricane Wilma.
Figure 6 shows the path of a typical debris mapping run where the robot was
programmed to autonomously complete a raster scan of the area of interest using
GPS waypoints (green dots). The Sea-RAI did not find debris that would obstruct
86 J Intell Robot Syst (2011) 64:7795
Fig. 5 DIDSON imagery:
a shows no scour at Rollover
Pass Bridge after Hurricane
Ike, b shows scour (dark
holes in front of pilings) at
Marco Island dock after
Hurricane Wilma
a
b
Fig. 6 Screen capture from
USV showing waypoints and
path for debris mapping
overlaid on Google map
pre-Ike photo
J Intell Robot Syst (2011) 64:7795 87
Fig. 7 DIDSON imagery:
pallet and debris in the
channel
navigation through the Rollover Pass channel or present a hazard to the bridge.
Figure 7 shows the typical debris scattered throughout the channel.
4 Comparison of UMVs
To be successful for post-disaster bridge inspection, or any other shallow, cluttered
littoral application, the UMV must be able to meet the mission requirements under
extreme environmental conditions and in the presence of debris and unpredictable
sea floor topologies and with unfavorable deployment conditions. As shown in
Fig. 8, the criteria for comparing types of UMVs can be divided into five categories,
Fig. 8 Comparison of USV,
ROV, and UUV for the
post-disaster bridge inspection
domain. Green means highly
suited, orange some concerns,
and red ill-suited
USV ROV UUV
Mission requirements
perceive damage
perceive scour
map debris field
Operate in extreme conditions
function in currents
agile station-keeping
function with turbidity
Operate in shallow, cluttered littoral regions
work in close proximity to structures
avoid underwater obstacles
avoid above water obstacles
Deployability
portability
tethered
Desirables
Supply real-time imagery
88 J Intell Robot Syst (2011) 64:7795
described below. In general, the USV is more suitable for the entire scope of tasks,
with the ROV and UUV highly capable for subsets.
The mission requirements are the essentials; in the case of post-disaster bridge
inspection a UAV must be able to perceive damage and scour in the substructure
as well as map the debris field. This requires both been able to get close enough to
the structures in order to map in order to perceive damages and scour, To do this,
the UMV must be must be able to get close enough to the structure and carry the
correct type of sensors that can penetrate turbidity if present. On the other on the
other hand in order to map the debris field, the UMV must be able to transit long
distances while also penetrating some turbidity.
The UMV must be able to operate in extreme environmental conditions. It must
be able to function in high currents and maintain accurate stationkeeping. As noted
before it must also function in turbidity. This last condition is redundant with the
perceptual tasks in the mission requirements category but is broken out for clarity.
Accurate stationkeeping means that the object of interest remains stationary while
the operators study the structure. It makes a difference whether the operator is
looking straight at or perpendicular to a structure, and it can be very frustrating if
the system is constantly moving off course and repositioning at a different viewing
angle.
The UMV must be able to operate in shallow, cluttered littoral regions. This means
working in close proximity to structures with only a few meters standoff distance.
The UMV should be able to avoid underwater obstacles as debris and the sea floor
topology may have changed. Operations in shallow water requires a sensitivity to the
tides; for example, a surface vehicle and UUV may be able to operate in a certain
region during high tide but become grounded at low tide. Methods and sensors for
detecting obstacles, such as boats, structures, buoys, and debris, above the water line
exist.
Deployability is a practical concern. The UMV must be portable and not require
the use of boat ramps, lifts, or any other aids that may have been destroyed. Tethers
interfere with deployability because of tether management; as noted in [32] and
by our experience with the ROV, it is extremely difficult to keep the tether from
tangling around debris or structures in these complex, high current environments.
In addition to the mission requirements and operating and deployment conditions,
it is desirable for a UMV to provide the operators with real-time imagery. This allows
the operators to inspect in real time in order to redirect the vehicle for further more
detailed observations if something is of interest. It also means that all data can be
captured immediately and if the vehicle malfunctions or sinks, data will not be lost.
Figure 8 captures the qualitative rating of the team for each of the robots at
Rollover Pass. Green means the vehicle generally meets that attribute, orange that
it meets it but with some notably drawback, and red means that it does not meet
it. A simplistic scoring scheme, green=3, orange=2, and red=1, for the entire set
of criteria suggests a preference ordering of USV, UUV, and ROV. However, this
interpretation of the table is misleading. Note that the USV is the only vehicle that
can perform both the close proximity portions of the mission (perceive damage to
the substructure) and the wider area component (map the debris field). The ROV
can perform the close proximity mission if the turbidity is low allowing the video
camera to see over several meters, but the UUVcannot because its navigation system
has significant errors and it might miss or collide with the structure and would be
difficult to be sure it had surveyed the entire structure. The UUV can map a debris
J Intell Robot Syst (2011) 64:7795 89
field with its side-scan sonar, assuming no navigational hazards are present, but the
ROV nether has sufficient tether length nor a positioning system that would permit
accurate mapping.
In terms of operating in extreme conditions, none of the vehicles function well in
high currents. The USB and UUV being larger have more power, while the smaller
ROP has the tether to prevent it from being swept away. None of the vehicles
perform acceptable stationkeeping. The USV has to contend with currents and with
the wind as documented by [33], while the ROV has deal with currents. The UUV
cannot operate in close proximity to structures. The sonars on the USV and UUV
can image through turbidity, while the ROV uses only a video camera which cannot
penetrate more than a meter through cloudy water.
The USV can operate in shallow, cluttered littoral regions, especially in close
proximity to structures, and can avoid underwater obstacles by virtue of its shallow
draft. The UUV can avoid above water obstacles by virtue of swimming under them.
The USV and ROV must rely on the operator to manage avoidance of above water
obstacles. None of these systems have autonomous obstacle avoidance though that
may change in the future.
Deployability in terms of the portability and launching of all three types of vehicle
is high. The ROV poses some concerns because of where an operator might have to
stand to deploy the tether; standing on a bridge that is suspect is worrisome and the
areas surrounding the collapse bridge may also be difficult to reach or unsteady. The
USV and UUV were rated higher because they were not tethered, due to problems
with tangling.
The UUV did not score well in regards to real-time imagery. Because a UUV
operates under water and can only communicate through a low baud rate acoustic
modem, it does not supply real-time imagery. Instead data must be stored for delivery
when the vehicle surfaces or downloaded when it returns to home. This poses a
significant risk of loss of data. In contrast, a USV operates above the surface with
presumably strong wireless conditions and the ROVs tether is a communications
line supplying real-time imagery.
5 General Findings
The evaluation produced findings in three areas: control challenges for UMVs,
human-robot interaction, and uncertain data. The control confirmed earlier findings
from Wilma [7] plus observations from other marine vehicle research [6, 17, 3234].
The human-robot interaction findings are similar to the Wilma findings [7] and are
consistent with small unmanned aerial vehicle issues [35]. The challenges in handling
and fusing the sensor data is new.
5.1 UMV Control Challenges
The fieldwork at Rollover Pass identified three major control challenges for UMVs.
The first is navigation and station-keeping in swift currents, a well-known problem
[33, 36] but as yet unsolved. The swift currents limited the times and duration the
Sea-RAI, EcoMapper, and VideoRay could be used. The Sea-RAI was actually put
on a safety line during the first run to make sure it could be recovered. The problems
90 J Intell Robot Syst (2011) 64:7795
with currents and station keeping is not unknown, see [33] for another example.
While station keeping and localization is difficult for all three types of UMVs, the
USV is especially challenging because of the higher degrees of freedom involved
(rotation/translation of the vehicle, pan/tilt of the sensor) and more environmental
impacts (wind plus water).
The second challenge is navigating in the presence of GPS loss or errors. As noted
in [7], operations near bridges interfere with GPS signals impacting the USV and
the surface operations of the UUV. The Sea-RAI had to be teleoperated for the
actual inspection task. The GPS loss was approximately 1% away from the bridge
and 22% under or in the shadow of the bridge. Therefore GPS will not be sufficient
for close inspection. And GPS errors in cluttered littoral environments can lead to
collisions, as seen when the Ecomapper while using GPS for a surface-based scan of
the channel bumped into a barricade. Figure 9 shows GPS outages based on location
and that while GPS errors are more frequent near the shadow of the bridge, they
also occur in relatively open spaces.
The third challenge is obstacle avoidance for underwater vehicles [17], as inci-
dents occurred in two of the five runs with the UUVs (40% incident rate). The
EcoMapper could not safely operate submerged through the channel until the Sea-
RAI mapped the underwater debris and gave some indication that the channel was
clear; unlike bays or open water, littoral regions after a disaster may be cluttered with
unmodeled obstacles. Without the Sea-RAI, it would have been risky to operate
submerged. Even operating on the surface using GPS, the UUV collided with a
channel barricade. The tethered VideoRay ROV became tangled around a pipeline.
This highlights the need for tether management and awareness of where the tether
is related relative to obstacles and the robot. Recent commercial developments have
created a smart tether which keeps up with its own position, which might solve the
problem in the future.
Obstacle avoidance for surface vehicles is less of a problem. Above the waterline
navigation should be possible with the addition of ranging sensors, while USV with
Fig. 9 Map of GPS outage for
a typical USV run
Red is GPS loss
J Intell Robot Syst (2011) 64:7795 91
a shallow draft are unlikely to collide with submerged vehicles. The only concern is
protecting the sonar payload.
5.2 Human-Robot Interaction
While imaging and perception difficulties in using UMVs are documented [34], the
fieldwork at Rollover Pass addressed the broader role of human-robot interaction.
It confirmed the role of humans in shared autonomy, reinforced the need for
multiple displays, and illustrated the lack of resilience in design and displays. The
USV and ROV were teleoperated, though the USV performed the debris mapping
and navigation in the open channel autonomously, resulting in a shared autonomy
scheme referred to as remote presence [37], where the operators work with the vehicle
in real-time as extensions of themselves.
The role of humans were piloting, payload specialists (operating the sensors),
subject matter experts (interpreting the data), and safety oversight. Aminimumof four
people were involved. These roles relied on a close interaction between all groups.
As noted above, the robot was teleoperated for a significant portion of the time
and the robot was in line of sight for the entire mission. Alternative teleoperation
mechanisms such as creating a virtual reality system proposed in [38] may be useful
and reduce the logistics footprint. Work reported in [39] in human-guided autonomy
work with ROV suggest operators prefer a exocentric, geolocative display but offer
no studies to confirm this.
A surprising human-robot interaction was the need for two-way audio, as there
was always a person involved in launching or recovering the robot. At Rollover
Pass, the base station was approximately 0.4 km from the launching area. Clegg
and Peterson [40] discusses trials for a REMUS UUV operating in shallow waters
where the operators were several kilometers away but does not describe the lessons
learned or posit any guidelines. Two-way audio would have permitted the operator
to coordinate with the handler in the field rather than be reduced to gesturing at a
camera and having the operator signal through panning or tilting and having a spare
team member run back and forth.
The need for multiple displays to accommodate multiple observers was noted in
general in [41] and the Hurricane Ike fieldwork reinforced this. Additional displays
are needed for the specialists and subject matter experts and the display should
be customizable. For example a civil engineer may just want to see the DIDSON
output while the operator sees the complete interface which includes vehicle health,
path, etc.
The deployment also showed problems with resilience and how hard it is for
humans to understand what is going on with vehicles. The DIDSON was knocked
out of alignment by the force of the water, causing the operators to get confused
about which way it was pointing relative to the vehicle. This led to coordination
challenges as the specialist had to give counter intuitive commands to the pilot
to maintain views of pilings. The design of the payloads should prevent normal
slippage and the display should provide diagnostics for confirmation of settings
and positions. This has been addressed in the newest version of the DIDSON
visualization software, by providing an indication of the sonar orientation relative
to the vessel it is deployed on. However, more comprehensive health monitoring
systems such as those proposed in [32] are needed.
92 J Intell Robot Syst (2011) 64:7795
5.3 Multi-Robot Cooperation
The experiences also suggested that multi-robot cooperation may be beneficial.
Homogeneous USV or UUV teams have been investigated. Prior work by [42]
has considered teams of surface vehicles acting as a water-based distributed sensor
network near coasts, while [43] explored UUVs for lane-finding through minefields
which might be useful for mapping ship channels. Heterogeneous teams have
also been examined. USV-UAV teaming was explored at post-Hurricane Wilma
deployment [7, 12, 44] discusses ground-sea-air coordination, while [45] describes
coordinating surface with untethered underwater vehicles. The USV-UUV teaming
is attractive given that a USV could map the region sufficiently for UUV navigation.
As noted in [7], the USV could also serve as a mother for a tethered ROV, seeing
where the USV could not, reducing the amount of tether and risk of tangling of a
ROV, and help keep overwatch on the tether. UMV researchers have proposed the
NEPTUS framework for multi-UMV control [46], while [47] has conceptualized a
multi-level global and local planning system for team coordination.
5.4 Uncertain Sensor Data
The purpose of the UMVs is inspection, to observe and record; this creates chal-
lenges in sensing processing, particularly in handling large datasets and managing
uncertainty. The Sea-RAI collected a continuous stream of imagery from above
and below the waterline. The imagery was color video and sonar, leading to large
datasets of information. Furthermore, the data had uncertainties in localization and
in content due to shadows and differing viewpoints from the vehicle and DIDSON
angles, presenting challenges for accurate 3D reconstruction and in understanding.
6 Conclusions
UMVPost-disaster inspection offers economic advantages by speeding up and reduc-
ing the cost of rescue and recovery while reducing human risk to divers who currently
manually inspect bridges. In order to effectively conduct post-disaster inspection of
structures in littoral regions, a UMV must be able to control changes in velocity and
direction to subject object investigation under extreme environmental conditions
(e.g., tides, currents, wind) in the presence of debris and changes in topology, and
under unfavorable deployment conditions (e.g., no boat ramps or nearby safe areas
of operation).
The experiences at the Rollover Pass Bridge showed that UMVs have sufficient
utility for immediate use in littoral inspection. While tethered ROVs have begun
to be explored by transportation departments, USVs appear to be more promising
than UUVs because of navigability in high currents, no tether to tangle, reduced
vulnerabilities to obstacles, ability to carry payloads such as acoustic cameras which
can penetrate turbidity, and real-time transmission of data. Regardless of surface
or underwater deployments, UMVs pose many open research questions in control
(especially with GPS dropout rates on the order of 20% or higher), human-robot
interaction,cooperation between surface and underwater vehicles, and handling of
large data sets of uncertain sensor readings.
J Intell Robot Syst (2011) 64:7795 93
The fieldwork combined with the review of literature suggest a possible roadmap
for improving post-disaster inspection of bridges and littoral regions. The ultimate
goals are to improve inspection and to reduce the number of humans on the human-
robot team. This may be accomplished through seven milestones; the milestones are
presented as sequential but in reality the research and development for several could
be done concurrently.
Standardize mission payloads. UMVs should have an acoustic camera and a
video camera to cover both high and lowlevels of turbidity. Given that inspection
may involve slow movements close to structures, side scan sonars which rely on
motion do not appear appropriate.
Health monitoring. A UMV should have sufficient sensors or encoders to cor-
rectly determine the position of its components and the overall health.
Improved teleoperation through better human-robot interaction. HRI is a sig-
nificant weakness in UMVs. Studies are needed to resolve competing displays
(see [11] versus [39]), while payloads such as two-way audio and extra displays
can facilitate team coordination.
3D obstacle avoidance. UMVs need to avoid obstacles above and below the
waterline but also consider the tides and depths. Above the waterline range
sensors must be hardened to work when splashed with water. Obstacle avoidance
is expected to speed navigation and improve the overall safety of operations.
Station-keeping. Significant advances in vehicle control must be made to enable
station-keeping. This is expected to reduce the load on the mission specialist and
engineers because they have a steady image to look at and eventually reduce the
number of people involved in the robot team.
Handle large data sets. As UMVs become more successful at inspection, a huge
amount of imagery and data will be generated and must be handled.
Cooperative sensing. Current systems put the responsibility for surveying the en-
tire structure and identifying damage or scour on the human. Advances in control
and in image processing should be able to ensure coverage of the structure and
assist the image interpretation by cueing areas of interest. These advances in
control theory and artificial intelligence would likely increase performance and
the reliability of the inspections.
Our current work has focused on creating a refined version of the Sea-RAI
platform and software to serve as a platform for further research in control, human-
robot interaction, and sensing.
Acknowledgements The authors thank Dr. Dennis Christiansen (TTI) for his help, the Texas
Engineering Extension Service (TEEX) for the generous loan of response equipment and personnel
(especially Daniel May for his expert help and suggestions), and Kimberly Mallett for logistics and
support.
References
1. Murphy, R.R., Steimle, E., Lindemuth, M., Trejo, D., Hall, M., Slocum, D., Hurlebas, S., Medina-
Cetina, Z.: Robot-assisted bridge inspection after Hurricane Ike. IEEE Workshop on Safety
Security Rescue Robotics, pp. 15. IEEE, Denver (2009)
2. Brown, J.: Robot inspects damaged bridge. Civ. Eng. 78(3), 40 (2009)
3. Merriam, E.: MDSU-2 Arrives in Minneapolis Prepared to Help. 1 May 2010 (2007)
94 J Intell Robot Syst (2011) 64:7795
4. Goldwert, L.: Minneapolis Honors Bridge Collapse Victims. 1 May 2010 (2007)
5. Leibholz, S.: Trends in robotics and implications for the marine environment. OCEANS 1984,
vol. 15, pp. 100105 (1983)
6. Ageev, M.D., Kiselev, L.V., Shcherbatyuk, A.P.: Tasks for autonomous underwater robot. Fifth
International Conference on Advanced Robotics, 1991. Robots in Unstructured Environments,
91 ICAR, pp. 13601364, vol. 2 (1991)
7. Murphy, R., Steimle, E., Griffin, E., Cullins, C., Hall, M., Pratt, K.: Cooperative use of unmanned
sea surface and micro aerial vehicle at Hurricane Wilma. Journal of Field Robotics 25(3), 164
180 (2008)
8. Steimle, E., Murphy, R., Hall, M., Lindemuth, M.: Unmmaned marine vehicle use at Hurricanes
Wilma and Ike. MTS/IEEE OCEANS 2009, pp. 16. (2009)
9. Murphy, R., Tadokoro, S., Nardi, D., Jacoff, A., Fiorini, P., Erkmen, A.: Rescue Robotics. In:
Sciliano, B., Khatib, O. (eds.) Handbook of Robotics, pp. 11511174. Springer-Verlag (2008)
10. Anderson, B., Crowell, J.: Workhorse AUV: a cost-sensible new autonomous underwater vehicle
for surveys/soundings, search & rescue, and research. In: Proceedings of MTS/IEEE OCEANS,
2005, pp. 16 (2005)
11. Lindemuth, M., Murphy, R., Steimle, E., Armitage, W., Dreger, K., Elliot, T., Hall, M., Kalyadin,
D., Kramer, J., Palankar, M., Pratt, K., Griffin, C.: Sea-RAI: a Marsupial USV-UAV team for
littoral inspection. IEEE Robot. Autom. Mag. (to appear, 2011)
12. Murphy, R., Stover, S., Pratt, K., Griffin, C.: Cooperative damage inspection with unmanned
surface vehicle and micro unmanned aerial vehicle at Hurricane Wilma. IEEE/RSJ International
Conference on Intelligent Robots and Systems, 2006, pp. 99 (2006)
13. Watanabe, K., Nakamura, A.: A design of tiny basin test-bed for AUV multi agent. In: Proceed-
ings of MTS/IEEE, OCEANS, 2005. pp. 10021008, vol. 2 (2005)
14. Meyrowitz, A.L., Blidberg, D.R., Michelson, R.C.: Autonomous vehicles. Proc. IEEE 84(8),
11471164 (1996)
15. Castelin, S., Bernstein, P.: A notional scenario for the use of unmanned system groups in littoral
warfare. 2004 IEEE/OES autonomous underwater vehicles, pp. 14 19 (2004)
16. Alleman, P., Kleiner, A., Steed, C., Hook, D.: Development of a new unmanned semi-
submersible (USS) vehicle. OCEANS 2009, MTS/IEEE BiloxiMarine Technology for Our
Future: Global and Local Challenges, pp. 16 (2009)
17. C. v. Alt, Allen, B., Austin, T., Forrester, N., Goldsborough, R., Purcell, M., Stokey, R.: Hunting
for mines with REMUS: a high performance, affordable, free swimming underwater robot.
OCEANS, 2001. MTS/IEEE Conference and Exhibition, pp. 117122, vol. 1 (2001)
18. Bick, E.T., Barock, R.T.: CENTURION harbor surveillance test bed. In: Proceedings of
MTS/IEEE OCEANS, 2005. pp. 13581363, vol. 2 (2005)
19. Healy, P.D., Bishop, B.E.: Sea-dragon: an amphibious robot for operation in the littorals. 41st
Southeastern Symposium on System Theory, 2009. SSST 2009. pp. 266270 (2009)
20. Farrell, J.A., Li, W., Pang, S., Arrieta, R.: Chemical plume tracing experimental results with a
REMUS A.U.V. Ocean 2003 Marine Technology and Ocean Science Conference (2003)
21. Ohata, S., Ishii, K., Sakai, H., Tanaka, T., Ura, T.: Development of an autonomous underwater
vehicle for observation of underwater structures. In: Proceedings of MTS/IEEE OCEANS, 2005,
vol. 3, pp. 19281933 (2005)
22. Williams, S., Mahon, I.: Simultaneous localisation and mapping on the Great Barrier Reef. IEEE
International Conference on Robotics and Automation, 2004. Proceedings. ICRA 04. 2004,
vol. 2, pp. 17711776 (2004)
23. Steimle, E.T., Hall, M.L.: Unmanned surface vehicles as environmental monitoring and assess-
ment tools. OCEANS 2006, pp. 15 (2006)
24. Caccia, M., Bono, R., Bruzzone, G., Spirandelli, E., Veruggio, G., Stortini, A.M., Capodaglio, G.:
Sampling sea surfaces with SESAMO: an autonomous craft for the study of sea-air interactions.
IEEE Robot. Autom. Mag. 12(3), 95105. 1070-9932 (2005)
25. Schiavon, R., Finotello, R., Terribile, T.: The supervisory control of ARAMIS, a system for
robotic inspection of sediments. OCEANS 2000 MTS/IEEE conference and exhibition, vol. 2,
pp. 823828 (2000)
26. Ribas, D., Ridao, P., Tardos, J.D., Neira, J.: Underwater SLAM in a marina environment.
IEEE/RSJ International Conference on Intelligent Robots and Systems, 2007. IROS 2007.
pp. 14551460 (2007)
27. Ribas, D., Ridao, P., Neira, J., Tardos, J.D.: SLAM using an Imaging Sonar for Partially Struc-
tured Underwater Environments. IEEE/RSJ International Conference on Intelligent Robots and
Systems, 2006, pp. 50405045 (2006)
J Intell Robot Syst (2011) 64:7795 95
28. Whalen, J.E., Wallace, G.: Multi-platform sonar-based systems for submerged threats operating
in littorals. Proceedings of MTS/IEEE, OCEANS, 2005. pp. 684689, vol. 1 (2005)
29. Bibuli, M., Bruzzone, G., Caccia, M., Indiveri, G., Zizzari, A.A.: Line following guidance control:
application to the Charlie unmanned surface vehicle. IEEE/RSJ International Conference on
Intelligent Robots and Systems, 2008. IROS 2008, pp. 36413646 (2008)
30. Manley, J.E.: Development of the autonomous surface craft. IEEE Oceans, vol. 2, pp. 827832
(1997)
31. Curcio, J., Leonard, J., Patrikalakis, A.: SCOUTa low cost autonomous surface platform for
research in cooperative autonomy. IEEE OCEANS, pp. 725729 (2005)
32. Rae, G.J.S., Dunn, S.E.: On-line damage detection for autonomous underwater vehicles. Pro-
ceedings of the 1994 Symposium on Autonomous Underwater Vehicle Technology, 1994. AUV
94., pp. 383392 (1994)
33. Pereira, A., Das, J., Sukhatme, G.S.: An experimental study of station keeping on an underactu-
ated ASV. IEEE/RSJ International Conference on Intelligent Robots and Systems, 2008. IROS
2008., pp. 31643171 (2008)
34. Duo-Min, H., Seet, G.G.L.: Underwater vision enhancement in turbid water by range-gated
imaging system. Summaries of papers presented at the Conference on Lasers and Electro-Optics,
2001. CLEO 01. Technical Digest, p. 378 (2001)
35. Murphy, R., Pratt, K., Burke, J.: Crew roles and operational protocols for Rotary-Wing Micro-
UAVs in close urban environments. ACM SIGCHI/SIGART Human-Robot Interaction, pp. 73
80 (2008)
36. Wood, S., Rees, M., Pfeiffer, Z.: An autonomous self-mooring vehicle for littoral & coastal
observations. OCEANS 2007Europe, pp. 16 (2007)
37. Murphy, R., Burke, J.: From remote tool to shared roles. IEEE ROBOT. AUTOM. MAG.
(special issue on New Vistas and Challenges for Teleoperation) 15(4), 3949 (2008)
38. Qingping, L., Chengi, K.: Virtual tele-operation of underwater robots. IEEE International Con-
ference on Robotics and Automation, 1997. Proceedings., 1997, vol. 2, pp. 10221027 (1997)
39. Murphy, C., Singh, H.: Human-guided autonomy for acoustically tethered underwater vehicles.
OCEANS 2008, pp. 18 (2008)
40. Clegg, D., Peterson, M.: User operational evaluation system of unmanned underwater vehicles
for very shallow water mine countermeasures. In: Proceedings OCEANS 2003, vol. 3, pp. 1417
1423 (2003)
41. Murphy, R., Stover, S.: Gaps analysis for rescue robots. ANS 2006: Sharing Solutions for Emer-
gencies and Hazardous Environments (2006)
42. Eickstedt, D.P., Benjamin, M.R., Schmidt, H., Leonard, J.J.: Adaptive Control of heterogeneous
marine sensor platforms in an autonomous sensor network. IEEE/RSJ International Conference
on Intelligent Robots and Systems, 2006, pp. 55145521 (2006)
43. Redfield, S.A.: Lane finding using homogeneous groups of cooperating autonomous vehicles
(2004)
44. Yavnai, A.: Distributed decentralized architecture for autonomous cooperative. Autonomous
Underwater Vehicle, pp. 6167 (1994)
45. Curcio, J., Leonard, J., Vaganay, J., Patrikalakis, A., Bahr, A., Battle, D., Schmidt, H., Grund, M.:
Experiments in moving baseline navigation using autonomous surface craft. IEEE OCEANS,
vol. 1, pp. 730735 (2005)
46. Dias, P.S., Fraga, S.L., Gomes, R.M.F., Goncalves, G.M., Pereira, F.L., Pinto, J., Sousa, J.B.:
Neptusa framework to support multiple vehicle operation. IEEE OCEANS- Europe, vol. 2,
pp. 963968 (2005)
47. Khorrami, F., Krishnamurthy, P.: A hierarchical path planning and obstacle avoidance system
for an autonomous underwater vehicle. American Control Conference, 2009. ACC 09, pp. 3579
3584 (2009)