The Detection and Disposal of IED Devices within
Harbor Regions using AUVs, smart ROVs and Data
Processing/Fusion Technology
Scott Reed, Jon Wood, Chris Haworth
SeeByte
Edinburgh, UK
Scott.Reed@seebyte.com
Abstract — Keeping our ports and waterways safe from threats
has become increasingly relevant. Hull and harbor infrastructure
inspections are not performed regularly as they are time
consuming, require careful planning and involve a huge amount
of risk as well as human and monetary resources. An
Autonomous Underwater Vehicle (AUV)-based approach will
speed up operations and help locate and identify possible threats
while removing divers from the high risk tasks. Providing smart
Remotely Operated Vehicle (ROV) technology that can
automatically prosecute contacts identified by the AUV as well as
carry out close video and sonar inspections of harbor
infrastructure will also help speed up and improve inspections. In
tandem with constant monitoring in sensitive diver incursion
areas this will provide a holistic risk-averse harbor inspection
solution.
Harbor areas may be divided into distinct open and constrained
regions. Within the open harbor regions there is a clear need to
remove divers from the tasks of detecting targets, discriminating
them from bottom clutter and positively identifying them.
SeeTrack Professional comprises a modular suite of software
components which provides multi-vehicle mission planning,
monitoring and post processing capabilities for rapid on-site
analysis and data fusion of the sensor data. It uses a modular
open architecture that provides a single integrated picture for all
assets being employed within the Harbor region. It has been used
successfully to coordinate multiple AUV operations. SeeTrack
Professional includes a Change Detection component which fuses
and compares data to highlight anything from possible threats to
foreign bodies. The Change Detection model considers the
location and spatial distribution of the targets within the Harbor
region data to flag possible new threats. Carrying out the fusion
at the meta-data level allows the Change Detection to be carried
out on data collected with different assets.
Within constrained regions of a harbor, ROVs can be deployed to
inspect areas such as the pier pilings, harbor walls and around
the ship hulls. ROVs are rapidly deployable and capable of
continuous, reliable operations in adverse conditions. They also
provide a platform on which multiple sensors may be mounted
and utilized to meet the harbor inspection problem. This paper
will present a smart ROV solution that employs automated
control technology to simplify close inspection. The advanced
technology enables the pilot, to focus on his primary task of
looking for possible threats (such as IEDs, Limpet Mines, signs of
Authorized licensed use limited to: Academia Navala Mircea cel Batran Constanta. Downloaded on December 12,2022 at 11:13:14 UTC from IEEE Xplore. Restrictions apply.
sabotage, etc). This same smart ROV can be used to
automatically prosecute contacts identified by the AUV in the
open areas.
In conjunction with a robust-repeatable real-time Diver
Detection system that can be operated with sonars mounted close
to sensitive areas of a harbor the waterways, the infrastructure
can be kept safe and free of threats. By using real-time sensor
processing capabilities including Automatic Target Recognition
(ATR) algorithms for Forward Looking Sonar and Video sensors
the AUV, Mosaicing and live 3D Re-construction modules,
ROV’s and harbor monitoring sensors can highlight possible
IED targets and divers, locating their position within the harbor
environment.
This presentation will present results from trial and operations.
Keywords-port and harbor security; autonomy; target
recognition;
I. INTRODUCTION
The security of our Port and Harbor areas is coming under
increased scrutiny due to growing concerns of a possible
terrorist attack. Environmental pressures to control vessel
discharges, along with counter drug concerns regarding
contraband smuggling further highlight the need for a robust-
repeatable solution to monitor and inspect these regions. These
surveys must provide detailed coverage information, allow
possible threats to be localized and re-acquired and should
allow comparisons with historic data sets. The proposed
solution should be man-portable, be simple to use, allow
frequent inspections and require little training.
Hull and Harbor infrastructure inspections are currently not
performed regularly as they require careful planning are time
consuming and require large amounts of monetary resources.
Manual Diver-centric operations offer an unacceptable level of
risk and require substantial training. Remotely Operated
Vehicles (ROV’s) may be deployed to inspect the constrained
regions of the harbor such as the pier pilings, ship hulls and
harbor walls. They can be deployed quickly, mounted with
increasingly sophisticated sensor payloads and operated
continuously in adverse conditions. However, controlling the
vehicle while also assessing the live sensor data and managing
 the ROV tether is a difficult task. Manual, joystick controlled The remainder of the paper is organized into two main
operations lack stability and cannot provide accurate navigation sections. The first section addresses the three operational
information. This uncertainty makes coverage estimation and solutions required for the proposed concept of operations. The
object re-localization very difficult. Achieving situational second section then goes on to discuss how cutting edge
awareness of where the ROV has inspected is also difficult due technical developments from SeeByte can be used for these
to a lack of sensor processing operator-aid tools. operational solutions. Results from recent field trials are
shown which demonstrate the readiness of these technologies.
Ports and Harbors may be split into distinct open and Finally the conclusions from this new concept for port and
constrained regions. Within the open water areas, there is a harbor security are discussed.
clear need to remove divers from the task of detecting possible
targets and discriminating real threats from bottom clutter.
Within the constrained areas, diver and manual ROV II. CONCEPT OF OPERATIONS
operations provide poor stability and coverage is hard to
achieve. A replacement that provides safer, cheaper and faster A. SURVEY: Open areas with AUVs
operations is clearly desirable. When inspecting the open water regions of the harbor
AUV’s may be used to survey large regions, searching for
Persistent surveillance is being widely adopted in a broad
potential threats. These vehicles may be programmed to
range of military and security scenarios to improve target
conduct lawnmower trajectories, providing high resolution
detection, tracking and identification performance. However,
sensor data while assuring 100% coverage rate. Navigation
the challenges of operating underneath the water make
sensors now provide positioning capabilities which enable an
persistent surveillance harder to achieve. This paper presents a
object to be localized and subsequently re-acquired. Modern
new “concept of operations” for port and harbor security based
sonar technologies provide large range, high quality data which
on persistent surveillance and rapid interrogation of discovered
provides a high coverage rate and the resolution required to
changes. The first key requirement is a robust-repeatable
detect potential threats.
ability to conduct frequent inspection surveys and provide the
operator with the situational awareness to interpret these However, historically AUV operations have required highly
surveys. The second requirement is the ability to inspect skilled operators, have been time consuming to perform and
smaller key regions including the ability to rapidly inspect have required specialist launch and recovery systems. Modern
discovered changes (or anomalies) to the known scene. Finally ultra-low logistic AUVs are changing this with one-man
this survey and inspection capability must be used in portable systems that can be rapidly deployed and recovered
conjunction with active, real-time threat detection systems for from a RIB or pier. In combination, operators often use a
monitoring high-value and sensitive locations to provide a variety of vehicle assets, making it impractical and unwieldy to
complete Port and Harbor security solution. have separate software packages for each vehicle and sensor
type. SeeByte addresses this issue with SeeTrack Professional,
The proposed “concept of operations” is described that
a modular suite of software components which provides multi-
makes use of three distinct technology solutions to address the
vehicle mission planning, monitoring and post processing
survey, inspection and monitoring tasks to provide a holistic
capabilities for rapid on-site analysis and data fusion of the
approach for modern port and harbor security. For open harbor
sensor data. It uses a modular open architecture that provides a
areas where the primary focus is reliable, consistent gathering
single integrated picture for all assets being employed within
of harbor floor data, the primary mode of operation is to survey
the Harbor region. It has been used successfully to coordinate
with AUVs. The AUVs provide the ability to survey a large
multiple AUV operations [1][2].
area gathering either video or side-scan sonar data all of which
can be fully geo-referenced. Furthermore, the danger faced by The critical capability delivered by AUV-based surveys is
divers is removed making an AUV the optimal choice. the ability to identify changes within a known region.
However, the ability to conduct change detection operations is
For constrained harbor areas, which are characterized by
dependent on being able to accurately register data from
complex 3D spaces, AUV surveys become impractical and of
limited value. In this case the primary focus is on close visual
inspection of the area, including for example the harbor walls
and pier pilings. Here the mode of operation is smart ROVs
gathering close inspection footage with the pilot’s capability
enhanced to enable accurate mission tracks, precise maneuvers
and improved inspection focus. The smart ROVs can also be
used to rapidly inspect and interrogate discovered changes in
the open harbor areas.
The final mode of operation is for the real-time monitoring
of high-value and sensitive locations, and this is achieved using
permanently mounted sonars combined with advanced sensor
processing to provide automatic monitoring. The system will
provide with operator alerts whenever potential targets are
detected in the field-of-view allowing rapid intervention with
the most appropriate asset.
Figure 1. SeeTrack Professional allows sensor data from multiple assets to
be geo-referenced, fused and compared.

Authorized licensed use limited to: Academia Navala Mircea cel Batran Constanta. Downloaded on December 12,2022 at 11:13:14 UTC from IEEE Xplore. Restrictions apply.
 Compare
previous survey
to most recent

Figure 2. Surveys may be conducted of the same area and possible threats identified by running Change Detection missions on the data.

multiple missions. The majority of approaches attempt change allows the operator to run pre-planned mission templates, or
detection at the data level by running the same vehicle and directly control the vehicle via an intuitive interface or via a
sonar setup through exactly the same mission plan. These joystick. This low-level control provides vehicle stability and
approaches are susceptible to modifications in the environment ensures external factors such as currents do not impact the
and require a level of navigational accuracy which is difficult ROV performance. This capability has been demonstrated on
to guarantee. They also require that the same system is used multiple vehicles in severe offshore conditions and provides
each time. vehicle stability and control capabilities in constrained
environments [9].
In contrast, SeeTrack Professional provides a change
detection model that operates at the meta-data level which The inclusion of an ROV within the deployable assets also
considers the location and spatial distribution of the targets allows rapid inspection and interrogation of discovered
within the monitored region to flag possible new threats. By changes from the AUV survey. In this instance, however, the
carrying out the change detection at the meta-data level it is difficulty in re-localization using standard ROV piloting tools
possible to compare data sets that have been collected with could result in significant time delay and potentially
different assets. Any sensor or vehicle which is compatible investigation of the wrong region. SeeTrack CoPilot provides
with SeeTrack Professional can therefore contribute to the the ability to perform pre-planned mission tracks ensuring
change detection process. Operator or ATR feedback may also rapid re-localization and allowing the pilot to focus on the
be used in the process. Fusing data in this manner removes the investigation task.
need for the vehicles to swim in exactly the same manner so
that the data can be compared from the same aspect point. The C. MONITOR: High-value and sensitive regions
operator therefore has greater flexibility; they may use a wide The final aspect of port and harbor security is for high-
range of assets and may plan missions based on the particular value and sensitive region protection utilizing sonars systems
asset and daily scenario. combined with real-time sensor processing to provide constant
B. INSPECT: Constrained areas with ROVs
When moving to the constrained areas such as around the
harbor walls and near the ship hulls, it is necessary to provide
an operator-in-the-loop inspection capability to regularly
inspect these complex regions. There are now several COTS
ROV systems which provide affordable, man-portable
solutions for carrying out inspections within constrained
environments. These provide a stable platform on which high
resolution sensors such as forward looking sonar and optical
video cameras may be mounted. Many underwater operations
that were once carried out by divers can now be carried out
more quickly, more efficiently and safely using ROVs
equipped with smart control and sensor processing software.
However, the manual control of an ROV using a joystick
remains a challenging and highly skilled task. Controlling the
ROV while also assessing the live sensor data, operating in a Figure 3. The UI for SeeTrack Co-pilot allows the user to control the
ROV, and monitor the mission against known maps of the area.
controlled environment makes the task even more difficult.
SeeByte’s SeeTrack Co-pilot provides a vehicle independent,
dynamic inspection and sensor-servo capability [3][4]. This

Authorized licensed use limited to: Academia Navala Mircea cel Batran Constanta. Downloaded on December 12,2022 at 11:13:14 UTC from IEEE Xplore. Restrictions apply.
 Figure 4. Real-time sensor processing capabilities allow possible divers to be detected and tracked over time.
surveillance of the particular area. Commercial systems,
generally referred to as Diver Intruder Detection, are available
utilizing a variety of sonar platforms. The current generation
of forward looking sonars can provide a field-of-view up to
135 degrees and ranges up to 100m, allowing good coverage
options with a limited number of sonars. However, the crucial
step for constant surveillance is real-time sensor processing for
robust, repeatable diver detection. SeeByte have developed
ATR algorithms which are able to robustly detect divers and
track them over time. These algorithms have been made
compatible with a large number of sonars and a solution can be
tailored to meet the needs of the operator.
III. KEY TECHNOLOGIES
A. Control Hierarchy
The control of an ROV or AUV may be described on an
autonomy hierarchy. At the highest level, the goal of the
mission is taken in as input and used to break the mission down
into a series of tasks. The completion of these tasks, in a
prescribed order, will result in the completion of the mission.
Full
Autonomy
Operator
selected tasks
Increased
Autonomy Operator
Scene Analyser
Assist
Manual
(Joystick)
Figure 5. An inspection task using an ROV/AUV may be split into 3 levels of autonomy. Autonomy modules may be provided at all levels to aid the operator.
Authorized licensed use limited to: Academia Navala Mircea cel Batran Constanta. Downloaded on December 12,2022 at 11:13:14 UTC from IEEE Xplore. Restrictions apply.
The level below on the hierarchy manages the execution of
specific tasks. An example would be the inspection of a ship
hull or a pipe. In manned operations, an operator would carry
out this task. In autonomous operations, a Task Manager
handles the execution of the task. The lowest level on the
control hierarchy is concerned with the carrying out the actual
motion and control requests required to complete a task. For
example, the complete inspection of an object of interest may
be described by a 360 degree turn around the object while
maintaining a constant range. This motion may then be
subsequently broken down into manageable operations that the
vehicle will be asked to execute.
In the case of a manual operation, the operator is
responsible for all 3 levels of this hierarchy. For example, if
the task is to inspect an oil riser, the operator may determine
that this operation is comprised of two core tasks: transiting to
the riser after which a vertical inspection is required. Each of
these tasks requires different behaviors (and so different Task
Managers when moving to an automated system) and the low
level control is conducted by the operator via the joystick.
Figure 5 presents the idea of the autonomy hierarchy along
Goal based Mission World Model
Mission Executive
Task Managers
Motion Planner
Waypoint Engine
Waypoint
Dynamic Positioning
Thrust command
 Figure 6. Real-time mosaics are built up as the vehicle conducts its mission. Independent sonar frames are fused to build up a large scale picture of the area
being inspected.
with the associated software modules that SeeByte’s
Autonomy software uses.
With SeeTrack CoPilot, the operator may use pre-planned
mission templates or specifically alter mission parameters via
the GUI while visualizing the mission objective, via 3D
models, available imagery, nautical charts, and historic data.
SeeTrack CoPilot currently has three working modes:
Automated piloting, semi-automated piloting (where the pilot
directly supervises the vehicles actions – point and click), and
full manual (joystick) control.
In automated control mode, SeeTrack CoPilot looks after
the vehicle control process and allows the vehicle to fulfill its
mission, sensing and countering the effects of external forces
such as water currents and surge. SeeTrack CoPilot
compensates for these effects, ensuring that the vehicle moves
where the mission planner module has requested it go. The
mission planner module provides SeeTrack CoPilot with a
series of waypoints describing where it would like the vehicle
to move to. These can be provided by the operator via the
GUI, via a Task Manager when running in Automated mode
(for example when conducting an automated ship hull
inspection) or via a pre-programmed way-point mission.
SeeTrack CoPilot ensures that the vehicle moves to these
waypoints, compensating for external factors such as sea
currents.
B. 2D Real-time Mosaicing
Typically, an operator assesses sensor data image by image.
This provides a snapshot “window” of what the vehicle is
seeing but does not provide any global context to aid the
operator. Relating the data to the actual scene is very difficult.
Real-time mosaicing and 3D re-construction capabilities
provide this information within the big picture; possible threats
will be seen relative to well known features, providing the
operator with a full situational awareness of the area being
surveyed and gaps in the coverage [5][6].
Authorized licensed use limited to: Academia Navala Mircea cel Batran Constanta. Downloaded on December 12,2022 at 11:13:14 UTC from IEEE Xplore. Restrictions apply.
The real-time mosaicing software provides the operator
with a large scale picture of the object as it is being inspected.
Potential targets may be localized relative to known structures,
making re-acquisition easier. The mosaic also provides the
user with coverage information, highlighting possible holidays
in the inspection mission.
The SeeByte mosaicing module receives live sonar
information and the vehicle navigation information. The
navigation provides information on the location of the vehicle
with respect to a frame of reference. This information is used to
calculate the relative motion between frames. The relative
motion can then be corrected using image registration
techniques to improve the quality of the mosaics.
This module has been demonstrated running real-time, both
on the surface, and onboard the Bluefin HAUV vehicle for hull
inspection [8]. The mosiacs are built up in real-time as the
vehicle is conducting its operation, providing the operator with
an immediate snapshot of the vehicle’s progress. A coverage
map may also be provided showing areas requiring re-
inspection. An example of the Mosaicing module running in-
mission may be seen in Figure 6.
C. Real-time 3D Reconstruction
Within complex regions of objects such as ship hulls, 2D
imaging sonars are generally not useful for providing accurate
coverage information. The flat surface assumption used for
mosaicing no longer applies, and it is more difficult to
determine the holidays in the coverage. The imagery is also
more difficult to evaluate. Within these regions, the imaging
sonar is replaced by a profiling sonar which can be used in
conjunction with the vehicle navigation information to build a
real-time 3D picture of the complex region being inspected.
This provides the operator with an image of the area being
inspected and highlights gaps in the coverage.
 Figure 7. Example 3D re-construction results produced from DIDSON imagery collected on the Bluefin HAUV vehicle.
The 3D re-construction algorithm is run in-mission and
real-time and is capable of processing around 10 frames per
second. A multi-threaded approach is used to ensure that
points already added to the 3D representation may be rendered
into a surface while new images are being processed and
added. Individual sonar frames are pre-processing using a
variety of morphology operations before relevant information
is transformed into 3D space and rendered as a surface.
Some examples of the 3D re-construction algorithm may be
seen in Figure 7.
D. Forward Looking ATR
The SeeByte forward looking ATR uses a modular design
and is based on 3 core components: Detection, Classification
and Tracking. The entire system may be run real-time, topside
or embedded. This ATR uses an anomaly detection algorithm
to highlight initial detections within the imagery that do not
statistically match their surroundings. The image is transferred
into a probability map of each pixel matching its surroundings.
This map is built up using very simple statistical filters
computed using Integral Images techniques to ensure the
process is fast. Anomalies which are outside the current target
parameters (e.g. diver-like or mine-like) are immediately
discarded. Remaining detections are fed into the classification
component [7].
The classification component uses a rule-based system to
determine whether a region highlighted by the anomaly
detector contains a possible threat or not. A supervised
approach is used where the rules are built up based on a series
Authorized licensed use limited to: Academia Navala Mircea cel Batran Constanta. Downloaded on December 12,2022 at 11:13:14 UTC from IEEE Xplore. Restrictions apply.
of training data. The training data is used to calculate a large
number of statistical and geometrical properties, which can be
used to represent an object within the imagery. The aim of the
training stage is to obtain a set of plausible rules for the
classification algorithm, which allows the model to
discriminate targets from other detection results. Typically
around 200 features are determined within the training phase.
However, it is common for only a small number of features to
be required to correctly classify a target. This sub-set of
features is determined using a data-mining algorithm, which
generates a set of classification rules.
The final phase of the algorithm requires that a classified
object is tracked over time. The forward looking arrangement
of the sonar ensures that any object of interest will appear in a
number of consecutive frames. Requiring an object to appear
in a number of adjacent sonar frames both confirms real threat
presence and reduces false alarms. Some examples of the ATR
detecting, classifying and tracking a cylindrical object may be
seen in Figure 8.
IV. CONCLUSION
This paper has presented a novel concept of operations
using technology development to enable persistent surveillance
for Port and Harbor security. The paper first presented
operational solutions for the survey, inspection and monitoring
missions required to achieve persistent surveillance. Combined
with the advanced functionality from SeeTrack Professional,
such as ATR and change detection algorithms, this allows rapid
processing of the gathered data for timely operation while
 Figure 8. Examples from the Forward Looking Sonar ATR processing DIDSON data from a ship hull. The ATR is capable of detecting,
classifying and tracking mine threats on the hull. When sonar resolution allows, the module can identify specific object types.
reducing human workload. These operational solutions
provide a powerful configuration, permitting rapid and accurate
search and inspection capabilities, where the operator is
constantly kept aware of status of the remote assets and the
environment around them
The paper then presented key technology developments
from SeeByte that help enable this new concept of operations.
The concept of Control Autonomy was discussed and how this
could be provided at a variety of levels. Operator Aid control
tools such as SeeByte’s SeeTrack Co-pilot may be used to ease
the ROV pilots workload, providing stability and Dynamic
Positioning capabilities. This low-level control solution has
been demonstrated to improve control performance and reduce
operation time. This level of control autonomy can be used to
aid joystick operations or may be run using an intuitive
interface.
Higher Levels of Autonomy may also be employed where
the operator hands over Inspection Tasks to the control
modules. SeeByte have robustly demonstrated the ability to
automate the inspection of ship hulls, oil structures and
possible mine threats through various ONR and offshore
programs. In this mode, the operator is able to focus on
analyzing the live sensor data while the Autonomy modules
control the vehicle.
Lastly, the paper presented operator-assist sensor
processing tools. These tools have been developed to provide
the user with an improved situational awareness as the vehicle
conducts its mission. Real-time mosaicing and 3D re-
construction modules provide a large scale picture of the
environment being inspected, allowing the operator to localize
possible threats, enabling future re-acquisition. ATR models
were also presented which provide the user with prompts to
possible threats. These models run real-time or may be run
embedded on the vehicle should an AUV solution be used.
These tools have also been developed to be sensor independent,
allowing them to be integrated with a range of COTS sensors.
Authorized licensed use limited to: Academia Navala Mircea cel Batran Constanta. Downloaded on December 12,2022 at 11:13:14 UTC from IEEE Xplore. Restrictions apply.
ACKNOWLEDGMENT
SeeByte would like to acknowledge the US Office of Naval
Research and Dr. Tom Swean for supporting some of this work
through the ONR Confined Area Search and VSW Mine
Neutralization programs. SeeByte would also like to thank
their project partners: Bluefin Robotics, iRobot, MIT, Florida
Atlantic University and Blueview Technologies.
REFERENCES
[1] A.Cormack, D.M.Lane, J.Wood, “Operational Experiences for Maritime
Security Operations”, Proceedings of IEEE Oceans 2010, Sydney, 24-
27th May 2010, .
[2] A.Cormack, “International Cooperation in the Development of
Scientific, Civil and Military Unmanned Systems Command and Control
for Surveys and Rapid Environmental Assessment”, Conf. Proc. US
Hydro 2009, Norfolk, Virginia, May 11-14th 2009
[3] http://www.seebyte.com/docs/smartROV_001.pdf
[4] http://www.seebyte.com/docs/ANewROVDP_000.pdf
[5] J. Vaganay, M.L. Elkins, S. Willcox, F.S. Hover, R.S. Damus, S.
Desset, J.P. Morash and V.C. Polidoro, “Ship Hull Inspection by Hull-
Relative Navigation and Control”, Proceedings of MTS/IEEE Oceans
2005, Vol.1, 761—766, 2005.
[6] S. Reed, A. Cormack, K. Hamilton, I. Tena Ruiz and D.Lane,
“Automatic Ship Hull Inspection using Unmanned Underwater Vehicles,
Proceedings of the 7th Int. Symposium on Technology and the Mine
Problem”, Naval Postgraduate School, Monterey Bay, USA, 2006.
[7] P.Y. Mignotte, S. Reed, A. Cormack, and Y. Petillot, "Automatic Ship
Hull Inspection - The Detection of mine-like targets in sonar data using
multi-cad fusion and tracking technologies," in Proceedings of Institute
of Acoustics International Conference on the Detection and
Classification of Underwater Targets, Sept. 2007, Heriot-Watt
University, Scotland.
[8] J. Vaganay, L. Gurfinkel, D. Jankins, K. Robinson, T. Stefanov-Wagner,
S., HAUV System Performance Enhancement for Use by EOD Units' -.
Summit -8th International Symposium on Technology and the Mine
Problem, Naval Post Graduate School, Monterey CA, 6-8 May, 2008
[9] D.Saul, I.Tena, BP’s AUV Development program, Long Term Goals –
Short Term Wins, OCEANS 2007, 2nd October 2007, Vancouver,
Canada
[10] S.Reed, J.Wood, P-Y.Mignotte,“SeeByte Autonomy Delivers Smart
ROV for Successful Ship Hull Inspection”, Proc. Conf AUVSI
Unmanned Systems North America 2010, Denver 24-27th August 2010