IEEE JOURNAL OF OCEANIC ENGINEERING, VOL. 47, NO.
4, OCTOBER 2022
Detecting Marine Organisms Via Joint
Attention-Relation Learning for Marine
Video Surveillance
Zhensheng Shi , Member, IEEE, Cheng Guan, Qianqian Li, Ju Liang, Liangjie Cao,
Haiyong Zheng , Member, IEEE, Zhaorui Gu , Member, IEEE, and Bing Zheng, Member, IEEE
Abstract—The better way to understand marine life and ecosys-
tems is to surveil and analyze the activities of marine organisms.
Recently, research on marine video surveillance is becoming in-
creasingly popular. With the rapid development of deep learning
(DL), convolutional neural networks (CNNs) have made remark-
able progresses in image/video understanding tasks. In this article,
we explore a visual attention and relation mechanism for marine
organism detection, and propose a new way to apply an improved
attention-relation (AR) module on an efficient marine organism
detector (EMOD), which can well enhance the discrimination of
organisms in complex underwater environments. We design our
EMOD via integrating current state-of-the-art (SOTA) detection
methods in order to detect organisms and surveil marine environ-
ments in a real time and fast fashion for high-resolution marine
video surveillance. We implement our EMOD and AR on the
annotated video data sets provided by the public data challenges in
conjunction with the workshops (CVPR 2018 and 2019), which are
supported by National Oceanic and Atmospheric Administration
(NOAA) and their research works (NMFS-PIFSC-83). Experimen-
tal results and visualizations demonstrate that our application of
AR module is effective and efficient, and our EMOD equipped with
AR modules can outperform SOTA performance on the experimen-
tal data sets. For application requirements, we also provide the
application suggestions of EMOD framework. Our code is publicly
available at https://github.com/zhenglab/EMOD.
Manuscript received 16 December 2020; revised 13 July 2021, 12 November
2021, and 19 January 2022; accepted 23 March 2022. Date of publication 1 June
2022; date of current version 13 October 2022. This work was supported in part
by the National Natural Science Foundation of China under Grant 62171421
and Grant 61771440, and in part by the Qingdao Postdoctoral Applied Research
Project of China. Preliminary work for this paper was presented at the IEEE/MTS
Global OCEANS 2020 Singapore - US Gulf Coast, October 5–31, 2020. [DOI:
10.1109/IEEECONF38699.2020.9389458]. (Zhensheng Shi, Cheng Guan, and
Qianqian Li contributed equally to this work.) (Corresponding authors: Haiyong
Zheng and Zhaorui Gu.)
Associate Editor: B. Thornton.
Zhensheng Shi is with the Frontiers Science Center for Deep Ocean Mul-
tispheres and Earth System, Ocean University of China, Qingdao 266100,
China, and also with the Underwater Vision Lab, College of Electronic
Engineering, Ocean University of China, Qingdao 266100, China (e-mail:
shizhensheng@ouc.edu.cn).
Cheng Guan, Qianqian Li, Ju Liang, Liangjie Cao, Haiyong Zheng, and
Zhaorui Gu are with the Underwater Vision Lab, College of Electronic Engineer-
ing, Ocean University of China, Qingdao 266100, China (e-mail: guancheng@
stu.ouc.edu.cn; liqianqian5957@stu.ouc.edu.cn; liangju@stu.ouc.edu.cn;
caoliangjie@stu.ouc.edu.cn; zhenghaiyong@ouc.edu.cn; guzhaorui@ouc.edu.
cn).
Bing Zheng is with the Underwater Vision Lab, College of Electronic En-
gineering, Ocean University of China, Qingdao 266100, China, and also with
Sanya Oceanographic Institution, Ocean University of China, Sanya 572024,
China (e-mail: bingzh@ouc.edu.cn).
Digital Object Identifier 10.1109/JOE.2022.3162864
1558-1691 © 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: Universidad Nacional de Colombia (UNAL). Downloaded on June 06,2023 at 14:18:36 UTC from IEEE Xplore. Restrictions apply.
959
Index Terms—Convolutional neural network (CNN), marine
organism detection, marine video surveillance, relation model,
visual attention.
I. INTRODUCTION
NDERSTANDING the behavior and abundance distribu-
U tion of marine organisms plays a very important role in
marine ecosystem, environment monitoring, and marine fish-
ery [1]–[3]. Detecting and recognizing marine organisms have
been a very challenging and crucial research topic. To surveil
the activity and master the distribution of marine organisms, re-
searchers have made great efforts to capture them in-situ and on-
site with large amounts of high-quality images or videos using
modern underwater vehicles, such as autonomous underwater
vehicles (AUVs) [4], [5] and remotely operated vehicles (ROVs).
Then, biologists are required to manually annotate and recognize
the species with their positions for further studies from these
large-scale images and videos, which is very time-consuming
and laborious as well as inapplicable to in situ and on-site
real-time analysis. Thus, it is necessary to devise a computer
vision-based method to detect and recognize organisms for
automatic marine video surveillance by using the cutting-edge
artificial intelligence technologies [6]–[8].
Recently, relying on large amounts of annotated data and com-
putational resources [i.e., graphical processing units, (GPUs)],
convolutional neural networks (CNNs) have first witnessed re-
markable progress in image recognition [9]–[12]. Besides, deep
CNNs have been widely applied to more complicated object
detection for images and videos, where the objects are not only
recognized but also marked with locations, achieving amazing
performance and even better results than our human beings in
applications, such as video surveillance [13]–[16]. However,
video surveillance in the marine environment is quite different
from that in the terrestrial environment, and faces several tough
challenges. First, underwater images and videos suffer from
strong absorption, scattering, color distortion, and noises from
artificial light sources as well as marine snow particles, causing
image blur, haziness, and bluish or greenish tone. Second, un-
derwater objects are usually hard to stay static especially when
marine organisms and underwater vehicles are also moving,
resulting in motion blur and various poses of organisms. Third,
underwater imaging environment is complicated and changeable
960
Fig. 1. Visualization of attention maps in AR module on scallop detection from HabCam data set. Our AR module can enhance the representation of organisms
in complex underwater environments. More visualizations are in Fig. 8.
due to the changes of illumination as well as spatio–temporal
scales. Although low-level image enhancement can to some
extent solve these problems and make the underwater image
data clearer, the general image/video understanding algorithms
still struggle to deal with such challenging problems in marine
environment [17], [18].
Visual attention and relation are two fascinating facets for
human intelligence, which have been modeled in CNNs and
proved to be effective for image/video understanding tasks.
Given an image or a video clip, human focus attention selectively
on salient parts of the visual space to acquire useful information
and also discover relations among them [19], [20]. In marine
video understanding tasks, the capture of visual attention and
relation becomes more difficult due to complex underwater envi-
ronments, therefore, how to overcome the effects of underwater
degradation and improve the discrimination of marine organisms
is crucial for the study of underwater attention and relation
modeling. Motivated by this observation, we propose a new
way to apply an improved attention–relation (AR) module on an
efficient marine organism detector (EMOD), which explores to
learn joint AR in CNNs. Our AR module can better understand
underwater specific video by enhancing the representation of
organisms (objects). As Fig. 1 shows, we visualize the attention
maps in AR module on scallop detection from HabCam data
set. It can be seen that, although the organism exists in the
environment with complex noises (e.g., the impurities in the first
column, the blurring and color distortion in the second column,
the haziness of organisms in the third and fourth columns), our
joint AR method can well enhance attentive features and capture
the target organism.
In this work, we use the challenging marine video data sets,
which are provided by the public data challenges [21] in con-
junction with the workshops hosted in IEEE/CVF Conference
on Computer Vision and Pattern Recognition (CVPR 2018 [22]
and 2019 [23] Workshops). The data sets are supported by
National Oceanic and Atmospheric Administration (NOAA)
and their research works (NMFS-PIFSC-83 [24]), and we take
the three sub-data sets for our study: MOUSS, MBARI, and
HabCam. Our AR module aims to study the visual attention
Authorized licensed use limited to: Universidad Nacional de Colombia (UNAL). Downloaded on June 06,2023 at 14:18:36 UTC from IEEE Xplore. Restrictions apply.
IEEE JOURNAL OF OCEANIC ENGINEERING, VOL. 47, NO. 4, OCTOBER 2022
and relation for precisely detecting marine organisms. In addi-
tion, for high-resolution marine video surveillance, we integrate
current state-of-the-art (SOTA) detectors and design an EMOD
framework, which can detect and surveil marine organisms in
a real-time and fast fashion. EMOD will be helpful for marine
scientists to effectively analyze the vast amount of survey data
collected from marine environment. The experimental results
also demonstrate that our application of AR module is effective
and efficient, and our EMOD equipped with AR module can
exceed the performance of SOTA detectors on the experimental
data sets.
A preliminary version of this work has been published in
Global OCEANS 2020: Singapore – U.S. Gulf Coast [25]. The
present work is a developed version of previously presented
version, which significantly adds to the initial work. First, we
propose a new way to apply an improved AR module on our
EMOD framework for exploring AR jointly, and also conduct
ablation study as well as extensive experiments to validate the
efficacy of our AR module. Then, we improved our original
EMOD framework by involving both our AR module and opti-
mal detector for marine video surveillance. Moreover, we pro-
vide suggestions of EMOD framework for different application
requirements.1
II. MARINE VIDEO SURVEILLANCE
In recent years, video surveillance of marine environment
and marine organisms has attracted wide attention and a lot
of related work has been carried out. Hu et al. [26] designed
a system that is capable of categorizing different underwater
habitat types to monitor and analyze the change and damage
of the marine environment. By using time-lapse photography
and a suite of marine instruments, Fujii et al. [27] constructed
a new underwater monitoring system to determine the dynamic
relationship between changing environmental conditions, bio-
logical activity, and the actual presence of marine infrastruc-
ture. Taormina et al. [2] developed and optimized a reliable
1 [Online]. Available: https://github.com/zhenglab/EMOD
SHI et al.: DETECTING MARINE ORGANISMS VIA JOINT AR LEARNING FOR MARINE VIDEO SURVEILLANCE
image scoring strategy to improve the underwater image for
monitoring benthic ecosystems. The purpose of Rose [1] was
to describe the methods and results of acoustic surveys of
Smith Sound over seven years in several seasons, and use these
results to design an acoustic-monitoring method for these fish.
Smale et al. [4] described the application of AUV technology
to conduct ongoing monitoring of benthic habitats at two key
locations in Western Australia. Effective fish sampling method
proposed by Salman et al. [6] was developed by using under-
water video and image processing techniques to automatically
estimate, and, thus, monitor fish biomass and groups in water
bodies.
A. Marine Organism Detection
Acoustic and optical measurements are widely used in the
detection of underwater organisms. There are many acoustic
techniques, from calculating the echo of a sonar beam [28],
[29] to the more complex methods [30]. Tiemann et al. [31]
developed a passive acoustic localization technique based on
a acoustic propagation modeling to monitor marine mammal
activities. Optical techniques include classical methods, such
as edge-based classifiers [32], and fish detector and tracker [33]
that combines an adaptive Gaussian mixture model detector [34]
with a continuously adaptive mean shift algorithm tracker [35].
Mizuno et al. [36] developed a towed optical camera array sys-
tem named speedy sea scanner (SSS) combined with a DL-based
estimation method to monitor benthic marine habitats.
Through fish detection and classification, Villon et al. [37]
studied the performance of an support vector machine (SVM)
classifier trained on HOG features, and compared it with perfor-
mance of fine-tuned CNNs, where the results showed that CNNs
are indeed better than traditional methods. Marburg et al. [38]
used various neural networks to train an underwater detector
and classifier for benthic macrofauna. Siddiqui et al. [39] used
deep-layer CNNs and cross-layer pooling methods to enhance
the discriminative ability to achieve SOTA performance in the
classification of fish species, which can solve the problem of lim-
ited training data for labeling. Lu et al. [40] proposed the identi-
fication of the species of six common tuna and billfish by using
machine vision. Rasmussen et al. [41] utilized YOLOv2 [16]
for scallop detection that was able to run in a real-time way.
Song et al. [42] applied sidescan sonar (SSS) and an onboard
GPU to implement automatic real-time object detection. Aznar
et al. [43] proposed a swarm behavior that was formalized with a
microscopic model to improve jellyfish detection system. Tseng
et al. [44] proposed an automatic approach for prescreening
harvested fish in the electronic monitoring system (EMS) videos
by using CNNs. Li et al. [3] constructed a phytoplankton micro-
scopic image data set, namely, PMID2019, to train the advanced
artificial intelligence model for phytoplankton detection.
B. Visual Attention
Attention, in a broad sense, can be regarded as a useful tool,
which tends to allocate available processing resources to the
most informative part of the input signal [45]–[49]. The benefits
of attention mechanism have been demonstrated in a range of
Authorized licensed use limited to: Universidad Nacional de Colombia (UNAL). Downloaded on June 06,2023 at 14:18:36 UTC from IEEE Xplore. Restrictions apply.
961
tasks, from image recognition and understanding [50], [51], to
sequence-based models [52], [53]. It is usually implemented in
combination with gating functions (such as softmax or sigmoid)
and sequential technology [54], [55]. Recent work has shown its
applicability to tasks, such as image captioning [56], [57] and
lip reading [58]. In these applications, it is often used on top
of one or more layers representing higher level abstractions for
adaptation between modalities.
Besides, Vaswani et al. [59] proposed a transformer architec-
ture based on self-attention mechanism for machine translation,
which calculates the response of a position (such as a sentence)
in the sequence by focusing on all positions and taking their
weighted average in the embedding space. Wang et al. [60]
viewed self-attention as a form of the nonlocal mean [61]
and bridged self-attention for machine translation to the more
general class of nonlocal filtering operations that are applicable
to image and video problems in computer vision.
C. Relation Model
Recently, relation models have been developed in the
area of visual question answering [62], [63], object detec-
tion/recognition [64]–[66], and intuitive physics [67], [68].
Cross-channel correlations are typically mapped as new com-
binations of features, either independent spatial structure [69],
[70] or joint convolutional structure of standard convolutional
filters [71] and 1 × 1 convolutions. Much of this work has
concentrated on the objective of reducing model parameters and
computational complexity, indicating an assumption that chan-
nel relationships can be formulated as a composition of instance-
agnostic functions with local receptive fields. In contrast,
Hu et al. [72] proposed a novel module “Squeeze-and-
Excitation” and provided the net unit with a mechanism to
explicitly model dynamic and nonlinear dependencies between
channels using global information, for easing the learning pro-
cess and enhance the representational power of network. More-
over, GENet [73] and PSANet [74] performed rescaling to
different channels to recalibrate the channel dependency with
global context.
D. Object Detection
The mainstream solutions for object detection usually be-
long to one of these two categories: 1) two-stage detector and
2) one-stage detector.
1) Two-Stage Detector: Current SOTA object detectors are
usually based on a two-stage, proposal-driven mechanism [75].
The first stage generates a sparse set of candidate object loca-
tions, which can be performed by using either basic algorithms
for region proposals or a dedicated region proposal CNN. The
second stage consists of a CNN that classifies each candidate as
one of the object classes or a background class. The two-stage
methods, such as the R-CNN series [76], have achieved SOTA
accuracy scores on many benchmark data sets [77].
Due to the great success of the R-CNN detector [78], two-
stage detector has become popular in recent years [77]. In
order to reduce redundant computation, SPP-Net [79] and Fast
R-CNN [80] introduced an idea of regional feature extraction
962
Fig. 2. Structure and details of our attention–relation (AR) module. We simplify the global attention module and then add channel relation to construct a joint
AR module.
that enabled objects to share a large number of feature compu-
tation. Faster R-CNN [13] proposed region proposal network
to improve the efficiency of detectors and allowed end-to-end
training of detectors. After this meaningful milestone, some
works aimed to enhance faster R-CNN from different details.
For example, R-FCN [81] utilized efficient regionwise full con-
volutions to avoid the heavy computations of faster R-CNN, and
Cascade R-CNN [14] proposed classic and powerful cascade
architecture to extend R-CNN to a multistage detector, while
mask R-CNN [76] added a mask branch that refined the detection
results with the help of multitask learning.
2) One-Stage Detector: A main disadvantage of the two-
stage detectors is that they are time-consuming, which limits
in real-time applications and encourages the development of
one-stage detectors [75], [77]. One-stage detectors classify all
image regions at once, thus are beneficial to computational
efficiency and become popular in some related tasks.
OverFeat [82] was one of the first modern one-stage object
detector based on deep networks. YOLO [83] output very sparse
detection results and implemented real-time object detection by
inputting image once through an efficient backbone network.
SSD [84] was one of the first attempts at using pyramidal feature
hierarchy to reuse the multiscale feature maps from different
layers computed in the forward pass. RetinaNet [15] proposed
a new focal loss to handle the class imbalance problem of dense
object detection, achieving results comparable to two-stage de-
tectors. The main limitation of one-stage detectors is that their
accuracy is usually lower than that of two-stage detectors.
III. LEARNING JOINT ATTENTION–RELATION
We propose a new way to apply an improved AR module
on our EMOD (see Section IV) to learn joint AR in CNNs for
marine organism detection, as shown in Fig. 2. Inspired by [60]
and [72], our AR module adopts nonlocal (NL) combined with
squeeze-excitation (SE) structure to learn the global attention
and relation in marine videos, which includes two submodules
[Fig. 2(d)] as follows: First, we model the global context in-
formation via utilizing a global attention submodule; then, a
bottleneck transform operation is implemented to model the
Authorized licensed use limited to: Universidad Nacional de Colombia (UNAL). Downloaded on June 06,2023 at 14:18:36 UTC from IEEE Xplore. Restrictions apply.
IEEE JOURNAL OF OCEANIC ENGINEERING, VOL. 47, NO. 4, OCTOBER 2022
channel relation among different perspectives. The AR mod-
ule is designed in a residual structure, which can be flexibly
incorporated to many existing CNN architectures. Thanks to the
bottleneck structure that greatly reduces model parameters, AR
module is lightweight which can be applied to multiple layers
for better modeling the joint AR. We employ the good practice
of Cao et al. [85] and implement an improved AR module (see
Section III-B and Table IV for details), where an AR module is
essentially a computational unit with the transformation map-
ping of an input x ∈ RC×H×W , which denotes the feature map
of an input instance (an image)
x = {xi }Pi=1 = [x1 , x2 , . . . , xP ] ∈ R
C×H×W
, xi ∈ RC×1
(1)
where i refers to the index of the query position; C is the channel
number; P = H × W is the number of positions in an input
feature map; and W and H are width and height of the feature
map.
A. Attention
Traditional CNN only focuses on the area of receptive field
size. Although the receptive field can be increased by stacking
convolutional layers, the receptive field of convolutional kernel
in a specific layer on the original image is still very limited.
In order to obtain global receptive field and extract more infor-
mation from the original image, Wang et al. [60] proposed a
self-attention model as the nonlocal (NL) block [Fig. 2(a)] to
capture the global attention. The core nonlocal operation can be
expressed as
P
1 
oi = f (xi , xj )g(xj ) (2)
C(x) j=1
where x and o represent the input signal and output signal,
respectively, f (xi , xj ) is used to compute the similarity between
position i and position j, C(x) represents the normalization
operation, and g(xj ) denotes the linear transformation of xj . In
this way, we actually obtain the probability (oi ) that represents
the effect of all positions on query position i, indicating the
global attention. Then the nonlocal block [60] can be defined in
SHI et al.: DETECTING MARINE ORGANISMS VIA JOINT AR LEARNING FOR MARINE VIDEO SURVEILLANCE 963

a residual structure as utilizes two adjacent fully connected layers to form a bottle-
P neck structure for modeling the relations between channels, and
 f (xi , xj )
zi = xi + Wz oi = xi + Wz g(xj ). (3) outputs the same number of channels as the input features. We
j=1
C(x) define our relation operation as

In particular, there are four different forms [60] to calculate R(·) = σ(Wc2 (δ(Wc1 (·)))) (9)
f (xi , xj ), here, we set f (xi , xj ) = exp((Wq xi )T (Wk xj )) and

C(x) = P j=1 f (xi , xj ), which are commonly used in self-
where Wc1 and Wc2 represent the two fully connected layers
attention module [59], [86], [87]. Similarly, we also employ the of MLP, δ(·) denotes ReLU activation function and σ· denotes
1 × 1 convoluational filter Wv as linear transformation matrix to Sigmoid activation function.
construct g(xj ) = Wv · xj . So the detailed nonlocal block can Thus, our AR residual framework (7) can be illustrated as
be represented as [Fig. 2(a)]
zi = x i
P
 T
exp((Wq xi ) (Wk xj )) ⎛ ⎛ ⎛⎛ ⎞⎞⎞⎞
 P
z i = x i + Wz P (Wv · xj ). exp(Wk x j )
j=1 m=1 exp((Wq xi )T (Wk xm )) + σ ⎝Wc2 ⎝δ ⎝Wc1 ⎝ P xj ⎠⎠⎠⎠.
(4) j=1 m=1 exp(Wk xm )
Further, Cao et al. [85] found that the attention maps for (10)
different query positions are almost the same so that the global
attention operation can be simplified by computing a global Note that, our improved AR module adopts the standard
(query-independent) attention map and sharing this global at- squeeze-excitation (SE) [72] structure (FC-ReLU-FC-Sigmoid)
tention map for all query positions. By doing so, Wz can be to learn relations across multiple channel perspectives in marine
dropped since the channel number will not be changed. video data set, while Cao et al. [85] only used Conv layers in their
Here, we also follow these observations and simplify the NL module and directly fused these convolutional features without
block to [Fig. 2(b)] capturing relations. We show the improved performance of our
AR module in Table IV.
P
 exp(Wk xj )
zi = x i + P (Wv · xj ) (5)
j=1 m=1 exp(Wk xm ) IV. EFFICIENT MARINE ORGANISM DETECTOR

where Wq and Wz are dropped from (4). Then, for further A. EMOD Framework
reducing the computation cost, according to distributive law, Our new way for the application of AR mechanism is to
the simplified NL block can be modified as [Fig. 2(c)] employ the AR module (Section III) on our designed EMOD
P
framework. The EMOD can detect and recognize marine or-
 exp(Wk xj ) ganisms from high-resolution marine surveillance videos in an
z i = x i + Wv P xj . (6)
j=1 m=1 exp(Wk xm ) efficient and fast way. The EMOD framework integrates current
SOTA detectors and consists of three parts as follows: The
The simplified nonlocal block can be viewed as a residual backbone (basic network), the SOTA detector, and the AR (joint
structure with two parts: 1) attention, which calculates a query- attention-relation) module. The backbone includes three kinds of
independent global attention map for all query positions; and basic networks: 1) ResNet-50, 2) ResNext-101, and 3) DarkNet-
2) relation, which captures the relation between channels. We 53. The SOTA detector integrates two-stage and single-stage
abstract the simplified NL block as an AR residual framework detection methods to complete organism detection based on
with the following definition [Fig. 2(d)]: marine surveillant videos. The two-stage detectors can achieve
high-precision detection performance, which are based on faster
zi = xi + R(y) (7)
R-CNN (FR, [13]) and Cascade R-CNN (CR, [14]), and one-
where y represents the output of attention module, which obtains stage detectors can achieve high-efficiency real-time detection
the global attention by grouping the features of all positions; and performance, which are based on YOLO (YL) and RetinaNet
R(·) denotes the relation module, which models channelwise (RN). By equipping with AR module, the EMOD can learn joint
dependencies. attention-relation from complex marine video contents, so as to
For the attention module, based on (6), the output y can be further improve the accuracy of marine organism detection. We
expressed as implement the AR module on EMOD framework via plugging
AR modules into every convolutional block (such as res block,
P
 exp(Wk xj ) see Fig. 3) of the backbone network. Experiments show that the
y= P xj . (8) EMOD framework can achieve high-efficiency, real-time, and
j=1 m=1 exp(Wk xm )
accurate detection results of marine organisms. We note that our
EMOD is an extensible framework, the experiments of this work
B. Relation
are based on the current version. We will enrich the framework in
In order to model channelwise feature dependencies, we em- future versions by integrating more backbones, SOTA detectors,
ploy a two-layer MLP to construct the relation module, which and marine video methods (such as AR module).

Authorized licensed use limited to: Universidad Nacional de Colombia (UNAL). Downloaded on June 06,2023 at 14:18:36 UTC from IEEE Xplore. Restrictions apply.
964
Fig. 3. Framework of our efficient marine organism detector (EMOD).
B. Backbone
1) ResNet-50 or R50 [12]: ResNet has become one of the
most popular CNN networks with its simple structure and ex-
cellent effect, which effectively solves the degradation problem
caused by the deepening of network layers. Due to moderate
network depth and complexity, ResNet-50 is popular and com-
monly used in research and application, and it contains a total of
five convolutional layers, in which the second to fifth layers are
residual learning layers that are, respectively, composed of 3, 4,
6, and 3 residual units and each residual unit adopts a bottleneck
structure that contains 1 × 1, 3 × 3, and 1 × 1 convolutional
layers.
2) ResNeXt-101 or X101 [88]: ResNeXt is an improvement
based on ResNet and inception models [11], [89]–[91], which
gains higher accuracy in the way of increasing cardinality (con-
volutional groups) instead of going deeper or wider, while reduc-
ing complexity. ResNeXt-101 (64 × 4 d) performs excellently in
ResNeXt series and achieves better accuracy than ResNet-200
with only 50% complexity. In the bottleneck convolutional layer,
the cardinality is set to 64, which means the channels are divided
into 64 groups for convolutional operation.
3) DarkNet-53 or D53 [83]: DarkNet is a lightweight and
efficient convolutional network for YOLO detector. DarkNet-53
is first proposed in YOLOv3 and consisted of 53 basic convo-
lutional units, which are composed of convolutional layers of
1 × 1 and 3 × 3 in a residual structure.
C. SOTA Detector
1) Faster R-CNN or FR [13]: Faster R-CNN (faster region-
based CNN) is an improvement of fast R-CNN [80] which first
adopts region proposal network (RPN) to generate high-quality
region proposals, and then trains a separate detection network
by fast R-CNN via using the proposals generated by RPN.
Authorized licensed use limited to: Universidad Nacional de Colombia (UNAL). Downloaded on June 06,2023 at 14:18:36 UTC from IEEE Xplore. Restrictions apply.
IEEE JOURNAL OF OCEANIC ENGINEERING, VOL. 47, NO. 4, OCTOBER 2022
Furthermore, FR uses the detection network to initialize RPN
training and fixes the shared convolutional layer to, respectively,
fine-tune the unique layers of RPN and fast R-CNN. In this way,
the two networks form a unified network sharing full-image
convolutional features and the step for region proposal is nearly
cost-free which greatly improves detection speed.
2) Cascade R-CNN or CR [14]: To address the problem
that a detector with low intersection over union (IoU) threshold
usually produces noisy detections, while detection performance
tends to degrade with increasing the IoU thresholds, CR is pro-
posed in a multistage structure, which trains stages sequentially
by using the output of one stage to train the next with increasing
IoU thresholds, and the same cascade procedure is applied at
inference. This architecture is shown to avoid the problem of
overfitting at training and quality mismatching at inference.
3) YOLO or YL [92]: Different from FR and CR, YL is
a one-stage detector without the stage of generating region
proposals. In our EMOD framework we adopt YLv3 as YL
detector. Compared to YLv1 and YLv2, YLv3 uses a new net-
work (DarkNet-53) for performing feature extraction and adopts
independent logistic classifiers instead of using a softmax as well
as employing binary cross-entropy loss for class predictions. For
determining bounding box priors, YLv3 still uses k-means clus-
tering and then chooses nine clusters and three scales arbitrarily
and divides up the clusters evenly across scales. After these
improvements, YLv3 achieves significant lift for small objects
detection, which has been a challenge for previous YL versions.
However, for medium and larger size object detection, the results
are comparatively worse.
4) RetinaNet or RN [15]: RN is a fully convolutional one-
stage detector designed to demonstrate the efficacy of the fo-
cal loss, which is proposed to solve the extreme foreground–
background class imbalance as it is considered as the primary
obstacle preventing one-stage object detectors from surpassing
SHI et al.: DETECTING MARINE ORGANISMS VIA JOINT AR LEARNING FOR MARINE VIDEO SURVEILLANCE
top-performing. Compare to YLv2, RN achieves a higher de-
tection accuracy and no matter for detecting small-, medium-,
or large-size objects, it makes a considerable improvement on
mean AP.
V. EXPERIMENTS
A. Data Sets
We conduct experiments on our proposed AR module and
EMOD framework with annotated video data sets, which show
many species of fish, shellfish, underwater plants, and other
biota in Fig. 4. Each data set contains different types of im-
ages, such as different lighting conditions, camera angles, and
wildlife.
We adopt the default “coarse-bbox-only” flavor of the data
set as our COCO-style annotations. We use the bounding box
annotated data sets (MOUSS, MBARI, HabCam) to evaluate our
proposed method, and show the experimental results.
The MOUSS [21] data sets include two sequences, MOUSS
seq0 (MOUSS0) and MOUSS seq1 (MOUSS1). MOUSS0
consists of 194 images belonging to the same category (Car-
charhiniformes) and the images have a resolution of 968 × 728.
MOUSS1 consists of 241 images within one category (Perci-
formes) and the images have a resolution of 720 × 480.
The MBARI [21] data set contains a single video consisting of
740 RGB frames with six classes. Each image has a 1920 × 1080
resolution.
The HabCam [21] data set contains objects, such as scallops,
sand dollars, rocks, sand, and the occasional fish. The annota-
tions include scallops and fish. There are 52 344 images with
2720 × 1024 of 11 classes in the data set.
B. Evaluation Metrics
Object detection requires to evaluate model performance of
both classification and location, and each image may have differ-
ent objects of different categories. Therefore, the standard evalu-
ation metrics used in the image classification problem cannot be
directly applied to object detection task. The evaluation metric
of COCO [93] is average precision (AP), which is calculated as
area under precision-recall curves across IoU thresholds from
0.5 to 0.95, with an interval of 0.05. All reported results of our
work follow standard COCO-style AP metrics that include AP
(averaged over IoU thresholds), AP50 (AP for IoU threshold
50%), and AP75 (AP for IoU threshold 75%). We also include
APS , APM , and APL , which correspond to the results on small,
medium, and large scales, respectively. In order to measure
the detection speed of different detectors, we use frames per
second (FPS), which means the number of video frames that the
model can process per second (on NVIDIA GTX 1080Ti GPU),
and is calculated as the detector’s speed indicator in the testing
process.
C. Implementation Details
We perform cross validation on the annotated training data
of the data sets. We randomly divide the data sets into three
splits, and each split includes a training set and a validation
Authorized licensed use limited to: Universidad Nacional de Colombia (UNAL). Downloaded on June 06,2023 at 14:18:36 UTC from IEEE Xplore. Restrictions apply.
965
set with a ratio of 4:1. We report the results by averaging over
all three splits. For the division of training/validation, we select
every ten consecutive frames according to the sequence of video
frames, and then randomly assign the ten frames to training and
validation set in a 4:1 ratio. In this way, the target organisms from
different temporal frames and spatial locations can be evenly
divided into each set, which ensures that the division of data
set can cover the diversity of original data. For the selection of
the splits, we use FR-R50 detector to conduct experiments on
multiple randomly generated splits. According to the principle
of high results and small fluctuation range (standard deviation),
we select the optimal three splits to ensure the reliability of
experimental results. Table I shows the results of the three
selected splits, we can see that, the values of standard deviation
for each data set are in a reasonable scope. Therefore, we use
these three splits of the four data sets for our experiments in the
following sections.
Besides, we further note that, we obtain the valuable data from
the data challenge [21] in conjunction with CVPR 2018 [22] and
2019 [23] Workshops, the division of the data sets refers to the
division of training and testing in the challenge, and our way
of the division also accords to the purpose and definition of
the data set provider. Meanwhile, due to the data insufficiency
and the particularity of marine video surveillance tasks, that is
what we strive to do to divide the data sets and experiment the
models based on the obtained data in order to realize a useful
application. Thus, it is still a challenging yet important future
work for collecting data sets with more diversity and dynamicity
for marine video surveillance.
Moreover, we take MOUSS1 and conduct two experiments to
further analyze the effectiveness of our data set division and
proposed method: 1) We randomly selected 50 frames from
the unlabeled testing set (provided by [21]), and then manually
annotated classes and bounding boxes. Our FR-R50 model still
worked well by 63.9 mAP (the result is 67.5 mAP in our
validation set), which indicates that our model also fits to the
data of testing set. 2) We adopted another division, which used
the first 4/5 frames as training and the last 1/5 as validation. We
observed that it led to a performance degradation (35.0 mAP
for validation and 48.2 mAP for testing by FR-R50 model), the
main reason might be that the division does not fully cover the
diversity of contents in the limited data. Furthermore, we also
experimented our AR module on this division, it still worked
well on both ablation study (35.8 versus 35.0 mAP, FR-R50
model) and SOTA comparison (38.5 versus 37.3 mAP, CR-X101
model), which can also demonstrate the effectiveness of our
proposed method.
In this work, we train models on training set, and report the
results of baseline models, ablation studies, and SOTA models
on validation set, and also additionally provide standard devi-
ation (std) of AP for reference. For two-stage detectors, 256
anchors are sampled by RPN per image with 1:1 ratio of positive
to negative anchors where the IoU threshold of positive and
negative anchors are 0.7 and 0.3, respectively, and span five
scales and three aspect ratios. In classification branch, region
proposals extracted by region proposal network have an overlap
with ground truth greater than 0.5 are regarded as positive
966
Fig. 4. Examples from three data sets of marine video surveillance.
TABLE I
RESULTS FOR THE DIVISION OF DATA SETS
samples. For fair comparison, all experiments are implemented
on PyTorch framework (version 1.3). We use COCO data set
for pretraining models and employ a weight decay of 0.0001, a
momentum of 0.9 and standard horizontal flipping with a ratio
of 0.5. All the experiments are implemented on two NVIDIA
GTX 1080Ti GPUs (two images per GPU) for 12 epochs with
an initial learning rate of 0.005, and decrease it by 0.1 after
8 and 11 epochs, respectively. The code of our EMOD frame-
work is publicly available, which can be directly used in marine
organism monitoring research.
D. EMOD Performance
We first conduct experiments on three data sets (four subsets)
to study the performance of EMOD with four SOTA detectors
(FR, CR, YL, and RN) and three backbones (R50, X101, and
D53), and the results are shown in Table II, where the detectors
FR, CR, and RN use two backbones R50 and X101, while
detector YL employs D53 backbone. In order to better present
the performance of each detector, we also provide the scatter
diagrams with x-axis and y-axis as FPS and AP, respectively, in
Fig. 5. It can be seen that:
1) in Fig. 5, the detection results of two-stage detectors (FR
and CR) are higher than those of one-stage detectors (YL
and RN), indicating that the performance of two-stage
detector is generally better than that of single-stage detec-
tor. In addition, the detection results of one-stage detector
are on the right side of two-stage detectors, showing a
higher efficiency. In particular, the accuracy of CR is the
best and the efficiency of YL is the highest. However,
there also exists an exception, i.e., in the experiments on
MOUSS0, the detection accuracy of two-stage detector
FR is not better than that of one-stage detectors RN and
YL, which might due to the small data set of MOUSS0
Authorized licensed use limited to: Universidad Nacional de Colombia (UNAL). Downloaded on June 06,2023 at 14:18:36 UTC from IEEE Xplore. Restrictions apply.
IEEE JOURNAL OF OCEANIC ENGINEERING, VOL. 47, NO. 4, OCTOBER 2022
tending to overfitting or underfitting with unstable results
of models (especially deep models). Analyzing the same
detector with different backbones on these three data sets,
we can conclude that the detector with X101 as backbone
has higher accuracy than that with R50 as backbone,
illustrating the efficacy of deeper models;
2) CR detector is the first choice with better performance for
the application scenarios that require high detection accu-
racy. The two-stage detection framework and the improved
cascade structure of CR detector effectively improves
the performance of FR. By comprehensive comparison,
CR-X101 is the most accurate detection configuration;
3) YL detector is the first choice with higher efficiency for
application scenarios that require fast-speed detection or
real-time detection. According to the comprehensive com-
parison of FPS results, the processing speed of YL-D53
detector is faster than that of other detectors, which can
reach the online real-time detection with an acceptable
detection accuracy;
4) for application scenarios that require consideration of both
accuracy and efficiency, RN detector is a good choice.
As a one-stage detector, RN can achieve a competitive
performance compared to two-stage detectors, which is
more efficient. Experimental results show that the detector
RN can balance accuracy and efficiency well.
E. Ablation Study for AR Module
We conduct ablation study to validate the efficacy of our AR
module, and show comparisons with other AR modules (NL
block [60], SE module [72], and GCNet. [85]). Considering
the tradeoff between accuracy and efficiency, we use FR-R50
detector as the baseline model to study the efficacy of our AR
module as well as its components: Attention (A, simplified NL
SHI et al.: DETECTING MARINE ORGANISMS VIA JOINT AR LEARNING FOR MARINE VIDEO SURVEILLANCE 967

TABLE II
EXPERIMENTAL RESULTS OF EMOD PERFORMANCE ON THREE DATA SETS

Authorized licensed use limited to: Universidad Nacional de Colombia (UNAL). Downloaded on June 06,2023 at 14:18:36 UTC from IEEE Xplore. Restrictions apply.
968 IEEE JOURNAL OF OCEANIC ENGINEERING, VOL. 47, NO. 4, OCTOBER 2022

TABLE III
ABLATION STUDY FOR AR MODULE

TABLE IV
COMPARATIVE RESULTS OF OUR AR MODULE AND CAO ET AL. [85]

block [60]) and Relation (R, SE module [72]). To clearly observe
the effect of AR module, we also present the scatter diagrams
with x-axis as FPS and y-axis as AP in Fig. 6, and the detailed
results are shown in Table III. It can be seen that:
1) both AR module and its component A or R can improve the
detection performance, indicating that the proposed visual
attention and relation method is effective for detecting
organisms from marine surveillant videos;
2) specifically, a model achieves almost the same perfor-
mance with A or R module, and neither of them works
better than the model with AR module, also the improve-
ment is almost twice of the AR module compared to single
A or R module. It also indicates that, through combining
A and R for joint attention–relation, our AR module is the
improved implementation for current AR modules (NL
block [60] and SE module [72]);
3) the comparative experiments are carried out on the basis
of feature pyramid network (FPN) [94] module, which
is an important module in FR that has been proved to
improve detection performance significantly, while, our

Authorized licensed use limited to: Universidad Nacional de Colombia (UNAL). Downloaded on June 06,2023 at 14:18:36 UTC from IEEE Xplore. Restrictions apply.
SHI et al.: DETECTING MARINE ORGANISMS VIA JOINT AR LEARNING FOR MARINE VIDEO SURVEILLANCE 969

TABLE V
RESULTS OF STATE-OF-THE-ART COMPARISON

Authorized licensed use limited to: Universidad Nacional de Colombia (UNAL). Downloaded on June 06,2023 at 14:18:36 UTC from IEEE Xplore. Restrictions apply.
970
Fig. 5. Scatter diagrams of EMOD performance on three data sets.
Fig. 6. Scatter diagrams of ablative results on AR module.
Fig. 7. Scatter diagrams of SOTA Comparison.
AR module and its components can further boost the
performance, showing the flexibility and effectiveness of
our design.
Furthermore, we show performance comparison of our AR
module and the module designed by Cao et al. [85] in Ta-
ble IV. We also use FR-R50 as the base detector, and report
AP value on each data set. We can see that our AR module
performs better, which indicates that our improvements for
learning relations across multiple channel perspectives is ef-
fective for marine video data sets. (See Section III-B for more
details).
Authorized licensed use limited to: Universidad Nacional de Colombia (UNAL). Downloaded on June 06,2023 at 14:18:36 UTC from IEEE Xplore. Restrictions apply.
IEEE JOURNAL OF OCEANIC ENGINEERING, VOL. 47, NO. 4, OCTOBER 2022
F. State-of-the-art Comparison
We finally conduct experiments to show SOTA comparison of
our EMOD+AR models. We adopt X101 backbone for detectors
FR, CR and RN, and D53 backbone for detector YL, due to their
outstanding performance in baseline experiments (Section V-D).
Similarly, the results are shown in Fig. 7 and Table V, which
illustrate that:
1) our AR module can further improve the performance of
EMOD, and outperform the SOTA detectors;
2) while our AR module can achieve performance gains with
slight decrease of FPS, indicating the lightweight design;
SHI et al.: DETECTING MARINE ORGANISMS VIA JOINT AR LEARNING FOR MARINE VIDEO SURVEILLANCE 971

Fig. 8. Detailed visualization of scallop detection on HabCam data set. We visualize the input images, backbone w/o. and w/. AR modules, and detection results.
It can be seen that our proposed AR module can enhance the representation of organisms obviously and bring benefits to marine organism detection.

3) the results provide valuable suggestions for model choice
considering different requirements, e.g., EMOD+AR
(CR) model is a good choice for situations that focus more
on accuracy, while EMOD+AR (YL) model is better to
preferentially care about the efficiency.
We note that, the SOTA comparison of Table V is based on
the current version of EMOD framework. We will integrate more
SOTA detectors and backbones in future version. Meanwhile, the
AR module will be implemented to the updated version, and will
be improved to a higher SOTA performance based on stronger
SOTA detectors.
G. Visualization of AR Module
Fig. 8 shows detailed visualization on HabCam data set,
and each row represents an instance of scallop detection. We
experiment the data set on EMOD+AR (CR-R50) model, and
provide attention maps of last residual layer from EMOD w/o.
AR (column 2) and w/. AR (column 3), and also the detection
results (column 4) and ground truth (column 5). The input
instances are affected by various underwater environments, e.g.,
lots of impurities in the first instance, the blurring and color
distortion in the second instance, the haziness of organisms in the
third and fourth instances. It can be obviously seen that, our AR
module can enhance the representation of organisms, and bring
benefits to marine organism detection for all cases. Furthermore,
even in the case of more target objects and more impurities (the
last instance), our AR module is still able to accurately detect all
target organisms in this more complex underwater environments.
VI. CONCLUSION
In this work, we propose a new way to apply an improved
AR module on an EMOD, which explores to learn joint AR
in CNNs for detecting marine organisms. In order to better
surveil the behaviors and understand the distribution of marine
organisms, our designed EMOD framework, which is equipped
with our AR modules can detect and surveil marine organisms
for high-resolution marine video surveillance in a real-time and
fast fashion. We integrate both one-stage and two-stage detection
methods to train the organism detector based on high resolution
data sets of marine videos. Experimental results and visualiza-
tions demonstrate that our application of AR module is effective
and efficient, and our EMOD can further boost the performance
based on SOTA methods. We hope our proposed AR module
and EMOD framework, which are publicly available, can benefit
for the research of marine video surveillance as well as marine
biology.

Authorized licensed use limited to: Universidad Nacional de Colombia (UNAL). Downloaded on June 06,2023 at 14:18:36 UTC from IEEE Xplore. Restrictions apply.
972
ACKNOWLEDGMENT
The authors would like to thank NOAA (NMFS-PIFSC-83)
and Kitware (viametoolkit.org) for providing the data and anno-
tations of the marine video data sets.
REFERENCES
[1] G. A. Rose, “Monitoring coastal northern cod: Towards an optimal sur-
vey of smith sound Newfoundland,” ICES J. Mar. Sci., vol. 60, no. 3,
pp. 453–462, 2003.
[2] B. Taormina et al., “Optimizing image-based protocol to monitor
macroepibenthic communities colonizing artificial structures,” ICES J.
Mar. Sci., vol. 77, no. 2, pp. 835–845, 2020.
[3] Q. Li et al., “Developing a microscopic image dataset in support of
intelligent phytoplankton detection using deep learning,” ICES J. Mar.
Sci., vol. 77, no. 4, pp. 1427–1439, 2020.
[4] D. A. Smale et al., “Regional-scale benthic monitoring for ecosystem-
based fisheries management (EBFM) using an autonomous underwater
vehicle (AUV),” ICES J. Mar. Sci., vol. 69, no. 6, pp. 1108–1118, 2012.
[5] S. B. Williams, O. Pizarro, M. How, D. Mercer, and R. Hanlon, “Surveying
noctural cuttlefish camouflage behaviour using an AUV,” in Proc. IEEE
Int. Conf. Robot. Automat., 2009, pp. 214–219.
[6] A. Salman et al., “Automatic fish detection in underwater videos by a deep
neural network-based hybrid motion learning system,” ICES J. Mar. Sci.,
vol. 77, no. 4, pp. 1295–1307, 2020.
[7] A. Mahmood et al., “Automatic detection of western rock lobster using
synthetic data,” ICES J. Mar. Sci., vol. 77, no. 4, pp. 1308–1317, 2020.
[8] C. S. Tan, P. Y. Lau, P. L. Correia, and A. Campos, “Automatic analysis of
deep-water remotely operated vehicle footage for estimation of Norway
lobster abundance,” Frontiers Inf. Technol. Electron. Eng., vol. 19, no. 8,
pp. 1042–1055, 2018.
[9] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learn-
ing applied to document recognition,” Proc. IEEE, vol. 86, no. 11,
pp. 2278–2324, Nov. 1998.
[10] K. Simonyan and A. Zisserman, “Very deep convolutional networks for
large-scale image recognition,” 2014, arXiv:1409.1556.
[11] C. Szegedy et al., “Going deeper with convolutions,” in Proc. IEEE/CVF
Conf. Comput. Vis. Pattern Recognit., 2015, pp. 1–9.
[12] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
recognition,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit.,
2016, pp. 770–778.
[13] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time
object detection with region proposal networks,” in Proc. Adv. Neural Inf.
Process. Syst., 2015, pp. 91–99.
[14] Z. Cai and N. Vasconcelos, “Cascade R-CNN: High quality object detec-
tion and instance segmentation,” IEEE Trans. Pattern Anal. Mach. Intell.,
vol. 43, no. 5, pp. 1483–1498, May 2020.
[15] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for
dense object detection,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2017,
pp. 2980–2988.
[16] J. Redmon and A. Farhadi, “YOLO9000: Better, faster, stronger,” in Proc.
IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2017, pp. 7263–7271.
[17] C. Beyan and H. I. Browman, “Setting the stage for the machine in-
telligence era in marine science,” ICES J. Mar. Sci., vol. 77, no. 4,
pp. 1267–1273, 2020.
[18] K. Malde, N. O. Handegard, L. Eikvil, and A.-B. Salberg, “Machine
intelligence and the data-driven future of marine science,” ICES J. Mar.
Sci., vol. 77, no. 4, pp. 1274–1285, 2020.
[19] C. Kemp and J. B. Tenenbaum, “The discovery of structural form,” Proc.
Nat. Acade. Sci., vol. 105, no. 31, pp. 10687–10692, 2008.
[20] J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. Lawrence
Zitnick, and R. Girshick, “CLEVR: A diagnostic dataset for compositional
language and elementary visual reasoning,” in Proc. IEEE/CVF Conf.
Comput. Vis. Pattern Recognit., 2017, pp. 2901–2910.
[21] K. Inc, “CVPR 2018 workshop data challenge,” Accessed: Sep. 15, 2019.
[Online]. Available: http://www.viametoolkit.org/cvpr-2018-workshop-
data-challenge/
[22] “CVPR Workshop: Automated analysis of marine video for environmental
monitoring,” 2018. Accessed: Sep. 15, 2019. [Online]. Available: http:
//www.viametoolkit.org/cvpr-2018-workshop/
Authorized licensed use limited to: Universidad Nacional de Colombia (UNAL). Downloaded on June 06,2023 at 14:18:36 UTC from IEEE Xplore. Restrictions apply.
IEEE JOURNAL OF OCEANIC ENGINEERING, VOL. 47, NO. 4, OCTOBER 2022
[23] “CVPR Workshop: Automated analysis of marine video for environmental
monitoring,” 2019. Accessed: Sep. 15, 2019. [Online]. Available: https:
//www.aamvem.com/
[24] B. L. Richards et al., “Automated analysis of underwater imagery: Ac-
complishments, products, and vision,” Nat. Ocean. Atmospheric Ad-
min., Washington, DC, USA, NOAA Tech. Memo NMFS PIFSC; 83,
2019.
[25] Z. Shi et al., “Detecting organisms for marine video surveillance,”
in Proc. Glob. OCEANS: Singapore-U.S. Gulf Coast, 2020, pp. 1–7,
doi: 10.1109/IEEECONF38699.2020.9389458.
[26] J. Hu, H. Zhang, A. Miliou, T. Tsimpidis, H. Thornton, and V. Pavlovic,
“Categorization of underwater habitats using dynamic video textures,” in
Proc. IEEE/CVF Int. Conf. Comput. Vis. Workshops, 2013, pp. 838–843.
[27] T. Fujii and A. J. Jamieson, “Fine-scale monitoring of fish movements and
multiple environmental parameters around a decommissioned offshore
oil platform: A pilot study in the north sea,” Ocean Eng., vol. 126,
pp. 481–487, 2016.
[28] H. Balk and T. Lindem, “Improved fish detection in data from split-beam
sonar,” Aquatic Living Resour., vol. 13, no. 5, pp. 297–303, 2000.
[29] Y. Xie, G. Cronkite, and T. J. Mulligan, A Split-Beam Echosounder
Perspective on Migratory Salmon in the Fraser River: A Progress Report
on the Split-Beam Experiment At Mission, BC, in 1995, vol. 11. Vancouver,
VC, Canada, British Columbia: Pacific Salmon Commission, 1997.
[30] J. A. Holmes, G. M. Cronkite, H. J. Enzenhofer, and T. J. Mulligan,
“Accuracy and precision of fish-count data from a ‘dual-frequency iden-
tification sonar’(DIDSON) imaging system,” ICES J. Mar. Sci., vol. 63,
no. 3, pp. 543–555, 2006.
[31] C. O. Tiemann, S. W. Martin, and J. Mobley, “Aerial and acoustic marine
mammal detection and localization on navy ranges,” IEEE J. Ocean. Eng.,
vol. 31, no. 1, pp. 107–119, Jan. 2006.
[32] G. Shrivakshan and C. Chandrasekar, “A comparison of various edge
detection techniques used in image processing,” Int. J. Comput. Sci. Issues,
vol. 9, no. 5, 2012, Art. no. 269.
[33] C. Spampinato, Y.-H. Chen-Burger, G. Nadarajan, and R. B. Fisher, “De-
tecting, tracking and counting fish in low quality unconstrained underwater
videos,” in Proc. 3rd Int. Conf. Comput. Vis. Theory Appl., vol. 2, 2008,
pp. 514–519.
[34] Z. Zivkovic, “Improved adaptive Gaussian mixture model for background
subtraction,” in Proc. Int. Conf. Pattern Recognit., vol. 2, 2004, pp. 28–31.
[35] K. Fukunaga, Introduction to Statistical Pattern Recognition. Amsterdam,
The Netherlands: Elsevier, 2013.
[36] K. Mizuno et al., “Development of an efficient coral-coverage estimation
method using a towed optical camera array system [speedy sea scanner
(SSS)] and deep-learning-based segmentation: A sea trial at the Kujuku-
Shima Islands,” IEEE J. Ocean. Eng., vol. 45, no. 4, pp. 1386–1395,
Oct. 2020.
[37] S. Villon, M. Chaumont, G. Subsol, S. Villéger, T. Claverie, and D.
Mouillot, “Coral reef fish detection and recognition in underwater videos
by supervised machine learning: Comparison between deep learning and
hog SVM methods,” in Proc. Int. Conf. Adv. Concepts Intell. Vis. Syst.,
2016, pp. 160–171.
[38] A. Marburg and K. Bigham, “Deep learning for benthic fauna identifi-
cation,” in Proc. IEEE/MTS OCEANS Conf., Monterey, CA, USA, 2016,
pp. 1–5.
[39] S. A. Siddiqui et al., “Automatic fish species classification in underwater
videos: Exploiting pre-trained deep neural network models to compensate
for limited labelled data,” ICES J. Mar. Sci., vol. 75, no. 1, pp. 374–389,
2018.
[40] Y.-C. Lu, C. Tung, and Y.-F. Kuo, “Identifying the species of harvested
tuna and Billfish using deep convolutional neural networks,” ICES J. Mar.
Sci., vol. 77, no. 4, pp. 1318–1329, 2020.
[41] C. Rasmussen, J. Zhao, D. Ferraro, and A. Trembanis, “Deep census:
AUV-based scallop population monitoring,” in Proc. IEEE/CVF Int. Conf.
Comput. Vis. Workshops, 2017, pp. 2865–2873.
[42] Y. Song, B. He, and P. Liu, “Real-time object detection for AUVs using self-
cascaded convolutional neural networks,” IEEE J. Ocean. Eng., vol. 46,
no. 1, pp. 56–67, Jan. 2021.
[43] F. Aznar, M. Pujol, and R. Rizo, “A swarm behaviour for jellyfish bloom
detection,” Ocean Eng., vol. 134, pp. 24–34, 2017.
[44] C.-H. Tseng and Y.-F. Kuo, “Detecting and counting harvested fish
and identifying fish types in electronic monitoring system videos using
deep convolutional neural networks,” ICES J. Mar. Sci., vol. 77, no. 4,
pp. 1367–1378, 2020.
[45] L. Itti and C. Koch, “Computational modelling of visual attention,” Nature
Rev. Neurosci., vol. 2, no. 3, pp. 194–203, 2001.
SHI et al.: DETECTING MARINE ORGANISMS VIA JOINT AR LEARNING FOR MARINE VIDEO SURVEILLANCE
[46] L. Itti, C. Koch, and E. Niebur, “A model of saliency-based visual attention
for rapid scene analysis,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 20,
no. 11, pp. 1254–1259, Nov. 1998.
[47] H. Larochelle and G. E. Hinton, “Learning to combine foveal glimpses
with a third-order Boltzmann machine,” in Proc. Adv. Neural Inf. Process.
Syst., 2010, pp. 1243–1251.
[48] V. Mnih et al., “Recurrent models of visual attention,” in Proc. Adv. Neural
Inf. Process. Syst., 2014, pp. 2204–2212.
[49] B. A. Olshausen, C. H. Anderson, and D. C. Van Essen, “A neurobiological
model of visual attention and invariant pattern recognition based on dy-
namic routing of information,” J. Neurosci., vol. 13, no. 11, pp. 4700–4719,
1993.
[50] C. Cao et al., “Look and think twice: Capturing top-down visual attention
with feedback convolutional neural networks,” in Proc. IEEE/CVF Conf.
Comput. Vis. Pattern Recognit., 2015, pp. 2956–2964.
[51] M. Jaderberg et al., “Spatial transformer networks,” in Proc. Adv. Neural
Inf. Process. Syst., 2015, pp. 2017–2025.
[52] T. Bluche, “Joint line segmentation and transcription for end-to-end hand-
written paragraph recognition,” in Proc. Adv. Neural Inf. Process. Syst.,
2016, pp. 838–846.
[53] A. Miech, I. Laptev, and J. Sivic, “Learnable pooling with context gating
for video classification,” 2017, arXiv:1706.06905.
[54] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural
Comput., vol. 9, no. 8, pp. 1735–1780, 1997.
[55] M. F. Stollenga, J. Masci, F. Gomez, and J. Schmidhuber, “Deep networks
with internal selective attention through feedback connections,” in Proc.
Adv. Neural Inf. Process. Syst., 2014, pp. 3545–3553.
[56] L. Chen et al., “SCA-CNN: Spatial and channel-wise attention in convolu-
tional networks for image captioning,” in Proc. IEEE/CVF Conf. Comput.
Vis. Pattern Recognit., 2017, pp. 5659–5667.
[57] K. Xu et al., “Show, attend and tell: Neural image caption generation with
visual attention,” in Proc. Int. Conf. Mach. Learn., 2015, pp. 2048–2057.
[58] J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman, “Lip reading
sentences in the wild,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern
Recognit., 2017, pp. 3444–3453.
[59] A. Vaswani et al., “Attention is all you need,” in Proc. Adv. Neural Inf.
Process. Syst., 2017, pp. 5998–6008.
[60] X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,”
in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2018, pp. 7794–
7803.
[61] A. Buades, B. Coll, and J.-M. Morel, “A non-local algorithm for image
denoising,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit.,
vol. 2, 2005, pp. 60–65.
[62] A. Santoro et al., “A simple neural network module for relational reason-
ing,” in Proc. Adv. Neural Inf. Process. Syst., 2017, pp. 4967–4976.
[63] L. Li, Z. Gan, Y. Cheng, and J. Liu, “Relation-aware graph attention
network for visual question answering,” in Proc. IEEE/CVF Int. Conf.
Comput. Vis., 2019, pp. 10313–10322.
[64] H. Hu, J. Gu, Z. Zhang, J. Dai, and Y. Wei, “Relation networks for object
detection,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2018,
pp. 3588–3597.
[65] M. Yatskar, L. Zettlemoyer, and A. Farhadi, “Situation recognition: Visual
semantic role labeling for image understanding,” in Proc. IEEE/CVF Conf.
Comput. Vis. Pattern Recognit., 2016, pp. 5534–5542.
[66] G. Gkioxari, R. Girshick, P. Dollár, and K. He, “Detecting and recognizing
human-object interactions,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern
Recognit., 2018, pp. 8359–8367.
[67] P. Battaglia et al., “Interaction networks for learning about objects,
relations and physics,” in Proc. Adv. Neural Inf. Process. Syst., 2016,
pp. 4502–4510.
[68] N. Watters, D. Zoran, T. Weber, P. Battaglia, R. Pascanu, and A. Tacchetti,
“Visual interaction networks: Learning a physics simulator from video,”
in Proc. Adv. Neural Inf. Process. Syst., 2017, pp. 4539–4547.
[69] F. Chollet, “Xception: Deep learning with depthwise separable convolu-
tions,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2017,
pp. 1251–1258.
[70] M. Jaderberg, A. Vedaldi, and A. Zisserman, “Speeding up convolutional
neural networks with low rank expansions,” 2014, arXiv:1405.3866.
[71] M. Lin, Q. Chen, and S. Yan, “Network in network,” 2013, arXiv:
1312.4400.
[72] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in Proc.
IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2018, pp. 7132–7141.
[73] J. Hu, L. Shen, S. Albanie, G. Sun, and A. Vedaldi, “Gather-excite:
Exploiting feature context in convolutional neural networks,” in Proc. Adv.
Neural Inf. Process. Syst., 2018, pp. 9401–9411.
Authorized licensed use limited to: Universidad Nacional de Colombia (UNAL). Downloaded on June 06,2023 at 14:18:36 UTC from IEEE Xplore. Restrictions apply.
973
[74] H. Zhao et al., “PSANet: Point-wise spatial attention network for scene
parsing,” in Proc. Eur. Conf. Comput. Vis., 2018, pp. 267–283.
[75] Z.-Q. Zhao, P. Zheng, S.-T. Xu, and X. Wu, “Object detection with deep
learning: A review,” IEEE Trans. Neural Netw. Learn. Syst., vol. 30, no. 11,
pp. 3212–3232, Nov. 2019.
[76] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask R-CNN,” in Proc.
IEEE/CVF Int. Conf. Comput. Vis., 2017, pp. 2961–2969.
[77] L. Liu et al., “Deep learning for generic object detection: A survey,” Int.
J. Comput. Vis., vol. 128, no. 2, pp. 261–318, 2020.
[78] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierar-
chies for accurate object detection and semantic segmentation,” in Proc.
IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2014, pp. 580–587.
[79] K. He, X. Zhang, S. Ren, and J. Sun, “Spatial pyramid pooling in deep
convolutional networks for visual recognition,” IEEE Trans. Pattern Anal.
Mach. Intell., vol. 37, no. 9, pp. 1904–1916, Sep. 2015.
[80] R. Girshick, “Fast R-CNN,” in Proc. IEEE/CVF Int. Conf. Comput. Vis.,
2015, pp. 1440–1448.
[81] J. Dai, Y. Li, K. He, and J. Sun, “R-FCN: Object detection via region-based
fully convolutional networks,” in Proc. Adv. Neural Inf. Process. Syst.,
2016, pp. 379–387.
[82] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun,
“OverFeat: Integrated recognition, localization and detection using con-
volutional networks,” 2013, arXiv:1312.6229.
[83] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once:
Unified, real-time object detection,” in Proc. IEEE/CVF Conf. Comput.
Vis. Pattern Recognit., 2016, pp. 779–788.
[84] W. Liu et al., “SSD: Single shot multibox detector,” in Proc. Eur. Conf.
Comput. Vis., 2016, pp. 21–37.
[85] Y. Cao, J. Xu, S. Lin, F. Wei, and H. Hu, “GCNet: Non-local networks
meet squeeze-excitation networks and beyond,” in Proc. IEEE/CVF Int.
Conf. Comput. Vis. Workshops, 2019, pp. 1971–1980.
[86] Z. Yang et al., “XlNet: Generalized autoregressive pretraining for language
understanding,” in Proc. Adv. Neural Inf. Process. Syst., 2019, pp. 5753–
5763.
[87] J. Fu et al., “Dual attention network for scene segmentation,” in Proc.
IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2019, pp. 3146–3154.
[88] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual
transformations for deep neural networks,” in Proc. IEEE/CVF Conf.
Comput. Vis. Pattern Recognit., 2017, pp. 1492–1500.
[89] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network
training by reducing internal covariate shift,” in Proc. Int. Conf. Mach.
Learn., 2015, pp. 448–456.
[90] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking
the inception architecture for computer vision,” in Proc. IEEE/CVF Conf.
Comput. Vis. Pattern Recognit., 2016, pp. 2818–2826.
[91] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. Alemi, “Inception-v4,
inception-resnet and the impact of residual connections on learning,” 2016,
arXiv:1602.07261.
[92] J. Redmon and A. Farhadi, “YOLOv3: An incremental improvement,”
2018, arXiv:1804.02767.
[93] T.-Y. Lin et al., “Microsoft COCO: Common objects in context,” in Proc.
Eur. Conf. Comput. Vis., 2014, pp. 740–755.
[94] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie,
“Feature pyramid networks for object detection,” in Proc. IEEE/CVF Conf.
Comput. Vis. Pattern Recognit., 2017, pp. 2117–2125.
Zhensheng Shi (Member, IEEE) received the B.Sc.
degree in electronic information science and technol-
ogy from Qufu Normal University, Qufu, China, in
2009, the M.Eng. degree in electronic and commu-
nication engineering, and the Ph.D. degree in intel-
ligent information and communication system from
the Ocean University of China, Qingdao, China, in
2012 and 2021, respectively.
He is currently a Postdoctor with the Frontiers Sci-
ence Center for Deep Ocean Multispheres and Earth
System, Ocean University of China. His research
interests include video understanding, multi-modal learning, and underwater
vision.
974
Cheng Guan received B.Sc. degree in electronic in-
formation engineering from Northeastern University,
Shenyang, China, in 2018, and the M.Eng. degree in
electronic and communication engineering from the
Ocean University of China, Qingdao, China, in 2021.
His research interests include deep learning and
video understanding.
Qianqian Li received the B.Sc. degree in applied
physics from Xihua University, Chengdu, China, in
2019. She is currently working toward the master’s
degree in electronic and communication engineering
from the Ocean University of China, Qingdao, China.
Her research interests include action recognition
and object detection.
Liangjie Cao received the B.Sc. degree in communi-
cation engineering from Shandong Agricultural Uni-
versity, Taian, China, in 2018, and the M.Eng. degree
in electronic and communication engineering from
the Ocean University of China, Qingdao, China, in
2021.
His research interests include action recognition
and video understanding.
Ju Liang received the B.Sc. degree in electronic in-
formation science and technology from Qufu Normal
University, Qufu, China, in 2009. She is currently
working toward the master’s degree in electronic and
communication engineering from the Ocean Univer-
sity of China, Qingdao, China.
Her research interests include action recognition
and video understanding.
Authorized licensed use limited to: Universidad Nacional de Colombia (UNAL). Downloaded on June 06,2023 at 14:18:36 UTC from IEEE Xplore. Restrictions apply.
IEEE JOURNAL OF OCEANIC ENGINEERING, VOL. 47, NO. 4, OCTOBER 2022
Haiyong Zheng (Member, IEEE) received the B.Eng.
degree in electronic information engineering and the
Ph.D. degree in ocean information sensing and pro-
cessing from the Ocean University of China, Qingdao,
China, in 2004 and 2009, respectively.
In 2009, he joined the Department of Electronic
Engineering, Ocean University of China, where he is
currently a Professor. His research interests include
computer vision, underwater vision, and deep learn-
ing.
Zhaorui Gu (Member, IEEE) received the B.Eng. de-
gree in electronic information science and technology
and the M.Eng. degree in electronic and communica-
tion engineering from the Ocean University of China,
Qingdao, China, in 2010 and 2013, respectively.
He is currently an Experimentalist with the Depart-
ment of Electronic Engineering, Ocean University of
China. His research interests include image process-
ing and underwater vision.
Bing Zheng (Member, IEEE) received the B.Sc.
degree in electronics and information system, the
M.Sc. degree in marine physics, and the Ph.D. degree
in computer application technology from the Ocean
University of China, Qingdao, China, in 1991, 1995,
and 2013, respectively.
He is currently a Professor with the Department of
Electronic Engineering, Ocean University of China.
His research interests include ocean optics, underwa-
ter imaging, and optical detection.