DESIGN OF IMINT MILITARY TARGET DETECTION

SOFTWARE USING DEEP LEARNING

Ghazi MARZOUK, Xu QIANG*, Wei LI
College of Information and Navigation
Air Force Engineering University
Xi’an, China
marzouk.ghazi@yahoo.com

Abstract: Detecting different targets from a High Resolution Remote Sensing image is one of the classical problems of
computer vision and is often described as a difficult task. This paper will present the appropriate tasks in com-
puter vision using Deep Learning technology with the constraint of small training data and to use a pretrained
Convolutional Neural Network (CNN) (by preprocessing the images dataset and developing the right training
process). Faster R-CNN method will be used for the object detection task. Due to practical use, this work will
detect 2 classes (Airplanes and storage tanks). The dataset used for the training is a combination of an existing
data set and collected images for the military aircrafts. The analyze of the large data of IMINT will be faster and
the human labor will be reduced to the minimum. The software will recognize different targets from large images
collected by Satellites, Reconnaissance UAVs or Aircrafts from the ISR missions by an average accuracy of
90% for the airplane class.
Index Terms: Imagery Intelligence; Remote Sensing; ISR; Deep Learning; Object Detection; Convolutional Neural Network; CNN;
Faster RCNN; Computer Vision.

1. INTRODUCTION (targets) from a Remote Sensing HR image. What

makes object detection a distinct problem is that it

T he growing quantity of Airborne and Satellite

images (Electro Optical, Infrared, SAR, etc.) ac-
quired for Military Intelligence or called IMINT (Im-
agery Intelligence), lead us to find a solution for de-
tecting and recognizing military targets such as
tanks, armored vehicle, land troops, aircrafts, ships,
bridges, storage tanks, buildings…). Due to the large
scale of the battlefield, the solution need to use an
automatic system to detect and recognize the tar-
gets. This system could enhance the intelligence ca-
pability of the Air force in the ISR (Intelligence Sur-
veillance and Reconnaissance) missions, but it can
also be employed by the Land Forces or the Navy.
involves both locating and classifying regions [3].
The location and size are typically defined using a
bounding box, in the form of corner coordinates. The
sub-image contained in the bounding box is then
classified by an algorithm that has been trained us-
ing machine learning or DL (in our case) [4].
In this paper, the use of the CNN in object detec-
tion task and present different existing methods will
be discussed first. Then the experimental work will
be presented which is the use of one of the current
method (Faster R-CNN) for the target detection
task.

In order to make this system autonomous, Artifi-

cial Intelligence (AI) could be the best choice to
achieve that goal. Nowadays, DL (Deep Learning), is
wide used in many fields (Medical, Social media, au-
tonomous vehicle navigation and also in military
use). Target recognition in large High Resolution
(HR) Remote Sensing (RS) images, such as aircraft,
and vehicle detection, is a challenging task due to
the small size and big number of targets and the
complex neighboring environments.
This work is a combination between Computer vi-
sion tasks and DL technology. The most common
technic used for this combination is the use of the
CNN (Convolutional Neural Network) and DCNN
(Deep CNN). The basic idea of the CNN was inspired
by a concept in biology called the receptive field [1].
They act as detectors that are sensitive to certain
types of stimulus, for example, edges. This biological
function can be approximated in computers using
the convolution operation detailed in this reference
[2]. My work is related with the object detection
2. CNN IN OBJECT DETECTION
Regardless of the visual task that we want to
achieve, a CNN or DCNN is necessary for the DL in
computer vision tasks. 2012 marked the first year
where a CNN was used to achieve a top 5 test error
rate of 15.4%. Alex Krizhevsky et al. [5] achieved
promising results with CNNs for the general image
classification task by developing the AlexNet. It
achieved excellent results on the ImageNet Large
Scale Visual Recognition Challenge (ILSVRC) [6] da-
taset. Trained on ImageNet data, the Alexnet net-
work is composed by 25 layers. It uses a relatively
simple layout. The network is made up of 5 convolu-
tional layers, max-pooling layers, dropout layers (to
combat the problem of overfitting to the training
data which is a good solution for making the transfer
learning with a small dataset hence the choice of this
model), and 3 fully connected layers [5].
Nowadays many methods exist for object detec-
tion and they are developing from year to year

* English to Chinese translation

according to the ILSVRC. Those methods are de-
pending on the purpose of the detection, we can di-
vide them into still image (R-CNN, Fast R-CNN,
Faster R-CNN…) and real-time (YOLO [7], SSD
[8]…) object detection. In this section, different ob-
ject detection methods for the still image that utilize
convolutional neural networks will be introduce and
compared. Then, the method used in my work will be
discussed, which is the Faster R-CNN [9].
2.1. R-CNN
In 2013, Girshick et al. published a method [4]
generalizing the results of the Alexnet introduced in
the beginning of section 2 to object detection. This
method is called R-CNN (CNN with region pro-
posals).
R-CNN, consisted of 3 simple steps: First, scan
the input image for possible objects using an algo-
rithm called Selective Search [10], generating
around 2000 region proposals. Second, run a CNN on
top of each of these region proposals. Then third,
take the output of each CNN and feed it into an SVM
(Support Vector Machine [11]) method to classify the
region and a linear regressor to tighten the bounding
box of the object, if such an object exists [4].
In other words, it first proposes regions, then ex-
tract features, and then classify those regions based
on their features. In essence, it has turned object de-
tection into an image classification problem. R-CNN
was very intuitive, but very slow.
2.2. Fast R-CNN
Fast R-CNN published in 2015 by Girshick Ross
to provide a more practical method for object recog-
nition [3]. The main idea is to perform the forward
pass of the CNN for the entire image, instead of per-
forming it separately for each Region of Interest
(RoI). It’s the immediate descendant of R-CNN.
This method generates region proposals based on
the last feature map of the network. As a result, we
can train just one CNN for the entire image. And in-
stead of training many different SVM’s to classify
each object class, there is a single Softmax layer [12]
that outputs the class probabilities directly. Now we
only have one neural net to train, as opposed to one
neural net and many SVM’s. Fast R-CNN performed
much better in terms of speed. There was just one
big bottleneck remaining: the selective search algo-
rithm for generating region proposals.
2.3. Region proposal generation
To use R-CNN and Fast R-CNN, we need a
method for generating the regions of interest. Many
methods exist for that like Sliding window technic,
Edge boxes [13], and the most popular unsupervised
methods is Selective Search [10] which utilizes an it-
erative merging of super-pixels.
Certain advanced object detection methods, such
as Faster R-CNN [9] described in subsection 2.4 be-
low, use parts of the same convolutional network
both for generating the region proposals and for de-
tection. We call these kinds of methods integrated
methods.
2.4. Faster R-CNN
Faster R-CNN [9] by Ren et al. is an integrated
method. The main idea is to use shared convolutional
layers for region proposal generation and for detec-
tion. The authors discovered that feature maps gen-
erated by object detection networks can also be used
to generate the region proposals. The fully convolu-
tional part of the Faster R-CNN network that gener-
ates the feature proposals is called a Region Proposal
Network (RPN). The authors used Fast R-CNN ar-
chitecture for the detection network.
A Faster R-CNN network is trained by alternat-
ing between training for RoI generation and detec-
tion. First, two separate networks are trained. Then,
these networks are combined and fine-tuned. The
trained network receives a single image as input.
The shared fully convolutional layers generate fea-
ture maps from the image. These feature maps are
fed to the RPN. The RPN outputs region proposals,
which are input, together with the said feature
maps, to the final detection layers. These layers in-
clude a RoI pooling layer and output the final classi-
fications as shown on Figure 1.
Figure 1: Faster R-CNN method in test mode
In this work, this method for the object detection
task will be used due to the results proven in the sub-
section below.
2.5. Comparing the methods
Liu et al. [8] compared the performance of Fast R-
CNN, Faster R-CNN and SSD on the PASCAL VOC
2007 [14] test set. When using networks trained on
the PASCAL VOC 2007 training data, Fast R-CNN
achieved a mean average precision (mAP) of 66.9.
Faster R-CNN performed better, with a mAP of 69.9.
SSD achieved a mAP of 71.6 with input size 512x512.
As the standard implementations of Fast R-CNN
and Faster R-CNN use 600 as the length of the
shorter dimension of the input image, SSD seems to
perform better with similarly sized images.
However, SSD requires extensive use of data aug-
mentation to achieve this result. Fast RCNN and
Faster RCNN only use horizontal flipping [8].
3. EXPERIMENTAL METHOD
3.1. Training data
In this section, the image dataset used for this
work will be presented and the image preprocessing
done on it will be discussed while explaining the rea-
sons.
3.1.1. Image source
The main source of the RS image dataset is the
well-known University of California(UC) Merced Da-
taset [15]. This dataset is a collection of aerial im-
ages (256×256 pixels in RGB space) depicting 21
land use classes of 100 images. Since each image
comes with a single label, the dataset can be only
used for image classification purposes. So, I needed
to enhance the images in the dataset as described in
subsection 3.1.2 below. Also, the number of images
per category is relatively small (100 images) hence
the need to collect more images especially for the
military airplane and I had to make preprocessing
and data augmentation described in the next subsec-
tion. For the practical purpose of this work, only 2
categories: Airplane and Storage Tanks has been
chosen.
Figure 2: Example images associated with the 2 land-use categories in
the UC Merced data set: Airplane (left) Storage tank (Right)
In addition, with the UC Merced Dataset, I have
collected other satellite images generally of military
transport airplanes using the GIS (Geographic Infor-
mation System) Software Global Mapper [16]. I have
searched for Air Bases around the world and down-
loaded the high-resolution images with different
sizes with a spatial resolution of 0.3m as shown in
Figure 3.
The total number of images in the dataset is 188
images with different sizes split into 2 categories.
Figure 3: Example of images from the dataset collected using Global
Mapper
3.1.2. Images preprocessing
The most time consuming in this work is the im-
ages preprocessing. I made many processing
technics to augment the dataset and trying to make
the network rotational invariant.
1. Rotating the images: The first step is to ro-
tate the images for two reasons. The first one is to
enlarge the dataset (188 images will be not enough
to get good results even if a pretrained network is
used). Second is to try to make the detection rota-
tional invariant. The CNN cannot distinguish be-
tween different rotation of the target. The idea is to
make a copy of every image from the dataset by add-
ing a rotation of 45° from 0° to 180°, so I will get 4
new images. I chose that interval because I will add
a data augmentation function in the input image
layer (see subsection 3.2.1 below) by flipping the im-
age vertically. At the end, while training the net-
work, I have a theoretical number of images aug-
mented by 9 images for each source image. (Total
output is 940 images)
2. Drawing the Ground Truth (GT) boxes: The
second step is to draw the GT boxes around the tar-
gets in the images manually. Training an object de-
tection CNN need to have the images with GT la-
beled with different target categories to train the
RPN and to learn how to localize the object from the
neighboring environment. I made a manual labelling
session using the built-in software implemented in
MATLAB 2017b [17] to get the GT boxes in the form
of [𝑥 𝑦 ℎ𝑒𝑖𝑔ℎ𝑡 𝑤𝑖𝑑ℎ𝑡] as shown in Figure 4.
Figure 4: Example of drawing the GT boxes
3. Extracting the target images from the image
dataset: As a third step, all the labeled target from
the images in the dataset after step 2 are extracted
by the crop method. I did this step to initially train
the CNN as a classification method. I believe that it
will initialize the CNN weights to get better result in
the Faster RCNN training (Total output is 1353 im-
ages). But in order the feed the CNN, images must
be resized into 𝑚 × 𝑚 and I have to keep the same
image ratio hence the need of the next step.
4. Image resizing: The CNN input squared im-
age (for our CNN it’s 227 × 227 and RGB channel) so
I resized all the cropped images from step 3. The
resize method idea is to fill the shortest side by black
(or said zeros intensity) and output a square image
so we don’t lose the image ratio and we don’t get a
distorted target for training. After that, the output
is resized into 227 × 227 pixels.
5. Create image datastore: this fifth step is to
prepare the data for the training purpose. I create an
image datastore with all the images in the previous
step with their respective classification label. After
that I split every category in 3 parts: A Training
Dataset representing 80% of the data, A Validation
Dataset (to follow the improvement of the network
while training) 10%, and a Test Dataset (to get the
accuracy) of 10%.
3.2. Architecture of the Network
3.2.1. CNN
For the CNN, I used the transfer learning tech-
nique in a pre-trained network which is the Alexnet
[5] (described in section 2). The advantage of this ap-
proach is that the network has already learned a rich
set of images features that is applicable to a wide
range of images. The time of the training can be too
long because of the heavy calculation needed to train
the network. To make the transfer learning, I de-
leted the last 3 layers that are trained for the classi-
fication (Fully-connected layer, Softmax layer and
the classification layer) because the output was 1000
classes, and our output is only 2 classes. I also
changed the first layer which is the input-layer and
add a data augmentation method for making a ran-
dom vertical flip of the input image on every itera-
tion of the mini-batch for the training. This data aug-
mentation, together with the image preprocessing
explained in the subsection 3.1.2, will help the
Faster R-CNN to detect the object with rotational in-
variance due to the limitation of the object detection
method explained in the subsection 3.2.2 below.
The network was trained on a laptop with a single
Intel core i5 CPU and 8GB RAM memory. The oper-
ating system was Windows 10, and the implementa-
tion environment was under MATLAB 2017b on the
Training Dataset with the Validation Dataset de-
scribed on the step 5 of the image preprocessing.
The CNN is trained using SGDM (Stochastic Gra-
dient Descent with Momentum) with a batch size of
128, a momentum of 0.9 and a learning rate of 10−4 .
The training required 8 epochs (shuffling the dataset
on every epoch) and took us around 385 minutes.
After running the trained network on the test da-
taset, I got a mean diagonal accuracy of 99.98% on
the 2 trained classes as shown in the confusion ma-
trix in Table 1.
Airplane Storage Tank
Airplane 0.9999 2.7189E-06
Storage
4,63E-04 0.9995
Tank
Table 1: Confusion Matrix of the trained CNN
This classification trained network will be used
for the object detection method.
3.2.2. Object detection method
For the object detection method, the Faster R-
CNN described in section 2.4 is used because of its
high accuracy result as shown in section 2.5. Faster
R-CNN don’t have rotation invariance implemented
[9]. Objects in RS imagery need to be rotational in-
variant due to the random direction of the targets in
the input image. This issue was overcome by the
data augmentation in the input layer and the image
preprocessing. The dataset created on step 2 de-
scribed in subsection 03.1.2 is used.
The Faster R-CNN have 2 CNN working to-
gether. One network is the pretrained CNN de-
scribed in the section 2 above, transformed into a
Fast R-CNN (see subsection 2.2) by adding a regres-
sion network in order to output the localization box
represented by 4 values [𝑥 𝑦 ℎ𝑒𝑖𝑔ℎ𝑡 𝑤𝑖𝑑ℎ𝑡]. The other
one is called the RPN to output the RoI. This net-
work shares the same weights as the previous CNN,
but the last layers concerned by the classification are
changed by an RoI output to feed the Fast R-CNN for
the classification [9]. The Faster R-CNN passes by 4-
steps alternating training [9], where: First, train the
RPN initialized by the pretrained CNN. Second,
train a separate detection network by Fast R-CNN
using proposals generated by the last step, initial-
ized by the CNN. Third, fix the convolutional layers,
fine tune unique layers to RPN, initialized by the de-
tector on the third step. Fourth, fix the convolutional
layer, fine-tune the Fully-Connected layers of Fast
R-CNN.
The chosen learning options of the Faster R-CNN
are almost the same for the 4 steps: SGDM with a
batch size of 256, a learn rate drop factor of 0.5 every
3 epochs and by shuffling the image dataset on every
epoch. The initial learn rate of the 2 first steps is
10−5 , and 10−6 for the rest of the steps (because
those steps are used to fine tune the previous ones).
The learning epochs is fixed at 10 for the 1st and the
3rd steps and 12 epochs for the 2nd and the 4th steps.
The positive IoU (Intersection over Union of the
anchor over the Ground Truth box, more details in
reference [13]) is also fixed by [0.6 1] (object) and
negative IoU by [0 0.3] (not object)
All the parameters were chosen after many prac-
tical tries to get a better result. The training took
around 72 hours on the same laptop environment
previously mentioned.
3.2.3. Evaluation Method
For the target detection performance, four com-
monly used criteria were computed: FPR, MR, Accu-
racy (AC), and Error Ratio (ER). These criteria are
defined as follows:
Number of falsely detected targets
𝐹𝑃𝑅 = × 100%
Number of detected targets
Number of missing targets
𝑀𝑅 = × 100%
Number of targets
Number of detected targets
𝐴𝐶 = × 100%
Number of targets
𝐸𝑅 = 𝐹𝑃𝑅 + 𝑀𝑅
4. CONCLUSION
The goal of this work is to show the ability of ob-
ject detection using DL technology using Faster R-
CNN method for military targets from a High-Reso-
lution RS image. To implement this method, a series
of process must be done on the Dataset and on the
chosen pre-trained CNN due to the small number of
images in the dataset. I demonstrated the feasibility
of this type of work on a single CPU laptop
Two object classes (Airplane, Storage tanks) are
used as an example. In the future, the software can
be trained to detect more classes (ships, armored ve-
hicles, bridges, deployed troops…). There will be no
limitation as long as we can collect the appropriate
Dataset from Remote Sensing images (including EO,
IR, SAR…).
5. REFERENCES
[1] K. Fukushima, "Neocognitron: A hierarchical
neural network capable," Neural Networks,
vol. 1, no. 2, pp. 119 - 130, 1988.
[2] D. Marr and E. Hildreth, "Theory of edge
detection," Proceedings, vol. 207, no. 1167, pp.
187 - 217, 1980.
[3] R. Girshick, "Fast R-CNN," Proceedings of
the IEEE International, pp. 1440-1448, 2015.
[4] R. Girshick, J. Donahue, T. Darrell and J.
Malik, "Rich feature hierarchies for accurate
object detection and semantic segmentation,"
in Proceedings of the IEEE conference on
computer vision and, 2014.
[5] A. Krizhevsky, I. Sutskever and G. E. Hinton,
"ImageNet Classification with Deep
Convolutional Neural Networks," Advances
in neural information processing systems, pp.
1097-1105, 2012.
[6] ImageNet, "Large Scale Visual Recognition
Challenge (ILSVRC)," Stanford Vision Lab,
[Online]. Available: http://www.image-
net.org/challenges/LSVRC/.
[7] J. Redmon, S. Divvala, R. Girshick and A.
Farhadi, "You Only Look Once: Unified, Real-
Time Object Detection," Computer Vision and
Pattern Recognition (cs.CV), 2016.
[8] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S.
Reed, C. Y. Fu and A. C. Berg, "SSD: Single
shot multibox detector," in European
Conference on Computer Vision , Amsterdam,
Netherlands, 2016.
[9] S. Ren, K. He, R. Girshick and J. Sun, "Faster
R-CNN: Towards Real-Time Object Detection
with Region Proposal Networks," IEEE
Transactions on Pattern Analysis and
Machine Intelligence, vol. 39, no. 6, pp. 1137-
1149, 01 06 2017.
[10] J. R. R. Uijlings, K. E. A. Sande, T. Gevers
and A. W. M. Smeulders, "Selective Search for
Object Recognition," International Journal of
Computer Vision, vol. 2, pp. 154-171, 2013.
[11] C. M. Bishop, Pattern Recognition and
Machine Learning, Secaucus,NJ, USA:
Springer-Verlag New York, Inc., 2006.
[12] I. Goodfellow, Y. Bengio and A. Courville,
DEEP LEARNING, MIT Press, 2016.
[13] C. L. Zitnick and P. Dollar, "Edge boxes:
Locating object proposals from edges,"
European Conference on Computer Vision,
pp. 391-405, 2014.
[14] M. Everingham, L. Van Gool, C. K. I.
Williams, J. Winn and A. Zisserman, "The
PASCAL Visual Object Classes Challenge
2007," [Online]. Available:
http://host.robots.ox.ac.uk/pascal/VOC/voc20
07/.
[15] Y. Yang and S. Newsam, "Bag-of-visual-
words and spatial extensions for land-use
classification," in 18th ACM SIGSPATIAL
Int. Symposium on Advances in Geograph.
Inf. Sys., San Jose, CA, USA, June, 2010.
[16] Blue Marble Geographics, "Global Mapper,"
Blue Marble Geographics , [Online].
Available:
http://www.bluemarblegeo.com/products/glob
al-mapper.php.
[17] Mathworks, "Matlab," [Online]. Available:
https://www.mathworks.com/products/matla
b.html.
[18] D. Rumelhart, G. Hinton and R. Williams,
"Learning representations by back-
propagating errors," Cognitive modeling, vol.
5, no. 3, 1986.
[19] R. Szeliski, Computer Vision: Algorithms and
Applications, Springer, 2010.
[20] F.-F. Li, A. Karpathy and J. Johnson, Writers,
Convolutional Neural Networks for Visual
Recognition. [Performance]. Stanford
University CS231n, 2016.