Aircraft Detection for Remote Sensing Images:

A comparative evaluation of object detection using

Ultralytics-YOLO
1st Huzair Saputra
Master of Artificial Intelligence,
Department of Informatics, Universitas
Syiah Kuala, Aceh, Indonesia
huzairsaputra@gmail.com
2nd M. Haekal Alfanshury
Master of Artificial Intelligence,
Department of Informatics, Universitas
Syiah Kuala, Aceh, Indonesia
haekalalfanshury@gmail.com
3rd Deny Joefakri Iwa Supriatna
Master of Artificial Intelligence,
Department of Informatics, Universitas
Syiah Kuala, Aceh, Indonesia
denyjoefakri@gmail.com

4th Farrasa Rani Faisyal 5th Kahlil Muchtar*

Master of Artificial Intelligence,
Department of Informatics, Universitas
Syiah Kuala, Aceh, Indonesia
farasa.lldikti13@gmail.com
Department of Electrical and Computer
Engineering, Universitas Syiah Kuala
Telematics Research Center (TRC)
Universitas Syiah Kuala, Aceh,
Indonesia
kahlil@usk.ac.id (corr.author)

Abstract— Aircraft detection in remote sensing images
plays a crucial role in various applications, such as aerial
surveillance, border security, and environmental
monitoring. In recent years, deep learning-based object
detection methods have shown remarkable performance
in identifying and localizing objects in complex visual
scenes. Among these methods, Ultralytics-YOLO (You
Only Look Once) has gained significant attention for its
real-time performance and high accuracy. This paper
presents a comprehensive comparison of all variants of
YOLOv5 and YOLOv8, two popular object detection
frameworks. The objective is to evaluate their
performance and identify the strengths and weaknesses of
each variant. A diverse range of datasets, namely RSOD-
aircraft, is used for evaluation. The experimental setup
involves configuring and training each variant, followed
by performance evaluation using standard metrics such
as precision, recall, and mean Average Precision (mAP).
The results highlight the performance differences and
trade-offs between the variants, aiding researchers and
practitioners in selecting the most suitable model for
object detection tasks. The findings contribute to the
advancement of object detection research and facilitate
informed decision-making in various domains.
Keywords— Aircraft detection, remote sensing images,
object detection, Ultralytics-YOLO
This study aims to conduct a comparative evaluation of
aircraft detection using Ultralytics-YOLO in remote sensing
images. The primary objective is to assess the performance
and efficacy of Ultralytics-YOLO in detecting aircraft, as
well as to compare it with other state-of-the-art object
detection methods. To achieve this, a comprehensive dataset
of remote sensing images encompassing diverse aerial scenes
will be utilized. Additionally, the execution time required for
aircraft detection will be measured to assess the real-time
capabilities of Ultralytics-YOLO [2].
In summary, this study aims to evaluate and compare
the performance of Ultralytics-YOLO in aircraft detection,
providing valuable insights into its capabilities and
limitations. The subsequent sections will present the related
work, proposed approach, results, and conclusion.
II. RELATED WORK
Previous research by Berkay Yaban, et al. in aircraft
detection in very high-resolution satellite images has gained
significant attention due to its potential applications in aerial
surveillance, military reconnaissance, and aviation safety.
Deep learning methods, such as the YOLO (You Only Look
Once) architecture, have demonstrated promising
performance in object detection tasks, including aircraft
detection. In this section, we review related works that have
employed YOLO-based deep-learning methods for aircraft
detection in very high-resolution satellite images [3].

I. INTRODUCTION
III. PROPOSED APPROACH
In recent years, deep learning-based object detection
methods have shown remarkable advancements in addressing This study suggests an Ultralytics-YOLO object
these challenges. One such approach that has gained attention detection method for use on aircraft. A total of 446 photos
is Ultralytics-YOLO (You Only Look Once), a state-of-the- with 4993 aircraft objects make up the dataset, called RSOD-
art object detection algorithm known for its real-time Dataset [4]. Examples of these images can be seen in Fig. 1.
performance and high accuracy. Ultralytics-YOLO has been
successfully applied in various domains, including computer
vision and robotics. However, its performance and
effectiveness in aircraft detection within remote sensing
images have not been extensively evaluated and compared to
other object detection algorithms [1].
 performance in real-world scenarios and identify areas for
improvement. By applying the model's prediction function to
detect all aircraft in an image, we could evaluate its accuracy
and efficiency in identifying these specific objects.

Fig.1. Example of aircraft image

In this paper, we propose a comprehensive evaluation to
compare all variants of YOLOv5 and YOLOv8, including
Yolov8n, Yolov5n, Yolov8s, Yolov5s, Yolov8m, Yolov5m,
Yolov8i, Yolov5i, Yolov8x, and Yolov5x, respectively. The
objective is to assess their performance and identify the
strengths and weaknesses of each variant in the context of
object detection. To conduct this comparison, we carefully
selected the RSOD aircraft dataset as it provides a
challenging and diverse set of images containing various
aircraft instances. The dataset encompasses different aircraft Fig.2. Comparison of model performance YOLOv5 and
types, poses, and environmental conditions, ensuring a YOLOv8 model
comprehensive evaluation of the models. The object prediction result can be seen in Fig. 3.
For each YOLOv5 and YOLOv8 variant, we configured
and trained the models using the appropriate training
strategies specific to each variant. This involved setting up
the network architectures, adjusting hyperparameters, and
initializing the models with suitable weights. To evaluate the
performance of each variant, we utilized standard evaluation
metrics for object detection, including precision, recall, mean
Average Precision (mAP), and F1-score. Additionally, we
considered class-specific metrics to evaluate their
performance on different aircraft categories present in the
dataset. Fig.3. The detection results of YOLOv5x model
Our comparative analysis revealed valuable insights
into the strengths and weaknesses of each YOLOv5 and V. CONCLUSION
YOLOv8 variant. We thoroughly examined their The results emphasize the positive impact of using the
performance in terms of accuracy, computational efficiency, YOLOv8 model for object detection tasks, showcasing
robustness to occlusion, and ability to handle variations in significant improvements over previous versions. This
lighting conditions and object scale. The analysis provided a knowledge can guide future research and development efforts
comprehensive understanding of the performance differences to enhance object detection capabilities, leading to more
and trade-offs between the variants. Based on the results and efficient and accurate systems in various applications.
analysis, this study offers guidance for selecting the most
suitable YOLOv5 or YOLOv8 variant for specific object ACKNOWLEDGMENT
detection applications, particularly in the domain of aircraft We thank the Master of Artificial Intelligence Study
recognition. Program and Erasmus+ DS&AI Program, for supporting this
research.
IV. EXPERIMENTAL RESULTS
REFERENCES
Overall, the results show a significant improvement in
object detection using the YOLOv8 variant model trained [1] A. Bochkovskiy, C.-Y. Wang, and H.-Y. M. Liao, “YOLOv4: Optimal
Speed and Accuracy of Object Detection.” 2020.
with images, but the highest mean Average Precision (mAP) [2] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You Only Look
score among the YOLO object detection models listed is Once: Unified, Real-Time Object Detection,” in 2016 IEEE
achieved by YOLOv5x, with an mAP of 0.742. The Conference on Computer Vision and Pattern Recognition (CVPR),
performance comparison between the model trained with 2016, pp. 779–788. doi: 10.1109/CVPR.2016.91.
[3] B. Yaban, U. Alganci, and E. Sertel, “Aircraft Detection in Very High
original images can be seen in Fig. 2. The mAP scores for Resolution Satellite Images using YOLO-based Deep Learning
different versions of YOLO object detection models. Methods,” in 4th Intercontinental Geoinformation Days (IGD), Tabriz,
We conducted an analysis using the YOLO prediction Iran, May 2022, pp. 270–273.
[4] Y. Long, Y. Gong, Z. Xiao, and Q. Liu, “Accurate Object Localization
model to gain further insights into the model's performance
in Remote Sensing Images Based on Convolutional Neural Networks,”
and understand the areas of focus during the object detection IEEE Transactions on Geoscience and Remote Sensing, vol. 55, no. 5,
process. This analysis provided valuable information on the pp. 2486–2498, 2017, doi: 10.1109/TGRS.2016.2645610.
model's strengths and limitations, allowing us to assess its