A SEMINAR PRESENTATION

ON
Multi-Sensory Object Detection For Autonomous
Drones

PRESENTED BY
Roll No - T.ECOB29
Name: Kshitij Patil

Guided By:
Mrs. Shailaja Lohar

Academic Year:2023-2024
DEPARTMENT OF COMPUTER ENGINEERING
PIMPRI CHINCHWAD COLLEGE OF ENGINEERING & RESEARCH PUNE

PCCOER
10/12/2023 1
Computer Engineering
CONTENT

⚫ Introduction.

⚫ Literature Survey

⚫ Problem Statement & Objectives

⚫ Proposed System/architecture/framework/algorithm

⚫ Feasibility and scope of the solution proposed such as its cost, targeted users, applicability
etc.

⚫ Conclusion & Future Scope

⚫ References

⚫ Conference attended certificate/Survey paper acceptance letter /Copyright information

PCCOER
10/12/2023 2
Computer Engineering
 INTRODUCTION

• Multi-sensor object detection (MSOD) is the process of detecting objects in a scene using data from multiple sensors. MSOD is a
critical technology for autonomous drones, as it allows them to perceive their environment more accurately and robustly.

• MSOD algorithms typically work by fusing data from multiple sensors, such as cameras, LiDAR, and radar, to generate a more
complete and accurate understanding of the scene. This is done by identifying and exploiting correlations between the data from
the different sensors.

• MSOD algorithms have been shown to achieve state-of-the-art results on a variety of object detection tasks, including pedestrian
detection, vehicle detection, and lane detection. MSOD algorithms are also becoming increasingly efficient to train and deploy,
making them more feasible for use in real-world applications.

PCCOER
10/12/2023 3
Computer Engineering

Research article
(Author/Year)
Multi-Sensor Object
Detection for Autonomous
Drone Navigation in GPS-
Denied Environments by A.
H. Al-Kaff et al.
Assessing Distribution Shift
in Probabilistic Object
Detection Under Adverse
Weather By M.Hildebrand
et al./2023
A Fine-Grained Object
Detection Model For Aerial
Images Based on YOLOv5
Deep Neural Network By
Zhang Rui et al./2023
10/12/2023
LITERATURE SURVEY
Objective/Proposed work
develop a multi-sensor
object detection framework
for autonomous drone
navigation in GPS-denied
environments.
Evaluate uncertainty as a
metric for distinguishing
true- and false-positive
detections in object
detection under adverse
weather conditions.
Improve fine-grained target
detection in
aerial images using
YOLOv5 neural network
Computer Engineering
Methods/Techniques
The proposed framework
integrates data from a
monocular camera, a LiDAR
sensor, and an inertial
measurement unit (IMU) to
achieve robust and accurate
object detection.
Bayesian neural network,
uncertainty estimation,
signal detection theory,
Probabilistic object
detection.
YOLOv5-based
modification with
CSL angle classification.
PCCOER
Datasets/ Relevant findings/
supporting data Limitations identified
The proposed framework is
KITTI, nuScenes evaluated on two real-world
datasets, but the datasets
are both from driving
scenarios. How the
framework would perform
in other scenarios, such as
search and rescue or
disaster response, is
unclear.
KITTI, WAYMO, Canadian Uncertainty distribution
Adverse Driving Conditions differences analyzed for
(CADC). Augmented true-positive and false-
datasets with simulated positive detections. KL
distortions. divergence used to quantify
differences AUC metrics
calculated for ROC curves.
ADR used for real-time
performance assessment.
FAIR1M, DOTA Proposed modification
enables precise recognition
of remote sensing objects
with arbitrary orientation.
Achieved mAP of 39.2 on
FAIR1M dataset and 72.68
on DOTA dataset.
4
R-YOLOv5: A Lightweight
Rotating Object Detection
Algorithm for Real-Time
Detection of Vehicles in
Dense Scenes By Zheng Wei
Li et al./2023
RGB-D Salient Object
Detection Using Saliency
and Edge Reverse Attention
By Tomoki Ikeda et al./2023
A CNN-Transformer Hybrid
Model Based on CSWin
Transformer for UAV Image
Object Detection
10/12/2023
Enhance vehicle detection
accuracy and application in
vehicle-road cooperative
system.
Improve RGB-D Salient
Object Detection.
Propose a CNN-transformer
hybrid model for efficient
object detection in UAV
images
Rotation angle branch for Drone-Vehicle UCAS-AOD. Improved accuracy in
orientation prediction, detecting rotation vehicles.
Circular smooth labels for Fast inference speed.
angle classification, Swin Compression in model size,
Transformer block for parameters and operations.
feature fusion, Feature
Enhancement Attention
Module, Adaptive Spatial
Feature Fusion Structure
Saliency and Edge Reverse Mixed dataset for training Improved object detection
Attention (SERA), Multi- (DUT-RGBD + NJUD + accuracy. Enhanced
Scale Interactive Module NLPR) – Evaluation on detection of important object
(MSIM). DUT-RGBD, NJUD, NLPR, parts. Sharper object
STEREO, and SIP boundaries. Future work:
Address limitations, explore
lighter weight networks
CSWin Transformer, Hybrid VisDrone2021-DET, Proposed model achieves
Patch Embedding Module UAVDT better results than classic
(HPEM), Slicing-Based models. CSWin Transformer
Inference (SI) enhances multiscale object
detection. HPEM improves
feature extraction. SI method
enhances small object
detection. Limitations
include varying contributions
of pretrained models and
gaps from SOTA methods
PCCOER
5
Computer Engineering
PROBLEM STATEMENT & OBJECTIVES
Developing a robust multi-sensory object detection system that integrates visual and auditory cues to
accurately identify and locate objects in real-time.

PCCOER
10/12/2023 6
Computer Engineering
 EXISTING SYSTEM

• The idea of the problem statement I have is through :

• Multi-Sensor Object Detection for Autonomous Drone Navigation in GPS-Denied Environments by A. H. Al-Kaff et al., IEEE
Access, 2022.

• This paper proposes a multi-sensor object detection framework for autonomous drone navigation in GPS-denied environments.
The framework integrates data from a monocular camera, a LiDAR sensor, and an inertial measurement unit (IMU) to achieve
robust and accurate object detection. The authors propose a novel deep learning architecture that is specifically designed for
multi-sensor fusion. The proposed framework is evaluated on two real-world datasets, and the results show that it outperforms
state-of-the-art methods in terms of object detection accuracy and robustness.

PCCOER
10/12/2023 7
Computer Engineering
 EXISTING SYSTEM ARCHITECTURE/
FRAMEWORK/
METHOD/TECHNIQUE

•Data preprocessing: The data from the different sensors is first preprocessed to remove noise and to align the data in time.

•Feature extraction: Features are extracted from the data from each sensor using a variety of techniques, such as deep learning and
traditional image processing methods.

•Sensor fusion: The features from the different sensors are then fused together to generate a more complete and accurate representation
of the scene.

•Object detection: An object detection algorithm is then used to detect objects in the fused feature representation.

PCCOER
10/12/2023 Computer Engineering 8
 PROPOSED SYSTEM/ARCHITECTURE/
FRAMEWORK/
ALGORITHM

•Use of advanced deep learning architecture. The authors of the paper propose a novel deep learning architecture for
multi-sensor fusion, but it is possible that more advanced architectures, such as transformer-based architectures, could
achieve even better performance.

•Use additional sensors. The framework proposed in the paper uses data from a monocular camera, a LiDAR sensor, and an
IMU. However, it is possible that using additional sensors, such as a thermal camera or a radar sensor, could further
improve the performance of the framework.

•Develop better data fusion algorithms. The framework proposed in the paper uses a simple weighted average approach to
fuse data from the different sensors. However, more sophisticated data fusion algorithms, such as those based on deep
learning, could potentially achieve better performance.

•Train the model on more data. The model proposed in the paper is trained on a relatively small dataset. Training the
model on a larger and more diverse dataset could potentially improve the performance of the model in real-world
scenarios.

•Develop a decision-making system that dynamically adapts to the reliability of each sensor's data. For example, if the
camera is obscured by fog, the drone can rely more on LiDAR and radar data for navigation.

PCCOER
10/12/2023 9
Computer Engineering
 PCCOER
10/12/2023 10
Computer Engineering
 FEASIBILITY AND SCOPE

• The cost of implementing a transformer-based architecture for multi-sensor object detection in autonomous drones
will vary depending on the specific requirements of the application. However, the cost of hardware and software has
been decreasing in recent years, making it more feasible to implement these systems.

• Transformer-based architectures for multi-sensor object detection in autonomous drones can be used in a variety of applications,
including:
1. Search and rescue: Transformer-based architectures can be used to develop autonomous drone systems that can search
for and rescue people in disaster zones.
2. Disaster response: Transformer-based architectures can be used to develop autonomous drone systems that can assess
the damage and provide relief to people in disaster zones.
3. Agriculture: Transformer-based architectures can be used to develop autonomous drone systems that can monitor crops
and livestock.
4. Surveillance: Transformer-based architectures can be used to develop autonomous drone systems that can monitor
borders and other sensitive areas.
5. Delivery: Transformer-based architectures can be used to develop autonomous drone systems that can deliver packages.

PCCOER
10/12/2023 11
Computer Engineering
 CONCLUSION& FUTURE SCOPE

• The problem statement of using transformer-based architectures for multi-sensor object detection in
autonomous drones is a promising one. Transformer-based architectures have the potential to significantly
improve the accuracy and robustness of object detection in autonomous drones, especially in GPS-denied
environments.

PCCOER
10/12/2023 12
Computer Engineering
 REFERENCES
[1] Hildebrand, M., Brown, A., Brown, S., & Waslander, S. L. (2023). Assessing distribution shift in probabilistic object
detection under adverse weather. IEEE Access, 11, 3270447-3270462.

[2] Zhang, R., Zhang, J., Zhang, X., Zhao, J., & Zhang, M. (2023). A fine-grained object detection model for aerial images based
on YOLOv5 deep neural network. IEEE Transactions on Geoscience and Remote Sensing, 61(6), 1-11.

[3] Li, Z. W., Liu, S., Chen, H., & Chen, C. S. (2023). R-YOLOv5: A lightweight rotating object detection algorithm for real-
time detection of vehicles in dense scenes. IEEE Transactions on Intelligent Transportation Systems.

[4] Ikeda, T., Kondo, K., & Takiguchi, T. (2023). RGB-D salient object detection using saliency and edge reverse attention.
IEEE Transactions on Image Processing, 32(5), 2091-2103.

[5] Zhou, Y., Hu, H., Wu, P., & Sun, J. (2023). A CNN-Transformer hybrid model based on CSWin transformer for UAV image
object detection. IEEE Transactions on Circuits and Systems for Video Technology.

PCCOER
10/12/2023 13
Computer Engineering
 [6] Multi-Sensor Object Detection for Autonomous Drone Navigation in GPS-Denied Environments by A. H. Al-Kaff et
al., IEEE Access, 2022

7] Multisensory Surveillance Drone for Survivor Detection and Geolocalization in Complex Post-Disaster Environment
by K. Andra et al., Semantic Scholar, 2021

[8] Multi-Sensor Fusion for Object Detection in Autonomous Drones by G. K. Gupta et al., 2022 IEEE Aerospace
Conference, 2022.

[9] Multi-Sensor Fusion for Object Detection in Autonomous Aerial Vehicles: A Survey by J. Liang et al., IEEE Sensors
Journal, 2022

[10]Tan, M., Pang, R., Le, Q. V., & Chen, H. (2019). EfficientDet: Scalable and efficient object detection. In
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10781-10790).

[11]Zhou, X., Wang, D., & Krähenbühl, P. (2019). Objects as points. arXiv preprint arXiv:1904.09755.

[12]Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., & Zhu, Y. (2017). Deformable convolutional networks.
In Proceedings of the IEEE international conference on computer vision (pp. 767-776).

PCCOER
10/12/2023 14
Computer Engineering
[13]Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017). Focal loss for dense object detection. In
Proceedings of the IEEE international conference on computer vision (pp. 2999-3007).

[14]Zhang, Z., Pang, J., Chen, K., Van Gool, L., & Tian, Y. (2020). DetNet: A lightweight object detection network for
mobile devices. In Proceedings of the European Conference on Computer Vision (ECCV) (pp. 19-34).

[15]Han, K., Wang, Y., Tian, Q., Guo, J., Xu, C., Xu, C., & Tang, Y. (2020). GhostNet: More features from cheap
operations. arXiv preprint arXiv:1911.11907

PCCOER
10/12/2023 15
Computer Engineering
 PCCOER
10/12/2023 16
Computer Engineering