Advanced Driver Assistance System

Dr. Shree Prasad M.

Asst. Prof., Dept. of E.E.E.,
BITS Pilani
O↵-Campus, Bangalore.

Advanced Driver Assistance System 1 / 41 BITS Pilani, Pilani Campus

Outline

1 Introduction to Vision Sensor and Image Formation.

2 Image Representation, Types, and Challenges

3 Image Processing Operations: Correlation, and Convolution.

Advanced Driver Assistance System 2 / 41 BITS Pilani, Pilani Campus

Why Vision Sensor???

What automotive Radars can do? - Detect objects.

* Range.
* Velocity.
* Angle/Direction.
What automotive Radars cannot do? - “Appearance (visual look)
information”.
Camera/Vision Sensors: Primary sensor in ADAS sensor suite.
) High resolution information (visual details).
) Mimics operation of human eye.
) Passive sensor: Collect light reflected by objects from the captured
scene.
) Low Cost.

Advanced Driver Assistance System 3 / 41 BITS Pilani, Pilani Campus

Visible Spectrum

Figure 1: Visible Spectrum 380 nm (780 THz) to 720 nm (415 THz); (Image
Source: Image Processing and Analysis by Stan Birchfield.)

Advanced Driver Assistance System 4 / 41 BITS Pilani, Pilani Campus

Image Formation

Figure 2: Image Source: “Digital Image Processing by R. C. Gonzalez, and R. E.

Woods.”

Advanced Driver Assistance System 5 / 41 BITS Pilani, Pilani Campus

Image Representation

Image Sensor On Semi AR0820AT Sony IMX324 Samsung S5K2G1

Resolution 3848 ⇥ 2168 3849 ⇥ 1929 3840 ⇥ 1920
Pixel Size (µm) 2.1 2.25 2.1
Pixel Array Size (mm) 8.081 ⇥ 4.553 8.660 ⇥ 4.340 8.064 ⇥ 4.032

Table 1: Source: F. E. Sahin, “Long-Range, High-Resolution Camera Optical Design for

Assisted and Autonomous Driving,” Photonics, 2019.

Advanced Driver Assistance System 6 / 41 BITS Pilani, Pilani Campus

Image Representation Types

1 Monochromatic or Gray scale Image.

2 Color/RGB Image.

Gray Scale Image

* Each pixel represents intensity level.
* No reflection (from object) - Intensity level = 0, and Maximum reflection -
Intensity level = 255.

Figure 3: Image Source: Cityscape Dataset.

Advanced Driver Assistance System 7 / 41 BITS Pilani, Pilani Campus

Image Representation: Color/RGB Image

Color percevied in an object: Nature of light reflected by object.

RGB/BGR Image

Figure 4: Image Source: Cityscape Dataset

Advanced Driver Assistance System 8 / 41 BITS Pilani, Pilani Campus

Computer Vision in ADAS: Visual Perception

Application of Computer Vision in ADAS

* Lane Detection.
* Traffic Light Color Recognition.
* Detection and Recognition of Traffic Signs.
* Recognizing and tracking various object in the scene.
* Depth estimation.
Computer Vision Techniques.
* Traditional.
* Machine Learning.

Advanced Driver Assistance System 9 / 41 BITS Pilani, Pilani Campus

Challenges:

IEEE-2020 Automotive Image Standard.

1 Noise: During acquisition, and transmission [1].
2 Occlusion: Objects occuled/partially blocked by background (or other
objects).
* Pedestrians occluded by the background such as other vehicles,
buildings, and mailboxes on the road [2, 3].

Figure 5: Image Source [4].

Advanced Driver Assistance System 10 / 41 BITS Pilani, Pilani Campus

Challenges Contd
3 Motion blur

Figure 6: Image Source: https://autonomousvision.github.io/motion-deblur/

4 Adversarial attacks

Figure 7: Image Source: https://arxiv.org/pdf/1707.08945.pdf

Advanced Driver Assistance System 11 / 41 BITS Pilani, Pilani Campus
Basic Camera Parameters

i Resolution (MP).
* Large resolution, object will be represented
with more pixels.
* Increasing resolution: Add more pixel
sensor or reduce pixel sensor size.
ii Frames per seconds (fps).
* Large fps - Track dynamic objects.
* Small fps - Track stationary objects.
iii FOV (Angle). Figure 8: Image Source:
* Wide Angle - Near Range View. Internet.
* Narrow Angle - Long Range View.
https://blogs.nvidia.com/blog/2019/04/15/how-does-a-self-driving-car-
see/

iv High Dynamic Range [dB].

https://www.sony.com/en/SonyInfo/News/Press/201704/17-034E/

v Mono or Stereo.
Advanced Driver Assistance System 12 / 41 BITS Pilani, Pilani Campus
Basics of Image Processing: Correlation

Advanced Driver Assistance System 13 / 41 BITS Pilani, Pilani Campus

Significance of Time Shifting

Advanced Driver Assistance System 14 / 41 BITS Pilani, Pilani Campus

Correlation Contd.

1
X
rfh [no ] = f [n] h [n no ] (1)
n= 1

Note: n no represents no samples behind the current index n.

Replace n no = i
1
X
rfh [no ] = h [i] f [i + no ] (2)
i= 1

Note: i + no represents no samples ahead of the current index m.

Advanced Driver Assistance System 15 / 41 BITS Pilani, Pilani Campus

Image Processing Basic Contd.

Basic Operation on Signals

1. Time shifting.
2. Time Reversal.
3. Upsampling or Downsampling.

Advanced Driver Assistance System 16 / 41 BITS Pilani, Pilani Campus

Time Shifting

x [n] ! x [n ± no ]

Advanced Driver Assistance System 17 / 41 BITS Pilani, Pilani Campus

Time Reversal

x [n] ! x [ n]

Advanced Driver Assistance System 18 / 41 BITS Pilani, Pilani Campus

Numerical Problem

i) x [ n 3] ?

Advanced Driver Assistance System 19 / 41 BITS Pilani, Pilani Campus

Basics of Image Processing: Convolution

Advanced Driver Assistance System 20 / 41 BITS Pilani, Pilani Campus

Convolution Contd.

1
X
v [no ] = f [n] g [n no ] (3)
n= 1

Note g [n] = h [ n]
1
X
v [no ] = g [i] f [i + no ] (4)
i= 1

Advanced Driver Assistance System 21 / 41 BITS Pilani, Pilani Campus

2D-Correlation

Figure 12: “Mask”,

“Kernal” or
“Filter.”
Figure 11: Image

Advanced Driver Assistance System 22 / 41 BITS Pilani, Pilani Campus

2D-Correlation Steps

Figure 13: Horizontal (right) shift = 0 and Verticle (down) shift = 0

Advanced Driver Assistance System 23 / 41 BITS Pilani, Pilani Campus

2D-Correlation Steps

Figure 14: Horizontal (right) shift = 1 and Verticle (down) shift = 0

Advanced Driver Assistance System 24 / 41 BITS Pilani, Pilani Campus
2D-Correlation Steps

Figure 15: Horizontal (right) shift = 2 and Verticle (down) shift = 0

Advanced Driver Assistance System 25 / 41 BITS Pilani, Pilani Campus
2D-Correlation Steps

Figure 16: Horizontal (right) shift = 6 and Verticle (down) shift = 0

Advanced Driver Assistance System 26 / 41 BITS Pilani, Pilani Campus

2D-Correlation Steps

Figure 17: Horizontal (right) shift = 0 and Verticle (down) shift = 1

Advanced Driver Assistance System 27 / 41 BITS Pilani, Pilani Campus

2D-Correlation Steps

Figure 18: Horizontal (right) shift = 1 and Verticle (down) shift = 1

Advanced Driver Assistance System 28 / 41 BITS Pilani, Pilani Campus

2D-Correlation Steps

Figure 19: Horizontal (right) shift = 6 and Verticle (down) shift = 3

Advanced Driver Assistance System 29 / 41 BITS Pilani, Pilani Campus

2D-Correlation Contd.

1
X 1
X
v [x0 , y0 ] = f [x, y ] h [x x0 , y y0 ] (5)
x= 1 y = 1

Replace x x0 = i, and y y0 = j
1
X 1
X
v [x0 , y0 ] = h [i, j] f [x0 + i, y0 + j] (6)
i= 1 j= 1

Change of variables x0 = x, and y0 = y

Correlation
1
X 1
X
v [x, y ] = h [i, j] f [x + i, y + j] (7)
i= 1 j= 1

Advanced Driver Assistance System 30 / 41 BITS Pilani, Pilani Campus

2D-Convolution

Mirror (or spatial reverse) h [x, y ], i.e., g [x, y ] = h [ x, y ]

Advanced Driver Assistance System 31 / 41 BITS Pilani, Pilani Campus

2D-Convolution Steps

Figure 20: Horizontal (right) shift = 0 and Verticle (down) shift = 0

Advanced Driver Assistance System 32 / 41 BITS Pilani, Pilani Campus

2D-Convolution Steps

Figure 21: Horizontal (right) shift = 1 and Verticle (down) shift = 0

Advanced Driver Assistance System 33 / 41 BITS Pilani, Pilani Campus
2D-Convolution Steps

Figure 22: Horizontal (right) shift = 2 and Verticle (down) shift = 0

Advanced Driver Assistance System 34 / 41 BITS Pilani, Pilani Campus
2D-Convolution Steps

Figure 23: Horizontal (right) shift = 6 and Verticle (down) shift = 0

Advanced Driver Assistance System 35 / 41 BITS Pilani, Pilani Campus

2D-Convolution Steps

Figure 24: Horizontal (right) shift = 0 and Verticle (down) shift = 1

Advanced Driver Assistance System 36 / 41 BITS Pilani, Pilani Campus

2D-Convolution Steps

Figure 25: Horizontal (right) shift = 1 and Verticle (down) shift = 1

Advanced Driver Assistance System 37 / 41 BITS Pilani, Pilani Campus

2D-Convolution Steps

Figure 26: Horizontal (right) shift = 6 and Verticle (down) shift = 3

Advanced Driver Assistance System 38 / 41 BITS Pilani, Pilani Campus

2D-Convolution

Convolution
1
X 1
X
v [x, y ] = g [i, j] f [x + i, y + j] (8)
i= 1 j= 1

where g [x, y ] = h [ x, y ]

Advanced Driver Assistance System 39 / 41 BITS Pilani, Pilani Campus

References I

[1] W. Liu, Z. Wang, X. Liu, N. Zeng, Y. Liu, and F. E. Alsaadi, “A

survey of deep neural network architectures and their applications,”
Elsevier Neurocomputing, 2017.
[2] A. Boukerche and M. Sha, “Design Guidelines on Deep
Learning–based Pedestrian Detection Methods for Supporting
Autonomous Vehicles,” ACMComputing Surveys, July 2022.
[3] S. Gilroy, E. Jones and M. Glavin, “Overcoming Occlusion in the
Automotive Environment—A Review,” IEEE Transactions on
Intelligent Transportation Systems, vol. 22, no. 1, pp. 23-35, Jan.
2021.
[4] C. Ning, et. al. “Survey of pedestrian detection with occlusion”.
Complex & Intelligent Systems, 2021.

Advanced Driver Assistance System 40 / 41 BITS Pilani, Pilani Campus

 Thank You

Advanced Driver Assistance System 41 / 41 BITS Pilani, Pilani Campus