Method of processing images collected from

self-driving car control camera

Abstract
An autonomous vehicle (AGV) is an intelligent machine capable of knowing enough to
determine its own state of motion based on environmental conditions. A typical form of self-
driving car or robot is the ability to operate and control without human movement in
constantly changing conditions when traveling on city lanes. In this paper, we present
algorithms and processing methods for self-driving cars that mainly focus on lane-aware
processing and dealing with unexpected situations to take passengers from point A. to point B
above. Simple image processing algorithms using lane vector method based on Non-uniform
theory Bspline (to determine lanes), Uncertainty Resolving System (URS) (for obstacle
detection and handling) and Point Minimal Solution (for motion estimation). The results of the
algorithm study show that the system can handle more than 12 frames per second, which is
suitable for the requirement of precise motion control of autonomous vehicles without human
intervention.

I. Introducation
Self-driving cars are widely applied in many different fields in practice and play an important
role in traffic environments with high risk of accidents such as steep, slippery roads or
intelligent traffic systems. ( giải thích) The vehicle's intelligent automatic motion control can
help reduce accidents and traffic jams. ( nêu ví dụ, giải thích)

In principle, a simple AGV system would consist of two main components: the Preprocessor
and the Control Unit.

The preprocessor uses sensors, radar, GPS or computer vision systems with attached cameras
to acquire information from the environment such as limits, direction, width, curvature,
flatness. lanes as well as the appearance of obstacles, etc. In which, the use of cameras to
identify lanes in processing probwlems is quite popular and widely used. The advantage of
using a camera is that it is easy to observe, provides color, contrast and optical character
recognition, and is affordable. Accompanied by a few outstanding disadvantages such as
limited scanning angle, poor performance in low light conditions, ...

The controller will control the car or autonomous robot to move along its lane limits and can
avoid obstacles in the road if necessary. Autonomous vehicle navigation includes three main
steps: Perception and localization – Planning – Control. In this, vehicle control is the final step
in the navigation system and is usually performed using one of two independent controllers.
Lateral Controller: Adjust the steering angle so that the vehicle follows the reference line. The
controller minimizes the error between the current vehicle position and the predetermined
path. Longitudinal Controller: The vertical controller minimizes the difference between the
direction of the vehicle and the direction of the reference line. Helps the vehicle to move
stably without shaking and accelerate and decelerate more smoothly.

In this study, we propose simple image processing algorithms and methods to help
autonomous vehicles move and control obstacles within a certain lane limit. By using the
concept of lane vectors based on the Non-uniform Bspline (NUBS) theory [1] to construct the
limit lines for the left and right lanes. Then describe the Uncertainty Resolving System (URS)
with the Visual Grounding (VG) model [2] to detect the object mentioned in the command in
the visual scene. Finally, using the motion estimation algorithm Point Minimal Solution [3] to
give the most suitable motion for the autonomous vehicle. Experimental results show that the
algorithms are very successful in identifying large urban lanes, where there is a large shadow
noise, as well as successful in identifying medium and large sized obstructions to correct
direction of movement. However, limitations are evident in detecting small obstacles such as
predicting the left while the subject is on the right or not recognizing a few colored obstacles
mixed with the lane. ( mở rộng giới thiệu các nhóm xử lý khác)

II. Methods
1. Vectơ làn đường dựa trên lý thuyết Non-uniform Bspline (NUBS)

a) Modeling the lane using a B-Spline . curve.

The NUBS curve can be represented by a set of control points on polynomials, the higher the
degree of polynomials, the more accurate the curve description will be. The distance between
the nodes is equal, we call the curve a uniform (uniform) B-Spline curve, otherwise, an
irregular B-Spline curve.
 b) Determine the boundary.

Figure 1: Lane vector model

- In there:

+ Al, Bl,… are the control points for the left lane; Ar, Br,… are the control points for the right
lane.

+ l⃗1=⃗ r 1=⃗
Al Bland ⃗ A r B r are the left and right lane vectors.

+ α l , α r are the lane inclination angles

+ C'l and C'r are hypothetical control points

Detailed content of the algorithm to find control points:

- Step 1: Set up 2 horizontal scan lines at the empty area at the end of the image, find the
control points respectively Al, Bl, Ar, Br

- Step 2: Build 2 lane vectors 1l and 1r , then calculate the angle l and r according to the
formula

( ) ( )
x A −x B x A −x B
α l=arctg l l
, α r =arctg r r
(1)
yA−yB
l l
y A − yB
r r

- Step 3: Divide the remaining space of the image into 4 parts (based on the length of the lane
map image) using 3 horizontal sweep lines. Vectors l⃗1and ⃗
r 1 stretching we get vector ⃗
l ' 2 and ⃗
r '2

. These two vectors intersect with the horizontal sweep line 3, we get two points C’ l and C’r .
Take the midpoint CM of the two points above and sweep to the sides, we get two control
points Cl and Cr. In case the lane is not continuous or noisy we will take C’l and C’r do the next
control points.

c) Lane curvature estimation algorithm.

Figure 2: Lane curvature estimation model.

- Step 1: Calculate the angles that determine the curvature of the lane
( i=1 … 3 ) :

( )
xl −x l
α li =arctg ( i+1) i

yl − yl
( i+1) i

( )
x r −x r
α ri =arctg (i+ 1) i (2)
yr − y r
(i+ 1) i

- Step 2: Calculate the difference between the angles:

∆ α l 1=α l 2−α l 1 , ∆ α l 2=α l 3−α l 2 (3)

∆ α r 1=α r 2−α r 1 , ∆ α r 2 =α r 3−α r 2 (4 )

- Step 3: Calculate the mean of the angles:

α l=a l ∆α +bl ∆ α +c l α l 1 (5)

l2 l1

α r=ar ∆ α + br ∆ α + c r α r 1 (6)
r2 r1

In there: al, bl, cl (left lane) and ar, br, cr (with the right lane) established and selected
experimentally. Sign of al, bl, cl take it in turn ∆ α , ∆α , α l 1 and ar, br, cr take it ∆ α , ∆α , α r 1 . In
l2 l1 r2 r1
the test here, we take:
|al|=|ar|=3 ,|bl|=|b r|=2,|c l|=|c r|=1.

d) Algorithm to calculate the steering angle of the car:

With the curvature of the left lane α l and the right lane α r , The car's position on the lane is
denoted by pVehicle ( 0 ≤ p Vehicle ≤ W ) (W: lane width and take W = 256), The current direction of
the car is denoted by α Vehicle.

Figure 3. Description of the location of the car.

- Step 1: Calculate 2 coefficients k l and k r for the left and right lanes respectively using the
following formulas:

( W − pVehicle )
k l=2 ; k r=2(1−k l ) (7)
W

- Step 2: Calculate the output values of steering angle:

+ α l and α r opposite sign:

( k l α l +k r α r )
α Out = −α Vehicle (8)
(kl + kr )

+ α l and α r sign together:

α Out =¿

2. Uncertainty Resolving System (URS)

a) Visual Grounding (VG) model
Before using URS, we ask a VG model to find the target object introduced (𝑜 ∗) by the
command (𝑐) in a given image 𝐼. We can create a model using the following statement:
 o=arg max p( o=o∗¿ I , c , θ)
o ∈O I
(10)

With: - OI the set of all objects in the image

- θ the model parameters

We create a set of exclusively initialized VG models. To calculate the adjusted probability

distribution for the set, we use the following equation:
E
1
p E ( OI|Φ , θ ) = ∑ pe ( O I|Φ ,θ ) (11)
E e=2

With: - 𝑝e is the probability output of the second VG model

- 𝑝E is the adjusted distribution of the full set

- Φ the set of all inputs

In our tests, we will use the notation 𝐸𝑛𝑠𝐸 where 𝐸 indicates the number of models in the
set. In addition to correcting a model, the use of a composite channel increases accuracy.
Detect uncertain and uncertain objects

2. Uncertainty detection methods.

Uncertainty detection allows directly predicting whether the model was certain about its
output or uncertain. In the latter case, the method selects the objects causing the uncertainty
and collects them in the candidate set 𝑂𝑐 ⊆ 𝑂I. However, the candidate set is still small, so in
case there are too many uncertain subjects, the car will not react in time. Therefore, a method
is needed that should strike a balance between maximizing the uncertainty and limiting the
size of 𝑂𝑐. The methods are described as follows

- Softmax Addition (SA): relies on the softmax probability distribution adding up to 1. The
top-𝑘 objects from 𝑂I are selected based on their probability from distribution 𝑝(𝑂I|𝛷,𝜽), such
that the sum of these 𝑘 probabilities is higher then the sum of remaining |𝑂I|−𝑘 probabilities,
with |𝑂I| is the number of objects in 𝑂I

+ 𝑘 = 1, the model isclassified as certain in the candidate set 𝑂𝑐

+ 𝑘 >1 the model is uncertain with alltop-𝑘objects in the candidate set 𝑂𝑐

- Centroid Agglomerative Hierarchical Clustering (CAHC): starts the clustering with every
object forming its own cluster with the probability of the object 𝑝(𝑂𝐼|𝛷,𝜽) representing it.
Next, the method iteratively merges pairs of clusters if their centroids’ absolute distance is
smaller than a parameter 𝛿. The centroid of the merged cluster is computed as the average of
the probabilities of its objects.

+ No more clusters can be merged, there are no clusters within a distance 𝛿 of each other, the
one with the largest probability is selected.

+ If there is only one object in the cluster, the model is classified as certain.

+ Otherwise, it is uncertain with all the objects in the highest probability cluster being
members of the candidate set 𝑂𝑐.

- Thresholding: makes use of a threshold 𝜂 for classifying the model as certain or not. In case
of the softmax output of the model, the threshold (trained on the validation set) is applied over
the probability distribution 𝑝(𝑂𝐼|𝛷,𝜽) to create the candidate set 𝑂𝑐 asfollows:

∀𝑜∈𝑂𝑐⟺𝑝(𝑜|𝛷,𝜽)> 𝜂 (12)

3. Point Minimal Solution

The paper chose to use the in-camera correspondence distance as the default case for our
implementation and rely on an additional inter-camera correspondence to retrieve the scale in
the degenerate case when cars are moving. In some very rare cases where inter-camera factors
are not found, we pass the scale from the previous friend using a Kalman filter.

Figure 4 : Motion line profile

 Specifically, the car is performing a circular motion about the Instantaneous Center of Motion
(ICR) with the Ackermann model. The radius of uniform circular motion will go to infinity
when the car is moving in a straight line. The main goal of motion estimation is to calculate
the relative motion between Vk and Vk+1.

[ ] [ ]
cosθ −sinθ 0 cos φv
R= sinθ cosθ 0 , t=ρ sin φv (13)
0 0 1 0

In there θ is the relative function angle and ρ is the scale of the comparative translation. Here,
the z-axis of Vk out of the paper. Further observations can be made from the graph showing the
angle between ρ and the line perpendicular to the circle at Vk, therefore φv = θ/2. We
immediately see that the relative motion between frame Vk and Vk+! depends only on 2 scale
parameters ρ and yaw angle θ.

Putting equation (10) into (11), we get equation (12):

[ ER R0 ](14)
EGC =

EGC =¿

For brevity, let us now drop all the indices on the Pl ̈uckerline vector from Equation 2 and
simply denote 2 correspon-dence Pl ̈ucker lines as l=[u T (t C × u)T ]T from frame k and

l '=[u' T (t C ' ×u ' )T ]T from frame k+1. Perform l × EGC ×l' , I get:

θ θ
acosθ +bsinθ +cρcos + dρsin + e=0(16)
2 2

Where

a=−u w ( t C u' y −t C u' x ) −u ' w ( t C ' u y −t C ' u x )

x x x y

+u y ( t C ' u' x −t C u ' x )+u x ( t C u ' y −t C ' u' y )

w w w w

b=u x ( t C ' u ' w −t C ' u ' x ) +u y ( t C' u ' w −t C ' u ' y )

x w y w

−u x (t C uw −t C u x )−u ' y (t C uw −t C u y )
x w y w

c=u w u ' y −u y u ' w

d=ux u' w +uw u ' x

e=u w ( t C ' u ' y −t C ' u ' x ) +u ' w ( t C u y −t C u x )
x y x y

Here, the subscripts x, y and w refer to the componentsin the vector. Equation 16 is our new
GEC with the Ack-ermann motion model. We need 2 Pl ̈ucker line correspon-dences to solve
for the 2 unknowns ρ and θ in Equation 16. Denoting each set of known coefficients obtained
from each Pl ̈ucker line correspondence by (a1,b1,c1,d1,e1) and (a2,b2,c2,d2,e2), and using
the trigonometric half-angleformula

θ
cosθ=1−2sin 2 (14 a)
2

θ θ
sinθ=2 sin cos (14 b)
2 2

−e1−a1 ( 1−2 β ) −b1 ( 2αβ )

2
ρ= (15 a)
c 1 α + d1 β

−e2−a2 ( 1−2 β ) −b2 ( 2 αβ )

2
ρ= (15 b)
c2 α + d2 β

III. Results
1. Vectơ làn đường dựa trên lý thuyết Non-uniform Bspline (NUBS)

a) Experiment.
Case 1: The lane is almost straight line, there is no interference, and the lengths of the left and
right lanes are approximately the same

Figure 5. Description of case 1

Case 2: The lane is curved to the left or to the right and the image is noisy by tree shadows on
the road.
 Figure 6. Description of case 2
Case 3: The lane is curved to the left or to the right and the image is heavily disturbed by the
shadows of trees or light poles on the road.

Figure 7 and 8 . Description of case 3

2. Simulation results
Case 1: The lane is curved to the right and the lane is slightly disturbed
 Figure 9. Simulation results of case 1

The simulation results show that α l=63.889 ° , α r =−18.142° , α out =22.874 ° . This indicates that
from this position, the car will turn to the right with a relatively large steering angle of value
22.8740

Case 2: The right lane curves heavily to the left, the right lane and the left lane are lost.

Figure 10. Simulation results of case 2

The simulation results show that α l=0 ° , α r =−66.914 ° , α out =−33.457 ° . This indicates that from
this position, the car will turn to the left with a relatively large steering angle of value -33.457 0

2. Uncertainty Resolving System (URS)

The results were evaluated according to the intersection over union (IoU5) criterion
and the Talk2Car dataset
Table 1.The 𝐼𝑜𝑈 scores on the Talk2Car test of the used VG-model with different top-𝑘
scoring objects as input
In Table 1, we see that using the top 32 subjects resulted in the highest score 𝐼𝑜𝑈.5.
On the other hand, using only the top 8 subjects will give the lowest score. This should
come as no surprise because with 8 predicted objects, the probability of having the
correct object among the predicted objects is lower. On the other hand, we found that
top-64 performed worse than top-32. It can be seen that when there are many such
boxes, some low quality boxes may also be included, which may eventually damage the
model.
 Figure 11. Graph 𝐼𝑜𝑈.5 of the VG model was used and the combined VG model with two
uncertainties from 𝐸𝑛𝑠4 + EV and 𝐸𝑛𝑠5 + CF + EV.
As discussed above, Figủe 11 shows that the combination of the VG model and the
proposed uncertainty detection models largely improves the 𝑇ℎ.𝐼𝑜𝑈.5 metric when confronted
with difficult visual situations(for the depth subset we witness an increase of 12.1%), with
long language commands (for the longest sentences subset we witness anincrease of 7.8%) and
with referred objects in the commands that havean ambiguous interpretation in the visual
scene (for the ambiguoussubset we witness an increase of 12.6%), while for easier
situationsthe increase is less pronounced (for the short sentences subset wewitness an increase
of 5.9%)

3. Point Minimal Solution

 Figure 12 .Trajectories before and after using the loop.
The relative motions estimated by our algorithm are aggregated together to form the full
trajectory of the car. The blue line on the figure shows the trajectory recovered from the
relative motions estimated using the Kalman filter. Cumulative biases lead to loop closure
errors, which we eliminated by performing a loop close that puts the graph. The orbit after
closing (red line) is significantly closer to the GPS/INS ground truth (green line).

Figure 13 .Top view of trajectory and 3D map points after pose-graph loop-closure
and full bundle adjustment compared with GPS/INSground truth
IV. Conclusions
The above articles have proposed simple and suitable image processing algorithms for the
problem of construction, lane detection, obstacle and steering angle control for autonomous
vehicles to provide appropriate motion on the lane. allow. By using properly installed camera
systems along with simple and powerful algorithms, experiments have shown that the system's
intuitiveness in controlling self-driving cars is completely feasible in remote areas. The traffic
area is not too crowded. High processing speed, image acquisition, more than 13 frames per
second, low latency, the ability to set lane trajectory and provide feedback to obstacles on the
road with relatively high accuracy for see that the car can completely operate freely without
human intervention. Some of the strengths of the system can be seen as follows:
- The technique of using lane-vector allows to overcome the difficulties of noise on the image
to fully detect the lanes that are discontinuous, or noisy by other objects.

- Successfully modeled the camera system as a general camera by 2-point minimal solution

- Combining the use of the URS system with the VG model gives a positive result, increasing
by more than 9% compared to using the VG model alone.

V. REFERENCES
[1] https://123docz.net//document/5855309-xu-ly-anh-xe-tu-hanh.htm

[2]https://husteduvn.sharepoint.com/:b:/s/
20213ME2021Technicalwritingandpresentation134869/Ef_a-
b7IWP9DpeBbF61i42UBfUHiiW77YGpPa5oP-B7xMQ?e=Ekor8R

[3]https://husteduvn.sharepoint.com/:b:/s/
20213ME2021Technicalwritingandpresentation134869/Ea5twn00E-
hNlzCiq5dM07ABw0Pxw0ky-AJMCPjV4OKRFQ?e=uUeZre