Research on Lane Detection using Deep learning
Antriksh Gupta
School of Mechanical engineering
Dr Vishwanath karad MIT world peace university Pune,
1032181729@mitwpu.edu.in
Abstract -Road boundary lanes are one of the serious
causes of road accidents and it affects the driver and
people’s safety. Detecting road boundary lanes is a
challenging task for both computer vision and machine
learning approaches. In recent years many machine
learning algorithms have been deploying but they failed
to produce high efficiency and accuracy. Unmanned
vehicles need to learn to observe the road from the visual
point of view if they want to achieve automatic driving,
which specifically is the detection of lane lines. This
includes identifying the positional relationship between
the lane line and the car, whether it is a solid line or a
dotted line. this paper proposes the use of Hough
transform to achieve straight-track detection of lane
detection, while for the detection of curved sections, the
tracking algorithm is studied. By controlling the slope of
the lane lines in the two frames before and after
comparison, a limitation is made near the previously
detected lane line area, ie, a region of interest (ROI) is set,
and a search for a corner pixel is performed in the
direction, for the corner portion Rebuild. The
experimental results show that the algorithm has the
characteristics of fast operation speed, high accuracy and
good robustness
Index Terms -driverless; lane detection; Convolutional neural
network; semantic segmentation; autonomous driving
INTRODUCTION
The last three decades have seen steadily increasing research
efforts, both in academia and in industry, towards developing
driverless vehicle technology. These developments have
been fueled by recent advances in sensing and computing
technology together with the potential transformative impact
on automotive transportation and the perceived societal
benefit: India is one of the busiest countries in the world in
terms of road traffic. The automotive industry across the
south Asian country became the fourth largest in the world in
2017. In 2019, there were almost three million new car
registrations in the country. The Indian road network,
spanning over five million kilometers, carried almost 90
percent of the country’s passenger traffic and about 65
percent of the goods. With the rapid increase in the number
of cars and the mercilessly congested Indian roads, road
safety has turned into a factor of utmost importance for the
country’s citizens. The number of road accidents across India
amounted to around 449 thousand in 2019. Each year, about
three to five percent of the GDP of the country was invested
in road accidents.Table 1 show you number of accident
caused by human error In India during year 2017-2018.
The data was taken from the Transport Research Wing of the
Ministry of Road Transport and Highways publishes an
annual report. Autonomous vehicles have the potential to
dramatically reduce the contribution of driver error and
negligence as the cause of vehicle collisions. They will also
provide a means of personal mobility to people who are
unable to drive due to physical or visual
disability.autonomous vehicles would facilitate more
productive use of the transit time, or simply reduce the
measurable ill effects of driving stress .The driverless system
is a complex intelligent control system that integrates
multiple modules such as mechanical control, path planning,
and intelligent sensing, and finally uses the in-vehicle
computer system to realize automatic driving operations. The
purpose of the development of the driverless system is to
provide better safety and comfort for car driving, develop
smart transportation in the city, and solve a series of problems
such as urban traffic congestion. Therefore, the research of
unmanned driving has great practical significance and the
future prospect is very broad. The sensing module is an
important module in the driverless system. It mainly senses
the driving environment during the driving process of the
vehicle, and senses vehicles, pedestrians, obstacles and other
objects in the surrounding environment of the vehicle, and
provides the result of the sensing and the path decision
module. The corresponding path planning is carried out, and
finally the mechanical control module realizes the relevant
mechanical control operation, so that the car can drive
automatically. Lane line detection is an important part of the
sensing module. Unmanned driving not only requires
obstacle avoidance and road traffic information perception,
but also needs to comply with traffic rules. The requirements
for lane line detection are relatively high. Many traffic rules
are designed so that pedestrians and vehicles must move
according to certain rules. In addition to the traffic signal, the
reference standard is the road lane line. By detecting the lane
line, the ground indicator can be further detected and the front
collision warning strategy can be designed. This paper tries
to propose effective lane detection algorithm and its methods
to solve one aspect of the problem stated . This lane detection
primarily focuses on urban and highway road condition that
can detect straight and curved road on these condition .open
source library such as tensor flow is used for the solution
implementation and its primarily done Python 3.9.algorithms
used are YOLOP (You only look once for panoptic driving
perception).Virtual driving simulation softer udacity driving
simulation model is used for training and testing.
Table 1.1 Road accident caused
Source - https://morth.nic.in/
LITERATURE REVIEW
In lane detection, there are lots of innovative researches based
on deep learning.[7]constructs a dual-branch network 3 to
perform semantic segmentation and pixel embedding on
images. It further clusters the dual-branch features to achieve
lane instance segmentation.
In this section, Lane detection and tracking algorithms are
discussed. Table 1 summarizes and presents a detailed
analysis of various lane detection and lane tracking
algorithms. This section also investigates the best lane
detection and tracking algorithms that can be selected for
a specific road conditions. Yim and Oh [8] developed a three
feature based lane detection algorithm. The features used are
starting position, orientation and intensity value. In the initial
step, a Sobel operator is applied to get the edge information.
The lane boundary is represented as a vector comprising of
the three features. The current lane vector is calculated based
on the input image and the previous lane model vector. Two
windows, one for each, is used for left and right boundaries.
Assuming N pixelin each horizontal line, N lane vector
candidates are generated. The best candidate is selected based
on the minimum distance from previous lane vector using a
weighted distance metric. For equalization each feature is
assigned a different weight. Then a lane inference system is
used top redict the new lane vector. If the road width changes
abruptly, the current vector calculated is discarded and the
previous one is taken as current vector. A lane detection
approach for urban environment is proposed by Sehestedt et
al. Since the lane markers are not clearly visible due to wear
and tear, occlusions and due to complex road geometry, a
weak model is used for detecting lane markers. In the inverse
perspective mapped image, particle filter is applied from
bottom row to top. The filter is tuned in such a way to track
multiple lanes. Aly [4] presents a real time and robust
approach to detect lane markers in urban roads. It first
generates a top view of the road image using inverse
perspective mapping for avoiding the perspective effect.
Then the top view is filtered using selective oriented two
dimensional Gaussian kernel. The filter is tuned specifically
for bright lines in dark background with specific width. So it
has high response to the line markers and retain only the
highest values by selecting q% quantile value from filtered
image and removing all values below the threshold. Then the
straight lines are detected using simplified Hough transform,
which is followed by RANSAC line fitting which provides
initial guess to the RANSAC spline fitting step. Then a post
processing step is done to localize the spline and extend it in
the image. The algorithm does not perform tracking. It can
detect any number of lane boundaries in the image not just
the current lane. Kim [3] developed a lane detection and
tracking algorithm which can handle challenging scenarios
such as faded lane markers, lane curvatures and splitting
lanes. In the initial step, agradient detector and an intensity
bump detector is used to eliminate the non lane markers.
Artificial Neural Networks (ANN) is applied on remaining
samples for lane detection. The detected lane markers pixels
are grouped using cubic splines. Hypotheses are generated
from random set of line segments. RANSAC algorithm helps
in validating the hypotheses. Particle filtering is used for
lane tracking. Cheng et al. [9] introduced a hierarchical
algorithm for lane detection. High dimensional feature points
are extracted based on feature color extraction. It is used to
distinguish structured roads from unstructured roads. Then
connected components is applied on the feature points. A
feature vector is constructed for feature points exceeding a
threshold. Eigen value Decomposition Regularized
Discriminant Analysis is done to reduce the variance of the
model. Then maximum likelihood Gaussian parameters are
estimated. The extracted feature points are used as detected
lanes in structured roads. For unstructured roads, the entire
scene is divided based on mean-shift segmentation. Each
region is considered to be homogeneous and lane markers are
detected using Bayes rule
 Table 2.1 Comparison of various lane detection and tracking algorithms

Author Preprocessing Detection Tracking Evaluation Comments

Xianwen Wei,et Image Hough transform Kalman filtering This algorithm is Further research
al.[1] preprocessing And region of not only Is needed for
interest applicable to selection of
structured roads region of
with lanes such interest.
as roads
Dong Wu,et al. SCNN, Enet- You only look Can perform real a simple and
(2021)[2] SAD once for panoptic time interface on efficient network
driving embedded device which can
perception Used for both simultaneously
(YOLOP) object and lane handle three
detection driving
perception.
Z. Kim (2008) Edge detector, Artificial Neural Particle filter Robust in Suitable for both
[3] Intensity bump Network (ANN) presence of straight and
detector leading vehicles curved roads
and low
illumination
Y. U. Yim and Sobel operator Hough transform Temporal Works fine for A priori
S.-Y. Oh (2003) and three feature predictor is used rainy and shady information on
[4] vector to predict current road road is not
lane vector needed

S. Sehestedt, et Inverse Weak model Clustered Robust in Suitable for both

al. (2007) [5] perspective based particle filter difficult lighting straight and
mapping conditions slightly curved
urban roads
H.Jung et al. Steerable filter Haar like Robust in Lane departure
(2013) [6] features illumination warning included
changes

PROPOSED METHODOLOGY
Out of all literature discussed above panoptic perception network,termed as YOLOP, contains one shared encoder
system was found most suitable for our applications and its and three subsequent decoders to solve specific tasks.
methodology is described . There are no complicated and redundant shared blocks
A panoptic driving perception system is an important part between different decoders, which reduces computational
of autonomous system. A high-precision and real-time consumption and allows network to be easily trained end
perception system can assist the targeted vehicle in making to end.
the reasonable decision while driving. panoptic driving
perception network (YOLOP) to perform traffic object
detection, drivable area segmentation and lane detection
simultaneously. It is composed of one encoder for feature
extraction and three decoders to handle the specific tasks.
This model performs well on the challenging BDD100K
dataset, achieving results on all three tasks in terms of
accuracy and speed.[2]

This is a simple and efficient feed-forward network

that can accomplish traffic object detection, drivable area
segmentation and lane detection tasks altogether. As shown
in Figure 1, our panoptic driving perception single-shot
Fig. 3.1 The architecture of YOLOP.
 A. Encoder
The network shares one encoder, which is composed of a
backbone network and a neck network.
1) Backbone: The backbone network is used to extract
the features of the input image. Usually, some classic
image classification networks serve as the backbone. Due
to the excellent performance of YOLOv4 [2] on object
detection, we choose CSP Darknet as the backbone, which
solves the problem of gradient duplication during
optimization . It supports feature propagation and feature
reuse which reduces the amount of parameters and
calculations. Therefore, it is conducive to ensuring the real-
time performance of the network.
2) Neck: The neck is used to fuse the features generated
by the backbone. Our neck is mainly composed of Spatial
Pyramid Pooling (SPP) module and Feature Pyramid
Network (FPN) module . SPP generates and fuses features
of different scales, and FPN fuses features at different
semantic levels, making the generated features contain
multiple scales and multiple semantic level information.
We adopt the method of concatenation to fuse features in
our work.[2]
B. Decoders
The three heads in our network are specific decoders for
the Three tasks.
1) Detect Head: Similar to YOLOv4, we adopt an
anchorbased multi-scale detection scheme. Firstly, we use
a structure called Path Aggregation Network (PAN), a
bottom-up feature pyramid network. FPN transfers
semantic features topdown, and PAN transfers positioning
features bottom-up. We combine them to obtain a better
feature fusion effect, and then directly use the multi-scale
fusion feature maps in the PAN for detection. Then, each
grid of the multi-scale feature map will be assigned three
prior anchors with different aspect ratios, and the detection
head will predict the offset of position and the scaling of
the height and width, as well as the corresponding
probability of each category and the confidence of the
prediction.
2) Drivable Area Segment Head & Lane Line Segment
Head: Drivable area segment head and Lane line Segment
head adopt the same network structure. We feed the bottom
layer of FPN to the segmentation branch, with the size of
(W/8, H/8, 256). Our segmentation branch is very simple.
After three upsampling processes, we restore the output
feature map to the size of (W, H, 2), which represents the
probability of each pixel in the input image for the drivable
area/lane line and the background. Because of the shared
SPP in the neck network, we do not add an extra SPP
module to segment branches like others usually do, which
brings no improvement to the performance of our network.
Additionally, we use the Nearest Interpolation method in
our upsampling layer to reduce computation cost instead of
deconvolution. As a result, not only do our segment
decoders gain high precision output, but also be very fast
during inference.
C. Loss Function
Since there are three decoders in our network, this multi-
task loss contains three parts. As for the detection loss Ldet,
it is a weighted sum of classification loss, object loss and
bounding box loss as in equation 1.
Ldet = α1Lclass + α2Lobj + α3Lbox, (1)
where Lclass and Lobj are focal loss , which is utilized
to reduce the loss of well-classified examples, thus forces
the network to focus on the hard ones. Lclass is used for
penalizing classification and Lobj for the confidence of one
prediction. Lbox is LCIoU , which takes distance, overlap
rate, the similarity of scale and aspect ratio between the
predicted box and ground truth into consideration.
Both of the loss of drivable area segmentation Lda−seg and
lane line segmentation Lll−seg contain Cross Entropy Loss
with Logits Lce, which aims to minimize the classification
errors between pixels of network outputs and the targets. It
is worth mentioning that IoU loss: LIoU = T N
T N+F P +F N is
added to Lll−seg as it is especially efficient for the
prediction of the sparse category of lane lines. Lda and
Lll−seg are defined as equation (2), (3) respectively.
Lda−seg = Lce, (2)
Lll−seg = Lce + LIoU . (3)
In conclusion, our final loss is a weighted sum of the three
parts all together as in equation (4).
Lall = γ1Ldet + γ2Lda−seg + γ3Lll−seg, (4)
where α1, α2, α3, γ1, γ2, γ3 can be tuned to balance all parts
of the total loss.
D. Training Paradigm
various paradigms were attempted to train our model. Most
suitable one is training end to end, and then three tasks can
be learned jointly. This training paradigm is useful when
all tasks are indeed related. In addition, some alternating
optimization algorithms also have been tried, which train
our model step by step. In each step, the model can focus
on one or multiple related tasks regardless of those
unrelated. Even if not all tasks are related, the model can
still learn adequately on each task with this paradigm.
CONCLUSION
The Hough line detection method is accurate and simple,
and the curve detection can be performed after adding the
tracking algorithm. The fitting method is unstable and has
the advantage of being able to detect curves. The advantage
of the affine transformation method is that multi-lane
detection can be performed. The disadvantage is that in a
complex situation, the vehicle or other object in front can
be easily obstructed and suffer severe interference.
Therefore, the detection of lane lines needs to adapt to
various conditions. Thus use of panoptic driving system is
proposed. Panoptic driving perception system is an
important part of autonomous system. A high-precision
and real-time perception system can assist the targeted
vehicle in making the reasonable decision while driving.
panoptic driving perception network (YOLOP) to perform
traffic object detection, drivable area segmentation and
lane detection simultaneously.
 REFERENCES

[1] Wei, X., Zhang, Z., Chai, Z., & Feng, W. (2018). Research on Lane
Detection and Tracking Algorithm Based on Improved Hough
Transform. 2018 IEEE International Conference of Intelligent
Robotic and Control Engineering (IRCE).
[2] Dong Wu and Manwen Liao and Weitian Zhang and Xinggang
Wang.(2021)YOLOP: You Only Look Once for Panoptic Driving
Perception.

[3] Z.Kim, "Robust lane detection and tracking in challenging

scenarios," IEEE Trans. Intell. Transp. Syst. , vol. 9, no. 1, pp. 16-
26, Mar. 2008.
[4] Y.U. Yim and S.- Y. Oh, "Three-feature based automatic lane
detection algorithm (TFALDA) for autonomous driving," IEEE
Trans. Intell. Transp. Syst. , vol. 4, no. 4, pp. 219-225, Dec. 2003.
[5] S.Sehestedt, S. Kodagoda, A. Alempijevic, and G. Dissanayake,
"Robust lane detection in urban environments," in Proc. IEEE Intell.
Robots Syst., Oct. 2007, pp. 123-128
[6] H.Jung, J. Min, and J.Kim,"An efficient lane detection algorithm for
lane departure detection," In Intelligent Vehicles Symposium (IV ),
2013 IEEE, pp. 976-981
[7] H.Tan, Y. Zhou, Y.Zhu, D.Yao, and K. Li, “A novel curve lane
detection based on Improved River Flow and RANSAC,” In
Intelligent Transportation Systems (ITSC), 2014 IEEE 17th
International Conference on, pp.133-138. IEEE, 2014.
[8] Y.U. Yim and S.- Y. Oh, "Three-feature based automatic lane
detection algorithm (TFALDA) for autonomous driving," IEEE
Trans. Intell. Transp. Syst. , vol. 4, no. 4, pp. 219-225, Dec. 2003.
[9] H.Y. Cheng, C.C. Yu, , C. C. Tseng, K. C. Fan,J. N. Hwang, and B.
S. Jeng, "Hierarchical lane detection for different types of roads,"
In Acoustics, Speech and Signal Processing (ICASSP), 2008 IEEE
International Conference on pp. 1349-1352. IEEE, 2008