IEEE/CAA JOURNAL OF AUTOMATICA SINICA, VOL. 7, NO.
2, MARCH 2020
Proximity Based Automatic Data Annotation
for Autonomous Driving
Chen Sun, Student Member, IEEE, Jean M. Uwabeza Vianney, Ying Li, Long Chen, Member, IEEE, Li Li, Fellow,
IEEE, Fei-Yue Wang, Fellow, IEEE, Amir Khajepour, and Dongpu Cao, Member, IEEE
    Abstract — The recent development in autonomous driving in-
volves high-level computer vision and detailed road scene under-
standing. Today, most autonomous vehicles employ expensive
high quality sensor-set such as light detection and ranging (LID-
AR) and HD maps with high level annotations. In this paper, we
propose a scalable and affordable data collection and annotation
framework, image-to-map annotation proximity (I2MAP), for af-
fordance learning in autonomous driving applications. We
provide a new driving dataset using our proposed framework for
driving scene affordance learning by calibrating the data samples
with available tags from online database such as open street map
(OSM). Our benchmark consists of 40 000 images with more than
40 affordance labels under various day time and weather even
with very challenging heavy snow. We implemented sample ad-
vanced driver-assistance systems (ADAS) functions by training
our data with neural networks (NN) and cross-validate the res-
ults on benchmarks like KITTI and BDD100K, which indicate
the effectiveness of our framework and training models.
    Index Terms—Affordance learning, autonomous vehicles, data syn-
chronization, scene understanding.
I.  Introduction
UTONOMOUS driving became popular in the research
A field in recent years. The information technology and
autonomous driving systems can in all be used to promote bet-
ter commuting choices, provide best route planning, improve
bus scheduling and routing and finally reduce travel time and
traffic congestion. A safe and robust autonomous driving sys-
Manuscript received January 21, 2020; revised February 4, 2020; accepted
February 12, 2020. Recommended by Associate Editor Lingxi Li. (Corres-
ponding author: Dongpu Cao.)
Citation: C. Sun, J. M. Uwabeza Vianney, Y. Li, L. Chen, L. Li, F.-Y.
Wang, A. Khajepour, and D. P. Cao, “Proximity based automatic data annota-
tion for autonomous driving,” IEEE/CAA J. Autom. Sinica, vol. 7, no. 2, pp.
395–404, Mar. 2020.
C. Sun, J. M. Uwabeza Vianney, A. Khajepour, and D. P. Cao are with the
Department of Mechanical and Mechatronics Engineering, University of Wa-
terloo, Waterloo ON N2L3G1, Canada (e-mails: chen.sun@uwaterloo.ca;
jmuviann@uwaterloo.ca; dongpu.cao@uwaterloo.ca; a.khajepour@uwater-
loo.ca).
Y. Li is with the Department of Geography and Environmental Manage-
ment, University of Waterloo, Waterloo ON N2L3G1, Canada (e-mail:
y2424li@uwaterloo.ca).
L. Chen is with the School of Data and Computer Science, Sun Yat-sen
University, Zhuhai 519082, China, and also with Waytous Inc., Qingdao
266109, China (e-mail: chenl46@mail.sysu.edu.cn).
L. Li is with the Department of Automation, Tsinghua University, Beijing
100084, China (e-mail: forcelee@gmail.com).
F.-Y. Wang is with the State Key Laboratory of Management and Control
for Complex Systems, Institute of Automation, Chinese Academy of Sci-
ences, Beijing 100190, China (e-mail: feiyue.wang@ia.ac.cn).
Color versions of one or more of the figures in this paper are available on-
line at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/JAS.2020.1003033
Authorized licensed use limited to: University of Exeter. Downloaded on June 20,2020 at 14:15:13 UTC from IEEE Xplore. Restrictions apply.
395
tem could also greatly reduce traffic accidents caused by hu-
man drivers [1]. To design a safe and robust autonomous driv-
ing system, the ability to understand the driving environment
as well as the current vehicle state is essential [2], [3]. The
techniques used in environment perception varied from simple
object marker detection by hand-crafted rules [4] to recent
deep learning approach [5]. The final goal is primarily to have
an affordable and robust system applicable under diverse en-
vironments.
One of the most frequently used autonomous driving
framework among car companies is the modular pipeline
approach where expensive LIDAR, high accurate global
navigation satellite system (GNSS) and 3-D high definition
maps are used to reconstruct the consistent world
representation of the surrounding environments [6], [7]. The
ego vehicle then takes all the information into account and
make further control decisions. However, such a way of
perception is costly and raises problems in both storage space
and processing speed. Furthermore, as mentioned in [8], a
human driver only needs relatively compact driving
information to make driving and control decisions. Instead of
reconstructing the three-dimensional high definition map with
bounding boxes of other traffic participants, a compact set of
driving affordance set, or even vehicle states [9], may be
enough for control decisions. On the other hand, end-to-end
learning [10] and direct perception [8] attempt to mapping
directly from camera images to either control inputs or driving
scene affordances. The end to end learning for autonomous
driving enjoys cheap annotation of training data set; however,
it is hard to interpret the control decisions. The direct
perception approach proposed in [8] leverage interpret-ability
by using compact annotations of driving scene affordances.
Autonomous driving systems trained in both ways are highly
dependent on the distribution and label accuracy of training
data sets. The data collection and attributes annotation for
neural network-based training methods raised several
problems, e.g., how to collect driving data in a scalable way in
a diverse environment, and how to ease the human annotation
effort in training data affordance labelling.
An alternative to manual annotation is offered by modern
computer graphic techniques which allow generating large-
scale synthetic data-sets with pixel-level ground truth.
However, the creation of photo-realistic virtual worlds is time-
consuming and expensive. Games like TORCS, limited by its
development, the image and video in the game is not photo-
realistic enough. Recently released open-sourced simulation
396
platform CARLA [11] and AIRSIM [12] are built on unreal
engine, which the image generated are more realistic and have
the flexibility to define various kinds of weather and
environment settings. Autoware [13] shares real public roads
and region data in Japan for the software test of self-driving
vehicles. However, the problem is that the simulator does not
represent reality. It is more reasonable to collect and to label
the driving data in a distributed approach. Xu et al. [14]
published BDD100K data set where the diverse driving data is
collected in a distributed way by Uber drivers across
California and New York and annotated by human labour. The
driving data are collected with a phone camera while the main
effort of manual annotation task is not relieved. In [15], the
authors deploy the APIs from open street map (OSM) [16] to
tag images from google street view (GSV). It is indeed a
clever implementation where the training samples are
available online (no driver needed) and the labels are free to
use from OSM. However, the major drawbacks of this scheme
are 1) low diversity of the driving samples due to GSV are
collected in sunny day time only; 2) low accuracy of the
sample labels due to the calibration between GNSS and OSM
and lack of label filtering.
This work provides the following contributions: 1) We
present an affordable, scalable driving data collection and
annotation scheme, image-to-map annotation proximity
(I2MAP) with GNSS calibration and confidence score
filtering for autonomous driving system, which provides
diverse samples and boosted label accuracy; 2) We propose
our driving scene understanding benchmark named Ontario
driving dataset-affordances (ODD-A), which is composed of
more than 40 000 samples with more than 40 labels under
various weather, light-condition and road segments. We
provide multiple labels for each image for driving scene
understanding with the proposed I2MAP method. The
effectiveness of calibration and confidence score filtering are
examined with the label accuracy correlation of human-expert
labelling; 3) We experiment on sample ADAS functions
including traffic flow prediction, and multi-labelling
affordance state observer is trained in a single multi-task
learning network with our dataset and cross-validated on
KITTI [6], BDD100K [17] benchmarks.
The rest of this paper is organized as follows: Section II
discusses related work in driving benchmarks, driving-related
affordance understanding and multi-task learning. We outline
our I2MAP data collection and annotation framework in
Section III, the detailed sensory fusion and proximity-based
mapping algorithms are discussed. In Section IV, we
demonstrate the ODD-A dataset for a typical multi-task
learning setting. We demonstrate the network structure and
experimental results with recent research works. Finally,
conclusion and future work discussion are presented in
Section V.
II.  Related Works
A. Driving Benchmarks
In the pursuit of understanding driving scenes and driving
autonomy, researchers have contributed massive effort to the
Authorized licensed use limited to: University of Exeter. Downloaded on June 20,2020 at 14:15:13 UTC from IEEE Xplore. Restrictions apply.
IEEE/CAA JOURNAL OF AUTOMATICA SINICA, VOL. 7, NO. 2, MARCH 2020
publicly available benchmarks. Geiger et al. [6] present KITTI
benchmark in 2012 containing both camera images and
LIDAR sweeps. They focused on local driving scenes in
Europe. The published benchmark is quite useful for modular
pipeline approach of autonomous driving where typical vision
tasks, including traffic agent detection, lane and road mark
detection, are demonstrated. In 2016, the Cityscape
benchmark [18] collected various urban driving scene across
50 cities for semantic segmentation tasks. Authors open-
sourced their benchmark for pedestrian, lane and road
marking detection under various weather and day time in [19],
[20]. However, accurate annotation is time and labour-
consuming. Baidu proposed their annotation pipeline along
with the Apolloscape [21] benchmark in 2018. At the same
time, UC Berkeley released BDDV dataset [17], which
provided a semantic evaluation benchmark containing large-
scale driving datasets distributed collected across four cities.
The driving data quality, labelling accuracy is proportional to
the human labelling power and sensor equipment costs in
most cases.
B. Driving Affordance Understanding
In the autonomous driving setting, the authors in [8] first
argue that traditional modular pipeline approach uses
redundant information and making the driving task even
harder. Typically, human drivers can estimate the affordances
of the environment, including the road attributes and relative
distances associated with related traffic participants instead of
identifying the bounding boxes or segmentation for all
instances, appeared in the image. Sauer et al. [22] further
examined the idea of direct perception by extending the
driving scenario in urban driving using more photo-realistic
simulation platform CARLA [11]. The images with the
affordance attribute attached in both works are collected easily
through the provided simulation API. However, affordance
annotation is a challenging task in real driving environments
since it requires a certain level of understanding of the current
diving environment. To annotate the driving affordances such
as heading angle, speed limit, road type as well as the distance
to the intersection, the human labeller may have to label along
the ride instead of following the annotation scheme mentioned
above image by image. In 2016, Seff et al. [15] present the
affordance learning methods by combining the GSV
panoramas and OSM road attributes. They use cropped GSV
panoramas to train a CNN model for a list of selected static
road attributes. This method was further examined in [23] for
various geographical locations. Ideally, we would able to use
a compact set of affordances to represent the background and
event of the driving scene with enough samples under the
chosen driving scenario.
C. Multi-Task Learning
Multi-task learning (MTL) is a sub-branch of supervised
learning, which can simultaneously solve multiple learning
tasks and make use of the commonness and differences
between tasks. The affordance learning task of understanding
a provided driving scene is similar to visual question
answering (VQA) task [24], where the model needs to extract
SUN et al.: PROXIMITY BASED AUTOMATIC DATA ANNOTATION FOR AUTONOMOUS DRIVING
the common features of the sample and be able to reason the
commonsense of the scene representation. Compared with the
training various NN models alone, the learning efficiency,
model size and prediction accuracy of the specific task model
can be improved by utilizing MTL. In the recent review [25],
the advantage of MTL with the ability to learn internal
representation which can improve the generalization ability of
trained NN has been illustrated. The effect of MTL in the
context of autonomous driving applications are demonstrated
in [14], [22].
III.  I2MAP Framework
In this section, we introduce proposed I2MAP data
collection and affordance annotation framework, as shown in
Fig. 1. We use Honda Civic LX 2017 as our ego-vehicle for
our later driving data collection. In the I2MAP set up, we
include a front camera, IPhone and Panda (Gray version)
OBDII Interface from comma.ai. All the sensory data are
synchronized with coordinated universal time (UTC). Once
the recordings are calibrated, we apply the extended Kalman
filter (EKF) as sensory fusion method to raw GNSS signal, the
phone sensor and CAN bus readings to obtain better
positioning result. The estimated pose and UTC are then used
as a query for online labels for the image tags. To reduce the
auto-labelling error, we propose a confidence score based data
filtering post-processing method, which is discussed in this
section.
A. Sensor Setup
1) Vehicle Proprioceptive Sensors: Vehicles have many
proprioceptive sensors such as steering angle, and throttle
input accessible via vehicle CAN Bus. We can access the
vehicle dynamics using Panda OBDII Interface and further
decoded based on BDC file matching with our vehicle model.
The Panda (Gray version) OBDII Interface also come up with
a Tallysman GPS antenna as GNSS receiver, the raw GNSS
signal can be retrieved in real-time. For our ODD-A data
collection, messages of interest such as longitudinal
acceleration and steering angle currently capture and save to a
separate file at 1 Hz.
2) Phone Data Collector App: We mounted an iPhone on
the dashboard of the ego-vehicle as a complement device for
vehicle dynamics estimation. We built an App capable of
logging the phone position and orientation estimates,
accelerometer and gyroscope sensor readings in real-time.
Notice that the accuracy of the IMU and magnetometer
equipped on the iPhone is relatively low due to the cheap
implementation. To compensate for the low accuracy
problem, we implemented on-board automatic accelerometer
and gyro calibration, where the iPhone is calibrated in an
upward portrait plane position demonstrated in Fig. 2. In this
regard, we only need to focus on bias estimation, and the
accelerometer sensor scaling can be ignored. As a
complement source of measurement of CAN bus, we can
estimate the vehicle heading by logging the iPhone heading,
which is determined using embedded magnetometer sensors.
We need to calculate the bias for the calibration process;
hence we drove slowly on flat ground for few minutes until
Authorized licensed use limited to: University of Exeter. Downloaded on June 20,2020 at 14:15:13 UTC from IEEE Xplore. Restrictions apply.
397
the Phone’s x-axis pointing to the true north matching with the
device attitude orientation. The biases are subtracted from
corresponding measurements in real-time as the driver collects
data.
3) Garmin Dash Camera: The Phone Data Collector App
can capture driving videos similarly with [14] which using
phone camera for recording. However, one drawback is that
the horizontal view angle for the captured iPhone videos was
only about 60° field of view (FOV). We use Garmin Dash
Cam 45 with a 122° FOV to overcome the raw data quality
issue in the distributed driving sampling applications in [14],
[17]. This camera records videos at 30 fps with a frame
resolution of 1920 × 1080. The camera gives 3 channels
(RGB) and has a night colour mode setting which helps
capture relatively good images at night. Hence, the raw image
quality is much better than the GSV samples used in [15] as
well as the capability to capture more abundant driving
scenes. Each frame of the dashcam recording is tagged with
UTC, GPS position and movement speed.
B. Proximity Mapping
The main idea of proximity mapping is to overlay road
feature attributes to recorded driving scenes. The OSM to
image matching algorithm was first proposed in [15] and
further examined in [23]. The contributed data in OSM is tied
together using location information in a world geodetic
coordinate system (WGS84). The tags for road attributes and
visibility can then be queried from OSM and online weather
API correspondingly. However, the major challenge of this
type of proximity mapping-based data auto annotation is how
to ensure label accuracy. The accuracy of feature association
is directly affected by the location accuracy in both GSV and
OSM and weather information in both sources was updated in
the same time frame. As highlighted later, we found some
mislabeled affordances due to unresolved location differences
especially at bridges, intersections and close road networks
where a small location deviation would associate features of
one road to an image showing a different road.
1) Extended Kalman Filter: Extended Kalman filter (EKF)
is an effective recursive filter to estimate the state of a
nonlinear dynamic system, which has been applied to vehicle
positioning with GPS and vehicle sensors in [26]–[28]. We
use the kinematic bicycle model [29] given by the following:
ẋ = v cos(ψ + β) (1)
ẏ = v sin(ψ + β) (2)
v
ψ̇ = sin(β) (3)
Lr
v̇ = a (4)
( ( ))
where β = tan−1 Lr tan δ f / Lr + L f with the steer input δ f
reading from CAN bus and a as acceleration. Notice that the
element in global pose x, y and velocity v are estimated from
the vehicle proprioceptive sensors; the inertial heading ψ is
estimated by the gyroscope sensor in the smart phone. Our
state vector is then X = [x, y, ψ, v] with input vector
u = [δ f , a]. By concluding the white noise uncertainties in the
398 IEEE/CAA JOURNAL OF AUTOMATICA SINICA, VOL. 7, NO. 2, MARCH 2020

Data from all devices synchronized by UTC time

Phone sensors: Proprioceptive Dash cam:

-GPS sensors: -40k key frames
-Speed & acceleration -Vehicle dynamics -30 fps
-Vehicle drife & heading -Driver control inputs -1920×1080 resolution

Calibrate Calibrate
Query

EFK-based
Online data-base: pre-rocessing
-Road static attributes sensor fusion
-Weather
-Road type, bus stop,
intersection, stop signs Annotated data sets

Confidence score based data filtering

Fig. 1. A framework demonstration of proposed I2MAP driving data collection and automatic annotation pipeline.
 
+Y  0 0 −v sin(ψ + β) cos(ψ + β) 
 0 v cos(ψ + β) sin(ψ + β) 
Roll  0 
JF =  sin β 
 (7)
 0 0 0

Pitch
 Lr
+X 0 0 0 0
Yaw
+Z JH = I4×4 . (8)
The process noise covariance P and measurement noise
(a) (b) covariance R are chosen based on experimental data
   
Fig. 2. The phone is mounted with its z-axis parallel to the vehicle’s x-axis,  1 0 0 0 
  4 0 0 0 
 0 1 0 0  
 0 4 0 0 
ego vehicle forwarding direction is the same as the –z direction in iPhone co-
P =   , R =  .
 0 0 2 0 
(9)
ordinate system. (a) IPhone coordinate reference system; (b) The phone and  0 0 0.5 0 
dash camera set up in the ego vehicle. 0 0 0 0.3 0 0 0 4
2) GNSS Preprocessing: With a minimum of four satellites,
dynamic model w , we have the nonlinear dynamic system for a good position for the GPS receiver is achieved by solving a
model forecast 4-dimensional trilateration problem [30]. The primary source
the GPS error is coming from user equivalent range errors
Ẋ = F(X) + w (5) (UERE) relate with the timing and path readings and dilution
where F(X) wrap up the functions in (1)–(4). The measure- of precision (DoP) caused by the arrangement of satellites in
ment model can be defined as follows, based on the need to the sky. The open-sourced Laika package in [31] provides
useful sources to get the information about satellites and
estimate the position (x, y) , velocity v , and the heading angle ψ
signal delays from the raw GNSS data. We can obtain the
   pseudorange delay from the summation of tropospheric delay,
 1 0 0 0  
 
x 

   ionospheric delay, clock error and DCB corrections in the
 0 1 0 0   y 
 recording area [32]. The positioning accuracy is then
zk =   

 + v = H(X) + v.
 (6)
 0 0 1 0   ψ  improved by query the position of the nearby continuously
   
 operating reference station (CORS) through Laika [31] and
0 0 0 1 v make compensation by differential correction scheme with the
∂F ∂H known pseudorange delay.
The Jacobian matrices JF = |bx and JH = | are
∂X ∂X bx Other than the UERE, we can also evaluate the DoP from
derived as the raw GNSS data to estimate the error propagation on

Authorized licensed use limited to: University of Exeter. Downloaded on June 20,2020 at 14:15:13 UTC from IEEE Xplore. Restrictions apply.
SUN et al.: PROXIMITY BASED AUTOMATIC DATA ANNOTATION FOR AUTONOMOUS DRIVING
positional measurement precision. If the elevations and
azimuths of the satellites in view are simial, the DOP is high.
Simultaneously, a high DoP means that small errors in UERE
can cause significant errors in the computed position [33]. In
our paper, we use DoP as one feature to specify the
correctness of the processed GNSS position result.
C. Confidence Score Filtering
One major issue in autonomous labelling is the label
accuracy and data quality. The annotated label needs to be as
accurate as possible to avoid introducing bias to the
subsequent usage on supervised learning. We have improved
the GNSS positioning through proposed raw GNSS
compensation and EKF filtering. Instead of the positioning
accuracy, the final label accuracy is strongly correlated with
the label confidence of the road segment structure complexity.
For example, we demonstrate an branching intersection in
Fig. 3(a), where at time t, vehicle is positioned with state Xt ;
after some time period δt , the true state of the vehicle is
portrait as Xt+δt and the estimated state is given as X̂t+δt . Based
on proximity mapping, we find the nearest intersection
distance from the corresponding ego-vehicle heading for the
ground truth label mapping. Consider the case at time t + δt ,
the image sampled at the real vehicle state at Xt+δt is then
tagged with the static road attributes annotated at nearby OSM
position (OSM-P2).
Labels like number of lanes, road type, vehicle to road
heading are likely to raise error in this case. The problem of
intersection label accuracy is inherited from the lack of formal
semantic definition of labels at the scenario transitions, which
was elaborated in [34]. One popular strategy is to have
multiple human labeller to evaluate the image labels and find
the final consensus as to the image tag. However, using
consensus label with even more human efforts is against the
intention of reducing human labelling workload. Moreover,
even by reaching a consensus tag, it remains vague and lack of
formal definitions. We introduce the confidence score to
measure the data label accuracy in the following three
categories
S c po (∆p) = e−α1 ∆p
2
S cDoP (∆d) = e−α2 ∆d
2
S cil (∆t) = σ(|∆t|)
where the S c po measures the confidence score for the pitch
offset as in our assumption, the ground is mostly flat for the
sake of sensor bias reduction. Hence a large pitch offset ∆p is
not expected. Similarly, the calculated DoP ∆d is a positive
real value state how errors in the measurement affect the final
state estimation. As mentioned in [33], a considerable ∆d > 10
is often representing a low confidence level, and the position-
al measurements should be discarded. The intersection label
confidence is measured by S cil (∆t) where ∆t is the frame off-
set from the given data sample frame to the sample frame cor-
responding to the intersection. Since the structure transition
mostly causes this label confidence issue, we use the sigmoid
function σ to map the frame difference ∆t to [0.5, 1]. Readers
Authorized licensed use limited to: University of Exeter. Downloaded on June 20,2020 at 14:15:13 UTC from IEEE Xplore. Restrictions apply.
399
can find a demonstrated plot for the confidence score in
Fig. 3(b) corresponding to (10)–(12).
D. Performance Evaluation
1) Position Accuracy: To prove the effectiveness of our
algorithm, we compare our result with the raw signal
positioning from Phone, DashCam, and Raw GNSS data.
Since the ground truth position of the receiver is never known,
it is hard to judge the filtering algorithms. However, with the
same assumption in [31], we can estimate the altitude
accuracy of a position by checking the variation of the
estimated road height over a small batch. It is reasonable to
assume that the vertical and horizontal accuracy is equally
correlated for the computed position. Fig. 4 shows the altitude
error distribution for position computed with Phone GPS,
DashCam, Raw GNSS, EKF-fusion algorithm and the EKF
with confidence score filtering algorithm. Overall the
positioning error was reduced by 50%–60% using EKF with
the following confidence score specification
Φ = (S c po > 0.8) ∧ (S cDoP > 0.6) ∧ (S cil > 0.6). (13)
Notice that the position error can be further reduced by
providing more stringent criteria Φ ; the downside is that we
have to filter out more data.
2) Label Accuracy: To compare the label accuracy, we
generated a single set of human labels as ground truth by
combining human annotators consensus following [35]. Each
human volunteer was first shown fifty example data samples
to help them understand the corresponding affordance
definition. We choose four classification tasks: number of
lanes, heading angle, road type and day or night. Notice that
estimating the real heading angle as a regression task is tough
for human annotators. We set a pseudo rule that we classify
the calculated heading angle corresponding to three types: left
(> 15◦), right (< −15◦ ), center (the rest of them). As evident in
Fig. 5, the proposed I2MAP label accuracy boosted 8% on
average by EKF fusion algorithm; the performance is further
improved by confidence filtering with the specification (13)
from 5% to 14%. We find that for the affordances have a
strong correlation with pose estimation, such as number of
(10) lanes, heading angle and road type; the corresponding label
accuracy can be greatly improved through our proposed
(11) algorithm. On the other hand, tags such as weather, visibility
are not very sensitive to the road structure change, and
(12)
localization error, hence the labelling quality of these
affordances are naturally good enough.
IV.  Benchmark Analysis and Experiments
A. Data Statistics
Our current dataset consists of about 40 000 images taken
during three weeks of driving in Waterloo. In total more than
40 label classes are automatically annotated. The available
tags and annotated attributes are shown in Table. I. We
recorded the data in various driving scenarios across various
road types, day time and night time.
The data balance of selected subset attributes is
demonstrated in Fig. 6. Our data set contains 53% secondary
400 IEEE/CAA JOURNAL OF AUTOMATICA SINICA, VOL. 7, NO. 2, MARCH 2020

Confidence score evaluation Sc (Δx) for

pitch offset, DoP and intersection label transition
1.0

The confidence score: Sc (Δx)

OSM-P1 0.8 Pitch off. score
DoP score
^
Xt+δt Intersection
d2 0.6 label scote
OSM-P2
d1 Xt+δt 0.4
Xt
0.2

0
−20 −15 −10 −5 0 5 10 15 20
The parameter for confidence scor evaluation: Δx
(a) (b)

Fig. 3. Label mapping error due to proximity and the proposed confidence score functions to filter the error. (a) An example scenario at branching intersec-
tion which induces the proximity-based label mapping error. The black dots are corresponding to the OSM reference location with road attributes annotation in
the database; the green dots represent the ground truth state of the ego-vehicle; the red dot is the estimated state at time t + δt ; (b) A set of confidence score com-
putation. Although they share the same x-axis, the meaning of the corresponding ∆x depends on the corresponding confidence score function.
(α1 = 0.1, α2 = 0.01).

Altitude error distributions Altitude error distributions

0.18
Phone Raw GNSS
0.16
DashCam EKF-fusion
0.08 0.14
Raw GNSS Cinfidence
0.12 score filtered
0.06
Density

Density

0.10
0.08
0.04
0.06
0.02 0.04
0.02
0 0
−20 −15 −10 −5 0 5 10 15 20 −20 −15 −10 −5 0 5 10 15 20
Altitude error (m) Altitude error (m)
(a) (b)

Fig. 4. Estimated altitude error distributions comparison for the signal from five schemes. (a) Error distribution comparison between Phone (purple), Dash-
Cam (green), and Raw GNSS (red); (b) Error distribution comparison between Raw GNSS (red), EKF-fusion (pink) and EKF with confidence score filtering
with our specification Φ (light-green).

roads which are the highest of all road types. The secondary the environmental labels, are listed in Table. I.
roads mostly indicate a route with two lanes and traffic
B. Affordance Observer Network
1.2 Raw GNSS One application of our proposed dataset ODD-A is to solve
EKF fusion the VQA problem as a multi-labelling problem. We use a
1.0 EKF fusion with cinfidence filtered
0.95 0.92 0.95 0.96 single neural network demonstrated in Fig. 7 as a affordance
Label accuracy

0.93 0.89 0.90

0.8 0.80
0.84 0.81 0.85 observer. The task blocks corresponding to different sub-tasks
0.68
0.6 on learning the affordances or even control outputs as an
0.4 imitation learning structure. The network we use here in Fig. 7
0.2
can be extended with controller modules as [11], [22], [36].
Here, we choose one example implementation on predicting
0
Num. of lane Heading angle Road type Day or night two sets of tasks with the image inputs and the history logs of
Label categories velocity and driver’s inputs. We use the ResNet50 [37] pre-
trained on ImageNet [38] as the encoder to extract features
Fig. 5. Label accuracy compared with human consensus on selected afford- from raw input images. The features are extracted from the
ances. input sequence and stacked as one set of N feature map. The
encoded features from images concatenate with N frame
moving both ways. The recorded high percentage is a true velocity and driver’s steer and throttle inputs forward pass to
reflection of most road networks that we collected data from. task block as an input. The task blocks are task-specific layer
The smallest represented road types with 6% are the service which is grouped as texture oriented task and geometric
roads which are used to provide access to business areas and oriented task. A texture oriented task is aiming to focus more
public gathering places such as business parks and campsites. on the texture instead of geometric relationships in the image.
The detailed image tags, including the vehicle dynamics and For example, the features to identify weather condition or

Authorized licensed use limited to: University of Exeter. Downloaded on June 20,2020 at 14:15:13 UTC from IEEE Xplore. Restrictions apply.
SUN et al.: PROXIMITY BASED AUTOMATIC DATA ANNOTATION FOR AUTONOMOUS DRIVING 401

TABLE I
Current Available Labels From the I2 MAP Data Collection Scheme
Labels from vehicle sensor Labels from Phone and DashCam Labels from OSM and online API
Longitudinal acceleration GPS & images Road type & intersection type
Engine torque & estimate Speed, heading angle & drift Intersection distance
Steer angle & steering wheel angle Estimated attitude: roll, pitch, yaw Bike lane? & one_way?
Engine RPM, odometer & pedal gas Gyro & accelerometer measurements Number of lanes
Throttle & brake Vertical, horizontal & heading accuracy Weather index & road condition
Front left & front right wheel speed Moving traffic? Bus stop? & stop sign?
Rear left & rear right wheel speed Wind speed & visibility

secondary n=2
of size 32. We use the class-weighted categorical cross-
20 000
entropy loss for classification, where the weights are chosen
17 500
Number of frames

sunny based on the data balance in the training set. The mean
15 000
12 500
cloudy average error is chosen for the regression task. The data
10 000 snowy
augmentation set is different between the weather task and the
tertiary n=1
7500 primary n=3 n=4 rest geometric oriented task blocks. Notice that we have an
5000 service rainy experiment with FCN (N = 1) and LSTMs (N = 5 or 10) for
2500 the task-specific layers.
0 We have evaluated the experimental results in Table. II with
Road type Num. of lane Weather
Label categories the cross-validation on KITTI [6] and BDD100K [17]. To
synchronize the labels for comparison, we obtain the image
Fig. 6. The sampled driving data distribution over a set of road type, num- tags for all testing datasets with the proposed I2MAP
ber of lanes and different weathers. algorithm. We found that by employing a temporal task block,
most of the prediction results improve, which demonstrates
Features Task blocks the additional temporal information could help to improve the
1
Texture oriented classification and regression ability. However, the number of
...
2 Affordances
feature maps stack on memory could affect the prediction
Image 3 FCN/LSTM result as well. In our experiment, we sample the data in 1 fps,
Encoder ...
N−1
which could result in the additional temporal trace (N = 10)
N Geometric oriented becomes disturbances in the final network prediction. Almost
...
N frames of velocity and Affordances all the data are collected in clear weather in KITTI [6], hence
driver’s inputs FCN/LSTM
we list NA in Table. II. Notice that the proposed dataset
ODD-A and BDD100K [17] are sampled similarly in terms of
driving location (both in North America) and driving scenario
Fig. 7. Overview of the multi-task learning network. The auto-encoder ex- (various lighting condition, urban). The ODD-A trained
tracts the features from the input image. We store the last N feature maps in network to generalize well when validating on the BDD100K
memory where N is the length of the image sequence for the designed percep- [17] data. Although a more diligent pre-processing needs to be
tion tasks. The N frames velocity and driver’s input are concatenated with the done for improving the validation result on KITTI [6], we can
N feature maps from images branched to each task blocks. The defined af- demonstrate the applicability and effectiveness of the
fordances are predicted from each task blocks. proposed ODD-A benchmark and the multi-task learning
network structure as shown in Fig. 7.
visibility from a front view image are different from those for
learning the cross-track error or lane structure. Different C. Traffic Flow Prediction
weighting combination should be learned corresponding to the We used the trained model to suggest actions of stop or
extracted features from the auto-encoder. Notice that the two drive given the driving scene as a sample ADAS function. By
types of tasks can be treated as augmentation noise to each restricting the moving direction of the Moving Traffic task
other during the training phase to improve the robustness of (must same as the ego-vehicle), as well as combining the
the neural network. historical driver's input and the vehicle speed, we can utilize
We choose the following seven affordances as the learning the result from Moving Traffic as the driving aid function.
tasks, namely heading angle, intersection distance, number of Some of the driving scenes, along with the predicted actions,
lanes, intersection type, weather type, moving traffic and road are presented in Fig. 8. We present scenes of snowy roads in
type. The weather task block is a texture oriented block, both day and night time. The image on top row in Fig. 8,
whereas the rest of the tasks are focus on learning the shows a road a snowy with unclear road geometry. Vehicles
embedded geometric relationships. We use the sixty per cent ahead of the ego vehicle appear far away. The image was
of the proposed ODD-A as the training set and the rest for correctly classified, and the model suggested that it is safe to
validation and testing. The network is trained on mini-batches drive. The top middle image in the same figure indicates

Authorized licensed use limited to: University of Exeter. Downloaded on June 20,2020 at 14:15:13 UTC from IEEE Xplore. Restrictions apply.
402 IEEE/CAA JOURNAL OF AUTOMATICA SINICA, VOL. 7, NO. 2, MARCH 2020

TABLE II
The Performance of the Multi-Task Network. The Higher the Accuracy (ACC) and the Lower the Mean Average Error
(MAE) the Better
ODD-A KITTI [6] BDD100K [17]
Task Metric Non-temporal Temporal Temporal Non-temporal Temporal Temporal Non-temporal Temporal Temporal
(N = 1) (N = 5) (N = 10) (N = 1) (N = 5) (N = 10) (N = 1) (N = 5) (N = 10)
Heading MAE 3.52 1.16 1.98 5.86 2.83 4.12 4.88 2.64 2.13
angle
Intersection MAE 4.77 2.28 3.02 8.12 6.33 6.98 6.73 6.02 6.97
distance
Number of 87.41% 91.27% 84.32% 90.37% 93.44% 90.08% 83.86% 89.37% 88.26%
ACC
lanes
Intersection ACC 93.52% 92.12% 88.48% 78.23% 83.12% 81.94% 80.09% 83.12% 81.88%
Weather ACC 91.74% 92.73% 92.53% NA NA NA 90.87% 91.33% 92.84%
Moving 81.26% 94.29% 88.65% 77.86% 82.61% 83.02% 80.54% 81.88% 81.79%
ACC
traffic
Road type ACC 91.53% 92.41% 91.88% 76.96% 81.49% 81.09% 91.35% 92.16% 91.37%

snowy road with a vehicle on the left lane but very close to
ego vehicle. The model can recognize that such scene-setting
suggest a safe to drive action. The top right image and the
bottom center image in Fig. 8 indicates that the model does
not just associate the green traffic light with clear to move
action but also considers the actions and positions of other
participants. It is also able to read the intention of the ego
vehicle given its orientation on the road. The top right image
clearly shows that the traffic lights are green and even another
vehicle (white) shown in the scene continued to move straight.
However, the model predicted that the ego vehicle intended to
turn right and given that there are pedestrians, the prediction is
a suggestion to stop action given the learned dynamic
affordances. Similarly, the bottom centre image in Fig. 8
shows a scene with clear green lights. However, the model
learned the difference in lighting between a moving and
stopping vehicle (in road setting). Consequently, it was able to
suggest that the correct action at that instance was to stop even
though the lights were green — the bottom right image in Fig. 8
indicates a case where the traffic light is red but are obstructed
by a large cargo vehicle moving in the adjacent traffic. This
traffic lights and of other participating vehicles might be
exaggerated and misleading. However, the model can learn
the most important traffic flow cues given the intent of the ego
vehicle.
V.  Conclusion
In this work, we introduced a scalable driving data
collection and automatic annotation framework driving by
EKF proximity mapping and confidence score filtering. The
collected data from distributed devices are synchronized and
annotated with filtered labels for direct perception scheme.
The proposed benchmark, ODD-A include vehicle dynamics
and road attributes under various scenarios including day time
and night time under various weather conditions (no rain, rain,
snow). We train and evaluate the affordance observer as a
multi-task learning network. One sample ADAS function on
traffic flow prediction for harsh weather are evaluated and
demonstrate its effectiveness with popular benchmarks. We
concluded that the proposed data collection and annotation
framework can be further employed in larger scale in order to
enhance the trained model generalization ability. Exploring
more advanced neural network structures and refining the
static OSM labels with dynamic observations is currently
under investigation.

References
[1] F.-Y. Wang, N.-N. Zheng, D. Cao, C. M. Martinez, L. Li, and T. Liu,
“Parallel driving in cpss: a unified approach for transport automation
Pred = safe to drive Pred = safe to drive Pred = suggest to stop and vehicle intelligence,” IEEE/CAA J. Autom. Sinica, vol. 4, no. 4,
pp. 577–587, 2017.
[2] C. Lv, D. Cao, Y. Zhao, D. J. Auger, M. Sullman, H. Wang, L. M.
Dutka, L. Skrypchuk, and A. Mouzakitis, “Analysis of autopilot
disengagements occurring during autonomous vehicle testing,”
IEEE/CAA J. Autom. Sinica, vol. 5, no. 1, pp. 58–68, 2017.
[3] Y. Xing, C. Lv, L. Chen, H. Wang, H. Wang, D. Cao, E. Velenis, and
F.-Y. Wang, “Advances in vision-based lane detection: algorithms,
Pred = safe to drive Pred = suggest to stop Pred = suggest to stop integration, assessment, and perspectives on acp-based parallel vision,”
IEEE/CAA J. Autom. Sinica, vol. 5, no. 3, pp. 645–661, 2018.
Fig. 8. Our model prediction on traffic flow driving suggestion under visib- [4] B. Jähne and H. Haußecker, Computer Vision and Applications: A
Guide for Students and Practitioners, Elsevier, 2000.
ility conditions.
[5] S. J. Russell and P. Norvig, Artificial Intelligence: A Modern Approach.
Malaysia; Pearson Education Limited, 2016.
scenario represents an obstruction object that the model can [6] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics:
recognize. Finally, the bottom left image in the same figure the kitti dataset,” The Int. J. Robotics Research, vol. 32, no. 11,
points out the difficulty of driving at night while raining. The pp. 1231–1237, 2013.

Authorized licensed use limited to: University of Exeter. Downloaded on June 20,2020 at 14:15:13 UTC from IEEE Xplore. Restrictions apply.
SUN et al.: PROXIMITY BASED AUTOMATIC DATA ANNOTATION FOR AUTONOMOUS DRIVING
[7] Z. Chen, J. Zhang, and D. Tao, “Progressive Lidar adaptation for road
detection,” IEEE/CAA J. Autom. Sinica, vol. 6, no. 3, pp. 693–702, 2019.
[8] C. Chen, A. Seff, A. Kornhauser, and J. Xiao, “Deepdriving: learning
affordance for direct perception in autonomous driving,” in Proc. IEEE
Int. Conf. Computer Vision, 2015, pp. 2722–2730.
[9] H. Guo, D. Cao, H. Chen, C. Lv, H. Wang, and S. Yang, “Vehicle
dynamic state estimation: state of the art schemes and perspectives,”
IEEE/CAA J. Autom. Sinica, vol. 5, no. 2, pp. 418–431, 2018.
[10] V. Rausch, A. Hansen, E. Solowjow, C. Liu, E. Kreuzer, and J. K.
Hedrick, “Learning a deep neural net policy for end-to-end control of
autonomous vehicles,” in Proc. IEEE American Control Conf., 2017,
pp. 4914–4919.
[11] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun,
“CARLA: an open urban driving simulator,” in Proc. 1st Annual Conf.
Robot Learning, 2017, pp. 1–16.
[12] S. Shah, D. Dey, C. Lovett, and A. Kapoor, “Airsim: high-fidelity visual
and physical simulation for autonomous vehicles,” in Field and Service
Robotics. Springer, 2018, pp. 621–635.
[13] S. Kato, S. Tokunaga, Y. Maruyama, S. Maeda, M. Hirabayashi, Y.
Kitsukawa, A. Monrroy, T. Ando, Y. Fujii, and T. Azumi, “Autoware
on board: enabling autonomous vehicles with embedded systems,” in
Proc. ACM/IEEE 9th Int. Conf. Cyber-Physical Systems, 2018, pp.
287–296.
[14] H. Xu, Y. Gao, F. Yu, and T. Darrell, “End-to-end learning of driving
models from large-scale video datasets,” in Proc. IEEE Conf. Computer
Vision and Pattern Recognition, 2017, pp. 3530–3538.
[15] A. Seff and J. Xiao, “Learning from maps: visual common sense for
autonomous driving,” arXiv Preprint arXiv: 1611.08583, 2016.
[16] M. Haklay and P. Weber, “Open street map: user-generated street
maps,” IEEE Pervas Comput., vol. 7, no. 4, pp. 12–18, 2008.
[17] F. Yu, W. Xian, Y. Chen, F. Liu, M. Liao, V. Madhavan, and T. Darrell,
“BDD100K: a diverse driving video database with scalable annotation
tooling,” arXiv Preprint arXiv: 1805.04687, 2018.
[18] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R.
Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset
for semantic urban scene understanding,” in Proc. IEEE Conf.
Computer Vision and Pattern Recognition, 2016, pp. 3213–3223.
[19] S. Lee, J. Kim, J. Shin Yoon, S. Shin, O. Bailo, N. Kim, T.-H. Lee, H.
Seok Hong, S.-H. Han, and I. So Kweon, “Vpgnet: vanishing point
guided network for lane and road marking detection and recognition,” in
Proc. IEEE Int. Conf. Computer Vision, 2017, pp. 1947–1955.
[20] G. Li, Y. Yang, and X. Qu, “Deep learning approaches on pedestrian
detection in hazy weather,” IEEE Trans. Industrial Electronics, 2019.
[Online]. Avaliable: https://ieeexplore.ieee.org/document/8880634/
[21] X. Huang, X. Cheng, Q. Geng, B. Cao, D. Zhou, P. Wang, Y. Lin, and
R. Yang, “The apolloscape dataset for autonomous driving,” in Proc.
IEEE Conf. Computer Vision and Pattern Recognition Workshops,
2018, pp. 954–960.
[22] A. Sauer, N. Savinov, and A. Geiger, “Conditional affordance learning
for driving in urban environments,” arXiv Preprint arXiv: 1806.06498,
2018.
[23] C. Sun, J. M. U. Vianney, and D. Cao, “Affordance learning in direct
perception for autonomous driving,” arXiv Preprint arXiv: 1903.08746,
2019.
[24] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Lawrence
Zitnick, and D. Parikh, “VQA: visual question answering,” in Proc.
IEEE Int. Conf. Computer Vision, 2015, pp. 2425–2433.
[25] S. Ruder, “An overview of multi-task learning in deep neural
networks,” arXiv Preprint arXiv: 1706.05098, 2017.
[26] S. Rezaei and R. Sengupta, “Kalman filter-based integration of dgps and
vehicle sensors for localization,” IEEE Trans. Control Systems
Technology, vol. 15, no. 6, pp. 1080–1088, 2007.
[27] S. E. Li, G. Li, J. Yu, C. Liu, B. Cheng, J. Wang, and K. Li, “Kalman
filter-based tracking of moving objects using linear ultrasonic sensor
array for road vehicles,” Mechanical Systems and Signal Processing,
vol. 98, pp. 173–189, 2018.
[28] S. Gao, Y. Hou, H. Dong, S. Stichel, and B. Ning, “High-speed trains
automatic operation with protection constraints: a resilient nonlinear
gainbased feedback control approach,” IEEE/CAA J. Autom. Sinica,
vol. 6, no. 4, pp. 992–999, 2019.
[29] J. Kong, M. Pfeiffer, G. Schildbach, and F. Borrelli, “Kinematic and
Authorized licensed use limited to: University of Exeter. Downloaded on June 20,2020 at 14:15:13 UTC from IEEE Xplore. Restrictions apply.
403
dynamic vehicle models for autonomous driving control design,” in
Proc. IEEE Intelligent Vehicles Symp. 2015, pp. 1094–1099.
[30] K. T. Leung, J. F. Whidborne, D. Purdy, and P. Barber, “Road vehicle
state estimation using low-cost GPS/INS,” Mechanical Systems and
Signal Processing, vol. 25, no. 6, pp. 1988–2004, 2011.
[31] H. Schafer, E. Santana, A. Haden, and R. Biasini, “A commute in data:
the comma2k19 dataset,” arXiv Preprint arXiv: 1812.057522018,
2018.
[32] S. Miura, L.-T. Hsu, F. Chen, and S. Kamijo, “GPS error correction
with pseudorange evaluation using three-dimensional maps,” IEEE
Trans. Intelligent Transportation Systems, vol. 16, no. 6, pp. 3104–3115,
2015.
[33] H. Sairo, D. Akopian, and J. Takala, “Weighted dilution of precision as
quality measure in satellite positioning,” IEE Proceedings-Radar, Sonar
and Navigation, vol. 150, no. 6, pp. 430–436, 2003.
[34] K. Czarnecki and R. Salay, “Towards a framework to manage
perceptual uncertainty for safe automated driving,” in Proc. Int. Conf.
Computer Safety, Reliability, and Security. Springer, 2018, pp.
439–445.
[35] E. Herrera-Viedma, F. Herrera, and F. Chiclana, “A consensus model
for multiperson decision making with different preference structures,”
IEEE Trans. Systems, Man, and Cybernetics–Part A: Systems and
Humans, vol. 32, no. 3, pp. 394–402, 2002.
[36] P. M. Kebria, A. Khosravi, S. M. Salaken, and S. Nahavandi, “Deep
imitation learning for autonomous vehicles based on convolutional
neural networks,” IEEE/CAA J. Autom. Sinica, vol. 7, no. 1, pp. 82–95,
2019.
[37] Z. Wu, C. Shen, and A. Van Den Hengel, “Wider or deeper: revisiting
the resnet model for visual recognition,” Pattern Recognition, vol. 90,
pp. 119–133, 2019.
[38] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and F. F. Li, “Imagenet: a
large-scale hierarchical image database,” in Proc. IEEE Conf. Computer
Vision and Pattern Recognition, 2009, pp. 248–255.
Chen Sun (SM’17) received the M.A.Sc degree in
electrical & computer engineering from University of
Toronto, Canada in 2017 and B.Eng. degree in auto-
mation from the University of Electronic Science and
Technology of China in 2014. He is currently work-
ing toward the Ph.D. degree in mechanical
& mechatronics engineering at University of Water-
loo. He is a Member of Cognitive Autonomous Driv-
ing Lab (CogDrive) and supervised by Prof. Dongpu
Cao and Prof. Amir Khajepour. His research in-
terests include end-to-end autonomous driving, safety validation for cyber-
physical systems, planning, and control for robots.
Jean M. Uwabeza Vianney received the BSc. de-
gree from University of Calgary, Canada in 2015. He
is currently a second year MASc Student in mechan-
ical & mechatronics engineering at University of
Waterloo. He is a Member of Cognitive Autonom-
ous Driving Lab (CogDrive) and supervised by Prof.
Dongpu Cao. Prior to starting a master’s program,
Jean was a Product Support Analyst at Applanix for
more than 2 years. His research interests include end-
to-end autonomous driving, vision-based systems,
sensor fusion, machine learning, geospatial data engineering, prediction, and
planning for autonomous driving.
Ying Li received the M.Sc. degree in remote sensing
from Wuhan University in 2017. She is currently
working toward the Ph.D. degree with the Mobile
Sensing and Geodata Science Laboratory, Depart-
ment of Geography and Environmental Management,
University of Waterloo, Canada. Her research in-
terests include autonomous driving, mobile laser
scanning, intelligent processing of point clouds, geo-
metric and semantic modeling, and augmented real-
ity.
404
Long Chen (M’12) received the B.Sc. degree in
communication engineering and the Ph.D. degree in
signal and information processing from Wuhan Uni-
versity. He is currently an Associate Professor with
the School of Data and Computer Science, Sun Yat-
sen University. His research interests include
autonomous driving, robotics, and artificial intelli-
gence where he has contributed more than 70 public-
ations. He serves as an Associate Editor for IEEE
Transactions on Intelligent Transportation Systems.
Li Li (S’05–M’06–SM’10–F’17) is currently an As-
sociate Professor with the Department of Automa-
tion, Tsinghua University, where he was involved in
artificial intelligence, intelligent control and sensing,
intelligent transportation systems, and intelligent
vehicles. He has authored over 90 SCI-indexed inter-
national journal papers and over 50 international con-
ference papers. Dr. Li was a Member of the Editorial
Advisory Board for Transportation Research Part C:
Emerging Technologies, a Member of the Editorial
Board of Transport Reviews and ACTA Automatica Sinica. He serves as an
Associate Editor for the IEEE Transactions on Intelligent Transportation Sys-
tems.
Fei-Yue Wang (S’87–M’89–SM’94–F’03) received
the Ph.D. degree in computer and systems engineer-
ing from the Rensselaer Polytechnic Institute, USA,
in 1990. In 1990, he joined the University of Ari-
zona, USA, where he became a Professor and the
Director of the Robotics and Automation Laboratory
and the Program in Advanced Research for Complex
Systems. In 1999, he founded the Intelligent Control
and Systems Engineering Center, Institute of Auto-
mation, Chinese Academy of Sciences (CAS), under
the support of the Outstanding Overseas Chinese Talents Program from the
State Planning Council and the 100 Talent Program from CAS. In 2002, he
joined the Laboratory of Complex Systems and Intelligence Science, CAS, as
the Director, where he was the Vice President for Research, Education, and
Academic Exchanges with the Institute of Automation from 2006 to 2010. In
2011, he was named the State Specially Appointed Expert and Director of the
State Key Laboratory for Management and Control of Complex Systems. His
current research interests include methods and applications for parallel sys-
tems, social computing, parallel intelligence, and knowledge automation. Dr.
Wang was elected as a Fellow of INCOSE, IFAC, ASME, and AAAS. In
2007, he was a Recipient of the National Prize in Natural Sciences of China
and the Outstanding Scientist by ACM for his research contributions in intel-
ligent control and social computing. He was a Recipient of the IEEE Intelli-
gent Transportation Systems (ITS) Outstanding Application and Research
Awards in 2009, 2011, and 2015 and the IEEE SMC Norbert Wiener Award
in 2014. He was the Chair of IFAC TC on Economic and Social Systems from
2008 to 2011. He has been the General or Program Chair of more than 30
IEEE, INFORMS, ACM, and ASME conferences. He was the Founding Edit-
or-in-Chief of the International Journal of Intelligent Control and Systems
Authorized licensed use limited to: University of Exeter. Downloaded on June 20,2020 at 14:15:13 UTC from IEEE Xplore. Restrictions apply.
IEEE/CAA JOURNAL OF AUTOMATICA SINICA, VOL. 7, NO. 2, MARCH 2020
from 1995 to 2000 and the IEEE ITS Magazine from 2006 to 2007. He was
the Editor-in-Chief of the IEEE Intelligent Systems from 2009 to 2012, and
the IEEE Transactions on ITS from 2009 to 2016. He is currently the Editor-
in-Chief of the IEEE Transactions on Computational Social Systems and the
Founding Editor-in-Chief of the IEEE/CAA Journal of Automatica Sinica, and
the Chinese Journal of Command and Control. He was the President of the
IEEE ITS Society from 2005 to 2007, the Chinese Association for Science and
Technology, USA, in 2005, and the American Zhu Kezhen Education Founda-
tion from 2007 to 2008. He was the Vice President of the ACM China Coun-
cil from 2010 to 2011. He is currently the President of the IEEE Council on
Radio Frequency Identification. Since 2008, he has been the Vice President
and the Secretary General of the Chinese Association of Automation.
Amir Khajepour received the Ph.D. degree in
mechanical engineering from the University of Wa-
terloo, Canada, in 1996. He is a Professor of the De-
partment of Mechanical and Mechatronics Engineer-
ing at the University of Waterloo. He holds the
Canada Research Chair in Mechatronic Vehicle Sys-
tems, and NSERC/General Motors Industrial Re-
search program that applies his expertise in several
key multidisciplinary areas including system model-
ing and control of dynamic systems. His research has
resulted in many patents and technology transfers. He is the author of more
than 400 journal and conference publications as well as several books. Dr.
Khajepour is a Fellow of the Engineering Institute of Canada, the American
Society of Mechanical Engineers, and the Canadian Society of Mechanical
Engineering. His research interests include the development of hybrid power-
trains (electric and air hybrid engines), component sizing and power manage-
ment design through concurrent optimization, vehicle modeling through real-
time simulation and hardware-in-the-loop, active and adaptive suspension
systems, vehicle stability, ultra-high-speed robotics, automated laser fabrica-
tion, and micro-electrical-mechanical systems.
Dongpu Cao (M’13) received the Ph.D. degree from
Concordia University, Canada, in 2008. He is a
Canada Research Chair in Driver Cognition and
Automated Driving, and currently an Associate Pro-
fessor and Director of Waterloo Cognitive Autonom-
ous Driving (CogDrive) Lab at University of Water-
loo, Canada. His current research interests include
driver cognition, automated driving, and cognitive
autonomous driving. He has contributed more than
180 publications, 2 books and 1 patent. He received
the SAE Arch T. Colwell Merit Award in 2012, and three Best Paper Awards
from the ASME and IEEE conferences. Dr. Cao serves as an Associate Edit-
or for IEEE Transactions on Vehicular Technology, IEEE Transactions on In-
telligent Transportation Systems, IEEE/ASME Transactions on Mechatronics,
IEEE Transactions on Industrial Electronics and ASME Journal of Dynamic
Systems, Measurement and Control. He was a Guest Editor for Vehicle Sys-
tem Dynamics and IEEE Transactions on SMC: Systems. He serves on the
SAE Vehicle Dynamics Standards Committee and acts as the Co-Chair of
IEEE ITSS Technical Committee on Cooperative Driving.