You are on page 1of 15

A Hybrid Approach for ADAS Data Fusion Algorithm

Development - From High-Level Prototypes to ECUs


Abstract
Current development of Advanced Driver Assistance Systems (ADASs) and automated
vehicles aims to address more and more complex scenarios. Not only the variety of
situations, also the area covered by sensors increases. This in turn is dealt with by
more and more complex sensors. In addition to the increased number of sensors even
the algorithms used for the signal processing are getting more complex to deal with
more challenging tasks given by innovative ADAS functions. This puts special focus
on the tool chain for ADAS development, as the overall time budget from an
algorithmic prototype to the series electronic control unit (ECU) remains the same, if
not even decreases. The paper at hand outlines design considerations for a data
fusion system of an ADAS and describes a novel methodology to support the
development process until series deployment. Furthermore, the novel hybrid
development approach is classified with respect to state-of-the-art methodologies.
The table below summarizes the conclusions.
Development

Transition to

Development

approach

series system

time for data

Flexibility

Algorithm

Sensor

support

interface

fusion
Direct ECU
implementation
Coding-based
Model-based
Hybrid

Not necessary
Manual reimplementation
Template-based
code generation
Novel code
generation

via

long

very low

none

ECU; direct

long

low

none

PC; direct

medium

low

limited

short

high

full

additional
hardware
PC; direct

Keywords: Data Fusion, Data Logging, ADAS, Perception Model, Bayesian Filtering,
Code Generation, Rapid Prototyping
Author: Norman Mattern, Head of Products & Services, BASELABS GmbH

A Hybrid Approach for ADAS Data Fusion Algorithm


Development - From High-Level Prototypes to ECUs

Introduction
Current developments in the domain of Advanced Driver Assistance Systems (ADASs)
handle more and more complex functions. This increased complexity leads to more
sensors being mounted on vehicles and puts a focus on the fusion of data of these
numerous sensors. This data fusion offers different degrees of freedom but also
algorithmic and implementation challenges. For that reason, the next section lists
typical approaches how the algorithmic processing of a system using multiple sensors
can be partitioned. Independent from the actual partitioning, the algorithms need to
be designed as well. They finally need to be available for an automotive electronic
control unit (ECU) for the later use in series production. Therefore, the paper at hand
additionally describes different scenarios including a novel approach how the
transition from a prototype to an ECU can be supported by tools in the context of
signal processing for ADAS sensors like radars, laser, and cameras. In the last section,
tools for the design of data fusion algorithms are compared.

Partitioning Approaches for Sensor Data Fusion System


The presence of various sensors, the capability to perform processing of data either in
the sensors or a separate ECU, and the interaction of OEMs and suppliers create some
degrees of freedom for the design of a data fusion system for ADAS. At the same time,
the algorithms to handle the increased amount of sensor data become more complex.
Additionally to that increasing complexity, different aspects for a data fusion system
need to be considered as well:
1. Fusion of more than one sensor, e.g. several radar and camera sensors
2. Ability to exchange sensor models and sensors suppliers
3. Ability to configure even the number and positions of sensors
4. Different applications and thus different models shall be implemented
The following sections describe typical partitioning approaches for such sensor data
fusion systems for ADAS.

www.baselabs.de | info@baselabs.de | +49 371 3371 51 10

A Hybrid Approach for ADAS Data Fusion Algorithm


Development - From High-Level Prototypes to ECUs

State of the Art Monolithic System


In the past, two tendencies could be observed:
1. Implementation of tracking and fusion functionality shifted to tier 1 supplier
2. As a result, tracking was also handled by those systems
However, that contradicts the aim to be flexible in choosing the supplier. While a
single sensor system such as a radar-only ACC can be treated as a monolithic
component as shown in Figure 1, a system using data of a multitude of sensors
cannot. For such systems, a more modular solution is desirable.

Radar

sensor data

Algorithm
(Tracking)

tracked objects

Algorithm
(Function)

Monolithic system of function supplier

Figure 1: Monolithic system architecture

Combining Track Lists - Track-to-Track Fusion


Depending on the requirements of an OEM, different architectures are possible. A
typical example is that different sensors are delivered by different suppliers. The
sensors may be stand-alone such as cameras or multi-sensor systems, for example
radars observing the vehicles surrounding. For the combination of the information of
the different sensors and sensor systems, their raw data (that has not been subject to
pre-processing in the sensor) is not used. Instead, the data that is provided by the
sensors is usually already pre-processed and tracking algorithms are applied already
in the sensor, resulting in object lists. As a result, putting together the information of
the different sensors means actually combining lists of tracked objects. This is
commonly referred to as track-to-track fusion. Figure 2 depicts an example of such
an architecture.
An advantage of this architecture is that the amount of data transferred between the
stages is limited. Furthermore, the combination of separate sensors requires less
sensor domain specific knowledge, as data is provided ready-to-use. At the same
time this pre-processing in terms of tracking always means reduction of information,
as only tracks considered as valid are provided to the track-to-track fusion. However,
in todays more and more complex scenarios every information is valuable and

www.baselabs.de | info@baselabs.de | +49 371 3371 51 10

A Hybrid Approach for ADAS Data Fusion Algorithm


Development - From High-Level Prototypes to ECUs

therefore centralized processing of all sensor information incl. raw data can be an
advantage. The central data fusion architecture offers this benefit and is addressed in
the next paragraph.

Stereo Cam

sensor data

Algorithm
(Tracking)

tracked
objects

System of camera supplier 1

Mono Cam

sensor data

Algorithm
(Tracking)

tracked
objects
Algorithm
(Track-to-track
Fusion)

System of camera supplier 2

Short Range
Radar 1

sensor data

Short Range
Radar 2

sensor data

Long Range
Radar

sensor data

Algorithm
(Data Fusion /
Tracking)

Algorithm
(Function)
System of OEM
or supplier

tracked
objects

Multi sensor system of radar supplier

tracked
objects

System of OEM
or supplier

Figure 2: Track-to-track fusion architecture

Combining Sensor Raw Data Central Data Fusion


Another approach of partitioning is to conduct the fusion of the sensor raw data in
one central data fusion instance. Clearly, that instance directly takes the
measurements from the sensors (such as vehicle positions in the image of a camera or
azimuth, range and range rate from a radar) instead of pre-processed data. Figure 3
illustrates the partitioning in that architecture.
An advantage of this configuration is that all information delivered by the sensors is
also available to the data fusion, which is the optimal basis for such a component.
However, the development and processing effort is higher.

www.baselabs.de | info@baselabs.de | +49 371 3371 51 10

A Hybrid Approach for ADAS Data Fusion Algorithm


Development - From High-Level Prototypes to ECUs

Stereo Cam

sensor data

Supplier 1

Mono Cam

sensor data
Algorithm
(Data Fusion /
Tracking)

Supplier 2

Radar 1

tracked
objects

Algorithm
(Function)
System of OEM
or supplier 6

sensor data

Supplier 3

Radar N

sensor data

Supplier 4

System of OEM
or supplier 5

Figure 3: Central data fusion architecture

Development and implementation of tracking and sensor data fusion


algorithms for ADAS
While the last chapter outlined different approaches to partition the processing of
data, i.e. one or multiple algorithms, the next chapter deals with the algorithms
implementation. That includes the design of the algorithm itself as well as the
transition to an automotive ECU as used in series systems.
Commonly, the development of systems for tracking and sensor data fusion for ADAS
starts with the implementation of an algorithmic prototype. After the functionality is
proven, that system is transformed to a representation which can be run on an ECU.
This representation finally needs to meet requirements specific to the automotive
domain, e.g. MISRA conformity.

General prototype development approaches


High level prototypes are used to get fast first results of the behaviour of the data
fusion system which shall be developed. Typical approaches are:
1. Model-based development
2. Coding-based development

www.baselabs.de | info@baselabs.de | +49 371 3371 51 10

A Hybrid Approach for ADAS Data Fusion Algorithm


Development - From High-Level Prototypes to ECUs

3. Hybrid development
The following sections will describe the different approaches in more detail.
Furthermore, the involved system development set-ups will be presented on a
simplified example. This example assumes that a certain algorithm, be it tracking or
other data fusion applications, may be designed and finally brought to an automotive
ECU. The hypothetical system uses a sensor which is connected via CAN bus.

Model-based development
As the name indicates, model-based development uses models, i.e. abstract
representation of knowledge. Simple examples for such models are PID controller or
state machines, which are configured to solve a certain problem. Pure model based
development utilizes tools like Simulink [1]. However, algorithms for sensor data
fusion for ADASs often include the perception of the environment. Prominent example
are multi object trackers (MOT), usually implemented by Bayes filters. One example
for that would be a bank of Kalman filters with existence estimation and measurement
association. Such systems offer many degrees of freedom in their design, as system
and measurement state and models may be chosen. They are therefore very complex
to handle in a model-based approach.
Figure 4 illustrates typical set-ups involved in the model-based development process.
Usually, logging tools (e.g. Vector CANalyzer [2]) are used to provide sensor data from
a test vehicle (Set-up M1). This data is used for offline playback algorithm design in a
model-based

development

environment

like

Simulink

(Set-up

M2).

If

the

model-based algorithm shall be tested in real-world environments, so-called rapid


control prototyping (RCP) hardware like the dSpace MicroAutoBox [3] offer a way to
connect the algorithms to the sensors, as it provides for example CAN bus access
(Set-up M3). The configured model is then transferred from the model-based
development to the RCP platform where it is executed. Any adoptions to the models
are usually made in the model-based development environment, which generates an
alternating process between the set-ups M2 and M3. Once the algorithm is fixed,
so-called series-coder products can generate code to be run on an automotive ECU.
However, series-coder may support not all models for code-generation, but only a
defined sub-set.

www.baselabs.de | info@baselabs.de | +49 371 3371 51 10

A Hybrid Approach for ADAS Data Fusion Algorithm


Development - From High-Level Prototypes to ECUs

Set-up M1
Recording of sensor data

Data
recorder

CAN reader

CAN data logging tool


CAN bus
Vehicle-under-test

Sensor
data
Offline data
player

Set-up M2
Playback of sensor data
Development of algorithms

Algorithm

Model-based development environment on PC


Code generation for rapid control
prototyping platform
CAN Reader

Algorithm

Feedback from
the real-world
Set-up M3
Test of algorithms during
their development

CAN writer

Rapid control prototyping hardware platform


CAN bus

CAN bus
Vehicle-under-test

Code generation for


automotive ECU platform

CAN Reader

i/f

Algorithm
i/f
(C-code)

Set-up M4
Validation of the system

CAN writer

ECU with automotive operating system

CAN bus

CAN bus
Vehicle-under-test

Figure 4: Model-based algorithm development approach

Coding-based development
The opposite of that is a pure coding-based approach, which is illustrated in Figure 5.
In that case the system is implemented in a high-level framework, usually given by
C++, C#, Java, or MATLAB [4] libraries (Set-up C1). However, even if supporting
software libraries or other code may exist in that case, in general the whole
complexity of the models needs to be handled by the programmer.

www.baselabs.de | info@baselabs.de | +49 371 3371 51 10

A Hybrid Approach for ADAS Data Fusion Algorithm


Development - From High-Level Prototypes to ECUs

CAN reader

Algorithm (highlevel language)

CAN data

CAN data

CAN writer

Data player/
recorder
PC-based high-level software framework
CAN bus

CAN bus
Vehicle-under-test

Manual re-implementation of
algorithm and interface (i/f)

CAN reader

CAN data

i/f

Algorithm
CAN data
i/f
(C-code)

CAN writer

Data player/
recorder
PC-based high-level software framework

CAN bus

Set-up C1
Recording of
sensor data
Playback of
sensor data
Development of
algorithms
Test of
algorithms
during their
development

Set-up C2
Playback of
sensor data
Test of
algorithms
code

CAN bus
Vehicle-under-test

Manual transfer of
algorithms code

CAN Reader

i/f

Algorithm
i/f
(C-code)

Set-up C3
Validation of the
system

CAN writer

ECU with automotive operating system

CAN bus

CAN bus
Vehicle-under-test

Figure 5: Coding-based development approach

An advantage is that C, C++ and C# offer efficient ways to access hardware of sensors.
Examples for such frameworks are ADTF [5], BASELABS Connect [6], or RTMaps [7].
Aim of the algorithm development process is (C-)code running on an automotive ECU.
For that reason, the code from set-up C1, yielding efficiency of development time
using high-level languages, needs to be transformed to that representation, as
features like using runtime-polymorphism of object oriented programming or
dynamic memory allocation are commonly not permitted by the MISRA standard used
for automotive ECU. Additionally to the manual re-implementation of the actual code
of the algorithm, even the environment-specific interfaces need to be implemented.
Those interfaces transform the data from a level of separate scalar signals (e.g.
radar-target 1, azimuth value in rads) on the CAN bus (for example given by a
dbc-file) to a data structure representation which can be used by the algorithm code.
www.baselabs.de | info@baselabs.de | +49 371 3371 51 10

A Hybrid Approach for ADAS Data Fusion Algorithm


Development - From High-Level Prototypes to ECUs

ECU implementation approaches


Systems described by a model-based approach can be transformed to an
ECU-capable representation by so-called Coder products, which generate C-code.
Even though this transformation is convenient and fast, the disadvantages of the
model-based approach remain.
For the coding-based approach, the transition from prototypical code to C-code is
usually performed manually. In case of C++ this usually means complete re-write. In
case of C the code still needs to meet requirements like no dynamic memory
allocation or no function pointers, which finally comes close to a re-write. In any case
this high-level to low-level transformation is very time consuming, which make the
requirements to both the hand-coded high-level prototype and low-level code even
higher.

Hybrid development approach


Hybrid development offers a way to combine the advantages of both model-based
and coding-based development. An SDK like BASELABS Create [6] provides
functionality to configure a data fusion system for the needs of the particular ADAS
algorithm architecture, be it monolithic, track-to-track fusion, or central. By means of
that, only valid filters can be created, still giving developers and researchers the yield
from the modelling capabilities and design efficiency of high-level languages like C#.
Already existing code (e.g. algorithms implemented in C++) can be used as well. At
the same time, the product BASELABS Code [6] offers a way to instantly generate the
necessary C-code for automotive ECUs from the algorithms implemented before.
Figure 6 shows the set-ups involved in the hybrid-development approach. The
Set-ups M1 and C1 (of the coding-based approach) are comparable, as both take care
of the sensor access, data record and replay, and algorithm development. However, in
the hybrid-development approach, the developed algorithm components can be
based on the powerful C# language as well as the BASELABS Create libraries for ADAS
specific tracking algorithms [6]. This enables the developer to directly generate
C-code as well as necessary interfaces. The actual code with attached interfaces is
automatically integrated in the high-level configuration of the software framework
BASELABS Connect. By means of that, the generated C-code can be stimulated by both
recorded and real-world data instantly, without the need to change the system design
or the development environment. Besides the effect of a consolidated tool chain,
www.baselabs.de | info@baselabs.de | +49 371 3371 51 10

A Hybrid Approach for ADAS Data Fusion Algorithm


Development - From High-Level Prototypes to ECUs

development and test time can be reduced significantly. In addition, the interface
generation is not limited to BASELABS Connect. The code and configuration
generation can be adopted not only to other high-level rapid prototyping frameworks,
but also to such frameworks targeting automotive ECUs such as AUTOSAR.

Sensor data

Algorithm (C#-Code)

CAN reader
Data player/
recorder
In-House-Framework

Set-up H1
Recording of
sensor data
Playback of sensor
data
Development of
algorithms
Test of algorithms
during their
development

Result
CAN writer

Automatic generation of
C-code und interfaces

Display

Algorithm (C-Code)

ADTF

PC-based high-level software framework


CAN bus

CAN bus
Vehicle-under-test

Direct re-use of the


algorithm s code

CAN reader

Set-up H2
Validation of the
system

CAN writer

Algorithm (C-Code)

RTOS, AUTOSAR, etc.

ECU
CAN bus

CAN bus
Vehicle-under-test

Figure 6: Novel hybrid development approach

By using the hybrid-development approach not only the given advantages for a
high-level prototype are exploited, the approach also enables:
1. The use of the same code in high-level prototype and ECU system
2. Quick changes of ECU code even in complex ADAS algorithm architectures
3. Reduced system development time

Comparison of the approaches


While

both

model-based

development

and

coding-based

development

are

successfully used in the practical development of automotive software, they both have
specific advantages and disadvantages. In the following, the flexibility in design and

www.baselabs.de | info@baselabs.de | +49 371 3371 51 10

10

A Hybrid Approach for ADAS Data Fusion Algorithm


Development - From High-Level Prototypes to ECUs

the development velocity will be compared. Furthermore, the tool-support of


signal-processing algorithms specific for the ADAS domain will be listed.

Flexibility and Velocity


Table 1 summarizes different aspects of the basic algorithms design and
development process. For example, while template-based code generation of
model-based approaches offer less error prone results in the series systems, the
flexibility in changing even complex models and architectures in data fusion
algorithms for ADAS may be limited. At the same time, real-time access to sensors
may require additional hardware. On the other side, coding-based development of
algorithms offers these possibilities. However, it generally lacks model-checks and
the availability of code generators. The hybrid development approach provided by the
BASELABS tools [6] unifies the advantages of both worlds: high flexibility, direct
sensor interfaces without additional hardware, strong algorithm support and a short
development time. In addition, it reduces the number of required set-ups in the
design and implementation process.

Development

Transition to

Development

approach

series system

time for data

Flexibility

Algorithm

Sensor

support

interface

fusion
Direct ECU
implementation
Coding-based
Model-based
Hybrid

Not necessary
Manual reimplementation
Template-based
code generation
Novel code
generation

via

long

very low

none

ECU; direct

long

low

none

PC; direct

medium

low

limited

short

high

full

additional
hardware
PC; direct

Table 1: Comparison of different approaches.

www.baselabs.de | info@baselabs.de | +49 371 3371 51 10

11

A Hybrid Approach for ADAS Data Fusion Algorithm


Development - From High-Level Prototypes to ECUs

Tool-Support for data fusion algorithm development


In the following, different tools for the development of ADAS algorithms shall be
compared. Additionally to the general user support in signal processing, special focus
is put on typical ADAS-domain specific challenges.
The basis of the algorithm blocks in outlined in Figure 1 - Figure 3 is state estimation,
typically using Bayes filters [8]. Those filters are used to estimate the state of one
single instance of an object. In that case, one filter consists of a system and at least
one measurement model, where the internal state of the filter can be represented by
different means, i.e. Bayes filter implementations. Table 2 summarizes the
tool-support for Bayesian state estimation in terms of ready-to-use algorithms and
models.

System/State

MATLAB R2013a

BASELABS Create

Linear

Linear

transition model
Space
Representation

Non-linear
Vector of floating point
numbers

Dedicated type for spaces


o Dimension names
o Dimension units ([m], [m/s], etc)

Bayes filter

Kalman filter

implementations

Kalman filter
Extended Kalman filter
Unscented Kalman filter
Particle filter

Build-in models

Linear

Linear
Non-linear and automotive specific

Table 2: Available tool-support on Bayesian state estimation

While one single filter is of course suitable to track only one target at the first stage,
additionally functionality is added to the algorithmic system to estimate the states of
multiple objects. That contains basically the association of particular measurements
from the sensors to certain tracks as well as the evaluation of the existence of tracks.
Furthermore, as real-world sensors are not perfect, clutter needs to be modelled as

www.baselabs.de | info@baselabs.de | +49 371 3371 51 10

12

A Hybrid Approach for ADAS Data Fusion Algorithm


Development - From High-Level Prototypes to ECUs

well. Table 3 summarizes how those challenges of multiple-object tracking are


handled by tools providing support on that topic.

Measurement
Association Algorithms

MATLAB R2013a

BASELABS Create

Hungarian

Single local nearest neighbor

assignment
(Complexity O(n))

Existence Estimation/

(Complexity O(n))
Multiple local nearest neighbor (O(n))
Sequential Probability ratio Testing

Clutter Handling

(SPRT)
-

Integrated Probabilistic Data


Association (IPDA)
generalized probabilistic data
association (GPDA)

Supporting
functionality

Gating
-

Detection models
Persistence models

Table 3: Available tool-support on multiple-object tracking

Finally, the application of multiple-object tracking to ADAS brings some typical


additional tasks, as different sensors with potentially overlapping field-of-views may
be used for more-and-more complex systems. Furthermore, different levels of
abstraction might be used for the sensor interfaces. Table 4 summarizes the
tool-support on ADAS specific multiple-object tracking using multiple sensors like
outlined in Figure 2 and Figure 3.

www.baselabs.de | info@baselabs.de | +49 371 3371 51 10

13

A Hybrid Approach for ADAS Data Fusion Algorithm


Development - From High-Level Prototypes to ECUs

MATLAB R2013a
Number of sensors/
measurement models
Track-to-Track fusion

1
-

Sensor field-of-view

BASELABS Create
Arbitrary
Covariance intersection
Different field-of-views of multiple

support

sensors including
-

o Overlapping regions
o Non-overlapping regions
o Region dependent detection and
persistence models

Table 4: Available tool-support on ADAS-specific data fusion systems using multiple sensors

Conclusion
Although the development of ADAS systems is specific in terms of the system
components (e.g. the sensors and ECU used in the system), the signal processing
algorithms for ADAS are system-independent on a mathematical level. To benefit
from this inherent independency, the underlying architecture needs to be sufficiently
generalized. Then, the algorithm development can be supported by the right tools.
The BASELABS tools focus on the automotive sector and provide domain specific
models and functionality. That is, data logging and handling on the one side as well as
comprehensive support by BASELABS Create in the development of the actual
algorithms for the data fusion, as it is based on a generalized architecture.
Each BASELABS tool can be used on its own to support engineers and developers
during particular process stages. The combination of the tools yields a unique
end-to-end chain from high-level algorithm and perception model design down to an
automotive ECU. In the hybrid approach, less development and implementation
set-ups are required compared to the other approaches. This in turn may significantly
decrease development time and error-proneness and at the same time increase the
users flexibility as well as the overall system quality.

www.baselabs.de | info@baselabs.de | +49 371 3371 51 10

14

A Hybrid Approach for ADAS Data Fusion Algorithm


Development - From High-Level Prototypes to ECUs

References
1. The Mathworks (2014), Simulink Simulation and Model-based Design,
http://www.mathworks.com/products/simulink/, last accessed 2014/01/14
2. Vector Informatik GmbH (2014), CANalyzer,
http://vector.com/vi_canalyzer_en.html, accessed 2014/01/14
3. dSpace GmbH (2014), dSpace MicroAutoBox,
http://www.dspace.com/en/inc/home/products/hw/micautob.cfm, accessed
2014/01/14
4. The Mathworks (2014) , MATLAB The Language of Technical Computing,
http://www.mathworks.com/products/matlab/, last accessed 2014/01/14
5. Elektrobit Cooperation (2014), Elektrobit: driver-assistance : EB Assist ADTF,
http://automotive.elektrobit.com/home/driver-assistance-software/eb-assist-a
dtf.html, accessed 2014/01/14
6. BASELABS GmbH (2014), Products BASELABS, http://www.baselabs.de, accessed
2014/01/14
7. Intempora S.A. (2014), RTMaps Overview,
http://www.intempora.com/rtmaps4/rtmaps-software/overview.html, accessed
2014/01/14
8. Sebastian Thrun, Wolfram Burgard, Dieter Fox, Probabilistic Robotics, MIT Press
2005

www.baselabs.de | info@baselabs.de | +49 371 3371 51 10

15