C OVER STORY   Ac oustic s | N V H

16 www.springerprofessional.com/automotive
AI-based, Automated
Acoustic Diagnostics in Vehicles
Automatically detecting anomalous noises in vehicles and identifying their causes is a challenge, which
has never been resolved in any satisfactory manner until now. Given the complexity of the input variables,
conventional methods of signal processing quickly reach their limits. IAV and TU Braunschweig have
investigated where AI-based methods of detecting and classifying anomalous acoustic events promise
significant advancements.

A UT H O R S

Prof. Dr.-Ing. Tim Fingscheidt

holds the Signal Processing
and Machine Learning Chair at the
Institute of Communications
Technology of Technische Universität
Braunschweig (Germany).

Jan Baumann
is Research Assistant in Signal
Processing and Machine Learning
at the Institute of Communications
Technology of Technische Universität
Braunschweig (Germany).

Michael Papendieck
is Senior Vice President Advanced
Diagnostics and Digital Product Life
Cycle at IAV GmbH in Berlin (Germany).

Dr.-Ing. Alexander Roy

is Senior Technical Consultant
on Advanced Diagnostics at
IAV GmbH in Berlin (Germany).
© IAV

ATZ worldwide 07-08|2021    17
C OVER STORY   Ac oustic s | N V H
FIGURE 1 Exemplary noise categories (© IAV)
g Acoustics is a broad field in the
product life cycle of a vehicle which con-
tinues to be made up of a large number
of mechanical parts. Depending on the
OEM, a distinction is drawn between
over 40 different noise categories (also
called event classes) – initially disre-
garding the source, FIGURE 1.
INFORMATION FROM NOISES
Acoustics in the operating state (Noise,
Vibration, Harshness, NVH) are opti-
mized in different ways from the vehicle
development stage on. Once a vehicle has
been produced and delivered, operating
noise becomes relevant for after sales
­service and quality assurance, as anoma-
lous noise can be an indication of wear
or even a critical problem. To date, the
driver has played the part of a smart sen-
sor. Drivers perceive strange noises as
anomalies compared to the normal oper-
ating noises they are familiar with, proba-
bly animating them in many cases to con-
tact after sales service. Usually, though,
the driver’s mind is occupied by diverse
demands from the actual activity of driv-
ing the vehicle. In addition, the road con-
ditions and individual driving style can
also produce many different noises. As
a result, the driver’s role as a “smart sen-
sor” is not always optimal.
Then, of course, there is the question
of what the situation will look like in
future, when more and more car-sharing
vehicles or autonomously driven shuttles
are used. Who will then know what con-
stitutes normal operating noises? Who
18
sures, noise databases, etc.). To begin
with, the reasons for this are found in
the diversity of normal operating noises,
which include noises produced by the
vehicle itself (engine, gearbox, blower,
etc.) as well as noises generated by the
vehicle’s movement (tires, aerodynam-
ics, etc.). In addition, there are external
noises such as rain, hail, other vehicles
passing by and the like. Below, these
noises are summarized as “background.”
However, the anomalous noises this
­a rticle focuses on, referred to below as
“events,” are also extremely complex
and vary in relation to the vehicle’s oper-
ating state. They also vary with vehicle
type, age, and mileage.
Some OEMs have databases of anoma-
lous noises. These provide a reference for
clearly identifying a noise that frequently
occurs in a particular vehicle type as well
will replace the driver’s sensor function? as the remedial measure that is needed
In this case, the only solution can be whenever the service team gets a vehicle
automated acoustic diagnostics. of the same type with related complaints.
Other databases have the goal of classify-
ing at least the noise category or event
HISTORY
class, FIGURE 1, in a distinctive manner.
The problem of automated detection of Classic methods of detection by pat-
anomalous noise in the vehicle is related tern matching are unable to meet the
to acoustic diagnostics and – although large variability of both events and back-
in existence for a rather long time – has ground. With the advent of deep learn-
never been resolved in any satisfactory ing, conventional machine learning
and universal way using conventional approaches [1] have been increasingly
signal-processing methods (frequency replaced by neural networks, which can
analytics, correlations, similarity mea- significantly exceed the detection rates
Vehicle bus
Microphones
Audio signals Operating state
Signal processing and feature extraction
Features
Detection of acoustic events Detection of anomalies
Probability of event occurring Probability of anomaly occurring
Post-processing
AI-based
acoustic diagnostics
Vehicle state
FIGURE 2 Main AI-based acoustic diagnostics (green) including optional ways (blue) (© IAV)
www.springerprofessional.com/automotive
 Visualization tool for acoustic diagnostics
Times (s)
0 20 40 60 80 100 120
-6

-10

-16
Amplitude
-inf
(dB)
-16
-10

Squeak

Rattle

Creak

Buzz

FIGURE 3 GUI for visualizing

event detection (© IAV)

achieved so far, provided sufficiently
large datasets are available.
SIGNAL PRE-PROCESSING
In the first step of AI-based acoustic
diagnostics, signals are pre-processed to
enhance the signal that is picked up, for
example, by a hands-free microphone in
the vehicle. If necessary, it is also possi-
ble at this point to compensate for echo-
ing, to deploy voice activity detection, or
to subtract infotainment signals [2]. The
signal is then fed to a properly config-
ured neural network – the core element
of acoustic diagnostics.
ARTIFICIAL INTELLIGENCE
Within neural networks or, to be more
precise, within their the layers, a dis-
tinction is made between various types:
fully connected layers, convolutional
layers and recurrent layers. By using
the latter two, it is possible to learn
context with a neural network. While
convolutional layers are particularly
good at processing information as a
function of the frequency, recurrent lay-
ers are capable of learning context over
a sequence of time. A neural network
architecture that combines these bene-
fits is referred to as a Convolutional
Recurrent Neural Network (CRNN) [3,
4]. An architecture of this type is opti-
mized and employed for the task under
discussion.
To process the audio signal in the
CRNN, frame-based Mel filterbank fea-
tures are extracted from the audio signal
as input data for the neural network. In
the first convolutional part of the net-
work, data is now processed, reducing
the dimension of the input features in
frequency domain. The resulting fea-
tures are merged and handed over to a
recurrent layer, where sequential infor-
mation is exploited. For the output layer,
a fully connected layer is used with each

Ceramics of the future.

GET YOUR
FREE COPY w w w.interceram -review.info

N O W!
04|2019

Expertise on all aspects of ceramic technology, in all application-relevant areas and presented at a w w w.interceram-review.info

technical-scientific level, together with the latest major trends and developments. Get the latest information
04|2019

five times a year with the exclusive combination of the print edition and interactive e-magazine,
as well as the online archive, a unique knowledge database with pdf download function. Ceramics Explore New Fields
Facing Challenges Ceramics Explore New Fields
Facing Challenges

MAX Phases Ceramic Batteries Additive Manufacturing

www.my-specialized-knowledge.com/interceram
MAX Phases Ceramic Batteries Additive
The Best of Ceramics Manufacturing
Powder Technology for Ceramic Foams with Defined
and Metals Page 10 Electrode Fabrication Page 18 Macrostructure Page 30
The Best of Ceramics Powder Technology for Ceramic Foams with Defined
and Metals Page 10 Electrode Fabrication Page 18 Macrostructure Page 30

ATZ worldwide 07-08|2021    19
 C OVER STORY   Ac oustic s | N V H

recordings from different vehicles in

No events detected
­various driving situations is mixed,
with proprietary anomalous automo-
Operating point tive noises from the arbitrarily chosen
(threshold
False positives False negatives value) classes buzz, creak, squeak, and rattle.
The mixed data contains an average of
six events per minute at various event-
to-background ratios.
Everything detected as event In FIGURE 3 the precision of a diagnos-
tic system based on acoustic event detec-
FIGURE 4 Exemplary selection of an operating point (© IAV) tion is illustrated using a Graphic User
Interface (GUI) with all events displayed
in separated tracks, each along the time-
line. This enables an observer to detect
output node simultaneously providing tion to signal pre-processing, feature events themselves and compare them
probabilities for the presence of multiple extraction and acoustic event detection, with the system’s outputs.
event classes (polyphonic output). One the post-processing estimates the cur- Robustness and reliability play a key
binary event activity track per class is rent vehicle or component status. part in practical use. For this reason,
finally generated by comparing these false positives should be avoided to the
probabilities to a threshold. greatest possible extent. By varying the
MEASUREMENT AND ANALYSIS
FIGURE 2 shows the overall system of detector’s operating point, the true posi-
AI-based acoustic diagnostics, including For the purpose of analysis, a dataset tives and the false positives are always
optional ways of extending it. In addi- from approximately 5 h of background influenced similarly, whereas the false
negatives are influenced in an opposite
manner, FIGURE 4. Assuming a total of
60 events during a 10-min test drive,
FIGURE 5 shows possible compromises
85
between false positives and the ratio
83.7 of true positives to all ground truth
­positives. If only one false positive is
81.5
80 to occur within 10 min, a good 75 %
79.3
(45 out of 60) of the ground truth posi-
True positives [%]

77.6 tives will still be detected. Further tech-

75 nical and scientific details can be found
74.7
in [5, 6].
72.5

70 SYSTEM OPTIMIZATIONS

67.9 In order to increase further the precision

of acoustic diagnostics, the system can
65
0.1 0.5 1 2 3 5 10 be combined with anomaly detection in
Number of false positives in 10 min [-] the next step, as shown in blue color in
FIGURE 5 Trade-off between false and true positives (© IAV) FIGURE 2, which potentially can improve
classification by reducing the amount
of data that needs to be evaluated. Simi-
larly, it is also possible to evaluate real-
AWS audio processing time information on the vehicle’s operat-
Audio signal
ing state (for example data available on
the vehicle bus), making it possible to
User
interaction restrict or exclude specific diagnoses.
Alexa skill Data collection

INTEGRATION INTO THE VEHICLE

Training data
Estimated Often, modern vehicles already come
diagnosis
with a link to cloud-computing services.
Acoustic diagnostics
These can be invoked from the vehicle’s
infotainment system, for example by
Vehicle with Alexa Skill implementation AWS cloud backend
Auto integration
means of a special “skill,” and have
access to various vehicle systems, such
FIGURE 6 AWS cloud-based implementation (concept) (© IAV) as hands-free microphones. AI-based
20 www.springerprofessional.com/automotive
acoustic diagnostics can also be imple-
mented in this way, with most of the
computation performed in the cloud.
This opens up the option of deploying
automatic acoustic diagnostics in a large
number of vehicles without having to
install additional hardware.
This somewhat deeper integration
of acoustic diagnostics into the vehi-
cle – FIGURE 6 shows an implementation
example based on the Amazon Web Ser-
vices (AWS) cloud – offers further bene-
fits: By controlling audio dialog accord-
ingly in the skill used, the aspect of
­private data protection can be covered
by only ever making a recording if the
driver explicitly and directly agrees to
such. At the same time, unwanted audio
sources can be muted during diagnosis.
After a recording, a confirmation can be
obtained from the driver that the anoma-
lous noise was in fact present during the
recording.
This way, a large amount of data can
be collected as the basis for continu-
ously improving the detector used, both
in terms of precision and of adding new
and up-to-now unseen event classes.

The Road in Your Lab

REFERENCES
[1] Transfeld, P.; Receveur, S.; Fingscheidt, T.:
An Acoustic Event Detection Framework and Eva­ Pioneering Test Systems
luation Metric for Surveillance in Cars. In: Proceed-
ings of Interspeech, Dresden, September 2015,
pp. 2927-2931
for Road Driving Simulation
[2] Matheja, T.; Buck, M.; Fingscheidt, T.:
A Dynamic Multi-Channel Speech Enhancement
System for Distributed Microphones in a Car Envi-
ronment. In: EURASIP Journal on Advances in Sig-
nal Processing, 2013, Vol. 1, pp. 1-21
[3] Cakir, E.; Parascandolo, G.; Heittola, T.; Hut-
tunen, H.; Virtanen, T.: Convolutional Recurrent
Neural Networks for Polyphonic Sound Event Detec-
tion. In: IEEE/ACM Transactions on Audio Speech
and Language 25 (2017), Vol. 6, pp. 1291-1303
[4] Strake, M.; Defraene, B.; Fluyt, K.; Tirry, W.;
Fingscheidt, T.: Fully Convolutional Recurrent Net- AIP designs and manufactures individual test
works for Speech Enhancement. In: Proceedings
of International Conference on Acoustics, Speech
systems for development and manufacturing
and Signal Processing, Barcelona, May 2020, centers, most vehicle OEM’s and their tier
pp. 6674-6678
[5] Baumann, J.; Lohrenz, T.; Roy, A.; Fingscheidt, suppliers as well as testing institutions, locally
T.: Beyond the DCASE 2017 Challenge on Rare and globally.
Sound Event Detection: A Proposal for a More Real-
istic Training and Test Framework. In Proceedings
of International Conference on Acoustics, Speech
and Signal Processing, Barcelona, May 2020,
pp. 611-615 AIP GmbH & Co. KG
[6] Baumann, J.; Meyer, P.; Lohrenz, T.; Roy, A.; 87490 Haldenwang / Germany
Papendieck, M., Fingscheidt, T.: A New DCASE Tel. +49 (0)8374-2409-0
2017 Rare Sound Event Detection Benchmark under info@aip-automotive.de Test Systems for a Clean
Equal Training Data: CRNN With Multi-Width Ker-
www.aip-automotive.de and Safe Environment
nels. Accepted for presentation on International
Conference on Acoustics, Speech and Signal
Processing, Toronto, June 2021

ATZ worldwide 07-08|2021    21
Testing Emission Real Drive Chassis Dynos Flat Track Hydraulic Windtunnel Powertrain e-Mobility Autonomous Test Bench Driving Vehicle
Technology Measurement Emission Systems ‘Shaker’ balance Test Benches Driving Automation Robot Cooling Fan
for R & D Systems Test Stands