applied
sciences
Article
Combined Use of 3D and HSI for the Classification of Printed
Circuit Board Components
Songuel Polat 1,2 , Alain Tremeau 2, * and Frank Boochs 1






Citation: Polat, S.; Tremeau, A.;
Boochs, F. Combined Use of 3D and
HSI for the Classification of Printed
Circuit Board Components. Appl. Sci.
2021, 11, 8424. https://doi.org/
10.3390/app11188424
Academic Editor: Elza Bontempi
Received: 23 July 2021
Accepted: 7 September 2021
Published: 10 September 2021
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2021 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Appl. Sci. 2021, 11, 8424. https://doi.org/10.3390/app11188424
1 i3mainz, Institute for Spatial Information and Surveying Technology, Mainz University of Applied Sciences,
Lucy-Hillebrand-Str. 2, D-55128 Mainz, Germany; songuel.polat@hs-mainz.de (S.P.);
frank.boochs@hs-mainz.de (F.B.)
2 Hubert Curien Laboratory, University Jean Monnet, 18 Rue Professeur Benoît Lauras,
42100 Saint-Etienne, France
* Correspondence: alain.tremeau@univ-st-etienne.fr
Abstract: Successful recycling of electronic waste requires accurate separation of materials such as
plastics, PCBs and electronic components on PCBs (capacitors, transistors, etc.). This article therefore
proposes a vision approach based on a combination of 3D and HSI data, relying on the mutual
support of the datasets to compensate existing weaknesses when using single 3D- and HSI-Sensors.
The combined dataset serves as a basis for the extraction of geometric and spectral features. The
classification is performed and evaluated based on these extracted features which are exploited
through rules. The efficiency of the proposed approach is demonstrated using real electronic waste
and leads to convincing results with an overall accuracy (OA) of 98.24%. To illustrate that the addition
of 3D data has added value, a comparison is also performed with an SVM classification based only
on hyperspectral data.
Keywords: hyperspectral imaging; 3D data; point cloud; classification; rule-based classification;
waste sorting; WEEE; recycling
1. Introduction
Humanity is producing more and more electronic waste. In 2019 worldwide, 53.6 million tons
of electronic waste were produced. Technological innovations, short production cycles and,
in particular, ever-shorter product life cycles are the reasons for this rapid increase. Only
17.4% of the electronic waste produced worldwide is recycled, while the rest ends up in
waste incineration plants [1]. Due to existing toxic substances, this is not only harmful
to the environment, but could also cause serious risks for the human health. This makes
recycling processes that help to achieve sustainability more important.
Prerequisite for recycling is the sorting of electronic waste based on the type of material.
Whether monitors, calculators, mobile phones or vacuums, these devices are shredded
when they end up in landfills where they remain in the form of cables, plastics, metals and
Printed Circuit Boards (PCBs). PCBs in particular represent a highly valuable resource
consisting of polymers and precious metals [2]. Identifying the individual components
on circuit boards (capacitors, integrated circuits, inductors, etc.) is a particular challenge.
This challenging issue has already been addressed in a number of studies using different
strategies. In [3], an approach was developed for the segmentation of Surface Mounted
Devices (SMDs) on PCBs. The authors use a segmentation based on assembly paint and
color distribution to detect two types of SMDs (small devices like resistors and integrated
circuits). A further article dealing with the segmentation of SMDs can be found in [4] and
they propose an algorithm which exploits shadows cast by SMDs to distinguish them from
invalid counterparts. The segmentation of Through Hole Components (THCs) is part of [5]
and is achieved by using a combination of RGB images and depth frames from a Microsoft
Kinect sensor. A further study based also on depth frames obtained from the Microsoft
https://www.mdpi.com/journal/applsci
Appl. Sci. 2021, 11, 8424 2 of 19

Kinect sensor and pixel classification can be found in [6]. The studies mentioned here
either focus on the segmentation of one type of component (e.g., SMDs or THCs) or are
designed for inspection tasks. The underlying datasets are based on clean and intact circuit
boards. However, in the recycling sector there are no clean and intact circuit boards. Due
to the processing at the landfill site, the objects are in the form of shredded parts and are
correspondingly dirty and demolished. This fact makes the classification of components
more difficult. Furthermore, these approaches do not cover the separation of different
materials. For example, capacitors mounted on circuit boards differ not only in shape but
also in their material composition. In order to be able to separate successfully in such cases,
an approach based purely on depth information or images cannot be really effective. The
use of physical information, on the other hand, offers much greater potential in terms of
material classification. Hyperspectral Imaging (HSI) technologies enable the acquisition of
physical information and can be easily integrated into sorting processes due to technological
advancements in instrumentation engineering in recent years [7]. As examples, [8,9] use
spectral information for the classification of metals on PCBs. The use of HSI in combination
with RGB images can be found in [10] and propose an object detection approach to detect
recyclable objects and elements on PCBs. The authors used spectral properties of the objects
(integrated circuits, gold connectors and electrolytic capacitors) recorded in the HSI to
localize the objects whereas the spatial properties given by the RGB image are used to
classify the localized objects. However, the results showed that not all components on the
PCBs are detected. As a reason, the authors stated that this can be attributed to the very
small training data set available and the parameters chosen for the prediction process. This
explanation is plausible and describes the generally problem of machine learning methods.
A comparison of some machine learning methods is conducted in [11] and shows that there
is no method that consistently provides the best performance and that the quality of the
prediction result mainly depends on factors such as the availability of training samples,
processing requirements, tuning parameters and speed of the algorithm. In particular, the
selection of training samples presents a problem due to the high variations with regard to
PCBs that are demolished and contaminated. Rule-based approaches are an alternative
to machine learning approaches as they need no prior training. They are based on expert
knowledge and a prior analysis of the data. The acquisition of knowledge enables a better
understanding and allows to structure and simplify a problem. Research papers from the
field of remote sensing, that use the advantage of knowledge can be found in [12–17]. A
comparison of deep learning and a knowledge-based method can be found in [18]. It shows
that a rule-based method can even be better than machine learning-based methods.
The classification of PCBs and the components on it is a difficult task, due to the
complexity of PCBs, both in terms of structure and big variety of electronic components
with small size and wide chemical composition [9]. In this article, we propose a general
approach based on a combination of topological, depth and spatial information obtained
from 3D sensors and physical information received from HSI. For the demonstration of
this general approach, the previously described problem from the field of waste sorting
is addressed. The 3D sensors and also HSI sensors have weaknesses and reach their
limits in certain situations. A structured light scanner, for example, faces difficulties in
detecting transparent objects, whereas HSI sensors are not affected by this. In contrast,
HSI sensors have their limits with black pigmented objects. A well-known problem is the
strong absorption of black colorants such as carbon black. Due to the strong absorption
of light from the UV to the SWIR, there is no reflected light that can be detected by the
sensor and thus no spectral information that can be used for classification [19,20]. The
combination is therefore intended to achieve mutual support among the data to improve
the interpretation and processing. A combination of geometry and spectral data brings
advantages that was shown for example in [21]. The authors used geometry in terms of
inclination angles in combination with spectral data for the detection of disease symptoms
on plants. Another example is given in [22] which aimed to use multispectral LiDAR data
for land cover classification of an urban area.
 The general concept of the proposed method consists of merging the datasets
lowed by extracting features in the 3D and hyperspectral domains. Especially the fus
of datasets is a challenging task and poses different requirements (e.g., calibration of l
sensors and methods for registration) depending on the sensor technologies used. Hyp
Appl. Sci. 2021, 11, 8424 3 of 19
spectral sensors with high spectral and spatial resolution are usually line sensors and d
fer in their imaging properties from frame-based sensors. Accordingly, adapted mod
are necessary to perform the calibration, which is a required step for the registration
The general concept of the proposed method consists of merging the datasets fol-
the datasets.
lowed by Methodologies
extracting featuresfor the3D
in the registration
and hyperspectralof linedomains.
scanning HSI sensors
Especially and 3D La
the fusion
Scanners in outdoor
of datasets applications
is a challenging task and can be different
poses found, requirements
for example,(e.g., in calibration
[23–25]. The of strate
line sensors and methods for registration) depending on the sensor
adopted in this article focuses on a laboratory application and follows a simple regis technologies used.
Hyperspectral
tion concept that sensors with high
can similarly bespectral
applied andtospatial resolutionproduction/sorting
an industrial are usually line sensors process.
and differ in their imaging properties from frame-based sensors. Accordingly, adapted
The basic steps proposed regarding the processing of the combined
models are necessary to perform the calibration, which is a required step for the registration
dataset can
easily
of generalizable or extended
the datasets. Methodologies forto theother application
registration fields. All
of line scanning HSI objects
sensors andtypically
3D hav
shape andScanners
Laser a material composition.
in outdoor applications Thiscanassures
be found, a generalization as 3DThe
for example, in [23–25]. andstrategy
HSI are in pr
adopted in this article focuses on a laboratory application and follows
ciple sensitive for features in these domains. The features obtained are next used as inp a simple registration
concept that can similarly be applied to an industrial production/sorting process.
for a rule-based approach. One of the main advantages of rule-based approaches is
The basic steps proposed regarding the processing of the combined dataset can be
simplicity. Once the knowledge
easily generalizable or extendedon which
to other rules arefields.
application basedAll has beentypically
objects workedhave out,aconditi
can be easily set up. Further advantages are the performance,
shape and a material composition. This assures a generalization as 3D and HSI are the ability to handle
in red
dantprinciple sensitive for
and irrelevant features in
attributes and thesethedomains.
flexibleThe features obtained
extensibility are next
of rule setsused
[26].as Acquir
inputs for a rule-based approach. One of the main advantages of rule-based approaches
knowledge can seem effortful, but in view of the resulting advantages, it should be s
is the simplicity. Once the knowledge on which rules are based has been worked out,
as a conditions
clear benefit
can bewhich
easily setallows to structure
up. Further advantages andare simplify problems
the performance, by using
the ability to exp
knowledge. The rules
handle redundant andused in this
irrelevant articleand
attributes arethebased
flexibleonextensibility
knowledge established
of rule sets [26]. for e
tronic waste sorting. A use of the procedure for other applications requires
Acquiring knowledge can seem effortful, but in view of the resulting advantages, it an adaptat
should
be seen as a clear benefit which allows to structure and simplify
of the rules to the respective application based on knowledge from the relevant field.problems by using expert
knowledge. The rules used in this article are based on knowledge established for electronic
waste sorting. A use of the procedure for other applications requires an adaptation of the
2. Materials and
rules to the Methods
respective application based on knowledge from the relevant field.
As shown in Figure 1, the general workflow of this work can be divided into f
2. Materials and Methods
main parts. The data acquisition, the data fusion, the use of 3D data to simplify and str
As shown in Figure 1, the general workflow of this work can be divided into four main
ture parts.
the data
Thebasis and the final
data acquisition, classification
the data based
fusion, the use ondata
of 3D obtained HSIand
to simplify andstructure
3D features. T
individual
the dataparts
basis are
and described in detail in
the final classification the on
based following
obtained subsections.
HSI and 3D features. The
individual parts are described in detail in the following subsections.

Figure 1. Workflow consisting of fusion, simplifying and classification steps.

Figure 1. Workflow consisting of fusion, simplifying and classification steps.

 a linear stage and an illumination unit. This system was chosen for the acquisitio
electronic waste samples consisting of plastics and PCBs from the landfill. Acco
these objects are damaged and dirty. Special attention was paid to the choice of PC
Appl. Sci. 2021, 11, 8424 different mounted electronic components. The required wavelength is 4 oflimited
19 to
range, therefore only the Specim FX17, working in the range from 900 nm to 1700
bands), is used for the measurements in this work. Acquired datasets are radiom
2.1. Data Set Acquisition
normalized by using dark reference image for dark-current (closed shutter) and
The measurement
reference image tosetup shown
reduce theininfluence
Figure 2 consists
of theofintensity
two Specimvariability.
Ltd. (Oulu, Finland)
For the white
hyperspectral pushbroom cameras
tion a 99% reflectance tile was used. (Specim FX10 and Specim FX17), a 3D scanner, a linear
stage and an illumination unit. This system was chosen for the acquisition of real electronic
For the acquisition of the geometry, a structured light scanner from GOM
waste samples consisting of plastics and PCBs from the landfill. Accordingly, these objects
(Braunschweig,
are damaged and dirty.Germany) is used.was
Special attention Thepaid
Atosto Core 500 of
the choice is PCBs
a high-resolution
with different optica
with a maximum resolution of 0.195 mm and a depth accuracy ofNIR
mounted electronic components. The required wavelength is limited to the 0.05 mm. This
range,
therefore only the
ner consists of Specim FX17, working
two stereoscopic in the range
cameras withfrom 900 nm to 1700
5-megapixel nm (224 bands),
resolution and a blue L
is used for the measurements in this work. Acquired datasets are radiometrically normal-
projector which projects structured light onto the object. The projected light, whic
ized by using dark reference image for dark-current (closed shutter) and a white reference
tiallytoencodes
image reduce the the surface
influence of of
the the object,
intensity is captured
variability. For theby thecalibration
white cameras aand 99%triangu
used to obtain
reflectance tile wasthe necessary 3D information.
used.

Figure
Figure 2. Measuring
2. Measuring setupsetup consisting
consisting of hyperspectral
of hyperspectral cameras,
cameras, linear linear stage,
stage, illumination unitillumination
and
3Dstructured
3D structured
lightlight scanner.
scanner.

For the acquisition of the geometry, a structured light scanner from GOM GmbH
2.2. Combining
(Braunschweig, Spectralisand
Germany) Spatial
used. Data
The Atos Sets
Core 500 is a high-resolution optical system
with a The
maximum resolution
first step of theofproposed
0.195 mmframework
and a depth is accuracy of 0.05
the fusion of mm. This 3D (show
the datasets
scanner consists of two stereoscopic cameras with 5-megapixel resolution
ure 3) consisting of HSI and 3D point cloud. This is achieved by using and a blue LEDtie points
light projector which projects structured light onto the object. The projected light, which
the technical setup of the sensors (fixed sensors looking vertically downwards), th
essentially encodes the surface of the object, is captured by the cameras and triangulation
isare
usedsuccessively captured3D
to obtain the necessary during one measuring process in both hyperspectral a
information.
three-dimensional. The transformation parameters are estimated by using tie point
2.2. Combining Spectral and Spatial Data Sets
existing in both datasets and allow the fusion in x-y plane. Due to the different re
The first step of the proposed framework is the fusion of the datasets (shown in
of the datasets, the assignment between 3D point and spectral signature is made
Figure 3) consisting of HSI and 3D point cloud. This is achieved by using tie points. Due to
pixel
the checking
technical setup the neighborhood.
of the The result
sensors (fixed sensors lookingisvertically
a 3D point cloud presented
downwards), the objectsin Figu
a spectral
are signature
successively capturedfor eachone
during individual
measuring3D point.
process Evenhyperspectral
in both though theassensor
well as technol
three-dimensional. The transformation parameters are estimated by using
sented in Figure 2 would not be readily applicable in an industrial context tie points that are due t
existing
sensorintechnology
both datasetsused,
and allow
the the fusion infor
potential x-ythe
plane. Due toconcept
overall the different resolutions
of the measuremen
of the datasets, the assignment between 3D point and spectral signature is made pixel by
and fusion strategy for use in industrial practice is given.
pixel checking the neighborhood. The result is a 3D point cloud presented in Figure 4
with a spectral signature for each individual 3D point. Even though the sensor technology
presented in Figure 2 would not be readily applicable in an industrial context due to the 3D
sensor technology used, the potential for the overall concept of the measurement system
and fusion strategy for use in industrial practice is given.
Appl.
Appl. Sci. 2021, 11,
Sci. 2021, 11, 8424
x FOR PEER REVIEW 55 of
of 19
19

(a)(a) (b) (b) (c) (c)

Figure 3.
3. Acquired
Acquired datasets
datasets of
of electronic
Figure
Figure 3. Acquired datasets of electronic
electronicwaste consisting
waste consisting of
waste of waste
consisting waste plastics
of plastics and PCBs:
and
waste plasticsPCBs: (a) RGB; (b)
(a) RGB;
and PCBs:(b) 3D
3D point
(a) point cloud
RGB; cloud with
with aa
(b) 3D point cloud wit
resolution
resolution of
of 0.5
0.5 mm
mm and
and aa depth
depth accuracy
accuracy of
of 0.05
0.05 mm;
mm; (c)
(c) HSI
HSI with
with aa resolution
resolution of
of 636
636 ×× 1118
1118 pixels
pixels and
and 224
224 bands
bands in
in the
the
resolution of 0.5
wavelength mmfrom
range and900
a depth accuracy
nm to 1700 nm. of 0.05 mm; (c) HSI with a resolution of 636 × 1118 pixels and 224 bands in
wavelength range from 900 nm to 1700 nm.
wavelength range from 900 nm to 1700 nm.

Figure 4. Combined 3D point cloud and HSI. Each 3D point is assigned by a spectrum. In this figure,
the average value over the entire spectrum is shown per pixel.

2.3. Processing of 3D Point Cloud

Figure 4.Combined
In4.these
Figure Combined
merged3D data
point cloudthe
3D point
sets andspatial
cloudHSI.
and Each 3D Each
HSI. point is
component 3Dassigned
point
directly by a spectrum.
is assigned
helps byIn athe
to simplify this figure,
spectrum.
prob- In th
the
the average
lem average value
as we can over
value the
over
reduce entire
thethe
data spectrum
entire is shown
set tospectrum per pixel.
regions ofisinterest,
shown suchper pixel.
as the objects, and remove
useless regions,ofsuch
2.3. Processing as theCloud
3D Point background. The second step of the proposed framework con-
sists in removing
2.3. Processing the
of 3D background.
Point CloudIn our study case, the background can be easily re-
In these merged data sets the spatial component directly helps to simplify the problem
moved
as weIn using
canthesethe depth
reducemerged information
the data data thanks
set to regions to a plane
of spatial
interest, estimation
such method.
as the objects, andThis has several
remove useless
advantages. sets the component directly helps to
onsimplify th
regions, suchFirstly, the amount of
as the background. data
The are reduced,
second which
step of the has aframework
proposed positive effect the
consists in
lem asprocessing
further
removing
we can reduce the dataother
steps. Secondly,
the background. In our study
setcase,
to regions
disturbing of interest,
effects
the background such such
as dirt
can be
as on
(e.g.,
easily
thethe
removed
objects,
con- and
using
useless
veyor regions,
belt) and such
shadow as the
areas background.
are removed. The
Areas ofsecond
shadow step
in of the
particular
the depth information thanks to a plane estimation method. This has several advantages. proposed
can have framewo
a
sists in removing the background. In our study case, the background can be ea
moved using the depth information thanks to a plane estimation method. This has
advantages. Firstly, the amount of data are reduced, which has a positive effec
further processing steps. Secondly, other disturbing effects such as dirt (e.g., on t
 Appl. Sci. 2021, 11, 8424 6 of 19

Appl. Sci. 2021, 11, x FOR PEER REVIEW 6 of 19

Firstly, the amount of data are reduced, which has a positive effect on the further processing
steps. Secondly, other disturbing effects such as dirt (e.g., on the conveyor belt) and shadow
negative impact on the quality of the results. An example for disturbing shadow effects
areas are removed. Areas of shadow in particular can have a negative impact on the
and the influence
quality on the
of the results. classification
An example result isshadow
for disturbing showneffects
in Figure 5. influence
and the Shadow on areas
the occur
mainly on the border
classification result isof 3D objects
shown and5.also
in Figure on the
Shadow border
areas occurof 3D components
mainly on the borderonofthe
3Dtop of
PCBs.
objects and also on the border of 3D components on the top of PCBs. The most disturbing are
The most disturbing shadows are on the border of the objects. Shadow areas
generally
shadowsa are problem
on the when
borderprocessing datasets
of the objects. Shadow as areas
they are
reduce the reliability
generally a problemandwhen success
processing
rate datasets
of algorithms forasobject
they reduce the reliability
recognition, and success
classification andrate of algorithms
other for object The
types of processing.
recognition,
removal classification
of shadow and
areas is other types
therefore of processing.
an essential stepThe removal
towards of shadowimage
improving areas is
quality
therefore an essential step towards improving image quality [27,28] and thus also improve
[27,28] and thus also improve the base for further analysis steps.
the base for further analysis steps.

(a) (b)
Figure 5. Influence
Figure 5. Influenceof shadow
of shadowon the classification
on the process.
classification Here
process. shadow
Here shadowoccurs onlyonly
occurs on the
on neighbor-
the
ingneighboring
area surrounding the object. the
area surrounding (a) Classification of pixels values
object. (a) Classification belonging
of pixels to the class
values belonging PS using
to the class spec-
tralPS
data. (b)spectral
using Classification
data. (b)ofClassification
pixels values of belonging to belonging
pixels values the class PS
to after removal
the class of removal
PS after the background.
of
the background.
Another advantage of adding 3D data is the possibility to partition the dataset by
Anothermethods,
using cluster advantage of adding
which 3D data is the
are addressing thepossibility to partition
shape content the dataset
of objects. by of a
The aim
using cluster methods, which are addressing the shape content of objects. The aim of a
cluster analysis is to find structures in a set of arbitrary data with different properties and
cluster analysis is to find structures in a set of arbitrary data with different properties and
to discover relationships in order to be able to group objects. This involves forming groups
to discover relationships in order to be able to group objects. This involves forming groups
in in
such a way
such a waythat
thatthe
theproperties ofthe
properties of thedata
datawithin
withinthethe group
group or cluster
or cluster are approximately
are approximately
identical. A variety of cluster algorithms exists in the state of the art for this purpose.
identical. A variety of cluster algorithms exists in the state of the art for this purpose. The The
task ofof
task cluster
clusteralgorithms
algorithms is toto form
formclusters
clusterswithin
within datasets.
datasets. Clustering
Clustering methods
methods differdiffer
in in
their
their approachesand
approaches and strategies.
strategies. This
Thisallows
allowsthe separation
the separation of the individual
of the objects
individual in the in the
objects
dataset.
dataset. Duetotothe
Due thehomogeneous
homogeneous data dataavailable
availablein in
ourour
dataset, a cluster
dataset, approach
a cluster basedbased
approach
on Euclidean distance was chosen for the separation of the individual
on Euclidean distance was chosen for the separation of the individual objects. Cluster objects. Cluster
methods can be not only used for the individual object’s separation (such as plastics and
methods can be not only used for the individual object’s separation (such as plastics and
PCBs, as shown in Figure 6b), but also can be used for the separation of components on the
PCBs, as shown in Figure 6b), but also can be used for the separation of components on
boards (as shown in Figure 6c).
the boards (as shown in Figure 6c).
2.4. Rules-Based Classification
The third step of the proposed framework consists in defining rules to process 3D data
and spectral data in order to classify individual 3D point clouds.
Based on the existing dataset, it can be stated that the hyperspectral point cloud
contains valuable geometric information that is implicitly represented by the spatial ar-
rangement of the individual 3D point clouds. In addition to the geometric information, the
dataset also contains valuable spectral information to describe the physical property of
the objects.
3D point clouds are a representation of spatial 3D information, which can be described
by geometric characteristics. The derivation of these geometric characteristics is done by
considering the local neighborhood of each 3D point. Based on this neighborhood, invariant
moments can be calculated for each point, representing the geometric properties of local
3D structures [29]. A representation of some 3D features is shown in Figure 7. The 3D
(a) (b)
 in such a way that the properties of the data within the group or cluster are approximately
identical. A variety of cluster algorithms exists in the state of the art for this purpose. The
task of cluster algorithms is to form clusters within datasets. Clustering methods differ in
their approaches and strategies. This allows the separation of the individual objects in the
Appl. Sci. 2021, 11, 8424 dataset. Due to the homogeneous data available in our dataset, a cluster approach 7 of 19 based
on Euclidean distance was chosen for the separation of the individual objects. Cluste
methods can be not only used for the individual object’s separation (such as plastics and
PCBs, as shown
features in this
tested in Figure 6b),
article arebut also can
nowadays be usedused
commonly forinthe separation
lidar of components
data processing. They on
the are
boards
based(as
onshown
normal in Figure
vectors 6c).
and eigenvalue-based 3D features like linearity, verticality,
planarity, scattering, omnivariance, anisotropy, eigenentropy and change in curvature [30].

(c)
of results of 3D point cloud clustering: (a) Background removed dataset; (b) Clustered objects; (c) PCB
mponents and clustering result. The clustering is completed after removing the board points using a
ethod.

2.4. Rules-Based Classification

pl. Sci. 2021, 11, x FOR PEER REVIEW 7 of 19
The third step of the proposed framework consists in defining rules to process 3D
data and spectral data in order to classify individual 3D point clouds.
Based on the existing dataset, it can be stated that the hyperspectral point cloud con-
(a)
tains valuable geometric information that is implicitly represented(b)
by the spatial arrange-
ment of the individual 3D point clouds. In addition to the geometric information, the dataset
also contains valuable spectral information to describe the physical property of the objects.
3D point clouds are a representation of spatial 3D information, which can be de-
scribed by geometric characteristics. The derivation of these geometric characteristics is
done by considering the local neighborhood of each 3D point. Based on this neighbor-
hood, invariant moments can be calculated for each point, representing the geometric
properties of local 3D structures [29]. A representation of some 3D features is shown in
Figure 7. The 3D features tested in this article are nowadays commonly used in lidar data
processing. They are based on normal vectors and eigenvalue-based 3D features like lin-
earity, verticality, planarity, scattering,(c) omnivariance, anisotropy, eigenentropy and
change in curvature [30].
Figure 6. Examples of resultsofof
Figure 6. Examples 3D point
results of 3D cloud clustering:
point cloud (a) (a)
clustering: Background
Backgroundremoved dataset;(b)
removed dataset; (b) Clustered
Clustered objects;
objects; (c) PCB
(c) PCB
with electronic components
with electronic and clustering
components result.
and clustering The
result. Theclustering
clustering isiscompleted
completedafterafter removing
removing thepoints
the board board points
using using a
a plane
estimation
plane estimation method.
method.

2.4. Rules-Based Classification

The third step of the proposed framework consists in defining rules to process 3D
data and spectral data in order to classify individual 3D point clouds.
Based on the existing dataset, it can be stated that the hyperspectral point cloud con-
tains valuable geometric information that is implicitly represented by the spatial arrange-
ment of the individual 3D point clouds. In addition to the geometric information, the dataset
also contains valuable spectral information to describe the physical property of the objects.
3D point clouds are a representation of spatial 3D information, which can be de-
scribed by geometric characteristics. The derivation of these geometric characteristics is
done by considering the local neighborhood of each 3D point. Based on this neighbor-
hood, invariant moments can be calculated for each point, representing the geometric
Figure 7. Visualization of 3D features. Omnivariance (left), Verticality (middle) and Normal Change Rate (right).
properties of local 3D structures [29]. A representation of some 3D features is shown in
Figure 7. Visualization
Figureof7.3D Thefeatures. Omnivariance
3D features tested in this (left),
articleVerticality (middle)
are nowadays and Normal
commonly used inChange
lidar data
Rate (right). processing. They are based on normal vectors and eigenvalue-based 3D features like lin-
earity, verticality, planarity, scattering, omnivariance, anisotropy, eigenentropy and
For the characterization
change in curvature of the spectral components, shape-describing features intro-
[30].
duced in [31] are used. Materials can be uniquely identified by their spectral signature.
Changes in material compositions lead to changes in the spectral signature and conse-
 Appl. Sci. 2021, 11, 8424 8 of 19

For the characterization of the spectral components, shape-describing features intro-

Appl.
Appl.Sci. 2021,
Sci. 11,11,
2021, x FOR PEER
x FOR REVIEW
PEER REVIEW 8 8ofof1919
duced in [31] are used. Materials can be uniquely identified by their spectral signature.
Changes in material compositions lead to changes in the spectral signature and conse-
quently to local changes in the geometric shape of these spectra. We have shown in [31] that
that curvature𝝒 𝝒isisa asuitable
thatcurvature
curvature suitablegeometric
geometricfeature
geometric featureand
feature andallows
and allowsaaadetailed
allows detaileddescription
detailed descriptionofofspec-
description spectral
spec-
tral signatures.
tralsignatures. Mathematically,
signatures.Mathematically,
Mathematically, the
thethecurvature
curvature
curvature can
cancan be
be be determined
determined
determined by Equation
by Equation (1) using
(1) using
by Equation the 1st
(1) using
Appl.the
Sci. 1st
1stand
and
the2021, 2nd
11,
andx 2nd derivative
derivative
FOR
2ndPEER for for
REVIEW
derivative foreach
each point
each P( x𝑷
point
point )𝑥(𝑡),
(t𝑷(𝑥(𝑡),
, y(t)) 𝑦(𝑡)
where
𝑦(𝑡)) where
where x xcharacterizes
x characterizes the
thewave-
the wavelength
characterizes wave- (in
length
length (in
nm), ynm),
(in the
nm),y ythe
thereflectance
reflectance and
and t the
reflectance and t the
t thesample
sample value,value,
sample value,respectively.
respectively. respectively.
|𝑥𝑦
 . .. . .. 
|𝑥̇𝑦− y𝑦𝑥|
ẍ − 𝑦̇ 𝑥ÿ |x 
−
that curvature𝝒 𝝒𝝒 ===is asuitable3geometric 3
feature and allows a (1) detailed
(1) (1) descrip
(𝑥(𝑥̇ +
2. +
tral signatures. Mathematically, 2𝑦 𝑦̇)2.)22 2 the curvature can be determined by Equat
x +y
the 1st and 2nd derivative for each point 𝑷 𝑥(𝑡), 𝑦(𝑡) where x characteriz
The
Theapproach
approachtotodetermine
determinethe thesignificant
significantparameters
parametersfor forthe
thedescription
descriptionofofthe
thespec-
spec-
The approach in length
to Figure (in nm),
determine the y the reflectance
significant parameters andfor
t thethe sample value,
description ofrespectively.
the spectral
tral
tral shape is presented in Figure 8. It is based on the computation of extreme values ofthe
shape is presented 8. It is based on the computation of extreme values of the
shape is presented in Figure 8. It is based on the computation of extreme values of the
2nd
2ndderivative.
derivative.For Fora abetter
betterquantification
quantificationofofabsorption
absorptionpeaks, peaks,the |𝑥𝑦 − signatures
thespectral
spectral𝑦𝑥|
signaturesare are
2nd derivative. For
normalized
normalizedbybyusing
a better quantification
usingContinuum
ContinuumRemovalRemoval[32,33].
of absorption peaks,
[32,33]. 𝝒 =the spectral signatures are
normalized by using Continuum Removal [32,33]. (𝑥 + 𝑦 )

The approach to determine the significant parameters for the description

tral shape is presented in Figure 8. It is based on the computation of extreme
2nd derivative. For a better quantification of absorption peaks, the spectral si
normalized by using Continuum Removal [32,33].

Figure 8. Use of minimum and maximum positions of 2nd derivative to select significant parameters
Figure
Figure 8.8.
UseUseofof
minimum
minimum and
andmaximum
maximum positions
positionsofof
2nd
2ndderivative
derivativetoto
select
selectsignificant
significantparameters
parameters
for for shape
shape description.
description. Parameters
Parameters are arelocation
the the location (min/max),
(min/max), the the curvature
curvature valuesvalues
atat at these
these locations
locations
for shape description. Parameters are the location (min/max), the curvature values these locations
(red
(red (red
lines) lines)
lines)and
andand
the
the the direction
direction
directionofof of curvature
curvature
curvature value value
value(up (up
for
(up for
convex
for convex
convex and
and and
down down
down forfor for concave
concave
concave behavior)
behavior)
behavior) [31]. [31].
[31].

In Figure 9,
InInFigure the normalized spectra and the calculated curvatures for polystyrene
Figure9,9,thethenormalized
normalizedspectra spectraand andthe thecalculated
calculatedcurvatures
curvaturesfor forpolystyrene
polystyrene(PS) (PS)
and (PS) and
polypropylenepolypropylene
(PP) are (PP)
shown, are shown,
which which
express a express
clear a clear
difference difference
in the in thefoot-
spectral spectral
and polypropylene (PP) are shown, which express a clear difference in the spectral foot-
footprint of both materials. Figure 8. The
Uselength
of minimumof the and green and redpositions
maximum lines indicates the strength
of 2nd derivative of significa
to select
print
printofofboth
bothmaterials.
materials.The The length
length ofofthethegreen
green andandred redlines
linesindicates
indicates the
thestrength
strength ofofthethe values at t
the curvature. In order fortoshape
takedescription.
only significant Parameters are the location
curvatures into account,(min/max), the curvature
a threshold value
curvature.
curvature.InInorderordertototake
takeonly
only
(red
significant
significant
lines)
curvatures
curvatures
and the direction
into
intoaccount,
of curvature account,a (up athreshold
threshold value
value must
must for concave b
must be used. The red lines represent the significant bandsvalue for convex
with a curvature and down
value higher
bebeused.
used.The Theredredlines
linesrepresent
representthe thesignificant
significantbands bandswith witha acurvature
curvaturevalue valuehigher
higherthan than
than the defined threshold and are derived on the basis of the curvature points presented
the
thedefined
definedthreshold
thresholdand andare are derived
In derived
Figure on
9,on the
the the basis
basisofofthe
normalized thecurvature and points
curvature points presented
presentedinin
in magenta and red colored dots. The direction of the spectra
lines (positive the calculated
or negative)curvatures
indicates for poly
magenta
magenta and
and red
redcolored
colored dots.
dots. The
The direction
direction ofofthe
the lines
lines (positive
(positive or ornegative)
negative) indicates
indicates
the type of curvature behavior (concave and convex curve) and is also an important source in the s
and polypropylene (PP) are shown, which express a clear difference
the
thetype
type ofofcurvature
curvature behavior
print(concave and convex curve) andof isthe
isalso ananimportant source
of information to behavior
describe of(concave
both
the shape and
materials. convex
of spectral The curve)
length
signatures.and also
green
Building important
andonred these source
lines indicates the st
extracted
ofofinformation
information totodescribe
describe thethe
curvature.shape
shape ofof spectral
In order spectral
to takesignatures.
signatures.
only significantBuilding
Building onon
curvaturesthese
these extracted
extracted
parameters regarding the spectra for each material and the 3D features into account,
describing a threshol
the
parameters
parameters regarding
regarding the bespectra
used.for each
red material and the 3D features describing the a ge-
geometric domain, the spectra
a collection for
The each
of conditionsmaterial
lines canand
represent the
be used the 3D features
significant
for describing
bands The
classification. with the ge-
curvature
combined value
ometric
ometric domain, a acollection ofofconditions can bebeused for forclassification. The combined
use ofdomain,
both domains collection
the
in defined
particular conditions
threshold
allows tocan and
mutually used
are support
derived classification.
on content
the the basis The
of of combined
theindividual
each curvature points
use
useofofboth
bothdomains
domainsininparticular
magentaallows
particular allows
and redtotomutually
mutually
colored support
dots. The the
support thecontent
directioncontentofof ofeach
the each
linesindivid-
individ-
(positive or negati
ual
ualdomain.
domain.For Forexample,
example, 3D3D
the features
features
type are
aremainly
of curvature mainly used
usedinin
behavior rule
ruleformation
(concave formation
and convexwhen
when thethespectral
curve) spectral
and is also an impo
information
informationdoes doesnot notprovide
of information to describe the shape of spectral signatures. in-
provide enough
enough information
information (e.g.,
(e.g.,black
black objects
objects on onPCB).
PCB). These
These in-
Building on the
clude,
clude,for forexample,
example,microchips.
microchips. IfIfthe
the rules
rules ininthethespectral
spectral domain
domain dodo
parameters regarding the spectra for each material and the 3D features descr not
not produce
produce any
any
ometric domain, a collection of conditions can be used for classification. Th
 cally rectangular and have a planar surface, the rule is defined as follows.

𝐈𝐅 (omnivariance > threshold A) AND (change in curvature < threshold B)

Appl. Sci. 2021, 11, 8424
THEN class → rectangular box 9 of 19

A rule formation should follow a logical approach and be based on the characteris
of the objects.
domain. ForAs example,
example, in the are
3D features case of theused
mainly microchip, the feature
in rule formation when‘change in curvat
the spectral
is used to detect
information thenot
does planar
providesurface
enoughofinformation
the object.(e.g.,
Thisblack
means thatonthe
objects threshold
PCB). These value
include, for example, microchips. If the rules in the spectral domain
this characteristic must be set low, because a planar surface has no curvature.do not produce anyThe thr
results, geometric rules are then checked. In the case of microchips, which are geometrically
olds used in our approach (threshold A = 10 and threshold B = 0.045) were empiric
rectangular and have a planar surface, the rule is defined as follows.
determined by investigations based on the underlying dataset. In the case of a diffe
data basisIF (due to different
omnivariance sensorAtechnology,
> threshold ) AND (change itinwould only
curvature < be necessary
threshold B) to adjust
THEN class → rectangular box
thresholds, as the features used reflect the characteristics of the objects.

(a) (b)
Figure 9.Figure
Continuum Removed
9. Continuum Reflectance
Removed spectra
Reflectance spectra (black line),calculated
(black line), calculated curvature
curvature valuesvalues (green
(green lines), lines), maximum
maximum points points
with concave behavior (red dots) and minimum points with convex behavior (magenta dots) for (a)
with concave behavior (red dots) and minimum points with convex behavior (magenta dots) for (a) Polystyrene (PS); (b) Polystyrene (PS);
(b) Polypropylene
Polypropylene (PP) [31].(PP) [31].
A rule formation should follow a logical approach and be based on the characteristics
ofTo
thebetter illustrate
objects. the in
As example, rule
the formation in the spectral
case of the microchip, domain,
the feature ‘change weinrefer to the exam
curvature’
of the spectral
is used curves
to detect the of PP and
planar PS of
surface from
the Figure 9 and
object. This express
means thethreshold
that the resulting rule as sho
value
for this
in Listing 1. characteristic must be set low, because a planar surface has no curvature. The
thresholds used in our approach (threshold A = 10 and threshold B = 0.045) were empirically
determined by investigations based on the underlying dataset. In the case of a different
Listing 1. Shape-based Rule for PS and PP.
data basis due to different sensor technology, it would only be necessary to adjust the
thresholds, as the features used reflect the characteristics of the objects.
𝐈𝐅 𝐶𝑉1108To better
< illustrate
−0.1 AND the rule formation in the𝐈𝐅 𝐶𝑉1128
spectral domain,< we−0.1
refer AND
to the example
of the spectral curves of PP and PS from Figure 9 and express the resulting rule as shown
𝐶𝑉1174 < 1.−0.1 AND
in Listing 𝐶𝑉1342 < −0.1 AND
𝐶𝑉1608Listing<1. Shape-based
−0.1 ANDRule for PS and PP. 𝐶𝑉1190 > +0.1 AND
IF CV1108 < −0.1 AND IF CV1128 < −0.1 AND
𝐶𝑉1143 CV1174
> <+0.1 AND
−0.1 AND CV𝐶𝑉
13421215
> +0.1
< −0.1 AND AND
CV1608 < −0.1 AND CV1190 > +0.1 AND
𝐶𝑉1204 CV1143
> >+0.1 AND
+0.1 AND CV𝐶𝑉
12151387
> +0.1
> +0.1 AND AND
CV1204 > +0.1 AND CV1387 > +0.1 AND
𝐶𝑉+1677
CV1677 > 0.1 > +0.1 CV𝐶𝑉 > +0.1 > +0.1
16941694
THEN class → PS THEN class → PP
𝐓𝐇𝐄𝐍 𝑐𝑙𝑎𝑠𝑠 → 𝑃𝑆 𝐓𝐇𝐄𝐍 𝑐𝑙𝑎𝑠𝑠 → 𝑃𝑃
The rule for each material includes a number of conditions. Each of these conditions
is based on the Curvature Value (CV) at a specific band and the sign of the curvature value.
IfThe
the sign
ruleisfor
negative, the curvature
each material behavior
includes is concave
a number of and CV must Each
conditions. be smaller than condit
of these
the defined threshold value. For a convex curvature behavior, the sign is positive, i.e., the
is based on the Curvature Value (CV) at a specific band and the sign of the curva
curvature value must be greater than the defined threshold value.
value. If the sign is negative, the curvature behavior is concave and CV must be sma
Appl. Sci. 2021, 11, x FOR PEER REVIEW 10 of 19

Appl. Sci. 2021, 11, 8424 10 of 19

than the defined threshold value. For a convex curvature behavior, the sign is positive
i.e., the curvature value must be greater than the defined threshold value.
As mentioned
As mentioned before, forfor
before, rule
ruleformation,
formation,mainly
mainlythethesignificant bandswith
significant bands withhigh
high curva
turecurvature
value arevalue are considered
considered (red lines).
(red lines). Each Each significant
significant bandband leads
leads totoa acondition.
condition. By
By analyz-
analyzing the spectra beforehand, the number of conditions can be reduced
ing the spectra beforehand, the number of conditions can be reduced to a minimum and no to a minimum
and not all significant curvature values (red lines) have to be taken into account. Therefore,
all significant curvature values (red lines) have to be taken into account. Therefore, only six
only six conditions are defined for sample PS in Listing 1, while in Figure 9 a total of seven
conditions
red linesare
aredefined
present. for
An sample
overview PSofinthe
Listing 1, while
used spectra forin
theFigure 9 a total of
rule-formation seveninred lines
is given
are present.
Figure 10.An overview of the used spectra for the rule-formation is given in Figure 10.

Figure 10. Spectral signatures used for the classification. As a result, the rule defined for a category is specific to that
Figure 10. Spectral signatures used for the classification. As a result, the rule defined for a category is specific to that
category and there is no possibility of confusion because the shapes are distinct enough. Such rules can therefore only be
category and there is no possibility of confusion because the shapes are distinct enough. Such rules can therefore only be
used if the shapes of the spectral curves are all sufficiently different.
used if the shapes of the spectral curves are all sufficiently different.
3. Results
3. Results
This section shows the steps of the rule-based classification. For a better understanding,
This
an sectionofshows
overview the steps
the process of theinrule-based
is shown Figure 11. classification. For a better
It can be understood understanding
as an iterative
an overview
refinementof the process
starting is shown
with simple in Figureconditions
and dominant 11. It canandbe continuing
understood as an
in the iterative
direction of refine
mentmore complex
starting withand composite
simple andconditions.
dominantThe first step and
conditions of thecontinuing
classificationinprocess is based of more
the direction
on theand
complex knowledge
composite about the spectralThe
conditions. behavior of plastics,
first step of thewhich is significantly
classification different
process to on the
is based
those of PCBs. We use this knowledge to separate these two groups and check the average
knowledge about the spectral behavior of plastics, which is significantly different to those
spectrum for each cluster. Since plastic objects are usually made of one material and not
of PCBs. We use this
of a combination of knowledge to separate
materials as for PCBs (board theseandtwo groups
different and checkthe
components), theaverage
average spec
trumspectrum
for eachprovides
cluster.a Since plastic objects
good approach are the
to classify usually
objectsmade of one(Figure
in question material12). and
Thisnot of a
combination
procedure ofnotmaterials
only ensures as afor PCBs
clean (board and
classification different
of these objects,components), the average spec
but also offers advantages
truminprovides
terms of performance,
a good approach as not to
every singlethe
classify pixel has toinbequestion
objects checked.(Figure 12). This procedure
not onlyBy assuming that all unclassified objects are objects consisting of composite materials,
ensures a clean classification of these objects, but also offers advantages in terms o
the classification of clusters can be refined in the next step. The knowledge about the
performance, as not
objects allows us toevery
adaptsingle pixel has
our approach to to
thebe checked. of the objects; in this case,
characteristics
circuit boards. That means, the PCBs are separated into individual parts consisting of
the board (a plane surface) and the individual electronic components on the board (ecob).
The separation between board and components is based on plane estimation [34], while
the components on the board are again processed based on Euclidean distance clustering.
 Appl. Sci. 2021, 11, 8424 11 of 19

Next, once again, a rule check of the average spectra for each cluster (board and board
components) is carried out. As a result (see example shown in Figure 13) the areas that11 of 19
Appl. Sci. 2021, 11, x FOR PEER REVIEW
were divided into boards and components can now be subdivided according to their
material properties. In the example dataset, we thus obtain a classification of three different
Appl. Sci. 2021, 11, x FOR PEER REVIEW 11 of 19
board types, three different types of capacitors and a connector consisting of the material
polystyrene (PS).

Figure 11. Steps of the rule-based classification process using spectral and spatial features for the rule formation. The
Figure 11.
Figure Steps of
11. Steps of the
the rule-based
rule-based classification
classification process
process using spectral
spectral and
and spatial
spatial features
features for
for the
the rule
rule formation. The
geometrical information is not only used for structuring using
but also supports the classification formation.
of components thatThe
cannot be
geometrical information
geometrical information isis not
not only
only used
used for
for structuring
structuring but
but also
alsosupports
supportsthe
theclassification
classification of
ofcomponents
componentsthatthatcannot
cannotbe
be
classified from their spectral signature only.
classified from
classified from their
their spectral
spectral signature only.

Figure 12. Example of result of classification (after step 1) of plastic objects by checking average spectra for each cluster.
Figure 12. Example of result of classification (after step 1) of plastic objects by checking average spectra for each cluster.
By assuming(after
Figure 12. Example of result of classification that step
all unclassified
1) of plasticobjects
objectsare
byobjects consisting
checking averageof composite
spectra materials,
for each cluster.
the classification of clusters can be refined in the next step. The knowledge about the objects
allows
By us to adaptthat
assuming our all
approach to theobjects
unclassified characteristics of the
are objects objects; inofthis
consisting case, circuit
composite materials,
boards. That means, the PCBs are separated into individual parts consisting of the board (a
the classification of clusters can be refined in the next step. The knowledge about the objects
plane surface) and the individual electronic components on the board (ecob). The separation
allows us to adapt our approach to the characteristics of the objects; in this case, circuit
between board and components is based on plane estimation [34], while the components on
boards. That means, the PCBs are separated into individual parts consisting of the board (a
the board are again processed based on Euclidean distance clustering. Next, once again, a
plane surface)
rule check andaverage
of the the individual electronic
spectra for components
each cluster (board andon the board
board (ecob).isThe
components) separation
carried
 Appl. Sci. 2021, 11, 8424 12 of 19

All other existing components do not provide clean spectra that can be used for
classification, either because of the black coloring (high absorption) or because of the
material composition (metals). However, they are surrounded by detected and classified
elements. This relation can be expressed by geometric and topological features allowing
to further refine the classification. Figure 14 shows an example of results where a total
of four additional groups of components were classified. The four groups include round
ppl. Sci. 2021, 11, x FOR PEER REVIEW capacitors, rectangular planar objects (such as microchips), resistors and other components 12 o
(such as potentiometers and connectors). Especially in this case of classification based on
geometric features, it can be seen that the classification of board components is challenging,
particularly when dealing with dirty and damaged elements. Even though most of the
example dataset,
components we been
have thus correctly
obtain aclassified,
classification
it canofbethree
seen, different board
for example, thattypes,
one of three
the diffe
typesconnectors
of capacitors andassigned
has been a connector
to theconsisting of the
class of radial material
capacitors polystyrene
(ecob 5). This is (PS).
due to
geometrical similarities shown in Figure 15.

Figure 13. Result of classification for PCBs using spectral features (after step 2). Classification of three board types and four
Figure 13. Resulttypes
different of classification
of components for PCBs(PS,
on board using
ecob spectral features
1, ecob 2 and ecob 3).(after step 2). Classification of three board types and
four different types of components on board (PS, ecob 1, ecob 2 and ecob 3).
The smaller the objects are, the more important it is to use 3D sensors that allow a
high-resolution detection
All other existing so that evendo
components thenot
smallest objects
provide can spectra
clean still be captured
that can in be
detail.
used for c
In this context, it is also important to mention that the sensors chosen for data acquisition
sification,
should either because
always be selectedofinthe black coloring
accordance (high absorption)
with the application. Capturing or because
small of the mate
objects with
composition (metals).
a low-resolution However,
3D scanner they
will not are
give surrounded
satisfactory by This
results. detected
is also and classified
reflected in the eleme
Thisstatistics
relationcomputed
can be expressed byshown
for each class geometric
in Tableand topological
1. With decreasingfeatures allowing
object size, an exact to furt
classification becomes more difficult. For the calculation of the statistics,
refine the classification. Figure 14 shows an example of results where a total of four ad the underlying
dataset was manually annotated and used for the computation of the confusion matrix.
tional groups of components were classified. The four groups include round capacit
rectangular planar objects (such as microchips), resistors and other components (such
potentiometers and connectors). Especially in this case of classification based on geom
ric features, it can be seen that the classification of board components is challenging, p
ticularly when dealing with dirty and damaged elements. Even though most of the co
 Appl. Sci. 2021, 11, 8424 13 of 19

Appl. Sci. 2021, 11, x FOR PEER REVIEW 13 of 19

lassification for PCBs using 3D features (after step 3). Classification of

The overall statistics for the entire dataset are shown in Table 2. It can be stated that with
an Overall Accuracy (OA) of 98.24%, a satisfactory result is achieved.
4, ecob 5, ecob 6 and ecob 7).

The smaller the objects are, the more important it i

high-resolution detection so that even the smallest objec
In this context, it is also important to mention that the se
should always be selected in accordance with the appl
with a low-resolution 3D scanner will not give satisfactor
the statistics computed for each class shown in Table 1
exact classification becomes more difficult. For the calcu
lying dataset was manually annotated and used for the c
trix. The overall statistics for the entire dataset are show
with an Overall Accuracy (OA) of 98.24%, a satisfactory
FigureFigure 14. Result
14. Result of classification
of classification for PCBs
for PCBs usingusing 3D features
3D features (after(after
step step 3). Classification
3). Classification of four
of four different
different types
types of
of compo-
components on board (ecob 4, ecob 5, ecob
nents on board (ecob 4, ecob 5, ecob 6 and ecob 7).6 and ecob 7).

The smaller the objects are, the more important it is to use 3D sensors that allow a
high-resolution detection so that even the smallest objects can still be captured in detail.
In this context, it is also important to mention that the sensors chosen for data acquisition
should always be selected in accordance with the application. Capturing small objects
with a low-resolution 3D scanner will not give satisfactory results. This is also reflected in
the statistics computed for each class shown in Table 1. With decreasing object size, an
exact classification becomes more difficult. For the calculation of the statistics, the under-
lying dataset was manually annotated and used for the computation of the confusion ma-
trix. The overall statistics for the entire dataset are shown in Table 2. It can be stated that
with an Overall Accuracy (OA) of 98.24%, a satisfactory result is achieved.

Figure 15. Misclassification due to geometric similarities in the point cloud.

Figure 15. Misclassification due to geometric similarities in the

Figure 15. Misclassification due to geometric similarities in the point cloud.
Appl. Sci. 2021, 11, 8424 14 of 19

Table 1. Classification accuracies for each class based on 3D and HSI data.

Class OA of Single Error of Single F1-Score Kappa

PA 0.9969 0.0030 0.9975 0.9584
PP 0.9961 0.0380 0.9966 0.8781
PS 0.9861 0.0138 0.9794 0.9723
Board type 1 0.9075 0.0924 0.9327 0.7950
Board type 2 0.9560 0.0439 0.9755 0.9628
Board type 3 0.9689 0.0311 0.9789 0.9325
ecob 1 0.8936 0.1063 0.9240 0.9947
ecob 2 0.7206 0.2793 0.7594 0.9947
ecob 3 0.7764 0.2235 0.7948 0.9970
ecob 4 0.9849 0.0150 0.9094 0.9745
ecob 5 0.8100 0.1899 0.7931 0.9848
ecob 6 0.7114 0.2885 0.7127 0.9890
ecob 7 0.6595 0.3404 0.5831 0.9981

For comparison purposes, a SVM based classification (C-SVC, RBF-Kernel, C: 2048,

gamma: 2.8284), which is one of the most widely used pixel-wise classifiers, was addi-
tionally performed. The SVM method was implemented in MATLAB using the LIBSVM
library [35]. The optimization of the hyperparameters was determined by a 10-fold cross-
validation. For the training data set, 10% of samples of each class were randomly selected.
The main purpose of the comparison is to show, that an approach, combining 3D and
HSI, has advantages in terms of classification. The results of the SVM classification are
also shown in Table 2. Considering the statistics, it is clearly evident that a classification
processing 3D and HSI data outperforms a SVM classification based only on HSI.

Table 2. Classification accuracies for real electronic wastes using (a) Rule-based approach with
combined 3D and HSI dataset; (b) SVM classification (C-SVC, RBF-Kernel, C: 2048, gamma: 2.8284)
with HSI dataset.

(a) (b)
Metrics Rule-Based SVM
(3D + HSI) (HSI)
Overall Accuracy 0.9824 0.8865
Precision 0.8812 0.7812
Sensitivity 0.8834 0.6576
False Positive Rate 0.0023 0.0124
F1-Score 0.8810 0.6956
Kappa 0.8672 0.2011

4. Discussion
The results presented in Section 3 show significant added value of a combined use
of 3D and HSI for classification of waste PCBs. In order to get the maximum benefit from
such a combination, it is important for the strategic approach to analyze the underlying
data sufficiently. A well-developed knowledge base with regard to the characteristics of
objects helps to improve the results. Especially the addition of the 3D information is helpful
in structuring and simplifying the datasets (background removal, removal of shadows,
separation between objects). Hyperspectral data, on the other hand, help to determine
the type of material and thus refines the classification. Even more important, each of the
very different datasets supports the other and compensates the weaknesses of the other.
Many of the components that are mounted on circuit boards have a black material color.
The use of purely spectral information is not appropriate in these cases due to the high
absorption of light. This is also confirmed by the SVM comparison (Table 2), where only
HSI data were used for classification. A purely 3D-based classification can be successful in
separating objects (e.g., board and components on board), but it would fail, for example, in
differentiating between different board materials or compositions.
Appl. Sci. 2021, 11, 8424 15 of 19

Especially with 3D data, the resolution and accuracy of the dataset used plays a major
role. The higher the resolution, the more accurate the description of very fine, small objects
on the board. We see in our experiments that we reach our limits with objects shown in
Figure 16 and that a clean separation of small elements such as metals (red and blue box),
resistors (green box) and capacitors (yellow box) is not successful. Nevertheless, even if a
distinction between these elements does not seem possible in detail, these elements can be
identified as components of PCBs through the given topological relationships, which could
provide an advantage for further analyses (e.g., determination of different metals).
The classification map shown in Figure 14 is the result of a total of 13 defined rules and
works well for the materials and components in the used dataset. An extension to further
objects and materials needs the exploitation of their characteristics and the definition of
additional rules. This has to be completed carefully in order to avoid conflicting definitions
and needs the existence of spatial and spectral characteristics allowing to differentiate
further object types. However, the iterative strategy going from more dominant to fewer
dominant rules has shown, that even smallest elements can be separated with acceptable
accuracy. This is already a reliable base for further extensions. Nevertheless, it is necessary
to pay attention to correct modelling, and thus it is important to study the spatial and
spectral properties of the objects beforehand and to investigate to what extent the acquired
data can make these properties visible. Generally, when setting up rules, it is important to
avoid overfitting. Thresholds that are overly sensitive can result in incorrect classifications
when applied to new datasets and should therefore be chosen carefully. In addition, the
underlying 3D data quality must be considered when forming rules based on 3D geometry.
As example, in this article the parameters and thresholds for rule formation were chosen
with regard to the used sensor system and the resulting point cloud. In the case of a
different 3D sensor with another resolution, accuracy and noise behavior, it would be
necessary to adjust the parameters.
Another aspect to consider is the performance of the system and the assessment of use
in industrial practice. We are aware that the use in industrial practice poses certain chal-
lenges such as high-speed processing, an appropriate integration of the sensor technology
into the sorting process and also the handling of dust and dirt in an industrial context is
a factor that should not be underestimated. This means that the sensor technology used
and the acquisition and processing strategy must meet these requirements. The approach
proposed in this article has the potential to fulfil these demands, but certain modifications
with regard to the 3D sensor technology are necessary. The strategy of the approach (sen-
sors for acquisition, registration approach and data processing) is general and is mainly
affected by the specifications and the capacity of the senor technology used. The structured
light scanner used in this article is not advisable in the context of a sorting process due
to the needed fast processing speed. However, it has the benefit to produce high quality
3D data, which in turn is advantageous in the context of small objects. The use of such
a sensor system would therefore be recommended in the context of randomized quality
checks where speed is not important.
However, there are alternatives that are suitable for use in production lines [36,37].
The study of [38] investigates the applicability of 3D sensor technologies in the context
of production lines. As a result, four techniques for capturing 3D data in the context of
industrial applications were identified, namely laser triangulation, time-of-flight (ToF),
shape-from-focus (SFF) and stereovision [38]. Mainly laser scanners based on triangulation
are widely used in production lines. Such systems are able to achieve depth resolutions
down to the range of a few micrometers and are a robust and standardized technology that
could also be used in the context of the proposed approach.
The use of hyperspectral sensors in industrial applications is also conceivable and
is particularly widespread in the food industry [39–41]. The Specim FX17 sensor used
was able to detect 670 FPS at full resolution. It is not always necessary to use the full
resolution. By analyzing the data beforehand, spectral ranges or even only a small number
of bands can be used, which ultimately leads to a further increase in the detection rate.
on 3D geometry. As example, in this article the parameter
mation were chosen with regard to the used sensor system
In the case of a different 3D sensor with another resolution
Appl. Sci. 2021, 11, 8424 16 of 19

it would be necessary to adjust the parameters.

Important in the context of hyperspectral sensor technology is the illumination unit. This
must be installed in the measurement setup in such a way that the 3D technology used is
not influenced.

Figure 16. Small components such as metals (red and blue box), resistors (green box) and capacitors (yellow box) are
difficult to distinguish because of limits in resolution. Nevertheless, they can be classified as components of a PCB due to
the topological relationships provided by the 3D data.
Figure 16. Small components such as metals (red and blue box), re
At this point we would also like to briefly discuss the influence of object soiling on the
(yellow box) aremeasurements
difficult of theto distinguish
hyperspectral because
sensors, since in of limits
practice it is precisely such objectsin
are involved. Dirt on objects influences the reflectance on the surface of the objects. This
that reso

classified as components
inevitably leads toof a PCB
changes and shiftsdue to the
in the spectral topological
signatures relations
as can be seen in Figure
for one of the samples in the used dataset. That our proposed approach is robust to such
17

influences can be seen from the achieved results. This is due to two main facts: First, we
use spectral features that describe the shape of spectral signatures, even if the expression
Another aspect to consider is the performance of the
of absorption or reflectance peaks is low due to the weak reflectance, the shape of the
spectral signature remains the same. Thus, the defined rules based on the shape apply in
both cases. The second point is due to the 3D component. Through the 3D information we
use in industrial practice. We are aware that the use in ind
can group components in clusters and for the rule check we refer to the average spectral
signature of this cluster. Thus, the influence of dirty areas is only significant if the object is
challenges such as high-speed processing, an appropriate
completely soiled.

nology into the sorting process and also the handling of

context is a factor that should not be underestimated.
Appl. Sci. 2021, 11, 8424 17 of 19

The processing time of the data is also a factor that needs to be taken into account. The
3D processing and point-by-point checking for conditions using MATLAB on a machine
with an Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz and 16 GB RAM was 110.29 s for
the background removed dataset (752.253 3D points and 224 bands) shown in Figure 4.
For comparison, processing the same dataset (only HSI) on the same machine using SVM
(LIBSVM for MATLAB) requires 2025.07 s.
For a practical application, 110.29 s is considered high. However, in this context it
must also be considered that a full spectral and high-resolution 3D point cloud serves
as the data basis. Optimizations in terms of processing time is conceivable and could be
Appl. Sci. 2021, 11, x FOR PEER REVIEW 17 of 19
achieved by reducing the data basis (e.g., reduction of spectral resolution), by using more
powerful hardware and by more efficient programming (e.g., parallel computing).

(a)

(b)
Figure 17. Influence of dirtFigure
on the17.
reflectance
Influenceofof
spectral
dirt onsignatures. (a) Spectra
the reflectance of a heavily
of spectral polluted
signatures. area (red
(a) Spectra of box). (b) Spectra
a heavily polluted
of areas without dirt (green box).
area (red box). (b) Spectra of areas without dirt (green box).

The processing time of the data is also a factor that needs to be taken into account. The
3D processing and point-by-point checking for conditions using MATLAB on a machine with
an Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz and 16 GB RAM was 110.29 s for the background re-
moved dataset (752.253 3D points and 224 bands) shown in Figure 4. For comparison, pro-
cessing the same dataset (only HSI) on the same machine using SVM (LIBSVM for
MATLAB) requires 2025.07 s.
Appl. Sci. 2021, 11, 8424 18 of 19

5. Conclusions
In this paper, a combined dataset consisting of a 3D point cloud and an HSI was
used to classify different types of materials and objects of electronic waste. In addition
to the classification of different plastic materials, it has been shown that a combination of
geometric and physical information can also help to distinguish components on circuit
boards, such as capacitors, transistors and other small parts on board. Using a rule-based
classification approach leads to satisfying results with an OA of 98.24%. The comparison
with an SVM approach based only on spectral information results in an OA of 88.65% and
confirms that a combined dataset helps to improve the quality of classification.

Author Contributions: Conceptualization, S.P., A.T. and F.B.; Investigation, S.P.; Methodology, S.P.,
A.T. and F.B.; Supervision, A.T. and F.B.; Writing—original draft, S.P.; Writing—review and editing,
A.T. and F.B.; All authors have read and agreed to the published version of the manuscript.
Funding: This research was funded by the European Union from the European Regional Develop-
ment Fund and the state of Rhineland-Palatinate.
Institutional Review Board Statement: Not applicable.
Informed Consent Statement: Not applicable.
Acknowledgments: We thank Pellenc ST for supplying the real waste materials.
Conflicts of Interest: The authors declare no conflict of interest. The funders had no role in the design
of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or
in the decision to publish the results.

References
1. Forti, V.; Baldé, C.; Kuehr, R.; Bel, G. The Global E-Waste Monitor 2020: Quantities, Flows, and the Circular Economy Potential; United
Nations University (UNU); United Nations Institute for Training and Research (UNITAR)–co-hosted SCYCLE Programme; Inter-
national Telecommunication Union (ITU); International Solid Waste Association (ISWA): Bonn, Germany; Geneva, Switzerland;
Rotterdam, The Netherlands, 2020; ISBN 978-92-808-9114-0.
2. Calvini, R.; Ulrici, A.; Amigo, J.M. Growing applications of hyperspectral and multispectral imaging. Des. Optim. Org. Synth.
2020, 32, 605–629. [CrossRef]
3. Li, W.; Esders, B.; Breier, M. SMD segmentation for automated PCB recycling. In Proceedings of the 2013 11th IEEE International
Conference on Industrial Informatics (INDIN), Bochum, Germany, 29–31 July 2013; pp. 65–70.
4. Jessurun, N.T.; Paradis, O.P.; Tehranipoor, M.; Asadizanjani, N. SHADE: Automated Refinement of PCB Component Estimates
Using Detected Shadows. In Proceedings of the 2020 IEEE Physical Assurance and Inspection of Electronics (PAINE), Washington,
DC, USA, 28–29 July 2020; Institute of Electrical and Electronics Engineers (IEEE): Piscataway, NJ, USA, 2020; pp. 1–6.
5. Herchenbach, D.; Li, W.; Breier, M. Segmentation and classification of THCs on PCBAs. In Proceedings of the 2013 11th IEEE
International Conference on Industrial Informatics (INDIN), Bochum, Germany, 29–31 July 2013; Institute of Electrical and
Electronics Engineers (IEEE): Piscataway, NJ, USA, 2013; pp. 59–64.
6. Li, D.; Li, C.; Chen, C.; Zhao, Z. Semantic Segmentation of a Printed Circuit Board for Component Recognition Based on Depth
Images. Sensors 2020, 20, 5318. [CrossRef]
7. Khan, M.J.; Khan, H.S.; Yousaf, A.; Khurshid, K.; Abbas, A. Modern Trends in Hyperspectral Image Analysis: A Review. IEEE
Access 2018, 6, 14118–14129. [CrossRef]
8. Ibrahim, A.; Tominaga, S.; Horiuchi, T. Spectral imaging method for material classification and inspection of printed circuit
boards. Opt. Eng. 2010, 49, 057201. [CrossRef]
9. Palmieri, R.; Bonifazi, G.; Serranti, S. Recycling-oriented characterization of plastic frames and printed circuit boards from mobile
phones by electronic and chemical imaging. Waste Manag. 2014, 34, 2120–2130. [CrossRef]
10. Sudharshan, V.; Seidel, P.; Ghamisi, P.; Lorenz, S.; Fuchs, M.; Fareedh, J.S.; Neubert, P.; Schubert, S.; Gloaguen, R. Object Detection
Routine for Material Streams Combining RGB and Hyperspectral Reflectance Data Based on Guided Object Localization. IEEE
Sens. J. 2020, 20, 11490–11498. [CrossRef]
11. Ghamisi, P.; Plaza, J.; Chen, Y.; Li, J.; Plaza, A.J. Advanced Spectral Classifiers for Hyperspectral Images: A review. IEEE Geosci.
Remote Sens. Mag. 2017, 5, 8–32. [CrossRef]
12. Andrés, S.; Arvor, D.; Mougenot, I.; Libourel, T.; Durieux, L. Ontology-based classification of remote sensing images using
spectral rules. Comput. Geosci. 2017, 102, 158–166. [CrossRef]
13. Berhane, T.M.; Lane, C.R.; Wu, Q.; Autrey, B.C.; Anenkhonov, O.A.; Chepinoga, V.V.; Liu, H. Decision-Tree, Rule-Based, and
Random Forest Classification of High-Resolution Multispectral Imagery for Wetland Mapping and Inventory. Remote Sens. 2018,
10, 580. [CrossRef]
Appl. Sci. 2021, 11, 8424 19 of 19

14. Cui, W.; Yao, M.; Hao, Y.; Wang, Z.; He, X.; Wu, W.; Li, J.; Zhao, H.; Xia, C.; Wang, J. Knowledge and Geo-Object Based Graph
Convolutional Network for Remote Sensing Semantic Segmentation. Sensors 2021, 21, 3848. [CrossRef]
15. Ghazaryan, G.; Dubovyk, O.; Löw, F.; Lavreniuk, M.; Kolotii, A.; Schellberg, J.; Kussul, N. A rule-based approach for crop
identification using multi-temporal and multi-sensor phenological metrics. Eur. J. Remote Sens. 2018, 51, 511–524. [CrossRef]
16. Houhoulis, P.F.; Michener, W. Detecting wetland change: A rule-based approach using NWI and SPOT-XS data. Photogramm. Eng.
Remote Sens. 2000, 66, 205–211.
17. Liu, S.; Shi, Q. Multitask Deep Learning with Spectral Knowledge for Hyperspectral Image Classification. IEEE Geosci. Remote
Sens. Lett. 2020, 17, 2110–2114. [CrossRef]
18. Ponciano, J.-J.; Roetner, M.; Reiterer, A.; Boochs, F. Object Semantic Segmentation in Point Clouds—Comparison of a Deep
Learning and a Knowledge-Based Method. ISPRS Int. J. Geo. Inf. 2021, 10, 256. [CrossRef]
19. Dvorak, R.; Kosior, E.; Moody, L. Development of NIR Detectable Black Plastic Packaging. Available online: http://www.wrap.
org.uk/sites/files/wrap/Recyclability_of_Black_Plastic_Summary.pdf (accessed on 11 January 2021).
20. Rozenstein, O.; Puckrin, E.; Adamowski, J. Development of a new approach based on midwave infrared spectroscopy for
post-consumer black plastic waste sorting in the recycling industry. Waste Manag. 2017, 68, 38–44. [CrossRef]
21. Roscher, R.; Behmann, J.; Mahlein, A.-K.; Dupuis, J.; Kuhlmann, H.; Plümer, L. Detection of disease symptoms on hyperspectral
3D plant models. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci. 2016, III-7, 89–96. [CrossRef]
22. Morsy, S.; Shaker, A.; El-Rabbany, A. Multispectral LiDAR Data for Land Cover Classification of Urban Areas. Sensors 2017,
17, 958. [CrossRef]
23. Igelbrink, F.; Wiemann, T.; Pütz, S.; Hertzberg, J. Markerless Ad-Hoc Calibration of a Hyperspectral Camera and a 3D Laser
Scanner. In Advances in Intelligent Systems and Computing; Springer Science and Business Media LLC: Berlin, Germany, 2018;
pp. 748–759.
24. Nieto, J.I.; Monteiro, S.T.; Viejo, D. 3D geological modelling using laser and hyperspectral data. IEEE Int. Geosci. Remote Sens.
Symp. 2010, 4568–4571. [CrossRef]
25. Wendel, A.; Underwood, J. Extrinsic Parameter Calibration for Line Scanning Cameras on Ground Vehicles with Navigation
Systems Using a Calibration Pattern. Sensors 2017, 17, 2491. [CrossRef]
26. Tan, P.-N.; Steinbach, M.; Karpatne, A.; Kumar, V. Introduction to Data Mining, 2nd ed.; Pearson: New York, NY, USA, 2019;
ISBN 9780133128901.
27. Shahtahmassebi, A.; Yang, N.; Wang, K.; Moore, N.; Shen, Z. Review of shadow detection and de-shadowing methods in remote
sensing. Chin. Geogr. Sci. 2013, 23, 403–420. [CrossRef]
28. Mostafa, Y. A Review on Various Shadow Detection and Compensation Techniques in Remote Sensing Images. Can. J. Remote
Sens. 2017, 43, 545–562. [CrossRef]
29. Weinmann, M.; Jutzi, B.; Mallet, C. Feature relevance assessment for the semantic interpretation of 3D point cloud data. ISPRS
Ann. Photogramm. Remote Sens. Spat. Inf. Sci. 2013, II-5/W2, 313–318. [CrossRef]
30. Weinmann, M.; Jutzi, B.; Hinz, S.; Mallet, C. Semantic point cloud interpretation based on optimal neighborhoods, relevant
features and efficient classifiers. ISPRS J. Photogramm. Remote Sens. 2015, 105, 286–304. [CrossRef]
31. Polat, S.; Tremeau, A.; Boochs, F. Rule-Based Classification of Hyperspectral Imaging Data. 2021. Available online: http:
//arxiv.org/pdf/2107.10638v1 (accessed on 9 August 2021).
32. Clark, R.N.; Swayze, G.A.; Livo, K.E.; Kokaly, R.; Sutley, S.J.; Dalton, J.B.; McDougal, R.R.; Gent, C.A. Imaging spectroscopy: Earth
and planetary remote sensing with the USGS Tetracorder and expert systems. J. Geophys. Res. Space Phys. 2003, 108. [CrossRef]
33. Clark, R.N.; Roush, T.L. Reflectance spectroscopy: Quantitative analysis techniques for remote sensing applications. J. Geophys.
Res. Space Phys. 1984, 89, 6329–6340. [CrossRef]
34. Torr, P.; Zisserman, A. MLESAC: A New Robust Estimator with Application to Estimating Image Geometry. Comput. Vis. Image
Underst. 2000, 78, 138–156. [CrossRef]
35. Chang, C.-C.; Lin, C.-J. LIBSVM: A library for support vector machines. ACM Trans. Intell. Syst. Technol. 2011, 2, 1–27. [CrossRef]
36. Bian, S.; Zhou, P.; Xu, J.; Zhang, J.; Shan, D. On-line detection device for high temperature forgings based on laser triangulation. J.
Phys. 2021, 1885, 052035. [CrossRef]
37. Sioma, A. Automated Control of Surface Defects on Ceramic Tiles Using 3D Image Analysis. Materials 2020, 13, 1250. [CrossRef]
38. Sioma, A. 3D imaging methods in quality inspection systems. In Photonics Applications in Astronomy, Communications, Industry, and
High-Energy Physics Experiments 2019; Romaniuk, R.S., Linczuk, M., Eds.; SPIE: Wilga, Poland, 2019; p. 91. ISBN 9781510630659.
39. Al-Sarayreh, M.; Reis, M.M.; Yan, W.Q.; Klette, R. Detection of Red-Meat Adulteration by Deep Spectral–Spatial Features in
Hyperspectral Images. J. Imaging 2018, 4, 63. [CrossRef]
40. Boldrini, B.; Kessler, W.; Rebner, K.; Kessler, R.W. Hyperspectral Imaging: A Review of Best Practice, Performance and Pitfalls for
in-line and on-line Applications. J. Near Infrared Spectrosc. 2012, 20, 483–508. [CrossRef]
41. Ma, J.; Sun, D.-W.; Pu, H.; Cheng, J.-H.; Wei, Q. Advanced Techniques for Hyperspectral Imaging in the Food Industry: Principles
and Recent Applications. Annu. Rev. Food Sci. Technol. 2019, 10, 197–220. [CrossRef] [PubMed]