Available online at www.sciencedirect.

com

ScienceDirect
IFAC PapersOnLine 56-2 (2023) 7609–7614
Predicting
Predicting Operators’
Operators’ Fatigue
Fatigue in
in a
a Human
Human
Predicting
in the
Predicting Operators’
Artificial
Operators’ Fatigue
Intelligence
Fatigue in a
Loop
in a Human
for
Human
in
in the
Predicting
the Artificial
Operators’
Artificial Intelligence
Fatigue
Intelligence Loop
in a
Loop for
Human
for
Defect
in the
Defect Detection
Artificial
Detection in Manufacturing
Intelligence
in Manufacturing Loop for⋆⋆⋆
in the Artificial
Defect
Defect Detection
Detection Intelligence
in
in Manufacturing
Manufacturing Loop for⋆
Defect
Jože M. Rožanec ∗,∗∗,‡
Detection
∗,∗∗,‡ Karelin
∗∗∗
Manufacturing
Križnar ⋆
∗∗∗∗,§
∗∗∗ Elias Montini ∗∗∗∗,§
Jože M. Rožanec
Vincenzo Karel Križnar
∗,∗∗,‡∗∗∗∗
Cutrona ∗∗∗ Elias
Erik| Montini ∗∗∗∗,§
Koehorst ‡
Jože
Jože M. Rožanec
M.Dunja
Rožanec Karel
∗,∗∗,‡∗∗∗∗
Karel Križnar
Križnar | BlažMontini
∗∗∗ Elias
Elias
Fortuna∗∗∗∗,§ ‡
Vincenzo Cutrona
Mladenić ∗∗∗∗ ∗∗ Erik Koehorst
Christos | BlažMontini
Emmanouilidis Fortuna † ‡
Vincenzo Cutrona ∗,∗∗,‡∗∗∗∗ ∗∗ Erik Koehorst ∗∗∗ | Blaž Fortuna † ∗∗∗∗,§
‡
Jože M.Dunja
Vincenzo Rožanec MladenićKarel
Cutrona ∗∗ Christos
ErikKrižnar
KoehorstEmmanouilidis
Elias
Blaž Montini
Fortuna †
Dunja Mladenić ∗∗∗∗ Christos Emmanouilidis |
Vincenzo
∗ DunjaCutrona
Mladenić Erik ∗∗ Koehorst
Christos Blaž Fortuna ‡
Emmanouilidis †
Jožef Stefan Institute,
∗ Dunja ∗∗ Ljubljana, Slovenia, (e-mail: †
Mladenić
∗ Jožef Stefan Institute, Christos
Ljubljana,Emmanouilidis
Slovenia, (e-mail:
∗ Jožef Stefan Institute,joze.rozanec@ijs.si)
Ljubljana, Slovenia, (e-mail:
∗∗ Jožef International joze.rozanec@ijs.si)
Stefan Institute, Ljubljana,School,
Slovenia, (e-mail: Slovenia
∗∗ Jožef Stefan
∗ Postgraduate
joze.rozanec@ijs.si)
joze.rozanec@ijs.si)
Ljubljana,
∗∗ Jožef StefanJožef International
Stefan Institute,Postgraduate
Ljubljana,
(e-mail: dunja.mladenic@ijs.si) School,
Slovenia, Ljubljana,
(e-mail: Slovenia
∗∗ Jožef Stefan International(e-mail:
Postgraduate School, Ljubljana, Slovenia
dunja.mladenic@ijs.si)
joze.rozanec@ijs.si)
Jožef
∗∗∗ Stefan International
University of Ljubljana, Postgraduate
Faculty of School, Ljubljana,
mathematics and physics,Slovenia
∗∗ ∗∗∗ (e-mail: dunja.mladenic@ijs.si)
Jožef University of Ljubljana, Faculty of mathematics
∗∗∗ Ljubljana, Slovenia, (e-mail: kk4876@student.uni-lj.si)Slovenia
Stefan International
(e-mail: Postgraduate
dunja.mladenic@ijs.si)School, and
Ljubljana, physics,
∗∗∗ University
Ljubljana,
of Ljubljana, Faculty of mathematics and physics,
University ∗∗∗∗ ofSlovenia,
(e-mail:
Ljubljana, (e-mail: kk4876@student.uni-lj.si)
dunja.mladenic@ijs.si)
Faculty of mathematics and physics,
∗∗∗ Ljubljana, ∗∗∗∗ SUPSI
Ljubljana, Slovenia,
Slovenia,
(e-mail:
(e-mail:
(e-mail:
elias.montini@supsi.ch,
kk4876@student.uni-lj.si)
kk4876@student.uni-lj.si)
University SUPSI
∗∗∗∗ of (e-mail:
Ljubljana, elias.montini@supsi.ch,
Faculty
vincenzo.cutrona@supsi.ch). of mathematics and physics,
∗∗∗∗ SUPSI (e-mail: elias.montini@supsi.ch,
vincenzo.cutrona@supsi.ch).
† Ljubljana,SUPSI Slovenia, (e-mail:
(e-mail: kk4876@student.uni-lj.si)
elias.montini@supsi.ch,
† University ∗∗∗∗ of Groningen (e-mail: c.emmanouilidis@rug.nl)
vincenzo.cutrona@supsi.ch).
† University‡ of
Qlector Groningen
SUPSI (e-mail:
d.o.o., (e-mail: c.emmanouilidis@rug.nl)
elias.montini@supsi.ch,
vincenzo.cutrona@supsi.ch).
Ljubljana, Slovenia (e-mail:
† University‡ of Groningen (e-mail: c.emmanouilidis@rug.nl)
University‡ Qlector d.o.o.,
of Groningen Ljubljana,
vincenzo.cutrona@supsi.ch).
(e-mail:
blaz.fortuna@qlector.com). Slovenia (e-mail:
c.emmanouilidis@rug.nl)
† ‡ Qlector d.o.o., Ljubljana, Slovenia (e-mail:
University § Qlector
of d.o.o., Ljubljana,
blaz.fortuna@qlector.com).
Groningen (e-mail: Slovenia (e-mail:
c.emmanouilidis@rug.nl)
ठPolitecnico di Milano, DEIB,
blaz.fortuna@qlector.com). Milan, Italy
| § Politecnico
Qlector di
d.o.o., Milano,
Ljubljana, DEIB,
blaz.fortuna@qlector.com). Milan,
Slovenia Italy
(e-mail:
| Philips Consumer § Lifestyle
Politecnico di BV,
Milano,Drachten,
DEIB, The Milan, Neatherlands
Italy (e-mail:
| Philips Consumer Lifestyle
Politecnico BV, Drachten,
blaz.fortuna@qlector.com).
di Milano, DEIB,
erik.koehorst@philips.com) The
Milan, Neatherlands
Italy (e-mail:
| Philips Consumer
§ Lifestyle BV, Drachten, The Neatherlands (e-mail:
Philips Consumer Politecnicoerik.koehorst@philips.com)
Lifestyle di BV,
Milano, DEIB, The
Drachten, Milan, Italy
Neatherlands (e-mail:
| erik.koehorst@philips.com)
Philips Consumer Lifestyle BV, Drachten, The Neatherlands (e-mail:
erik.koehorst@philips.com)
Abstract: Quality inspection, typically performed manually by workers in the past, is now
erik.koehorst@philips.com)
Abstract:
rapidly switchingQuality inspection, typically performed
to automated manually by workers in the past, is This now
Abstract: Quality inspection, solutions, using artificial
typically performed intelligence
manually by workers(AI)-drivenin themethods.
past, is now
rapidly
Abstract:
elevates switching
the Quality
job to automated
inspection,
function of thesolutions,
typically
quality using
performedartificial
inspection team intelligence
manually from bythe (AI)-driven
workers
physical in the methods. This
past, istasks
inspection now
rapidly switching to automated solutions, using artificial intelligence (AI)-driven methods. This
elevates
Abstract:
rapidly the
to tasksswitching job function
Quality
related tomanaging
automated of the
inspection, quality performed
typically
solutions, inspection
using team
manually
artificial from bythe
intelligence physical
workers
(AI)-drivenin theinspection
past,
methods. istasks
now
This
elevates the job to function workflows
of the quality ininspection
synergy with
teamAI fromagents, for example,
the physical interpreting
inspection tasks
to tasks
rapidly
elevates
inspection related
switching
the job
outcomes toto managing
automated
function of
or labeling workflows
thesolutions,
quality
inspection in synergy
using
inspection
image with
artificial
team
data AIfrom
forAIthe agents,
intelligence the
AI models. for example,
(AI)-driven
physical interpreting
methods.
inspection
In this interpreting This
tasks
context, we
to
to tasks
tasks related
related to managing workflows in synergy with agents, for example,
inspection
elevates
have the
studied
inspection howto
outcomes
job
outcomes
managing
or labeling
function
defect workflows
of the
inspection
or labeling
inspection
quality
can be
inspection
ininspection
synergydata
image
enhanced, with
teamforAI
image dataproviding
the
from
for the AI
agents,
AI
defect for
themodels. example,
physical
models.hints the interpreting
In inspection
this
Intothis
context,
operator
context,
taskswe
to
we
have
to studied
tasks
inspection
ease defect howtodefect
related
outcomes
identification. inspection
managing
or labeling workflows can we
inspection
Furthermore, be enhanced,
indeveloped
synergydata
image providing
withforAIthe
machine defect
agents,
AI
learning hints
for
models. Intoto
example,
models the
this operatorand
interpreting
context,
recognize to
we
have studied how defect inspection can be enhanced, providing defect hints to the operator to
ease
have
predictdefect
inspection
studied identification.
outcomes
how
operators’ or
defect Furthermore,
labeling
inspection
fatigue. Furthermore, inspection
By doing so, we can we
be developed
image
enhanced, machine
data for
providingthe learning
AI models.
defect models
hints Into to
this
therecognize
context,
operator and
we
to
ease defect identification. wecan proactively
developed machinetake mitigation
learning models actions toto enhanceand
recognize the
predict
have
ease operators’
studied
defect
workers’ how fatigue.
defect
identification.
well-being and ensure By doing
inspection
Furthermore, so,
canwe
the highest becan
we proactively
enhanced,
developed
defect machine
inspection take
providing mitigation
defect
learning
quality actions
hints
models
standards. toWe to
tothe enhance
operator
recognize
consider the
and
such to
predict
predict operators’
operators’ fatigue.
fatigue. By
By doing
doing so,
so, we
we can
can proactively
proactively take
take mitigation
mitigation actions
actions to
to enhance
enhance the
the
workers’
ease defectwell-being and
identification.
processeswell-being
to empower ensure the
Furthermore, highest we defect inspection
developed machine quality standards.
learning models We
to consider
recognize such
and
workers’ andhuman
ensure and non-human
the highest defectactors in manufacturing
inspection quality standards. and the Wesociotechnical
consider such
processes
predict
workers’
production to empower
operators’
well-being fatigue.
and human
ensure and
By doing non-human
so, we can
the highest defect actors
theproactively in manufacturing
inspection take mitigation
quality standards.and
actionsthe
Weto sociotechnical
enhance
consider the
such
processes tosystem.
empower Thehuman
paper first
and outlines
non-human conceptual approach
actors in manufacturing for integrating the
and the sociotechnical operator
production
workers’
processes to
in the AI-drivensystem.
well-being
empower The
and
quality paper
ensure
human first
the
and
inspection outlines
highest
non-human
process the the
defectconceptual
inspection
actors
while in approach
quality
manufacturing
implementing for
a fatigue integrating
standards.and We
the
monitoring the
consideroperator
sociotechnical such
production system. The paper first outlines conceptual approach for integrating thesystem
operator to
in the AI-driven
processes
production
enhance tosystem.
work quality
empower Thehuman
conditions. inspection
paper first
Furthermore, process
and outlines
non-humanit thewhile implementing
actors
conceptual
describes how approach
this was a fatigue
in manufacturing monitoring
and
for integrating
implemented system
the sociotechnical
bytheleveraging
operator to
in
in the
the AI-driven
AI-driven quality
quality inspection
inspection process
process while
while implementing
implementing a
a fatigue
fatigue monitoring
monitoring system
system to
to
enhance
production
data and workwork conditions.
system.
experiments The paperFurthermore,
performed first outlines it
for a real-world describes
the how
conceptual
manufacturingthis
approachwas implemented
for integrating
useimplemented
case. bytheleveraging
operator
enhance conditions. Furthermore, it describes how this was by leveraging
data
in theand
enhance experiments
AI-driven
work quality
conditions. performed
inspection forprocess
Furthermore, a real-worldwhile manufacturing
it describes implementing
how this was ause case.monitoring
fatigue
implemented system to
by leveraging
data and experiments
Copyright © 2023 The performed
Authors. This for
is a open
an real-world
access manufacturing
article under the useBY-NC-ND
CC case. license
enhance
data and work conditions.
experiments Furthermore,
performed it describes
for a real-world how this was
manufacturing useimplemented
case. by leveraging
(https://creativecommons.org/licenses/by-nc-nd/4.0/)
Keywords:
data manufacturing
and experiments plant control;
performed intelligentmanufacturing
for a real-world manufacturing; usehuman-centric
case.
Keywords:
manufacturing; manufacturing
fatigue monitoringplant control; intelligent manufacturing; human-centric
Keywords: manufacturing plant control; intelligent manufacturing; human-centric
manufacturing;
Keywords: fatigue monitoring
manufacturing plant control; intelligent manufacturing; human-centric
manufacturing; fatigue monitoring
Keywords:
manufacturing; manufacturing
fatigue monitoringplant control; intelligent manufacturing; human-centric
1. INTRODUCTION
manufacturing; fatigue monitoring activities. A typical example is in manufacturing quality
1. INTRODUCTION activities.
inspection.A typicalwasexample is inmanual
manufacturing
activity quality
1. INTRODUCTION activities. AThistypical a common
example is in manufacturing under-
quality
1. INTRODUCTION inspection.
activities. A This
typicalwas a common
example is in manual activity
manufacturing under-
but quality
Human-centricity 1. one of the core values of the evolv- taken
is INTRODUCTION taken
by workers
inspection.
by
activities. A
This was
workers
typical
in the
in the
quality control
a common
quality
example is
manualteams
control
in teams
activity
manufacturing but
is now
under-
is from
now
quality
Human-centricity is one of the core values of the evolv- inspection.
served by a This
range wasof a common
automated manual
solutions, activity
startingunder-
ing fifth industrial revolution. New approaches
Human-centricity is one of the core values of the evolv- inspection. must be taken by workers in the quality control teams but is now
ing fifth industrial
Human-centricity is revolution.
one of the New
core approaches
values of themust
evolv- be served
taken
simple bybyby a
computer range
This
workers was
inof automated
a
the common
quality solutions,
manual
control teams starting
activity
but is from
under-
now
devised
ing fifthtoindustrial
ensure that AI-driven
revolution. New solutions
approaches developedmust for be served a rangevision-based
of automated solutions.
solutions, The job profile
starting from
devised
ing fifth to ensure
Human-centricity
industrial that
is one AI-driven
of
revolution. the New solutions
core values
approaches developed
of the evolv-
must for
be simple
taken
served
of by
workerscomputer
by workers
a range vision-based
inof the
in suchvision-based quality
automated
teams increasingly solutions.
control
solutions, The
teams job
but
startingprofile
is now
from
manufacturing
devised to ensuresettings
that serve human
AI-driven workers
solutions and opera-
developed for simple computer solutions.involvesThe job sampling
profile
devised to
manufacturing
ing ensure that
settings AI-driven
serve human solutions developed
workers and opera-for of workers
served
simple by a
computerin such
range ofteams
vision-based increasingly
automated solutions. involves
solutions, The sampling
starting
job from
profile
tors fifth industrial
better,
manufacturing enhancing revolution.
settings their
serve work
Newconditions,
human
approaches
workers and
must
while be outcomes
also
opera- of workersofinthe
outcomes ofinthe
suchautomated
automated
visual inspection
teams increasingly
visual
involvestosampling
inspection to
confirm
confirm
tors better,
devised
contributing enhancing
to ensure
manufacturing to settings
operationaltheir
that AI-driven
serveand work
human conditions,
solutions
workers
enterprise while
developed
and
goals. opera-
Looking for simple
also of
the workerscomputer
automated
outcomes of
such
the
vision-based
teams increasingly
classification
automated
solutions.
of
visual parts or
inspection
The
involves job
larger
to
profile
sampling
compo-
confirm
tors better, enhancing their work conditions, while also of the automated
workers
outcomes of inthe classification
such teams
automated of
increasingly
visual parts or larger
involves
inspection to compo-
sampling
confirm
contributing
manufacturing
tors better, to operational
settings
enhancing
at artificial intelligence serve
their and
(AI) and workenterprise
human
- enabled workers
conditions, goals.
manufacturingand Looking
while opera-
also
from nents. Similarly, classification
the automated it involves curating of parts inspection
or larger data, such
compo-
contributing to operational enterprise goals. Looking nents. Similarly, itautomated
involves
at artificial
tors better,
contributing
such
at
intelligence
enhancing
to operational
a viewpoint,
artificial
(AI)
this paper
intelligence (AI)
- enabled
theirfocuses
and
-
work manufacturing
conditions,
enterprise
enabled on a keygoals. while
pattern
manufacturing
from
Looking also
infrom
outcomes
the automated
as inspection
the nents.
of the
Similarly, images,
it involves and curating
classification visual
of parts
labeling
curating
inspection
inspection
them
inspection
data,
for to
or larger trainingsuch
confirm
compo-
data, such a
at artificial
such a
contributing intelligence
viewpoint,
to this (AI)
paper
operational -
and enabled
focuses on manufacturing
enterprisea key pattern
goals. infrom
Looking the as
the inspection
nents. automated
Similarly, images,
it and
classification
involves labeling
of
curating parts them
or
inspection for training
larger compo-
data, sucha
emerging
such transformation
a viewpoint, this paper of jobs
focuses transitioning
on a key pattern in the machine
to functions as inspection
machine
nents.
as
learning
learning
Similarly,
inspection it
solution.
images,
solution.
images,
and This
involves
and
labeling
This
curating
labeling
requires
thema for
requires a
inspection
them
cognitive
training
cognitive
for data,
training
shift
a
shift
sucha
emerging
at artificial
such a
that involve transformation
intelligence
viewpoint, this
further cognitiveof
(AI)
paper jobs
- transitioning
enabled
focuses on a key to
manufacturing
pattern functions
infrom
the
human in the workers’
machine learning activities,
solution. resulting
This in higher
requires added-value
aa cognitive shift
emerging transformation of jobsinvolvement
transitioningfrom to functions machine
in
as the learning
workers’
inspection solution.
activities,
images, and This
resulting
labelingrequires
in higher
them cognitive
for shift
added-value
training
that
such a
workers
that
involve
emerging further
viewpoint,
and operators
involve
this
transformation cognitive
paperof focuses
jobs
instead of
further cognitive
involvement
on a key
transitioning
largely physical
involvement
from
pattern
to human
in
functions the
human outcomes.
from routine in the workers’ Such activities,
outcomes resultingcan hardly in be
higher achieved by thea
added-value
workers
emerging
that and
involve operators
transformation instead
further cognitiveof jobsof largely physical
transitioning
involvement from human machine
routine
to functions outcomes.
in
automated Such
learning
the workers’ outcomes
solution solution.
activities,
alone, can
Thishardly
resulting
without in be
requireshigher
human aachieved
cognitive byshift
added-value
involvement. the
workers and operators instead of largely physical routine outcomes. Such outcomes can hardly be achieved by the
workers
that
⋆ and
involve operators
further instead
cognitive of largely
involvement physical
from routine
human automated
in the
outcomes. workers’ solution
Such alone,
activities,
outcomes without
resulting
can hardly human
in higher
be involvement.
added-value
achieved by the
⋆ This work was supported by the Slovenian Research Agency and automated
Within thisSuch
outcomes.
automated
solution
context, alone,
outcomes
solution alone,
without
the present
can
without hardly
human
paper offers
be
human
involvement.
a novel
achieved by
involvement. view
the
⋆ This
workers
the work
and
European was supported
operators
Union’s by the
instead
Horizon 2020 Slovenian
of largely
program Research
physical
project
This work was supported by the Slovenian Research Agency and
Agency
STAR routine and
under Within this context, theloop
present paper offersvisual a novel view
⋆
the European
This work was Union’s Horizon
supported 2020Slovenian
by the program Research
project STARAgency under
and of the human
automated
Within thissolution in the
context, alone, of AI-driven
without
the present human
paper offers quality
involvement.
a novel view
grant
the agreement
European number
Union’s H2020-956573.
Horizon 2020 program project STAR under
⋆
grant
the agreement
European number
Union’s H2020-956573.
Horizon 2020Slovenian
program Research
project STAR under of the human
Within this context,in the theloop
present of AI-driven
paper offersvisual quality
a novel view
This
grant work was
agreement supported
number by the
H2020-956573. Agency and of the human in the loop of AI-driven visual quality
grant agreement number H2020-956573.
the European Union’s Horizon 2020 program project STAR under Within
of the this
human context,
in the the present
loop of paper
AI-driven offers a novel
visual view
quality
of the
2405-8963 Copyright © 2023 The Authors. This is an open access article under the CC BY-NC-ND license.
grant agreement number H2020-956573. human in the loop of AI-driven visual quality
Peer review under responsibility of International Federation of Automatic Control.
10.1016/j.ifacol.2023.10.1157
7610 Jože M. Rožanec et al. / IFAC PapersOnLine 56-2 (2023) 7609–7614
inspection, with interventions targeting the empowerment
of both the human and the AI agent to jointly perform the
intended activities in ways that would have been hard for
either type of actor alone to achieve. The empowerment
of the AI actor is achieved through the integration of
the human cognitive capabilities in the AI loop, in the
form of labeling instances of data and engaging with
the AI process through an Active Learning mechanism.
The empowerment of the human actors is twofold: first,
the AI-driven automated visual inspection system delivers
not only inspection outcomes but offers also hints and
explanations as to how this outcome was derived; second,
it supports the operator through a fatigue monitoring
system, thus enabling more optimal human engagement
and triggering fatigue prevention or mitigation actions.
The remainder of the paper is structured as follows. The
next section outlines related work on automated visual
quality inspection, human fatigue monitoring, and human
in the AI loop. Section 3 presents the use case that offers
the industrial context for the presented research. The
active learning methodology and experiment design are
provided in Section 4. Results are presented and analyzed
in Section 5. A brief discussion and the main conclusions
drawn from the research are provided in Section 6.
2. RELATED WORK
A brief overview of literature related to (a) visual quality
inspection; (b) humans in the AI loop; and (c) fatigue
monitoring is provided here. Visual quality inspection
is the main manufacturing operation application focus
and serves as a research baseline. The involvement of
the human in the AI loop is aimed to achieve dual
empowerment: of the worker; and of the AI approach.
Therefore key concepts about such involvement are part of
the methodology baseline. Finally, human workers, either
performing manual inspections or engaged in interacting
with AI for achieving a joint quality inspection outcome,
are subject to fatigue and relevant monitoring needs to be
considered.
2.1 Visual quality inspection
Quality inspection on engineered products may involve
simple visual features, such as colors, as well as more
complex ones, needed to detect defects, cracks, orientation
deficiencies, or other anomalies (Rajan et al. (2020)).
Quality assessment based on surface visual inspection re-
lates to products of varying shapes and sizes, and there-
fore the complexity of the task varies as well (Cao et al.
(2018)) (Tsai and Jen (2021)) (Yun et al. (2020)). Pro-
duction process quality monitoring may also involve con-
siderable complexity, especially in assembly operations.
(Frustaci et al. (2022)). Moreover, quality inspection is
particularly relevant also for remanufacturing production
processes, involving the assessment of individual parts
(Saiz et al. (2021)). Whether quality inspection applies to
the beginning of life (production), middle of life (operation
and maintenance), or end of life (such as in remanufac-
turing), a commonly encountered problem is that of data
sparsity: rarely the available sample images are sufficient
for training, a problem which can partly be handled by
data augmentation (Yun et al. (2020)). Whether it is in
the management and curation of image data exemplars,
the tuning of algorithmic parameters or the labeling of
instances for training, the role of the human in such pro-
cesses is undeniable and yet not sufficiently appreciated.
2.2 Human in the AI Loop
When considering the joint involvement of human and
AI actors in collaborative settings, the issue of trust
attains paramount importance. Trust in itself depends
on (a) AI offering an explanation about its outcomes,
related to the concept of explainable AI; (b) humans
exercising their own judgment regarding domain-specific
decision-making circumstances, related to the concept of
interpretable AI; (c) AI outcomes to achieve to narrow
the distance between decisions and actions, related to the
concept of actionable AI. All these require a careful design
for involving the human in the AI loop (Emmanouilidis
et al. (2019)) (Lyytinen et al. (2020)).
2.3 Human fatigue monitoring
Research on fatigue estimation produced several contribu-
tions in the last decade, with models to estimate human
fatigue while doing different kinds of activities, including
working, rehabilitation, and sports activities. In the re-
habilitation and sport domains, monitored activities are
usually predefined, with a desired level of fatigue demand,
with involved subjects just repeating an exercise (e.g.,
sit-to-stand exercise (Aguirre et al. (2021)), or shoulders
flexion and abduction, and elbow extension (Papakostas
et al. (2019))). When coming to work activities, the same
activity might be executed in different ways by each oper-
ator, possibly showing different fatigue demands. Studies
have been conducted to estimate the physical or mental
fatigue of operators while monitoring their job routine,
including driving and piloting (Hooda et al. (2022)), pal-
letizing and transporting of weighted containers (Maman
et al. (2020)), supply carrying (Lambay et al. (2021)),
construction (masonry) (Zhang et al. (2019)). Zhang et al.
(2014) monitored operators while just walking (involving
firefighters and construction workers).
Fatigue estimation is realized in different ways, including
biological features, physical features, interactions monitor-
ing, and hybrid approaches (Sikander and Anwar (2018)).
While there are many examples of driver fatigue estima-
tion, few can be identified in the manufacturing context.
A physiological feature like eye blinking has been applied
for fatigue estimation during the execution of push and
pull tasks (Biondi et al. (2023)), while biological charac-
teristics, such as heart rate variability and galvanic skin
conductance, have been used to estimate fatigue in the ex-
ecution of human-robot collaboration task (Bettoni et al.
(2020)).
The majority of considered contributions investigated
physical fatigue (Aguirre et al. (2021); Hooda et al. (2022);
Maman et al. (2020); Papakostas et al. (2019); Zhang et al.
(2014, 2019); Lambay et al. (2021)), sometimes focusing
on specific muscles (Papakostas et al. (2019)). Recently,
mental fatigue is getting more attention (Hooda et al.
(2022)), even combined with physical fatigue evaluation
(Ramos et al. (2020)). Specifically, mental fatigue can be
 Jože M. Rožanec et al. / IFAC PapersOnLine 56-2 (2023) 7609–7614
classified into active (mental depletion caused by active
engagement in a task), passive (experienced because of
monotonous tasks, causing distractions), and sleep-related
fatigue (Sikander and Anwar (2019)).
However, a commonly agreed definition of fatigue is lacking
in the literature, with fatigue being considered both a bi-
nary (detection problem) and numerical property (predic-
tion problem). The binary definition is the most adopted
one, but a few studies tried to predict fatigue on a three-
level scale (Aguirre et al. (2021)). Approaches to predict
fatigue in larger ranges (e.g., the full Borg CR-10 scale)
is still lacking in the literature, possibly because a wider
range of values increases the task complexity. Besides, it is
hard to collect a balanced dataset with a sufficient amount
of samples for each considered value of the scale, especially
in non-controlled environments. This aspect prevents the
training phase of large machine learning models (as a
matter of fact, the largest dataset among the considered
studies contains only 660 records (Aguirre et al. (2021))),
with some trials conducted in recent years, spreading from
traditional machine learning models (e.g., SVM (Aguirre
et al. (2021); Hooda et al. (2022); Maman et al. (2020);
Papakostas et al. (2019); Ramos et al. (2020); Zhang
et al. (2014, 2019)), Random Forest (Aguirre et al. (2021);
Maman et al. (2020); Papakostas et al. (2019)), K-nearest
neighbors (Aguirre et al. (2021); Hooda et al. (2022);
Maman et al. (2020))), to modern deep learning models
(e.g., neural networks (Aguirre et al. (2021); Hooda et al.
(2022)), Autoencoder (Hooda et al. (2022)), recurrent neu-
ral networks (Lambay et al. (2021))). For supervised mod-
els, most of the conducted experiments preferred to label
collected data by verbally asking operators whether they
feel tired or not; in other cases, some approaches proposed
approaches to infer fatigue from sensors data: Ramos et al.
(2020) proposed the Global Fatigue Descriptor (GFD);
Zhang et al. (2014) used the maximum voluntary isokinetic
exertion (MVE)); various automobile companies rely on
the interaction of the driver with vehicle control (e.g.,
pedals and wheel), or driver’s physical features (e.g., eye
blinking). Of the drowsiness-detection measures available
in the literature, ”PERCLOS” represents the most reliable
and valid determination of alertness level, which reflects
slow eyelid closures (”droops”) rather than blinks (Junaedi
and Akbar (2018)), however, it does not consider individ-
ual differences.
3. USE CASE
The research we present in this paper was performed with
data provided by Philips Consumer Lifestyle BV, from
Drachten, The Netherlands. The manufacturing plant is
considered one of the largest development and produc-
tion centers in Europe Philips has. The use case con-
cerns manual visual inspection of Philips logos printed on
shavers. In particular, Philips Consumer Lifestyle BV has
printing machine setups for various products and logos.
Many products whose logos are printed on these machines
are manually handled and inspected to determine their
visual quality. When a defect is observed, the product is
removed from the manufacturing line, leaving only those
that comply with the quality standards. Operators usually
spend several seconds handling, inspecting, and labeling
the products. This process could be enhanced with artifi-
7611
cial intelligence in multiple ways. First, a machine learning
model could be developed to partially automate defect
identification. The operators would therefore focus only on
those cases where the machine could not determine with
enough confidence that a certain defect exists. For those
cases, artificial intelligence could generate defect hints,
thus providing valuable information on where to look for a
defect for a given piece. Generative models could be used
to mitigate class imbalance and therefore induce a higher
degree of attention from the operators and periodically
validate the quality of their labeling. Finally, we envision
machine learning could be used to predict operators’ fa-
tigue and the consequent labeling quality decay.
The original dataset provided by Philips Consumer Lifestyle
BV has 3.518 images, with three possible categories (see
Fig. 1): good printing, double printing, and interrupted
printing.
Fig. 1. Examples for the three classes present in the
dataset provided by Philips Consumer Lifestyle BV.
The dataset is imbalanced. Most images correspond
to non-defective pieces (good print), while the defec-
tive prints (double or interrupted prints) introduce
different levels of imbalance.
4. METHODOLOGY
For this research, we developed a web application for
defect inspection (see 2). The application required the
participants to log in with a username and password and
execute a series of experiments (labeling 600 images in each
case). The application forced them to execute the experi-
ments sequentially to ensure that the learning path for all
participants would be the same. To that end, the following
experiment was unlocked only when the current was com-
pleted. As a collateral effect, all the participants had an
increased learning process between the single experiments
and the hinting techniques, which we could not mitigate.
For each experiment, we collected data regarding the start
and end time of each experiment, the time required to label
each image, and the label the participants assigned to the
image. Furthermore, we also persisted the participant’s ID
executing the experiments, but no personal information
persisted in the database that could allow associating the
ID to a particular person. Finally, we must highlight that
for each image, we had complementary data to determine
whether it was obtained from the original dataset or gener-
ated with a generative model. Furthermore, for each image,
we had its ground truth label. We detail the experiments
in Table 1. Results on how different hinting techniques
affected the participants’ labeling accuracy were reported
in Rožanec et al. (2022).
We conducted the experiments among two groups of par-
ticipants: four researchers and students from an artificial
intelligence laboratory and three operators working on
manual visual inspection at Philips Consumer Lifestyle
7612 Jože M. Rožanec et al. / IFAC PapersOnLine 56-2 (2023) 7609–7614
Fig. 2. Three screenshots describing the application we de-
veloped to test six labeling scenarios. In the image, we
highlight three different screens: (1) the login screen,
(2) the experiments menu (to choose the next avail-
able experiment), and (3) the particular experimental
setup. Among the experiments, when performing a
labeling task, we always display a good image on the
left side, the image to be inspected is shown in the
center, and on the right, we eventually provide some
defect hinting. In (3), the image corresponds to a
defective item (interrupted print), and the defect hint
was created with the GradCAM technique (Selvaraju
et al., 2017)).
Experiment Balanced dataset? Defect hinting technique
1 NO No defect hinting
2 YES No defect hinting
3 NO DRAEM anomaly map
4 YES DRAEM anomaly map
5 YES GradCAM heatmap
6 YES Nearest labeled image, considering SSIM.
Table 1: Experiments description. SSIM is the acronym for
Structural Similarity Index Measure.
BV. We asked them to execute the experiments daily to
avoid fatigue affecting their performance.
Experiments 1-4 were designed to determine whether the
class imbalance affects the quality of labeling, while Ex-
periments 4-6 aimed to examine whether defect hinting
could enhance the quality of labeling and which hinting
techniques would be most helpful. For this research, we
were interested in the quality of labeling dynamics over
time and whether a consistent decrease in the quality of
labeling could be observed across the experiments. To that
end, we computed precision and recall for each participant
over a rolling window comprehending the last hundred
images. When computing precision and recall, we con-
sidered a binary classification (defective vs. non-defective
images, with defective samples as the class of interest).
We confirmed that decay in precision and recall could be
observed for most experiment participants. Furthermore,
the highest precision and recall were obtained for Experi-
ment 6. We, therefore, chose the precision and recall time
series to develop machine learning models to detect the
participants’ fatigue (see Fig. 3).
When analyzing the precision and recall time series, we
considered that for this research, we would consider the
participant fatigued once their performance drops below
0.75, either considering precision or recall, over a rolling
Participant ID Fatigue point (Image ID)
1 580
2 594
4 242
10 305
11 310
14 133
Table 2: Fatigue points identified for the participants.
Participants who did not meet the fatigue criteria were
excluded from the table.
time window considering the last hundred images labeled
by a given participant within an experiment. In other
scenarios, such thresholds could be adjusted based on the
use case requirements (e.g., defined acceptance quality
levels for each defect). We then checked when such events
(we call them fatigue points) took place and reported them
in Table 2. We also analyzed the time required to label each
image and whether participants required more or less time
to annotate the images. We found no strong signals in this
regard and therefore did not consider labeling time when
creating features for a machine-learning model.
We defined fatigue as a performance drop below a 0.75
threshold for precision or recall and decided to solve the
problem with two regression models: one to predict the
expected precision and one to predict the expected recall.
We did so for two-time horizons: one and thirty images
ahead. We report the features created for each model in
Table 3.
Both models were created with a Linear Regression al-
gorithm, given its simplicity and good results obtained.
We trained the models with the first half of the dataset
(three hundred images) and predicted the second half. Our
decision to do so was mainly guided by the fact that (a)
we were interested in identifying the first fatigue point
for each user, (b) we had only seven participants, and (c)
most fatigue points were located in the second half of the
dataset.
5. RESULTS AND ANALYSIS
We present the results in Table 4 and Table 5. From Table
5 we see that the forecasts’ performance between an image
ahead and thirty images ahead forecast strongly degrades.
Nevertheless, in Table 4 we found that the fatigue points
were predicted accurately. The fatigue points were pre-
dicted perfectly for all relevant cases in a one-image-ahead
forecast. For the thirty images ahead, on the other hand,
a perfect prediction was issued for participant id = 2, and
a prediction close to the target for participants id = 10
and id = 11, predicting a fatigue point three and one
images before the actual event occurrence. We consider
the model failed to accurately predict the fatigue point
for participant id = 1, given it predicted a fatigue point
140 images earlier than the actual event. We, therefore,
consider this prediction a false positive.
6. DISCUSSION AND CONCLUSION
From multiple experiments we performed for data labeling
of manufacturing defects, we observed that most of the
participants’ quality of labeling (measured as precision
and recall) declined over time. Therefore, we hypothesize
 Jože M. Rožanec et al. / IFAC PapersOnLine 56-2 (2023) 7609–7614 7613

Fig. 3. Precision (left) and recall (right) plots for each of the participants. Participants 1-4 correspond to researchers
from an artificial intelligence laboratory, while the rest correspond to operators from Philips Consumer Lifestyle
BV. A detailed analysis of discrepancies between precision and recall among researchers and operators is presented
in Rožanec et al. (2022).
Model Feature Explanation
p-t Precision value at t, computed over the last 100 images. We considered t=0, 1, 2, 3.
Precision
p-t1/p-t2 Ratio between p-t1 and p-t2.
r-t Recall value at t, computed over the last 100 inspected images. We considered t=0, 1, 2, 3.
Recall
r-t1/r-t2 Ratio between r-t1 and r-t2.

Table 3: Features used to predict future precision and recall.

fatigue point Precision Recall
participant id predicted predicted predicted
target target target
1 s.a. 30 s.a. 1 s.a. 30 s.a. 1 s.a. 30 s.a.
1 580 580 440 0,8089 0,8117 0,8033 0,7333 0,7450 0,7398
2 594 594 594 0,8261 0,8241 0,8169 0,7500 0,7445 0,7394
10 305 305 302 0,7439 0,7454 0,7309 0,9385 0,9382 0,9347
11 310 310 309 0,7560 0,7464 0,7270 0,9841 0,9840 0,9821

Table 4: Predicted fatigue points and the corresponding predicted precision and recall. s.a. abbreviates steps ahead,
indicating the forecasting horizon.

One image ahead Thirty images ahead research, we considered the first case when either precision
Precision Recall Precision Recall or recall had a value lower than 0.75, more complex rules
RMSE 0,0474 0,0442 0,2622 0,2465 can be established. For example, we could consider accep-
MAE 0,0057 0,0059 0,1079 0,1049
tance quality levels for each type of defect or establish the
R2 0,8470 0,8450 0,1433 0,1910
extent to which performance dropping below the estab-
lished acceptance quality level is tolerated and produce
Table 5: Predicted RMSE, MAE, and R2 values for both
predictions for multiple horizons. This would allow us to
forecasting horizons: one image ahead and thirty images
determine participants’ fatigue with higher confidence and
ahead.
robustness. Furthermore, while there is a clear decreasing
trend in the operators’ labeling performance, such perfor-
such a decline in the quality of labeling is associated with mance fluctuates. Therefore, this must be considered to
the operators’ fatigue. In this research, we successfully understand how such fluctuations affect the overall visual
predict operators’ performance in one and thirty images inspection performance and what mitigation actions can
in advance. If operators’ fatigue is measured through their be taken to prevent such a performance drop when ap-
labeling performance, they can decide when to stop the proaching lower bounds of acceptable quality thresholds.
labeling activity based on the expected performance. This
could be achieved by generating short streams of labeled, Our future work will be focused on upgrading our exist-
GAN-generated synthetic images on which the labeling ing software to incorporate (a) a machine learning model
accuracy can be measured and the operators’ fatigue de- for eye blinking detection, (b) incorporate information
termined. Nevertheless, including random streams of syn- regarding eye blinking into our fatigue detection model,
thetic data affects the labeling throughput of unlabeled (c) include questionnaires and cognitive tests, to have an
data. Therefore, this approach could complement predic- objective measurement and baseline understanding of the
tive models aiming to predict operators’ fatigue based on participants (e.g., regarding fatigue and cognitive abili-
some sensor data. ties) before labeling, (d) complement the application data
with data obtained from wearable sensors, and (e) use
We have identified several work limitations based on the active learning techniques to improve our fatigue detec-
experience acquired through this research. While for this
7614 Jože M. Rožanec et al. / IFAC PapersOnLine 56-2 (2023) 7609–7614
tion machine learning models. We also plan to conduct
experiments with a larger population in a homogeneous
environment to enhance the robustness of our findings.
ACKNOWLEDGEMENTS
Philips Consumer Lifestyle B.V. provided industrial cases
and data. Credits are due to Jelle Keizer and Paulien Dam
for the use case.
REFERENCES
Aguirre, A., Pinto, M.J., Cifuentes, C.A., Perdomo, O.,
Diaz, C.A., and Munera, M. (2021). Machine learning
approach for fatigue estimation in sit-to-stand exercise.
Sensors, 21(15), 5006.
Bettoni, A., Montini, E., Righi, M., Villani, V., Tsvetanov,
R., Borgia, S., Secchi, C., and Carpanzano, E. (2020).
Mutualistic and adaptive human-machine collaboration
based on machine learning in an injection moulding
manufacturing line. Procedia CIRP, 93, 395–400.
Biondi, F.N., Saberi, B., Graf, F., Cort, J., Pillai, P.,
and Balasingam, B. (2023). Distracted worker: using
pupil size and blink rate to detect cognitive load during
manufacturing tasks. Applied ergonomics, 106, 103867.
Cao, G., Ruan, S., Peng, Y., Huang, S., and Kwok,
N. (2018). Large-complex-surface defect detection
by hybrid gradient threshold segmentation and im-
age registration. IEEE Access, 6, 36235–36246. doi:
10.1109/ACCESS.2018.2842028.
Emmanouilidis, C., Pistofidis, P., Bertoncelj, L., Kat-
souros, V., Fournaris, A., Koulamas, C., Ruiz-Carcel,
C., C., P.P.B.L.K.V.F.A.K.C.E., and Ruiz-Carcel, C.
(2019). Enabling the human in the loop: Linked
data and knowledge in industrial cyber-physical sys-
tems. Annual Reviews in Control, 47, 249–265. doi:
10.1016/j.arcontrol.2019.03.004.
Frustaci, F., Spagnolo, F., Perri, S., Cocorullo, G., and
Corsonello, P. (2022). Robust and high-performance
machine vision system for automatic quality inspec-
tion in assembly processes. Sensors, 22, 2839. doi:
10.3390/s22082839.
Hooda, R., Joshi, V., and Shah, M. (2022). A compre-
hensive review of approaches to detect fatigue using
machine learning techniques. Chronic Diseases and
Translational Medicine, 8(1), 26–35.
Junaedi, S. and Akbar, H. (2018). Driver drowsiness
detection based on face feature and perclos. In Journal
of Physics: Conference Series, volume 1090, 012037.
IOP Publishing.
Lambay, A., Liu, Y., Morgan, P., and Ji, Z. (2021). A data-
driven fatigue prediction using recurrent neural net-
works. In 2021 3rd International Congress on Human-
Computer Interaction, Optimization and Robotic Appli-
cations (HORA), 1–6. IEEE.
Lyytinen, K., Nickerson, J.V., and King, J.L. (2020).
Metahuman systems = humans + machines that learn.
Journal of Information Technology, 0268396220915917.
doi:10.1177/0268396220915917.
Maman, Z.S., Chen, Y.J., Baghdadi, A., Lombardo, S.,
Cavuoto, L.A., and Megahed, F.M. (2020). A data ana-
lytic framework for physical fatigue management using
wearable sensors. Expert Systems with Applications, 155,
113405.
Papakostas, M., Kanal, V., Abujelala, M., Tsiakas, K., and
Makedon, F. (2019). Physical fatigue detection through
emg wearables and subjective user reports: a machine
learning approach towards adaptive rehabilitation. In
Proceedings of the 12th ACM International Conference
on PErvasive Technologies Related to Assistive Environ-
ments, 475–481.
Rajan, A.J., Jayakrishna, K., Vignesh, T., Chandradass,
J., and Kannan, T.T. (2020). Development of computer
vision for inspection of bolt using convolutional neural
network. Materials Today: Proceedings, 45, 6931–6935.
doi:10.1016/j.matpr.2021.01.372.
Ramos, G., Vaz, J.R., Mendonca, G.V., Pezarat-Correia,
P., Rodrigues, J., Alfaras, M., and Gamboa, H. (2020).
Fatigue evaluation through machine learning and a
global fatigue descriptor. Journal of healthcare engi-
neering, 2020.
Rožanec, J.M., Zajec, P., Keizer, J., Trajkova, E., Fortuna,
B., Brecelj, B., Šircelj, B., and Mladenić, D. (2022).
Enhancing manual revision in manufacturing with ai-
based defect hints. In Central European Conference on
Information and Intelligent Systems, 357–363. Faculty
of Organization and Informatics Varazdin.
Saiz, F.A., Alfaro, G., and Barandiaran, I. (2021). An
inspection and classification system for automotive com-
ponent remanufacturing industry based on ensemble
learning. Information 2021, 12(12), 489.
Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R.,
Parikh, D., and Batra, D. (2017). Grad-cam: Visual
explanations from deep networks via gradient-based
localization. In Proceedings of the IEEE international
conference on computer vision, 618–626.
Sikander, G. and Anwar, S. (2018). Driver fatigue detec-
tion systems: A review. IEEE Transactions on Intelli-
gent Transportation Systems, 20(6), 2339–2352.
Sikander, G. and Anwar, S. (2019). Driver fatigue detec-
tion systems: A review. IEEE Transactions on Intel-
ligent Transportation Systems, 20(6), 2339–2352. doi:
10.1109/TITS.2018.2868499.
Tsai, D.M. and Jen, P.H. (2021). Autoencoder-
based anomaly detection for surface defect inspec-
tion. Advanced Engineering Informatics, 48. doi:
10.1016/j.aei.2021.101272.
Yun, J.P., Shin, W.C., Koo, G., Kim, M.S., Lee, C., and
Lee, S.J. (2020). Automated defect inspection system
for metal surfaces based on deep learning and data
augmentation. Journal of Manufacturing Systems, 55,
317–324.
Zhang, J., Lockhart, T.E., and Soangra, R. (2014). Clas-
sifying lower extremity muscle fatigue during walking
using machine learning and inertial sensors. Annals of
biomedical engineering, 42(3), 600–612.
Zhang, L., Diraneyya, M., Ryu, J., Haas, C., and Abdel-
Rahman, E. (2019). Automated monitoring of physi-
cal fatigue using jerk. In ISARC. Proceedings of the
International Symposium on Automation and Robotics
in Construction, volume 36, 989–997. IAARC Publica-
tions.