sensors
Article
Hybrid Virtual Commissioning of a Robotic Manipulator with
Machine Vision Using a Single Controller
Marek Noga * , Martin Juhás






Citation: Noga, M.; Juhás, M.;
Gulan, M. Hybrid Virtual
Commissioning of a Robotic
Manipulator with Machine Vision
using a Single Controller. Sensors
2022, 22, 1621. https://doi.org/
10.3390/s22041621
Academic Editors: Samad
Sepasgozar, Rafiq Ahmad,
Limao Zhang and Sara Shirowzhan
Received: 22 January 2022
Accepted: 17 February 2022
Published: 18 February 2022
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Sensors 2022, 22, 1621. https://doi.org/10.3390/s22041621
and Martin Gulan
Institute of Automation, Measurement and Applied Informatics, Faculty of Mechanical Engineering,
Slovak University of Technology in Bratislava, 812 31 Bratislava, Slovakia; martin.juhas@stuba.sk (M.J.);
martin.gulan@stuba.sk (M.G.)
* Correspondence: marek.noga@stuba.sk
Abstract: Digital twin (DT) is an emerging key technology that enables sophisticated interaction
between physical objects and their virtual replicas, with applications in almost all engineering fields.
Although it has recently gained significant attraction in both industry and academia, so far it has no
unanimously adopted and established definition. One may therefore come across many definitions
of what DT is and how to create it. DT can be designed for an existing process and help us to
improve it. Another possible approach is to create the DT for a brand new device. In this case, it can
reveal how the system would behave in given conditions or when controlled. One of purposes of
a DT is to support the commissioning of devices. So far, recognized and used techniques to make
the commissioning more effective are virtual commissioning and hybrid commissioning. In this
article, we present a concept of hybrid virtual commissioning. This concept aims to point out the
possibility to use real devices already at the stage of virtual commissioning. It is introduced in a
practical case study of a robotic manipulator with machine vision controlled with a programmable
logic controller in a pick-and-place application. This study presents the benefits that stem from the
proposed approach and also details when it is convenient to use it.
Keywords: hybrid virtual commissioning; digital twin; robotics; simulation; machine vision; PLC
1. Introduction
In recent years, automation has undeniably gained a significant role in many engi-
neering fields. The effort to speed up and simplify the processes as much as possible
necessarily implies the requirement to make the time needed for design, development and
commissioning a device as short as possible. It is also similar in the case of designing new
machines or even entire production processes. There is a number of different approaches
and procedures on how to achieve the desired result. The arrival of the COVID-19 pan-
demic, which markedly affected the market and the entire economy, has exposed even
more the advantage of using digital tools to reveal or propose possible solutions to the
impacts of various events [1]. To a great extent, they use the concept of digital twin.
1.1. Digital Twin
In the literature, one can find many definitions of a DT; however, there is none
that is recognized both in academia and across industry sectors [2]. At the same time,
there are many approaches to the creation of a DT, see, e.g., [3–6]. In short, according to
Schluse et al. [7], DT can be understood as a representation of a real object or subobject
with its data, functions and means to communicate in a digital environment. Qin et al. [6]
describe the DT in a similar way and add that DT can also be used for process optimization,
monitoring, diagnostics and predictions based on artificial intelligence (AI), machine learn-
ing and software analysis. DTs are hence used for simulation and assessment of system
or process behavior, based on which it is possible to take appropriate actions. Note also
https://www.mdpi.com/journal/sensors
Sensors
Version2022, 22, 1621
February 14, 2022 submitted to Sensors 22of
of17
17

37 with DT [8]. The use of DTs for manufacturing systems can be aimed at the support of
that, within the Industry 4.0 concept, AI is very often used together with DT [8]. The
38 technical condition analysis for the purpose of improvement and predictive planning of
use of DTs for manufacturing systems can be aimed at the support of technical condition
39 maintenance activities, management and optimization of properties of devices during
analysis for the purpose of improvement and predictive planning of maintenance activities,
40 their lifecycle [9].
management and optimization of performance of devices during their lifecycle [9].
41 Before starting to create a DT, it is necessary to ask questions; in particular whether
Before starting to create a DT, it is necessary to ask questions, in particular, whether it
42 it makes sense and what benefits it would bring. Given our requirements on the model,
makes sense and what benefits it would bring. Given our requirements on the model, is it
43 is it even necessary to create a DT? Would the investment into DT return or would it be
even necessary to create a DT? Would the investment into DT return or would it be lossy?
44 lossy? It may also happen that creation of DT will result in a model that is too complex,
It may also happen that creation of DT will result in a model that is too complex, which
45 which tends to be the case when striving for a DT model identical to the device [10].
tends to be the case when striving for a DT model identical to the device [10].
46 Every simulation requires power and time to perform computations. When de-
Every simulation requires power and time to perform computations. When designing
47 signing a DT it is therefore of utmost importance to consider which parts of the device
a DT, it is therefore of utmost importance to consider which parts of the device or process
48 or process are meaningful to simulate and which in turn would be useless and only lead
are meaningful to simulate and which in turn would be useless and only lead to an increase
49 to increase of computational effort. There is of course an effort to approximate the real
in computational effort. There is of course an effort to approximate the real system as
50 system as good as possible. It is therefore necessary to know what precision to achieve
good as possible. It is therefore necessary to know what precision to achieve already at the
51 already at the design stage.
design stage.
52 All these factors significantly affect the required resources—computational, time,
All these factors significantly affect the required resources—computational, time,
53 financial, and last but not least, human.
financial and last but not least, human.
54 Although there is no uniform definition of a digital twin, currently an international
Although there is no uniform definition of a digital twin, an international standard
55 standard ISO 23247 is currently published. This standard defines a framework to sup-
ISO 23247 is currently published. This standard defines a framework to support the creation
56 port the creation of digital twins of observable manufacturing elements including per-
of digital twins of observable manufacturing elements including personnel, equipment,
57 sonnel, equipment, materials, manufacturing processes, facilities, environment, prod-
materials, manufacturing processes, facilities, environment, products and supporting
58 ucts, and supporting documents. It will consist of four parts [11]:
documents. It consists of four parts [11]:
59 • ISO 23247-1: General principles and requirements for developing digital twins in
• ISO 23247-1: General principles and requirements for developing digital twins
60 manufacturing;
in manufacturing;
61 • ISO 23247-2: Reference architecture with functional views;
• ISO 23247-2: Reference architecture with functional views;
62 • ISO 23247-3: List of basic information attributes for the observable manufacturing
• ISO 23247-3: List of basic information attributes for the observable manufacturing elements;
63 elements;
• ISO 23247-4: Technical requirements for information exchange between entities within
64 • ISO 23247-4: Technical requirements for information exchange between entities
the reference architecture.
65 within the reference architecture.
The main principle of creating a DT according to ISO 23247 is illustrated in Figure 1.
66 The main principle of creating a DT according to ISO 23247 is illustrated in Figure 1.

Observe
Collect Data Model – Digital Twin Learn and Act
Manufacturing Elements

Figure 1. Principle of creating a digital twin according to ISO 23247.

Figure 1. Principle of creating a digital twin according to ISO 23247.
67 In the first step the standard assumes observing the device or process for which a
68 In theisfirst
model to bestep, the standard
created. Based on assumes observing
this it shall the device
be determined which orsignals
processand
fordata
which a
shall
model
69 is to be created.
be obtained from the Based on this,
process and itfurther
shall be determined
integrated intowhich
the DTsignals and
model. databased
Next, shall be
on
obtained
70 from we
this model the shall
process andhow
decide further integrated
to improve theinto the DT
process model.
itself, Next,actions
i.e. what based shall
on this
be
model
71 we shall decide how to improve the process
taken in order to improve its behavior or performance. itself, i.e., what actions shall be taken in
order to We
72 improve its behavior
refer the interested orreader
performance.
to [12] for a demonstration of various use case sce-
73 We refer the interested reader
narios for digital twin implementation to [12] for a demonstration
based on ISO 23247. of various use case scenarios
for
74 digital twin implementation based on ISO 23247.
Another possible approach is described in [13], according to which a DT shall con-
75 Another
sist possible approach
of the following four parts:is described in [13], according to which a DT shall consist
of the following four parts:
76 • real space;
• •Realvirtual
77 space;space;
• •Virtual
78 space;
the data link from real space to virtual space;
• •Theand
79 datainformation
link from real linkspace
fromto virtual
virtual space;
space to real space and virtual sub-spaces.
• In addition, an information link from virtual space to real space and virtual sub-spaces.
80 Note that the digital representation of the product can be built also from multiple parts.
Note
81 This that the digital
approach allows representation of a physical
to create a detailed entity can
physics-based be also built
simulation modelfrom
formultiple
the pur-
parts. This approach allows
82 pose of process analysis [13]. to create a detailed physics-based simulation model for the
purpose of process analysis [13].
Version February
Sensors 14,1621
2022, 22, 2022 submitted to Sensors 3 of317of 17

83 1.2. Virtual Commissioning

1.2. Virtual Commissioning
84 Digital twin is often associated with the concept of virtual commissioning (VC). It
Digital twin is often associated with the concept of virtual commissioning (VC). It de-
85 denotes
notes an approach
an approach when when virtual
virtual prototypes
prototypes of devices
of devices areto
are used used to validate
validate function-
functionality
86 ality of the program as well as the mechanics [14]. The employment of
of the program as well as the mechanics [14]. The employment of virtual commissioning virtual commis-
87 sioning brings many other advantages that include e.g. avoiding the risk
brings many other advantages that include, e.g., avoiding the risk of damaging the device of damaging
88 the device
during during testing
commissioning whilewhile ensuring
ensuring safetysafety
of the of the personnel.
personnel.
89 Virtual
Virtual commissioning
commissioning typically
typically builds
builds on twoon main
two main approaches:
approaches: software
software in the
in the loop
90 loopand
(SIL) (SIL) and hardware
hardware in the
in the loop loopThese
(HIL). (HIL).are
There are followed
followed by hybridbycommissioning
hybrid commissioning
(HC),
91 (HC),eventually
which which eventually leads
leads to real to real commissioning,
commissioning, as illustrated
as illustrated in Figure 2.in Figure 2.

Virtual commissioning

Software Hardware Real

in the loop Hybrid commissioning commissioning
in the loop

Figure 2. Variants of virtual commissioning. Reproduced with permission from authors of [15].
Figure 2. Variants of virtual commissioning. Reproduced with permission from authors of [15].
92 Within the SIL approach, in addition to the simulated process control hardware is
Within the SIL approach, in addition to the simulated process, control hardware is
93 emulated as well. It is conveniently used in both testing and design phase, since not a
emulated as well. It is conveniently used in both testing and design phase, since not a
94 single piece of hardware is required and programming of software is accelerated [16].
single piece of hardware is required, and programming of software is accelerated [16]. It is
95 It is thus a purely off-line analysis where all hardware components are simulated.
thus a purely off-line analysis where all hardware components are simulated.
96 Withinthe
Within the
HILHIL approach,
approach, realreal control
control hardware
hardware is connected
is connected to thetoreal-time
the real-time
simula-sim-
97 ulation of the virtual devices. This enables to test even complex automation
tion of the virtual devices. This enables to test even complex automation tasks with the tasks with
98 the control
control systemsystem
that is that
to beisutilized
to be utilized in real commissioning.
in real commissioning. The HIL simulation
The HIL simulation can be usedcan
99 be used at different levels of
at different levels of production [14]. production [14].
100 Hybridcommissioning
Hybrid commissioning proposed
proposed in [17]
in [17] represents
represents an incremental
an incremental procedure,
procedure, whichwhich
101 startsasasHIL
starts HIL and
and stepwise
stepwise replaces
replaces virtual
virtual devices
devices by realbydevices
real devices andleading
and thus thus leading
to real to
102 real commissioning.
commissioning. In [17], In
the[17] the authors
authors introduced introduced the HC by
the HC approach approach
means of byanmeans
example of an
103 ofexample
a PROFIBUS driven production
of a PROFIBUS drivenplant where they
production plantcompare
where and
theycombine
compare theand
signal valuesthe
combine
104 ofsignal
real and simulated
values of realcomponents.
and simulated components.

105 1.3.
1.3.Motivation
Motivation
106
Let
Letususnow
nowrecall
recallthetheISO
ISO23247
23247standard
standard and
and itsits
implementation
implementation presented in [12]
presented in [12]
107
and demonstrated by means of three use case scenarios. The first two,
and demonstrated by means of three use case scenarios. The first two, “Machine Health “Machine Health
108
Digital
DigitalTwin”
Twin” and
and“Scheduling
“Scheduling andand
Routing
RoutingDigital Twin”,
Digital run above
Twin” a real-world
run above processpro-
a real-world
or machine and focus more on their optimization or Product Lifecycle Management (PLM).
109 cess or machine, and focus more on their optimization or Product Lifecycle Manage-
It is however the third use case scenario, “Virtual Commissioning Digital Twin”, which
110 ment (PLM). It is however the third use case scenario, “Virtual Commissioning Digital
is the most interesting from the perspective of this study. It differs by not having a real-
111 Twin”, which is the most interesting from the perspective of this study. It differs by not
time connection to the device. Specifically, it describes the use of VC for a CNC machine
112 having
with a real-time connection
a programmable to the(PLC)
logic controller device. Specifically,
which it describes
is not physically the useyet,
constructed of VC
but for
113 a CNC machine with a Programmable Logic Controller (PLC)
it is possible to perform its VC for various scenarios using a DT. The use case examplewhich is not physically
114 constructed
assumes the HILyet, but it isfor
approach possible to perform
this purpose. its VC for
It also presents various scenarios
possibilities using a DT.
of using simulation
115 The use casetoexample
environments assumes the
acquire appropriate HIL approach
signals for testing for this purpose.
or verification and It also presents
validation (V&V). pos-
116 sibilities
The authors ofnote
using
thatsimulation environments
after performing to acquire
all the tests, appropriate
it is possible signals
to deploy such for testing or
a program
117 verification
on and validation (V&V). The authors note that after performing all the tests it
the real device.
118 is possible to
Verificationdeploy such a program
and validation on the real
of simulation modelsdevice.
represent an integral part of the
119 Verification
development and The
process. validation
model, of
ergosimulation
the DT, ismodels
created represent an integral
for a specified purpose. part of the
Even
120 before its design,
development it is therefore
process. necessary
The model, ergo totheformulate questions
DT is created for athat the system
specified will be
purpose. Even
121 asked
beforeorits
tasked to solve
design and that the
it is therefore model has
necessary to be able questions
to formulate to answer. that the system will be
122 asked or tasked to solve and that the model has to be able to answer.
123 In [18] the author discusses different approaches to V&V of simulation models. He
124 also points out the fact that despite a model can be valid for various uses, it does not
 Sensors 2022, 22, 1621 4 of 17

ersion February 14, 2022 submitted to Sensors 4 of 17

In [18], the author discusses different approaches to V&V of simulation models. He

also points out the fact that, despite that a model can be valid for various uses, it does not
125 guarantee
guarantee itsits
validity
validityforforuse
usein
in all
all applications
applications ininaagiven
givenfield.
field.Figure
Figure 3 illustrates
3 illustrates the the
126 relationship between value and cost of model and its confidence. It implies
relationship between value and cost of a model and its confidence. It implies that, beyond that beyond
127 a required
a required model
modelconfidence
confidence, itsits value
valuetotothe
user
userstarts
startstoto decrease rapidly,
decrease rapidly, while
while thethe
timetime
128 required
requiredto to
create
createmodel has ahas
the model dependence
a dependence similar
similartotothe
theoverall
overall value
value ofof the
model.
model.Note
129 thatNote
the that
costthe
of cost
modelof the model
tends tends
to be to be
quite quite significant,
significant, especially
especially whenwhen a verya very
highhigh
model
130 model confidence
confidence is required.
is required. It is given
It is given by a by
suma sum of incurred
of all all incurred costs,from
costs, from development
development to
131
to V&V.
V&V.
Value of model

Overall value
Value to user
Cost of model

0% required 100 %
Model confidence
Figure 3. Illustration of relationship
Figure 3. Illustrationbetween valuebetween
of relationship / cost of the model
value/cost of and its confidence.
the model and its confidence.

132 A particular
A particularform
form ofof V&V
V&V isisessentially
essentially thethe
VCVC approach,
approach, which which enables
enables to shorten
to shorten the
133 the time needed for programming and to resolve various software errors—and in con-
time needed for programming and to resolve various software errors—and in conjunction
134 with a with
junction suitable simulation
a suitable environment,
simulation also construction
environment defects. Thedefects.
also construction connectionTheofconnec-
VC
135 and DT offers an effective perspective on the product lifecycle. VC,
tion of VC and DT offers an effective perspective on the product lifecycle. VC howeverhowever, also entails
136
somealso
entails drawbacks. In particular,
some drawbacks. In it is to a great
particular, extent
it is affected
to a great by accuracy
extent affectedofbythe model. of
accuracy
Low accuracy can cause grave problems during commissioning and thus increase both its
137 the model. Low accuracy can cause grave problems during commissioning and thus in-
cost and required time. Hence, although VC can significantly accelerate the development
138 crease bothitits
process, may cost and required
happen time.
that its cost willHence,
be too although VC can
high, especially in significantly accelerate
cases of increasingly
139 thecomplex
development
devices.process, it may happen that its cost will be too high, especially in case
140 of increasingly
How would complex
it thus devices.
be possible to lower the cost while maintaining the advantages of
141 How
VC? Onewould it be thus
of the options is topossible to lower
save on the number the
of cost
used while maintaining
software, and thus onthe
theadvantages
number
142 of VC? One of
of licenses as the
welloptions is toprogrammers.
as necessary save on the number of usedis software,
Another option to employ and thus on the
our proposed
143 concept
number ofof hybrid as
licenses virtual
wellcommissioning (HVC).
as necessary programmers. Another option is to employ our
144 proposed concept of Hybrid Virtual Commissioning (HVC).
2. Hybrid Virtual Commissioning
145 HVC
2. Hybrid represents
Virtual an approach combining VC with features of HC. Its main idea is
Commissioning
to reduce the cost of simulation by using the available equipment, the cost of which can
146
beHVC
lowerrepresents an approach
than the license fees, andcombining
to simulate VConlywith features
the parts of HC.
that are Its main idea
less affordable or is
147 to reduce
currently unavailable. Use of this approach makes sense if the utilized devices or parts are can
the cost of simulation by using the available equipment, the cost of which
148 be so
lower thanthat
complex theitlicense
is more fees, and totosimulate
convenient only
directly use thethan
them parts that arethem.
to simulate less affordable
Such a
149 or solution
currently unavailable.
then requires evenUse less of this
time forapproach makes
commissioning sense
since if the
it allows forutilized devices or
a more accurate
150 testing
parts are by
so using DT, which
complex, that itfurther
is moreincreases the likelihood
convenient of detecting
to directly use them defects.
than to simulate
151 In comparison with VC, HVC does not require a purely virtual
them. Such a solution then requires even less time for commissioning since model but combines
it allowsit for
152
with accurate
a more available devices
testingwhich
by usingdo not
DT,need to befurther
which modeled or simulated.
increases In comparison
the likelihood with
of detecting
HC (as described in [19]), HVC does not assume a successive transition from VC to HC but
153 defects.
154 In comparison with VC, HVC does not require a purely virtual model, but com-
155 bines it with available devices which do not need to be modeled or simulated. In com-
156 parison with HC (as described in [19]), HVC does not assume a successive transition
 Sensors 2022, 22, 1621 5 of 17

a direct use of components already in the VC phase. Similar to VC, HVC enables to employ
both the SIL and the HIL approaches. Likewise, it can be followed by application of HVC,
which will also require less time.
Figure 4 shows a flowchart for deciding which commissioning approach to choose.
The feasibility of modeling can be understood as a software limitation, i.e., absence of
Version February 14, 2022 submitted tolicense
Sensorsand other. At this step, it is necessary to consider the cost factor. Unless software
limitations are an issue, efficiency of modeling needs to be taken into account as well.

Start

Select
subsystem

Modeling No
feasible?

Yes

Modeling No
effective?

Yes

No All subsystems
checked

Yes

Use VC Use HVC

approach approach

End

4. Flowchart
Figure Figure of the
4. Flowchart proposed
of the proposedcommissioning approach
commissioning approach including
including HVC.HVC.

160 Figure 4 shows a flowchart for deciding which commissioning approach to c

161 The feasibility of modeling can be understood as a software limitation, i.e. absen
162 license and other. At this step it is necessary to consider the cost factor. Unless sof
163 limitations are an issue, efficiency of modeling needs to be taken into account as
Sensors 2022, 22, 1621 6 of 17

Feasibility involves the overall complexity of modeling and therefore poses a question
as to what amount of resources is required to model of a given element. The higher the
complexity, the more demanding the feasibility—which, in most cases, implies more time
needed for modeling and increase in costs, either of labor or hardware/software equipment
used for modeling. During the entire decision-making process, one has to keep in mind that
a model for DT is concerned; therefore, its repeatable use for future purposes needs to be
considered as well. If it turns out that it is efficient to use at least one physical device (except
for the control element, which would imply only a change from SIL to HIL simulation) in a
subsystem, we no longer deal with VC but HVC.
It may seem that this concept does not fulfill the requirements for DT. If we, however,
recall the definition of a DT, e.g., “A digital twin is a virtual representation of a physical
object or system across its lifecycle, using real-time data to enable understanding, learning
and reasoning” [20], it holds that also by using this approach it is possible to gather from
DT data about behavior of the system during its commissioning but also at a later stage
of lifecycle.
Our concept will be presented in more detail in a case study using a delta robot with
machine vision and PLC control. Delta robots belong to a class of robotic manipulators
with fast dynamics and are therefore suitable for tasks where their speed can be benefited
from [21]. The most common in practice are pick-and-place applications where the delta
robots replace the mostly monotonous manual labor. A pick-and-place task typically
consists of handling of objects transported on a conveyor belt, while the necessary position
information necessary for their gripping is acquired by a machine vision system. Note that
particular systems need to be properly interconnected, while required resources will differ
depending on used technologies.
Our device, situated in the authors’ “Learning factory for Industry 4.0” [22], combines
the proposed solutions to reduce costs by employing HVC as well as by using less soft-
ware environments. Validation of our model is performed by means of animation of DT.
According to [23], it is possible to use the tracing method for the purpose of V&V. The
author describes this method as a dynamical representation of the simulated system. In
order to use the tracing, it is however assumed that the creators are well familiar with the
real-world system the model corresponds to. In such a case, they shall be able to reveal
errors in the control program.
Figure 5 illustrates the proposed conceptual design of our device that will be subject
to hybrid virtual commissioning. The reference task represents a modern use of robot
manipulator in a pick-and-place application in combination with machine vision. The
objective of the robot is to pick randomly positioned objects and to sort them into containers
according to their type. The system for object recognition that determines the type and
position of parts is realized by a smart camera. A fully commissioned solution requires
physical availability of all of its elements—mechanics of the manipulator, gripper, work
space with the parts and sort containers, servo drives, control system, as well as the machine
vision system. This raises the price of possible mechanical adaptations, optimization of
robot arms and way of gripping the parts, as well as hinders parallel commissioning of
software during development and production work on the mechanics. It is therefore logical
to employ the virtual commissioning approaches. Modeling of the camera system intended
for this device would be hardly feasible and, given its availability, also ineffective. The HVC
concept seems optimal in this application since it enables to mirror the physical assembly
of parts scanned by the real camera into the virtual world of the digital twin, which leads
to a full realization of the environment for the pick-and-place application, i.e., robot with
gripper, containers and sorted parts. As it can be observed, the purpose of the device is to
scan the position of objects in real time and to subsequently convert them into digital space,
where virtual commissioning of the delta robot will take place. Its task is to sort the objects
based on their position data. To achieve an effective and reliable control, we propose to
create a digital twin of the delta manipulator.
 187 Our device, situated in the authors’ “Learning factory for Industry 4.0” [22], com-
188 bines the proposed solutions to reduce costs by employing HVC as well as by using less
189 software environments. Validation of our model is performed by means of animation
190 of DT. According to [23] it is possible to use the tracing method for the purpose of V&V.
191 The author describes this method as a dynamical representation of the simulated sys-
Sensors 2022, 22, 1621 192 tem. In order to use the tracing it is however assumed that the creators are well
7 of familiar
17
193 with the real-world system the model corresponds to. In such case they shall be able to
194 reveal errors in the control program.

PLC HMI

Smart IOs
camera

Physical Delta NX MCD

parts picker robot

Virtual
parts

Figure 5. Conceptual design of the delta robot workplace used in our case study.
Figure 5. Conceptual design of the delta robot workplace used in our case study.
195 Figure 5 illustrates the proposed conceptual design of our device that will be sub-
2.1. Used Components
196 ject to hybrid virtual commissioning. The reference task represents a modern use of
The197computer providing in
robot manipulator necessary power for
a pick-and-place running in
application thecombination
simulationwith environment,
machine vision.
the programming
198 environment and the OPC UA server features a 3.1 GHz
Objective of the robot is to pick randomly positioned objects and to sort them i7-8705G CPUinto con-
and 32 GB199 of RAM.according to their type. The system for object recognition that determines the
tainers
For200thetype
purpose of controlling
and position of partsthe delta manipulator,
is realized we chose
by a smart camera. a Siemens
A fully PLC from
commissioned solution
the S7-1500TF
201 series,physical
requires which provides
availability sufficient computing
of all of its power for control
elements—mechanics of multiple grip-
of the manipulator,
technology per, work
202 objects spaceEach
(TOs). withversion
the partsofand
the sort containers,
technology servo
series drives,
offers control
a limited system,ofas well
number
technology as the machine
203 sources. These vision
can besystem.
combined Thisin raises the price
various ways,ofwhich
possible mechanical
allows adjustments,
to simultane-
ously control
204 optimization
a larger numberof robot armsdifferent
of even and wayTOs. of gripping
The PLC theenables
parts, asto well
also as hinders
create parallel
direct
or inverse commissioning
205 kinematics of software
calculations duringor
of motion development
to use someand production
of the predesignedworkkinematic
on the mechan-
TOs. One 206 of ics.
themIt is
is delta
therefore logical to and
manipulator, employ the virtual
calculations forcommissioning
its TO are based approaches. Modeling
on the inverse
of the camera system intended for this device would be hardly
kinematics problem. This PLC can also process a safety program for the automation tasks
207 feasible and given its
and allows to create an OPC UA server. In our case, OPC UA server SIMATIC Net is used
for communication between the controller and the real-time simulation environment. In
Section 2.4, we will discuss how the choice of server affects the real-time simulation itself.
Programming of PLC is performed by means of the TIA Portal platform. Specifically,
the project is programmed in version 15.1. This work is greatly simplified by a good
availability of libraries and examples that can be used in one’s own program. This environ-
ment also enables to program a Human–Machine Interface (HMI) which allows effective
operation and control of the device and displays information about its state.
For the input-output (IO) signals, a distributed IO system SIMATIC ET 200SP is
utilized. It is connected via Profinet network to the PLC where IO signals are processed.
Information about the position of physical parts is acquired from a smart camera,
namely SIMATIC MV540 by Siemens. The camera is capable of recognizing digital codes as
well as objects. It uses the Profinet protocol for data communication. Its configuration via
web interface, specifically Web Based Management (WBM), and integration into TIA Portal
facilitate the programming.
In order to create a DT of the manipulator, we chose the Siemens NX Mechatronic
Concept Designer (MCD) environment which enables real-time simulations.
After HVC, the control is applied on the physical device. The controller determines
the required angular displacement of particular drives so that the effector achieves the
desired position within the manipulator’s coordinate system. The angles are calculated
within the TO in PLC and sent to corresponding frequency converters SINAMICS S210 by
Siemens. The PLC also performs the safety part of the program over every drive. All safety
functions achieve Safety Integrity Level (SIL) 2.
Mechanics of our delta picker robot, designated D4-500-S010 Demo, is manufactured
by Codian Robotics. As its working tool, we assume the Bernoulli gripper OGGB by Festo.
Sensors 2022, 22, 1621 8 of 17

2.2. Creating Digital Twin in NX MCD

NX MCD is an extension of the CAD software Siemens NX that, among other features,
enables to simulate physical behavior of the device. The MCD environment allows to
specify which parts will be considered as dynamic objects with physical properties and
which will be kept as static objects. This specification plays an essential role in terms of
realizing to which extent it is necessary to define the model. As pointed out in the previous
section, overdetermination of model would increase computational complexity and hence
the length of the operating cycle. In case of devices with fast dynamics, which include also
delta manipulators, it is necessary to keep this time as short as possible. That is why we
defined only the objects of arms and working tool of the delta robot. For CAD models, the
program determines basic parameters such as mass, center of gravity, etc. These can also be
set or modified manually, which allows the designers to test various settings. It was also
necessary to consider what types of joints are suitable for particular connections. Note that
the joints significantly affect the computational complexity.
Signals have been defined as well. They are fed to specific elements of the model by
means of UPC UA protocol. There are also functions which enable to calculate some values
directly in the software environment, e.g., angular displacement of the motors or position
based on data form PLC. In addition, we created signals that are acquired from model and
passed to PLC. They substitute the signals from sensors based on events that shall occur for
its activation.

2.3. Creating Control Program for the Device

In this subsection, we will discuss the advantages of using a single programming
environment for designing the entire device. The program for the device will be divided
into several parts and for each, these advantages will be pointed out.

2.3.1. Smart Camera

In today’s industry practice, one can encounter a variety of camera types. They are
used for object recognition, quality control, as well as position determination—which is also
assumed in our case study. Similarly, cameras can be programmed in various languages.
The acquired data can be subsequently utilized by other devices, such as in our application.
In that case, it is however required that they are all well mastered by a single programmer
or that more programmers cooperate. This implies certain mutual dependencies which
may lead to undesired delays.
Our camera allows for a simple interface with PLC. Although the program for the
camera is created via WBM, it concerns a very intuitive and undemanding programming.
Moreover, once the program is created in this environment, it is no longer necessary to use
it unless requirements for camera shooting need to be configured, e.g., adding the type of
scanned object, etc. All other operations are carried out directly in PLC. After entering the
required parameters, these functions also take care of synchronization of the coordinate
systems of camera and robot.
The objects that are to be recognized by the camera are, in reality, printed on a paper
that is scanned. This allows to easily alter the scanned scenes. Figure 6a illustrates the
object recognition by the camera system. Figure 6b depicts the working surface of the
manipulator in the NX MCD environment where a virtual image of the detected objects is
created. Therein, the picked-and-placed parts also inherit their physical properties.
Sensors 2022, 22, 1621 9 of 17

(a) (b)

Figure 6. Object recognition using smart camera. (a) Detection of parts. (b) Visualization of detected
parts in NX MCD.

2.3.2. Implementation of Kinematic System for Delta Manipulator in PLC

As outlined in Section 2.1, kinematic calculations for the manipulator were carried
out within the PLC program. In order to process the TOs, the TIA Portal environment
uses motion control (MC) functions. When using an arbitrary TO, organization blocks
(OBs) are automatically inserted into the project. These have their own execution level
and are called within the MC application cycle. It concerns the MC-SERVO (OB91) and
MC-Interpolator (OB92) blocks, which are know-how protected and cannot be unmasked
or edited. The calculations are performed in the MC-SERVO OB, which is executed before
the MC-Interpolator OB that evaluates MC instructions, generates reference values and
monitors functionalities [24].
Among the available TOs within MC, one may choose different object types, while two
of them are particularly important for this work. The first object type relates to selecting
the function of an axis, which may be set as a speed, positioning or synchronous one. The
axis TOs can be used independently to control a single actuator, e.g., an asynchronous
motor, or they can be used for a kinematic object, which is the second object type essential
for this work. The axis TOs can be simulated directly in the environment, which does not
necessitate a connection to the real device.
The kinematic objects, as the name implies, relate to kinematic systems. It is possible
to choose from a range of predefined objects and thus to create a custom kinematic system.
The predefined systems use preprogrammed kinematic transformations, which only need
to be assigned geometric parameters and axis TOs. On the other hand, custom systems
require to program kinematic transformations beforehand. A great advantage is the option
to employ virtual commissioning directly in the TIA Portal environment. The predefined
systems also include the kinematic system of a delta manipulator.

2.3.3. Used Libraries

The Siemens company offers strong software support with its products. Within
its Industry Online Support, it provides many libraries for various tasks together with
sample application problems. This facilitates the work of programmer and shortens the
required time.
To create the program for our device, we used the following libraries and
sample applications:
• LKinCtrl [25]—a library that facilitates the work with TOs by providing functions
for control of an entire kinematic system and for execution of path motion, as well
Sensors 2022, 22, 1621 10 of 17

as sample applications, one of them being a pick-and-place application for a delta

manipulator. We used some of its functions and modified them for our purposes.
• Virtual commissioning for kinematics in NX MCD with Software in the Loop [26]—
demonstrates how to interface NX MCD with PLC by providing several sample appli-
cations, including one for the delta manipulator from which we adopted functions to
process data fed to the simulation.
• Transformation of MV440 camera coordinates into robot coordinates [27]—demonstrates
how to use the smart camera by describing its integration, the identification procedure
and functions for its control. We adopted a function from this application that takes
care of the entire communication between PLC, camera and interface for position
data from the camera. Other functions are responsible for controlling the camera and
conversion between the coordinate systems of the camera and the robot.
• LDrvSafe [28]—a library that provides fail-safe blocks to implement various safety
applications. We used it for safety when transitioning to hybrid commissioning.

2.4. Communication Between Model and Controller Using PLC OPC UA Server
Communication between the PLC and the model was handled by OPC UA, a protocol
based on server–client framework. The server provides access to data and functions which
are object oriented. The client can access the server via a line with various security levels.
As mentioned in the previous section, the PLC that we employed allows to create an
OPC UA server. The initial idea was to use the PLC as the server. Such a server can be
assigned a sampling interval. Its minimum value for the given PLC can be set as 100 ms.
As shown in Figure 7, PLC contains data blocks which are used by NX MCD to read or log
data by means of so-called signal adapters. Particular signals need to be mapped in NX
Version February 14, 2022MCD. This
submitted is facilitated by the possibility of automatic mapping when signals with the
to Sensors 10 of 1
same name in signal adapters and external signals are automatically connected.

PC

PLC NX MCD
OPC UA server OPC client

OPC UA
IODataMCD SA I_O_Signals

Positions SA PartsPositions

MCDInterface SA Kinematics

Figure 7. Communication structure between PLC OPC UA server and NX MCD (DB – data block, SA – signal adapter.)
Figure 7. Communication structure between PLC OPC UA server and NX MCD (DB—data block,
SA—signal adapter.)
343 face for position data from camera. Other functions are responsible for controllin
When
344 connecting NX MCD
the camera and to the OPC UA
conversion server,
between it coordinate
the is possible to set an update
systems time, and th
of the camera
i.e., how345often the server
robot. data are to be read or logged. As it would not be meaningful to
make it346shorter
• than the OPC
LDrvSafe [28]UA– asampling time,
library that they were
provides set as
fail-safe same.to implement various safet
blocks
There
347 was however a fundamental
applications. We used it forproblem with the
safety when communication,
transitioning to hybridwhere the
commissioning.
communication speed was not sufficient. We will discuss it in more detail in Section 3.1.
2.4. SIMATIC
It led to348using Communication Betweenwhich
PC Station, Modelisand Controllercomponent
a software using PLC OPC that UA Server the
manages
SIMATIC 349 software products andbetween
Communication interfaces
theonPLC
a PC.
andItthe
was used was
model to set up an by
handled OPCOPCUAUA, a pro
NET server
350 that allowed us to achieve a much lower cycle time, which we
tocol based on server-client framework. The server provides access to data chose as 20 ms.and func
The impact
351 of cycle time on real-time simulation will be discussed later as well. Note
tions which are object-oriented. The client can access the server via a line with variouthat
352 security levels.
353 As mentioned in the previous section, the PLC that we employed allows to creat
354 an OPC UA server. The initial idea was to use the PLC as server. Such a server can b
355 assigned a sampling interval. Its minimum value for the given PLC can be set as 100 m
356 As shown in Figure 7, PLC contains data blocks which are used by NX MCD to read o
Sensors 2022, 22, 1621 11 of 17

thetochange
Version February 14, 2022 submitted of server also caused the structure of communication between the
Sensors OPC
11 of 17 UA
server and the client to change. This structure is shown in Figure 8.

PC

SIMATIC PC Station NX MCD

PLC OPC UA server OPC UA client

Profinet OPC UA Symbols OPC UA

IODataMCD SA I_O_Signals

Positions SA PartsPositions

MCDInterface SA Kinematics

Figure 8. Communication structure between PLC, PC Station and NX MCD.

Figure 8. Communication structure between PLC, PC Station and NX MCD.

ResultsFigure
3.377 9 depicts the model of our device designed in NX MCD. In addition to the
and Discussion
378 mechanics itself and the manipulated parts, this DT does not contain a model of the
379
In this section, we present our findings resulting from testing of the DT of our delta
camera system, which from our perspective has no impact on the mechanics and is not
manipulator
380 simulated and application
in any way. of the proposed HVC approach. Within the HVC phase of
our case study, we simulate everything except for the camera system and the controller. As
outlined in previous sections, all physical devices are simulated either in TIA Portal or NX
MCD environment.
Figure 9 depicts the model of our device designed in NX MCD. In addition to the mechan-
ics itself and the manipulated parts, this DT does not contain a model of the camera system,
which from our perspective has no impact on the mechanics and is not simulated in any way.

Figure 9. Digital twin of the delta robot manipulator designed in NX MCD environment.

Figure 9. Digital twin of the delta robot manipulator designed in NX MCD environment.
Sensors 2022, 22, 1621 12 of 17

As we already pointed out, the smart camera that we used allows for easy WBM
programming. Importing the library into the control program enabled all the following
work in the TIA Portal environment. For using the camera system, it was necessary to
synchronize the coordinate systems of robot and camera. Note that the camera’s connection
to the system and information obtained from [27] made it easier to tackle this problem. This
turned out to be convenient within the next step in which we applied the created program
on real mechanics of the delta manipulator. The positions of scanned objects are written
to the data blocks which are used for control of the delta robot manipulator. Via the OPC
UA server, these data were simultaneously sent to NX MCD, where initial positions of the
picked-and-placed parts were set according to scanned images; recall Figure 6.

3.1. Hybrid Virtual Commissioning of Robotic Manipulator

The call structure of the PLC program for this testing is depicted in Figure 10. Com-
munication between PLC and NX MCD was taken care of by the PLC OPC UA server as
illustrated in Figure 7. To work with the camera, it was necessary to create a program for
calibration as well as a program for object recognition. Its objective is to set the position of
parts according to the scans obtained from the camera system and, subsequently, to control
the manipulator so as to pick and stack the parts in a chosen position. To make the control
even more user-friendly, we have also created an HMI which allows us to switch between
Version February 14, 2022 submitted to Sensors 12 of 17
the programs for the camera as well as to track the evaluated scans.

Main AxisControl LKinCtrl_MC_GroupPower

CameraPositionHandler LKinCtrl_MC_GroupReset

PickAndPlaceCamera LKinCtrl_MC_GroupHome

DeltaPick LKinCtrl_MC_JogFrame

MV

MVControl MV540

ScalingFB

Controls HMI

IODataMCD

Positions KinemAxis1 MCPostServo

PathData KinemAxis2 MCDInterfaceHandler

Servo DeltaPicker3D KinemAxis3 MCDInterface

Figure 10. Call structure of the PLC program using HVC. Note: OB – organization block, FB – function block, DB – data block.
Figure 10. Call structure of the PLC program using HVC. Note: OB—organization block,
FB—function block, DB—data block.
381 As we already pointed out, the smart camera that we used allows for easy WBM
382 programming.
By testing, weImporting
validatedthe
thelibrary into
transfer ofthe control
data program
for signal enabled
adapters ofall the following
position control,
383 work in the TIA Portal environment. For using the camera system it was necessary to
which simulate angular displacement of manipulator’s axes. Next, by comparing simula-
384 synchronize the coordinate systems of robot and camera. Note that camera’s connection
tion of the manipulator in NX MCD with its tracking using Kinematics trace in TIA Portal,
385 to the system and information obtained from [27] made it easier to tackle this problem.
we validated synchronization of the coordinate systems.
386 This turned out to be convenient within the next step in which we applied the created
387 program on real mechanics of the delta manipulator. The positions of scanned objects
388 are written to the data blocks which are used for control of the delta robot manipulator.
389 Via OPC UA server these data were simultaneously sent to NX MCD, where initial posi-
390 tions of the picked-and-placed parts were set according to scanned images; recall Figure
391 6.
Sensors 2022, 22, 1621 13 of 17
Version February 14, 2022 submitted to Sensors 13 of 17

By means of the DT, we were able to test functionalities, communication between

405 By means
environments andofcorrect
the DTcommand
we were able to test functionalities,
execution communication
in advance. During between
the testing, we also
406 environments and correct command execution in advance. During the testing
tracked the trajectory of tool motion in NX MCD and also in Tia Portal; see Figure we11.
also
407 tracked the trajectory of tool motion in NX MCD and also in Tia Portal; see Figure 11.

Figure 11. Trace of the tool visualized in TIA Portal.

Figure 11. Trace of the tool visualized in TIA Portal.
The motion of tool in NX MCD however did not follow the programmed trajectory. As
shown
408 isThe motion
Figure 12a,of toolare
there in NX MCD
loops however
arising. Sincedid
thenot follow
tool tracesthe
inprogrammed trajectory.
particular environments
As shown
differed,
409 is Figure signal
we included 12a there are loops
adapters forarising.
motionSince theinto
control tool the
traces in particular
tracking in NXenvi-
MCD.
410 ronments differed, we included signal adapters
Time profile of the signal is depicted in Figure 12b. for motion control into the tracking in
411 NX MCD. Time profile of the signal is depicted in Figure 12b.
Position [◦ ]

Home
position

0.0 4.0 8.0 12.0 16.0

Time [s]
(a) (b)
Figure 12. Results from simulation in NX MCD using PLC OPC UA server. (a) Trace of the tool. (b) Time profile of the
Figure 12. Results from simulation in NX MCD using PLC OPC UA server. (a) Trace of the tool.
signal adapter for axis position control.
(b) Time profile of the signal adapter for axis position control.
412 A detailed look at the signal (Figure 13) clearly shows its step changes which cause
413
A detailed
the looktrace.
loops in tool at theInsignal
fact, it(Figure
is due to13)communication
clearly shows its stepthat
speed changes which cause
we mentioned in
the
414 loops
Section 2.4. This problem thus needs to be solved for verification and validation (V&V)in
in tool trace. In fact, it is due to the communication speed that we mentioned
Section
415 of our2.4. This problem
model, which wethus needs to
performed bybe solved
means for verification
of animation method.andMore
validation (V&V)
specifically,
of our model, which we performed by means of animation method.
416 we used tracing to observe motion of the tool. For V&V to be acceptable, the tool has More specifically,
we used
417 tracing
to follow to observewithout
the trajectory motionany of the tool.
loops. Fora V&V
Such to be
behavior acceptable,
could in some the
casestool has
lead toto
follow
418 the readings
faulty trajectoryfrom
without any loops.
NX MCD, which Such a behavior
would could,
impact the in some
executed cases, lead
program. to faulty
Therefore,
readings
419 from
although NX
the MCD, which
program could be would
correct,impact the executed
deviations program.would
in the animation Therefore, although
misrepresent
the program
420 the resultscould be correct,prolong
and eventually deviations in the animation would misrepresent the results
the work.
and eventually
421 prolong
As already the work.
mentioned, the OPC PLC UA server was replaced with the faster OPC
422 UA NET server with cycle time of 20 ms. After this change the procedure of signal map-
423 ping was had to be performed again. The update time for NX MCD was also adjusted
424 to match the cycle time of server. Next, we again observed the tool trace and the signal
 Sensors 2022, 22, 1621 14 of 17
Version February 14, 2022 submitted to Sensors 14 of 17

Version February 14, 2022 submitted to Sensors 14 of 17

Zoom: 300 %

Position [◦ ]
Version February 14, 2022 submitted to Sensors 14 of 17
Zoom: 300 %

Position [◦ ]
Zoom: 300 %

Position [◦ ]
Home
position
Home
position
Home
position

3.8 5.1 6.5

Time [s]
3.8
6.5 5.1
Figure 13. Detail of the signal obtained using PLC OPC UATime
server.
[s]server.
Figure 13. Detail of the signal obtained using PLC OPC UA
3.8 5.1 6.5
Figure 13. Detail of the signal obtained using
TimePLC
[s] OPC UA server.
425 Figure 13. As
adapter of already
one
Detail ofofthe
thementioned,
position
signal the
axes.
obtained OPC
As PLC
using PLC
shown UA
inUA
OPC server
Figure wasthe
14a,
server. replaced with
trace no the contains
longer faster OPC UA
426 anyNET
425
serverloops.
apparent
adapter with
of oneaTime
cycle time
profile
of the ofof20
position ms.
the
axes. After
signal this change,
depicted
As shown in in the14a,
Figure
Figure procedure
14btheis traceofno
signal
smoother, mapping
with
longer contains
427
425
had
noadapter
abrupt
426
to be performed
changes,
of one
any as again.
shown
of the position
apparent in
loops. TimeThe update
detail in
axes.profile
As shown time
Figure
of the for
15. NX
in Figure MCD
signal 14a, was
the trace
depicted also adjusted
no longer
in Figure is smoother, the
to
14bcontains match with
426
cycle
any time
427 apparent
no ofloops.
abrupt the server. Next,
Time profile
changes, as wethe
of
shown again
in observed
signal
detail depicted
in Figurethe tool trace
in15.
Figure 14b and the signal
is smoother, adapter of
with
427 one
no of the
abrupt positionasaxes.
changes, Asinshown
shown detail in
in Figure 14a, the trace no longer contains any apparent
Figure 15.
loops. Time profile of the signal depicted in Figure 14b is smoother, with no abrupt changes,
as shown in detail in Figure 15.
Position [◦ ]

Position [◦ ]
Position [◦ ]

Home
position
Home
Home position
position

0.0 4.0 8.0 12.0 16.0

Time [s]
(a) 0.0 0.0 4.0 4.0 8.0 (b) 8.0 12.0 12.0 16.0 16.0
Time [s] Time [s]
(a)
(a) simulation
Figure 14. Results from in NX MCD using OPC UA NET server. (a) Trace (b) of the (tool.
b) (b) Time profile of the
signal adapter for axis position control.
FigureFigure 14. Results
14. Results from simulation
from simulation in NX
Figure in
14.MCDNXusing
MCD
Results using
OPC
from UAOPC
NETUA
simulation NET(a)
server.
in NX server.
MCDTrace (a)the
of
using Trace of(b)
tool.UA
OPC the tool.
Time
NET (b) Time
profile
server. of profileofofthe
(a)the
Trace thetool.
signalsignal
adapter for axisfor
adapter position control.
axis position
(b) Timecontrol.
profile of the signal adapter for axis position control.
Zoom: 220 %
Position [◦ ]

Zoom: 220 %Zoom: 220 %

Position [◦ ]

Position [◦ ]

Home
position
Home
position Home
position

1.8 3.6 5.4 7.3 9.1 10.9

1.8 3.6 Time
10.9[s] 5.4 7.3 9.1
Figure 15. Detail1.8
of the Time
UA [s] server.
NET
9.1 10.9 signal
3.6 obtained
5.4 using 7.3
OPC
Time [s]
Figure 15. Detail of the signal obtained using OPC UA NET server.
Figure Detail
15.15.
Figure ofof
Detail the signal
the obtained
signal obtainedusing
usingOPC
OPC UA NET
UA NET server.
server.
Sensors 2022, 22, 1621 15 of 17

3.2. Testing Scenarios

During the tests, we had encountered various errors, from minor to major ones that
could damage the device or even endanger safety of persons. One of the revealed errors
was an opposite alignment of the coordinate systems used by TIA Portal and NX MCD.
We fixed this error by additional recalculation of variables in NX MCD. Another error had
been revealed in the code, which allowed us to rectify misalignment of coordinate systems
of the camera and the manipulator. The errors that could cause damage to the real device
included one that could make the arms overturn and hence collide with the base and axes
of the manipulator. At the same time, it was possible to test how lighting affects the camera
resolution and hence the scanned parts. Already in the design phase, this helped us to
better understand what the optimal conditions for accurate object recognition are.
Hybrid virtual commissioning may however entail a disadvantage in combination
with available real-time simulation capabilities. In the current implementation, real-time
simulation for systems with fast dynamics, such as the delta robot, is due to limited
computational power not feasible for a full speed of motion attainable by the device.

3.3. Partial Application on Real Manipulator

Finally, we implemented the developed and by-HVC-validated program to control
the physical device. When transitioning to the real robot manipulator, safety functions
were added to the project and virtual axes were replaced with frequency converters of
the SIMATIC S210 series. Since we had not had the gripper components at our disposal,
its function remained in NX MCD where signals indicating gripping were obtained from.
Figure 16 depicts the real, partially implemented delta robot manipulator together with its
digital twin implemented in NX MCD and an HMI for controlling the manipulator.

Figure 16. Real, partially implemented delta robot manipulator and its digital twin implemented in
NX MCD.

After commissioning of the device, the DT can be reused, for example, when testing
a new type of work tool. This will shorten the time required for changing the work tool
of the real device. It can also be used for integration into a production line, where it will
Sensors 2022, 22, 1621 16 of 17

collaborate with other devices on performing given tasks, or for testing the feasibility of
new trajectories when modifying the control program.

4. Conclusions
Digital twins offer a great potential by predicting future behavior of manufacturing
systems and processes instead of analyzing the past. When combined with appropriate
equipment, it is possible to obtain more from the DT than what was invested in it. It is
therefore no surprise that there is an ongoing effort to make processes and their imple-
mentation more efficient. It is the implementation of solutions where one may observe
a diversity of environments and control approaches which allow to achieve the desired
result. This diversity often introduces additional requirements into the implementation of
a solution, which can lead to an increase in cost, required time or personnel. Hence, there is
also an effort to make the time from designing up to commissioning as short as possible.
This can be achieved already in the design stage by means of testing using various models—
mathematical, statistical or animative ones. These models are subsequently subject to the
aforementioned approaches of VC or HC.
Our proposed approach of HVC represents a concept that combines properties of VC
and HC. It makes sense to use it if the cost of simulating a device was higher or the work
required to model a certain subsystem was more demanding than direct implementation of
a particular device into the model. The HVC concept can be conveniently used as a tool in
the training and teaching process. Thanks to virtualization and simulation of processes,
there is no risk of damaging equipment or causing injuries.
The HVC concept has been demonstrated in a case study featuring a robotic manipula-
tor with machine vision and a single PLC in a pick-and-place application. Its DT created in
NX MCD allowed to accelerate the commissioning. At the same time, during verification
and validation of the control program, using an animation method, we were able to reveal
certain errors before implementing the program on a real device. Besides the concept
itself, we also used the case study to demonstrate the benefits of using fewer software
environments. In particular, we used three software environments, one of them via WBN.
This implies fewer programmers, although the requirement on programmer’s knowledge
is higher.
Our further research will focus on the possibility of using digital twin to determine
suitability of a device for improving a given process.

Author Contributions: Conceptualization, M.N., M.J. and M.G.; methodology, M.J.; software, M.N.
and M.J.; validation, M.N. and M.J.; formal analysis, M.N. and M.J.; investigation, M.N. and M.J.;
resources, M.N. and M.J.; data curation, M.N.; writing—original draft preparation, M.N. and M.G.;
writing—review and editing, M.J. and M.G.; visualization, M.N.; supervision, M.J. and M.G.; project
administration, M.J. and M.G.; funding acquisition, M.G. All authors have read and agreed to the
published version of the manuscript.
Funding: This research was funded by the Slovak Research and Development Agency under the
grants APVV-18-0023, APVV-20-0046 and APVV SK-AT-20-0021. The APC was funded by the project
‘Predictive control of mechatronic systems and industrial processes’ under the program ‘Excellent
creative teams at Slovak University of Technology in Bratislava’.
Institutional Review Board Statement: Not applicable.
Informed Consent Statement: Not applicable.
Data Availability Statement: The data presented in this study are available on request from the
corresponding author. The data are not publicly available due to privacy.
Acknowledgments: M.J. and M.G. would like to thank the European Institute of Innovation and
Technology under the activities PowerLIFT and FactoRIS for their financial contribution.
Conflicts of Interest: The authors declare no conflict of interest. The funders had no role in the design
of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript or
in the decision to publish the results.
Sensors 2022, 22, 1621 17 of 17

References
1. Shen, W.; Yang, C.; Gao, L. Address business crisis caused by COVID-19 with collaborative intelligent manufacturing technologies.
IET Collab. Intell. Manuf. 2020, 2, 96–99. [CrossRef]
2. Bellalouna, F. Case study for design optimization using the digital twin approach. Procedia CIRP 2021, 100, 595–600. [CrossRef]
3. Strelets, D.Y.; Serebryansky, S.A.; Shkurin, M.V. Concept of creation of a digital twin in the uniform information environment
of product life cycle. In Proceedings of the 13th International Conference “Management of Large-Scale System Development”,
Moscow, Russia, 28–30 September 2020; pp. 1–4.
4. Martinez, E.M.; Ponce, P.; Macias, I.; Molina, A. Automation pyramid as constructor for a complete digital twin, case study: A
didactic manufacturing system. Sensors 2021, 21, 4656. [CrossRef] [PubMed]
5. Stary, C. Digital twin generation: Re-conceptualizing agent systems for behavior-centered cyber-physical system development.
Sensors 2021, 21, 1096. [CrossRef] [PubMed]
6. Qin, H.; Wang, H.; Zhang, Y.; Lin, L. Constructing digital twin for smart manufacturing. In Proceedings of the 24th International
Conference on Computer Supported Cooperative Work in Design, Dalian, China, 5–7 May 2021; pp. 638–642.
7. Schluse, M.; Priggemeyer, M.; Atorf, L.; Rossmann, J. Experimentable digital twins—Streamlining simulation-based systems
engineering for Industry 4.0. IEEE Trans. Ind. Inform. 2018, 14, 1722–1731. [CrossRef]
8. Huang, Z.; Shen, Y.; Li, J.; Fey, M.; Brecher, C. A survey on AI-driven digital twins in Industry 4.0: Smart manufacturing and
advanced robotics. Sensors 2021, 21, 6340. [CrossRef] [PubMed]
9. Negri, E.; Fumagalli, L.; Macchi, M. A Review of the Roles of Digital Twin in CPS-based Production Systems. Procedia Manuf.
2017, 11, 939–948. [CrossRef]
10. Zhang, L.; Zhou, L.; Horn, B.K. Building a right digital twin with model engineering. J. Manuf. Syst. 2021, 59, 151–164. [CrossRef]
11. ISO 23247–1:2021 ; Automation Systems and Integration—Digital Twin Frame-Work for Manufacturing—Part 1: Overview and
General Principles. International Organization for Standardization: Geneva, Switzerland, 2021.
12. Shao, G. Use Case Scenarios for Digital Twin Implementation Based on ISO 23247; NIST Advanced Manufacturing Series 400–402;
National Institute of Standards: Gaithersburg, MD, USA, 2021.
13. Donoghue, I.; Hannola, L.; Papinniemi, J.; Mikkola, A. The benefits and impact of digital twins in product development phase of
PLM. In Product Lifecycle Management to Support Industry 4.0; Chiabert, P., Bouras, A., Noël, F., Ríos, J., Eds.; Springer International
Publishing: Berlin/Heidelberg, Germany, 2018; pp. 432–441.
14. Reinhart, G.; Wünsch, G. Economic application of virtual commissioning to mechatronic production systems. Prod. Eng. 2007,
1, 371–379. [CrossRef]
15. Allmacher, C.; Schumann, M.; Klimant, P.; Putz, M. Optimizing development time through hybrid commissioning of control
software. Procedia CIRP 2019, 79, 450–455. [CrossRef]
16. Demers, S.; Gopalakrishnan, P.; Kant, L. A Generic Solution to Software-in-the-Loop. In Proceedings of the 2007 IEEE Military
Communications Conference, Orlando, FL, USA, 29–31 October 2007; pp. 1–6.
17. Dominka, S.; Schiller, F.; Kain, S. Hybrid commissioning–speeding-up commissioning of field bus driven production plants.
In Proceedings of the IEEE International Conference on Mechatronics, Kumamoto, Japan, 8–10 May 2007; pp. 1–6.
18. Sargent, R.G. Verification and validation of simulation models. In Proceedings of the 2010 Winter Simulation Conference,
Baltimore, MD, USA, 5–8 December 2010; pp. 166–183.
19. Kain, S.; Dominka, S.; Merz, M.; Schiller, F. Reuse of HiL simulation models in the operation phase of production plants.
In Proceedings of the 2009 IEEE International Conference on Industrial Technology, Churchill, VIC, Australia, 10–13 February 2009;
pp. 1–6.
20. National Infrastructure Commission. Data for the Public Good. Available online: https://nic.org.uk/app/uploads/Data-for-the-
Public-Good-NIC-Report.pdf (accessed on 2017 ).
21. Bouri, M.; Clavel, R. The Linear Delta: Developments and Applications. In Proceedings of the ISR 2010 (41st International
Symposium on Robotics) and ROBOTIK 2010 (6th German Conference on Robotics), Munich, Germany, 7–9 June 2010; pp. 1–8.
22. Juhás, M.; Gulan, M.; Grandová, K. A learning factory concept for Slovak manufacturing SMEs. In Proceedings of the 11th
Conference on Learning Factories, online, 1–2 July 2021.
23. Kleijnen, J.P. Verification and validation of simulation models. Eur. J. Oper. Res. 1995, 82, 145–162. [CrossRef]
24. Siemens AG. Organization Blocks for Motion Control. Available online: https://support.industry.siemens.com/cs/mdm/
109749263?c=100323337227 (accessed on 3 October 2021).
25. Siemens AG. SIMATIC S7-1500T Kinematics Control. Available online: https://support.industry.siemens.com/cs/document/
109755891/ (accessed on 3 October 2021).
26. Siemens AG. SIMATIC S7-1500T–Virtual Commissioning for Kinematics in NX MCD with Software in the Loop. Available online:
https://support.industry.siemens.com/cs/document/109760078/(accessed on 3 October 2021).
27. Siemens AG. Transformation of MV440 Camera Coordinates into Roboter Coordinates. Available online: https:
//support.industry.siemens.com/cs/document/109755137/(accessed on 3 October 2021).
28. Siemens AG. SIMATIC–Failsafe Library LDrvSafe to Control the Safety Integrated Functions of the SINAMICS Drive Family.
Available online: https://support.industry.siemens.com/cs/document/109485794/(accessed on 3 October 2021).