Available online at www.sciencedirect.

com
Available online at www.sciencedirect.com
ScienceDirect
ScienceDirect
Available online at www.sciencedirect.com
Procedia Manufacturing 00 (2019) 000–000
www.elsevier.com/locate/procedia
ScienceDirect
Procedia Manufacturing 00 (2019) 000–000
www.elsevier.com/locate/procedia
Procedia Manufacturing 45 (2020) 37–42

10th Conference on Learning Factories, CLF2020

10th Conference on Learning Factories, CLF2020
Method to measure competencies - a concept for development,
Method to measure competencies - a concept for development,
design and validation
designa,and validation a
Rupert Glass *, Joachim Metternich
a Rupert
Institute of Production Management, Glass
Technology
a,
*, Joachim
and Machine Metternich
Tools, Germany,
a
Otto-Berndt-Str. 2, 64287 Darmstadt, Germany
a
Institute of Production Management, Technology and Machine Tools, Germany, Otto-Berndt-Str. 2, 64287 Darmstadt, Germany

Abstract
Abstract
Production companies face the challenge to adapt their manufacturing processes and technologies to changing market demands and
to stay ahead in global competition. To master this challenge successfully competencies at all hierarchical levels are a central
Production companies face the challenge to adapt their manufacturing processes and technologies to changing market demands and
success factor. Learning factories have proven to be a successful approach to develop these competencies. Trainings in learning
to stay ahead in global competition. To master this challenge successfully competencies at all hierarchical levels are a central
factories are particularly beneficial, when their learning success can be assessed. In order to verify the learning success and to
success factor. Learning factories have proven to be a successful approach to develop these competencies. Trainings in learning
revise trainings afterwards precisely, a method to determine the acquisition of competencies is necessary. This paper addresses the
factories are particularly beneficial, when their learning success can be assessed. In order to verify the learning success and to
question of how production-relevant competencies can be determined if they can’t be assessed directly. An approach on how a
revise trainings afterwards precisely, a method to determine the acquisition of competencies is necessary. This paper addresses the
measurement can be performed is presented, as well as the direction of further refinement of the method. Among other things, a
question of how production-relevant competencies can be determined if they can’t be assessed directly. An approach on how a
structural equation analysis will be used to empirically test hypotheses on competency acquisition. Using a variety of different data
measurement can be performed is presented, as well as the direction of further refinement of the method. Among other things, a
sources (e.g. written test, observation or self-assessment) diverse parameters will be tested for the determination of competencies
structural equation analysis will be used to empirically test hypotheses on competency acquisition. Using a variety of different data
with the aim to develop a statistically sound measurement procedure. This will make it possible to compare different approaches
sources (e.g. written test, observation or self-assessment) diverse parameters will be tested for the determination of competencies
with a large dataset created in the learning factory “Center of industrial Productivity” (CiP) at the Technical University Darmstadt.
with the aim to develop a statistically sound measurement procedure. This will make it possible to compare different approaches
with a large dataset created in the learning factory “Center of industrial Productivity” (CiP) at the Technical University Darmstadt.
© 2020 The Authors. Published by Elsevier Ltd.
This is an
© 2020
2020 open
The accessPublished
Authors. article under the CC BY-NC-ND
by Elsevier
Elsevier Ltd. license https://creativecommons.org/licenses/by-nc-nd/4.0/)
© The Authors. Published by Ltd.
Peer-review
This is an statement:
open access Peer-review
article under under
the CCresponsibility
BY-NC-ND of the scientific
license committee of the 10th Conference on Learning Factories
(https://creativecommons.org/licenses/by-nc-nd/4.0/)
This is an open access article under the CC BY-NC-ND license https://creativecommons.org/licenses/by-nc-nd/4.0/)
2020
Peer-review under responsibility of the scientific committee of the 10th Conference on Learning Factories 2020.
Peer-review statement: Peer-review under responsibility of the scientific committee of the 10th Conference on Learning Factories
2020
Keywords: Learning factories; trainings; measurement of competencies; method development

Keywords: Learning factories; trainings; measurement of competencies; method development

________
________
* Corresponding author. Tel.: +49-6151-16-20577; fax: +49-6151-16-20087.
E-mail address: r.glass@ptw.tu-darmstadt.de
* Corresponding author. Tel.: +49-6151-16-20577; fax: +49-6151-16-20087.
E-mail address: r.glass@ptw.tu-darmstadt.de
2351-9789 © 2020 The Authors. Published by Elsevier Ltd.
This is an open access article under the CC BY-NC-ND license https://creativecommons.org/licenses/by-nc-nd/4.0/)
2351-9789
Peer-review©statement:
2020 The Peer-review
Authors. Published by Elsevier Ltd.
under responsibility of the scientific committee of the 10th Conference on Learning Factories 2020
This is an open access article under the CC BY-NC-ND license https://creativecommons.org/licenses/by-nc-nd/4.0/)
Peer-review statement: Peer-review under responsibility of the scientific committee of the 10th Conference on Learning Factories 2020

2351-9789 © 2020 The Authors. Published by Elsevier Ltd.

This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/)
Peer-review under responsibility of the scientific committee of the 10th Conference on Learning Factories 2020.
10.1016/j.promfg.2020.04.056
38 Rupert Glass et al. / Procedia Manufacturing 45 (2020) 37–42
2 Author name / Procedia Manufacturing 00 (2019) 000–000

1. Introduction

Production-relevant competencies of their workforce are a key requirement for companies to remain competitive
[1,2]. This proposition was recognized years ago, but stays significant with new developments in the fields of
digitalization and Industrie 4.0. Studies show that there is a great need for teaching and further training in this area in
particular, for which existing educational institutions are not sufficiently prepared [3]. Learning factories are
established as a promising approach to achieve this necessity with competency-oriented trainings. However, such
trainings can only be useful if they can be made verifiable [4]. To ensure the learning success of competency-oriented
trainings and to accurately rework and optimize training courses an efficient competency measurement is necessary.
The first method to measure competencies in learning factories was presented by Tisch et al. [5]. Nevertheless, the
CIRP collaborative working group on learning factories pointed out that larger scale studies need be conducted to
investigate learning success and its measurement methods in learning factories [6]. In 2018 Glass et al. [7] also
confirmed, that existing approaches are only applicable in real production environment production with limitations.
Their set up is too time-consuming and complex and not acceptable for operational staff. Also they are not suitable to
measure technical methodological competencies.
This paper therefore presents an approach on how a measurement can be realised, as well as the procedure for
further refinement of the method. Among others, a structural equation model (SEM) will be used to empirically test
the interdependencies between an achieved competency level and different parameters that are easier to assess and
quantify. In literature the term structural equation analysis is also used synonymously [8]. Chapter 2 will therefore
give a brief introduction in the field of competencies, competency measurements and SEM. While chapter 3 will
clarify the aim of the research, chapter 4 will present the current state of the measurement method and describe the
approach on how to apply a SEM to further refinement. The paper closes with a conclusion and an outlook for the near
future.

2. Theoretical background

2.1. What are competencies in general?

In literature, the term “competency” has been gaining more and more importance in the fields of education and
training, in which the focus shifted from certified qualifications to acquired competencies. In everyday language,
knowledge, qualification and competencies are often used synonymously. While knowledge and qualifications are the
necessary prerequisites, a competency can be summarized as the ability to act in unknown and complex situations [9].
Erpenbeck and Rosenstiel [10] categorize competencies in four different classes:
• personal competencies, which describe the reflexive and self-organized actions of a person. Examples are
the ability to evaluate one’s own skills.
• social-communicative competencies, which contribute to the communicative actions along with basic
interactions with other people.
• activity and implementation orientated competencies, to which all abilities that contribute to the acting by
oneself count. They include the will to direct one’s own actions towards implementing plans into reality.
• professional-methodical competencies, which a person needs in order to solve objective problems. The
application of professional knowledge, skills and methods is part of these.
While the presented approach can be applied to examine different classes, the focus of this research is on the
professional-methodical competencies.

2.2. What are competency measurements?

Competencies neither can be measured directly, nor can they be easily assessed or evaluated. Competencies of an
acting person can, however, be observed indirectly through corresponding knowledge and actions. By the evaluation
of these a coherent evaluation of competencies is possible. If the measurement parameters are clearly defined, a self-
assessment also becomes possible. However, subjective distortions of the results must be taken into account. The
 Rupert Glass et al. / Procedia Manufacturing 45 (2020) 37–42 39
Author name / Procedia Manufacturing 00 (2019) 000–000 3

objectives of the measurement are particularly relevant here, as participants may be motivated to influence the results
in their favor. In principle, competency evaluations can be separated into the following categories on the basis of their
approach: Subjective or objective evaluation, open or closed evaluation, self-evaluation or external evaluation,
summative or formative evaluation, direct or indirect evaluation, quantitative or qualitative evaluation. In addition,
combinations in the form of hybrid approaches are also possible. While none of the existing approaches are superior
to the others, the selection must be made individually according to their objectives and context. It is evident that there
are numerous procedures in literature. For further details see [10].

Main Sub-
Observable action Required knowledge
Competence competencies

Integrating customer, supplier and

PPS into the Value stream map Knowledge of all the
available symbols and
Utilizing the
Conducting Drawing the different processes the overall process
right VSM
the value steps
symbols
stream
mapping Knowledge of the
Drawing the connecting symbols
connecting symbols

… … …

Fig. 1. Section of a competency matrix with generalized knowledge aspects.

A prerequisite to measure competencies is to define them first. Competency matrices are used for this purpose. In
a competency matrix a main competency (e.g. the application of a method for production optimization) is divided into
sub-competencies [11]. The sub-competencies are than described by their required knowledge aspects and relating
observable actions. A reduced section of an exemplary competency matrix on the subject of value stream analysis is
shown in Fig 1. These components can then be assessed by a competency measurement.

2.3. What are structural equation analyses / structural equation models?

SEM offer the ability to test theoretically formulated statements in the form of hypotheses using empirical methods.
By applying SEM, a model (based on preliminary findings) is created, examined by a study and can finally be adapted
for the best fit possible. SEM map complex relationships between variables in a linear equation system in order to
investigate them statistically. Thus, a measurement method can not only be developed from theoretical assumptions,
but through statistically justified decisions based on the results of a large scale study.
In SEM, independent variables are referred to as exogenous variables (in this case different competency measuring
instruments e.g. the results of interviews, questionnaires or observations of actions and the latent variable motivation).
These are given "from outside" and used to explain dependent, endogenous variables (in this case knowledge, action
Section A Section A
δ1 X1 λ1
Results of an
δ2 X2 λ2 δ1 expert knowledge test

(…)
ζ2 λ1
(…) (…)
Knowledge
β1 Results of a
(…) δ2 self-assessment
λ2
δ4 X3 λ3 ζ1 (…) λ5 X5 5 ζ2
λ4 λ6
δ5 X4 Motivation β3 Competence X6 6 (…)
(…) β2
ζ3 (…)
(…)
(…)
Results of other knowledge Knowledge
(…) measurment instruments
(…) (…) (…)

(…) Action
(…) (…)

Caption:
Manifest exogenous Latent endogenous
Xn = Results of different λn = Correlation ζn and δn = Residuals
Direction of effect of measuring instruments and variances for latent
variable variable coefficient between latent
variables among each other and manifest variables
and manifest variable

Fig. 2. Exemplary illustration of the used structural equation model.

40 Rupert Glass et al. / Procedia Manufacturing 45 (2020) 37–42
4 Author name / Procedia Manufacturing 00 (2019) 000–000

and competency). Endogenous variables are thus explained by their influence on other variables. For each latent
variable a regression equation is formulated, which is called a structural equation. Fig. 2 shows an exemplary structural
equation model which is used in the presented research, Formula 1 shows the structural equation in Matrix notation
for the latent variables, whereas B and Γ are the coefficient matrices, Ψ is the matrix of disturbance variables 𝜂𝜂 are
the latent endogenous variables (knowledge, action and competence) and ξ are the latent exogenous variables
(motivation). The presented structure will be further examined in chapter 4 of this paper.
𝜂𝜂 = 𝐵𝐵𝐵𝐵 + 𝛤𝛤𝛤𝛤 + 𝛹𝛹 (1)
In contrast to classical regression analysis, relationships between empirically directly measurable (manifest)
variables and empirically not directly observable (latent) variables are analyzed. Therefore, a SEM is quite suitable
for the investigation of competencies since they are not directly measurable. Missing connections of variables, which
might not be measurable like values of an individual, are reflected in the disturbance variables. SEM can be adjusted
using the empirical data collected in a study to improve the model structure. This gives the analysis an explorative
character. The model structure can be changed by excluding previously specified parameters from the model in order
to improve the model's fit. In this way, suitable measuring instruments can be identified and a statistically reasoned
measuring method for competencies can be developed.

3. Research goals

Competency measurement methods can pursue different goals. These include, for example, competency
diagnostics, personnel selection, potential analyses or the evaluation of training courses [12]. The main objective of
the presented research is the development of a measurement method to determine the acquisition of competencies by
training in companies and learning factories. Applicable in learning factories but also in production it should be
possible to improve training courses in a precise manner and prove the learning success of learning factories. The
procedure for competency-oriented training design in learning factories [11] is to be complemented by the
measurement of learning success in order to facilitate an improvement cycle. Therefore, the following central research
question and two sub questions are derived:
 How can an existing level of production-relevant competencies be determined objectively and in a
practice-oriented way?
o Which measuring parameters and instruments provide significant information about competency?
o How can a measurement method developed in learning factories be applied to production?

4. Concept for development of the measurement

Schuh and Warschat propose a research methodology in six steps [13] which provides the framework for this
research (see Fig. 3). Starting with the research questions discussed above existing solutions are reviewed based on
the competency model of Erpenbeck and Rosenstiel which includes the motivation of participants as prerequisite of
knowledge acquisition and its practical application. This makes it possible to derive the structure of the SEM in the
next step as shown in Fig. 3. It explains the indicators used and ensures their validity.
Practical orientation Scientific orientation

Question: How can competency development in Evaluation of existing solutions:

learning factories and production be determined? Methods and measuring instruments

Data collection: Specification of the own procedure:

Studies in the process learning factory CiP Structural equation analysis

Validation in the application context: Derivation of design rules: Explorative SEM &
Competence measurement in practice final designation of the measurement method

Fig. 3. Research design to develop the competency measurement.

 Rupert Glass et al. / Procedia Manufacturing 45 (2020) 37–42 41
Author name / Procedia Manufacturing 00 (2019) 000–000 5

In the specification of the own procedure phase, it is necessary to check which indicators of competency allow a
clear and quantifiable measurement. The latent and manifest variables and their properties must be defined as concrete
as possible. It has to be described who should judge which variables and how it is supposed to be realized in
application. It is important that the definition does not restrict the general applicability of the model too much. For the
specification on the measuring instruments the applicability in learning factories and in a production environment are
a key requirement, as the measurement should not interfere too much in the everyday work, of the workforce. The
time and effort should remain proportionate for the user.
For the latent variable knowledge of the SEM, a self-estimation questionnaire of the theoretical knowledge as well
as a knowledge test is used. The test consists of questions aiming at different areas of knowledge (expert-, reasoning-
and process knowledge) derived from previous research [14]. The questions can be extracted from the competency
matrix which has to be set up during the conception of every competency oriented workshop [11]. The motivation is
measured with a questionnaire in the four fields “interest in the topic”, “fear of failure”, “assessment of one's own
prospects of success” and “evaluation of the challenge in the application of the examined competency” [15].
The action of the participants is observed and rated by the seven parameters “sequence”, “error rate”,
“independence”, “duration”, “general assessment of the observer” and “external assessment of another participant”.
These parameters are the result of an extensive literature research and a validation through interviews with trainers of
learning factories. During the examination of possible observational parameters, it became apparent, that there are still
a lot of unanswered research questions this particular field of study. Part of the following research will be the test of
these parameters. It is quite possible that multiple of these have to be excluded due to a poor applicability within a real
production environment or no clear statistical evidence for their application.
To apply the SEM, an indicator for competency itself is needed. Here, a self-evaluation as well as a comparison of
the results in a practical exercise with a prepared sample solution is used. This can only be realized in case of a well
prepared study, in which the examined competency, the task of the participants and the sample solution are aligned
perfectly in order to make all presented parameters comparable. In a regular application of a competency the results
will always be open ended, thus not comparable with a sample solution.
This leads into the phase of data collection, in order to service the SEM with sufficient data to evaluate the model.
Therefore, a large scale study is conducted in the Process Learning Factory CiP at the PTW in Darmstadt with around
120-180 participants. Here, participants (students of technical studies) will first take part in workshops to learn
professional-methodical competencies (methods of lean production), and have to practically apply them in a new
environment of the learning factory (see Fig. 4). In preparation of the study, two independent pre-tests of the workshops
were held in which the content and duration was tested as well as observers trained for their deployment in the field.
The focus here was on the calibration of observers. They must have fully comprehended the measuring instruments
and the contents of the training course in order to achieve a high degree of standardization in the evaluation of the
actions taken by the participants. In the second pre-test, participants were evaluated by two observers at the same time.
The measuring instruments proved to be reliable and valid.

Theoretical test of Observation of: Theoretical test of

expert-, reasoning- and Theoretical course • Sequence expert-, reasoning- and
process knowledge • Error rate process knowledge
• Independence
Self-evaluation of • Duration Self-evaluation of
competency Practical exercise • General assessment competency
and motivation and motivation
• External participant

Preliminary investigation: Competency-oriented Application of learned competencies in Follow-up investigation:

examine the current learning factory training a new environment & application of examination of actual
status different observational parameters competencies

Fig. 4. Sequence of the workshops for data collection.

42 Rupert Glass et al. / Procedia Manufacturing 45 (2020) 37–42
6 Author name / Procedia Manufacturing 00 (2019) 000–000

In the derivation of design rules phase the data collected will be applied within the SEM to test the fit of the model
as well as the correlation of the presented parameters with the model. Through the explorative use of the SEM,
adjustments can be tested that strengthen the correlations. The indicators (measuring instruments/parameters) that
support the model structure can be selected for future measurements. Thus, measuring instruments for competencies
can be quantitatively evaluated and compared. Non-applicable ones can be excluded and the most suitable ones can
be selected [8]. The goal of the explorative use of the SEM will be to define the parameters suitable for measuring
competencies, while keeping the costs and expenses in a proportionate scope in order to be able to be used in industry
as well. Once the final parameters are selected, the “trained model” can be used to calculate the resulting competency
through the input of the chosen measuring instruments and parameters. The measurement method can be applied in
further workshops and be validated with participants from the industry. This study will be part of the evaluation of the
research project “Mittelstand 4.0 Kompetenzzentrum Darmstadt” funded by the German Federal Ministry for
Economic Affairs and Energy (BMWi). For further information, see www.mit40.de.

5. Conclusion and outlook

At the point of writing this paper, the workshops have just taken place. A total of 127 participants took part in the
workshops. These are students of technical courses (56% mechanical engineering, 33% industrial engineering with a
specialization in mechanical engineering and 11% others). On average, the participants were in their 5th semester,
with a standard deviation of 4.6. On average, the participants were 23 years old, the majority were male (85%) with
only very little previous experience in the field of lean production. Throughout the study, 234 individual parameters
were examined, resulting in over 20.000 individual data points. Currently these are held in an Excel database, the SEM
will be done by SPSS Statics within the near future.
The field of competency measurements in production shows great potential for further research. In this paper a new
approach to design and validate a method to measure competencies in learning factories and industrial production was
presented. After the completion of ongoing studies and the statistical analysis of the acquired data, the next steps will
be a validation in industry. Additionally, the results will be applied to improve existing trainings in learning factories.
The possibility to close a feedback loop after a training to precisely revise workshops and trainings looks very
promising.

References

[1] J.B. Barney, D.N. Clark, Resource-based theory: Creating and sustaining competitive advantage, Oxford University Press, Oxford, New
York, 2007.
[2] E. Abele, G. Reinhart, Zukunft der Produktion, Carl Hanser Verlag, München, 2011.
[3] A. Gottburgsen, K. Wannemacher, J. Wernz, J. Willige, Ingenieursausbildung für die digitale Transformation: Zukunft durch Veränderung,
2019.
[4] R. Tenberg, Lerndiagnostik im kompetenzorientierten Unterricht, ZBW (2012) 482–490.
[5] M. Tisch, C. Hertle, J. Metternich, E. Abele, Goal-oriented improvement of learning factory trainings, The Learning Factory, An Annual
Edition from the Network of Innovative Learning Factories 1 (2015) 7–12.
[6] E. Abele, G. Chryssolouris, W. Sihn, J. Metternich, H. ElMaraghy, G. Seliger, G. Sivard, W. ElMaraghy, V. Hummel, M. Tisch, S.
Seifermann, Learning factories for future oriented research and education in manufacturing, CIRP Annals 66 (2017) 803–826.
[7] R. Glass, P. Miersch, J. Metternich, Influence of learning factories on students’ success – a case study, Procedia CIRP 78 (2018) 155–160.
[8] K. Backhaus, B. Erichson, R. Weiber, Fortgeschrittene Multivariate Analysemethoden, Springer Berlin Heidelberg, Berlin, Heidelberg, 2015.
[9] E. Abele, J. Metternich, M. Tisch, Learning Factories: Concepts, Guidelines, Best-Practice Examples, Springer International Publishing,
Cham, 2019.
[10] J. Erpenbeck, L.v. Rosenstiel, S. Grote, W. Sauter (Eds.), Handbuch Kompetenzmessung: Erkennen, verstehen und bewerten von
Kompetenzen in der betrieblichen, pädagogischen und psychologischen Praxis, 3rd ed., Schäffer-Poeschel Verlag, Stuttgart, 2017.
[11] E. Abele, J. Metternich, M. Tisch, M. Abel, C. Hertle, S. Eißler, J. Enke, L. Faatz, Innovative Lernmodule und -fabriken - Validierung und
Weiterentwicklung einer neuartigen Wissensplatform für die Produktionsexzellenz von morgen (2015).
[12] J. Erpenbeck, Was "sind" Kompetenzen?, in: Kompetenz, Persönlichkeit, Bildung Festschrift - Prof. Dr. John Erpenbeck zum 70. Geburstag,
Steinbeis-Ed, Stuttgart, 2012, pp. 1–57.
[13] G. Schuh, J. Warschat, Potenziale einer Forschungsdisziplin Wirtschaftsingenieurwesen, München, 2013.
[14] R. Tenberg, Vermittlung fachlicher und überfachlicher Kompetenzen in technischen Berufen: Theorie und Praxis der Technikdidaktik,
Steiner, Stuttgart, 2011.
[15] F. Rheinberg, R. Vollmeyer, B.D. Burns, QCM: A Questionnaire to assess current motivation in learning situations, Diagnostica 47 (2001)
57–66.