Proceeding of the IEEE
International Conference on Information and Automation
Hailar, China, July 2014
Evaluation Indicators Architecture for Industrial Automation System Integration
Hao Luo
Department of Automation
Tsinghua University
Beijing, China
e-mail: wing012k@163.com
Abstract—The evaluation of industrial automation system (IAS)
integration to support decisions is critical for enterprise
management. This paper proposes an evaluation meta-model,
which absorbs the ideology and principles of IDEF0 and
defines the essential elements when conducting an evaluation
for system. Based on the idea that management process plays a
key role on IAS integration, a generic Evaluation Indicators
Architecture (EIA) for IAS integration evaluation is derived
from the proposed meta-model by mapping Views of GERAM
to the meta-model.
Keywords-Industrial Automation Systems; integration;
system evaluation; Evaluation Indicator Architecture;
I. INTRODUCTION (HEADING 1)
With the process of globalization and the emergence of
new information technologies such as Internet of Things,
Cloud computing, the complexity of industrial environment
that enterprises encounter is increasingly expanding. In this
context, the progress in overall integration of Industrial
Automation System (IAS) is taken as a critical factor for
every enterprise to acquire its competitiveness advantage. To
effectively achieve a higher level of integration, enterprise
managers need to perform evaluation of integration to
identify problems and opportunities as a basis for
decision-making. It is in urgent need of developing an EIA
for managers to evaluate IAS integration correctly.
IAS integration evaluation has been widely discussed in
literature. Several researchers have developing indicator
architectures for specific IAS evaluation objectives, such as
interoperability, security, and MES assessment, etc. [1-3].
Most indicator architectures are built based on experts’
experience and preference, which makes it not objective
enough and lack of scalability. Li et al. [4] developed a
method to built EIA for IAS integration by organizing
indicators based on Views defined in GERAM. However, the
indicators of the proposed EIA focus on technology
application too much and neglect the importance of
management process to achieve overall integration.
This paper suggests a meta-model as a reference to
develop EIA for system evaluation. This meta-model adapts
the basic “boxes and arrows” model in IDEF0 [5] to the
requirement of evaluation. We derived an EIA for IAS
integration by mapping Views of the Generalized
Enterprise-reference Architecture and Methodologies
(GERAM) to the meta-model, which refers to the method
1082
978-1-4799-4100-1/14/$31.00 ©2014 IEEE
Qing Li
Department of Automation
Tsinghua University
Beijing, China
e-mail: liqing@tsinghua.edu.cn
proposed in [4]. But different from the method in [4], we
consider the process of IAS integration progress as the
process of effective management of a series of projects about
IAS integration and construct the indicators model from the
perspectives of project management process.
The rest of the paper is organized as follows: Section 1
provides a meta-model for system evaluation adapted from
IDEF0. In Section 2, the generic IAS integration EIA
developed from the meta-model is introduced. Section 3
presents the conclusion and future work.
II. SYSTEM EVALUATION META-MODEL
IDEF0 is a modeling language for requirements
definition, functions specification and analysis supporting.
Combined graphics and text are presented in an organized
and systematic way to model systems. The most primary
modeling components are functions and data, which are
expressed by boxes and arrows. As shown in the top level of
Fig.1, a box represented a function activity of system, while
arrows represents data or object related to functions. There
are four kinds of standard arrows: Inputs are transformed or
consumed by the function to produce outputs; Controls
specify the conditions required for the function to produce
the correct outputs; Mechanisms identify some of the means
that support the execution of the function; Outputs are data
or objects produced by function [5].
We focus on the arrows in IDEF0 to identify aspects of
systems for evaluation. Evaluation meta-model is built by
conceptualizing corresponding evaluation elements from
data or object that arrows represent in IDEF0. Corresponding
with IDEF0, there are four elements in evaluation
meta-model: Sufficient and Adequate of Input measures
whether enough and appropriate resources are ready to be
consumed for production; Control effectiveness defines
criteria about whether the conditions or constrains on
systems facilitate an effective operation of systems;
Rationality of Mechanism evaluates elements about how the
system works, which usually includes evaluation of people
and equipment executing the system functions; Benefits of
Output covers performance indicators that evaluating the
performance of systems. The map path between IDEF0
diagram and evaluation meta-model is also shown in Fig.1.
Noted that, in meta-model, we ignore the functions of system
out of the idea that sometimes systems are black boxes to
outsiders and appropriateness of the aforementioned
 Figure 1. Meta-model, EIA and Mapping path
elements in systems assure functions meeting the system
requirements.
In instantiation process of meta-model, Evaluators of
complex system gather criteria information according to the
four evaluation elements in meta-model and built a specific
construct of EIA based on system characteristics. Similar to
IDEF0, it may be ambiguous when identifying whether a
specific set of metrics are included in Input or Control, but
the ambiguity has no affect on completeness of EIA.
Evaluators just need classify the ambiguous set of metrics
into any cluster in meta-model. It should be noted that other
models or architectures that depict the systems could be
taken as a reference to gather metrics and instantiate the
meta-model to a specific EIA. These models and
architectures provide a comprehensive or partial description
of systems from several aspects based on its objectives. By
mapping views of relevant models or architectures to the
meta-model, evaluators of a specific system can build the
basic constructs of EIA. A case of EIA with relation to IAS
integration is presented in next section.
III. DEVELOPING EIA WITH META-MODEL
As discussed in the previous section, a generic EIA,
which is an instantiation of evaluation meta-model, is
developed for managers to perform evaluation of IAS
integration. The enterprise managers may construct the final
EIA by adapting the general indicators architecture to meet
the particular requirement of the individual enterprises.
This EIA is instantiated from evaluation meta-model
and gather sets of indicators reference to the GERAM.
According to the evaluation meta-model, when conducting
an evaluation of system, four aspects need to be considered:
Sufficient and Adequate of Input, Control effectiveness,
Rationality of Mechanism, Benefits of Output. In order to
discover what should be included in each aspect of IAS
integration evaluation, GERAM is used as a reference to
instantiate. GERAM is documented as Annex A in ISO
1083
15704:2000 Industrial automation systems— Requirements
for enterprise-reference architectures and methodologies [6]
and specify four views as essential elements when modeling
enterprise, including Organization View, Information View,
Resource View and Function View. In addition, ISO 15704
Amendment 1:2005 provide an Economic View to measure
enterprise economic performance.
In this paper, the process of IAS integration progress is
considered as the process of effective management of a
series of projects about IAS integration. So the success of
IAS integration in a company is largely dependent upon the
continual, successful management of IAS integration project,
which comes from mature, reliable management process of
integration and underlying resource support. (Fig.2) Based
on the above thoughts, we map the views of GERAM to the
meta-model and define the specific concept of each view.
Because IAS integration evaluation is transformed into
evaluation of management process of integration, we take
critical management process of integration as elements in
Function View, which is classified into Control effectiveness.
And other view defines elements supporting management:
Organization View, classified into Rationality of Mechanism,
depicts organization structure, role setting and relevant issues
related to organization and roles; Information View, mapped
to Sufficient and Adequate of Input, defines information
resource in companies; Resource View specifies necessary
resources of IAS integration, such as investment and
facilities, covering elements including in both Sufficient and
Adequate of Input and Rationality of Mechanism; Economic
View links with Benefits of Output and measures enterprise
economic performance relater with integration. Additionally,
Strategy is considered as a key control element for IAS
integration, so it is another set of metrics included in Control
effectiveness. Finally, as shown in the bottom level of Fig.1,
the basic EIA of integration includes the following sets of
criteria: Management Process of Integration, Strategy,
Organization and Roles, Infrastructure and Data, Investment,
Performance Indicators. Criteria in each subset are detailed
below and mainly from literature and industrial practice
[7][8][9].
Management
 Process growth. It has to be noted that an organizational structure
Strategy Economic
Performance
Organization
Of
Infrastructure
 IAS the following factors:
Information
Investment
Figure 2. Framework of IAS management
First, confirm that you have the correct template for your
paper size. This template has been tailored for output on the
US-letter paper size. If you are using A4-sized paper, please
close this template and download the file for A4 paper
format called “CPS_A4_format”.
Management Process of Integration is the process that
facilitates the initiation, sustainability and success of the
application of IAS integration in enterprise business. The
purpose of these factors in this subset is evaluating the
effectiveness of enterprise integration management. The
factors are obtained from PDCA cycle [10], which represents
plan-do-check-action. So the factors are as below:
• The processes to plan a project of IAS integration
progress
• The way in which technology, business process and
organization are optimized
• The process associated with integration project
evaluation and reviewing
• The extent of normalization and institutionalization of
business process
Strategy defines strategic aspects in relation to IAS
integration to guarantee implement IAS integration at
strategic level, express enterprise integration policy and
objective, drive a consensus of integration, coordinate
integration strategy with strategies in other aspects, and get
commitment of support from top management in integration.
Since the strategy is the basic premise that drives the
company to move towards the goal of IAS integration, this
set plays a key role among all the sets. The factors in this set
are as follows:
• Integration planning and policy at strategic level
• Interconnection of integration strategy with other
strategies
1084
• Top management support
Organizational and Roles define the structure of
organization and the guarantee system of human resources
for integration, which is related to the definition of
authorities and responsibilities of each role and employee
that the department and leaders responsible for integration
can drive the business process optimization and
reengineering is necessary to assure depth of integration. In
addition, the guarantee system of human resources aims to
foster skills and awareness of integration as well as
encourage staff to form a shared vision. This set consists of
• Authorities of department and leaders responsible for
IAS integration
• Clarity of responsibilities of all the staff
• The way of human resources development
• Staff satisfaction with integration
Infrastructure and data, corresponding to Resource
View and Information View in GERAM, is the necessary
infrastructure and data resource to support research,
manufacture, management, and other business activities. It is
constituted of the factors：
• Industrial facilities and information system
• Information resource planning and data standardization
• Information management
• Information security
Investment measures the amount of money that
enterprise invest in IAS integration, the factors are as
bellows:
• Automation resources investment
• Investment of Information system construction
• Investment of Information system maintenance
• R&D investment
Performance Indicators aim to evaluate to what extent
enterprise achieves its economic target under the present IAS
integration circumstance. These indicators are structured by
Balance Score Card (BSC), i.e., grouped into four clusters:
financial, customer, Process, and innovation and learning,
and indicators in each cluster may impact on others, which
form the cause and effect chain of balanced scorecards
(Fig.3). Each cause and effect chain reflects a particular
strategy, through which we can determine indicators
coherence with enterprise strategy.
Noted that the criteria in EIA are not independent and
each criterion has impacts on others. The interrelations of
these criteria form a system of mutual influences that makes
the consequence bias when using methods like weighted
addition to comprehensive evaluate. In this context, methods
considering these interrelations such as ANP are better for This work is sponsored by the China High-Tech 863
evaluation. Program, No 2001AA415340 and No.2007AA04Z1A6, the
China Natural Science Foundation, No.9072007, and the
Performance
 Indicators
 BSC Aviation Science Foundation of China, No.20100758002
and 20128058006.
Financial
Customer REFERENCES
[1] Chen, David, Bruno Vallespir, and Nicolas Daclin. "An Approach for
Enterprise Interoperability Measurement." MoDISE-EUS. 2008.
Process [2] Torres, Jose M., et al. "Managing information systems security:
critical success factors and indicators to measure effectiveness."
Innovation& Information Security. Springer Berlin Heidelberg, 2006. 530-545.

Learning
Figure 3. Balance Scorecard
IV. SUMMARY AND CONCLUSION
This paper proposed a meta-model for system evaluation
based on principles of IDEF0. We developed an Evaluation
Indicators Architecture for Industrial Automation System
integration that was instantiated from the evaluation
meta-model. The EIA is constructed based on the idea of
management process. The GERAM is used as a reference to
identify elements included in EIA.
In the future, more research is needed to discover the
methods suitable for indicators architecture with
interrelations between each criteria and find quantitative
relations among the indicators by analyzing data collected
from enterprises. This may provide a more accuracy
instruction to managers for IAS integration progress.
ACKNOWLEDGMENT
[3] Liang, Chao, and Qing Li. "Enterprise information system project
selection with regard to BOCR." International Journal of Project
Management 26.8 (2008): 810-820.
[4] Li, Jun, et al. "Performance evaluation for Industrial Automation
System Integration based on enterprise architecture standards and
application in Cotton Textile Industry." System Science, Engineering
Design and Manufacturing Informatization (ICSEM), 2011
International Conference on. Vol. 2. IEEE, 2011.
[5] Q. Li and Y.L. Chen in: Modelling and Analysis of Enterprise and
Information System from requirements to realization, Springer and
High Education Press (2009)
[6] ISO15704: Automation systems and integration — Framework for
enterprise architectures and models, (2009), p. 50-64.
[7] Verdecho, María-José, et al. "A multi-criteria approach for managing
inter-enterprise collaborative relationships." Omega 40.3 (2012):
249-263.
[8] Li, Jun, Qing Li, and Jian Zhou. "Evaluation Indicator Architecture
Development for Industrial Automation Systems Using ISO15704
Annex B: GERAM." Key Engineering Materials 474 (2011):
466-471.
[9] Morton, Michael S. Scott, ed. The corporation of the 1990s:
Information technology and organizational transformation. Oxford
University Press, 1991.
[10] Moen, Ronald, and Clifford Norman. "Evolution of the PDCA cycle."
(2006).

1085