See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/276373845

Organizational Reliability Capability Assessment: A Case Study in China R&D

Enterprise for Aviation Products

Article  in  IEEE Transactions on Reliability · June 2015

DOI: 10.1109/TR.2015.2394360

CITATIONS READS

4 750

3 authors, including:

Xing Pan Li Guosheng

Beihang University (BUAA) Huazhong University of Science and Technology
71 PUBLICATIONS   304 CITATIONS    2 PUBLICATIONS   5 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

PH.D paper View project

All content following this page was uploaded by Xing Pan on 24 April 2016.

The user has requested enhancement of the downloaded file.

550 IEEE TRANSACTIONS ON RELIABILITY, VOL. 64, NO. 2, JUNE 2015

Organizational Reliability Capability Assessment:

A Case Study in China R&D Enterprise
for Aviation Products
Xing Pan, Member, IEEE, Ziling Xin, and Guosheng Li

Abstract—The IEEE Standard 1624 provides a foundation for RSE-CMM Reliability Systems Engineering Capability
the assessment of organizational reliability capability (ORC). Maturity Model
However, to better match the individual characteristics of different WBS Work Breakdown Structure
industries, this standard needs to be suitably clipped and adjusted,
so as to develop customized models for the reliability capability SEM Structural Equation Modeling
evaluations of different organizations. To evaluate the ORC of AMOS Analysis of Moment Structures
China's aviation enterprises effectively, we propose a reliability
engineering capability maturity model (RE-CMM) for China's Test Chi-Square Test
aviation industry, with adequate consideration of the features Ratio of Chi-Square and Degree of Freedom
closely associated with China's aviation enterprises and China's
military standards. The proposed RE-CMM model is established GFI Goodness of Fit Index
on the theoretical basis of Capability Maturity Model Integration AGFI Adjusted Goodness of Fit Index
(CMMI), combined with the IEEE standard and China's military
standards for Reliability, Maintainability, and Supportability RMR Root Mean Square Residual
(RMS). Structural equation modeling (SEM) is used to verify the NFI Normed Fit Index
validity of the RE-CMM model. We use a fuzzy method for the
evaluation of reliability engineering capability. Detailed steps of CFI Comparative Fit Index
reliability engineering capability assessment are also provided for RMSEA Root Mean Square Error of Approximation
the application of the model. Finally, a case study on a Chinese
aviation enterprise is given to illustrate the model and method. CR Critical Ratio
Index Terms—Fuzzy integral, multiple index evaluation, relia-
bility capability, triangular fuzzy number.
I. INTRODUCTION

ABBREVIATIONS AND ACRONYMS

R&D Research and Development
ORC Organization Reliability Capability
REC Reliability Engineering Capability
CMM Capability Maturity Model
CMMI Capability Maturity Model Integration
RE-CMM Reliability Engineering Capability Maturity
Model
Manuscript received March 15, 2014; revised August 10, 2014, September
17, 2014, and October 27, 2014; accepted December 31, 2014. Date of publica-
tion February 12, 2015; date of current version June 01, 2015. This work was
supported in part by the National Natural Science Foundation of China under
Grants No. 70901004/No. 71171008, the in part by China Scholarship Council
and the Fundamental Research Funds for the Central Universities. Associate
Editor: C. Smidts.
X. Pan is with the School of Reliability and Systems Engineering, Beihang
University, Beijing 100191, China (e-mail: panxing@buaa.edu.cn).
Z. Xin is with the North China Institute of Computing Technology, Beijing
100083, China (e-mail: zilinxin@dse.buaa.edu.cn).
G. Li is with the Wuhan Maritime Communications Research Institute,
Wuhan 430079, China (e-mail: guoshengli@dse.buaa.edu.cn).
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TR.2015.2394360
T HE product research and development (R&D) process
is often very comprehensive and complex, as there are
many factors that can impact the reliability of final prod-
ucts, including manpower organization, process management,
and technical risks. Process capability can indicate product
reliability, and a direct correlation exists between an organ-
ization's capability maturity and an organization's capability
to develop mature, reliable products [1], [2]. So improvement
in reliability-related work in the product R&D process will
be helpful in increasing product reliability. Understanding
the reliability-related criteria of an organization, and how the
organizational reliability capability (ORC) is evaluated. are
important to improve organizational reliability-related work.
ORC is a measure of the effectiveness of an organization's
reliability program, practices, and activities that could help in
meeting a customer's requirements for product reliability. The
IEEE Standards Board in 2008 formulated the ORC standard
[3], which divides ORC into five levels. The standard has
established an ORC maturity model to assess ORC through
three dimensions of capability: product demand capability,
engineering capability, and feedback process capability, in-
cluding eight key practices, and a number of process activities.
The model provides a basic standard for the assessment of the
enterprise's ORC.

0018-9529 © 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
PAN et al.: ORGANIZATIONAL RELIABILITY CAPABILITY ASSESSMENT
The assessment of ORC brings many advantages, such
as specifying or planning reliability practices throughout an
organization, and evaluating reliability practices to determine
the extent to which the organization is capable of providing a
product that meets the reliability requirements [1]. Generally,
the evaluation for reliability capability of an organization can
be used as a criterion for supplier selection; meanwhile, a
self-evaluation can be used to find an effective path for the im-
provement of an enterprise's reliability-related work. Currently,
quite a few methods have been presented for the assessment of
ORC with respect to the characteristics of different industries.
Sanjay and Michael [4]–[8], and Gullo et al. [9] established
a reliability capability assessment methodology for electronic
product manufacturers to help them carry out reliability-related
work, and to evaluate their reliability capabilities so that they
could improve their reliability-related works and choose sup-
pliers effectively. Williams et al. [10] presented a reliability
capability evaluation method, and several improvement strate-
gies for subsea equipment suppliers. To assess the ORC of an
organization, these above methods all attempted to output a
single level number through an evaluation process. However,
because defining reliability capability evaluation output by a
single level number is complex and tedious, these methods
gave the ORC level number of an organization directly by
expert judgment, guided by many reliability-related criteria
which indicate different capability levels.
IEEE Standard 1624 presented a model ORC that has been
applied to several industrial areas, and it is suitable for man-
ufacturing enterprises, especially for electronic manufacturers
[3]–[9]. However, to better serve China's aviation industry, the
reliability capability assessment methods related to ORC need
to improve in two ways. First, it should be extended to take into
account other characteristics, such as maintainability and sup-
portability. Second, a quantitative evaluation method should be
developed to give more accurate results. The Reliability Engi-
neering Capability Maturity Model (RE-CMM) [11] and the Re-
liability Systems Engineering Capability Maturity Model (RSE-
CMM) [12] use the reliability-related work listed in China's na-
tional military standard as key practices. However, they do not
provide an index system for the evaluation, and do not take the
impact of maintainability and supportability on the quality of
aviation products into consideration, which makes these models
unsatisfactory for China's aviation enterprises.
In this paper, we have combined the IEEE standard for ORC
with China's national military standard requirements to develop
an evaluation method for organizational Reliability Engineering
Capability (REC). Previously, Li and Pan [13] had built an index
system for the evaluation method with emphasis on reliability
technology. However, they ignored the relation between relia-
bility management and reliability-related work, which is impor-
tant for conducting an accurate evaluation. Furthermore, their
model has not been validated, and therefore the outcomes of the
assessment are not reliable. To validate the assessment model,
Sanjay and Michael [8] used a quantitative assessment process.
In this paper, we investigate the reliability engineering prac-
tices of China's aviation R&D enterprise, and present a relia-
bility engineering capability maturity model for China's avia-
tion R&D enterprise. A Structural Equation Modeling (SEM)
551
is developed to verify the model. Then, to eliminate inconsis-
tency and inter-subjectivity of evaluation results conducted by
different experts with different context on a same organization, a
fuzzy evaluation method and process for reliability engineering
capability based on triangular fuzzy numbers is provided. Fi-
nally, a case study of China's aviation R&D enterprise is used
to illustrate the application of the evaluation method.
II. RELIABILITY ENGINEERING CAPABILITY MATURITY
MODEL FOR AVIATION R&D ENTERPRISE
In general, the whole life cycle of a system or product gen-
erally includes six phases: conceptual design, preliminary de-
sign, detail design, production, product operation, and phase out
and disposal. China aviation enterprises generally consider re-
liability-related work in the first three phases [14]–[16]. There-
fore, the reliability-related work has a significant impact on the
reliability of final products. Taking into account the special fea-
tures of aviation products, reliability, maintainability, and sup-
portability are considered separately, and therefore cannot meet
the needs of the aviation industry. This paper, based on the
ORC model of IEEE Standard 1624, combined with the char-
acteristics of China's aviation enterprises, we have established
a reliability engineering capability maturity model for China's
aviation industry that includes reliability, maintainability, and
supportability.
A. The Concept of Reliability Engineering Capability
Reliability engineering capability is a measure of the relia-
bility-related works within an organization that contributes to
the reliability of the final product and the effectiveness of this
work in meeting the reliability requirements of customers. As-
sessing the REC maturity of an organization is necessary to de-
termine if an organization knows what reliability-related work it
is doing, and if it understands what that reliability-related work
is expected to do.
In aviation product R&D, carrying out reliability activities
requires technology, management, and other tools. These
tools require an organization to apply effective management
methods, so as to improve the efficiency of reliability-re-
lated work. Therefore, this paper, according to the areas that
reliability engineering activities cover, classifies reliability en-
gineering capability into four kinds of capability: organization
capability, management capability, technology capability, and
support capability, as shown in Fig. 1.
B. Reliability Engineering Capability Maturity Model
Framework
According to the four sub-capabilities that reliability
engineering capability contains, the reliability engineering ca-
pability maturity model was created. Process areas of reliability
engineering capability include four categories: engineering or-
ganization, engineering management, engineering technology,
and engineering support.
Each process area category includes a number of process
areas, and each process area contains a number of process prac-
tices. All process areas and process practices contain five ca-
pability maturity levels, and all process practices in each ca-
pability maturity level contain some objectives and activities
552
Fig. 1. Reliability engineering capability classification.
Fig. 2. Reliability engineering capability maturity model framework.
that are designed to achieve the intended capability maturity
level. These objectives and activities are evaluation rules that
evaluate the process practice maturity level. Limited in space,
this paper does not list each evaluation rule in detail. When
all activities and objectives of one process practice are met in
one capability maturity level, the process practice achieves the
capability maturity level. All process practice capability matu-
rity levels are integrated into the upper phase to get the process
area capability maturity levels, and the process area category
capability maturity levels. Then the reliability engineering ca-
pability maturity model of the aviation R&D enterprise can be
reached. The reliability engineering capability maturity model
proposed in this paper focuses on improving the reliability-re-
lated work process for the Chinese aviation enterprise, so as
to improve the reliability engineering capability level. The re-
liability engineering capability maturity model framework is
shown in Fig. 2.
C. Process Areas and Process Practices in the Reliability
Engineering Capability Maturity Model
Process area and process practice are the key indices of eval-
uating reliability engineering capability. The determination of
process area and process practice should reflect the actual situ-
ation of product development, namely, which practices have a
IEEE TRANSACTIONS ON RELIABILITY, VOL. 64, NO. 2, JUNE 2015
positive impact on reliability engineering capability in the R&D
product process.
A comprehensive evaluation index system that reflects avi-
ation R&D enterprise reliability engineering capability is the
basis of evaluating reliability engineering capability correctly.
Due to many reliability activities in product R&D, reliability,
maintainability, and supportability are blended into product
research and development, with reliability being considered
less in the manufacturing stage. This paper mainly considers
reliability activity characteristics, and establishes a reliability
engineering capability index system that contains maintain-
ability and supportability. The existing Chinese military
standards formulate specific reliability, maintainability, and
supportability requirements in product R&D [14]–[16]. These
requirements ensure that new products meet their reliability,
maintainability, and supportability requirements; and that ex-
isting products achieve improved reliability, maintainability,
and supportability levels so as to meet the requirements of
system task success, reduce the need for support resources,
and reduce life cycle cost. Therefore, this paper, based on
China's military standards for RMS engineering, combines re-
liability capability standards formulated by the IEEE reliability
standards committee with the reliability-related work charac-
teristics of China's aviation R&D enterprise. This combination
PAN et al.: ORGANIZATIONAL RELIABILITY CAPABILITY ASSESSMENT 553

TABLE I
PROCESS AREAS AND PRACTICES OF THE RELIABILITY ENGINEERING CAPABILITY MATURITY MODEL

determined the process areas and process practices, as shown

in Table I.

III. VALIDATION OF RELIABILITY ENGINEERING

CAPABILITY MATURITY MODEL
The basic objective of any measurement tool is to render a
measured value that is similar to the true value. In a reliability
engineering capability maturity model for the aviation R&D en-
terprise, there are many variables that cannot be measured di-
rectly, and the model can only use some of its related observ-
able variables (also known as manifest variables) as the identity
of the unpredictable variables (also known as latent variables).
It is a hard problem to consider whether manifest variables can
reflect the information of latent variables truly and reliably or Fig. 3. Path diagram of the SEM.
not as alternatives to latent variables. To solve this problem, the
Structural Equation Model (SEM) is used to find the relation-
ship between the manifest variables and the latent variables to observed variables. The structural model refers to the relation-
verify the validation of the whole model. ships among the latent variables. The SEM is essentially a type
The SEM is usually used to study the structural relationship of verification model that establishes a theoretical hypothesis
among latent variables through latent variables [17]. Therefore, through empirical data, and verifies the theoretical hypothesis
this method can easily find relationships between each latent through data collection. Such verification is intended to com-
variable and its significant variable sets. In the SEM, there are pare the difference between the covariance matrix of the hypo-
three types of variables: latent variables, observed variables, and thetical model proposed by the researchers and the actual co-
unique variables, as shown in Fig. 3. Latent variables are im- variance matrix derived through empirical data collection. The
possible to measure, and are shown with ovals in the SEM. Ob- steps that use the SEM to verify the reliability engineering capa-
served variables (also known as measurement variables or man- bility maturity model for aviation R&D enterprises are describes
ifest variables), which are question items in a questionnaire, can in the following subsections.
be measured directly, and are shown with rectangles in the SEM.
Each observed variable will have deviation variables that cannot A. Model Setting
be measured, and are shown with circles in the SEM. The SEM can find the relationship among latent variables,
An SEM path diagram includes two sub-models: a measure- and configuring latent variables is the basis of the SEM.
ment model, and a structural model. The measurement model Through an extensive literature search, and empirical research,
refers to the relationship between the latent variables and the it has been determined that reliability engineering capability
554 IEEE TRANSACTIONS ON RELIABILITY, VOL. 64, NO. 2, JUNE 2015

Fig. 4. Initial SEM of the reliability engineering capability maturity model.

includes organization capability, management capability, • H5. Engineering organization has a positive impact
technology capability, and support capability, each of which has on engineering management . When organizational
an impact on reliability engineering capability. Therefore, in capability is improved, management capability level will
a reliability engineering capability maturity model evaluation rise to some extent.
index system, the four process area categories are engineering • H6. Engineering support has a positive effect on en-
organization , engineering management , engineering gineering management . When hardware and software
technology , and engineering support , each of which tools and knowledge management levels increase, the en-
has an impact on target variable reliability engineering capa- gineering management level will rise too.
bility. To establish the SEM of the REC of China's aviation • H7. All the process areas in the process area category have
enterprises, we have the following hypotheses. positive effects on the process area category.
• H1. Engineering organization provides effective or- Based on the above hypotheses, the initial structural model
ganizational support for the implementation of engineering of the reliability engineering maturity evaluation index system
technology . That is to say, engineering organization is established, as shown in Fig. 4.
not only affects reliability engineering capability ,
but also affects the implementation of engineering tech- B. Data Collection
nology . Engineering organization has a positive Before we evaluated the REC of a Chinese aviation enterprise
impact on engineering technology , and reliability en- in our case study, we performed a validation of the REC model
gineering capability . based on China's aviation industry reliability engineering prac-
• H2. Engineering management provides scientific tices. A questionnaire was developed to collect data. The per-
and effective management for the implementation of sonnel involved in the investigation were mainly engaged in re-
engineering technology . That is to say, engineering liability design, research, development, and other relevant work
management not only affects reliability engineering ca- in China's aviation industry. The sample size requirements can
pability directly, but also affects it indirectly through be referred to as an analysis of relevant statistical theory. When
the implementation of engineering technology . En- SEM analysis is commonly used to analyze the initial model,
gineering management has a positive impact on reliability due to the validity of the test, the appropriate sample size
engineering capability , and engineering technology should be 100 to 200. Therefore, the survey questionnaires were
. distributed to 185 people, and a total of 173 valid questionnaires
• H3. Engineering support provides strong support were returned; thus, the effective rate of the questionnaire was
for hardware and software for the implementation of 93.51%.
engineering technology , and affects reliability engi-
neering capability directly. That is to say, engineering C. SEM Analysis of the Initial Model
support has a positive impact on reliability engi- In the SEM, it attempts to calculate the model parameters that
neering capability , and engineering technology . make the smallest differences between the sample variance-co-
• H4. Engineering technology provides advanced variance matrix and the theoretical model variance-covari-
technical support for high reliability product development ance matrix through parameter estimation. If the theoretical
under the effect of another three process area categories. model structure for the collected data is reasonable, then the
That is to say, engineering technology has a positive sample variance-covariance matrix with the theoretical model
impact on reliability as a mediating variable. variance-covariance matrix is not very different; specifically,
PAN et al.: ORGANIZATIONAL RELIABILITY CAPABILITY ASSESSMENT 555

TABLE II is the basis of reliability engineering capability evaluation for

FIT INDEX China's aviation R&D enterprise.

IV. RELIABILITY ENGINEERING CAPABILITY

FUZZY EVALUATION METHOD

A. The Theory of Fuzzy Evaluation

TABLE III
FIT DEGREE TEST OF REC MATURITY MODEL
each element of the residual matrix is close to 0. The fit
index is built based on this idea. Analysis of Moment Structures
(AMOS) provides several fit indexes, as shown in Table II. If the
model is not fitted well, the model needs to be modified.
Note that the fit index is designed to examine the fit degree
between the theoretical model and data, and not to determine
whether the model is good as the only basis. Sometimes the fit
index shows that the model is not optimal. However, if it can
explain the model well by corresponding theories, the fit index
of the model need not be the best. According to the theoretical
assumption of the first step, AMOS is used to test the fit degree
of the initial SEM, and the test results are shown in Table III.
See that the fit indexes meet the requirements. So from the
perspective of fit degree, the initial model can be accepted. After
fit degree testing, SEM also needs to conduct a path load factor
significance test. The path load factor reflects the relationship
of two variables and the connecting path. When the significance
test is done, the original hypothesis for the load factor is 0; vari-
ables are not relevant. AMOS uses CR (Critical Ratio, Z sta-
tistics), and the statistical accompanied probability P of CR to
conduct a significance test of the path load factor. The typically
used value of P is 0.05. If P is less than 0.05, the initial hy-
pothesis is rejected. There is no reason to believe that the corre-
sponding load factor is 0. The standardized path load factor test
results for the reliability engineering capability maturity model
are shown in Table IV. In the table, the symbol “ ” signifies
that P is far less than 0.05, and the symbol “ ” and the symbol
“ ” indicate the effect directions.
As can be seen from the table, although some standardized
path load factors were more than 0.05, almost all the path load
factors were around 0.05. The results show that the corre-
sponding path existed in the 95% confidence level, and the load
factor of each path was positive. This result means that there
is a positive correlation between variables, which is consistent
with the hypothesis. So the paths of the original model are
reasonable.
Through the steps above, the validation of the RE-CMM is
verified. From the results, it can be seen that the initial relia-
bility engineering capability maturity model is reasonable and
acceptable. At the same time, the path load factor acquired from
the SEM analysis can be a weight factor of the variable, which
1) Linguistic Variable: Linguistic variables turn natural lan-
guage words into measurable values. For example, worse, bad,
general, good, better are used to express the experts' feeling to
the evaluation index. Linguistic variables can be used to express
these subjective judgments appropriately to deal with unclear or
vague information. Trapezoidal fuzzy numbers and triangular
fuzzy numbers are usually used to represent the membership
function of the linguistic variable. This paper uses triangular
fuzzy numbers to evaluate reliability engineering capability.
2) Triangular Fuzzy Numbers: Triangular fuzzy numbers are
commonly used as fuzzy numbers. A triangular fuzzy number
can be expressed as , and its membership function can
be defined as
(1)
.
The two triangular fuzzy numbers and
have addition and multiplication regulation:
(2)
(3)
3) Making a Triangular Fuzzy Number a Clear Value:
There are many ways to transform the linguistic variables or
fuzzy numbers into a clear value. This paper uses a central
value method, which is as follows. From the overall view of
triangular fuzzy numbers, the center is better able to express
importance. Therefore, a fuzzy number (take for
example) can be transformed into a clear value:
(4)
where refers to a clear value that the triangular fuzzy number
is transformed into.
4) Fuzzy Measure, and Fuzzy Integral:
a) Fuzzy measure: A fuzzy measure, which is the improve-
ment of the classical probability measure, replaces the addi-
tive condition in classical probability with monotonicity which
has weak constraints. Typically, when fuzzy measures are used
in decision making, the candidate set represents the evaluation
items, and fuzzy measure is the weight of the evaluation items.
is assumed to be a non-empty set, and is a -algebra,
which is composed of some sub-set of .
Definition 1: Assume the mapping satisfies
these conditions.
• Boundedness: .
• Monotonicity: , if , then .
• Continuity: If , and is monotonous,
namely or
, then .
556 IEEE TRANSACTIONS ON RELIABILITY, VOL. 64, NO. 2, JUNE 2015

TABLE IV
STANDARDIZED PATH LOAD FACTOR TEST OF REC MATURITY MODEL

Then is called a fuzzy measure. is called a Currently, the most widely used type of fuzzy measure used is
fuzzy measurable space. is called a fuzzy mea- the fuzzy measure, which uses to express an additive degree.
sure space. Its value is limited by .
PAN et al.: ORGANIZATIONAL RELIABILITY CAPABILITY ASSESSMENT
TABLE V
SET PRINCIPLE OF IMPORTANCE DEGREE AND
Definition 2: Assume the fuzzy measure meets the fol-
lowing additional properties.
If , then
, where . is called the fuzzy
measure, or measure, where, and are called
fuzzy densities.
If is a finite set, and the fuzzy density
function of variable is , then is
(5)
in (5) is determined by experts based on the purpose of the
evaluation. As there is a dependency relationship between de-
termining by expert scoring and the importance degree of the
evaluation index, the importance degree can be used to deter-
mine , which can meet a variety of evaluation purposes. Spe-
cific set principles of the importance degree and the value are
shown in Table V.
The main idea of the calculated fuzzy measure is that, ac-
cording to the purpose of evaluation and the determined index
weight (fuzzy density) of the reliability engineering capability
by the survey results, the experts judge the scores to determine
through set principles of importance degree and . And then
(5) is used to calculate the initial fuzzy measure, which is nor-
malized to get fuzzy measures. The fuzzy measure method
can meet an actual evaluation objective by setting different .
b) Fuzzy integral: A fuzzy integral is a nonlinear func-
tion defined on the basis of a fuzzy measure. Examples include
Sugeno fuzzy integral [18], and the Choquet fuzzy integral [19].
This study used the Choquet fuzzy integral to evaluate reliability
engineering capability.
Definition 3: Make a hypothesis that is a fuzzy mea-
sure space, and is a measurable function of in . In
a hypothetical case, without loss of generality, if
557
Fig. 5. Choquet fuzzy integral comprehensive evaluation method.
, then the Choquet fuzzy integral for fuzzy
measure of at is defined as
(6)
Equation (6) is the basic content of the fuzzy integral eval-
uation method, where is the value of the fuzzy integral
evaluation, is the output value of the evaluation objects
at index , and represents the importance degree that
is taken into account, specifically
The relation of function and its fuzzy measure function
is as shown in Fig. 5. In the assessment of the REC, repre-
sents the performance of evaluation objects at a specific index,
namely, the score of the index; represents the subjective im-
portance degree that the corresponding index is considered, and
it can be determined by the weight (fuzzy density) of the index.
The fuzzy integral of and is a comprehensive evaluation
value of reliability engineering capability.
B. The Steps of Fuzzy Evaluation for Reliability Engineering
Capability
As the reliability engineering capability evaluation for
China's aviation R&D enterprises is a multi-criteria, multi-index
comprehensive evaluation, because the indexes are vague
and the values are difficult to quantify, and because there is
correlation between indexes, this paper used a fuzzy integral to
evaluate reliability engineering capability. A language-based
triangular fuzzy number was used to quantify qualitative in-
dexes, and a fuzzy integral was used to solve the correlation
between indexes, which enhanced the accuracy and reliability
of the evaluation results. Basic steps in fuzzy comprehensive
evaluation for reliability engineering are shown in Fig. 6.
Table V shows the evaluation index system for reliability
engineering. The steps are described as follows.
Step 1. Determine each index and its weight.
Process area categories, process areas, and process practices
in the reliability engineering maturity model are used as an eval-
558 IEEE TRANSACTIONS ON RELIABILITY, VOL. 64, NO. 2, JUNE 2015

all experts are the same. Triangular fuzzy numbers in are

calculated to get the fuzzy value set of the process practice.

(7)

where the element represents the value of process practice

Fig. 6. Flow of reliability engineering capability fuzzy evaluation.
TABLE VI
LINGUISTIC VARIABLE OF QUALITATIVE INDEX
after integrating experts views.
2.3. Fuzzy evaluation values are cleared to obtain an accurate
value set for process practice. Equation (4) is used to clear the
fuzzy number to get the accurate evaluation value set
for process practice, where represents the value of the
process practice .
Step 3. Use the fuzzy integral to determine the value of each
process area in the secondary index level.
3.1. Determine the fuzzy density of all indexes. Through
every index weight determined, it can get the fuzzy density
(weight) of every index , where represents the
fuzzy density set in the third index level, and represents
fuzzy density of process practice . The fuzzy density of all
indexes can be reached by the same method.
3.2. Determine in fuzzy measures. experts determine
according to index importance, and the set principle in
Table V through the questionnaire data. Then the value can
be found by the formula
(8)

where represents the value determined by expert .

3.3. The evaluation values of all process practices that
belong to the process area are re-sorted by size as
, where , and
represents that there are process practices belonging to .
3.4. Equation (5) is used to get the fuzzy measure of ,
which is normalized to obtain
uation index to assess the reliability engineering maturity level
(see Table I). And according to the path load factor of SEM anal-
ysis results in Section III, the weight of each index can be deter-
3.5. Obtain the evaluation value of process area through
mined. In the issue of reliability engineering capability assess-
a fuzzy integral with (6):
ment, the process practices are the third-level rank
indexes, the process areas are the second-level rank
indexes, the process area category is the first rank
index, and reliability engineering capability is the final rank
target.
Step 2. Determine the evaluation values of the process prac- 3.6. Repeat steps (1) through (5) to get evaluation value set
tices in the third rank index level. of all process areas in the second index level.
2.1. After the investigation through the questionnaire, the 5 Step 4. Determine the evaluation values of all process area
ranked results of process practices were transformed into a lin- categories in the first rank index level.
guistic triangular fuzzy set, . This Using the results from Step 3, and repeating, the evaluation
transformation reflects the reliability engineering capability of value set ( ,2,3,4) of all process area categories
China aviation enterprises, where represents experts in- can be calculated, where represents the evaluation value of
volved in the investigation, and represents expert evalu- process area category .
ates process practice to get a triangular fuzzy number. In the Step 5. Comprehensive evaluation.
linguistic variables table, fuzzy ratio-scales correspond to the Using the results from Step 3, the fuzzy density of process
five ranks in the questionnaire, as shown in Table VI. area category ( ,2,3,4) can be reached. Using the fuzzy
2.2. Fuzzy values of all process practices are calculated integral, repeating the same method of Step 3, and combining
through comprehensive opinions by all experts. The weights of with the evaluation value ( ,2,3,4) of all process
PAN et al.: ORGANIZATIONAL RELIABILITY CAPABILITY ASSESSMENT 559

TABLE VII TABLE IX

INDEX RANK DETERMINATION EFFECT AND WEIGHT OF PROCESS AREA

TABLE VIII
EFFECT AND WEIGHT OF PROCESS CATEGORY

TABLE X
EFFECT AND WEIGHT OF PROCESS PRACTICE

area categories found by Step 4, the evaluation value of re-

liability engineering capability for China's aviation R&D enter-
prise can be obtained.
Step 6. Rank determination.
Capability maturity level can be determined by contrasting
with the rank determination table, as shown in Table VII.
The capability levels of all process practices and reliability
engineering capability maturity levels for China's aviation R&D
enterprise can be reached by the six steps above. The fuzzy in-
tegral evaluation method can solve the problem that qualitative
indexes are difficult to quantify, and that there is correlation be-
tween indexes, which makes the evaluation results more accu-
rate and reliable.

V. A CASE STUDY OF CHINA AVIATION R&D ENTERPRISE

As high input cost, high value-added, and high-risk products,
aviation products usually are asked to have a high reliability
level. The reliability engineering capability of the aviation R&D
enterprise reflects the ability of the aviation R&D enterprise
to meet customer requirements in reliability engineering, and
can measure to what extent the reliability technology, organiza-
tion, and management of the aviation R&D enterprise meet cus-
tomers' requirements. Reliability engineering capability evalu-
ation is a method for choosing an aviation R&D enterprise, and
reliability engineering capability self-evaluation can help an en-
terprise to improve reliability engineering capability. Therefore,
effective reliability engineering capability evaluation for avia-
tion R&D enterprises is of great significance. One Chinese avi-
ation R&D enterprise intended to evaluate its reliability engi-
neering capability so as to find shortcomings, and improve its re-
liability-related work. Based on the reliability engineering ma-
turity model, and the evaluation method mentioned above, the
reliability engineering capability of the enterprise can be eval-
uated. The first step is to make investigations, and collect and
analyze data on the enterprise. The next step is to use the re-
liability engineering capability evaluation method to evaluate
reliability capability. Additional steps are described below.
Step 1. Determine index weight.
In Section III, the validation of the REC model was verified
through 173 empirical data, and these data also can be used to
determine index weight in the model. According to path load
factor of the SEM analysis results in Section III, each index
weight can be found. Path load factors represent the effects that
independent variables (lower level variables in the REC model)
make on a dependent variable (upper level variable in the REC
model). The effects include direct and indirect effects, the sum
of which represents the ultimate effect of independent variables
on a dependent variable. Path load factor normalization can be
used to find the relative weight of each variable.
The effects that four process area categories make on reli-
ability engineering capability and corresponding weights are
shown in Table VIII.
There are only indirect effects that each process area makes
on target variable reliability engineering capability. The effects
and weights of all process areas are shown in Table IX.
Process practices are observational variables in SEM, which
have direct effects on upper latent variables. The weight of all
560
TABLE XI
EVALUATION VALUE AND RANK OF PROCESS PRACTICE AND PROCESS AREA
TABLE XII
EVALUATION VALUE AND RANK OF TARGET
VARIABLE AND PROCESS AREA CATEGORY
process practices can be directly derived from the SEM. The
effects and weights of process practices are shown in Table X.
Step 2. Determine the evaluation value of each process
practice.
To assess the reliability capability level of the aviation R&D
organization, another questionnaire was developed and con-
ducted by 10 experts to judge the fuzzy value of each process
practice in Table I by using linguistic variables in Table VI.
The Evaluation value set of all process practices
can be found by using Step 2 of Section IV through sorting the
inventory data.
Step 3. Determine the evaluation value of all process areas in
the fuzzy integral method.
With the calculation results of Step 1 and Step 2, refer to Step
3 of Section IV. The evaluation value set of all
process areas can be found.
Step. 4. Determine the evaluation value of all process area
categories in the fuzzy integral method.
IEEE TRANSACTIONS ON RELIABILITY, VOL. 64, NO. 2, JUNE 2015
With the calculation results of Step 1 and Step 3, repeat Step
3, with evaluation value set ( ,2,3,4) for all
process area categories that can be found, where represents
the evaluation value of .
Step 5. Comprehensive evaluation.
With a fuzzy density of the process area category (
,2,3,4) created by step one, and evaluation value
( ,2,3,4) of the process area categories created by Step 4,
the evaluation value of the reliability engineering capability
can be found using the same fuzzy integral method.
Step 6. Grade determination.
The capability maturity level of all indexes can be found by
contrasting , , , and with
the level determination in Table VII. The detailed results can be
seen in Table XI and Table XII, as follows.
From the results above, it can be seen that the aviation en-
terprise's reliability engineering capability maturity is in level
1. The enterprise can analyze existing problems, and improve
its reliability engineering capability according to the results and
analysis above.
VI. CONCLUSIONS
Assessing the REC of an aviation R&D enterprise effectively
has important significance to improve its REC, so as to improve
reliability technology, organization, and management works for
aviation R&D enterprise. In addition, REC provides important
help on the selection of suppliers.
This paper, based on a Chinese aviation R&D enterprise reli-
ability engineering capability evaluation case study, combined
the reliability capability evaluation model of the IEEE Stan-
dards Committee, integrated maintainability and supportability
into reliability, and considered reliability technology and man-
agement. The paper proposed a reliability engineering maturity
model for a Chinese aviation R&D enterprise, and used a fuzzy
evaluation method to quantify the evaluation of reliability engi-
neering capability to improve reliability-related work.
ACKNOWLEDGMENT
This work was conducted during Dr. Pan's visit to the De-
partment of Systems and Industrial Engineering, University of
Arizona, as a visiting scholar. The authors wish to thank Dr.
Varlerdi and Benjamin Reidy, who have provided valuable and
helpful revisions of this work.
REFERENCES
[1] L. J. Gullo, “Advantages of IEEE P1624 for assessing organizational
reliability capability,” in Proc. Annu. Reliability and Maintainability
Symp., 2009 (RAMS 2009), 2009, pp. 285–291.
[2] B. Ramakrishnan, P. Sandborn, and M. Pecht, “Process capability
indices and product reliability,” Microelectron. Rel., vol. 41, pp.
2067–2070, 2001.
[3] IEEE Std. 1624, IEEE Standard for Organizational Reliability Capa-
bility, IEEE Reliability Standards Committee, 2008.
[4] S. Tiku, M. Azarian, and M. Pecht, “Using a reliability capability ma-
turity model to benchmark electronics companies,” Int. J. Qual. Rel.
Manage., vol. 24, pp. 547–563, 2007.
[5] S. Tiku and M. Pecht, “Reliability capability assessment method-
ology,” in Proc. Int. Tech. Symp. Packaging, Assembling and Testing
& Exhibition, Brazil, 2003, pp. 6–8.
PAN et al.: ORGANIZATIONAL RELIABILITY CAPABILITY ASSESSMENT
[6] S. Tiku and M. Pecht, “Organizational reliability capability,” in Proc.
Can. Reliability and Maintainability Symp., Ottawa, ON, Canada,
2003.
[7] S. Tiku and M. Pecht, “Auditing the reliability capability of electronics
manufacturers,” in Proc. Int. Electronic Packaging Technical Conf.
Exhib., Maui, HI, USA, 2003.
[8] S. Tiku and M. Pecht, “Validation of reliability capability evaluation
model using a quantitative assessment process,” Int. J. Qual. Rel.
Manage., vol. 27, pp. 938–952, 2010.
[9] L. J. Gullo, M. H. Azarian, D. Das, F. Schenkelberg, and S. Tiku,
“Assessment of organizational reliability capability,” IEEE Trans.
Compon. Packag. Technol., vol. 29, pp. 425–428, 2006.
[10] K. Williams, N. Robertson, R. C. Haritonov, and J. Strutt, “Reliability
capability evaluation and improvement strategies for subsea equipment
suppliers,” Underwater Technol.: Int. J. Soc. Underwater, vol. 25, pp.
165–173, 2003.
[11] X. Wang, “Research of reliability engineering capability maturity
model,” M.S. thesis, Sch. Syst. Rel. Eng., Beihang Univ., Beijing,
China, 2006.
[12] J. Wang, “Research of reliability engineering capability maturity
model,” Ph.D. dissertation, Sch. Syst. Rel. Eng., Beihang Univ.,
Beijing, China, 2008.
[13] L. Guosheng, P. Xing, C. Wenbing, and M. Zhao, “Research of orga-
nizational RMS engineering capability assessment method,” in Proc.
2010 IEEE Int. Conf. Management of Innovation and Technology
(ICMIT), Singapore, 2010, pp. 1013–1018.
[14] GJB3872, Equipment Integrated Supportability General Requirements,
China National Military Standard, 1999.
[15] GJB450A, Equipment Reliability Work General Requirements, China
National Military Standard, 2004.
[16] GJB368B, Equipment Maintainability General Outline, China National
Military Standard, 2009.
[17] R. D. Bock and R. E. Bargmann, “Analysis of covariance structures,”
Psychometrika, vol. 31, pp. 507–534, 1966.
[18] M. Sugeno, Theory of Fuzzy Integrals and Its Applications. Tokyo,
Japan: Tokyo Inst. Technol., 1974.
View publication stats
561
[19] Z. Wang and G. J. Klir, Fuzzy Measure Theory. New York, NY, USA:
Springer, 1992.
Xing Pan received the B.S. degree in mechanical engineering, and the Ph.D. de-
gree in systems engineering from Beihang University (BUAA), Beijing, China,
in 2000 and 2005, respectively.
From 2005 to 2009, he was an Assistant Professor with the School of Re-
liability and Systems Engineering, Beihang University, Beijing, China. Since
2009, he has been an Associate Professor. From 2012 to 2013, he was a visiting
scholar at the Department of Systems and Industrial Engineering, University of
Arizona, Tucson, USA. His research interest includes reliability engineering,
systems engineering, and system risk analysis.
Dr. Pan is the representative of the School of Reliability and Systems Engi-
neering, Beihang University as an INCOSE Corporate Advisory Board member.
Ziling Xin received the B.S. degree in mathematics from the University of Sci-
ence and Technology Beijing, China, in 2011, and the M.E. degree in systems
engineering from Beihang University (BUAA), Beijing, China, in 2014.
Since 2014, she has been with the North China Institute of Computing Tech-
nology, Beijing, China. Her research interests are reliability engineering, sys-
tems engineering, and quality management.
Guosheng Li received the B.S. degree in industrial engineering from the Wuhan
University of Science and Technology, Wuhan, China, in 2007, and the M.E. de-
gree in systems engineering from Beihang University (BUAA), Beijing, China,
in 2011.
Since 2012, he has been with the Wuhan Maritime Communications Research
Institute, Wuhan, China. His research interests are reliability engineering and
systems engineering.