Received: 25 September 2018 Revised: 14 January 2019 Accepted: 4 February 2019

DOI: 10.1002/smr.2161

REVIEW ARTICLE

Process maturity models: Classification by application sectors

and validities studies

Donghun Lee1 | Ja‐Whan Gu2 | Ho‐Won Jung1

1
Korea University Business School, Seoul,
South Korea
2
KCA Co. Ltd., Seoul, South Korea
Correspondence
Ho‐Won Jung and Donghun Lee, Korea
University Business School, Anam‐dong 5Ka,
Sungbuk‐gu, Seoul 02841, South Korea.
Email: hwjung@korea.ac.kr; donghun.dh.
lee@gmail.com
Abstract
A variety of maturity models have been developed and evolved as a tool for
assessments/benchmarking and continuous improvements for the past two decades.
The rapidly increasing number of maturity models led to problems in locating proper
maturity models and identifying model validities. To identify maturity models and
their validities as well as a classification of models across industrial sectors, a mapping
study was applied to initially retrieved 2381 articles on maturity models published in
peer‐reviewed journals between 1993 and June 2015. We identified 194 maturity
models from 285 articles using a set of quality assessment steps. Among those
maturity models, besides the five partially validated ones, this study also identified
26 maturity models that were empirically tested on the basis of tested hypotheses
and associated validity types, such as predictive validity and unidimensionality. Addi-
tionally, despite the wide presence of maturity models in most application sectors,
validated models were observed only in five sectors, eg, IT, industrials, and general.
Practitioners and researchers as well as standard organizations can utilize our results
to find out maturity models meeting their needs, to identify their validities, and/or to
seek research opportunities.

KEY W ORDS

mapping study, maturity models, model building, model testing, validity

1 | I N T RO D U CT I O N

A variety of maturity models have been developed since the publication of capability maturity models (CMMs) (eg, capability maturity model for soft-
ware [SW‐CMM]1 in 1993 and capability maturity model integration [CMMI]2 in 2010). A maturity model is a benchmarking concept assuming that a
less‐effective organization or projects adopt the practices of an excellent organization or projects. Thus, it is a codification of best practices arranged
to a set of evolutionary improvement stages. Its rationale is that higher stages of maturity are associated with improved performance (eg, within
budget, on schedule, improved quality, and high‐level productivity) in processes and products. Most of maturity models assert that they have very
similar architecture to CMMs as an implicit evidence of their reliabilities and/or validities.
A report3 showed that more than 2200 CMMI appraisals (91% of CMMI for development4) were conducted around the world in 2016. CMMs'
success stories as a tool for process improvements and assessments/benchmarking motivated and encouraged practitioners and researchers to
develop maturity models across diverse industrial domains over software engineering. However, on account of the variety of existing maturity
models and the scarce evidence of their usefulness, domestic and international standard developers as well as practitioners and researchers
who seek to find their appropriate maturity models need answers for some questions, such as “what kinds of maturity models (i.e., model name)
have been published in the industrial domains that I am interested in?” and “which one has been empirically validated?”

J Softw Evol Proc. 2019;e2161. wileyonlinelibrary.com/journal/smr © 2019 John Wiley & Sons, Ltd. 1 of 30
https://doi.org/10.1002/smr.2161
2 of 30 LEE ET AL.

Mapping study (also referred to as a scoping study)5,6 and systematic literature reviews (SLRs)7 are known as appropriate methods to answer
those questions, to understand previous studies, and to accumulate knowledge on them. A few related studies on maturity models have been
reported. All of previous mapping studies8-10 showed that maturity model publications have been expanding beyond IT sector. On the other hand,
they raised concerns of the validity of models; several SLRs11-14 showed the status of maturity models in the limited domains such as business
process management (BPM) and usability, which would be an echo of the mapping studies. However, to the best of our knowledge, previous
studies did not provide the specific names of the maturity models used across domains and those that have been statistically validated. Thus,
we do not know what maturity models are available and/or empirically tested in the industrial domains of our interest.
Given the importance of the decisions influenced by assessment results, maturity model users including both contractors and acquirers must
be able to have confidence in the assessment results.15 The confidence mainly derives from the reliability and validity of the model as well as the
competence of the assessor. In this respect, users (practitioners and researchers) and standard developers of maturity models should first locate
maturity models they want and identify the validities in model building and testing of those models located and/or seek research opportunities in
the domains of their interest.
The purpose of this study is to identify maturity models and their validity tests across industrial domains as well as article publication trends
and research gaps. For this, we used a mapping study approach, a suitable technique to provide an overview of research trends (eg, publication
trends over time and topics covered in the literature) in a specific topic area.5,7 Two requirements for maturity models16-18 are used as “quality
criteria” in this mapping study. The first one is the requirements for building maturity models (ie, model‐building requirements) where behavioral
and/or design sciences were utilized.16,19 Maturity models structured and analyzed under this study are closely selected to pass the
model‐building requirements, the criteria of a mapping study. The other one, which is built on the first one, is the requirement for model validation
(ie, model‐testing requirements),16,20,21 whether maturity models were empirically evaluated as a means for validities.16
Results show that a total of 194 distinct models in 285 articles were associated with maturity models; these articles were extracted from 2381
initially retrieved articles between 1993 and June 2015. Among them, besides the five partially validated ones, 26 empirically tested maturity
models were also identified. In addition, although maturity models have been widely spread across most industrial sectors, validated models were
observed only in five sectors, eg, IT, industrials, and general. This study was conducted on the basis of the recommendations from mapping studies,
such as Kitchenham et al22 and Petersen et al.5
Section 2 of this study provides related works, and Section 3 briefly describes steps for our mapping study. The results and discussions are
presented in Sections 4 and 5. Lastly, Section 6 contains final remarks.

2 | R E LA T E D WO R K S

According to Kitchenham et al,7 there are two methods to answer a specific research question from articles published: A mapping study provides
an overview of major topics and classifies the primary research papers. SLR, the second method, aims to synthesize of evidence on the very
specific research question.
There are a few related studies, and in this section, we summarize them focusing on mapping studies and SLRs that analyzed the articles on
maturity models. Wendler10 performed a mapping study on maturity models. He selected 237 articles published by journals and in five
conferences on IS, software engineering, and business between 1993 and 2010. He classified the selected articles by each of five categories
(ie, research design, research method, research content, application domains, and maturity models) for 10 research questions. However, the study
did not provide any specific information on the maturity models used in the study, such as names, industrial sectors, and their validity.
Pöppelbuß et al9 analyzed 76 articles on maturity models concerning IS discipline, published between 1996 and 2010 from the three perspec-
tives: research (theoretical and methodological aspects), publication (interests of potential authors, editors, and reviewers), and practice (issues
relevant to the end users). He mentioned the current academic focus on “how to improve the theoretical grounding and empirical validity of
maturity models.” A mapping study of Helgesson et al8 identified 59 maturity model articles between 1995 and 2009 on software engineering.
She classified the articles from two databases into six categories, such as the maturity model under evaluation, type of evaluation, relation of
the evaluators/authors to the maturity model, level of objectivity, main purpose of the paper, and size of study. None of them indicated the names
of the maturity models identified in their reference publications.
Previous mapping studies showed that CMM is the dominant theoretical background in the maturity model research, and the growth of
research is an ongoing trend expanding the sectors of application. On the other hand, they raised concerns on maturity model validity. In the
software process improvement (SPI) context, Unterkalmsteiner et al23 reached a conclusion that the validity of SPI initiative evaluation is
challenged because of insufficient consideration of the evaluation context and the potential confounding factors.
Several SLRs on maturity models have been reported as well. They were limited to specific domains such as BPM, usability, supply chain,
knowledge management (KM), and agility. Tarhan et al13 conducted an SLR on BPM maturity models. They analyzed 61 selected articles in terms
of their research focus, such as development, application, and validation. In the area of the usability engineering, Lacerda and von Wangenheim24
conducted an SLR with 15 usability capability/maturity models. SLR studies on supply chain maturity models, KM maturity models, and agile
LEE ET AL. 3 of 30

maturity models were performed by Correia et al,11 Pour et al,14 and Henriques and Tanner,12 respectively. Similar to the previous mapping
studies, one or more concerns on model validity such as a lack of information on model design/development, insufficient support for model
usage/adoption in practice, scarce empirical evidence, and questionable validity and usefulness of the models were mentioned in their study areas
of interest. Even in the specific domains, however, none of the SLRs provided information on how the models were available in the industrial
domains of interest and how they were validated by, for example, tested hypothesis and associated statistical tests.

3 | METHOD: MAPPING STUDY

This study adopted a mapping study method to discover trends (eg, publication over time, identification of maturity models and their validities, and
classification by sectors) of maturity model development. For this purpose, we followed recommendations of the mapping studies by Petersen
et al5 and Kitchenham et al.7 Our mapping study consisting of three phases (ie, planning, conducting, and reporting) with 11 steps (shown in
Figure 1) was performed as follows:

• planning (five steps): need for a mapping study, research questions, mapping study protocol formation, search process, and mapping study
protocol evaluation;
• conducting (four steps): article selection, quality assessment, data extraction, and data synthesis; and

• reporting (two steps): threats and conclusions.

This section describes the mapping study steps from 1 to 8. The remaining steps 9 through 11 are presented in Sections 4 to 6.

3.1 | Need for a mapping study

The “need for a mapping study” (step 1) is based on the following rationale: Previous mapping studies

• did not fully identify and report maturity models,

• cast doubts on the validity of maturity models, and

• were limited to certain sectors in spite of the diversity in maturity models.

ISO/IEC JTC 1/SC 7/WG 10 (process assessment), an international standardization group, has developed capability and process maturity
models. The needs of these models have been growing within the ISO process assessment standardization working group, from other standards

FIGURE 1 Phases and steps in our mapping study

4 of 30 LEE ET AL.

TABLE 1 Research questions (RQs)

No. Question

RQ1 What maturity models appeared in journal articles?

RQ2 How many articles on maturity models were published over time?
RQ3 What are the sectors of maturity model articles?
RQ4 Which maturity models have been statistically tested?

bodies, and from external agencies.25 Those organizations need to know a map of current maturity models developed and validated so far, as the
first step of standardization. ISO report shows 13 standardization projects on maturity model development across diverse sectors within ISO.
Before starting this study, this author contacted some of ISO/IEC SC 7 international delegates to evaluate the value of this mapping study. All
of them wanted to know how many maturity models with details such as model names have been published across industrial sectors over IT.
Furthermore, delegates in WG 10 also were interested in empirical validation because they developed a process capability models (ISO/IEC
15504‐5) in accordance with a set of empirical evaluation named SPICE Trials.26 Our findings on identified maturity models across industrial
sectors and their statistical evaluation were formally reported to delegates interested in maturity model standardization including WG 10.
To identify similar works, we searched the title, keyword, and abstract of peer‐reviewed studies written in English of K‐eArticles* with a set of
keywords. The search terms applied, the combination of “maturity model” and synonyms of “mapping studies” and “systematic literature review,”
are as follows:

((“maturity model” OR “maturity models”) AND (“systematic review” OR “mapping study” OR “systematic mapping study” OR “systematic
literature review” OR “literature search” OR “research review”))

Articles found in this search as a quasi‐gold standard27 were consulted for establishing keywords in search strings and for assuring the results
of the database search. This search identified several previous literature studies of maturity models described in Section 2. Not all of them were
enough to accommodate our objectives described above.

3.2 | Research questions

Step 2 of this mapping study is to formulate “research questions” consistent with the aims of mapping study (eg, categorizing and structuring
maturity models).5,22 Table 1 shows research questions of this study. We first identified articles in peer‐review journals and applied the quality
criteria (ie, the requirements of model building). All of the four questions in Table 1 were answered on the basis of data extracted from articles
that passed the criteria.
RQ4 is a question on whether maturity models were statistically tested,28-30 which was intensively discussed in Section 4.4. To avoid probable
research bias and ambiguity as well as to enhance research repeatability,7 we developed a mapping study protocol (called a review protocol in
Kitchenham et al7) for research questions, search string, search process, selection criteria, article selection, quality assessment, data extraction,
and data synthesis (step 3). It was refined in an iterative manner among the authors throughout the study.

3.3 | Search process

A proper search string is essential in finding relevant studies. We identified keywords and constructed a search string using the research questions.
Because many maturity models followed the architectures of the SW‐CMM, CMMI, or ISO/IEC 15504, we examined the key terminologies of
those standards; all of them had the very similar terms. We also consulted previous mapping studies on maturity models (Section 2) to establish
the final search string. This result was reexamined in the pilot study for review protocol validation.
The terms such as model building and model testing (ie, validities) in research questions were not included in the search string because they
should be dealt with in depths by quality criteria in this study.
Table 2 shows the search string used for this study. The following is a brief explanation of the keywords in the string:

• Although a primary interest group of this study is standardization organizations for IT, this study is not limited to maturity models in the IT
sector because there is an emerging need for establishing a new standard organization for maturity model across diverse sectors.25

*An integrated academic journal search engine provided by Korea University.

LEE ET AL. 5 of 30

TABLE 2 Search string

“maturity model” OR “capability model” OR “maturity grid” OR “process improvement model” OR “process assessment model”

TABLE 3 Search database and results

Database No. of Articles Retrieved

K‐eArticle 1145
Scope 497
Wiley 330
Proquest 224
EBSCO 54
Emerald 39
Science Direct 73
Web of Science 19
Total 2381

• In some studies, maturity and capability models use the same process areas with different perspectives. In CMMI,31 a set of process areas is
formulated as a stage of maturity level. Thus, we included a keyword “capability” in the search string.
• “Grid” was included in the search string because maturity grid implied maturity level some years ago.
• Two powerful organizations for standardization—ISO and SEI—that developed (process) capability† and maturity models have their own
principles for the use of models such as process assessment32 and process improvement,31 respectively.

The search using the above search string (Table 2) was performed on K‐eArticles and the other seven electronic databases covering diverse
disciplines with a set of search criteria (ie, search field: title, keywords, and abstract; period: January 1993‐June 2015; language: English; and
publication type: academic journals). Table 3 shows those databases, and 2381 articles were retrieved as a result.
In addition, this study compared relevant references in previous maturity model mapping studies with search results in Table 3 as a means to
get assurance of our database search (known as a backward snowballing27). Besides articles in German or those published in conferences or
proceedings, the “relevant references” were found in the database search result. No discrepancy was found. We used EndNote to handle the high
number of articles.

3.4 | Review protocol evaluation

As part of the “mapping study protocol evaluation” (step 5), we piloted randomly chosen 200 articles from the 2381 articles retrieved against our
mapping study protocol. The results were fed into the review protocol revision.
The inclusion criteria were finalized as follows:

• articles (i) that associated a maturity model as the main topic, (ii) that could categorized into at least one area of the following, development,
application, or validation of maturity models, and (iii) that described model contents (eg, levels, process areas, best practices, and rating
methods);
• empirical studies associated with maturity models.
The exclusion criteria were refined as follows:

• articles (i) mentioning a maturity model without model contents, (ii) dealing with subjects such as general discussion, review paper, and model
comparison, and (iii) simply borrowing parts of structure of maturity model structure for irrelevant topics (eg, cost estimation model and
measurement model design).

During this step, a sensitivity of each search string component (Table 2) in retrieving the 200 pilot articles was experimented, and our search
string did not show any issues with robustness.

†
CMMI maturity level based on capability levels is determined by “equivalent staging,” aggregating the capability ratings of process areas (CMMI for development,
version 1.3, 2010 (pp34‐37)).
6 of 30 LEE ET AL.

3.5 | Article selection

Articles irrelevant to research questions were excluded from the “article selection” (step 5).7 The first two authors applied the inclusion/exclusion
criteria to titles, keywords, and abstracts, respectively. This resulted in 502 articles. For disagreed articles (ie, articles of which only one author
agreed to the inclusion), we included them to full‐text reading. In addition, we discarded dozens of duplicated articles created by the changes
in the authors' name (eg, first and last name reversed).
During the full‐text reading, 153 articles (irrelevant articles [145], an unidentified article [1], and articles unobtainable within the schedule of
the study [7]) were removed. Finally, 349 articles were selected for the next step.

3.6 | Quality assessment in model building

Although Kitchenham et al22 addressed that quality assessment in mapping study is not essential, this study conducted it because it may be
important to potential audiences of this study. It is noteworthy that Petersen et al5 also performed quality assessment in a systematic mapping
study of systematic mapping studies.
Our mapping study uses phase 1 requirements (ie, theoretical definition(s), operationalization, and model components) as shown in Table 4 for
maturity model development as criteria for quality assessment. The first two in Table 4 can be referenced to Ahire and Devaraj,20 Edwards and
Greenberg,21 Venkatraman,33 and Peter,34 and model components are based on CMMI31 and ISO/IEC 15504‐5.35
In this study, the phase 1 requirements in Table 4 can be fulfilled using behavioral and/or design sciences. Behavioral science seeks to develop
and verify (justify) maturity models and/or theories that explain or predict interesting phenomena including organizations and human beings.19,36
As an example, an ISO draft37 shows the types of theoretical models that may be utilized to develop maturity models. Schweiger38 also insists that
maturity model development should be based on both theoretical and practical aspects of models. Van Looy et al39 discussed the role of theory in
the development of their decision tool for selecting business process maturity models as well. However, because behavioral science drawing on
the known theories for creating artifacts is often insufficient for emerging sectors' maturity models, design science can also be utilized to create
new and innovative artifacts (eg, ideas, practices, technical capabilities, and/or products) to address unsolved problems.19,36 Thus, the phase 1
requirements can be fulfilled using an integrated approach of behavioral and design sciences. Wendler40 proposed an organizational agility
maturity model based on the design science. In consideration of this background, this study accommodated behavioral science and/or design
science in quality assessment of phase 1.
From 349 articles in step 6, a total of 285 articles passed all of the three criteria in Table 4, and 194 maturity models were identified from the
285 articles. Appendix A provides those 194 maturity models by sector and their publication articles.
A data extraction template in excel format was developed to record quality assessment results in Table 4 and the relevant information
(eg, article ID, author(s), title, journal, and publication year). The first two authors filled out the template for each article, and the third
author reviewed it. This was repeated throughout the process.
The extent of meeting the criteria in Table 4 for each article varies. However, the completeness of meeting the criteria was not quantitatively
evaluated. In this regard, Kitchenham and Brereton27 also noted “quality assessment of studies using a variety of empirical methods remains a
major problem” in SLR.

3.7 | Data extraction

This study recorded extracted data on excel sheets (created in step 7) to answer four research questions. Sections 3.7.1 to 3.7.4 describe what was
actually extracted for each extraction: name of maturity model (RQ1), publication year (RQ2), application sector (RQ3), and quality assessment in
model testing (RQ4).

TABLE 4 Phase 1: criteria of quality assessment in model building and descriptions

Criterion Descriptions
Theoretical background(s) Yes: Theoretical background(s) is (are) described and/or inferred.
No: Theoretical background(s) cannot be inferred.
Operationalization Yes: Best practices (or base activities) are defined and/or inferred.a
No: Any operationalization method applied cannot be inferred.
Model components Yes: Components (ie, maturity levels, process (areas), and rating methods) are addressed and/or inferred.
No: One or more of model components are not inferred.
a
For example, case study, pilot test, survey, expert feedback, or comparison with reference models is also regarded as evidence.
LEE ET AL. 7 of 30

3.7.1 | Name of maturity model (RQ1)

Names of maturity models extracted from articles were listed on the excel sheet for RQ1. They were collected as stated in the articles to secure
their uniqueness. Appendix A shows the extracted names (194 names) of maturity models by sector and their associated articles.

3.7.2 | Publication year (RQ2)

Article publication years were extracted to show the trend. A lot of publications are related to the CMMI models (eg, the SW‐CMM, CMMI‐DEV,
and CMMI‐SVC). Thus, we categorized article publications into those with CMMI and non‐CMMI models from 1994 to 2015.

3.7.3 | Application sector (RQ3)

Application sectors of maturity models were categorized on the basis of global industry classification standard (GICS).41 Its 11 sectors are energy,
materials, industrials, consumer discretionary, consumer staples, health care, financials, information technology, telecommunication services,
utilities, and real estate.
During the work, we deleted “real estate” sector from the sectors listed above because no maturity model was identified on the real estate.
For completeness, this study added three sectors: general, government, and law. Maturity models applicable to multiple sectors or without a
corresponding sector were placed in the general sector.

3.7.4 | Quality assessment in model testing (RQ4)

The validity in testing16,20,21,33,34 statistically demonstrates how well a model measures what it purports to measure. Validity claims of models with
regard to performance and quality can be confirmed, supported, and/or elaborately refined by evidence from empirical evaluation.42 In this study,
quality assessment in model testing implies examination of validity studies on maturity models in phase 2. We believe the results provide an
answer to RQ4.
The classification scheme of model‐testing validities is a key to acquiring reliability in the mapping study. Clear classification scheme was
emphasized by Kitchenham et al,22 Petersen et al,5 and Wohlin et al.43 Thus, we followed reliability and validity requirements specified in
ISO/IEC 33003 as mandatory requirements in phase 2. They are not the same as those on software engineering described by Petersen et al.5 Note
that validity requirements may depend on sectors or granularity of models.
This study determined the validity in testing as “Yes,” “Partially,” or “No” based on the three criteria in Table 5. The validity studies of a
maturity model can appear one or more articles. Validity is discussed further in later sections.

4 | RESULTS

This section presents the results and discussions of the RQ1 to RQ4.

4.1 | RQ1: What maturity models appeared in journal articles?

The number of maturity models developed was speculated from dozens to hundreds.9,10,42,44,45 Our mapping study identified 194 distinct
maturity models (listed in Appendix A) published in 285 articles.

4.2 | RQ2: How many articles were published?

Figure A1 (Appendix A) depicts the article publications by year between 1993 and June 2015. The two bars in different colors for each year
represent the numbers of articles on CMMI and non‐CMMI, respectively. From 1994 to June 2015, article publication related to the SW‐CMM

TABLE 5 Phase 2: criteria of model‐testing validity and descriptions

Criterion Description

Yes “Hypothesis” has been statistically tested.

Partially Any descriptive statistics and/or graphical methods for evaluating a maturity model have been applied.
No None of the above methods can be inferred.
8 of 30 LEE ET AL.

and CMMI (47, 16%) has ranged from zero to six articles per year at most. On the other hand, starting from 2004, articles associated with
non‐CMMI (238, 84%) display a noticeable upward trend and reached its peak with 33 articles in 2012. To sum up, a growth in maturity model
research seems to be an ongoing trend. The articles were published in 198 journals (list omitted).

4.3 | RQ3: What are the sectors of maturity model articles?

The article publications by application sector from 1994 to 2015 are depicted in Table A1 (Appendix A). This indicates the sectors on which articles
have been published are becoming diverse. From 2012 onwards, articles of maturity model in non‐IT sectors have been published more than those
articles in IT sector have been.

4.4 | RQ4: Which maturity models have been statistically tested?

A total of 194 models were examined on the basis of the model‐testing validity in Section 3.7.4. In addition to the five partially validated ones
(eg, graphical representation of the relationship between levels and performance), 26 validated maturity models were identified. The remaining
models (163, 84%) were not validated by any articles published. A detailed list of the validated models and their reference articles is given in
Table B1 (Appendix B). As shown in the table, validity studies of CMMI, continuous improvement maturity model, and supply chain maturity
model are published in more than one article.
Furthermore, this study investigated the hypotheses that were statistically tested for the 26 validated models (indicated by “Yes” as a result in the
table). Their hypotheses and associated validity16,21,33,34,46 are provided in Table B2 (Appendix B). The last column shows the types of the statistical
tests such as predictive validity, unidimensionality, and multidimensional construct (refer to ISO/IEC 33003 for detailed description on the tests16).
Table B3 (Appendix B) indicates the distribution of 194 maturity models and 26 statistically validated maturity models with their associated
validities by application sectors. Although maturity models have been spread out in 14 application sectors, validated models were observed only in
five sectors such as IT, general, and industrials.

5 | DISCUSSIONS

Standard practitioners addressed concerns of standards regarding the lack of evidence to support best practices in the models; Pfleeger et al47
stated “Standards have codified approaches whose effectiveness has not been rigorously and scientifically demonstrated. Rather, we have too
often relied on anecdote, ‘gut feeling’, the opinions of experts, or even flawed research.” Similar arguments can be found in Jung et al,48 Fenton
et al,49 and Littlewood et al.50 As previously noted, validities in process model building and testing have been core issues in standard development.
To mitigate those concerns, standardization organizations invested their efforts in validating their process models: Appendix A includes empirical
evaluations of the SW‐CMM (maturity model) and CMMI (capability and maturity models). SPICE Trials26,51 evaluated ISO/IEC 15504 (capability
models), and recently, ISO and OECD developed their guidelines relating to the development of process models16 and index models,52 respectively.
In this context, present study employed a set of quality criteria regarding modeling building and testing to answer our research questions with
associated studies. From the 349 articles extracted for article selection, 62 models and 64 articles were removed on the basis of the quality assess-
ment in model building. Its results are shown in Table 6. Among those 62 disqualified models, 13 (21%) and 58 (94%) models did not meet the
criteria of the theoretical background(s) and operationalization, respectively. Furthermore, 10 of them even failed in both criteria. On the other
hand, all of them met the criterion of the model components. It is no wonder because the model components were part of the inclusion criteria.
Through this process, we were left with 285 articles, excluding the 64 articles.
Each of 194 maturity models from the 285 articles was examined on the basis of model‐testing validity (quality assessment in model testing). A
total of 26 models, in addition to the five partially validated ones, were determined to be statistically tested (eg, predictive validity and unidimension-
ality). On the other hand, the other remaining models (163, 84%) were found to be not having been statistically tested by any articles published yet.
However, just because those models may be under the validation process, they are not all bad or insufficient models. Their progress (eg, under
development, developed, and statistically tested) can be identified at one of those stages. The progresses should be updated regularly.

TABLE 6 Distribution of disqualified models by the quality assessment in model building

Criterion Yes No
No. of models (%) Theoretical background(s) 49 (79%) 13 (21%)
Operationalization 4 (6%) 58 (94%)
Model components 62 (100%) 0 (0%)
LEE ET AL. 9 of 30

6 | FINAL REMARKS

As most of maturity model literature studies, this study has its limitations that should be made clear before interpreting the results (the “threats”
(step 11)). This study selected the articles from academic journals. Therefore, some models and their associated evidences that published in other
sources such as technical reports, work in progress, and unpublished might have been omitted from the scope of this study.7 Those sources of
evidence are also important because there may be models that are influential in the field but less active in the publications, eg, control objectives
for information and related technologies (COBIT) [A6, A125] and enterprise architecture (EA) maturity model.53‡ Research parts (eg, model
development and model validation) of such models may have been published in other forums besides academic journals. However, as stated in
Sections 4 and 5, their validation based on a scientific research cycle should be openly shared and updated as well because much of the confidence
in those models may result from their reliability and validity.
In addition, specialized and/or emerging topics in maturity models might be missing or partially included in this study. One of them is the
development of maturity models in agile context, which has become one of the main stream of process maturity models in software
engineering.54-57 Another one is the model‐based system engineering (MBSE)58 approach for developing models; “focus area maturity
models”59,60 belong to the MBSE models, the architecture of which is different from that of the process maturity models. We hope that further
validation work on those models will be performed soon as they have not been validated yet.
To cover a variety of disciplines as well as to mitigate the threats mentioned above, a search process was conducted on eight digital databases,
and the relevant references in the previous mapping studies and the initial retrieval were compared (ie, backward snowballing). We analyzed 285
articles extracted from the initially retrieved 2381 articles. However, incompleteness of the initial retrieval and the article selection for analysis and
possible omissions of several models and their evidences are not deniable.
However, we need to remember that the aim of this study is to identify maturity models and their validities, along with a classification of
models across industrial sectors, and to that end, each of the identified models was examined by the two kinds of validities (ie, “model building
and testing”) associated with a scientific research cycle. Therefore, industrial activities such as a take‐up of models and application/assessment
in practice are not our concerns. We believe that practitioners and researchers as well as standard organizations can utilize our results to locate
the maturity models of their interest, to identify their validities in model building and testing, and/or to seek research opportunities because our
work provides the names of the maturity models, their validities (ie, model building and testing), a classification of industrial sectors, and associated
studies. For those who are in search of suitable models and/or validated models, our results could be a practical guidance.
This mapping study reveals the current status of maturity model validation using a set of quality criteria. Among the 256 maturity models
identified from 349 articles prior to the quality assessment in model building, not a little proportion of those models was disqualified by not meet-
ing the criteria of model‐building (25%, 62/256) and model‐testing (84%, 163/194) validity, respectively. Thus, our approach and study results
would serve as a base for a subsequent systematic review on the maturity models. In addition, because we based our study on maturity models
as titled, capability models (eg, ISO/IEC 1550461,62 and KM capability assessment model63) with maturity models as part of them were not dealt
with by the study. Replication of our study for the capability models would be worthy of study using a set of quality criteria.

ACKNOWLEDGEMEN TS
We would like to thank the anonymous referees whose valuable comments (eg, agility and MBSE in system engineering) have improved the paper.

ORCID
Donghun Lee https://orcid.org/0000-0001-8099-1032

RE FE R ENC E S
1. Paulk MC. The Capability Maturity Model: Guidelines for Improving the Software Process. Boston, MA, USA: Addison‐Wesley Professional; 1995.
2. SEI. CMMI® for acquisition, version 1.3 (CMU/SEI‐2010‐TR‐032); CMMI® for development, version 1.3 (CMU/SEI‐2010‐TR‐033); CMMI® for services,
version 1.3 (CMU/SEI‐2010‐TR‐034). 2010.
3. CMMI Institute. CMMI® maturity profile report. 2016.12. Available at: https://goo.gl/info/FjQxGs.
4. SEI. CMMI® for development, version 1.3 (CMU/SEI‐2010‐TR‐033). 2010. Available at: http://goo.gl/YpvL.
5. Petersen K, Vakkalanka S, Kuzniarz L. Guidelines for conducting systematic mapping studies in software engineering: an update. Inform Softw Technol.
2015;64:1‐18.
6. Kitchenham BA, Budgen D, Brereton OP. The value of mapping studies—a participant‐observer case study. In: EASE; 2010.
7. Kitchenham B, Charters S, Budgen D, Brereton P, Turner M, Linkman S, Jorgensen M, Mendes E, Visaggion G. Guidelines for performing systematic
literature reviews in software engineering; 2007.
8. HelgessonYYL, Höst M, Weyns K. A review of methods for evaluation of maturity models for process improvement. J Softw Evol Process. 2012;24:436‐454.

‡
EA maturity model developed by National Information Society Agency (NIA) in Korea has been used for evaluating IT capabilities of public institutions.
10 of 30 LEE ET AL.

9. Pöppelbuß J, Niehaves B, Simons A, Becker J. Maturity models in information systems research: literature search and analysis. Commun Assoc Inf Syst.
2011;29:505‐532.
10. Wendler R. The maturity of maturity model research: a systematic mapping study. Inf Softw Technol. 2012;54(12):1317‐1339.
11. Correia E, Carvalho H, Azevedo SG, Govindan K. Maturity models in supply chain sustainability: a systematic literature review. Sustainability. 2017;9(1):64.
12. Henriques V, Tanner M. A systematic literature review of agile maturity model research. Interdiscipl J Inf Knowl Manag. 2017;12:53‐73.
13. Tarhan A, Turetken O, Reijers HA. Business process maturity models: a systematic literature review. Inf Softw Technol. 2016;75:122‐134.
14. Pour MJ, Manian A, Yazdani HR. A theoretical and methodological examination of knowledge management maturity models: a systematic review. Int J
Bus Inf Syst. 2016;23(3):330‐352.
15. Jung H‐W, Goldenson DR. The internal consistency and precedence of key process areas in the capability maturity model for software. Empir Softw Eng.
2008;13(2):125‐146.
16. ISO/IEC 33003. Process assessment—requirements for process measurement frameworks. 2015.
17. Corley KG, Gioia DA. Building theory about theory building: what constitutes a theoretical contribution? Acad Manage Rev. 2011;36(1):12‐32.
18. Bollen KA. Structural Equations with Latent Variables. New York: Wiley; 1989.
19. Hevner AR, March ST, Park J, Ram S. Design science in information systems research. MIS Q. 2004;28(1):75‐105.
20. Ahire SL, Devaraj S. An empirical comparison of statistical construct validation approaches. IEEE Trans Eng Manag. 2001;48(3):319‐329.
21. Edwards JR, Greenberg IJ. Construct validation in organizational behavior research. Organizational Behavior: A Management Challenge, New York: Psychol-
ogy Press; 2003:311‐354.
22. Kitchenham BA, Budgen D, Brereton OP. Using mapping studies as the basis for further research—a participant‐observer case study. Inf Softw Technol.
2011;53(6):638‐651.
23. Unterkalmsteiner M, Gorschek T, Islam A, Cheng CK, Permadi RB, Feldt R. Evaluation and measurement of software process improvement—a systematic
literature review. IEEE Trans Softw Eng. 2012;38(2):398‐424.
24. Lacerda TC, von Wangenheim CG. Systematic literature review of usability capability/maturity models. Comput Stand Interfaces. 2017;55:95‐105.
25. ISO. Position paper: some considerations regarding transitioning from ISO/IEC JTC1/SC7/WG10 to an ISO Technical Committee. 2017.
26. Jung HW, Hunter R, Goldenson DR, El‐Emam K. Findings from phase 2 of the SPICE Trials. Softw Process Improv Pract. 2001;6(4):205‐242.
27. Kitchenham B, Brereton P. A systematic review of systematic review process research in software engineering. Inf Softw Technol. 2013;55(12):2049‐2075.
28. Handfield RB, Melnyk SA. The scientific theory‐building process: a primer using the case of TQM. J Oper Manag. 1998;16(4):321‐339.
29. Trochim WM, Donnelly JP. Research methods knowledge base. 2001.
30. Hughes MA, Price RL, Marrs DW. Linking theory construction and theory testing: models with multiple indicators of latent variables. Acad Manage Rev.
1986;11(1):128‐144.
31. SEI. CMMI for development, version 1.3; 2010.
32. ISO/IEC 33001. Information technology—process assessment—concepts and terminology: ISO/IEC JTC1/SC 7 WG10; 2014.
33. Venkatraman N. Strategic orientation of business enterprises: the construct, dimensionality, and measurement. Manag Sci. 1989;35(8):942‐962.
34. Peter JP. Construct validity: a review of basic issues and marketing practices. J Market Res. 1981;18(2):133‐145.
35. ISO/IEC 15504‐5. Information technology—process assessment—part 5: an exemplar process assessment model. 2012.
36. Hevner AR, March ST. The information systems research cycle. Computer. 2003;36(11):111‐113.
37. ISO. An organizational capability maturity model based on ISO/IEC 15504‐5 (draft) (Author: Ho‐Won Jung) (N 1090). 2016.
38. Schweiger J. A theory‐based perspective on maturity models in purchasing and supply management. Innov Methods Logist Supply Chain Manag.
2014;19:531‐555. Available at: https://hicl.org/publications/2014/19/531.pdf
39. Van Looy A, De Backer M, Poels G, Snoeck M. Choosing the right business process maturity model. Inf Manag. 2013;50(7):466‐488.
40. Wendler R. Development of the organizational agility maturity model. In: Computer Science and Information Systems (FedCSIS), 2014 Federated
Conference on: IEEE; 2014.
41. MSCI, S&P. Global industry classification standard (GICS). Available at: https://www.msci.com/gics. (Accessed Mar. 2016).
42. El‐Emam K, Goldenson DR. An empirical review of software process assessments. National Research Council of Canada, 1999.
43. Wohlin C, Runeson P, da Mota Silveira Neto PA, Engström E, do Carmo Machado I, de Almeida ES. On the reliability of mapping studies in software
engineering. J Syst Softw. 2013;86(10):2594‐2610.
44. Becker J, Knackstedt R, Pöppelbuß D‐WIJ. Developing maturity models for IT management. Bus Inform Syst Eng. 2009;1(3):213‐222.
45. Mettler T, Rohner P, Winter R. Towards a classification of maturity models in information systems. In: Management of the Interconnected World. Verlag
Berlin Heidelberg: Springer; 2010, 333‐340.
46. Hattie J. Methodology review: assessing unidimensionality of tests and ltenls. Appl Psychol Measur. 1985;9(2):139‐164.
47. Pfleeger SL, Fenton N, Page S. Evaluating software engineering standards. Computer. 1994;27(9):71‐79.
48. Jung HW, Kim S‐G, Chung C‐S. Measuring software product quality: a survey of ISO/IEC 9126. IEEE Soft. 2004;21(05):88‐92.
49. Fenton N, Littlewood B, Page S. Evaluating software engineering standards and methods. Software Engineering: A European Perspective 1993:463‐470.
50. Littlewood B, Brocklehurst S, Fenton N, et al. Towards operational measures of computer security. J Comput Secur. 1993;2(2‐3):211‐229.
LEE ET AL. 11 of 30

51. SPICE Trials. SPICE phase 2 trials final report.

52. Nardo M, Saisana M, Saltelli A, Tarantola S, Hoffman A, Giovannini E. Handbook on Constructing Composite Indicators. OECD publishing: European Com-
mission; 2005.
53. Nam KC, Oh SW, Kim SK, Shin DU. Case study of EA implementation in the Korean public sector: guidelines, lessons, and future research model.
Informatization Policy J. 2015;22:45‐67.
54. Schweigert T, Nevalainen R, Vohwinkel D, Korsaa M, Biro M. Agile maturity model: oxymoron or the next level of understanding. In: International
Conference on Software Process Improvement and Capability Determination: Springer; 2012.
55. Ambler SW. The agile maturity model (AMM). 2010. Available at: http://www.drdobbs.com/architecture‐and‐design/the‐agile‐maturity‐model‐amm/
224201005.
56. Fontana RM, Fontana IM, da Rosa Garbuio PA, Reinehr S, Malucelli A. Processes versus people: how should agile software development maturity be
defined? J Syst Softw. 2014;97:140‐155.
57. Fontana RM, Meyer V Jr, Reinehr S, Malucelli A. Progressive outcomes: a framework for maturing in agile software development. J Syst Softw.
2015;102:88‐108.
58. Estefan JA. Survey of model‐based systems engineering (MBSE) methodologies. Incose MBSE Focus Group. 2007;25:1‐12.
59. van Steenbergen M, Bos R, Brinkkemper S, van De Weerd I, Bekkers W. The design of focus area maturity models. In: International Conference on
Design Science Research in Information Systems: Springer; 2010.
60. van Steenbergen M, Bos R, Brinkkemper S, van de Weerd I, Bekkers W, Improving IS. Functions step by step: the use of focus area maturity models.
Scand J Inf Syst. 2013;25:35‐56.
61. Coletta A. An industrial experience in assessing the capability of non‐software processes using ISO/IEC 15504. Softw Process Improv Pract.
2007;12(4):315‐319.
62. Kozina M, Kirinic V. Analyzing the PAM's structure using the ISO/IEC 15504‐5 standard (SPICE). DAAAM International Scientific Book 2013:475.
63. Andrade J, Ares J, GarcÍa R, Rodríguez S, Suárez S. A knowledge‐based system for knowledge management capability assessment model evaluation.
WSEAS Trans Comput. 2010;9:506‐515.

How to cite this article: Lee D, Gu J‐W, Jung H‐W. Process maturity models: Classification by application sectors and validities studies. J
Softw Evol Proc. 2019;e2161. https://doi.org/10.1002/smr.2161

APPENDIX A
MAT URITY M ODELS BY SECT OR
A total of 194 maturity models identified from the 285 articles were classified by application sectors. A distinct maturity model may appear more
than one sector.

Reference Article ID(s)

Consumer discretionary (2)

Logistic maturity model (LMM) [A20]
Maturity framework for commercial culture [A215]
Consumer staples (2)
Maturity model for chain digitization [A201]
Typology for chain digitization maturity [A200]
Education (10)
Capstone project management maturity model (CPMMM) [A65]
e‐Learning maturity model (eMM) [A160], [A161], [A195]
ICT in school education maturity model (ICTE‐MM) [A237]
ICT maturity model [A17]
Intellectual capital maturity model (ICMM) [A227]
Learning institution maturity model [A91]
Online course design maturity model (OCDMM) [A181]
Quality maturity model (QMM) [A274], [A275]
Quality model for educational products based on information and [A218]
communication technology (eQETIC)
12 of 30 LEE ET AL.

(Continued)

Reference Article ID(s)

Teaching CMM (T‐CMM) [A37]
Energy (3)
Jurisdictional maturity model for abandoned mine management [A254]
Safety culture maturity model [A66]
Simulation maturity model [A245]
Financials (2)
Business continuity management (BCM) maturity model [A205]
CMMI [A206]
General (24)
Berkeley project management (PM) process maturity model [A135]
Business continuity management system (BCMS) capability model [A230]
CMMI [A57]
Collaboration maturity levels (CollabMM) [A157], [A158]
Continuous improvement (CI) maturity model [A49], [A116]
Contract management maturity model (CMMM) [A74], [A75], [A76], [A212]
Customer relationship management maturity model (CRM3) [A235]
Maturity model for knowledge‐intensive business process [A113]
Ecodesign maturity model [A198]
Energy management maturity model (Antunes et al) [A12]
Energy management maturity model (Introna et al) [A107]
Ergonomic maturity model (EMM) [A265]
Industrial research institute (IRI) sustainability maturity model [A106]
Integrated knowledge management maturity model (I‐KMMM) [A208]
Knowledge management maturity model (Khatibian et al) [A127]
Knowledge management maturity model (Kruger et al) [A133], [A134]
Knowledge management maturity model (Oliva) [A188]
Knowledge navigator model (KNM) [A103]
Leadership maturity model [A101]
New service development (NSD) maturity model [A111]
Process management maturity model (process management 1‐2‐3) [A46]
Quality and maturity method (QMM) [A25]
Sales and operations planning (S&OP) maturity model [A267]
Supply chain maturity model [A168]
Government (16)
Capital project portfolio management (CPPM) capability model [A54]
Climate data records (CDR) maturity matrix [A18]
e‐Government maturity model [A129]
e‐Government procurement observatory maturity model (eGPO‐MM) [A42]
e‐Government maturity model (eGov‐MM) [A256]
Internal audit capability model (IA‐CM) [A154]
Knowledge management CMM [A59]
Maturity model for health in all polices (MM‐HiAP) [A244]
Open data maturity model (OD‐MM) [A236]
Open government maturity model (OGMM) [A138]
Program and project management maturity model (P3M3) [A280]
Project management maturity model (Prado‐PMMM) [A51]
Public application maturity model [A62]
Public sector process rebuilding (PPR) maturity model [A10]
Risk management capability model (RMCM) model [A191]
Smarter‐city maturity model [A104]
LEE ET AL. 13 of 30

(Continued)

Reference Article ID(s)

Health care (11)
African health profession regulatory collaborative (ARC) CMM [A167]
CMM for hospital process management [A41]
CMMI [A270]
General practice information maturity model (GPIMM) [A79], [A80]
Maturity model for enterprise interoperability (MMEI) [A88]
Maturity model for supplier relationship management (SRM) systems in [A174]
hospitals
Networkability maturity model [A67]
Patient safety culture improvement model [A69]
Picture archiving and communication system (PACS) maturity model [A258], [A259], [A260]
(PMM)
Risk management capability model (RMCM) for the medical device (MD) [A166]
software industry
Telemedicine maturity model (TMMM) [A262]
Industrials (31)
Building information modeling (BIM) maturity model [A248]
Built environment management maturity model (BEM3) [A155], [A156]
CMM for human social capital [A52]
CMMI [A7]
Construction industry macro maturity model (CIM3) [A271], [A272], [A273]
Energy and utility management maturity model (EUMMM) [A182]
Information integration maturity model (IMMM) [A122]
Knowledge maturity model [A114]
Lean enterprise transformation maturity model [A186]
Learning organization maturity model [A39]
Life cycle management CMM (LCM CMM) [A178]
Maturity model for industrial supply chains [A92]
Maturity model for new service development (NSD) [A207]
National building information modeling standard interactive CMM [A169]
(NBIMS I‐CMM)
Organizational project management maturity model (OPM3) [A78], [A179]
Product lifecycle management (PLM) maturity model [A225]
Program management organization maturity integrated model for mega [A109]
construction programs (PMOMIM‐MCPs)
Project management maturity model (PMMM) [A45], [A85], [A175], [A194]
Project management maturity model (ProMMM) [A98], [A232]
Project risk maturity model (PRMM) [A96]
Reliability capability maturity model [A252]
Risk management maturity model (RM3) [A284]
Road safety maturity model [A9]
Safety maturity model (SMM) [A136]
Safety meeting quality measurement (SMQM) maturity model [A5]
Structured process improvement for construction environments (SPICE) [A222], [A223]
Supplier chain management (SCM) maturity model [A137]
Supply chain CMM (S (CM)2) [A213]
Supply chain maturity model [A145], [A146], [A234]
Supply chain relationship maturity model [A173]
Verification and validation maturity model (V2M2) [A23]
IT (92)
Attraction website maturity model (AWMM) [A283]
Brazilian software process model (MPS model) [A176]
14 of 30 LEE ET AL.

(Continued)

Reference Article ID(s)

CMMI [A1], [A8], [A11], [A15], [A16], [A19], [A24], [A26], [A29], [A35], [A44], [A50],
[A53], [A55], [A58], [A64], [A68], [A73], [A86], [A93], [A95], [A105],
[A110], [A115], [A117], [A118], [A121], [A132], [A147], [A159], [A172],
[A189], [A197], [A203], [A204], [A224], [A226], [A247], [A250], [A253],
[A261], [A269], [A278]
Cognizant enterprise maturity model (CEMM) [A94]
Communities of practice (CoP) maturity model [A140]
Community maturity model (CoMM) [A27], [A28]
Complex product and systems risk management CMM (CoPS RM‐CMM) [A211], [A279]
Configuration management (CM) maturity model [A187]
Consulting services maturity (CSM) model [A233]
Control objectives for information and related technologies (COBIT) [A125]
Corrective maintenance maturity model (CM3) [A119], [A120]
Cost engineering maturity model (CEMM) [A243]
Data‐centric security model (DCMS) maturity model [A257]
Data quality management maturity model [A220]
Data warehousing process maturity model (DWP‐M) [A228], [A229]
Decision support system CMM (DSS‐CMM) [A61]
Design safety CMM (DCMM) [A246]
Documentation process maturity model [A266]
Employee competency maturity model (ECMM) [A60]
Enterprise business intelligence maturity model (EBI2M) [A40]
Enterprise manufacturing service maturity model (EMSMM) [A143]
eProcurement maturity model [A33]
ERP maturity model (EMM) [A192]
ERP maturity stage model [A102]
ERP support for pull production CMM [A202]
Evolutionary software project management maturity model (ESPM3) [A249]
Focus area maturity model (FAMM) [A264]
Formal specifications maturity (FSM) model [A72]
General knowledge management (KM) maturity model (G‐KMMM) [A193]
Green information and communication technology (ICT) maturity model [A30]
for SMEs (small and medium enterprises) (GICTMM4SME)
Information process maturity model (IPMM) [A90]
Information security management (ISM) maturity model [A165]
Information security maturity model [A216]
Information systems interoperability maturity model (ISIMM) [A263]
Infosys's knowledge management maturity (KMM) model [A171]
Infrastructure management–process maturity model (IM‐PMM) [A282]
Initial, developed, evolved, advanced, leader (IDEAL) maturity model [A285]
Inspection CMM (ICMM) [A130], [A131]
Integrated CMM‐ISO model [A219]
IT‐enabled collaborative networked organizations maturity model [A14]
(ICoNOs MM)
IT capability maturity framework (IT CMF) [A34], [A43]
IT dependability in emergency management maturity model (IDEM3) [A268]
IT infrastructure flexibility maturity model (ITIF‐MM) [A281]
IT marketing maturity model [A99]
IT maturity model (Leem et al) [A141]
IT maturity model (Santos et al) [A221]
IT portfolio management maturity model (ITPM) [A108]
IT service CMM [A185]
LEE ET AL. 15 of 30

(Continued)

Reference Article ID(s)

Knowledge management maturity model (KMMM) (Chen and Fong) [A38]
Knowledge navigator model (KNM) [A144]
Maintenance IT maturity model [A123]
Management control systems maturity model (MCSMM) [A164]
Master data management maturity model (MD3M) [A242]
Maturity model for designing and implementing supply chain [A36]
performance measurement system (SCPMS)
Maturity model for enterprise interoperability (MMEI) [A87], [A89]
Maturity model for interoperability potential (MM‐IRIS) [A32]
Maturity model for IT functions [A170]
Maturity model for IT outsourcing relationships [A83], [A239]
Maturity model for service systems in heavy equipment manufacturing [A180]
enterprises
Maturity model of software customer satisfaction [A142]
Maturity model of strategic information systems planning (SISP) [A199]
MIS‐PyME software measurement CMM (MIS‐PyME MCMM) [A56]
MND‐ESPAM (software process assessment model invented for [A139]
evaluating the software process capability of Korean military software
organizations)
Multisourcing maturity model [A97]
Open source maturity model (OMM) [A196]
Open source software test maturity model (OSS‐TMM) [A177]
Open source usability maturity model (OS‐UMM) [A209]
Outcome and learning‐based maturity model (OLMM) [A128]
People‐CMM [A48], [A255]
Process management maturity assessment (PMMA) model [A217]
Process maturity model for requirements engineering (PMM‐RE) [A238]
Relationship management maturity model (RMMM) [A162], [A163]
Reliability engineering CMM (RE‐CMM) [A190]
Requirement engineering maturity measurement framework (REMMF) [A183], [A231]
Requirements CMM (R‐CMM1) [A21], [A22]
Requirements engineering (RE) process maturity model [A241]
Service oriented architecture (SOA) maturity model (Foreman et al) [A70]
Service oriented architecture (SOA) maturity model (Hirschheim et al) [A100]
Software engineering maturity model [A77]
Software maintenance maturity model (S3M) [A13]
Software process improvement (SPI) implementation maturity model [A184]
Software product line engineering maturity model: Architecture [A4]
Software product line engineering maturity model: Business [A2]
Software product line engineering maturity model: Organization [A3]
Software talent cultivation maturity model (STP‐CMM) [A276]
Strategy alignment maturity model (SAMM) [A126], [A148], [A149]
Sustainable supply chain management (SSCM) maturity model [A210]
Technology management maturity model [A277]
Test employee CMM (TEC‐MM) [A112]
Test improvement model (TIM) [A63]
Ultra large systems interoperability maturity model [A214]
Unified requirements engineering process maturity model (Uni‐REPM) [A251]
Law (3)
Digital investigation CMM (DI‐CMM) [A124]
Maturity model for criminal organizations [A82]
16 of 30 LEE ET AL.

(Continued)

Reference Article ID(s)

Maturity model for geographic information systems (GIS) applications [A84]
Materials (5)
Control objectives for information and related technologies (COBIT) [A6]
People‐CMM [A240]
Safety maturity model for UK coal mining [A71]
SME‐specified maturity model for new forming processes [A81]
Structural elements of coordination mechanisms–maturity model [A47]
(SECM‐MM)
Telecommunication (1)
Verification and validation maturity model (V2M2) [A150]
Utilities (2)
Electricity subsector cybersecurity CMM (ES‐C2M2) [A31]
Risk management CMM (RM‐CMM) [A151], [A152], [A153]

FIGURE A1 Articles by publication year.

CMMI, capability maturity model integration

TABLE A1 Article publications by application sector

Application Sector '94 '95 '96 '97 '98 '99 '00 '01 '02 '03 '04 '05 '06 '07 '08 '09 '10 '11 '12 '13 '14 '15 No. of Article(s)
Consumer discretionary 1 1 2
Consumer staples 1 1 2
Education 2 1 1 1 1 1 2 1 3 13
Energy 2 1 3
Financials 1 1 2
General 1 1 3 1 1 6 2 2 2 1 3 6 1 30
Government 1 1 2 2 5 1 4 16
Health care 2 2 1 3 1 2 3 14
Industrials 1 1 1 2 2 3 2 1 4 4 5 7 4 3 2 42
IT 1 1 2 7 2 3 4 3 5 3 6 7 6 8 11 10 10 14 15 12 14 4 148
Law 1 1 1 3
Materials 1 1 2 1 5
Telecommunication 1 1
Utilities 2 1 1 4
No. of articles 1 1 2 7 3 3 8 4 6 5 10 11 12 16 22 18 25 26 35 26 32 12 285
LEE ET AL. 17 of 30

APPENDIX B
VALIDATED MATURITY MODELS

TABLE B1 Validated maturity models

Result Maturity Model Ref Article ID(s)

Yes Berkeley project management (PM) process maturity model [A135]
CMMI [A1], [A57], [A58], [A95], [A110],
[A117], [A118], [A132], [A189], [A247]
Complex product and systems risk management CMM (CoPS RM‐CMM) [A211]
Continuous improvement (CI) maturity model [A49], [A116]
Customer relationship management maturity model (CRM3) [A235]
Data quality management maturity model [A220]
Data warehousing process maturity model (DWP‐M) [A229]
Industrial research institute (IRI) sustainability maturity model [A106]
Initial, developed, evolved, advanced, leader (IDEAL) maturity model [A285]
IT maturity model (Leem et al) [A141]
Knowledge management maturity model (Khatibian et al) [A127]
Knowledge management maturity model (KMMM) (Chen and Fong) [A38]
Knowledge management maturity model (Oliva) [A188]
Knowledge navigator model (KNM) [A103]
Maturity model for IT outsourcing relationships [A239]
Organizational project management maturity model (OPM3) [A78]
Picture archiving and communication system (PACS) maturity model (PMM) [A259]
Project risk maturity model (PRMM) [A96]
Reliability engineering CMM (RE‐CMM) [A190]
Road safety maturity model [A9]
Safety culture maturity model [A66]
Safety maturity model (SMM) [A136]
Safety meeting quality measurement (SMQM) maturity model [A5]
Software product line engineering maturity model: Business [A2]
Software product line engineering maturity model: Organization [A3]
Supply chain maturity model [A145], [A168], [A234]
Partially Knowledge management maturity model (Kruger et al) [A134]
Maturity model for strategic information systems planning (SISP) [A199]
Requirements engineering (RE) process maturity model [A241]
Sales and operations planning (S&OP) maturity model [A267]
Strategy alignment maturity model (SAMM) [A126], [A149]
18 of 30 LEE ET AL.

TABLE B2 Hypotheses and associated validity

Maturity Model Ref Article ID Hypotheses Validity Tested

Berkeley PM process [A135] “Positive correlation between an organizational project Predictive

maturity model management maturity level and its actual project
performance”
CMMI [A1] “Impacts of highly mature processes on effort, quality, Predictive
and cycle time”
[A57] “Maturity in the new product development process Predictive
will be correlated positively with project
outcomes”
[A58] “Customers and prospective customers have high Predictive
expectations with regard to the service quality
of software providers appraised in
CMM/CMMI”
[A95] (1) “Higher levels of process‐maturity lead to Predictive
higher product‐quality in software products”
(2)“Higher levels of process‐maturity are
associated with increased cycle‐time in
software products”
(3) “Higher product‐quality is associated with
lower cycle‐time in software products”
(4) “Higher levels of process‐maturity are
associated with increased development‐effort
in software products”
(5) “Higher product‐quality is associated with
lower development‐effort in software products”
[A110] “Relationship between CMM software process Predictive
development activities and project performance”
[A117] (1) “Achievement of the Maturity Level (ML) 2 Convergent and discriminant
KPAs (Project Implementation) enables the
implementation of the ML 3 KPAs
(Organization Implementation)”
(2) “Achievement of the ML 3 KPAs
(Organization Implementation) enables the
implementation of the KPAs at the MLs
4 and 5 (Quantitative Process Implementation)”
(3) Achievement of the ML2 KPAs
(Project Implementation) cannot directly enable
the implementation of the KPAs at MLs 4 and 5
(Quantitative Process Implementation)
[A118] “Increasing ML reduces schedule deviation in Predictive
software maintenance”
[A132] “Consistent adoption of the CMM practices Predictive
will be associated with lower number of defects”
[A189] “There is no significant change in the important Predictive
indicators of organizational
performances (IOPs) (i.e., customer satisfaction,
employee morale, profitability, overall productivity,
reduction in quality cost, overall financial
performance and overall operational
performance) due to the implementation of CMM”
[A247] “Organizations in different levels of CMM Predictive
(CMM II,CMMIII, and CMM IV)
exhibit different levels of IS project outcomes
as measured by software quality and
project performance”
CoPS RM‐CMM [A211] “Risk management capability maturity level Predictive
correlates positively with project
performance”
CI maturity model [A49] (1) “The CI maturity model consists of six reliable Predictive
and valid CI abilities”
(2) “CI is practiced on five different maturity levels”
(3) “Increases in CI maturity, in terms of
adoption of CI behaviors, will correspond to
improvements in operating performance”
[A116] “Increases in CI maturity, measured in terms Predictive
of the level of adoption of CI behaviors, will
correspond to improvements in performance”
CRM3 [A235] “Factors affecting CRM maturity are associated with Predictive
culture, information infrastructure of the
organization, vison of change, management
support and structure”

(Continues)
LEE ET AL. 19 of 30

TABLE B2 (Continued)

Maturity Model Ref Article ID Hypotheses Validity Tested

Data quality management [A220] (1) “Data quality level improves as data quality Predictive and discriminant
maturity model management level matures”
(2) “The growth of enterprise integration data
quality is greater for level 2 ➔ level 3
than level 1 ➔ level 2”
DWP‐M [A229] (1) “Theorized indicators converge together on Convergent, discriminant, and predictive
appropriate constructs and discriminate
across multiple constructs”
(2) “Constructs (e.g., data quality, alignment of
architecture, and change management) are
associated with perceptions of data
warehousing process maturity”
IRI sustainability [A106] (1) “The tool provides a valid statistical Convergent, discriminant, and predictive
maturity model discrimination between the various
maturity levels”
(2) “There is a linear relationship between IRI
sustainability tool scores and increased
operating margins”
IDEAL model [A285] “Top management support factors have a Predictive
positive impact on project success”
IT maturity model (Leem et al) [A141] “IT maturity model are defined by five stages, Convergent and discriminant
and each of which has unique characteristics”
Knowledge management maturity [A127] (1) “There is positive link between an indicator Convergent, discriminant, and
model (Khatibian et al) and knowledge management maturity” construct specification
(2) “There is no significant difference in the
current situation of each factor's variables number”
(3) “There is no significant difference in the
suggested levels of factors (or indicators)”
Knowledge management maturity [A38] (1) “The knowledge management capability can Construct specification,
model (KMMM) (Chen and Fong) be measured by the dimensions of both unidimensionality, and predictive
knowledge governance mechanisms
and knowledge process”
(2) “The association between knowledge governance
mechanisms and business performance is
mediated by knowledge processes”
Knowledge management maturity [A188] “KM practices can be classified into five Construct specification and
model (Oliva) explanatory factors, which are associated with unidimensionality
four level of maturity of KM”
KNM [A103] (1) “The KM maturity level can be defined as five Unidimensionality
stages: knowledge chaotic stage, knowledge
conscientious stage, KM stage, KM
advanced stage, and KM integration stage”
(2) “The evaluation framework for KNM consists
of three aspects: three target management
objects, 68 KM activities, and 16 key
areas (KAs)”
Maturity model for IT outsourcing [A239] “The maturity model of IT outsourcing relationships Discriminant
relationships shows predictable patterns of growth from
cost stage, to resource stage and into
partnership stage”
OPM3 [A78] “There is meaningful contingency between the Predictive
maturity state of companies with their
proven capabilities for winning
international projects tenders”
PMM [A259] “The alignment of PACS, as represented by the Discriminant and predictive
multifactorial nature of five organizational sectors
and their related maturity levels, has a
positive relationship on PACS performance,
as represented by the
multifactorial nature in terms of hospital efficiency
and clinical effectiveness and their related items”
PRMM [A96] “The original model which classify items within sub Convergent and discriminant
dimensions on the basis of theories, concepts,
or past models are supported by
empirical evidence”

(Continues)
20 of 30 LEE ET AL.

TABLE B2 (Continued)

Maturity Model Ref Article ID Hypotheses Validity Tested

RE‐CMM
Road safety maturity model
Safety culture maturity model
SMM
SMQM maturity model
Software product line engineering
maturity model: Business
Software product line engineering
maturity model: Organization
Supply chain maturity model
[A190] “Each of the four processes (e.g., engineering Construct specification
organization, engineering management,
and engineering technology) has an
impact on target variable reliability
engineering capability”
[A9] “Correlation between the maturity of a county's Predictive
road safety practices and road safety incidents”
[A66] “Five aspects of organizational safety are an Construct specification
indicator of five levels of cultural maturity”
[A136] “Correlation between metros' safety maturity Predictive
between their actual performance”
[A5] “The use of the maturity model that is reinforced Predictive
by the mnemonic cognitive cues does not
improve the proportion of hazards identified
and discussed prior to commencing work.”
[A2] “The maturity of the business dimension of Predictive
software product line as a function of how
a set of business practices are aligned
with product line engineering in an
organization”
[A3] Organizational factors are associated with Multidimensional construct
the maturity levels
[A145] “There is a statistical significant association Predictive
between the levels of supply
chain process maturity and performance”
[A168] “The impact of process maturity and uncertainty Predictive
on supply chain performance”
[A234] (1) “Higher levels of SCM maturity lead to Predictive
better supply chain performance”
(2) “Higher levels of SCM maturity lead to
better financial performance”

TABLE B3 Validated maturity models by application sector

No. of Validated Models Validity Tested
Application Sector No. of Modelsa Partially Yes Statistically Validated Maturity Model(s) U CS MC C D P
Consumer 2 0 0
discretionary
Consumer staples 2 0 0
Education 10 0 0
Energy 3 0 1 Safety culture maturity model ■
Financials 2 0 0
General 24 2 9 Berkley project management (PM) maturity model ■
CMMIb ■ ■ ■
Continuous improvement (CI) maturity model ■
Customer relationship management maturity model (CRM3) ■
Industrial research institute (IRI) sustainability maturity ■ ■ ■
model
Knowledge management maturity model (Khatibian et al) ■ ■ ■
Knowledge management maturity model (Oliva) ■ ■
Knowledge navigator model (KNM) ■
Supply chain maturity modelb ■
Government 16 0
Health care 11 1 Picture archiving and communication system (PACS) ■ ■
maturity model (PMM)
Industrials 31 0 6 Organizational project management maturity model (OPM3) ■
Project risk maturity model (PRMM) ■ ■

(Continues)
LEE ET AL. 21 of 30

TABLE B3 (Continued)

No. of Validated Models Validity Tested

a
Application Sector No. of Models Partially Yes Statistically Validated Maturity Model(s) U CS MC C D P
Road safety maturity model ■
Safety maturity model (SMM) ■
Safety meeting quality measurement (SMQM) ■
maturity model
Supply chain maturity modelb ■
IT 92 3 11 CMMIb ■ ■ ■
Complex product and systems risk management ■
CMM (CoPS RM‐CMM)
Data quality management maturity model ■ ■
Data warehousing process maturity model ■ ■ ■
(DWP‐M)
Initial, developed, evolved, advanced, leader ■
(IDEAL) maturity model
IT maturity model (Leem et al) ■ ■
Knowledge management maturity model ■ ■ ■
(KMMM) (Chen and Fong)
Maturity model for IT outsourcing relationships ■
Reliability engineering CMM (RE‐CMM) ■
Software product line engineering maturity ■
model: Business
Software product line engineering maturity ■
model: Organization
Law 3 0 0
Materials 5 0 0
Telecommunication 1 0 0
Utilities 2 0 0

Abbreviations: C, convergent validity; CS, construct specification; D, discriminant validity; MC, multidimensional construct; P, predictive validity; U, unidi-
mensionality; ■, area of the validity tested.
a
A distinct maturity model may appear in multiple sectors.
b
The models that appear in multiple sectors.

APPENDIX C
S E L E C T ED A R T I C LE S F O R A N A L Y S I S
[A1] Agrawal M, Chari K. Software effort, quality, and cycle time: a study of CMM level 5 projects. IEEE Transactions on Software Engineering 2007,
33:145‐156.
[A2] Ahmed F, Capretz L. A business maturity model of software product line engineering. Information Systems Frontiers 2011, 13:543‐560.
[A3] Ahmed F, Capretz LF. An organizational maturity model of software product line engineering. Software Quality Journal 2010, 18:195‐225.
[A4] Ahmed F, Capretz LF. An architecture process maturity model of software product line engineering. Innovations in Systems and Software
Engineering 2011, 7:191‐207.
[A5] Albert A, Hallowell MR, Kleiner BM. Enhancing construction hazard recognition and communication with energy‐based cognitive mnemonics
and safety meeting maturity model: multiple baseline study. Journal of Construction Engineering and Management 2014, 140:1‐12.
[A6] Aliquo Jr JF, Zhiwei. DuPont drives continuous improvement with COBIT 5 process assessment model. COBIT Focus 2014, 2:1‐7.
[A7] Allenbach RL, Huffman JE. Improving simulation engineering practices I—a capability maturity model for simulation processes improvement.
International Journal of Industrial Engineering: Theory Applications and Practice 1998, 5:150‐156.
[A8] Alyahya M, Ahmad R, Lee S. Impact of CMMI‐based process maturity levels on effort, productivity and diseconomy of scale. International Arab
Journal of Information Technology 2012, 9:352‐360.
[A9] Amador L, Willis CJ. Demonstrating a correlation between the maturity of road safety practices and road safety incidents. Traffic Injury
Prevention 2014, 15:591‐597.
[A10] Andersen KV, Henriksen HZ. E‐government maturity models: extension of the Layne and Lee model. Government Information Quarterly
2006, 23:236‐248.
[A11] Antoniol G, Gradara S, Venturi G. Methodological issues in a CMM level 4 implementation. Software Process: Improvement and Practice 2004,
9:33‐50.
[A12] Antunes P, Carreira P, da Silva MM. Towards an energy management maturity model. Energy Policy 2014, 73:803‐814.
[A13] April A, Abran A. A software maintenance maturity model (SMMM): measurement practices at maturity levels 3 and 4. Electronic Notes in
Theoretical Computer Science 2009, 233:73‐87.
22 of 30 LEE ET AL.

[A14] Asadi Dizaji F, Mohammadian A, Asadi Dizaji A, Aghdam Shahryar H. Business‐IT alignment in collaborative networked organizations. Inter-
national Business Management 2012, 6:99‐101.
[A15] Ashrafi N. The impact of software process improvement on quality: in theory and practice. Information & Management 2003, 40:677‐690.
[A16] Basavaraj MJ, Shet KC. Estimating and prediction of turn around time for incidents in application service maintenance projects. Journal of
Software 2008, 3:12‐21.
[A17] Bass JM. An early‐stage ICT maturity model derived from Ethiopian education institutions. International Journal of Education & Development
using Information & Communication Technology 2011, 7:5‐25.
[A18] Bates JJ, Privette JL. A maturity model for assessing the completeness of climate data records. Eos 2012, 93:441.
[A19] Batista J, Dias de Figueiredo A. SPI in a very small team: a case with CMM. Software Process: Improvement and Practice 2000, 5:243‐250.
[A20] Battista C, Schiraldi MM. The logistic maturity model: application to a fashion company. International Journal of Engineering Business
Management 2013, 5.
[A21] Beecham S, Hall T, Austen R. Defining a requirements process improvement model. Software Quality Journal 2005, 13:247‐279.
[A22] Beecham S, Hall T, Britton C, Cottee M, Rainer A. Using an expert panel to validate a requirements process improvement model. The Journal
of Systems & Software 2005, 76:251‐275.
[A23] Belt P, Oiva‐Kess A, Harkonen J, Mottonen M, Kess P. Organisational maturity and functional performance. International Journal of
Management and Enterprise Development 2009, 6:147‐164.
[A24] Berenbach BA, Spool PR, Bitterle D. The application of modern software engineering practices to control engineering. Journal of Advanced
Manufacturing Systems 2003, 2:127‐141.
[A25] Berg P, Pihlajamaa J, Poskela J, Smedlund A. Benchmarking of quality and maturity of innovation activities in a networked environment.
International Journal of Technology Management 2006, 33:255‐278.
[A26] Billings C, Clifton J, Kolkhorst B, Lee E, Wingert WB. Journey to a mature software process. IBM Systems Journal 1994, 33:46‐61.
[A27] Boughzala I. A community maturity model: a field application for supporting new strategy building. Journal of Decision Systems 2014, 23:82‐98.
[A28] Boughzala I, Bououd I. The development and application of a community maturity model. International Journal of Information Technology and
Management 2013, 12:273‐297.
[A29] Brodman JC, Johnson DL. Return on investment (ROl) from software process improvement as measured by US industry. Software Process:
Improvement and Practice 1995:35‐47.
[A30] Buchalcevova A. Green ICT maturity model for Czech SMEs. Journal of Systems Integration 2015, 6:24‐36.
[A31] Cacas M. New model sparks safeguards to the grid. Signal 2012, 66:35‐36.
[A32] Campos C, Chalmeta R, Grangel R, Poler R. Maturity model for interoperability potential measurement. Information Systems Management
2013, 30:218‐234.
[A33] Caniato F, Golini R, Luzzini D, Ronchi S. Towards full integration: EProcurement implementation stages. Benchmarking 2010, 17:491‐515.
[A34] Carcary M. IT risk management: a capability maturity model perspective. Electronic Journal of Information Systems Evaluation 2013, 16:3‐13.
[A35] Cepeda S, Garcia S, Langhout J. Is CMMI useful and usable in small settings? One example. CrossTalk 2008, 21:14‐18.
[A36] Chafik O, Zitouni B, Said M, Said B. A maturity model for SCPMS project‐an empirical investigation in large sized Moroccan companies.
International Journal of Computer Science Issues 2011:203‐212.
[A37] Chen CY, Chen PC, Chen PY. Teaching quality in higher education: an introductory review on a process‐oriented teaching‐quality model.
Total Quality Management and Business Excellence 2013, 25:36‐56.
[A38] Chen L, Fong PSW. Revealing performance heterogeneity through knowledge management maturity evaluation: a capability‐based
approach. Expert Systems with Applications 2012, 39:13523‐13539.
[A39] Chinowsky P, Molenaar K, Realph A. Learning organizations in construction. Journal of Management in Engineering 2007, 23:27‐34.
[A40] Chuah M‐H, Wong K‐L. A framework for accessing an enterprise business intelligence maturity model (EBI2M): Delphi study approach.
African Journal of Business Management 2012, 6:6880‐6889.
[A41] Cleven A, Winter R, Wortmann F, Mettler T. Process management in hospitals: an empirically grounded maturity model. Business Research
2014, 7:191‐216.
[A42] Concha G, Astudillo H, Porrua M, Pimenta C. E‐Government procurement observatory, maturity model and early measurements.
Government Information Quarterly 2012, 29:S43‐S50.
[A43] Costello T. A new management framework for IT. IT Professional 2010, 12:61‐64.
[A44] Craig R. Measure twice and cut once. CrossTalk 2007, 20:8‐12.
[A45] Crawford JK. The project management maturity model. Information Systems Management 2006, 23:50‐58.
[A46] Cronemyr P, Danielsson M. Process management 1‐2‐3—a maturity model and diagnostics tool. Total Quality Management and Business
Excellence 2013, 24:933‐944.
[A47] Cuenca L, Boza A, Alemany MME, Trienekens JJM. Structural elements of coordination mechanisms in collaborative planning processes and
their assessment through maturity models: application to a ceramic tile company. Computers in Industry 2013, 64:898‐911.
LEE ET AL. 23 of 30

[A48] Curtis B, Hefley WE. Developing organizational competence. Computer 1997, 30:122‐124.
[A49] Dabhilkar M, Bengtsson L, Bessant J. Convergence or national specificity? Testing the CI maturity model across multiple countries. Creativity
and Innovation Management 2007, 16:348‐362.
[A50] Dangle KC, Larsen P, Shaw M, Zelkowitz MV. Software process improvement in small organizations: a case study. IEEE Software 2005,
22:68‐75.
[A51] das Neves LAL, Nunes LENP, Corrêa VA, Rezende MC. Application of the Prado—project management maturity model at a R&D institution
of the Brazilian Federal Government. Journal of Aerospace Technology and Management 2013, 5:459‐465.
[A52] Davis PR, Walker DHT. Building capability in construction projects: a relationship‐based approach. Engineering Construction & Architectural
Management 2009, 16:475‐489.
[A53] de Oliveira SB, Valle R, Mahler CF. A comparative analysis of CMMI software project management by Brazilian, Indian and Chinese
companies. Software Quality Journal 2010, 18:177‐194.
[A54] Dettbarn Jr JL, Ibbs CW, Murphree Jr EL. Capital project portfolio management for federal real property. Journal of Management in
Engineering 2005, 21:44‐53.
[A55] Diaz M, Sligo J. How software process improvement helped Motorola. IEEE Software 1997, 14:75‐81.
[A56] Diaz‐Ley M, Garcia F, Piattini M. MIS‐PyME software measurement capability maturity model—supporting the definition of software
measurement programs and capability determination. Advances in Engineering Software 2010, 41:1223‐1237.
[A57] Dooley K, Subra A, Anderson J. Maturity and its impact on new product development project performance. Research in Engineering Design
2001, 13:23‐29.
[A58] Dos Santos RP, De Oliveira KM, Da Silva WP. Evaluating the service quality of software providers appraised in CMM/CMMI. Software Qual-
ity Journal 2009, 17:283‐301.
[A59] Ekionea JPB, Fillion G, Koffi V. Improving municipal information and knowledge management capabilities: case study. Journal of Information
and Knowledge Management 2012, 11:1‐21.
[A60] El‐Baz H, Zualkernan IA. Employee competency maturity model and its application in global software outsourcing. International Journal of
Computer Applications in Technology 2011, 40:170‐180.
[A61] El‐Gayar OF, Deokar AV, Tao J. DSS‐CMM: A capability maturity model for DSS development processes. International Journal of Decision
Support System Technology 2011, 3:14‐34.
[A62] Eom S‐J, Kim JH. The adoption of public smartphone applications in Korea: empirical analysis on maturity level and influential factors.
Government Information Quarterly 2014, 31:S26‐S36.
[A63] Ericson T, Subotic A, Ursing S. TIM—a test improvement model. Software Testing, Verification and Reliability 1997, 7:229‐246.
[A64] Falessi D, Shaw M, Mullen K. Achieving and maintaining CMMI maturity level 5 in a small organization. IEEE Software 2014, 31:80‐86.
[A65] Fielden K. Evaluating capstone project coordination and supervision with a maturity model: a comparative case study. New Zealand Journal
of Applied Computing & Information Technology 2004, 8:30‐34.
[A66] Filho APG, Andrade JCS, Marinho MMD. A safety culture maturity model for petrochemical companies in Brazil. Safety Science 2010,
48:615‐624.
[A67] Fitterer R, Rohner P. Towards assessing the networkability of health care providers: a maturity model approach. Information Systems and e‐
Business Management 2010, 8:309‐333.
[A68] Fitzgerald B, O'Kane T. A longitudinal study of software process improvement. IEEE Software 1999, 16:37‐45.
[A69] Fleming M, Wentzell N. Patient safety culture improvement tool: development and guidelines for use. Healthcare quarterly 2008, 11:10‐15.
[A70] Foreman RD, Jones JW, Ducharme DJ, Forrester D. BP describes maturity model for IT in its E&P organization. Oil & Gas Journal 2007,
105:36‐44.
[A71] Foster P, Hoult S. The safety journey: using a safety maturity model for safety planning and assurance in the UK coal mining industry.
Minerals 2013, 3:59‐72.
[A72] Fraser MD, Vaishnavi VK. A formal specifications maturity model. Communications of the ACM 1997, 40:95‐103.
[A73] Galin D, Avrahami M. Are CMM program investments beneficial? Analyzing past studies. IEEE Software 2006, 23:81.
[A74] Garrett GA. How to evaluate a purchasing system: tools, techniques, and best practices. Contract Management 2013, 53:40‐51.
[A75] Garrett GA, Rendon RG. Managing contracts in turbulent times: the contract management maturity model. Contract Management 2005,
45:48‐57.
[A76] Garrett GA, Rendon RG. Improving the U.S. Fedral Acquisition Workforce, part 2 of 3—contract management process maturity: the key for
organizational survival. Contract Management 2015, 55:78‐87.
[A77] Garzas J, Pino FJ, Piattini M, Fernandez CM. A maturity model for the Spanish software industry based on ISO standards. Computer
vStandards & Interfaces 2013, 35:616‐628.
[A78] Ghoddousi P, Amini Z, Hosseini MR. A survey on the maturity state of Iranian grade one construction companies utillzing OPM3 maturity
model. Technics Technologies Education Management‐Ttem 2011, 6:69‐77.
24 of 30 LEE ET AL.

[A79] Gillies A. Information support for general practice in the new NHS. Health Libraries Review 2000, 17:91‐96.
[A80] Gillies A. Assessing and improving the quality of information health evaluation and promotion. Methods of Information in Medicine 2000,
39:208‐212.
[A81] Gökhan A, Albert W, Alexander R. Design of a maturity model for new forming processes considering uncertain values. Quality ‐ Access to
Success 2012:47‐52.
[A82] Gottschalk P. Maturity levels for criminal organizations. International Journal of Law, Crime and Justice 2008, 36:106‐114.
[A83] Gottschalk P, Solli‐Sæther H. Maturity model for IT outsourcing relationships. Industrial Management & Data Systems 2006, 106:200‐212.
[A84] Gottschalk P, Tolloczko PC. Maturity model for mapping crime in law enforcement. Electronic Government 2007, 4:59‐67.
[A85] Grant KP, Pennypacker JS. Project management maturity: an assessment of project management capabilities among and between selected
industries. IEEE Transactions on Engineering Management 2006, 53:59‐68.
[A86] Grossi L, Calvo‐Manzano JA, San Feliu T. High‐maturity levels: achieving CMMI ML‐5 in a consultancy company. Journal of Software:
Evolution and Process 2014, 26:808‐817.
[A87] Guédria W. A conceptual framework for enterprise interoperability. International Journal of e‐Business Research 2014, 10:54‐64.
[A88] Guédria W, Bouzid H, Bosh G, Naudet Y, Chen D. eHealth interoperability evaluation using a maturity model. Studies in Health Technology &
Informatics 2012, 180:333‐337.
[A89] Guédria W, Naudet Y, Chen D. Maturity model for enterprise interoperability. Enterprise Information Systems 2015, 9:1‐28.
[A90] Hackos JT. From theory to practice: using the information process‐maturity model as a tool for strategic. Technical Communication 1997,
44:369‐381.
[A91] Hallam G, Hiskens A, Ong R. Conceptualising the learning organisation: creating a maturity framework to develop a shared understanding of
the library's role in literacy and learning. Australian Library Journal 2014, 63:78‐93.
[A92] Hameri AP, McKay KN, Wiers VCS. A maturity model for industrial supply chains. Supply Chain Forum 2013, 14:2‐15.
[A93] Hardgrave BC, Armstrong DJ. Software process improvement: it's a journey, not a destination. Communications of the ACM 2005, 48:93‐96.
[A94] Harigopal U, Satyadas A. Cognizant enterprise maturity model (CEMM). IEEE Transactions on Systems, Man and Cybernetics Part C:
Applications and Reviews 2001, 31:449‐459.
[A95] Harter DE, Krishnan MS, Slaughter SA. Effects of process maturity on quality, cycle time, and effort in software product development.
Management Science 2000, 46:451‐466.
[A96] Hartono B, Wijaya DFN, Arini HM. An empirically verified project risk maturity model. International Journal of Managing Projects in Business
2014, 7:263‐284.
[A97] Herz TP, Hamel F, Uebernickel F, Brenner W. A multisourcing maturity model as an IT governance mechanism for business groups.
International Journal on IT/Business Alignment & Governance 2011, 2:1‐14.
[A98] Hillson D. Assessing organisational project management capability. Journal of Facilities Management 2003, 2:298‐311.
[A99] Hirschheim R, Schwarz A, Todd P. A marketing maturity model for IT: building a customer‐centric IT organization. IBM Systems Journal 2006,
45:181‐199.
[A100] Hirschheim R, Welke R, Schwarz A. Service‐oriented architecture: myths, realities, and a maturity model. MIS Quarterly Executive 2010,
9:37‐48.
[A101] Hogan TJ. The adaptive leadership maturity model. Organization Development Journal 2008, 26:55‐61.
[A102] Holland CP, Light B. A stage maturity model for Enterprise Resource Planning systems use. Data Base for Advances in Information Systems
2001, 32:34‐45.
[A103] Hsieh PJ, Lin B, Lin C. The construction and application of knowledge navigator model (KNM™): an evaluation of knowledge management
maturity. Expert Systems with Applications 2009, 36:4087‐4100.
[A104] Huestis EM, Snowdon JL. Complexity of legacy city resource management and value modeling of interagency response. IBM Journal of
Research & Development 2011, 55:1:1‐1:12.
[A105] Hyde K, Wilson D. Intangible benefits of CMM‐based software process improvement. Software Process: Improvement and Practice 2004,
9:217‐228.
[A106] Hynds EJ, Brandt V, Burek S, Jager W, Knox P, Parker JP, Schwartz L, Taylor J, Zietlow M. A maturity model for sustainability in new
product development. Research Technology Management 2014, 57:50‐57.
[A107] Introna V, Cesarotti V, Benedetti M, Biagiotti S, Rotunno R. Energy management maturity model: an organizational tool to foster the
continuous reduction of energy consumption in companies. Journal of Cleaner Production 2014, 83:108‐117.
[A108] Jeffery M, Leliveld I. Best practices in IT portfolio management. MIT Sloan Management Review 2004, 45:41‐49.
[A109] Jia G, Chen Y, Xue X, Chen J, Cao J, Tang K. Program management organization maturity integrated model for mega construction programs
in China. International Journal of Project Management 2011, 29:834‐845.
[A110] Jiang JJ, Klein G, Hwang HG, Huang J, Hung SY. An exploration of the relationship between software development process maturity and
project performance. Information and Management 2004, 41:279‐288.
LEE ET AL. 25 of 30

[A111] Jin D, Chai KH, Tan KC. New service development maturity model. Managing Service Quality 2014, 24:86‐116.
[A112] Jithinan S, Peraphon S, Decha D. Using test employee capability maturity model for supporting gaps bridging in software testing. Journal of
Software Engineering and Applications 2012:417‐428.
[A113] Jochem R, Geers D, Heinze P. Maturity measurement of knowledge‐intensive business processes. The TQM Journal 2011, 23:377‐387.
[A114] Johansson C, Hicks B, Larsson AC, Bertoni M. Knowledge maturity as a means to support decision making during product‐service systems
development projects in the aerospace sector. Project Management Journal 2011, 42:32‐50.
[A115] Johnson DL, Brodman JG. Applying CMM project planning practices to diverse environments. IEEE software 2000, 17:40‐47.
[A116] Jørgensen F, Boer H, Laugen BT. CI implementation: an empirical test of the CI maturity model. Creativity & Innovation Management 2006,
15:328‐337.
[A117] Jung H‐W, Goldenson DR. The internal consistency and precedence of key process areas in the capability maturity model for software.
Empirical Software Engineering 2008, 13:125‐146.
[A118] Jung H‐W, Goldenson DR. Evaluating the relationship between process improvement and schedule deviation in software maintenance.
Information and Software Technology 2009, 51:351‐361.
[A119] Kajko‐Mattsson M. Problem management maturity within corrective maintenance. Journal of Software Maintenance and Evolution: Research
and Practice 2002, 14:197‐227.
[A120] Kajko‐Mattsson M, Forssander S, Andersson G, Olsson U. Developing CM3: maintainers' education and training at ABB. Computer Science
Education 2002, 12:57‐89.
[A121] Kandt RK. Experiences in improving flight software development processes. IEEE Software 2009, 26:58‐64.
[A122] Kang Y, O'Brien WJ, O'Connor JT. Information‐integration maturity model for the capital projects industry. Journal of Management in
Engineering 2015, 31.
[A123] Kans M. Maturity based evaluation of IT systems for maintenance management. International Journal of COMADEM 2012, 15:15‐25.
[A124] Kerrigan M. A capability maturity model for digital investigations. Digital Investigation 2013, 10:19‐33.
[A125] Khadra HA, Zuriekat M, Alramhi N. An empirical examination of maturity model as measurement of information technology governance
implementation. International Arab Journal of Information Technology 2009, 6:310‐319.
[A126] Khaiata M, Zualkernan IA. A simple instrument to measure IT‐business alignment maturity. Information Systems Management 2009, 26:138‐
152.
[A127] Khatibian N, pour THg, Jafari HA. Measurement of knowledge management maturity level within organizations. Business Strategy Series
2010, 11:54‐70.
[A128] Killen CP, Hunt RA. Robust project portfolio management: capability evolution and maturity. International Journal of Managing Projects in
Business 2013, 6:131‐151.
[A129] Kim DY, Grant G. E‐government maturity model using the capability maturity model integration. Journal of Systems and Information
Technology 2010, 12:230‐244.
[A130] Kollanus S. Experiences from using ICMM in inspection process assessment. Software Quality Journal 2009, 17:177‐187.
[A131] Kollanus S. ICMM—a maturity model for software inspections. Journal of Software Maintenance and Evolution: Research and Practice 2011,
23:327‐341.
[A132] Krishnan MS, Kellner MI. Measuring process consistency: implications for reducing software defects. IEEE Transactions on Software
Engineering 1999, 25:800‐815.
[A133] Kruger CJ, Snyman MMM. Formulation of a strategic knowledge management maturity model. South African Journal of Information
Management 2005, 7:1‐1.
[A134] Kruger CJN, Johnson RD. Is there a correlation between knowledge management maturity and organizational performance? VINE 2011,
41:265‐295.
[A135] Kwak YH. Calculating project management's return on investment. Project Management Journal 2000, 31:38‐49.
[A136] Kyriakidis M, Hirsch R, Majumdar A. Metro railway safety: an analysis of accident precursors. Safety Science 2012, 50:1535‐1548.
[A137] Lahti M, Shamsuzzoha AHM, Helo P. Developing a maturity model for supply chain management. International Journal of Logistics Systems
and Management 2009, 5:654‐678.
[A138] Lee G, Kwak YH. An open government maturity model for social media‐based public engagement. Government Information Quarterly 2012,
29:492‐503.
[A139] Lee I‐L, SunTae K, Soojin P. Validating the software process assessment model for Korean military software industry. International Journal of
Software Engineering and Its Applications 2014, 8:95‐112.
[A140] Lee J, Suh Eh, Hong J. A maturity model based CoP evaluation framework: a case study of strategic CoPs in a Korean company. Expert
Systems with Applications 2010, 37:2670‐2681.
[A141] Leem CS, Kim BW, Yu EJ, Paek MH. Information technology maturity stages and enterprise benchmarking: an empirical study. Industrial
Management & Data Systems 2008, 108:1200‐1218.
26 of 30 LEE ET AL.

[A142] Leem CS, Yoon Y. A maturity model and an evaluation system of software customer satisfaction: the case of software companies in Korea.
Industrial Management & Data Systems 2004, 104:347‐354.
[A143] Li H, Ji Y, Gu X, Bao Z, Qi G. A universal enterprise manufacturing services maturity model: a case study in a Chinese company. International
Journal of Computer Integrated Manufacturing 2014, 27:434‐449.
[A144] Lin C, Wu JC, Yen DC. Exploring barriers to knowledge flow at different knowledge management maturity stages. Information and Manage-
ment 2012, 49:10‐23.
[A145] Lockamy III A, Childerhouse P, Disney SM, Towill DR, McCormack K. The impact of process maturity and uncertainty on supply chain
performance: an empirical study. International Journal of Manufacturing Technology & Management 2008, 15:12‐27.
[A146] Lockamy III A, McCormack K. The development of a supply chain management process maturity model using the concepts of business
process orientation. Supply Chain Management: An International Journal 2004, 9:272‐278.
[A147] Lowe DE, Cox GM. Implementing the capability maturity model for software development. Hewlett‐Packard Journal 1996, 47:6‐14.
[A148] Luftman J. Assessing IT/business alignment. Information Systems Management 2003, 20:9‐15.
[A149] Luftman J, Kempaiah R. An update on business‐IT alignment: “a line” has been drawn. MIS Quarterly Executive 2007, 6:165‐177.
[A150] Maatta J, Harkonen J, Jokinen T, Mottonen M, Belt P, Muhos M, Haapasalo H. Managing testing activities in telecommunications: a case
study. Journal of Engineering and Technology Management 2009, 26:73‐96.
[A151] MacGillivray BH, Pollard SJT. What can water utilities do to improve risk management within their business functions? An improved tool
and application of process benchmarking. Environment International 2008, 34:1120‐1131.
[A152] MacGillivray BH, Sharp JV, Strutt JE, Hamilton PD, Pollard SJT. Benchmarking risk management within the international water utility
sector. Part II: a survey of eight water utilities. Journal of Risk Research 2007, 10:105‐123.
[A153] MacGillivray BH, Sharp JV, Strutt JE, Hamilton PD, Pollard SJT. Benchmarking risk management within the international water utility
sector. Part I: design of a capability maturity methodology. Journal of Risk Research 2007, 10:85‐104.
[A154] Macre E. A framework for audit evolution. Internal Auditor 2010, 67:68‐69.
[A155] Madritsch T, Ebinger M. A management framework for the built environment: BEM2/BEM3. Built Environment Project and Asset
Management 2011, 1:111‐121.
[A156] Madritsch T, Ebinger M. Performance measurement in facility management the environment management maturity model Bem3. Research
Journal of Economics, Business and ICT 2013:4‐10.
[A157] Magdaleno AM, Cappelli C, Araujo Baiao F, Maria Santoro F, Araujo R. Towards collaboration maturity in business processes: an explor-
atory study in oil production processes. Information systems management 2008, 25:302‐318.
[A158] Magdaleno AM, De Araujo RM, Borges MRDS. A maturity model to promote collaboration in business processes. International Journal of
Business Process Integration and Management 2009, 4:111‐123.
[A159] Margarido IL, Faria JP, Vidal RM, Vieira M. Challenges in implementing CMMI® high maturity: lessons learned and recommendations.
Software Quality Professional 2013, 16:4.
[A160] Marshall S. A quality framework for continuous improvement of e‐learning: the e‐learning maturity model. Journal of Distance Education
2010, 24:143‐165.
[A161] Marshall S. Improving the quality of e‐learning: lessons from the eMM. Journal of Computer Assisted Learning 2012, 28:65‐78.
[A162] Martin VA, Hatzakis T, Lycett M, Macredie R. Building the business/IT relationship through knowledge management. Journal of Information
Technology Cases & Applications 2004, 6:27‐47.
[A163] Martin VA, Hatzakis T, Lycett M, Macredie R. Cultivating knowledge sharing through the relationship management maturity model.
Learning Organization 2005, 12:340‐354.
[A164] Marx F, Wortmann F, Mayer JH. A maturity model for management control systems: five evolutionary steps to guide development.
Business and Information Systems Engineering 2012, 4:193‐207.
[A165] Matrane O, Talea M. A maturity model for information security management in small and medium‐sized Moroccan enterprises: an empirical
investigation. International Journal of Advanced Research in Computer Science 2014, 5:206‐210.
[A166] Mc Caffery F, Burton J, Richardson I. Risk management capability model for the development of medical device software. Software Quality
Journal 2010, 18:81.
[A167] McCarthy CF, Kelley MA, Verani AR, St. Louis ME, Riley PL. Development of a framework to measure health profession regulation
strengthening. Evaluation and Program Planning 2014, 46:17‐24.
[A168] McCormack K, Bronzo Ladeira M, Paulo Valadares de Oliveira M. Supply chain maturity and performance in Brazil. Supply Chain
Management: An International Journal 2008, 13:272‐282.
[A169] McCuen TL, Suermann PC, Krogulecki MJ. Evaluating award‐winning BIM projects using the national building information model standard
capability maturity model. Journal of Management in Engineering 2012, 28:224‐230.
[A170] McKeen JD, Smith HA. Developments in practice XXVII: delivering IT functions: a decision framework. Communications of the Association
for Information Systems 2007, 19.
LEE ET AL. 27 of 30

[A171] Mehta N, Oswald S, Mehta A. Infosys Technologies: improving organizational knowledge flows. Journal of Information Technology 2007,
22:456‐464.
[A172] Menezes W. Transitioning to the CMMI SM‐SE/SW—an evolutionary process. Systems Engineering 2002, 5:32‐40.
[A173] Meng X, Sun M, Jones M. Maturity model for supply chain relationships in construction. Journal of Management in Engineering 2011,
27:97‐105.
[A174] Mettler T. Transformation of the hospital supply chain: how to measure the maturity of supplier relationship management systems in
hospitals? International Journal of Healthcare Information Systems and Informatics 2011, 6:1‐13.
[A175] Mittermaier HK, Steyn H. Project management maturity: an assessment of maturity for developing pilot plants. South African Journal of
Industrial Engineering 2009, 20:95‐107.
[A176] Montoni MA, Rocha AR, Weber KC. MPS.BR: A successful program for software process improvement in Brazil. Software Process: Improve-
ment and Practice 2009, 14:289‐300.
[A177] Morasca S, Taibi D, Tosi D. OSS‐TMM: guidelines for improving the testing process of open source software. International Journal of Open
Source Software and Processes 2011, 3:1‐22.
[A178] Moreno OAV, Swarr TE, Asselin AC, Milà i Canals L, Colley T, Valdivia S. Implementation of life cycle management practices in a cluster of
companies in Bogota, Colombia. International Journal of Life Cycle Assessment 2015, 20:723‐730.
[A179] Mullaly M. Longitudinal analysis of project management maturity. Project Management Journal 2006, 37:62‐73.
[A180] Neff AA, Hamel F, Herz TP, Uebernickel F, Brenner W, vom Brocke J. Developing a maturity model for service systems in heavy equipment
manufacturing enterprises. Information & Management 2014, 51:895‐911.
[A181] Neuhauser C. A maturity model: does it provide a path for online course design? Journal of Interactive Online Learning 2004, 3:17p.
[A182] Ngai EWT, Chau DCK, Poon JKL, To CKM. Energy and utility management maturity model for sustainable manufacturing process. Interna-
tional Journal of Production Economics 2013, 146:453‐464.
[A183] Niazi M, Cox K, Verner J. A measurement framework for assessing the maturity of requirements engineering process. Software Quality
Journal 2008, 16:213‐235.
[A184] Niazi M, Wilson D, Zowghi D. A maturity model for the implementation of software process improvement: an empirical study. The Journal
of Systems & Software 2005, 74:155‐172.
[A185] Niessink F, Van Vliet H. Towards mature IT services. Software Process: Improvement and Practice 1998, 4:55‐77.
[A186] Nightingale DJ, Mize JH. Development of a lean enterprise transformation maturity model. Information Knowledge Systems Management
2002, 3:15‐30.
[A187] Niknam M, Bonnal P, Ovtcharova J. Configuration management maturity in scientific facilities. International Journal of Advanced Robotic
Systems 2013, 10.
[A188] Oliva FL. Knowledge management barriers, practices and maturity model. Journal of Knowledge Management 2014, 18:1053‐1074.
[A189] Padma P, Ganesh LS, Rajendran C. An exploratory study of the impact of the capability maturity model on the organizational performance
of Indian software firms. Quality Management Journal 2008, 15:20‐34.
[A190] Pan X, Xin Z, Li G. Organizational reliability capability assessment: a case study in China R&D enterprise for aviation products. IEEE
Transactions on Reliability 2015, 64:550‐561.
[A191] Pangeran MH, Pribadi KS, Wirahadikusumah RD, Notodarmojo S. Assessing risk management capability of public sector organizations
related to PPP scheme development for water supply in Indonesia. Civil Engineering Dimension 2012, 14:26‐35.
[A192] Parthasarathy S, Ramachandran M. Requirements engineering method and maturity model for ERP projects. International Journal of
Enterprise Information Systems 2008, 4:1‐14.
[A193] Pee LG, Kankanhalli A. A model of organisational knowledge management maturity based on people, process, and technology. Journal of
Information and Knowledge Management 2009, 8:79‐99.
[A194] Pennypacker JS, Grant KP. Project management maturity: an industry benchmark. Project Management Journal 2003, 34:4‐11.
[A195] Petch J, Calverley G, Dexter H, Cappelli T. Piloting a process maturity model as an e‐learning benchmarking method. Electronic Journal of
e‐Learning 2007, 5:49‐58.
[A196] Petrinja E, Succi G. Assessing the open source development processes using OMM. Advances in Software Engineering 2012:1‐17.
[A197] Phan DD. Software quality and management. Information Systems Management 2001, 18:56‐67.
[A198] Pigosso DCA, Rozenfeld H, McAloone TC. Ecodesign maturity model: a management framework to support ecodesign implementation into
manufacturing companies. Journal of Cleaner Production 2013, 59:160‐173.
[A199] Pita Z, Cheong F, Corbitt B. A maturity model of strategic information systems planning (SISP): an empirical evaluation using the analytic
network process. International Journal of Enterprise Information Systems 2011, 7:30‐57.
[A200] Plomp MGA, Batenburg RS. Measuring chain digitisation maturity: an assessment of Dutch retail branches. Supply Chain Management: An
International Journal 2010, 15:227‐237.
28 of 30 LEE ET AL.

[A201] Plomp MGA, Rijn GV, Batenburg RS. Chain digitisation support by point‐of‐sale systems: an analysis of the Dutch product software market.
International Journal of Information Technology and Management 2012, 11:257‐272.
[A202] Powell D, Riezebos J, Strandhagen JO. Lean production and ERP systems in small‐ and medium‐sized enterprises: ERP support for pull
production. International Journal of Production Research 2013, 51:395‐409.
[A203] Puus U, Mets T. Software development maturity evaluation: six cases from Estonian SMEs. Baltic Journal of Management 2010, 5:422‐
443.
[A204] Ramasubbu N, Mithas S, Krishnan MS, Kemerer CF. Work dispersion, process‐based learning, and offshore software development
performance. MIS Quarterly: Management Information Systems 2008, 32:437‐458.
[A205] Randeree K, Mahal A, Narwani A. A business continuity management maturity model for the UAE banking sector. Business Process
Management Journal 2012, 18:472‐492.
[A206] Rangarajan A, Parthasarathy R. Practical issues with CMM implementation in the banking industry: a real world practitioner's perspective.
ICFAI Journal of Systems Management 2007, 5:28‐35.
[A207] Rapaccini M, Saccani N, Pezzotta G, Burger T, Ganz W. Service development in product‐service systems: a maturity model. Service
Industries Journal 2012:300‐319.
[A208] Rašula J, Bosilj Vukšić V, Indihar Štemberger M. The integrated knowledge management maturity model. Zagreb International Review of
Economics and Business 2008, 11:47‐62.
[A209] Raza A, Capretz LF, Ahmed F. An open source usability maturity model (OS‐UMM). Computers in Human Behavior 2012, 28:1109‐1121.
[A210] Reefke H, Ahmed MD, Sundaram D. Sustainable supply chain management—decision making and support: the SSCM maturity model and
system. Global Business Review 2014, 15:1S‐12S.
[A211] Ren Y, Yeo KT, Ren Y. Risk management capability maturity and performance of complex product and system (CoPS) projects with an Asian
perspective. Journal of Engineering, Project & Production Management 2014, 4:81‐98.
[A212] Rendon RG. Procurement process maturity: key to performance measurement. Journal of Public Procurement 2008, 8:200‐214.
[A213] Reyes HG, Giachetti R. Using experts to develop a supply chain maturity model in Mexico. Supply Chain Management: An International
Journal 2010, 15:415‐424.
[A214] Rezaei R, Chiew TK, Lee SP. An interoperability model for ultra large scale systems. Advances in Engineering Software 2014, 67:22‐46.
[A215] Rhee H. Applying maturity framework to US cinema for analyzing the commercial culture industry. Asian Social Science 2015, 11:255‐
261.
[A216] Rigon EA, Westphall CM, Dos Santos DR, Westphall CB. A cyclical evaluation model of information security maturity. Information Manage-
ment and Computer Security 2013, 22:265‐278.
[A217] Rohloff M. Advances in business process management implementation based on a maturity assessment and best practice exchange.
Information Systems and e‐Business Management 2011, 9:383‐403.
[A218] Rossi R, Mustaro PN. eQETIC: A maturity model for online education. Interdisciplinary Journal of E‐Learning & Learning Objects 2015,
11:11‐23.
[A219] Rozman I, Vajde Horvat R, GyÓrkÓs J, Hericùko M. PROCESSUS—integration of SEI CMM and ISO quality models. Software Quality Journal
1997, 6:37‐63.
[A220] Ryu KS, Park JS, Park JH. A data quality management maturity model. ETRI Journal 2006, 28:191‐204.
[A221] Santos RS, Borges MRS, Canos JH, Gomes JO. The assessment of information technology maturity in emergency response organizations.
Group Decision and Negotiation 2011, 20:593‐613.
[A222] Sarshar M, Haigh R, Amaratunga D. Improving project processes: best practice case study. Construction Innovation 2004, 4:69‐82.
[A223] Sarshar M, Haigh R, Finnemore M, Aouad G, Barrett P, Baldry D, Sexton M. SPICE: a business process diagnostics tool for construction
projects. Engineering Construction & Architectural Management 2000, 7:241‐250.
[A224] Satrio S, Lumban GF. Measurement of capability maturity model integration implementation impact for application development process
improvement in telkomsigma. International Journal of Software Engineering and its Applications 2014, 8:103‐124.
[A225] Savino MM, Mazza A, Ouzrout Y. PLM maturity model: a multi‐criteria assessment in southern Italy companies. International Journal of
Operations and Quantitative Management 2012, 18:159‐180.
[A226] Schwomeyer W, A., Barner D, J., Gundrum V, McCray W, Vogel J, R. CMMISM transition experiences from an integrated product and
process development (IPPD) perspective. Systems Engineering 2002, 5:41‐51.
[A227] Secundo G, Perez SE‐, Martinaitis Ž, Leitner K‐H. An intellectual capital maturity model (ICMM) to improve strategic management in
European universities. Journal of Intellectual Capital 2015, 16:419‐442.
[A228] Sen A, Ramamurthy K, Sinha AP. A model of data warehousing process maturity. IEEE Transactions on Software Engineering 2012, 38:336‐
353.
[A229] Sen A, Sinha AP, Ramamurthy K. Data warehousing process maturity: an exploratory study of factors influencing user perceptions. IEEE
Transactions on Engineering Management 2006, 53:440‐455.
LEE ET AL. 29 of 30

[A230] Sheth S, McHugh J, Jones F. A dashboard for measuring capability when designing, implementing and validating business continuity and
disaster recovery projects. Journal of Business Continuity & Emergency Planning 2008, 2:221‐239.
[A231] Shrivastava A, Tripathi SP. A requirements engineering maturity measurement framework tool for medium and small scale software
companies. International Review on Computers and Software 2011, 6:725‐729.
[A232] Simangunsong E, Da Silva EN. Analyzing project management maturity level in Indonesia. South East Asian Journal of Management 2013,
7:72‐84.
[A233] Simon A, Schoeman P, Sohal AS. Prioritised best practices in a ratified consulting services maturity model for ERP consulting. Journal of
Enterprise Information Management 2010, 23:100‐124.
[A234] Söderberg L, Bengtsson L. Supply chain management maturity and performance in SMEs. Operations Management Research 2010, 3:90‐97.
[A235] Sohrabi B, Haghighi M, Khanlari A. Customer relationship management maturity model (CRM3): a model for stepwise implementation.
International Journal of Human Sciences 2010, 7:1‐20.
[A236] Solar M, Daniels F, López R, Meijueiro L. A model to guide the open government data implementation in public agencies. Journal of
Universal Computer Science 2014, 20:1564‐1582.
[A237] Solar M, Sabattin J, Parada V. A maturity model for assessing the use of ICT in school education. Journal of Educational Technology & Society
2013, 16:206‐218.
[A238] Solemon B, Sahibuddin S, Ghani AAA. A new maturity model for requirements engineering process: an overview. Journal of Software
Engineering and Applications 2012:340‐350.
[A239] Solli‐Sæether H, Gottschalk P. Maturity in IT outsourcing relationships: an exploratory study of client companies. Industrial Management &
Data Systems 2008, 108:635‐649.
[A240] Soltani I, Joneghani RBN, Bozorgzad A. A study on determining the level of individual, procedural and organizational maturity based on inte-
grated pattern of people–capability maturity model: (P‐CMM) and 3‐dimensional pattern of organizational maturity in production and industrial
organizations. International Business Research 2011:234‐242.
[A241] Sommerville I, Ransom J. An empirical study of industrial requirements engineering process assessment and improvement. ACM
Transactions on Software Engineering & Methodology 2005, 14:85‐117.
[A242] Spruit M, Pietzka K. MD3M: The master data management maturity model. Computers in Human Behavior 2014:1068‐1076.
[A243] Stephenson HLCCC. Cost engineering maturity model (CEMM). Cost Engineering 2011, 53:13‐41.
[A244] Storm I, Harting J, Stronks K, Schuit AJ. Measuring stages of health in all policies on a local level: the applicability of a maturity model.
Health policy 2014, 114:183‐191.
[A245] Stothard P, Swadling P. Assessment of maturity of mining industry simulation. Transactions of the Institutions of Mining and Metallurgy,
Section A: Mining Technology 2010, 119:102‐109.
[A246] Strutt JE, Sharp JV, Terry E, Miles R. Capability maturity models for offshore organizational management. Environment International 2006,
32:1094‐1105.
[A247] Subramanian GH, Jiang JJ, Klein G. Software quality and IS project performance improvements from software development process
maturity and IS implementation strategies. The Journal of Systems & Software 2007, 80:616‐627.
[A248] Succar B, Sher W, Williams A. Measuring BIM performance: five metrics. Architectural Engineering and Design Management 2012, 8:120‐142.
[A249] Sukhoo A, Barnard A, Eloff MM, Van Der Poll JA. An evolutionary software project management maturity model for Mauritius.
Interdisciplinary Journal of Information, Knowledge, and Management 2007, 2:99‐118.
[A250] Sunetnanta T, T., Choetkiertikul M. Quantitative CMMI assessment for software process quality and risk monitoring in software process
improvement. International Journal of Digital Content Technology & its Applications 2012, 6:95.
[A251] Svahnberg M, Gorschek T, Nguyen T, Nguyen M. Uni‐REPM: a framework for requirements engineering process assessment. Requirements
Engineering 2015, 20:91‐118.
[A252] Tiku S, Azarian M, Pecht M. Using a reliability capability maturity model to benchmark electronics companies. International Journal of Qual-
ity and Reliability Management 2007, 24:547‐563.
[A253] Twaites G, Sibilla ML. Software engineering in an SEI level‐5 organization. International Journal of Reliability, Quality & Safety Engineering
2002, 9:347‐365.
[A254] Unger CJ, Lechner AM, Kenway J, Glenn V, Walton A. A jurisdictional maturity model for risk management, accountability and continual
improvement of abandoned mine remediation programs. Resources Policy 2015, 43:1‐10.
[A255] Vakaslahti P. Process improvement frameworks—a small case study with people capability maturity model. Software Process: Improvement and
Practice 1997, 3:225‐234.
[A256] Valdes G, Solar M, Astudillo H, Iribarren M, Concha G, Visconti M. Conception, development and implementation of an e‐government
maturity model in public agencies. Government Information Quarterly 2011, 28:176‐187.
[A257] van Cleeff A. Future consumer mobile phone security: a case study using the data‐centric security model. Information Security Technical
Report 2008, 13:112‐117.
30 of 30 LEE ET AL.

[A258] van de Wetering R, Batenburg R. A PACS maturity model: a systematic meta‐analytic review on maturation and evolvability of PACS in the
hospital enterprise. International Journal of Medical Informatics 2009, 78:127‐140.
[A259] van De Wetering R, Batenburg R. Towards a theory of PACS deployment: an integrative PACS maturity framework. Journal of Digital Imag-
ing 2014, 27:337‐350.
[A260] van de Wetering R, Batenburg R, Lederman R. Evolutionistic or revolutionary paths? A PACS maturity model for strategic situational
planning. International Journal of Computer Assisted Radiology & Surgery 2010, 5:401‐409.
[A261] Van Der Velden MJ. Experiences with the capability maturity model in a research environment. Software Quality Journal 1996, 5:87‐95.
[A262] van Dyk L, Schutte CSL. Development of a maturity model for telemedicine. South African Journal of Industrial Engineering 2012:61‐72.
[A263] Van Staden S, Mbale J. The information systems interoperability maturity model (ISIMM): towards standardizing technical interoperability and
assessment within government. International Journal of Information Engineering & Electronic Business 2012, 4:36‐41.
[A264] van Steenbergen M, Bos R, Brinkkemper S, van de Weerd I, Bekkers W. Improving IS functions step by step: the use of focus area maturity
models. Scandinavian Journal of Information Systems 2013, 25:35‐56.
[A265] Vidal MC, Guizze CLC, Bonfatti RJ, Silva e Santos M. Ergonomic sustainability based on the ergonomic maturity level measurement. Work
2012, 41:2721‐2729.
[A266] Visconti M, Cook CR. Evolution of a maturity model—critical evaluation and lessons learned. Software Quality Journal 1998, 7:223‐237.
[A267] Wagner SM, Ullrich KKR, Transchel S. The game plan for aligning the organization. Business Horizons 2014, 57:189‐201.
[A268] Weyns K, Höst M. Evaluation of a maturity model for IT dependability in emergency management. International Journal of Information
Systems for Crisis Response & Management 2012, 4:47.
[A269] Wiegers KE, Sturzenberger DC. A modular software process mini‐assessment method. IEEE Software 2000, 17:62‐69.
[A270] Williams P. A practical application of CMM to medical security capability. Information Management and Computer Security 2008, 16:58‐73.
[A271] Willis CJ, Rankin JH. Measuring the maturity of Guyana's construction industry using the construction industry macro maturity model
(CIM3). Journal of Construction in Developing Countries 2010, 15:87‐116.
[A272] Willis CJ, Rankin JH. Demonstrating a linkage between construction industry maturity and performance: a case study of Guyana and New
Brunswick. Canadian Journal of Civil Engineering 2012, 39:565‐578.
[A273] Willis CJ, Rankin JH. The construction industry macro maturity model (CIM3): theoretical underpinnings. International Journal of
Productivity and Performance Management 2012, 61:382‐402.
[A274] Wilson F. The quality maturity model: your roadmap to a culture of quality. Library Management 2015, 36:258‐267.
[A275] Wilson F, Town JS. Benchmarking and library quality maturity. Performance Measurement and Metrics 2006, 7:75‐82.
[A276] Wu T, Ning F, Xiao LY, Chen CB. STP‐CMM: a grading practice capability maturity model for software talent cultivation. Journal of Software
2011, 6:636‐642.
[A277] Wu Ww, Liang Dp, Yu B, Yang Y. Strategic planning for management of technology of China's high technology enterprises. Journal of
Technology Management in China 2010, 5:6‐25.
[A278] Yamamura G. Process improvement satisfies employees. IEEE Software 1999, 16:83‐85.
[A279] Yeo KT, Ren Y. Risk management capability maturity model for complex product systems (CoPS) projects. Systems Engineering 2009,
12:275‐294.
[A280] Young M, Young R, Zapata JR. Project, program and portfolio maturity: a case study of Australian Federal Government. International Journal
of Managing Projects in Business 2014, 7:215‐230.
[A281] Zainon N, Salleh H, Rahim FA. A prototype framework of information technology infrastructure flexibility maturity model (ITIF‐MM).
Advanced Science Letters 2013, 19:273‐278.
[A282] Zeb J, Froese T, Vanier D. Infrastructure management process maturity model: development and testing. Journal of Sustainable Development
2013, 6:1‐15.
[A283] Zhong L, Leung D, Law R, Wu B, Shao J. An application of the capability maturity model for evaluating attraction websites in Mainland
China. International Journal of Tourism Research 2014, 16:429‐440.
[A284] Zou PXW, Chen Y, Chan TY. Understanding and improving your risk management capability: assessment model for construction
organizations. Journal of Construction Engineering and Management 2010, 136:854‐863.
[A285] Zqikael O, Levin C, Rad PF. Top management support—the project friendly organization. Cost Engineering 2008, 50:22‐30.