See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/305909880

Methods and techniques for maturity assessment

Conference Paper · June 2016

DOI: 10.1109/CISTI.2016.7521483

CITATIONS READS

24 18,416

1 author:

Diogo Proença
Inesc-ID
34 PUBLICATIONS 375 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

4C Project View project

European Archival Records and Knowledge Preservation (E-ARK) View project

All content following this page was uploaded by Diogo Proença on 12 November 2018.

The user has requested enhancement of the downloaded file.

 Methods and Techniques for Maturity Assessment
Diogo Proença
Departamento de Engenharia Informática
INESC-ID / IST
Lisbon, Portugal
diogo.proenca@tecnico.ulisboa.pt
Abstract — A Maturity Model is a widely used technique that is
proved to be valuable to assess business processes or certain
aspects of organizations, as it represents a path towards an
increasingly organized and systematic way of doing business. A
maturity assessment can be used to measure the current maturity
level of a certain aspect of an organization in a meaningful way,
enabling stakeholders to clearly identify strengths and
improvement points, and accordingly prioritize what to do in
order to reach higher maturity levels. However, in order to make
that possible, maturity assessments must be performed. Doing that
can range from simple self-assessment questionnaires to full blown
assessment methods, such as recommended by the ISO15504 or the
SEI CMMI. However, a main caveat of these assessments is the
resources they encompass, as well as, a lack of actual automation,
which many times renders benchmarks not possible. Assuming
that the modeling of business domains is becoming a fact with the
wide spread of Enterprise Architecture practices, and also
considering the recent state of the art on the representation of
Enterprise Architecture models using ontologies, this work
proposes to follow that trend and innovate by using existing
semantic technology to automate maturity models assessment
methods.
Keywords - Maturity Model, Maturity Assessment, Ontology
Analysis.
I. INTRODUCTION
A Maturity Model (MM) is a technique that has been proved
to be valuable in measuring different aspects of a process or an
organization. It represents a path towards increasingly organized
and systematic way of doing business in organizations.
A MM consists of a number of “maturity levels”, often five,
from the lowest to the highest, Initial, Managed, Defined,
Quantitatively Managed and Optimizing (however, the number
of levels can vary, depending on the domain and the concerns
motivating the model). This technique provides organizations:
(1) A measuring for auditing and benchmarking; (2) A
measuring of progress assessment against objectives; (3) An
understanding of strengths, weaknesses and opportunities
(which can support decision making concerning strategy and
project portfolio management).
MMs history goes back to 1973 [1], and had its maximum
visibility with the Software Engineering Institute Capability
Maturity Model Integration (CMMI) [2] and the ISO 15504 [3].
Both these key references were born in the Software
Engineering domain, culminating decades of development and
refinement of the corresponding models. Moreover, there is
certification for these two references as they are considered as
the de facto assessment techniques to be used when
benchmarking organizations for their software engineering
process implementation and maturity. As such, in order for the
results to be comparable, there is a detailed maturity assessment
method behind each of these MMs. These methods define in
detail how an assessment should be planned, conducted, the
maturity levels calculated and how the results must be presented
to the organization. These methods make each assessment
repeatable and comparable with results from other
organizations, allowing for benchmarking.
In the computer science domain, several definitions for
ontologies have been proposed. One of the most widely used
definitions describes ontologies as a “formal, explicit
specification of a shared conceptualization” [4].
Conceptualization refers to an “abstract, simplified view of the
world” [5], containing “the objects, concepts, and other entities
that are assumed to exist in some area of interest and the
relationships that hold among them” [6]. The Web Ontology
Language (OWL) is described as a “semantic web language
designed to represent rich and complex knowledge about things,
groups of things, and relations between things” [7]. Moreover,
the use of ontologies and computational inference mechanisms
as a means for representing and analyzing Enterprise
Architecture (EA) models has already be proven in [8].
Furthermore, there is some previous work [9][10] in the domain
of software engineering where authors used ontologies as a way
to automate the process assessment in software engineering and
have defined ontologies for CMMI, ISO15504 and ISO9001 to
aid in this effort. However, this works seems to be abandoned as
of the time of writing this proposal.
II. RESEARCH PROBLEM
Current maturity assessment methods focus on highly complex
and specialized tasks being performed by competent assessors
in an organizational context. These tasks mainly focus on
manually collecting evidence to substantiate the maturity level
calculation. Because of the complexity of these methods,
maturity assessment becomes an expensive and burdensome
activity for organizations.
These methods usually start by creating an assessment plan
which describes how the assessment will be conducted, as well
as, the schedule, people involved, necessary documents and
how evidence will be collected. Then a group of assessors,
denominated assessment team follows the assessment plan,
they collect all the necessary evidence, calculate the maturity
levels and assemble the assessment report which details the
findings and maturity levels of the assessment. Then, based on
the assessment results, the organization can plan for
improvement by following an improvement plan.
As such, the objective of this research is to develop methods
and techniques to automate maturity assessment. There are
several examples of models used to represent an organization
architecture, such as, Archimate, BPMN or UML. These
models are descriptive and can be detailed enough to allow to
perform, to some extent, maturity assessment. For example, the
collected evidence from an organization can be synthetized into
a set of model representations that can then be used when
analyzing and calculating the maturity levels.
However, in order for these models to become relevant for
maturity assessment there should be a formal representation for
both MMs and model representations. One hypothesis is that
building on the knowledge of ontologies from the computer
science and information science domains, these can be used to
represent MMs and model representations.
Then, by representing MMs and models representations of
concrete organizational scenarios using ontologies we can
verify if an organization models representations matches the
requirements to reach a certain maturity level using ontology
query and reasoning techniques, such as SPARQL and
Description Logics inference.
The final objective is thus to identify how these methods and
techniques can be used in existing maturity assessment
methods, so that they can be proven as relevant to enable the
automation of certain aspects of maturity assessment, such as,
the maturity level determination. In order to do this, there
should be an exploration of what types of analysis can be
performed using the information on model representations that
is relevant in a maturity assessment effort.
III. RESEARCH QUESTIONS
To meet the pointed objectives, this work proposes to consider
the following assumptions and explore the related generic
research question:
Assumptions:
 The wide spread of modeling practices of business
domains is becoming a fact, making it possible to
have access, for processing, to the data managed by
these tools;
 The recent state of the art demonstrating how
business processes and EA models in general can
be represented as ontologies has demonstrated the
potential relevance of the semantic techniques for
the automated processing of EA models.
Generic Research Question (RQ):
 What is the potential, and what are the limitations,
of the existing semantic techniques to automate
methods for the assessment of MM through the
analysis of an existing model representation of a
reality?
This generic research question can be detailed in the following
ones:
RQ1. Formal representation of a MM: How can the core
universe of discourse of MM be formalized? This question is
related to the capture and formalization of the terms, properties
and relationships that exist in the MM universe of discourse.
For example, the hypothesis is to use an ontology that expresses
all these core concepts relationships among them (including
rules for their assessment).
RQ2. Maturity Assessment: How can model
representations that are already common in business
domains be used to support maturity assessment? This
question is related to the expressiveness of existing model
representations and their original languages (such as
Archimate, BPMN, UML, etc.) and their relevance for maturity
assessment. The assumption is that the modelling frameworks
can be used to represent an organizational reality.
RQ3. Effectiveness of the approach: How can model
representations, when represented as ontologies, be used in
automated activities to support the assessment of the reality
they express against existing reference MM? This question
is related to the effectiveness of the semantic techniques to
expose the potentials and limitations for this purpose.
IV. FUTURE WORK
This section details the research approach, expected
contributions and how the research will be validated.
1.1 Research Approach and Expected Contributions
The research approach to be taken in this proposal is based on
the design science research method (DSRM) as described in
[11]. In this method, research focuses on designing solutions to
be applied to solve a problem, as opposed to the more
descriptive paradigm followed in social and natural sciences.
The process includes six steps: (1) problem identification and
motivation; (2) definition of the solution objectives (3) design
and development of the solution artefact; (4) demonstration; (5)
evaluation; and (6) communication.
Figure 1 depicts the DSRM as applied to this research. Then
each of the steps are described in more detail. The research
entry point for this proposal is a Problem-centered initiation.
Problem identification and motivation: this step deals with
the definition of the research problem and with the justification
of the value of a solution. The process to be followed during the
elaboration of this dissertation will consist of several iterations,
firstly initiated by the generic research question “What is the
potential, and what are the limitations, of the existing
 Figure 1. Design science research method to be applied in this work according to the guidelines present in [11].
semantic techniques to automate methods for the
assessment of MM?”.
Definition of the solution objectives: this step deals with the
definition of the objectives that a solution artefact should
achieve. The objectives are derived from the definition of the
problem performed in the previous step and should be defined
quantitatively or qualitatively. In this case, the objectives are
derived from the research questions raised at Section 2.
Design and development of the solution artefact: this step
concerns the creation of the solution artefact, which can be
formed by constructs, models, methods or instantiations [11].
Following the delineated research questions, the design and
development of the artefact focused on the use ontology
representations of maturity models and model representations
to support maturity assessment methods. For that purpose,
fundamental concepts must be defined and related to each other,
and the types of possible analysis must also be defined and
described.
Demonstration: this step deals with the demonstration of the
use of the artefact in solving one or more instances of the
problem. In the context of this proposal, this will be
demonstrated in the context of synthetic scenarios, with the
application of formal logic-based ontologies to represent
maturity models pertaining to different domains and using
computational inference to perform the analysis. The aim is to
show that the use of a logic-based formalization of maturity
models effectively assists in the assessment of maturity, thus
addressing the research questions raised in this proposal.
Evaluation: this step concerns the assessment of how well the
artefact supports a solution to the identified problem. For that
purpose, in the context of this proposal, a validation of the
proposed artefact is performed by applying it to a set of real
organizational scenarios to demonstrate the value of the artefact
and how it addresses the raised research questions.
Communication: this step concerns the communication to the
academic community of the problem and its relevance, and of
the artefact and its effectiveness in addressing it. Several
publications must be achieved in the context of this proposal,
such as conference, workshop and journal articles which are
directly related to the results obtained.
As a conclusion the expected contribution will be to innovate in
the domain of MMs assessment, by demonstrating how the state
of the art of semantic technology based on ontologies can be
used to automate activities for that purpose. As such, the
contributions of this work will be:
1. The development of a reference ontology to represent
MMs, which must comprise the components used to
define a MM and also its assessment;
2. The development of a method for assessment of model
representations of realities which are represented as
ontologies according to a given MM.
1.2 Validation of the Research
The validation of this work will be done in the context of these
possible scenarios:
Scenario 1. The E-ARK Project, E-ARK is a 3-year
multinational research project co-funded by the European
Commission under its ICT Policy Support Programme (PSP)
within its Competitiveness and Innovation Framework
Programme (CIP). The goal of the E-ARK Project is to pilot
archival services to keep records authentic and usable based on
current best-practices. These will address the three main
activities of an archive – acquiring, preserving and enabling re-
use of information. E-ARK brings together a group of European
national archives, four leading research institutions, three
providers of archiving software solutions and services, two
government agencies, and two international membership
organizations that represent the communities who stand to
benefit from the project: data owners/providers, archives,
software vendors and solution providers. In the context of this
project a maturity model for Information Governance is being
developed and used to assess organizations on the maturity of
their current Information Governance practice.
Scenario 2. The ELIXIR Project, which goal is to orchestrate
the collection, quality control and archiving of large amounts of
biological data produced by life science experiments. Some of
these datasets are highly specialized and would previously only
have been available to researchers within the country in which
they were generated. Each participant country in the project has
what is called an ELIXIR node, which run the resources and
services that are part of ELIXIR. These include, data resources;
bio-compute centers; services for the integration of data,
software, tools and resources; training; and standards expertise.
The Portuguese ELIXIR node main focus in on the woody
plants domain. In the context of this project a maturity model is
being developed to assess the maturity of each node of the
ELIXIR project.
Scenario 3. Synthetic scenarios, which are scenarios artificially
created with the purpose of validating distinct aspects of the
expected contributions.
V. CONCLUSIONS
This proposal aims to innovate on the methodological
approaches for the assessment of maturity using ontology
reasoning with the ultimate goal of maximizing the automation
of the overall maturity assessment process. For that purpose, the
follow-up work will aim on the development of an ontology for
maturity models that capture the universe of discourse of the
maturity models and assessment domain. This together with the
integration of other domain specific ontologies such as, BPMN
and Archimate will allow to analyze existing model
representations from a maturity model perspective. Ultimately
with the integration of these methods together with an existing
View publication stats
maturity assessment method will allow the automation of
certain aspects of the maturity assessment process.
As preliminary work, we described the concepts which form the
foundation of this research, in particular the concept of maturity
model and assessment as well as the multiple domains where it
was adopted and used. With basis on that analysis, a description
of the different aspects of current maturity models was
presented, combining knowledge from the different domains
analyzed. A description of relevant modeling frameworks also
encompassing different domains was also provided. The
requirements engineering domain was also presented and
relevant sources were analyzed. A description of the current
practice in the semantic technology domain was provided
detailing the concept of ontology, OWL, Description Logics,
among others. Finally, the future work resulting from this
proposal was detailed, as well as, how the research will be
validated.
ACKNOWLEDGMENT
This work was supported by national funds through
Fundação para a Ciência e a Tecnologia (FCT) with reference
UID/CEC/50021/2013, and by the European Commission under
the Competitiveness and Innovation Programme 2007-2013, E-
ARK – Grant Agreement no. 620998 under the Policy Support
Programme. The authors are solely responsible for the content
of this paper
REFERENCES
[1] R. L. Nolan, "Managing the Computer Resource: A Stage Hypothesis",
Communications of the ACM, vol. 16, pp. 399-405, 1973.
[2] D. M. Ahern, A. Clouse, R. Turner, “CMMI Destilled: A Pratical
Introduction to Integrated Process Improvement, Third Edition,” Addson
Wesley Professional, 2008.
[3] ISO/IEC 15504:2004, “Information technology - Process assessment,”
International Organization for Standardization and International
Electrotechnical Commission Std. 2004.
[4] R. Studer, R. Benjamins, and D. Fensel, “Knowledge engineering:
Principles and methods,” Data & Knowledge Engineering, vol. 25, pp.
161–198, 1998.
[5] N. Guarino, D. Oberle, and S. Staab, Handbook on Ontologies, ch. What
Is an Ontology?, pp. 1–17. Springer Berlin Heidelberg, 2009.
[6] M. R. Genesereth and N. J. Nilsson, Logical Foundations of Artificial
Intelligence. Morgan Kaufmann, Los Altos, CA, 1987.
[7] W3C, OWL 2 Web Ontology Language Structural Specification and
Functional-Style Syntax (Second Edition). World Wide Web Consortium
Recommendation, 2012. Retrieved January 2015 from
http://www.w3.org/TR/owl2-syntax/.
[8] G. Antunes, “Analysis of Enterprise Architecture Models: An Application
of Ontologies to the Enterprise Architecture Domain,” PhD Thesis,
University of Lisbon, 2015.
[9] G.Grambow, R. Oberhauser, M. Reichert, "Towards Automatic Process-
aware Coordination in Collaborative Software Engineering," In ICSOFT
2011 - Proceedings of the 6th International Conference on Software and
Data Technologies, Volume 1, Seville, Spain, 18-21 July, 2011.
[10] G. Soydan, M. Kokar, “An OWL ontology for representing the CMMI-
SW model,” Semantic Web Enabled Software Engineering (SWESE
2006), 2006.
[11] K. Peffers, T. Tuunanen, M. Rothenberger, S. Chatterjee, “A design
science research methodology for information systems research,” Journal
of Management Information Systems, vol. 24, pp. 45–77, 2008