You are on page 1of 9

See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/252019625

A Quality Measurement Framework for IT Services

Article · March 2011


DOI: 10.1109/SRII.2011.84

CITATIONS READS
21 2,065

3 authors:

Marion Lepmets Eric Ras


SoftComply Luxembourg Institute of Science and Technology (LIST)
50 PUBLICATIONS   431 CITATIONS    104 PUBLICATIONS   1,447 CITATIONS   

SEE PROFILE SEE PROFILE

Alain Renault
Luxembourg Institute of Science and Technology (LIST)
49 PUBLICATIONS   572 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Collaboration with TIPA team View project

Reuse in Software Engineering (RISE) View project

All content following this page was uploaded by Marion Lepmets on 09 September 2015.

The user has requested enhancement of the downloaded file.


2011 Annual SRII Global Conference

A Quality Measurement Framework for IT Services


Marion Lepmets, Eric Ras, Alain Renault
Service Science and Innovation
Public Research Centre Henri Tudor
Luxembourg, Luxembourg
e-mail: Marion.Lepmets, Eric.Ras, Alain.Renault@tudor.lu

Abstract— Our research aims to evaluate the impact of process improvements are not implemented or their benefits not
improvement on IT service quality by conducting a series of quantified.
studies in industry. In order to do that we first establish a The main research question of our research work is to
framework for IT service quality measures and indicators. evaluate the impact of process improvement on IT service
Stemming from the fact that you cannot improve what you quality. Our research extends on various works describing
cannot measure, this paper proposes IT service quality different approaches to measure process improvement and
measures. We adapt two widely known quality measurement innovation and their impact on organizational performance.
frameworks of software engineering to IT services. We apply The study is carried out in the area of IT Service
the software product quality model described in ISO/IEC
Management, and focuses on IT service management
25010 and the quality measurement framework of Practical
models, IT service process improvement and innovation and
Software and Systems Measurement to the IT service quality
attributes, which are derived from ITIL, ISO/IEC 20000 and
their impact on IT service quality. In order to address the
SERVQUAL. As a result of the study, we will have IT service research question, we will first identify the IT service quality
quality measures that industry can collect in order to evaluate measures.
their progress. Little has been published on measuring the quality of IT
services although there is an overall agreement on the
Keywords-IT service quality; quality measurement importance of this topic. This paper describes the first phase
framework; IT service quality measures; SQuaRE; PSM; ITIL; of our study proposing a theoretical framework for
ISO/IEC 20000; SERVQUAL measuring IT service quality. The framework is based on the
relevant related literature to the subject at hand from IT
I. INTRODUCTION service management and software engineering areas.
Software engineering is a domain where measurement and
Service Science is the emerging integration of multiple process improvement have played a key role in research as
methods and approaches from different disciplines related to well as in practice and we aim to transfer this insight into IT
understanding, designing, improving, and scaling service service management domain.
systems >1@. Although many IT organizations have The paper is structured as follows: we will first provide
understood the importance of IT Service Management, most the overview of the related literature on IT service quality.
of them are still far from mastering their IT service Related works also include the description of measurement
processes at the same level as their engineering processes. frameworks from software engineering that we aim to apply
While services have become the largest part of most to measuring the quality of IT services. We then map IT
industrialized nations’ economies, there is still no widely service quality characteristics to the measurement framework
accepted definition of service; and service productivity, in order to propose IT service quality measures that will be
quality, and innovation remain hard to measure. used in our further studies.
Process improvement is about improving the current state II. RELATED LITERATURE ON IT SERVICE QUALITY AND
of practices and is based on the evaluation of these practices,
QUALITY MEASUREMENT
also referred to as process assessment. Process improvement
aims to increase efficiency and performance of This paper relates to two different topics: On the one hand it
organizations. Successful process improvement is one that is related to IT service quality and quality attributes and on
supports the achievement of organizational and operational the other hand it is related to quality measurement
goals. Process assessment helps organizations improve frameworks that illustrate software and systems quality
themselves by identifying their critical problems and concepts and measurement approaches.
establishing improvement priorities. There is a large number
of process assessment and improvement models available but A. IT Service Quality
there is little research conducted to evaluate the impact of Quality of service refers to the extent to which the service
these models on organizational performance in industry. fulfills the requirements and expectations of the customer
There is also an extensive amount of literature about the key (including users) >2@. To be able to provide quality, the
success factors of process improvement but still a lot of

978-0-7695-4371-0/11 $26.00 © 2011 IEEE 784


785
777
753
DOI 10.1109/SRII.2011.84
supplier should continuously assess how the service is being The five dimensions of service quality and the related
experienced and what the customer expects in the future. attributes of SERVQUAL described in >10@ are illustrated in
A customer will be dissatisfied with a service provider Table 1 below.
who occasionally exceeds the expectations but who
disappoints at other times. Providing constant quality is one TABLE I. SERVICE QUALITY DIMENSION AND ATTRIBUTES OF
SERVQUAL
of the most important but also one of the most difficult
aspects of the service industry >2@. Service Quality Service Quality Attributes
Although service quality is agreed between the service Dimensions
provider and its customer, commonly accepted attributes of Tangibles Modern equipment
Visually appealing facilities
service quality are the following >2@, >3@, >4@, >5@: Employees who have a neat, professional
x Availability – IS (information system) is available to appearance
users at agreed time and place Visually appealing materials associated with the
x Capacity – all capacity related issues regarding IS and service
IT in general Reliability Providing service as promised
Dependability in handing customers’ service
x Performance – speed of information processing, viewed problems
from user’s position. Performance of a component is not Performing services right the first time
of a direct interest of customer/user: the user Providing services at the promised time
experiences the “consolidated output” of the Maintaining error-free records
performances of all IS components. Responsiveness Keeping customers informed about when services
x Security – is always covered in IT service agreements will be performed
Prompt service to customers
x Confidentiality – as a part of security, is playing an Willingness to help customers
important role depending on customer’s business Readiness to respond to customer’s requests
x Scalability – service provider should ensure growth at
Assurance Employees who instill confidence in customers
required speed and without disturbing the business
Making customers feel safe in their transactions
x Adjustability – provider has to choose his development Employees who are consistently courteous
methods, his infrastructure architecture and anything Employees who have the knowledge to answer
else that supports the adjustability of IS customer questions
x Portability – important for customers who want to Empathy Giving customers individual attention
change providers, whether frequently or in view of Employees who deal with customers in a caring
fashion
dealing with early or planned end of contract.
Having the customer’s best interest at heart
Employees who understand the needs of their
Brooks proposes several measures in >6@ for measuring customers
progress in IT service processes on three levels – strategic, Convenient business hours
tactical, and operational levels depending on the type of
goals: short term (operational), medium term (tactical), and Each SERVQUAL dimension consists of four or more
long term (strategic). Each process has a process goal and attributes (Table 1) that are also related to the service quality
each process goal has one or more metrics related to it. Many attributes of capacity, availability, adjustability, security, and
processes are internal processes of the IT service confidentiality suggested in the previous list of commonly
organization: there is no direct interaction with the users of accepted IT service quality attributes. The major difference
the IT services. In these cases the customers are internal as between these two quality attributes lists is that SERVQUAL
well, either management or another process. Since there is addresses primarily the issue of how a service can be
always at least one stakeholder for each process, each provided better, i.e., the provider’s attitude and empathy that
process has a customer satisfaction metric. influences directly the overall customer satisfaction.
Parasuraman and his colleagues suggest that service
quality can be assessed by measuring customer’s
expectations and perceptions of performance levels for a B. Quality Measurement Frameworks
range of service attributes. Parasuraman et al. worked on an Effective management has become increasingly
instrument called SERVQUAL for assessing the gap important to the success of both government and commercial
between customer expectations and perceptions of service enterprises. Measurement has long been of high importance
quality in service and retailing organizations. They argue that in engineering disciplines and should also become a primary
SERVQUAL’s items measure the core criteria of service tool of IT service managers to ensure that the delivered IT
quality >7@. They assert it transcends specific functions, service meets the defined objectives. >6@
companies, and industry. It has been debated and applied on There are various quality measurement frameworks
several studies described in >8@. The study of Pitt et al. >9@ applied in software engineering. Probably the two best-
provides further evidence that SERVQUAL can be used as a known ones are described in the international standard for
measure of information systems’ (IS) success and in Software Product Quality Requirements ISO/IEC 25020,
measuring the progress of IS service quality. also known as SQuaRE >12@ and the Practical Software and
Systems Measurement (PSM) >13@.

754
778
786
785
SQuaRE is an international standard and it describes the Goal/Question/Metric approach >15@, still popular among
relationship between a quality model, its associated quality researchers. PSM has rapidly become the most widely
characteristics and software product attributes with the practiced approach to the management of software projects
corresponding software quality measures, measurement >13@.
functions, quality measure elements, and measurement The goal of PSM is to provide project and technical
methods >11@. SQuaRE >12@ describes eight software quality managers with the quantitative information required to make
attributes either for a software product or a computer system: informed decisions that impact project cost, schedule, and
x Functional suitability technical performance objectives. The PSM describes
x Reliability measurement as a systematic, but flexible process that can be
x Usability applied to both software and system engineering, as well as
x Performance efficiency management activities. The PSM measurement process is
x Maintainability based on a proven set of principles derived from actual
x Portability experience on government and industry projects.
x Security The PSM organizes issues the managers can use for
decision-making into seven classes or categories that are
x Compatibility
further decomposed into measurable concepts:
Software product quality can be evaluated by measuring x Schedule and progress
internal attributes (typically static measures or intermediate x Resources and cost
products), external attributes (typically by measuring the x Product size and stability
behavior of the code when executed), or by measuring x Product quality
quality in use attributes (when the product is in real or x Process performance
simulated use) >12@. The quality in use of a system x Technology effectiveness
characterizes the impact that the product has on the x Customer satisfaction
stakeholders: effectiveness, efficiency, satisfaction, safety,
and context comprehensiveness. Project success depends on achieving the defined
Improving process quality contributes to improving objectives. Measurements provide insight for making
product quality and product quality contributes to improving decisions that help to achieve these objectives and to assure
system quality in use >12@. Therefore, assessing and project success. An effective measurement process helps
improving a process is a means to improve product quality. project managers to identify and manage problems or risks
Evaluating and improving product quality is one of the that could adversely impact the project. The PSM is an issue-
means to improve system quality in use. The relationships driven measurement process, where these obstacles are
between the SQuaRE quality attributes are illustrated below referred to as issues that must be managed to achieve project
on Fig. 1. success (Fig. 2). The PSM measurement specification
guidance is designed to simplify the mapping of project
Process Software product Effect of software product

influences Internal External


influences influences
Process quality quality Quality in
quality attributes attributes use attributes
depends on depends on depends on

Process measures Internal measures External measures Quality in use


measures

Figure 1. Quality in lifecycle

The Practical Software and Systems Measurement (PSM) issues to applicable measures. A project manager requires a
initiative began as an effort by the US Department of lot of information from the project to make informed
Defense to improve the management of large software decisions and effective communication throughout the
acquisition projects. PSM was developed as the result of an project organization. The PSM process is adapted to meet
unusual collaborative effort among industry, government and these information needs. Issue areas are derived from
academic organizations >13@. It represents the best management information needs of a project. Issues are areas
measurement practices used within the software and system of concern that may impact the achievement of a project
acquisition and engineering communities >14@. Many of the objective. An indicator is a measure or combination of
basic concepts of PSM have been formalized in ISO/IEC measures that provides insight into an issue or concept. Most
Standard 15939 – Software Measurement Process and are indicators compare actual values with baselines.
closely related to other measurement approaches, like Measurement categories define groups of related measures.

755
779
787
786
A measure is the quantification of a characteristic of a groups of measures: process measures, internal IT service
process or a product.
p measures, external IT service measures, and customer
satisfaction measures. Process measures are related to
process performance evaluated with the process assessment
standards. Internal quality measures are related to the quality
of the information system that the IT service is built upon.
The external quality measures are related to the quality of the
IT service provided to the customer. Customer satisfaction is
related to the perception and expectations of the customer
about the IT service and is based on the customer feedback
to the IT service. The adopted SQuaRE approach to IT
service quality has been illustrated below on Fig. 3.
We now present the mapping of IT service quality
attributes to the PSM framework by common issue areas in
the tables. Common issue areas are derived from SQuaRe
and the PSM and target process, internal, external, and
Figure 2. PSM measurement process elements customer satisfaction measures. Indicators for the measures
are derived primarily from ITIL.
Each indicator in the PSM is related to a common issue We first describe the external measures of IT service
area, measurement category, and measure. An indicator is a quality in (Table 3) that support the stability of the IT
combination of measures that provide insight into one or service. These measures are directly related to the customer
more issues or concepts. Table 2 (for better readability, the experience of the IT service but are measured and improved
tables are in the end of the paper) illustrates a fragment of the
by the IT service provider. Based on ITIL >16@, IT service
indicator examples in the PSM.
stability is related to attributes that aim to provide IT service
III. MAPPING IT SERVICE QUALITY ATTRIBUTES TO continuously in the way agreed with the customer.
QUALITY MEASUREMENT FRAMEWORKS Availability means that the delivered IT service availability
matches or exceeds current and future agreed needs of
Based on >16@, a service is a means of delivering value to business. IT service continuity addresses the IT service
customers by facilitating outcomes customers want to continuity and recovery in agreed time, and the management
achieve without the ownership of specific costs and risks. An of risks of the IT service continuity. IT service capacity
IT service, according to >2@, is an output of an IT relates to the performance and capacity of the operational IT
organization. Both customers and users use the service but services. The speed of information processing is very critical
the users don’t necessarily buy it. In our study, we talk only for the IT service customers and is addressed in IT service
about customers combining all the users as we aim to performance measurement category. Information security
measure the quality based on the feedback of all who use the means that information is available to those who have the
service.
IT Service processes Information System IT Service Effect of IT Service

influences IT service
influences influences
Process IS quality quality Customer
quality attributes attributes satisfaction
depends on depends on depends on

Process Internal measures External measures Customer survey


performance measures
measures

Figure 3. IT Service quality measures


In our research we focus on IT service quality attributes rights (confidentiality), that information is complete,
and will use the fragment of indicator examples derived from protected and accurate (integrity), and that information is
PSM and shown in Table 2 as the basis for defining IT available and usable when the customer requires it
service quality attributes. Relating to different levels of IT (availability). IT service dependability is related to the
service process metrics described in >6@, we focus only on provider’s ability to handle customers’ service problems
process measures on operational and tactical levels, as the >10@.
processes on strategic level are dealing less with operational Next we illustrate in (Table 4) the quality measures that
IT services and more with the management of the IT service were called Product Quality in the PSM. Product Quality can
organizations themselves. We also adapt the SQuaRE be viewed in IT service management as the internal quality
approach to group the IT service quality attributes into four of the IT services, i.e., the quality of the Information System.

756
780
788
787
IS Quality relates to the system’s ability to support the IT applied to IT service quality attributes described in ITIL,
services delivered. Based on the PSM, ISO/IEC 20000 and ISO/IEC 20000 and SERVQUAL.
ITIL, the internal measurement categories are: functional As a result of the study we have identified IT service
correctness, usability, availability/reliability/maintainability/, quality measures and indicators grouped into four common
component availability, component capacity, scalability, issue areas: IT service quality, IS quality, process
adjustability and portability. Functional correctness is performance and customer satisfaction measures. These
related to the functional defects of the system. Portability indicators could assist industry to evaluate their progress of
stands for the system’s compliance with the standards. providing better IT service and providing the IT service
Usability is related to the problem prevention and resolution better. These indicators will also be used in our further
of the problems. The availability of components at agreed studies for data gathering from industry as we aim to
time and the time it takes to have the system working after evaluate the impact of process performance on the IT service
the failure both support the reliability of the system. and IS quality.
Component capacity is the power to store and process data,
and absorb changes in the required time. Scalability focuses ACKNOWLEDGMENT
on the provider’s ability to ensure growth at the required The present project is supported by the National
speed, without disturbing the business. Adjustability of an Research Fund, Luxembourg and cofounded under the Marie
information system relates to the development method, Curie Actions of the European Commission (FP7-
architecture and infrastructure that can support adjustments COFUND).
according to the needs of the customer organization.
We illustrate the process performance measures in (Table REFERENCES
5) that can be collected from process assessment or [1] J. Spohrer, P. P. Maglio, et al. "Steps toward a science of service
measuring the effectiveness or efficiency of the processes. systems." Computer 40(1): 71pp. (2007).
The performance of IT service processes can be measured in [2] S. Polter, et al., ISO/IEC 20000 - An Introduction, First Edition ed.
three ways: process assessment is an evaluation of process Zaltbommel: Van Haren Publishing, 2008.
performance against process reference model requirements. [3] L. Shu-Ping, et al., "What is the valuable service quality gap for
ISO/IEC 15504 conformant IT service process assessment improving customer satisfaction?," in 6th International Conference on
Service Systems and Service Management, 2009, pp. 242-247.
can be conducted based on TIPA methodology >17@. Process
[4] I. MacDonald, “A simple, low cost but innovative approach to end-to-
effectiveness in software engineering is measured by the end service reporting”, itSMF Whitepaper, 2009
absence of defects in its outputs and process efficiency by the [5] T. Barroero, et al., "Aligning IT Service Levels and Business
increase of software productivity. IT service process Performance: A Case Study," in IEEE International Conference on
effectiveness is measured through the defects discovered and Services Computing (SCC), 2010, pp. 570-577.
rework required after the IT service design phase. [6] P. Brooks, Metrics for IT Service Management, First edition ed.
Finally, we illustrate the measures about the perceptions Zaltbommel: Van Haren Publishing, 2006.
of the IT service from the customer viewpoint in (Table 6). [7] A. Parasuraman, et al., "Refinement and Reassessment of the
SERVQUAL method suggests measuring the gap between SERVQUAL Scale," Journal of Retailing, vol. 67, pp. 420-450, 1991.
the expectation and perception of the IT service user. [8] R. T. Watson, et al., "Measuring information systems service quality:
Customer satisfaction survey results provide the most Lessons from two longitudinal case studies," Mis Quarterly, vol. 22,
pp. 61-79, Mar 1998
important insight into how well the IT services have been
[9] L. F. Pitt, et al., "Service Quality - A Measure of Information Systems
provided and received by the customer. A customer Effectiveness," MIS Quarterly, vol. 19, pp. 173-187, Jun 1995
satisfaction survey should include SERVQUAL quality
[10] V. Zeithaml, et al., Delivering quality service: balancing customer
indicators about the service provider’s attitude and empathy perceptions and expectations: Free Press, 1990, Chapter 2
from the customer’s viewpoint. Customer satisfaction is in [11] ISO/IEC JTC1 SC7, "ISO/IEC 25020, Software engineering -
fact the sum of how good the IT service is and how well the Software product Quality Requirements and Evaluation (SQuaRE) -
IT service is provided. We should therefore gather customer Measurement reference model and guide," 2007
feedback also on IT service quality from the users and IS [12] ISO/IEC JTC1 SC7, "ISO/IEC FCD 25010, Systems and software
quality from the customers who are internal to the IT service engineering - System and software product Quality Requirements and
providing organization. Customer support is the point of Evaluation (SQuaRE) - System and software models," 2009
contact for IT service users on a day-to-day basis known as [13] D. N. Card and C. L. Jones. “Status report: practical software
measurement.” Proceedings of Third International Conference on
the service desk. Quality Software, 2003.
IV. CONCLUSIONS [14] "Practical Software and Systems Measurement: A Foundation for
Objective Project Management, v. 4.0b1”, www.psmsc.com, 2000
This paper describes the first step in our research work [15] R. V. Basili and G. Caldiera, et al. “The Goal Question Metric
on the impact of process improvement in IT service Approach”. Encyclopedia of Software Engineering. Ed: J. J.
management. The aim of this paper was to identify IT Marciniak. New York, John Wiley & Sons, Inc., 1994, pp. 528-532.
service quality measures with the help of quality [16] V. Lloyd and C. Rudd, "ITIL Service Design," OGC, London, 2007
measurement frameworks of SQuaRE and PSM from [17] B. Barafort, et al., ITSM Process Assessment Supporting ITIL.
software engineering domain. These frameworks were Amersfoort: Van Haren Publishing, 2009

757
781
789
788
TABLE II. INDICATOR EXAMPLES OF PSM (PSM)

Common Issue Area Measurement Category Measure Indicator


Product size and stability Physical size and stability Interfaces Interface stability
Lines of Code x Software size – by lines of code
x Software size – lines of code
Physical dimensions Electrical power budget
Functional size and stability Requirements x Requirements stability
x Requirements stability by type of
change
Functional change workload x Multiple indicators by change
requests
x Change requests by priority
Product quality Functional correctness Defects Defect density, etc.
Supportability – Maintainability Time to Restore x System failures
x Mean Time to Restore System, etc.
Cyclomatic Complexity x Software complexity
x Software units complexity
Efficiency Timing x Response time – on-line functions
x Response time during test
Portability Standards compliance Interface compliance validation
Usability Operator errors x Problem reports by type of
problem data
x Operator error distribution by
reason
x Device complexity distribution
Dependability - Reliability Failures x MTBF ranges based on historical
data
x Reliability growth tracked with
mean time to failure
Process performance Process compliance Reference model rating Reference model level
Process audit findings Process audit findings
Process efficiency Productivity x Software Productivity – Historical
vs Proposal
x Evaluating options using software
productivity
Process effectiveness Defect containment Requirements defects discovered after
requirements phase
Rework x Rework effort
x Development effort by
activity – compared to total
rework effort
Customer satisfaction Customer feedback Survey results Customer Satisfaction Survey
Performance rating Performance Award Scores
Customer support Requests for support x Total calls per month by priority
x Mean response time by priority

758
782
790
789
TABLE III. IT SERVICE QUALITY ATTRIBTUES – EXTERNAL MEASURES

Common Issue Area Measurement Category Measure Indicator


IT Service Stability IT Service Availability Maintainability MTRS – mean time to restore IT service
after failure
IT Service Continuity Business Impact Analysis x Business impact on the loss of IT
(BIA) service (financial)
x Mean time to recovery
x Incidents related to IT service
continuity
x Changes related to IT service
continuity
Management of Risks x Number of identified risks and
(M_o_R) threats
x Weighted average of the impact of
aggregated risks
IT Service Capacity Capacity of operational x Incidents related to IT service
services capacity
x Changes related to IT service
capacity
IT Service Performance Speed of information x Incidents related to IT service
processing performance
x Changes related to IT service
performance
Information Security Confidentiality x Incidents related to information
confidentiality
x Changes related to information
confidentiality
Integrity x Incidents related to information
integrity
x Changes related to information
integrity
Availability x Incidents related to information
availability
x Changes related to information
availability
IT Service Reliability Dependability x Incidents, RFCs and problems
handled daily
x Mean time to achieve incident
resolution

TABLE IV. IS QUALITY ATTRIBUTES – INTERNAL MEASURES


Common Issue Area Measurement Category Measure Indicator
IS Quality Functional Correctness Defects Defect density
Portability Standards Compliance Interface compliance validation
Usability Problems and Errors x Number of prevented problems daily
x Number of incidents related to problems
daily
Availability – reliability – Time to Restore x MTBSI – mean time between system
maintainability incidents
x MTBF – mean time between system
failures
x MTRS – mean time to restore system
x Number of incidents resolved out of all
incidents daily
Component capacity Performance of Number of system failures related to
technical components component performance
Capacity of technical Number of system failures related to
components component capacity
Scalability Growth without Incidents related to the speed of growth
business disturbance
Adjustability Complexity System complexity

759
783
791
790
TABLE V. IT SERVICE PROCESS PERFORMANCE

Common Issue Area Measurement Category Measure Indicator


Performance of IT service Process Compliance Reference Model Rating Process Assessment Results
processes Process Audit Findings Process Audit Results
Process Efficiency Productivity Historical vs proposed and actual
Process Effectiveness Defect Containment Requirements defects discovered
after design phase
Reword Rework effort

TABLE VI. CUSTOMER SATISFACTION MEASURES OF IT SERVICES

Common Issue Area Measurement Category Measure Indicator


Customer Satisfaction Customer Feedback Survey Results x Appearance of physical facilities,
equipment, personnel and
communications material
x IT service is provided dependably and
accurately
x IT service provider is willing to help
customers and provide prompt service
x IT service provider conveys trust and
confidence
x IT service provider provides caring,
individualized attention
x Perception of IT service stability
x Perception of IS quality
x Perception of IT service processes’
performance
Customer Support Requests for Support x Total call per day answered,
abandoned
x Average call response time
x Incidents handled daily by service
desk

760
784
792
791

View publication stats

You might also like