ITIL Key Performance Indicators and the IT Balanced

Scorecard

IT-Management 2

Supervisors: Prof. Dr. Andreas Breiter, Dipl.-Inf. Jörg Hofmann

28 August 2009

Tobias Hildebrandt
Master Digitale Medien, Fachbereich 3
Universität Bremen
Matrikelnummer: 2337614
Tobias_Hildebrandt@gmx.de
Abstract
This paper deals with the IT Service Management Framework ITIL and the business
performance measurement method Balanced Scorecard and its IT equivalent, the IT
Balanced Scorecard. The Balanced Scorecard aggregates low level measures, which
are called KPIs. Usually there are relations between these measures, which are
called cause-and-effect relationships. This paper tries to find out what features
current software implementations of the Balanced Scorecard and of the IT Balanced
scorecard offer, especially in regards to cause-and-effect relationships.

Keywords: ITIL, Performance measurement, Key Performance Indicator (KPI),

Balanced Scorecard (BSC), IT Balanced Scorecard (IT BSC), IT Management, IT
Service Management, IT Governance
Table of Contents
1 Introduction ........................................................................................................................ 1
2 Balanced Scorecard ............................................................................................................ 3
2.1 Motivation ................................................................................................................... 3
2.2 Concepts ...................................................................................................................... 4
3 IT Balanced Scorecard ....................................................................................................... 8
3.1 Differences to the Business Balanced Scorecard ........................................................ 9
3.2 Considerations for implementation ........................................................................... 10
4 Linking ITIL Key Performance Indicators and the IT Balanced Scorecard .................... 11
4.1 ITIL Key Performance Indicators ............................................................................. 11
4.2 Combining ITIL Key Performance Indicators with the IT Balanced Scorecard....... 11
4.3 Translating the strategy ............................................................................................. 13
5 Cause-and-effect Relations .............................................................................................. 16
5.1 State of the art in research ......................................................................................... 17
5.2 Software implementation .......................................................................................... 18
5.3 How a software IT BSC could look like ................................................................... 27
6 Conclusion........................................................................................................................ 29
7 Sources ............................................................................................................................. 31
List of abbreviations
BSC Balanced Scorecard
CIO Chief Information Officer
CobiT Control Objectives and Related Technology
IT Information Technology
IT BSC IT Balanced Scorecard
ITIL IT Infrastructure Library
KPI Key Performance Indicator
OGC Office of Government Commerce
Introduction

1 Introduction
The ITIL (Information Technology Infrastructure Library) framework is an approach to
IT Service management and according to [Th@] also the most widely accepted one.
It was developed by Great Britains’ Office of Government Commerce (OGC) and is a
documentation of best practices for IT Service Management. It consists mainly of a
series of advising books. The ITIL framework has been revised since its initial
development in the 1980s, the current version as this paper is being written is
Version 3 which was released in 2007. The OGC claims, that using ITIL might
provide benefits like reduced costs, improved IT services and improved customer
satisfaction. It provides best practices for all of the lifecycles of an IT service, which
according to [Th@] are Service Strategy, Service Design, Service Transition, Service
Operation and Continual Service Improvement. This paper will mostly deal with the
Continual Service Improvement phase.
Since its first introduction the ITIL framework has become more and more accepted,
being the most used framework for IT service management and a de facto standard
in this area nowadays. This success might be supported by the shift from IT
Management to IT Service Management.
While IT Management was more concentrated on infrastructure and the operational
level of IT, IT Service Management focuses on the services that are provided to its
customers. In the course of this shift, new methods were needed to cope with these
changes, and the ITIL framework was able to provide these methods. At the same
time, the importance of the alignment of business and IT increased, which made the
role of the IT Manager more important, leading to the nowadays common role of the
CIO (Chief Information Officer), who is often a member of the board of executives.
The next shift of roles and mindsets in IT that is frequently discussed is the IT
Governance, which is even more business oriented than IT Service Management.
While IT Service Management can still be considered to be in the IT domain, the IT
Governance responsibilities are to be found almost completely in the business
domain. According to [Va07] the ITIL framework is clearly a management framework,
and therefore it does not cover IT Governance – in contrary to other frameworks like
for example CobiT, which are explicitly frameworks for IT Governance.

1
Introduction

What ITIL is missing however is a business-centered way of controlling the IT and

evaluating its support in reaching the business goals. These aspects can be covered
by the IT Balanced Scorecard, which can be used to link the business goals and IT
Service Management. According to [Sa04@], the IT Balanced Scorecard is a method
that can be used to support IT Governance.
This paper tries to answer the question “What are the current features of (IT)
Balanced Scorecard software regarding cause-and-effect relationships
between measures in general and ITIL Key Performance Indicators in special
and how can they be improved?”
In order to answer this question, firstly the concepts and methods of the Balanced
Scorecard (BSC) are explained in chapter 2, which is followed by remarks on the IT
Balanced Scorecard (IT BSC) in chapter 3. Afterwards methods to combine ITIL and
the IT BSC are discussed in chapter 4, which is then follow by extensive
considerations on cause-and-effect relationships in theory and in software products
(in chapter 5). This paper ends with the conclusion.

2
Balanced Scorecard

2 Balanced Scorecard
The BSC is a method for strategic business performance measuring that was
suggested by Robert S. Kaplan and David P. Norton in 1992. Since then it received a
lot of attention by managers as well as by scientists of different sectors. These
sectors include big companies like banks, insurance companies or software
manufacturers, but also smaller or governmental enterprises, like for example
hospitals. This chapter first tries to explain the motivation that led to the development
of the BSC, and then shows the underlying concepts.

2.1 Motivation
Historically, business performance measurement was based mostly on financial
techniques and measures like budget planning, Return on Investment (Ratio of
money earned or lost on an investment), Cash flow (movement of cash into or out of
a business) or earnings per share (Earned money divided by the number of shares)
[Ba01@].
The concentration on these aspects may lead to very short term thinking – usually
managers plan ahead at most until the next fiscal year, where a new budget plan is
being created. Long term goals and developments may be neglected by this
paradigm. This kind of thinking may also lead to actions and decisions which can
help in the short term, but impede long term improvements. One example might be
expenditures on employee trainings. When just focussing on “traditional” financial
performance measuring, a manager will, in many cases, come to the conclusion to
cut expenditures in this area – in doing so he will better be able to keep his budget
plan for the current year. And when a Return on Investment analysis on expenditures
on training is being calculated, there might not be any measurable or predictable
“return” at all, let alone in the current fiscal year. This is because it is very difficult to
calculate the detailed benefits of an improved employee training. But in cutting this
expenditure there might be a lot of additional costs in the future years coming up, for
example a lower productivity or a decrease in customer satisfaction due to a worse
service.
This is where the BSC comes into play. It tries to help to see a business from a more
“integrated” perspective, and also aims to put performance measurement on a more
long term strategic level.

3
Balanced Scorecard

Its’ basic concept is to select a hand full of goals from the following areas, which are
called “perspectives”: user orientation, business contribution, operational excellence
and future orientation (see figure 1).
Each of these perspectives has a mission. One example for such a mission might be
“be the supplier of choice for our customers” for the customer perspective. Besides
the overall mission, each perspective is made up of several subgoals that support the
mission. For each of these goals it has to be defined a way to measure the state of
their completion. In this example “reach a high customer satisfaction” might be a
good goal to reach the mission of the customer perspective. In this example “Score of
evaluation by customer” might be a way to measure this goal. [Ka92]

Figure 1: The four perspectives of a Balanced Scorecard

It is very important that these values on the lowest hierarchy level of the BSC are
actually measurable.

2.2 Concepts
One of the reasons why the concept of the BSC is so successful may be that it claims
to help seeing a business from an integrated meta-perspective. It especially is
intended to help to see interconnections between areas of business (the
perspectives), which is illustrated by figure 2.

4
Balanced Scorecard

Figure 2: The interconnections of a Balanced Scorecard

One area that is often discussed about the BSC is the finding of cause-and-effect
relationships between the perspectives in general, and between the goals of them in
special.
These relations shall now be explained by one example. Figure 3 shows on the left
side a so-called “strategy map” (an overview of the perspectives of an organization
with its goals) of a BSC, which consists of four perspectives - each of which have two
subgoals.
Now one might come to the conclusion that there might be relationships between the
goals, for example between the customer support and the customer satisfaction. A
manager of a company could for example suspect that by improving his customer
support, a higher customer satisfaction might be the result. This possible relation was
added to the strategy map in figure 3. The right side of figure 3 offers a more detailed
view with metrics measuring the subgoals defined in the strategy map. The
superficial cause-and-effect relation defined between the goals “Customer Support”
and “Customer Satisfaction” may here be displayed more detailed by building
relations between the metrics themselves. For example assuming that the
imaginative company of this example is a data center, especially the metric “% SRs
Resolved on Initial Contact” might have a big effect on customer satisfaction. Maybe
this relation can even be specified more detailed: the relation may exist especially
with the measurement “Customer Satisfaction - Service Support”. But the finding of
cause-and-effect relationships usually does not end here – generally this is a process
with multiple steps.
The relations between goals that have been defined in figure 3 can now be used as a
base for finding more relations.

5
Balanced Scorecard

Figure 3: Strategy Map and IT BSC part 1

As pointed out in figure 3, there might also be a cause-and-effect relationship

between customer satisfaction and the revenue. Given that improving the customer
support leads to a higher customer satisfaction, it can further be concluded that this
may lead to increased revenue. Figure 4 links this relationship to the metric “IT
Revenue – actual”.

6
Balanced Scorecard

Figure 4: Strategy Map and IT BSC part 2

The examples that were pointed out here can be completed by many more relations
that might exist between these goals. Once found, these possible relations should
then be evaluated – possibly in group workshops with discussions, using historical
business data if available.

7
IT Balanced Scorecard

3 IT Balanced Scorecard
After the concept of the BSC was introduced, it was soon picked up by different
sectors and areas. Thus, a BSC especially for IT Management / IT Governance was
created. The first suggestions to use the BSC in the IT context came up in 1992 by
Gold [Go92]. However, Willcocks found out in 1994 [Wi94], that Golds’ approach was
leading too much to see IT from an IT departments’ perspective. After that, the
concept of the IT BSC has been refined in 1997 [Va97] and 1998 [Va98] again. Since
then it has further been improved, for example by Wim Van Grembergen in 2000
[Va00], who pointed out how to link the IT BSC to the business BSC.
The main motivation to develop the IT BSC was to demonstrate the value of IT to the
business, address IT Governance issues and make IT more efficient and cut costs
[Cr07]. According to its popularity, [Cr07] created a hype cycle for the IT BSC. A hype
cycle is:
“[…] a graphic representation of the maturity, adoption and business
application of specific technologies […]”1.
This cycle (see figure 5) was published in 2007, according to which the IT BSC
already reached the “Plateau of productivity”. This is the last phase of a Hype Cycle
and means, that a product or method is no longer a hype theme which is heavily
discussed, but somehow accepted and also widely used.

1
[Un@]

8
IT Balanced Scorecard

Figure 5: Gardner Hype Cycle of the IT BSC [Cr07]

One of the points of motivation to use the IT BSC is that it delivers faster feedback of
the status of goals than traditional methods, what van der Zee expresses as follows:
“The time lag between business planning processes and IT planning
processes is often just too long to be acceptable.”2.
This chapter first points out the differences to the business BSC, and then states
considerations that should be taken into account when implementing an IT BSC.

3.1 Differences to the Business Balanced Scorecard

One difference to the business BSC is that often the perspective “financial” is
replaced by “business contribution” [Va00]. While the naming of the perspective can
differ, the content can be considered the same (IT expenses and business value of IT
projects). The term “customer” from the customer perspective is often replaced by the
term “user” [Va00], which may sometimes fit better. The “learning and growth”
perspective is sometimes replaced by “future orientation”, which still contains
strategies like offering a good employee training, but also includes for example the
research of new technologies and applications in order to provide a better position for
the future.

2
[Va99], Page 141

9
IT Balanced Scorecard

3.2 Considerations for implementation

There are some things to consider when implementing an IT BSC. For example
instead of just one IT BSC there can also be a cascade of hierarchical grouped IT
BSCs (which applies to the business BSC also) [Va99]. There could be for example
BSCs for each ITIL-Service or for different phases of service lifecycles, like a
scorecard for Service Design. These scorecards on the lower levels may then be
aggregated to a single high level IT BSC.
However, there are some critical success factors that need to be considered when
implementing an IT BSC. One of them is the communication between the IT
department and business managers, as J.T.M. Van der Zee states:
“Potential success is highly dependent on the communication skills of all who
are involved. In many organizations, business managers and IT planners have
shown that they are unable to express themselves in a common language, […]
so that successful links between business objectives, the IT strategy, and the
IT architecture were insufficiently build.”3.
This ability to communicate is especially crucial in the development phase of an IT
BSC. In a case study ([Va99]), the IT manager of a company was not involved in the
planning process of an IT BSC introduced in his company. In that example, the CIO
of said company was not a member of the board and not involved in any decision
making - neither in the initial development, nor in the process of finding cause-and-
effect relationships. This resulted in unrealistic time estimations of steps that needed
to be completed in order to meet goals from the BSC. For example some business
executives shared the opinion
“[…] that the replacement of a core information system could be done almost
overnight.” 4
, what resulted in a delay of more than one year. The conclusion of the study is that
communication barriers between disciplines have to be recognized and addressed
before developing a BSC, as
“[…] the Balanced Business Scorecard does not strictly enforce an integration
of business and IT management processes by itself.” 4.
When business- and IT managers are working together the probability that an IT BSC
is successful is higher.

3
[Va99], Page 142
4
[Va99], Page 153

10
Linking ITIL Key Performance Indicators and the IT Balanced Scorecard

4 Linking ITIL Key Performance Indicators and the IT

Balanced Scorecard
This chapter tries to point out how to link ITIL Key Performance Indicators (KPIs) to
the concept of the IT BSC. After a short introduction about ITIL KPIs, the question is
tackled how to combine these with the IT BSC. This chapter ends with remarks on
how to translate the organizations strategy top-down up to low level measurements.

4.1 ITIL Key Performance Indicators

The lowest level of performance measuring is defined by metrics, which are in the
context of BSCs most of the times called KPIs. Although Kaplan and Norton [Ka92]
did not speak of KPIs when they first introduced the BSC (instead, they used the
more generic term “measure”), the term was soon associated with the BSC.
Nowadays, in most cases the measures or metrics in a BSC are called KPIs. ITIL
also uses the term KPI to describe its metrics.
ITIL defines KPIs as atomic measures that provide low level measurements of how
well the ITIL services are running. For every KPI there should be a target value, to
which the current value is compared to – without a target, there would simply be no
way of telling if a goal is reached or not. There is a number of KPIs suggested in the
ITIL framework for every state of the service lifecycle. It is recommended by the ITIL
framework that these measurements are used to invoke a continual service
improvement – if the actual measures differ from the planned values, actions have to
be taken.
It has also been recommended frequently, that for every KPI a responsible person is
defined (the so called “metric owner”). This person is the responsible for gathering
the necessary data to keep up-to-date values of a metric and sometimes also for
reaching the goal that is measured by “his” KPI.

4.2 Combining ITIL Key Performance Indicators with the IT

Balanced Scorecard
As the goals and strategies of the perspectives of the IT BSC have to be supported
by metrics, using ITIL KPIs for that may come to mind. And in fact many of the
metrics typically found in IT BSCs are recommend by the ITIL framework.

11
Linking ITIL Key Performance Indicators and the IT Balanced Scorecard

But often ITIL KPIs itself can not cover all aspects of an IT BSC; this may especially
be true for the financial perspectives. This is why ITIL KPIs are often coexisting
beside other KPIs or also purely financial measures (like displayed in figure 6).

Figure 6: Combining ITIL KPIS and the IT BSC

According to [Ke05] it is a deadly sin to organize an IT BSC in a way that it offers a

very IT-centric view of its performance - according to the authors an IT BSCs’
missions and goals should instead be business-centered. One might therefore come
to the conclusion that an IT BSC might run the risk of being too IT-centric by
focussing too heavily on ITIL KPIs (as they are often very operational and IT-
oriented). On the other hand [Ke05] says that another sin might be using measures
that lack standard metric definitions. This problem might occur less when using KPIs
that are recommended in the ITIL framework, as there is already existing knowledge
about them. And it should not to be forgotten, that even the OGC recommends in its
book on Continual Service Improvement [Ca07] to use KPIs generated from ITIL
services to measure higher level IT goals and later subsequently aggregate these in
IT BSCs.

12
Linking ITIL Key Performance Indicators and the IT Balanced Scorecard

A conclusion might be to use ITIL KPIs, but also other measures and KPIs where
adequate. If several KPIs are combined to measure one goal, the risk of having a too
short-time-oriented, too IT-focussed measurement can be reduced. When merging
several of these KPIs, the fusion of these might help to see the IT from a more
distanced, business-like perspective.
However, it has to be kept in mind to have a filtering process at work, which helps to
select just KPIs that are relevant for the respective business and IT goals.

4.3 Translating the strategy

The translation of the business goals into IT goals may be a crucial step when
implementing an IT BSC. Figure 7 shows an example of such a transition process.
The financial perspective of the Business BSC has in this example, among others, a
business goal called “provide a good return on investment of IT-enabled business
investments”, which can be mapped to an IT goal “Improve IT’s cost-efficiency and its
contribution to business profitability”. This makes sense, as when the IT cost is going
down the investment in IT will also decrease, which may make it easier to get a
positive Return on Investment.
However, this figure covers only the “investment” part of the business goal to
increase the Return on Investment. However, the term Return on investment
indicates, that there is besides an investment also some kind of Return (or income).
However in this case the IT goal only covers the expenditures of IT, not its incomes
or benefits – therefore, the business goal is only partly translated into an IT goal. The
moneywise benefit of IT however may be difficult to measure, unless the affected
company is a data center, which has concrete revenues for its services. But even if
not, with ITIL successfully in place it should be possible to charge the business for
the IT services it consumed.
Returning to the example in figure 7, the IT goal “Improve IT’s cost-efficiency and its
contribution to business profitability” is being covered by two processes, one of them
is ”Identify and allocate costs”. This process makes sense, as if the cost-efficiency
has to be improved, the costs have to be known. This process is measured by a
bunch of KPIs (in figure 7 they are called “IT Performance Metrics”), which in this
case are not KPIs recommend by the ITIL framework, as the financial perspective is
not covered very deeply by ITIL.

13
Linking ITIL Key Performance Indicators and the IT Balanced Scorecard

Figure 7: Mapping of business goals to IT metrics (screenshot of the software Metricus)

Other IT goals and perspectives could probably be very well measured by ITIL KPIs –
for example the goals of the Internal perspective, as it focuses on internal business
processes. These internal business processes should be easily translatable into IT
processes, which may be managed using the ITIL framework anyway. In this case it
would make sense to use also KPIs that are recommended by the ITIL framework.
Another example of how to combine different types of KPIs could be the IT BSC for a
software developing company. The customer perspective for example could have as
a business goal “support the software development team in developing new
products”. This business goal could be translated into the more specific IT goal
“Support the development team by delivering the necessary tools at high availability”.
In order to measure the fulfilment of this goal, the processes could be for example
“find out which tools the development team needs”, “provide and maintain the
adequate tools” and “provide the availability of development tools”.
Concerning the process “provide and maintain the adequate tools”, one could think of
the following KPIs to measure the fulfilment of this process:

• Number of requests for change (regarding the tools)

• Average ratio of number of computers where provided tools are installed
compared to average number of computers existing in development
department

14
Linking ITIL Key Performance Indicators and the IT Balanced Scorecard

• Percentage of tools that are installed in the latest available product version
• Number of customer complaints (regarding the tools)
• Number of incidents (regarding the tools)
• Percentage of tools that are generally the most popular tools of their category

This list is just for the purpose of giving a general idea, one could certainly think of
more KPIs. What is noteworthy in this example is that metrics defined by the ITIL
framework (like number of requests, number of customer complaints or number of
incidents) coexist beside KPIs that have been developed for this specific process.

15
Cause-and-effect Relations

5 Cause-and-effect Relations
The principle of cause-and-effect relationships in BSCs was explained in chapter 2.2.
There has some theoretical research being done in this field already, for example by
Torben Hügens [Hü08], whose results will be picked up later.
In comparison to this theoretical existing knowledge, there are a lot of software
manufacturers that claim to offer software products which support the creation and
maintenance of BSCs or even IT BSCs. This chapter will evaluate the product
“Metricus” regarding its functionality of the IT BSC and ITIL KPIs in general and then
regarding the finding and evaluation of cause-and-effect relations in special. This is
followed by a broader overview over the market of products, stating if the most
popular products realize the finding and evaluation of cause-and-effect relationships
in their software, and how. This chapter ends with suggestions on how the finding
and evaluating of cause-and-effect relations could be implemented in a software
product.
The selection criteria for choosing the tool “Metricus” for a more detailed evaluation
were primarily, that the software supports creating and maintaining a BSC. Besides
that another criterion was, as this paper specializes on ITIL KPIs and the IT BSC, that
ITIL KPIs and ITIL Service Dashboards are supported.
Three software products were evaluated for this: Hyscore BSC, Opalis Business
Process Centric IT Dashboard and Metricus.
Out of these, “Opalis Business Process Centric IT Dashboard” offered the support of
ITIL KPIs and ITIL Service dashboards, but no BSC. Hyscore BSC did this, but
without supporting ITIL KPIs and ITIL Service dashboards. Only Metricus managed to
support all three criteria (see figure 8).

16
Cause-and-effect Relations

ITIL KPIs,
Tool Balanced Scorecard ITIL Service
Dashboards

Hyscore BSC Yes No

Opalis Business
Process Centric IT No Yes
Dashboard

Metricus Yes Yes

Figure 8: Selection criteria for the software evaluation

5.1 State of the art in research

One of the most comprehensive books in the area of cause-and-effect relationships
in BSCs is “Balanced Scorecard und Ursache-Wirkungs-beziehungen” from Torben
Hügens [Hü08]. He criticizes that the way of finding relations between measures
suggested by Kaplan and Norton - the strategy map - does actually not state
causalities, as it is not clear which metric is influencing the other. Further he criticizes
that this system is based on subjective expectations from managers, as the possible
relations have to be found manually, without a clear model to evaluate these
suggestions.
He further presents a model suggested by the consulting agency Horváth & Partners,
in which the possible relations are determined in the form of a workshop. This seems
better to Hügens, as this enables an extensive discussion about the causalities. It is
recommended however, to evaluate these possible relations afterwards. Regarding
the traditional ways of finding the relations, Hügens sees big problems in the fact that
they point out relations, but they do not quantify these. This means they do not show
how strong the dependency of a relation is. He concludes this by saying that the
status quo in research, as well as in current software products, provides only
methods to find qualitative, not quantitative relations. He further states that statistical
methods are available, which can help to find correlations between the measures.
These are according to the author already in use in order to find cause-and-effect
relationships, despite the fact that these methods can just find correlations, not
causalities.

17
Cause-and-effect Relations

He suggests to use a method called sequence analysis instead, which can help to
find out the time which passes between a cause and the effect it has on another
variable.
He concludes that there is no simple way to solve this challenge to find these
relations due to the complexity of the topic. He sees need for further research in this
area, among other things in how to simulate the effects. This might be one area
where software would be very helpful. He states that one way to solve this problem
might be qualitative reasoning, which is a technique that is usually used in physics,
and can help by means as simulation to find causalities even without much
quantitative data. Like Hügens, Akkermans et al. [Ak02] also suggest to build a model
of the expected relationships, and then test this model using simulation methods
which are fed by historical data available in the company. For doing so, they claim
that most relations between KPIs are not unidirectional, but that instead most metrics
will build cyclic loops of some kind. They tried to prove this by conducting a case
study with a company, for which they build a quantified simulation model. The
expected (loop) causalities where established in a workshop, the model was filled
with company data. The outcome of the study, among other things, established loop-
causalities, as well as the successful calculation of time lags between changes in
related KPIs. In summary, it can be said that the study proves that simulation can in
fact be very helpful to test out relationship models. Unfortunately, according to
Hügens, these simulation methods are hardly ever implemented in software products
(or at least were at the time his book was written) – if this is still the case will be
verified in the course of this chapter.
At least, according to Hügens, statistical methods like correlation analysis are already
in place and used in developing BSCs. If these methods are also embedded in
common BSC software will also be a topic of this chapter.

5.2 Software implementation

Metricus
Metricus is a web-based Software-As-A-Service application for IT-Management by
the Dutch company ITpreneurs. It is a modular system that offers an IT BSC, service
Scorecards and dashboards for various ITIL processes such as for example Service
Desk and Change Management. These modules come with a set of over 600 KPIs
within a “best practice IT KPI library”.

18
Cause-and-effect Relations

Also available are modules to support IT Governance such as a module for project
management, as well as a module for green IT and one for Cost-control.
This evaluation is giving a general overview of the functionality first, afterwards the
features regarding relations are being discussed. The first steps when using Metricus
is the data entry. Here Metricus offers some user interfaces to enable the data entry
and processing from different sources, such as databases or CSV-files. When
building the scorecards, one can use a set of predefined KPIs or define his own
ones. When the system is configured and running, one can choose mainly between
several dashboards, ITIL-service scorecards and the IT BSC. The dashboards are
left aside for this evaluation, as they are not topics of this paper.
Figure 9 shows one of the ITIL-service scorecards, the change Management
Scorecard. It offers an easy to understand and comprehensive overview of the
perspectives, goals and measures.

Figure 9: Change Management Scorecard in Metricus

Figure 10 shows the user interface for IT BSCs, which has a similar structure as the
ITIL-service scorecards.

19
Cause-and-effect Relations

Figure 10: IT BSC in Metricus

Besides that view of the IT BSC, the online demo that was used for this evaluation
contained no features that seemed to support the finding or simulation of
relationships. However, the menu entry “Service Desk Benchmarking” contained a
window that was named “Cause and Effect”, which was unfortunately blank and did
not offer any options.
After asking ITPreneurs about that and about the overall possibilities of Metricus to
support the finding or evaluation of these relationships, the company responded that
Metricus offers the display of MS-Visio diagrams that show relationships between
measures. However, these diagrams have to be created by the customer, and also
the relationships itself have to be found, defined and tested by the customer.
Therefore it can be concluded that Metricus does not support the process of finding
relations.

20
Cause-and-effect Relations

Of course there is more BSC software that might support relations – according to
[Fi06@] in 2006 there were 23 vendors whose products were certified by the
Balanced Scorecard Collaborative, and around 80 more that were not. The most
popular ones are now described according to the features that are stated by their
publishers.

IBM Cognos 8
IBM Cognos 8 is a business intelligence tool that also contains a BSC. According to
IBM it offers cause-and-effect diagrams as well as impact analysis diagrams, which
should help to visualize how strong the impact of a change of one KPI on a related
KPI is. IBM states the following in its product brochures:
“Automatically generated HTML displays of the relationship between metrics
visually guide analysis to the root of performance problems.” 5.
IBM claims that its tools can answer the following questions:
“What are the factors driving the performance?
What other processes or metrics are affected?” 6.
It has, however, not further been researched for this paper, if the cause-and-effect
diagrams have to be created manually or if up to which extent Cognos can help to
find relations and causalities. In the whitepapers and videos IBM offers about Cognos
(at http://www-01.ibm.com/software/data/cognos/library.html) there is also no
mentioning of any tools than can simulate changes, so it can be assumed that this is
not supported.

CP-BSC
The tool CP-BSC by the company “Corporate planning” offers a software supported
BSC. It contains views for cause-and-effect relations and their impacts. However, it
can be assumed that as in Metricus these relations have to be modelled manually,
and that no simulation tool is supported:
„Es ist möglich, sowohl positive als auch negative Wirkungen mit
verschiedenen Ausprägungen (Stärken) zu definieren und zu
dokumentieren.“7.

5
[Co02@]
6
[Mo09@]
7
[Cp@]

21
Cause-and-effect Relations

Artemis Balanced Scorecard

The company Artemis offers a product called Artemis Balanced Scorecard. It offers,
among other, a view that shows cause-and-effect relations. As shown in figure 11,
these relations can also been shown quantitatively with a weighed index.

Figure 11: cause-and-effect relations in Artemis [Ar@]

However, again the product description might lead to the conclusion that the relations
have to be established manually:
„A simple yet powerful feature links Strategic Objectives, Measures (KPIs) and
Initiatives, allowing easy navigation and management of dynamic cause and
effect relationships.“ 8.
One might come to this conclusion because the words “navigation” and
“management” are used, instead of words like suggestion, finding, exploring or
simulation.

8
[Ar@]

22
Cause-and-effect Relations

SBS - STRAT&GO Business Scorecard

The product SBS-STRAT&GO Business Scorecard by the company PROCOS AG
offers, beside the common features for BSC software, possibilities to simulate the
relations between KPIs:
“[…] Simulation of scenarios using the defined hierarchical KPI relationships
[…]” 9.
Figure 12 shows that an extensive simulation of the impact of KPIs on other KPIs is
possible – regarding [St@] however, the relations itself have to be found and defined
manually, PROCOS offers workshops to support the customer in this task.

Figure 12: Cross-Impact-Analysis in STRAT&GO Balanced Scorecard [St@]

SAP SEM Business Planning and Simulation

SAP offers the tool SAP Strategic Enterprise Management (SAP SEM), which
includes a software enabled BSC. Regarding [Cr@] the tool can help in finding
relationships between KPIs:

9
[Sb@]

23
Cause-and-effect Relations

“It supports the development and maintenance process of a Balanced

Scorecard through the following functionality […] Definition of an influence
diagram (cause-and-effect linkage) to visualize dependencies among strategic
objectives on a Balanced Scorecard. “10.
However, it is not mentioned that this “definition of an influence diagram” is supported
by automatization features or that SAP SEM suggests possible relations. Therefore, it
can be assumed that the relations have to be defined manually. This assumption is
supported by the following statement:
“[…] Hypothesis definition through development of influence diagrams to
visualize dependencies among strategic objectives and KPIs in a Balanced
Scorecard. Influence diagrams are stored in the SEM database for shared
access (controlled by defined authorizations) [...]” 11.
However, Regarding [Sa04@2], the relationships between metrics can be used for
“what-if” simulations. And once defined, the relations can also be quantified:
“[…] Quantification of influence diagram cause-and-effect linkages by creating
a dynamic simulation model with SEM Business Planning and Simulation
functionality." 12.
Figure 13 shows the simulation functionality of SAP SEM.

10
[Cr@]
11
[No99@]
12
[No99@]

24
Cause-and-effect Relations

Figure 13: SAP SEM simulation functionality [Sa@]

In addition, according to [No99@] the created simulation models are even assessed
automatically on a regular basis, where actual metric values are compared to their
targets. If simulation models are no longer representing the reality, they are
automatically adapted.

SAS Balanced scorecard:

The company SAS offers a tool called Balanced Scorecard. SAS claims to offer
extensive automatization and simulation support for their tool:
“As noted before, users can create data-driven, dynamic links within strategy
diagrams to virtually any element within a SAS Strategic Performance
Management project (measures, objectives, initiatives etc…This means that
one can “logically” dig deeper into a specific piece of information using
appropriate software for developing insight and foresight.” 13.

13
[Cr07@]

25
Cause-and-effect Relations

In contrast to all other tools evaluated so far, SAS claims not only to be able to
simulate user defined relationships, but also to suggest and find these relations at
first hand:
„Correlation analysis can reveal previously unknown relationships between
metrics, making it clear which metrics are important and where to set
thresholds. This analysis shows a tight correlation between customer churn
and total revenue. Once cause-and-effect relationships are determined and
validated, their relationship to one another and the strengths of the
relationships can be displayed. Users can better predict potential outcomes
based on achieving certain results.“14.
The ability of finding relation is further described as follows:
“That is, the movement of one variable may have been caused by the
movement of another. Using advanced modelling techniques, these causal
relationships can then be isolated and highlighted. These capabilities can help
uncover cause-and-effect relationships. Regression analysis is a common
form of predictive modelling that can reveal previously unknown relationships
between KPIs in an easy and intuitive way. In the process, it reveals broader
and deeper insights into the way an organization really operates. To further
hone your strategy, a variety of analytical methods such as neural nets,
genetic algorithms, experimental design and optimization can be applied.” 15.
Simulation also seems to be supported:
“Example 2: Forecasting and “what-if analysis” help you model scenarios to
determine the best course of action. Change the values for one or more
variables to see the effect on the forecasted margin.“16.
And also the impact that a change in one KPI has on another KPI seems to be
simulatable in form of a sensitivity analysis:
“Applying rigorous analytics to business problems enables managers to more
quickly anticipate future challenges and opportunities with confidence assess
the impact of changing KPI values and respond more quickly with fact-based
decisions.” 17.

14
[Co@]
15
[Pr09@]
16
[Co@]
17
[Pr09@]

26
Cause-and-effect Relations

According to [In08@], every SAS program can be loaded into their data mining
workbench. One might expect that by means of data mining SAS can offer extensive
possibilities to find correlations between measures (for example by the association
method) or conduct simulations.

Conclusion
After evaluating Metricus practically by the means of an online demo, and the
features of six other tools according to the descriptions of their publishers, it can be
concluded that software implementations of the BSC leave in many cases a lot to be
desired. All of the evaluated tools deliver an implementation of the BSC itself via
some kind of dashboard or cockpit view with different degrees of comfort. But only
the tool from SAS really seems to help in finding and suggesting possible relations
between metrics. The situation regarding simulation features looks a bit brighter: here
at least the products from SBS, SAP and SAS seem to offer extensive tools to
simulate quantitative impact changes between KPIs. In the short evaluation that was
executed here, the tool from SAS seems to offer the best features. As the same
company also offers modules for ITIL KPIs and an IT BSC, it seems therefore much
more powerful than the practically evaluated Metricus, which lacks any features
regarding relation finding or simulation.

5.3 How a software IT BSC could look like

It has to be noted up-front that it seems very unrealistic that a BSC system could find
out all important cause-and-effect relations with their linkage in terms of the time lag
between changes and in terms of their impact on each other without any human
intelligence. As the business reality is probably not a deterministic model that can
completely be calculated and predicted, it is and will probably stay a task of
managers to at least evaluate automatically suggested relations and also to suggest
ones that might not be found by correlation analysis. However a well working
methodology could be that a system suggests through correlation analysis or other
means an initial set of cause-and-effect relations. This list can then be expanded by
managers for example in the form of a workshop. For example could a software
solution, based on historical business data, create a list of all correlating measures
that exceed a predefined correlation threshold. This list could also contain the impact
that the metrics have on each other and the time lag that lies between cause and
effect.

27
Cause-and-effect Relations

Managers and employees can then use their knowledge to verify or falsify this list, set
the causalities of the suggested correlations and add own suspected relations. These
new relations can then again be subject of a correlation analysis. Once in place, this
system of causalities can then always be used for simulations, for example for
sensitivity analyses. And even if such a system fails to provide correct relations, at
least the process can help to sensibilize managers, as concluded by [Ho98].
The software from SAS seems to enable all these features - however, one could
imagine even more. One thing that could be useful is a feature to suggest possible
relations not only using historical data of one company, but maybe using data from
several different companies of the same sector or industry. Online based software-
as-a-service solutions like Metricus could collect the data from all its users in an
anonymized way to facilitate analysis and simulation by providing a bigger data pool.
This might especially be useful for companies that do not have much historical data
for KPIs yet, for example because they just introduced ITIL to their company. Such a
comparison within companies that operate in the same sector might be especially
easy if a company is using a lot of KPIs that are recommend by the ITIL framework or
that are very common. However, the suggested relations have to be evaluated
thoroughly, as even in companies of the same sector they might sometimes not be
directly transferable. What can be suggested for further research is to find out how
KPIs of companies of the same sector resemble each other.

28
Conclusion

6 Conclusion
This paper began by pointing out the motivation for the change from IT Management
to IT Service Management and finally to IT Governance. It described the need for IT
Service Management frameworks like ITIL and performance measurement concepts
like the IT BSC. The motivation for and concepts of the BSC were explained, followed
by considerations on the IT BSC. It was further evaluated how to link ITIL KPIs to the
IT BSC and how to translate the business strategy into goals and measures of an IT
BSC. Then the state of the art in research regarding the finding and evaluation of
cause-and-effect relations in BSCs was illustrated, which made clear that the
research theories are somehow ahead of its implementations in software solutions.
The evaluation of the software IT BSC Metricus showed, that even as it offers good
support of ITIL and the evaluation of defined goals in an IT BSC, it lacked features
regarding cause-and-effect relation concepts, as most of the other evaluated
software solutions did too. Three of the seven evaluated products did offer
mentionable simulation support, however only one of the seven products (SAS
Balanced Scorecard) did show signs of features that help in finding possible
relations. This evaluation was followed by considerations on how software enabled
(IT) BSCs could be improved, which can be summarized as the implementation of
simulation and correlation analysis techniques (an approach that already has been
suggested by researchers). The research question of this paper
“What are the current features of (IT) Balanced Scorecard software regarding cause-
and-effect relationships between measures in general and ITIL Key Performance
Indicators in special and how can they be improved?”
has therefore been answered in the course of this software evaluation, as it pointed
out the available features by examining a few representative software products. The
part of the research question about possible improvements of software products has
been answered subsequently.
What remains to be seen is if and how fast the industry will adapt the new mindset of
IT Governance and replace or adjust its existing concepts of IT Service Management.
In this context, it will be interesting to see if ITIL can still be adequate in an
environment where IT Governance is in place – maybe the combination with more
business-centered concepts of performance measurement like the IT BSC can help
ensuring ITIL its popularity.

29
Conclusion

It is hard to predict future features of BSC software but it can be assumed, as there is
already indication for that in some products, that software solutions will be more and
more connected to business intelligence and data analysis tools. This linkage will
provide the desired functionalities by methods like data mining and simulation.
There is probably not much need for further theoretical basic research on finding and
evaluating cause-and-effect relations, as there has been done a lot of work already.
What can and should be done however is to research what mix of methods will
provide the best output for practically implemented software solutions. Then it can
only be hoped that this knowledge will be subsequently implemented by software
manufacturers.

30
Sources

7 Sources
[Ak02] Henk Akkermans and Kim van Oorschot: Developing a Balanced Scorecard
with System Dynamics, Proceedings of the 20th International Conference of
the System Dynamics Society, July 28 - August 1, at Palermo, Italy, 2002

[Ar@] Artemis Balanced Scorecard, http://sg.aisc.com/product/3, last checked on

8 August 2009

[Ba01@] D. R. Backinsell: Performance Measurement and Evaluation,

http://www.gfkmysteryshopping.eu/imperia/md/content/gfk_nop/mysterysho
pping/cs_performancemeasurement.pdf, 2001, last checked on 8 August
2009

[Ca07] Gary Case: Continual service improvement, The Stationary Office, Norwich,
2007

[Co@] Gary Cokins et. Al: Discover how analytics makes the difference between
managing and improving performance, http://www.sas.com/solutions/pm/
pmsupp-analytics.pdf, last checked on 8 August 2009

[Co02@] Cognos Metrics Manager, https://www.bscol.com/pdf/Cognos_Facts.pdf,

2002, last checked on 8 August 2009

[Cp@] CP-BSC Balanced Scorecard, http://www.corporateplanning.com/d/pages/

soft/cpbsc.php, last checked on 8 August 2009

[Cr@] Creating Strategic Focus: The Balanced Scorecard, http://help.sap.com/

saphelp_sem40bcs/helpdata/de/5a/546637a04c2367e10000009b38f8cf/
content.htm, last checked on 8 August 2009

[Cr07] Alec Cram: The IT Balanced Scorecard Revisited, Information Systems

Control Journal, Volume 5, Pages 33-36, Rolling Meadows, 2007

[Cr07@] Creating focus, alignment and agility with SAS® Strategic Performance
Management, http://www.sas.com/resources/whitepaper/wp_3549.pdf,
2007, last checked on 8 August 2009

[Fi06@] Joseph M. Firestone: The Balanced Scorecard: Developments and

Challenges, KMCI Online Press, 2006,
http://www.adaptivemetricscenter.com/media/
BSCdevelopmentsandchallenges.pdf, last checked on 8 August 2009

[Go92] C. Gold: Total quality management in information services – IS measures: a

balancing act, Research Note, Ernst & Young Center for Information
Technology and Strategy, Boston, 1992

[Ho98] Peter Horvath and Lutz Kaufmann: Balanced Scorecard - ein Werkzeug zur

31
Sources

Umsetzung von Strategien, Harvard Business Manager, Heft 5, Pages 39-

48, Hamburg, 1998

[Hü08] Torben Hügens: Balanced Scorecard und Ursache-Wirkungs-beziehungen,

Auflage 1, Gabler Verlag, Wiesbaden, 2008

[In08@] Integrating Your Favorite SAS Program into a SAS Data Mining
Workbench, http://www.sas.com/resources/whitepaper/wp_6874.pdf, 2008,
last checked on 8 August 2009

[Ka92] Robert S. Kaplan and David P. Norton: The Balanced Scorecard-Measures

that drive performance, Harvard Business Review, January-February,
Pages 71-79, Boston, 1992

[Ke05] Jessica Keyes: Implementing the IT balanced scorecard: aligning IT with

corporate strategy, CRC Press, Boca Raton, 2005

[Mo09@] Monitor.Manage.Perform. Scorecarding with IBM Cognos 8 Business

Intelligence, http://www.ibm.com/software/data/cognos/pdfs/whitepapers/
wp_c8bi_scorecarding_monitor_manage_perform.pdf, 2009, last checked
on 8 August 2009

[No99@] David Norton et al.: SAP Strategic Enterprise Management,

http://www.hochschul-management.de/BSC-SAPSEM.pdf, 1999, last
checked on 8 August 2009

[Pr09@] Predictive Performance Management, http://www.sas.com/apps/

whitepaper/index.jsp?cid=3496, 2009, last checked on 8 August 2009

[Sa04@] Mathias Sallé: IT Service Management and IT Governance: Review,

Comparative Analysis and their Impact on Utility Computing,
http://www.cioindex.com/nm/articlefiles/67864-
ITServiceManagementITGovernance.pdf, 2004

[Sa04@2] SAP Strategic Enterprise Management (SAP SEM), http://www.sap.com/

germany/media/50071524.pdf, 2004, last checked on 8 August 2009

[Sa@] SAP SEM simulation functionality, http://help.sap.com/saphelp_sem40bcs/

helpdata/de/f8/4569377f2b8032e10000009b38f8cf/Image113.gif, last
checked on 8 August 2009

[Sb@] SBS-Strat&GO Business Scorecard, http://www.procos.com/en/

pro_mod_sbs.asp, last checked on 8 August 2009

[St@] Strategieprozess innovativ unterstützt,

http://www.procos.com/it/img/pdf/Strategieprozess_innovativ.pdf, last
checked on 8 August 2009

[Th@] The Official ITIL® Website, http://www.itil-officialsite.com/home/home.asp,

last checked on 8 August 2009

32
Sources

[Un@] Understanding Hype Cycles, http://www.gartner.com/pages/

story.php.id.8795.s.8.jsp, last checked on 8 August 2009

[Va00] W. Van Grembergen: The Balanced Scorecard and IT Governance,

Information Systems Control Journal, Vol. 2, Pages 40-43, Rolling
Meadows, 2000

[Va97] W. Van Grembergen and R. Van Bruggen: Measuring and improving

corporate information technology through the balanced scorecard
technique, Proceedings of the Fourth European Conference on the
Evaluation of Information Technology, October 1997, Pages 163-171, Delft,
1997

[Va98] W. Van Grembergen and D. Timmerman: Monitoring the IT process through

the balanced scorecard, Proceedings of the 9th Information Resources
Management (IRMA) International Conference, May 1998, Pages 105-116,
Boston, 1998

[Va99] J.T.M. van der Zee et al.: Alignment is not enough: integrating business and
information technology management with the balanced business scorecard,
Journal of Management Information Systems archive, Volume 16, Issue 2,
Pages 137–156, M. E. Sharpe Inc., Armonk, 1999

[Va07] Jan Van Bon: IT service management: an introduction, Van Haren

Publishing, Zaltbommel, 2007

[Wi94] L. Willcocks: Information Management: The Evaluation of Information

Systems Investments, Chapman & Hall, London, 1994

33