You are on page 1of 26

Economic Notes by Banca Monte dei Paschi di Siena SpA,

vol. 34, no. 2-2005, pp. 231256

Risk Mapping and Key Risk Indicators in


Operational Risk Management
SERGIO SCANDIZZO*

In this article I describe a methodology for the mapping of Operational


Risk with the objective of identifying the risks inherent in the different
steps of a business process, selecting a set of variables providing an
estimate for the likelihood and the severity of operational risk (Key
Risk Indicators KRIs) and designing the most appropriate control
activities. I then present two examples of how the methodology described
can be applied to map risks and of how a set of relevant KRIs can be
identified in the front office of a trading business and in the back office of
a lending business. Finally, I discuss how the information conveyed by the
KRIs can be organised and summarised in order to provide a comprehen-
sive look at the risk profile of the various business lines. The structured
presentation of KRIs covering the business processes of a bank is what we
call an Operational Risk Scorecard.
(J.E.L.: G24).

1. Introduction

Risk mapping is often mentioned both in describing various approaches


to operational risk management and, in an audit context, in formulating the
key steps to control self-assessment, as the cornerstone of the risk identifica-
tion process. Yet there is little published guidance on how to perform it
effectively and on how to ensure that the resulting map is indeed complete
and consistent. In other words, although the term is widely used by bankers,
auditors, regulators and consultants alike, and although all these profes-
sionals may even agree on what constitutes an acceptable final product,
they will most likely give widely different explanations on how to get such
product, the resources needed and the costs involved.
Risk mapping is difficult for a number of reasons, all of which can be
summarized by reminding ourselves that the map is not the territory. No
matter how accurate and thorough our analysis is, what really goes on in

* Head of Operational Risk Unit, European Investment Bank, 100 Boulevard Konrad
Adenauer, L-2950, Luxembourg. E-mail: s.scandizzo@eib.org
The views expressed in this article are those of the author and do not necessarily reflect those
of the European Investment Bank.

# Banca Monte dei Paschi di Siena SpA, 2005. Published by Blackwell Publishing Ltd,
9600 Garsington Road, Oxford, OX4 2DQ, UK and 350 Main Street, Malden, MA 02148, USA.
232 Economic Notes 2-2005: Review of Banking, Finance and Monetary Economics

the business is never exactly what is written in the manual. Here are just a
few of the key dimensions:
1 People: Processes are affected by people, and people, no matter how
formalized the process is, adapt, interpret and improvise in response to
circumstances.
2 Specialization: Very few people really understand a specific business
process and its interactions with other people and systems within the
bank. When one of these people leaves or is just absent for a while, the
potential for an operational failure appears.
3 Processes: Processes change all the time and any mapping becomes
obsolete almost overnight after being completed.
In this article, I describe a methodology for the mapping of operational
risk with the objective of identifying the risks inherent in the different steps
of a business process, selecting the key risk indicators (KRIs) (Hoffman,
2002; Davis and Haubenstock, 2002) and designing the most appropriate
control activities. In my approach, therefore, risk mapping is the basis for
all the key components of operational risk management identification,
assessment, monitoring/reporting and control/mitigation as defined by the
Basel Committee on Banking Supervision (2003).
There is more than one way to map risks. The most common tech-
nique is probably the mapping on a probability/severity chart (Figure 1) so
as to identify the key priorities for management. The result in most cases
helps to distinguish between high severity/low frequency and high fre-
quency/low severity losses, but which in general gives no indication as to
what management actions to take in order to change the existing risk
profile. Another way is to map the risks to the phases of a business activity
where they can occur and identify the key risk factors and drivers in the
process. This leads to a somewhat more complex result, rich in qualitative
information rather than in quantitative assessment, but giving very clear
indications as to which parts of the process should be changed in order to
make a difference to the overall risk exposure. It also allows for the
identification of the KRIs that are more relevant to each risk exposure.
Pursuing the application of KRIs to operational risk assessment is
suggested by the need to capture the various issues we find with purely
statistical approaches as well as the impact that managerial decisions may
have on the operational risk profile. In market and credit risk measure-
ment, the key managerial decisions are taken in deciding portfolio compo-
sition, thereby affecting the resulting risk profile directly and in a manner
that measurement models have no problem in capturing. In operational
risk measurement, on the other hand, managerial decisions may affect the
risk profile in a number of different ways (through changes in control
procedures, systems, personnel, to name but a few), none of which any
measurement model can capture in a simple and direct way. Statistical

# Banca Monte dei Paschi di Siena SpA, 2005.


S. Scandizzo: Operational Risk Management 233

Disastrous
Non-threatening
High frequency/
High frequency/
high severity
low severity
Likelihood

De minimis Threatening
Low frequency/ low frequency/
low severity high severity

Severity

Figure 1: Likelihood/Severity Risk Map

approaches in particular will be at a loss in taking into account such changes,


as historical data will reflect a risk and control environment which by and
large no longer exists. The requirement of the new Basel Accord (Basel
Committee on Banking Supervision, 2004) to base risk assessment on 5
years of historical data if taken too literally will have banks generating risk
capital charges on the basis of information largely unrelated to the current
and, even less, the future risk and control environment.1
Statistical and other quantitative techniques (for a survey, see Cruz,
2002) are still very useful in modelling high-frequency/low-impact losses,
like those related to credit card frauds or forged cheques, both because of
the sheer scale of the data available and because the related risk and
control environment is stable. High-frequency/low-impact losses are meas-
ured accurately, but managed to a lesser extent, as it is normally too
expensive or unpractical to eliminate them completely. By contrast, high-
severity/low-frequency losses are not tolerated, and management action is
triggered immediately at each instance with the objective of mitigating the
risk drastically. As a consequence, high-severity-loss events are not very
useful in modelling future exposure, as the risk and control environment,
and hence the statistical distribution underlying such events, changes
sharply immediately thereafter.

1
Think, for example, of how much the recent overhaul in corporate governance, enshrined in
US legislation by the SarbanesOxley Act, will impact the risk and control environment of most
large banks and of how irrelevant extreme loss of data predating such changes already are.

# Banca Monte dei Paschi di Siena SpA, 2005.


234 Economic Notes 2-2005: Review of Banking, Finance and Monetary Economics

2. Objectives of Risk Mapping and KRIs

Risk mapping is the basis of operational risk as, unlike market and
credit risks, it is not product specific. The market risk of a derivative
contract depends strictly on the contracts features and on the relevant
market risk factors. Once the deal is concluded, the underlying process, by
and large, does not matter to the related market risk exposure. It is
impossible, on the other hand, to analyse the operational risk in the
trading activities of a bank without a thorough understanding of the
whole trading process from initial negotiation to final accounting.
It is also not enough to analyse operational risk on a business unit
basis. Although this may seem natural in the light of the need to allocate
responsibility and reward performance and good behaviour, it will give a
biased view of operational risk exposures and may even miss some of them
altogether. In fact, failures in one part of the process can generate failures
in others as well as materialize into losses within units that are organiza-
tionally separate, while being part of the same business process. Controls,
on the other hand, are often performed by an organizational unit in order
to prevent or detect failures happening elsewhere. In many cases, the
organizational separation within the same process (segregation of duties)
is a key control feature in itself. For a more general discussion on opera-
tional risk measurements frameworks and methodologies, see Crouhy
et al. (1998), van der Brink (2002) and Ebnother et al. (2003).
Risk mapping is an analysis tool whereby risk exposures are linked to
the relevant parts of the business process. Designing this tool requires a
methodology to identify and cover all the relevant risks. The mapping will
then allow a bank to analyse the causes of operational failures as well as to
link the consequent financial loss to the part of the organization at the origin
of the problem. In turn, this will be the key step to a transparent measure-
ment and reporting of the corresponding operational risk exposure as well as
to foreseeing and acting upon (through internal controls and other manage-
ment tools) those exposures that are not in line with the banks risk appetite.
As shown in Figure 2, risk mapping may be described as a systematic
way of extracting task-specific information on the various ways a process
can fail and specific indications along the different dimensions of risk
measurement and management. The core of risk mapping always revolves
around the question: What can go wrong?, which can only be answered
on the basis of a thorough analysis of the specific business process.2 Such
analysis provides to distinct, but highly complementary outputs. The first

2
Hence, there are inevitable limitations of ready-made risk management frameworks and of
commercial software solutions. Unlike for market and credit risks, there is no standardization in
operational risk, to the extent that there is no (or only limited) standardization in banking activities,
processes and resources.

# Banca Monte dei Paschi di Siena SpA, 2005.


S. Scandizzo: Operational Risk Management 235

How
People
Process
System

Where
Activity analysis What can go wrong? Business
units

How much
Risk
categories

Assess exposure Foresee risks Control risks


Likelihood Key risk Internal
Impact indicators control
system

Figure 2: Objectives of Risk Mapping

is an understanding of the causes and consequences of specific events: (i)


which particular resource failed (people, process or system); (ii) in which
part of the organization the failure originated and what other parts were
affected; (iii) what was the impact of the failure (financial or otherwise).
The second is a database of quantitative or at least quantifiable informa-
tion that can be used to model the operational risk profile of the organization
as well as to guide management action both in corrective and in preventive
terms. This encompasses (i) the straightforward measurement through a
statistical methodology (based on an estimation of probability and impact
of each risk); (ii) the design of key control activities, and related allocation of
resources, in accordance with the relative importance of each exposure; (iii)
the identification and computation of KRIs as a way to foresee changes in
exposures and hence be able to react promptly and anticipate problems.
Specifically, the role of KRIs is very relevant in the monitoring and in
the forward-looking analysis of operational risk both in complementing
any statistical analysis in areas where data are not readily available and in
ensuring all information about the evolution of the risk and control
environment is taken into account (Finlay, 2004; Vinella, 2004).
A KRI is an operational or financial variable that provides a reliable
basis for estimating the likelihood and the severity of one or more opera-
tional risk events. It can be a specific causal variable as well as a proxy for
the drivers of the events and/or the loss related to an operational risk. It
can be strictly quantitative, like the turnover rate in a business unit or the
number of settlement errors, or more qualitative, like the adequacy of
system or the competence of personnel. It can be perfectly objective, like
the number of hours of system downtime, or more subjective, like the
overall complexity of a portfolio of derivatives. But in order to be useful, it

# Banca Monte dei Paschi di Siena SpA, 2005.


236 Economic Notes 2-2005: Review of Banking, Finance and Monetary Economics

will always have to be somehow linked to one of the risk drivers, or better
to one of the mechanisms generating an operational failure.
It follows that indicators have to be regularly reviewed and updated by
discarding those that have become irrelevant or redundant, changing the
way key data are collected and processed and developing new ones accord-
ing to the evolution of the risk and the control environment.

3. A Methodology for Risk Mapping

The methodology I am proposing rests on the following key concepts:


1 The drivers of operational risk are also the key resources present in
each banking activity: people, process, technology and external factors.
2 An operational failure will occur every time one or more of these
resources is inadequate to the task being performed. This may happen
because the resource is insufficient either in quality or in quantity
(capacity and capability), unavailable at a critical stage (availability
and criticality), or because they break down altogether.
3 Meaningful KRIs will measure and anticipate the inadequacies
described above, and key control activities will be designed to address
them (through prevention, reduction and detection).
The methodology is pictorially described in Figure 3, which shows the
central role of key resources as drivers of operational risk and the relation-
ship between failures in those resources and operational events and losses.
It also shows, as discussed more in detail further below, that each KRI
needs to give a direct measure of the extent or likelihood of failure in one
or more resources drivers.

Activities
People Quality
Process Quantity
Risk drivers Key resources Systems Criticality
External events Failure

Insufficient
Inadequate
Unavailable
KRIs Breakdown
Measure
Monitor
Anticipate Operational event

Loss

Figure 3: A Methodology for Risk Mapping

# Banca Monte dei Paschi di Siena SpA, 2005.


S. Scandizzo: Operational Risk Management 237

This approach is not different in principle from the one adopted in


market risk where we start from the so-called risk factors, equity and
commodity prices, interest and exchange rates. Then we examine the
exposure of the bank to these factors. This is the result of all the existing
positions the bank has opened at a given point in time. Then we look at the
way the portfolio of positions is affected by the behaviour of the risk
factors. This is what we call sensitivity of the position (in the language of
option-pricing theory indicated by Greek letters). Finally, by combining
positions, sensitivities and statistical information on the risk factors, we
estimate the potential loss, that is, the maximum change in value for the
portfolio on a given time period with a given probability: value at risk
(VAR). Figure 4 shows how resources/risk drivers applied to specific
combinations/portfolios of activities expose the bank to risks that depend
critically on how these resources can fail to perform as expected.
The key steps in risk mapping can be summarized as follows:
1 Identification of the key activities (process mapping): This will offer a
clear picture of what activities are carried out as part of each process,
where such activity is carried out and how they are performed. A map
allows examining a business process clearly, without the distraction of
the organizational structure or internal politics. In process mapping,
the level of detail can be from a broad organizational process perspect-
ive down to a micro-detail approach of the smallest unit of work. It is
often useful to map business processes at a high level and then drill
down to successive lower levels. This enables to identify the critical
element and the potential flaws or inefficiencies in processes.

Market risk Operational risk

Market factors Operational factors


Interest and exchange rates People, process, technology, external
Equity and commodity prices dependencies

Portfolio of positions Portfolio of business activities


Exposing the bank to risk factors Exposing the bank to risk factors

Sensitivities of positions to factors Sensitivities of positions to factors


Greeks: , , vega, , Availability, capacity, capability

Financial loss Financial loss


One kind Many different types
Change in value for the portfolio (Regulatory fines, legal liability,
fraud and rogue trading, higher taxes, etc.)

Figure 4: Market and Operational Risks

# Banca Monte dei Paschi di Siena SpA, 2005.


238 Economic Notes 2-2005: Review of Banking, Finance and Monetary Economics

Furthermore, process maps present information about a business pro-


cess in its organizational context. In other words, with one diagram a user
can see all the steps (or events) involved in a business process, the organiza-
tional function that performs the steps, the dependencies in the process and
the order in which the steps are generally performed. The user can also see
the sequential and concurrent nature of activities and the decision points.
Unlike data flow diagrams, process maps take into consideration organ-
izational units (process owners) and characterize how information moves
throughout an organization as business is performed.
Detailed task instructions may accompany the map and use the
process numbers as a reference. For example, a sub-process could be
accompanied by a detailed document that clearly articulates the steps
and specific instructions used by the relevant department to perform it.
These detailed step-by-step instructions also include what department
or person owns the process, the inputs and outputs of the process
and any dependencies of the process. Not only do the instructions
thoroughly document the system, but they can also be used as a
foundation for training materials.
2 Analysis of the risk drivers: People, process, systems and external depend-
encies will influence different activities in different ways. The main tools
that line managers can use to fulfill their organizational responsibility are,
in fact, the key internal drivers of operational risk people, systems and
facilities. This is the basic reason why line management bears primary
responsibility for managing operational risks. Analysing the role and the
relative relevance of each factor within an activity allows understanding
how, in what circumstances and why that resource may fail.
3 Analysis of the risk factors quantity, quality, criticality and failure:
In each activity, the same resources can fail in different ways depending
on the nature of the task performed as well as on the specific risk and
control environment. Capacity, for instance, may be the main risk
factor in certain back office activities while dependency on critical
people may be the key risk factor in a trading front office and so on.
4 Identification of the risks: What happens? is the next question to
answer, following the consequences of the failure all the way down
the process (and through the related ones). The really important thing
during risk identification is not to miss any risks out. You can decide to
ignore some of them at a later stage, after you have assessed them, but
they all need to be included at this stage. Whatever technique (or
techniques) you use, it is important to provide an audit trail so that
you can be sure of what happened and that no risks were omitted.
5 Identification and analysis of the losses: This is the key step, not only for
future categorization and statistical analysis, but also to prioritize exposures
and subsequent control actions. Although at the beginning, in the absence of
a reliable database of historical losses, this task may be primarily based on

# Banca Monte dei Paschi di Siena SpA, 2005.


S. Scandizzo: Operational Risk Management 239

managements expert judgement, it is important to ensure that the resulting


estimates are then updated constantly following any operational event.
Information contained in commercial databases of operational losses
can be used in the development of specific scenarios for risk analysis as
well as to supplement limited internal data. It could not, however, be
the main basis for a statistical estimate of operational risk.
Finally, because of the swift actions normally taken to correct con-
trol weaknesses emerging from operational events, a reliable process
for identification and assessment of losses must take into account the
changes occurring in the risk and control environment and reflect them
quantitatively in the overall estimate of operational risk.
6 Identification and analysis of KRIs. KRIs will be identified on the
basis of the information gathered in the previous steps, namely the
drivers, the factors and the potential losses, and ranked according to
their predictive ability. KRIs should be:
* Relevant, strongly related to the frequency of operational failure
and/or severity of impact.
* Non-redundant: If two indicators are strongly correlated, only one
should be considered.
* Measurable: As much as possible, indicators should be objectively
(and independently) quantifiable and verifiable.
* Easy to monitor: Indicator tracking should not be too cumbersome
and expensive.
* Auditable: Indicators and their sources should be properly
documented.
Because many different operational and financial variables can be
used as risk indicators, KRIs form an absolute heterogeneous set. There
are thus many different ways to classify them. The following simple
classification stresses the relationship between KRIs and the two key
features of an operational risk exposure: likelihood and severity.
Descriptive indicators are variables that give information about some
key business dimension, such as size, volume and amounts, and can there-
fore be thought as somehow linked to the impact (loss) of an operational
failure. Examples of descriptive indicators are number of transactions,
volume of trades and size of assets.3
Performance indicators are usually related to the output of a business
process and give an indication on how well a certain process is working.
Therefore, they are normally related to problems in the process and can
also be used to get an indication of how likely a certain operational failure

3
The basic and standardized approaches as suggested by the Basel Committee on Banking
Supervision are in fact using a single, descriptive KRI as a basis for the calculation of the
operational risk capital charge.

# Banca Monte dei Paschi di Siena SpA, 2005.


240 Economic Notes 2-2005: Review of Banking, Finance and Monetary Economics

is. Typical performance indicators are number of settlement errors and


amount of related losses, number of cancellations and other manual
interventions, and hours of system downtime.
Control indicators are linked to management actions and represent
variables that management can usually directly control. Their main feature
is that management can predict their evolution and can thus use them as
indicators of how the control environment will be in the immediate future.
Examples of control indicators are compensation alignment, percentage of
complex products in a portfolio and age of IT systems.
Descriptive KRIs will be to an extent related to the potential impact of
operational risk, but their ability to predict operational events will be
minimal. Performance KRIs, on the other hand, will be more related to
the likelihood of an event but will need to be combined with some descrip-
tive indicator to give an idea of the potential impact.
In general, control indicators that are related to management actions
will give information on the likelihood of future events that are neither
captured by VAR, to the extent that the latter as a statistical technique
only captures information related to the past, nor by descriptive or
performance-related indicators.
In Figure 5, I have mapped the above categories in terms of their
ability to work as predictors of operational events and proxies for opera-
tional risk exposures. I have also added VAR as a benchmark that is,
hopefully, strongly related to both likelihood and impact of future events.

4. Case Studies

In this section, I will present an example of how the methodology just


described can be applied to map risks to the various components of the

Proximity to risk exposure (impact)


Predictive ability (likelihood)

Control KRIs

VAR (Events)
(Losses)

Performance KRIs

Descriptive KRIs

Figure 5: Predictors of Operational Events and Proxies for Operational Risk Exposures

# Banca Monte dei Paschi di Siena SpA, 2005.


S. Scandizzo: Operational Risk Management 241

task of negotiating and capturing a deal in the front office of a trading


business and of how a set of relevant KRIs can be identified.
Table 1 summarizes in matrix form (risk-mapping matrix) the results
of the methodology for a simple example. It describes the main ways
each key resource/risk factor (people, process, technology and external
dependency) can fail and identify risk, loss and a KRI for each case. It
should be noted that, albeit referring to a specific activity, the example is
still quite generic. In a real case, specific conditions may generate different
or more numerous risk exposures for each kind of failure and different
kinds of losses may occur. Likewise, other KRIs may be pertinent depend-
ing on the particular risk and control environment involved. In a real case,
for instance, only one risk may be really relevant which could generate
many kinds of losses and be tracked by more than one indicator, thus
giving rise to an asymmetric and/or sparse risk-mapping matrix.
In the example chosen (deal negotiation and capture), the people
dimension of operational risk is probably the most relevant both because
the activity is people intensive, even with the help of the best technology,
and because of the potential consequences of human failures.
Another key element of the risk profile, also linked to people, is the winner-
takes-all quality of the trading profession. Few top traders tend to concentrate
in their hands most of the business and of client relationships. This creates a
critical dependency on few individuals that needs to be carefully managed and
monitored, for instance, by looking at a concentration index and at how in line
with market compensation levels and patterns are.
On the process side, approval and authorization procedures are of the
utmost importance, especially given that, by and large, negotiation is a
quite unstructured activity. Lack of appropriate approval processes, for
both existing and new products, can lead either to lose business or, worse,
to concluding transactions which may turn out to be unsuitable for the
client or outright unauthorized. The numbers of new products introduced
as well as an indicator of product complexity (like the percentage of
exotics in a derivative portfolio) are key indicators in monitoring the
ability of existing processes to cope with the business activity.
Although technology is probably not at the top of the risk drivers list,
it still can do a lot of damage at this stage of a deal by slowing down,
disrupting or preventing altogether proper input and recording from hap-
pening. Monitoring system capacity, error rate, downtime and the effec-
tiveness of the business continuity plans is the best way to foresee
problems and act upon them before they become critical.
Finally, a number of risk indicators can be used to monitor opera-
tional risks driven by external events. The ones that I have listed
(concentration of key clients, regulatory ratios, existence of alternative
suppliers and business performance) are but a sample of the factors to
keep under control in order to manage externally driven risk exposures.

# Banca Monte dei Paschi di Siena SpA, 2005.


Table 1: Risk-mapping Matrix Deal Negotiation and Conclusion

Risk driver Risk factor Risk Loss Key risk indicator


242

People Quantity Failure to conclude Opportunity loss/ % of to be filled position on total staff
(sufficient staff) deals and missing of targets lost clients
Quality Loss due to mistakes Market loss (e.g. due to Number of cancellations
(competent staff) (i.e. incorrect hedging) higher than intended VAR)
Criticality Foregone clients/business Opportunity loss/lost clients Concentration of business in staff
(key staff) due to loss of key individuals (i.e. Herfindal Index)
Failure Rogue trading, fraud, etc. Direct financial loss or Above market returns/abnormal
(unauthorized unanticipated market loss trading pattern
behaviour)
Process Quantity (existing Foregone business (i.e. Opportunity loss/lost clients/ Number of new products introduced
process can handle inability to obtain unauthorized deals/unsuitable
all instances) approval) products
Quality (appropriate Foregone business (i.e. Opportunity loss/lost clients Time to produce confirmation
processes) delays in process)
Criticality (appropriate Failure to conclude deals Opportunity loss/lost clients Existence of an alternative approval
process unavailable) (i.e. key process unavailable process
and no backup)
Failure Rogue trading, fraud, etc. Direct financial loss or Audit score
unanticipated market loss
Technology Quantity (system Failure to conclude deals Opportunity loss/lost clients Number of deals/system capacity
capacity) (i.e. trading platform too slow)
Quality (incorrect Error (i.e. wrong pricing) Market loss (e.g. due to Error rate
market information) due to incorrect information higher than intended VAR)
Criticality (critical Failure to conclude deals Opportunity loss/lost clients Downtime
application) (i.e. trading platform unavailable)
Failure (infrastructure Inability to conduct business Opportunity loss/lost clients Time since latest BCP test
breakdown)
External Clients Foregone business due to Opportunity loss/lost clients Concentration of business in key clients
dependencies loss of key clients
Regulators Compliance failure Fines or other sanctions Capital adequacy/liquidity ratio
Suppliers Unavailability of key inputs Opportunity loss or market loss Existence of alternative
Competitors Foregone business Opportunity loss Performance

Notes: VAR denotes value at risk; BCP denotes business continuity plan.

# Banca Monte dei Paschi di Siena SpA, 2005.


S. Scandizzo: Operational Risk Management 243

Table 2: Deal Negotiation and Conclusion Numerical Example

Evolution
Indicators Units Value (%) Threshold Limit Scores Weight

Deal negotiation and conclusion


% of to be filled % 1.5 15 5 10 1.0 1
position on total staff
Number of cancellations # 154 5 150 250 2.0 1
Concentration of % 30 5 20 35 2.0 2
business in staff (i.e.
Herfindal Index)
Above market returns % 50 75 25 40 3.0 3
(% on average)
Number of new products # 12 9 10 15 2.0 1
introduced
Time to produce Hrs 2.5 13 2 4 2.0 1
confirmation
Existence of an Yes/no Yes 56 1.0 1
alternative approval
process
Audit score 14 3 0 3 2 1.0 2
Number of deals/system % 93 36 95 105 1.0 1
capacity
Error rate % 4.7 12 5 10 1.0 2
Downtime Hrs 4.3 0.5 2 5 2.0 1
Time since latest Months 9 12 12 18 1.0 2
BCP test
Concentration of % 20% 0 15 30 2.0 2
business in key clients
Capital adequacy % 115 5 110 100 1.0 3
Existence of alternative Yes/No Yes 0 1.0 1
suppliers
Relative performance % 105 0.4 110 90 2.0 2
(ROE compared to
competitors)
Activity Score (weighted average) 2.0

Notes: BCP denotes business continuity plan; ROE denotes return on equity.

Table 2 contains a numerical example using the KRIs identified in Table 1.


The table shows the value and the unit of measure for each KRI. It also shows
the percentage change over last quarter (Evolution), a threshold representing
the expected values of the indicator (Threshold) and a limit representing the
threshold beyond which management action is required (Limit). The resulting
scores are calculated as follows on the basis of thresholds and limits:
1 Indicator below Threshold is scored 1 acceptable;
2 Indicator above Threshold, but below Limit, is scored 2 acceptable,
but to watch;

# Banca Monte dei Paschi di Siena SpA, 2005.


244 Economic Notes 2-2005: Review of Banking, Finance and Monetary Economics

3 Indicator above Limit is scored 3 unacceptable.


Weights reflect the relative relevance of the indicators and are used to
estimate a weighted average of indicator scores within each activity. An
average of activity scores within each process can also be estimated.
Scores higher than 1 are therefore early warning signals for the specific
process on the basis of specific threshold and limit values. Such values are
set based on historical experience of the process performance as well as on
knowledge about the evolution of the risk and control environment. In
particular, it is important to adjust thresholds and limits to reflect changes
such as increasing volumes, system modifications/replacements, new staff
policies, new regulatory requirements, etc.
In Table 2, only one indicator is in the red zone (unacceptable), the
one indicating an above market return for the trading room of 50 per
cent. This particular indicator will be particularly powerful if applied to
each individual trader, who will in turn have to show the appropriate audit
trail for his/her exceptional returns. However, for the business as a whole,
it is still an indication that something out of the ordinary is happening and
worth of management analysis, especially in the light of the sharp increase
of the indicator from the previous observation.
The number of cancellations slightly above the threshold may indicate
an increase in error rate, but also simply an increase in volume of activity,
whereas the time to produce a confirmation may have increased beyond
the threshold due to staff reduction or, again, to increased volumes. The
level of system downtime may be due to a number of events occurring in
the IT department while the increased level of concentration of business in
key clients coupled with the declining relative return on equity is a warning
on the institutions exposure to external risk factors.
The weighted average of the various scores is 1.62 (rounded to 2.0)
and shows that the activity is to be watched with some tasks to be analysed
with care but that no major failure is on the horizon.
In Table 3, (loan disbursement) we have chosen a back office activity
and are therefore faced with some key processing issues. As back office
activity is driven by the volumes sold in the front office, capacity and
quality tend to be the key risk drivers, in terms of people as well as in terms
of process and technology. One overriding feature is certainly that the
same key indicators can be used to measure performance as well as
operational risk exposure. This also means that risk managers are always
on the verge of crossing the line between risk management and perfor-
mance management when dealing with back office activities, often without
fully realizing it. The misunderstandings and arguments that are system-
atically raised can be easily imagined.
In our case study, most of the KRIs identified (backlog, error rate,
failures and delays) can be used both for risk management and for

# Banca Monte dei Paschi di Siena SpA, 2005.


Table 3: Risk-mapping Matrix Loan Disbursement

Risk driver Risk factor Risk Loss Key risk indicator

People Quantity (sufficient Inability to process disbursements Failure to disburse: reputation Backlog
staff) within deadlines and client relationship
Quality (competent Errors in disbursement processing Interests, legal costs Error rate
staff) (e.g. wrong account)
Criticality Inefficiencies and disruption due All of the above All of the above
(key staff) to loss of key staff plus financial losses
Failure (unauthorized Collusion and/or fraud Direct financial loss Reconciliation failures
behaviour)
Process Quantity (existing process Inability to process (committee Opportunity loss/lost clients Number of transactions
can handle all instances) approving payments does not approved vs. submitted
meet often enough)
Quality (appropriate Errors in disbursement processing Opportunity loss/lost clients Error rate
processes)
Criticality (appropriate Inability to process Opportunity loss/lost clients Existence of alternatives
process unavailable) (e.g. backups)
Failure Unauthorized behaviour Direct financial loss or Audit score
unanticipated market loss
Technology Quantity (system capacity) Inability to process all Failure to disburse: reputation Backlog
disbursements on time and client relationship
Quality (incorrect Errors in data enrichment due Delays and mistakes in Processing delays Error rate
counterpart information) to wrong market information executing disbursements
Criticality (critical Inability to conclude transaction All of the above All of the above
application)
Failure (infrastructure Inability to conduct business Lost business System downtime
breakdown)
External Counterparts Non-reception of payment Liquidity problem, interest Overdraft on nostros
dependencies charges
Regulators Compliance failure Fines Results of audit or
regulatory review
Suppliers Lack of key input Delays and mistakes in Monitoring of suppliers
executing disbursements performance
245

Competitors Competitors Foregone business Opportunity losses

# Banca Monte dei Paschi di Siena SpA, 2005.


246 Economic Notes 2-2005: Review of Banking, Finance and Monetary Economics

management control. However, it is important to understand that a risk


manager will mainly be interested in the ability of the indicator to give early
warning before an operational failure happens rather than in its ability to
measure poor performance. For example, a backlog in processing (for instance,
due to some missing IT functionality) which is stable over time and system-
atically cleared within a certain delay may be of interest to a performance
expert but will become of interest to a risk manager only if it goes up abruptly
for no apparent reason and completely out of line with past experience.4
A delay in nostro reconciliation, even a long one, may be perfectly
explained by the non-receipt of a bank statement, when there has been
only one perfectly known movement on the account, and be of no great
interest to the risk manager. A failure in reconciling on time an otherwise
very active account is, on the other hand, of immediate concern, as more
than one item may result unaccounted for, without the bank realizing it,
until the reconciliation is performed.
Table 4 contains a numerical example using the KRIs identified in Table
3. Here, the percentage of reconciliation failures, the delays in processing,
overdraft on nostro accounts and opportunity losses are all well above the
limits established. Reconciliation failures and processing delays can be read
in conjunction with backlog and error rate (both above their respective
thresholds), all indicating perhaps staffing or other organizational problems
in the back office. Undesired overdraft usually indicates poor liquidity
management, whereby opportunity losses due to business foregone to com-
petitors are a sign that, even with everything working properly, someone else
is outsmarting us in one or more fields and actions need to be taken.
The weighted average of the various scores is 2.08 (rounded to 2.0)
and shows that a number of activities need to be carefully checked to avoid
major problems in the near future.

5. Organizing and Summarizing KRIs: The Operational Risk Scorecard

There are of course two basic ways to look at KRIs: they can be either
considered as individual variables to monitor and from which to get
assessments and warnings related to risk exposures to be reported and
controlled independently or they can be looked at as a system allowing us
to get a picture of the overall risk exposure of a business. In this last case,
KRIs need not simply have to be identified, gathered and analysed as we
have discussed above, but also have to be quantified coherently, aggre-
gated and compared. They must cover all the key risks without

4
On the other hand, in some other back office task, as, for instance, those linked to certain
trading activities, any backlog is unacceptable, and even a minor one will raise a red flag to the
operational risk manager.

# Banca Monte dei Paschi di Siena SpA, 2005.


S. Scandizzo: Operational Risk Management 247

Table 4: Loan Disbursement Numerical Example

Evolution
Indicators Units Value (%) Threshold Limit Scores Weight

Loan disbursement
Backlog % 7.4 1.5 5 10 2.0 1
Error rate % 4.9 3.4 3 5 2.0 1
Financial losses EUR 200.000 5.6 100.000 1,000.000 1.0 3
Reconciliation % 9.8 15.3 1 5 3.0 2
failures
Number of % 93 5 95 85 2.0 1
transactions
approved vs.
submitted
Existence of Yes/no Yes 0 1.0 2
alternatives
(e.g. backups)
Audit score 14 3 0 3 2 2.0 2
Processing delays Hrs 25 45 8 16 3.0 2
System downtime Hrs 3.4 7 2 5 2.0 3
Overdraft on EUR 1,500.000 19 100.000 1,000.000 3.0 3
nostros
Results of 14 4 0 3 2 1.0 2
regulatory review
Monitoring of Yes/no Yes 0 1.0 1
suppliers
performance
Opportunity losses EUR 5,000.000 35 1,000.000 3,000.000 3.0 3
Activity Score (weighted average) 2.0

overlapping and give a balanced overview of the key phases of each


process. The structured presentation of KRIs covering the business pro-
cesses of a bank is what we call an operational risk scorecard (Scandizzo
and Setola, 2003; Anders and Sandstedt, 2003).
The information organized and summarized in the scorecard provides
a comprehensive look at the risk profile of the various business lines. The
scorecard is organized according to the key risk drivers (size and complex-
ity of activities, people, processes, systems and external events) and con-
tains risk indicators, operational losses and more qualitative information
on changes happening in the risk profile and in the structure of internal
controls. This qualitative information should reflect improvements in the
risk control environment that will alter both the frequency and the severity
of future operational risk failures.
It should be noted that combining quantitative information, which
itself is already of a diverse nature, with qualitative information is a

# Banca Monte dei Paschi di Siena SpA, 2005.


248 Economic Notes 2-2005: Review of Banking, Finance and Monetary Economics

process that is both subjective and judgemental. However, as also recom-


mended by the Basel Committee on Banking Supervision, it is a key step in
combining historical, backward-looking data with qualitative, forward-
looking information.
Subjectivity in quantification also comes into play when we try to
aggregate several indicators. This is a well-known problem in multi-criteria
decision-making when applied, for example, to investment analysis (see for
example Loia and Scandizzo, 1995). Suppose we are trying to rank a
number of different companies based on a selection of financial ratios.
Although financial ratios are very crisp quantities, at least two problems
arise in trying to aggregate them. First, when quantities are not commen-
surable, for example debt/equity ratios versus profitability, we need to
bring different indicators to a commons scale in order to aggregate them.
But, within the scope of each indicator, an additional problem arises when
the same value may have a totally different meaning for different kinds of
companies. For example, a debt/equity ratio of three or four may be
perfectly acceptable for a financial institution, but absolutely unacceptable
for a pharmaceutical company. Likewise, when we consider an indicator
such as turnover ratio, the same value of 5 per cent may be considered low
in the front office, but very high in a back office function.
Some of the key issues that arise when trying to aggregate KRIs along
with two different mathematical techniques are discussed in Appendix to
this paper.
A scorecard is also a monitoring tool for operational risk facilitating a
number of preventive controls by combining risk indicators in the evalu-
ation and reporting of the impact of new controls and other changes in the
banks operating environment. A scorecard can also be used for risk
capital allocation purposes on the basis of the various business lines
performances in managing and controlling operational risk.
A sample structure for an operational risk scorecard is reported here-
under (Table 5).

6. Conclusions

In this paper, I have discussed the concepts of risk mapping and KRIs
in operational risk management. I have examined the main objectives of
these tools and proposed a general methodology to map operational risks to
business activities and to select KRIs. I have shown how to apply the
methodology to two specific, albeit rather stylized, business cases, and
I have identified the key issues that arise in organizing KRIs through an
operational risk scorecard. Two different methodologies to aggregate KRIs
and to derive operational risk scores are discussed in Appendix to this work.

# Banca Monte dei Paschi di Siena SpA, 2005.


Table 5: Operational Risk Scorecard

Agency
Corporate Trading Retail Commercial Payment services and Asset Retail
finance and sales banking banking and settlement custody management brokerage

Risk driver Ind. Unit Weight Ind. Unit Weight Ind. Unit Weight Ind. Unit Weight Ind. Unit Weight Ind. Unit Weight Ind. Unit Weight Ind. Unit Weight

Activity

Complexity

People

Process

Systems

Aggregate
indicator

Realized losses

Writedowns
Loss of recourse
Restitutions
Legal liability
Regulatory and
taxation
Loss/damage
to assets

Total losses

Corrective actions

New procedures N. Weight N. Weight N. Weight N. Weight N. Weight N. Weight N. Weight N. Weight
Information
systems
HR and training

Total
(weighted)

Aggregate
indicator
(after corrective
249

actions)

Notes: Ind. denotes indicator.

# Banca Monte dei Paschi di Siena SpA, 2005.


250 Economic Notes 2-2005: Review of Banking, Finance and Monetary Economics

As automation and complexity in banking operations reach new


heights, risk mapping is bound to take a central role both in business
and in risk management. With the recent increase in scrutiny and pressure
for tightening controls coming from regulators and legislators alike, the
need for understanding, documenting and monitoring banking activities is
becoming a major concern at top management level and is not anymore an
exclusive endeavour for auditors and operations managers.
Amongst all the control tools available to managers, KRIs have one
clear advantage, if properly selected and interpreted: they are forward
looking. They ideally complement statistical models, with their sophisti-
cated analysis of past information, through a snapshot, imperfect and
error-prone as it may be, of what might happen in the immediate future.
In activities where timing is always tight, consequences are swift and
unforgiving, and where complexity often clouds the real mechanics of
events, monitoring a set of well-identified KRIs can substantially enhance
the effectiveness of risk management and substantially reduce operational
exposures. A systematic, factor-driven analysis of processes, going from
risk factors through failure and consequences of failures can help identify
the most important indicators almost as a by-product of risk mapping.
Furthermore, a more holistic view of processes and risks can be
achieved by considering a structured set of indicators as a means of
drawing an overall picture of operational risk exposures. This bank-wide
analysis can be implemented through the construction of an operational
risk scorecard, summarizing, aggregating and reporting KRIs by business
activity and risk category.
Finally, it should be noted that, no matter how sophisticated the tools
are and how penetrating the analysis is, risk mapping, KRIs and scorecard
will only be effective with the full involvement of the business people
concerned. Not only are they the most important source of information,
and therefore the basis for the whole analytical process, but they are those
that will make the measurement and monitoring activities meaningful by
helping in interpreting, updating and improving them. In the end, it is
never an analytical or mathematical model that makes the difference, but
rather its effective implementation within the day-to-day management of
the business and the ability to update it in order to respond to the changes
in the risk and control environment.

# Banca Monte dei Paschi di Siena SpA, 2005.


S. Scandizzo: Operational Risk Management 251

REFERENCES

U. ANDERS M. SANDSTEDT (2003), An Operational Risk Scorecard Approach,


RISK, January.
BASEL COMMITTEE ON BANKING SUPERVISION (2003), Sound Practices for the
Management and Supervision of Operational Risk, Consultative Paper,
Basel.
BASEL COMMITTEE ON BANKING SUPERVISION (2004), International Convergence of
Capital Measurements and Capital Standards, Basel.
M. CROUHY D. GALAI R. MARK (1998), Key Steps in Building Consistent
Operational Risk Measurement and Management, in Operational Risk and
Financial Institutions, London: RISK Books.
M. CRUZ (2002), Modelling, Measuring and Hedging Operational Risk, New York:
J. Wiley & Sons.
J. DAVIS M. HAUBENSTOCK (2002), Building Effective Indicators to Monitor
Operational Risk, The RMA Journal, May, pp. 4043.
S. EBNOTHER P. VANINI A. MCNEIL P. ANTOLINEZ (2003), Operational Risk:
A Practitioners View, The Journal of Risk, 5(3), pp. 115.
M. FINLAY (2004), KRIs: An Industry Framework, Operational Risk, July.
D. HOFFMAN (2002), Managing Operational Risk, New York: J. Wiley & Sons.
V. LOIA S. SCANDIZZO (1995), A Fuzzy-based Approach to the Analysis of
Financial Investments, Lecture Notes in Artificial Intelligence, Montreal,
Canada: Springer-Verlag, 1188: pp. 12843.
S. SCANDIZZO (2000), Operational Risk Measurement in Financial Institutions: A
Fuzzy Logic Approach, in B. Bouchon-Meunier, R. R. Yager and
L. A. Zadeh (eds), Uncertainty in Intelligent and Information Systems,
Singapore: World Scientific.
S. SCANDIZZO R. SETOLA (2003), A Scorecard for the Measurement of Operational
Risk, Operational Risk, December.
G. J. VAN DER BRINK, (2002), Operational Risk, The New Challenge for Banks, New
York: Palgrave Macmillan.
P. VINELLA (2004), A Foundation for KPI and KRI, Operational Risk,
November.
L. A. ZADEH (1965), Fuzzy Sets, Information and Control, 8, pp. 33853.

# Banca Monte dei Paschi di Siena SpA, 2005.


252 Economic Notes 2-2005: Review of Banking, Finance and Monetary Economics

Appendix: KRIs Aggregation Methodologies

1. Weighted Average

In order to calculate an overall risk rating for each business line, the
individual indicators will have to be normalized, weighted and aggregated.

Normalization: Every indicator is normalized, i.e. expressed on a common


[0, 1] range by using the following simple transformation (example for the
indicator a):

a
anormalized
maxa

where max(a) is the maximum value indicator a can take.

Estimation of weights: Each bank will need to develop a set of parameters


to assign an appropriate relative weight to each risk indicator. Such
weights will be based on loss of data, empirical evidence, scientific
literature available on the subject, management information, auditors
opinion, sector experience and best practice.
Indicators can also be weighted on the basis of strategic objectives,
with the idea of providing incentives for desired behaviours.

Aggregation and risk ratings. The calculation of an overall indicator for


each risk category is based on a weighted average of the individual indicators
and of the weights discussed above. Such aggregation can provide specific
indicators for each risk category as well as for each line of business.
For example, indicator Ii for risk category i within a specific business
line could be calculated as follows:

X
m
Ir w j ij
j1

where ij is the indicator for risk r and wj is the corresponding weight.


A risk rating for each business line can be calculated by aggregating all
the relevant indicators for that unit. In our example, risk rating for a
specific business line will be calculated as:

X
n
R Ir
r11

# Banca Monte dei Paschi di Siena SpA, 2005.


S. Scandizzo: Operational Risk Management 253

2. Fuzzy Logic

An alternative technique for summarizing the results in a scorecard is


based on the applications of fuzzy logic, originally developed by Zadeh
(1965), to the problem of aggregating risk indicators. The use of fuzzy
logic in operational risk management has been explored by Scandizzo
(2000).
More specifically,5 let us consider a KRI whose range of values is
characterized using the following linguistic variables:
1 Very good (standing for the range of values below the lower boundary
of the tolerance range the operational benchmark or target);
2 No problem (characterizing the lower part of the tolerance range);
3 Tolerable (characterizing the upper part of the tolerance range);
4 Intolerable (stretching from the upper boundary of the tolerance range
the minimal performance standard to double the size of the upper
boundary value to reflect the 10-score in the traditional scoring
approach);
5 Catastrophic (reflecting the 50-score).
Let us choose, for instance:
KRI 1 takes on a value of 3.8 per cent and KRI 2 takes on a value of
9.7 per cent
which implies that
KRI 1: Tolerable 1
KRI 2: Intolerable 0.3
Catastrophic 0.7
The result for KRI 2 corresponds to a managers view that KRI 2 is
rather catastrophic (expressing the word rather by formulas).
Let us now assume that KRI 1 and KRI 2 together are the building
blocks of the (intermediate or final) score that we are interested in. Rules
are defined tying together the linguistic variables of the KRIs to form the
linguistic variables and membership functions of the score. To do this, we
first need a formal representation of the score itself. Scores are most
frequently represented in a symmetric way. We will keep the linguistic
description of the KRIs, and let the score vary between Very good,
No problem, Tolerable, Intolerable and Catastrophic, and thus let
it range on a range between 0 and 5.
Examples of rules:

5
The following example was originally developed by Accenture (2003).

# Banca Monte dei Paschi di Siena SpA, 2005.


254 Economic Notes 2-2005: Review of Banking, Finance and Monetary Economics

Low High Low High


Very low Medium Very high Very low Medium Very high
1 1

0 0
1% 20 KRI 1yr 5yr KRI

Figure 6: Two Examples of Fuzzy Rules

Rule i: If KRI 1 Tolerable and KRI 2 Intolerable, then score


Intolerable
Rule j: If KRI 2 Catastrophic, then score Catastrophic
Because rule i does not provide one single, exclusive value, we have to
apply the minimization law to rule i and get:
KRI 1: Tolerable 1, KRI 2: Intolerable 0.3 ) Score: Intolerable
min(1, 0.3) 0.3
Rule j is exclusive and holds:
KRI 2: Catastrophic 0.7 ) Score: Catastrophic 0.7.
Rule j implies that the score is catastrophic to a degree of 0.7.
We can rephrase our results from steps 1 and 2 saying that the score is
intolerable to a degree of 0.3 and is catastrophic to a degree of 0.7. These
two results can be aggregated using different possibilities of pulling
together the different values for scores. One intuitive approach can be
represented graphically as shown in Figure 6.

Non-technical Summary

In this article, I describe a methodology for the mapping of opera-


tional risk with the objective of identifying the risks inherent in the
different steps of a business process, selecting a set of variables providing
an estimate for the likelihood and the severity of operational risk (key risk
indicators KRIs) and designing the most appropriate control activities.
In my approach, therefore, risk mapping is the basis for all the key
components of operational risk management identification, assessment,
monitoring/reporting and control/mitigation as defined by the Basel
Committee on Banking Supervision.
Risk mapping is often mentioned both in describing various approaches
to operational risk management and, in an audit context, in formulating the

# Banca Monte dei Paschi di Siena SpA, 2005.


S. Scandizzo: Operational Risk Management 255

key steps to control self-assessment, as the cornerstone of the risk identifica-


tion process. Yet there is little published guidance on how to perform it
effectively and on how to ensure that the resulting map is indeed complete
and consistent. In other words, although the term is widely used by bankers,
auditors, regulators and consultants alike, and although all these profes-
sionals may even agree on what constitutes an acceptable final product, they
will most likely give widely different explanations on how to get such
product, the resources needed and the costs involved.
Risk mapping is difficult for a number of reasons, all of which can be
summarized by reminding us that the map is not the territory. No matter
how accurate and thorough our analysis is, what really goes on in the
business is never exactly what is written in the manual. Here are just a few
of the key dimensions:
1 People: Processes are affected by people, and people, no matter how
formalized the process is, adapt, interpret and improvise in response to
circumstances.
2 Specialization: Very few people really understand a specific business
process and its interactions with other people and systems within the
bank. When one of these people leaves or is just absent for a while, the
potential for an operational failure appears.
3 Processes: Processes change all the time, and any mapping becomes
obsolete almost overnight after being completed.
Operational risk, unlike market and credit risks, is not product specific.
The market risk of a derivative contract depends strictly on the contracts
features and on the relevant market risk factors. Once the deal is con-
cluded, the underlying process, by and large, does not matter to the related
market risk exposure. It is impossible, on the other hand, to analyse the
operational risk in the trading activities of a bank without a thorough
understanding of the whole trading process from initial negotiation to final
accounting.
Risk mapping is an analysis tool whereby risk exposures are linked to
the relevant parts of the business process. Designing this tool requires a
methodology to identify and cover all the relevant risks.
The methodology presented in this paper is based on the following key
ideas:
1 The drivers of operational risk are also the key resources present in each
banking activity: people, process, technology and external factors.
2 An operational failure will occur every time one or more of these
resources is inadequate to the task being performed. This may happen
because the resource is insufficient either in quality or in quantity
(capacity and capability), unavailable at a critical stage (availability
and criticality), or because they break down altogether.

# Banca Monte dei Paschi di Siena SpA, 2005.


256 Economic Notes 2-2005: Review of Banking, Finance and Monetary Economics

3 Meaningful KRIs will measure and anticipate the inadequacies


described above, and key control activities will be designed to address
them (through prevention, reduction and detection).
In this paper, I also present two examples of how the methodology
described can be applied to map risks and of how a set of relevant KRIs
can be identified in:
1 the negotiation and capture of a deal in the front office of a trading
business;
2 the disbursement and administration of a loan in the back office of a
lending business.
Finally, I describe how the information conveyed by the KRIs can be
organized and summarized in order to provide a comprehensive look at
the risk profile of the various business lines. The structured presentation of
KRIs covering the business processes of a bank is what we call an opera-
tional risk scorecard.
A scorecard is also a monitoring tool for operational risk facilitating a
number of preventive controls by combining risk indicators in the evalu-
ation and reporting of the impact of new controls and other changes in the
banks operating environment. A scorecard can also be used for risk
capital allocation purposes on the basis of the various business lines
performances in managing and controlling operational risk.

# Banca Monte dei Paschi di Siena SpA, 2005.