You Cant Manage It If

You Cant Measure It

ISACA
March 2006
Deloitte & Touche LLP and affiliated entities.
Agenda
Do you know how well your information security program is
working?
Key Performance Indicator (KPI)
Key Performance Index (KPX)
Information Collection
Examples
Summary
Deloitte & Touche LLP and affiliated entities.
What do we have to be worried about?
The time between
the discovery of a
vulnerability and
the potential
exploit is
diminishing from
months to days if
not hours
Deloitte & Touche LLP and affiliated entities.
IT Security Governance Maturity Model
The Maturity Model is sponsored by the IT Governance
Institute
It is used to rank the maturity of an organizations practices
and standards against industry best practices and standards
It can be used to help guide an organization on the areas that
will improve their overall information security posture
Deloitte & Touche LLP and affiliated entities.
How do you know if you have an information security
program that effectively manages risks?
Obtain a high score on an ISO 17799 assessment?
Complete regular, active penetration tests with no discovered
vulnerabilities?
Have an acceptably low # of security incidents reported using
the Incident Response process?
Have an effective virus program (few or no infections and any
infections are managed effectively with little interruption)?
Have Measurable Service Level Expectations (SLE) that are
consistently being achieved?
Have an effective IDS program (# and type of alerts are being
managed effectively, little impact on the business, in line or
better than industry benchmarks)?
Obtain certification against an information security reference
standard (ISO 27001)?
Deloitte & Touche LLP and affiliated entities.
There are several problems to avoid when establishing
an information security measurement program
Lack of management commitment
Measuring too much, too soon
Measuring too little, too late
Measuring the wrong things
Imprecise metrics definitions
Using metrics data to evaluate individuals
Using metrics to motivate, rather than to understand
Collecting data that is not used
Lack of communication and training
Misinterpreting metrics data
Deloitte & Touche LLP and affiliated entities.
Key Performance Indicators (KPIs) can help
determine the current status of the information
security program
A key performance indicator is a measure of a particular
organizational performance activity, or an important indicator
of a precise health condition of an organization
Used as an indication of the current state of a component of
the business to take the surprise out of risk
To be effective, the KPI must be defined as succinctly as
possible
Can be measured as an improvement from a known state or
a reference standard
Deloitte & Touche LLP and affiliated entities.
A Key Performance Indicator . . .
Must be something that can be measured and continued to be
measured
Must be precise, meaningful and understandable
Must be relevant to the business
May be required by legislation and/or Regulations
Must have a measurement index that has meaning
Must have an appropriate life (Stickiness)
Should be tied to the organizations vision and strategy
Deloitte & Touche LLP and affiliated entities.
Types of Key Performance Indicators (KPIs)
Threshold when an index reaches set targets or falls into
set ranges
e.g., ETS scores on defined risks
Milestone when a specific condition is reached
e.g., certification
Quantitative measure of value (number, time, $, etc.)
e.g., number of reported security incidents, lost time due to
viruses
Qualitative measure of acceptability or health
e.g., survey ratings, rating of risks
Deloitte & Touche LLP and affiliated entities.
Examples of Key Performance Indicators
Awareness
Knowledge of policies, standards and procedures (surveys and
tests)
Risk Assessment
Depth and breadth of regular risk assessments across the
enterprise (When was the last assessment? Qualitative
measure of the risks, risk index)
Risk Management
Number of incidents reported, amount of loss incurred,
number of situations managed
Audit
Noted deficiencies against the policy and standards (measured
year over year)
Benchmarks and Certification
Maintaining/following IT security certifications such as FIPS
140-1, ISO 27001, ISO 15408 (Common Criteria)
Deloitte & Touche LLP and affiliated entities.
Possible Non-Risk Key Performance Indicators (KPIs)
People
Training & Certifications
Competence Turnover
Technology
Currency
Cost management
Compliance / licensing
Investment
Trends per area
Effectiveness & Return on Investment
Key Risk Indicator experience vs. cost
Productivity
Missed Deadlines
Deloitte & Touche LLP and affiliated entities.
KPIs can be used to measure the Effectiveness of
Investment (EOI)
A Return on Investment (ROI) for information security is
difficult to measure since risk, and especially risk reduction, is
challenging to quantify in terms of dollars.
The Effectiveness of Investment (EOI) could be the
comparison of the effectiveness of the security measures with
the value of the investment.
For example, the number and impact of viruses and worms
can be compared with the investment in virus detection
technology and support programs.
A collection of KPIs could be used to measure the EOI for
information security
Deloitte & Touche LLP and affiliated entities.
A Key Performance Index (KPX) is a summary or
correlation of one or more KPIs that provides an
indication of the overall performance of a defined
area of the security program
May prompt the organization to change strategic direction in
information security
Levels may be triggered by a variety of factors
Must be meaningful and understandable
Must be relevant to the business
Must have a measurement index that has meaning
Must have an appropriate life (Stickiness) and
Should be tied to the organizations vision and strategy
Deloitte & Touche LLP and affiliated entities.
Example KPI Format
Any additional information or comments? Is this a requirement from
legislation or regulations?
Comments
___ Day ___ Week ___ Month
___ Quarter ___ Year ___ Year+
Frequency
Any potential tools used to support the measurement and reporting
process?
Tools
Method used to measure the KPI Method
What does it apply to? Unit/Dept
__ Low __ Medium __ High Effort
__ Quantitative ___ Qualitative ___ Milestone ___ Threshold Type
Who is this KPI relevant to? Stakeholder
What are the objectives of the KPI what is it measuring? Why is it
important?
Objective
Description of the KPI what does it address? Description
Short name or title for the KPI KPI Name
Deloitte & Touche LLP and affiliated entities.
Example Key Performance Indicator (KPI)
Need to have confidence in the detection and reporting mechanisms to be
able to measure changes to the index over time. A lower index will then
mean less risk
Comments
___ Day _X_ Week _X_ Month _X_ Quarter _X_ Year ___ Year+ Frequency
IDS and/or security management/reporting software Tools
Count number of reported security incidents/events at low, medium and
high severity over the past week
Method
Information Security Unit/Dept
__ Low _X_ Medium __ High Effort
_X_ Quantitative ___ Qualitative ___ Milestone ___ Threshold Type
CSIO, CIO, Operations Management, Technology Management Stakeholder
A measure of the relative size and effectiveness of the organizations risk
management processes
Objective
Provides a relative index on the current number of reported security
incidents/events at differing security levels for the recent reporting week
Description
Weekly Reported Security Incidents KPI Name
Deloitte & Touche LLP and affiliated entities.
Example Key Performance Index (KPX)
Need to have confidence in the detection and reporting mechanisms to be
able to measure changes to the index over time. A lower index will then
mean less risk
Comments
___ Day _X_ Week _X_ Month _X_ Quarter _X_ Year ___ Year+ Frequency
IDS and/or security management/reporting software Tools
Count number of reported security incidents/events at low, medium and
high severity over a defined time frame
Method
Core Systems Unit/Dept
__ Low _X_ Medium __ High Effort
_X_ Quantitative ___ Qualitative ___ Milestone ___ Threshold Type
CSIO,CIO Stakeholder
A measure of the relative size and effectiveness of the organizations risk
management processes
Objective
Provides a relative index on the current number of reported security
incidents/events at differing security levels within a specified time frame
Description
Information Security Risk Management Index KPI Name
Deloitte & Touche LLP and affiliated entities.
Several automated tools can provide a view of
security incidents and trends
Deloitte & Touche LLP and affiliated entities.
Security Incidents - Advanced Forensic Tools
Deloitte & Touche LLP and affiliated entities.
The Information Security Program should include a
reporting mechanism that provides a single point of
reference for concise, executive-level information for
business and technology owners.
The dashboard aims to transform data from operations to actionable information for
decision makers
Sample Security Dashboard
Operator Event View
Reports
Incident Tracking
(Ticketing System)
Geographic Threat View
Trend View
Advanced Forensic Tools
Geographical Dashboard
View
Deloitte & Touche LLP and affiliated entities.
An analysis of security incidents will contribute to the
current status of the Information Security Program
Deloitte & Touche LLP and affiliated entities.
Keep track of each area of concern that is the object
of a KPI or KPX definition
Any additional information or comments? Comments
Any required acknowledgement or reporting for this KPI? Reporting
How does the KPI(s) map to the individual performance goals? Map KPI(s) to
Performance Goals
What summary index(s) can be defined that is a high-level representation of
one or more KPIs that are vitally important to the organization?
KPX(s)
What Key Performance Indicators(s) should be defined for this objective? KPI(s)
What are the measurements that may be available to report on this area? Measurements
What are the key control objectives and controls that should be in place for the
organization? The controls should be based on international reference
standards
Key Control
Objectives and
Controls
What is the main objective how is it measured? Why is it important? Objective
What is the Vision and Mission statement that directs IT security? Vision/Mission
Topic - <What is the KPI or area of concern?>
Deloitte & Touche LLP and affiliated entities.
Presentation Name (View / Header and Footer)
An example KPI for Inappropriate Use
Inappropirate Use - KPX
The impact of recorded
inappropriate use events
compared to the amount of
IT security awareness
training per person.
KPI - 1
Number of verified
instances of inappropriate
use over a set time
period. (weekly or by
reporting period)
KPI - 2
Impact of inappropriate use
events to the business in
terms of resources and or
loss over time (weekly or
by reporting time)
KPI - 3
Number of verified
inappropriate use events
compared with the number
of IT security awareness
training days per person
compared over time
Measurement - 1
Number of inappropriate
use cases opened and
verified
Measurement - 2
Amount of service lost to
inappropriate use
Measurement -3
Number of IT security
awareness training days
Deloitte & Touche LLP and affiliated entities.
An example KPX for Inappropriate Use
KPX
Deloitte & Touche LLP and affiliated entities.
An example KPI for Intrusion Detection
KPI - 4
Cost of the IDS program in
relation to the number and
impact of detected events
Measurement - 4
The number of systems with
active monitoring capabilities
KPI - 3
Number of IDS
program failures
Measurement - 5
Number of Sensors per
network segment
Measurement - 6
Cost of the hardware and/or
software to implement intrusion
detection sensors
IDS KPX
The measureable amount of
productivity loss attributed
to intrusions in relation to the
the number of events and the
cost of the IDS program.
KPI - 1
Average amount of Loss
(productivity time) per intrusion
within a set time period (weekly
or per reporting period).
Measurement - 1
Number of incidents of intrusions
detected and reported
Measurement - 3
Amount of downtime or productivity
loss caused by intrusion incidents.
Measurement - 2
Number of incidents of intrusions impacting
the organization that were not reported
KPI - 2
Number of events caught and
prevented by the IDS within a
set time period
Deloitte & Touche LLP and affiliated entities.
An example KPX for Threat Management
Intrusion Detection System (IDS)
Number of Resolved Major and
Catastrophic Incidents Over Time
Time/ Reporting Period
#of
Resolved
Major and
Catastrophic
Incidents
11 22
33
Number of Major and Catastrophic
Incidents Over Time
Time/ Reporting Period
#of Major
and
Catastrophic
Incidents
High Risk Incidents High Risk Incidents
Critical Incidents Critical Incidents
Average Time to Resolve a Number
of Major and Catastrophic Incidents
Average Time to Resolve Major and
Catastrophic Incidents
#of
Resolved
Major and
Catastrophic
Incidents
Major Incidents
Catastrophic
Incidents
Number of Resolved
Major and
Catastrophic Incidents
Major Incidents Major Incidents
Catastrophic
Incidents
Catastrophic
Incidents
Number of Resolved
Major and
Catastrophic Incidents
>4<10hrs/month/
systemproductivity
loss
>4<10hrs/month/
systemproductivity
loss
>10hrs/month/ system
productivity loss
>10hrs/month/ system
productivity loss
Deloitte & Touche LLP and affiliated entities.
Summary
A good collection of Key Performance Indicators will provide
an overview of the current status of risk management within
the organization
Use the collection of KPIs as an information security dashboard
The KPIs can be used to help comply with legislative or
regulatory requirements
Provide the information that can be used for reporting purposes
The KPIs must be carefully selected and defined to be useful
Must be meaningful and measurable
Effective KPIs can be used to demonstrate good management
of risk
For example, KPIs may provide a financial institution the ability to
reduce the percentage of reserve required to offset operational risk
defined by the Basel II Accord
Questions?
Glen Bruce, glebruce@deloitte.ca
Member of
Deloitte Touche Tohmatsu
Deloitte & Touche LLP and affiliated entities.
Deloitte, one of Canada's leading professional services firms, provides audit, tax, consulting, and
financial advisory services through more than 6,100 people in 47 offices. Deloitte operates in Qubec
as Samson Blair/Deloitte & Touche s.e.n.c.r.l. The firm is dedicated to helping its clients and its
people excel. Deloitte is the Canadian member firm of Deloitte Touche Tohmatsu.
Deloitte refers to one or more of Deloitte Touche Tohmatsu, a Swiss Verein, its member firms, and
their respective subsidiaries and affiliates. As a Swiss Verein (association), neither Deloitte Touche
Tohmatsu nor any of its member firms has any liability for each other's acts or omissions. Each of the
member firms is a separate and independent legal entity operating under the names "Deloitte,"
"Deloitte & Touche," "Deloitte Touche Tohmatsu," or other related names. Services are provided by
the member firms or their subsidiaries or affiliates and not by the Deloitte Touche Tohmatsu Verein.