Copyright © 2005 Information Systems Audit and Control Association. All rights reserved. www.isaca.org.
How Can Security Be Measured?
By David A. Chapin, CISA, CISM, CISSP, IAM, and Steven Akridge, JD, CISM, CM, CISSP, IAM
T
raditional security metrics are haphazard at best; at worst
they give a false impression of security that leads to
inefficient or unsafe implementation of security measures.
This paper presents an approach whereby maturity and quality
are combined to provide a more complete and orderly picture of
an organization’s security posture. The approach will be referred
to as the Security Program Maturity Model.
Security metrics—the measurement of the effectiveness of
the organization’s security efforts over time—have always
been difficult to evaluate. How can an organization determine
whether it is secure? The measure of the quality of the security
program can be truly tested only when the organization is
stressed by a crisis. Yet, this situation is exactly what the
security effort is designed to prevent.
Management needs some measure of how secure the
organization is. Organizations need to ask themselves:
• How many resources does it take to be “safe”?
• How can the cost of new security measures be justified?
• Is the organization getting its money’s worth?
• When does the organization know it is “safe”?
• How does the organization compare its posture with others in
the industry and with best practice standards?
The traditional answer to these questions relates to risk
assessment and the residual risk the organization is willing to
take based on business needs and budget limits. Risk
management may beg the question, not necessarily leading to a
stronger security stance.
Imagine, for instance, a risk assessment that lists a threat
matrix and the cost to mitigate the risks. Some items on the list
would be of insignificant cost. Other items will be very
expensive (figure 1). Often, management may decide to mitigate
Figure 1—Weighing the Cost of Security Controls
Is it better to buy more for the same amount of money?
$ Control
$ Control
$$$ Control
$ Control
INFORMATION SYSTEMS CONTROL JOURNAL, VOLUME 2, 2005
the most items for the least amount of money, possibly
bypassing the most expensive items. The assumption is that
aggregating risk reduction controls is a better buy. Thus, there is
a tendency to buy large quantities of security tools and avoid the
more expensive, less glamorous controls. The more difficult
controls tend to be organizational in nature requiring cultural
change (such as a disaster recovery plan) rather than specific
turnkey solutions [such as firewalls and intrusion detection
systems (IDSs)]. Management thinks it is buying more security
for less money.
However, who is to say that more security is purchased?
How can the organization measure the relative protection
gained by each purchase? Is the organization purchasing the
security safeguards in the right order? Is the organization being
exposed to more risk because of the unsystematic approach
toward implementation?
Building security programs from the ground up allows for
the development of new approaches toward these traditional
security metrics problems. A fresh look at these problems
enables the development of a comprehensive solution for any
industry.
This newer, more systematic approach toward security
metrics will:
• Generate reproducible and justifiable measurements
• Measure something of value to the organization
• Determine real progress in security posture
• Apply to a broad range of organizations while producing
similar results
• Determine the order in which security controls should be
applied
• Determine the resources needed to apply to the security
program
Traditional Security Metrics—
What to Count?
A measurement, by itself, is not a metric. Time has to be
brought into the picture, and a metric alone is not the answer
to all the organization’s problems. People have to think
through and analyze the time meaning of the metrics. The trick
is to develop metrics that are simple and provide useful
management information matching security-related, goal-
setting objectives. The metrics have to enlighten the
organization by showing some type of progress.
Obviously, the task of security metrics is to count or
measure something. But what should be counted? How can
security be measured? Figure 2 shows samples of traditionally
used security metrics. Many organizations count incidents
handled, e.g., cyberviruses caught or logged events. How does
this provide a measure of the quality of the security program?
How does this show progress?
 people bring to an organization are their brainpower and the
Figure 2—Traditional Security Metrics time they spend using it. But from a security standpoint,
people’s time may not be a valuable metric. For instance, when
Metric Presumed Measurement Pitfalls measuring time spent on security investigations, if more time is
Number of Effectiveness of automated Why are so many spent, is that necessarily an indication of improved security
computer antivirus controls viruses getting
viruses/ through in the first posture? It could be that time is inefficiently used investigating
malicious code place? How many security incidents because the organization’s procedures are
caught got through that never weak—triggering more incidents to be handled by the security
got caught? group when they could be prevented in other ways, such as
Number of Activity of monitoring What threshold causes better training.
security security events an incident or Finally, another classic metric is cost to the business from
incidents and investigation to get
investigations triggered? Are incidents damage by a security incident. This assumes something bad
triggered because of has happened in the first place. While it may measure the
organizational process effectiveness of disaster response, it is not necessarily a good
flaws? measure of the quality of the security program. Some incidents
Cost of True business loss related What residual risks did are a function of the residual risk the business is willing to
security to security failures the business choose to take, combined with unlucky circumstances. Alternatively,
breaches take? Is it a measure of
a crisis or disaster other incidents may be the result of poor security practices
response, but not opening the door for disaster. How can these be differentiated?
necessarily a function of Or, maybe one of these two events occurred and the crisis
reasonable safeguards management was so good, it caused minimum business impact.
in place? Safeguards provide only a certain degree of security; there will
Time and True business cost of Are the tools, always be some risks.
materials running a security assignments or
assigned to program procedures inefficient, The key to security metrics is obtaining measurements that
security causing people to have the following ideal characteristics:
functions waste time? • They should measure organizationally meaningful things.
Compliance Level of compliance How does compliance • They should be reproducible.
with security matching security relate to effectiveness? • They should be objective and unbiased.
rules program goals What is the order of • Over time, they should be able to measure some type of
compliance? Once
compliance is achieved, progression toward a goal.
is the security program In practice, almost all published security metrics are
“finished”? missing one or more of these characteristics. Traditional
security metrics were a “take what you can get” affair, i.e.,
Incident totals are unreliable measures for this reason: whatever metrics were available were grabbed and reported.
imagine a small town with one police officer. He does no other This way of thinking should change.
police work other than patrolling the highway with a radar gun, A more systematic approach is needed for the development of
pulling over hundreds of speeders. Now imagine a large town metrics that directly fit the previously mentioned characteristics.
with many police officers. They do not use radar guns and have
caught very few speeders but have a large defensive driving
Security Program Maturity
program and an active anti-drunk-driving program. Is the small
One piece of the security metrics puzzle is to measure
town safer than the large town? The count of speeders is only
progress of the security program against a maturity model. This
as good as the sensing mechanism, but that number has no
approach directly targets at least two of the four characteristics
depth to it. What about the small town with nonspeeders who
listed above: it measures organizationally meaningful things and
are drunk—are they not potentially
progression toward a goal.
more dangerous?
The few published security maturity models are
Now compare this to an antivirus tool in an information
summarized in figure 3. For some reason, each has only five
systems environment. The fact that it reports a large number of
levels of maturity. Each model seems to suffer from its own
virus infections may make the security team feel good that its
biases about the definition of maturity.
antivirus tool is working, but what does it really say about
It is herewith proposed that a new standard for maturity be
security? Why are so many viruses getting through in the first
used. Maturity should be a measure of only the program’s
place? How many get through and remain undetected? How
progress over time, not necessarily the quality of the elements
does that measure the quality of the security program? It
of the program. This definition of maturity has several
should be considered a great success if the antivirus tool never
important characteristics:
finds any viruses because none ever gets into the environment!
• It provides the blueprint for a complete security program.
Another traditional security metric is time spent on a task—
• It tells management the order in which to implement security
how long people spend doing security-related functions. In
elements.
some cases, from a project management point of view, this
• It leads toward the use of best practice standards
may be a valuable metric because the only two real resources
(e.g., ISO 17799).1

INFORMATION SYSTEMS CONTROL JOURNAL, VOLUME 2, 2005

 • As long as a standard is used, it provides a way to compare With security, the result should be more like an insurance
one organization’s security program to another. policy. It is not like manufacturing a product. Instead, the
Following this maturity standard, previous models suffer corporation is socializing infrastructure and culture.
from three key deficiencies: This new approach toward a detailed security maturity
1. They confuse quality with existence. One must learn to walk model (called the Security Program Maturity Model) takes a
before one can run. The quality of how well one walks is not management systems approach. Therefore, it follows the
necessarily an indication of one’s running ability. ISO 17799 standards for developing a complete security
2. The existing models need to be customized specifically for program. It involves the existence or nonexistence of a large
the organization. Therefore, it is difficult to directly compare number of elements (figure 4).
results from one organization to another.
3. The current models tend to come from engineering or project Figure 4—General Outline of the
management perspectives. Thus, they focus on program Security Maturity Model
elements meeting certain engineering-style specifications.
They do not necessarily drive the program toward a particular No. of
ISO 17799 Elements Items Covered
organizational goal and therefore work poorly for a security
Categories Measured by Elements
program. The incorporation of total quality management
1. Overall Business need, strategy,
(TQM) and six Sigma philosophies are examples of this. security 11 commitment, roles and
management responsibilities, policies and procedures
2. Asset Valuation, risk assessment, business
Figure 3—Published Security Maturity Models classification 5 ownership, labeling and handling,
and control inventory
Model Description Comments 3. Personnel Hiring and termination, roles and
NIST CSEAT IT Five levels of progressive Focused toward levels security 8 responsibilities, screening, training,
Security Maturity maturity: of documentation reporting, review
Model2 1. Policy 4. Physical and Perimeters, environmental hazards,
2. Procedure environmental risk assessment, access controls,
3. Implementation security 12 safety, asset removal and destruction,
4. Testing monitoring, incident handling,
5. Integration awareness, cooperation
Citigroup’s Five levels of progressive Focused toward 5. Access Perimeters, risk assessment, access
Information maturity: organizational awareness control 11 controls, authentication, need to know,
Security 1. Complacency and adoption user responsibility, access updating,
Evaluation 2. Acknowledgment monitoring, mobile computing, incident
Model 3. Integration handling
(CITI-ISEM)3 4. Common practice
5. Continuous 6. System Standards, life cycle model, review,
improvement development gap analysis, requirements planning,
and 9 testing integrity and certification,
COBIT® Five levels of progressive Focused toward auditing maintenance code repository, release management,
Maturity maturity: specific procedures retirement
Model4 1. Initial/ad hoc
2. Repeatable but intuitive 7. Communi- Standards, all methods of
3. Defined process cations and e-communications, operations
4. Managed and operations procedures, monitoring, backups,
measurable management 16 exception handling, updates and
5. Optimized patches, help desk, change
management, cryptographic systems,
SSE-CMM Five levels of progressive Focused toward security
media handling, malicious code, system
Model5 maturity: engineering and software
acceptance, documentation library,
1. Performed informally design
capacity planning
2. Planned and tracked
3. Well-defined 8. Organizational Security function, monitoring,
4. Quantitatively controlled security 11 advisory, auditing, forum, awareness
5. Continuously improving training, segregation of duties,
CERT/CSO Five levels of Focused toward penetration and vulnerability testing,
Security progressive maturity: measurement of quality incident handling, cooperation
Capability 1. Exists relative to levels of 9. Business Risk assessment, prioritization,
Assessment6 2. Repeatable documentation continuity 7 backups, business continuity/
3. Designated person management disaster recovery planning, testing,
4. Documented updates
5. Reviewed and updated 10. Compliance Regulatory, contractual, intellectual
10 property, labeling and handling, record
Measures using four retention, auditing, sanctions
levels:
1. Initial
2. Evolving
3. Established
4. Managed

INFORMATION SYSTEMS CONTROL JOURNAL, VOLUME 2, 2005

 Since very little judgment is involved, the results are
reproducible and objective. Quality or effectiveness of the
element implementation is not measured, though certain
elements (such as auditing programs), if executed, can lead
toward other quality controls. This is similar to health
department inspections of restaurants. Inspection scores can
tell which restaurants to avoid, but they tell nothing about
whether one will get a good meal at those that pass inspection.
Maturity level is an important measure when comparing one
organization to another. If an organization’s maturity score is
75 percent, would it want to network into another organization
whose score is only 25 percent?
Maturity level leads an organization to better understand its
security program relative to its peers. It provides a yardstick
with which to assess the degree of trust that can be placed with
interconnected computer systems between different
organizations.
This security maturity model is also a blueprint for the order
that program elements should be implemented. Figure 5 provides
an example of the step-by-step approach toward the
implementation of elements. In a maturing program, the elements
are executed based on the outcome of previous implementation
steps. Consequently, it directly tells management when to “buy”
security. It answers the question posed in figure 1.
This is why a separate measurement tool must be employed to
measure quality or effectiveness of the actual implementation.
Security Posture
An improvement to the maturity model is the security
posture, which essentially modifies the maturity model based
on the quality of the implementation of each element. An
advantage of adding a quality measure is that, unlike the
maturity score, security posture is not a static achievement
level. It is dynamic and can change based on the quality of the
continued execution of the program elements. It requires active
management of the security program to maintain a certain
security posture.
Quality is a subjective measure. But, since it is separate
from maturity, the two values cannot be confused, as in other
models. A three-tiered factor (high, medium and low), as
shown in figure 6, is suggested. With detailed threshold
descriptions, it is possible to be objective and produce results.
This scheme is similar to the US National Security Agency
(NSA) Infosec Assessment Methodology (IAM) criticality
model, where items also have three tiers of quality.7 But, unlike
the NSA IAM model, which is only focused on data and
systems, a richer picture of the entire organization’s security
program is provided.

Figure 5—Sample Elements From Figure 6—Sample of a Quality Measure From

Security Maturity Model Security Maturity Model
Order Program Maturity Elements— Program If maturity element is implemented, then…
Performed 2. Asset Classification and Control Maturity Low-quality Medium-quality High-quality
1 2.1 Valuation is performed to identify and understand Element Threshold Threshold Threshold
information assets to protect. 2.4 Information Procedures Assets Pervasive
2 2.2 Risk assessment is performed to identify and quantify Assets developed but partially classification
threats to information assets. classification not classified throughout
3 2.3 Information assets have defined system custodians labeling and implemented entire
and business owners. handling organization
procedures
4 2.4 Information assets classification labeling and developed
handling procedures are developed.
5 2.5 An asset management inventory program is installed
to handle assets on an ongoing basis. An ideal security quality metric could be used as a dashboard
display for management. It could give a near real-time view of
It also avoids the pitfall of implementing security measures the organization’s security posture (see figures 7 and 8). It
in the wrong order, introducing security risks precisely because should be measured on a weekly basis; ownership of individual
safeguards are unsystematically implemented. For example, in program maturity elements should be assigned to specific
figure 5 the organization could actually be harmed if an active
inventory management system (element 2.5) is implemented Figure 7—Simulated Example of Improving
before valuing the assets and performing a risk assessment Program Maturity With Time
(elements 2.1 and 2.2). If it is completed in the wrong order, it Maturity

could take years of wasteful redesign work to the inventory ISO 17799 Categories
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

system to properly categorize and ultimately protect the assets. 1. Overall Security Managementt (11)

Since this model is essentially a detailed compliance tool, 2. Asset Classification and Control (5)

management can possibly misinterpret program maturity. A 3. Personnel Security (8)

high level of maturity may give the false impression of project 4. Physical & Environmental Security (12)

management closure. It might indicate to management that the 5. Cyber Access Control (11)

organization is now “safe” and there is no more need to 6. System Development and Maintenance (9)

7. Communications and Operations Management (16)

support the security effort, while it may instead be an
8. Organizational Security (11)
indication of a safe posture only if the elements have been
9. Business Continuity Management (7)
executed at a high-quality level. But, on the other hand, just
10. Compliance (10) Time 1
because an organization has completed a particular security Time 2
Overall ISO 17799
element, it does not necessarily mean it is doing a good job of it.

INFORMATION SYSTEMS CONTROL JOURNAL, VOLUME 2, 2005

 Figure 8—Simulated Example of Improving Quality Table 9—Simulated Example Showing a
With Time Using a Security Posture Metric Management Dashboard Comparison of
Security Performance by Department
Quality—Time 1
Depart- Maturity Maturity Quality of Implemented
High
ment Elements Level Security Elements
Owned
(implemented
Low elements in bold) high medium low
1 1.1, 3.2, 4.1, Three (3) of
7.5 four (4)
implemented—
Medium
75 percent
2 7.1, 7.2, 7.3, Twelve (12) of
7.4, 7.6, 7.7, 15 implemented—
Quality—Time 2 7.8, 7.9, 7.10, 80 percent
7.11, 7.12,
High
Low 7.13, 7.14,
7.15, 7.16
3 4.2, 4.3, 4.4, Eight (8) of 11
4.5, 4.6, 4.7, implemented—
4.8, 4.9, 4.10, 73 percent
4.11, 4.12

Medium
departments. Thus, in sorting the elements by departments,
management can gain a security understanding specifically
customized by the way the organization is structured.
Figure 9 provides a simulated example of one such fictional
organization. At a glance, it can be seen how each department
rates in program maturity and quality. For example, department
2 is more mature, yet its quality is lower than the other two
departments, and while departments 1 and 3 are at nearly the
same level of maturity, the quality of department 1’s
implementation is higher.
These appraisals need active management on an ongoing
basis. By using security metrics in this manner, the organization
incorporates security deeply into its structure. Security metrics
then become a meaningful gauge of organizational performance,
because they were designed to meet the initial objectives for
metrics. An organization can easily demonstrate security posture
improvements over time. Moreover, as security elements become
adopted in a more systematic way, management can begin to
understand the costs and benefits of an organized, mature and
high-quality security program.
They are sorted by ISO 17799 categories with the numbers
in parentheses representing the actual number of program
elements used for the maturity score.
All the quality measures for existing program elements are
aggregated at two different times.
Endnotes
1
ISO/IEC 17799:2000(E), Information Technology—
Code of Practice for Information Security Management,
December, 2000
2
National Institute of Standards and Technology (NIST)
Computer Security Expert Assist Team (CSEAT) IT
Security Model, csrc.nist.gov/cseat/CSEAT_IT_Security_
Maturity_Levels.htm
3
Dunbar, Thomas M.; “Information Metrics @ Citigroup,”
14 June 2000, Computer System Security and Privacy
Advisory Board (CSSPAB) workshop “Approaches to
Measuring Security,” http://csrc.nist.gov/ispab/june13-
15/Citigroup.pdf
4
IT Governance Institute (ITGI), Control Objectives for
Information and related Technology (COBIT), USA, 2000,
www.itgi.org. Also see www.auckland.ac.nz/security/
InfomationSecurityMaturityAssessment.htm, draft version 0.1,
2003.
5
Systems Security Engineering Capability Maturity Model,
(SSE-CMM), Carnegie Melon University, 1999,
www.sse-cmm.org
6
“CERT Security Capability Assessment Tool,” CSO Online,
CXO Media Inc. and Carnegie Melon University, 2003,
www.csoonline.com/surveys/securitycapability.html
7
G. Miles, et al., Security Assessment: Case Studies for
Implementing the NSA IAM, Sygress Publishing Inc., 2004

INFORMATION SYSTEMS CONTROL JOURNAL, VOLUME 2, 2005

David Chapin, CISA, CISM, CISSP, IAM
is a private security consultant. His current interests include
security infrastructure design, security management,
information assessment and organizational security. He holds
three patents in business processes. He can be contacted at
dchapin@earthlink.net.

Steven Akridge, JD, CISM, CM, CISSP, IAM

is a private security consultant. His current interests include
security management, organizational security, assessment
methodology, forensics, cryptography and security intelligence.
He has served on numerous security industry committees,
including the Federal Public Key Infrastructure Steering
Committee, the Partnership for Critical Infrastructure Security
and the National Association of State CIOs. He may be
contacted at steveakridge@cissp.com.

Information Systems Control Journal, formerly the IS Audit & Control Journal, is published by the Information Systems Audit and Control Association, Inc.. Membership in the association, a voluntary
organization of persons interested in information systems (IS) auditing, control and security, entitles one to receive an annual subscription to the Information Systems Control Journal.

Opinions expressed in the Information Systems Control Journal represent the views of the authors and advertisers. They may differ from policies and official statements of the Information Systems Audit
and Control Association and/or the IT Governance Institute® and their committees, and from opinions endorsed by authors' employers, or the editors of this Journal. Information Systems Control Journal
does not attest to the originality of authors' content.

© Copyright 2005 by Information Systems Audit and Control Association Inc., formerly the EDP Auditors Association. All rights reserved. ISCATM Information Systems Control AssociationTM

Instructors are permitted to photocopy isolated articles for noncommercial classroom use without fee. For other copying, reprint or republication, permission must be obtained in writing from the
association. Where necessary, permission is granted by the copyright owners for those registered with the Copyright Clearance Center (CCC), 27 Congress St., Salem, Mass. 01970, to photocopy articles
owned by the Information Systems Audit and Control Association Inc., for a flat fee of US $2.50 per article plus 25¢ per page. Send payment to the CCC stating the ISSN (1526-7407), date, volume,
and first and last page number of each article. Copying for other than personal use or internal reference, or of articles or columns not owned by the association without express permission of the
association or the copyright owner is expressly prohibited.

www.isaca.org

INFORMATION SYSTEMS CONTROL JOURNAL, VOLUME 2, 2005