You are on page 1of 42

IFAC Information Technology Guideline 6

Information April 2002

Technology
Committee

IT Monitoring

Issued by the
International
Federation of

Accountants
This Guideline of the Information Technology Committee was approved
for publication by the International Federation of Accountants in April
2002.

In this guideline series, the International Federation of Accountants,


through its Information Technology Committee, seeks to promote
executive understanding of the key issues affecting the management of
information and communications. This series of guidelines is written for
management.

This guideline is the sixth of the series and covers Information


Technology Monitoring. In addition to providing an approach to
developing effective information technology plans, it provides an
understanding of the nature and importance of information technology
planning.

IFAC welcomes any comments you may have. Comments should be sent
to:

Technical Director
International Federation of Accountants
535 Fifth Avenue, 26th Floor
New York, NY 10017 USA
Fax: (212) 286-9570

Copies of this paper may be downloaded free of charge from the IFAC
website at http://www.ifac.org.

The approved text of this Guideline is that published in the English


language.

Copyright © April 2002 by the International Federation of Accountants.


All rights reserved. No part of this publication may be reproduced, stored
in a retrieval system, or transmitted in any form or by any means,
electronic, mechanical, photocopying, recording, or otherwise, without the
prior written permission of the International Federation of Accountants.

ISBN 1-887464-84-0.
CONTENTS

PAGE

EXECUTIVE SUMMARY .................................................................... 1

WHY IS MONITORING OF IT IMPORTANT? ....................................... 5

WHAT IS IT MONITORING? ............................................................... 8

HOW DOES MANAGEMENT MONITOR IT? ..................................... 13

WHAT IT MONITORING TOOLS ARE AVAILABLE? .......................... 14

WHAT IS THE BEST APPROACH TO IMPLEMENT IT MONITORING? 19

WHEN? ........................................................................................... 22

WHO?.............................................................................................. 22

M1 – MONITORING THE PROCESSES .............................................. 23

M2 – ASSESSING INTERNAL CONTROL ADEQUACY ....................... 26

M3 – OBTAINING INDEPENDENT ASSURANCE ............................... 29

SECURITY AND RISK MONITORING................................................ 33

ADDITIONAL IT MEASURES TO MONITOR ..................................... 35

KEY PERFORMANCE INDICATORS (PERFORMANCE DRIVERS)...... 36


PREFACE
In a digital world, the management and use of information, information
systems and communications are of crucial importance to the success of
an organization. This arises from the:
increasing dependence on information and the systems and
communications that deliver the information;
scale and cost of current and future investments in information; and
potential of technologies to dramatically change organizations and
business practices, create new opportunities and reduce costs.

Many organizations recognize the potential benefits that technology can


yield. But, with those potential benefits come certain risks. To be
successful, organizations must also understand and manage the risks
associated with implementing new technologies. To provide effective
direction and adequate control, therefore, executive management needs to
have an appreciation of the benefits, risks and constraints of information
technology (IT).

In this series of guidelines, the International Federation of Accountants’


Information Technology Committee seeks to promote executive
understanding of key issues affecting the management of information and
communications. This series of guidelines is written for management.

This guideline is the last of the series and covers managing IT


monitoring. In addition to emphasizing the nature and need for IT
monitoring and its impact on IT governance, this guideline provides an
understanding of the main principles on which IT monitoring should be
founded, and a generic approach for implementing effective IT
monitoring.

Executives in various capacities (for example, accountants, financial


controllers, auditors or business managers) are frequently asked to
manage, participate in, assess or comment on the IT monitoring process.
They can do this only if they have a sound knowledge of the principles and
practices required to manage IT monitoring.

IFAC’s Information Technology Committee would like to acknowledge the


support from the Information Systems Audit and Control Association
(ISACA) and thank its various contributors who provided valuable input
for this document:
Susan M. Caldwell, ISACA
Erik Guldentops, IT Governance Institute
John W. Lainhart IV, PricewaterhouseCoopers
Akira Matsuo, Chuo Aoyama Audit Corporation
Ron Saull, Great West Life Assurance Company/London
Life/Investors Group
Michael Schirmbrand, Ernst & Young
Executive Summary

Executive Summary

Why monitor IT?


1. Information, and the systems and communications that deliver
the information, are truly pervasive throughout today’s
organizations. Executive management has a responsibility to
ensure that the organization monitors its use of information and
information technology (IT). IT monitoring is fundamental to IT
governance and part of management’s stewardship responsibility
to make certain that what was agreed to be done is being done
and is being done in line with directions and policies set by the
board. Monitoring is needed to make sure that those to whom
responsibility has been delegated are doing the right things, are
doing them right and can be held accountable if they do not.

What is IT monitoring?
2. Those responsible for IT governance need first to set measurable
goals, then delegate the execution to executive management and,
finally, regularly verify that performance matches the goals. If
goals and measures are not in line, the governance body needs to
take corrective action, provide redirection or, possibly,
reconsider the original goals. Monitoring of IT is enabled by the
definition of relevant performance indicators, the systematic and
timely reporting of performance and prompt acting on any
deviations identified. IT monitoring is especially important
because of the complexity and risk involved in IT activities. It
has the business goals of ensuring the delivery of information to
help the organization achieve its objectives and ensuring the
achievement of performance objectives for the IT function.

IT monitoring covers:
how IT sustains the business with operational processes and
risk and control systems;
whether IT complies with business strategy, standards and
policy;
how IT improves the business with technology, process and
organizational changes; and

1
IT Monitoring

how IT supports enterprise growth through process


knowledge and service capability.

How does management monitor IT?


3. While IT monitoring processes are unique to the needs and
circumstances of each organization, they are generally developed
using seven core principles.

KEY/CORE PRINCIPLES
COMPREHENSIVENESS— Any monitoring activity has to
be comprehensive, based on simple and consolidated
measures focusing on exceptions.
RELEVANCE — Any monitoring activity has to be relevant
to the mission, vision, goals and strategy of the enterprise.
ACCEPTABILITY — An effective monitoring approach has
to be acceptable to those being monitored. This means not
invading their privacy and not intruding into their day-to-
day responsibilities.
TIMELINESS — To make correct and expedient decisions,
monitoring data must be available to detect deviations that
need to be reported immediately.
VERIFIABILITY — Information obtained by the monitoring
process should be verifiable by other means – thus, it should
be accurate and, whenever possible, it should be based on
fact.
ACTION-ORIENTED — Any form of monitoring must
enable expedient corrective action.
FLEXIBILITY/ADAPTABILITY — The monitoring system
should be easily adaptable to provide accurate, relevant and
timely information in a changing environment.

What tools are available?


4. While many IT monitoring tools are available, management is
effectively using seven key tools in performing IT governance
responsibilities:

2
Executive Summary

traffic light reports to follow up on projects and strategic


initiatives;
performance management through balanced scorecards (and
dashboards);
benchmarking for decision making relative to IT
investments for risk and control;
active monitoring of the IT infrastructure;
brainstorming for risk management and improvements;
internal and external audit for independent assurance;
management reporting for executive management review.

What is the best approach?


5. Although monitoring of IT is unique to the organizational
environment, the monitoring process and the underlying
activities are similar. Usually, the process consists of the
following six phases:
Phase I: Orientation. This start-up phase is required to
determine the scope of monitoring and the methodology and
techniques to be applied. In this phase, the resources required for
monitoring IT are mobilized.
Phase II: Criteria Definition. This phase is regularly covered in
the planning/design phase of each IT and business process. Goals
or performance measurement indicators are set up for
monitoring.
Phase III: Ongoing Monitoring. Ongoing monitoring is a
continuous supervisory function over key IT activities and
control processes. Exceptional events have to be identified and
tracked. Performance measures need to be established, involving
both IT and the stakeholders, aligned with the strategy and
reviewed on an ongoing basis.
Phase IV: Separate Periodic and Ad Hoc Monitoring. Besides
ongoing monitoring, separate periodic and ad hoc monitoring is
vital to ensure the ongoing monitoring and other control
functions operate properly, to periodically review IT-related
risks and opportunities and to obtain comfort relative to major IT
decisions. Periodic monitoring includes internal audit

3
IT Monitoring

procedures, external assurance, self-assessments and


brainstorming sessions.
Phase V: Subsequent Actions. Subsequent actions include
corrective actions to redirect IT activities and processes and
bring them back in line with goals, strategy and policy;
minimization of adverse effects; refinement of goals and
measures; changes to strategy, policy and standards; and
initiation of reassessment activities.
Phase VI: Reporting. For monitoring to be able to support
effective IT governance, management reporting about all phases
of the monitoring process, including subsequent actions and
escalation procedures, is an essential element of the
recurring/iterative control cycles.

When?
6. Monitoring is necessary always whenever IT is used within an
organization: from planning and organization, acquisition and
implementation to delivery and support. For effective IT
governance, monitoring is absolutely crucial.

Who?
7. Everyone who has a specific role and/or responsibility for
achieving IT goals and processes must be involved in monitoring
IT. Effective monitoring involves the entire organization, as
information is captured, consolidated and reported up the various
management levels.

4
Why is Monitoring of IT Important?

Why is Monitoring of IT Important?


8. In a global information society, where information travels
through cyberspace on a routine basis, the significance of
information is widely accepted. In addition, information, and the
systems and communications that deliver the information, are
truly pervasive throughout organizations – from the user's
platform to local and wide area networks to servers to mainframe
computers. As such, information technology (IT) has become an
essential part of most enterprises, and management increasingly
needs to address:
IT’s enabling capacity for new business models and
changing business practices;
IT’s increasing costs and information’s increasing value;
the risks of doing business in an interconnected digital
world and the dependence on entities beyond the direct
control of the enterprise;
IT’s impact on business continuity due to increasing reliance
on information and IT in all aspects of the enterprise;
IT’s ability to build and maintain knowledge essential to
sustain and grow the business; and
the failures of IT increasingly having a major impact on
reputation and enterprise value.

9. Boards and executive management generally expect their


organization’s IT to deliver business value, i.e., provide fast,
secure, high-quality development; generate maximum return on
investment; and move from efficiency and productivity gains
toward value creation and business effectiveness.

10. While some have been successful, many organizations find that
their expectations and reality do not always match.
Unfortunately, too often, boards and executives end up with:
business losses, reputational damage and a weakened
competitive position;
the failure of IT initiatives to deliver the innovation and
benefits they promised;

5
IT Monitoring

inadequate or even obsolete technology;


un-met deadlines and budget overruns.

11. Boards and management who exercise proper IT monitoring


often uncover and address problems in time to ensure a
successful outcome. Accordingly, executive management has a
responsibility to ensure that the organization monitors its use of
information and IT, to continuously check whether goals are
being achieved, value is being delivered to the business and risks
are mitigated.

12. Increasingly, top management is looking at IT performance. This


is best illustrated by the balanced scorecard, which translates
strategy into action to achieve goals with a performance
measurement system that goes beyond conventional accounting.
Balanced scorecards augment traditional financial measures,
incorporating measures for those relationships and intangible
(e.g., knowledge-based) assets necessary to compete in the
information age: customer focus, process effectiveness and the
ability to learn and grow.

13. Because it is so essential to an enterprise’s operations, IT needs


its own scorecard. Defining clear goals and good measures that
unequivocally reflect the business impact of the IT goals is a
challenge that needs to be resolved in cooperation among the
different governance layers within the enterprise.

6
Why is Monitoring of IT Important?

14. The following illustration depicts some examples of outcome


and performance measures for the different dimensions of the IT
balanced scorecard. (More information on the IT balanced
scorecard is provided in the “Tools” section and an extra set of
possible IT metrics is provided in Appendix 3.)

Financial

• # of IT
customers
• Cost per IT
customer
• Cost-efficiency
of IT processes
up
• Delivery of IT
value per
employee
Customer
Process
• Level of service
delivery up • Availability of
• Satisfaction of systems and
existing services
customers • Developments
• # of new on schedule
customers Information and budget
reached • Throughput
• # of new and response
service delivery times
channels • Amount of
errors and
rework

Learning

• Staff productivity
and morale
• # of staff trained
in new
technologies/servi
ces
• Value delivery per
employee up
• Increased
availability of
knowledge 7
systems
IT Monitoring

Control
Device Comparison
with standard
Observed (assessor)
information about
what is happening
(detector)

Entity being
Controlled Behavior-altering
communication, if
necessary (effector)

What is IT Monitoring?
15. Monitoring is fundamental to any control system. Monitoring is
the process of observing what is happening (detection) and
comparing it to a standard that has previously been set
(assessing). While strictly not part of monitoring, the
communication that results with the aim of altering observed
behavior may not be ignored.

16. In the context of IT governance, this means that measurable


goals need to be set first and then delegated to executive
management. Those responsible for IT governance need to verify
regularly that performance matches goals. If goals and measures
are not in line, the IT governance body needs to take corrective
action, provide redirection or, possibly, reconsider the original
goals.

8
What is IT Monitoring?

Act if not
aligned

Set Deliver
measurable Compare against
results
goals the goals

Measure performance

17. IT monitoring is fundamental to IT governance; it is part of


management’s stewardship responsibility to make certain that
what was agreed to be done is being done. IT monitoring is
needed to make sure that those to whom responsibility has been
delegated are carrying it out correctly and can be held
accountable if they are not.

18. IT monitoring is enabled by the definition of relevant


performance indicators, the systematic and timely reporting of
performance and prompt acting on deviations.

19. Monitoring of IT is especially important because of the


complexity and risk involved in IT activities. It has the business
goals of ensuring the delivery of information to help the
organization achieve its objectives and ensuring the achievement
of performance objectives for the IT function.

9
IT Monitoring

20. The performance objectives are best illustrated by the IT


governance framework proposed by the IT Governance Institute.
That model implies that the governance entity needs to monitor:
whether IT delivers value in supporting the enterprise’s
strategy, in sustaining the enterprise on a day-to-day basis
and in enabling new products and services;
whether IT’s risks are mitigated; and
whether IT is in compliance with laws and regulations and
with standards and policies.

Provide
Direction

Set Objectives IT Activities

• IT is aligned • Increase
with the automation
business, (make the
enables the
business
business,
and Compare effective)
maximizes • Decrease cost
benefits (make the
• IT resources enterprise
are used efficient)
responsibly • Manage risks
• IT related risks (security,
are managed reliability and
appropriately compliance)

Measure
Performance

10
What is IT Monitoring?

21. Additionally, executives put control frameworks in place to


provide assurance that enterprise objectives are being met and
that agreed strategy is executed within established policies.
Hence, monitoring of the proper functioning of internal control
systems, as predicated by Internal Control – Integrated
Framework, issued in 1992 by the Committee of Sponsoring
Organizations of the Treadway Commission (COSO),1 is an
important, but not the sole, subject of IT monitoring.

22. Management should establish the means for monitoring, either


through independent evaluations or ongoing, structured and
independent process checks:
compliance with strategic goals;
the achievement of tactical projects and activities;
the performance of people, systems and processes;
the proper functioning of the internal control systems;
adherence to internal standards and policies; and
observance of laws and regulations

23. Furthermore, it is essential to monitor the fulfillment of


committed improvements resulting from these monitoring
activities:
process, systems and people performance improvement
programs;
self-assessments;
quality management;
risk management; and
internal and external audit.

24. Most monitoring activities have the ultimate goal of:


sustaining the business;
ensuring compliance; or

1
COSO is a voluntary private sector organization dedicated to improving the quality of
financial reporting through business ethics, effective internal controls and corporate
governance.

11
IT Monitoring

improving the business.

25. Each of these goals is illustrated by specific monitoring


processes of the Control Objectives for Information and related
Technology (COBIT) Control Framework, i.e., M1 - Monitor the
Processes, M2 - Assess Internal Control Adequacy and M3 -
Obtain Independent Assurance.

26. A detailed description of the COBIT monitoring processes are


provided in Appendix 1, covering:
high level control objectives;
detailed control objectives;
critical success factors (the most important things to do to
increase the probably of the IT process achieving its goals);
outcome measures of key goal indicators or KGIs
(measuring whether the process achieves its goals); and
performance drivers of key performance indicators or KPIs
(measuring whether the process performs well).

27. These COBIT monitoring processes provide valuable tools for


management in assessing its effectiveness in managing the IT
monitoring process.

12
How Does Management Monitor IT?

How Does Management Monitor IT?


28. While IT monitoring processes are unique to the needs and
circumstances of each organization, they are generally developed
using seven core principles.

KEY/CORE PRINCIPLES:
COMPREHENSIVENESS — Any monitoring activity has to
be comprehensive, based on simple and consolidated
measures, focusing on exceptions.
RELEVANCE — Any monitoring activity has to be relevant
to the mission, vision, goals and strategy of the enterprise.
Alignment of the IT strategy to the enterprise strategy is a
critical success factor for successful IT governance.
ACCEPTABILITY — An effective monitoring approach has
to be acceptable to those being monitored. This means not
invading their privacy and not intruding into their day-to-
day responsibilities. The “tone at the top” and maturity level
of the internal control systems are essential to achieving
acceptability.
TIMELINESS — To make correct and expedient decisions,
monitoring data must be available to detect deviations that
need to be reported immediately. The frequency of
monitoring different activities of an organization should be
determined by considering the risks involved and the
frequency and nature of changes occurring in the operating
environment.
VERIFIABILITY — Information obtained by the monitoring
process should be verifiable by other means. Thus, it should
be accurate and, whenever possible, it should be based on
fact. (It should be noted that obtaining an opinion/feeling is
also part of the governance process – actually, future
predictions to drive the strategy are not always based on
fact.)
ACTION-ORIENTED — Any form of monitoring must
enable expedient corrective action. Executive management
must ensure that the monitoring function is properly defined

13
IT Monitoring

and structured within the organization to take the actions


needed.
FLEXIBILITY/ADAPTABILITY — The monitoring system
should be easily adaptable to provide accurate, relevant and
timely information in a changing environment. Boards of
directors, audit committees and executive management
should obtain unbiased information about line activities for
dynamic, flexible and adaptable decision making.

What IT Monitoring Tools are Available?


29. While many IT monitoring tools are available, management is
effectively using seven key tools in performing IT governance
responsibilities:
Traffic light reports to follow up on projects and
strategic initiatives
Traffic light reports have become a preferred reporting
mechanism for executives and boards. They exemplify the
principle of comprehensiveness and the practice of
exception reporting. They provide a “green” condition when
the committed action is on schedule and on budget. When
issues are known that suggest that the commitment might go
over budget or schedule, or might not achieve all of its
objectives in the future, an “orange” condition is reported.
When budgets or schedules are exceeded, or when it is clear
that goals will not be achieved without major changes or
investments, then a condition “red” is provided.
These commitments can be of different types, such as:
◦ a project,
◦ an improvement initiative, or
◦ the closure of an audit recommendation.
The improvement initiatives can result from process re-
engineering activities, risk brainstorming sessions, quality
reviews, control self-assessment, etc. It is evident that
orange and red conditions will require a short explanation of
status and corrective actions. It is also good practice for the
board and executive and operational management to have

14
What IT Monitoring Tools are Available?

clear agreements on when orange and red conditions need to


be raised and what needs to be reported when they do occur.
Performance management through balanced scorecards
(and dashboards)

as s
Financial

Ta ures
Me ctive

es
Ini ts
tiv
rge
je

tia
“Tosucceedfinancially, how

Ob
does ITcontributetothe
organizations success?”

Customer Internal Business Process


V isionn

as s
VVisio

Ta ures
Me ctive

as s
es

Ta ures
“Toachieveour “Tosatisfyour

Me ctive

es
Ini ts
ision

tiv

Ini ts
tiv
rge
je

rge
an dd

tia
vision, how shareholders and

je

tia
an
Ob

Ob
shouldITsupport and customers, what
our customers?” sstrate
Stra te gy
teg
trategygy IT processes
must weexcel at?”

LearningandGrowth

as s
Ta ures
Me ctive

es
Ini ts
tiv
rge
je

tia
Ob
“Toachieveour vision,
howwill wesustainour
abilitytochangeimprove
our ITenvironment?”

IT management is increasingly applying the balanced


scorecard method (Kaplan, Norton) and performance
dashboards to the measurement of the value and overall
contribution delivered by IT. In this method, alignment is
achieved between business and IT plans through visible
alignment of IT goal (outcome) measures of IT processes
and the business goals they support. They also identify the
key performance indicators (drivers) of success for each
process.
Management implements a set of measurement and
monitoring activities to collect information on the
achievement of the outcomes using the goal measures and
on the performance of IT processes using the key

15
IT Monitoring

performance indicators. This information, and the


correlation of results over time, enable IT management to
determine whether its IT strategies and approaches are
effective and to decide on corrective or adjusting actions as
appropriate.
It is vital to note that the linkage between the business
balanced scorecard and the IT balanced scorecard is a strong
method of alignment. Many of the outcome measures of IT
influence how well the enterprise is doing and are, therefore,
performance measures for the enterprise. It is equally vital
to stress that the balanced scorecard should demonstrate the
value that IT delivers to the enterprise.
Benchmarking for decision making about IT investments
for risk and control
Maturity modeling and benchmarking are other
management practices for monitoring the return of
investment and risk mitigation of IT. Maturity models
provide for measurable, recognizable levels of maturity, for
example, in control maturity, risk management, operational
proficiency, etc. On these scales, an enterprise can define
where it is (As-Is) and where it wants to be (To-Be). This
can then drive strategy and help decision making on
improvement projects. Monitoring these improvements and
regularly reassessing the enterprise’s maturity level through
benchmarking (i.e., comparing it to where others are) are
becoming best practice in management monitoring in the
domain of IT.
Active monitoring of the IT infrastructure
Concern for IT infrastructure risks at enterprise and
national/international levels has changed how IT security
and risk are being managed. Awareness that the traditional
approach of defining policy, selecting safeguards and
implementing them is too static for a highly volatile
environment has pushed organizations into a more fluid
continuous approach of actively monitoring the IT
infrastructure. This consists of continuously monitoring and
performing self-assessments to detect and then fix the
problems identified. Results of this monitoring activity need
to be brought to top management’s attention when

16
What IT Monitoring Tools are Available?

appropriate, and management needs to give guidance on


when notifying management is appropriate and when not.
Such systems will detect and stop unauthorized activity in
systems and networks on a 24/7 basis through constant
information gathering and analysis, looking for attack
signatures, viruses, vulnerabilities, non-compliance with
basic rules and misuse.
These systems need to be supplemented with appropriate
response procedures and, thus, provide for assurance about
the security of the IT infrastructure. They usually are
complemented with intrusion detection exercises, testing the
controls of the infrastructure and resulting in exception
reports to management.
Brainstorming for risk management and improvements
Apart from the above techniques, for the most part, only
high-risk enterprises will implement processes and assign
responsibilities to monitor enterprise risks on a continuous
basis. A cost-effective alternative exists in well-prepared
and facilitated risk brainstorming sessions between top
management and those responsible for IT, security, risk and
audit. These brainstorming sessions might be performed, for
example, on an annual basis and the professionals involved
should prepare and document a list of the most important
vulnerabilities and threats for consideration.
These brainstorming sessions should result in clear
improvement actions and responsibility commitments.
Executive management and boards can then follow up with
traffic light reports (see above). Appendix 2 provides
additional guidance on security and risk monitoring that
should be useful in these brainstorming sessions.
Internal and external audit for independent assurance
Internal and external audit reports are key monitoring tools
for executive management. These reports should include a
statement of the audit objectives, a description of the audit
scope and methodology, the period of coverage and the
nature and extent of the audit work performed. The report
should include a full discussion of the audit findings and
conclusions, the cause of the problem areas noted in the

17
IT Monitoring

audit and recommendations for actions to correct the


problem areas and improve operations. The report should
include a statement that the audit was made in accordance
with generally accepted auditing standards and disclose,
when applicable, standards that were not followed. It should
include the pertinent views of responsible officials of the
organization, program, activity or function audited
concerning the auditors' findings, conclusions and
recommendations, and what corrective action is planned.
Although resolution of audit comments rests with executive
management, follow-up by audit staff is a continuous
process to determine if promised corrective actions actually
have been implemented. Auditors should update information
on previous findings, conclusions and recommendations to
determine whether appropriate actions are being
implemented in a timely manner.
Management reporting
Management reporting is an essential element of IT
governance. Executive management should receive, for
review, reports on the organization’s progress toward
identified goals. It should also receive status reports on the
extent to which planned objectives have been achieved,
deliverables obtained, performance targets met and risks
mitigated. Once the reports have been reviewed,
management should ensure that any required remedial
actions are taken in a timely manner.

18
What is the Best Approach to Implement IT Monitoring?

What is the Best Approach to Implement IT Monitoring?


30. Although monitoring of IT is unique to the organizational
environment, the monitoring process and the underlying
activities are similar. Usually, the process consists of the six
phases described below.

APPROACH - Phase I: Orientation


This start-up phase is required to determine the scope of
monitoring and the methodology and techniques to be applied. In
this phase the resources required for monitoring IT are
mobilized.

Determine scope: The scope of IT monitoring normally includes


the following key activities:
determining if monitoring incorporates all business units or
that separate monitoring activities will be developed for
selected business units;
assessing the quality of the IT monitoring performance
objectives set for each business unit; and
evaluating the extent of independent assurance involvement
in IT monitoring.
At the end of this step, the scope for IT monitoring will have
been determined.

Establish methodology/techniques and mobilize resources: IT


monitoring can be a time-consuming process depending on the
size of the organization and the scale of its current or desired IT
dependence. Once the scope has been determined, the
methodology and techniques need to be established, and the
background information and resources necessary for the
planning effort need to be mobilized, including a clear
delineation of reporting lines. Key activities include:
Gathering necessary background information on the
organization, its IT profile and capabilities and IT
monitoring activities.
Selecting a proven methodology to support the IT
monitoring activities. This methodology may be provided

19
IT Monitoring

by external consultants, internally developed or acquired


from a third party.
Determining techniques that will be used for collecting and
analyzing information, including IT monitoring tools (e.g.,
performance measurement, balanced scorecard,
benchmarking).
Establishing an IT monitoring project team. Typically, this
will be a multidisciplinary team, comprising persons with
both IT and business skills. Management should ensure that
the team possesses the technical competence, and skills and
knowledge necessary to perform the needed analyses in an
efficient, effective and economical manner. Frequently, the
team is supplemented by external consultants with expertise
in IT monitoring.
Formalizing the reporting mechanism for the project team.
Generally, the team reports to a steering committee headed
by the chief executive officer, chief information officer or
another senior business executive and comprising key
business unit managers, the IT manager and an information
systems audit manager.

APPROACH - Phase II: Criteria Definition


This phase is regularly covered in the planning/design phase of
each IT and business process. Goals or performance
measurement indicators are defined and established for IT
monitoring. Periodic re-evaluation of the performance measures
must also be a standard part of the criteria definition phase, as
these will change over time and they must be updated to
incorporate the changes.

For the IT and internal control processes, management should


ensure that relevant performance indicators (e.g., benchmarks)
from both internal and external sources are being defined, and
that data are being collected for the creation of management
information reports and exception reports regarding those
indicators. Controls also should be aimed at validating the
propriety and integrity of both organizational and individual
performance measures and indicators.

20
What is the Best Approach to Implement IT Monitoring?

APPROACH - Phase III: Ongoing Monitoring


Ongoing monitoring is a continuous supervisory function over
key IT activities and control processes. Exceptional events have
to be identified and tracked. Performance measures need to be
established, involving both IT and the stakeholders, aligned with
the strategy and reviewed on an ongoing basis.

IT services should be measured (key performance indicators


and/or critical success factors) by management and be compared
with target levels. Independent assessments and evaluations of
the IT should be performed on a continuous basis to ensure IT’s
continued effectiveness.

APPROACH - Phase IV: Separate Periodic and Ad Hoc


Monitoring
Besides ongoing monitoring, separate periodic and ad hoc
monitoring is essential to (1) ensure that ongoing monitoring and
other control functions operate properly, (2) periodically review
IT related risks and opportunities and (3) obtain comfort relative
to major IT decisions. Periodic monitoring includes internal
audit procedures, external assurance, self-assessments and
brainstorming sessions.

At regular intervals, management should measure customer


satisfaction regarding the IT services delivered to identify
shortfalls in service levels and establish improvement objectives.
Independent assessments and evaluations of IT processes should
be performed on a routine cycle to ensure IT’s continued
effectiveness.

APPROACH - Phase V: Subsequent Actions


Subsequent actions include corrective actions to redirect IT
activities and processes and bring them back in line with goals,
strategy and policy; minimization of adverse effects; refinement
of goals and measures; changes to strategy, policy and standards;
and initiation of reassessment activities.

21
IT Monitoring

Appropriate management action should be initiated to correct


deficiencies and to ensure that effective corrective actions are
taken in a timely manner.

APPROACH - Phase VI: Reporting


If monitoring is to support effective IT governance, management
reporting about all phases of the monitoring process (including
subsequent actions and procedures to alert top management) is
an essential element of recurring/iterative control cycles.

Executive management should receive, for review, reports on the


organization’s progress toward identified goals. Management
should also receive status reports on the extent to which planned
objectives have been achieved, deliverables obtained,
performance targets met and risks mitigated.

WHEN?
31. Monitoring is necessary always whenever IT is used within an
organization: from planning and organization, acquisition and
implementation to delivery and support. Monitoring is crucial for
effective IT governance and occurs in a planned manner
whenever goals are being verified but also in a continuous and
ad hoc fashion when monitoring for risks, faults or defects.

WHO?
32. Chief executive officers, chief information officers, other
executive management, process owners, users and information
systems auditors all have roles and responsibilities in monitoring
IT’s goals and processes. An effective monitoring system, like
any successful management information system, involves the
whole organization in that monitoring information is captured,
consolidated and reported up, at all levels.

22
M1 – Monitoring the processes

CONTROL OBJECTIVES FOR THE IT


MONITORING PROCESS DOMAIN

M1 – Monitoring the processes

High level control objective


Control over the IT process of
monitoring the processes
that satisfies the business requirement
to ensure the achievement of the performance objectives set for
the IT
processes
is enabled by
the definition of relevant performance indicators, the
systematic and
timely reporting of performance and prompt acting on
deviations
and takes into consideration
• scorecards with performance drivers and
outcome measures;
• customer satisfaction assessments;
• management reporting;
• knowledge base of historical
performance;
• external benchmarking.

Detailed control objectives

1. Collecting Monitoring Data


For the IT and internal control processes, management should ensure that
relevant performance indicators (e.g., benchmarks) from both internal and
external sources are being defined, and that data are being collected for
the creation of management information reports and exception reports
regarding those indicators. Controls also should be aimed at validating the
propriety and integrity of both organizational and individual performance
measures and indicators.

23
IT Monitoring

2. Assessing Performance
Services to be delivered by the IT function should be measured (key
performance indicators and/or critical success factors) by management
and be compared with target levels. Assessments of the IT function should
be performed on a continuous basis.

3. Assessing Customer Satisfaction


At regular intervals, management should measure customer satisfaction
regarding the services delivered by the IT function to identify shortfalls in
service levels and establish improvement objectives.

4. Management Reporting
Executive management should receive, for review, reports on the
organization’s progress toward identified goals. Management should also
receive status reports on the extent to which planned objectives have been
achieved, deliverables obtained, performance targets met and risks
mitigated. Once management has reviewed the reports, it should take
whatever appropriate action is deemed necessary.

Management Guidelines
Critical Success Factors
Useful, accurate and timely management reports are available.
Processes have defined and understandable key goal indicators and
key performance indicators.
Measurements of IT performance include financial, operational,
customer and organizational learning criteria that ensure alignment
with organization-wide goals and that can be integrated with tools
such as the IT balanced business scorecard.
There are clearly understood and communicated process objectives.
A framework is established for defining and implementing IT
governance reporting requirements.
A knowledge base of historical performance is established.

24
M1 – Monitoring the processes

Key Goal Indicators


Consistent application of the correct limited number of performance
indicators.
Increased number of process improvement opportunities detected and
acted on.
Satisfaction of management and the governance entity with
performance reporting.
Reduced number of outstanding process deficiencies.

Key Performance Indicators


Time lag between the process deficiency occurrence and reporting.
Time lag between the reporting of a deficiency and action initiated.
Ratio between process deficiencies reported and deficiencies
subsequently accepted as requiring management attention follow-up
(noise index).
Number of processes monitored.
Number of cause and effect relations identified and incorporated in
monitoring.
Number of external benchmarks of process effectiveness.
Time lag between business changes and any associated changes to
performance indicators.
Number of changes to the set of performance indicators without the
business goals changing.

25
IT Monitoring

M2 – Assessing internal control adequacy

High level control objective


Control over the IT process of
assessing internal control adequacy
that satisfies the business requirement
to ensure the achievement of the internal control objectives set
for the
IT processes
is enabled by
the commitment to monitoring internal controls,
assessing their
effectiveness and reporting on them on a regular basis
and takes into consideration
• responsibilities for internal control;
• ongoing internal control monitoring;
• benchmarks;
• error and exception reporting;
• self-assessments;
• management reporting;
• compliance with legal and regulatory
requirements.

Detailed control objectives

1. Internal Control Monitoring


Management should monitor the effectiveness of internal controls in the
normal course of operations through management and supervisory
activities, comparisons, reconciliations and other routine actions.
Deviations should trigger analysis and corrective action. In addition,
deviations should be communicated to the individual responsible for the
function and also to at least one level of management above that
individual. Serious deviations should be reported to senior management.

2. Timely Operation of Internal Controls


Reliance on internal controls requires that controls operate promptly to
highlight errors and inconsistencies, and that these are corrected before

26
M2 – Assessing internal control adequacy

they have an impact on production and delivery. Information regarding


errors, inconsistencies and exceptions should be kept and systematically
reported to management.

3. Internal Control Level Reporting


Management should report information on internal control levels and
exceptions to the affected parties to ensure the continued effectiveness of
its internal control system. Actions should be taken to identify what
information is needed at a particular level of decision making.

4. Operational Security and Internal Control Assurance


Operational security and internal control assurance should be established
and periodically repeated, with self-assessment or independent audit to
examine whether or not the security and internal controls are operating
according to the stated or implied security and internal control
requirements. Ongoing monitoring activities by management should look
for vulnerabilities and security problems.

Management Guidelines

Critical Success Factors


Management clearly defines what components of the processes need
to be controlled.
Internal control, compliance and internal audit responsibilities are
clearly understood.
Competence and authority of the internal control compliance function
exist, addressing delegation as appropriate.
A properly defined IT control process framework is in place.
A clear process is used for timely reporting of internal control
deficiencies.
Internal control monitoring data are accurate, complete and timely.
There is management commitment to act on internal control
deficiencies.
There is alignment with risk assessment and security processes.

27
IT Monitoring

A process is in place to support knowledge sharing on internal control


incidents and solutions.

Key Goal Indicators


Index of senior management satisfaction and comfort with reporting
on internal control monitoring.
Decreased probability of internal control incidents.
Positive external qualification and certification reports.
Number of control improvement initiatives.
Absence of regulatory or legal non-compliance events.
Decreased number of security incidents and quality defects.

Key Performance Indicators


Number and coverage of control self-assessments.
Timeliness between internal control deficiency occurrence and
reporting.
Number, frequency and coverage of internal compliance reports.
Number of timely actions on internal control issues.
Number of control improvements stemming from root cause analysis.

28
M3 – Obtaining independent assurance

M3 – Obtaining independent assurance

High level control objective


Control over the IT process of
obtaining independent assurance
that satisfies the business requirement
to increase confidence and trust among the organization,
customers
and third-party providers
is enabled by
independent assurance reviews carried out at regular
intervals
and takes into consideration
• independent certifications and
accreditation;
• independent effectiveness evaluations;
• independent assurance of compliance
with laws and regulatory requirements;
• independent assurance of compliance
with contractual commitments;
• third-party service provider reviews and
benchmarking;
• performance of assurance reviews by
qualified personnel;
• proactive audit involvement.

Detailed control objectives


1. Independent Security and Internal Control Certification/Accreditation
of IT Services
Management should obtain independent certification/accreditation of
security and internal controls prior to implementing critical new IT
services and re-certification/re-accreditation of these services on a routine
cycle after implementation.

29
IT Monitoring

2. Independent Security and Internal Control Certification/Accreditation


of Third-Party Service Providers
Management should obtain independent certification/accreditation of
security and internal controls prior to using IT service provides and re-
certification/re-accreditation on a routine cycle.

3. Independent Effectiveness Evaluation of IT Services


Management should obtain independent evaluation of the effectiveness of
IT services on a routine cycle.

4. Independent Effectiveness Evaluation of Third-Party Service Providers


Management should obtain independent evaluation of the effectiveness of
IT service providers on a routine cycle.

5. Independent Assurance of Compliance with Laws and Regulatory


Requirements and Contractual Commitments
Management should obtain independent assurance of the IT function’s
compliance with legal and regulatory requirements and contractual
commitments on a routine cycle.

6. Independent Assurance of Compliance with Laws and Regulatory


Requirements and Contractual Commitments by Third-Party Service
Providers
Management should obtain independent assurance of third-party service
providers’ compliance with legal and regulatory requirements and
contractual commitments on a routine cycle.

7. Competence of Independent Assurance Function


Management should ensure that the independent assurance function
possesses the technical competence, skills and knowledge necessary to
perform such reviews in an effective, efficient and economical manner.

8. Proactive Audit Involvement


IT management should seek audit involvement in a proactive manner
before finalizing IT service solutions.

30
M3 – Obtaining independent assurance

Management Guidelines

Critical Success Factors


There is continuous alignment with stakeholder needs.
The organization has defined processes for IT assurance activities,
especially overall internal control, certification and major decisions.
Benchmarking of external service providers is routinely performed.
Major IT decisions have an up-front requirements analysis for a third-
party assurance opinion.
Prior to obtaining independent assurance, a high-level risk assessment
is performed with the key stakeholders.
There is a commitment to leverage independent assurance for
sustainable improvement.
Assurance activities are performed in accordance with generally
accepted practices, such as SysTrust.
There is a partnership between auditor and auditee, to encourage
cooperation.

Key Goal Indicators


Increased number of accepted opinions on the overall system of
internal control for all agreed domains.
Increased number of quality certifications or accreditations for all
agreed domains.
Increased number of second opinions reported to the stakeholders for
major IT decisions such as going live, contract negotiations, joint
ventures and major acquisitions.
Percentage of recommendations closed on time relative to
independent internal control reviews, quality certifications or
accreditations and second opinions.
Reduced number of failed or reversed major IT decisions.
Index of confidence and trust of stakeholders.

31
IT Monitoring

Key Performance Indicators


Reduced overhead of obtaining assurance and certifications.
Timeliness of assurance reporting.
Timeliness of assurance activities.
Number of assurance processes initiated.
Number of iterations before assurance reports are accepted.
Number of IT decisions requiring assurance where no assurance was
sought.
Number of IT decisions not requiring assurance where assurance was
sought.
Reduced number of failed or reversed major IT decisions after a
positive assurance was obtained.

32
Security and Risk Monitoring

Security and Risk Monitoring

Information systems are subject to a wide range of disruptive incidents of


varying degrees of intensity. The business processes that rely on these
systems, and the environment in which both these systems and processes
operate, also are continually subject to change and new risks.

Preventive measures may not always be feasible or cost-effective to


minimize loss, disclosure, damage or disruption. Hence, monitoring
measures need to be established to detect and ensure correction of risk and
security breaches, so that all actual and suspected breaches and risk
exposures are promptly identified, investigated and acted on. This also
will ensure ongoing compliance with policy, standards and minimum
acceptable security and risk practices.

The immediate benefit of instituting risk monitoring measures and


procedures over systems, processes and their environment is to identify
issues promptly, contain damage and expedite recovery. The most
important consequential benefit is that it increases the ability to prevent
future damage and inconvenience, while increasing the predictability of
actions involving failures, risk exposures or breaches of security. An
associated benefit is the deterrence value of effective monitoring
processes.

Actions that may result from monitoring practices are:


disciplinary or corrective actions;
minimization and recovery of losses;
refinement of security levels;
changes to policy or standards;
changes to design and implementation of security and risk
management processes;
initiation of reassessment programs, including root cause and pattern
analysis;
initiation of intelligent monitoring systems with interactive feedback;
and
initiation of network or system penetration studies.

33
IT Monitoring

Follow-up of security and other risks is as important as its


implementation, especially in the light of new technological
developments, whether those adopted by the system owner or those
available for use by others. Issues that need to be addressed in achieving
effective monitoring include:
the appointment of a responsible manager with adequate tools and
resources;
the performance of independent and objective assessments of security
controls such as those provided by security audits;
the establishment of clear and expedient investigative procedures;
the massive amount of management audit trail information from a
large variety of system components that may need to be examined;
the timeliness of processes to alert management when electronic
transactions are practically instantaneous; and
the dynamic and ever-changing business and information systems
environment.

It is management’s responsibility to ensure that such processes and


associated responsibilities are embedded into the organization with clear
objectives and accountabilities. Subsequently, management should
monitor whether the processes function well and whether the results of
those processes are appropriate and acted on.

34
Additional IT Measures to Monitor

Additional IT Measures to Monitor

Key Goal Indicators (or outcome measures)


Enhanced performance and cost management
Improved return on major IT investments
Improved time to market
Increased quality, innovation and risk management
Benchmarking comparisons of IT’s return on investment, unit cost,
etc.
Creation of new service delivery channels
Increased level of service delivery
Absence of integrity and confidentiality risks
Cost efficiency of processes and operations
Confirmation of reliability and effectiveness
Number of timely changes to processes and systems
Enhanced performance and cost management
Measurable contribution from IT to fast introduction of innovative
products and services
Reaching new and satisfying existing customers
Meeting stakeholder requirements and expectations on budget and on
time
Adherence to laws, regulations, industry standards and contractual
commitments
Transparency on risk taking and adherence to the agreed
organizational risk profile
Business cases that demonstrate a high potential return on investment.
Absence of integrity and confidentiality risks
Availability of appropriate bandwidth, computing power and IT
delivery mechanisms
Confirmation of reliability and effectiveness

35
IT Monitoring

Deviation between estimated and actual costs


Improved productivity (e.g., delivery of value per employee, number
of customers and cost per customer served)
Cost efficiency of processes and operations

Key Performance Indicators (performance drivers)


Improved cost-efficiency of IT processes (costs vs. deliverables)
Increased number of IT action plans for process improvement
initiatives
Increased utilization of IT infrastructure
Increased availability of knowledge and information for managing the
enterprise
Improved performance as measured by IT balanced scorecards
System downtime
Throughput and response times
Amount of errors and rework
Number of staff trained in new technology and customer service
skills
Benchmark comparisons for operational excellence, best practice, etc.
Number of non-compliance reportings
Reduction in development and processing time
Increased number of enterprise transformation projects enabled by IT
Increased satisfaction of IT users and stakeholders (surveys and
number of complaints)

36