Pics from : http://www.pragroup.ca/Services/InformationTechnology/tabid/70/Default.

aspx

Evaluation of Information System

Asset Safeguarding and Data Integrity

Effectiveness and Efficiency
Drs. Haryono, Ak. M.Com
& Dimas M. Widiantoro, SE., S.Kom., M.Sc.
 Intorduction
• Lets strat from the video

• http://www.youtube.com/user/Networking4a
ll?v=iRoenMHx6LQ&feature=pyv&ad=791097
6668&kw=network
 Definition
• Explicitly or implicitly, safeguarding assets is an
inescapable fiduciary obligation bestowed on
managers; whether the entity exists for-profit or
not-for-profit.
– Fiduciary duties are an inherent managerial
responsibility correlated to accountability that can be
conveyed through legislation, regulation, or
expectation.
– Foundationally, an operating entity’s very existence is
usually heavily dependent on how well employees
safeguard assets utilized in fulfilling the organizational
mission.
 Assumption
• Assumption for safeguarding assets should
span the entity’s total tangible and intangible
resources. Specifically, information and
associated technologies are assets requiring
appropriate investments in protective
measures to retain intrinsic value.
 Requirement
• Safeguarding IT resources usually requires an
information security governance (ISG) framework
rendering essential information asset coverage.
• An entity’s management can adopt the
– Information Systems Audit and Control Association’s
(ISACA’s) Control Objectives for Information and
related Technology (COBIT) framework, promulgated
by the Information Technology Governance Institute,
to ensure adequate ISG and/or the International
Organization for Standardization (ISO) 27002
methodology.
Control Objectives for Information and
Related Technology.

COBIT FRAMEWORK
IT Resources From COBIT
The Flow Chart of Information Syste,
The Structure of The Audit
The Flow Chart of IS Audit from COBIT
 Information, Application, and
Infrastructure
• Information encompasses utilizable objects,
structured and non-structured data, and
presentation formats.
• Applications are deemed the sum of manual
and programmed procedures.
• Whereas, the infrastructure is defined as
hardware, operating systems, configuration
systems, facilities, and support structure.
The Cube
COBIT AUDIT STEPS
Acquisition and Implementation
Delivery and Support
Monitoring
 IT gov. = indispensable
• With IT considered indispensable for providing
processing efficiencies, communication
expediency and information reliability, entities
should govern safeguarding information
assets through an ISG program. To accomplish
this security necessity, management normally
needs a governance framework enabling
organizational alignment, adequate resource
allotments, risk management, value delivery
and performance measurement.
 Governance - subset
• Whether information security governance is
abstractively viewed as a distinct governance
classification supporting entity governance or
a subset of information technology
governance, safeguarding IT normally
mandates addressing responsibilities
separation and
‘protection-of-information-assets’ to assure
managerial due diligence.
Example
Control Environment Consideration
Control Environment Consideration
Information and Communication
Risk Assessment Consideration
Monitoring Consideration
Monitoring Consideration
 Evaluating information system
effectiveness
and efficiency
•SECTION ONE - Why study effectiveness?

•Problems have arisen or criticisms have been voiced in connection with

a system;

•Some indicators of the ineffectiveness of the hardware and software

being used may prompt the review;,

•Management may wish to implement a system initially developed in one

division throughout the organization, but may want to first establish its

effectiveness;

•Post-implementations review to determines whether new system is

meeting its objectives.

Indicators of System Ineffectiveness

excessive down
time and idle time

slow system
data loss
response time

excessive
slow system
maintenance
response time
costs

inability to
unreliable system interface with new
outputs hardware/softwar
e
Two approaches to measurement of
system
effectiveness
Goal-centered view - does • Conflicts as to priorities, timing etc. can lead to objectives
met
system achieve goals set in the short run by sacrificing fundamental system qualities,
out? leading to long run decline of effectiveness of the system

System resource view - • If the qualities exist, then information system objectives, by
desirable qualities of a inference, should be met. By measuring the qualities of the
system system
are identified and their levels may get a better, longer-term view of a system's
are measured. effectiveness.

The main problem–

measuring system qualities is
much
more difficult than
measuring goal
achievement.
2 Types of Eval'ns for Sys.
Effectiveness
Relative evaluation - auditor compares the
state of goal
accomplish. after the system implemented,
with the state
of goal accomplishment before system
implemented.
• Improved task accomplishment, and
• Improved quality of working life.
Absolute evaluation - the auditor assesses the
size of the
goal accomplish. after the system has been
implemented.
• Operational effectiveness,
• Technical effectiveness, and
• Economic effectiveness.
Task Accomplishment - an effective I/S
improves the task accomp. of its users.

Performance measures for task

Providing specific measures of
past accomplishment that
auditor can use to evaluate IS is
difficult.
accomplishment differ
across applications and
sometimes across organizations.
•For a manufacturing control system
might be:
•number of units output,
•number of defective units reworked,
units scrapped
•amount of down time/idle time.
Important to trace task
accomplishment over time.
System may appear to have
improved for a short time
after implementation, but fall
into disarray thereafter.
Quality of Working Life

High quality of working life for users of a system is a

major objective in the design process. Unfortunately,
there is less agreement on the definition and
measurement of the concept of quality of working life.

Different groups have different vested interests - some

productivity, some social

Major advantages - relatively objective, verifiable, and

difficult to manipulate. Data required is relatively easy to
obtain.

Major disadvantages - it is difficult to link them to IS quality

and difficult to pinpoint what corrective action is needed
Operational Effectiveness Objectives

Auditor
examines how
well a system • Frequency of use,
meets its goals
from the • Nature of use,
viewpoint of a
user who • Ease of use, and
interacts with
the
system on a • User satisfaction.
regular basis
Frequency and Nature of Use

• sometimes a high quality system

leads to low frequency of use
Frequency - because the system permits more
employed work to be accomplished in a
widely, but shorter period of time.
• sometimes a poor quality system
problematic leads to a low frequency of use
since users dislike the system

Nature - can • lowest level: treat as black box

use systems providing solutions to the
in many • highest level: use to redefine how
tasks, jobs performed and viewed
ways
Ease of Use and User Satisfaction
Ease of use - positive
correlation betw. users'
feelings about
• Terminal
systems and the degree
to which the systems
location,
were easy
to use. In evaluating
ease of use, it is
flexibility of
important to
identify the primary and
secondary users of a
reporting, ease of
system.
error correction

User satisfaction -
has become an
important measure
of
• problem finding,
operational
effectiveness problem
because of the
difficulties and
problems
solving, input,
associated with
measures of
processing,
frequency of use,
nature of use, and report form
ease of use.
Technical Effectiveness Objectives -

•Has the appropriate hardware and software technology

been used to support a system, or, whether a change in

the support hardware or software technology would

enable the system to meet its goals better.

•Hardware performance can be measured using hardware monitors

or more gross measures such as system response time, down time.

•Software effectiveness can be measured by examining

the history of program maintenance, modification and

run time resource consumption. The history of program

repair maintenance indicates the quality of logic existing

in a program; i.e., extensive error correction implies:

inappropriate design, coding or testing; failure to use

structured approaches, etc.

Economic Effectiveness Objectives -

• Requires the identification of costs and benefits and the

proper evaluation of costs and benefits - a difficult task
since
costs and benefits depend on the nature of the IS.
•For example, some of the benefits expected and derived from an
IS
designed to support a social service environment
would differ significantly from a system designed to
support manufacturing activities. Some of the most
significant costs and benefits may be intangible and
difficult to identify, and next to impossible to value.
SECTION TWO - Evaluating system
efficiency
Why would an auditor get involved in a
study of system
efficiency?
• evaluate an existing operational system to determine
whether its performance can be improved;
• evaluate alternate systems that the installation is
considering purchasing or leasing. For example,
management may be considering two systems with
different database management approaches.

To determine whether a system is

efficient, the auditor
will need to identify:
• an appropriate performance index to assess system
efficiency.
• an appropriate workload model to measure the
system's
performance in the context of that workload.
Performance Indices

Measure system efficiency;

quantitatively how well system Expressed in terms of
achieves an efficiency criterion. Have
several functions:
Expressed using ranges or workload - e.g., response time
• allow users to decide whether a system
probability distributions - avg. of an interactive system will
will meet needs, may be deceiving (look at vary depending on the
• permit comparison of alternate response time variations) number and the nature of the
systems, and jobs in the system.
• show whether changes to the
hardware/software configuration of
system have produced the desired
effect.
Indices - Timeliness

How quickly a system For a batch system, typically

is turnaround time - the
is able to provide length of time between
submission of a job and
users with the receipt
output they require. of the complete output.
For interactive systems, the
response time - the length
of time between submission
of an input transaction to
the system and receipt of the
first character of output.

Must be defined in
terms of a unit of In an interactive system it
In a batch system the unit of may be a job consisting of
work and the priority work usually is a job. multiple transactions, or a
categorization given single transaction.

to the unit of work.

Indices - Throughput & Utilization
Throughput indices measure how much work
is
done by the system over a period of time.
• Throughput rate of a system is the amount of work done
per unit of time.
• The system capability is the maximum achievable
throughput rate.
• Throughput indices must be defined in terms of some
unit of work: a job, a task, or an instruction.
• The more responsive a system, the greater its throughput.

Utilization indices measure the proportion of

time a
system resource is busy.
• For example, the CPU utilization index is calculated by
dividing the amount of time the CPU is busy by the
total amount of time the system is running.
Workload

• Using the real workload of the system

for evaluation may be
too costly and too disruptive.
• To measure efficiency for a
representative workload, the
time period for evaluation may be too
long.
• Also, the real workload cannot be
used if the system to be evaluated is
not operational.

A system's workload is the

set of resource demands Need a workload model
imposed upon the system representative of the real
resources by the set of jobs workload
during a given time period.
Workload Models
• In a time subset, the content of the
Natural workload models, or workload model is the same as the
benchmarks, are real workload, but the time interval
constructed by taking some subset of for performance indices is smaller
the real workload. than the interval for the real
workload.
Artificial workload models not
constructed from jobs in the
real workload; useful when system
unable to process the
natural workload

Natural - more representative and less

costly to construct

Artificial - more flexible and more

compact
SECTION 3- Comparison of 3 Audit
Approaches - Objectives

Effectiveness audit - express an Audits of system efficiency -

F/S audit - express an opinion as to
opinion on whether a whether maximum output is
whether financial
system achieves the goals set for achieved at minimum cost or with
statements are in accordance with
the system. These minimum input
GAAP
goals may be quite broad or specific. assuming a given level of quality.
Comparison of 3 Approaches - Planning

F/S audit - part is identifying controls upon which the

auditor could rely and reduce other audit verification
procedures; or, id controls upon which the auditor is forced
to rely

Effectiveness audit - id goals, measures for determining

whether the goals obtained during a specific period,if
explicit measures are more straight-forward; however,
when
broad and multi-dimensional, the auditor may need to
develop relevant measures and indicators of achievement.

Audits of system efficiency - often comparable to a scientific

experiment. A scheme for obtaining measurements must
be
developed explicitly for the performance index defined. For
example, if average turnaround time is used as a measure of
efficiency, then the experimental task must control for
various job sizes, time of day, etc.
Comparison of 3 Approaches -
Execution

F/S audit - controls analysis and CAATs

Effectiveness - Once the system goals have been

identified, measures of goal achievement have been
selected, and the population to be studied has been
identified, it is necessary to actually obtain measures of
goal achievement and analyze the results.

Efficiency - During the execution phase the benchmark

or workload model test is actually run and the result are
subjected to analysis. Care must be taken to
control for interference by factors other than those built
into the model. And measurements must be taken
carefully.
Comparison of 3 Approaches -
Reporting

F/S audit - letter re I/C deficiencies

Effectiveness - the analysis will likely highlight areas of

successful attainment of objectives as well as failures.
Explanations of the causes of significant successes and
failures should be sought out and included in the report.

Efficiency - reports of studies of system efficiency must

typically contain specific recommendations identifying
ways in which the identified inefficiencies can be
eliminated.