You are on page 1of 91

Incident Response Plan

A technical guide to aid in preparing for, detecting and


responding to computer security incidents

July 2004
Incident Response Plan

Preface

This Incident Response Plan was developed in consultation with WA Government


agencies, and both national and state organisations. Feedback received has been
incorporated into the document. It is a technical guide to aid in preparing for, detecting
and responding to computer security incidents. It has been written to document the
steps agencies may take to effectively and efficiently prepare for and manage computer
security incidents should they occur. The style and language of this document is in the
‘second person’, and this follows the standard approach taken by other respected
incident response-type manuals.

This Incident Response Plan is a higher level companion document to the Forensic
Plan, which provides lower-level technical guidance in understanding how digital
evidence collected as part of an incident investigation may be preserved such that it
may be admissible in a court of law.

Both the Incident Response Plan and the Forensic Plan are available from the Office of
e-Government website (http://www.egov.dpc.wa.gov.au) in PDF format from the Policies
and Guidelines section.

If you have any comments, queries or need additional information, please contact:

The Office of e-Government,


Department of the Premier and Cabinet,
Locked Bag 10,
Cloisters Square,
PERTH WA 6850

email: egov@dpc.wa.gov.au
Phone: 9213 7100

Contact:

Sven Bluemmel
Gail Holt

Department of the Premier and Cabinet – Office of e-Government 2


Incident Response Plan

Table of Contents
PREFACE ...................................................................................................................................... 2
TABLE OF CONTENTS ................................................................................................................... 3
Background ..........................................................7
INTRODUCTION ............................................................................................................................ 8
1.1 Disclaimer ............................................................................................................... 8
1.2 Introduction............................................................................................................. 8
1.3. Purpose and Scope .................................................................................................. 9
1.4 Related Documents ............................................................................................... 11
Incident Response Plan.......................................... 12
PHASES OF THE INCIDENT RESPONSE PLAN ................................................................................ 13
PREPARATION ............................................................................................................................ 14
3.1 Establish policies and procedures for responding to incidents ............................. 14
3.1.1 Information disclosure policy ........................................................................... 15
3.1.2 Networked systems security policy .................................................................. 16
3.1.3 Configuration redundancy policy ..................................................................... 18
3.1.4 Presumption of privacy policy .......................................................................... 19
3.1.5 Document a response procedure that implements your policies....................... 19
3.1.6 Conduct a legal review of your policies and procedures .................................. 19
3.1.7 Train designated staff about your response policies and procedures................ 20
3.2 Use Proactive Techniques to Prevent Incidents.................................................... 20
3.2.1 Subscribe to resources for notification of new threats and vulnerabilities ....... 21
3.2.2 Establish standard build procedures.................................................................. 21
3.2.3 Patch known system vulnerabilities.................................................................. 22
3.2.4 Perform network auditing and penetration testing............................................ 22
3.2.5 Display warning banners................................................................................... 22
3.2.6 Ensure that systems run antivirus with current definitions............................... 23
3.2.7 email client configuration ................................................................................. 23
3.2.8 Keep all network information current............................................................... 23
3.2.9 Improve technical security skills ...................................................................... 23
3.2.10 Establish automated intrusion detection ........................................................... 24
3.2.11 Know who has access to your systems ............................................................. 24
3.2.12 Keep track of remote computing users ............................................................. 24
3.2.13 Establish extranet agreements and monitoring ................................................. 24
3.2.14 Identify critical assets and servers .................................................................... 24
3.2.15 Review the physical security of critical assets and servers............................... 25
3.2.16 Use strong account policies .............................................................................. 25
3.2.17 Domain and network information current in whois databases.......................... 25
3.2.18 Situational awareness........................................................................................ 26
3.3 Form Your Incident Response Capability............................................................. 26
3.3.1 Create an incident response plan....................................................................... 26
3.3.2 Authority to act ................................................................................................. 26

Department of the Premier and Cabinet – Office of e-Government 3


Incident Response Plan

3.3.3 Pick the team..................................................................................................... 26


3.3.4 Establish a primary point of contact ................................................................. 27
3.3.5 Establish visibility and a compensation plan for the team................................ 27
3.3.6 Decide on the scope of operation...................................................................... 27
3.3.7 Prioritise your incident response procedures .................................................... 27
3.3.8 Establish your agency approach to incident handling....................................... 28
3.3.9 Let people know you are there.......................................................................... 28
3.3.10 Publish a list of indicators of an incident.......................................................... 28
3.3.11 Maintain incident reporting forms .................................................................... 29
3.3.12 Set up a planning/training meeting on scenarios .............................................. 29
3.4 Prepare to Respond ............................................................................................... 29
3.4.1 Build an archive of boot disks and distribution media ..................................... 29
3.4.2 Build an archive of security-related patches..................................................... 29
3.4.3 Identify and install tools that support the reinstallation of systems.................. 30
3.4.4 Backups backups backups................................................................................. 30
3.4.5 Generate information required to verify systems and data integrity................. 31
3.4.6 Ensure supporting equipment is available for system backups ........................ 31
3.4.7 Ensure that test systems and networks are configured and available ............... 32
3.4.8 Build a resource kit of tools and hardware devices .......................................... 32
3.4.9 Build an inventory of computing resources ...................................................... 33
3.4.10 Set up tools and techniques training ................................................................. 33
3.4.11 Build and maintain a database of contact information...................................... 33
3.4.12 Create an incident notification call tree ............................................................ 34
3.4.13 Identify your Public Affairs Office contact ...................................................... 34
3.4.14 Provide checklists and network diagrams......................................................... 34
3.4.15 Set up secure communications mechanisms ..................................................... 34
3.4.16 Involve your system administrators .................................................................. 35
3.4.17 Ensure that auditing and logging are enabled and sufficient ............................ 35
3.4.18 Utilise the power of your logs........................................................................... 35
3.4.19 Interfaces with External Agencies .................................................................... 35
DETECTION ................................................................................................................................ 37
4.1 Determine if the event discovered is an incident .................................................. 37
4.2 Begin a log ............................................................................................................ 38
4.3 Triage - identify the nature of the incident and its impact.................................... 38
4.3.1 What has happened? ......................................................................................... 38
4.3.2 What is the scope and impact?.......................................................................... 38
4.4 Take immediate steps if necessary to contain the incident ................................... 39
4.5 Protect incident information ................................................................................. 39
4.6 Notify all parties that need to be aware of the incident ........................................ 40
4.6.1 Notify those with key roles ............................................................................... 40
4.6.2 Use secure communication mechanisms .......................................................... 40
4.6.3 Inform upstream and downstream sites of attacks and intrusions .................... 40
4.6.4 Log all contacts made ....................................................................................... 41
CONTAINMENT ........................................................................................................................... 42
5.1 Exercise precautions to limit the extent of the incident........................................ 42
5.1.1 Temporarily shut down the compromised system ............................................ 43

Department of the Premier and Cabinet – Office of e-Government 4


Incident Response Plan

5.1.2 Disconnect the compromised system from the network ................................... 43


5.1.3 Disable system services, if possible.................................................................. 43
5.1.4 Change passwords or disable accounts ............................................................. 44
5.1.5 Monitor system and network activities ............................................................. 44
5.1.6 Verify that redundant systems and data have not been compromised .............. 44
5.2 Collect and protect information about the compromised systems and networks.. 44
5.2.1 Collect all information related to the incident .................................................. 45
5.2.2 Collect and preserve evidence .......................................................................... 45
5.2.3 Contact law enforcement if you decide to pursue and prosecute...................... 46
5.3 Backup the system ................................................................................................ 46
5.3.1 Avoid potentially compromised code ............................................................... 46
5.3.2 Backup .............................................................................................................. 46
5.3.3 Safely store backups ......................................................................................... 47
ERADICATION ............................................................................................................................ 48
6.1 Analyse all information about the incident........................................................... 48
6.1.1 Isolate the compromised system(s)................................................................... 49
6.1.2 Search on other systems for signs of compromise............................................ 49
6.1.3 Examine logs generated by firewalls, network monitors, and routers.............. 50
6.2 Identify the attack method used and the damage caused ...................................... 50
6.2.1 Identify the attacks used to gain access to your systems .................................. 50
6.2.2 Identify what an intruder did while accessing your systems ............................ 51
6.3 Eliminate the means of attack and vulnerabilities, and restore the system........... 52
6.3.1 Virus infections................................................................................................. 52
6.3.2 Other malicious code infestations..................................................................... 52
6.3.3 Network intrusion ............................................................................................. 52
6.3.4 Change all passwords........................................................................................ 53
6.3.5 Restore from backups or reload the entire system ............................................ 53
6.3.6 Remove any means for intruder access............................................................. 54
6.3.7 Restore programs and binary files .................................................................... 54
6.3.8 Restore user data from trusted backup media ................................................... 55
6.3.9 Review system configurations .......................................................................... 55
6.4 Verify that the means of attack and potential vulnerabilities are correctly
eliminated.......................................................................................................................... 56
6.4.1 Perform system vulnerability analysis .............................................................. 56
6.4.2 Search for related vulnerabilities ...................................................................... 56
6.5 Strengthen security and detection mechanisms .................................................... 57
6.5.1 Improve protection mechanisms to limit network and system exposure.......... 57
6.5.2 Improve detection mechanisms to enable better reporting of attacks............... 57
RECOVERY AND CLOSURE ......................................................................................................... 58
7.1 Determine the requirements and timeframe for resuming normal operations ...... 58
7.2 Enable system and application services................................................................ 58
7.3 Reconnect the restored system to the network...................................................... 59
7.4 Validate the restored system ................................................................................. 59
7.5 Monitor for any suspicious activity ...................................................................... 59
7.6 Notify all parties that the incident is closed.......................................................... 59
FOLLOW-UP ............................................................................................................................... 60

Department of the Premier and Cabinet – Office of e-Government 5


Incident Response Plan

8.1 Hold a post-mortem analysis and review meeting with all involved parties ........ 60
8.2 Conduct a lessons learned meeting ....................................................................... 61
8.3 Send recommended changes to management ....................................................... 61
8.4 Revise security plans, policies, procedures, and user and administrator training to
prevent incident recurrence............................................................................................... 62
8.5 Determine whether or not to perform a new risk analysis based on the severity and
impact of an incident......................................................................................................... 62
8.6 Take a new inventory of your system and network assets.................................... 62
8.7 Participate in investigation and prosecution, if applicable ................................... 62
ACKNOWLEDGEMENTS............................................................................................................... 63
APPENDIX A GLOSSARY........................................................................................................ 64
APPENDIX B COMPUTER AND NETWORK INCIDENT TAXONOMY .......................................... 70
APPENDIX C INCIDENT NUMBERING SCHEME ......................................................................... 72
APPENDIX D TYPES OF INCIDENTS ........................................................................................ 73
APPENDIX E SECURITY INCIDENT CATEGORIES...................................................................... 74
APPENDIX F LOG OF EVENTS FORM ..................................................................................... 76
APPENDIX G INCIDENT REPORTING FORM .............................................................................. 78
APPENDIX H CONTACT LIST ................................................................................................. 79
APPENDIX I NETWORK DIAGRAMS AND INFORMATION ......................................................... 80
APPENDIX J SERVICENET INCIDENT RESPONSE PROCEDURE ................................................. 81

Department of the Premier and Cabinet – Office of e-Government 6


Incident Response Plan

Background

Department of the Premier and Cabinet – Office of e-Government 7


Incident Response Plan

Introduction
1.1 Disclaimer

This document is a guide to help agencies have a planned, systematic approach to


handling computer security incidents that occur within Western Australian Government
computer networks. It is intended to provide a general understanding of the subject
matter, and to help people assess whether they need more detailed information and
plans. Agencies should tailor a copy of this generic document so that it is applicable to
their own networks. Agencies should seek their own legal advice where appropriate.
No proprietary products are endorsed by this document. Where examples of software
are cited, these are non-proprietary software products that are freely available from the
Internet.

Every effort is made to ensure that the material in this document is accurate and up to
date. However, the Department of the Premier and Cabinet does not guarantee or
warrant the accuracy, completeness or currency of the information provided.

The material in this document is not and should not be regarded as legal advice.

1.2 Introduction

The increased computer literacy both within Government and the general community,
as well as the low cost and efficiency of the implementation of technological solutions,
has increased the likelihood of technology being used to commit an unauthorised act or
augment its commission. People who would not dream of stealing or maliciously
damaging other people’s property in the ‘real world’ have no qualms or second thoughts
in relation to the opportunities and challenges presented by technology, the Internet,
and the ‘virtual world’.

Computers are used in all facets of Government work to create messages, write reports
and other documents, compute financial information, transfer funds and browse the
Internet. As well, Government has established a business requirement to make more
extensive use of public data networks, such as the Internet. Connections to these
networks carry a number of additional threats and risks over and above those already
identified and for which safeguards have been provided. Government needs to
understand these threats and to respond to computer security incidents with speed and
skill so that incidents can be identified, contained, resolved, and in the longer term
prevented, in a timely and cost-effective manner.

Government must also be in a position to prosecute any individual who damages


departmental information assets or disrupts its normal business activities through

Department of the Premier and Cabinet – Office of e-Government 8


Incident Response Plan

connection to Government computers from any network. As part of the Western


Australian Government’s initiative to establish a proactive approach to managing
security incidents, the Office of e-Government WA Computer Security Incident
Response Team (WACSIRT) has been formed to provide help and advice to agencies
when dealing with computer security incidents. This document has also been created
as a guide to help ensure that agencies are prepared for, and have the capability to
resolve and recover from, such an incident.

1.3. Purpose and Scope

The purpose of this Incident Response Plan is to document the steps necessary for
effectively and efficiently managing a computer security incident (hereafter referred to
as an incident).

It is aimed at the technical management of incidents, and intended for system and
network administrators, managers of information systems, and security personnel
responsible for networked information resources. Without an incident response
structure, agencies may be inadequately prepared to deal with incidents. When one is
discovered or suspected, decisions may be made in haste, responses may be
haphazard and ineffective, and may lead to:

• extensive damage to data, systems, and networks due to not taking timely action
to contain an incident. This can result in increased costs, loss of productivity,
and loss of business;

• the possibility of an incident affecting multiple systems both inside and outside an
agency because staff did not know who else to notify and what additional actions
to take;

• negative exposure in the news media that can damage Government stature and
reputation with customers and the community;

• possible legal liability and prosecution for failure to exercise an adequate


standard of due care when your systems are inadvertently or intentionally used to
attack others.

By having a systematic and well-defined incident response procedure in place, agencies


are better able to:

• understand the extent and source of an incident;

• protect sensitive data contained on systems;

• protect the systems, the networks, and their ability to continue operating as
intended;

Department of the Premier and Cabinet – Office of e-Government 9


Incident Response Plan

• recover systems;

• collect information to better understand what happened. Without such


information, you may inadvertently take actions that can further damage your
systems; and

• support legal investigations

This plan defines and categorises incidents and the actions to be taken when they
occur. The strategy processes for a systematic incident response approach have been
addressed in this plan under the following phases:

• Preparation
• Detection
• Containment
• Eradication
• Recovery and closure
• Follow-up

These phases can be grouped into 4 categories:

• Preparing to respond to an incident


• Detecting an incident
• Responding to an incident
• Taking necessary follow-up steps to ensure that your security practices are
improved. This will result in a more secure operational environment that is based
on what you learned as you responded to an incident.

Category Phase
Prepare Preparation
Detect Detection
Respond Containment
Eradication
Recovery and closure
Follow-up Follow-up

Note. Adverse events such as floods, fires, power-related disruptions, excessive heat
that causes system crashes, and other natural disasters, though certainly undesirable

Department of the Premier and Cabinet – Office of e-Government 10


Incident Response Plan

incidents, are not within the scope of this document and should be addressed in
Agencies’ business continuity (contingency) plans. For the purpose of this document,
therefore, an incident refers to an adverse event that is related to Information Security.

1.4 Related Documents

This Incident Response Plan forms part of an overall Office of e-Government Western
Australian Computer Security Incident Response Team (WACSIRT) incident response
framework. This plan refers to, and should be read in conjunction with, the WACSIRT
Forensic Plan. The Forensic Plan documents in more detail the mechanisms by which
evidence relating to security incidents should be handled by Government staff and
contractors. Also see the WACSIRT Operational Manual that documents how the
WACSIRT operates, and the WACSIRT Framework that is the head document
describing the objectives and scope of the WACSIRT.

Department of the Premier and Cabinet – Office of e-Government 11


Incident Response Plan

Incident Response
Plan

Department of the Premier and Cabinet – Office of e-Government 12


Incident Response Plan

Phases of the Incident Response Plan

FOLLOW-UP

To review the incident


response process to
identify and capture the
knowledge acquired.

RECOVERY &
CLOSURE
To restore the affected systems and
networks to operational status.

….

ERADICATION

To resolve the damage and eliminate


all potential causes of the incident.

CONTAINMENT

To stop or limit the extent of the incident


and to determine operational continuity.

DETECTION

To determine whether an incident


has occurred and if so, the nature of
the incident and its impact.

PREPARATION

To be prepared with a set of established Detection, Containment, Eradication


guidelines, procedures and mechanisms phases may be iterated until there
for responding to incidents. are no further signs of the incident.

|_______________|_________________|________________|_____________| …. |_____________|_____________|_→
T0 T1 T2 T3 Tn Tn+1

Department of the Premier and Cabinet – Office of e-Government 13


Incident Response Plan

Preparation

The goal of this phase is to be prepared with a set of established policies, procedures,
and mechanisms for responding to incidents, and to be proactive in preventing them
occurring.

Part of handling an incident is being prepared to respond to an incident before it occurs


in the first place. By establishing policies, procedures and agreements in advance, you
provide a basis upon which decisions can be made and responses actioned in an
effective manner. Having a formal incident response plan that is defined and exercised
in advance ensures that everyone understands how security incidents will be handled.
Furthermore, there are many proactive measures you can take that may reduce the
number of incidents from occurring in the first place.

3.1 Establish policies and procedures for responding to incidents

A security policy defines the rules that regulate how your agency manages and protects
computing resources to achieve security objectives. For responding to incidents, one of
the policy’s primary purposes is to document the threats you intend to guard against
and the actions you intend to take in response to a successful attack.

Response procedures describe how the response policies will be implemented


throughout your agency, e.g. who to notify, at what point in the response procedure, and
with what types of information. From these procedures, all concerned parties are able
to determine what operational steps they need to take to comply with your policies and,
thereby, respond in a manner that upholds the security objectives for your agency’s
information and networked systems.

This section describes a subset of the topics your incident response policies and
procedures should address. It is important that you are willing to enforce your policies,
and that they will support the investigators of any incident. Ensure that you have a
mechanism to show that staff is aware of, and have accepted, your policies.

Why this is important. Policies and supporting procedures that are documented,
communicated, and enforced prepare you to respond to incidents in a timely, controlled
manner. This gives you the ability to exercise your procedures and eliminate potential
errors or omissions in advance of an incident. During or after an incident, you do not
want to be in a position of needing to determine what actions to take, what data to
gather and preserve, and how to protect your data, systems, and networks from further
damage.

Department of the Premier and Cabinet – Office of e-Government 14


Incident Response Plan

Having documented plans, conducted training, and tested procedures in advance will
allow staff members to efficiently coordinate their activities when responding to an
incident. Without the knowledge conveyed through training and test exercises, users
may inadvertently expose parts of the organisation to security threats. For example,
users might reveal sensitive information if they contact the wrong person when
observing an incident.

3.1.1 Information disclosure policy


Your agency’s information disclosure policy should indicate what information is shared
with others, under which circumstances, and who has the authority to initiate information
disclosure beyond that specified in your policy. This policy is especially relevant to the
detection phase. Your agency’s information disclosure policy should also include
information dissemination policy that:

• specifies who should be notified in the event of an incident and in what order.
The order of notification may depend on the type of incident or other
circumstances. You should contact the responsible manager and your local
response team immediately. The remaining contacts are listed in no particular
order and are in this list as a ‘picking list’ for you to tailor to your requirements.
o the responsible manager and other managers who need to know
o CSIRT, if one exists
o public relations
o system and network administrators
o security officer and personnel
o your Internet Service Provider (ISP)
o human resources (in the event an employee is implicated)
o help desk personnel (who may have to answer questions about an
intrusion)
o legal counsel
o corporate investigations group (if one exists)
o law enforcement (state, federal)
o users
o vendors
o other CSIRTs, the national body AusCERT

• defines specific roles and responsibilities for each contact within your
organisation including their range of authority;

• specifies how much information should be shared with each class of contact and
whether or not sensitive information needs to be sanitised (removed or filtered)
prior to sharing it;

• identifies who to notify and when to notify them by using specified communication
mechanisms (eg phone, email, fax, pager) and whether or not these mechanisms
need to be secure;

Department of the Premier and Cabinet – Office of e-Government 15


Incident Response Plan

• identifies who has the authority to initiate information disclosure beyond that
specified in your policy.

3.1.2 Networked systems security policy


Establish guidelines and rules at the management level for responding to incidents and
include these in your agency’s networked systems security policy. These guidelines
and rules can be categorised as follows:

Priority and sequence of actions


Document the priority and sequence of actions to be taken when dealing with an
incident. Actions necessary to protect human life and safety are likely to have priority
over those that ensure operational continuity, protect classified and sensitive data, or
prevent damage to systems. Document the criteria by which the priority and sequence
of actions can change:

• over time;
• as you obtain additional information (such as the extent of an intrusion);
• as you become aware that an incident you are analysing interacts with one or
more other incidents.

Actions to be taken can include:

• denying access to an intruder, possibly by disconnecting the affected system


from the network and shutting down the system;

• containing an intrusion and limiting the actions of an intruder;

• continuing operation to gather additional information;

• restoring the affected system. You need to specify the order in which services
will be restored, if this is a consideration (for example, restore your email service
before restoring ftp). This ensures that the order meets your business objectives
and priorities while minimising negative effects on users, where possible.

Authority to act
This policy should indicate what types of incident response actions require management
approval and which are pre-approved. Document the circumstances under which you
intend to:

• stay connected to pursue an intruder by gathering additional information;


• protect your systems by disconnecting and shutting down;
• conduct covert monitoring of network traffic and file access.

Department of the Premier and Cabinet – Office of e-Government 16


Incident Response Plan

Ensure that the individuals or team responsible for incident response have pre-
authorisation from management to disconnect from the network and shut down the
affected system(s), if appropriate. Management must understand that shutting down
will cause a denial of service condition on the affected system until it is returned to
operation.

Document the actions to be taken in dealing with incidents involving remotely connected
computers used by your employees and vendors if these actions differ from those taken
for other types of incidents.

Your networked systems security policy should clearly state:

• the acceptable level of risk to business processes and the systems and networks
that support them;

• to what extent business processes and the systems and networks that support
them must remain operational, even in the event of intrusions or attacks;

• which additional systems that reside on local networks with compromised


systems will be disconnected, even if these systems are known to not (yet) be
affected. This is a preventative measure;

• who is empowered to make a final decision, under which circumstances, and


who has the authority to decide in situations not covered by the policy;

• how to manage password compromises, including when and how to change


passwords for all users and accounts at a specific site or an organisational level.

Incident response resources


Determine how you will structure and staff your incident response activity. One option is
to create a computer security incident response team (CSIRT). For this option,
determine if the team will be distributed or centralised and identify the roles to be filled
by team members based on your agency’s security policies and procedures. Identify
qualified people to assume these roles and determine how much of their time will be
devoted to team activities. Ensure that you have a knowledgeable team trained and in
place to handle the full incident response and recovery process.

If you choose not to create a CSIRT, ensure that all response roles and responsibilities
are clearly assigned to system and network administrators, security personnel, and
other staff.

Your agency’s networked systems security policy should:

• require that designated system and network administrators, and response team
members be trained in the use of incident response tools and environments.

Department of the Premier and Cabinet – Office of e-Government 17


Incident Response Plan

This training should include participation in response practice drills or simulations


using the tools and environments;

• require that the inventory of all applications software, operating systems,


supporting tools, and hardware be kept up to date;

• require quick access to backups in an emergency, even if they are stored offsite.
This may include defining procedures that give specific managers the
responsibility to authorise such access;

• state that staff members dealing with an incident may need to gain access to
restricted systems and data. This may include
o specifying how staff access is granted and how they will obtain
administrator passwords and encryption keys
o establishing the authority for staff access
o establishing the authenticity of the staff member obtaining access
o requiring that all access is documented and tracked.

The above points are especially relevant to 3.4 Prepare to Respond.

Gathering evidence
Your agency’s networked systems security policy should document the roles,
responsibilities, authority, and conditions under which data, systems, and networks
suspected of having been compromised can be examined in the analysis of incidents.
Your agency’s networked systems security response procedures should ensure that a
provable chain of custody is maintained for protection of potential evidence. Refer to
your Forensic Plan for more information about the chain of custody. This level of rigor is
required even if you choose not to enter into a legal investigation with law enforcement.
Other affected organisations may decide to take action and request your assistance and
any evidence you have collected, and other organisations may initiate legal proceedings
against you if an intruder attacked their systems from yours.

3.1.3 Configuration redundancy policy


If a critical machine is compromised as a result of an incident, having redundant
equipment in place enables you to restore service quickly while preserving all of the
evidence on the compromised machine to perform ongoing analysis. You need to
describe, for example, where and when to use hot, warm, and cold backups. (Hot
backups provide the capability to immediately switch configurations as the backup
system is being run in parallel with the primary system. Warm backups require some
degree of reconfiguration before being used since they are not run in full parallel with
operational systems. Cold backups need to be started from a shutdown state and
brought up to date before being used.)

Ensure that this policy is consistent with your business continuity policy.

Department of the Premier and Cabinet – Office of e-Government 18


Incident Response Plan

3.1.4 Presumption of privacy policy


You should have decided on and formalised a policy of presumption of privacy, and be
willing to enforce that policy. For example, you may need to clearly state that your
network, web access, and system logs may be/are being monitored.

This is probably as good a place as any to make you aware of the Telecommunications
(Interception) Act 1979 (the Act). This legislation prohibits interception of ‘a
communication passing over a telecommunication system’ except where this is
necessary for the operation or maintenance of such a system or pursuant to an
interception warrant. At the time of writing, there is a Telecommunications (Interception)
Amendment Bill 2004 before Parliament that seeks to amend the Act in relation to,
among other things, new offences, allowing access to ‘stored’ or ‘delayed access’
communications, and amending the definition of ‘interception’ to accord with recent
technological advances. Telecommunications interception is a huge area in itself, and
you should acquaint yourself with the requirements of the Act in terms of what
constitutes interception and how it may affect monitoring of your network and
information held on servers such as email.

3.1.5 Document a response procedure that implements your policies


Document the roles, responsibilities, and authority of all staff involved in executing this
procedure. Identify who performs each activity, when, and under what conditions.

Ensure that your incident response procedure is consistent and integrated into your
business continuity and disaster recovery processes.

3.1.6 Conduct a legal review of your policies and procedures


This should be performed by your agency’s legal counsel, or other qualified legal
professionals you consult with. The legal review should ensure that your policies and
procedures:

• are legally defensible and enforceable;


• comply with overall agency policies and procedures;
• reflect known industry best practices demonstrating the exercise of due care;
• conform to national, state, and local laws and regulations;
• protect your staff from lawsuits; and
• protect your agency from being held legally responsible in the event of a
compromise. This part of the review should include a consideration of the legal
implications of continuing to allow intruders access as you gather additional
information about their activities. The risk is that they might continue to use your
system to attack others.

Department of the Premier and Cabinet – Office of e-Government 19


Incident Response Plan

This is especially important if your agency intends (or may intend) to take legal action in
the event of an incident, as it will prevent technicalities from undermining an otherwise
strong legal case.

3.1.7 Train designated staff about your response policies and procedures
During the training process, users should learn:

• how to communicate appropriately with the news media including forwarding


inquiries to your agency’s public relations staff;

• the actions they need to take to report a suspected incident, including who to
notify, how (web, email, phone), and what information they should report;

• the use of incident response tools and environments based on their roles and
responsibilities.

Create and conduct periodic training about your response policies and procedures.
This training should be mandatory for all new employees and should review specific
policies and procedures relevant to the employee’s knowledge and responsibilities.

Test the effectiveness of the training and each employee’s readiness. Conduct practice
drills (e.g. responding to network intrusions and viruses) that test procedures and
execute operational activities, making sure all staff members are aware of their roles
and responsibilities. Conduct post-mortem meetings with trainees. Provide remedial
training as required.

Regularly conduct mandatory security awareness refresher training for designated staff.
Highlight recent change to policies or procedures and summarise recent incidents.
Make this subject a recurring topic at executive and management-level staff meetings to
maintain awareness.

Ensure that your system and network administrators, incident response staff, and their
managers are aware of and familiar with your Forensic Plan for evidence handling.

To stay current with the fast rate of technological change, ensure that system and
network administration staff set aside time to maintain and update the knowledge and
skills they need in technical topics required to implement your policies and procedures.

3.2 Use Proactive Techniques to Prevent Incidents

Everything in this section 3.2 either constantly needs auditing and updating, or is an
audit process that needs to be run on a timely basis. Lessons learned from the
investigation of any incident and identified in the Follow-up phase feed directly back in
to this section.

Department of the Premier and Cabinet – Office of e-Government 20


Incident Response Plan

3.2.1 Subscribe to resources for notification of new threats and


vulnerabilities
The majority of computer intrusions occur through well-known vulnerabilities. The
Office of e-Government WA Computer Security Incident Response Team (WACSIRT) is
a member of the Australian Computer Emergency Team (AusCERT) on behalf of all WA
government agencies. WACSIRT receives AusCERT security alerts, bulletins, and
advisories that contain information about threats, vulnerabilities, patches and
workarounds of an IT security nature. WACSIRT makes this information freely available
to agencies in two ways:

• via an email distribution list of those people/agencies who wish to receive all
notifications; and

• via a mailing list server, so that people/agencies can subscribe only to those
email lists of interest to their infrastructure.

You should already be subscribing to this service. If you are not, please:

• email gholt@dpc.wa.gov.au or wacsirt@dpc.wa.gov.au and request that you be


added to the distribution list to receive all notifications;

• point your browser to http://list.bs.wa.gov.au/mailman/listinfo to see all the


WACSIRT lists available for you to subscribe to. All lists prefixed csirt_ are run
by the WACSIRT. Click on the list you are interested in for directions on how to
subscribe; or

• contact Gail Holt at 9213 7118 or gholt@dpc.wa.gov.au for more information.

There are many freely available sites/resources to provide guidance in securing


systems, and to help you to stay on top of the latest threats and vulnerabilities, for
example http://www.ntbugtraq.com, http://isc.incidents.org, http://xforce.iss.net,
http://www.auscert.org.au, http://www.sans.org.

At the very minimum you should be subscribing to the service offered by WACSIRT.

3.2.2 Establish standard build procedures


All machines on your network should be built and configured to a standard. You may
have more than one standard, for example one build standard for Unix, one for
Windows PCs and one for Windows 2000 servers. All operating system services that
are not needed for the function of the machine should be disabled; this is especially
important for bastion host machines that face the Internet, and is termed ‘hardening of
the kernel’. If a service is not running, it is not available to be compromised. As part of
this process, and part of your network information, you should document what services
each machine is running – if you are able to recognise what ordinary processes your
machines are running, you will be able to quickly spot programs that are unusual.
These programs could be Trojans or backdoors placed on a system compromised by

Department of the Premier and Cabinet – Office of e-Government 21


Incident Response Plan

hackers. Consider running cryptographic hashing algorithms like MD5 or SHA-1 or


install software like Tripwire to monitor the integrity of critical system files. These
checksums can be used later to verify the integrity of your system files. (Refer to 3.4.5
Generate information required to verify systems and data integrity.)

3.2.3 Patch known system vulnerabilities


Your systems must be kept current with vendor security patches, and any other patches
recommended by your alerting service that are applicable to your network. Work with
your system administrators to ensure that critical patches are made in the shortest
possible time frame.

3.2.4 Perform network auditing and penetration testing


Conduct regular audits and penetration tests to make sure that your systems are
configured securely and that system patches have been applied. Run an external test
to see what an outsider (from the Internet) can discover and exploit about your systems,
and an internal one to see what an insider can discover and exploit. Also consider
running a ‘war-dialling’ test to discover unauthorised modems and poorly configured
authorised modems.

Rogue wireless LANs are easy and cheap to install, and are proving to be a real
security headache. As well, many major computer vendors are selling an increasing
number of laptops with built-in wireless LAN access cards. If your agency has a policy
that bans wireless LANs completely, or more precisely prohibits employees from
deploying their own wireless networks, you must decide how to enforce that policy
across the enterprise by choosing and using mechanisms to detect rogue wireless
networks.

3.2.5 Display warning banners


From a legal perspective, displaying warning banners is one of the most important steps
you can take to prepare for an incident. Every system should display an approved
warning banner visible to all users who attempt to login to the system. The banner
should state that the system is the property of your agency, is subject to monitoring, and
that unauthorised use or access is prohibited. You should have your banner reviewed
by your legal counsel. The banner should not reveal any details about the operating
system, machine, or the purpose of the machine.

Department of the Premier and Cabinet – Office of e-Government 22


Incident Response Plan

A sample banner could be:

******************************************************************
Only authorised users may access this machine.

The information on this machine and network is private


property and is protected by intellectual rights. You must
be assigned an account on this machine to access
information and are only allowed to access information
as defined by a system administrator.
All activities on this machine may be monitored.

******************************************************************

3.2.6 Ensure that systems run antivirus with current definitions


Antivirus software is only effective when definitions are updated regularly. Make sure
that you can easily distribute antivirus updates in the event of an outbreak. Antivirus
software should be present on all 3 layers of your network – the desktop, your servers,
and your email gateways. It is also a good idea to run different products on each level,
and to check from time to time to make sure your products are actually running;
malicious code will attempt to terminate antivirus and personal firewall programs.

3.2.7 email client configuration


Worms and viruses (for example the recent W32/Sobig.F worm) typically are email
borne malicious programs with specially crafted attachments. Typically a user has to
execute (open) the attachment for the code to be executed. Some email clients are
configured to automatically open email attachments, thus automatically executing the
malicious code and not giving the user the opportunity to scan the attachment before
opening, or simply deciding to delete the suspicious email. Ensure your email client
configuration does not automatically open attachments.

3.2.8 Keep all network information current


Network diagrams showing network topology, IP addresses, equipment types and
configurations should always be kept up-to-date to show the current state of your
network. Network diagrams should show where all machines (including switches,
routers and servers) are located, what connections and interconnections you have, and
where your network perimeter is. If you can’t define your perimeter you can’t protect it.
Network information should always be kept secure, as the information could be used to
compromise your network if it fell into the wrong hands.

3.2.9 Improve technical security skills


Often, vulnerabilities are not removed because system and network administrators are
unaware of them, or do not know what to do about them. It is important that your
system and network administrators and security staff have the opportunities to keep

Department of the Premier and Cabinet – Office of e-Government 23


Incident Response Plan

abreast of security developments and trends, emerging threats and vulnerabilities, and
have the chance to upgrade their skills.

3.2.10 Establish automated intrusion detection


An intrusion detection system (IDS) can provide an early alerting mechanism when a
system intrusion is in progress. Commercial products and freeware tools such as Snort
(http://www.snort.org) can be set up to monitor critical network segments to watch for
attacks. In addition, host-based agents on critical servers can issue an alert when
individual systems may be under attack. Prioritise alerts and tune the IDS to reduce
false positives and produce quality alarms. Some IDS products can be configured to
automatically respond to an incident, for example, by terminating an attacker’s
connection or reconfiguring the firewall access control list. Caution should be employed
when configuring an IDS in this manner because clever attackers will be able to use
automatic responses to trick your IDS into taking unwanted actions.

3.2.11 Know who has access to your systems


Establish a system through which you routinely record every outside person, such as a
consultant or an employee working from home, who is given access to systems and
information. Record what access was granted. Also establish guidelines and
mechanisms to remove their access when they cease employment, are transferred, or
when access is no longer required. Where possible, make security guidelines
mandatory in contracts with all outside vendors, contractors and consultants.

3.2.12 Keep track of remote computing users


Establish a system through which you routinely record every outside person, such as a
consultant or an employee working from home, who is given access to systems and
information. Record what access was granted. Also establish guidelines and
mechanisms to remove their access when they cease employment, or are transferred or
when access is no longer required. Where possible, make security guidelines
mandatory in contracts with all outside vendors, contractors and consultants.

3.2.13 Establish extranet agreements and monitoring


Establish agreements with all outside organisations connected to your network that give
your agency the right to disconnect and monitor as needed. Consider including in the
contract a clause that requires all parties to inform the others immediately if an incident
occurs. Ensure that security requirements are included in service level agreements
(SLAs).

3.2.14 Identify critical assets and servers


Perform a risk assessment to identify your agency’s critical computing assets.
Determine the impact that a system might have if it were to become unavailable or have
confidential data exposed. Spend extra time protecting critical assets and take
additional steps to secure these systems such as encrypting sensitive data and using
host-based intrusion detection. The Department of the Premier & Cabinet has
developed an Information Security Management System (ISMS) methodology to help
you implement ‘best practice’ risk management and information security management

Department of the Premier and Cabinet – Office of e-Government 24


Incident Response Plan

based on Australian and International standards. Please go to


http://www.govsecure.wa.gov.au for more information regarding the ISMS.

3.2.15 Review the physical security of critical assets and servers


Recent thefts of government computers containing corporate information have been
reported in the press. Apparently the thieves were ‘disguised’ as technicians, and
simply waltzed in and took the machines. Servers and laptops containing sensitive
information must be physically secured, and only be accessible by nominated
personnel.

3.2.16 Use strong account policies


Idle accounts and accounts with weak passwords are incidents waiting to happen.
Define and enforce strong password policies that include expiration, password
complexity rules (such as upper/ lower case characters, use of numbers) and password
history so that passwords are not reused. Tools like John the Ripper
(http://www.openwall.com/john/) can be used to assess passwords. There are also tools
for Unix and Windows to assess the strength of passwords as they are created.

3.2.17 Domain and network information current in whois databases


Public directories are available on the Internet so that other sites can contact you if they
detect that your systems are involved in an incident at their site. Directories based on
your domain name and your IP address must be kept up to date with current point of
contact information.

• Domain name. The Registrar for .gov.au domain names is the National Office for
the Information Economy (NOIE). For more information on .gov.au domain
names, or to see or update details for your domain name, point your browser to
http://www.domainname.gov.au. If you have a .com.au domain name, these are
now managed by a host of Registrars accredited by .au Domain Administration
Ltd (auDA). To see a list of auDA accredited Registrars, point your browser to
http://www.auda.org.au/registrars/. To check the Registrar for your .com.au
domain name, use the whois facility at http://whois.ausregistry.net.au. Then you
can update your domain name using your Registrar’s website.

• IP address. Regional Internet Registries (RIRs) manage, distribute, and register


public, numeric Internet address space (and related resources) within their
respective regions. The Asia Pacific Network Information Centre (APNIC) is one
of 4 RIRs in the world, and is responsible for distributing and registering Internet
address resources throughout the Asia Pacific region, of which Australia is part.
(APNIC is not involved in domain name registration.) APNIC is responsible for
Internet Protocol (ip) address space (both ipV4 and ipV6), Autonomous System
(AS) numbers, and in-addr.arpa (reverse address) domain delegations.
Information about IP addresses allocated within Australia is available from the
APNIC databases. For more information, or to see or update details for your
address space, point your browser to http://www.apnic.net.

Department of the Premier and Cabinet – Office of e-Government 25


Incident Response Plan

3.2.18 Situational awareness


Be aware of what is happening in the world, and how it could affect you. Things
change, leading to new vulnerabilities.

3.3 Form Your Incident Response Capability

Senior management must provide ongoing commitment and support for an incident
response capability. Before establishing an incident response capability, senior
management must serve as an advocate of security and be proactive in preventing
incidents. Senior management must ensure that the security policies of the agency are
supported with reliable security and detection mechanisms, and that IT security
awareness and responsibility is conveyed throughout the agency. Senior management
must also commit to and support the initiative for an incident response capability.

3.3.1 Create an incident response plan


Thorough preparation for an effective incident response strategy involves establishing a
set of guidelines and supporting procedures that define the course of actions to exercise
when responding to incidents. Creating a formal incident response plan will ensure that
everyone understands how security events will be handled. This plan should be
regularly updated and should include escalation paths, a contact list, current network
information, and sample forms.

3.3.2 Authority to act


Having the authority to make difficult decisions in the midst of a crisis can mean the
difference between success and failure. Determine the level of authority you are
granting the incident response team to make critical decisions, including the authority to
take servers offline, shutdown affected systems, and disconnect networks. This should
already be documented as policy (refer to 3.1.2 Networked systems security policy) and
be pre-approved by management.

Ensure that management supports and understands the level of authority the incident
response team will have in the event of a problem, and document this in your incident
response plan.

3.3.3 Pick the team


Select team members that includes more than your system and network administrators.
Include trained management and representatives from information security, risk
management, human resources and the legal department to help make the hard
decisions about whether or not to shut down core business systems to preserve your
agency from even greater harm. Identify subject matter experts and keep their contact
information on file. Contact details for all team members should be included in your
contact list.

Department of the Premier and Cabinet – Office of e-Government 26


Incident Response Plan

3.3.4 Establish a primary point of contact


Effective coordination requires a single point of contact (POC) otherwise no one knows
who is in charge. Nominate someone in the team to be the single POC.

3.3.5 Establish visibility and a compensation plan for the team


Incident handlers and the system administrators who support them often have to put in
extraordinary effort in responding to an incident. Negotiate with management in
advance, to provide mechanisms for recognition and compensation. Set the ground
rules for what overtime will be paid and what time off can be granted to those who work
long hours responding to security incidents.

3.3.6 Decide on the scope of operation


You need to decide, and document, what types of incidents will be handled by the team,
and what types of incidents will not be handled by the team. This needs to be
communicated within your agency. For instance, the types of incidents that may or may
not be handled could include:

• intrusions
• software vulnerabilities
• viruses
• SPAM
• illegal activities such as software piracy
• requests to investigate suspected staff
• requests to perform on-site security audits
• requests to perform on-site training
• requests to speak at conferences
• requests for security information

In addition, you must decide what level of assistance will be provided. Will the team
merely forward notification of security incidents onto the affected people, or will they
work completely with staff to determine the extent of the incident and help them to better
secure their systems?

3.3.7 Prioritise your incident response procedures


Decide on and document your priorities when dealing with an incident. For example:

Priority 1: Protect human lives


Priority 2: Protect classified systems
Priority 3: Protect other systems
Priority 4: Prevent damager to systems

Department of the Premier and Cabinet – Office of e-Government 27


Incident Response Plan

Priority 5: Minimise disruptions to operation

Once again, this should already be documented in a policy and be pre-approved by


management. Refer to 3.1.2 Networked systems security policy, Priority and
sequence of actions.

3.3.8 Establish your agency approach to incident handling


You need to choose between two approaches to incident handling. The first, and
generally the simplest, is to ‘contain, clean, and deny access’. The focus of this
approach is to eradicate the problem as quickly as possible and get back into business.
The alternative approach, requiring more technical skill and planning, is to ‘monitor and
gather information’. In this case, you allow the intruder to continue the attack, perhaps
with subtle restrictions that minimise further damage. Often, the decision between the
two approaches rests on whether or not you intend to prosecute the intruder. The
simpler ‘contain, clean, and deny access’ may not provide evidence needed to identify
and prosecute the criminal. Both approaches have advantages and disadvantages.
Your approaches should already be documented in a policy and be pre-approved by
management. (Refer to 3.1.2 Networked systems security policy). Management
must be involved in this decision, as ultimately the risk to the business environment
belongs to them. Each system or application, based upon its business criticality, can
have an incident handling approach assigned to it. This decision needs to be made
prior to an incident, and formalised in the incident handling procedures.

After you determine your approach to incident handling, it will be much easier to
determine under what conditions you are going to approach law enforcement.

3.3.9 Let people know you are there


When users do not know who to contact or what to say, they may not report information
about possible security incidents. Advertise your presence and let people know what
you are there for. New employees should be briefed as part of their orientation process.
Encourage reporting.

3.3.10 Publish a list of indicators of an incident


A list of indicators can assist your team and others in recognising an incident when they
see one. Examples of indicators could include:

• alerts from your Intrusion Detection Systems


• denial of service
• unexplained poor system performance
• suspicious probes (numerous unsuccessful login attempts from another node)
• activity in previously idle accounts
• unusual offsite access
• new setuid root scripts

Department of the Premier and Cabinet – Office of e-Government 28


Incident Response Plan

• transfer of /etc/passwd in ftp and web logs


• system crashes
• unexplained filling of file systems
• gaps in accounting logs
• new or unfamiliar file names

This list is by no means comprehensive; it is simply lists examples of some common


indicators. Work with your system administrators to establish a list of indicators that are
suited to your environment. Let system administrators know that it is worth the extra
effort to look into suspicious activity and report it as soon as possible.

3.3.11 Maintain incident reporting forms


Maintain incident reporting forms, such as the ones included in this Plan (see
Appendices). Using prepared forms will ensure that incident notes are thorough enough
to capture all required information. You may decide to include information about your
incident response team on your intranet, and make the incident reporting form available
there.

3.3.12 Set up a planning/training meeting on scenarios


Prepare and train your staff to handle a real incident. Plan and conduct a session for
your incident response team to discuss how to handle basic scenarios. Examples might
include a successful website defacement, or the discovery of a virus on an email server.
Determine how these incidents would be handled, and decide whether resources
outside your agency would need to be involved.

3.4 Prepare to Respond

Refer to section 3.1.2 Networked systems security policy for more information on
policy considerations that may be needed to be in place to underpin some of the
following steps.

3.4.1 Build an archive of boot disks and distribution media


Build an archive of boot disks and distribution media for all applications and all
operating systems and versions in use at your agency. Having an archive of original or
trusted boot disks (or CDROMs) provides the capability to restart a specific computer
from a known, pre-existing configuration. This ensures, to a large extent, that
compromised files, programs, and data are not reloaded onto the system. All media
should be hardware-write-protectable to avoid intentional or accidental tampering.

3.4.2 Build an archive of security-related patches


Build an archive of security-related patches for all applications and all operating
systems and versions in use at your agency. Every new version of every operating

Department of the Premier and Cabinet – Office of e-Government 29


Incident Response Plan

system or application contains some unknown vulnerabilities and errors. Vendors


provide security-related patches (also called bug-fixes, hot-fixes, etc) to correct these.
Usually they provide such patches free of charge to their customers.

Having an archive of patches allows you to initiate a specific operating system or


application in a known, secure configuration by applying patches whenever the software
is initially installed or subsequently reinstalled.

3.4.3 Identify and install tools that support the reinstallation of systems
Identify and install tools that support the reinstallation of systems, applications, and
patches. This includes:

• installation servers containing trusted, generic versions of the original distribution


for all operating systems, application, and versions in use at your agency. The
reinstallation process can be executed from these servers for a network with a
large number of hosts with prompting for any host-specific information such as
IP address; and

• all tools that are needed to retrieve, unpack, verify, and install software patches
that are intended to correct errors and eliminate vulnerabilities.

3.4.4 Backups backups backups


Your daily generation of backups provides you with copies of your systems and data.
You depend on these copies to be able to restore the most recent version of a system
or specific data files in the event of damage to your systems or data. Inadequate
backups have been the cause of many catastrophes in recovering from security
incidents. You need to have high confidence that the restored assets are as you
intended by regularly testing your ability to restore these assets from previously stored
backups. Critical systems should be backed up at least daily. Now might be a good
time to review your systems backup regime. Also ensure that offsite backups can be
retrieved in a timely manner.

Department of the Premier and Cabinet – Office of e-Government 30


Incident Response Plan

Think of your backups as being your


reserve parachute. In the event that
your main system fails, you rely on
your reserve. If your reserve fails,
there is only one way to go….. down!

3.4.5 Generate information required to verify systems and data integrity


Prepare a set of test results to use for comparison purposes. This will allow you to have
some level of confidence that your systems have been properly restored after an
incident occurs. Such results may include a scan for services on the network level
(building up an authoritative list of such services) and a file of cryptographic checksums
for critical configuration files (building up an authoritative list of such checksums). (Also
refer to 3.2.2 Establish standard build procedures.)

3.4.6 Ensure supporting equipment is available for system backups

You need to ensure that high capacity, removable and hardware-write protectable
media and supporting equipment are available to make and restore system backups.
Backups you make during the response process are used:

• to save the latest version of data and configuration information. This is


necessary for restoring systems to their last known state, which will include the
most recent modifications made by users and system administrators not
available from the routine backups;

• to ensure that evidence on compromised systems is preserved; and

Department of the Premier and Cabinet – Office of e-Government 31


Incident Response Plan

• to provide an easy way to establish the compromised environment on other


systems required to conduct analysis (for example, on an isolated test network).

The media that you use to store backups must have sufficient capacity to contain all
backup information.

To make backups and to load backups on other systems, all of the necessary devices,
cables, plugs, and terminators need to be available as well as the software that was
used to create the backups.

If possible, use media that cannot be written again once it has been used (i.e. protected
for read only access such as WORM media). This safeguards the information and
avoids accidental overwriting.

3.4.7 Ensure that test systems and networks are configured and available
Using compromised systems for any kind of analysis or test may expose these systems
to further damage and, given the systems are compromised, any results produced by
them are unreliable. In addition, using such systems may inadvertently inform an
intruder of the tests you are executing through the generation of network messages by
malicious or compromised programs.

If you have sufficient resources available, consider setting up test systems and
networks that are both physically and logically separate from your operational system
and network. You can then move the compromised systems to the test network, and
deploy newly installed and fully patched and secure systems onto the operational
network so as to continue operations. Newly installed systems may have the same
vulnerability that an intruder used to gain access. However, making the original
systems available for analysis may give you an advantage over other approaches for
reconstituting the compromised systems on your test network. In addition, the newly
installed system will not contain any software that the intruder may have left behind.

If your equipment is limited or you do not want equipment to be idle when no analysis is
being performed, you could instead choose to have a plan and process in place to
configure a test environment quickly, on an as-needed basis.

After analysis is complete, clear all disks to ensure that any remnant files or malicious
programs do not impact future analysis or any ongoing work on the test system, or are
inadvertently transferred to other operational systems. Refer to your Forensic Plan.
This is particularly critical in the event that your test system is used for other purposes.

3.4.8 Build a resource kit of tools and hardware devices


The resource kit should contain all tools that you may need to use in the response
process. Examples of such tools include those that make and restore backups,
compare files, build and compare cryptographic checksums, take system snapshots,
review system configurations, list services and processes, trace routes to a destination
etc.

Department of the Premier and Cabinet – Office of e-Government 32


Incident Response Plan

Ensure that the resource kit is available on clean, hardware-write-protectable media.

Ensure that hardware devices such a printers or laptops are reserved for use in the
incident response process. Hardcopy materials are often required as part of an
incident log and archives. For example, laptops can be used to monitor your network
for suspicious activity, to retrieve contact information from Internet directory servers,
and to access previously collected information such as default configurations and user
id lists.

3.4.9 Build an inventory of computing resources


Having an up-to-date inventory of all your computing resources will help in the response
process by enabling you to quickly identify machines that may have related or the same
vulnerabilities. Compile and keep current a list of all IP address ranges you own and
systems you are running in your network – hardware platforms, operating systems,
vendor packages, third party packages, public domain packages and patch
levels/versions of all systems. Your inventory should include such information as:

• function (e.g. firewall, public web server, internal email server);


• hardware platform make, model, serial number;
• operating system, version, patch level;
• application software, version, patch level; and
• IP addresses. Document all interfaces if multi-homed.

Documenting the IP address of the machine will enable you to quickly locate it on your
network diagram. This information should always be kept secure, as the information
could be used to compromise your network if it fell into the wrong hands.

3.4.10 Set up tools and techniques training


If you decide that your incident response team will be handling the forensics of an
incident investigation you will need to ensure that they have the tools and techniques to
do this. Establish training for your team members on using tools and techniques for
backup, evidence collection, and analysis. Create a CD or floppy with forensics tools,
including trusted command interpreters. It is useful to ensure that your legal team
review these tools in advance if you plan to use their output as evidence. Ensure that
team members are fully conversant with your Forensic Plan.

3.4.11 Build and maintain a database of contact information


Create a database containing contact information of all people that may need to be
contacted if an incident occurs within your agency. As well as members of the incident
response team, system administrators, network administrators, and subject matter
specialists you will need managers and senior managers in the event an incident needs
to be escalated. Also consider what external contacts you may need - your Internet

Department of the Premier and Cabinet – Office of e-Government 33


Incident Response Plan

Service Provider, Computer Crime Investigation Unit, and the Office of e-Government
WA Computer Security Incident Response Team. :)

To ensure database availability during an incident, store a backup copy off-site, have a
copy available on an accessible system that is not connected to any network, such as a
laptop, and have hardcopies available in an Appendix of your Plan. Keep all contact
information up to date, and protect it as you would any other piece of critical information.
Getting to know all your contacts before an incident occurs helps to make your
response process more efficient.

3.4.12 Create an incident notification call tree


Create, use and maintain an incident notification call tree that shows the sequence of
people to call and who will call whom. Also document procedures for informing people
quickly. Your Incident Response Team should carry this contact information them at all
times in the event they need to us it

3.4.13 Identify your Public Affairs Office contact


Some agencies have a Public Affairs Office that is responsible for answering questions
from the public regarding agency activity. When a security-related incident occurs it
may be the Public Affairs Office’s responsibility to disseminate appropriate information
to the public. Your public affairs team may also be valuable support for incident
handling, as they usually have very good access to senior management and can help to
co-ordinate communications between the incident response team and senior
executives. All press interaction should take place through the Public Affairs Office,
with the help of management and the incident response team.

3.4.14 Provide checklists and network diagrams


Checklists for incident handling procedures help team members to avoid errors and take
the right steps toward incident resolution. Include network diagrams and
platform/system information documents that reflect current network topology, including
router Access Control Lists (ACLs) and firewall rule sets. These documents must also
be kept in an Appendix of your Plan.

3.4.15 Set up secure communications mechanisms


Incident information dissemination during the handling of an incident needs to be
authentic and secure to ensure the confidentiality of sensitive information. Determine
whom you may need to communicate with using secure mechanisms during the
handling of an incident. Communication may take place using electronic means such
as email. All your points of contact need to agree on what technology to use, and if
necessary, exchange authenticated encryption and signature keys in advance. You
may want to consider protecting other communication mechanisms such as fax or
phone using encryption technology.

Be aware that even the act of two people communicating could indicate to an intruder
that they have been detected. A sudden flurry of encrypted email from your internal
security group to users, system administrators, CSIRTs, and others is a sure sign that

Department of the Premier and Cabinet – Office of e-Government 34


Incident Response Plan

something is happening. Your response procedures need to take this into account
including the possibility of conducting all communications without the use of email (eg
using only phone and fax).

3.4.16 Involve your system administrators


Your system administrator is the individual most often responsible for the operational
security of machines at your agency. They are involved in the day-to-day running of
your systems, and (hopefully) have their fingers on the pulse of your computer systems.
Alert system administrators have detected many incidents by noticing some event that
is strange, and by reviewing a system’s log files. System administrators also have the
potential to do great harm in an incident – with privileged accounts they can alert an
intruder that they have been detected, destroy evidence or even destroy system files in
an attempt to eradicate an intruder.

Invite your system administrators to help in the incident handling process. Because of
their intimate knowledge of your computer systems they can often bring the perspective
needed to solve a problem.

3.4.17 Ensure that auditing and logging are enabled and sufficient
Make sure that sufficient auditing and logging are enabled before an incident occurs.
Also, make sure that system clocks are synchronised from a trusted time source. This
will ensure that log files accurately reflect the real time that an incident occurred. Where
possible send logs to duplicate log servers. For instance, with the syslog facility, it is a
simple matter to keep a local log and send a duplicate copy to a syslog server. A
duplicate log on a separate logging server can be a huge help in the eradication phase
if an intruder has deleted or manipulated log entries on the local machine.

3.4.18 Utilise the power of your logs


The indicators of many never-detected incidents are buried in log files. It is not enough
to collect system logs, it is important to read them, as they are the machine’s ‘diary’ of
what has happened. Obtain tools that facilitate log file analysis and that automate the
process as much as possible. Logs can be extremely large and it is usually simply not
possible to dedicate a system administrator to the task of manually reading log files (nor
do they generally want to!). Ensure that your log retention policy is sufficient to support
in-depth investigations, where logs from weeks or months prior might be required.

3.4.19 Interfaces with External Agencies


The Office of e-Government Western Australian Computer Security Incident Response
Team (WACSIRT) is your first point of contact for all incident escalation outside your
agency. WACSIRT has already established relationships with such agencies as
AusCERT, the WA Police Computer Crime Investigation Unit, WA Police Child Abuse
Unit, and the Australian Federal Police, and can act as a central communication
interface with these and other agencies. Your WACSIRT contact is:

Department of the Premier and Cabinet – Office of e-Government 35


Incident Response Plan

Gail Holt
Principal Technical Officer
Office of eGovernment
Department of the Premier & Cabinet
Level 10 Dumas House
2 Havelock Street
West Perth WA 6005
p: (08) 9213 7118
f: (08) 9213 7101
m: 0407 195 279
e: gholt@dpc.wa.gov.au
e: wacsirt@dpc.wa.gov.au
e: wacsirt@iinet.net.au

Department of the Premier and Cabinet – Office of e-Government 36


Incident Response Plan

Detection

The goal of this phase is to determine whether an incident has occurred and if so, the
nature of the incident and its impact.

Incidents can be detected in numerous ways. System administrators or network


engineers may detect an incident by monitoring systems and networks for unexpected
activities, someone may simply notice that something doesn’t look quite right on a
system, or you may receive a phone call from an external agency notifying you that
suspicious activity has been tracked back to your network. When an intrusion, attack
or anomaly is discovered or suspected, you must take steps to determine if the event is
an incident, and if so, the nature of the incident and its impact. Immediate containment
is necessary if the incident may cause escalating damage to information assets or
computing resources that are of critical business value to you. Information observed
about the incident must be logged and protected to ensure the preservation and
admissibility of evidence. The incident must then be notified to your incident response
team in a timely and controlled manner. Your incident response team will determine if
the incident is a crisis, and what action should be taken.

The following steps may not necessarily be carried out in the sequence listed. For
instance, if one of your staff suspects that an incident has occurred, they may decide to
immediately contact your Incident Response POC to triage the incident, rather than
trying to identify the nature of the incident and its impact themselves. Or a system
administrator may identify that an incident occurring is of such catastrophic
consequences that he decides to immediately take steps to contain it before notifying
your Incident Response POC or beginning a log of the incident. For all incidents, these
steps should all be carried out regardless of the sequence.

Refer to 3.1.1 Information disclosure policy and 3.1.2 Networked systems


security policy for more information on policy considerations that may be needed to be
in place to underpin some of the following steps.

4.1 Determine if the event discovered is an incident

Is this a real incident? You need to determine if a problem really exists. Use the list of
indicators you developed during the preparation phase to quickly assess the possible
type of incident. Check for simple mistakes such as errors in system configuration,
hardware failures, and user or system administrator errors. Too often people leap to the
conclusion that there is a hostile intent behind every problem. Once you have ruled out
simple mistakes it is easier to determine the total scope of the incident. To help in
identifying whether there really is an incident, you may need to examine system logs
and logs from any detection systems you have.

Department of the Premier and Cabinet – Office of e-Government 37


Incident Response Plan

4.2 Begin a log

As soon as an incident is suspected, the person handling the incident should start
taking notes to keep a Log of Events, or Running Sheet. (Have a read of your Forensic
Plan section on Contemporaneous Notes.) Log all information observed about the
incident, and record all steps that you take. Log entries should include such details as
who did what, when, how and why they did it. The notes should be chronological with
the date and time of each entry notated. Record facts only, and avoid speculation. If
any equipment is to be removed and sealed as evidence, be sure to record any
identifying information such as make, model, or serial number. Have a look at a sample
Log of Events form included in an Appendix F of this document.

4.3 Triage - identify the nature of the incident and its impact

You need to evaluate the scope and impact of the problem. It is important to correctly
identify the boundaries of the incident in order to effectively deal with it and prioritise
responses.

4.3.1 What has happened?


Document what is being defined as an incident.

4.3.2 What is the scope and impact?


In order to identify the scope and impact, you should define a set of criteria that are
appropriate to your site and to the type of connections available. Some of the issues
include:

• Is this a multi-site incident?


• How many machines have been compromised?
• Is the compromised machine critical?
• Do any critical machines ‘trust’ the compromised machine?
• Are any critical machines on the same subnet as the compromised machine?
• Do any critical machines trust a machine on the same subnet as the
compromised machine?
• Is sensitive information held on the compromised machine?
• Do the user accounts on the compromised machine have rights within other
systems or applications?
• Does the compromised machine have network shares with any other machine?

Department of the Premier and Cabinet – Office of e-Government 38


Incident Response Plan

• What is the entry point of the incident (network, phone line, internal etc)?
• Is the press involved?
• What is the potential damage of the incident?
• What resources could be required to handle the incident?
• Is law enforcement involved?

You should document the answers to as many of these questions as possible as part of
gathering background information for handling the incident.

Previous risk assessments undertaken within your agency will have already identified
critical servers/machines, and so provide important input to this step. (You have done a
risk assessment of critical systems, haven’t you☺.)

At this point it is not necessary, or usually possible, to absolutely determine the cause or
extent of the incident; doing so may require days of careful analysis. The goal is to
determine the most likely and worst case risks, based on what is known about the
incident and/or compromised system(s).

4.4 Take immediate steps if necessary to contain the incident

If the nature of the incident and the answers to questions asked in incident triage
indicate that this incident is of high severity, and may cause escalating damage to
information assets or computing resources that are of critical business value to the
organisation, you may need to advance immediately to the Containment phase.

4.5 Protect incident information

You need to protect incident information to ensure preservation and admissibility of


evidence. The events that you see and the evidence that you collect while investigating
this incident may end up in a legal case if the incident is escalated to law enforcement
for prosecution. It won’t be worth much in a courtroom unless you can prove much later
on that these are the exact events that happened, and are the same evidence you
collected during the incident. You must protect all your Logs of Events and every piece
of evidence by controlling access to them. Identify an evidence custodian to ensure that
you can prove who has access to the secure container used to store the evidence – and
this should be a very small group of people. If there is a key lock, each key should be
stamped do not duplicate. Make sure, by policy and practice, that each person with
access understands they are required to control access to these items.

A chain of custody should also protect each piece of evidence – have a read of your
Forensic Plan sections on Seizing the Evidence and The Chain of Custody.

Department of the Premier and Cabinet – Office of e-Government 39


Incident Response Plan

4.6 Notify all parties that need to be aware of the incident

As soon as possible after an incident has been suspected or detected, it should be


notified to your incident response team. Any documentation relating to the incident,
including any Logs of Events should be handed over to the team. The team will
complete an Incident Reporting Form and take responsibility for the handling of the
incident.

4.6.1 Notify those with key roles


You need to notify, and keep informed, all those designated as having key roles in
responding to incidents. This is important because they cannot execute their
responsibilities if they are not notified in a timely manner that an incident is occurring or
has occurred, and if they are not kept informed as an incident progresses. These
people should have already been defined in your contact lists and call notification tree
lists.

Share all information on a need-to-know basis and sanitise sensitive information as per
your information dissemination policy.

4.6.2 Use secure communication mechanisms


Use secure mechanisms for all communication, as you have already established as part
of your preparation (see 3.4.15 Set up secure communications mechanisms). Do not
send email from compromised systems or networks.

4.6.3 Inform upstream and downstream sites of attacks and intrusions


Upstream sites are those that were involved in an intrusion prior to your system
becoming involved. Downstream sites are those that were involved after your site
experienced an intrusion.

In the process of analysing an intrusion, you are often able to gain information about
systems not belonging to your agency that were:

• used by an intruder to attack your systems;


• attacked by an intruder from your systems;
• used by an intruder to access your systems;
• accessed by an intruder from your systems.

Such information is usually obtained from logs about connections attributed to an


intruder or from remnant files left behind by an intruder. Remnant files may include
scripts with the IP addresses of the attacked hosts or the output files of attack scripts an
intruder neglected to delete.

Department of the Premier and Cabinet – Office of e-Government 40


Incident Response Plan

You have a responsibility to inform the administrators of all other organisations about
the involvement of their systems so that they can take the necessary steps to respond
to an intrusion. This includes any ISPs that may have been involved in transmitting and
receiving intruder messages. However, in the event of an ongoing legal investigation,
your ability to inform other organisations may be restricted. For example, action on your
part may affect the outcome of the investigation in some way or users from the other
organisation may, in fact, be the subjects of the investigation.

Depending upon what type of incident has occurred, it may be worthwhile reporting it to
ServiceNet, our WA Government ISP, even if ServiceNet is not your ISP. This is
because they may be able to:

• help collate information on the same incident occurring in multiple government


agencies;

• be proactive in helping prevent malicious traffic from spreading to other agencies


by placing filters on inter-agency routers;

• be proactive in helping to block malicious traffic from entering the government


intranet by placing filters at the external firewalls.

Included in this plan is the ServiceNet Incident Response Procedure that details their
site procedure of what actions are to be taken when an incident occurs at ServiceNet.
Refer to Appendix J for this plan and ServiceNet contact details.

4.6.4 Log all contacts made


Keep an accurate, detailed log of all contacts made and of the information exchanged.
Record all phone calls and even conversations. These details should be made as
entries in your Log of Events.

Department of the Premier and Cabinet – Office of e-Government 41


Incident Response Plan

Containment

The goal of this phase is to stop or limit the extent of the incident and to determine
operational continuity.

When an incident has been detected and notified, steps must be taken to stop or limit
the extent of the incident. This can be achieved by exercising tactical precautions to
deny further attacks and to limit the extent of damage caused. The compromised
systems and networks may also be isolated to regain control of these systems for
investigating and resolving the incident. Incident response personnel must determine
whether or not to discontinue operations. All information about the compromised
systems and networks must be acquired and secured. Information collection must also
be on going throughout the incident response process. Containing an incident provides
a short-term security solution until sufficient information about the incident is acquired
for resolving and recovering from the incident.

Refer to 3.1.2 Networked systems security policy for more information on policy
considerations that may be needed to be in place to underpin some of the following
steps.

5.1 Exercise precautions to limit the extent of the incident

Depending on the incident you are containing, you may need to decide whether or not
to shut systems down or disconnect from the network. These decisions require
management involvement. However, if you have an intruder whose activities are
malicious, your system administrators or Incident Response personnel may need to
take more immediate action. This is why the decision-making process and the level of
authority need to be articulated in your policies and made operational in your
procedures. Determine whether you need to:

• shutdown the system;


• isolate the affected system(s) (disconnect from the network);
• disable system services (eg r commands);
• change passwords or disable accounts;
• monitor system and network activities.

Refer to your Forensic Plan for more detail on what information may be needed for
evidence before proceeding. You may need to capture and record system information

Department of the Premier and Cabinet – Office of e-Government 42


Incident Response Plan

that may be lost or not captured during the execution of your backup procedure, or lost
if the system is shutdown.

5.1.1 Temporarily shut down the compromised system


You need to temporarily shut down a compromised system when there is no other
means by which to deny an attacker or intruder access to the system. This action will
prevent more serious damage and give you time to perform a more detailed analysis.
However, be aware that by doing this you also deny access to legitimate users of the
system.

Shutting down a compromised system may also destroy important information that you
need for analysis. For example, an intruder’s programs may only be available in main
memory because he or she deleted the files on the hard drive to avoid their detection.
In an emergency, you need to be able to weigh the risk of further damage against the
possibility of losing this information by shutting down.

5.1.2 Disconnect the compromised system from the network


Alternatives to shutting down a compromised system are to:

• disconnect the local area network or corporate network to which the system is
connected from the local or public networks that an intruder is using; or

• disconnect the compromised system from the local network.

The first alternative ensures that the system and all communication with it is still
available from within your agency. However, this may alter critical information on the
compromised system as normal use continues, and will prevent your users accessing
the local or public networks you have disconnected from.

The second alternative allows your users of systems other than the compromised
system to continue to access the local or public networks. However, if you incorrectly
determined the scope of an intrusion (that may include other systems on the same local
network), you run the risk of an intruder continuing to have access to that network and
the remaining systems connected to it, which may also be compromised.

5.1.3 Disable system services, if possible


If you have performed sufficient analysis to correctly limit the scope of an intrusion to
specific services, you should disable them. Disabling services is especially necessary if
no patch is immediately available to eliminate the vulnerability than the intruder used.
By disabling only the specific services used by the intruder, you can continue to provide
users access to all other services on the system.

This is where the network information you documented on the services each machine is
running comes in handy. Refer to 3.2.2 Establish standard build procedures

Department of the Premier and Cabinet – Office of e-Government 43


Incident Response Plan

5.1.4 Change passwords or disable accounts


Although this is not a totally reliable containment action (an intruder may have other
means of access), disabling the accounts or changing the passwords associated with
the accounts that an intruder is using will terminate that access path if the account was
the primary means used to gain access (versus the use of a vulnerability that bypassed
user authentication).

5.1.5 Monitor system and network activities


Regardless of what other steps are taken, you need to monitor all system and network
activities for unusual and suspicious events. Watch the network to detect attacks
previously used by an intruder, and connections coming from systems you know to be
used by an intruder.

Using this approach, you can identify subsequent intruder access attempts more quickly
and more easily. This may also reveal other access paths not previously identified
which allows you to disable these access paths.

5.1.6 Verify that redundant systems and data have not been compromised
If you are running hot, warm, or cold backups in support of your configuration
redundancy policy (if you have one), you need to ensure that an intruder’s actions have
not been replicated, thereby affecting those systems and data.

In addition, ensure that any containment, elimination and restoration steps you take on
the primary system are also taken on your backup systems including the elimination of
vulnerabilities that were used by the intruder to gain access.

5.2 Collect and protect information about the compromised systems


and networks

Great care should be taken to collect all necessary information about the compromised
system(s) and the cause of the incident, as they will likely be lost when cleaning up the
system. All information about the compromised system(s) and cause(s) of an incident
needs to be captured and securely stored. This may include system and network log
files, network message traffic, user files, results produced by intrusion detection tools,
analysis results, system administrator console logs and notes, and backup tapes that
capture the before-incident and after-incident states of the affected system. All
information must be carefully collected, labelled, catalogued, and securely stored at
each stage of your incident analysis.

This is important because if you do not collect and protect this information, you may not
be able to learn from the experience and improve your systems, their operation, and
your staff capabilities. If the incident is internally generated, you may not be able to
take appropriate action to educate, reprimand, or terminate an employee found
responsible.

Department of the Premier and Cabinet – Office of e-Government 44


Incident Response Plan

If you intend to prosecute the perpetrator of an incident, you need to have complete,
thorough, and convincing evidence that has been protected through a secure chain of
custody procedure that tracks who has been involved in handling the evidence and
where it has been stored. Refer to your Forensic Plan for more information about the
chain of custody.

5.2.1 Collect all information related to the incident


Collect information about all relevant system and network logs from the compromised
system(s), including written log records made by incident response staff, any other
auditing information produced by tools, full backups, partial backups (snapshots),
screen dumps, videotapes, and photographs.

Document all information in a notebook that addresses the questions who, what, where,
when, why, and how. This includes:

• name of system;
• date/time of each entry;
• what actions were taken;
• what was said;
• who was notified;
• who had access;
• what data was collected;
• what information was disseminated, to whom, by whom, when, and for what
purpose;
• what was submitted to legal counsel, to whom, by whom, and how it was verified
(eg notarised).

It is important to be aware that your notes may be subject to subpoena in any legal
proceedings, so document responsibly. Make sure you use a separate notebook for
each incident so that if it is subpoenaed it doesn’t contain information about other
incidents.

5.2.2 Collect and preserve evidence


Designate a person to be the point of contact who is responsible for maintaining liaison
with law enforcement and other external agencies. To ensure that your evidence will be
acceptable to the legal community, its collection should be done following predefined
procedures in accordance with laws and legal regulations. Refer to your Forensic Plan
for more details on preserving evidence.

Department of the Premier and Cabinet – Office of e-Government 45


Incident Response Plan

5.2.3 Contact law enforcement if you decide to pursue and prosecute


Depending on the nature of the incident, you may decide to contact law enforcement to
help you deal with this incident. This is especially advised if you think a crime may have
been committed, and need further clarification or advice on this. In the first instance,
please contact your WACSIRT contact Gail Holt, who will then liaise with the
appropriate authorities.

If you choose to keep your systems running and connected to collect more information
about an intruder and an intrusion (as contrasted with disconnecting or shutting down
your systems), you may be liable if your system is used as a launch point to attack
another site. You need to have law enforcement involved ASAP, and take action as
they advise to limit this liability.

5.3 Backup the system

Information you collect and interpret through analysis is key to your decisions and
actions throughout the response process. It is extremely important to obtain a full
backup, preferably using disk imaging of the compromised system, with known good
binaries. In specific instances or for critical environments, the backups may be used to
reinstall the compromised system on another host in a controlled environment (an
‘isolation ward’) for further investigation, while the original host is preserved as
evidence.

5.3.1 Avoid potentially compromised code


Intruders may install Trojan horses and other malicious code in system binaries, so it is
not advisable to run any programs on the affected system. Unless you can check that
the cryptographic fingerprint of the binary your want to run (in this case the backup), and
validate that it has not been altered, you should run any programs from your own read-
only media, such as CD-ROM or write-protected floppy disk.

This is where the cryptographic checksums of critical system files generated as part of
your preparation to respond is useful. Refer to 3.4.5 Generate information required to
verify systems and data integrity

5.3.2 Backup
Do not allow the system to be altered in any way until you have completed a full system
backup. Make at least two full backups of the system(s) that have been identified as
compromised, as well as the user data on those systems, preferably using disk imaging
of the system with known good binaries. Do so using hardware-write-protectable or
write-once media.

Use new media for your backups because it may appear that the evidence is faulty or
contaminated if it is written over old information. Besides, the old information is not part
of the incident and it may not be appropriate for it to be handed over to an external
party.

Department of the Premier and Cabinet – Office of e-Government 46


Incident Response Plan

5.3.3 Safely store backups


To protect the backup as evidence, preserve one of the backup media in a secure
location and preserve the chain of custody. This copy should not be used for any
operational or analysis tasks. The other may be reinstalled on other test systems for
further analysis, and the data may be used for system recovery.

Department of the Premier and Cabinet – Office of e-Government 47


Incident Response Plan

Eradication

The goal of this phase is to eliminate or mitigate the factor(s) that resulted in a
compromise of system security.

After an incident has been contained, steps must be taken to completely resolve the
incident. This requires a thorough investigation of the compromised systems and
networks to determine the attack method used and damage that may have been
caused. To exercise caution and ensure the preservation of evidence, it may be
necessary to reconstruct the compromised systems on a test environment for
investigating the incident. All system backups used to reconstruct the compromised
systems must be protected to ensure the preservation and admissibility of evidence.
Information acquired about the compromised systems and networks must be
consolidated and analysed. The attacks and potential vulnerabilities used to
compromise the systems and networks may then be identified and eliminated, and the
damage resolved.

Compromised systems and networks must be restored using trusted backups and
reliable programs. After recovery, incident response personnel must verify the integrity
of the restored systems and networks using audit information generated during the
Preparation phase. To exercise caution, signs of the incident on similar systems must
also be corrected. Security and detection mechanisms currently installed must also be
strengthened to prevent further incidents.

Refer to 3.1.2 Networked systems security policy for more information on policy
considerations that may be needed to be in place to underpin some of the following
steps.

In the event that an incident evolved into a crisis situation, you may also (thru the
WACSIRT) decide to communicate with the WA Computer Crime Investigation Unit
and/or AusCERT to help monitor and resolve the incident.

6.1 Analyse all information about the incident

Information collected during the Detection and Containment phases may be sufficient to
determine the root cause of the incident. However, in most incidents there are multiple
causes for a compromise. For example, the absence of adequate technical controls
may result from:

• failure to adequately select and train system administrators;


• lack of written security policies and procedures;

Department of the Premier and Cabinet – Office of e-Government 48


Incident Response Plan

• absence of management emphasis;


• failure to proactively keep abreast of vendor patches;
• failure to apply fixes to known vulnerabilities in a timely manner;
• any combination or all of these reasons.

For this reason, you should conduct a comprehensive review of the data gathered, and
not just assume that one factor alone contributed to the incident. When analysing
incident information, remember that you are looking for the answers to the questions
what, where, when, why, how, who? Information gathered might also reveal other
systems that have been compromised.

6.1.1 Isolate the compromised system(s)


This can be accomplished by:

• transferring one copy of the backup files to a test system that is isolated from
your operational systems and restoring the compromised system(s) on the test
system for analysis; or

• disconnecting the compromised system(s) and performing analysis on those


systems directly, keeping in mind that this will destroy the original source of
information.

6.1.2 Search on other systems for signs of compromise


Intruders regularly establish more entry points into a network once they have gained
initial access. Whenever an attack or an intrusion is detected, you need to check all
other systems that are “similar” to the system that was accessed. “Similar” can have
various meanings depending on your operational environment, including:

• systems that are in the same IP address range or are on the same network
segment. Intruders perform scans across large ranges of IP addresses to locate
security vulnerabilities;

• systems that are in the same “trusted” domain. These systems provide access to
users from other systems within the same domain without further authentication;

• systems that have at least one network service in common. Intruders often
check for well-knows services such as DNS (Domain Name System), FTP (File
Transfer Protocol), HTTP (Hyper-Text Transfer Protocol), and SMTP (Simple
Mail Transfer protocol); and

• systems that have the same operating system.

Why this is important. If you find other systems that have been compromised, this
must be documented as part of the incident. All the steps in the Detection,

Department of the Premier and Cabinet – Office of e-Government 49


Incident Response Plan

Containment, and Eradication must be iterated for each system compromised until there
are no further signs of the incident.

6.1.3 Examine logs generated by firewalls, network monitors, and routers


Attacks often leave trace information that can lead you to the system that was used (or
abused) by an intruder. Such traces include log and audit files, files left behind by an
intruder, or information about the use of servers and services on other systems used in
moving thru the network. This trace information can be used to search for other events
or connections originating from that system that were previously unnoticed. In this way,
you can identify other systems to which an intruder may have gained access.

Firewall, network monitor, and router logs often remain intact and contain valuable
information even if an intruder gains local access, manages to get administrator
privileges, and deletes the local system logs to hide the information about the attack
method, an intrusion itself, and the access methods used. These programs and
devices, if properly configured, may record connections and message traffic generated
by an intruder. The logs that they each produce may reveal intruder activity. For
example, firewall logs can be configured to store information about the source of any
connection, the destination, and characteristics of connections such as the amount of
data transferred.

By analysing this information (including date, time, and systems attacked) you can
locate related records in these logs that reveal more detailed information about an
intrusion or information missed altogether. Look for similar connections, including
connections from the same source or going to the same destination.

Building a complete picture can be a tedious task. Log formats of many systems are
not compatible and no general purpose tools currently exist that chronologically
synchronise logs produced by multiple systems. However, there are monitoring tools
that will collect information from multiple systems and consolidate the logs produced by
those systems. In addition, the timing source and time zone used by systems may
differ. Protocols such as NTP (Network Time Protocol) can be used to synchronise the
time for multiple systems.

6.2 Identify the attack method used and the damage caused
6.2.1 Identify the attacks used to gain access to your systems
You cannot fix the security problem that led to a system compromise if you don’t know
what happened. You need to isolate the attack and determine how it was executed.

Search the local logs of the compromised systems for information that reveals what kind
of attack was used in addition to making use of logs produced by firewalls, routers, and
network monitors. Usually, intruders attempt a number of different attacks, or scan for
the presence of one particular vulnerability before they gain initial access. Depending

Department of the Premier and Cabinet – Office of e-Government 50


Incident Response Plan

on how your systems are configured to detect signs of intrusion, your system and
network logs may contain:

• denied access messages if an intruder tried to guess passwords;


• messages pointing to old vulnerabilities; or
• blocked accesses to specific services collected by an installed tool, such as TCP
Wrapper.

You may also find date, time, and source in the logs that can be quite useful.

After gaining access, an intruder may try to delete all logs or specific log entries.
Therefore, it is possible that you will not find any useful entries. Sometimes the deletion
of log entries is detectable. If so, this notifies you that something suspicious has
happened to your system and you need to conduct further analysis. This is where it is
extremely useful to have a duplicate log on a logging server separate to the
compromised system.

6.2.2 Identify what an intruder did while accessing your systems


You need to understand how an intruder attacked and gained access to your systems.
However, this understanding will not reveal what an intruder actually did once access
was gained. Without more information, you cannot identify what damage was done to
systems and data. On many systems, you can easily identify attempts to write and
modify files while, unfortunately, read access (of potentially sensitive data) is rarely
captured due to the high volume of such access. Without any further information, you
should assume that once intruders gain access, they are able to obtain any data and
access any service or program on the compromised system. That is, you need to
assume the worst case for the purpose of assessing damage.

To identify what an intruder did, you can:

• analyse various log files;


• compare cryptographic checksums of known, trusted files to those on the
compromised machine;
• use other intrusion detection and analysis tools.

Having a trusted cryptographic checksum with which to compare is particularly


important to detect if an intruder modified your operating system kernel. Examples of
traces that intruders may leave behind include:

• changes to log files to hide their presence;


• actions to modify a system utility to not list processes started by an intruder to
protect against easy recognition of installed back doors;

Department of the Premier and Cabinet – Office of e-Government 51


Incident Response Plan

• alteration of system binaries and files;


• installation of attack programs;
• Trojan horses, back doors, or new versions of system commands.

6.3 Eliminate the means of attack and vulnerabilities, and restore


the system

Deciding what to do to remove the cause of an incident is often a challenge. Depending


on what has occurred, the actions below provide some high-level guidance.

If your incident is a network intrusion, eradication is more difficult.

6.3.1 Virus infections


In the case of a virus, eradication simply requires removing the virus from all systems
and media (e.g. floppy disks), usually with virus eradication software. Check that your
systems are running the most up-to-date virus patterns, and that this virus will be
cleaned successfully with your virus eradication software. Check with your software
vendor for recommended Service Pack upgrades or patches that you may need to apply
to your systems in order to protect yourself from the vulnerability that this virus
exploited.

6.3.2 Other malicious code infestations


Commercial software may exist to eradicate common or “in the wild” malicious code.
Most commercial programs will identify computer viruses, well-known Trojan horses,
and certain worms – all forms of malicious software for the Windows and Macintosh
environments. For Unix environments there are few commercial programs available.
However, in most cases detection and removal tools have been released into the public
domain by dedicated professionals to address the influx of Trojan horses and worms
that have appeared in Unix systems within the last few years. You should familiarise
yourself with all commercial and free detection/eradication software available to you in
anticipation of an infection.

6.3.3 Network intrusion


Complete eradication of the root cause(s) of an intrusion is a long-term goal that can
only be achieved by implementing an ongoing security improvement process. In
response to a specific intrusion, you need to ensure that the affected systems are
protected against the same or similar types of access and attacks in the future, once
you have contained this intrusion and systems have been returned to normal operation.

A successful intrusion usually occurs in two parts. The first phase is when a
vulnerability is exploited and the system is accessed. The second phase is that once in,
the intruder typically installs back doors or other means for obtaining future access to
the compromised system, or sniffer programs to collect additional passwords and user

Department of the Premier and Cabinet – Office of e-Government 52


Incident Response Plan

ids. The following steps describe how to eliminate this access to the greatest possible
extent. These steps are likely to be sufficient if you have prepared properly, performed
thorough analysis, and if you are able to identify all changes made by an intruder.

If you are not prepared, then your only option is to reinstall the systems and restore all
user data to the extent possible. You may wish to reinstall the systems in any event to
ensure that all intruder changes are eliminated, or if you are in doubt of the accuracy of
the information you relied on to identify the system changes.

Why this is important. Intruders widely advertise compromised systems and regularly
trade system addresses and access information in exchange for attack tools or similar
information. Victims of intrusions indicate that intruders try to access their system
addresses long after the original intrusion has occurred. Such addresses are not only
distributed amongst intruders, but also are available in databases on the Internet.

You need to ensure that your systems are no longer vulnerable before returning them to
service. Otherwise, they can be compromised by the same kind of attack that resulted
in the successful intrusion. The absence of elimination actions undermines your ability
to operate securely and, in effect, negates all efforts you expend to respond to the
original intrusion.

6.3.4 Change all passwords


Change all passwords on all systems to which the attacker may have had access. This
step should be taken in all cases, however it is essential when there is evidence that an
intruder may have had access to your password file or used a password sniffer tool that
captures user passwords transmitted in clear text on the network.

You need to cater for the fact that doing this will create a fair amount of work for users
and may confuse them. If passwords cannot be changed, an alternative is to lock out
the compromised accounts and reassign new accounts.

6.3.5 Restore from backups or reload the entire system


If you are insufficiently prepared, you probably do not have access to information such
as cryptographic checksums to verify the authenticity of the operating system version
used on the compromised system. In this event, you need to reinstall the system.

A root kit attack embeds so much malicious code in so many hidden places in a
computer system that cleaning the system is almost never a viable option. If there is
evidence of a root kit style attack, it may be better to avoid restoring from backups. You
should completely wipe the disk and rebuild the operating system from original
distribution media.

If you decide to restore the system from backups, it is essential to first determine the
integrity of the backup itself. You will need to restore from the most recent backup
made before the system was compromised. You need to make every effort to ensure

Department of the Premier and Cabinet – Office of e-Government 53


Incident Response Plan

that you are not restoring compromised code. If in doubt, reinstall the system from
original distribution media or copies you trust.

If you have reinstalled the system from distribution media, you must then patch the
vulnerability, configure the operating system safely, patch the operating system with the
latest patches, and reload the data.

6.3.6 Remove any means for intruder access


Remove any means for intruder access, including changes made by an intruder. Use
your analysis results to determine how the intruder gained access and eliminate them.
Examples include:

• known vulnerabilities for which patches are available; install the patches.

• the presence of malicious code (back door, Trojan horse) with an unknown and
hidden function such as deleting log files or starting a service on unused ports to
permit access without requiring a password; reinstall a trusted version of the
affected software.

• adding new users to the list of authorised users including levels of system
privileges up to full privileges; reinstall a trusted version of your authorised user
file.

• setting new passwords for existing users; reinstall a trusted version of your
password file.

• weak or inadequate procedures (e.g. password setting and aging); update the
procedures and enforce them.

• corrupted configuration files containing previously disabled entries (such as a


rshd daemon entry within the inetd.conf configuration file on Unix systems);
correct them.

• inspect all files executed at boot time for the presence of Trojan horses or back
doors; reinstall trusted versions of files executed at boot time.

6.3.7 Restore programs and binary files


Restore executable programs (including application services) and binary files from
original media or trusted deployment methodology. You need to identify all files that
have been added, changed, or deleted by an intruder because you cannot determine
what hidden functions may have been left behind or what critical functions may have
been influenced by these changes. For executable files (including non-binary files such
as Unix shell scripts) and other binary files (such as libraries), verify cryptographic
checksums to ensure that the files were not compromised. Refer to 3.4.5 Generate
information required to verify systems and data integrity.

Department of the Premier and Cabinet – Office of e-Government 54


Incident Response Plan

If you possess detailed logging or audit information about files an intruder accessed and
programs an intruder executed, this step is made easier by allowing your staff to
concentrate on specific files instead of all existing and deleted files.

6.3.8 Restore user data from trusted backup media


An intruder may have altered user data and application program areas. Examples
where this may occur include:

• installing back doors to provide future access. For example, an intruder installs a
program in a local user directory that is called each time the user logs in,
providing an unprotected login shell that can be accessed by anyone via the
Internet;

• compromising user data to sabotage the user’s work. For example, an intruder
makes small changes to spreadsheets that go unnoticed. Depending on how the
spreadsheets are used, this can cause minor to major damage.

Use the latest trusted backup to restore user data. For files that have not been
compromised, you can consider using the backup that was made closest in time to
when an intrusion was detected to avoid user rework. This should be done with caution
and is based on having a high level of confidence that restored user files were not
compromised. Regardless, you need to encourage users to check for any unexpected
changes to their files and warn them about the risk of compromise.

Users should review all restored data files that resided on the compromised system to
ensure they were not affected by an intruder’s activities.

All executables or binary files residing in user areas should be handled in the same way
as system executables and binary files. If no authenticated, cryptographic checksum is
available, you need to reinstall from the original distribution media.

6.3.9 Review system configurations


System configuration files include the following types of data:

• user accounts
• system services and their configuration
• audit and monitoring facilities
• access control lists

Compare all system configuration files with authoritative copies of these files or
compare cryptographic checksums with trusted checksums collected before the
incident.

Department of the Premier and Cabinet – Office of e-Government 55


Incident Response Plan

If authoritative copies are available, you may choose to overwrite the current set of files
with these copies.

In the absence of trusted copies or checksums, you need to review all files manually.
Clearly, this will take significant time and resources. The review can be effectively
conducted as a peer review performed by multiple system and network administrators
or by members of your response team.

6.4 Verify that the means of attack and potential vulnerabilities are
correctly eliminated

In response to a specific incident, you need to ensure that the affected systems are
protected against the same or similar types of access and attacks in the future, once an
incident is contained and systems are returned to normal. The use of automated
vulnerability assessment tools can assist you not only to validate the correctness of
eradication procedures, but also to anticipate and correct factors that might facilitate an
attack.

6.4.1 Perform system vulnerability analysis


Use a system vulnerability tool to determine whether configuration and software
versions at your site need to be updated. You can do this by executing public domain
or vendor tools yourself, or arrange for external experts to check for the presence of
known vulnerabilities using their own tools.

Both host-based and network-based assessment tools can determine the robustness of
your system and network configurations. Host-based tools provide a higher degree of
accuracy than do network-based tools. Increasingly, host-based tools such as bastille,
available at http://www.bastille-linux.org or the tools from the Center for Internet Security
http://www.cisecurity.org/ have the capability to actually correct or to strengthen the
security of a system. Network-based tools such as nmap, available at
http://www.insecure.org/nmap or nessus, available at http://www.nessus.org can
identify, and in some cases stress, common vulnerability exposures that may have
resulted in a compromise. A combination of tools is beneficial because each tool may
have unique capabilities.

6.4.2 Search for related vulnerabilities


Ensure that other platforms within your agency are not vulnerable to the factor(s) that
allowed the compromise. Perform your system vulnerability analysis to search for other
potentially vulnerable systems on other platforms. This is where your computing
resources inventory (see 3.4.9 Build an inventory of computing resources) comes in
handy. Having a centralised inventory and standard build procedures (see 3.2.2
Establish standard build procedures) will allow you to focus your attention and
resources productively when searching out and correcting related vulnerabilities.

Department of the Premier and Cabinet – Office of e-Government 56


Incident Response Plan

6.5 Strengthen security and detection mechanisms

Depending on your environment, many procedures may be outside the control of the
compromised system’s owner. For example, typically in large agencies, firewalls and
router configurations are the responsibility of another organisational entity. In this case,
your IR team should serve as the liaison between the system’s owner and these entities
to ensure adequate protection.

6.5.1 Improve protection mechanisms to limit network and system


exposure
All protection mechanisms (such as firewalls) should be reviewed and their
configurations adjusted based on what you have learned in responding to your incident.
Pay particular attention to the following aspects.

• Determine if protection mechanisms need to be configured differently, such as


changing or adding new IP addresses to the router filter for allowing or
disallowing connections;

• Determine if protection mechanisms need to be placed in a new or additional


location on your network that was previously unprotected or insufficiently
protected; and

• Review available information on vulnerabilities, patches, and new versions of


your protection mechanism software, ensuring that your configurations are up to
date.

6.5.2 Improve detection mechanisms to enable better reporting of attacks


Update your detection mechanisms, such as intrusion detection systems and other
types of intrusion reporting tools, to ensure that similar attacks are detected by these
mechanisms in the future. Perform the following analysis and take appropriate actions.

• Determine if detection mechanisms need to be configured differently, such as


adding in new attack patterns to be detected, or changing logging options;

• Determine if detection mechanisms need to be placed in a new or additional


location on your network that was previously insufficiently covered;

• Review available information on vulnerabilities, patches, and new versions of


your detection mechanism software, ensuring that your configurations are up to
date; and

• Review and update the conditions under which your detection mechanisms
generate alerts to system and network administrators and the forms in which the
alert is made (email, phone, pager, printouts etc).

Department of the Premier and Cabinet – Office of e-Government 57


Incident Response Plan

Recovery and Closure

The goal of this phase is to return the affected systems and networks to operational
status.

After an incident is completely resolved, the compromised systems and networks that
were isolated must be restored to operational status to allow business continuity.
Incident response personnel must determine how to restore compromised systems and
networks with minimum risk of incident recurrence and impact to current operations. All
systems and networks must also be continually monitored to prevent incident
recurrence. After recovering business operations, your legal counsel may advise on
whether to report the incident to law enforcement.

7.1 Determine the requirements and timeframe for resuming normal


operations

You need to fully determine the requirements to be met and the priority they should
have before you return the affected system to normal operations. This determination
requires input from both the management of the affected system and their system
administrators.

If the requirements do not include completion of incident analysis and the eradication of
vulnerabilities due to a business-critical need to return the system to operations as soon
as possible, continue analysis in parallel and upgrade the system as soon as possible.
In this case you must recognise that the system is vulnerable to another occurrence of
the same type of incident until you do so. One way to mitigate the risk is to increase the
level of monitoring and intrusion detection to ensure that a new intrusion does not go
unnoticed.

Your agency’s networked systems security policy should specify the order in which
system services are returned to normal operation. Refer to 3.1.2 Networked systems
security policy for more information on policy considerations that may be needed to be
in place to underpin some of the following steps.

7.2 Enable system and application services

Enable previously disabled services that are either known as not vulnerable or have
been corrected to eliminate the vulnerability used by an intruder.

Only enable those services that are required by the users of the system (as contrasted
with enabling all available services).

Department of the Premier and Cabinet – Office of e-Government 58


Incident Response Plan

7.3 Reconnect the restored system to the network

Reconnect the restored system to its local network. In the case where a local or
corporate network containing the compromised system was previously disconnected
from the public network used by an intruder, reconnect the local or corporate network.

7.4 Validate the restored system

Management and users want to know whether the problem has actually been
eradicated. Once the system has been restored, verify that the operation was
successful and the system is back to its normal condition. Ideally you have a system
test plan to evaluate the system. More commonly, the system is run through its normal
tasks while being closely monitored by a combination of techniques such as network
loggers and system log files. Take notice that sometimes patches or techniques used
to prevent a vulnerability may cause the system to function differently than it did before
the incident.

7.5 Monitor for any suspicious activity

You need to watch for any scans, probes or other suspicious activity that may signal the
return of an intruder. Back doors and malicious code can be very well hidden. Once
your system is back online, continue to monitor for backdoors that escaped detection
and eradication.

In many cases, you can gather much more information about an intruder when they
attempt to return than you can immediately after an intrusion. This is particularly true if
you have installed improved monitoring tools and procedures as a result of lessons
learned from a previous intrusion.

Monitor for failed login attempts, attempts to access back doors, attempts to re-exploit
the original vulnerability, and attempts to exploit new vulnerabilities. Each occurrence of
these should be analysed further.

Once your system has been compromised and this becomes known in the hacker
community, the system becomes a bigger target for future attacks. Improved monitoring
may reveal new attacks more easily and provide the opportunity to defeat the attack.

7.6 Notify all parties that the incident is closed

As part of the Detection phase, you will have notified all parties that needed to be aware
of the incident. All these parties now must be informed that the incident has been
resolved and closed.

Department of the Premier and Cabinet – Office of e-Government 59


Incident Response Plan

Follow-up

The goal of this phase is to review the incident response process to identify and
implement security lessons learned. It is important that this phase feeds information
directly back into the Preparation phase so that lessons learned as you responded to an
incident result in a more secure operational environment.

As soon as possible after business operations are recovered, all involved parties should
hold a post-mortem analysis and review of the incident response process to identify
what worked well and what did not. Experience must be captured quickly. A follow-up
report, including lessons learned, is the accepted method of protecting the knowledge
so that it can be used in the future.

An executive summary of the incident, including costs and impacts, should be prepared
for management so as to assist them in assessing the cost of reconstituting information
assets and computing resources damaged by the incident. This assessment conveys
the value of an incident response capability, and provides information for revising the
incident response plan and the security policy of the organisation.

Why this is Important. If you do not learn from the experience of responding to a
successful intrusion, you will continue to operate at risk and will likely have to deal with
the same or similar type of incident again. Every successful intrusion indicates
weaknesses in your systems, networks, and operations that provide opportunities to
make them more secure. It may also point to an inadequate level of staff preparedness
that can be remedied through additional training or other forms of skill development.

8.1 Hold a post-mortem analysis and review meeting with all


involved parties

It is best to do this within 3 to 5 working days of completing the incident investigation,


otherwise participants are likely to forget critical information.

Capture the following information:

• Did your detection and response processes and procedures work as intended? If
not, where did they not work? Why did they not work?

• Methods of discovery and monitoring procedures that would have improved your
ability to detect an intrusion;

Department of the Premier and Cabinet – Office of e-Government 60


Incident Response Plan

• Improvements to procedures and tools that would have aided you in the
response process. For example, consider using updated router and firewall
filters, placement of firewalls, moving the compromised system to a new name or
IP address, or moving the compromised machine’s function to a more secure
area of your network;

• Improvements that would have enhanced your ability to contain an incident;

• Correction procedures that would have improved your effectiveness in recovering


your systems;

• Updates to policies and procedures that would have allowed the response and
recovery processes to operate more smoothly;

• Topics for improving user and system administrator preparedness;

• Areas for improving communication throughout the detection and response


processes.

Make a monetary estimate of the costs associated with an intrusion to support the
business case for the level of investment you should make in security improvements.
This should include the estimated value of proprietary or sensitive information assets
that may have been accessed by an intruder.

Document and review the meeting results.

Prepare a final report. Prepare an Incident Executive Summary and brief these results
to senior management for their review and comment. This ensures they are aware of
the vulnerability of their systems and continue to be educated about these issues.

8.2 Conduct a lessons learned meeting

Distribute the incident report in advance so that the meeting can focus on recounting the
incident and identifying what lessons have been learned.

8.3 Send recommended changes to management

Provide management with a prioritised set of recommended changes from the lessons
learned process, along with cost estimates, high-level schedule, and impact of
implementing or not implementing the recommended actions.

Department of the Premier and Cabinet – Office of e-Government 61


Incident Response Plan

8.4 Revise security plans, policies, procedures, and user and


administrator training to prevent incident recurrence

Include any new, improved methods resulting from lessons learned in your current
security plans, policies, and procedures.

Make sure that you are regularly reviewing the public, legal, and vendor information
sources necessary to protect your systems from further attacks of this type. These
sources regularly report current intruder trends, new attack scenarios, and tools that will
improve the effectiveness of your response process.

8.5 Determine whether or not to perform a new risk analysis based


on the severity and impact of an incident

This is a decision that management would need to make after reviewing the Executive
Incident Summary report and the recommended changes from the lessons learned
process.

8.6 Take a new inventory of your system and network assets

You may need to generate new information required to verify the integrity of your
systems and data. Review your Preparation phase step 3.4.5 Generate
information required to verify systems and data integrity.

8.7 Participate in investigation and prosecution, if applicable

Refer to your Forensic Plan for more information about evidence handling.

Department of the Premier and Cabinet – Office of e-Government 62


Incident Response Plan

Acknowledgements

1. The SANS Institute. “Computer Security Incident Handling V2.3.1”


March 2003

2. B Fraser. “Site Security Handbook” RFC 2196 September 1997


URL:http://www.ietf.org/rfc/rfc2196.txt

3. Kossakowski, Klaus-Peter et al. “Responding to Intrusions”. February 1999

4. Howard J, Longstaff T. “A Common Language for Computer Security Incidents”


October 1998 URL: http://www.cert.org/research/taxonomy_988667.pdf

5. Smith, D. AusCERT, “Forming an Incident Response Team”


URL: http://www.auscert.org.au/render.html?it=2252

6. IT Audit & Consulting (ITAC) Pty Limited. Strategy paper for the establishment of
a Whole of Government Incident Response Team. 2000

7. Defence Signals Directorate’s Information Security Group ISIDRAS Security


Incident Categories. URL: http://www.dsd.gov.au/infosec/services/isidras2.html

8. Keith J. Jones. “Incident Response”. November 2001, Volume 26, Number 7 of


;login: The Magazine of USENIX and SAGE.
URL: http://www.usenix.org/publications/login/2001-11/pdfs/jones1.pdf

9. Standards Australia. “Guidelines for the management of IT evidence”. HB 171-


2003

10. Information was also drawn from various papers and documents freely available
on the Internet.

Department of the Premier and Cabinet – Office of e-Government 63


Incident Response Plan

Appendix A Glossary

access – establish logical or physical communication or contact.

account – a domain of user access on a computer or network, which is controlled


according to a record of information which contains the user’s account name, password
and use restrictions.

action – a step taken by a user or process in order to achieve a result, such as to


probe, scan, flood, authenticate, bypass, spoof, read, copy, steal, modify, or delete.

artefact – instances of malicious code or other file remnants left behind by intruders.
Examples include Trojan horse programs, Ethernet sniffer log files, password files,
exploit scripts, and other program source code.

attack – a series of steps taken by an attacker to achieve an unauthorised result; any


attempt to gain knowledge of or penetrate a system; includes scanning, probing,
mapping, and all adverse events described under incident.

attacker – an individual who attempts one or more attacks in order to achieve an


objective.

authenticate – present an identity of someone to a process and, if required, verify that


identity, in order to access a target.

authorised – approved by the owner or administrator.

autonomous agent – a means of exploiting a vulnerability by using a program, or


program fragment, which operates independently from the user. Examples are
computer viruses or worms.

breach – same as intrusion.

bypass – avoid a process by using an alternative method to access a target.

chain of custody – verifiable documentation that indicates the sequence of individuals


that have handled a piece of evidence and the sequence of locations where that
evidence has been stored, including dates and times. For a proven chain of custody to
occur, the evidence is accounted for at all times.

component – one of the parts that make up a computer or network.

Department of the Premier and Cabinet – Office of e-Government 64


Incident Response Plan

computer – a device that consists of one or more associated processing units and
peripheral units, that is controlled by internally stored programs, and that can perform
substantial computations, including numerous arithmetic operations, or logic operations,
without human intervention during execution. Note: May be stand alone, or may
consist of several interconnected units.

contingency plan – documentation describing the actions needed to allow business


operations to continue if a primary facility, personnel, systems, networks, etc are unable
to operate. A contingency plan is most commonly put into effect during events such as
power outages, unexpected system shutdowns, fires, and other natural disasters.

copy – reproduce a target leaving the original target unchanged.

corrective action – an action taken during or after an incident to prevent further


attacks, repair damage, or punish offenders.

corruption of information – unauthorised alteration of data on a computer network.

configuration vulnerability – a vulnerability resulting from an error in the configuration


of a system, such as having system accounts with default passwords, having “world
write” permission for new files, or having vulnerable services enabled.

corporate raiders – employees who attack computers of competitors for financial gain.

CSIRT – Computer Security Incident Response Team

data – representations of facts, concepts, or instructions in a manner suitable for


communication, interpretation, or processing by humans or by automatic means. Data
can be in the form of files in a computer’s volatile or non-volatile memory, or in a data
storage device, or in the form of data in transit across a transmission medium.

data in transit – data that are being transmitted across a network, or otherwise
emanating from a source. Examples include packet data travelling across the Internet,
and the data content of electromagnetic radiation surrounding a computer terminal or
network cable.

data tap – a means of monitoring the electromagnetic radiation emanating from a


computer or network using an external device.

delete – remove a target, or render it irretrievable.

denial of service – intentional degradation or blocking of computer or network


resources.

design vulnerability – a vulnerability inherent in the design or specification of hardware


or software whereby even a perfect implementation will result in a vulnerability.

Department of the Premier and Cabinet – Office of e-Government 65


Incident Response Plan

disclosure of information – dissemination of information to anyone who is not


authorised to access that information.

distributed pool – a tool that can be distributed to multiple hosts, which can then be
coordinated to anonymously perform an attack on the target host simultaneously after
some time delay.

DNS – Domain Name System.

ending data – the date of the last known incident activity.

event – an action directed at a target, which is intended to result in a change of state


(status) of the target.

file – a collection of data, which is designated by name and treated as a unit by a user
or process.

flood – access a target repeatedly in order to overload the target’s capacity.

FTP – File Transfer Protocol.

hackers – attackers who attack computers for challenge, status, or the thrill of obtaining
access.

HTTP – Hyper Text Transfer Protocol.

implementation vulnerability – a vulnerability resulting from an error made in the


software or hardware implementation of a satisfactory design.

incident –Any real or suspected adverse event in relation to the security of computer
systems or computer networks.

incident handling – actions taken to protect and restore the normal operating condition
of computers and the information stored in them when an adverse event occurs;
involves contingency planning and contingency response.

incident response – same as incident handling.

incident number – a reference number used to track an incident, or identify incident


information.

increased access – an unauthorised increase in the domain of access on a computer


or network.

Department of the Premier and Cabinet – Office of e-Government 66


Incident Response Plan

information exchange – a means of obtaining information either from other attackers


(such as through an electronic bulletin board), or from the people being attacked
(commonly called social engineering).

internetwork – a network of networks.

intrusion – any intentional event where an intruder gains access that compromises the
confidentiality, integrity, or availability of computers, networks, or the data residing on
them.

IP – Internet Protocol.

ISP – Internet Service Provider.

modify – change the content or characteristics of a target.

network – an interconnected or interrelated group of host computers, switching


elements, and interconnecting branches.

number of sites – the overall number of sites known to have reported or otherwise to
have been involved in an incident.

NTP – Network Time Protocol.

objective – the purpose or end goal of an incident.

other sites – the site names of sites known to have been involved in an incident, but
that did not report the incident.

physical attack – a means of physically stealing or damaging a computer network, its


components, or its supporting systems (such as air conditioning, electric power, etc).

probe – access a target in order to determine its characteristics.

process – a program in execution, consisting of the executable program, the program’s


data and stack, its program counter, stack pointer and other registers, and all other
information needed to execute the program.

professional criminals – attackers who attack computers for personal financial gain.

read – obtain the content of data in a storage device, or other data medium.

reporting date – the first date that the incident was reported to a response team or
other agency collecting data.

reporting sites – the sites names of sites known to have reported an incident.

Department of the Premier and Cabinet – Office of e-Government 67


Incident Response Plan

scan – access a set of targets sequentially in order to identify which targets have a
specific characteristic.

script or program – a means of exploiting a vulnerability by entering commands to a


process through the execution of a file of commands (script) or a program at the
process interface. Examples are a shell script to exploit a software bug, a Trojan horse
login program, or a password-cracking program.

site – the organisational level with responsibility for security events; the organisational
level of the site administrator or other authority with responsibility for the computers and
networks at that location.

site name – the portion of the fully qualified domain name which corresponds to a site.

SMTP – Simple Mail Transfer Protocol.

sniffer – any hardware or software device that monitors network traffic. The traffic can
be stored and archived for later viewing. Intruders commonly use sniffers to capture
userid and password data that are passed in clear text over a network.

spies – attackers who attack computers for information to be used for political gain.

spoof – masquerade by assuming the appearance of a different entity in network


communications.

starting date – the date of the first known incident activity.

steal – take possession of a target without leaving a copy in the original location.

target – a computer or network logical entity (account, process, or data) or physical


entity (component, computer, network or internetwork).

taxonomy – a classification scheme that partitions a body of knowledge and defines the
relationships among the pieces. It is used for classifying and understanding the body of
knowledge.

TCP - Transmission Control Protocol.

theft of resources – unauthorised use of computer or network resources.

tool – a means of exploiting a computer or network vulnerability.

toolkit – a software package which contains scripts, programs, or autonomous agents


that exploit vulnerabilities. An example is the widely available toolkit called rootkit.

Department of the Premier and Cabinet – Office of e-Government 68


Incident Response Plan

terrorists – attackers who attack computers to cause fear for political gain.

unauthorised – not approved by the owner or administrator.

unauthorised result – an unauthorised consequence of an event.

user command – a means of exploiting a vulnerability by entering commands to a


process through direct user input at the process interface. An example is entering Unix
commands through a telnet connection, or commands at an SMTP port.

vandals – attackers who attack computers to cause damage.

voyeur – attackers who attack computers for the thrill of obtaining sensitive information.

vulnerability – a weakness in a system allowing unauthorised action.

WORM – Write Once, Read Many

WWW – World Wide Web

Department of the Premier and Cabinet – Office of e-Government 69


Incident Response Plan

Appendix B Computer and


Network Incident Taxonomy

Department of the Premier and Cabinet – Office of e-Government 70


Incident Response Plan

Computer and Network Incident Taxonomy

|……………………………………………………………………… incident ……………………………………………………………………… |


| |
| |………………………………………………… attack(s) ………………………………………….. | |
| | | |
| | | | | |
| | |……………… event ………… | | |
| | | | | |

Attackers Tool Vulnerability Action Target Unauthorised Objectives


Result
Hackers Physical Design Probe Account Challenge,
Increased Status, Thrill
Attack
Access
Spies Implementation Scan Process Political
Information
Exchange Disclosure of Gain
Configuration Flood Data Information
Terrorists Financial
User
Command Corruption of Gain
Corporate Authenticate Component Information
Script or Damage
Raiders Denial of
Program Bypass Computer
Professional Service
Criminals Autonomous
Agent Spoof Network Theft of
Vandals Resources
Toolkit
Read Internetwork
Voyeurs
Distributed
Copy
Tool
Data Tap Steal

Modify

Delete

Department of the Premier and Cabinet – Office of e-Government 71


Incident Response Plan

Appendix C Incident Numbering


Scheme
The WACSIRT incidents are numbered #yyyy-nn where yyyy is the year and nn (or
nnn!) is a sequential number starting from 1 for the first incident of the year. So incident
#2004-13 is incident number 13 that occurred in 2004.

You may wish to devise your own incident numbering scheme, use the WACSIRT
scheme within your own agency, or notify WACSIRT when an incident is detected and
ask them to allocate a number from their records. Regardless of what numbering
scheme you decide to use, I encourage you to notify the WACSIRT of your incidents so
that they can build a central repository of incidents that occur across WA Government
agencies.

Also refer to your Forensic Plan, section 5.1 Property / Exhibit Numbering Scheme that
details how equipment of a computer forensic nature is subject to a standardised
numbering scheme, which is based around an incident number.

Department of the Premier and Cabinet – Office of e-Government 72


Incident Response Plan

Appendix D Types of Incidents

The term "incident" encompasses, but is not limited to, the following examples of
adverse events (based on those defined by AusCERT).

• Electronic theft of proprietary/confidential information


• Computer facilitated fraud
• Unauthorised access to, or modification of, system or data files
• Website defacement
• Disruption or denial or service through electronic means (DoS or DDos)
• Interception of telecommunications data
• Virus, Worm, or Trojan infection
• Malicious probes or scans (including war dialling and war driving)
• Hoax or scam
• Prohibited content
• Other

Department of the Premier and Cabinet – Office of e-Government 73


Incident Response Plan

Appendix E Security Incident


Categories

The following incident categories are based on those used by the Defence Signals
Directorate, Information Security Incident Detection, Reporting and Analysis Scheme
(ISIDRAS). I have included some additional incident types into the categories based
upon incident investigations undertaken by the WACSIRT.

Category 1

Category 1 incidents include events that cannot definitively be identified as attacks, and
have no effect on system operation, such as:

• Isolated and non-repeated scans or pings from an external uncontrolled network.


• Virus/Trojan/worm detected and removed prior to being placed on an operational
system or network.
• Inappropriate content on a machine.
• Abuse of privileges or password confidentiality by Agency employee (not
extending to superuser or root or administrator privileges).

Category 2

Category 2 incidents also have no effect on system operations, and comprise identified
but unsuccessful attempts to actively breach an information system security policy.
Such events include:

• Multiple repetitions of category 1 incidents.


• Repeated active probes or port mapping from an external network.
• Attempt to gain unauthorised access to Agency resources, either from within or
outside of an Agency network.
• Unsuccessful DOS/DDOS attempts (blocked at firewall or router).
• Virus/trojan/worm found on a single system which has been successfully
contained or removed.

Category 3

Category 3 incidents include any successful attempt to actively breach an information


system security policy on a single system, and may result in a minor or moderate effect
on system operations. These could include:

Department of the Premier and Cabinet – Office of e-Government 74


Incident Response Plan

• Unauthorised access acquired by one or more unauthorised people to any


account, at any access level.
• Abuse of privileges or password confidentiality by an Agency employee,
extending to superuser, root or administrator privileges.
• Unauthorised use of a system for the transmission, processing, or storage of
data.
• Unauthorised collection of information for facilitating future intrusion attempts or
espionage purposes.
• Virus/Trojan/worm found on more than one system, or an inability to contain and
remove the code from a single system.
• Defacement, alteration or deletion of web server files.
• A successful attack against system services, for example, NIS, DNS, NFS, email,
WWW, etc, including denial of service attacks.
• A prank or hoax perpetrated from an external uncontrolled network.
• Unauthorised access to a firewall.
• Unauthorised modification to system files and system access controls.
• Unauthorised modification to system hardware or software without the owner’s
knowledge or permission.
• State or National security compromises and disclosures arising from accidental
or deliberate breaches of security policy.
• Equipment loss whether it is theft or accidental.

Category 4

Category 4 incidents include any situation in excess of the examples given above,
particularly where high-level intervention or crisis management is required. Such
incidents will usually have a major effect of system operations. This category also
includes the use of any system or network resources for criminal activity, and those
incidents of criminal activity (eg fraud, prohibited content) resulting in escalation to law
enforcement.

Department of the Premier and Cabinet – Office of e-Government 75


Incident Response Plan

Appendix F Log of Events Form

Department of the Premier and Cabinet – Office of e-Government 76


Incident Response Plan
Department of the Premier & Cabinet

LOG OF EVENTS
Incident Number: _______________
Sheet: _______

Date Time Actions Taken Comments/Reason Action Taken Who Took Action Signature

Department of the Premier and Cabinet – Office of e-Government 77


Incident Response Plan

Appendix G Incident Reporting


Form
<This form is in the final stages of design… it will be inserted here when complete>

Department of the Premier and Cabinet – Office of e-Government 78


Incident Response Plan

Appendix H Contact List

Insert your agency’s contact list here

Department of the Premier and Cabinet – Office of e-Government 79


Incident Response Plan

Appendix I Network Diagrams and


Information

Insert your agency’s network diagrams and information here

Department of the Premier and Cabinet – Office of e-Government 80


Incident Response Plan

Appendix J ServiceNet Incident


Response Procedure

Department of the Premier and Cabinet – Office of e-Government 81


Procedure Number:
SNET 021.

Procedure Name:

Incident Response
Procedure
REVISION HISTORY
PROCEDURE FOR INCIDENT RESPONSE

Date Rev. Status or Amendment Prepared Approved


30/03/03 0.1 First Issue Angelo Giaros
INCIDENT RESPONSE PROCEDURE

1. Purpose

The purpose of this document is to detail the site procedure of what actions are to be taken
when an incident occurs or is identified/discovered at ServiceNet. This document covers or
provides references for all actions when such an event occurs.

2. Responsibilities

The person responsible for implementing a procedure change or creation must provide
details of the new procedure to all other ServiceNet staff and to clients who may be affected
by the change. It is the responsibility of the person implementing change to ensure the
authenticity of information reported as part of this procedure. ‘DTF Online Services’ and the
engineer’s responsibilities are further defined in respect to this procedure.

Any person that updates this document must complete the Revision History Table at the
start of this document.

3. Procedure

3.1 Required Knowledge

• Thorough understanding of the ServiceNet environment


• A good understanding of Network topologies
• Excellent understanding of Security
• Thorough understanding of the Windows / Unix Operating System

3.2 Incident Management Defined

In ITIL terminology, an incident is defined as, ‘any event which is not a part of the standard
operation of a service and which causes, or may cause, an interruption to, or a reduction in,
the quality of service’.

It is important that all incidents are reported to the Help Desk and logged in every instance.

As the Incident Management process is mostly reactive, the primary goal is to restore
normal service operation as quickly as possible and minimise the adverse impact on ‘Online
Services’ operations.

The Engineer who first attends to the incident is responsible for the resolution process and
final reporting. The Engineer attending to the incident is also responsible for it’s Escalation
Process, as per the ‘Rules of Engagement’.
3.3 Escalation & Control of information

Control of information during the course of an incident or investigation of a possible incident


is very important. Once escalated to the ServiceNet Project Manager, it is then the
responsibility of the ServiceNet Project Manager to monitor progress and escalate as per the
agreed ‘Rules of Engagement’ escalation table. When an agreed time interval has elapsed,
the ServiceNet Project Manager is responsible for the ‘functional escalation’ of transferring
the incident to a Senior Engineer (specialist).

3.4 Incident Response Defined

Incident Response is defined to ensure System Application, Infrastructure, Communication


faults and the potential for breaches in Security are acted upon accordingly by ServiceNet
staff. An Incident Response can be identified by:

• Loss of Internet Services


• A virus has infected a system
• Denial of service or inability of one or more users to login to an account
• Network Communications failure
• System crashes
• Any intrusion detection and potential exposure to intrusion
• Any attempt to obtain access to secured or restricted systems – including manual or
automated attempts – regardless of their success or otherwise.
• Loss or provision (deliberate or otherwise) of information (passwords, network diagrams
etc.) to unauthorised parties.
• Any event not listed but identified by support staff or management as a threat to – or
breach of – security.
3.5 Incident coordination team members

Alphawest Help Desk

support.centre@alphawest.com.au 1800 647 716

Responsibility: Provides a single point of contact for all ServiceNet customers and DFT
users; and to facilitate the restoration of normal operational service within agreed service
levels and business priorities.

ServiceNet Technical Support


Managed Service Team 1800 647 716
Backup/Escalation: ServiceNet Project Manager 0417 014 052
Responsibility: Technical Administrators for all ServiceNet Infrastructure. These members
can detect incidents and can take technical measures to limit damage.

Alphawest Site Manager


ServiceNet Project Manager 0417 014 052
Backup/Escalation: National Operations Centre Manager 9429 6000
Responsibility: Coordination of technical resources and escalation of incidents.

Manager, Online Services (Dept. Treasury & Finance)


Jack Hondros (Tel. 9222 5327)
Backup: Sean Quintal (Tel. 9222 5139)
Responsibility: Notification & Authority for all incidents, as appropriate.

Director, Online Services (Dept. Treasury & Finance)


Steve Price (Tel. 9222 5433)
Backup: Jack Hondros (Tel. 9222 5327)
Responsibility: Handles interfaces to the media, public statements & co-ordinates
department communications.
3.6 Follow-up Analysis

After an incident has been addressed and all systems have been restored to a normal mode
of operation, a follow-up postmortem analysis should be performed.
Performing the follow-up is one of the most critical activities in responding to incidents. It
helps improve the incident handling procedures. Follow-up analysis is dependent upon the
history of the actual incident itself.

3.7 Incident Documentation

All Incidents need to be fully detailed. The logs of every task, discussion, action and
escalation made during the incident, by every party involved in the incident, needs to be
collated and checked for accuracy and documented.

The current ServiceNet SLA requires an Incident Report to be submitted within 24 hours of
an incident occurring. If the service is not restored within 24 hours, a progress report is
required daily. All progress report updates are to be appended as a historical event at the
end of the initial Incident Report, until the service is restored and a final copy submitted.

3.8 Incident Logging

The following is an overview of the procedures associated with Fault Resolutions.

1. The Alphawest Support Centre contact details as follows:


a. e-mail address: support.centre@alphawest.com.au.
b. Phone: 1800 647 716
2. During core business hours (7am-7pm), the Alphawest Support Centre has the
ultimate responsibility and accountability for the coordination of all fault resolutions.
a. The Alphawest Support Centre will escalate critical faults to the ServiceNet
Project Manager.
b. If the fault has not been addressed within the hour, the Alphawest Support
Centre will escalate critical faults to the ServiceNet Project and is to advise all
parties as required.
3. Outside core business hours (7pm-7am), the Alphawest Support Centre will forward
to a paging service, which will notify the ServiceNet on-call engineer.
4. Any work request must have an associated Alphawest Support Centre Call Reference
number.
5. Any phone calls or e-mails that are received direct from the client must be forwarded
to the Alphawest Support Centre (support.centre@alphawest.com.au) so that they
can be logged accordingly.
6. It is the responsibility of the ServiceNet engineers to action and update the HelpDesk
calls within SLA requirements with appropriate update comments.
7. With any specialised services outside, The ServiceNet Project Manager will establish
which team should be involved and forward the call.
3.9 Incident Escalation Table

Outage duration Contact Advise next working


day
Less than 30 mins On call engineer SN Project Manager
DTF Rep NOC Delivery Manager
Greater than 30 less than On call engineer NOC Delivery Manager
1 hour DTF Rep/Project Manager Account Manager
Greater than 1 hour On call engineer Account Manager
DTF Rep/Project Manager Strategic Business
NOC Delivery Manager Manager
Operations Director

3.10 Client Contact details

Client contact information can be obtained by using the Internet to access the following
page http://www.servicenet.wa.gov.au/cgi-bin/logon.pl.
The page provides both business hours and out of hours contact details for all monitored
agencies.

3.11 Related Procedures

• Snet018.doc – Security Breach Event


• Fault Resolution Procedures for resolving communications link failures

ServiceNet Infrastructure Fault


Core Hours Mon-Fri 7am-7pm
From ServiceNet Hardw are Fault
7am - 7pm

ServiceNet Notified
Refer 7.2
NMC
Responds AlphaWest Help Desk &
and SN support Notif ied
investigates via automated e-mail

No
NMC
Call NMC Contacts
ServiceNet Contacts 120 Mins Senior
Hardw are Fault Yes Call Resolved No Yes Call Resolved
Rep Supplier elapsed Product
Refer 7.8 Ref er 7.6 Eng.
Ref er 7.8

No

No
No
Yes
Call
Operating ServiceNet NMC 60 Mins
Yes Call Resolved No
Softw are f ault Rep investigates elapsed No
Refer 7.8

Yes

No
Eng. Notif ies
Ops Man
Refer 7.8

Call Resolved

Ops manager liases


w ith ServiceNet
Yes Representitive
Yes

Yes

To ServiceNet Link Fault


7am to 7pm
Refer Call Closed
Ser viceN et e vent or
interuption to ser vice.

O nC all S erviceN et Lin k F au lt


O ncall Engineer
M o n -F ri 7pm -7am , S at & S un
Eng
investigates Alarm R ec ieves
page

No
15 Min C all
Yes
Elapsed resolved

Ser viceN et N otified


O n-C all D oes c lient R efer 7.2
Eng. w ant to b e Yes O perations Man ager
Attends site contacted? Inform ed
R efer 7.8

Send e-m ail


No
notific ation

No
No

Eng Lo gs
Link Pro vider 60 Mins
Yes call with Yes Yes C all resolved
res pons ibility elaps ed
Pro vider

No
No

No

Yes

D oes client C ontact C lient, the y


C lient Eng S tands
w ant to b e Yes acc ept resposibility? Yes Yes C all resolved
R es ponsibility D ow n
c ontac ted? R efer 5

No Send e-m ail


No notific ation

F rom Go To
Infras tructure Infras tructure
H ardw are Yes
F ault 7pm to F ault 7pm to
7am Procedure 7am Procedure

Yes

No

C all resolved C AL L C L O SED


N MC Ad vises c lient Yes
(R efer 7.2 )and
Yes M anager(s) (R efer 7.8)
C all D etails logged and
call closed
No

N MC
Escalates to
O ps Man
R efer 7.8
End of Document

You might also like