You are on page 1of 325

HSE Auditing:

Fundamentals, Skills
and Techniques for
Team Members

NPC Course Manual

2005

Arthur D. Little Limited


Science Park, Milton Road
Cambridge CB4 0XL
Telephone 01223 392090
Fax 01223 420021

Reference 20365
Copyright © 2005 by Arthur D. Little Limited. All rights reserved.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 2
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Table of Contents

Chapter Page
No
Overview of Environmental, Health, and Safety Auditing 5
Audit Approach 12
Basic Steps in the Typical Audit Process 23
Pre-Audit Activities 25
On-site Opening Activities 42
Understanding HSE Management Systems 55
Effective Interviewing 75
Preparing Working Papers 105
Assessing Strengths and Weaknesses 135
Gathering Audit Evidence 161
Sampling Strategies 189
Evaluating Audit Results 220
Writing Audit Findings 242
Post Audit Activities 260

Appendix A – Confirmation Letter


Appendix B – Roles and Responsibilities of the Audit Team
Appendix C – Guide to Acronyms

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 3
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
List of Exercises

Exercise Page
No
1 Understanding HSE Management Systems 69

2 Effective Interviewing - Difficult Interview Situations 88

3 Effective Interviewing - Conducting Interviews 93

4 Preparing Working Papers 126

5 Assessing Strengths and Weaknesses 152

6 Gathering Audit Evidence - 177


Developing Verification Strategies

7 Sampling Strategies 1 190

8 Sampling Strategies 2 200

9 Evaluating Audit Results - 230


Specific ‘Local’ vs ‘Report’

10 Writing Audit Findings - 255


Critiquing Audit Findings

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 4
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Overview of Health, Safety and Environmental
Auditing

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 5
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Overview of Health, Safety and Environmental
Auditing

Recognising auditing as a powerful tool for managing and


communicating HSE performance, organisations around the
world are developing audit programmes to:

• Provide assurance to corporate officers about the company’s


compliance status with HSE requirements and good industry
practices.

• Assess potential HSE liabilities.

• Demonstrate effective management of HSE obligations to


companies’ key stakeholders.

Audit Programme

Annual Report Audit Report

Community CEO and Management

Investors/Shareholders Facilities

An increasing number of companies are including status reports


on their HSE audit programmes in their annual reports and/or
annual environmental reports (e.g., Union Carbide, ARCO
Chemical, NOVA, Deere & Company, WMX, DuPont, etc.).

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 6
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Overview of Health, Safety and Environmental
Auditing

Moreover, there are many sound business benefits of performing


HSE audits.

Improve public image

Increase awareness and


understanding of HSE hazards

Reduce employee and community


To address exposure hazards
Why Audit? stakeholder
needs Improve compliance

Reduce costs by operating efficiently


and safely

Reduce exposure to fines

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 7
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Overview of Health, Safety and Environmental
Auditing –
What is Health, Safety and Environmental Auditing?

Environmental Protection Hazard Management

Management Tool

Employee Health and


Compliance Management
Safety

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 8
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Overview of Health, Safety and Environmental
Auditing –
Definition of Health, Safety and Environmental
Auditing

Auditing has become recognised throughout the world by various


organisations as a useful tool in managing HSE issues. Although
a variety of definitions have been established for HSE auditing,
they all share common elements and themes. For example,
auditing has been defined as…

• A systematic, documented, periodic, and objective review by a


regulated entity of facility operations and practices related to
meeting environmental requirements. (U.S. EPA Policy
Statement on Environmental Auditing, July 1986)
• Internal evaluations by companies and governmental agencies
to verify their compliance with legal requirements as well as their
own internal policies and standards. (Environment Canada, May
1988, Environmental Protection Act, Enforcement and
Compliance Policy)
• A series of activities undertaken on the initiative of an
organisation’s management to evaluate environmental
performance. (International Chamber of Commerce)
• An activity directed at verifying a site’s or organisation’s
environmental, health, or safety status with respect to specific,
predetermined criteria. (U.S. Environmental Auditing
Roundtable)

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 9
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Overview of Health, Safety and Environmental
Auditing –
Definition of Health, Safety and Environmental
Auditing

…and the definition of auditing has broadened over the years to


include the evaluation of management systems in determining a
facility’s HSE performance status.

• A systematic, documented verification process of objectively


obtaining and evaluating evidence to determine whether
specified environmental activities, events, conditions,
management systems, or information about these matters,
conform to audit criteria, and communicating the results of this
process to the client. (International ISO 14000 Standard)
• A management tool comprising a systematic, documented,
periodic, and objective evaluation of the performance of the
organisation, management system, and processes designed to
protect the environment with the aim of: facilitating management
control of practices which may have impact on the environment;
and assessing compliance with company environmental policies.
(Eco-Management and Audit Scheme—European Union)
• A systematic evaluation to determine whether or not the
environmental management system and the environmental
performance it achieves conform to planned arrangements, and
whether or not the system is implemented effectively, and is
suitable to fulfill the organisation’s environmental policy and
objectives. (ISO)

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 10
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Overview of Health, Safety and Environmental
Auditing –
Standards Against Which to Audit

Since auditing has emerged as a systematic process intended to


verify compliance with established standards and, in some cases,
to review the effectiveness of management systems, it tends to
be most effective for those HSE issues that are well defined by
specific audit criteria.

Audit Criteria

Corporate/Division Facility Standard


Policies and Operating
Procedures Procedures
(SOPs)
Laws and
Regulations
(Federal/National,
State/Provincial,
and Local)

Best Management Management


Practices Systems

Increasingly, management system requirements are being


incorporated into the scope of the audit in recognition that well-
designed and well-implemented management systems are an
important vehicle for maintaining compliance over time.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 11
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Audit Approach

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 12
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Audit Approach

Most companies engage in a three-phased audit process to fulfill


the objectives of the audit programme.

Pre-Audit Activities

Three-Phased On-Site Activities


Audit Approach

Post-Audit Activities

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 13
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Audit Approach

Pre-Audit Activities

The pre-audit activities are designed to ensure that the audit


team members and facility personnel understand the audit
process, and their roles and responsibilities within that audit
process, and are prepared to implement them.

Pre-Audit Activities Outcome

1. Initial planning
activities Identification of key site
issues
2. Document review
Preparation of detailed
plans
3. Audit plan preparation

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 14
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Audit Approach

On-Site Activities

Arthur D. Little has developed a well-defined and systematic five-


step process to provide organisation and structure to the on-site
activities. This five-step process facilitates a review of how a
facility manages its HSE obligations.

Basic Step On-Site Activities Outcome

Opening meeting Strong working knowledge


Step 1: Understand Tours of key systems on site
Management Systems Initial Interviews Identification of key issues
Document review to review
Review of Step 1 Develop verification
information strategies
Step 2: Assess Team meetings Reallocate team resources,
Strengths &
if required
Weaknesses of
Management Systems Identify potential impacts
and management system
weaknesses
Physical inspections Analyse site programs
Focused interviews Develop evidence to
Step 3: Gather Audit substantiate findings
Data and records
Evidence
examination Confirm status of
Verification testing compliance
Review data collected Prepare draft findings
Review factual accuracy of Confirm accuracy
Step 4: Evaluate Audit findings
Results Identify potential root
Analyse/integrate findings of causes
team
Daily debrief meetings Early, clear, consistent
Close-out meetings communication
Step 5: Report Audit
Understand facility concerns
Findings
Prepare preliminary draft
report

In addition, the five-step process has been recognised by:


• Hundreds of companies
• Environmental Auditing Roundtable (EAR)
• Canadian Environmental Auditing Association (CEAA)
• International Chamber of Commerce in its Guide to Effective
Environmental Auditing
• European Community in the Eco-Management and Audit
Scheme (EMAS)

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 15
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Audit Approach

Two Components of HSE Auditing

There are two important components in completing the five-step


process:
• Assessment—Process to develop an opinion (judgment) on
the strengths and weaknesses of the activities under review.
• Verification—Process to determine adherence to specific
standards.

Although assessment- and verification-based techniques play a


key role in the audit process, they each provide the auditor with
different information.

Principal Activities
What you look for Output
Deficiencies, problems, risks, Professional opinion as to
conformance with good practice performance with regard to
Assessment
(performed during Steps 1, 2 and 4) accepted practice and
recommendations for improvement

Evidence of compliance with Statement of performance against


regulations, policies, and standards with identification of
procedures (performed during Verification shortcomings/areas for
Step 3) improvement

Auditors need to use a mix of assessment and verification to


complete the five-step process. Some skills are better suited
toward assisting an auditor in the assessment stage while others
suit the verification stage.

Assessment is dependent upon the auditor’s knowledge of:


• Site operations
• Management systems (e.g., policies, procedures, etc.)
• HSE requirements
• Environmental technology

Verification is dependent upon the auditor’s knowledge of:


• Auditing skills and techniques
• HSE regulations
• Internal standards

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 16
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Audit Approach

The balance between assessment and verification will vary


depending on where you are in the five-step process.

Five-Step Audit Process

Step 1: Understand
Management Systems

Step 2: Assess Strengths and


Weaknesses of Management Systems

Step 3: Gather Audit Evidence

Step 4: Evaluate Audit Results

Step 5: Report Audit Findings

Assessment-related activities Verification-related activities

Based on the information gathered, the auditor may move


forward in the process or may need to reassess the information
gathered in previous steps. For example, results obtained during
gathering audit evidence (Step 3) or evaluating audit results
(Step 4) can lead an auditor back to reassess his/her
understanding of the management systems in place and/or the
strengths and weaknesses of those management systems. The
resulting finding may be one that identifies gaps in the facility’s
management systems or the proximate or root cause of the
finding.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 17
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Audit Approach

Audit Skills, Techniques, and Tools

In implementing the five-step audit process, there are some


essential audit skills, techniques, and tools that can be utilised to
increase on-site efficiency and effectiveness.

Audit Skills and Techniques

Conducting interviews

Documenting the audit (working papers)

Gathering audit data

Using sampling strategies

ls Writing audit findings


T oo
A udit
s
ocol
Prot d i t
a u
Pre- naires Communicating audit results
s t i on es
que i on guid
u s s
Disc

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 18
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Audit Approach

Post-Audit Activities

The purpose of the post-audit activities is to ensure that:


• Audit results are communicated to the facility and appropriate
levels of management.
• Audit findings are addressed.

Post-Audit Activities Outcome

1. Develop report Audit report


(performance status)

2. Distribute report

Corrective action plan


3. Develop and implement
corrective actions

4. Track corrective Status reports


actions

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 19
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Audit Approach -
Context for Health, Safety and Environmental
Auditing

Although HSE auditing has been recognised as a prominent and


useful HSE management tool, it is not the only means by which
an organisation manages its HSE issues. Auditing is one of
many key activities within the typical HSE management
processes.

Typical HSE Management Process

Assessing Planning Implementing Reviewing


Identifying/managing Influencing Managing compliance Measuring
issues issues/requirements Managing significant performance
Identifying/ Setting policy risks Assuring performance
understanding direction Preventing/reducing Communicating
requirements Establishing unwanted impacts performance
Identifying/evaluating performance Remediating past
risks standards/guidance damages
Obtaining needed Responding to
permits/approvals emergences
Improving value to Auditing
final customers

Supporting

Training & awareness Documenting / record keeping Managing information

Sound HSE management processes should include activities that


address all four HSE management processes (assessing,
planning, implementing, and reviewing) and incorporate the three
key supporting activities (training and awareness,
documenting/recordkeeping, and managing information). These
same management processes are typically present at facilities in
order to meet their HSE obligations.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 20
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Audit Approach -
Context for Health, Safety and Environmental
Auditing

Auditing can provide a basis for guiding, measuring, and


evaluating the performance of a facility’s HSE management
activities. In general, an audit looks at and evaluates the
management processes to comply with HSE requirements for
each functional area being reviewed (e.g., air pollution control,
hazardous waste management, industrial hygiene, employee
safety, etc.). The specific activities that will be audited will
depend upon the programme objectives. For example:

Typical HSE Management Process

Assessing Planning Implementing Reviewing


Identifying confined Developing entry Issuing entry permits Reviewing cancelled
spaces procedures Monitoring confined permits
Identifying jobs/tasks Establishing rescue spaces Supervisor review of
procedures Using personal confined spaces
protective equipment

Supporting

Training & awareness Documenting / record keeping Managing information


- Training for entrants, - Permit retention - Training database
attendants, & supervisor - Training records
- Written programme

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 21
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Audit Approach –
Conducting Effective Audits

Conducting effective and efficient audits is dependent upon:

• Utilising the team’s collective experience and knowledge of


the site operations, HSE standards, and auditing skills and
techniques.

• Prioritising the topics to review in terms of importance/impact.

• Basing your prioritisation on a thorough review and


assessment of management systems and controls.

• Developing verification strategies to gather data that will


provide meaningful insights regarding compliance.

• Communicating, communicating, communicating—both oral


and written communications that occur within the team and
externally with affected parties are essential factors
influencing the success of auditing efforts.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 22
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Basic Steps in the Typical Audit Process

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 23
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Basic Steps in the Typical Audit Process –
Overview

Pre-Audit Activities On-Site Activities Post-Audit Activities

Step 1: Understand Prepare Draft Report


Management Systems
Conduct opening meeting Obtain Review Comments from
Select & Schedule Facility Audit Conduct orientation tour Corporate HSE
Review audit strategy Law department
Understand details of Facility management
management systems

Step 2: Assess strengths Issue Final Report to


& Weaknesses Facility management
Select Team Members & Confirm Consider potential impacts Operations
their Availability Evaluate management Corporate HSE
systems Law department
Set priorities for verification

Step 3: Gather Audit Evidence


Plan the Audit: Evaluate what needs to be
Correspond with the done Develop Action Plan
facility Determine depth & rigor of Develop proposed action(s) to
Assemble & distribute review address each finding
background information Select types of evidence Assign responsibility for
Assign & communicate needed & methods to gather corrective action
audit responsibilities them Develop timetable
Conduct pre-audit Compare practices against
meeting requirements
Document results

Conduct Follow-up
Step 4: Evaluate Audit Results Track status of corrective
Evaluate audit results actions
Write audit findings Confirm closure of findings

Step 5: Report Audit Findings


Conduct exit meeting

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 24
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Pre-Audit Activities

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 25
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Pre-Audit Activities

These are primarily the responsibility of the team leader, and


have therefore been treated in detail later in this manual. It is
important to remember that the team participates in the pre-audit
preparation, by reviewing the background information to develop
an initial understanding of the facility’s operations, modifying
audit protocols as/if necessary, making their own travel
arrangements, having liaised with the team leader on the time for
the pre-audit team meeting, obtained and reviewed applicable
regulations, and organising any materials or equipment
necessary to perform the audit.

Before the audit team arrives at the opening meeting, it should


know enough about the site to be able to formulate some
preliminary hypotheses about the major risks and HSE issues.
There are several ways to accomplish this:

• Auditors’ basic familiarity with company operations, policies,


and procedures

• Pre-audit questionnaire

• Other background material provided by the facility

• Conversations between the team leader/members and plant


personnel

• Review of applicable regulations

In our experience, it is not uncommon for team members to begin


their pre-audit preparation only on the plane that is taking them to
the site. Even if this were adequate for team members, it is
definitely not sufficient for team leaders.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 26
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Pre-Audit Activities

Pre-Audit Activities On-Site Activities Post-Audit Activities

Step 1: Understand Prepare Draft Report


Management Systems
Conduct opening meeting Obtain Review Comments from
Select & Schedule Facility Audit Conduct orientation tour Corporate HSE
Review audit strategy Law department
Understand details of Facility management
management systems

Step 2: Assess strengths Issue Final Report to


& Weaknesses Facility management
Select Team Members & Confirm Consider potential impacts Operations
their Availability Evaluate management Corporate HSE
systems Law department
Set priorities for verification

Step 3: Gather Audit Evidence


Plan the Audit: Evaluate what needs to be
Correspond with the done Develop Action Plan
facility Determine depth & rigor of Develop proposed action(s) to
Assemble & distribute review address each finding
background information Select types of evidence Assign responsibility for
Assign & communicate needed & methods to gather corrective action
audit responsibilities them Develop timetable
Conduct pre-audit Compare practices against
meeting requirements
Document results

Conduct Follow-up
Step 4: Evaluate Audit Results Track status of corrective
Evaluate audit results actions
Write audit findings Confirm closure of findings

Step 5: Report Audit Findings


Conduct exit meeting

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 27
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Pre-Audit Activities

Pre-Audit Activities

Administrative Planning

Schedule audit Assemble & review


Select team members background information &
Contact & coordinate with the applicable regulations
facility Develop audit assignments &
Arrange for travel areas to focus on
Gather & distribute audit Review & discuss audit team
materials (reference responsibilities
documents, working paper
pads)

Goal
Hit the ground running

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 28
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Pre-Audit Activities

The pre-audit activities provide the foundation upon which the


team members build their understanding of the facility’s
operations and establish an audit strategy. They also provide the
facility with its first impressions of the audit programme and set
the tone for audit team/facility interactions. The pre-audit
activities should meet the following objectives:

• Provide the audit team with sufficient plant information to


enable the team members to develop a basic understanding
of the facility, the processes, and the HSE management
systems employed. For example, the audit team should
understand:

− The type of facility being audited (e.g., chemical


manufacturing, injection molding, pulp and paper,
distribution center, research laboratories, etc.).

− The employee population (e.g., 60-person plant or 1,000-


person plant, extent of contract employees and contractors
used on site, business organisation).

− The general applicability of regulatory requirements to the


facility’s operations (e.g., presence of wastewater
treatment plant, hazardous waste generator status,
number of permitted air sources, applicability of industrial
hygiene programmes [e.g., respiratory protection, hearing
conservation, bloodborne pathogens, etc.], presence of a
fire brigade and/or spill response team, availability of
routine and emergency medical personnel, etc.).

• Inform the facility as to audit programme goals, objectives,


and procedures. Typically, the more information facility
personnel receive prior to the audit, the less anxious and
more comfortable they will feel about the on-site portion of the
audit.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 29
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Pre-Audit Activities

Using the information obtained, allow the audit team to develop a


basic audit strategy prior to arrival at the site. In developing an
audit strategy, the audit team should consider the following:
• Audit objectives to be achieved, areas to receive emphasis,
and a preliminary division of responsibility among the team
members.
• Questions or issues that need to be resolved during the
preliminary meetings and points to be clarified.
• A preliminary agenda and schedule to be used during the
audit.

To help facilitate the conduct of an efficient and thorough audit,


the following pre-audit activities are frequently undertaken by the
audit team leader and team members.

1. Corresponding With the Facility

Within the designated time frame established by the audit


programme, the team leader should contact the facility to confirm
the exact dates of the planned audit and to address the following:

• Audit process and activities

• Types of documents to be reviewed:


− Pre-audit information request
− Pre-audit questionnaire

• Planning/logistical details:
− Safety and security requirements
− Administrative/logistical details
− Initial interview schedule

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 30
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Pre-Audit Activities

Audit Process and Activities


In general, the team leader should explain the audit process and
the types of activities to be undertaken while on site. The audit
programme objectives, purpose, and scope, and the types of
interviews and tours that will be conducted, should be explained.
In addition, the team leader typically requests that the facility
prepare a short presentation describing the plant operations for
the opening meeting. The facility should have a general
understanding of how the audit will proceed and what type of
time commitment will be required for the various levels of staff
affected by the audit.

Types of Documents to be Reviewed


It is important to determine the background information to be
requested and reviewed before arriving at the facility.

A protocol represents a plan to be used by the auditors in


conducting an audit. Protocols are produced in advance, for
each of the issues to be audited, sometimes annotated with
relevant regulatory standards references.The purpose is to
provide the audit team with a step-by-step guide to collecting
evidence about a facility’s programmes and practices included
within the scope of the audit. In addition, the protocol identifies
selected topics and requirements included in the audit and
provides guidance regarding how the team may audit or review
against those requirements.

However, it typically does not include all applicable performance


requirements that the team may need to review a facility’s
compliance status. Typically, audit team members review
applicable performance requirements prior to the audit to
determine whether any other requirements are appropriate for in-
depth review and verification.

Other purposes of an audit protocol include:

• Tool for audit planning.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 31
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Pre-Audit Activities

• Record for audit procedures—planned and completed.

• Outline for working papers.

• Record of changes in audit scope, procedures, etc., and the


rationale for any changes.

• Basis for reviewing/critiquing an individual audit.

Pre-audit information request. There are some materials that will


be essential to have prior to the audit while other materials will
only need to be looked at while on site. For some materials (e.g.,
training records, inspection logs, material safety data sheets,
etc.), it is in the best interest of the audit team to see this material
during the on-site visit. The team leader would be better advised
to ask the plant to leave this information in existing files until the
audit. As a result, the audit team will be able to see actual
recordkeeping conditions and practices. The tables on the
following pages list the types of information that are typically
requested prior to the audit and information that should be
available on site for review during the audit.

The information listed in these tables serves only as an example.


The type of documentation that should be requested will depend
upon the staffing resources utilised (e.g., full-time corporate
auditors, corporate or facility personnel with other full-time
responsibilities, third-party auditors, etc.) and the time these
resources realistically have for pre-audit preparation.

When conducting pre-audit activities, it is important to keep in


mind that the background materials should be requested early
enough to ensure that there is enough time for:

• The facility to assemble and send the information to the team


leader.

• The team members to receive and review the materials prior


to the audit.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 32
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Pre-Audit Activities

Audit Information to be Sent in Advance


Facility plot plan or map

Directions to the facility

Visitor safety requirements (e.g., personal protective equipment,


orientation, specialised training, special clearances, etc.)

Completed pre-audit questionnaire

Description of the facility’s operations/processes

Facility organisation chart, showing HSE responsibilities

Local laws, regulations, and ordinances related to the scope of the audit

List of current environmental licenses, certificates, and authorisations

Copies of permits for wastewater discharges and example air emission


permits

Recent regulatory agency inspection/enforcement correspondence

Recent internal and intra-company environmental, health, and safety


audit reports
Table of contents for facility-specific environmental, health, and safety
policies and procedures

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 33
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Pre-Audit Activities

Audit Materials to be Reviewed Upon Arrival On Site


Management objectives
Copies of all current environmental licenses, certificates, registrations,
authorisations, and applications for such (including air and wastewater
discharges; hazardous waste treatment, storage, or disposal activities;
underground storage tanks; drinking water supplies; etc.)
Facility procedures and programme manuals (e.g., spill prevention plan,
hazardous waste management contingency plan, respiratory protection
plan, exposure control plan, hazard communication plan, etc.)
Effluent and emission monitoring reports

Training records

Hazardous waste manifests

Material safety data sheets (MSDSs)

Inventory of chemicals, including oils, in use or stored on site

Purchase orders for chemicals

Annual PCB reports (for the five years preceding the audit)

PCB transformers inspection records

Monitoring instrument calibration records and maintenance logs

Records of safety inspections, including reports of loss prevention


surveys by insurance underwriters

First aid/dispensary records

Records of exposure monitoring and results, respirator fit testing,


audiograms, etc.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 34
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Pre-Audit Activities

Pre-audit questionnaire. In order to obtain a quick and concise


facility profile to provide the audit team members with a basic
understanding of the facility’s operations, a pre-audit
questionnaire is typically sent to be filled out and returned by the
facility. The pre-audit questionnaire should be a tool that is easily
processed by the facility and provides the auditors with basic
information enabling them to begin planning for the audit. An
example of a pre-audit questionnaire is provided in Appendix A of
this manual.

Although the pre-audit questionnaire may provide the audit team


members with a basis to begin their understanding, the auditors
should critically review the information and remember that the
information provided may not be completely reflective of the
facility’s operations. For example, the facility may indicate on the
pre-audit questionnaire that it does not have underground
storage tanks on site. However, the facility may be unaware that
there are two abandoned underground storage tanks that were
inadvertently omitted from the facility’s initial spill prevention plan
and have long been forgotten. During the on-site phase, the
auditor may need to change his/her original audit strategy (based
on the pre-audit information) to ensure that the protocol areas
assigned are properly addressed.

Planning/Logistical Details
During the team leader’s communications with the facility contact,
any planning details should be resolved, such as:
• Safety and security requirements
• Administrative/logistical details
• Initial interview schedule

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 35
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Pre-Audit Activities

Safety and security requirements. The team leader should clearly


understand the personal protective equipment requirements for
visitors who will be touring and inspecting any and all site areas
(e.g., safety glasses, hard hats, safety shoes, safety clothing,
etc.). The team leader should also obtain information regarding
whether:

• Safety orientations and specialised training (e.g., hydrogen


sulfide, respiratory protection, underground mine safety, etc.)
are required to enter the general facility or regulated areas.
The team leader should inquire as to the time needed to
complete the necessary orientation or specialised training
(e.g., 15 minutes, two hours, eight hours, etc.). Depending
upon the training necessary, the team leader may need to
adjust the audit schedule.

• Security clearances/passes are required for audit team


members or vehicles to enter the facility or regulated areas.

• Restrictions apply to team members who are not citisens of


the country where the facility is located.

• Escorts are required for team members touring and inspecting


facility areas.

Administrative/logistical details. The team leader should cover the


administrative and logistical details with the facility contact as
early as possible in the pre-audit process. Administrative and
logistical details that may need to be addressed include:

• Requesting information regarding travel to the facility, as well


as lodging in the vicinity of the site.

• Arranging for badges, clearances, car passes, safety


orientation/ training, and escorts.

• Arranging for the audit team to stay after normal business


hours and observe second or third shift operations, where
applicable.
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 36
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Pre-Audit Activities

• Scheduling the opening meeting and setting a tentative time


for the closing meeting.

• Establishing a time during the day when the audit team can
meet daily with the key HSE staff to discuss their preliminary
findings and concerns. These meetings are typically referred
to as daily debriefs.

• Arranging for meals (working breakfasts, lunches, or dinners).

• Requesting that the facility prepare a brief presentation on the


facility’s operations for the opening meeting.

• Arranging for a meeting room to be available to the audit team


during its visit.

• Informing facility personnel of any documentation/equipment


(e.g., paper copy of the facility’s presentation, telephone
directory, employee rosters, telephone, overhead projector,
slide projector, printer, fax machine, photocopy capabilities,
etc.) that needs to be made available for the audit team.

Initial interview schedule. The team leader may want to obtain the
names of facility contacts to begin scheduling interviews with key
HSE staff for the first one or two days of the audit. The benefits
associated with having the team leader and the facility put
together an initial interview schedule with key HSE staff are two-
fold: 1) the facility is given an opportunity to feel a part of the
audit process and to establish interviews at convenient times,
lessening the impact on their day-to-day activities; and 2) the
audit team will be able to begin their understanding of how the
facility manages environmental, health, and safety areas
immediately after the opening meeting and orientation tour.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 37
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Pre-Audit Activities

Finally, after arrangements have been agreed upon, the team


leader should send a letter to the facility contact confirming those
arrangements along with an outline of the audit process and
activities, and the audit objectives, purpose, and scope. In
addition to the pre-audit questionnaire, a list of the materials that
the audit team would like sent prior to the audit is typically sent
as an attachment to the confirmation letter. An example
confirmation letter is provided in Appendix B.

2. Assembling and Distributing Background Information

Assembling Background Information


Following the receipt of materials from the facility, the team
leader should begin the task of assembling the available
background information. This step, in general, will enable the
team to develop an effective audit strategy tailored to an
individual facility. This task typically involves:

Reviewing facility information and responses to the pre-audit


questionnaire and/or contacting the facility if background
information has not been received within the specified time
period.

Contacting the facility to clarify any ambiguous or incomplete


information received.

Contacting the legal department, as appropriate, to ascertain


whether the facility has any outstanding litigation or history of
compliance problems.

Obtaining relevant company policies and procedures and


applicable federal/national, state/provincial, and local regulations.

Identifying site-specific situations and requirements that may


require modification of the standard audit protocols.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 38
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Pre-Audit Activities

Assembling the appropriate audit protocols to be used for the


audit, and assigning them to individual auditors according to their
skills, experience and background.

Distributing Background Information


The background information collected by the team leader should
be distributed in a timely fashion to allow the team members
enough time to review the information prior to the audit.

3. Assigning and Communicating Audit Responsibilities

As the necessary background information is gathered and


reviewed, the team leader should make an initial allocation of the
functional areas (e.g., air pollution control, hazardous waste
management, industrial hygiene, employee safety, etc.). This
task involves matching the talent and expertise of the team
members with specific tasks or protocols, as well as taking into
consideration the team members’ prior audit assignments.

These assignments are typically made by the audit team leader


with input from the individual auditors. If the audit team is not in
one geographical location, a conference call or video conference
can be set up to establish and communicate the audit
responsibilities. In addition to the assignment of functional areas,
the team leader should clearly communicate all pertinent
information and audit team responsibilities during the pre-audit
phase. For example, audit team members are typically
responsible for:

• Reviewing the background information supplied by the team


leader to begin developing an initial understanding of the
facility’s status with respect to each assigned functional
area(s).

• Modifying the audit protocols, as appropriate, to incorporate


state/provincial, local, or facility-specific requirements or
special facility conditions, plans, procedures, etc.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 39
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Pre-Audit Activities

• Making travel/lodging plans.

• Obtaining and reviewing the applicable federal/national, state/


provincial, and local regulations.

• Bringing the necessary audit materials/equipment for the audit


as directed by the team leader (e.g., personal protective
equipment/ clothing, background information, audit protocols,
working paper pads, regulations, computer, etc.).

4. Conducting Pre-Audit Meeting

A pre-audit meeting may be the last step of the pre-audit


planning phase. This meeting is typically held immediately
before the audit (i.e., evening or early morning prior to the audit)
if team members are not located in the same geographical area.

The purpose of the pre-audit planning meeting is to clarify any


details regarding the protocols and to develop an overall audit
strategy. An audit strategy is essentially an outline of the tasks
that may need to be done in order to complete the five-step on-
site audit process, how each task is to be accomplished, and the
time required to complete each step. Audit protocols serve as a
basis for developing this strategy.

During this meeting, team members also:

• Discuss and evaluate background information received from


the facility and determine if there are any overlapping areas
(e.g., industrial hygiene and employee safety regarding
personal protective equipment).

• Identify protocol steps and questions that have been modified


to reflect special facility conditions or unique state/provincial
or local regulatory requirements.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 40
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Pre-Audit Activities

• Confirm that they understand how much time to allocate


during the on-site phase to complete the five-step audit
process. Typically, each auditor should allocate his/her time
on site as indicated below.

Allocation of Time On Site

Percentage
Activity
of Time
15 Understand the management system for assigned topics
Assess the apparent strengths and weaknesses of those
10
management systems
40 Gather audit evidence
30 Re-assess strengths and weaknesses and evaluate audit results
5 Formally report the audit findings to site management

In addition, some audit programmes find it beneficial, whenever


possible, to conduct a pre-audit visit. During this pre-audit visit,
the audit team leader will have a one-day meeting with facility
personnel to review the audit programme objectives, scope, and
approach; establish a preliminary interview schedule; and/or tour
the facility to better understand the operations. The results from
this pre-audit visit are shared with the audit team members prior
to the audit to assist in the development of an audit strategy.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 41
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
On-Site Opening Activities

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 42
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
On-Site Opening Activities

Pre-Audit Activities On-Site Activities Post-Audit Activities

Step 1: Understand Prepare Draft Report


Management Systems
Conduct opening meeting Obtain Review Comments from
Select & Schedule Facility Audit Conduct orientation tour Corporate HSE
Review audit strategy Law department
Understand details of Facility management
management systems

Step 2: Assess strengths Issue Final Report to


& Weaknesses Facility management
Select Team Members & Confirm Consider potential impacts Operations
their Availability Evaluate management Corporate HSE
systems Law department
Set priorities for verification

Step 3: Gather Audit Evidence


Plan the Audit: Evaluate what needs to be
Correspond with the done Develop Action Plan
facility Determine depth & rigor of Develop proposed action(s) to
Assemble & distribute review address each finding
background information Select types of evidence Assign responsibility for
Assign & communicate needed & methods to gather corrective action
audit responsibilities them Develop timetable
Conduct pre-audit Compare practices against
meeting requirements
Document results

Conduct Follow-up
Step 4: Evaluate Audit Results Track status of corrective
Evaluate audit results actions
Write audit findings Confirm closure of findings

Step 5: Report Audit Findings


Conduct exit meeting

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 43
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
On-Site Opening Activities –
A “Typical” Audit

Monday Tuesday Wednesday Thursday Friday


Travel to Site Daily Debrief Daily Debrief Daily Debrief Meet with HSE
Staff
8:00 - 8.30 8:00 - 8.30 8:00 - 8.30 8:00 - 8.30
Understand Gather Audit Continue to Wrap Up Loose
Details of Evidence Gather Audit Ends
Systems Evidence
8:30 - 12.00 8:30 - 11.00
Opening Meeting 8:30 - 12.00 8:30 - 12.00
Conduct Close-
11:00 - 12.00 Out Meeting

11:00 - 12.00
Lunch Lunch Lunch Lunch Lunch

Orientation Tour Assess Strengths Continue to Continue to Travel Home


and Weaknesses Gather Audit Gather Audit
1:00 - 2.00 Evidence Evidence
1:00 - 2.00
12:30 - 4.00 12:30 - 2.00
Review Audit Gather Audit Evaluate Audit
Plan Evidence Results

2:00 - 2.30 2:00 - 4.30 2:00 - 6.00??

Understand
Details of
Systems Team Meeting
Team Meeting
2:30 - 5.30 4:00 - 5.30
4:30 - 5.30

Team Activities Individual Activities

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 44
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
On-Site Opening Activities -
Purpose of the On-Site Opening Activities

The opening meeting and the orientation tour are the initial
activities undertaken by the audit team to begin their
understanding of the facility’s management systems.

On-Site Activities

Interviews Document Review Observation

Opening meeting Key plans, procedures, Orientation tour


policies

Goal
Understand, who, what, where, how

The on-site opening activities are intended to provide the audit


team with a broad and general overview of facility operations and
issues. The audit team members typically obtain this broad
overview by:

• Conducting an opening meeting with facility management


upon arrival at the site to discuss overall facility operations
and the organisational structure used to help facilitate the
implementation of compliance activities.

• Conducting an orientation tour with key facility personnel to


obtain a general orientation to the plant, including its layout
and size, location of operations, and location of those
activities pertaining to the audit scope.

• Reviewing the audit strategy as a team to ensure that the


audit scope includes all the applicable audit topics and that
resources are allocated appropriately, based upon the
information gathered thus far, for each of the audit topics
under review.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 45
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
On-Site Opening Activities -
Conducting the Opening Meeting

The purpose of the opening meeting is to:

• Describe to facility personnel the overall objectives of the


audit programme, and the purpose, scope, and approach of
the audit.

• Gain an initial overview of the facility’s programmes and


practices established to manage environmental, health, and
safety issues relevant to the scope of the audit.

The opening meeting will, to a large extent, influence the overall


outcome of the audit; therefore, it is important that this meeting
be conducted in a professional manner which allows a
comfortable exchange of information between the audit team and
facility personnel.

The following table outlines the typical activities and topics


included in the opening overviews of the:

• Audit process presented by the team leader.

• Site operations, programmes, and procedures presented by a


facility representative.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 46
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
On-Site Opening Activities -
Conducting the Opening Meeting

Key Activities in Presenting the Overview of the Audit Process


Meet with the facility manager and key facility personnel

Have the team leader open the meeting

Use an opening meeting discussion guide

Use an overhead projector with viewgraphs, as appropriate

Encourage discussion among meeting participants

Tentatively schedule the exit meeting or reconfirm the previously


established schedule

Reconfirm the time for the daily debriefs


Topics Typically Addressed in the Facility’s Overview
Production operations, capacities, raw materials, and product lines

Operating hours (staff and operations personnel)

Employee profile (number of salaried and hourly staff, union status)

Overview of HSE programmes

Any major HSE issues/problems

Community issues

Government inspections and compliance history

Regulatory climate

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 47
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
On-Site Opening Activities -
Conducting the Opening Meeting

Before beginning the orientation tour, the team leader should


also reconfirm the following logistical and administrative
arrangements discussed during the pre-audit phase:

Necessary clearances
Safety rules and procedures
Work space for the team
Names of facility contacts

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 48
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
On-Site Opening Activities -
Conducting the Orientation Tour

The purpose of the orientation tour is to provide the audit team a


general familiarity with the plant layout and operations. Typically,
this tour takes one to two hours.

During the tour, the team members should:

• Stay together.

• Focus on obtaining an
overview, not on making
inspections.

• Carry and annotate, as


necessary, a site plot plan.

• Take notes on areas the


auditor wishes to revisit and
employees/ staff that he/she
may need to interview.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 49
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
On-Site Opening Activities -
Reviewing the Audit Strategy

After the team has gained a general understanding of the


facility’s operations and issues, they should meet to review the
audit strategy and discuss the next steps. For instance, the audit
team should finalise an interview schedule for the remainder of
the day or for the next day. Often, the team may need to sit
down with the facility HSE coordinator(s) to clarify HSE roles and
responsibilities as well as to gather additional information
regarding the management systems in general and the overall
applicability of performance requirements, if not already
addressed in the pre-audit phase.

Team Team Team Team Team


Time Member Member Member Member Member
7:00 am Breakfast
Breakfast with
with Team
Team
8:00 am
9:00 am
10:00 am
11:00 am
12:00 pm Lunch
Lunch
1:00 pm
2:00 pm
3:00 pm
4:00 pm Daily
Daily Debrief
Debrief with
with Facility
Facility Staff
Staff

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 50
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
On-Site Opening Activities -
Example of an Opening Meeting Discussion

Chemplant
Opening Meeting Discussion Guide

Audit Team: NPC Contacts:


Lead Auditor: Plant Manager:
John Clarke Jerry Osborne
Auditor: Safety and Health Supervisor:
Peter Tillson Pat Dawson
Auditor: Environmental Co-ordinator:
Paula Brown Chris Carson

Local Address: Facility Address:


Best East Inn 875 Willow Street
Any road Anytown, Anyplace
Anytown, Anyplace (222) 222-2222
(222) 333-3333

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 51
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
On-Site Opening Activities -
Example of an Opening Meeting Discussion

Purpose

The purpose of this audit is to conduct a comprehensive review


of the environmental, health, and safety activities in order to:

• Verify compliance with applicable national, regional (local,


state etc.), and local environmental, health, and safety laws
and standards

• Verify conformance with corporate, company, and facility


environmental, health, and safety policies and procedures

• Determine whether activities are consistent with good


environmental, health, and safety management practices and
whether systems are in place and functioning

Scope

This environmental, health, and safety audit will address the


facility’s compliance and management systems in the following
areas:

• Water Pollution Control, including Spill Prevention and Control

• Air Pollution Control

• Solid and Hazardous Waste Management

• Employee Safety

• Industrial Hygiene

• Loss Prevention

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 52
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
On-Site Opening Activities -
Example of an Opening Meeting Discussion

Approach

The audit is based on:

• A physical survey of the facility

• Examination of a sample of environmental, health, and safety


administrative, technical, and operating records available at
the facility

• Interviews and discussions with key facility management and


staff

• Verification procedures designed to examine the facility’s


application of and adherence to environmental, health, and
safety laws and regulations

Period of Review

January 3, 2000 through the last day of the audit

Reporting

A hierarchical reporting scheme will be used:

Who What When


Facility/HSE Supervision All deficiencies noted When noted

Facility Manager All deficiencies noted Periodic, exit interview,


final report

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 53
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
On-Site Opening Activities -
Example of an Opening Meeting Discussion

Report Schedule

A draft report will be issued within three weeks of the close-out


meeting. After receiving comments on the draft report, a final
report will be issued.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 54
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Understanding HSE Management Systems

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 55
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Understanding HSE Management Systems -
Why Understand HSE Management Systems?

Pre-Audit Activities On-Site Activities Post-Audit Activities

Step 1: Understand Prepare Draft Report


Management Systems
Conduct opening meeting Obtain Review Comments from
Select & Schedule Facility Audit Conduct orientation tour Corporate HSE
Review audit strategy Law department
Understand details of Facility management
management systems

Step 2: Assess strengths Issue Final Report to


& Weaknesses Facility management
Select Team Members & Confirm Consider potential impacts Operations
their Availability Evaluate management Corporate HSE
systems Law department
Set priorities for verification

Step 3: Gather Audit Evidence


Plan the Audit: Evaluate what needs to be
Correspond with the done Develop Action Plan
facility Determine depth & rigor of Develop proposed action(s) to
Assemble & distribute review address each finding
background information Select types of evidence Assign responsibility for
Assign & communicate needed & methods to gather corrective action
audit responsibilities them Develop timetable
Conduct pre-audit Compare practices against
meeting requirements
Document results

Conduct Follow-up
Step 4: Evaluate Audit Results Track status of corrective
Evaluate audit results actions
Write audit findings Confirm closure of findings

Step 5: Report Audit Findings


Conduct exit meeting

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 56
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Understanding HSE Management Systems -
Why Understand HSE Management Systems?

Purpose

The first step of the on-site activities is to develop an


understanding of how the facility manages its HSE activities so
that the audit team can:

• Gather information that will be used to help set priorities


among the audit topics to review.

• Gain insights regarding how effectively and efficiently an EHS


topic is being managed and, thus, establish a context for
evaluating the audit results.

• Identify potential underlying causes that contribute to


compliance-related deficiencies.

Other Driving Forces

In addition, audit standards—such as the ICC Charter for


Sustainable Development, ISO 14000, EMAS, and BS7750—
require a thoughtful review and evaluation of the facility’s
systems for managing HSE obligations.

Our approach to Step 1: Understand Management Systems will


be to:

• Explain what HSE management systems are and some of the


specific activities involved.

• Describe how the auditor should go about understanding HSE


management systems in Step 1 of the audit process.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 57
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Understanding HSE Management Systems -
What Are Management Systems?

In the simplest sense, HSE management systems are the actual


processes used by a facility to achieve and maintain
conformance with established standards, including programmes,
policies, equipment, administrative controls, etc. The auditor’s
role is to find out how the facility really manages its HSE
obligations.

Source: Systemation, January 15, 1959, published by Systemation, Inc.,


Colorado Springs, Colorado.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 58
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Understanding HSE Management Systems -
What Are Management Systems?

With the onset of emerging international environmental


management standards, HSE management systems can be
described in terms of key processes that are aligned with an
appropriate organisation and resources.

Typical HSE Management Processes

Assessing Planning Implementing Reviewing


“What do we need “How should
“Let’s manage it!” “How are we doing?”
to manage?” we manage it?”

Supporting
Training and awareness Documenting/recordkeeping Managing information

Organisation Resources

Foundation

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 59
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Understanding HSE Management Systems -
What Are Management Systems?

The key processes in management systems can be defined and


categorised as follows:

Basic Step On-Site Activities Outcome

Assessing is used by the facility for identifying Identify / evaluate HSE issues
Assessing conditions / aspects & materials on facility that Identify on-site hazards
“What do we need to
manage?”
have HSE implications, & for determining the Review HSE activities and projects
applicability of regulations Review applicable regulations
Planning is used by the facility for designing and Formulate HSE strategies & policies with clear
establishing programs & systems for HSE objectives & targets that reflect the importance of the
compliance HSE issues applicable on site
Develop HSE procedures for compliance & record
Planning keeping activities, as well as prevention plans with
“How should we manage
it?” specific operating criteria
Identify & design engineered controls & equipment
Develop emergency response procedures; create
procedures regarding critical HSE activities / issues or
departures from established criteria
Implementing is used by the facility for ensuring Acquire permits
effective and consistent implementation of its Disseminate policies & procedures
HSE programs Assign & communicate roles & responsibilities
Implementing Install, calibrate & maintain engineered controls
“Let’s manage it”
Handle situations that deviate from an established
standard
Undertake activities in accordance with established
schedules
Reviewing is used by the facility for measuring & Conduct drills
assuring HSE program effectiveness Conduct inspections / self-audits in accordance with
documented procedures
Reviewing Review compliance data / performance
“How are we doing?” Track continuous improvement
Review the effectiveness of management systems
Undertake corrective actions in response to identified
directors from procedures or established criteria

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 60
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Understanding HSE Management Systems -
What Are Management Systems?

Supporting processes (training and awareness, documenting/


recordkeeping, and managing information) tend to cross over the
four key categories of assessing, planning, implementing, and
reviewing.

Training programmes, both formal (e.g., classroom) and


informal (e.g., on-the-job), in-house and external, for
ensuring that each HSE programme element is understood
and implemented properly.
Training and
Programmes and/or other means for communicating HSE
Awareness
procedures, and addressing critical HSE issues and
potential consequences of departure from specified
operating procedures.
Competent staff performing HSE work.
Systems for legibly documenting and dating activities (e.g.,
routine reports, inspections, audits, training, etc.)
conducted during the implementation of the HSE
Documenting/ programmes.
Recordkeeping Systems for retaining documentation in an accessible and
orderly manner for periods of time required by regulation
or internal standards and for removing obsolete
documents.
Systems for communicating relevant information across
the various levels and functions of the organisation.
Managing
Mechanisms to track and evaluate compliance information.
Information
Systems for responding to relevant communications from
external parties.

Organisation and resources should be aligned to support


effective HSE management systems.

Clear assignments and understanding of HSE


responsibilities and accountabilities among HSE and line
Organisation management.
Visible HSE commitment and support by management.
Appropriate high-level HSE reporting.
Sufficient number of qualified HSE staff.
Adequate HSE staff/responsibilities to cover all
Resources
business/organisation groups within the facility.
Availability of needed financial and technological resources.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 61
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Understanding HSE Management Systems -
The Methodology for Understanding HSE
Management Systems

There are three principal approaches used to understand HSE


management systems within the context of an audit.

Talk to Key People

Look at Key Equipment/Facilities Review Key Programme Documents


60
50
40
30
20
10
0

Warningwith, if applicable)
Contains (or manufactured
(insert name of substance), a substance which har ms
public health and environ ment b y destro ying ozone
in the upper atmosphere

Warningwith, if applicable)
Contains (or manufactured
(insert name of substance), a substance which har ms
public health and environ ment b y destro ying ozone
in the upper atmosphere

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 62
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Understanding HSE Management Systems -
The Methodology for Understanding HSE
Management Systems

Technique Activities
Talk to several people (e.g., line management, HSE staff,
Talk to Key
operating personnel, maintenance personnel) to obtain a
People
comprehensive understanding of the activities that are in place
to manage compliance and to increase your sense of
confidence in the information obtained.

Summarise the information obtained from each interviewee to


verify the completeness of your understanding.

Probe to understand inconsistencies in the information


obtained.

Examples of what auditors should endeavor to understand


during interviews include:

• What is meant by the scope of the facility’s programmes? For


example, when the facility says that, “All employees receive
hazard communication training,” does this mean all
employees, all employees who work in certain areas, or
something even narrower?
• How does the facility handle seasonal, situational, or “non-
normal” activities? For example, how does a particular activity
work on the off-shift, or when a key person is on vacation,
etc.?
• How does the facility develop data for preparing compliance-
related reports, for determining compliance, or for identifying
HSE-related problems? To understand this, the auditor may,
for example, want facility personnel to describe or
demonstrate how they reconcile the monthly inventory for an
aboveground storage tank

Look at Key Walk around the facility to understand the nature of the HSE
Equipment/ issues that need to be managed and the types of engineered
Facilities controls in place. (This is primarily accomplished during the
orientation tour.)

At this stage, auditors should focus on understanding the


nature and rationale of the engineered controls used to
manage HSE hazards.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 63
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Understanding HSE Management Systems -
The Methodology for Understanding HSE
Management Systems

Technique Activities
Review Key Briefly review procedures, plans, etc., that explain how the
Programme facility manages HSE obligations. For example:
Documents • Compliance-related programme documents
• Operational procedures
• Checklists or inspection forms
• Training programme description

At this stage, auditors should focus on understanding:


• How the programme is supposed to work.
• Tools used by the facility for ensuring that the programme
works as designed and is effective.

There are some key questions auditors should try to answer


when understanding each stage of the management process.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 64
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Understanding HSE Management Systems -
The Methodology for Understanding HSE
Management Systems

Management
Things the Auditor Wants to Learn
Process
• How are regulations tracked (i.e., identifying, tracking,
Assessing
interpreting, and communicating regulations)?
• How are HSE risks and effects assessed (e.g., waste
stream inventory, air emissions inventory, natural
resources consumed, likelihood and magnitude of
unplanned events, potential exposure to hazardous
substances, etc.)?
• How does the facility manage changes in procedures or
facility design (e.g., HSE review and consideration for
new products, processes, equipment, acquisitions and
divestitures, maintenance modifications, etc.)?
• What type of basic compliance programmes (e.g.,
Planning
permitting, monitoring, training, recordkeeping,
reporting, etc.) have been or are being established and
do they include critical operating parameters and
schedules?
• How does the facility prepare for emergencies (e.g.,
developing scenarios, response capability, response
plans, etc.)?
• What type of issue-specific risk reduction programmes
(e.g., groundwater monitoring, pollution prevention,
waste management practices, spill containment
programmes, ergonomics, engineered controls, etc.) are
developed?
• What engineered controls or alarms are in place to help
achieve desired results?
• What measures have been taken to reduce the
likelihood of nonconformance with established criteria?
• What types of policy and related goals and objectives
are established (e.g., vision statements, basic policies
and guiding principles, specific goals and milestones,
etc.)?
• What strategies are developed for managing HSE risks
and effects?

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 65
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Understanding HSE Management Systems -
The Methodology for Understanding HSE
Management Systems

Management
Things the Auditor Wants to Learn
Process
• Has the facility fully implemented the various HSE
Implementing
programmes?
• How does the facility maintain operating equipment
(e.g., preventive maintenance programmes, testing and
monitoring, etc.)?
• How is nonconformance with established criteria
handled?
• Are facility operations being inspected (e.g., routine
Reviewing
walk-throughs, use of checklists, etc.)?
• How are HSE effects being measured?
• Does the facility analyse its performance (e.g., evaluate
“findings,” “lessons learned,” trends, etc.)?
• Are programmes in place for developing, implementing,
and tracking corrective actions?
• What formal training have key personnel had to assist
Training and
them in performing their HSE tasks and functions?
Awareness
• What training and awareness activities are conducted to
provide an understanding of HSE obligations and
responsibilities?
• What type of training programmes (e.g., compliance-
related training, emergency drills, on-the-job training,
etc.) are available?
• What types of experience or background are required to
perform HSE tasks?
• Are procedures and practices for both compliance and
Documenting
remediation generally written down?
and Record-
• What records are routinely developed and retained in
keeping
carrying out various tasks and functions?
• What exception reports are developed?
• What is the general nature or character of the
documentation that is developed?
• How does the relative importance of HSE activities
correspond to the nature and level of documentation that
is developed?
• Where is information retained?
• How long is information retained?

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 66
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Understanding HSE Management Systems -
The Methodology for Understanding HSE
Management Systems

Management
Things the Auditor Wants to Learn
Process
• How is important HSE information conveyed to
Managing
personnel?
Information
• Can the facility readily access HSE information?
• How does the facility ensure that reports are submitted
in a timely manner and that essential records are
retained?
• Are there procedures in place for responding to external
requests for information?
• How are responsibilities and accountabilities defined,
Organisation
established, and communicated?
• How are assignments of responsibilities reinforced?
• Are any key responsibilities overlapping, shared, or
conflicting?
• What potential exists for “conflict of interest” in
accomplishing key HSE tasks and functions?
• How is authority granted to carry out assigned
responsibilities?
• How have responsibilities for implementation been
Resources
communicated to personnel who need to know?
• Are there sufficient resources to carry out the various
HSE programmes?
• Who has the authority to waive adherence to, or
conformance with, an established standard or
requirement and are deviations recorded?
• Are there perfunctory approvals—authorisations without
understanding what is involved?

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 67
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Understanding HSE Management Systems -
How Much is Enough?

Auditors frequently wonder whether they have collected enough


information and the right kind of information to substantiate their
understanding of a facility’s key management
systems/programmes and physical controls. Listed below are
tips for determining how much is enough.

You have probably gathered enough information if:

You understand both system design (e.g., facility policies and


procedures regarding regulatory tracking) and implementation
(e.g., availability of the latest regulations on site).

You have interviewed all of the key personnel involved in key


HSE functions or tasks, and you can summarise to their
satisfaction the basic programmes, practices, and control
systems.

You understand the probable cause(s) of any differences


between management’s and employees’ perspectives, or
between environmental and operational personnel’s
perspectives.

You understand the types of activities that are applicable and


the range of activities being managed by the facility.

You understand the roles and responsibilities of HSE staff, the


mechanisms to share relevant information, and the processes
for retaining information in an accessible manner.

Once the auditor has developed a basic understanding of the


management systems associated with assigned audit topics,
he/she is now ready to move on to Step 2 and assess the
systems under review.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 68
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 1A
Understanding HSE Management Systems -
Confined Space Entry

Objective

The purpose of this exercise is to demonstrate how the auditor


obtains information to understand the management systems in
place at a facility (Step 1 of the audit process).

Background

Based upon the preliminary information provided to you by the


facility, you know the following:

• The facility has confined spaces, which are entered by facility


personnel and contractors.

• The facility has a confined space entry programme along with


permits.

• The facility is a chemical manufacturer that handles a variety


of toxic and flammable substances.

Instructions

Describe the steps you would take to understand the


management systems used by the facility to implement the
confined space entry programme. Include in your description:

• Whom you would talk to.

• What questions you would ask.

• What documents and physical facilities you would want to look


at.

• Any other activities you would conduct while completing Step


1 of the audit process for this topic.
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 69
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 1A
Understanding HSE Management Systems -
Confined Space Entry

Whom would you talk to?

What questions would you ask?

What documents and physical facilities would you look at?

Other activities?

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 70
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 1A – Potential Answers
Understanding HSE Management Systems -
Confined Space Entry

Whom would you talk to?


HSE Manager
Maintenance Manager
Maintenance Staff

What questions would you ask?


What is the procedure for confined space entry?
Where are they formalised (HSE manual)?
To Maintenance Staff – What do they do before and while entering
confined space?
Training
PPEs?

What documents and physical facilities would you look at?


Permit to work
Inventory of confined spaces

Other activities?
Ask to go and see the confined spaces
Ask to observe if any maintenance is scheduled during the audit

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 71
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 1B
Understanding HSE Management Systems -
Spill Control

Objective

The purpose of this exercise is to demonstrate how the auditor


obtains information to understand the management systems in
place at a facility (Step 1 of the audit process).

Background

Based upon the preliminary information provided to you by the


facility, you know the following:
• The facility has aboveground storage tanks containing
petroleum and chemical products.
• The facility has tank truck loading/unloading operations.
• The facility has a spill control team that has been provided
training.
• A Spill Prevention Control and Countermeasures plan is on
site.

Instructions

Describe the steps you would take to understand the


management systems used by the facility to implement the spill
control programme. Include in your description:
• Whom you would talk to.
• What questions you would ask.
• What documents and physical facilities would you want to look
at.
• Any other activities you would conduct while completing Step
1 of the audit process for this topic.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 72
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 1B
Understanding HSE Management Systems -
Spill Control

Whom would you talk to?

What questions would you ask?

What documents and physical facilities would you look at?

Other activities?

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 73
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 1B – Potential Answers
Understanding HSE Management Systems -
Spill Control

Whom would you talk to?


HSE Manager
Maintenance Manager and staff who supervise loading/unloading
Spill control team

What questions would you ask?


What is the procedure for spill control and countermeasure?
Who is in charge?
What training is carried out?
Is external personnel (truck drivers) trained/supervised?
What happens if a spill occurs on a Sunday night?

What documents and physical facilities would you look at?


Spill prevention control and counter measure plan
(Accident/Incident/Near misses register) – Loading/unloading,
secondary containment
Spill control kits – Waste registers
Hazard material inventory – Quantities and storage locations

Other activities?
Observe loading/unloading

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 74
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Effective Interviewing

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 75
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Effective Interviewing

Interviewing is one of the primary techniques used in gathering


audit information. Interviews provide auditors with:

• A time-efficient way of gathering both broad general


information and specific details from people who should know.

• A means to confirm hypotheses about site conditions,


changes, needs, and opportunities.

• A current and credible source of facts and perceptions that


complement written information and physical observations.

Thus, good interviewing skills are essential to the successful


completion of the audit. While in one sense interviewing is a skill
that comes naturally to most people, good interviewing
techniques, which emphasise interaction between interviewer
and interviewee, must be developed. By remembering some
basic elements of good interviewing skills, the interviewer not
only will be successful in gathering the information he/she
desires, but will also find the interview process much more
pleasant.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 76
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Effective Interviewing –
The Basic Interview Process

A significant portion of the auditor’s field time is spent asking


questions of and engaging in discussions or conducting
interviews with facility staff. While the setting, duration, and
degree of formality of such interviews can vary, all audit
interviews follow this common pattern.

Planning

Opening

Documenting
Conducting

Closing

1. Planning the Interview

Prior to conducting the interview, the auditor should identify


personnel to be interviewed, outline the objectives to be
accomplished, and plan how to maximise the effectiveness of the
interview. Key considerations include:

• Iron out logistics—set a specific time and place for the


interview.

• Define the desired outcome—identify the types of information


desired and/or areas to be addressed. The types of
information gathered during interviews can be characterised
as “hard” or “soft.”

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 77
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Effective Interviewing –
The Basic Interview Process

• Quantitative data regarding emissions, exposure monitoring,


Hard
etc.
• Records prepared for compliance purposes
• Historical information about a site
• Why did something (not) happen?
Soft
• How does the process/system work?
• How does the work really get done?

Most HSE audit interviews mix “hard” and “soft” information


needs.

• Organise your thoughts—develop a logical sequence of


questions.

• Be prepared—HSE interviews take place under all types of


distracting conditions, so plan appropriately.

2. Opening the Discussion

The quality of information gathered during an interview is closely


related to the interviewee’s sense of comfort. The level of
openness that develops during an interview, along with the
interviewee’s confidence in the topic being discussed, depends a
great deal on the rapport and atmosphere established during the
initial contact.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 78
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Effective Interviewing –
The Basic Interview Process

In constructing an atmosphere where positive rapport can be


established, auditors should follow these general guidelines:
• Arrive on time.
• Introduce yourself.
• Ensure appropriateness of time.
• Explain the purpose of the discussion.
• Explain how the information gathered in the discussion will be
used.

In addition, auditors should strive to build the desired sense of


comfort and confidence by:

• Cultivating a friendly, nonthreatening discussion.


• Attempting to respond to the interviewee’s social style. For
example:

Example Interviewee
Social Style Orientation
Questions/Responses
Analytical/ Technically- “What’s your methodology?”
Thinker oriented
“How will the results be used?”
Driving/Doer Results- “What can I do for you in the next ten
oriented minutes?”

“What is the outcome of all this time


spent talking with auditors?”
Amiable/ Relationship- “How are you enjoying our plant/town?”
Feeler oriented
“Why are we having this interview?”
Expressive/ Social “Did you notice how effectively our
Intuitionist recognition- waste management process operates?”
oriented
“We believe we’ve made the most
improvements in this area of anyone in
the industry.”

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 79
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Effective Interviewing –
The Basic Interview Process

• Refraining from portraying a condescending, arrogant, know-


it-all attitude.
• Acting in a supportive and nonjudgmental manner.
• Ensuring that the auditor and the interviewee are “on equal
ground.”

3. Conducting the Interview

Style, Flow, and Tone


Once a comfortable interview setting and rapport have been
established, the auditor should focus toward obtaining specific
information from the interviewee. Some examples of specific
items that an auditor should address include:

• Request a brief overview of the interviewee’s job.

• Discuss the interviewee’s responsibilities in relation to the


topic(s) being reviewed.

• Use language that the interviewee can understand.

• Start with some general questions, then gather more detailed


information.

• Resolve ambiguities through constructive probing.

• Do not exceed the agreed-upon time limit without first


obtaining the interviewee’s approval.

• Provide feedback as appropriate.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 80
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Effective Interviewing –
The Basic Interview Process

Establishing an appropriate interview style, flow, and tone is


essential to the interview process. For example:

Style • Avoid interrogation


• Be empathetic
• Avoid defensiveness
• Be calm, objective, and non-partisan
• Be courteous, alert, and responsive
Flow • Avoid disjointed transitions that damage rapport (e.g., cutting
the interviewee off because you are trying to transition to a new
topic)
• Use word association to change topic focus (e.g., if the
interviewee mentions his/her training, use this opportunity to
raise any training questions)
Tone • Be genuine and take an interest in the interviewee’s responses
• Use a soft, friendly voice

Types of Interview Questions


Appropriate questioning should be utilised to obtain the desired
information. For example:

Type of Example Questions Typical Response Relative Value of


Question Outcome Response
Information to
Auditor
Leading “Of course you notify the Often
state of a planned unintentionally
discharge.” “lead” the
interviewee to the −
“You do test the wells desired answer.
every month, don’t
you?”
Yes/No “Do you have a spill Usually receive only
response procedure?” a “yes” or “no”
answer. +
“Have you conducted
waste audits?”
Close- “What is your current Usually receive a

+
Ended production capacity?” one-word answer.

“What is the capacity of


your wastewater
treatment plant?”

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 81
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Effective Interviewing –
The Basic Interview Process

Type of Example Questions Typical Response Relative Value of


Question Outcome Response
Information to
Auditor
Open- “How are new chemicals May prompt a more
Ended selected for use in your detailed response.

+++
process?”

“What analyses are


performed to assess
HSE risks and
liabilities?”
How Do “How does your facility Can provide the
You Know ensure that workers most insight into
receive appropriate HSE how things are
training?” actually managed. +++
“How does your facility
ensure that PCB
transformer inspections
are performed?”

Active Listening
Another important component to conducting successful
interviews involves active listening. Active listening allows the
interviewer to:
• Summarise information accurately.
• Test the interviewee’s understanding of the topic being
discussed.
• Probe for confirmation.
• Facilitate the interview.
• Display empathy/establish greater rapport.

All auditors should develop the following listening techniques:

• Wait until the current question is answered before asking


another question.

• Encourage the interviewee in a nonverbal manner (e.g.,


maintain eye contact, display attentiveness, etc.).
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 82
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Effective Interviewing –
The Basic Interview Process

• Interrupt only if you sense avoidance—people often feel more


secure when given an opportunity to speak without being
interrupted.

• Imagine the interviewee’s situation (i.e., put yourself in the


interviewee’s job scenario).

• Listen for emotions and attitudes as well as facts.

In addition, the interviewer should remember to listen over 90


percent of the total interview time.

Paraphrasing/Summarising Information Learned


The technique of paraphrasing can aid the auditor in confirming
or clarifying something said or implied by the interviewee. There
are three levels of paraphrasing:

Level Accomplishment Example Paraphrase


1 Confirms or clarifies “So there are three factors that
expressed thoughts and determine the present situation...”
feelings

2 Confirms or clarifies implied “You would really like to change this


thoughts and feelings situation...”

3 Surfaces core thoughts and “You are concerned that your


feelings company’s engineering approach is
outdated...”

In using this technique, it is important to:


• Paraphrase completely.
• Match the levels of emotional intensity and factual content.
• Use levels 1 and 2 freely; treat level 3 with caution.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 83
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Effective Interviewing –
The Basic Interview Process

Nonverbal Communication
A significant portion of the information exchanged in any
interview is done nonverbally. Nonverbal communication is a
combination of the meanings expressed by the interviewer and
the interviewee through gestures, facial expressions, voice
inflections, and posture. Auditors should be aware of nonverbal
communication and pay attention to the following:
• Shake hands.
• Maintain eye contact.
• Keep the right distance.
• Tolerate silence.
• Mirror the interviewee’s body movements.
• Be sensitive to culture and customs (especially outside of your
native country).

Actions to Avoid
In addition to the positive actions that should be taken in
preparing for and conducting the interview, there are also some
actions the interviewer should avoid. These include:
• Debating with the interviewee or wasting time disagreeing on
any one point.
• Forcing a meeting with the interviewee if he/she is otherwise
occupied.
• Rushing the interview.
• Amplifying criticism given.
• Using sarcasm or subtle humor.
• Telegraphing your assessment of the interview.
• Jumping to conclusions/not watching your assumptions.
• Communicating incomplete or unsubstantiated findings or
conclusions.
• Using a tape recorder.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 84
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Effective Interviewing –
The Basic Interview Process

4. Closing the Interview

The closing phase of the interview should appear to reach a


natural conclusion. In closing the interview, the auditor should
strive to:
• Pace the interview to avoid rushing and, if necessary,
schedule additional interview time when mutually convenient.
• Summarise and confirm information learned.
• Try to bring closure to sensitive issues/questions raised earlier
in interview.
• Never exceed the allotted time without confirming with the
interviewee.
• Ensure that the interviewee is psychologically ready to leave
the interview (e.g., not unduly worried, upset, etc.).
• End with an “open door” for communication in either direction.
• End on a positive note—thank the interviewee for his/her time
and cooperation.

Because the interviewee may feel that this phase of the interview
is his/her final opportunity to be heard, closing questions used by
the auditor can be extremely productive. Some examples
include:
• Is there anything you expected me to ask about that I did not
mention?
• A little earlier you said that...
• We have heard that...Is that really the way it is?
• Can you elaborate on that point?
• Would others say the same?
• Is it like that at other times/places/divisions/companies?
• Playing devil’s advocate, would some people fundamentally
disagree?

These questions clarify open issues and help to resolve


contradictory statements.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 85
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Effective Interviewing –
The Basic Interview Process

5. Documenting the Interview

Most of us have limited memories, especially when it comes to


recalling what people have told us; if we have no record of an
interview and have to rely entirely on our memories, we are
bound to make mistakes and omit important points. Therefore, in
audit interviews, it is safer to take notes and not to rely on an
imperfect memory, especially when you are interviewing several
people during the course of any given day, which is most often
the case during audits.

The only problem with taking working paper notes is how to do so


without distracting or intimidating the interviewee. The break in
eye contact that occurs during the writing process can be
distracting for the non-writing individual. Similarly, not knowing
what or why an auditor is writing may intimidate the interviewee.
With these points in mind, review the following suggestions on
how to take notes during interviews:

• Never try to hide the fact that you are taking notes. Rather,
draw the interviewee’s attention to what you are doing by
explaining the need to take notes and involving the
interviewee in the process. If necessary, address the issue
regarding the confidentiality of working papers.

• Offer the interviewee the opportunity to review the notes taken


during your interview to put him/her at ease with the process,
if necessary.

• Keep your working paper pad within easy reach, such as


attached to a clipboard on a desk or table. Always make sure
you have a pen handy—and that it works.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 86
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Effective Interviewing –
The Basic Interview Process

• Take notes about facts that are relevant to the audit. Avoid
listing points that do not relate to the audit topics of concern.
For example:
− Record name, title, and job description of person with
whom you spoke.
− Reference the appropriate protocol step(s) addressed in
the interview.
− Note relevant interview information.
− Highlight key statements/observations.

• Put the interviewee’s words in quotation marks to distinguish


them from your own comments.

• Spend time immediately following the interview summarising


the key points obtained from the interview in your working
papers

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 87
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 2
Effective Interviewing - Difficult Interview Situations

Objective

The purpose of this exercise is to identify different types of


interview situations that could potentially occur while conducting
an audit and to discuss various strategies useful in resolving
difficult interview situations.

Instructions

Read the following scenarios and answer these questions for


each:

• What is happening in this interview?


• What would you do in this situation?

Scenario A

You are responsible for assessing the environmental training


provided to site personnel during an environmental audit of the
company Mining Samples. During an interview with the lab
training co-ordinator , Jo Lopes, you ask ‘Could you tell me what
types of environmental training site employees receive?’ Mr.
Lopes responds ‘Sure. All site employees attend an initial eight-
hour environmental awareness course during their new employee
orientation week. Employees also attend and must successfully
pass a first aid/cardiopulmonary resuscitation course.’ You say,
‘Getting back to the environmental awareness course, can you
describe the types of information that are discussed in this
training?’ Mr. Lopes replies, ‘Sure. We train new site employees
on environmental issues such as pollution prevention and air and
water contamination. Our first aid courses instruct site
employees on how to respond to health and safety emergencies
that may occur here at Mining Samples. Let me get a copy of the
training material for you.’

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 88
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 2
Effective Interviewing - Difficult Interview Situations

Scenario B

During an internal audit of the company Big Motors, you are in


charge of evaluating the company’s health and safety
management systems. The facility manager suggests you talk
with Harry Thomas, the facility health and safety co-ordinator to
obtain a detailed understanding Big Motor’s health and safety
programmes. After leaving three messages with his secretary,
Mr. Thomas returns your call. You explain to Mr. Thomas that
you wish to meet with him at a mutually convenient time. He
responds, ‘I don’t have any free time to meet in person. Why
don’t you ask your questions now?’ You begin, ‘Could you
describe in general the different health and safety programmes
that are in place at this facility?’ Mr. Thomas states, ‘You can find
all that information in the site health and safety manual.’ You ask,
‘ Where could I find a copy of that manual?’ Mr. Thomas replies,
‘I’m not sure. There’s probably one floating around in my office
somewhere.’ You then inquire, ‘Well, could you briefly describe
your duties to me?’ Mr. Thomas answers, ‘I’m in charge of all the
programmes listed in the site health and safety manual.’

Scenario C

As the auditor responsible for assessing the management of


hazardous wastes at Gesher Manufacturing, you are in the
middle of a scheduled interview with Richard Seaton, the facility
maintenance manager. Although slightly preoccupied, Mr.
Seaton has been responding amicably to your questions. As you
listen to Mr. Seaton explain the major product lines at the facility,
you recall the environmental co-ordinator, Jessica Schaeffer, did
not mention any site waste recycling activities. You ask Mr.
Seaton, ‘Does the facility participate in any types of pollution
prevention or recycling activities?’ Mr. Seaton’s demeanor rapidly
changes as he responds, ‘No. I don’t believe in those types of
new-fangled practices.’

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 89
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 2
Effective Interviewing - Difficult Interview Situations

Scenario D

On the last day of the audit of the Orecrush Mill water pollution
control programme, you realise you have a specific question
about the site’s backflow prevention devices. You decide to call
Mr Ahraby, the wastewater treatment operator you spoke with
earlier in the week. After calling the treatment plant, you learn
that Mr. Ahraby is out sick and that Mr Karami, one of the on-duty
water treatment operators, is covering for Mr. Ahraby. You ask to
speak with Mr Karami; after he answers the telephone, you
explain that you spoke with Mr. Ahraby at length earlier in the
week and have a specific question regarding water backflow
prevention devices. Mr Karami answers, ‘I’d be happy to answer
any of your questions. In Mike’s absence, I’m responsible for the
entire wastewater treatment plant. Do you know this plant
handles 20,000 gallons of wastewater a day, seven days a
week? That’s over seven million gallons annually.’ You interrupt,
‘How interesting. Can you tell me if the backflow prevention
devices are routinely inspected?’ Mr Karami replies, ‘Sure. Why
just last week we had a full plant inspection. The inspectors even
made us clean out our personal lockers. I’ll tell you, that took
some doing. I had so much stuff in my locker that I couldn’t even
get it open . . . ‘

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc c 90
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 2
Effective Interviewing - Difficult Interview Situations

Scenario E

As part of a four-person audit team assigned to audit D&R


Chemical’s facility, you are in the process of interviewing Janet
Kowalski, the facility health and safety co-ordinator. In an
attempt to obtain information regarding site personnel health and
safety training from Ms. Kowalski, you ask, ‘Could you explain
how the facility ensures that new employees receive appropriate
health and safety training?’ Ms Kowalski answers, ‘How do you
think the facility should monitor such training.’ You respond, ‘I’m
really trying to understand how this particular facility handles
personnel health and safety training. Yesterday, I spoke with site
training co-ordinator, John Golder, who suggested I talk with you.
Could you please describe your training responsibilities to me?’
Ms Kowalski replies, ‘What did the training co-ordinator tell you
that I do?’

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 91
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 2 – Potential Answers
Effective Interviewing - Difficult Interview Situations

The following are general techniques to use in any difficult


interview situation:

• Establish and maintain mental and physical control of


yourself.

• Vary your questioning pattern to obtain maximum information


from the interviewee

• Make sure you are adequately prepared for the interview


beforehand so you portray confidence and can maintain
control of the interview

• Don’t be afraid to openly acknowledge the difficulty, take a


break, or ask if you can reschedule for a later time

Some additional techniques to apply to specific interview


situations include:

• If the interviewee becomes hostile and aggressive, remain


calm and do not worsen the problem. Limit your questions to
just the facts, and convey to the interviewee that the purpose
of the interview is to uncover the truth

• If the interviewee can’t stop talking, do not be afraid to politely


interrupt in order to move on in the interview

• If the interviewee goes off course, sum up what the


interviewee has said and either move onto the next question
or return to the point where things went wrong in the interview

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 92
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 3
Effective Interviewing - Conducting Interviews

Objective

The purpose of this exercise is to practice questioning patterns


that elicit the maximum amount of relevant information from the
interviewee. We will also want to pay attention to planning the
interview and to opening the discussion.

Instructions

In this exercise, we will role play an interview between an auditor


and a plant representative.

If you are No. 1 you will be the Observer for this role play. If you
are No. 2, you will be the Auditor. If you are No. 3, you will be
the Plant Representative. There are different sets of instructions
for your assigned role, which are on the following pages:

Health and Safety Specialists


No. 1 Observer Page 94
No. 2 Auditor Page 95 - 97
No. 3 Plant Representative Page 98 - 99

Environmental Specialists
No. 1 Observer Page 100
No. 2 Auditor Page 101 - 102
No. 3 Plant Representative Page 103 - 104

We will break into groups (Observers, Auditors, and Plant


Representatives in separate rooms). You will have 10 minutes to
prepare for your role, 15 minutes to conduct the interview, and 5
minutes for feedback.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 93
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 3A
Effective Interviewing - Conducting Interviews:
‘Lockout/tagout’

Observer’s Instructions

Your role in this exercise is to observe the interview being


conducted by the auditor, to note the techniques being used, to
document the information in your working papers, and to provide
feedback to the auditor and plant representative on your overall
impressions of the interview at its conclusion.

The auditor’s assignment is to understand and document how the


facility manages its lockout/tagout procedures. The auditor will
be interviewing the facility’s safety supervisor.

To assist you with noting the techniques used during the


interview, an index card with an outline of key interviewing
techniques will be provided to you/ You may wish to note the
number of leading, yes/no, and open-ended questions asked by
the auditor by putting tick marks near these items on the index
card each time a question is asked by the auditor.

Helpful Hint

In order to minimise the distraction created by your role in this


exercise, position yourself off to the side between the other two
participants and make your note-taking as unobtrusive as
possible. Keep track of the time and signal the auditor to wrap
up when the interview period is approaching 15 minutes.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 94
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 3A
Effective Interviewing - Conducting Interviews:
‘Lockout/tagout’

Auditor’s Instructions

Your role in this exercise is to play the part of a safety and


industrial hygiene auditor. During your interview with the facility’s
safety supervisor, you wish to develop and document your
understanding of how the facility manages its lockout/tagout
procedures (refer to the protocol steps provided on the following
pages).

The facility’s lockout/tagout programme is designed to protect


maintenance and production personnel and outside contractors
from contact with energised equipment during maintenance and
overhaul. In its operations, the facility utilises steam to clean
process vessels, and electric and pneumatic motors to operate
mixers, and has numerous product transfers lined located
throughout the plant. Personnel involved in these manufacturing
operations can often be exposed to high temperature liquids and
electrical hazards due to we floors and working surfaces.

You may want to consider the following in preparing for your


interview:
• Identification of equipment requiring lockout/tagout
• Written lockout/tagout procedures
• Training of ‘affected’ and ‘authorised’ employees
• Issuance and use of locks and tags
• Inspections of the lockout/tagout procedures performed
• Group lockout/tagout
• Contractors

Take time now to plan for the interview. Think how you will open,
conduct, and close the interview.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 95
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 3A
Effective Interviewing - Conducting Interviews:
‘Lockout/tagout’

Auditor(s)/ Working
Comments Paper
Reference
Control of Hazardous Energy (Lockout/Tagout)
13. Examine documentation and interview key
personnel to confirm that a written hazardous
energy control programme has been
developed.
14. Review the written hazardous energy control
programme, interview affected personnel, and
review records. Confirm that:
a. Site and local management have
developed and implemented a written
hazardous energy control programme for
all operations where the unexpected
energising, startup, or release of stored
energy could occur and cause injury
b. Local management has provided the
necessary material (e.g. locks, chains,
tags) to their employees for implementing
the hazardous energy control programme
c. The facility has implemented a training
programme of lockout/tagout policies and
procedures, including locking isolation
valves
d. The facility has established a programme
to perform and document periodic
inspections (at least annually) of the
energy control programme
e. There are procedures for briefing
contractor personnel on the emergency
control procedures
f. Only authorised employees implement
energy control measures

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 96
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 3A
Effective Interviewing - Conducting Interviews:
‘Lockout/tagout’

Auditor(s)/ Working
Comments Paper
Reference
15. Examine lockout/tagout devices dedicated to
controlling hazardous energy to confirm that
the element is included
a. Standardised throughout the facility
b. Able to withstand environmental conditions
c. Substantial enough to prevent accidental
removal
d. Traceable to the employee applying the
device
16. Interview employees who work in areas where
lockout/tagout devices are used to confirm that
they are notified prior to the application and
removal of a lockout/tagout device
17. Confirm that there are procedures in place to
ensure the continuity of lockout at shift change
and during group lockout/tagout activities

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 97
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 3A
Effective Interviewing - Conducting Interviews:
‘Lockout/tagout’

Plant Representative’s Instructions

Your role in this exercise is to play the part of the facility safety
supervisor, who is fairly co-operative, but is concerned about how
the audit findings will be presented to management. Although
you answer specific questions, you tend not to readily volunteer
information.

Your facility does have a written lockout/tagout procedures.

In this exercise, you are embarrassed by the fact that the facility
does not rigorously enforce its ‘one lock/one key’ policy.
Combination locks are sometimes used instead of keyed locks,
and several employees may know the combination. Also,
because of inadequate labelling of sources, there have been a
few recent instances in which a supervisor observed that the
wrong energy source was locked/tagged out. Although no actual
incidents occurred, this indicated a failure to verify isolation of the
equipment. You will withhold this information unless asked a
question that requires you, in all honesty, to reveal it. Do not lie,
but make the auditor probe.

Follow these guidelines when conducting the interview:

• Assume one of the following difficult interviewee stances for


part of the review:
− Take control of the interview
− Become hostile and aggressive
− Give inadequate answers and/or become unresponsive
− Continue talking throughout the entire interview
− Go off course from the subject matter

• Insert a ten-second pause into the conversation.

• Give key information that should be recorded and one piece of


superficial information that should not be recorded.
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 98
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 3A
Effective Interviewing - Conducting Interviews:
‘Lockout/tagout’

If you are asked difficult questions for which not enough


information has been provided to you, please feel free to make
up information. Perhaps the next thing to do will be to imagine a
facility with which you are familiar when creating your responses.
Be imaginative and enjoy it!

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 99
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 3B
Effective Interviewing - Conducting Interviews:
Air Pollution Control

Observer’s Instructions

Your role in this exercise is to observe the interview being


conducted by the auditor, to note the techniques being used, to
document the information in your working papers, and to provide
feedback to the auditor and plant representative on your overall
impressions of the interview at its conclusion.

The auditor’s assignment is to understand and document how the


facility manages its air pollution control programmes. The auditor
will be interviewing the facility’s environmental co-ordinator.

To assist you with noting the techniques used during the


interview, an index card with an outline of key interviewing
techniques will be provided to you. You may wish to note the
number of leading, yes/no, and open-ended questions asked by
the auditor by putting tick marks near these items on the index
card each time a question is asked by the auditor.

Helpful Hint

In order to minimise the distraction created by your role in this


exercise, position yourself off to the side between the other two
participants and make your note-taking as unobtrusive as
possible. Keep track of time and signal the auditor to wrap up
when the interview period is approaching 15 minutes.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 100
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 3B
Effective Interviewing - Conducting Interviews:
Air Pollution Control

Auditor’s Instructions

Your role in this exercise is to play the part of an environmental


auditor. During your interview with the facility’s environmental
co-ordinator, you wish to develop and document your
understanding of how the facility manages its air pollution control
programmes (refer to the protocol step provided on the following
page).

The facility has over 20 sources that require air permits. In


addition, the facility has programmes for monitoring the ambient
air, preparing emissions inventories, and inspecting control
equipment.

You may want to consider the following in preparing for your


interview:
• Emission source identification and inventory, registration, and
permitting
• Emission control equipment
• Monitoring programmes
• Maintenance and inspection programmes
• Odour control programme
• Indoor air pollution issues
• Training
• Reporting and recordkeeping

Take time now to plan for the interview. Think how you will open,
conduct, and close the interview.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 101
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 3B
Effective Interviewing - Conducting Interviews:
Air Pollution Control

Auditor(s)/ Working
Comments Paper
Reference
Understanding Management Systems
1. Obtain and document your understanding as to
how the facility manages its air emission
control programmes. Considerations may
include:
a. Emission source identification, registration,
and permitting
b. Emission control equipment
c. Monitoring programmes
d. Maintenance and inspection programmes
e. Training
f. Reporting and recordkeeping etc.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 102
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 3B
Effective Interviewing - Conducting Interviews:
Air Pollution Control

Plant Representative’s Instructions

Your role in this exercise is to play the part of the facility


environmental co-ordinator. You are fairly skeptical of the audit
being conducted and its usefulness to the corporation. You are
very proud of your plant and its environmental record. You have
been inspected by the regulatory agency several times in the
past year and have received no citations from any of these
inspections.

You can feel free to share your skepticism of the audit and your
perspective that the plant has been and is in great shape.

Your facility does have 23 air permits, 26 air sources (stacks from
furnesses, process vents and drains), and 6 emission control
devices (scrubbers and dust collectors). You do monitor ambient
air for fugitive emissions but are not required to report the results.
Your maintenance department does inspect all control equipment
every six months as specified in your permit applications. You
are responsible for completing all emission inventories and
submitting them to the regulator.

In this exercise, you are embarrassed by the fact that the facility
has just recently (last month) undergone expansion that has
resulted in the installation of two new ovens for which the facility
did not obtain the required installation permits and does not have
operating permits. The facility has also added a flare without
revising the permit. Also, some of your permits are past their
expiration dates and timely re-applications were not submitted.
You will withhold this information unless asked a question that
requires you to tell the interviewer honestly about these
oversights. Do not lie, just make the auditor probe to discover
this fact.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 103
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 3B
Effective Interviewing - Conducting Interviews:
Air Pollution Control

Follow these guidelines when conducting the interview:

• Assume one of the following difficult interviewee stances for


part of the interview:
− Take control of the interview
− Become hostile
− Give inadequate answers and/or become unresponsive
− Continue talking throughout the entire interview
− Go off course from the subject matter

• Insert a ten-second pause into the conversation

• Give key information that should be recorded and one piece of


superficial information that should not be recorded.

If you are asked difficult questions for which not enough


information has been provided to you, please feel free to make
up information. Perhaps the next thing to do will be to imagine a
facility with which you are familiar when creating your responses.
Be imaginative and enjoy it!

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 104
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Preparing Working Papers

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 105
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Preparing Working Papers
Definition of Working Papers

Working papers consist of any item that documents information


gathered by an auditor (i.e., rough notes, worksheets, company
records and policies, etc.). Working papers serve as the basis
for the audit findings.

Working papers should contain:

• A description of the environmental, health, and safety


management systems in place at the facility.

• A description of the specific actions taken to address each


step of the protocol (tests conducted, sources(s) of
information, evidence accumulated).

• A summary of the auditor’s findings and observations.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 106
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Preparing Working Papers

Audit Planning Document scope in working papers

Step 1:
Understand Management Systems Record understanding in working papers

Step 2:
Assess Strengths &
Weaknesses of Management Record assessment of soundness of
Systems system design

Step 3:
Gather Audit Evidence Document verification testing plan and results

Step 4: Note explanation & disposition of all findings


Evaluate Audit Results

Step 5: Working
Report Audit Findings Papers
Document audit findings
at exit meeting

Audit
Audit Follow-Up Report

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 107
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Preparing Working Papers -
Purpose of Working Papers

The overall purpose of preparing working papers is to aid the


auditor in providing reasonable assurance that an adequate
audit, consistent with programme goals and objectives, was
conducted. Working papers clearly document the information
gathered by each auditor during the audit, and the information
included in these documents should substantiate both
compliance and noncompliance areas. Audit working papers,
therefore, provide the principal evidential support for the audit
report.

Audit working papers:

• Provide an organised method for ensuring that all audit steps


have been addressed in a manner consistent with the
objectives and established procedures of the audit
programme.

• Supplement the protocols by providing audit planning details


such as the time budgeted to individual audit tasks and the
auditor’s evaluation of the management systems that may
have influenced the conduct of the audit.

• Provide a record of tests conducted and evidence


accumulated.

• Provide data that support the audit report and that may be
useful in subsequent action-planning and follow-up activities.

• Provide information to assist in answering questions that may


arise during subsequent action planning and follow-up.

• Provide a basis for quality assurance and aid in the planning,


performance, and review of future audits.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 108
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Preparing Working Papers -
Retention of Working Papers

Typically, companies will retain working papers according to


policies such as the following:

• Until the final report and/or corrective actions are completed.

• Until the next audit of that facility.

• For a particular retention time based on corporate policy (e.g.,


five years).

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 109
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Preparing Working Papers -
Working Paper Standards

The minimum standards for working papers will vary based on


the company’s audit programme objectives and working paper
retention policy. The principles presented below reflect the types
of working paper standards that have been adopted by various
companies.

Examples of Working Paper Standards


Methods/actions to complete each protocol step or rationale for not
completing the step (e.g., not applicable) are documented.
Potential ambiguities or misleading comments have been clarified.
Facility compliance and noncompliance with applicable requirements
are documented.
All exceptions (deficiencies), observations, and local attention items
(items not for the formal report) are identified. The rationale for local
attention items is documented.
If a “sampling” approach is used, the types and sizes of the “samples”
are identified.
Key requirements to be reviewed are identified.
The auditor’s assessment of the management systems is documented.
The sources by which the auditor gains information are identified.
The auditor’s daily goals and the results achieved are documented.
Protocols and working papers are cross-referenced.
The nature and scope of the audit are identified.
Any changes to the scope are documented.
All exhibits are listed on an exhibit list and referenced in the working
papers, as appropriate.
All pages are numbered, dated, and initialed. Any changes made by the
auditor to the working papers are initialed.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 110
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Preparing Working Papers -
Confidentiality of Working Papers

Working papers should be carefully safeguarded during and after


all auditing activities. Upon completion of the working papers
review at the end of the audit, all audit working papers should be
stored in a central file under the control of the audit programme
manager, until they are destroyed in accordance with an
established records retention policy. (A specific retention period
should be established at the corporate level for the audit working
papers that is consistent with audit objectives and supported by
the legal department.)

Since working papers represent documentation of the scope and


conduct of the audit, they may be needed to support findings in
the audit report should questions arise at a later date. Excerpts
may be copied upon approval of the audit programme manager;
however, working papers should not be copied without prior
approval from audit management.

Because there is no absolute guarantee that an audit report will


not be discovered by an outside party at some future time,
careful preparation of working papers offers the best protection
against potential future liabilities.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 111
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Preparing Working Papers -
Techniques for Recording Information

A number of basic techniques to be used in developing thorough


and well-organised working papers should be kept in mind.

• Write while conducting the audit. Notations that serve as


reminders of key points are helpful in gaining a complete
understanding of facility systems or activities. An auditor
should avoid relying on his/her memory and putting off
documenting items until he/she “has more time.”

• Start each new topic on a new page. Many times an auditor


will obtain additional information even after he/she feels the
particular topic has been completely documented. Any
additional information or notes to clarify particular items can
easily be inserted in existing text, and dated if the information
was obtained on a different day from when the page was
prepared, if each topic is entered on a new page. Also, cross-
references can be added to indicate where additional
information on the topic can be found.

• Clearly label each working paper page. Initial, date, and


sequentially number each page. Labelling each page with the
protocol step makes it easier for both the auditor and the audit
team leader to review the work performed and helps locate
specific topics in the working papers. A single notation
identifying the relevant protocol step is generally sufficient
(e.g., Protocol Step 8b: Off-Site Shipment of Wastes).

• Keep entries factual. Each statement should be based on


sound evidence, with unconfirmed data or information
qualified, and speculation and generalities avoided. For
example:
− Do not say, “It appears that...”; rather, state the facts that
create the appearance.
− Avoid extreme language (e.g., “terrible,” “dangerous,”
“incompetent”).
− Distinguish clearly between information obtained by word-
of-mouth and information observed, verified, or concluded.
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 112
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Preparing Working Papers -
Techniques for Recording Information

− Document the source of all information.

• Keep entries legible. Although no one’s penmanship is


perfect, working papers should be written in a legible hand.
Avoid crowding, leave plenty of space, and write only on one
side of the page. This practice will aid in the audit team
leader’s review of the working papers and will help the audit
programme manager confirm that audit programme goals and
objectives have been achieved. At a minimum, each auditor
should be able to read his/her own notes. In the event of a
mixed language team, the team leader will decide what the
language to use in the working papers will be.

• Write clearly in an understandable style. An auditor should


strive to write clearly, so that a person not involved in the audit
can understand the steps taken and can reach the same
conclusions. Avoid uncommon abbreviations.

• Include photocopies of selected documents. An auditor


should sequentially number and reference selected facility
documents as “exhibits” in his/her working papers (e.g.,
Exhibit A1). If any notations on any exhibit are made, they
should be documented in the auditor’s working papers. For
example: “Exhibit G1 is a copy of the facility’s air pollution
control permit. Page 3 of the permit contains notes that I used
to confirm all emission sources.”

• Maintain an exhibit list. To keep track of the exhibits, an


auditor should develop and update an exhibit list as each new
exhibit is identified and numbered.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 113
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Preparing Working Papers -
Techniques for Recording Information

• Highlight “to do” items and findings. Many auditors find it


convenient to keep a running list of “to do” items (items that
call for further investigation or additional information) during
the audit. This can be done by either writing them on a
separate page or in some way identifying them where they are
noted. Any “to do” items should be indicated as complete and
then cross-referenced when they have been finalised.

• Develop and use standard “tick marks.” To increase


efficiency in developing useful working papers, many auditors
develop standard “tick marks,” or legend codes (i.e., a
personal type of shorthand) for many of the more common or
cumbersome working paper notations.

Tick Mark Examples


Item needing further auditor attention

Item where subsequent attention has been given and noted on


PCC-15 page 15 in the working papers

Potential report or exit interview exception/observation

Exception/observation after reporting to team leader and/or facility

Exception confirmed by auditor (as item 3) on exit meeting


3
discussion sheet

Potential concern later determined by auditor not to be an


PCC-17
exception (explanation on page 17 of the working papers)

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 114
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Preparing Working Papers –
Format

The handwritten information and photocopied facility


documentation that comprise an auditor’s working papers are
normally prepared on site during the audit. The following pages
illustrate examples of working paper entries and documentation,
including examples of flowcharting, organisational charts,
interview notes, descriptions of actions taken and tests
performed, interim summary, pending and completed “to do”
items, and an exhibit list.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 115
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Preparing Working Papers –
Format

2. Date
1. Applicable
protocol step

5. Source of
information

6. Tick marks

7. Referenced
exhibit

8. “To Do” items 3. Auditor’s initials

9. Initialed
cross-outs

10. Working
paper
references

4. Page number

11. Confidentiality

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 116
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Preparing Working Papers –
Example of Flowcharting

MAC p1 of 675

Auditor’s initials and page #

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 117
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Preparing Working Papers -
Example of Organisational Charts

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 118
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Preparing Working Papers -
Example of Interview Summary Notes

M DG 7 of 42

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 119
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Preparing Working Papers -
Example of Descriptions of Actions Taken and Tests
Performed

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 120
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Preparing Working Papers -
Example of an Interim Summary

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 121
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Preparing Working Papers -
Example of “To Do” Items (pending and completed)

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 122
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Preparing Working Papers -
Example of an Exhibit List

MDG - 85

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 123
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Preparing Working Papers -
Review of Working Papers

Working papers should undergo a two-stage review process: (1)


on-site review by each individual auditor of their own working
papers, and (2) post-audit review by the team leader of each
auditor’s working papers.

1. On-Site Review

Throughout the audit, each auditor should review his/her working


papers frequently (i.e., at least daily and preferably several times
during each audit day) to ensure that all audit tasks have been
completed, open items have been resolved, and adequate
evidence has been gathered to support the findings. The
information included in the working papers should cover all
elements of the audit protocol and leave no unanswered
questions or open items. Working papers should be complete,
free-standing records of the actions taken by the auditor that can
be used to verify and document compliance and noncompliance
situations.

2. Post-Audit Review

Immediately following each audit, and before the draft audit


report is issued, all working papers should be reviewed by the
audit team leader. This review provides for a quality assurance
check on both the individual auditor and the audit topic(s)
covered. The reviewer should document his/her review by
signing and dating each set of working papers reviewed. The
audit report and the exit meeting discussion sheets should be
included in this review to ensure that the findings are reported
appropriately and are substantiated by sufficient evidence
documented in the working papers.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 124
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Preparing Working Papers -
Review of Working Papers

Sample Checklist to Review Working Papers

Format

Each working paper page is clearly labeled with the applicable protocol
step.

Sources of information are clearly identified.

All exhibits are referenced in the working papers.

Each page is sequentially numbered, initialed, and dated.

Cross-outs are initialed; postscripts or afterthoughts are written in a


manner that provides appropriate context.

Content

Each protocol step was addressed in accordance with the instructions


provided.

Any departures from the protocol are described and explained.

A description of actions taken to complete each protocol step has been


documented.

An understanding of how the facility is managing the items under


review has been documented.

The conclusions reached as a result of testing have been documented.

All audit findings have been clearly identified.

All findings in the working papers have been included on the audit exit
meeting discussion sheet.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 125
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 4
Preparing Working Papers

In this exercise, we will role play another interview between an


auditor and a plant representative. The principal objective of this
exercise is to practice documenting relevant information obtained
from the interviewee. Of course, you will also want to practice
the skills from the last exercise – planning the interview, opening
the discussion, varying your question patterns, etc.

You will have the same role number (1, 2, or 3) as in the previous
interviewing exercise. However, the roles have been switched. If
your number is ‘1’, you will be the ‘Observer’ for this role play. If
your number is ‘2’, you will be the ‘Plant Representative’. If your
number is ‘3’, you will be the ‘Auditor’. There are different sets of
instruction for each role; please read only the instruction for your
assigned role, as listed on the following pages

Health and Safety Specialists

No. 1 Observer Page 127


No. 2 Plant Representative Page 128 - 129
No. 3 Auditor Page 130

Environmental Specialists
No. 1 Observer Page 131
No. 2 Plant Representative Page 132 – 133
No. 3 Auditor Page 134

We will break into groups (Observers, Auditors, and Plant


Representatives in separate rooms). You will have 10 minutes to
prepare for your role, 15 minutes to conduct the interview, and 5
minutes for feedback.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 126
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 4A
Preparing Working Papers –
Safety Inspection Programme

Observer’s Instructions

Your role in this exercise is to document the interview being


conducted and to critique the auditor’s working papers. The
auditor’s assignment is to understand and document the
implementation of the safety inspection programme as conducted
within Department 901’s Superintendent.

During the interview, document the responses of the


superintendent in your working papers. At the completion of the
interview, obtain the auditor’s working papers and: (1) compare
them with your own working paper notes (e.g., Identification of
topic and interviewee, content and completeness, legibility, etc.);
(2) compare them with working paper principles and techniques;
and (3) note strengths and weaknesses of these working papers.

Helpful Hint:

To minimise the distraction created by your role in this exercise,


position yourself to the side of the other two participants and
make your note-taking as unobtrusive as possible.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 127
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 4A
Preparing Working Papers –
Safety Inspection Programme

Plant Representative’s Instructions

Your role in this exercise is to play the part of the Department


901 Superintendent. You should co-operatively and
comprehensively provide information to the auditor.

Some information about the equipment safety inspection


programme for Department 901 is listed below:

• Department 901 has a safety committee (composed of


yourself, two other management representatives) that meets
twice per month to discuss safety issues of concern and to
tour the department to conduct safety inspections.

• The pressure of production over the last two years have


resulted in certain periods where the committee has failed
both to meet and to conduct the inspections

• The result of each inspection are quickly reviewed by the


committee; the two management representatives convey the
results to specific work areas for corrective actions.

• No formal corrective action plans are developed but you


believe things are getting addressed.

During the interview, you should project a friendly,


accommodating demeanor. You are free to develop additional
information in responding to the auditor’s questions, and should
provide an abundance of information (all of which need not be
relevant to the aforementioned topic). The purpose of your role
in this exercise is to force the auditor and the observer to think
before they write: rather than acting as stenographers who record
every word, the auditor and the observer should ideally record
only that information which is important and pertinent to the
aforementioned topic.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 128
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 4A
Preparing Working Papers –
Safety Inspection Programme

Be imaginative in developing your description of the facility’s


safety inspection programme and your responses to the auditor’s
questions.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 129
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 4A
Preparing Working Papers –
Safety Inspection Programme

Auditor’s Instructions

Your role in this exercise is to play the part of a safety and


industrial hygiene auditor. During your interview with the
Department 901 Superintendent, you wish to develop and
document your understanding of how the department manages
its safety inspections. You may want to consider the following
elements in preparing for your interview:
• Responsibility for conducting inspections
• Frequency of inspections
• Documentation/Reporting of results
• Corrective actions/follow-up
• Training

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 130
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 4B
Preparing Working Papers – Spill Response

Observer’s Instructions

Your role in this exercise is to document the interview being


conducted and to critique the auditor’s working papers. The
auditor’s assignment is to understand and document how the
facility manages its spill response equipment, distribution, and
training programme. The auditor will be interviewing the facility’s
emergency spill response team leader.

During the interview, document the responses of the facility’s


emergency spill response team leader in your working papers.
At the completion of the interview, obtain the auditor’s working
papers and: (1) compare them with your own working paper
notes (e.g., identification of topic and interviewee, content and
completeness, legibility, etc.); (2) compare them with working
paper principles and techniques; and (3) note strengths and
weaknesses of these working papers.

Helpful Hint

To minimise the distraction created by your role in this exercise,


position yourself to the side of the other two participants and
make your note-taking as unobtrusive as possible.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 131
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 4B
Preparing Working Papers – Spill Response

Plant Representative’s Instructions

Your role in this exercise is to play the part of the facility’s


emergency spill response team leader, who is co-operative and
proud of the facility’s emergency spill response team and of its
training and performance.

Your understanding of the quantities of materials currently stored


at the facility is:

Tank Farm
Acrylonitrile 2 x 15,000 gallons
Toluene 1 x 10,000 gallons
Fuel Oil 2 x 10,000 gallons
Sulfuric Acid 1 x 10,000 gallons
Hexane 2 x 10,000 gallons
Isobutyl Methacrylate 1 x 5,000 gallons

Other
Chlorine 6 x 1-ton cylinders manifolded
at wastewater plant
12 x 1-ton full spares
Ammonia 1 x 20-ton tank
Emulsifiers 55-gallon drums
Catalysts 55-gallon drums
Other additives 55-gallon drums; 50-pound
bags

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 132
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 4B
Preparing Working Papers – Spill Response

Some brief information about the facility is listed below:

• Absorbent pads, pillows, along with absorbent granule


supplies are maintained in a central warehouse for distribution
upon request.

• The spill team should receive monthly refresher training and


conduct a mock spill event once every six months

• The spill team has its own meeting room and storage area for
self-contained breathing apparatus, chemical resistant suits,
etc.

• The spill team’s mock spill conducted in August revealed that


some of the newer members of the team were not familiar
with procedure and did not know what action to take in the
event of a facility spill. As spill team training has been patchy
since August, follow-up on this shortcoming has not yet taken
place.

During the interview you should project a friendly,


accommodating demeanor. You are free to develop additional
information in responding to the auditor’s questions, and should
provide an abundance of information (all of which need not be
relevant to the aforementioned topic). The purpose of your role
in this exercise is to force the auditor and observer to think before
they write; rather than acting as stenographers who record every
word, the auditor and the observer should ideally record only that
information which is important and pertinent to the
aforementioned topic.

Be imaginative in developing your description of the facility’s spill


response programme and your responses to the auditor’s
questions.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 133
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 4B
Preparing Working Papers – Spill Response

Auditor’s Instructions

Your role in this exercise is to play the part of an environmental


auditor. During your interview with the facility’s spill team
captain, you wish to develop and document your understanding
of the activities conducted at the facility relating to spill response.
You may want to consider the following elements in preparing for
your interview:
• Hazardous materials on-site
• Spill equipment available for use
• Storage and maintenance of spill equipment
• Spill team training
• Mock drills and follow-up

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 134
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Assessing Strengths and Weaknesses

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 135
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Assessing Strengths and Weaknesses

Pre-Audit Activities On-Site Activities Post-Audit Activities

Step 1: Understand Prepare Draft Report


Management Systems
Conduct opening meeting Obtain Review Comments from
Select & Schedule Facility Audit Conduct orientation tour Corporate HSE
Review audit strategy Law department
Understand details of Facility management
management systems

Step 2: Assess strengths Issue Final Report to


& Weaknesses Facility management
Select Team Members & Confirm Consider potential impacts Operations
their Availability Evaluate management Corporate HSE
systems Law department
Set priorities for verification

Step 3: Gather Audit Evidence


Plan the Audit: Evaluate what needs to be
Correspond with the done Develop Action Plan
facility Determine depth & rigor of Develop proposed action(s) to
Assemble & distribute review address each finding
background information Select types of evidence Assign responsibility for
Assign & communicate needed & methods to gather corrective action
audit responsibilities them Develop timetable
Conduct pre-audit Compare practices against
meeting requirements
Document results

Conduct Follow-up
Step 4: Evaluate Audit Results Track status of corrective
Evaluate audit results actions
Write audit findings Confirm closure of findings

Step 5: Report Audit Findings


Conduct exit meeting

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 136
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Assessing Strengths and Weaknesses -
Purpose of Assessing Strengths and Weaknesses

Once the auditors have developed an understanding of how the


facility manages each of the environmental, health, and safety
programmes included within the scope of the audit, the next step
is to evaluate the soundness of these management systems in
the context of the potential environmental, health, and safety
impacts.

The purpose of this assessment is to determine verification


priorities for Step 3 activities. Spending the time to evaluate
priorities will enable the auditor to maximise the effectiveness
and efficiency of gathering the evidence needed to achieve the
objectives of the audit.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 137
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Assessing Strengths and Weaknesses -
Process for Assessing Strengths and Weaknesses

Consider Potential Determine the range of potential impacts if a particular HSE issue is not
Impacts managed appropriately

Evaluate the management systems to determine if they are designed


Evaluate Management soundly. That is, consider if the systems, coupled with the controls, are
Systems appropriate given the potential impacts

Set priorities for verification so as to provide the optimum allocation of


Set Priorities for available team resources to ensure that issues representing high risk and
Verification weak management / control systems receive sufficient attention

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 138
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Assessing Strengths and Weaknesses -
General Approach for Considering Potential Impacts

Potential impacts refer to the range of potential consequences for


the facility or company arising from an event or activity. In
evaluating potential impacts, the auditor should ask: “If a health,
safety and environmental activity at the facility is not managed
appropriately, what consequences could ensue?”

Potential impacts might include:


• Catastrophic events (e.g., explosion)
• Loss of life (e.g., confined space entry procedures)
• Injury or illness (e.g., levels of airborne pollutants in the
workplace)
• Environmental damage (e.g., spills of hazardous materials to
navigable waters or drinking water supplies)
• Legal or financial liability (e.g., air permit violations)
• Loss of operation and production
• Adverse publicity (e.g., release of visible hazardous toxic
substance to air)
• Recordkeeping or reporting exceptions
• Occasional spills or releases to the environment
• Employee/community exposures to hazardous/toxic
substances
• Loss of property (e.g., fire)

The auditor should evaluate the potential impacts for each


protocol topic based upon a high to low spectrum.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 139
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Assessing Strengths and Weaknesses -
Assessing Health, Safety and Environmental
Management Systems

Evaluating the soundness of a facility’s health, safety and


environmental management of key programmes is inherently a
subjective process. While regulations tend to stipulate explicit
performance or technology requirements, explicit
criteria/standards as to what constitutes an adequate
management system are only beginning to emerge. In some
instances, however, a corporation, division, or facility may have
developed its own guidelines or policies as to how a particular
activity or function is to be managed.

In those situations where there are established management


criteria—be they regulatory or internal—the auditor can look to
the criteria for assistance in assessing the soundness of
management systems. In all other instances, the guidance on
the following pages provides a framework for assessing the
strengths and weaknesses of the health, safety and
environmental management systems.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 140
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Assessing Strengths and Weaknesses -
General Principles for Assessing Management
Systems

Given the absence of explicit criteria for assessing management


systems, the following guidance may be helpful.

Management System Descriptors

Assessing The facility has a process in place for: 1) identifying and


evaluating conditions and/or materials on site that have
regulatory implications, and 2) understanding the scope of
applicable regulatory requirements.
• Assessments are performed to identify potential risks (e.g.,
leaking underground storage tanks or pipes, carcinogens in
the workplace).
• Environmental, health, and safety evaluations are
conducted for modified products, processes, and operating
ventures and signed off by health, safety and environmental
staff.
• Programmes are in place to keep abreast of regulatory
changes, to interpret the applicability of those changes to
facility operations, and to develop procedures to address
those changes.
Planning The facility has developed procedures and systems for
managing compliance (e.g., plans, procedures, policies).
• Appropriate procedures have been established to respond
to unintended events, such as process shutdowns, as well
as to notify appropriate groups within the corporation and in
the community.
• Usable programme and procedure guidance exists to direct
facility activities to achieve health, safety and environmental
goals consistently.
• A system has been developed for recordkeeping which
provides documentation of health, safety and environmental
activities and compliance with governmental requirements
and company policy.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 141
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Assessing Strengths and Weaknesses -
General Principles for Assessing Management
Systems

Management System Descriptors

Implementing The facility has structures and/or equipment in place to


manage or implement compliance (e.g., pH meters, approved
vendor lists, methods to address nonconformance situations,
etc.).
• Controls for equipment and storage areas are maintained in
an operable manner.
• Systems are in place to address nonconformance situations
(e.g., permit exceedances, internal alarms).
Reviewing The facility has developed a process for periodically reviewing
and monitoring compliance programmes (e.g., self-
inspections, audits, supervisory review of data to cross-verify
reports).
• The functionality of the controls is periodically tested.
• Periodic and comprehensive inspection programmes are in
place.
• Deficiencies identified during inspections are corrected in a
timely manner.
Organisation Clear roles and responsibilities have been established to
manage compliance with applicable regulations.
• Roles and responsibilities are clearly understood with
respect to health, safety and environmental functions.
• Health, safety and environmental staff have access to
appropriate line management to discuss issues and
concerns.
Resources Qualified and sufficient health, safety and environmental staff
and/or line personnel are involved in compliance
management.
• Staffing levels are appropriate to obtain environmental,
health, and safety compliance management goals.
• Clear accountability exists for health, safety and
environmental performance.
Training Programmes are in place to familiarise staff with the nature
and scope of compliance programmes.
• Health, safety and environmental staff have appropriate
education, training, and experience to fulfill assigned duties.
• Information and education programmes are sufficient to
enable employees to carry out health, safety and
environmental functions.
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 142
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Assessing Strengths and Weaknesses -
General Principles for Assessing Management
Systems

Management System Descriptors

Documentation/ The facility has an accessible and orderly recordkeeping


Information system for compliance-related activities (e.g., training,
Management monitoring, governmental correspondence) and mechanisms
in place to communicate relevant information between the
various levels and functions of the organisation (i.e.,
memoranda, weekly meetings).
• Records are accessible and managed in an orderly fashion.
• Sufficient information is reported to management and
outside agencies, as appropriate.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 143
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Assessing Strengths and Weaknesses -
General Principles for Assessing Management
Systems

Based on the information gathered during Step 1: Understand


Management Systems, the auditor should assess the strengths
and weaknesses of the facility’s approach to managing each
protocol topic. An example of such an assessment (to verify
compliance with a wastewater permit) is illustrated below.

Management
Strengths Weaknesses
System Activity
Assessing Hired an outside Do not have formal procedure
consultant five years ago to review capital projects for
to review operations and wastewater impacts.
identify wastewater issues.
Do not keep P&IDs current.
Have hard copies (2004) of
regulations on site. Have not reviewed latest
Corporate HSE provides stormwater regulations.
monthly updates on
changes.
Planning Have formal inspection Do not have formal inspection
and preventive and PM programme for
maintenance (PM) laboratory equipment.
programme for
Wastewater Treatment Do not have written pollution
Plant equipment. prevention plan.

Have site-specific Sampling and analysis manual


wastewater sampling and does not address stormwater
analysis manual based on sampling.
procedures in 40 CFR 136.
Plant manager’s secretary
Have reduced total types final Discharge
wastewater volume by Monitoring Report for
25% in past two years. signature, keeps file copies,
and does any exceedance
Lab chemist prepares draft reporting. HSE coordinator
Discharge Monitoring does not review final
Report. HSE coordinator documents.
reviews report and raw
data before sending to Stormwater pollution
plant manager for typing prevention plan is based on
and signature. draft general permit.
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 144
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Assessing Strengths and Weaknesses -
General Principles for Assessing Management
Systems

Management
Strengths Weaknesses
System Activity
Implementing Have continuous monitors Do not do sampling and
for wastewater flow, analysis of process-specific
temperature, and pH at inputs to Wastewater
outfall. Wastewater Treatment Plant.
Treatment Plant operator
is responsible for Have not identified points for
preventative measures. stormwater sampling.
Inspected and calibrated
weekly. Process upsets cause “slug”
flow to Wastewater Treatment
Periodically have outside Plant; no advance notice given.
laboratory analyze split
samples as quality control Laboratory wastewater goes to
check. Publicly owned treatment
works, not Wastewater
Treatment Plant.
Reviewing Reviewed and updated site Do not have formal self-
policies and procedures in inspection programme.
June 2000.

HSE coordinator spends


50% of time out in plant;
sees/hears about changes
as they happen.
Organisation Roles and responsibilities Operations manager clearly
are defined in job considers wastewater
descriptions for compliance to be the
Wastewater Treatment responsibility of others (e.g.,
Plant operator, lab Wastewater Treatment Plant
chemist, HSE coordinator; operator); end-of-pipe
well understood. mentality.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 145
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Assessing Strengths and Weaknesses -
General Principles for Assessing Management
Systems

Management
Strengths Weaknesses
System Activity
Resources Have sufficient number of
qualified staff for
Wastewater Treatment
Plant and lab.

Person who does sampling


and analysis is trained
chemist with >10 years
experience.

HSE coordinator has


degree in environmental
engineering.
Training Chemist and HSE Have not trained internal staff
coordinator are on stormwater issues.
encouraged to attend
outside meetings/courses Have not provided operations
2-3 times per year. personnel awareness training
on wastewater issues.
Documentation Excellent laboratory
recordkeeping.

Issue weekly newsletter to


all plant personnel that
includes information on
compliance with
Wastewater Treatment
Plant effluent limitations.
Information Have automated system Do not have formal records
Management for tracking training retention policy.
requirements.
Do not give notice to HSE
HSE coordinator has coordinator when DMR is
automated system to alert actually sent.
him when Discharge
Monitoring Report is due.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 146
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Assessing Strengths and Weaknesses -
Balancing Management Systems/Controls Strengths
vs. Potential Impacts

Your overall judgment reflects, to a large extent, whether the


facility has strong enough systems in place to minimise the
potential impacts.

Balanced Approach “Heavyweight” System “Lightweight” System


to Manage to Manage
Low Potential Impacts High Potential Impacts
s
tem
Pote
n Sys d
Imp tial a n
Systems acts trols
Potential Con
and Sys l
Impacts tem ntia
Controls
and s Pote cts
a
Con Imp
trols

(pretty much OK) (may reflect an imprecise (may be vulnerable


understanding of to surprises)
impacts)

Often strengths or weaknesses in the management systems are


linked to the presence or absence of several of the activities.
The key is to identify which are most relevant to achieve a
balanced approach.

For example, a redundant (“belt plus suspenders”) system may


be appropriate for high-impact situations, while a more
straightforward (“strong leather belt”) approach may suffice
where impacts are not as significant.

Poor or deficient management systems can lead to “errors or


omissions,” (e.g., error—failure to perform atmospheric testing
prior to entry into a confined space; omission—failure to
reschedule new employees for training who missed the initial
training) and there are some health, safety and environmental
situations in which an error and/or omission could have a
substantial impact. Strong controls also reduce the risk of either
an error or omission.
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 147
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Assessing Strengths and Weaknesses -
Setting Priorities for Verification

Following the completion of the previous tasks, the auditor is now


in a position to establish priorities for verification, that is, an
approach to gathering data that will provide the auditor with
sufficient data to draw conclusions regarding compliance with
established standards and the effectiveness of the management
systems.

Setting priorities for verification should be dependent on the


auditor’s evaluation of the strengths and weaknesses of the
management systems, combined with an assessment of the
potential impacts associated with a particular topic. This
approach is based on the premise that strong management
systems can (and should) be employed to mitigate otherwise
high potential impacts.

As a conceptual model, the auditor can “grid” each of the protocol


topics covered to assist in ranking the priorities for verification.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 148
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Assessing Strengths and Weaknesses -
Setting Priorities for Verification

In the empty matrix presented below, indicate how you would


prioritise your time for the balance of the audit. Put a “1” in the
box corresponding to your highest priority, a “2” in the box
corresponding to your second highest priority, etc. That is, graph
the strength of the management system on the y axis against the
potential impacts on the x axis.

Management Potential Impacts


Systems
Low
Low High

Weak
Weak

Strong

The process of “gridding” assists the auditor in setting the


verification priorities. That is, the gridding helps auditors select
an approach to data gathering that emphasises an in-depth
review of areas where potential impacts are high and
management systems are weak.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 149
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Assessing Strengths and Weaknesses -
Setting Priorities for Verification

For example:

• If there are significant potential impacts associated with


noncompliance with a particular topic, and the management
systems associated with that topic are judged to be weak, the
auditor should ensure that sufficient time is allocated to
reviewing that topic.

• Conversely, if there are low potential impacts associated with


noncompliance with a particular topic and the management
systems associated with that topic are judged to be strong,
the auditor need only spend a relatively small amount of time
reviewing that topic.

By developing priorities for verification based on an assessment


of the strengths and weaknesses of management systems, and
potential impacts associated with noncompliance, the auditor
helps ensure that the highest priority issues are covered in
significant depth during the audit.
Verification Priorities

High/Weak

High/Strong

Low/Weak

Low/Strong

Level of Effort

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 150
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Assessing Strengths and Weaknesses -
Setting Priorities for Verification

Auditors should recognise that it is generally not useful, in terms


of setting priorities for verification, to conclude that all protocol
topics are of equally high (or low) priority. Furthermore, it is
generally not true that potential impacts are equally high (or low)
and management systems are equally strong (or weak) for all
topics in a given protocol. It is important for the auditor to use the
Step 2 process to set priorities for Step 3. The assessment of
priorities can be discussed as a team to provide a broader
perspective of potential impacts and evaluation of the
management systems.

Following the development of verification priorities, the team


should once again confirm that resources have been
appropriately allocated to complete the audit.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 151
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 5A
Assessing Strengths and Weaknesses

Objective

The purpose of this exercise is to:

• Assess the potential strengths and weaknesses of an HSE


programme and the potential impacts if that particular issue is
not managed appropriately.

• Prioritise the order in which the protocol elements should be


completed based on your assessment of the management
systems and the potential impacts.

Instructions

You have been assigned the Hazardous Waste Management


Audit Protocol. During Monday and Tuesday of the week that
you are auditing the Woodmount Company, you have interviewed
several key people who are involved in hazardous waste
management. Attached is a summary of the information that you
have compiled as you begin to understand the management
systems surrounding hazardous waste management. Based on
your understanding of the management systems and on your
assessment of the risks involved, rank the following elements of
your protocol:

Potential Management
Protocol Element Impacts Systems Priority
(low/high) (weak/strong)
Hazardous Waste
Manifests
Waste Accumulation
and Storage
Waste Classification
Training

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 152
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 5A
Assessing Strengths and Weaknesses

Based on interviews with selected key staff, you have learned the
following:

• The work force is quite stable and most of the employees on


site have been there for at least five years.

• The facility has a hazardous waste generator number which


was obtained about ten years ago. The number is CAG 110
070 001.

• The facility generates hazardous wastes, including used


solvents and oils generated in the process of maintaining and
lubricating machinery, support equipment, and machine shop
activities.

• Annually, production supervisors are required to review the


waste profiles and initial the file copy to confirm that there
have been no process changes that would alter the waste
composition. Every five years, the facility hires a recognised
consultant to review its waste characterisation programme.

• Hazardous wastes are stored in a locked shed adjacent to the


main building.
− The only light in the shed comes from the indirect light
through the window openings near the ceiling. Employees
are instructed not to enter the shed after dark. Exterior
spot lights are available in the event of a nighttime
emergency.
− Floor drains lead to a small sump, which can be pumped
out manually if necessary.
− You were told that the facility has an ample supply of
sorbent materials, but have not seen it yet.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 153
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 5A
Assessing Strengths and Weaknesses

• The treatment, storage, and disposal facility (TSDF), which is


also the transporter, prepares the manifests for the facility
when it picks up a waste load.
− The TSDF comes whenever the facility calls.
− The TSDF leaves the generator copy of the manifest with
the facility.
− Six different Woodmount employees have signed
manifests in the last ten months.

• The receptionist keeps the facility’s manifests.


− During the previous year, 25 shipments of hazardous
waste were sent off site to the TSDF.
− Whenever a signed TSDF copy of a manifest comes in the
mail, the receptionist inserts it in the file with the original
generator copy. (The receptionist noted that this usually
occurs within a month of the shipment.)
− The current year’s manifests are kept in the receptionist’s
files; older records are boxed and sent to a storage area in
the manufacturing building.
− The sample manifest you pulled from the file was done on
a 1984 form.

• The maintenance supervisor prepares the annual reports by


recording the volume of waste per manifest and submits them
to the regulatory agency every year by March 1.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 154
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 5A
Assessing Strengths and Weaknesses

• Facility hazardous waste training programmes are conducted


once per year.
− They last approximately eight hours.
− The topics covered include HAZCOM, spill cleanup, and
an overview of hazardous waste management rules,
including waste characterisation.
− Training is provided by an outside consultant, who submits
a contract every year describing the approach and scope
of the services rendered. The contract includes a
provision for “back-up” training of staff who are absent
from the on-site session.
− An agenda is prepared to reflect the topics to be covered
during the day-long session.
− Production, maintenance, and selected clerical staff and
supervisors attend the yearly sessions. Attendees are
required to sign in for both the morning and afternoon
sessions. Copies of the attendance records are kept by
the maintenance supervisor.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 155
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 5B
Assessing Strengths and Weaknesses

Objective

The purpose of this exercise is to:

• Assess the potential strengths and weaknesses of an HSE


programme and the potential impacts if that particular issue is
not managed appropriately.

• Prioritise the order in which the protocol elements should be


completed based on your assessment of the management
systems and the potential impacts.

Introduction

You have been assigned the Process Safety Management (PSM)


Audit Protocol. During Monday and Tuesday of the week that
you are auditing the Woodmount Company, you have interviewed
several key people who are involved in process safety
management. Woodmount is subject to the OSHA PSM
Standard because it stores and uses more than 10,000 pounds
of flammable materials, chlorine, and ammonia on site. The
facility has three process areas that are covered by the standard.
Two of the process areas use flammable materials only. The
third process area uses chlorine and ammonia. Attached is a
summary of the information that you have compiled as you begin
to understand the management systems surrounding process
safety management. Based on your understanding of the
management systems and on your assessment of the risks
involved, rank the following elements of your protocol:

Potential Impacts Management Systems


Protocol Element Priority
(low/high) (weak/strong)
Operating Procedures
Mechanical Integrity
Training
Process Hazard
Assessments/HAZOPs

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 156
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 5B
Assessing Strengths and Weaknesses

Based on interviews with selected key staff, you have learned the
following:

• The work force is quite stable and most of the employees on


site have been there for at least five years.

• The facility has an objective of keeping all operating


procedures up to date. Every three years, the operating
procedures are reviewed and updated as necessary.

• The facility has a standard for how to develop and write


operating procedures.

• The supervisor for each department is responsible for


maintaining operating procedures.

• The operations manager approves all operating procedures.

• The availability of operating procedures is verified during pre-


startup reviews.

• Operating procedures are updated by production engineers


and are reviewed by a committee consisting of the safety
manager and senior production operators.

• Changes to operating procedures are summarised in memos


which are kept in a separate section of the operating
procedures manual.

• The facility does not maintain documentation of maintenance


performed on individual pressure relief devices.

• The facility has not completed an inventory of all pressure


relief devices.

• Maintenance personnel have not yet received formal training


in plant safety.
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 157
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 5B
Assessing Strengths and Weaknesses

• Associated piping to and from raw material and product


storage tanks is not included in the facility’s preventive
maintenance (PM) programme.

• The facility’s computerised PM system has not been set up to


archive the completion of the PMs, nor does the facility
maintain a hard copy of this information.

• The facility has not yet implemented a periodic visual


inspection and/or nondestructive testing programme for its
critical equipment (e.g., storage tanks, piping, and process
vessels).

• The facility maintains a matrix of all job titles and their


applicable training requirements.

• The facility has developed training “blocks” in order to


facilitate consistency. A refresher training programme has not
yet been established to provide training every three years to
operators. Most operators are due to receive refresher
training within the next four to six months.

• The facility’s training programme concludes with a testing of


the employee’s understanding of the material presented using
a written exam and a practical.

• The facility has completed process hazard analyses (PHAs)


for all three covered processes using the hazard and
operability (HAZOP) methodology.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 158
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 5B
Assessing Strengths and Weaknesses

• The HAZOP studies included representatives from the


following facility organisations:
− Operations
− Maintenance (instrumentation and process)
− Engineering
− HSE department, if deemed necessary by Engineering

• HAZOPs conducted included formal feedback from operators.

• Results of the HAZOPs are not consistently communicated to


personnel.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 159
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 5A – Potential Answers
Assessing Strengths and Weaknesses

Potential Management
Protocol Element Impacts Systems Priority
(low/high) (weak/strong)

Hazardous Waste Manifests Low Weak 3

Waste Accumulation and


High Weak 1
Storage

Waste Classification High Strong 2

Training Low Strong 4

Exercise 5B – Potential Answers


Assessing Strengths and Weaknesses
Potential Management
Protocol Element Impacts Systems Priority
(low/high) (weak/strong)

Operating Procedures High Strong 4

Mechanical Integrity High Weak 1

Training High Strong 3

Process Hazard
High Weak 2
Assessments / HAZOPs

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 160
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Gathering Audit Evidence

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 161
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Gathering Audit Evidence

Pre-Audit Activities On-Site Activities Post-Audit Activities

Step 1: Understand Prepare Draft Report


Management Systems
Conduct opening meeting Obtain Review Comments from
Select & Schedule Facility Audit Conduct orientation tour Corporate HSE
Review audit strategy Law department
Understand details of Facility management
management systems

Step 2: Assess strengths Issue Final Report to


& Weaknesses Facility management
Select Team Members & Confirm Consider potential impacts Operations
their Availability Evaluate management Corporate HSE
systems Law department
Set priorities for verification

Step 3: Gather Audit Evidence


Plan the Audit: Evaluate what needs to be
Correspond with the done Develop Action Plan
facility Determine depth & rigor of Develop proposed action(s) to
Assemble & distribute review address each finding
background information Select types of evidence Assign responsibility for
Assign & communicate needed & methods to gather corrective action
audit responsibilities them Develop timetable
Conduct pre-audit Compare practices against
meeting requirements
Document results

Conduct Follow-up
Step 4: Evaluate Audit Results Track status of corrective
Evaluate audit results actions
Write audit findings Confirm closure of findings

Step 5: Report Audit Findings


Conduct exit meeting

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 162
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Gathering Audit Evidence -
Purpose of Gathering Audit Evidence

The purpose of Step 3 is to gather data to verify that the facility is


in compliance with applicable regulations and that HSE
management systems are functioning as intended.

Auditors gather data during Step 3 by developing and


implementing a verification strategy. A verification strategy, in
the simplest sense, is an expansion of the Step 2 assessment in
that it details:

• The specific focus or goal of the data gathering;

• The types of data to be gathered and why; and

• The range of tests to be performed to confirm the validity of


facility information.

This section of the manual explains the process by which (and


the factors to consider) to develop a verification strategy that
complements the priorities established in Step 2.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 163
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Gathering Audit Evidence –
Basic Tasks

There are several basic tasks the auditor should follow in


formulating a sound verification strategy.

Task Description
1. Evaluate what needs to be done “What do I need to look at and how
much time do I have to do it?”

2. Determine the depth and rigor of “How deep do I need to dig? How many
review kinds of evidence do I need to gather?”

3. Select the types of evidence “What types of data will I gather and
needed and the methods to how can I collect them?”
gather them
4. Compare practices against “How does the facility rate against the
requirements applicable requirements?”

5. Document results “What information do I need to record


to help me remember what I have
learned and substantiate my
conclusions?”

Each of these tasks is described on the following pages.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 164
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Gathering Audit Evidence –
Basic Tasks

1. Evaluate What Needs to Be Done

To initiate the formulation of a sound verification strategy, the


auditor should first develop a good appreciation for what he/she
needs to review and how much time there is to do it all. As such,
the auditor should:

• Determine the specific objectives of the protocol steps for


review and then prioritise them.

• Review the audit resources.

Determine the Specific Objectives of the Protocol Steps for


Review and Then Prioritise Them
As the auditor develops a verification strategy, he/she may still
need to determine the specific objectives of the protocol steps for
review and then prioritise them. This task builds on what the
auditor has identified as priorities based on the assessment
made in Step 2.

The task of understanding and appreciating the objectives of the


protocol steps can be more subtle than merely identifying a topic.
For instance, suppose that an auditor has identified spill
response training as an item that must be verified during the
audit. In this situation, there could be two different aspects to
verifying the training requirement. On the one hand, the auditor
may only want to verify that all applicable employees have had
training during the past year. Accordingly, the auditor will likely
review training records and attendance sheets. Alternatively, if
the auditor wanted to verify that training was adequate or
appropriate to the level of the trainees, then he/she may take a
different approach toward verification, such as talking with
employees who were supposed to be trained and/or evaluating
the content of the training manuals and comparing them to
regulatory requirements and/or industry practices.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 165
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Gathering Audit Evidence –
Basic Tasks

Prior to determining the types of data to gather, the auditor


should quickly review the protocol steps to develop a full
understanding of what needs to be verified. For example, a
protocol step might direct the auditor to verify that a written
training programme has been developed or to verify the
effectiveness of the training programme. Understanding these
types of differences will be important when selecting/identifying
the types of data to review.

Following this step, the auditor should prioritise the steps that will
be completed. This task builds upon the priorities established
during Step 2. That is, if an auditor determined in Step 2 that
priority should be given to reviewing a facility’s respiratory
protection programme, he/she now needs to decide what
emphasis should be given to the various elements of the
respiratory protection programme—verifying that a written
programme has been established, training is conducted,
inspections are performed, etc. The auditor should base his/her
decisions on those elements that will provide the greatest
insights regarding the functioning/implementation of the
respiratory protection programme.

Review the Audit Resources


The auditor should review the audit resources to determine
whether the audit strategy established in the pre-audit planning
stage is still appropriate or if any modification or reallocation of
audit resources is necessary to meet the objectives of the
assignment. As a team, auditors should discuss whether:

Audit team resources have been appropriately allocated to


adequately cover all functional areas.

Protocol topics do not overlap in such a manner that on-site


activities are duplicated by two or more team members.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 166
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Gathering Audit Evidence –
Basic Tasks

2. Determine the Depth and Rigor of Review

In formulating a reliable and defensible verification strategy, the


auditor needs to determine how much data to collect and how
deep to dig—in other words, the depth and rigor of review. To
make this determination, the auditor should rely on the
assessment of the potential impacts of the issues or activities
being reviewed and the assessment of the soundness of the
management systems made in Step 2. For example:

Management Potential Impacts


Systems
Low
Low High
3 1
Weak
Weak Dig Less Deep Dig Deepest

4 2
Strong Don’t Dig? Dig Deep

After the auditor has determined how deep to dig, he/she is now
in a position to select what types of information to gather and
how.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 167
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Gathering Audit Evidence –
Basic Tasks

3. Select the Types of Evidence Needed and the Methods to


Gather Them

The auditor should now determine what types of evidence are


potentially available to be gathered and what methods would
need to be utilised to gather them.

In developing an approach for collecting evidence, the auditor


should take into account what he/she has learned thus far in the
audit, especially in Step 1, so as to get right to the business of
verification. In this task, the following activities take place:
• Consider what was learned in Step 1
• Select the type(s) of audit evidence
• Decide whether and how to design a test
• Develop sampling strategies

Consider What Was Learned in Step 1: Understand


Management Systems
As auditors identify the types of evidence they will gather and the
methods to be employed in gathering them, they should consider
what was learned in Step 1: Understand Management Systems,
because the nature of the environmental, health, and safety
activities and management structures at the facility can influence
the verification strategies. For example:

• If the auditor is trying to confirm that a facility completely lacks


a hazard communication programme, he/she should not
expect to find any physical or documentary evidence because
obviously there is no written document describing the non-
existent programme. Instead, the auditor will base this
conclusion on the testimonial evidence.

• If the spill plan has not been updated in four years, then
perhaps the auditor will need to focus efforts on gathering
physical data to determine whether and how current activities
compare against the elements of the plan.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 168
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Gathering Audit Evidence –
Basic Tasks

• If a facility has different operations and/or different


organisational structures at one location, then the auditor will
want to ensure that any data-gathering activity is
representative of each operation. For instance, if a facility has
two different product lines, each of which is managed by a
different division, then the auditor may need to go to two
different sources to verify that employees received safety
training. This factor should be taken into account when
evaluating how much time is needed to complete a particular
protocol step.

Select the Type(s) of Audit Evidence


In general, there are four different types of evidence (also
referred to as “audit data” or “audit information”) that can be
gathered during an audit, and each can affect the quality and
reliability of the audit results. Each type of evidence is
associated with a particular method, and each has advantages
and limitations.
• Physical evidence
• Documentary evidence
• Testimonial evidence
• Circumstantial evidence

The auditor can strengthen the weight of the evidence by


obtaining several different types and sources of
evidence/information.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 169
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Gathering Audit Evidence –
Basic Tasks

Physical Evidence

Definition Something that is seen or can be touched.

Method Physical evidence is obtained through observation whereby the


auditor collects information through physical examination.

Examples High level alarm, secondary containment, label on a container of


chemicals, ethylene oxide monitor, etc.

Advantages Physical evidence is usually one of the most reliable and


persuasive types or sources of data. In many situations, the
physical presence of an object or operation can, by itself, satisfy a
particular requirement of compliance and, therefore, observing that
physical object or operation is essential during an audit to verify
compliance.

Limitations The mere presence of an object or conduct of an operation does


not ensure that it is appropriate for the situation, that it is designed
and functioning properly, or that it will continue to function.

Documentary Evidence

Definition Something written down on paper or recorded electronically.

Method Documentary evidence is obtained through the collection and


ensuing review of something written or recorded.

Examples Facility HSE policies, standard operating procedures, reports,


inspection sheets, etc.

Advantages Documentary evidence allows the auditor to “see” the facility’s


practices in a formal sense (i.e., the documentation) via a paper
trail. As with physical data, the documentation itself is often a
requirement for compliance. Furthermore, documentation of
something makes a strong argument that an activity is indeed
performed.
Limitations Documentary evidence by itself does not tell the auditor that an
activity actually took place. Also, its reliability can be questioned
since documents can be generated or altered while preparing for
the audit.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 170
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Gathering Audit Evidence –
Basic Tasks

Testimonial Evidence

Definition Evidence received from interviews with facility representatives.

Method Testimonial evidence is obtained through inquiry and simply


involves asking questions, both formally and informally.

Examples Anything told to the auditor by facility personnel.

Advantages Testimonial evidence gained through inquiry is a very efficient


method of gathering data during an audit. It is especially helpful
when one is acquiring basic knowledge on HSE systems at a
facility, trying to clarify contradictory information, or obtaining
explanations to unclear items.

Limitations Testimonial evidence obtained from inquiry must consider factors


such as the competency of the questioned individual concerning the
topic (e.g., the person’s training); the interest the person providing
the response(s) has in the subject discussed; any biases that the
individual questioned may have; and the logic and reasonableness
of the question (i.e., was the question understood and appreciated).

Circumstantial Evidence

Definition Indirect evidence which conveys an overall impression.

Method Circumstantial evidence is obtained through an auditor’s developing


a general impression or intuitive feeling about something at the
facility.

Examples The order and neatness of records and files, the attitudes of facility
personnel, the apparent relevance of facility staff’s background and
experience to their HSE responsibilities.

Advantages Circumstantial evidence can be useful in directing where potential


deficiencies may lie within a facility’s HSE management systems.

Limitations Circumstantial evidence is the most unreliable type of evidence


and, thus, should never be used to verify compliance.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 171
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Gathering Audit Evidence –
Basic Tasks

Decide Whether and How to Design a Test


Testing in an audit situation refers to a wide variety of activities1
that can be employed to verify that the facility has implemented
its HSE systems, programmes, and procedures and that the
implementation has been effective in achieving the intended
results. Testing leads to increased confidence that what the
auditor was told in interviews or saw on tours is, in fact, effective.
The focus of testing is frequently on compliance with specific
applicable regulatory and internal requirements.

Testing is most commonly done on documentary evidence, such


as:
• Training records
• Monitoring data and reports
• Material safety data sheets (MSDSs)
• Emission source inventories
• Inspection and maintenance logs
• Incident investigation reports

In the case of documentary evidence, the testing can be done


through vouching, recomputation, retracing data, and/or
confirmation (see the table on the next page). Depending upon
the nature of the specific audit step, any one of these four may
be appropriate.

1 Testing in this context does not mean effluent or emissions sampling or chemical analysis.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 172
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Gathering Audit Evidence –
Basic Tasks

Types of Testing

Vouching This test would uncover errors in reported data and


involves following the paper trail back to the raw data. For
example, hazardous waste shipments recorded in the
annual report might be tracked back to shipping
department records and the hazardous waste manifests.
This process verifies that all reported data are supported.

Recomputation This test checks for the accuracy of arithmetic


calculations. This would include, for example, recalculating
the results of employee monitoring to determine time-
weighted averages.

Retracing Data This test would uncover omissions in reported data and
involves reviewing the original data records to ensure that
all results are appropriately reported. For example, pH
strip charts might be reviewed to identify all excursions
from permit conditions. The auditor would then verify that
no excursions were omitted from the monthly Discharge
Monitoring Reports.

Confirmation This test seeks written confirmation of something from


independent third parties. This test may be used where an
auditor cannot physically observe a condition, such as the
operation of an automatic sprinkler system.

An important part of the concept of “testing” is that auditors


deliberately, independently, and systematically select the specific
pieces of evidence they will look at. Examples of tests are
provided below.

• A test of the MSDS system might involve choosing a sample


of chemicals from the facility’s master inventory, from
purchasing or receiving records, or from chemicals observed
during tours, and verifying that current and accurate MSDSs
are on site for each chemical in the sample. (In contrast,
simply noting whether there are books of MSDSs present in
various locations during a tour would not constitute a “test”,
but an observation).

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 173
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Gathering Audit Evidence –
Basic Tasks

• An auditor could start with personnel department or payroll


records and develop a sample of employees who should have
received particular types of training, then review training
records to see whether all of these people had been trained.

Although testing is most commonly applied to documentary


evidence, it can apply to testimonial or physical (but not
circumstantial) evidence as well. For example, the auditor might
test the effectiveness of spill response training by gathering
testimonial evidence—asking a selected sample of employees to
describe proper spill response procedures. Similarly, a sample of
emergency eyewash stations might be selected and inspected by
the auditor (gathering physical evidence) as a test to determine if
they are in good working order and unobstructed.

It is important to note that an auditor does not design a formal,


rigorous “test” for every protocol step or topic. Conducting
testing is likely to be appropriate for the protocol steps/topics
associated with “Dig Deepest” and “Dig Deep” ratings (see p.167)
resulting from the Step 2 assessment of strengths and
weaknesses. For lower priority protocol steps, the auditor may
decide to rely on inquiry and observation, rather than testing, in
formulating a conclusion.

Develop Sampling Strategies


Even after developing a sound verification strategy, auditors can
still find themselves with more data to review than time allows.
When this happens, the auditor, as part of his/her verification
strategy, needs to also develop a sampling strategy. Sampling is
the tool the auditor utilises to look at a portion of a whole
population of items. Like verification, there are strategies for
effective sampling that serve to minimise bias and ensure that
what the auditor looks at is representative of actual conditions at
the audited facility. Indeed, the sampling strategy itself can affect
the validity of the data gathered and, consequently, the validity of
the conclusions reached. Sampling strategies will be discussed
in detail in the next chapter of the manual.
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 174
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Gathering Audit Evidence –
Basic Tasks

4. Compare Practices Against Requirements

Following the gathering of data, the auditor needs to compare


facility practices against applicable performance requirements to
determine whether the facility is in compliance with these
requirements, and whether there are enough data to evaluate
whether systems are being implemented as designed. If the
auditor finds deficiencies in compliance and/or the
implementation of management systems, he/she should
remember that there still may be time to explore potential
underlying causes of selected deficiencies (as discussed in the
chapter on evaluating audit results). As the auditor compares
practices against requirements, he/she begins to enter into Step
4: Evaluate Audit Results.

5. Document Results

Auditors should document their verification strategy as well as


the data gathered. In particular, the auditor should make sure to
document the following in his/her working papers:

Information What Auditor Documents

Which protocol steps were The verification strategy (i.e., what topics were
reviewed and which were not examined in depth and why).

Any changes to the If the auditor’s strategy called for review of


verification strategy once certain records, but those records could not be
initiated reviewed because they were inaccessible, then
this fact should be documented along with
whatever alternative verification strategies were
employed.

What data were gathered and The testing plans that were utilised and the
the source information actually collected/reviewed by the
auditor.

Conclusions Summary(ies) of the results of the verification


strategy.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 175
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Gathering Audit Evidence –
Basic Tasks

Summary: Developing a Verification Strategy

Task Question Asked How to Do it


• Determine the specific objectives
Evaluate what “What do I need to
of the protocol steps for review
needs to be look at and how
and then prioritise them.
done much time do I have
• Review the audit resources.
to do it?”
• Determine where you want to
Determine the “How deep do I need
engage in a rigorous review of
depth and to dig? How many
facility programmes and practices,
rigor of review kinds of evidence do
based on your assessment of the
I need to gather?”
following:
− High priorities
− Low priorities
− Where few or no systems exist
• Take into account what you
Select the “What types of data
learned in Step 1.
types of will I gather and how
• Determine the types of audit
evidence can I collect them?”
information needed.
needed and
• Determine the most appropriate
the methods
methods for collecting the audit
to gather
data.
them
• Determine the areas where you
need to design a test.
• Determine the most effective
sampling strategy.
• Determine if you have enough
Compare “How does the facility
data to evaluate whether systems
practices rate against the
are being implemented as
against applicable
designed.
requirements requirements?”
• Explore selected deficiencies for
underlying cause(s).
• Begin Step 4.
• Document what was looked at.
Document “What information do
• Document what was not looked at
results I need to record to
and why.
help me remember
• Summarise conclusions in working
what I have learned
papers.
and substantiate my
conclusions?”

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 176
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 6
Gathering Audit Evidence -
Developing Verification Strategies

Objective

The purpose of this exercise is to develop a verification strategy


to most effectively conduct the Step 3 portion of the audit (Gather
Audit Evidence) and to determine how best to manage the
remaining time on site.

Instructions

Based on the information provided in one of the following


scenarios, outline your strategy for verifying that the facility is in
compliance with applicable requirements highlighted in the
protocol steps provided in this exercise and that management
systems are being implemented and followed at the facility.

You should assume that you are in fact at Step 3 in the audit
process, that you have already spoken with the key HSE
personnel, and that you have a general understanding of
management systems.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 177
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 6A
Gathering Audit Evidence -
Developing Verification Strategies: Contractor’s
Safety

Background

It is now Tuesday afternoon of a one-week audit (the close-out


meeting is scheduled for 10:00 a.m. Friday morning). After
spending all day Monday and Tuesday morning learning how the
facility manages safety, you feel that you have a good
understanding of the systems in place, and you are now ready to
go out and “verify.”

What You Understood from Step 1 About the Facility’s


System for Managing Contractor Safety

The facility, which is subject to the OSHA process safety


management standard, has established a contractor/visitor
orientation prior to commencing work.

The HSE coordinator, human resources department, and


maintenance manager have all been trained to provide the
orientation. The orientation covers the facility’s safety rules,
emergency/evacuation procedures, and hazard communication.
In addition, the facility has various modules (i.e. confined space
entry, lockout/tagout, and hazardous waste practices) which are
added to the basic orientation depending on the type of work the
contractor will be engaged in. Once the contractor has
completed the orientation, his/her badge is given a colour-coded
stamp which indicates that the person has received orientation
and the topics covered (e.g., yellow—basic orientation; green—
includes lockout/tagout; red—includes confined space entry; and
blue—includes hazardous waste practices). Security personnel
and all facility personnel are aware of the facility’s contractor
orientation programme and are encouraged to check contractor
badges to ensure that they have received orientation.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 178
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 6A
Gathering Audit Evidence -
Developing Verification Strategies: Contractor’s
Safety

Protocol Steps Relating to Contractor Safety

Contractor/Visitor Safety Programmes


1. Determine if the facility has a programme for contractor activities that have
the potential for affecting process safety, including maintenance or repair,
turnaround, major renovation, or specialty work on or adjacent to a
process. If so, review the programme and confirm that the facility:

a. Obtains and evaluates information regarding the contractor’s safety


performance and programmes when selecting a contractor.

b. Informs contractors, prior to the initiation of the contractors’ work at


the site, of the known potential fire, explosion, or toxic release hazards
related to the contractor’s work and the process and obtains a signed
confirmation that the contractor has received and understood the
information.

c. Explains to contractors, prior to the initiation of the contractors’ work at


the site, the applicable provisions of the emergency action plan.

d. Develops and implements safe work practices to control the entrance,


presence, and exit of contract employees in process areas.

e. Issues appropriate permits to work (hot and cold) as necessary.

f. Periodically evaluates the performance of contract employees in


fulfilling their obligations.

g. Maintains a contract employee injury and illness log related to the


contractor’s work in process areas.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 179
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 6A
Gathering Audit Evidence -
Developing Verification Strategies: Contractor’s
Safety

Instructions

1. Evaluate What Needs to Be Done


Review the protocol steps highlighted and determine the specific
objectives of those steps.

2. Select the Types of Evidence Needed and the Methods to


Gather Them
List the types of evidence that could be gathered to address the
highlighted protocol steps. In addition, identify the items or
systems you will want to test, and how you will accomplish
verification testing. In developing your verification strategies,
consider the following:

• What evidence could you gather to address the protocol steps


highlighted?

• Where or from whom could you gather the evidence?

• How could you design a test of the facility’s contractor safety


programme with respect to the highlighted protocol steps?

• When you have decided what you could do, decide which of
these data-gathering activities you would do to address the
highlighted protocol steps.

3. Consider What Was Learned During Step 1


What factors learned in Step 1 drove your verification strategy?

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 180
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 6A
Gathering Audit Evidence -
Developing Verification Strategies: Contractor’s
Safety

To assist in developing your verification strategy, you should use


the worksheet provided below which will take you through the
basic steps as described in this section.

Physical Documentary Testimonial


Evidence Evidence Evidence

What evidence
would you
gather?

Where or from
whom would
you gather the
evidence?

How would you


design a test of
the facility’s
performance
with respect to
the highlighted
protocol steps?

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 181
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 6B
Gathering Audit Evidence -
Developing Verification Strategies: Hazardous Waste
Manifests

Background

It is now Tuesday afternoon of a one-week audit (the close-out


meeting is scheduled for 10:00 a.m. Friday morning). After
spending all day Monday and Tuesday morning learning how the
facility manages its hazardous waste, you feel that you have a
good understanding of the systems in place, and you are now
ready to go out and “verify.”

What You Understood from Step 1 About the Facility’s


System for Managing Hazardous Waste Manifests

• The facility’s is a large quantity generator of special


(hazardous) wastes.

• The HSE coordinator and maintenance manager have been


designated to fill out the waste manifests.

• Once the accumulation storage log indicates that wastes must


be shipped off site because they are within two weeks of
reaching an accumulation time of 90 days, the waste
transporter is called and a pick-up is scheduled.

• The log must be periodically reviewed since there are no


automatic systems to notify facility personnel that a shipment
must be scheduled.

• Once the waste transporter picks up the hazardous waste, the


facility copy of the waste manifests is retained by the Human
Resources records clerk. The clerk is responsible for filing the
waste manifests with the receipt copy once the signed receipt
copy from the disposal facility is received.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 182
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 6B
Gathering Audit Evidence -
Developing Verification Strategies: Hazardous Waste
Manifests

Protocol Steps Relating to Special Waste Manifests

Hazardous Waste Manifests


1. By reviewing a representative sample of Waste Manifests, verify that the
facility has a program in place to accurately prepare these shipping
documents and track waste loads from the point of generation to final
disposition. In particular:

a. Note whether or not signed Waste Manifests are returned to the


producer from the disposal or treatment facility. For any that were not,
document facility actions to locate the waste shipment.

b. Compare the waste streams shipped off site to your list developed
previously. For those materials not covered by Waste Manifests
during the review period, interview staff in the operating area(s) where
the waste is usually produced and determine if the waste was
generated during the review period. If so, resolve how this stream
was disposed of without being covered by Waste Manifests.

c. Review the Waste Manifests and determine if each box or information


entry has been filled out correctly.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 183
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 6B
Gathering Audit Evidence -
Developing Verification Strategies: Hazardous Waste
Manifests

Instructions

1. Evaluate What Needs to Be Done


Based on the information provided above, review the protocol
steps highlighted and determine the specific objectives of those
steps.

2. Select the Types of Evidence Needed and the Methods to


Gather Them
Identify the types of evidence that could be gathered to complete
the highlighted protocol steps. In addition, identify the items or
systems you will want to test, and how you will accomplish
verification testing. In developing your verification strategies,
consider the following:

• What evidence could you gather to address the protocol steps


highlighted?

• Where or from whom could you gather the evidence?

• How would you design a test of the facility’s hazardous waste


consignment note programme with respect to the highlighted
protocol steps?

• When you have decided what you could do, decide which of
these data-gathering activities you would do to address the
highlighted protocol steps.

3. Consider What Was Learned During Step 1


What factors learned in Step 1 drove your verification strategy?

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 184
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 6B
Gathering Audit Evidence -
Developing Verification Strategies: Hazardous Waste
Manifests

To assist in developing your verification strategy, use the


worksheet provided below which will take you through the basic
steps as described in this section.

Physical Documentary Testimonial


Evidence Evidence Evidence

What evidence
would you
gather?

Where or from
whom would
you gather the
evidence?

How would you


design a test of
the facility’s
performance
with respect to
the highlighted
protocol steps?

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 185
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 6A– Potential Answers
Gathering Audit Evidence -
Developing Verification Strategies

Scenario A: Contractor Safety

Physical Evidence Documentary Testimonial


Evidence Evidence

What evidence • Badge colours • Contractor training • Contractor


observed on site. manual. understanding of
would you
• Contractor activities • Contractor training the training
gather? materials and facility
on site against the records.
facility’s safety • Facility contractor safety rules.
rules. safety rules. • Affected facility
• Contractor sign-in personnel’s
log. understanding of
the contractor safety
• Contract language.
programme and its
implementation.
Where or from • Locations where • HSE coordinator. • HSE coordinator.
contractor work is in • Site security. • Site security.
whom would
progress. • Contracting / • Maintenance
you gather the
evidence? Purchasing manager.
personnel. • Facility personnel
who manage on-site
contractors.
• On-site contractors
(if practical).
How would • Compare a sample • Compare a sample • Survey a sample of
of colour badges on of contractors on contractors to test
you design a
contractors to the the sign-in log to the their understanding
test of the activity being training records. of the training
facility’s conducted. provided.
• Obtain a sample of
performance • Review contractor contractor names • Survey affected
with respect to activities in and the colour facility personnel’s
the highlighted comparison to the badge observed understanding of
protocol facility’s contractor and compare the the contractor safety
steps? safety rules. colour badge to the programme (e.g.,
level of training badges).
received through a
records review.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 186
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 6B– Potential Answers
Gathering Audit Evidence -
Developing Verification Strategies

Scenario B: Hazardous Waste Manifests

Physical Evidence Documentary Testimonial


Evidence Evidence

What evidence • Accumulation start • Hazardous waste • HSE coordinator’s


dates on hazardous accumulation and maintenance
would you
waste drums. storage logs. manager’s
gather? understanding of
• Hazardous waste • Filed Waste
streams generated Manifests. Waste Manifests.
by the facility. • Chemical • Human resources
purchasing or records clerk’s
inventory lists. understanding of
• Annual hazardous consignment note
waste reports. record keeping.
• Purchase orders for • Affected facility
waste disposal. personnel’s
understanding of
hazardous waste
accumulation.
Where or from • Hazardous waste • Human resources • HSE coordinator.
accumulation records clerk. • Maintenance
whom would
storage area. • Facility personnel manager.
you gather the
• Process areas responsible for the • Facility personnel
evidence?
where potential accumulation area. responsible for the
hazardous waste • Purchasing accumulation area.
streams are personnel. • Human resources
generated. records clerk.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 187
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 6B– Potential Answers
Gathering Audit Evidence -
Developing Verification Strategies

Physical Evidence Documentary Testimonial


Evidence Evidence

How would • Compare the drums • Check consignment • Ask the HSE
in the hazardous note signed copy coordinator /
you test the
waste storage area from the waste maintenance
facility’s to the information on disposal company, manager how they
management the accumulation where applicable. obtain the
systems to storage logs. • Compare Waste information
address the • Check the Manifests versus the regarding various
highlighted accumulation start annual hazardous wastes generated
protocol dates on the waste report. (i.e., waste
steps? hazardous waste • Compare Waste characterisation,
drums to determine Manifests versus time generated etc.).
if drums are being waste profiles and/or • Ask affected
stored over 90 days. process information employees how they
• Review facility to verify correctness manage the
processes to identify of waste hazardous waste
potential hazardous classification. accumulation area.
waste streams and • Compare lists • Ask the records
compare to the versus Waste clerk how she/he
facility’s identified Manifests. manages the
waste streams. • Review purchasing / hazardous waste
shipping records consignment note
and verify that records (e.g.,
Waste Manifests are retention, signed
on file for a sample copy from the waste
of shipments. disposal company
etc.).

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 188
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Sampling Strategies

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 189
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 7
Sampling Strategies 1

Objective

The purpose of this exercise is to illustrate some key


considerations when developing and implementing a sampling
strategy.

Instructions

Your assignment is to verify that secondary containment has


been provided for all bulk storage tanks. The facility’s Spill Plan
indicates that there are 30 bulk storage tanks and that all of them
have secondary containment. The facility has a self-inspection
checklist that lists the 30 tanks. Because time is limited, you
have decided to choose a sample of six storage tanks from the
facility’s list. The first tank in your sample has no secondary
containment. On the way to the second sample location, you
discover a tank that is not listed in the SPCC Plan and note that
the drain valve in the containment area is locked “open.” The
second tank in your sample has a visible crack in the
containment.

At this point, what do you do?

1. Stop right there.

2. Finish the sample of six.

3. Check the containment at all 30 tanks.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 190
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Sampling Strategies –
Overview

A fundamental component of any audit situation involves the


review of a facility’s HSE management systems and activities to
obtain evidence needed to substantiate compliance or
noncompliance. Because auditing is basically a check on the
overall compliance status of a facility and is conducted over a
relatively short period of time, auditors generally do not examine
entire populations of records, documents, or employees.
Instead, auditors sample populations in order to draw
conclusions regarding compliance with performance standards.

To help ensure the gathering of appropriate sampling


information, the following process is frequently followed by
auditors:

1. Determine objective of protocol step


2. Identify population for review
3. Select sampling method
4. Determine sample size
5. Document results

Each of these steps is described on the following pages.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 191
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Sampling Strategies –
Basic Sampling Process

1. Determine Objective of Protocol Step

• Specify what you are trying to confirm.

• Consider the nature of the regulatory or internal standard to


accurately identify the boundaries of the population under
review.

2. Identify Population for Review

• Estimate size of population through:


− Review of selected documents.
− Observations made during initial understanding of health,
safety and environmental management systems in place.
− Interviews with facility personnel.

• Pay attention to major subsets or key segments of the


population that need to be included in the review.

• Define population before starting to sample.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 192
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Sampling Strategies –
Basic Sampling Process

Sampling Methods

Judgmental Probabilistic

Random

Block

Interval

Stratification
3. Select Sampling Method

Judgmental Sampling
Judgmental sampling is used to gather examples of deficiencies
or problems to support an auditor’s assessment of a weak or
improper health, safety and environmental management system.
Sampling is directed toward segments of the population where
problems are likely to exist. Judgmental sampling cannot be
used to draw compliance conclusions about an entire population
because it focuses on only a portion or subset of that population.
Judgmental sampling can be used as a first step to provide the
auditor with an indication of whether to use a probabilistic
sampling technique such as random, block, interval, or
stratification sampling.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 193
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Sampling Strategies –
Basic Sampling Process

Probabilistic Sampling
Random. The objective is to select items by a statistically-based
chance. If properly done, each item in the population should
have an equal chance of being selected, and there should be no
subjective determinations to bias the sample. (Note: For auditing
purposes, “random” is not the same as “haphazard.” Haphazard
means “characterised by lack of order or planning; aimless;
independent of any reasoning process.” An example of
haphazard sampling would be closing your eyes and grabbing
one file out of a drawer.

• Block. The objective is to analyze certain segments of


records or areas of the facility. For example, if files were
arranged alphabetically, in numerical order, or chronologically,
one or more blocks (e.g., all the E’s, records numbered 51
through 75, or January and June files) could be selected.
While the block method is easy to use, it neglects entire
segments of the population.

• Interval. The objective of interval sampling is to select


samples at specific intervals (e.g., every nth segment of the
population is analyzed) with the first item selected at random.
Increased confidence is achieved where several intervals with
different random starts are used.

• Stratification. The objective of stratification sampling is to


arrange items by categories (e.g., high versus low effluent
volumes; new versus experienced employees; regular versus
weekend or off-shift transactions) based on the auditor’s
judgment that the probability of finding an exception is
different for different segments of the population and/or that
there are categories within the population that represent
higher inherent risks. Higher risk categories may, thus,
receive greater review and testing. Once the population has
been stratified, random, block, or interval sampling can be
applied to select items within each segment.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 194
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Sampling Strategies –
Basic Sampling Process

4. Determine Sample Size

• There are two ways to determine sample size:


− Statistically
− Auditor’s judgment

• In most HSE audit situations, it is both appropriate and


adequate to develop sample sizes based upon professional
judgment.

• The auditor must be sure that the sample size is large enough
to be representative of the total population.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 195
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Sampling Strategies –
Basic Sampling Process

5. Document Results

• Document rationale for selecting sample.


• Document how sample was selected.
• Include in working papers:
− Population under review
− How and why population was selected
− Type of sampling method employed
− Reasons sampling method was used
− Potential bias in sample
− Sample size and reasons for selecting sample size

Example Guideline for Selection of Sample Size

Suggested Minimum Size of


Size of Population Sample

A B C

2–10 100% 100% 30%


11–25 100% 40% 20%
26–50 50% 20% 15%
51–100 25% 15% 10%
101–250 15% 10% 5%
251–500 10% 5% 3%
501–1,000 5% 3% 2%
Over 1,000 3% 2% 1-2%

(These percentages do not imply any specific confidence level but are intended as
guidelines only.)

A Suggested minimum sample size for a population(s) being reviewed which is


considered to be extremely important in terms of verifying compliance with
applicable requirements and/or is of critical concern to the organisation in
terms of potential or actual impacts associated with noncompliance.
B Suggested minimum sample size for a population(s) being reviewed that will
provide additional information to substantiate compliance or noncompliance
and/or is of considerable importance to the organisation in terms of potential or
actual impacts associated with noncompliance.
C Suggested minimum sample size for a population(s) being reviewed that will
provide ancillary information in terms of verifying overall compliance with a
requirement.
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 196
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Sampling Strategies –
Basic Sampling Process

Random Number Table

The random number table (on the following page) is intended for
use when the auditor has chosen random sampling as the
preferred sampling method. (See discussion above on “random
sampling.”) To use the table, start at any location on the table;
then, moving in a given direction (across, down, or up), identify
the numbers in sequence. For example, in randomly selecting 25
of 90 records, designate values of 1 to 90 for each record, then
moving along the table in the direction you have chosen, use the
first two digits of each number to choose 25 records. Assuming
you start in the upper left-hand corner of the random number
table, and move down, your first number is 104 (record 10); the
second number is 465 (record 46); the third number is 225
(record 22); etc. If you get a duplicate or a number outside the
defined range, ignore it. Stop when you have 25 records to
review.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 197
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Sampling Strategies –
Basic Sampling Process

Random Number Table


104 150 015 020 816 916 691 141 625 362 209 995 912 907 223
465 255 853 309 891 279 534 939 340 526 191 396 995 241 483
225 972 763 648 151 248 493 320 306 196 633 586 421 930 062
616 078 163 394 535 713 570 008 749 977 163 375 399 818 166
061 917 604 813 496 606 141 069 012 546 779 069 110 427 277
534 186 706 906 150 219 818 443 428 995 729 564 699 988 310
711 187 440 488 632 210 106 129 963 919 054 079 188 209 945
568 690 600 184 849 425 323 895 143 636 102 170 181 577 843
253 125 586 449 055 569 854 368 533 539 530 595 388 623 081
179 164 114 185 649 289 695 882 332 709 799 568 058 901 315
015 855 916 781 635 409 482 034 496 694 186 726 521 208 122
905 337 903 094 939 526 927 889 334 363 176 300 082 841 271
306 749 109 611 875 856 482 522 676 933 015 263 851 202 299
898 071 973 710 081 772 139 475 810 977 859 293 744 285 907
510 127 518 512 774 163 607 921 494 539 709 639 756 407 023
213 524 602 893 198 553 448 011 652 648 449 059 551 010 540
333 949 312 041 185 298 715 850 511 019 927 649 521 539 463
585 232 145 831 987 234 643 947 177 351 357 070 976 337 099
426 066 769 136 518 461 889 195 256 581 486 912 858 143 091
301 902 047 591 221 304 616 999 328 541 584 224 741 470 253
764 263 581 066 215 152 969 112 326 323 055 242 133 380 943
287 358 069 170 641 182 228 293 270 876 873 587 002 458 153
465 411 103 076 361 185 024 330 288 073 197 924 609 612 500
676 325 866 507 949 132 168 741 920 246 366 007 228 021 516
072 795 972 459 212 003 304 038 946 894 415 175 273 639 415
491 822 241 990 478 810 649 662 804 657 832 341 132 305 977
358 919 001 509 986 384 878 946 397 574 675 776 443 112 711
110 605 064 287 378 079 987 985 271 312 806 444 978 704 954
145 507 354 590 875 481 029 009 481 047 212 208 929 902 124
350 011 386 281 680 109 100 542 064 508 654 793 538 106 218
724 779 565 559 873 696 451 003 257 008 968 306 476 231 395
569 206 217 517 331 726 326 415 761 915 211 362 278 739 206
378 638 710 847 524 223 780 174 961 183 709 669 997 724 011
421 113 207 543 369 700 232 654 596 996 947 114 181 813 804
906 525 020 851 885 478 002 825 720 157 438 998 104 769 259
035 215 834 439 907 229 442 340 655 857 558 388 593 137 351
013 395 762 224 832 322 795 290 041 162 153 128 662 383 224
733 887 094 825 052 926 826 270 325 170 276 982 638 119 346
880 561 349 570 239 258 400 670 122 027 148 232 350 997 375
116 355 851 099 963 059 979 283 141 008 807 704 756 767 887
378 401 590 333 266 622 699 761 508 438 866 709 793 938 281
192 687 695 889 496 467 633 566 004 733 914 152 069 570 541
176 008 643 607 889 610 997 306 264 115 443 347 603 609 719
602 635 711 056 438 582 261 321 634 354 571 109 073 546 936
851 643 291 443 144 552 787 341 303 484 513 095 259 276 112
652 528 508 222 055 995 737 857 292 703 602 183 198 428 082
432 470 426 456 000 206 146 499 945 563 596 091 580 290 443
457 707 056 490 269 574 992 241 746 756 286 392 528 627 726
980 672 727 018 134 146 876 897 139 778 691 700 354 345 154
813 587 354 948 755 006 977 966 864 964 864 544 864 554 413

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 198
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Sampling Strategies –
Basic Sampling Process

Sampling Strategies

Random Sampling

Select items in entire population


by chance (e.g. using a random
Number table)

Block Sampling

Arrange items by certain segments


or clusters and randomly select
some clusters as your sample

Interval Sampling

Select samples at intervals


(every nth item starting randomly)

Stratification Sampling

Arrange items by important


categories or subsets, then
sample within the groups

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 199
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 8
Sampling Strategies 2

Objective

The purpose of this exercise is to:

• Use each sampling method (random, block, interval, and


stratification) on a set of hazardous waste training records to
determine if affected employees have received refresher
training within the last year.

• Determine whether there are any advantages or


disadvantages in using one sampling method versus another.

Background

During your audit of the ACCO Chemical Company, you need to


verify whether the site is complying with the corporate
requirement that all employees who have responsibility for
managing or handling hazardous waste have received annual
refresher training and that the training is documented.

You have learned that there are three categories of employees


who are involved with hazardous waste management:

• Within each production department, there are designated


operators, who have routine, day-to-day responsibility for
managing a variety of wastes that are generated as a result of
frequent cleaning of reactors used for batch specialty
chemical production. Other production employees, who might
on occasion generate hazardous waste, know that they are
not to handle it themselves but are to contact one of the
designated operators. If the designated operators are not
properly trained, the probability of waste mismanagement is
high.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 200
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 8
Sampling Strategies 2

• Maintenance workers handle a much smaller number of


relatively constant hazardous waste streams on an infrequent
basis. The wastes most frequently handled are used oil and
spent solvents from cleaning operations. Occasionally,
maintenance workers encounter other wastes (when they
service plant equipment (pumps, compressors), for example).

• Supervisors have virtually no hands-on responsibility for


waste management. However, because they are ultimately
responsible for the actions of the workers they supervise, it is
company policy that all supervisors (not just those in
production or maintenance) must receive annual training to
maintain awareness.

Based on the above job descriptions and the employee roster,


you have developed the list shown in Table 1, which indicates
that there are 80 employees total who require annual hazardous
waste refresher training according to company policy.

The plant environmental coordinator has provided you with his


list of employees who attended the two most recent refresher
training sessions, both held within the past 12 months
(see Table 2).

Instructions

Depending on the group to which you are assigned, you are to


use one of the following methods to create a sample of
approximately 25 percent of the 80-employee total
(corresponding to the suggested minimum size of sample in
column A using the table on page 9-10):
• Random
• Block
• Interval
• Stratification

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 201
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 8
Sampling Strategies 2

For your sample, determine how many of the employees in each


job category have received the required training according to the
documented records (Tables 1 and 2 attached).
After you have completed your sampling and analysis, we will
compare the results of the various methods.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 202
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 8
Sampling Strategies 2

Table 1: Employee List Developed by Auditor

No Job Category Employee Sampling Trained in


Name Method Chosen Past Year
(Y/N)

R B I S
1 Maintenance Domingo, P.
2 Maintenance Bell, M.
3 Maintenance Cort, A.
4 Maintenance Greeno, L.
5 Maintenance Herald, M.
6 Maintenance Higgins, H.
7 Maintenance Getchell, M.
8 Maintenance Jones, T.
9 Maintenance Monteiro, L.
10 Maintenance Obbagy, J.
11 Maintenance Plunkett, J.
12 Maintenance Reid, R.
13 Maintenance Rotberg, F.
14 Maintenance Savoie, M.
15 Maintenance Sellers, G.
16 Maintenance Young, R.
17 Operator Allen, M.
18 Operator Arnold, E.
19 Operator Bach, J.S.
20 Operator Bateman, R.
21 Operator Butler, L.
22 Operator Catmur, J.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 203
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 8
Sampling Strategies 2

No Job Category Employee Sampling Trained in


Name Method Chosen Past Year
(Y/N)

R B I S
23 Operator Cavalcanti, C.
24 Operator Coburn, T.
25 Operator Davanzo, L.
26 Operator DiBerto, M.
27 Operator Fitch, T.
28 Operator Fletcher, J.
29 Operator Harris, J.
30 Operator Hasselreis, D.
31 Operator Hryciuk, R.
32 Operator Jones, M.
33 Operator Lennon, J.
34 Operator Lewis, S.
35 Operator Loren, S.
36 Operator McGinnes, M.
37 Operator McLean, R.
38 Operator Moody, C.
39 Operator Murray, E.
40 Operator Neville, N.
41 Operator Nott, M.
23 Operator Nureyev, R.
43 Operator Patel, B.
44 Operator Picasso, P.
45 Operator Ryder, W.
46 Operator Schmidt, H.
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 204
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 8
Sampling Strategies 2

No Job Category Employee Sampling Trained in


Name Method Chosen Past Year
(Y/N)

R B I S
47 Operator Smith, F.
48 Operator Smith, J.
49 Operator Smith, N.
50 Operator Stone, O.
51 Operator Tallchief, M.
52 Operator Umscheid, M.
53 Operator Voeller, R.
54 Operator Wescott, W.
55 Operator Windsor, C.
56 Operator Yetskalo, V.
57 Supervisor Berstein, L.
58 Supervisor Boehm, P.
59 Supervisor Canton, C.
60 Supervisor Capogna, S.
61 Supervisor Dolittle, E.
62 Supervisor Ferguson, S.
63 Supervisor Fonteyn, N.
64 Supervisor Gedanke, M.
65 Supervisor Harris, R.
66 Supervisor Hedstrom, G.
67 Supervisor Hill, R.
68 Supervisor Hogwood, C.
69 Supervisor Lopez, D.
70 Supervisor Major, J.
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 205
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 8
Sampling Strategies 2

No Job Category Employee Sampling Trained in


Name Method Chosen Past Year
(Y/N)

R B I S
71 Supervisor Moore, T.
72 Supervisor Morrison, J.
73 Supervisor Nutty, P.
74 Supervisor Parigot, M.
75 Supervisor Presley, E.
76 Supervisor Robinson, J.
77 Supervisor Shrimpton, J.
78 Supervisor Starr, R.
79 Supervisor Stricoff, S.
80 Supervisor Thomas, M.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 206
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 8
Sampling Strategies 2

Table 2: Training Records by Employee Name

Attendance at First Annual Refresher Training

Employee Name Job Category


Domingo, P. Maintenance
Arnold, E. Operator
Bach, J.S. Operator
Boehm, P. Supervisor
Butler, L. Operator
Capogna, S. Supervisor
Cavalcanti, C. Operator
Coburn, T. Operator
Cort, A. Maintenance
Ferguson, S. Supervisor
Harris, R. Supervisor
Hogwood, C. Supervisor
Hryciuk, R. Operator
Lennon, J. Operator
McGinnes, M. Operator
McLean, R. Operator
Monteiro, L. Maintenance
Morrison, J. Supervisor
Neville, M. Operator
Nott, M. Operator
Obbagy, J. Maintenance
Parigot, M. Supervisor
Schmidt, H. Operator

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 207
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 8
Sampling Strategies 2

Attendance at First Annual Refresher Training

Employee Name Job Category


Sellers, G. Maintenance
Shrimpton, J. Supervisor
Smith, J. Operator
Stone, O. Operator
Tallchief, M. Operator
Voeller, R. Operator
Young, R. Maintenance

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 208
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 8
Sampling Strategies 2

Attendance at Second Annual Refresher Training

Employee Name Job Category


Bell, M. Maintenance
Berstein, L. Supervisor
Canton, C. Supervisor
Catmur, J. Operator
Davanzo, L. Operator
DiBerto, M. Operator
Dolittle, E. Supervisor
Fletcher, J. Operator
Fonteyn, M. Supervisor
Hasselreis, D. Operator
Higgins, H. Maintenance
Hill, R. Supervisor
Lewis, S. Operator
Loren, S. Operator
Moody, C. Operator
Moore, T. Supervisor
Murray, E. Operator
Nutty, P. Supervisor
Nureyev, R. Operator
Plunkett, J. Maintenance
Robinson, J. Supervisor
Savoie, M. Maintenance
Smith, N. Operator
Starr, R. Supervisor
Stricoff, S. Supervisor
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 209
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 8
Sampling Strategies 2

Attendance at Second Annual Refresher Training

Employee Name Job Category


Thomas, M. Supervisor
Umscheid, M. Operator
Wescott, W. Operator
Yetskalo, V. Operator

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 210
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 8 – Potential Answers
Sampling Strategies

Determine Objective of Protocol Step

Review facility training records to confirm that employees with


hazardous waste management responsibilities have received
refresher training within the past year.

Determine Total Population and Important Subpopulations

There are 80 employees in three categories who require training:


16 maintenance workers, 40 operators and 24 supervisors.

Select Sampling Strategy and Sample Size

1. Random
The total population is 80 and you have decided that this is an
important issue for the audit. The suggested minimum sample
size is therefore 25%. You therefore want to pick 20 random
numbers between 1 and 80.

One way of generating your sample is to use a random number


table. If you start at the lower right-hand corner of the table and
read up, the first 20 random numbers (eliminating duplicates)
between 1 and 80 are:

13 36 24 6
54 19 51 18
26 41 59 77
43 75 4 15
12 46 11 16

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 211
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 8 – Potential Answers (continued)
Sampling Strategies

The attached worksheet indicates with an “R” the sample of


employees selected by this method. To summarise, the random
method gives a reasonably representative sample, with all three
job categories included:

7 of 16 maintenance workers
10 of 40 operators
3 of 24 supervisors

2. Block
For this type of information, the most logical “block” method to
use might be by the first letter of the last name, by generating
random numbers between 1 and 26. Using the random number
generator on a programmable pocket calculator, you might have
selected the numbers : 8, 18, 1, 14, 23 and 4 which correspond,
respectively, to last names beginning with H, R, A, N, W and D.
A quick count tells you that the first four of these letters will be
enough to give a sample of 20.

The attached worksheet indicates with a “B” the sample of


employees selected by this method. To summarise, the block
method also gives a reasonably representative sample, with all
three job categories included:
• 5 of 16 maintenance workers
• 9 of 40 operators
• 6 of 24 supervisors

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 212
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 8 – Potential Answers (continued)
Sampling Strategies

3. Interval
To get a 25% sample, you need to select every fourth name on
the list. You could choose any number between 1 and 4 as the
starting point.

The attached worksheet indicates with an “I” the sample of


employees selected by this method. To summarise, the interval
method also gives a reasonably representative sample, with all
three job categories included:
• 4 of 16 maintenance workers
• 10 of 40 operators
• 6 of 24 supervisors

4. Stratified
Suppose you decided that annual retraining was very important
for the operators, less important for the maintenance workers,
and a relatively low priority for the supervisors. The guidelines
would then suggest sampling 50% of the operators, 40% of the
maintenance workers, and 20% of the supervisors. Applying the
interval method within each stratum would require taking every
third maintenance worker, every other operator, and every fifth
supervisor.

The attached worksheet indicates with an “S” the sample of


employees selected by this method. To summarise, the
stratification method also gives a reasonably representative
sample, with all three job categories included:
• 6 of 16 maintenance workers
• 20 of 40 operators
• 5 of 24 supervisors

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 213
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 8 – Potential Answers (continued)
Sampling Strategies

Analyze Results

1. Random
Five of the sample of 20 had not received training, according to
the records. Of these, 3 were maintenance workers, 1 was an
operator, and 1 was a supervisor.

2. Block
Seven of the sample of 20 had not received training, according to
the records. Of these, 3 were maintenance workers, 3 were
operators, and 1 was a supervisor.

3. Interval
Six of the sample of 20 had not received training, according to
the records. Of these, 1 was a maintenance worker, 4 were
operators, and 1 was a supervisor.

4. Stratification
Three of the sample of 6 maintenance workers and 7 of the
sample of 20 operators had not received training, according to
the records. All of the supervisors in this sample of 5 had
received training. (Note: When using this method, it is not correct
to combine the results and say “10 out of a sample of 31”.)

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 214
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 8 – Potential Answers (continued)
Sampling Strategies

Conclusion From This exercise

No two probabilistic sampling methods give exactly the same


answer, in quantitative terms. However, qualitatively all four
methods give comparable results.

The three methods that looked at the population as a whole


indicated that between 25% and 35% of the people who should
have been trained were not and that persons in all three job
categories has missed training. The stratification method
indicated that 50% of maintenance workers and 35% of operators
had not been trained. This latter method did not pick up the fact
that some supervisors also had missed training. However, all
four approaches picked up on the fact that a substantial fraction
of the people in the two “high risk groups” had not received the
training.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 215
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 8 – Potential Answers (continued)
Sampling Strategies

Table 1

Employee Roster
No Job Category Employee Sampling Trained in
Name Method Chosen Past Year
(Y/N)

R B I S

1 Maintenance Domingo, P. B S Y
2 Maintenance Bell, M.
3 Maintenance Cort, A. I Y
4 Maintenance Greeno, L. R S N
5 Maintenance Herald, M. B N
6 Maintenance Higgins, H. R B Y
7 Maintenance Getchell, M. I S N
8 Maintenance Jones, T.
9 Maintenance Monteiro, L.
10 Maintenance Obbagy, J. S Y
11 Maintenance Plunkett, J. R I Y
12 Maintenance Reid, R. R B N
13 Maintenance Rotberg, F. R B S N
14 Maintenance Savoie, M.
15 Maintenance Sellers, G. R I Y
16 Maintenance Young, R. R S Y
17 Operator Allen, M. B S N
18 Operator Arnold, E. R B Y
19 Operator Bach, J.S. R I S Y
20 Operator Bateman, R.
21 Operator Butler, L. S Y

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 216
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 8 – Potential Answers (continued)
Sampling Strategies

No Job Category Employee Sampling Trained in


Name Method Chosen Past Year
(Y/N)

R B I S
22 Operator Catmur, J.
23 Operator Cavalcanti, C. I S Y
24 Operator Coburn, T. R Y
25 Operator Davanzo, L. S Y
26 Operator DiBerto, M. R Y
27 Operator Fitch, T. I S N
28 Operator Fletcher, J.
29 Operator Harris, J. B S N
30 Operator Hasselreis, D. B Y
31 Operator Hryciuk, R. B I S Y
32 Operator Jones, M.
33 Operator Lennon, J. S Y
34 Operator Lewis, S.
35 Operator Loren, S. I S Y
36 Operator McGinnes, M. R Y
37 Operator McLean, R. S Y
38 Operator Moody, C.
39 Operator Murray, E. I S Y
40 Operator Neville, N. B Y
41 Operator Nott, M. R B S Y
23 Operator Nureyev, R. B Y
43 Operator Patel, B. R I S N
44 Operator Picasso, P.
45 Operator Ryder, W. B S N
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 217
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 8 – Potential Answers (continued)
Sampling Strategies

No Job Category Employee Sampling Trained in


Name Method Chosen Past Year
(Y/N)

R B I S
46 Operator Schmidt, H. R
47 Operator Smith, F. I S N
48 Operator Smith, J.
49 Operator Smith, N. S Y
50 Operator Stone, O.
51 Operator Tallchief, M. R I S Y
52 Operator Umscheid, M.
53 Operator Voeller, R. S Y
54 Operator Wescott, W. R Y
55 Operator Windsor, C. I S N
56 Operator Yetskalo, V.
57 Supervisor Berstein, L. S Y
58 Supervisor Boehm, P.
59 Supervisor Canton, C. R I Y
60 Supervisor Capogna, S.
61 Supervisor Dolittle, E.
62 Supervisor Ferguson, S. S Y
63 Supervisor Fonteyn, N. I Y
64 Supervisor Gedanke, M.
65 Supervisor Harris, R. B Y
66 Supervisor Hedstrom, G. B N
67 Supervisor Hill, R. B I S Y
68 Supervisor Hogwood, C. B Y
69 Supervisor Lopez, D.
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 218
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 8 – Potential Answers (continued)
Sampling Strategies

No Job Category Employee Sampling Trained in


Name Method Chosen Past Year
(Y/N)

R B I S
70 Supervisor Major, J.
71 Supervisor Moore, T. I Y
72 Supervisor Morrison, J. S Y
73 Supervisor Nutty, P. B Y
74 Supervisor Parigot, M.
75 Supervisor Presley, E. R I N
76 Supervisor Robinson, J. B Y
77 Supervisor Shrimpton, J. R S Y
78 Supervisor Starr, R.
79 Supervisor Stricoff, S. I Y
80 Supervisor Thomas, M.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 219
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Evaluating Audit Results

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 220
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Evaluating Audit Results

Pre-Audit Activities On-Site Activities Post-Audit Activities

Step 1: Understand Prepare Draft Report


Management Systems
Conduct opening meeting Obtain Review Comments from
Select & Schedule Facility Audit Conduct orientation tour Corporate HSE
Review audit strategy Law department
Understand details of Facility management
management systems

Step 2: Assess strengths Issue Final Report to


& Weaknesses Facility management
Select Team Members & Confirm Consider potential impacts Operations
their Availability Evaluate management Corporate HSE
systems Law department
Set priorities for verification

Step 3: Gather Audit Evidence


Plan the Audit: Evaluate what needs to be
Correspond with the done Develop Action Plan
facility Determine depth & rigor of Develop proposed action(s) to
Assemble & distribute review address each finding
background information Select types of evidence Assign responsibility for
Assign & communicate needed & methods to gather corrective action
audit responsibilities them Develop timetable
Conduct pre-audit Compare practices against
meeting requirements
Document results

Conduct Follow-up
Step 4: Evaluate Audit Results Track status of corrective
Evaluate audit results actions
Write audit findings Confirm closure of findings

Step 5: Report Audit Findings


Conduct exit meeting

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 221
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Evaluating Audit Results

The purpose of Step 4 is to evaluate and summarise the audit


results so as to write clear and concise findings that are
supported by sufficient audit evidence.

Finding A conclusion based on documentary,


physical, and testimonial evidence, with
respect to health, safety and
environmental performance

There is a Everything is
problem OK

Exception Good Management Management Local Attention


Practice Systems Item
Observation Observation
An identified An identified An identified An isolated anomaly
deficiency with weakness with weakness with found in existing
respect to a respect to general respect to the programs where
regulatory (industry) standards processes used by regulatory or
requirement or a of good practice in a facility to achieve company standards
company policy health, safety and and maintain of performance exist
environmental conformance with
management established
standards, including
programs, policies,
equipment,
administrative
controls etc.

In order to be sufficient to support the audit findings, audit


evidence should be:
• Relevant
• Objective
• Persuasive

The first two properties relate to the appropriateness of evidence.


The last requirement, persuasiveness, refers to its strength.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 222
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Evaluating Audit Results

Relevance

HSE audit evidence should provide a logical basis of support for


the findings. As a check on the relevance of the evidence, an
auditor should be explicit about the hypothesis he/she is trying to
verify. Thus, information developed through a review of the
facility’s file of hazardous waste manifests is relevant as
verification of the hypothesis that “[a sample of] shipments of
hazardous waste are appropriately documented in terms of the
EPA’s manifest requirements.” However, the manifest files, by
themselves, would not be relevant to verify the hypothesis that
“all hazardous waste shipments made by the facility have, in fact,
been manifested.” In the latter case, other documents, such as
shipping logs, would also be relevant.

Objectivity

Audit evidence is objective if it is free from bias. The objective


quality of evidence should lead two auditors examining the same
evidence to reach the same conclusion. Auditors need to ask
themselves (and each other) whether the evidence collected in
Step 3 presents an unbiased and, therefore, representative,
picture of the true situation. Bias, and a resulting lack of
objectivity, could arise from at least three sources:

• The auditor’s sampling strategy could introduce bias. For


example, he/she might review all of the incident investigation
reports in the Safety Coordinator’s files and conclude that they
were being filled out correctly and comprehensively, without
realising that there was a completely separate file in the
Maintenance Supervisor’s office, which addressed all of the
incidents relating to maintenance personnel.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 223
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Evaluating Audit Results

• The auditor could be making some implicit assumptions,


based on his/her prior knowledge and experience at other
facilities, that are, in fact, not true for the facility being audited.
This could introduce an inappropriate subjective element into
the evaluation process. For example, an auditor might
assume that all maintenance personnel handle hazardous
waste and, therefore, require annual refresher training, when
this might not be the case.

• Facility personnel could overstate their adherence to internal


procedures if they feel a need to appear more confident and
efficient than they really are.

Persuasiveness

Evidence is persuasive when it forces a specific conclusion to be


drawn and when another reasonable and knowledgeable person
would not challenge the validity of the conclusion nor propose a
conceptually different alternative. For example, the evidence that
life does not exist on the planet Mercury is very persuasive, but
the evidence that there is no extraterrestrial intelligence
elsewhere in the universe is less so. In the HSE context, the
evidence is persuasive that the facility has a fire extinguisher
inspection programme when fire extinguishers are tagged
with inspection stickers; the facility has records indicating
that inspections are conducted; and the person in charge of the
inspections knows when they were last done and when the
next inspections are due.

The table below indicates some examples of relevant, objective,


and persuasive evidence to support particular hypotheses.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 224
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Evaluating Audit Results

Topic Hypothesis Examples of Sufficient Data


Evidence Gathered
• Written hazardous waste
Hazardous The facility has √
training programme has
waste training implemented a
been prepared.
programme. hazardous waste
training • Training records for

programme. personnel involved in
hazardous waste
management are on file.
• Training instructor has the

necessary qualifications
and expertise to teach the
course.

• Written respiratory
Respiratory The facility has √
protection programme has
protection implemented an
been prepared.
programme. effective
• Fit testing and medical
respiratory √
protection clearance have been
programme. performed.
• Periodic inspections of

respirators are conducted.
• Personnel are trained in

the use and maintenance
of respirators.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 225
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Evaluating Audit Results

Listed below are some basic techniques for the auditor and audit
team to consider in evaluating audit results and determining if
there are sufficient data to substantiate the findings.

As an Individual

1. Confirm the Appropriateness of the Data Gathered


Relative to the Audit Scope
As each auditor completes an assigned protocol step—or a
group of protocol steps on a particular topic, such as “Spill
Response Training”— he/she should take time to review the
actions taken to ensure that sufficient data have been gathered.
In doing this, he/she should carefully weigh the data against the
criteria discussed on the previous pages:
• Relevance
• Objectivity
• Persuasiveness

Auditors should also ask themselves whether they have collected


an appropriate mix of physical, testimonial, and documentary
evidence, using appropriate data-gathering techniques
(observation, inquiry, and testing).

If a sampling strategy was used, the auditor should ask


him/herself one more time whether the sample taken was, in
retrospect, the right one.

Finally, individual auditors should be sure they have correctly


identified the regulatory and internal requirements relevant to the
particular protocol topic.

2. Summarise Conclusions
Having satisfied him/herself that he/she has gathered sufficient
evidence for each assigned protocol topic, the auditor should
next summarise his/her conclusions and outline his/her findings.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 226
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Evaluating Audit Results

As a Team

3. Ensure That All Findings Are Substantiated by the


Evidence Collected
Having reviewed and evaluated individual audit conclusions, the
next step is to meet as a team to review all protocol topics
included within the scope of the audit. With the team leader
presiding, each of the team members should briefly discuss the
actions taken and conclusions reached for each protocol area.
As the responsible team member reviews the actions he/she took
to address a particular area, other team members should politely
and constructively challenge the conclusions.

• Critically review the conclusions. It is essential in this step


to critically analyse the rationale for those protocol topics
where the team did not identify any findings. For example, if
no problems were found with respect to the use of respiratory
protection equipment, ask yourselves: “Did we talk to the right
people? Are we sure that we accurately identified the affected
population? Could there be other plant areas or job tasks
where respirators are used (e.g., in the laboratory, unloading
rail cars, etc.)?”

• Play “devil’s advocate.” Finally, pause for a moment to ask


yourselves what you could have missed. Given what you
have learned about the facility’s programmes, ask yourselves,
what is the worst that could happen, and make sure the
actions you took to address the protocol area were likely to
catch the “worst case,” at least for your “dig deepest” and “dig
deep” priorities.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 227
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Evaluating Audit Results

4. Prepare Written Findings


The final step in evaluating audit results is to prepare written
findings to present to the facility at the exit meeting. The written
findings can be presented either in a draft report or simply in a list
of the findings. The purpose here is to provide an organised,
complete, written summary of the exceptions noted by the team.
It is important to present the finding in writing because the true
nature of the issue may not be fully appreciated until the facility
sees the finding in black and white. As the audit team prepares
this document, they should pay careful attention to the following:

• Ensure factual accuracy. Because the written findings


provide the basis for the exit meeting with facility
management, it is critical that each finding listed be factually
accurate. Each team member, in reviewing the wording of the
finding, should make sure he/she has the facts to substantiate
each exception noted.

• Review with facility HSE coordinator. Prior to the exit


meeting, the audit team should review the written findings with
the facility individual who is responsible for day-to-day HSE
compliance. He/she will want to know what the team is
presenting to “the boss” before it is presented. Also, the
coordinator may have some legitimate questions and
comments on the findings that the team needs to clarify prior
to the exit meeting.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 228
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Evaluating Audit Results

5. Agree Upon Local Attention Items


Local attention items should be identified and categorisation
agreed upon as a team. Typically, local attention items are those
that meet one or more of the following criteria:

• Isolated anomalies found in existing programmes where


regulatory or company standards of performance exist (e.g.,
an occasional signature omission from documents requiring
signatures; a single exit sign not lighted).

• Minor items that lack specific criteria (e.g., MSDSs available


but not in each of several locations; responsibilities for
responding to complaints or incidents not well coordinated).

• Items that are outside the audit scope (e.g., a safety


deficiency observed during an environmental audit).

An item should never be for local attention if it is:


• An exception to a regulatory or company policy requirement
and is associated with a flaw in the management system.
• A repeat of a finding from a previous audit.
• An immediate danger to health, safety, or the environment.

Example Local Attention Only Items


The large alcohol tank in the tank farm does not have a drain plug in place to
back up the spring-loaded valve.

Two unlabelled 25-liter drums were observed on the concrete pad behind the
fire water pump building. (Containers elsewhere at the site were labelled
properly.)

One wooden ladder being used outside the aerosol gas house was found to be
unstable. Facility personnel removed and disposed of it immediately. (The
team did not note any other defective ladders.)

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 229
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 9
Evaluating Audit Results -
Specific “Local” vs. “Report” Decisions

Objective

The purpose of this exercise is to apply the guidelines for


classifying local attention items to audit results so as to gain
experience in determining whether findings are isolated
anomalies or whether the information is sufficient to justify
determination as an audit finding.

Instructions

On the following pages you will find several sets of findings


developed by audit teams. What we have attempted to do is to
describe situations that represent varying degrees of “severity” of
deficiency with regard to a particular requirement.

You are to assume that these situations are mutually exclusive.


That is, the auditor observed either the situation described in a.
or the situation described in b. or the situation described in c.
Also, you should assume that the situation statement describes
all of the observed deficiencies with respect to the indicated
requirement; there are no other closely related exceptions to the
requirement (i.e., if the exercise says you saw on employee not
wearing hearing protection, you may infer that all other
employees observed were wearing the required protection).

You challenge is to work in small groups to decide, within each of


these sets of specific findings, where you would draw the line
between “local attention” and “for report”. Note that it is quite
possible to decide that all three of the versions are in the same
category (e.g., all local attention items or all reportable).

Note: For purposes of this exercise, do not concentrate too much


over whether you would have one or more than one finding
based on the information given.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 230
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 9
Evaluating Audit Results -
Specific “Local” vs. “Report” Decisions

Set 1: Hearing Conservation


Requirement: The facility has several areas in which noise
levels exceed 90 dBA (8-hour TWA); therefore, hearing
protection is required. In addition, it is company policy that signs
be posted in these areas and that the entrances to these areas
be marked off with pink or green painted lines on the floor.

a. A door opens to an air compressor equipment area where


hearing protection is required. The area is posted as
“Hearing Protection Required”, but no pink/green marking
lines have been marked on the floor.

b. A door opens to an air compressor equipment area where


hearing protection is required. The area is posted as
“Hearing Protection Required”, but no pink/green marking
lines have been marked on the floor. One employee was
observed working in the air compressor area with no ear
protection.

c. A door opens to an air compressor equipment area where


hearing protection is required. The area is posted as
“Hearing Protection Required”, but no pink/green marking
lines have been marked on the floor. One employee was
observed working in the air compressor area with no ear
protection. The team also observed one employee (out of
ten) in the machine shop and one employee in the pump
room (both posted areas) not wearing hearing protection.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 231
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 9
Evaluating Audit Results -
Specific “Local” vs. “Report” Decisions

Set 2: Ventilation Flow Measurement


Requirement: As a matter of good management practice, the
facility should conduct and document inspection and
maintenance activities on ventilation systems to ensure that
filters are changed regularly and that accurate air flow tests are
conducted to confirm performance.

a. The facility has not measured the air flow for the hood in the
quality control laboratory at any time since installation, when
the flow was checked by the vendor.

b. The facility has not measured the air flow for the hood in the
quality control laboratory at any time since installation, when
the flow was checked by the vendor. There are no flow
measurement instruments available on site for testing the
performance of ventilation systems used for air contaminant
exposure control in the process areas; flows in process areas
are checked once yearly by a contractor.

c. The facility has not measured the air flow for the hood in the
quality control laboratory at any time since installation, when
the flow was checked by the vendor. There are no flow
measurement instruments available on site for testing the
performance of ventilation systems used for air contaminant
exposure control in the process areas; flows in process areas
are checked once yearly by a contractor. The site has
installed three new ventilation systems since the last
contractor inspection; no flow measurements have been
made on these systems.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 232
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 9
Evaluating Audit Results -
Specific “Local” vs. “Report” Decisions

Set 3: Chemical Containment


Requirement: The company’s spill control plan states that
secondary containment must be provided in all chemical storage
areas.

a. In the storeroom, there was one five-gallon container (not


empty) overhanging the edge of the spill containment
structure.

b. In the storeroom, there was one five-gallon container (not


empty) overhanging the edge of the spill containment
structure. In this area, there were also ten five-gallon
containers set on the floor beside the spill containment
structure. The containment structure was full of cans.

c. In the storeroom, there was one five-gallon container (not


empty) overhanging the edge of the spill containment
structure. In this area, there were also ten five-gallon
containers set on the floor beside the spill containment
structure. The containment structure was full of cans. In the
maintenance area, there were five-gallon drums of solvent
with no containment. In the boiler room, there was a 55-
gallon drum with no spill containment.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 233
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 9
Evaluating Audit Results -
Specific “Local” vs. “Report” Decisions

Set 4: Document Control


Requirement: As a matter of good management practice, there
should be a method of ensuring that critical HSE documents are
periodically reviewed and revised as necessary.

a. The facility does not have a formal document control


programme to ensure that plans such as the emergency
response plan are periodically reviewed, amended, and
reissued if necessary.

b. The facility does not have a formal document control


programme to ensure that plans such as the emergency
response plan are periodically reviewed, amended, and
reissued if necessary. The current emergency response plan
dates from 1990 and does not include the correct home
telephone number for the alternate response coordinator.

c. The facility does not have a formal document control


programme to ensure that plans such as the emergency
response plan are periodically reviewed, amended, and
reissued if necessary. The current emergency response plan
dates from 1990 and does not include the correct home
telephone number for the alternate response coordinator. It
also does not contain specific plans for responding to spills at
the aqueous ammonia tank installed in 1993.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 234
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 9
Evaluating Audit Results -
Specific “Local” vs. “Report” Decisions

Set 5: Management of “Used Oil” Drums


Requirement: Company policy states that no more than 55
gallons of used oil is to be stored in working areas. All full drums
are to be stored in designated areas, which are to be managed
as 90-day accumulation areas.

a. One of the six designated “used oil” storage areas showed


evidence of past spills and partially illegible container labels.

b. One of the six designated “used oil” storage areas showed


evidence of past spills and partially illegible container labels.
Three apparently full drums labeled “used oil” were observed
in the maintenance shop and two at the south side fuelling
area.

c. One of the six designated “used oil” storage areas showed


evidence of past spills and partially illegible container labels.
Three apparently full drums labeled “used oil” were observed
in the maintenance shop and two at the south side fuelling
area. Facility personnel interviewed could not identify
individual(s) specifically responsible for “used oil” drum
handling.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 235
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 9 – Potential Answers
Specific “Local” vs. “Report” Decisions

Set 1: Hearing Conservation

a. local b. report c. report

Set 2: Ventilation Flow Measurement

a. local b. local c. report

Set 3: Chemical Containment

a. local b. report c. report

Set 4: Document Control

a. report b. report c. report

Set 5: Management of “Used Oil” Drums

a. report b. report c. report

You may find that this opinion may change in the light of
additional background information. For example the
recommended answers for set 2 would be dependent on what
the ventilation hood is used for, its toxicity and the frequency of
exposure to employees.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 236
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Evaluating Audit Results

Management Systems Information

In addition to the above, there are several other methods the


auditor can apply to determine if the data are sufficient to develop
a management systems observation. These are:

Identify trends among the compliance findings.


Perform a cause and effect analysis.

This process is generally conducted as a team.

1. Identify Trends Among the Compliance Findings

Once the auditors have developed the complete list of findings,


they should, as a team, review the list to develop an integrated,
organised summary. In this step, the team should do the
following:

• Identify common findings. Look for situations where two or


more of the individual findings listed may relate to one basic
problem, and may represent a system deficiency.

• Look for patterns or trends. In reviewing the findings, try to


find whether any patterns emerge which suggest that several
findings should be combined. Ask yourselves whether there
are several findings that, when viewed as a group, may have
greater significance.

• Be alert to systemic issues. Ask yourselves whether the


symptoms observed (errors, omissions, etc.) are
manifestations of a more fundamental systems weakness.

For example, the audit team may have several findings all related
to the same general topic, such as training, and the findings
themselves all point to the underlying management systems
deficiency that there is no means to identify and track HSE-
related training.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 237
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Evaluating Audit Results

Individual
Individualfindings
findingsrelated
relatedto:
to: The
Themanagement
managementsystem
system
deficiency:
deficiency:
Hazardous
Hazardouswaste
wastetraining
training
Spill
Spillresponse
responsetraining
training No
Nomeans
meanstotoidentify
identifyand
andtrack
track
Hazard
Hazardcommunication
communicationtraining
training HSE-related
HSE-relatedtraining
training
Respirator
Respiratoruse
usetraining
training
Confirmed
Confirmedspaces
spacesentry
entrytraining
training

However, management systems observations are not always so


easy to recognise during an audit. Thus, the team may want to
relate the findings back to the management systems processes.

In Step 1, the team gained a lot of insight into how HSE


compliance is supposed to be managed at the facility. As the
team verified conformance with the facility’s system, they most
likely uncovered some gaps and breakdowns. As a means of
discerning the management systems deficiencies, the team
should try to relate individual findings to one or more of the
management processes utilised by the facility to manage its HSE
matters. The table on the following page describes how the
individual findings can be categorised within the various
management processes to assist in developing management
system observations.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 238
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Evaluating Audit Results

Compliance Findings Management Management Systems


System Observation
Activity
• The facility has not Assessing The facility has not
developed an air emissions assessed its HSE issues
inventory. and risks in several areas,
• The facility has not including:
conducted a noise survey.
• Air emissions inventory
• The facility has not
• Noise survey
characterised 12 different
• Waste characterisation
waste streams.
• Workplace montinoring for
• The facility has not
XYZ chemicals
conducted workplace
monitoring for xyz
chemicals.
• The facility has not Reviewing There is no comprehensive
conducted 7 of 12 monthly or system for HSE-related
safety inspections. Organisation/ inspections.
• The facility does not
Resources or
document its fire
Responsibility and
extinguisher inspections.
accountability for the HSE
• Respirators were observed
inspection function have not
stored out on shop floor. been clearly defined.

2. Perform a Cause and Effect Analysis

Another technique to develop a management systems


observation is to identify the underlying causes associated with
compliance and good management practice findings. The
following examples illustrate this approach.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 239
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Evaluating Audit Results

Finding
The facility exceeded its wastewater pH limit on several occasions during
the review period. Upon investigation, the team learned that the wastewater
treatment system controls had not been maintained regularly and, thus,
failed to properly adjust the pH of the effluent.

Incident/Effect
An exceedance of the pH (level of acid/base) was noted at the wastewater
treatment plant.

Possible Causes
• Failure of pH controls resulting from lack of maintenance
• Training
• Inspection
• Lack of redundant systems
• Wrong equipment
• System to ensure QA (procedures)
• Labor shortage

Finding
During the audit, the team observed that the employee’s hand could be
pulled into the mobile pumping unit because the unit does not have a guard.
Upon investigation, the team learned that the guard had been removed to
make access to the equipment easier and a system is not in place to
periodically verify that guards are in place where required.

Incident/Effect
The belt-driven pulley on mobile pumping unit 5123 does not have a guard.

Possible Causes
• Training
• Inspection
• Lack of equipment

To make this technique truly useful, the cause-effect model


begins at the incident, or effect, and proceeds backward by
asking “why” until the appropriate management system
conclusion is reached.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 240
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Evaluating Audit Results

Cause 1 Incident
Cause 2
Cause 3

Root cause

After the cause (i.e., the main point to communicate to


management) is identified, the team is in a position to write a
management systems observation.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 241
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Writing Audit Findings

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 242
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Writing Audit Findings

As the team prepares the list of audit findings, it is important to


ensure that each is written in a manner to clearly and accurately
communicate the facts. It is important to keep in mind several
principles when wording audit findings:

1. Do Not Overstate the Facts

State the facts as you have discovered them but avoid drawing
overly broad conclusions.

Don’t say... If you mean...

The facility does not have a PCB The facility’s PCB inspection
inspection programme. programme does not include PCB-
contaminated transformers.

The facility does not have a The facility’s respiratory protection


respiratory protection programme. programme does not include fit
testing or routine inspection and
maintenance of respirators.

2. Distinguish Between Performance and Documentation

Some regulatory or corporate requirements specify that a


particular activity or programme be conducted, but do not specify
that the completion of the activity or programme be documented.
In other cases, regulatory or corporate requirements specify that
the activity or programme be conducted, and that it be
documented to verify that it was conducted.

Don’t say... If you mean...

Weekly hazardous waste inspections Weekly hazardous waste inspections


are not conducted. are not documented.

Weekly ladder inspections are not Weekly ladder inspections are not
conducted. documented.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 243
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Writing Audit Findings

3. Avoid Generalities

Generalities and vague reporting will confuse and mislead the


reader. The specific problem should be succinctly
communicated. In addition, although a finding may be worded
factually, it may not contain enough information to fully
communicate the nature and extent of the problem.

Too general More helpful

The facility’s contingency plan is The facility’s contingency plan does


incomplete. not include the following elements:

a. Agreement with local authorities.

b. Types and location of fire


protection equipment.

c. Listing of emergency telephone


numbers.

Employees have not received safety Four of 30 maintenance mechanics


training. have not received lockout/tagout
training.

4. Do Not Draw Legal Opinions

Legal judgments, interpretations, and conclusions should be


avoided when writing audit findings. Generally speaking, legal
conclusions can be characterised by words such as “in violation
of…,” “not in compliance with…,” “as required by…,” etc.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 244
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Writing Audit Findings

Legal conclusion Factual conclusion

Hazardous waste drums were not Drums storing hazardous waste were
labeled as required by 40 CFR not labeled with the words, “Hazardous
262.34. Waste,” or other words indicating their
contents. (40 CFR 262.34(c)(1)(ii))

During the review period, the facility During the review period, the facility did
was not in compliance with 29 CFR not conduct annual hearing
1910.95. conservation training. (29 CFR
1910.95)

5. Give Regulatory or Company Policy References

Because the basis for a finding may not always be clear to the
report recipient, particularly if the report recipient is an individual
who is not involved with environmental, health, and safety issues
on a daily basis, regulatory or company policy references should
be included.

Poor Improved

Required annual hazardous waste Annual hazardous waste training has


training has not been conducted not been conducted within the past
within the past 18 months. 18 months. (40 CFR 265.16 and XYZ
Company Policy, HAZWASTE 3.2)

Required annual hearing Annual hearing conservation training


conservation testing has not been has not been conducted within the
conducted within the past 18 months. past 18 months. (29 CFR 1910.95
and XYZ Company Policy,
HEARCON 4.7)

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 245
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Writing Audit Findings

6. Avoid Extreme Language

Refrain from using such deprecating words as careless, terrible,


dangerous, intentional, severe, reckless, incompetent, and the
like. These words can be broadly interpreted and are not helpful
in communicating the exact nature of the problem.

Poor Improved

The lack of spill containment Spill containment provisions are not


provisions on the loading dock may available at the loading dock area
lead to a dangerous situation. where hazardous chemicals are
loaded/unloaded adjacent to a drain
that discharges to surface waters.

The lack of documented confined The manufacturing operations do not


space entry procedures for the have written confined space entry
manufacturing operations may lead procedures.
to an injury/accident.

7. Use Familiar Terminology

Not all recipients of the report will be involved in health, safety


and environmental activities on a daily basis and, thus, they may
not be as familiar with the health, safety and environmental
acronyms, abbreviations, and regulatory jargon as the auditors
are.

Poor Improved

The facility does not have pollution The facility does not have pollution
prevention equipment to prevent prevention equipment to prevent
exceedances of TSS, BOD, and oil and exceedances of total suspended solids
grease in its discharges to the POTW. (TSS), biochemical oxygen demand (BOD),
and oil and grease in its discharges to the
publicly owned treatment works (POTW).

Four of 12 P&IDs reviewed were out of Four of 12 piping and instrumentation


date. diagrams (P&IDs) were out of date.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 246
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Writing Audit Findings

8. Do Not Focus Criticism on Individuals or Their Mistakes

Do not identify the individual(s) involved in a performance finding


or as sources of information. The audit is a review of facility
programmes and practices, not of individuals.

Poor Improved

John Doe and Jane Smith were The team observed maintenance
observed . . . . personnel . . . .

9. Avoid Contradictory Messages

Activities or programmes presented in a positive light, when the


ultimate message will involve pointing out deficiencies, may
confuse the reader and obscure the real message being
conveyed.

Poor Improved

Although the facility has a well-written The facility’s waste analysis plan
waste analysis plan, it does not does not include the following:
include parameters for each
hazardous waste analyzed or the a. Parameters for each hazardous
frequency of analysis. waste analyzed.

b. Frequency of analysis.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 247
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Writing Audit Findings

Poor Improved

Although the facility has a site- In reviewing the facility’s hazard


specific written hazard communication practices, the team
communication programme, the noted the following:
hazard communication inventory is
not up to date, nor does the facility a. The facility’s hazard
have a procedure to ensure that communication inventory is not up
employees receive hazard to date.
communication training upon initial
assignment. b. The facility does not have a
procedure to ensure that
employees receive hazard
communication training upon initial
assignment.

10. Group Similar Findings

Rather than state several individual findings, look for patterns or


trends and group similar findings to more fully convey the
message.

Individual Grouped

Quarterly Discharge Report Reporting of Sample Events to


The analysis results for one of 13 POTW
samples of the wastewater In reviewing quarterly monitoring
discharged to the publicly owned reports and analytical data, the team
treatment works (POTW) was not noted the following:
included in the quarterly discharge
report. a. One of 13 samples taken and
analyzed of the wastewater
Reporting of Analytical Results discharged to the publicly owned
Analytical results for samples taken treatment works (POTW) was not
at internal outfalls were not reported included in the quarterly discharge
to the POTW. report.

b. Analytical results of samples taken


at internal outfalls were not
reported to the POTW.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 248
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Writing Audit Findings

Individual Grouped

Labelling Confined Spaces Permit-Required Confined Spaces


The facility has not posted danger In reviewing the facility’s confined
signs or used any other equally space entry programme, the team
effective means to identify the noted the following:
existence and location of and the
danger posed by the permit-required a. The facility has not posted danger
confined spaces. signs or used any other equally
effective means to identify the
Confined Space Permit existence and location of and the
Programme danger posed by the permit-
The confined space permit required confined spaces.
programme does not include
procedures: b. The confined space permit
programme does not include
a. To be taken in the event that a procedures:
hazardous atmosphere is detected
during entry. 1) To be taken in the event that a
hazardous atmosphere is
b. For verifying that conditions in the detected during entry.
permit space are acceptable for
entry throughout the duration of 2) For verifying that conditions in
the authorised entry. the permit space are acceptable
for entry throughout the duration
of the authorised entry.

11. Write Management Systems Observations

To be effective, management systems observations need to be:


• Crisp
• To the point (one point at a time)
• Clearly framed
• Expressed in terms meaningful to managers

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 249
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Writing Audit Findings

What’s Wrong With This Finding?


Clearly Defined Responsibilities
A reorganisation of the plant management functions had recently occurred.
In conjunction with this reorganisation, a newly created position of Manager,
Health, Safety, and Environment Special Projects had been established with
the main job activities listed in the personnel job description form. The
Safety department organisation has also been expanded. Greater emphasis
is being placed on health, safety and environmental issues as evidenced by
top plant management now having HSE concerns listed in their formal job
description outlines. Plans are under way to expand this emphasis by
including these same issues in the job descriptions of first line supervision.

In the past, environmental concerns were jointly addressed by the


Engineering department and the Maintenance department. The majority of
the contacts with governmental agencies were handled by these two
departments. Under the recent reorganisation, the Engineering and
Maintenance departments have been combined under one manager and
most of the previous job activities and responsibilities of both departments
have been combined. Although the job duties and responsibilities of the
newly created position of Manager, HSE have been defined by management,
no accountability document has been prepared to indicate specific
responsibility or joint accountability with other departments. In addition, it
was not clearly defined who can sign HSE documents and correspondence.

This management systems observation regarding “clearly defined


responsibilities”:

• Uses 208 words

• Includes extraneous information on:


− The reorganisation itself
− The Safety department
− Greater emphasis on HSE issues
− Top management and supervisors job descriptions

• Unnecessarily muddles the basic accountability issue with the


lesser question of who should sign HSE documents and
correspondence.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 250
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Writing Audit Findings

A Sharper Picture
Clearly Defined Responsibilities
The recent organisation has left undefined the issue of whether the new HSE
Manager position shares any joint responsibilities or accountabilities with the
now combined Engineering and Maintenance department.

This revised finding is:

• Twenty-nine words, or an 86 percent reduction in length.

• Crisp and to the point, which is that of joint accountability.

• Clearly framed as a management issue, that of clarifying


accountabilities.

• Much more easily read, understood, and acted upon.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 251
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Writing Audit Findings

Example Management Systems Observations

Tracking of Training
The facility has not developed a comprehensive system for tracking the
movement of employees who change job positions to ensure that regulatory
required training is provided in a timely fashion. For example, environmental
management staff do not know which employees (some may be security
contractors) are responsible for chemical and petroleum product loading and
unloading.

Promotion of Safety and Health Management


Management commitment to the recently reissued health and safety policy
does not appear fully evident in that:
• A facility inspection programme that includes senior management
participation is not in place.
• Promotional activities such as safety/environmental recognition awards,
safety contests, etc., have not been developed.
• There are gaps in the attendance of employee, supervisory, and
management personnel in mandatory safety training sessions.
• Middle management participation in at least two safety, health, and loss
prevention meetings with their employees per year does not occur.

Safety Coordinator Programme


In reviewing the facility’s safety coordinator programme, the team noted
several issues that impact the effectiveness of this programme. For example:
• The roles and responsibilities of the safety coordinators have not been clearly
defined and approved by line management.
• The safety coordinators do not regularly attend and participate in safety
coordinator meetings.
• Some of the safety coordinators lack the training necessary to perform their
responsibilities in accordance with facility policies and procedures.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 252
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Writing Audit Findings

Example Management Systems Observations


Management of Health and Safety Issues
The management of health and safety issues is not fully integrated into day-to-
day operational activities as evidenced by the concerns noted in the areas of
contractor compliance, electrical safety, and gaps in the safety coordinator
inspection programme. Based on interviews with facility staff, several barriers
were noted that hinder the effective integration of health and safety with
operations. For example:
• The absence of management directives outlining the role of health and safety
staff in the day-to-day administration of compliance programmes throughout
the facility.
• The lack of periodic meetings among line management and health and safety
staff to discuss and foster integration of health and safety into operations.
In addition, the process for developing and revising safety, health, and loss
prevention procedures does not include a formal review by line management to
help ensure approval and effective implementation.

12. Consider Using a Template

Consider using a template to prepare a preliminary draft of your


findings.

Template for Writing Findings


Template Finding
The Statistical Finding
Fifteen of a sample of 30 of the 52 [# of deficiencies] of a sample of [# in
employees who routinely enter sample] of [# in universe] of what is
confined spaces did not receive wrong
training during the last year
The Have/Do Finding
The facility does not have an air Who or what / does not / do not have /
pollution permit to operate the what they do not have
three boilers in Powerhouse B on
site

Facility personnel do not conduct Who / does not do it / does not do


or document inspections of the (perform) / what they do not do
hazardous waste accumulation
area

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 253
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Writing Audit Findings

Template for Writing Findings


Template Finding
The Grouped Finding
In touring/reviewing / the area
In reviewing the facility’s practices
toured/document reviewed / the
with respect to
team noted the following:
flammable/combustible liquid
handling, the team noted the
a. List the exceptions.
following:

a. An oxidiser (nitric acid) was


stored in the same flammable
liquid storage cabinet as
organic solvents
b. Self-closing valves have not
been installed on
combustible/flammable drums
in the dispensing area
c. Devices for bonding dispensing
drums to portable metal
containers were not in use

[29 CFR 1910.106]


The Observed Finding
During the facility tour, the team Who or what / observed / what was
observed an oil sheen near the observed?
outfall to the lake
The Told Finding
The team was told that there have The team was told that what the team
been several spills of hazardous was told
waste to the storm sewers

13. Avoid These Words

Adequate Occasionally
Inadequate Insufficient
Some Compliance
Few Noncompliance
Many Required
Not all Violation
Sometimes

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 254
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 10
Writing Audit Findings - Critiquing Audit Findings

Objective

The purpose of this exercise is to review a set of written audit


findings and determine whether the findings clearly and
appropriately communicate the audit results.

Instructions

Based on the principles regarding how to properly word findings,


critique the following exit meeting discussion sheets. Also offer
suggestions on how to rewrite the findings.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 255
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 10
Writing Audit Findings - Critiquing Audit Findings

Exit Meeting Discussion Sheet


Facility XYZ Facility Discipline HSE

Audit Team Present John Collins, Brenda Fields, Bob Taylor, Jane Smith

Facility Management Present Ralph Gold

Others Present Bill Dunn

Discussion Date MM/DD/YY Prepared by John Collins

# Exception Critique

1 We could not verify that waste manifests were


received from TSDFs (Treatment, Storage and
Disposal Facility) within 45 days of shipment.

2 Current storage of emergency response


equipment may result in increased likelihood of
failure.

3 There is minimal on-site compliance with


corporate or department contractor safety
policy and procedures.

4 Some of the air sources are being operated


without proper permits and some are not
adequately maintained.

5 The facility’s central MSDS file is very neat and


accessible to those employees who should see
it. Not all materials used or stored by the
facility have MSDSs (Material Safety Data
Sheets) in the central file. Those MSDSs
reviewed appeared complete and contained the
appropriate information.

6 There are no toe boards and missing hand


rails.

7 A discrepancy exists among the frequency of


safety inspections.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 256
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 10
Writing Audit Findings - Critiquing Audit Findings

Exit Meeting Discussion Sheet (continued)


# Exception Critique

8 Ron Kline and Seth McGee were not familiar


with the company’s hazard communication
programme or could identify where MSDSs
were located.

9 The facility’s computer monitoring programme


for permit expiration was found to lack a
procedure to make sure that all permits were
entered into the computer system to begin with.

10 No inspection and maintenance records were


available to the audit team and no documented
procedures for rail car loading / unloading.

11 There is insufficient personnel to manage all


HSE matters given the requirements put forth
in the operating manuals which describe the
gamut of environmental, health, and safety
regulations.

12 The facility often goes through changes in


operations which result in additional
environmental impacts only to find notifications
to permit conditions and variation in employee
safety conditions. A management system
should be addressed to fix this problem.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 257
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 10 – Potential Answers
Writing Audit Findings - Critiquing Audit Findings

Exit Meeting Discussion Sheet


Facility XYZ Facility Discipline HSE

Audit Team Present John Collins, Brenda Fields, Bob Taylor, Jane Smith

Facility Management Present Ralph Gold

Others Present Bill Dunn

Discussion Date MM/DD/YY Prepared by John Collins

# Exception Critique

1 We could not verify that waste manifests − Implies auditor has not done job
were received from TSDFs within 45 days properly
of shipment.
− Need to include citation
− What does TSDF stand for?
2 Current storage of emergency response − Should not use ambiguous terms
equipment may result in increased (e.g. may, increased)
likelihood of failure.
− So what?
− Which equipment?
3 There is minimal on-site compliance with − Not specific
corporate or department contractor safety
policy and procedures. − Does not describe problem to help
site correct it
4 Some of the air sources are being − Do not use “some” or “proper” or
operated without proper permits and some “adequately”
are not adequately maintained.
− Be specific; which sources, how
many?
5 The facility’s central MSDS file is very neat − What does MSDS stand for?
and accessible to those employees who
should see it. Not all materials used or − Combines good and bad findings
stored by the facility have MSDSs in the − Do not use “not all” or “appeared”
central file. Those MSDSs reviewed
appeared complete and contained the
appropriate information.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 258
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Exercise 10 – Potential Answers
Writing Audit Findings - Critiquing Audit Findings

Exit Meeting Discussion Sheet (continued)


# Exception Critique

6 There are no toe boards and missing − So what?


hand rails.
− How many?
− Where?
7 A discrepancy exists among the − What is the discrepancy?
frequency of safety inspections.
− Needs clearer description
8 Ron Kline and Seth McGee were not − Avoid using names
familiar with the company’s hazard
communication programme or could
identify where MSDSs were located.

9 The facility’s computer monitoring − Bad English


programme for permit expiration was
found to lack a procedure to make sure − Irrelevant information
that all permits were entered into the
computer system to begin with.

10 No inspection and maintenance records − So what?


were available to the audit team and no
documented procedures for rail car − What about the procedures for
loading / unloading. loading / unloading?

11 There is insufficient personnel to − Judgmental not factual


manage all HSE matters given the
requirements put forth in the operating − Do not use “insufficient”
manuals which describe the gamut of
environmental, health, and safety
regulations.

12 The facility often goes through − Bad English


changes in operations which result in
− What’s the problem?
additional environmental impacts
only to find notifications to permit − Should the finding include a
conditions and variation in employee recommendation?
safety conditions. A management
system should be addressed to fix
this problem.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 259
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Post-Audit Activities

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 260
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Post-Audit Activities

Pre-Audit Activities On-Site Activities Post-Audit Activities

Step 1: Understand Prepare Draft Report


Management Systems
Conduct opening meeting Obtain Review Comments from
Select & Schedule Facility Audit Conduct orientation tour Corporate HSE
Review audit strategy Law department
Understand details of Facility management
management systems

Step 2: Assess strengths Issue Final Report to


& Weaknesses Facility management
Select Team Members & Confirm Consider potential impacts Operations
their Availability Evaluate management Corporate HSE
systems Law department
Set priorities for verification

Step 3: Gather Audit Evidence


Plan the Audit: Evaluate what needs to be
Correspond with the done Develop Action Plan
facility Determine depth & rigor of Develop proposed action(s) to
Assemble & distribute review address each finding
background information Select types of evidence Assign responsibility for
Assign & communicate needed & methods to gather corrective action
audit responsibilities them Develop timetable
Conduct pre-audit Compare practices against
meeting requirements
Document results

Conduct Follow-up
Step 4: Evaluate Audit Results Track status of corrective
Evaluate audit results actions
Write audit findings Confirm closure of findings

Step 5: Report Audit Findings


Conduct exit meeting

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 261
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Post-Audit Activities –
Introduction

The objectives and associated responsibilities of the post-audit


activities are outlined below.

Objectives Responsibilities
• To ensure that the audit results Meeting this objective is typically part
are clearly communicated to the of the audit team’s responsibility. This
appropriate levels of management. is accomplished by means of a formal,
written audit report, as described in
this section of the manual.
• To ensure that all audit findings Meeting this objective is typically the
are addressed by management responsibility of line management,
through the implementation of a although the audit team may be asked
formal corrective action process.
to make recommendations, review
proposed action plans, and/or track
the implementation and closure of
corrective action.
• To evaluate the effectiveness of Meeting these last two objectives is
the audit and provide suggestions frequently among the responsibilities
for improving future efforts. of the audit programme manager and
team leader(s).
• To share lessons learned during
the audit, especially to similar
facilities.

In this section of the manual, we discuss:


• Preparation of audit reports
• Protection of audit results
• Audit policy statements
• Quality assurance in the audit process

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 262
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Post-Audit Activities –
Preparation of Audit Reports

Purpose of Audit Reports

The overall goal of an audit report is to document the audit


findings clearly and accurately. Within this overall goal, an audit
report has three basic purposes:

• To document the scope of the audit and the audit team’s


conclusions regarding the facility’s compliance status.

• To provide appropriate levels of management with information


on the results of the audit—information sufficient to meet the
needs of the report’s recipients and consistent with the overall
objectives of the audit programme.

• To initiate corrective action so that once exceptions to


applicable requirements have been identified, action steps are
set in motion to correct the deficiencies found.

A strong linkage exists between the report’s purpose and the


overall audit programme’s objective. For example, where the
primary objective of the audit programme is to provide assurance
to management, the purpose of the audit report is to provide top
management with information on the more significant findings.
Likewise, when the primary purpose of the audit programme is to
provide plant management with information on the
environmental, health, and safety status of the facility, the
purpose of the audit report is to help facility managers.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 263
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Post-Audit Activities –
Preparation of Audit Reports

Report Recipients

What Are “Appropriate” Levels of Management?


Most audit programmes use a hierarchical reporting scheme.

• Most, but not all, programmes today require dissemination of


audit reports to high levels of management.

• Increasingly, top corporate managers and/or boards of


directors expect to be informed of audit results.

• There may, however, be some audit findings (“local attention


items”) that do not require reporting beyond the facility
manager level.

An example of a hierarchical reporting scheme is shown on the


following page.

Example of a Hierarchical Reporting Scheme

Who How What


Facility HSE staff Daily communication All deficiencies noted
Facility manager Exit meeting; draft and All deficiencies noted
final reports
Corporate HSE Affairs; Draft and final reports All deficiencies noted,
Law Department; except local attention
division or group items
management
Corporate management Periodic status summary Significant matters;
overall patterns and
trends; general
programme status
Board of directors Summary presentation Overall HSE
at a board meeting performance; most
significant matters

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 264
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Post-Audit Activities –
Preparation of Audit Reports

Report Form and Content

The health, safety and environmental audit report, according to


international and industry standards for audit programme
performance, must be in writing in order to allow results to be
widely communicated, misunderstandings reduced, and follow-up
facilitated.

Audit reports must be:


• Objective
• Clear
• Concise
• Timely

The type of information and level of detail provided in an audit


report depends upon the objectives of the audit programme, the
needs of the report recipients, and the problems identified. An
example of a report outline that meets the minimum requirements
specified by the Environmental Auditing Roundtable is provided
below.

I. Introduction
Purpose, date, and scope of the audit
Name, location, and description of the audited site
Names of the audit team members
Criteria utilised in performing the audit
Deviations from the planned scope

II. Exceptions and observations discovered during the audit

III. General instructions for response and follow-up

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 265
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Post-Audit Activities –
Preparation of Audit Reports

Similarly, ISO 14000 (ISO 14010, Section 5.7 and ISO 14011,
Section 5.4.2) describes the contents of the audit report as
follows:

The audit report should be dated and signed by the lead auditor. The audit
report should contain the audit findings or a summary thereof with reference to
supporting evidence. Subject to agreement between the lead auditor and the
client, the audit report may also include the following:

a. The identification of the organisation audited and of the client.


b. The agreed objectives, scope, and plan of the audit.
c. The agreed criteria, including a list of reference documents against which the
audit was conducted.
d. The period covered by the audit and the date(s) the audit was conducted.
e. The identification of the auditee’s representatives participating in the audit.
f. The identification of the audit team members.
g. A statement of the confidential nature of the contents.
h. The distribution list for the audit report.
i. A summary of the audit process, including any obstacles encountered.
j. Audit conclusions, such as:
– EMS conformance to the EMS audit criteria.
– Whether the system is properly implemented and maintained.
– Whether the internal management review process is able to ensure the
continuing suitability and effectiveness of the EMS.

Audit results should be reported in a manner that is clear and


easily understood by the recipient of the report. Audit findings
need to be described in an appropriate managerial context, which
takes into account the recipient’s extent of familiarity with the
subject matter. They also need to be free of jargon and
unfamiliar terminology.
Audit reports should also be factual, unbiased, and free from
distortion. Findings should be prepared without prejudice and
expressed as pertinent statements of fact, which are supported
by sufficient, valid, and documented evidence gathered during
the audit. The focus of the audit report should be on the findings
developed during the audit and, depending on the scope of the
audit programme, recommendations where necessary.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 266
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Post-Audit Activities –
Preparation of Audit Reports

The length of the audit report varies widely, depending on the


scope of the audit and the depth of detail needed to meet the
recipients’ needs.

Potential Components of an Audit Report

A strong relationship exists between the format and content of


the audit report and the needs of the report’s recipients.
Although audit reports should satisfy the minimum criteria
described on the previous pages, there are also a number of
potential components to consider. The decision to incorporate
one or more of these alternatives into the audit report is really a
programme design issue and not within the discretion of an
individual auditor or audit team. Nevertheless, auditors need to
have a clear understanding of how their company’s audit reports
are designed, so that they can contribute effectively to the
reporting process.

Examples of audit report components are described on the


following pages.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 267
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Post-Audit Activities –
Preparation of Audit Reports

Exceptions Only

An exception report is essentially designed to meet the minimum


report requirements. The “exceptions and observations” section
of such a report is its distinguishing characteristic in that it simply
communicates departures from established governmental or
internal standards and observations with respect to general
(industry) standards of good practice in HSE management. This
format does not necessarily provide the report recipients with a
means of interpreting the overall significance of the audit findings
and may not represent the most desirable option for readers who
lack sufficient context to understand and assess the status of the
facility’s performance. This alternative is illustrated below.

II. Exceptions and Observations

A. Water Pollution Control

1. Sampling Frequency (Regulatory)


The facility does not sample National Pollutant Discharge Elimination System
(NPDES) parameters such as Biological Oxygen Demand (BOD) and pH on a
biweekly schedule. [40 CFR 122.41]

2. Stormwater Pollution Prevention Plan (Regulatory)


The facility has not completed a Stormwater Pollution Prevention Plan
(SWPPP) and submitted the corresponding certification form to the Bureau of
Stormwater Permitting. [NJPDES General Permit NJ0088315]

3. Etc.....

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 268
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Post-Audit Activities –
Preparation of Audit Reports

Recommendations

It is becoming more common for an audit programme to assist


the facilities by developing suggested recommendations for
corrective action to address each of the exceptions and
observations. This is particularly helpful when the facility HSE
personnel may not be completely familiar with the requirements
(especially regulatory requirements) or to facilitate the
development of corrective actions within a specified time frame.
It is important to ensure that recommendations are written in a
way to convey what should be done, without dictating to facility
management how it is to be done. Some examples that illustrate
this what/how distinction are provided in the table below.

Don’t tell them how Do tell them what

The facility should require that the The facility should establish
plant environmental coordinator authorisation procedures to ensure
review all proposed changes to that accountable personnel assess the
process chemicals. impacts of process chemical changes
on waste stream characteristics.

The facility should hire an industrial The facility should assess the need for
hygienist and two additional process additional industrial hygiene and
safety engineers. process safety resources to perform
an employee exposure assessment
and process hazards analysis in units
A, B, and C.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 269
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Post-Audit Activities –
Preparation of Audit Reports

Audit Opinion

An audit opinion provides an overall classification of the facility’s


performance in addition to a list of exceptions and observations.
An opinion report provides management with a means for
interpreting the significance of the audit results and assists them
in focusing their resources on areas where improvement is most
needed.

Arthur D. Little has developed a five-level opinion scheme, as


described in the table on the following page.

When an audit programme design calls for an audit opinion as


part of the report design, it is important to be as explicit as
possible about the criteria for assigning each of the alternative
opinions. Inconsistency between audit teams in assigning
opinions could, for example, result in a facility’s appearing to
have “lost ground” in successive audits when the only real
difference was two different audit teams with two different
interpretations of what “requires improvement” means. There is
also an inherent concern of “grade inflation” over time. The
criteria for the Arthur D. Little five-level opinion scheme are
described in the table.

Formulating an audit opinion is frequently the responsibility of the


team leader and/or the audit programme manager, with input
from individual team members. An audit opinion would not be
given in instances where the team’s review was not sufficiently
rigorous to substantiate conclusions about the facility’s
compliance.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 270
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Post-Audit Activities –
Preparation of Audit Reports

Arthur D. Little’s Scheme

Opinion Criteria
On the basis of its review, the audit The facility is in compliance with all (or
team believes that the environmental, virtually all) of the applicable
health, and safety programmes and requirements included in the audit scope.
practices that were reviewed meet Isolated exceptions to a few requirements
governmental and internal are noted, but are judged to be
requirements. occasional, anomalous, and
inconsequential in comparison to the
level of compliance achieved.

On the basis of its review, the audit Audit results substantiate a high degree
team believes that the environmental, of compliance. Only a few requirements
health, and safety programmes and are not satisfied. These represent
practices that were reviewed isolated weaknesses in implementation of
substantially meet governmental and an otherwise effective compliance
internal requirements. programme.

On the basis of its review, the audit Several exceptions to applicable


team believes that the environmental, requirements are noted. These
health, and safety programmes and exceptions are more than anomalies and
practices that were reviewed reflect weaknesses in the design and/or
generally meet governmental and implementation of certain aspects of
internal requirements, except as compliance programmes.
noted below.

On the basis of its review, the audit Several exceptions to applicable


team believes that the environmental, requirements are noted. Some of the
health, and safety programmes and exceptions reflect the absence of one or
practices that were reviewed require two required programmes, significant
improvement to meet governmental departures from a few established
and internal requirements. criteria, or lapses in programme
implementation.

On the basis of its review, the audit Many exceptions to applicable


team believes that the environmental, requirements are noted. They included
health, and safety programmes and significant departures from various
practices that were reviewed require established criteria, the absence of
significant improvement to meet several required programmes, or
governmental and internal prolonged inattention to the resolution of
requirements. previously identified compliance or
liability issues.
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 271
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Post-Audit Activities –
Preparation of Audit Reports

Strengths

Oral acknowledgment of the facility’s strengths frequently occurs


during the team’s daily interactions with facility personnel and in
the exit meeting. Increasingly, however, companies are
recognising the value of giving credit to areas of real strength in
the written report as well by explicitly acknowledging those areas
in which the facility is performing particularly well, through design
and implementation of an effective management system, which
provides some context within which to evaluate the audit
exceptions and observations. This acknowledgment is most
commonly done in the executive summary section of the report.
For example:

In its review of the facility’s programmes and practices, the team noted the
following as areas of real strength:

• The hazardous waste management programme is clearly understood by


staff throughout the facility and is documented to demonstrate commitment
to compliance.
• The bloodborne pathogens programme is well developed, organised, and
tracked.

To avoid confusion, it is important that both the auditors and


facility personnel have a common understanding of what is
meant by a “strength.” A strength need not necessarily be a
unique, world class, or extraordinary practice. On the other
hand, the mere fact that a facility is in compliance with a
regulatory requirement is not an example of a strength.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 272
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Post-Audit Activities –
Preparation of Audit Reports

Executive Summary

The purpose of the executive summary is to provide a one- or


two-page section at the beginning of the report that conveys the
highlights of the audit to readers. An example of an executive
summary is illustrated below.

A health, safety and environmental (HSE) audit was conducted at ABC Company’s XYZ facility
on Month/Day/Year. The time period under review was January 1, 199X, through the last day
of the audit. The purpose of the audit was to verify compliance with applicable federal, state,
and local safety and health laws and regulations. The scope of the audit included air pollution
control, water pollution control, solid and hazardous waste management, industrial hygiene,
employee safety, and loss prevention. Specific deficiencies in the HSE programmes and
practices are described in detail in the body of this report.

On the basis of its review, the audit team believes that XYZ facility will need to develop and
implement more formal HSE management programmes to achieve the HSE goals recently
established by facility management. In the audit team’s opinion, the HSE programmes
currently in place are heavily dependent upon the capability and good intention of facility
personnel and are being implemented in the absence of clearly defined written procedures and
plans. In addition, the level of familiarity with regulatory obligations varied among the staff
responsible for HSE compliance activities.

The management of XYZ facility in Anytown, Anystate, has initiated a programme to ascertain
the status of HSE programmes on site, to develop appropriate corrective actions to remediate
problems, and to develop overall systems to ensure compliance with governmental rules and
regulations. In addition, the facility has undergone a recent management reorganisation
creating a more centralised HSE group. Thus, these activities represent an increased
commitment to manage compliance.

To develop more formal HSE programmes suitable to the operations of XYZ facility, we
recommend the following for consideration:

• Developing clearly stated and well-defined descriptions of roles and responsibilities for all
HSE activities, and communicating those to facility staff.

• Establishing well-developed plans and written procedures, as appropriate, for undertaking


compliance activities which are communicated and understood by all key HSE staff.

• Undertaking a facility programme to periodically review and monitor HSE compliance and
to identify problems.

• Developing an active, formal training and awareness programme for key staff in all areas
of HSE requirements.

Lastly, we believe the facility should continue in its efforts to identify and assess site
contamination and to develop appropriate remedial plans.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 273
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Post-Audit Activities –
Preparation of Audit Reports

Explanation of Requirements

In many instances, the readers of audit reports are not all equally
familiar with the regulatory or company policy requirements to
which the team has identified exceptions. In these situations, it
may be helpful for the audit report to include a brief description of
the regulation or company policy that is being cited as the basis
for the finding. For ease of reading, it is useful to identify the
requirement and the exception separately. For example:

Management of Satellite Accumulation Areas

Description Federal regulations allow for the accumulation of up to 55


of gallons of hazardous waste in containers at or near the point of
Requirement generation and under the control of the operator of the process
generating the waste (commonly referred to as a “satellite
accumulation area”), without a time limit, permit, or interim
status, contingent upon the generator complying with specific
container labelling and management conditions. [40 CFR
262.34(c) and 265.171-265.173]

Exception In inspecting a sample of areas where wastes are being


accumulated, the team noted the following:
• In the Rastex process, solvent-contaminated filters are
routinely changed and disposed of as hazardous waste.
The “satellite” accumulation area for filters taken from the
RASTEX 679 reactor is located across the manufacturing
building at the DACSTAB UVX production area. In the
team’s opinion, the DACSTAB UVX area is not “at or near”
the RASTEX reactor, nor is it “under the control of the
operator” of the RASTEX process.
• In several areas, the team observed containers that were
used to collect solvent-contaminated rags. The containers
were not marked with the words “Hazardous Waste” or with
other words to describe the container contents and were not
kept closed during times when waste was neither being
added nor removed.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 274
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Post-Audit Activities –
Audit Policy Statement

One of the most common concerns with audit reports and audit-
related documentation is the risk of disclosure in an enforcement
or litigation situation. Therefore, many organisations will
establish or utilise one or several mechanisms to protect their
audit results. Some of the various means that can be used to
protect audit results include:
• Physical limitations on distribution
• Attorney-client privilege
• Attorney work product
• Audit privilege

Physical Limitations on Distribution


This protection is not founded in law, but simply requires the
audit programme to exercise due care in the release and
handling of all information relating to the audit. There are no
specific requirements in utilising this protection other than
following common sense and whatever systems or procedures
are in place in individual organisations for maintaining
confidentiality.

Attorney-Client Privilege
Attorney-client privilege protects the confidential communications
between a “person” (who may, in fact, be a company) and his/her
attorney. Legal protection under this privilege requires that:

• The person asserting the privilege is a client of the attorney to


whom the information is entrusted.

• The communication is made to the attorney or someone


working for the attorney (e.g., an audit team).

• The attorney is engaged in preparing a factual investigation or


legal opinion for the client.

• The communication is kept confidential and the privilege is not


breached voluntarily or inadvertently waived, i.e., information
is not shared freely or recklessly.
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 275
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Post-Audit Activities –
Audit Policy Statement

In the context of an HSE audit, the corporation is the client and


the audit results (including the report, the working papers, etc.)
can be designated as the confidential communications between
the client (the facility/ company) and the attorney (via the audit
team, who is acting as the agent of the attorney).

Attorney Work Product


Another potential means of protection of audit results is the work
product doctrine. The work product rule protects
information/material prepared by an attorney in anticipation of
litigation. This privilege, however, is not absolute, and can be
vacated by a showing of “undue hardship and substantial need”
on the part of the party seeking to discover the privileged
information.

Privilege under the work product rule is seldom utilised to protect


audit results in the course of routine HSE audits. For one thing,
most audits are not typically conducted in anticipation of litigation,
but rather are used as a management tool to assure or measure
compliance. Furthermore, the privilege under the work product
rule rests with the attorney, unlike the attorney-client privilege,
which rests with the client. Therefore, in the context of HSE
audits, this type of privilege would typically require substantial
involvement of counsel in the audit process.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 276
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Post-Audit Activities -
Quality Assurance in the Audit Process

Many audit programmes have quality assurance mechanisms


such as:
• Team leader reviews
• Customer/facility feedback
• Periodic programme reviews

Team Leader Reviews

One important aspect of team leader reviews of auditor


performance is working paper review, which was discussed
earlier (see checklist in Tab 6). In addition, team leaders
frequently evaluate and provide feedback to individual auditors
regarding their technical and interpersonal skills as auditors. If a
person is less than effective in one area, he/she can receive
some training or other assistance in that area prior to or during
the next audit. Many companies have formalised this process
with questionnaires or forms for the team leader to complete. An
example of a feedback form, which is based on the ISO 14012
qualification criteria for auditors, is provided in the table on the
following page.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 277
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Post-Audit Activities -
Quality Assurance in the Audit Process

Auditor Name

Facility Audited

Audit Dates

Team Leader/Reviewer

Audit Element Score* Comments

1 2 3 4 5

Technical Knowledge
• HSE science and
technology
• Facility operations
• Regulatory requirements
• HSE management systems
• Audit procedures and
techniques
− Interviewing
− Using the protocol
− Keeping working papers
− Writing findings
Personal Attributes
• Clarity in oral
communication
• Foreign language capability
• Diplomacy, tact, and
listening skill
• Independence and
objectivity
• Personal organisation and
time management
• Ability to reach sound
judgments based on
objective evidence

*1 = poor; 3 = average; 5 = exceptional; N/A = not applicable

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 278
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Post-Audit Activities -
Quality Assurance in the Audit Process

Customer/Facility Feedback

Upon completion of the field activities, it is quite common for the


audit team to solicit feedback from the “auditees.” This may be
done informally, for example by means of a debriefing
conversation between the team leader and facility management
after the closing meeting, or more formally. Companies that
solicit such feedback have generally found it to be a valuable
mechanism for improving the overall effectiveness and
acceptance of the audit programme. Managers of audit
programmes have also found, however, that it is important to
keep the feedback form simple, so as not to overburden the
facility. The table on the following page illustrates the feedback
form utilised in one audit programme.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 279
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Post-Audit Activities -
Quality Assurance in the Audit Process

Auditee Feedback Form

Location

Date of Audit

Responder

Strongly Neither Strongly No


Agree Agree or Disagree Basis
Disagree

1. The audit objectives were clearly


communicated to me.

2. The audit took an acceptable amount of


time (from entrance to exit).

3. The disruption of daily activities was


minimised as much as possible during
the audit.

4. My business concerns and perspective


were adequately considered during the
audit.

5. Communication of audit results and


status to me during the audit was timely
and adequate.

6. The audit team demonstrated technical


proficiency in the audit areas.

7. The audit team demonstrated courtesy,


professionalism, and a constructive and
positive approach.

8. The audit team’s conclusions were


logical and well documented.

9. Audit results were accurately reported


and appropriate perspective was
provided.

10. The audit report was clearly written and


logically organised.

11. Overall the audit provided “value


added” to my organisation.

Comments:
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________
_____________________________________________________________________________

Source: AlliedSignal Inc.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 280
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Post-Audit Activities -
Quality Assurance in the Audit Process

Periodic Programme Reviews

Audit programmes should keep pace with changes in the


business and regulatory climate and cannot be static.
Periodically, it pays to step back and take a hard look at the
overall programme design and implementation. In these periodic
reviews, all elements of the design—objectives, scope, coverage,
organisation, resources, and approach—should be critically
examined to confirm whether or not they are (still) meeting the
needs of the programme’s stakeholders. It is also important to
verify that audit teams in the field are actually implementing the
programme as designed.

There are a variety of mechanisms for conducting these periodic


reviews. One possibility is to assemble a team or task force to
examine the entire audit process and look for improvement
opportunities. The team should probably include representatives
of the audited facilities and of upper management, as well as
some of the members of the audit programme itself. Another
approach that has been used successfully in some companies is
to ask the law department or the internal audit department to
review the programme. Yet another possibility is to engage an
independent third party from outside the company to confirm the
adequacy of programme design and implementation versus self-
established standards, industry and international standards, and
best practices.

Regardless of the composition of the reviewing body, a “high


performance business” approach may prove useful. In such an
approach, the review begins by identifying the key stakeholders
of the audit programme and determining/confirming their needs.
It is then possible to evaluate the various elements of the audit
process to see if it is structured to meet those needs. Once any
necessary improvements to the process have been identified, it is
possible to assess whether the underlying organisation and
resources are adequate for effective implementation.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 281
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Appendix A
Confirmation Letter

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 282
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Appendix A
Confirmation Letter
*[Date]

*[Name]
*[Title]
*[Company]
*[Address]
*[City, State]

Re: Environmental, Health, and Safety Compliance Audit

Dear __________:

Confirming our telephone conversation of *[Date], we will conduct an


environmental, health, and safety compliance audit of the *[Facility name] Plant the
week of *[Date].

The audit will address air pollution control, water pollution control, spill control and
emergency response planning, solid and hazardous waste management, underground
storage tanks, soil and groundwater contamination, drinking water management,
PCB management, employee safety, loss prevention, and industrial hygiene as well
as company policies, guidelines, etc. The audit team will arrive on site on *[Date].
They would like to meet with you and other appropriate personnel on Monday
morning to briefly describe the audit and to answer any questions. At that opening
conference, it would be helpful if the team could receive a brief description of the
current organisation and operations at the facility as well as an orientation tour.

Those facility personnel involved with environmental, health, and safety activities
will be needed on site for discussions during the audit week.

At the conclusion of the audit, the audit team will again meet with you and other
appropriate personnel to discuss the team’s findings. This closing conference is a
critical part of the audit, and it is very important that you be present.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 283
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Appendix A
Confirmation Letter
*[Date] Page 2

*[Name]
*[Title]
*[Company]

The audit team will make its own lodging and transportation arrangements,
including cars for use during the week. A dedicated meeting room (work area) is
needed at the facility with one large conference table or several smaller tables to
accommodate the team members and their working papers. Also, access to a
telephone and a copier would be helpful.

To assist us in our preparation, we would appreciate having the facility complete the
enclosed pre-audit questionnaire (Attachment 1), and provide a copy of as many as
possible of the applicable items listed in Attachment 2. The pre-audit documents
should be sent to *[The Team Leader] ’s attention by *[Date].

Please do not hesitate to call me if you have any questions about this audit or the
Environmental, Health, and Safety Audit Programme in general.

Thank you for your assistance in these matters.

Sincerely,

[Team Leader]

Attachments

cc:

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 284
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 285
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________
I General Information
1. Number of Employees on-site

2. Number of Contractors (Firms) on-


site

3. Number of Contractors’ employees


on-site (daily average)

4. Number of Employees in Company

5. Number of shifts worked


(please specify times)

6. Products

7. Annual production volume

8. Time company has operated at this


site

9. Nature of any previous industrial 1.


activities on this site
2.

3.

10. Name(s) of Doctors and 1.


Occupational Nurses on-site and %
time in attendance 2.

3.

4.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 286
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________
II Locality/Neighbourhood Description
1. Nature of facility location e.g.,
commercial, industrial, residential,
agricultural, rural

2. Are there any schools, hospitals,


nursing homes, prisons, churches or
other public buildings within two
kilometres of the facility?

3. Are there any nature reserves,


national parks or sites of specific
scientific interest within two
kilometres of the facility?

4. Distance to nearest industrial


neighbour from the fence line

5. Nature of neighbouring industrial


activities

6. Describe any major hazard


installations within two kilometres of
the facility and distance

7. Distance to nearest residential


property

8. Approximate size of the population


near this facility (see table below)

0-1 kilometre 0-100 100-1000 +1000


1-2 kilometres 0-100 100-5000 +5000
2-5 kilometres 0-100 100-10,000 +10,000

9. Number of complaints per year


received from the local community
related to HSE issues

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 287
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________

10. Nature of any local community


activities in which the site
participates

11. Distance to nearest surface water more than 1 kilometre


(please tick) 0.5-1 kilometre
adjacent to facility
boundary
within facility
12. Is the facility affected by any natural
hazards, e.g., earthquake, hurricane,
flooding?

13. List any off-site facilities


(warehouses, processing units)

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 288
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________
III Policy and Organisation

A Environmental Management System

1. Does the facility have any of its own


specific policies, procedures or
guidelines pertaining to:

(a) Health and Safety Yes No

(b) Environment Yes No

List site-specific written HSE policies and procedures below or attach an


index if preferred

2. Please provide an organisation chart


showing HSE management structure
and responsibilities

3. Who is responsible for identifying


training needs?

4. Who is responsible for developing


and implementing training
programmes?

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 289
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________
5. List site specific training programmes/topics below or attach an index if
preferred

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 290
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________
IV Functional Areas

A Occupational Environment

Who is responsible for occupational Name:


environment programmes at the Title:
facility?

1. Are risk assessments carried out to


evaluate:
(a) Internal air quality Yes No N/A
(b) Temperature Yes No N/A
(c) Lighting levels Yes No N/A
(d) Ergonomics Yes No N/A
(e) Manual handling Yes No N/A
(f) Exposure to hazardous Yes No N/A
substances

2. Is there a preventative maintenance


programme covering Heating, Yes No N/A
Ventilation and Air Conditioning
(HVAC) and lighting equipment?

3. Is there a formalised safety


inspection programme? Yes No N/A

(a) Who inspects?

(b) How often?

4. Does the facility have procedures to


report and respond to occupational Yes No N/A
health hazards?

5. How many reports/complaints have


been made in the past two years?

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 291
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________
B Technical Equipment and Machinery Protection

Who is responsible for technical Name:


equipment and machinery protection Title:
programmes at the facility?

1. Are risk assessments carried out to


evaluate:
(a) Moving parts Yes No N/A
(b) Cutting/blades Yes No N/A
(c) Rollers and pinch points Yes No N/A
(d) Excessive temperatures Yes No N/A
(e) High pressure Yes No N/A
(f) Other (please specify)

2. Is there a preventative maintenance


programme covering maintenance of Yes No N/A
equipment and machinery, and
associated protection?

3. Are equipment inspections


conducted to review the safe Yes No N/A
working condition(s) of the
equipment/machinery?

4. Are all lifting appliances examined


and certified? Yes No N/A

5. Are all pressure vessels tested and


certified? Yes No N/A

6. Does the facility have procedures for


employees to report faults and for Yes No N/A
corrective action?

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 292
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________
C Materials, Goods and Hazardous Substances

Who is responsible for materials,


goods and hazardous substances?

1. Do you maintain an inventory of


hazardous materials used and Yes No N/A
stored on site?

2. Have you conducted risk


assessments on Yes No N/A
use/storage/transport of hazardous
materials, goods and substances?

3. Have you a risk reduction


programme, which includes, for Yes No N/A
example, elimination, reduction of
use or personal protective
equipment?

4. Are Safety Material Data Sheets


giving physical, chemical and toxic Yes No N/A
properties, and environmental data
kept on site for each hazardous
material?

5. What are the separation distances of


hazardous material areas from the
boundary fence and other buildings?

6. Does the facility control the use of


hazardous materials by contractors? Yes No N/A

7. Has the facility undertaken any


renovation or demolition activities in Yes No N/A
the last 24 months that involved the
removal of asbestos?

8. Have you completed plans for the


phasing-out and replacement of all Yes No N/A
CFC’s?
NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 293
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire
__________________________________________________________________________________________________________
11. Please complete the following table or supply copy of internal list:

Main Hazardous Materials, Goods and Substances used and stored on-site
Type of material¹ Use² Approximate Maximum quantity Type of storage4 Storage Area Spill6 containment
quantity used per stored on site Description5 type and capacity
year³ (m3)
e.g. Solvent Plant degreasing 1000 kg 10 kg AST Covered concrete 15m³ with retention
pad basin

Notes:
1 Indicate major hazardous materials used in operations or activities, and include potentially hazardous waste materials
2 Describe use of hazardous material listed, for example, boiler/furnace fuel, degreasing/metal cleaning chemical, wastewater treatment
chemical, etc.
3 Indicate use per year in kilograms or litres
4 Please describe the type of container(s) the material is stored in using one of the following symbols :
AST: Aboveground non-buried storage tank D200: 200 Litre drums
UST: Buried underground storage tank D25: Approx. 25 litre or smaller
O: Other (please specify) e.g., IBC/containers/piles
5 Briefly describe storage area (for example, inside, outside, covered, fenced, locked, restricted access, fireproofed etc.)
6 Indicate type of spill containment provided, if any (for example, retention basin, collection sump, oil/water separator, paved/sealed area,
concrete pad, curbing, bunded area, etc,) and indicate by 'yes' or 'no' if the containment will retain firewater in the event of an emergency.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 294
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________
D New or Modified Processes

Who is responsible for new or Name:


modified processes at the facility? Title:

1. Does the facility have HSE


procedures for change Yes No N/A
management?

2. Have any major new processes


been introduced/installed over the Yes No N/A
past two years? e.g. new production
lines, closing of production lines

If so, what were they?

3. Have any major modifications been


made to processes over the past two Yes No N/A
years?
e.g. machinery replacements

If so, what were they?

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 295
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________
E Hazardous Work

Who is responsible for hazardous Name:


work programmes at the facility? Title:

1. Has the facility conducted formal risk


assessments of hazardous work
activities? Yes No N/A
(a) Hot work
Yes No N/A
(b) Confined spaces
Yes No N/A
(c) Energy isolation work
Yes No N/A
(d) Working at height
(e) Other (please specify)

2. Has the facility established


documented procedures for
hazardous activities, e.g. Yes No N/A
(a) Permit-to-Work/Approval
Yes No N/A
procedures
Yes No N/A
(b) Pre-use/activity inspections
Yes No N/A
(c) Issue of personal locks
Yes No N/A
(d) ‘Buddy’ system ( for lone working)
Yes No N/A
(e) Engineering controls, e.g guarding

(f) Provision and use of personal


protective equipment

(g) Other (please specify)

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 296
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________

3. Are Work Permits/Approvals issued


for: Yes No N/A
(a) Hot work
Yes No N/A
(b) Confined spaces/Entry work
Yes No N/A
(c) Electrical work
Yes No N/A
(d) Working at heights
Yes No N/A
(e) Contractor Control
(f) Other (please specify)

4. Is specific training provided or


qualifications required for employees
involved in hazardous activities? Yes No N/A
(Please ensure training topics are
included in your answer to section
III, question 5.)

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 297
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________
F Electrical Safety

Who is responsible for electrical Name:


safety programmes at the facility? Title:

1. Has the facility conducted risk


assessments for work on electrically Yes No N/A
energised equipment and electric
circuits?

2. What preventative and protective


safety measures are in place, e.g.
earthing, fuses and circuit breakers?

3. Is a lock-out/tag-out programme in
place at the facility for electrical and Yes No N/A
other energy isolation?

4. Are regular inspections conducted to


review the safe working condition(s) Yes No N/A
of the electrical equipment?

5. Is specific training provided


to/qualifications required for
employees involved in electrical Yes No N/A
work?
(Please ensure training topics are
included in your answer to section
III, question 5.)

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 298
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________
G Work at Height

Who is responsible for working at Name:


height programmes at the facility? Title:

1. Does the facility have personnel or


contractors working at heights?

(a) Routinely, as part of their work Yes No N/A


activities, e.g. Facilities engineer

(b) Non-routinely Yes No N/A

2. Has the facility conducted risk


assessments on working at height Yes No N/A
activities?

3. What controls are in place to reduce


risks, e.g.
(a) Permits-to-Work Yes No N/A
(b) Harnesses Yes No N/A
(c) Personal protective equipment Yes No N/A
(d) Buddy system Yes No N/A
(e) Others (Please specify)

4. Does the facility have an inspection


and maintenance programme for Yes No N/A
ladders and scaffolding?

Does this include pre-use Yes No N/A


inspections, tagging and approvals?

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 299
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________
H Noise Control

Who is responsible for noise control Name:


programmes at the facility? Title:

1. What are the main sources of noise


at the facility?

Source Internal External

e.g. compressors Yes Can be heard


outside factory
building
e.g. transport movements No Yes

2. Are there any regulatory limits for


noise at the facility?

(a) environmental Yes No N/A

(b) occupational Yes No N/A

3. In the last three years, how many None


times
has the facility received complaints Less than 5
from
different neighbours relating to 5 - 10
noise?
(please tick) More than 10

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 300
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________

4. Does the facility operate its own


freight transport? Yes No N/A

5. Are there transport movements


before 06.00 and after 22.00 hours? Yes No N/A

6. Are there any loading or unloading


operations before 06.00 and after Yes No N/A
22.00 hours?

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 301
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________
I Personal Protective Equipment (PPE)

Who is responsible for PPE Name:


programmes at the facility? Title:

1. For which activities is PPE required?

2. What types of PPE are provided?

3. Does the facility have an approved


list of PPE and/or an approved list of Yes No N/A
suppliers?

4. Which PPE does the inspection and


maintenance activities cover?

5. Is there a formalised
inspection/maintenance Yes No N/A
programme?

(a) Who inspects?

(b) How often?

6. Is training provided to employees


involved requiring PPE? Yes No N/A
(Please ensure training topics are
included in your answer to section
III, question 5.)

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 302
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________
J Vehicles and Driving

Who is responsible for vehicles and Name:


driving programmes at the facility? Title:

1. Does the facility use (please indicate


approximate numbers): Owned? Leased?
Contractors’?
(a) Forklift trucks
(b) Industrial plant (e.g. diggers,
cranes)
(c) Heavy Goods Vehicles (HGV)
(d) Company cars
(e) Other (please specify)

2. Are there procedures for meeting


designated vehicle specifications? Yes No N/A

3. Is there a formalised
inspection/maintenance Yes No N/A
programme?

(a) Who inspects?

(b) How often?

4. Is there a pre-use inspection


procedure for vehicles (other than Yes No N/A
company cars)?

5. Is training provided to employees


driving company vehicles? Yes No N/A
(Please ensure training topics are
included in your answer to section,
III, question 5.)

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 303
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________
K First Aid, Medical Examinations and Care

Who is responsible for first aid, Name:


medical examinations and care Title:
programmes at the facility?

1. Does the facility have an equipped


medical centre/clinic? Yes No N/A

If yes, is this staffed:

(a) Doctor 24 hrs Full-time Part-time


(b) Nurse 24 hrs Full-time Part-time
(c) First Aiders 24 hrs Full-time Part-time

2. How many formally qualified First


Aid providers does the facility have?

3. Where does the facility provide first


aid equipment?

(a) Medical Centre Yes No N/A


(b) Designated locations Yes No N/A
(c) Per First Aider Yes No N/A
(d) Security/Gate keepers Yes No N/A
(e) Other (please specify)

4. Who inspects and maintains first aid


equipment regularly and how often?

Please attach a summary of accidents/incidents over the past 12 months (if


easily available).

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 304
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________

5. Are medical examinations provided


by the facility?

(a) Pre-employment Yes No N/A


(b) Annual check-ups Yes No N/A
(c) Critical situations, e.g.
– exposure to hazardous Yes No N/A
substances Yes No N/A
– pregnancy Yes No N/A
– return to work after Yes No N/A
injury/illness Yes No N/A
– persons under 18/over 60
– terminal diseases
– other (please specify)

6. Which general welfare and wellness


programmes are provided for
employees?

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 305
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________
L Fire Protection and Control

Who is responsible for fire protection Name:


and control programmes at the Title:
facility?

1. Is this facility regulated in relation to


flammable/explosive substances Yes No N/A
and/or for fire protection?

If so, what
certificates/permits/licences does the
facility hold?

2. Does the facility have any areas


designated as explosion hazard Yes No N/A
zones?

3. Has the facility conducted a risk


assessment of ignition, fire and Yes No N/A
explosion sources? e.g. through
insurance inspections

4. Has the facility installed fire


protection equipment:

(a) Fire detection (heat or smoke)? Yes No N/A


(b) Alarms? Yes No N/A
(c) Sprinklers or other dousing Yes No N/A
systems?
(d) Other (please specify)?

5. Does the facility have an emergency


and evacuation plan? Yes No N/A

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 306
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________

6. Does the facility provide instruction,


training, and testing of emergency Yes No N/A
plans?
Please specify

7. When was the last drill carried out? Date:

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 307
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________
L Fire Protection and Control (continued)

8. Does the facility have a trained


emergency response team or Yes No N/A
nominated fire marshals?

Please specify

9. Has the facility experienced any


explosions or fires over the past:

(a) ten years Yes No N/A


(b) five years Yes No N/A
(c) two years Yes No N/A

10. Has the facility experienced any


near-misses involving explosions or
fires over the past:

(a) ten years Yes No N/A


(b) five years Yes No N/A
(c) two years Yes No N/A

11. Of these, how many have been


reportable to a government agency

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 308
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________
M Air Pollution Control

Who is responsible for air pollution Name:


control programmes at the facility? Title:

1. Has the facility identified and


documented all sources of air Yes No N/A
emissions?

2. Has the facility identified the nature


and quantities of pollutants emitted to Yes No N/A
atmosphere?

3. Which air emissions from the facility


are regulated by authorities?

4. Which air pollutants emitted from the


facility are required to be monitored
by a government agency? If none,
state ‘none’

5. Which types of air pollution control


equipment are installed, e.g.
scrubbers, dust filters?

6. Does the plant operate planned


maintenance procedures for its air Yes No N/A
pollution control equipment?

7. State average number of public


complaints for last three years
attributed to air emissions from the
facility, e.g. odour, dust, VOCs,
smoke

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 309
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________

8. How many cooling towers or static


water tanks are located in the
facility?

9. Are 6-monthly chlorinations carried


out on these to prevent growth of Yes No N/A
legionella pneumophilia?

10. Does the facility dispense fuel to


motor vehicles? Yes No N/A

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 310
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________
N Water Pollution Prevention

Who is responsible for water Name:


pollution prevention programmes at Title:
the facility?

1. Does the facility have any targets or


programmes for reducing water Yes No N/A
consumption?

Please specify

2. Are waste water discharges


regulated by a permit or consent? Yes No N/A

3. Does the facility conduct any effluent


monitoring? Yes No N/A

4. Does the facility make use of an on-


site wastewater treatment system Yes No N/A
prior to effluent discharge?

5. If the facility has own wastewater


treatment plant, how is the sludge
disposed of

6. Is any process wastewater recycled? Yes No N/A

7. Does any portion of the facility’s


drinking water supply come from on- Yes No N/A
site wells or surface water sources?

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 311
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________
O Waste Management

Who is responsible for waste Name:


management programmes at the Title:
facility?

1. Does the facility generate wastes


that are defined as ‘hazardous’ or Yes No N/A
‘special’ under government
regulations? Please provide details
in the Table below (question 6).

2. Does the facility require/have a


permit or license for its waste Yes No N/A
activities?

3. Does waste treatment and/or


disposal take place/ever taken place Yes No N/A
on-site?

Please specify

4. Are any waste materials separated


and sent for recycling? Please Yes No N/A
provide details in the Table below
(question 6).

5. Does the facility monitor off-site


disposal activities? Yes No N/A

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 312
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire
__________________________________________________________________________________________________________
6. Please complete the following table.

Solid and Hazardous Waste Generation, Treatment and Disposal


Description of Process1 Classification Estimate of Annual Method of Waste minimisation
Waste e.g. hazardous, Quantity Generated (tons Treatment and target and date (if
Generation liquid, inert or Kg/year) (if available) final Disposal2 any)
e.g. cardboard raw materials/ inert 6 tons Re/L 20% reduction by 1999
packaging supplies

Notes :
1 Indicate which type of equipment or operation generates this waste stream.
2 Please enter one of the following letters as appropriate. If the disposal or treatment is on-site, please circle the letter.
Re = Recycled externally S = Sold for further use (please specify) T = chemical or physical I = Incinerated
Treatment
Ri = Recycled Internally L = Landfill O = Other (please
specify

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 313
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________
P Soil and Groundwater Protection

Who is responsible for soil and Name:


groundwater protection programmes Title:
at the facility?

1. Have you conducted a survey to


identify actual areas of soil and
groundwater contamination resulting Yes No N/A
from previous industrial practices or
activities?

2. Has there been any remediation of


land which is contaminated? Yes No N/A

3. Does the facility have any


underground (buried) tanks in or out Yes No N/A
of service with associated piping?
(See also 7)

4. Does the site have procedures to


protect soil and groundwater (e.g. Yes No N/A
spill prevention and containment
programmes/facilities) and a written
spill reporting procedure?

5. Does the stormwater drainage


system include any interceptors? Yes No N/A

6. Could excessive stormwater cause


secondary containment, interceptors Yes No N/A
etc. to flood and overflow?

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 314
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________
7. Please complete the following table.

Bulk Storage Tanks


Tank Above In/Out of Material Leak Tank age Leak
Volume ground service stored (or detection (years) test
AST or previously and/or
Buried stored) release
UST prevention
e.g 50m³ UST In fuel oil none 35 none

Notes:
1 Indicate volume of tank in litres (I) or cubic metres (m³)
2 Indicate whether tank is above ground - use letters AST, or below ground - use
letters UST
3 Indicate whether the tank is equipped with leak detection measures, overfill
protection, or corrosion protection
4 Indicate date of most recent leak test (including hydrostatic pressure testing,
sampling and analysis of surrounding soil, etc.). If not tested, indicate ‘None’.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 315
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________
Q Product Stewardship

Who is responsible for product Name:


stewardship programmes at the Title:
facility?

1. Has the facility determined the


environmental impacts of all its
product and packaging materials,
including transport packaging Yes No N/A
materials?

2. Are environmental impacts formally


considered during development and
marketing of products? Yes No N/A

3. Do raw material specifications


routinely include HSE requirements? Yes No N/A

4. Do manufacturing machinery
specifications routinely include HSE
requirements? Yes No N/A

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 316
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 1
Example of Pre-Audit Questionnaire

__________________________________________________________________
R Energy Conservation

Who is responsible for energy Name:


conservation programmes at the Title:
facility?

1. Has the facility carried out a


systematic review of all its uses of Yes No N/A
energy, to identify major energy
consuming equipment or activities?

2. Have energy conservation objectives


and targets been set? Yes No N/A

Please specify

3. Has the facility defined Energy


Accountable Centres (i.e., discrete Yes No N/A
operating units within the site for
which energy consumption figures
are available)?

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 317
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Letter Attachment 2
Pre-Audit Information Request

• Facility plot plan or map

• Directions to the facility

• Visitor safety requirements (e.g., personal protective


equipment, orientation, specialised training, special
clearances, etc.)

• Completed pre-audit questionnaire

• Description of the facility’s operations/processes

• Facility organisation chart, showing HSE responsibilities

• Local laws, regulations, and ordinances related to the scope


of the audit

• List of current environmental licenses, certificates, and


authorisations

• Copies of permits for wastewater discharges and example air


emission permits

• Recent regulatory agency inspection/enforcement


correspondence

• Recent internal and intra-company environmental, health, and


safety audit reports

• Table of contents for facility-specific environmental, health,


and safety policies and procedures

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 318
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Appendix B
Roles and Responsibilities of the Audit Team

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 319
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Appendix B
Roles and Responsibilities of the Audit Team -
Audit Team Leader Responsibilities

Pre-Audit Activities

• Select team members and assign audit responsibilities.

• Gather and distribute background information.

• Identify applicable federal/national, state/provincial, and local


regulations and company policies and procedures.

• Conduct advance visit to the facility (if necessary).

• Review and revise audit strategy and assigned duties as


necessary.

• Determine and confirm arrangements with the team members


and the facility:
− Travel arrangements
− Hotel/travel reservations

• Prepare items for audit (forms, supplies, protocols).

• Coordinate pre-audit team meeting(s).

On-Site Activities

• Lead opening meeting presentation.

• Serve as liaison between team and facility personnel to


ensure that all team members are appropriately scheduled to
meet with facility personnel.

• Solicit feedback from each team member on the status of


work accomplished throughout the audit.

• Perform audit duties as determined by the audit strategy.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 320
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Appendix B
Roles and Responsibilities of the Audit Team -
Audit Team Leader Responsibilities

• Review assigned protocol steps with each auditor to ensure


that all steps are covered appropriately.

• Document the rationale for changing the scope of the audit (if
necessary).

• Understand the context for and meaning of each finding


reported by the team.

• Provide periodic feedback to facility personnel on the status of


the audit and the findings of the team.

• Prepare the exit meeting discussion sheets listing all findings


as summarised by the team.

• Ensure that all exit meeting discussion sheets are reviewed by


each team member.

• Review all findings with key facility personnel prior to the exit
meeting to ensure the accuracy of all findings.

• Lead presentation of exit meeting discussion.

• Summarise reporting schedule and format.

Post-Audit Activities

• Review all working papers to ensure that all topics were


covered and that all findings are corroborated by working
paper notes.

• Prepare draft report.

• Distribute draft report for comments.

• Incorporate comments where appropriate into the final report.


NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 321
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Appendix B
Roles and Responsibilities of the Audit Team -
Audit Team Member Responsibilities

Pre-Audit Activities

• Make travel arrangements (if required).

• Attend pre-audit team meeting (if required).

• Prepare for the audit by reviewing appropriate


federal/national, state/provincial, and local regulations,
company policies and procedures, and available background
information.

• Modify or annotate the protocol to reflect facility-specific


requirements, state and local regulations, and information
gained during review of background information.

On-Site Activities

• Perform duties assigned by the team leader during the audit.

• Serve as a resource for other audit team members during the


audit.

• Report on progress to the team leader throughout the audit,


including any problems encountered.

• Share observations/concerns with other team members during


the audit to ensure that each is addressed appropriately.

• Keep facility personnel apprised of findings as they are noted.

• Summarise all findings and report them to the team leader


before the close-out meeting.

• Assist with preparing the exit meeting discussion sheets.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 322
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Appendix B
Roles and Responsibilities of the Audit Team -
Audit Team Member Responsibilities

• Ensure that all findings noted in your working papers are


presented on the exit meeting discussion sheets and
accurately reflect the facts as you understand them.

• Contribute during the exit meeting when questions are raised


about your findings.

Post-Audit Activities

• Review draft audit reports for:


− Wording changes
− Suggested changes in placement of findings within the
report

• Provide input as necessary when findings in the draft report


are challenged.

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 323
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Appendix C
Guide to Some Health, Safety and Environmental
Acronyms

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 324
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.
Appendix C
Guide to Some Health, Safety and Environmental
Acronyms
AAQS Ambient Air Quality Standards NSR New Source Review
ACGIH American Conference of Governmental Industrial OCAW Oil, Chemical, and Atomic Workers
Hygienists
AHERA Asbestos Hazard Emergency Response Act of 1986 OEL Occupational Exposure Limit
(Title II of TSCA)
ANSI American National Standards Institute OSHA Occupational Safety and Health Administration
ARAR Applicable or Relevant and Appropriate OTA Office of Technology Assessment
Requirements
AQCR Air Quality Control Region PEL Permissible Exposure Limit (for workplace)
ASHRAE American Society of Heating, Refrigeration, and Air PCBs Polychlorinated Biphenyls
Conditioning Engineers
BACT Best Available Control Technology pH Measure of acidity or alkalinity
BOD Biochemical Oxygen Demand PHA Process Hazard Analysis
BPCT Best Practicable Control Technology P&ID Piping and Instrumentation Design
BPT Best Practical Treatment PM Particulate Matter (in air)
BTU British Thermal Unit PM10 PM with <10 micron diameter; respirable
CAA Clean Air Act PMN Premanufacture Notification (under TSCA)
CAAA Clean Air Act Amendments POTW Publicly Owned Treatment Works
CAS Chemical Abstracts Service Registration Number PRPs Potentially Responsible Parties (under CERCLA
CAG EPA’s Carcinogen Assessment Group PSD Prevention of Significant Deterioration
CAIR Comprehensive Assessment Information Rule (under PSM Process Safety Management
TSCA)
CERCLA Comprehensive Environmental Response, RACT Reasonably Available Control Technology
Compensation, and Liability Act (The Superfund Law)
CFCs Chlorofluorocarbons RCRA Resource Conservation and Recovery Act
CFR Code of Federal Regulations RQ Reportable Quantity (for spill reporting)
CMA Chemical Manufacturers Association RTECS Registry of Toxic Effects of Chemical Substances
CO Carbon Monoxide SARA Superfund Amendments and Reauthorisation Act of
1987
COD Chemical Oxygen Demand SDWA Safe Drinking Water Act
CPSC Consumer Product Safety Commission SERC State Emergency Response Commission
CWA Clean Water Act SIC Standard Industrial Classification
DMR Discharge Monitoring Report SIP State Implementation Plan
DOT Department of Transportation SNUR Significant New Use Rule
EPA Environmental Protection Agency SPCC Plan Spill Prevention Control and Countermeasures Plan
EPCRA Emergency Planning and Community Right-to-Know SQG Small Quantity Generator (of hazardous waste)
Act (Title III of SARA, commonly called Right-to-
Know or SARA Title III)
FIFRA Federal Insecticide, Fungicide, and Rodenticide Act STS Standard Threshold Shift
GEP Good Engineering Practice SWDA Solid Waste Disposal Act
HAZOP Hazard and Operability Study TCLP Toxicity Characteristic Leaching Procedure (under
RCRA)
Hazwoper Hazardous Waste Operations and Emergency Title III Emergency Planning and Community Right-to-Know
Response Act
HCS OSHA Hazard Communication Standard (Worker TLV Threshold Limit Values (for workplace exposure)
Right-to-Know)
HMTA Hazardous Materials Transportation Act TPQ Threshold Planning Quantity (for emergency planning
HSWA Hazardous and Solid Waste Amendments (1984 TSCA Toxic Substances Control Act
RCRA Amendments)
IARC Internal Agency for Research on Cancer TSDF Treatment, Storage, and Disposal Facility
IDLH Immediate Danger of Life and Heath TSP Total Suspended Particulate (in air)
IH Industrial Hygienist TSS Total Suspended Solids (in water)
LAER Lowest Achievable Emission Rate UST Underground Storage Tank
LEPC Local Emergency Planning Committee VHAP Volatile Hazardous Air Pollutant
LOTO Lockout/Tagout (Control of Hazardous Energy) VOC Volatile Organic Compound
MACT Maximum Available Control Technology WWTP Wastewater Treatment Plant
MSHA Mine Safety and Health Administration Z List OSHA list of hazardous chemicals (29 CFR 1910
Subpart Z, Worker Right-to-Know)
MSDS Material Safety Data Sheet
NAAQS National Ambient Air Quality Standards
NEPA National Environmental Policy Act
NESHAP National Emission Standards for Hazardous Air
Pollutants
NIOSH National Institute for Occupational Safety and Health
NOAA National Oceanic and Atmospheric Administration
NOx A mixture of nitrous oxide and nitrogen dioxide
NRC National Response Center
NPDES National Pollutant Discharge Elimination System
NSPS New Source Performance Standards

NPC/20365/Mod6_158Audit_Skills_Training_Handbook.doc 325
Copyright © 2005 by Arthur D. Little Ltd. All rights reserved.

You might also like