Auditing-Techniques-Audit Performance 2022

Auditing Techniques –
Performing of Audits
1 © 2022 Copyright Fly 88, s.r.o.
Learning objectives
At the end of this module you should be
able to understand:
1. Principles of Auditing technique
2. Preparation for the audit / scope, methodology, goals
using Performance Based Oversight principles
3. Definition of applicable legislation
4. Checklist preparation
5. Inclusion of risk assessment
6. Evaluation of service provider’s documentation and
records
Learning objectives
At the end of this module you should be
able to understand:
7. Opening meeting agenda
8. Assessment of conformity
9. Non-conformities & Closing Meeting
10. Collection / security of data
1
Principles of Audit
Preparation
Modular training
 Three modules prescribed for the auditors training;
1. Auditing Technique – Initial, with following agenda:
1. Refences to the legislation used during the audit based on audit
scope
2. Preparation of checklists
3. Different methods monitoring of compliance with limitations of the
service provider
4. Working with service providers’ documentation.
2. Next one is this module ‘Performing of audits’
Modular training
 Three modules prescribed for the auditors training;
3. Last one is this module ‘Auditing Techniques – Audit follow-ups’,
with the agenda focused on;
1.Overview of audit preparation activities
2.Review of the audit checklists and data collected
3.Evaluation of the corrective action plan
4.Verification of effectiveness of the corrective actions
5.Performance evaluation and risk assessment as part of the corrective
actions
6.After-audit activities
7.Review of the audit plan based on data collected
2
Target Audience
Audit Preparation
First module is dedicated to everyone including those having no auditor training before.
Audit Performance
Second module is for those who passed the first module or any formal auditor training.
Audit follow-up
Third module is for those who are or will be appointed as lead auditors or team leaders of
inspection team.
Principles of Audit Preparation

1. Preparation of the audit program taking into account;
1. Specific nature of the organisation 
2. Complexity of its activities 
3. Results of past monitoring and oversight activities 
2. Audit program shall be based on the assessment of
associated risks - Operational risk:
Safety risk connected with the performance of operations 
Auditing cycle for each scope or organisation shall take into
account:
• the risk profile and
• the assessment of the safety performance. When the risk profile relies
on expert judgment, the decision making should be made by
8 consensus by Fly
© 2022 Copyright a 88,
teams.r.o. of experts.

Auditing cycle for each scope or organisation shall take into
account:
 the risk profile and
 the assessment of the safety performance.
Note: When the risk profile relies on expert judgment, the
decision making should be made by consensus by a team
of subject matter experts (SMEs).
Audit program from the very beginning should not prescribe
same intervals and workload for different audits.
3
10
11
Audit Objective, Scope and Criteria
 Audit Objective:
• The audit objective really means the purpose or the goal for this
particular audit.
• What are we trying to achieve by doing this audit? What is the aim?
What are we doing it for?
• Options:
a) Certification audit (Authority, ISO, IOSA, ISAGO, code-share
etc.)
b) Oversight audit (Internal, External follow-up etc.)
12
4
 Certification Audit:
• Different names can be used: Certification, Compliance, Initial,
Approval, Conformance etc.
• Performed once, normally as a precondition to obtain privileges
• Compliance means more formal and a legal requirement (e.g.
obtaining AOC, approval of a manufacturer or maintenance
organization etc.). Compliance refers to the government or
international standards (FARs, EASA regulations etc.)
• Conformance is voluntary. Conformance refers to the norms,
expectations, standards, and policies that an organization abides by.
These may be established by the organization itself or by another
organization (e.g. airline alliance, IOSA, ISO)
13
 Certification Audit:
• Normally performed by the Authority, Registrar or other independent
organization.
• May be performed by organization itself to achieve internal
approval of:
–External service providers (e.g. ground services provider for an
airline)
–Vendors (e.g. suppliers of spare parts for the maintenance
organization)
–Facilities (e.g. for performing of ditching training)
14
 Oversight Audit:
• Different names can be used: Recertification, Oversight,
Surveillance, Supervision, Monitoring, Recurrent, Follow-up etc.
• Performed for those processes/organizations which passed the
certification and are currently performing activities for which they are
certified/approved
• May be either scheduled or unscheduled
• Verifying whether the certified activity keeps all required parameters
and maintain the control over operations
15
5
 Oversight Audit:
• Follow-up audit:
–Specific type of the oversight audit
–This audit is required as part of the corrective action verification
process
–In most cases, follow-up audit scope is more narror than primary
audit
16

Develop Audit Design
Perform Audit, Report Findings & Risks
Schedule Checklist
GACA GM
Validate
Request Audit of proposed audit
Service Provider Validate findings
logistics
and riska
Identify Audit
Category,Type Assign Audit
&Scope,Schedul Team Leader Communicate
e (Date & Audit Report and
Lead Auditor
Feequency) CAP
Review the
hazards database
and previous audit
reports. Prepare/ Compose Audit
Refine Audit
adjust the report and
Schedule, Arrange
checklist prepare CAP
logistics, assign
audit team
Submit final Identify findings Categorize

checklist & risk findings and
Auditor
assessment evaluate risks
Perform Audit
Agree Audit
Audited Service
Timing and
Confirm Receipt
ensure
Provider
of Audit Report
availability of
required records
17
What should you remember?

• Inspections and audits are one the the highest levels of
oversight – but not the highest ones
• Audit preparation starts from the preparation of the long term
audit programme containing the schedule of audits
• Schedule is not the forever fixed document but rather a
‘skeleton’ on which you can continuously adjust the oversight
activities
• Preparation of the audit schedule (and later on, also the
checklists etc.) requires performance-based principles to be
followed
18
6
Applied exercise
• Explain which criteria and why you will use for the
planning of audit of organization processes:
• In case of airlines
• In case of maintenance organization
• In case of training organization
• In case of airport – ground services
• Work in groups
• Select a group representative to present the conclusion
Time: 15 m
19
Definition of applicable
legislation
20
Applicable Legislation
Define:
1. Legislation based on which audits are performed. It can
be:
a) NAA regulations
b) Legally binding agreement (i.e. IOSA, ISAGO, IS-BAO, Alliance
application)
c) Decision of appropriate authority (NAA, ICAO, enforcement agency
etc.)
d) Internal standards of the audited organization
21
7
Define:
2. Legislation to which audit will refer. It can be:
a) NAA regulations
b) Agreed standards used (IOSA Standards Manual, ISO 9001 etc.)
c) Documented requirements of the customer
d) Internal standards of the audited organization
e) Such legislation should be available within the Q-Pulse software
22
In some cases, the used legislation contains templates for the
audits;
• FAA FSIMS checklists
• IOSA checklists
• Checklist - Safety Management System Manual Structure Evaluation
ICAO Doc. 9859 etc.
In such case auditor should use these checklists either:
• As is (in such case results are comparable with other users)
• Adjust based on PBO principles (it means with changes based on
auditee pre-assessment)
• Software Q-Pulse may offer modules containing such checklists
23
Very often, audits are performed refering to more that one
legislation source:
24
8
During the audit, you should find cross-references between the
applicable regulations and the documentation used by the
audited organization:
25
1. Referring to documentation should be synchronized
between different auditors if an audit team is used:
2. In case, if reference is in electronic format only, refer to

“Documentation type” field…
3. Important: Referred documentation should always directly
refer to the legislation requirement
26
Applicable Legislation - Terms

Be aware of terms used!
1. System: A combination of interacting or interrelated
elements within an organization functioning in a coordinated
manner to achieve desired outcomes.
Examples:
• ICAO PQ 2.003: How is the civil aviation safety oversight system
organized in the State?
• IOSA ORG 1.3.1: The Operator shall ensure the management system
defines the safety accountability, authorities and responsibilities of
management and non-management personnel throughout the organization
27
9
2. Program: An organized set of processes directed toward a
common purpose, goal or objective.
Examples:
• ICAO PQ 3.113: Is there a formal training programme detailing what type
of training should be provided to personnel licensing officers?
• IOSA ORG 1.6.5: The Operator shall have a program that ensures its
personnel are trained to understand SMS responsibilities and competent to
perform associated duties
28

3. Policy: A policy sets out the strategic direction of the
organisation as decided by senior management.
Examples:
• ICAO PQ 1.055: Has the State established a policy and procedures for
enforcement?
• IOSA ORG 3.1.8: The Operator should have a policy and procedures for
the transport of items in the cargo compartment
29

4. Process: One or more actions or procedures implemented
in a coordinated manner to achieve a goal, a defined result or
to satisfy a requirement.
Examples:
• ICAO PQ 3.805 Has the State established a process for the review and
validation of application forms for aircraft maintenance personnel for the
issuance, renewal, validation and extension of licences and ratings?
• IOSA ORG 1.8.1: The Operator shall ensure the management system
includes planning processes for operations
30
10
5. Procedure: An organized series of actions accomplished in
a prescribed or step-by-step manner to achieve a defined
result.
Examples:
• ICAO PQ 1.009: Has the State established procedures for the amendment
of its specific regulations
• IOSA ORG 4.1.17: The Operator should have procedures under the
corporate ERP that ensure a central coordination and control of all
communications with external entities:
31

6. Action / task: An activity accomplished when following a
procedure.
Examples:
• ICAO PQ 1.019: If the State has delegated or transferred safety oversight
tasks to a regional organization or an independent non-governmental
organization, is there a legal basis for this delegation/transfer?
• IOSA MNT 4.5.1: : The Operator shall have a process to ensure each
maintenance organization that performs maintenance for the Operator has
a training program that requires all maintenance personnel to receive initial
and recurrent training that is appropriate to individually assigned tasks and
responsibilities, and provides maintenance personnel
32

Be aware of terms used! Task
Procedure
Process
Program
System
33
11
• There is a hierarchy of information used – system 
program process procedure. Policy shows intention to
achieve some goal and should be promoted by top
management
• When thinking of audit as a mean for compliance monitoring,
it must be clear:
• Based on which fact you are allowed to do so
• Based on which standards you will be looking for a compliance
34
Applied exercise
• Explain which type of information should be used to
describe:
• Signing of an accomplished AD
• Collection and work with the flight data
• Intention to reach lowering level of emissions
• How to get an AOC
• Work individually
• Defend your position
Time: 10 m
35
Checklist preparation
36
12
Checklist Preparation
Principles are described in the first module of the training:
1. Proper type of a checklist should be chosen based on audit
objective
2. Checklist should clearly specify where the requirements are
taken from
3. Checklist should be organized, and checklist questions
should be grouped:
• Based on the topic (one person can answer the group)
• Based on location
• Based on complexity and cross-references etc.
37
Different types of audits and different type of questions may be
used (see Module 1).
However, now there are available different tools for the audit
checklist preparation (Qpulse, IMX, Q5 etc.).
Such tools ease audit checklist preparation, however, following
is required:
• To be prepared by scope qualified author or with help of SME
• To be trained to use the applicable software
• To be reviewed and coordinated by the Lead auditor
• Such software should be accepted by NAA
38
Questions used during the audit:
1. Should have its rationale clear both to auditor and audited
subject (to avoid “Why are you asking this?”)
2. Should be reviewed before and after the audit to evaluate
their applicability and effectiveness (do not repeat the
questions which are not applicable for the audited
organization)
3. Whenever possible, should be taken from the question
database (avoid “ad-hoc questions coming to your mond
during the audit)
39
13
4. There should be clear and unambiguous options to reply:
a) Yes or No
b) Satisfactory or unsatisfactory
c) Documented or not documented
d) Implemented or not implemented
5. There should be an option to identify those questions where
there was no reply:
a) Not applicable, Not performed etc.
40
6. In some audits there are “scoring” options:
Such option gives auditor change to subjectively evaluate the

compliance
41
However, scoring requires:
• In depth knowledge and experience of audited processes by
the auditor
• Full impartiality of the auditor
• Clear rules how to change the scoring / close the findings
More details will be included in the third module for the Lead
auditors
42
14
Reference to evidences:
1. In discretion of the auditor (e.g. in IOSA audits)
2. Based on auditors' consideration (e.g. ICAO PQ using
“Review Evidence & References” box0) whereas evidences
are considered both documentation and records
3. Including a field “Evidence” into checklist or referring to
evidences in the flied “Note” or “Comments”
43
Reference to evidences:
4. It is in the discretion of the auditor to collect, maintain and
disclose the evidences.
5. However, evidences containing information which are
subject to data security, should be maintained based on the
applicable legislation.
6. Therefore, it is recommended to highlight which information
you considered as evidence but do not to keep it (e.g.
Aircraft technical log page number etc.). Such rules should
be agreed cduring the audit preparation.
44
Working with checklist before audit commencement:
• Based on the reference document, the auditor will prepare
the audit checklist in Q-Pulse Software.
• It shall be specific for the verified area/process, containing a
series of questions.
• For each question the auditor actions may be described in
Checklist template Guidance (In Q-Pulse)
45
15
1. Fill those parts which you know, such as:
a) Details of audited unit
b) Names of the auditors in the audit team and individual responsibilities
c) Facilities / items required to be visited / reviewed
d) Documentation to be reviewed including its revision status
e) Details of the previous audit
f) Detailed schedule of the planned audit
g) If member of an audit team, coordinate these activities with a lead
auditor and/or other team members
46
2. Identify, whether there will be necessary any additional
resources to complete specific checklist questions (e.g.
magnifying glass, measurement instruments, torchlight,
etc.). It is not a duty of audited organization to provide you
with such items.
3. Identify, which resources should be required from the
audited organization during the audit (PPEs, ladders,
specific communication means etc.).
47

• Audit checklist shall be available in the Q-Pulse prior to the
audit
• Both representant of the audited organization, lead auditor
and auditing auditor shall fully understand the question in the
checklist and its meaning
• Question must have its origin in the standards based on
which you perform the audit
• Checklist should be reviewed before commencement of the
audit and all related activities should be planned and
coordinated
48
16
Applied exercise
• Prepare a checklist for one of the following processes:
• Availability of controlled documentation
• Managing access to the maintenance stores
• Scheduling of the cabin crew
• SMS training for ground handling staff
Scope of checklist: 5 questions

Type of audit: Audit of AOC holder
• Send your project to r.domcek@gmail.com today
Time: 1 hour
49
Inclusion of risk
assessment
50
Inclusion of risk assessment

1. Compliance monitoring should be focused not only on the
availability or required processes but also to:
1. Identifying hazards to operations;
2. Assessing the effectiveness of safety risk controls
2. Service provider’s compliance monitoring manager shall
have a formal training in risk management
3. Certain audit findings might fall under the category of
hazards to operations. In such cases, the hazard would be
subject to the risk assessment and mitigation process in the
development of corrective action.
51
17
4. To ensure effective monitoring, different range of internal
and external methods for use in the oversight of service
providers.
5. Methods might include;
– auditing,
– systematic review and risk assessment of reported hazards and/or
occurrences,
– monitoring of performance output (SPIs/KPIs),
– reporting and governance processes;
– monitoring and analysis of targeted risk areas,
– establishment of an effective two-way communication link with the
service provider.
52
Performance Risk based

based oversight
regulations Plan Do
Act Study
53

Risk Based Oversight:
Compliance monitoring, where planning is driven by the
combination of
• risk profile and
• safety performance;
And execution focuses on the:
• management of risks,
• ensuring compliance.
54
18
55
“As in many of the previous years, all the accidents were

suffered by small domestic or regional carriers which are
probably little known – or totally unknown – outside the
markets they serve, many of them happening in remote areas
with limited support structures, few navigation aids, and limited
weather reporting.”
- Director of Safety at Ascend by Cirium, Paul Hayes, sums up
2021
56
Compliance monitoring program shall be developed taking into

account:
• the specific nature of the organisation,
• the complexity of its activities,
• the results of past certification and/or oversight activities
required by national regulations or applicable standards, and
• shall be based on the assessment of associated risks
57
19
RISK BASED OVERSIGHT: a way of performing monitoring

allowing to:
i) Prioritize and plan its activities based on compliance, risk
profiling and assessment of the safety performance; and
ii) Verify compliance with a focus on management of
operational risks.
58
Risk Profile:
The elements of risk that are
inherent to the nature and the
operations of the audited entity
59
Safety
performance:
The demonstration
of how effectively
can an audited entity
mitigate its risks
60
20
Safety profile – example of Maintenance organization:
Scope of Approval
Ratings & Level of Number of Specialized Maintenance

Limitations maintenance staff service sites
61
Risk profile– example of Maintenance organization:
Organization Activity
Non
Permanent
staff Use of the
Number of
approval &
& years using Fabrication
Outsourcing Other
the NAA of parts
Part-66 approval
approval
licensed held
Engineers
62
Organisation Performance Risk Profile
Stability of
the
Quality & Organization Management
Safety of task &
Management contracted
System activities
Compliance Organization Safety

history Performance reports
63
21
• Both representant of the audited organization, lead auditor
and auditing auditor shall fully understand the question in the
checklist and its meaning
• Question must have its origin in the standards based on
which you perform the audit
• Checklist should be reviewed before commencement of the
audit and all related activities should be planned and
coordinated
64
Audit Types
After certification, Authority should check-up on Service
Provider periodically using surveillance audits to verify
service providers are still upholding national requirements.
Surveillance audits are very much like certification audits, with
the exception that they are not issuing or re-issuing a
certificate. These are typically conducted by DCA inspectors
annually.
65
Audit Types
Based on the scope of the audit, they are generally divided into
three groups:
1. System Audits
2. Process Audits
3. Product Audits
Where:
System Audit focuses on the organization’s management
system as a whole, and compares the planning activities and
broad system requirements to ensure that each clause or
requirement has been implemented. Typical are the audits of
SMS.
66
22
Audit Types
Process Audit is an in-depth analysis which verifies that the
processes comprising the management system are performing
and producing in accordance with desired outcomes. The
process audit also identifies any opportunities for improvement
and possible corrective actions. Process audits are used to
concentrate on any special, vulnerable, new or high-risk
processes. Typical is the case of introduction of DG operations.
67
Audit Types
Product Audit may be a series of audits, at appropriate stages
of design, production and delivery to verify conformity to any
specified product requirements, such as dimensions,
functionality, packaging and labeling, at a defined frequency.
Such audits are performed not only within Airworthiness or
Aerodrome regulations but also in approval of 83bis
operations.
68
What you should remember?

When starting to think about using of audit as a service
provider’s surveillance tool, you should define:
1. Whether the audit is part of the certification of service
provider or part of regular oversight process (Certification or
Surveillance Audits)
2. Whether audit should be performed on-site, remotely or you
will be satisfied by self audit of service provider
69
23
Applied exercise
• Define typical risk profile of:
• Regional airlines
• Maintenance organization with Part 145 and Part 21 approval
• Ground handling company with 6 airports domestic and 2
international
• Airport with no LVO, one RWY, 300,000 pax per year, international
• Work in groups of 2-3

• Defend your work
Time: 20 minutes
70
Workload planning and

preparation of the audit
team / tasks assignment
71
Workload planning and preparation of

the audit team / tasks assignment
Workload planning role:
• studying the current processes used by your organization to
determine their staffing needs for aviation auditors and
• to provide guidance on ways to improve the staffing
process through the usage of a standard methodology, and
• determining the required auditors manpower to enable
standard compliance monitoring and oversight obligations.
Responsibility for the workload planning:
• Lead auditor for the availability of resources
• Every auditor for his/her currency including currency of
personal documents
72
24
Normally, auditors participates in following regular activities:
a) Initial Certification or Renewal certification or
Approval/Acceptance of process / service provider
b) On-site inspections and audits of operating procedures;
c) Remote inspections and audits
d) Review of self-audits / self-evaluations performed by the
service providers / vendors
e) Initial / recurrent trainings and evaluations
f) Participation in safety/quality/ERP events by service
providers
73

For the planning purposes, auditors should be able to define
the overhead time – as percentage of working time occupied
by other activities.
Lead auditor will prepare a detailed workload plan covering all

required activities.
However, workload plan is subject to regular and ad-hoc

revision based on the results of performance-based oversight
and change management.
74

Several activities are performed off-site, without visiting of the
monitored organization.
In such case, following rules should be introduced:
• Remote audits:
– Agreed and available communication means
– Availability of a facility where auditor will not be disturbed
– Recording of the online meeting
• Review of self evaluation records:
– Validation of the records by two auditors
75
25
Documentation of the workload:
– Should be included in the organizations manual
– Should include basic rules for planning, e.g.:
– Normal duration of the audits (manhours should be
adjusted to days in case of audits outside of HQ)
– Standard schedule of audits during the day
– Reporting to Lead auditor
– Travelling and logistics rules should comply with the
organization's internal procedures
– Changes in audit timing is performed only by Lead auditor
76
What you should remember?

Auditing organization should be able to prove:
1. Availability of staff to perform the audits
2. Presence of the current Audit Plan reflecting the
Performance based oversight principles
3. Personal records of the auditor proving their education,
training and recency
4. Records of the audits performed should be kept and are
subject of internal audits
77
Applied exercise
• Define three criteria based on which should appointed
auditor to perform specific audit
• Work inindividually
Time: 5 minutes
78
26
Evaluation of service
provider’s
documentation &
records
79
Evaluation of service provider’s

documentation & records
• As a part of the audit preparation, auditor should familiarize
with the service provider's supplied documentation.
• Normally, the first step in the evidence collection process is a
review of the service provider's-controlled manuals and other
relevant documentation, to determine if and how the
requirements provisions are documented by the service
provider.
• The review of documents should also provide the auditor
with descriptive information that indicates how the service
provider exercises management and control of its operations
(i.e. systems, programs, standards, policies, processes and
procedures).
80

• Reviewing documentation is one audit activity that can be
accomplished in advance of the on-site audit.
• In an ideal situation:
• Auditee will provide various documents (manuals, handbooks, etc.)
prior to the Audit, thus permitting the auditor to accomplish a review
before arriving on site.
• To further improve efficiency of the documentation review process, the
auditor should request the auditee to provide all references from its
manual system that directly correspond to checklist questions. This will
greatly assist the auditor by eliminating the necessity of having to
search through various manuals on-site when attempting to verify that
specifications are properly documented.
81
27
• In a real situation:
• It will not always be possible to receive documents prior to an Audit; so
in many cases the auditor will be required to review documents in the
early stages of the on-site visit.
• This obviously leads to a certain degree of inefficiency because of the
on-site time needed for this review.
• Count your workload based on the fact whether documentation:

– Was provided prior to audit
– Cross references were prepared
82

• Documentation and Records – Definitions:
Documentation:
• The written information considered necessary to define and support the
performance of administrative or operational functions.
• Documentation may be displayed via electronic or paper media, and may serve
various purposes (e.g. communicating, presenting processes and procedures,
proving conformity, knowledge sharing).
• Specific examples of documentation include operations manual, management

manual, quality manual, training manual and policy manual.
83

Records:
• Information created, received, and maintained as evidence and information by an
organization or person, in pursuance of legal obligations or in the performing any
action.
• There are many examples of records, however a few common examples include:
a) Reports, (e.g. safety reports, occurrence reports, audit reports, service difficulty
reports);
b) Correspondence, (e.g emails, letters);
c) Minutes of meetings, agendas, action items from the meetings;
d) Completed forms, (e.g. hazard reporting form, maintenance approval request,

audit forms, training forms); rvation records).
84
28
Records/2:
e) Rosters and schedules, (e.g flight crew rosters, flight schedules, audit schedules,
maintenance schedules);
f) Certificates and registers, (e.g. training certificates, attendance registers,

signature registers); and
g) Records from activities, (e.g. maintenance records, modifications, pilot training &
checking, inspections, LOSA observation records).
Note:
Templates, forms and other documents that are blanks do not constitute a record,
they do not demonstrate that a requirement is active, and does not support the
assessment of implementation.
85

Working with records:
• In order to assess compliance, the auditor must examine
records that demonstrate that the requirement(s) is (are)
established, activated, integrated, etc., where applicable, in
accordance with the documented systems, programs,
processes, procedures and plans in use by the operator.
• The fact that specifications are properly documented does
not always mean that they are really used and implemented.
86

• Examining the records provides an additional source of
information to contribute to the assessment of compliance.
• Records are general information that is generated through
implementing the documented systems, programs,
processes, procedures and/or plans.
• When examining the records, auditor should ensure that the
record is the most recent and from a reliable source in an
attempt to ensure it is factual in its content.
• (For example, getting copies of meeting minutes from an email, may not
be the most reliable source, getting it from the network/server where all
the official minutes are kept may be more reliable)
87
29
In no records exists because activity is new or never applied
(for example, emergency response or abnormal situations, or the
confidential safety reporting system, when the system has not been used)
In these cases, the auditor must find other sources of

evidence to support the assessment of the ISARP, usually
through interview by asking if staff are aware of the system /
actions / response, and if any confidential reports or abnormal
situations have occurred.
88

In no records exists because activity is new or never applied
(for example, emergency response or abnormal situations, or the
confidential safety reporting system, when the system has not been used)
In some cases, the records might not be available due to the

recent commencement of a new approval by the operator.
For example, the operator was approved to carry Dangerous
Goods two days before the on-site audit; in this situation, the
auditor should not use the availability of Dangerous Goods
records as the only means to verify the implementation.
89

Working with records – Continued effectiveness:
• Records may support the assessment that ISARP
requirement(s) is/are monitored and evaluated, as
necessary, for continued effectiveness / Performance
Based Oversight (e.g. minutes of meetings are distributed,
action items from management reviews are carried out and
closed, control charts, job descriptions, workouts, process
improvement strategies are actively managed, carried out
and closed).
90
30
Working with records – Continued effectiveness:
• For some audits, a number of Mandatory Observations must
be completed and they often contains requirements to verify
records, e.g,:
MO-7-MNT AD/SB Management:
• Direct examination of engineering/planning processes,
documentation, records; interviews of personnel
91
Applied exercise
• Define auditors actions to be taken to verify compliance
with requirement;
• “Safety review Board meeting shall be performed at least
quarterly with presence of all nominated postholders and
relevant safety personnel.”
Time: 10 minutes
92
Other Evidence of
Conformity
93
31
Other Evidence of Conformity
Evidence:
• The auditor needs to secure sufficient factual or objective
evidence, derived from all available sources of evidence,
information, documentation and activities assessed during
an Audit, to determine that the Auditee is in conformity with
the ISARP or not.
• Conversely, conformity or non-conformity must never be
based on subjective evidence or opinion.
94

Evidence:
• As mentioned in the module 1, Auditor Actions (AAs), should
are customized for each audit question, and provide the
following:
a) guidance to auditors for the actions to be taken to confirm
implementation;
b) a record of the actions taken to confirm implementation for
Operator’s internal assessments; and
c) a means of standardizing the assessment of
implementation
95

Evidence based on AA may be:
(a) identifying and assessing documentation;
(b) interviewing personnel;
(c) observing facilities, equipment and other physical resources;
(d) observing the conduct of operational activities and processes;
(e) observing implementation of provisions of appliable standards;
(f) examining data collected from day-to-day operations (e.g. flight data
analysis, quality control inspections); and
(g) examining records (accident report, performance reports, supplier
evaluation, maintenance, training, checking, inspections, audits, agendas,
minutes, action items).
96
32
BUT: Not all evidence is objective or factual. Auditors must
exercise healthy skepticism and professional judgment when
evaluating information derived from:
(a) individuals that might be operationally uninformed, misinformed or not
fully aware of all audit requirements;
(b) representatives of the auditee who may be attempting to influence the
objectivity of the auditor;
and/or
(c) sources that could have negative intentions designed specifically to
mislead, hinder or prejudice the auditor.
97

Uncertainty:
• In determining conformity, the sampled evidence must be crosschecked
with other sources of objective evidence to gain certainty for the auditor to
exercise ‘Auditor Judgement’ in making the final assessment.
• A single sample is not enough to support an assessment of conformity or
nonconformity.
• If the auditor is not certain about an assessment, further evidence must be
collected and crosschecked.
98

Interviewing Personnel:
• Auditors must conduct interviews with operational personnel during an
Audit, often referred to in the Auditor Actions as the Responsible Manager
or designated representative, primarily for the purpose of gathering the
supporting evidence needed to determine conformity with IOSA
specifications and in the assessment of ‘implemented’.
• The interviews help determine the level of awareness and understanding
by staff of the system, program, process, procedure or plan.
• The Responsible Manager may be at any level (frontline manager, middle
manager, executive level, post holders, or Accountable Executive), that is
responsible for that operational activity, function or area.
• This responsibility may be defined within the organization chart, job
descriptions, duty statements or within the relevant manual(s).
99
33
Interviews must:
• be of operational personnel, frontline managers and supervisors; and
• cover all levels of the organization (from the Accountable
Manager/Executive/CEO down to the staff on the workshop floor) and
across the operational functions or departments of the operator.
Interviews must not:
• be focused only on the quality department staff, one manager, staff that
compiled the internal auditing against the ISARPs, nor staff/managers from
an external Service Provider. However, the internal auditors should be
encouraged to attend the interviews for professional development, assist in
standardization of interpretation, and to address questions in the
verification of the internal audit results against the checklist questions
100

Interpreting of interviews:
• Information gained from interviews should normally be considered
subjective evidence and will seldom be enough by itself to justify a final
audit assessment of conformity or nonconformity.
• Additionally, one interview may seldom be enough to justify an assessment
of conformity or nonconformity.
• The evidence from the interview should always be accompanied by cross-
referenced evidence (preferably objective), which must be analyzed
together in order to arrive at a conclusive determination of conformance or
nonconformance.
101

During the interviewing process, the auditor must take into
consideration:
(a) the tough and stressful process, and the effect on the interviewee;
(b) cultural factors, (i.e. breaks for prayer time etc.);
(c) operational priorities of the person being interviewed; and
(d) the well-being of the interviewee, with regular comfort breaks..
102
34
Observations:
Observing and assessing facilities and/or actual operations
during an audit is required for the evaluation of evidence and
determining of conformity or nonconformity with standards, in
terms of being implemented by the operator.
103

Observations:
• Observe activities that are indicative of normal operations by
typical front line personnel.
• Operational activities performed by individuals with a
significantly higher level of qualification (e.g. instructors,
supervisors) would not be indicative of normal operations.
• Rather select a random flight and/or maintenance activity if
possible or perform an additional mandatory observation if
feasible.
104

Other sources:
Other sources of evidence may present themselves during an Audit.
Evidence from any source is acceptable for use in determining conformity or
nonconformity, as long as it can be verified as factual, e.g.:
(a) direct observation of facilities and/or front-line operations;
(b) records that reflect completion of operational requirements (e.g. training,
checking, inspections, audits, maintenance, component changes, modifications);
(c) records that provide the history or output of management activities (e.g.
meeting agendas, minutes, audit reports and action items);
(d) statistical summaries of operational performance (e.g. accidents, incidents,
failure rates);
(e) regulatory requirements or approvals;
(f) audits of the operator by Regulators (or other organizations); and
(g) reports of accidents, incidents, irregularities or other events.
105
35
Opening Meeting
106
Opening meeting
• Every audit must be started with a formal opening meeting.
• The opening meeting is conducted by the designated Lead
Auditor and should include the audit team members, unless
logistically not possible, and representatives of the auditee.
• Ideally, auditees representatives include members of senior
management, however it is at their discretion to nominate
participants for the opening as well as the closing meetings.
Attendance needs to be recorded.
107
Opening meeting
Typical agenda:
1) introduction of members of the Audit Team and
representatives of the auditee;
2) roles and responsibilities of the Audit Team and the auditee;
3) exchange of contact information and lines of
communication during the Audit;
4) Audit objective, scope and application:
5) planned schedule of all Audit activities, including the closing
meeting;
6) methods and procedures used to conduct the Audit;
108
36
Opening meeting
Typical agenda:
7) criteria for establishing conformity with standards;
8) administrative arrangements and facilities to be used during
the Audit;
9) arrangements for travel to, and security/ramp passes for,
various Audit venues;
10)arrangements for observations of operational activities;
11)language to be used during the Audit;
12)method of keeping the auditee informed of Audit progress;
109
Opening meeting
Typical agenda:
13)method of reporting findings or observations to the auditee;
14)confidentiality of the audit;
15)safety, security, quarantine or emergency procedures
applicable to the Audit Team;
16)availability and roles of guides or escorts during the Audit;
17)miscellaneous
110
Applied exercise
• Which agenda you will add into point of “Miscellaneous” in

case of auditing:
I. Flight operations department
II. External service provider of fuelling
III. Pilot training academy
• Work in pairs
• Defend your opinion
Time: 10 min
111
37
Assessment of Conformity
112
Evidence of documentation of the process:
• Auditor must find information that addresses the
requirements as specified in the applicable standard(s)
published in a controlled document, such as a manual,
handbook or other similar publication, that comprises
content approved by the operator, and where applicable the
Authority;
• The wording of standard must not to be copied verbatim into
the operator's manuals and/or controlled manuals as a mean
to conform to it’s provision. Instead, the real matters shall be
described for the purpose of conformity to the standard's
provision.
113
• Shall be based specifically on whether the provision requires
a system, plan, policy, process or procedure. For example: if
standard states that a process is required, then the auditor
must assess the documentation to see that a process has
been documented.
• It is often an incorrect assessment is made due to the
misunderstanding of these terms.
114
38
• A controlled document must have associated processes for
the positive control of content, revision, publication,
distribution, availability and retention, what ensures
appropriate operational personnel always have access to the
current version of the document;
115
• Controlled documents are always under the control of the
operator and not another entity;
• Manuals of the DCA are not acceptable for evidence of
documentation.
• Manuals produced by external companies for the operator,
such as navigation manuals, and the manuals produced by
the manufacturer of the aircraft/components can be used for
evidence of documentation.
• Auditor shall ensure that these manuals/documents are
controlled for the latest version/revision.
116
Assessment of documentation:
1. Identify the manuals and/or other document(s) that contain
the information to address the specification(s) in the
particular standard – it must be contained in a controlled
document
2. The documentation must be under the direct or indirect
control of the operator, not the rulemaker
3. Paper or electronic forms of documentation are both
acceptable, as long as the memory medium meets the
criteria for a controlled document and is traceable
117
39
4. The manuals and/or documents being assessed shall be
available for use by all the staff, crews and/or external
service providers (if applicable) involved
5. Confirming that the process, procedure, etc., is documented
and sufficient
6. The content of a document must be written in a language,
style and format that clearly and accurately represents
specification(s), and will be understood by relevant
operational personnel.
118
7. Documentary references are not necessarily required to be
listed individually for each sub requirement. However, all
requirements (including sub-requirements) of the standard
must be traceable from the documentary reference(s)
8. The references for documents or manuals must include a
title, edition or revision number, and/or a date of issue, or
other means of recording traceability of the information
9. When a standard is applicable to multiple fleets, then there
must be a document reference applicable to each of the
fleets utilized
119
10. Auditor shall record either the full document name,
abbreviation or acronym listed in the Document Reference,
followed by detailed subreferences (e.g. chapter, section,
sub-section number) as applicable to the documented
format and the ISARP specification. (For example: “OMA
Chapt 2.5.3 vi)
11. Documents such as meeting minutes, bulletins, records,
plans, checklists, etc., generally cannot be used as
documentation references unless those documents fulfill all
the document control requirements.
120
40
12. Documents of a temporary or transitory nature, or records,
(e.g. letters, email, memos, flyers, posters, MS PowerPoint
presentations), are not acceptable as sources of controlled
documentation, unless the information has been
reproduced and included as part of the content in a
controlled document.
13. If the standard covers a broad range of procedures and
there are document references within the majority of the
sections of a manual, a generalized reference may be used,
i.e. a phrase such as “CAME – complete document” or
“OMA – all sections”.
121
Assessment of implementation:
• Determine that the requirements as specified in the
applicable IOSA specifications have been established,
activated, integrated, incorporated, deployed, installed,
maintained and/or made available, as part of the operational
system:
• as a part of the operator's organization or operations, or;
• as a contracted or outsourced operational function.
122
1. In assessing implementation, the auditor must be familiar
with the content of the auditee’s documentation
2. Provision(s) of requirement must be effectively implemented
in a manner consistent with the way the requirement is
documented (write what you do, do what you write)
3. Based on auditor judgement following, thorough
observations, interviews and reviews of records (among
other audit techniques). Evidence must then be identified to
confirm that the specification(s) is/are being used on a daily
basis by the personnel concerned, in accordance with the
documented requirement
123
41
4. Evidence must then be identified to confirm that the
specification(s) is/are being used on a daily basis by the
personnel concerned, in accordance with the documented
requirement
5. Assessment of implementation should be conducted
following the assessment that the requirement(s) is/are
documented
6. Requirements are subject to monitoring to ensure desired
outcomes are achieved
124
7. Auditor Actions provide specific information on the actions
that would normally be used to confirm implementation
8. Specific Observations may be prescribed to provide
activities that would contribute in the assessment to confirm
implementation
125
Not applicable requirements:
• “When making a determination as to the applicability of
individual requirements, it is important to take into account
operations that are conducted within stations and locations
throughout the operator's network”.
• Only when a specific standard is determined to be not
applicable across the whole network, it is not audited and
recorded as N/A
126
42
Not applicable requirements for outsourced functions:
• Functions currently outsourced cannot be recorded as N/A,
but must be audited through the oversight program of
outsourced functions.
• Standards related to the outsourced functions can only be
recorded as N/A when it has been confirmed that the
specifications do not apply to the processes anywhere within
the organization, or throughout its operational system.
127
Conditions for N/A Assessments:
1. Can only be used if the process, function, equipment
requirement, etc., is completely inactive, or does not apply
to any part of the operator's system
2. Descriptions for N/A assessments must be clear, complete
and not contradict other assessments
3. Sub-specifications not applicable (usually referred to as
mini N/As) must be independently assessed as N/A,
irrespective of whether the standard is in conformity or not.
128
Conditions for N/A Assessments:
4. An N/A assessment cannot be used for active outsourced
functions
5. N/A assessments are not required when a sub-specification
is contained in a Table; or a sub-specification is separated
by an ‘or’
129
43
Applied exercise
• Find in ISM ISARPs applicable for your scope and

highlight one:
• Compliant
• Not applicable
• Work in groups
• Do not forget to use proper references to the documentation

Time: 10 min
130
Non-conformities
131
Non-conformities
A determination of conformity or non-conformity must always
be based on the analysis of appropriate factual or objective
evidence collected by the auditors.
• Within IOSA:
• if the nonconformity is due to a deficiency in documentation, then the
assessment is ‘Not Documented’.
• If the nonconformity is due to the requirements not being implemented
in practice, then the assessment is ‘Not Implemented’.
• The final assessment for a nonconformity may be ‘Not Documented Not
Implemented’, ‘Implemented, Not Documented’ or ‘Documented, Not
Implemented’.
132
44
Non-conformities
Within MNA QMS, nonconformity is presented as:
Non-satisfactory
• A nonconformity with a standard results in a Finding; a
nonconformity with a Recommended Practice results in an
Observation.
• A Finding or Observation will be generated only when the
auditor has not been able to obtain the required objective
and factual evidence needed to assess conformity with the
standard.
• All factual evidence that supports a determination of
nonconformity must be recorded on the checklist.
133
Non-conformities
• Before being finalized, evidence or lack of evidence leading
to each Finding and Observation must have been discussed
and agreed upon by the Audit Team.
• Ideally, all Audit Team members and the auditee should have
an awareness of all Findings and Observations prior to the
Closing Meeting.
134
Non-conformities
Description of a non-conformity:
• Description of the non-conformity, i.e. the narrative needs to
define the deviation from the standard and/or documented
processes/procedures in a factual, clear and effective
manner
• Auditors can test the clarity and completeness of their
evidence descriptions by stepping back, looking at the
description though the eyes of the Auditee and asking
themselves – is it clear what type and extent of corrective
actions will be needed?
135
45
Non-conformities
Description of a non-conformity – what NOT TO DO:
• Broad generalization using words such as “incompletely”,
“insufficiently”, “not fully”, etc., must not be used.
• First person or personal terms such as: “I confirmed that”,
“we consider”, “we are of the opinion that” are subjective and
must also not be used for factual evidence
• Information gained from one interview alone will generally
not be enough to verify evidence of implementation; such an
assessment must be supported by other sources of
evidence.
136
Non-conformities
Non-Conformity Classification:
Safety hazard
A ‘safety hazard’ classification is any condition, event or
circumstance based on factual evidence, which could induce
risk of an accident or incident. It directly affects operational
safety or airworthiness of aeroplanes. Safety hazards must be
eliminated (or avoided) immediately.
137
Non-conformities
Level 1 finding
A level 1 finding is any significant non-compliance with
standards, and it must be corrected before next flight. It shows
non-compliance with requirements which lowers the safety
standard and hazards seriously the flight safety. If it is not flight
critical, the final corrective action may be taken with an
implementation date of a maximum of 10 days from the closing
meeting
138
46
Non-conformities
Level 2 finding
A level 2 finding requires a planned action with an
implementation date within a maximum of 1 month from the
closing meeting. If the responsible department requests, a
maximum of 1 additional month implementation period may be
allowed by Director of Quality (Corporate).
139
Non-conformities
Level 3 finding
Level 3 finding requires a planned action with an
implementation date within a maximum 3 months from the
closing meeting. If the responsible department requests, a
maximum of 3 additional month implementation period may be
allowed by Director of Quality (Corporate).
140
Non-conformities
Observation
Observations do not represent non-compliance, but
observations are intended to give information in order to
improve process/procedure effectiveness and quality standard.
Observations do not require a planned corrective actions.
141
47
Non-conformities
Non-Conformity records:
• After completing audit, auditor shall enter non-conformity
into Q-Pulse Software. A non-conformity report (CAPA
record) is notified to responsible manager via email
generated Q-Pulse Software for Internal Audit.
142
Non-conformities
Closing Meeting:
• At the end of an audit, prior to preparing the preliminary
audit report, the audit team will hold a closing meeting with
the auditees. The main purpose of this meeting is to present
audit preliminary results to the auditee in such a manner so
as to ensure that they clearly understand them.
143
Non-conformities
Closing Meeting:
Non-conformities shall be agreed by all parties at this meeting
by:
• Summarizing audit results and present them appropriately to
the auditee management,
• Discussing corrective actions preliminary proposed by
concerned Managers and Team Leaders,
• Re-defining corrective action time limits, if necessary,
• Agreement about distribution of preliminary audit report and
outline the use of the CAPA to concerned Managers.
144
48
Applied exercise
• Define which requirements were not fulfilled in the

following statement:
FLT 1.10.2 The Operator shall have an audit planning process and
sufficient resources to ensure audits of flight operations functions are:
• (i) scheduled at intervals that meet management system requirements;
• (ii) completed within a specified time period. (GM)
Documented not implemented
Narrative: The scheduled audit of Flight Operations training records, as
specified on the annual Audit Plan, had not taken place within the specified time
frame.
Document Reference: CTM 3.5.

Time: 5 min
145
Data Collection and

security
146
Data Collection and Security

Collection of the audit data:
• Within MNA, for auditing there is used specific software
Qpulse.
• Security of the data should be reviewed on a regular interval
or in cae of any update of the software
• Software provider shall provide the measures which are
acceptable by the operator’s IT security procedures.
147
49
Progress check
Where are we now?
• Review the Module objectives
• How confident do you feel in reaching these objectives?
• What are the main takeaways from this module?
• Do you have any questions, concerns?
148
Questions?
149
50

Auditing-Techniques-Audit Performance 2022

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Auditing-Techniques-Audit Performance 2022

Uploaded by

Copyright:

Available Formats

Auditing Techniques –

1 © 2022 Copyright Fly 88, s.r.o.

2 © 2022 Copyright Fly 88, s.r.o.

3 © 2022 Copyright Fly 88, s.r.o.

4 © 2022 Copyright Fly 88, s.r.o.

2. Next one is this module ‘Performing of audits’

5 © 2022 Copyright Fly 88, s.r.o.

6 © 2022 Copyright Fly 88, s.r.o.

7 © 2022 Copyright Fly 88, s.r.o.

Principles of Audit Preparation

Principles of Audit Preparation

9 © 2022 Copyright Fly 88, s.r.o.

10 © 2022 Copyright Fly 88, s.r.o.

Principles of Audit Preparation

11 © 2022 Copyright Fly 88, s.r.o.

Audit Objective, Scope and Criteria

12 © 2022 Copyright Fly 88, s.r.o.

13 © 2022 Copyright Fly 88, s.r.o.

Audit Objective, Scope and Criteria

14 © 2022 Copyright Fly 88, s.r.o.

Audit Objective, Scope and Criteria

15 © 2022 Copyright Fly 88, s.r.o.

16 © 2022 Copyright Fly 88, s.r.o.

Audit Objective, Scope and Criteria

Submit final Identify findings Categorize

assessment evaluate risks

17 © 2022 Copyright Fly 88, s.r.o.

What should you remember?

18 © 2022 Copyright Fly 88, s.r.o.

20 © 2022 Copyright Fly 88, s.r.o.

21 © 2022 Copyright Fly 88, s.r.o.

22 © 2022 Copyright Fly 88, s.r.o.

23 © 2022 Copyright Fly 88, s.r.o.

24 © 2022 Copyright Fly 88, s.r.o.

25 © 2022 Copyright Fly 88, s.r.o.

2. In case, if reference is in electronic format only, refer to

26 © 2022 Copyright Fly 88, s.r.o.

Applicable Legislation - Terms

27 © 2022 Copyright Fly 88, s.r.o.

28 © 2022 Copyright Fly 88, s.r.o.

Applicable Legislation - Terms

29 © 2022 Copyright Fly 88, s.r.o.

Applicable Legislation - Terms

30 © 2022 Copyright Fly 88, s.r.o.

31 © 2022 Copyright Fly 88, s.r.o.

Applicable Legislation - Terms

Applicable Legislation - Terms

33 © 2022 Copyright Fly 88, s.r.o.

34 © 2022 Copyright Fly 88, s.r.o.

36 © 2022 Copyright Fly 88, s.r.o.

37 © 2022 Copyright Fly 88, s.r.o.

38 © 2022 Copyright Fly 88, s.r.o.

39 © 2022 Copyright Fly 88, s.r.o.

40 © 2022 Copyright Fly 88, s.r.o.

Such option gives auditor change to subjectively evaluate the

41 © 2022 Copyright Fly 88, s.r.o.

42 © 2022 Copyright Fly 88, s.r.o.

43 © 2022 Copyright Fly 88, s.r.o.

44 © 2022 Copyright Fly 88, s.r.o.

45 © 2022 Copyright Fly 88, s.r.o.

46 © 2022 Copyright Fly 88, s.r.o.

47 © 2022 Copyright Fly 88, s.r.o.

What should you remember?