Professional Documents
Culture Documents
OP 1110 Audit Failure and Risk Mitigation
OP 1110 Audit Failure and Risk Mitigation
1.0 PURPOSE
This procedure defines failure criteria and establishes the processes for audit
failure and risk mitigation.
2.0 SCOPE
3.0 DEFINITIONS
4.0 PROCEDURE
4.1 An audit may be failed for any of the reasons categorized by the modes of
failure defined below.
• The Total NCRs or Major NCRs exceed the thresholds defined in Table 1
− When applicable, compressed audits shall be assessed against the
criteria for the uncompressed (normal) length
• For Verification of Corrective Action Audits (VCA), any non-sustaining
NCRs are identified
• Initial (Re-entry) Audits shall be evaluated using Reaccreditation Audit
failure criteria in Table 1
• Add Scope audits shall be evaluated using Initial Audit failure criteria in
Table 1
i-frm-1101 20-May-2020
Nadcap OPERATING PROCEDURE OP 1110
AUDIT FAILURE AND RISK MITIGATION Page 2 of 14
Document Owner: Michael Graham Issue Date: 19-Apr-2015
Revision Date: 30-Mar-2021
• All NCRs are not Accepted/Closed/Void after the review of the fourth cycle
as indicated by the cycle count displayed in eAuditNet
− An allowance of up to two (2) additional response cycles as displayed
in eAuditNet are permitted at the discretion of the Audit Report
Reviewer. This is to allow for situations where a full cycle has not been
completed; such as where the audit was submitted to SE Review in
error, missing objective evidence, communication issues, etc.
• In the case where the audit has been submitted for Task Group Review,
all Task Group Ballot comments are not addressed after two (2) cycles
4.2 Audits meeting the Mode A failure criteria do not require a failure ballot and
shall follow the failure process defined in 4.5.
4.3.1 A VCA audit failure ballot does not require a failure ballot for the main audit.
4.3.2 The Task Group has a minimum of seven (7) days to review the failure ballot.
4.3.3.2 Votes to not fail an audit shall include a rationale for voting not to fail.
4.3.4 Task Group Subscriber Voting Members voting to fail an audit shall also vote
on withdrawal of the current accreditation.
4.3.4.1 In the case where the audit was failed by the Task Group, any vote to not fail
the audit shall also be considered a vote to not withdraw the current
accreditation.
4.4 If the Task Group decides to not fail an audit, the audit shall be returned to
Auditee Review to continue the audit review process per OP 1106 Audit
Report Review.
4.5 If the Task Group decides to fail an audit, the following actions shall be taken:
4.5.2 If there is a current accreditation associated with the failed audit, it shall be
withdrawn unless otherwise agreed by the Task Group in the failure ballot.
4.5.3 The Auditee shall be notified of the audit failure and the requirements for risk
mitigation.
4.5.4 The Subscriber Voting Members and PRI Staff associated with Task Groups
where the Auditee holds current accreditation or has an in-process audit, shall
be notified of the audit failure.
4.6.1 When a VCA audit fails, the main audit shall also be failed per Mode A.
4.6.2 A linked AC7004 audit shall fail per Mode A or accreditation withdrawn per
OP 1107 Post Accreditation Actions, if a commodity audit fails and the
Auditee does not hold another Nadcap accreditation.
4.6.3 If a commodity audit fails and the Auditee holds AC7004 and at least one
other Nadcap accreditation, the AQS Task Group shall review the failed audit
to determine if actions are required on the AC7004 accreditation.
4.6.3.1 Evidence of AQS Task Group review and subsequent decision(s) shall be
documented in the failed audit.
4.6.4 When an Add Scope Audit fails, the linked accreditation may be withdrawn
per OP 1107 Post Accreditation Actions or any in process audits may be
balloted for failure.
4.6.5 When an Add Scope Audit is linked to an audit that fails, the Add Scope Audit
shall fail per Mode A.
4.6.6 Failures of linked Main and Satellite audits shall follow the process defined by
the applicable Task Group in OP 1114 Task Group Operation.
4.6.7 When an Option B Audit fails, the Nadcap Management Council (NMC)
Subscriber Voting Members on the applicable Review Team shall review the
failed audit and consider if any action(s) is warranted on other audits in the
scope of accreditation. Actions may include, but are not limited to the
following:
• Issue certificate for all linked audits excluding any failed audits
• Withhold all or a subset of linked audits (based upon site and or
commodity) until Risk Mitigation is completed
• Fail all or a subset of linked audits (based upon site and or commodity)
• Require an Observation of an internal audit performed by the Auditee
• Require additional PRI verification audits on the following year audit plan
Nadcap OPERATING PROCEDURE OP 1110
AUDIT FAILURE AND RISK MITIGATION Page 4 of 14
Document Owner: Michael Graham Issue Date: 19-Apr-2015
Revision Date: 30-Mar-2021
4.6.7.1 Evidence of Review Team review and any decision(s) on linked audits shall
be documented in the Option B HQ audit.
4.7.1 When an initial or reaccreditation AC7004 audit fails and the Auditee does not
have an acceptable alternative quality system per PD 1100 Nadcap Program
Requirements, the following actions shall be taken:
4.7.1.1 The status of any scheduled Initial Audit shall be changed to Initiated.
4.7.1.3 Any existing accreditation shall be withdrawn per OP 1107 Post Accreditation
Action.
4.7.2 When an AC7004 Reaccreditation audit fails, any existing accreditation shall
be handled per OP 1107 Post Accreditation Action.
4.8.1 The purpose of the Risk Mitigation process is to provide the opportunity for:
4.8.2 All failed audits must successfully complete Risk Mitigation prior to an Initial
(Re-entry) Audit being initiated.
4.8.2.1 If 24 months have elapsed since the date of failure, Risk Mitigation is not
required.
4.8.2.2 The new scope of a failed Add Scope Audit cannot be included in a future
accreditation audit within 24 months of failure unless Risk Mitigation has been
completed for the failed Add Scope Audit.
4.8.2.3 Where the same checklists are used by more than one Task Group, a
checklist that was included in an audit that failed cannot be added to a future
accreditation audit within 24 months of failure unless Risk Mitigation has been
completed for the failed audit or otherwise approved by the Task Group of the
failed audit.
Nadcap OPERATING PROCEDURE OP 1110
AUDIT FAILURE AND RISK MITIGATION Page 5 of 14
Document Owner: Michael Graham Issue Date: 19-Apr-2015
Revision Date: 30-Mar-2021
4.8.2.3.1 To obtain approval, the Auditee shall submit an appeal per OP 1113 Appeals
to the Task Group that failed the audit.
4.8.3 The Auditee shall agree to pay the required fees prior to starting Risk
Mitigation.
4.8.4 Audits with zero Open NCRs upon entering Risk Mitigation shall be moved
directly to Risk Mitigation Completed without ballot.
4.8.5.1 Auditees shall provide responses to all Open NCRs within 21 days of the date
the Audit enters or resumes the Risk Mitigation process.
4.8.5.2 Subsequent Auditee responses are due within seven (7) days.
4.8.5.3 The Audit Report Reviewer shall review responses within 14 days.
4.8.6 If responses are not adequate to close all NCRs after four (4) cycles in Risk
Mitigation, or the audit has accumulated greater than 30 days of Cumulative
Response Delinquency in Risk Mitigation, the Audit Report Reviewer shall
ballot the audit for suspension of Risk Mitigation, identifying the reason in the
ballot.
4.8.6.1 The Task Group has a minimum of seven (7) days to review the suspension
ballot.
4.8.6.4 Votes to not suspend Risk Mitigation shall include the rationale for not
suspending.
4.8.6.6 To resume the Risk Mitigation process, the Auditee must submit an appeal in
accordance with OP 1113 Appeals.
4.8.7 Once all NCRs are resolved, the audit shall be submitted for Task Group
Review.
4.8.8 Audit reports shall be in Task Group Review for a minimum of seven (7) days.
4.8.9 Accept with Comments votes shall have the comments resolved, and when
requested, the resolution shall be agreed to by the commenter.
4.8.9.1 Resubmission of the audit for Task Group Review is not required.
4.8.10 A Disapprove vote requires the comment associated with the disapproval to
be resolved.
4.8.10.1 After resolution, the audit report shall be submitted to Task Group Review.
4.8.10.2 When there is a Disapprove vote on a second ballot for the same issue, the
Task Group Subscriber Voting Members shall resolve the disapproval with a
2/3 majority required to reject the disapproval.
4.8.11 All Subscriber comments made within the audit report (Task Group
Discussion, Ballot Screen, Forum, etc.) shall have documented
acknowledgement, and where necessary, evidence of action or resolution will
be attached in eAuditNet.
4.8.12 The status of all nonconformances shall be Closed or Void prior to completing
the Risk Mitigation process.
4.8.13 Completion of the Risk Mitigation process shall not result in accreditation of
the audit.
4.9.1 An Initial (Re-entry) Audit shall only be scheduled when the Risk Mitigation
process is completed.
4.9.1.1 For audits with a start date greater than 24 months after the date of failure,
the completion of the Risk Mitigation process is not required and the audit
shall be scheduled as an Initial Audit.
4.9.2 The Initial (Re-entry) Audit shall not be conducted within 90 days of the failure
date.
4.9.3 The length of the Initial (Re-entry) Audit may be increased to allow time to
verify the corrective actions of the failed audit.
Revision Summary
Date
19-Apr-2015 New Procedure – Transitioned from NOP-011 and PD 3000
27-Apr-2015 Annual Mode B Failure Criteria Update. (CP, COMP, CMSP, MTL,
NDT, SEAL, WLD)
29-Mar-2016 Annual Mode B Failure Criteria Update.
Nadcap OPERATING PROCEDURE OP 1110
AUDIT FAILURE AND RISK MITIGATION Page 7 of 14
Document Owner: Michael Graham Issue Date: 19-Apr-2015
Revision Date: 30-Mar-2021
Revision Summary
Date
5-April-2016 Editorial change to add AC 7004 and OP 1103 to
“Referenced Documents”.
09-May-2016 Revised Section 4.1.4 to clarify that the 4-round limit on the number of
rounds of response applies prior to balloting an audit for accreditation.
The number of rounds permitted to resolve issues identified during the
ballot is at the discretion of the Task Group.
Sections 4.4 and 4.5: Revised process to require the Risk Mitigation
Review to be performed by the Staff Engineer/Consultant Reviewer
and to require Suspension or Closure of an audit via Risk Mitigation to
be approved by Task Group Subscriber Ballot.
30-Sep-2016 Remove 4.4.7.1 which allowed Task Groups to require a VCA audit
prior to scheduling a Re-entry audit. Revised 4.5.1.1 to allow the
length of time of the Re-entry audit to be adjusted to allow time for
verification of corrective actions. Revised 4.5.1 to clarify that the 90-
day timeframe starts from the date of audit failure.
It should be noted that the May 09, 2016 revision was held pending
programming changes required to support the revision in eAuditNet.
During “requirements gathering” the changes above were
recommended (and approved via ballot) to support the eAuditNet
enhancement. This revision of the procedure is being released
September 30, 2016 and is effective January 1, 2017.
16-Mar-2017 Annual Annex A (Mode B Failure Criteria) update.
Revision Summary
Date
28-Aug-2018 Complete re-write Document Transition Activity. Even Year Review.
Removed Risk Mitigation definitions; Separated out and clarified failure
modes; Mode D requirement for explanation if audit was not balloted
for number of cycles moved to OP 1106; removed 120-day
requirement from Mode E; new outcomes when an AC7004 audit fails
4.7.1; new 4.8.9, 4.8.10, 4.8.11; Process for establishing Mode B
criteria moved to Annex A; added A3.4.1
18-Mar-2019 Annual Annex A (Mode B Failure Criteria) update. (Changes to HT,
NDT, NMMM, NMMT, NMSE. Option B HQ removed due to limited
number of audits.)
ANNEX A
A1.0 Each Task Group shall establish Mode B failure criteria for allowed number of
Major and Total NCRs per audit day for Initial and Reaccreditation Audits at
the first Task Group meeting of the year.
A1.1 For new Task Groups, Mode B failure criteria shall be established when a
data set meeting the requirements of A2.0 can be compiled.
A2.0 A data set for Initial Audits and Reaccreditation Audits shall be compiled for
each Task Group using audit NCR data from audits conducted in previous
years (January 1 to December 31).
A2.1 At a minimum, each data set shall include audit NCR data from the previous
year. Task Groups may elect to include multiple years of audit NCR data in
their data set.
A2.2 The Reaccreditation Audit data set shall include Reaccreditation Audits and
Initial (Re-entry) Audits.
A2.3.1 Additional years of data shall be added in full year increments as needed to
attain the minimum number of audits.
A2.4 The NCRs per audit day shall be calculated for each audit in the data set.
A2.4.1 The number of audit days in the data set shall account for multiple Auditor
audits and compressed audits.
A2.4.2 For example, the data set for an Initial Audit conducted with two (2) Auditors
lists the number of audit days as four (4). The audit scope in this case graded
to eight (8) days; however, the number of days was reduced to four (4) by
adding a second auditor. To accurately reflect the number of NCRs per audit
day, the number of audit days in the data set must be changed from four (4)
to eight (8).
A3.0 Failure criteria shall be established using the compiled data sets.
A3.1 Select failure criteria per audit day where the allowed number of Major and
Total NCRs are within the limits established by the 95th and 98th percentiles.
A3.2 Determine the allowed number of Major and Total NCRs by multiplying the
calculated percentiles by the number of audit days.
A3.3 Task Groups may define an upper limit of NCRs per audit day.
Nadcap OPERATING PROCEDURE OP 1110
AUDIT FAILURE AND RISK MITIGATION Page 10 of 14
Document Owner: Michael Graham Issue Date: 19-Apr-2015
Revision Date: 30-Mar-2021
A3.4 Task Groups may select failure criteria outside the limits established by the
95th and 98th percentiles based on the size and maturity of the Task Group.
A3.4.1 A rationale for failure criteria selected outside the limits established by the
95th and 98th percentiles shall be noted in meeting minutes.
A3.5 Task Group approval of Mode B Failure Criteria shall be recorded in the
meeting minutes.
A3.6 When new failure criteria are approved, Table 1 shall be updated.
A3.6.1 Changes to Table 1 are not required to be balloted per OP 1101 Document
Control.
Table 1
Reaccred
1 2 3 4 5
Major 2 2 - - - Y
Total 3 3 - - - Y
CP Initial
1 2 3 4 5+
Major 2 3 5 6 6 Y
Total 5 9 13 18 18 Y
Reaccred
1 2 3 4 5+
Major 2 3 5 6 6 N
Total 3 6 9 12 12 Y
COMP Initial
1 2 3 4 5
Major 2 3 4 5 7 Y
Total 4 7 10 12 15 Y
Reaccred
1 2 3 4 5
Major 1 2 3 4 5 Y
Total 2 5 7 10 12 Y
CMSP Initial
1 2 3 4 5
Major 2 4 4 5 6 Y
Total 3 6 8 9 10 Y
Reaccred
1 2 3 4 5
Major 1 2 2 3 3 Y
Total 3 6 6 6 6 Y
CT Initial
1 2 3 4 5
Major 2 3 4 5 6 Y
Total 5 9 14 19 23 Y
Reaccred
1 2 3 4 5
Major 2 3 4 5 6 Y
Total 4 7 10 13 16 Y
Nadcap OPERATING PROCEDURE OP 1110
AUDIT FAILURE AND RISK MITIGATION Page 12 of 14
Document Owner: Michael Graham Issue Date: 19-Apr-2015
Revision Date: 30-Mar-2021
Reaccred
1 2 3 4 5
Major 1 2 4 6 8 N
Total 3 6 9 12 15 Y
FLU Initial
1 2 3 4 5
Major 2 4 6 6 6 N
Total 3 6 9 12 15 Y
Reaccred
1 2 3 4 5
Major 2 4 6 6 6 N
Total 3 6 9 12 15 Y
HT Initial
1 2 3 4 5
Major 2 4 6 8 8 Y
Total 3 6 9 12 12 Y
Reaccred
1 2 3 4 5
Major 1 2 3 4 4 Y
Total 2 5 7 9 9 Y
MTL Initial
1 2 3 4 5+
Major 2 3 5 6 7 Y
Total 8 15 22 29 36 Y
Reaccred
1 2 3 4 5+
Major 1 2 3 4 5 Y
Total 4 8 12 16 20 Y
M&I Initial
1 2 3 4 5+
Major 2 3 5 6 7 N
Total 5 10 15 20 25 Y
Reaccred
1 2 3 4 5+
Major 2 2 3 4 5 Y
Total 3 8 11 14 18 Y
Nadcap OPERATING PROCEDURE OP 1110
AUDIT FAILURE AND RISK MITIGATION Page 13 of 14
Document Owner: Michael Graham Issue Date: 19-Apr-2015
Revision Date: 30-Mar-2021
Reaccred
1 2 3 4 5
Major 1 2 3 4 5 Y
Total 3 6 9 12 15 Y
6 7 8 9 10
Major 6 7 8 9 10 Y
Total 18 21 24 27 30 Y
NMMM Initial
1 2 3 4 5
Major 1 1 2 2 2 Y
Total 3 5 7 10 12 Y
Reaccred
1 2 3 4 5
Major 1 1 2 2 2 Y
Total 2 4 5 7 9 Y
NMMT Initial
1 2 3 4 5
Major 1 1 2 2 - Y
Total 2 4 7 9 - Y
Reaccred
1 2 3 4 5
Major 1 1 2 2 - Y
Total 2 3 5 7 - Y
NM & SE Initial
1 2 3 4 5
Major 1 2 4 4 4 Y
Total 3 6 12 12 12 Y
Reaccred
1 2 3 4 5
Major 1 2 4 4 4 Y
Total 2 4 8 8 8 Y
Nadcap OPERATING PROCEDURE OP 1110
AUDIT FAILURE AND RISK MITIGATION Page 14 of 14
Document Owner: Michael Graham Issue Date: 19-Apr-2015
Revision Date: 30-Mar-2021
Reaccred
1 2 3 4 5
Major 4 4 6 - - N
Total 6 6 9 - - N
SLT Initial
1 2 3 4 5
Major 3 4 4 - - N
Total 6 10 10 - - N
Reaccred
1 2 3 4 5
Major 3 4 - - - N
Total 6 10 - - - N
WLD Initial
1 2 3 4 5
Major 1 2 3 3 3 Y
Total 3 5 8 8 8 Y
Reaccred
1 2 3 4 5
Major 1 2 3 3 3 Y
Total 2 4 6 6 6 Y