Baxter P.reese Calibration in Regulated Industries Federal Agency Use ISO-17025 ANSI Z540.3 28

Calibration in Regulated Industries:
Federal Agency use of ANSI Z540.3 and ISO 170251

Speaker/Author: Paul Reese
Baxter Healthcare Corporation
25212 West Illinois Route 120
Mail Stop WG2-2S
Round Lake, IL 60073
Phone: (224) 270-4547 Fax: (224) 270-2491
e-mail: paul_reese@baxter.com
Abstract
ANSI/NCSL Z540.3-2006 and ISO/IEC 17025:2005 are voluntary consensus standards which
prescribe requirements for the calibration of measuring and test equipment and for the technical
competency of the performing laboratories. Many agencies in the U.S. which are part of, or
regulated by, the Federal Government are required to use instruments which have been calibrated
in accordance with one or both of these standards. The National Technology and Transfer
Advancement Act (NTTAA) of 1995 encourages all federal agencies to use technical standards
that are developed by consensus standards bodies, in lieu of “government-unique” standards.
ISO 17025 and ANSI Z540.3 have evolved over a half-century of metrological advancement,
drawing upon expertise in the public and private sector. They are now supported by a mature
infrastructure of third party assessment and accreditation that facilitates mutual recognition and
global trade, ensuring calibrations are accepted worldwide. However, some federal agencies and
regulatory bodies in the U.S. do not yet recognize these standards. Calibrations are commonly
performed which may not conform to these requirements. This paper discusses the history of risk
mitigation techniques applied to products and processes when declaring in-or-out of tolerance
conditions. Particular focus is given to the Food and Drug Administration’s (FDA) regulation of
calibration requirements in the Quality System Regulation (QSR) found in Title 21 Part 820 of
the Code of Federal Regulations (CFR). Currently, a paucity of official guidance exists with
respect to what constitutes an acceptable calibration program in medical device and
pharmaceutical industries. Ambiguities may persist due to lack of recognition of consensus
standards such as ISO 17025 and Z540.3 as guidance documents. Fundamental requirements
such as traceability, measurement uncertainty, measurement decision-rules, as well as basic
metrological definitions are somewhat ill-defined in the CFR. The objective of this paper is to
provide relevant background information and to encourage constructive dialogue between
government agencies, standards writing committees, industry partners, and third party
assessment/accreditation bodies. Cooperation of this type is consistent with public law and
White House policy objectives. Ultimately, such dialogue may foster recognition of these
voluntary consensus standards as guidance documents for calibration in regulated industries,
resulting in regulatory efficiencies, improved quality, and reduced risk to consumers or patients.
Note: Throughout this paper, “emphasis” is denoted via red-highlight and/or underline for quoted references. Where applied,
emphasis is the author’s and not that of the original source document.
1 th
Revised July 31, 2016. Originally submitted May 24 , and published in the July 2016 NCSLI Proceedings.
2016 NCSL International Workshop & Symposium

1. Background
Many federal agencies, in diverse roles, have established minimum quality standards and
regulatory requirements governing the calibration of test and measurement equipment. These
agencies differ from commercial non-regulated industries in several respects. Requirements of
most federal agencies are codified into law and non-compliance with such rules and regulations
can result in legal action, particularly where regulatory agencies are concerned. For example,
medical device and pharmaceutical manufacturers, regulated by the FDA, must comply with
Title 21 of the Code of Federal Regulations (CFR), including the calibration requirements of
section §820.72 (see Table 2) and other applicable sections [1]. Non-compliance can subject a
manufacturer to warning letters, recalls, injunctions, seizures, and/or debarment. Federal
agencies exist where legislation has been enacted to create them and where the criticality,
importance, or cost of such a program is deemed too great to relegate the responsibility to private
industry. USA.gov currently lists more than 600 different federal agencies, each charged with
matters related to the general public well-being, with some establishing their own quality and
calibration standards. Examples of Federal Agencies include:
 National Aeronautics and Space Administration (NASA)

 Food and Drug Administration (FDA)
 Environmental Protection Agency (EPA)
 Department of Defense (DOD)
 Department of Energy (DOE)
 Nuclear Regulatory Commission (NRC)
 Bureau of Alcohol Tobacco Firearms and Explosives (ATF)
 National Security Agency (NSA)
 Department of Transportation (DOT)
It has been the aspiration of government and industry alike to standardize on quality
requirements for more than half a century. The largest standardization effort in quality history
began with the wide spread adoption of the ISO 9000 series in 1987, the documentary roots of
which began in the 1950’s. See Table 1 and Figures 1 and 2. Standardization has also been the
goal with respect to the calibration requirements of test and measuring equipment used to
manufacture goods or monitor critical processes in a broad range of industries. Groups
developing voluntary consensus standards for calibration requirements have worked diligently to
aid in this endeavor. A cursory review of Table 1 will reveal a litany of documents, developed
over 65 years, governing calibration in both the public and private sector. Figure 1 provides a
simplified graphical evolution of these historical standards and guidelines, with a particular focus
on the FDA-regulated medical device and healthcare industry. The calibration documents in
Table 1 can generally be divided into two categories:
 Government-unique standards (e.g. FDA, DoE, DoD, NASA, NRC, etc.)

 Voluntary consensus standards (e.g. ISO, ASTM, ANSI, ASME, IEEE, etc.)
Full document titles for these standards are provided in Table 1 such that publications may
henceforth be referenced simply by their document numbers for convenience.

Table 1A. Chronology of calibration system standards and guidelines
YEAR DOCUMENT NUMBER TITLE OF DOCUMENT
1950 MIL-STD-120 Military Standard, Gage Inspection [Chng#1 added in 1963; MIL-STD-120 was cancelled in 1996]
1955 Tech Memo 63-106 Navy: Factors Affecting Measurement Reliability [Jerry Hayes, Naval Ordnance Lab – Oct 24, 1955]
1958 AFR 74-2 Air Force Metrology and Calibration (AFMETCAL) Program [Established Project “Test-Shop” for PMELs]
1958 SLIM Navy: Standards Laboratory Information Manual [BuWepsRep – Pomona, CA, Feb 1958, Revised July 1959]
1959 SECNAV 4355.11 Navy Metrology and Calibration (METCAL) Program [March 1959; Also NavOrd 4355.28 & BuWeps 4355.5]
1960 BU-520 USAF Bulletin No. 520: Calibration and Certification of Measuring and Testing Equipment [May 17, 1960]
1960 MIL-C-45662 (Ord) Military Specification: Calibration of Standards
1962 MIL-C-45662A Military Specification: Calibration System Requirements [Renamed; Supersedes MIL-C-45662 (1960)]
1962 SECNAV 4355.11A Navy Metrology and Calibration (METCAL) Program [January 16, 1962]
1962 MIL-C-55163 Calibration of Test and Measuring Equipment [Army: Cancelled in 1967]
1962 NASA NPC 200-2 Quality Program Provisions for Space System Contractors [Chap 9 – Inspection, Measuring, & Test Equipment]
1962 AR 750-25 Army Test, Measurement, and Diagnostic Equipment Calibration and Repair Support Program
1962 T.O. 33-1-14 Air Force: Repair, Calibration and Certification of Precision Measurement Equipment [October 1962]
1963 FDA 21 CFR 133.4* Code of Federal Regulations: cGMP – Equipment, [Note: No equipment calibration requirements]
1964 MIL-HDBK-52 Military Handbook – Evaluation of a Contractor’s Calibration System [for use with MIL-C-45662A]
1965 NHB 5300.2 NASA Handbook – Apollo Metrology Requirements Manual
1965 MIL-C-24133 Preparation of Calibration Procedures [Navy – Space and Naval Warfare Systems Command]
1969 RDT F3-2T Calibration System Requirements [Atomic Energy Commission, ORNL Feb 1969; See also NE F3-2T - 1973]
1969 NBS SP 300-1 Precision Measurement and Calibration – Statistical Concepts and Procedures [NIST]
1969 MIL-C-24133A Preparation of Calibration Procedures [Navy; cancelled in 1970; superseded by MIL-M-38793]
1969 MIL-M-38793 Performance Specification: Manuals, Technical: Calibration Procedures, Preparation of
1969 SECNAV 4355.11B Navy Metrology and Calibration (METCAL) Program [July 16, 1969]
1970 AEC 10 CFR 50 Appendix B, XII Code of Federal Regulations: AEC – Control of Measuring and Test Equipment [Basic Requirement 12]
1970 NATO AQAP-6 NATO Measurement and Calibration System Requirements for Industry [December 1, 1970]
1971 NCSL RP-4 Recommended Practice: Calibration System Specification [Withdrawn: Replaced by ANSI/NCSL Z540.1-1994]
1972 NATO AQAP-7 Guide for the Evaluation of a Contractor’s Calibration System for Compliance with AQAP-6
1973 SECNAV 4355.11C Department of the Navy Metrology and Calibration (METCAL) Program [November 7, 1973]
1973 DEFSTAN 05-26/1 Measurement and Calibration System Requirement for Industry
Guide for the Evaluation of a Contractor’s Measurement and Calibration System for Compliance with
1973 DEFSTAN 05-27/1 DEFSTAN 05-26
1973 NE F3-2T Calibration Program Requirements [DoE; Supersedes RDT F3-2T – 1969. NE F3-2T cancelled July 1996]
1975 EIA QB4-1975 Calibration System Requirements [Electronic Industries Alliance]
IEEE STD 498-1975 IEEE Standard Supplementary Requirements for the Calibration and Control of Measuring and Test Equipment
1975 Used in the Construction and Maintenance of Nuclear Power Generating Stations [Revised in 1980]
(ANSI N45.2.16)
1976 ASTM E548-76 Recommended Practice for Generic Criteria for Use in the Evaluation of Testing and Inspection Agencies.
1976 DEFSTAN 05-26/2 Measurement and Calibration System Requirement for Industry
1976 NATO AQAP-6 (2nd Ed) NATO Measurement and Calibration Systems Requirements for Industry [July 1976; Cancelled March 7, 1995]
1976 DEFSTAN 05-32/1 Quality Control Requirements for Test House and Laboratory Organizations [Withdrawn 1982: See BS 6460-1]
Guide for the Evaluation of a Contractor’s Measurement and Calibration System for Compliance with
1978 DEFSTAN 05-27/2 DEFSTAN 05-26
1978 ISO Guide 25-1978 (1st Ed) Guidelines for Assessing the Technical Competence of Testing Laboratories [October 1, 1978]
1978 ISO Guide 24:1978 Guidelines for the Acceptance of Testing and Inspection Agencies by Certification Bodies
1978 NATO AQAP-7 Guide for the Evaluation of a Contractor’s Calibration System for Compliance with AQAP-6
1978 FDA 21 CFR 820.61* Code of Federal Regulations: cGMP - Measurement Equipment [Supersedes 21 CFR 133.4, 1963]
Device Good Manufacturing Practices – An Industry Self-Inspection Program [pp 30 – 34, 820.61 Measurement
1978 FDA GMP Self Inspection Program* Equipment]
Device Good Manufacturing Practices – A Quality Audit Program for Industry [pp 88-91, 820.61 Measurement
1979 FDA GMP Quality Audit Manual* Equipment]
1979 BS 5781-1:1979 Measurement and Calibration Systems. Specification for System Requirements
1979 NCSL RP-2 Recommended Practice: Evaluation of Measurement Control Systems and Calibration Laboratories
1980 MIL-STD-45662 Military Standard: Calibration System Requirements [Superseded by MIL-STD-45662A in 1988]
1980 AS 2415-1980 Calibration System Requirements [Australia – Superseded by AS 3912.1-1993]
IEEE Standard Requirements for the Calibration and Control of Measuring and Test Equipment Used in the
1980 IEEE-498-1980 Construction and Maintenance of Nuclear Power Generating Stations [Revised in 1985]
1981 NMI 5330.9A NASA Management Instruction, Metrology and Calibration
1981 BS 5781-2:1981 Quality Assurance Requirements for Measuring Equipment. Guide to the use of BS 5781-1
1982 ISO/IEC Guide 25:1982 (2nd Ed) General Requirements for the Technical Competence of Testing Laboratories
1982 FDA GMP Workshop Manual* Device Good Manufacturing Practices – A Workshop Manual [Chap. 4 Buildings, Equipment & Calibration]
Accreditation of Testing Laboratories. Specification of the General Requirements for the Technical Competence
1983 BS 6460-1:1983 of Testing Laboratories [Withdrawn 23 Nov 2007. See ISO 17025:2005]
1983 DEFSTAN 05-55/2 Ministry of Defence Calibration Laboratories Operation and Management
1983 UKAS/NAMAS M10 General Requirements for the Competence of Testing Laboratories [See ISO Guide 25]
Table 1B. Chronology of calibration system standards and guidelines (continued)
1984 MIL-HDBK-52A Military Handbook – Evaluation of a Contractor’s Calibration System [for use with MIL-STD-45662]
1984 ASTM E548-84 Standard Practice General Criteria for Use in the Evaluation of Testing and Inspection Agencies
Maintenance of Supplies and Equipment Army Test, Measurement, and Diagnostic Equipment (TMDE)
1984 TB 750-25 Calibration and Repair Support (S&RS) Program
1984 FDA 85-4179 (3rd Ed)* Medical Device Good Manufacturing Practices Manual [Chap. 4 Equipment and Calibration]
Requirements for the Calibration and Control of Measuring and Test Equipment Used in the Construction and
1985 IEEE-498-1985 Maintenance of Nuclear Power Generating Stations [Supersedes IEEE-498-1980; Replaced by IEEE-498-1990]
1986 MIL-STD-1839 Military Standard – Calibration and Measurement Requirements
1986 NCSL RP-6 (1st Ed)** Recommended Practice – Medical Device and Pharmaceutical Industry Calibration Control System
1987 FDA 87-4179 (4th Ed)* Medical Device Good Manufacturing Practices (GMP) Manual [Chap. 5, Equipment and Calibration]
1987 ANSI/ASQC M1-1987 American National Standard for Calibration Systems [Reaffirmed in 1996 as ANSI/ASQC M1-1996]
1988 MIL-STD-45662A Military Standard: Calibration System Requirements [Supersedes MIL-STD-45662 (1980); Cancelled in 1995]
1988 MIL-STD-1839A Military Standard – Calibration and Measurement Requirements
1988 DEFSTAN 05-55/3 Ministry of Defence Calibration Laboratories Operation and Management
1989 EN 45001:1989 General Criteria for the Operation of Testing Laboratories [See ISO Guide 25]
1989 EN 45002:1989 General Criteria for the Assessment of Testing Laboratories [See ISO Guide 25]
1989 BS 7501:1989 General Criteria for the Operation of Testing Laboratories [See ISO Guide 25]
1989 BS 7502:1989 General Criteria for the Assessment of Testing Laboratories [See ISO Guide 25]
1989 MIL-HDBK-52B Military Handbook – Evaluation of a Contractor’s Calibration System [for use with MIL-STD-45662A]
1989 OIML G7 Guide to Calibration [International Organization for Legal Metrology]
1990 ISO/IEC Guide 25:1990 (3rd Ed) General Requirements for the Competence of Calibration and Testing Laboratories [See ISO 17025:1999]
1990 DOE 4330.4A DoE Maintenance Management Program [Section 12 – Control & Calibration of Measuring & Test Equipment]
1990 NAVAIR 17-35TR-4 Requirements for Preparation of Instrument Calibration Procedures
1991 FDA 91-4179 (5th Ed)* Medical Device Good Manufacturing Practices (GMP) Manual [Chap. 5, Equipment and Calibration]
1991 ASTM E548-91 Standard Guide for General Criteria Used for Evaluating Laboratory Competence [See ISO Guide-25]
1991 NHB 5330.9(1A) NASA Handbook: Metrology, Calibration & Measurement Processes Guidelines [see also NMI 5330.9]
Guidelines for third-party certification and accreditation – Guide 25- General Requirements for the Competence
1991 HB 18.25-1991 of Calibration and Testing Laboratories [Australia: Superseded by ISO 17025:1999]
1991 NCSL RP-11 Recommended Practice: Reports and Certificates of Calibration [Replaced by ANSI/NCSL Z540.1-1994]
1991 IEEE-498-1990 Maintenance of Nuclear Power Generating Stations [Supersedes IEEE-498-1985. Withdrawn 7/14/1995]
1992 ANSI/ASQC M2-1992 American National Standard for the Quality Control of Measurements
1992 TSE – TS 10006 Military Quality Assurance Systems – Measurement and Calibration System Requirements for Industry [Turkey]
Quality Assurance Requirements for Measuring Equipment – Metrological Confirmation System for Measuring
1992 BS 5781-1:1992 Equipment [See ISO 10012-1:1992]
Quality Assurance Requirements for Measuring Equipment Part 1 – Metrological confirmation system for
1992 ISO 10012-1:1992 measuring equipment [AS 3912.1-1993 in Australia, which superseded AS 2415-1980]
Calibration and Testing Laboratory Accreditation Systems — General Requirements for Operation and
1993 ISO/IEC Guide 58:1993 Recognition [Superseded by ISO/IEC 17011:2004]
1993 MIL-PRF-38793A Performance Specification – Technical Manuals: Calibration Procedures – Preparation
Guideline to Good Practices for Control and Calibration of Measuring and Test Equipment (M&TE) at DoE
1993 DOE-STD-1054-93 Nuclear Facilities [cancelled in 2001]
Guideline for third-party certification and accreditation – Guide 58- Calibration and testing Laboratory
1993 HB 18.58-1993 Accreditation Systems – General Requirements for Operation and Recognition
1994 ILAC-G5:1994 Guidelines for Calibration and Maintenance of Test and Measuring Equipment
Quality Assurance Requirements for Measuring Equipment. Metrology Confirmation System for Measuring
1994 BS EN 30012-1:1994 Equipment [See BS 5781-1:1992 and ISO 10012-1:1992. Superseded by ISO 10012:2003]
1994 NASA RP-1342 Metrology – Calibration and Measurement Processes Guidelines [NASA Reference Publication]
1994 NMI 5330.9B NASA Management Instruction: Metrology Calibration and Measurement Processes [Converted to NPD 8730.1]
DoE Maintenance Management Program [Section 12 – Control & Calibration of Measuring & Test Equipment.
1994 DOE 4330.4B Cancelled in 1996]
1994 ANSI/NCSL Z540.1-1994 Calibration Laboratories and Measuring and Test Equipment – General Requirements
1995 NCSL Z540.1 Handbook Handbook for the Application of ANSI/NCSL Z540-1-1994
1995 ASTM E548-94(E1) Standard Guide for General Criteria Used for Evaluating Laboratory Competence [See ISO Guide-25]
1995 MIL-STD-1839B DoD Standard Practice for Calibration and Measurement Requirements
1995 MIL-STD-45662A (Cancelled) Note: Cancelled. Relegated requirements to ANSI Z540-1:1994 and ISO-10012-1:1992
1995 MIL-HDBK-52B (Cancelled) Note: Cancelled without replacement [last revised 1989]
1995 NATO AQAP-6 (Cancelled) Note: Cancelled. Superseded by ISO 10012-1:1992
1995 IEEE-498-1990 (Cancelled) Note: Cancelled 7/14/1995by IEEE Nuclear Power Engineering Committee
1996 MIL-STD-120 (Cancelled) Note: Cancelled without replacement [last revised 1963]
1996 NE F3-2T (Cancelled) Note: Cancelled – No longer needed per Department of Energy [last revised Feb 1973]

Table 1C. Chronology of calibration system standards and guidelines (continued)
1996 MIL-HDBK-1839 DoD Handbook – Calibration and Measurement Requirements
1996 ANSI/ASQC M1-1996 American National Standard for Calibration Systems [Reaffirmation of ANSI/ASQC M1-1987]
Handbook: A Guide to the Selection, Care, Calibration and Checking of Measuring Instruments
1996 SAA/SNZ HB-86.1:1996 in Industry Part 1:General Principles [Australia / New Zealand]
Medical Device Quality Systems Manual: A Small Entity Compliance Guide [Chapt.7, Equipment and
1996 FDA 97-4179 (1st Ed)* Calibration. Note: 1th Edition (renamed); supersedes FDA 91-4179, 5th edition 1991]
Code of Federal Regulations: QSR – Inspection, Measuring, and Test Equipment [supersedes FDA 21 CFR
1997 FDA 21 CFR 820.72* 820.61, 1978. QSR harmonized with ISO 9001:1994 and ISO-13485:1996 CD in 1997]
1997 ISO 10012-2:1997 Quality Assurance for Measuring Equipment Part-2: Guidelines for Control of Measuring Processes
1997 MIL-PRF-38793B Performance Specification – Technical Manuals: Calibration Procedures – Preparation
Maintenance of Supplies and Equipment Army Test, Measurement, and Diagnostic Equipment (TMDE)
1997 TB 750-25 Calibration and Repair Support (S&RS) Program
1998 NPD 8730.1 NASA Policy Directive: Metrology and Calibration [Replaced NMI 5330.9B. Superseded by NPD 8730.1B]
General Requirements for the Competence of Testing and Calibration Laboratories [Supersedes ISO Guide
1999 ISO 17025:1999 (1st Ed) 25:1990]
1999 NCSL RP-6 (2nd Ed)** Recommended Practice – Calibration Control Systems for the Biomedical and Pharmaceutical Industry
2000 MIL-STD-1839C DoD Standard Practice for Calibration and Measurement Requirements
2000 MIL-HDBK-1839A DoD Handbook – Calibration and Measurement Requirements
NRC – Nuclear Procurement Issues Committee: Commercial Grade Calibration Services Implementation
2000 NUPIC Document 29 (Rev.1) Guidelines [Applicable to NUPIC Document 28, “Commercial Grade Calibration Services Checklist” – undated]
2002 ISPE GAMP (1st Ed)** Good Practice Guide: Calibration Management
2002 ANSI/NCSL Z540.1-1994 (R2002) Calibration Laboratories and Measuring and Test Equipment – General Requirements [Reaffirmed]
2002 TB 43-180 Calibration and Repair Requirements for the Maintenance of Army Material
Measurement Management Systems – Requirements for Measurement Process and Measuring Equipment
2003 ISO 10012:2003 [Supersedes ISO 10012-1:1992 & ISO 10012-2:1997. AS/NZS 10012:2004 supersedes AS3912.1-1993]
Conformity assessment -- General requirements for accreditation bodies accrediting conformity assessment
2004 ISO/IEC 17011:2004 bodies [Supersedes ISO/IEC Guide 58:1993]
2004 NPD 8730.1B NASA Policy Directive: Metrology and Calibration [Superseded by NPD 8730.1C]
2005 TB 43-180 Calibration and Repair Requirements for the Maintenance of Army Material
2005 DI-QCIC-80798B Calibration Certificate
2005 ISO 17025:2005 (2nd Ed) General Requirements for the Competence of Testing and Calibration Laboratories
NRC – Nuclear Procurement Issues Committee: Commercial Grade Calibration Services Implementation
2006 NUPIC Document 29 (Rev.2) Guidelines [Supersedes NUPIC Document 29, Rev.1 – 2000. See also NUPIC Doc. No. 28]
Measurement and Calibration System Requirement for Ministry of Defence Test & Measurement Equipment
2006 DEFSTAN 05-55(PT1)/1 Part 1: Ministry of Defence Calibration Laboratories Operation and Management
2006 DEFSTAN 05-55(PT2)/1 Part 2: Unit Level Test
2006 DEFSTAN 05-55(PT3)/1 Part 3: Sub Contract of Calibration
Note: First published as M1-1987, M1-1996 was not reaffirmed within 10 years of last reaffirmation; still
2006 ANSI/ASQC M1-1996 (Expired) available for reference via ASQ Catalog.
2006 ANSI/NCSL Z540.3-2006 Requirements for the Calibration of Measuring and Test Equipment
2007 ANSI/NCSL Z540.1 (Withdrawn) Note: Part I replaced by ANSI/ISO/IEC 17025:2005. Part II replaced by ANSI/NCSL Z540.3-2006
2008 NCSL RP-6 (3rd Ed)** Recommended Practice – Calibration Quality Systems for the Healthcare Industries
2008 MIL-PRF-38793C Performance Specification – Technical Manuals: Calibration Procedures – Preparation
Handbook for the Application of ANSI/NCSL Z540.3-2006 – Requirements for the Calibration of
2009 NCSL Z540.3 Handbook Measuring and Test Equipment
2010 MIL-STD-1839D DoD Standard Practice – Calibration and Measurement Requirements
2010 NATO STANAG 4704 NATO Requirements for Calibration Support of Test and Measurement Equipment
2010 ISPE GAMP (2nd Ed)** Good Practice Guide: A Risk-Based Approach to Calibration Management [Replaces 2002 version]
2011 NPD 8730.1C NASA Policy Directive: Metrology and Calibration [Supersedes NPD 8730.1B. Expires 2016-Jun-27]
2011 T.O. 00-20-14 USAF Air Force Metrology and Calibration Program [previous versions too numerous to list]
2012 NCSL RP-21 Recommended Practice: Assessment of ANSI/NCSL Z540.3-2006 Sub-clause 5.3 [false accept risk]
2013 FDA 97-4179 (Withdrawn)* Note: Withdrawn without comment or replacement [Chapt. 7, Equipment and Calibration]
2013 ANSI/NCSL Z540.3-2006 (R2013) Requirements for the Calibration of Measuring and Test Equipment [Reaffirmed]
2014 AR 750-43 Maintenance of Supplies and Equipment: Army Test, Measurement, and Diagnostic Equipment
2015 DIN VG-96910:2015 Documentation of Calibration Services [Germany]
2015 NCSL RP-6 (4th Ed)** Recommended Practice for Calibration Quality Systems for the Healthcare Industries
2016 ISO 17025:2017 (CD2 Draft) General Requirements for the Competence of Testing and Calibration Laboratories [Committee Draft 2]
*FDA Requirements or GMP Guidance; ** Guidance via private sector healthcare industry (NCSLI or ISPE/GAMP)

1950
Government Standards & Guides

MIL-STD-120
1955
1963
Army: AR 750-25
FDA 21 CFR
Air Force: AFR 74-2 & BU-520
§133.4 (GMP)
(for calibration)
Navy: SLIM & SecNav 4355.11 No Cal Requirements
Navy: Tech Memo 63-106
1978
1970 1960
FDA 21 CFR
NATO AQAP-6 MIL-C-45662 §820.61 (GMP)
NATO AQAP-7 MIL-HDBK-52
1973 1976 1997
DEFSTAN 05-26 FDA 21 CFR

DEFSTAN 05-32
DEFSTAN 05-27 §820.72 (QSR)
1979 1976 1971

1987
BS 5781-1 ASTM E548 NCSL RP-2
(Quality Standards)
Voluntary Consensus Standards & Guides

BS 5781-2 NCSL RP-4
ISO 9000 Series
(E36 Committee)
NCSL RP-11
~1977
1992 1987
ISO CERTICO[1]
BS EN 30012-1 (Now CASCO) ANSI/ASQC M1
ISO 10012-1 Guidelines ANSI/ASQC M2
ISO 10012-2
ISO 10012*
(for calibration)
1978
ISO Guide 25
1999 ASTM E548 1994
UKAS M10 ANSI/NCSL Z540.1

ISO/IEC 17025* BS 6460-1 & Z540.1 Handbook
HB 18.25
BS 7501/7502
EN 45001/45002
2006
ANSI/NCSL Z540.3*
& Z540.3 Handbook
* Asterisk designates active voluntary consensus standards for calibration.

[1]
CERTICO was created in 1970 as the “Committee on Certification”. In 1985, it became CASCO or “Committee on Conformity Assessment”
Dates shown reflect initial publication of earliest document; see Table 1 for revision histories. Dotted lines indicate “indirect” predecessors.
Figure 1. Simplified evolution of international and national calibration standards and guidelines

Military & Aerospace Quality Standards 1950 – 1967
MIL-STD-120 (1950) Military Standard, Gage Inspection
Government-Unique Quality & Calibration System Standards

MIL-Q-5923 (1950) General Equipment Quality Control Requirements
MIL-G-14461 (1957) General Specification for Quality Control
MIL-Q-9858 (1959) Quality Control System Requirements (MIL-Q-9858A , Quality Program Requirements - 1963)
MIL-Q-21549A (1959) Product Quality Program Requirements for Fleet Ballistic Missile Weapon System Contractors
MIL-C-45662 (1960) Calibration of Standards (MIL-C-45662A, Calibration System Requirements in 1962) (See NATO AQAP-6)
MIL-I-45208 (1961) Inspection System Requirements
MIL-H-110 (1960) Quality Control and Reliability Handbook
NPC 200-2 (1962) Quality Program Provisions for Space System Contractors (NHB 5300.4(1B)). See also Apollo RA-001-007-1 (1966)
NPC 200-3 (1962) Inspection System Provisions for Suppliers of Space Materials, Parts, Components, and Services (NHB 5300.4(1C))
MIL-C-55163 (1962) Calibration of Test and Measuring Equipment
MIL-HDBK-52 (1964) Evaluation of a Contractor’s Calibration System (See NATO AQAP-7)
MIL-HDBK-50 (1965) Evaluation of a Contractor’s Quality Program
MIL-HDBK-51 (1967) Evaluation of a Contractor’s Inspection System
(See also American Standards Association (now ANSI) Z1.1, Z1.2, Z1.3 (1941/1942) and BS 9000 (1967) )
Allied Quality Assurance Publications (NATO) 1968 - 1972

NATO AQAP-1 (1968) Quality Control System Requirements for Industry
NATO AQAP-2 (1968) Guide for the Evaluation of a Contractor’s Quality Control System for Compliance with AQAP-1
NATO AQAP-4 (1970) Inspection System Requirements for Industry
NATO AQAP-5 (1972) Guide for the Evaluation of a Contractor’s Inspection System for Compliance with AQAP-4
NATO AQAP-6 (1970) Measurement and Calibration Systems Requirements for Industry (See DEFSTAN 05-26 & 05-27)
NATO AQAP-7 (1972) Guide for the Evaluation of a Contractor’s Calibration System for Compliance with AQAP-6
NATO AQAP-9 (1970) Basic Inspection Requirements for Industry
(See also BS 4891 (1972) A Guide to Quality Assurance)
British Ministry of Defence Quality Standards (MoD) 1973 - 1976

DEFSTAN 05-21/1 (1973) Quality Control System Requirements for Industry
DEFSTAN 05-22/1 (1973) Guide for the Evaluation of a Contractor’s Quality Control System for Compliance with DEFSTAN 05-21
DEFSTAN 05-24/1 (1973) Inspection System Requirements for Industry
DEFSTAN 05-25/2 (1976) Guide for the Evaluation of a Contractor’s Inspection System for Compliance with DEFSTAN 05-24/2
DEFSTAN 05-26/1 (1973) Measurement and Calibration System Requirement for Industry (see BS 5781-1 and 5781-2)
DEFSTAN 05-27/1 (1973) Guide for the Evaluation of a Contractor’s Measurement and Calibration System for Compliance with DEFSTAN 05-26
DEFSTAN 05-29/1 (1973) Quality System Requirements for Final Inspection and Test
DEFSTAN 05-32/1 (1976) Quality Control Requirements for Test House and Laboratory Organizations (see ISO Guide 25, BS 6460-1, BS 7501/ 7502)
Calibration System Voluntary Consensus Standards

British Standards (BSI Group) 1974 1979 - 1982 1978
BS 5179 (1974) Guide to the Operation and Evaluation of Quality Assurance Systems:
 BS:5179-1 Part 1 – Final Inspection System BS 5781-1 (1979) ISO Guide 25 (1978)
Updated in 1982 & 1990
Quality System Voluntary Consensus Standards
 BS:5179-2 Part 2 – Comprehensive Inspection System Measurement and

 BS:5179-3 Part 3 – Comprehensive Quality Control System Calibration Systems: Guidelines for Assessing
Specification for System the Technical
British Standards (BSI Group) 1979 Requirements Competence of Testing
BS 5750 (1979) Quality Systems. Updated in 1987 Laboratories
BS 5750-1 Specification for Design, Development, Production, Installation and Servicing BS 5781-2 (1982) (See also ASTM E548,
BS 5750-2 Specification for Manufacture and Installation Quality Assurance BS 6460-1,
BS 5750-3 Specification for Final Inspection and Test Requirements for BS 7501 / 7502
Measuring Equipment – EN 45001 / 45002,
(See also ANSI/ASQ Z1.15-1979 Generic Guidelines for Quality Systems)
Guide to the use of BS HB 18.25-1991,
International Standards (ISO) 1987 5781-1 and UKAS M10)
ISO-9000 Series (1987) Updated in 1994, 2000, 2005, 2008, & 2015
ISO 9000 Quality Management & Quality Assurance Standards – Guidelines for Selection 1999
and Use
ISO 9001 Quality Systems – Model for Quality Assurance in Design/Development, ISO 17025 (1999) Updated in 2005, CD2 2016 Draft
Production, Installation and Servicing
ISO 9002 Quality Systems – Model for Quality Assurance in Production and Installation General Requirements for the Competence of Testing
ISO 9003 Quality Systems – Model for Quality Assurance in Final Inspection and Test and Calibration Laboratories
ISO 9004 Quality Management and Quality System Elements – Guidelines (See also ISO 10012-1:1992, ISO 10012-2:1997,
(See also EN 29000 Series, ANSI/ASQC Q90 Series, etc.) BS EN 30012:1994, and ISO 10012:2003)
(See also ANSI/ASQC M1-1987, ANSI/ASQC M1-1996,
Current International Standards (ISO) 2015
ANSI/ASQC M2-1992, ANSI/NCSL Z540.1-1994 (R2002),
ISO-9000:2015 Quality Management Systems – Fundamentals and Vocabulary ANSI/NCSL Z540.3-2006 (R2013))
ISO 9001:2015 Quality Management Systems – Requirements
ISO 9004:2009 Managing for the Sustained Success of an Organization – A
Quality Management Approach (Note: ISO 9002 & 9003 are now obsolete)
1997
Medical Device International Standard 2016 FDA Harmonized
FDA 21 CFR §820
ISO-13485 Quality Systems (1996) Updated in 2003, and 2016. QSR in 1997 w/
Medical Devices – Particular Requirements for the Application of ISO 9001 ISO 9000:1994 & Quality System
13485:1996 (CD) Regulation (QSR)
(See also BS EN 46001:1997, ISO 13488:1996, EN 46002, etc.)
Figure 2. Divergence of ISO quality and calibration system standards

2. Historical Perspective
Born primarily out of the defense and aerospace quality efforts of the 1950’s and 60’s, stringent
quality and calibration requirements had to be met by industries wishing to supply goods and
services to these government agencies; see Figure 2. One of the most important and recurring
themes was how to ensure that when suppliers judged items to be In-Tolerance during calibration
or inspection, they actually were In-Tolerance. A common method to achieve a reasonable level
of confidence in such decisions was to require the calibration standards to be considerably more
accurate than the items they were being used to test or calibrate (i.e., test accuracy ratio or TAR).
Another method was to temporarily assign acceptance limits, which were tighter than the
tolerance limits, during the calibration process – a practice now called guardbanding.
Concepts of test accuracy (or uncertainty) ratios for calibration (e.g. 10:1 and 5:1), as well as
false-accept and false-reject risk, have existed since at least 1950, as evidenced in MIL-STD-120
[2]; see Appendix B. Origins of the ubiquitous 4:1 test accuracy ratio have been documented by
Castrup [3] and Mimbs [49] [A125] and attributed to Crandon and Hayes of the U.S. Navy in the
1950’s. Hayes [A5] published the first known statistical analysis of calibration quality (false
accept/reject) in 1955, invoking the concept of accuracy ratios, based primarily on the 1954
seminal works of Eagle [A1] and Grubbs & Coon [A2]. Concurrently in 1953-1954, Wiesen and
Clark [A3] at Sandia Corporation were independently carrying out similar analysis of how
accuracy ratios and guardband limits affect incorrect accept/reject decisions, as was Mandel [A4]
in the UK. They were followed shortly thereafter by Tingey & Merrill [A6] at the Atomic
Energy Commission in 1957. A few years later in 1959, Wiesen & Owen [A8] at Sandia along
with David et-al [A7] published additional work on the calculus of how accuracy ratios and
guardbanding impact incorrect pass/fail decisions or misclassification of items judged to be in-
or-out of tolerance. All used the bivariate normal joint probability density model to calculate
consumer and producer risk2. See Appendix A for a partial chronology of related works on
measurement decision risk for calibration and testing (pass/fail) decisions.
In the 1960’s, suppliers to federal agencies had to contend with a myriad of different calibration
requirements. Many of these government-unique documents espoused different calibration
quality mandates including different test accuracy ratios which had to be met. In most cases,
minimum accuracy ratios were required, rather than detailed statistical analysis of decision-risk.
See Appendix B for a history of some TAR requirements. Private industry sometimes struggled
to keep up with the proliferation of these government standards and the vast array of calibration
requirements they imposed, particularly those requiring specific test accuracy ratios. This was
reflected in statements 50 years ago by Condon [4], Russell [5], and Fruechtenicht [6].
2
In 1926, Shewhart addressed measurement errors as impacting ‘True Versus Observed Quality’.
In 1948 , Grubbs also investigated effects of measurement errors on observed product variability and discussed
isolating these variances into separate components, as did Bennett in Jan 1954.
In 1962, Traver at GE, and engineers at GM, developed the Gauge R&R Study which ANOVA is now commonly used
to estimate “gauge capability”, e.g. Precision to Tolerance Ratio (P/T), Signal-to-Noise Ratio (SNR), Discrimination
Ratio (DR), etc. However, as stated by Montgomery, “None of these quantities really describe the capability of the
gauge in any sense that is directly interpretable. The effective capability of a measuring system is best described in
terms of how well it discriminates between good and bad parts… The joint probability density function… is bivariate
normal… A very useful way to describe the capability of a measurement system is in terms of producer’s risk and
consumer’s risk”. Appendix H.5 in ISO GUM further describes why ANOVA Gauge R&R alone is insufficient.
In 1963, Eisenhart also published a seminal work, “Realistic Evaluation of the Precision and Accuracy of Instrument
Calibration Systems”, but did not directly address measurement decision risk.

Condon: NCSL (1966): [4]
“NASA’s policy on ratio-of-accuracy as stated in NPC 200-2 requires that ‘Within the state-of-the-art limitation,
the standards used for calibration of inspection, measuring, and test equipment shall have a tolerance no
greater than 10 % of the allowable tolerance for the equipment being calibrated.’ …many measurement
requirements are becoming so sophisticated that they approached the limits of the science of metrology. In
such cases, it becomes impossible to maintain the 10 to 1 ratio of accuracy in the calibration of the instrument”
Russell: NCSL (1966)

“…our discussion centered around the accuracy ratio of standard to instrument during measurement and
calibration operations… Basically, the problem revolves around the actual or implied requirement that the
accuracy of an instrument or standard used to measure a quantity, or to calibrate another instrument, shall be
10 times as accurate as the quantity or the instrument being calibrated.
There is also the implication that the 10-to-1 ratio of accuracy shall exist between every level or echelon in the
traceability chain for product to National Standards. This requirement could create an impossible situation…
Most contractors indicated that the ratio-of-accuracy requirements imposed upon them ranged from 10:1 to
4:1, or ‘state-of-the-art’. In nearly every case, they stated that the 10-to-1 requirement was considered
unrealistic from an economic as well as a practical point of view…
I am of the opinion that that there are too many documents that basically state parallel requirements, the
majority of which are, to a degree, unrealistic… Where 4-to-1 is maintained… the reliability of the calibrated
instrument accuracy is assured…”
Fruechtenicht: NCSL (1968): [6]

“Specifications often referenced or required by contracts that delineate calibration requirements are:
 MIL-Q-9858A
 MIL-I-45208A
 MIL-C-45662A
 MIL-C-55163 (Sig.C)
 MIL-Q-21549B (WEP)
 NASA NPC-200-1A
 NASA NPC-200-2
 MIL-I-45607
 MIL-I-8500B and
 MIL-Hand-Book 50, 51, and, 52.
These specifications differ as to the extent of the requirements for calibration and measurements. As
examples of confusion created, some specifications require adherence to a strict 10 to 1 accuracy ratio, while
others make no reference to such ratios. Due to the problems created, Government contract evaluation
agencies find it difficult to enforce and administer contractual provisions as envisioned by the procuring
agencies. It is important that NASA and DoD calibration system requirements be consolidated into one
specification…
Recommendations: That MIL-C-45662A be revised and adopted as the standard calibration specification to be
referenced in all Government contracts including small business and R&D contractors where measurements
are to be performed.”
Indeed, MIL-C-45662A [7] and its successors did become the ubiquitous calibration standard for
critical industries for more than 30 years. This was true even outside of government agencies
and their contractors. It was adopted by a multitude of industries in the U.S., particularly those
which operated a quality management system, even though MIL-C-45662 had begun as a
government-unique military standard. In 1982, the Food and Drug Administration [8] [9] stated:

FDA Device Good Manufacturing Practices – A Workshop Manual (1982): [8]
(Also published in the NCSL Newsletter, Dec. 1980) [9]
“The medical device GMP equipment requirements are based on MIL-C-45662 and were written to assure that
production and quality assurance measurement equipment (mechanical, electronic, automated) used in the
manufacture of medical devices is suitable for its intended use.”
Like the FDA calibration requirements in the Good Manufacturing Practices (GMP’s), many
Federal agencies developed their own government-unique standards and requirements based, to
some extent, on this venerable military standard. This included the Department of Energy
(DOE), the Nuclear Regulatory Commission (NRC), and the National Aeronautics and Space
Administration (NASA), among many others. General equipment calibration requirements were
eventually codified into law by many government agencies via the Code of Federal Regulations
(CFR). For example:
 FDA: 21 CFR §820.72 Inspection Measuring, and Test Equipment [1A]
 NRC: 10 CFR §50 B-XII Control of Measuring and Test Equipment [10]
 FAA: 14 CFR §145.109 Equipment, Materials, and Data Requirements [11]
MIL-C-45662 was first published in 1960 [12]. It had grown out of the military metrology and
calibration programs established just a few years earlier in the mid 1950’s. Together, MIL-C-
45662A in 1962 [7] and its companion MIL-HDBK-52 in 1964 [13] formed the developmental
foundation for future international/allied military calibration standards and guidelines. The
North Atlantic Treaty Organization’s (NATO’s) Allied Quality Assurance Publication AQAP-6
[14] and AQAP-7 [15] for calibration were developed in the early 1970’s based to some extent
these U.S. military requirements. Afterwards, the British Ministry of Defence (MoD) looked to
these NATO standards when developing their own DEFSTAN 05-26 [16] and DEFSTAN 05-27
[17] calibration standards in 1973. In turn, these British MoD calibration standards were
influential in the development of BS 5781-1:1979 [18] and BS 5781-2:1982 [19] which would
subsequently form much of the basis for ISO 10012-1:1992 [20] and ISO 10012-2:1997 [21]
(also see footnote 3 in the following section). In 2003, these two ISO documents were combined
into a single document, the current ISO 10012:2003, Measurement Management Systems –
Requirements for Measurement Process and Measuring Equipment [22]. See Figure 1.
Concurrent to this evolution, the first edition of ISO Guide 25 had been published in 1978 [23].
Guide 25 was developed by ILAC Task Force 1, established during the first ILAC meeting of
1977 in Copenhagen [24]. Guide 25 emerged from guidelines developed earlier by the ISO
Committee on Certification (CERTICO, formed 1970), now called the Committee on Conformity
Assessment (CASCO, renamed in 1985). ISO Guide 25 was also derived from ASTM E548-76
(Committee E36, formed 1973) and, to some degree, DEFSTAN 05-32 in 1976. These standards
addressed accreditation and quality requirements for “test house” and laboratory organizations.
Guide 25 was just 4 pages long when first published in 1978. It was revised in 1982 and again in
1990. This third revision, ISO Guide 25:1990 [25] became the primary source document for the
development of ISO 17025:1999 [26], with the current version being ISO 17025:2005 [27]. See
Figure 1. At the time of this writing, ISO 17025 is presently under revision again by CASCO
Working Group 44. A Committee Draft (CD1) was released in September 2015 for circulation
and comment, with CD2 released in March 2016 [28]. A Final Draft International Standard
(FDIS) is expected in late 2016 with final publication tentatively projected for September 2017.

3. The 1990’s: ISO Guide 25, MIL-STD-45662A, and ANSI/NCSL Z540.1
The previous background information sets the stage in the early 1990’s for the two3 competing
documents that U.S. calibration programs and laboratories were being driven to comply with:
 ISO Guide 25 (1990) voluntary consensus standard
 MIL-STD-45662A (1988) government-unique standard
While sharing many common objectives, there were significant differences in these documents.
Maintaining compliance to multiple calibration standards was a challenge for U.S. industry,
similar to the situation alluded to earlier in the late 1960’s. ISO Guide 25 was the prevailing
voluntary consensus standard for calibration labs in 1990. Meanwhile, the government-unique
MIL-STD-45662A [29] in 1988 had evolved substantially since its initial introduction in 1960 as
MIL-C-45662. Most notably perhaps, MIL-STD-45662A was the first version to formally
require the ubiquitous 4:1 Test Accuracy Ratio (TAR), although not specifically referenced by
name. Section 5.2 of MIL-STD-45662A required that:
MIL-STD-45662A (1988): [29]

“5.2 Adequacy of Measurement Standards. Measurement standards used… for calibrating M&TE and other
measurement standards shall be traceable and shall have the accuracy, stability, range and resolution
required for the intended use. Unless otherwise specified in contract requirements, the collective uncertainty
of the measurement standards shall not exceed 25 percent of the acceptable tolerance for each characteristic
being calibrated. The… calibration system description may include provisions for deviating for the uncertainty
requirements, provided the adequacy of the calibration is not degraded. All deviations must be documented.”
A brief summary of the TAR requirements throughout the life of MIL-STD-45662A and its
associated handbook is given.
 1960: MIL-C-45662: Required a 10:1 TAR [12]

 1962: MIL-C-45662A: No TAR specified [7]
 1964: MIL-HDBK-52: Required 4:1 to 10:1 TAR [13]
 1980: MIL-STD-45662: No TAR specified [30]
 1984: MIL-HDBK-52A: Mentions 10:1, 4:1, 3:1, 2:1, & 1:1 TAR examples only [31]
 1988: MIL-STD-45662A: Required 4:1 TAR [29]
 1995: Cancelled: Replaced by Z540.1-1994; retained 4:1 TAR [32]
MIL-STD-45662A and many of its predecessors had invoked test accuracy ratios as a risk
mitigation technique. This was done to ensure that calibration standards were adequate for their
intended use by limiting the risk of incorrect decisions during calibration. This was accomplished
by ensuring that the laboratory standards were sufficiently more accurate than the instruments
3
The ANSI/ASQC M1-1987 standard had been previously published in 1987 and was reaffirmed in 1996 [74]. A
history of its development was given by one of the M1/M2 authors (Schumacher) [32A] [32B]. However (arguably),
the M1 standard was not as widely adopted as the later ANSI/NCSL Z540.1-1994 standard, despite M1’s
comprehensive nature, a focus on “measurement assurance”, and co-authorship by Dr. Belanger of NBS/NIST. M2
was also influential in ISO-10012-2:1997. Appendix B provides a summary of M1’s treatment of accuracy ratios.
In 1993, the first edition of the ISO Guide to the Expression of Uncertainty in Measurement (GUM) was also
published, standardizing the way uncertainties were calculated internationally. The term Test Uncertainty Ratio
(TUR) slowly began to replace Test Accuracy Ratio (TAR). A distinction is made in the Z540.3-2006 Handbook [45]
between Test Uncertainty Ratio (TUR) and Test Accuracy Ratio (TAR), as elaborated by Mimbs [49].

they were employed to calibrate. Incorrect pass/fail decisions can result from excessive
measurement uncertainty (inaccuracy) associated with the laboratory’s standards and processes.
A ≥4:1 TAR had become a quality hallmark of calibration in the U.S., the popularity of which
would be difficult to overstate. See Appendix B.
By contrast to MIL-STD-45662A (1988), ISO Guide 25 did not contain any requirement or
mention of test accuracy ratio throughout its history from 1978 to 1990. This and several other
differences between ISO Guide 25 and MIL-STD-45662A caused calibration requirements in the
U.S. to become somewhat splintered into two different paradigms (ISO Guide 25 vs. MIL-STD-
45662A). Once again, many industries and calibration laboratories in the U.S. were struggling to
resolve the apparent conflict, often having to comply with both to satisfy a broad range of
interests. As reported by Held [33] in 1992 when quoting Randall [34]:
T. Held – Abbott Laboratories (1992): [33]
“In response to this need, the NCSL [National Conference of Standards Laboratories] formed a standing TQM
committee on calibration systems and tasked it with two immediate objectives:
1. Bring together the various government bodies which audit calibration quality systems for the purpose
of establishing a single American National Calibration Quality System; and
2. Promote the establishment of a nationally recognized calibration laboratory accreditation system.

This committee quickly evolved, gathering representatives from
 NIST
 FAA
 FDA
(The FDA has from the onset strongly supported the first of the two objectives, but has until only
recently withheld their support for the second. Currently, the FDA supports both of the TQM
Committee's objectives)
 NASA
 U.S. Dept of Energy (DoE)
 U.S Nuclear Regulatory Commission (NRC)
 U.S. Naval Warfare Assmt. Ctr.
 U.S. Army T.M.D.E.
 U.S. Air Force
 Defense Logistics Agency (DLA)
 Office of the Secretary of Defense (DoD)
 various private sector interests (Including Abbott Laboratories)
After having spent months comparing various American Quality System requirements with the new European
standards for compatibility, the committee overwhelmingly voted to adopt a proposed U.S. consensus
standard consisting of ISO/IEC Guide 25 and some additional requirements to make it compatible with other
European standards and satisfy some requirements important to U.S. industry and government agencies [i.e.
MIL-STD-45662A].
Note that the incorporation of ISO/IEC Guide 25 will include ISO 9000 compliance and international
recognition. A final draft of this document was written by Dr. Joe D. Simmons, Chief of Calibration Services at
NIST. This draft was reviewed and unanimously endorsed by the TQM Committee. At this point the TQM
Committee is seeking a group which will publish the standard for distribution and use.”

The NCSL TQM committee would eventually become the ANSI Z540 Committee, formally
known as the ANSI Accredited Standards Committee on General Requirement for Calibration
Laboratories and Measuring and Test Equipment. The document they developed would
ultimately be published as ANSI/NCSL Z540.1-1994, “American National Standard for
Calibration – Calibration Laboratories and Measuring and Test Equipment – General
Requirements” [35]. Although the FDA was a member of the NCSL TQM Committee, it was not
part of the final ANSI Z540 Committee for Z540.1 [35; see foreword]. However, additional
comments on the two objectives of the TQM Committee and the FDA’s position on these issues
were provided in 1992, also by Held [33]:
T. Held – Abbott Laboratories (1992): [33]

“Of interest within the U.S., is the position being taken by the FDA in their representation on the NCSL TQM
Committee. The FDA from the onset strongly supported the objective of developing a single American National
Calibration Control System Standard stating that they would probably use it as a supplement to existing
GMP/GLP's.
However, the FDA initially felt obligated to withhold their support of the concept of establishing and utilizing a
nationally recognized calibration laboratory accreditation system. The FDA was apprehensive that the
utilization of accredited laboratories could be perceived of as the FDA's transferring its legislative responsibility
to possible private for-profit third party laboratories. At the NCSL TQM Committee meeting of July 92' the FDA
announced a change in position. Their position now is that they support both of the TQM Committee's
objectives:
 the development of a single standard and
 laboratory accreditation.”
Efforts to assimilate ISO Guide 25 and MIL-STD-45662A into a single U.S. Calibration
Standard, and to develop a nationally recognized laboratory accreditation program are further
documented by Cigler [36] of NIST; a history of accreditation is given by Neumann [36A].
J. Cigler – NVLAP Calibration Laboratory Program (1993): [36]

“I am working closely with the NCSL TQM Committee to tailor ISO Guide 25 to the needs of U.S. Government
and private industry. It should be noted that, with very minor exceptions, nothing is being tailored out of the
basic ISO Guide 25 requirements. Rather, an attempt is being made to add requirements which satisfy
government audit requirements (for example, MIL-STD-45662A for the Department of Defense).
This is being accomplished through the NCSL TQM Committee in the form of a proposed U.S. Standard
currently titled ‘General Requirements for Calibration Laboratories and Measuring and Test Equipment’ [Z540].
When completed, it will serve as this country’s interpretation and implementation of ISO Guide 25 in
determining the competence of calibration laboratories as part of the NVLAP accreditation process.”
Schumacher in 1995 [32B] and Harris in 2001 [36B] would later recount the efforts of the NCSL
TQM Committee (ANSI Z540 Committee) and the participation of various federal agencies.
Schumacher, NCSL (1995): [32B]
“…an NCSL [TQM] committee was formed by Gary Davidson with the purpose to eliminate the various similar
calibration control system requirements standards issued by various agencies of the US federal government
with a single standard [Z540] they could all agree upon. For that purpose, committee members were invited
from numerous agencies of the federal government… Hence, the NCSL committee proceeded with writing
such a standard. Its two parts are based on the ISO/IEC Guide 25 and on MIL-STD-45662A”.

Harris, NIST SP-986. NIST Workshop on Conformity Assessment for a Changing Government (2001): [36B]
“ANSI/NCSLI Z540, Standards Writing Committee
Participation
- Representation on NACLA Board
- Sponsorship of NACLA (Patron Member)
- Stakeholder in ILAC
- Annual meetings with NMI management (NIST, CENAM, NRC)
Collaboration
- Requested NIST establish a laboratory accreditation program for “calibration laboratories” in NVLAP
- Includes government representatives on writing committees to encourage adoption of standards…
- Adoption of Z540-1-1994 in NVLAP & OWM laboratory criteria at NIST
- Adoption of Z540-1-1994 by DoD DoE etc for calibration laboratories. DOE adoption of ISO/IEC 17025
- FAA contacted Airline Industry Committee – regarding Z540-1 applicability to FAA regulation”.
Indeed, when Z540.1 was published in 1994, it was quickly embraced by industry and enjoyed
widespread adoption in some agencies for more than a decade. With its ≥4:1 TAR requirement
carried over from the military, the sanctity of this calibration quality metric in Z540.1 remained
firmly entrenched in U.S. industry. It was thought to provide evidence that incorrect acceptance
decisions during calibration would be rare. A 4:1 TAR instilled confidence that when a
laboratory’s calibration certificate stated an item was found to be “In-Tolerance”, it actually was.
Less than a year after Z540.1 was published in 1994, the Department of Defense formally
cancelled MIL-STD-45662A in February of 1995, referring future requirements to Z540.1 (and
ISO 10012-1) as alternatives. NATO’s AQAP-6 was also cancelled the same year, citing ISO
10012-1 as its replacement. At the time, guidance within the ISO 10012-1:1992 document
alluded to both 3:1 and 10:1 TAR as guidance (see Appendix B). Eleven years later in 2003,
when ISO 10012-1:1992 and 10012-2:1997 were combined into a single standard (ISO
10012:2003) the conceptual reference to test accuracy ratio was removed altogether.
4. The 2000’s: ISO 17025 and ANSI/NCSL Z540.3

During the evolution of ISO Guide 25 (3rd edition, 1990) into the 1st edition of ISO 17025 in
1999, there was much debate regarding how the accuracy (or uncertainty) of the calibration
process should be taken into account during compliance decisions. As discussed earlier, taking
uncertainty into account is necessary to mitigate the risk of incorrect decisions during claims of
In-or-Out of tolerance. By requiring a 4:1 test accuracy ratio, MIL-STD-45662A had taken the
global or program-level approach to risk management; if the accuracy of the measurement
standard(s) was 4 times better than the item being tested, then, on the average, incorrect
decisions would occur infrequently. The effectiveness of this simple 4:1 rule did require some
rather significant assumptions regarding equipment reliability [3] [37] [38] [39], but the
simplicity of the 4:1 TAR rule made it an attractive technique (see footnote 5 in Section 5 of this
paper for a brief discussion of global vs. specific risk).
As stated, in 1990, ISO Guide 25 contained no test accuracy ratio requirement and did not
address how measurement uncertainty might result in incorrect decisions. It simply stated that
appropriate methods and procedures should “…be consistent with the accuracy required”.

It did require that calibration certificates “…shall provide the measurement results and
associated uncertainty of measurement and/or a statement of compliance with an identified
metrological specification”. Early drafts, of what was to become the 1st edition of ISO 17025,
reveal that the authors seriously contemplated including accuracy (or uncertainty) ratio
considerations, as well as a specific decision rule for In-Tolerance declarations. Similar to the
3:1 TAR referenced in ISO 10012-1:1992, Draft V of ISO Guide 25 [40], which circulated in
August 1996, proposed the following:
ISO Guide 25 Draft V (1996): [40]

“3.9.2.5 When, in the case of calibration certificates, parameters(s) are claimed to be within specified
tolerances, the measurement values(s), extended by the estimated uncertainty of measurement, shall fall
within the appropriate specification limit.
Notes
1. A statement of compliance should only be made if the ratio of the uncertainty of measurement to the
specified tolerance is reasonably small, e.g. 1:3.
2. If the measurement value, extended by the measurement uncertainty, exceeds the specified tolerance
while the measurement value itself falls within the tolerance, neither compliance nor non-compliance
can be proved. Only the measurement results and the associated uncertainty can then be given in the
certificate without any statement of compliance.”
Ultimately, when ISO Guide 25 did evolve into the 1st edition ISO 17025 and was finally
published in 1999, the language in the earlier draft relating to any type of accuracy or uncertainty
ratio (or decision rules) to mitigate the risk of incorrect acceptance decisions had been removed.
Moreover, when ISO 17025 was later revised in 2005 (2nd edition), no additional treatment of
accuracy ratios or measurement decision risk was added at that opportunity in time. Today, the
pertinent requirement is found in section 5.10.4.2 which simply requires that:
ISO 17025:2005 (2005): [27]

“5.10.4.2 When statements of compliance are made, the uncertainty of measurement shall be taken into
account.”
Ten years later in September 2015, the first committee draft (CD1) of the pending 3rd edition of
ISO 17025 [41] did not necessarily add any additional technical requirements to this matter.
However, it did propose some much needed direction, stating:
ISO 17025 CD1 Draft (2015): [41]
“7.1.1 (d) when the customer requests a verification of conformity to a specification or standard for the test or
calibration (e.g. pass/fail, in-tolerance/out-of-tolerance):
 the specification is clearly defined in the procedure selected;
 the decision rule for conformity, its level of risk and statistical assumptions is documented in the test
method/procedure or is documented by the laboratory and communicated to the customer;
 the decision rule is agreed to by the customer.”

When the second draft (CD2) of ISO 17025 was released in March 2016 [28], it proposed:
ISO 17025 CD2 Draft (2016): [28]
“7.1.1.3 When the customer requests a statement of conformity to a specification or standard for the test or
calibration (e.g. pass/fail, in-tolerance / out-of-tolerance) the specification and the decision rule shall be clearly
defined and communicated.
7.7.1 When statement of conformity to a specification or standard for test or calibration is requested, the
laboratory shall:
a) Document the decision rules employed taking into account the level of risk (such as false accept and
false reject and statistical assumptions) associated with the decision rule employed;
b) Apply the decision rule.
NOTE For further information see ISO/IEC Guide 98-4”.
The note in the CD2 draft of ISO 17025 is of great significance, as it references ISO Guide 98-4
[42], which is also known as JCGM 106:2012, “Evaluation of Measurement Data – The Role of
Measurement Uncertainty in Conformity Assessment” [43]. Compliance with ISO/IEC Guide
98-4 is not specifically required by CD2 of ISO 17025. However, Guide 98-4 provides a
thorough technical treatment of the statistical considerations which should be addressed any time
a measured value is asserted to fall inside or outside of acceptable tolerance limits. There is
always a risk that such a pass/fail decision or assertion is incorrect, due to measurement
uncertainty. If the uncertainty is small by comparison with the acceptable tolerance limits (high
TUR), then, on the average, the risk or probability incorrect decisions will be low. Conversely,
if this uncertainty is comparable in magnitude with the tolerance limits (low TUR), the average
risk of incorrect conformance decisions is elevated. This is the nature of calibration and
verification.
Six years prior to the publication of JCGM 106:2012, the NCSLI 174 Standards Writing
Committee and ANSI had published ANSI/NCSL Z540.3-2006 [44], replacing Z540.1-1994
(R2002). For the first time in a national or international standard, Z540.3 directly addressed, in a
formal manner, the rather complex issue of false accept risk during calibration. The Z540.3
standard requires that:
ANSI/NCSL Z540.3 (2006): [44]
“5.3 (b) Where calibration provide for verification that measurement quantities are within specified tolerances,
the probability that incorrect acceptance decision (false accept) will result from calibration tests shall not
exceed 2% and shall be documented. Where it is not practicable to estimate this probability, the test
uncertainty ratio shall be equal to or greater than 4:1.
NOTE: Achieving these requirements may involve adjustment and management of calibration system
parameters such as: measurement reliability, calibration intervals, measurement uncertainty, calibration
tolerances, and/or guard bands.
Calibration-servicing components may be considered competent to provide calibration services when they
have been accredited to meet ANSI/ISO/IEC 17025, including the requirements of this sub-clause…”

Additionally, NCSL International published a handbook to accompany Z540.3 [45] which
contains a substantial amount of information on measurement decision risk, including six
methods which can be employed to comply with the 2 % false-accept risk requirement. This
guidance consumes eight pages, while Appendix A of the Z540.3 Handbook spans 45 pages. In
total, more than 50 pages are dedicated to the subject of risk (probability) of incorrect decisions
during in-or-out of tolerance declarations. Subsequently, additional methods have also been
published [37] [38] [39] [39A]. Two NCSLI Recommended Practices, RP-18 [46] and RP-21
[47], also address the risk of incorrect decisions during calibration. Caldwell [48] has given an
overview of the Z540.3 standard. A distinction is also made in the Z540.3 handbook between test
uncertainty ratio (TUR) and test accuracy ratio (TAR), as elaborated by Mimbs [49].
As discussed earlier, the concept of measurement decision risk had been formalized in the
1950’s. Over the years, a proliferation of papers have been published from individuals on the
subject of accuracy (or uncertainty) ratios and the risk of incorrect pass/fail decisions during
calibration and testing. Appendix A provides a partial list of these papers and serves to
demonstrate the considerable attention this subject has garnered, particularly in the 21st century
where ISO 9001:2015 has adopted a “risk-based-thinking” model [50]. In addition to
independent authors, there exists many guidelines from a variety of engaged organizations which
deal directly with measurement decision risk as applied to calibration and the adequacy of
measurement processes. Several prominent publications on measurement and calibration
decision risk (misclassification) are listed here, many of which are available via public domain.
 1984: NBS (NIST) Special Publication 673 [51]
 1994: NASA RP-1342 – Sections 4.10, 6.5, 6.6, Appendix C.5 & F.5 [52]
 2001: ASME B89.7.3.1-2001 [53]
 2002: UKAS LAB34 – Section 4 [54]
 2003: ISO 10576-1:2003 [55]
 2004: AFNOR FD x07-022, Section 6 [A79]
 2005: ASME B89.7.4.1-2005 [56]
 2007: Eurachem/CITAC Guide [57]
 2008: Eurolab Cook Book Doc No. 8.0 [58]
 2009: Z540.3 Handbook – Section 5.3 and Appendix A [45]
 2009: ILAC-G8:03/2009 [59]
 2010: NASA HDBK 8739.19-4 [60]
 2010: VDA-5 [61]
 2010: AIAG MSA-4 – Chapter 1, Section B & Chapter 3 Section C [62]
 2012: UKAS M3003 – Appendix M [63]
 2012: NCSL RP-21 [47]
 2012: JCGM 106:20012 [43]
 2012: ISO/IEC Guide 98-4 [42]
 2012: ISO 22514-7 [64]
 2012: ISO/TR 14253-6 [A140]
 2013: ISO 14253-1 [65]
 2014: WADA TD2014DL [66]
 2014: NCSL RP-18 [46]
 2015: ISO 14253-5 [A157]

5. Calibration in the Healthcare Industry
Medical device and pharmaceutical manufacturers, regulated by the FDA, must comply with the
applicable sections in Title 21 of the Code of Federal Regulations (CFR). Table 2 provides an
historical summary of the prominent FDA equipment calibration requirements, including the
“suitable for intended purpose” clause. These may be considered government-unique standards.
The CFR calibration requirements do not specifically indicate how instruments should be
assessed to ensure they are “suitable for their intended purpose” (i.e. no accuracy ratios or
uncertainty considerations) nor what the qualifications or objective evidence for “traceability”
are. In 1997, the FDA [67] stated that:
FDA, Guide to Inspections of Medical Device Manufacturers (1997): [67]
“Manufacturers must assure all inspection, measuring and test equipment (including mechanical, automated or
electronic inspection and test equipment) is suitable for its intended use and is capable of producing valid
results. This would normally be done through installation, operation and performance qualification [IQ/OQ/PQ]
of the equipment”
The first statement above is a direct requirement of 21 CFR §820.72. However, the second
statement is the FDA guidance on how that requirement would normally be achieved. From a
literal perspective, it is difficult to imagine that “all” measuring and test equipment (to include
micrometers, calipers, rulers, hand-held meters, etc.) would be subjected to rigorous IQ/OQ/PQ
type qualification and validation activities. For medical device manufacturers, such extensive
qualification activities for measuring equipment are typically limited to analytical and/or
automated/computerized measuring apparatus addressed in USP <1058> [68] (e.g. “Group C”).
This is contrasted with Group B instruments which may only require calibration and preventative
maintenance, rather than qualification. ISPE GAMP-5 [69] further categorizes software and
computerized systems for analytical instrument qualification. Efforts to harmonize these two
guidance documents on qualification were published in 2014 by Vuolo-Schuessler et-al [70].
However, a certain degree of subjectivity exists when deciding which measuring instruments
must be qualified via the IQ/OQ/PQ process to show suitability for the intended use, and which
instruments must simply be calibrated. Thus, the type and extent of objective evidence necessary
to satisfy the “suitable for intended use” clause remains somewhat nebulous. Recent standards
such as ISO 22514-7:2012 [64], provide some guidance on how objective evidence to show
suitability might be achieved. Similar to the ubiquitous process capability index of 1.33, ISO
22514-7 recommends maintaining a “measuring system capability index (CMS)” of ≥1.33 and
Minitab® notes that a “gage capability index (Cgk)” of ≥1.33 is a common benchmark to assess
repeatability and bias for a “capable gage” [64A]. ANOVA Gauge R&R techniques often
espouse “gauge capability” requirements for “precision to tolerance ratio” (see footnote 2 in
Section 2 of this paper). However, a thorough statistical treatment of the subject is provided in
JCGM 106:2012 as a discussion of measurement capability index (Cm), mathematically
equivalent to test uncertainty ratio or TUR in Z540.3. In such an approach, suitability might be
determined based on the ability to provide acceptably low false-accept risk. That is, suitability is
related to the probability that a measuring instrument or system may erroneously indicate that a
product or process is in-tolerance. Measuring instruments exhibiting an acceptably low false-
accept risk may then be deemed suitable and “capable of providing valid results”.

Table 2. Review of historical FDA GMP/QSR calibration requirements
FDA 21 CFR §133.4 (GMP)* FDA 21 CFR §820.61 (GMP) FDA 21 CFR §820.72 (QSR)
Equipment Measuring Equipment Inspection, Measuring, & Test Equipment
(1963 – Obsolete) [71] (1978 – Obsolete) [72] (1997 - Current) [1A]
Equipment used for the All production and quality assurance measurement a) Control of inspection, measuring, and test
manufacture, processing, equipment, such as mechanical, automated, or equipment. Each manufacturer shall ensure that all
packaging, labeling, holding, or electronic equipment, shall be suitable for its inspection, measuring, and test equipment,
control of drugs shall be intended purposes and shall be capable of producing including mechanical, automated, or electronic
maintained in a clean and orderly valid results. Such equipment shall be routinely inspection and test equipment, is suitable for its
manner and shall be of suitable calibrated, inspected, and checked according to intended purposes and is capable of producing
design, size, construction, and written procedures. Records documenting these valid results. Each manufacturer shall establish
location in relation to surroundings activities shall be maintained. When computers are and maintain procedures to ensure that equipment
to facilitate maintenance and used as part of an automated production or quality is routinely calibrated, inspected, checked, and
operation for its intended purpose. assurance system, the computer software programs maintained. The procedures shall include
The equipment shall: shall be validated by adequate and documented provisions for handling, preservation, and storage
testing. All program changes shall be made by a of equipment, so that its accuracy and fitness for
(a) be so constructed that any designated individual(s) through a formal approval use are maintained. These activities shall be
surfaces that come into contact procedure. documented.
with drugs are suitable, in that they
are not reactive, additive, or (a) Calibration. Calibration procedures shall include (b) Calibration. Calibration procedures shall
absorptive to an extent that specific directions and limits for accuracy and include specific directions and limits for accuracy
significantly affects the identity, precision. There shall be provisions for remedial and precision. When accuracy and precision limits
strength, quality, or purity of the action when accuracy and precision limits are not are not met, there shall be provisions for remedial
drug or its components. met. Calibration shall be performed by personnel action to reestablish the limits and to evaluate
having the necessary education, training, whether there was any adverse effect on the
(b) be so constructed that any background, and experience. device's quality. These activities shall be
substances required for the documented.
operation of the equipment, such as
lubricants or cool ants, may be (b) Calibration standards. Where practical, the (1) Calibration standards. Calibration standards
employed without hazard of calibration standards used for production and quality used for inspection, measuring, and test
becoming additive to drug assurance measurement equipment shall be traceable equipment shall be traceable to national or
products. to the national standards of the National Bureau of international standards. If national or international
Standards, Department of Commerce. If national standards are not practical or available, the
(c) be constructed to facilitate standards are not practical for the parameter being manufacturer shall use an independent
adjustment, cleaning, and measured, an independent reproducible standard reproducible standard. If no applicable standard
maintenance as necessary to assure shall be used. If no applicable standard exists, an in- exists, the manufacturer shall establish and
the reliability of control house standard shall be developed and used. maintain an in-house standard.
procedures, to assure uniformity of
production, and to assure the (c) Calibration records. The calibration date, the (2) Calibration records. The equipment
exclusion from drugs of calibrator, and the next calibration date shall be identification, calibration dates, the individual
contaminants, including those from recorded and displayed, or records containing such performing each calibration, and the next
previous and current information shall be readily available for each piece calibration date shall be documented. These
manufacturing operations. of equipment requiring calibration. A designated records shall be displayed on or near each piece of
individual(s) shall maintain a record of calibration equipment or shall be readily available to the
(d) Be of suitable size and accuracy dates and of the individual performing each personnel using such equipment and to the
for use in any intended measuring, calibration. individuals responsible for calibrating the
mixing, or weighing operations. equipment.
* The 1963 GMP equipment regulations (21 CFR §133.4) did not contain any calibration requirements for equipment.

Concepts of measuring system “suitability” or “fitness for use”, based on the ability to provide
valid results with adequately low false accept risks, were recently addressed in a stimuli article in
the U. S. Pharmacopeial Forum. This article is entitled, Fitness for Use: Decision Rules and
Target Measurement Uncertainty [A159].
U.S. Pharmacopeial Forum Stimuli Article (2016): [A159]
“There is a long history behind DRs [Decision Rules], which have been used to provide organizations with
procedures for accepting and rejecting products… Some regulatory bodies in other industrial sectors are
already using DRs in formulating laws and regulations for dealing with commodities, including the food and
nuclear industries, environmental regulations, and highway traffic laws…
The DR defines the use of the reportable result, and can provide the information, such as acceptable
probabilities, needed to set the target measurement uncertainty (TMU), which is defined in the International
Vocabulary of Metrology as a ‘measurement uncertainty specified as an upper limit and decided on the basis
of intended use of measurement results.’
…DRs can take into account an acceptable level with regard to the probability of making a wrong decision.
The wrong decision can lead to accepting an out-of-specification (OOS) reportable result which is not true or
rejecting an OOS reportable result which is true, or a false failure or a missed fault…”
Target Measurement Uncertainties (TMU’s)4 are similar to minimum Test Uncertainty Ratios
(TUR’s) in that each can be used to specify the maximum allowable uncertainty, based on the
intended use of the measurement result. For calibration, the target measurement uncertainty
might be specified as a minimum 4:1 TUR, and the intended use might be to verify that test and
measurement equipment complies with established metrological specifications or allowable
tolerances. For pharmacopeial assays, the TMU might be specified directly, and the intended use
of the measurement result might be to verify that a drug has the required potency, within
specified limits. In each case, the required/minimum Test Uncertainty Ratio (TUR) or Target
Measurement Uncertainty (TMU) provides an upper limit on the magnitude of the acceptable
measurement uncertainty for the intended use, i.e., deciding conformity to specifications with
acceptably low risk of incorrect decisions. The Eurachem/CITAC guide on Setting and Using
Target Uncertainty in Chemical Measurements [A158] (section 5.1.2, equation 1) recommends
the Target Measurement Uncertainty (Utg), be 8 times smaller than the interval range (or span) of
the specification limits (Qmax – Qmin) for the measurand. This Target Measurement Uncertainty
is mathematically identical to a Test Uncertainty Ratio of 4:1, as defined in Z540.3.
ANSI/NCSL Z540.3-2006 (R2013): [44]

“3.11 Test uncertainty ratio: The ratio of the span of the tolerance of a measurement quantity subject to
calibration, to twice the 95% expanded uncertainty of the measurement process used for calibration.
5.3(b) …the probability that incorrect acceptance decision (false accept) will result from calibration tests shall
not exceed 2% and shall be documented. Where it is not practicable to estimate this probability, the test
uncertainty ratio shall be equal to or greater than 4:1”.
4
Target Measurement Uncertainties (also called International Target Values or ITV’s) emerged from efforts in 1979
by the nuclear industry to safeguard the quantity or inventory of nuclear and/or fissile materials held in various
locations, which are subject to periodic verification and accountancy via analytical measurement techniques [196].
Target Measurement Uncertainties have since also found favor with the analytical chemistry community [A158].
Although defined in the VIM, this term has not, thus far, been embraced by the calibration community at large,
particularly in the U.S. As of this writing, a Google search of the term “target measurement uncertainty” together
with “test uncertainty ratio” yields zero results returned (although Weitzel [173D] [173E] does relate TMU & TAR).

Eurachem/CITAC Guide on Setting and Using Target Measurement Uncertainty (2015): [A158]
“This document discusses how to set a maximum admissible uncertainty, defined in the third edition of
the International Vocabulary of Metrology as the ‘target uncertainty’, to check whether measurement
quality quantified by the measurement uncertainty is fit for the intended purpose…
The target uncertainty can be inferred from the compliance interval, defined by a minimum and a maximum
limit (section 5.1.2), or from the quantity value, above or below a single limit, beyond which there should be a
low probability of an incorrect compliance decision.
If a compliance interval is defined for the measurand… the uncertainty should be small enough to distinguish
max min
quantities within this interval. If the compliance interval is defined by a maximum Q and a minimum Q
tg
quantity, the target expanded uncertainty, U , should typically be 8 times smaller than the interval range:
𝑄 𝑚𝑎𝑥 − 𝑄 𝑚𝑖𝑛
𝑈𝑡𝑔 =
8
…an important reason for setting a target uncertainty is the use of uncertainty in the assessment of
compliance… The key to the assessment of compliance is the concept of ‘Decision rules’. These rules give a
prescription for the acceptance or rejection of a product based on the measured quantity value, its uncertainty
and the specification limit or limits, taking into account the acceptable level of the probability of making a
wrong decision.”
Target Measurement Uncertainties have also been employed to identify the “end-use”
uncertainty objective of the final result (or legal limit), such that, yet a sufficiently smaller
measurement uncertainty should be used to determine compliance to such a limit. Section 2.1.5
of the Eurachem guide on Terminology in Analytical Measurement – An introduction to VIM 3
[202] appears to advocate a 5:1 Test Uncertainty Ratio “rule of thumb” in traceability chains:
Eurachem Guide on Terminology in Analytical Measurement (2011): [202]

“2.1.5 ...if the intention is to determine if a specific test sample complies with a legal limit…
As a rule of thumb the measurement uncertainty for those steps in the measurement procedure that have a
significant effect on the result should be ≤1/5 of the target measurement uncertainty for the final result. When
this condition is met, the individual steps concerned will make a negligible contribution to the overall
measurement uncertainty.”
An example in this Eurachem guide is also given with respect to EU legislation on maximum
allowable uncertainty for the testing of surface water for lead content, where a 2:1 Test
Uncertainty Ratio is apparently acceptable [202]. WELMEC uses the terms Maximum
Permissible Uncertainty (MPU) and Maximum Permissible Error (MPE), recommending that
“where practically possible, MPU ≤ MPE/3”, equivalent to a 3:1 TUR [A86].
Eurachem Guide on Terminology in Analytical Measurement (2011): [202]
“3.1.2 The laboratory will aim to remove sources of significant uncertainty until the measurement procedure is
deemed to be fit for purpose. This means that the laboratory should know the maximum measurement
uncertainty that can be accepted by the customer for a specific application. This is called the target
measurement uncertainty. For example, the EU legislation regarding the official control for monitoring water
status state that laboratories performing measurements should use measurement procedures capable of
providing results with an ‘uncertainty of measurement of 50 % or below (k=2) estimated at the level of relevant
environmental quality standards.’ For example, the environmental quality standard for lead in surface waters
-1 -1
is 7.2 µg L , so the target measurement uncertainty is 3.6 µg L .”

TUR (or TMU or MPU) can be used to calculate (or can be based on) the probability of making
incorrect decisions (i.e. misclassification, missed faults, false failure, etc.), to formulate decision
rules, and/or to determine acceptable guardband limits if necessary. In both cases, the acceptable
measurement uncertainty is directly related to the global probability of making incorrect
accept/reject decisions5. Global probabilities associated with making incorrect decisions are also
dependent on the a priori probability that the device (or process) being tested complies with its
specification(s) [A96]. For calibration, this is often called End of Period Reliability (EOPR) or
“as-found in-tolerance probability” and can be estimated from observed historical reliably data.
Mathematical boundary conditions have been shown to exist for TUR and EOPR, which can
effectively limit global false-accept risk to selected levels [37] [38] [39] [39A]. Similar
reliability data is often available for manufacturing processes, analytical assays, control charts,
etc., where the historic ability (or confidence) to meet specified limits can be used to estimate the
variability of the product or process under test. Such boundary conditions and risk mitigation
techniques are equally applicable to these endeavors, not just calibration processes.
Rather than requiring (sometimes arbitrary) minimum Test Uncertainty Ratios or maximum
allowable Target Measurement Uncertainties, Z540.3 addresses the root intent of these attempts
at risk mitigation. Rather than placing requirements on secondary parameters which are merely
related to measurement decision risk, Z540.3 advocates placing allowable limits on false accept
risk itself. Risk can then be mitigated by controlling Test Uncertainty Ratio, reliability of the
end-item/process, and/or guardband limits. In addition to the US Pharmacopeial Forum stimuli
article and the Eurachem/CITAC guides previously discussed, Target Measurement
Uncertainties (and guardbanding), along with their utility in establishing “fitness for purpose or
intended use”, have been addressed by several authors. These include Weitzel & Johnson
[A138], Majcen, Skubic & DeBièvre [197], DeBièvre [198] [199], DaSilva & Williams [200],
among many others. Accreditation bodies have also recognized TMU’s with respect to
establishing uncertainty requirements in medical laboratories accredited to ISO 15189 [201].
Weitzel & Johnson - USP: (2012) [A138]

“The target measurement uncertainty can be decided by following a process which involves determining…the
acceptable level of risks of incorrect decisions of compliance; developing a suitable decision rule, with guard
bands if appropriate; using the probability of making an incorrect decision of compliance based on the decision
rule… The fitness for purpose of a quantitative method… when assessing compliance to requirements can
be described by the maximum measurement uncertainty that is acceptable based on the use of the data.
Decision rules, measurement uncertainty, risk and probability have been used in many scientific areas for
many years. We can leverage this experience to more effectively meet cGMP requirements… The USP is
using the TAR [Test Accuracy Ratio] decision rule to determine if its Certified Reference Materials are fit-for-
use.” Weitzel [173E]
See also comments from the FDA [173A] and USP [173B] [173C] [173D] [173E] regarding test
accuracy ratios, measurement uncertainty, decision risk, and fitness-for-use toward the end of
Section 7 of this paper.
5
JCGM 106:2012 [43] invokes the term “global” consumer/producer risk when addressing probabilities of
incorrect decisions, prior to obtaining any specific individual measurement result. The terms “program-level” risk
[A96] and “average” risk [56] have also been used in this capacity. By contrast, JCGM 106 uses the term “specific”
risk when addressing probabilities of incorrect decisions associated with a specific measurement result . The terms
“bench-level” [A96] risk, “individual” [56] risk, and “local” [A85A] risk have also been used in discussing this level
of risk. The distinction between global and specific risk is critically important, but is beyond the scope of this paper.

The FDA introduced measuring and test equipment calibration requirements into the GMP’s in
1978 [72]. Through a series of workshops, along with publication of Good Manufacturing
Practice manuals from 1978 to 1997, the FDA provided guidance to the industry on how to
comply with the many aspects of the GMP’s. Most of the GMP manuals dedicated an entire
chapter to equipment calibration. However, none of the FDA GMP manuals addressed accuracy
ratios or uncertainty considerations during calibration for in-or-out of tolerance decisions. The
one exception to this appears merely as an example in a calibration procedure in the FDA
guidance for mechanical measuring tools, which stated, “The calibration shall be done by a
comparison to standard gage blocks… with an accuracy 3 to 10 times greater than that of the
measuring tool.” See Appendix F for the 1997 version of the FDA GMP/QSR guidance on
calibration. Note that this guidance was officially withdrawn by the FDA on December 12,
2013. There are currently no (known) official FDA guidelines on instrument calibration for
compliance with 21 CFR §820.72.
Meanwhile, private industry was publishing their own calibration guidance for medical device
and pharmaceutical manufacturers. These groups were often comprised of individuals and
experts with direct knowledge of metrology and calibration principles. The Association for the
Advancement of Medical Instrumentation (AAMI) published the following in 1998 [73]:
AAMI Quality System Compendium – GMP Requirements & Industry Practice (1998): [73]
2nd edition published in 2007; 3rd edition in 2015.
“Most manufacturers consider the necessary accuracy, precision, and resolution required for the measuring
and test equipment during the establishment of their manufacturing and test procedures. Otherwise, the
manufacturer may discover at a later date that the equipment is not suitable (i.e., not accurate or precise to
provide reliable measurement/test results)…
Generally, measuring equipment is at least four times, preferably ten times, more accurate than specified
tolerances. Traceable standards are typically at least four times, preferably ten times, more accurate than the
particular measuring or test equipment being calibrated.
The ANSI/ASQC standard, ‘Calibration Systems’ [ANSI/ASQC M1-1987], discusses quantification of

calibration errors and provides additional information on determining the degree of uncertainty of a calibration
system to ensure that traceable standards are accurate enough for use in a specific calibration program”.
AAMI chose to cite the ANSI/ASQC M1-1987 standard “Calibration Systems” [74] while also
referencing 4:1 and 10:1 accuracy ratios. Like AAMI, the FDA also made reference to
ANSI/ASQC M1-1996 in a list of “useful standards” under the calibration section (line 1336) of
the bibliography in the revised-draft guidance entitled, “Guidance for Industry: PAT – A
Framework for Innovative Pharmaceutical Manufacturing and Quality Assurance” in 2003.
However, this reference to the ANSI/ASQC M1 Calibration Standard was removed upon final
publication of the PAT Guidance [75]. See Appendix B for treatment of accuracy ratios in
ANSI/ASQC M1-1996.

Like the calibration guidance provided by AAMI, two other industry groups have published
calibration guidance tailored specifically for the FDA-regulated pharmaceutical and medical
device industry. These groups are the ISPE/GAMP (International Society of Pharmaceutical
Engineers – Good Automated Manufacturing Practice) and the NCSL International Healthcare
Metrology Committee. The ISPE/GAMP document is entitled, “A Risk Based Approach to
Calibration Management” [76]. The NCSLI document is known as RP-6 and is entitled,
“Recommended Practice: Calibration Quality Systems for the Healthcare Industries [77].”
The ISPE GAMP document does not address accuracy ratios directly. However, it does provide
guidance on how measurement uncertainty should be taken into account when making decisions
regarding the in-or-out of tolerance status of calibration results.
ISPE/GAMP Risk Based Approach to Calibration Management (2010): [76]

1st edition published in 2002
“5.5.5 Introduction to the Application of Measurement Uncertainty

When measuring a critical parameter, measurement uncertainty should be understood, in order to evaluate the
potential risk associated with obtaining either a false pass or a false fail…
5.5.7 Review of Third Party Calibration Certificates for Reference Standards

Reference standards should be calibrated by a competent (appropriately qualified) organization… The
calibration ranges, test points, and configuration should be suitable for the intended use. It also should show
the errors at each test point, prior to and following any adjustment, along with a statement of the estimated
measurement uncertainty.
The certificate should be checked to determine whether residual errors are within agreed limits. This
assessment should include consideration of the potential influence of measurement uncertainty when trying to
determine pass or fail decisions (see Figure 5.3)... Given the ambiguity and inconsistency this can generate, it
is considered good practice to have a policy on dealing with these various influence conditions.
…the contract should state how the specification limits are selected and how the potential influence of
measurement uncertainty will be treated. (Note: it is usually considered better to select larger tolerances than
the manufacturer’s specification, where the application allows, as compliance with manufacturer’s specification
may be difficult to achieve or demonstrate in the long term.) Measurement uncertainty can lead to the incorrect
or false acceptance and rejection of calibration results.
Terms which may be used by pharmaceutical organizations include:
• False accept occurs when an instrument appears to pass specification limits when it actually fails:
- Declaring an instrument to be within tolerance when it is out of tolerance has the potential for the
instrument’s reading to cause product to be made outside the intended process range.
• False reject occurs when an instrument appears to fail specifications when it actually passes:
- False rejects may have a significant impact on business efficiency. For critical measurements, the calibration
false accept rate should be considered.”

NCSLI Recommended Practice RP-6 addresses measurement decision risk and accuracy ratios as
follows:
NCSL International RP-6: Calibration Quality Systems for the Healthcare Industries (2015): [77]
(1st edition 1986, 2nd edition 1999, 3rd edition 2008, 4th edition 2015)
“5.3 Adequacy of Measuring and Test Equipment (M&TE)

Calibration tolerances should be based on the product/process requirements and limits. While the
manufacturer’s specifications are often used as the assigned tolerance, a different (usually wider) tolerance is
often adequate for the application. It the ratio between the product/process limits and the instrument tolerance
is 4:1 or greater, the instrument tolerance is generally considered insignificant and no additional compensation
for instrumental measurement uncertainty or accuracy is necessary.
5.4 Adequacy of Calibration Equipment and Standards

The calibration quality system should ensure that the adequacy of calibration equipment is not compromised.
This system should contain the following:
Items should be calibrated with measurement systems, standards, and reference materials that have
adequate accuracy, stability, and range to completely verify the performance of the calibrated item within its
specified tolerance limits.
Each organization should establish a program that includes measurement uncertainty, uncertainty ratios,
accuracy ratios, false accepted risks, false rejected risks, or coverage factors to support the adequacy of its
measurement system.
5.6.2 Measurement Uncertainty

Measurement uncertainties should be calculated when…
Test accuracy ratio parameters are below measurement quality thresholds
There are significant uncertainty contributors that can affect the measurement result…
7. Appendix B. Adequacy of M&TE for Intended Use Decision Guideline…

Depending upon the criticality of the product/process, the PAR [Process Accuracy Ratio] requirement may
vary. Typically, a PAR of 4:1 or greater is considered sufficient to minimize measurement risk (e.g. false
accept risk or consumer risk). However, the correct level of acceptable PAR and/or risk should be determined
and documented by the organization.
NOTE: Other methods of controlling measurement decision risk, such as guardbanding may also be used”.

An overview of the NCSLI Healthcare Metrology Committee that published RP-6 has been
given by McNeely [78].
M. McNeely, NCSLI (2013): [78]
“4. Goals of the 151 Healthcare Metrology Committee
4.1. Work with the US FDA on recognition of 17025 (ILAC MRA) accredited calibration laboratories.
4.1.1. This effort aspires to create a guidance document for abbreviated inspection of such laboratories, by
providing greater efficiency for documentation and lab qualification, however, it does not advocate in any way
the mandated use of such laboratories. It is stressed to be promoted as a voluntary guidance document.
4.1.2. If a calibration laboratory was accredited to ISO/IEC 17025 for the applicable calibration discipline by a
mutually recognized accreditation body (AB), and the scope of uncertainty was considered acceptable by the
contracting party to ultimately support its process, the calibration would be considered traceable to national or
international standards under the scope of such accreditation, and the scope of accreditation would then be
considered acceptable documentation of the unbroken chain of traceability to satisfy the requirements stated
in 21 CFR 820.72(1).
4.1.3. As of May, 2013, the 151 awaits formal notification from The US FDA on the ultimate status of this
proposed draft guidance document.”
As stated by McNeely, the NCSL International 151 Healthcare Metrology Committee has
petitioned the FDA to create a new draft guidance document that would allow minimized FDA
assessments of calibration laboratories that are accredited to ISO 17025. This would be
somewhat similar to the course taken by NRC, which now readily accepts calibrations from
accredited laboratories for commercial third party calibrations (see section 6.2). Such an FDA
guidance document may facilitate an abbreviated inspection of calibration laboratories that
voluntarily chose to become accredited. The request by the NCSLI 151 Committee was
submitted to the Dockets Management Branch (HFA-305) of the FDA on October 11, 2011. See
Appendix C. Such activities are consistent with 21 CFR §10.115 [79] governing FDA Good
Guidance Practices and the participation in their development; see Appendix D. As of April
2016, the NCSLI 151 Committee awaits formal notification from the FDA on the status of this
proposed draft guidance document.
Similar advocacy regarding the recognition of national and international calibration standards
was previously submitted to the FDA in 1998 by James [80] in Docket No. 97N-0477.
James, Comments Submitted to FDA; Docket No. 97N-0477 (1998): [80]
“All calibration systems that assure the integrity of medical measurements should employ methods similar to
those set forth in national and international standards such as IS0 GUIDE 25, ANSI/NCSL Z540-1, NASA
5300.4, or U.S. DOD MIL-STD-45662A. These standards require periodic intervals and methods be
established to maintain acceptable accuracy and measurement reliability. Measurement reliability [End of
Period Reliability or EOPR] is defined as: the probability that the equipment under test and the measurement
standard will remain in-tolerance throughout the established interval. This kind of system is designed to be
both effective and efficient at addressing the needs of uncertainty growth.”

Another group that has published calibration guidance for the healthcare industry is The Global
Harmonization Task Force (GHTF). However, as the 1997 FDA Quality System Regulation in
21 CFR had been harmonized with ISO-13485 and ISO 9001, the GHTF calibration guidance
[81] largely addresses the basic calibration requirements in section 4.11 of ISO 9001:1994 and
section 13 of ISO 9004-1:1994.
GHTF.SG3.N99-8 Quality Systems For The Design And Manufacture Of Medical Devices (1999): [81]
Published (1999) – Archived June 21, 2005.
Note: The GTHF participated in ISO/TR 14969:2004(E), which effectively replaced the GHTF guidance here.
“4.11 Control of inspection, measuring and test equipment

4.11.1 General
4.11.2 Control procedure
The requirements of this clause in ISO 9001 spell out in considerable detail what is to be implemented.
Although the requirements pertain explicitly to inspection, measuring and test equipment, it is helpful to
approach the subject from the perspective that measuring is itself a process involving raw materials,
equipment and procedures. The requirements of ISO 9001 explicitly involve elements of the measurement
process; elements whose collective purpose is to choose suitable measurements, suitable measuring
equipment, and suitable measurement procedures. These elements are specified to provide confidence in the
ability of the supplier's measuring systems to control adequately the production and inspection of the product.
For both product- and process-measurement systems, statistical methods are valuable tools for achieving and
demonstrating fulfilment of requirements. In particular, statistical methods are the preferred tools in fulfilling
the requirement that ‘Inspection, measuring and test equipment shall be used in a manner which ensures that
the measurement uncertainty is known and is consistent with the required measurement capability’.
The requirements of this clause also should be applied by the supplier insofar as ‘demonstrating the
conformance of product to the specified requirements’ contractually involves measurements subsequent to
production and inspection of a product (e.g. during subsequent handling, storage, packaging, delivery or
servicing) as may be required under other clauses of ISO 9001.
Note: See also ISO 10012-1.

See 13 of ISO 9004-1.”
Many entities regulated by the FDA voluntarily choose to be certified to ISO 13485:2016 [82] by
a registrar. The International Accreditation Forum (IAF) has published an overview of recent
enhancements to this certification process [82A]. Registration/certification to ISO 13485 is not
specifically required by the FDA. However, a new Medical Device Single Audit Program
(MDSAP) is currently in the final stages of a pilot program. Under this pilot program, FDA
regulated entities can voluntarily choose to be audited, not by the FDA, but by a 3rd party
auditing organization. Auditing organizations must be approved by the International Medical
Device Regulators Forum (IMDRF), of which the FDA is a member of, and which evolved out
of the previous GHTF. The MDSAP will take the place of routine FDA audits and will use ISO
13485 as its primary requirements document. The calibration requirements for control of
monitoring and measuring devices in ISO 13485:2003 are given below. Note that no mention of
test accuracy ratios, uncertainty, or measurement decision risk is provided.

ISO 13485:2003: Medical Devices – Quality Management Systems – Requirements for Regulatory Purposes
(2003) [82]
“7.6 Control of Monitoring and Measuring Devices
The organization shall determine the monitoring and measurement to be undertaken and the monitoring and
measuring devices needed to provide evidence of conformity of product to determined requirements (see
7.2.1).
The organization shall establish documented procedures to ensure that monitoring and measurement can be
carried out and are carried out in a manner that is consistent with the monitoring and measurement
requirements.
Where necessary to ensure valid results, measuring equipment shall

a) Be calibrated or verified at specified intervals, or prior to use, against measurement standards
traceable to international or national standards; where no such standards exist, the basis used for
calibration or verification shall be recorded;
b) Be adjusted or re-adjusted as necessary;
c) Be identified to enable the calibration status to be determined;
d) Be safeguarded from adjustments that would invalidate the measurement result;
e) Be protected from damage and deterioration during handling, maintenance and storage.
In addition, the organization shall assess and record the validity of the previous measuring results when the
equipment is found not to conform to requirements. The organization shall take appropriate action on the
equipment and any product affected. Records of the results of calibration and verification shall be maintained
(see 4.2.4).
When used in the monitoring and measurement of specified requirements, the ability of computer software to
satisfy the intended application shall be confirmed. This shall be undertaken prior to initial use and
reconfirmed as necessary.
NOTE See ISO 10012 for guidance related to measurement management systems.”
As stated, the IMDRF’s Medical Device Single Audit Program will use ISO 13485 as the core
framework for auditing purposes. The IMDRF recently published an MDSAP Companion
Document [83], which contains guidance on auditing calibration requirements, to be used by
approved auditing organizations. Although the concepts of test accuracy ratio and measurement
decision risk are not directly discussed in the guidance, the primary requirement is the ubiquitous
clause, “…suitable for its intended use, and capable of giving valid results.” The IMDRF
guidance is provided below.

IMDRF Medical Device Single Audit Program (MDSAP) Companion Document (2014): [83]
“13. Confirm that the organization has determined the monitoring and measuring devices needed to provide
evidence of conformity to specified requirements. Verify that the monitoring and measuring equipment used in
production and service control has been identified, adjusted, calibrated and maintained, and capable of
producing valid results.
Clause and regulation: [ISO 13485:2003: 7.5.1.1, 7.6; TG(MD)R Sch3 P1 1.4(5)(e); RDC ANVISA 16/2013:
5.1.5, 5.4; 21 CFR 820.70(g), 820.72]
Additional country-specific requirements: None
Assessing conformity:
Maintenance and calibration
While reviewing the selected production process, make note of significant pieces of process equipment and
significant pieces of measuring or test equipment. Consider selecting process and test equipment that, if not
properly controlled, could cause devices to not meet specified requirements; or produce inaccurate results that
could lead to unrecognized nonconformities. Confirm that the production and test equipment selected for
review is suitable for its intended purpose and capable of giving valid results.
Review the maintenance, control, and calibration procedures (and records) for the equipment selected for
review. The initial frequency with which measuring and test equipment is calibrated and maintained is usually
based on the equipment manufacturer’s recommendations. As the organization gains experience with the
piece of equipment, the frequency of calibration and maintenance may be adjusted, based on a documented
rationale.
Accuracy and precision
When accuracy and precision is a factor in the validity of the result of the measuring equipment, the required
accuracy and precision should be defined during the planning of product realization to ensure the equipment is
suitable and capable of providing valid results
Reviewing records
If production equipment or test equipment is found to be outside of its maintenance or calibration
requirements, verify that the organization made an assessment of the effect of the out-of-tolerance situation on
in-process, finished, or released devices, based on risk. Equipment adjustment, calibration, and maintenance
procedures and records may provide insight into nonconformities. Review these procedures and records to
determine whether inadequate procedures or the organization’s failure to comply with adequate procedures
contributed to the nonconformity. For example, determine whether the lack of specified equipment adjustment
or maintenance contributed to the production of nonconforming product.”
Much like the FDA’s database on recognized consensus standards [84] [85] [86] , ISO 17025 is
not currently listed as a “recognized standard” by the IMDRF N15 document, “Final Report: List
of International Standards Recognized by IMDRF Management Committee Members as of
March 2014” [87] [88].
As previously stated, the older 1999 GHTF SG3.N99-8 guidance document was archived in 2005
and has been essentially replaced by ISO/TR 14969:2004 [89]. While again not mentioning test
accuracy ratios or measurement decision risk, the ISO Technical Report 14969 does indicate
that, “Statistical methods are important in showing which monitoring and measuring devices are
used in a manner which ensures that the measurement uncertainty is known and is consistent
with the required measurement capability.”

ISO/TR 14969:2004(E) Technical Report: Medical Devices – Quality Management Systems – Guidance on the
Application of ISO 13485:2003 (2004): [89]
“7.6.1 The requirements refer explicitly to monitoring and measuring devices, including test software. It is
helpful to approach the subject of control of monitoring and measuring devices from the perspective that
measuring is itself a process involving materials, equipment and procedures. The intent of the requirements is
to give the organization confidence in the monitoring and measuring devices that it uses to ensure that product
meets customer and regulatory requirements.
Statistical methods are important in showing which monitoring and measuring devices are used in a manner
which ensures that the measurement uncertainty is known and is consistent with the required measurement
capability.
The requirements of this subclause are also applied by the organization when demonstrating the conformity of
product to the specified requirements. This can involve measurements subsequent to production and
inspection of product (e.g. during handling, storage, packaging, preservation, delivery or servicing).
Documented procedures should include details of equipment type, unique identification, location, frequency of
checks, check method and acceptance criteria…
NOTE Additional information regarding the management of monitoring and measuring equipment is available in
ISO 10012.”
The calibration requirements in ISO 13485 may not be sufficiently prescriptive to adequately
minimize or control incorrect decisions due to measurement uncertainty or to provide guidance
on instrument suitability. In the healthcare industry, falsely accepting a measuring device during
calibration (i.e., declaring it as In-Tolerance, when it is actually Out-of-Tolerance, due to
measurement uncertainty) can have serious consequences. Accuracy is of paramount importance
for many instruments, not just those used during calibration and for the manufacture of medical
devices, but for the actual devices themselves. Grim [90] published the following conclusions in
2002 regarding measurement error and uncertainty in blood pressure measuring instruments:
Grim, Summary Report – NHBPEP / NHLBI / AHA National Institutes of Health (2002): [90]
“Unless blood pressure (BP) is measured accurately, the proven benefits of diagnosing and treating an
unhealthy blood pressure will not be transferred to the population. …errors as small as 2-5 mmHg can have
astounding costs to the individual patient, to the health care system, to a research project, for government
planning and for society.
…an error of ‘only’ -5 mmHg at the 90-95 mmHg range will miss the 21 million US hypertensives in the US in
this range (42 % of all with HTN). Over the next 6 years, these 21 million untreated HTN will experience
125,000 CAD [coronary artery disease] deaths of which at least 20 % would have been prevented by
treatment. About the same number of fatal strokes would have also been prevented. Thus a -5 mmHg error
will cause about 50,000 preventable deaths not to mention preventing perhaps twice this many non-fatal CAD
and CVAs [cerebrovascular accident]. Aneroid devices out of calibration most often read too low.
Measuring BP falsely high increases costs by treating those who do not truly have high BP. Thus an error of
+5 mmHg would move 27 million people from 85-89 [mmHg] into the high BP range. As the estimated cost of
treating one person for high BP is $1000/yr, this will cost $27 billion/yr to treat a ‘non-disease’. Even a 2
mmHg error will misclassify about 6 million persons into the 90–95 range. Current standards permit devices ±3
mm of the mercury standard. This seems too lenient.”

Interestingly, Lozy [91] had published a risk analysis method for the misclassification of
hypertension diagnosis due to measurement error in 1982. Lozy had applied the same
mathematical false accept/reject or misclassification technique (bivariate normal joint probability
density function) that Hayes had applied to incorrect calibration decisions in 1955 [A5]. Other
researchers such as Friedman-Gerlicz and Lilly [91A] have also employed this mathematical
model to quantify misclassification rates in hypertension diagnosis due to measurement errors.
Indeed, this technique is now commonly employed to model global consumer risk and producer
risk anytime pass/fail decisions (or diagnoses) are based on measurements which are subject to
error or uncertainty (see Appendix A). Greaves [92] when citing Forsman [93] states that 60 %
to 70 % of medical decisions are based on test results. Thus, the influence of measurement
uncertainty on incorrect decisions extends beyond calibration and into all areas of measurement,
including manufacturing processes, analytical chemistry, diagnoses of medical conditions, etc.
Kuselman et-al provided the following in 2003 [94] and Stamm in 2013 [A143A].
Kuselman et-al (2003): [94]

“Traceability and uncertainty of measurement results are basic technical elements of quality systems in
analytical laboratories whose competence is recognized by accreditation according to ISO/IEC 17025.
However, GLP and GMP standards widely used since the 1960s for the quality system assessment in
pharmaceutical industry do not include requirements for the measurement uncertainty evaluation. It is clear
today that the issue discussed above should be taken into account by pharmacopoeial committees and
regulatory bodies involved in drug quality assurance for harmonization of requirements to analytical results
and improvement of their quality…”
Stamm (2013): [A143A]

…FDA regulations…require calibration of instrumentation and gauging used in the manufacture of
pharmaceuticals and medical devices and also require limits for accuracy and precision of these devices...
However, they do not specify a method for determining accuracy and precision estimates… While the FDA
does specify calibration for measurement instrumentation… the requirement stops short of requiring an
estimation of uncertainty in the measurement results…
The preceding review of the standards and guidelines employed by the healthcare industry to
maintain and calibrate equipment may appear rather stringent. However, a recent article by
Simpson [95] entitled, “FDA Standards on Calibrating Lab Equipment Are Not So Tough”
highlights the basic FDA calibration requirements found in 21 CFR §820.72. These
requirements have remained largely unchanged since the 1978 version of 21 CFR §820.61
which, as stated previously, were based to some degree on calibration requirements from MIL-C-
45662 published in 1960. One significant change did occur when the GMP’s were revised in
1997; a requirement was added to investigate product impact or “adverse effects” resulting from
out-of-tolerance instruments (see Table 2). This same impact assessment requirement had also
been added to MIL-STD-45662 in 1980, along with the addition of the 4:1 TAR requirement in
MIL-STD-45662A in 1988. However, the revised 1997 GMP’s in 21 CFR §820.72 contained no
accuracy or uncertainty ratio requirement. The basic FDA calibration requirements in 21 CFR
§820.72 are similar to those found in ISO 9001 and ISO 13485. They are not as prescriptive as
the modern requirements found in ISO 17025 or Z540.3 which many industries, both regulated
and unregulated, have adopted or employ as guidance documents. Refer to Table 1 for a
historical context of calibration documents and requirements and guidelines that have evolved
over several decades to prescribe technical requirements for many critical calibration programs.

Without agreement on the use national and/or international standards (even as guidance
documents), the very definitions in the text of the CFR may be ill-defined or subject to
interpretation. Despite established metrological definitions in Z540.3, ISO 17025, and other
related documents, there have been recent accounts where entities regulated by the FDA have
received Form-483 Investigational Observations and Warning Letters arising from the failure to
always adjust instruments during calibration, whether out-of-tolerance or not [96]. These
incidents may be attributable to a nebulous distinction between the definitions of calibration,
verification, and adjustment – which are well defined terms outside of the CFR [97] [98].
References to similar events in regulated industries have also been published where the
definition of calibration has been inferred to mandate adjustment during calibration [99] [99A]
[100] [101]. Without agreement on standards and guidance documents for calibration, similar
events may again manifest – where the definition of other metrological terms in the CFR, such as
traceability, may become the subject of future regulatory contention. Recognition of ISO 17025
and Z540.3 as guidance documents could provide a common framework for all parties to
reference in such instances.
An article in 2010 by Schmitt [102] provides an interview with Trautman – at the time, the FDA
Associate Director of International Affairs and author of the 1997 (current) 21 CFR 820 Quality
System Regulation. This article deals directly with FDA calibration expectations, particularly
those required of 3rd party calibration laboratories (outsourced “test houses”). Since little exists
in the public domain regarding official FDA guidance on calibration, the article by Schmitt
becomes highly relevant. Owing to this, the article is reproduced in its entirety, with the author’s
permission, in Appendix E of this paper. Inherent to the theme of the article is that the FDA
calibration requirements are very stringent – so stringent, that care is advised when selecting
outsourced third-party calibration vendors, because they may not meet rigorous FDA
requirements. Cautions are given in the article that such calibration laboratories often service
other “unregulated” industries – the apparent implication being that some of these other
industries may not have such strict calibration requirements.
6. Calibration in Other Government/Regulated Industries

The preceding inferences from the Schmitt article compare the calibration requirements of the
FDA with those of other unregulated industries. However, there are a plethora of both
government and non-government organizations that require calibrations to be performed in
compliance with, if not accredited to, ISO 17025 and/or ANSI Z540.3. By contrast, compliance
with such standards is not an FDA requirement of the calibration function of entities regulated by
the FDA, nor have these documents been cited as official guidance documents by the FDA for
calibration. A comparative review of some other federal government agencies is provided here.

6.1 NASA – National Aeronautics and Space Administration
Historical NASA calibration and metrology documents are provided in Table 1. NASA’s current
policy directive on metrology and calibration [103] states that:
NASA Policy Directive NPD 8730.1C (2011): [103]
“a. It is NASA policy to…

(3) Control the accuracy, reliability, and use of Measuring and Test Equipment (MTE) through the use of a
calibration system compliant with the requirements of American National Standards Institute/National
Conference of Standards Laboratories (ANSI/NCSL) Z540.3-2006 and applicable requirements of
Society of Automotive Engineers (SAE) AS9100, subject to the clarifications and modifications
provided in Attachment B of this NASA Policy Directive (NPD).
ATTACHMENT B: ANSI/NCSL Z540.3 Clarifications and Modifications…
B.3. End-of-period-reliability (EOPR) values equal to or greater than 89 percent are considered
acceptable evidence of compliance to ANSI/NCSL Z540.3 subclause 5.3.b, false acceptance
requirements, and subclause 5.3.3, measurement uncertainty requirements, where such values are
derived from statistically significant empirical data…
B.5. Guidance concerning estimation and evaluation of measurement decision risk, including false accept risk,
is provided in NASA-HDBK 8739.19-4, Estimation and Evaluation of Measurement Decision Risk, and in the
NCSLI Handbook for the Application of ANSI/NCSL Z540.3-2006, Requirements for the Calibration of
Measuring and Test Equipment.
B.6. Guidance and methods for controlling the accuracy of measurement results are provided in the NASA-
HDBK 8739.19 series of measurement handbooks. The handbooks are found at
http://www.hq.nasa.gov/office/codeq/doctree/hdbk873919.htm.
B.7. Compliance to ANSI/NCSL Z540.3-2006 is considered to meet or exceed the requirements of ANSI/NCSL
Z540.1-1994 (R2002)”.
Thus, Z540.3 is listed as a mandatory compliance document in NASA contract solicitations

(Request for Proposals or RFP’s) for calibration programs. This is evidenced under the ISC
contract at the Kennedy Space Center [104], the CAMS contract at Johnson Space Center [105],
and the TIALS contract at Glenn Research Center [106]. The Measurement Standards and
Calibration Laboratory (MSCL) at Johnson Space Center has been accredited by ACLASS (now,
ANAB) to both ISO 17025 and Z540.3 [107]. Accreditation, per se, is not required by NASA
policy, but compliance with the requirements of the Z540.3 standard is a requirement.
Compliance is assessed via the NASA Metrology and Calibration Program Office, under the
Office of Safety and Mission Assurance (OSMA).

6.2 DOE/NRC – Department of Energy and the Nuclear Regulatory Commission
Some federal agencies may not specifically require all calibrations to be performed in strict
compliance with national or international standards. However, many of these agencies still
recognize, and will readily accept, calibrations which have been performed by an accredited
laboratory. Department of Energy (DoE) laboratories have been accepting accredited
calibrations from outside commercial suppliers for well over a decade now [108]. See Figure 3.
Many of the DoE calibration laboratories have themselves, attained accreditation. In 1998, Pettit
[109] published the following account:
Pettit – Sandia Primary Standards Laboratory (1998): [109]
“Accreditation can offer many benefits to a testing or calibration laboratory, including increased marketability of
services, reduced number of outside assessments, and improved quality of services. Compared to IS0 9000
registration, the accreditation process includes a review of the entire quality system, but in addition a review of
testing or calibration procedures by a technical expert and participation in proficiency testing in the areas of
accreditation.
Within the DOE, several facilities have recently become accredited in the area of calibration, including Sandia
National Laboratories, Oak Ridge, AlliedSignal FM&T; Lockheed Martin Idaho Tech. Co., and Pacific
Northwest National Lab.
Recently, representatives from these and other DOE facilities throughout the U.S. formed a DOE Accreditation
Committee under the sponsorship of the DOE Technical Standards Program. This committee will hold its first
open meeting on September 23-24, 1998 at the National Institute of Standards and Technology in
Gaithersburg, MD with the Goal of developing procedures for sharing and coordinating information within DOE
on accreditation issues.”

Figure 3. Department of Energy approves the use of accredited calibration laboratories (used w/ permission)
The Nuclear Regulatory Commission (NRC) is an associated federal agency whose calibration
requirements have been codified into law via the CFR for over 45 years.
10 CFR 50 Appendix B, Requirement XII. (1970 - present): [10]
“Measures shall be established to assure that tools, gages, instruments, and other measuring and testing
devices used in activities affecting quality are properly controlled, calibrated, and adjusted at specified periods
to maintain accuracy within necessary limits.”
For many years the NRC’s calibration requirements, specified by this “Basic Requirement 12” in
Appendix B of 10 CFR §50, were augmented by NQA-1 supplementary requirements [110]
[111].
ANSI/ASME NQA-1 Supplement 12 S-1, Supplementary Requirements for Control of Measuring and Test
Equipment (1986): [110], [111]
1. General: This Supplement provides amplified requirements for control of measuring and test
equipment. It supplements the requirements of Basic Requirement 12 of this Standard and shall be
used in conjunction with that Basic Requirement when and to the extent specified by the organization
invoking this Standard.
2. Selection: Selection of Measuring and Test Equipment shall be controlled to assure that such items
are of proper type, range, accuracy, and tolerance to accomplish the function of determining
conformance to specified requirements.
3. Calibration and Control

3.1 Calibration: Measuring and test equipment shall be calibrated, adjusted and maintained at
prescribed intervals or, prior to use, against certified equipment having known valid
relationships to nationally recognized standards. If no nationally recognized standards exist,
the basis for calibration shall be documented.
3.2 Control: The method and interval of calibration for each item shall be defined, based on the
type of equipment stability characteristics, required accuracy, intended use, and other
conditions affecting measurement control. When measuring and test equipment is found to
be out of calibration, an evaluation shall be made and documented of the validity of previously
inspected or tested. Out-of-calibration devices shall be tagged or segregated and not used
until they have been recalibrated. If any measuring or test equipment is consistently found to
be out of calibration, it shall be repaired or replaced. A calibration shall be performed when
the accuracy of the equipment is suspect.
3.3 Commercial Devices: Calibration and control measures may not be required for rulers, tape
measures, levels, and other such devices, if normal commercial equipment provides adequate
accuracy.
4. Handling and Storage: Measuring and test equipment shall be properly handled and stored to
maintain accuracy.
5. Records: Records shall be maintained and equipment shall be suitably marked to indicate calibration
status.

Various revisions of IEEE STD 498 over the years (1975, 1980, 1985, 1990) [112] [113] [114]
[115], also specified calibration requirements/guidance for NRC facilities, as stated in NRC
Inspection Manual, Inspection Procedure 35750 [116].
NRC Inspection Manual: Procedure 35750 (1992): [116]
“General Guidance:
The regulatory basis for control of M&TE is found in 10 CFR 50, Appendix B, Criterion XII. The method by
which the licensee will comply with this requirement is normally described in Chapter 17 of the FSAR [Final
Safety Analysis Report].
IEEE 498-1980, ‘IEEE Standard Requirements for the Calibration and Control of Measuring and Test
Equipment Used in Nuclear Facilities’ provides guidance for establishing a program for the control and
verification of the accuracy of M&TE.
Even though a regulatory guide does not currently exist to endorse IEEE Standard 498, this inspection
procedure was written using the guidelines of IEEE Standard 498 because the majority of the guidance is
generic in nature.”
IEEE STD 498 was withdrawn in July of 1995. See Appendix B for the requirements of IEEE
STD 498, as they related to test uncertainty ratio to mitigate incorrect calibration decisions, and
how these requirements evolved over two decades.
For a number of years, the nuclear industry has also relied on the Nuclear Procurement Issues
Committee (NUPIC) Document No. 28 and No. 29 for the procurement of 3rd party supplier
“commercial-grade” calibrations. Document No. 28 is the Commercial Grade Calibration
Service Checklist [117] and Document No. 29 is the Commercial Grade Calibration Services
Checklist Implementation Guidelines [118].
An endeavor of the NRC to use and accept commercial calibrations from laboratories accredited
to ISO 17025, in lieu of surveys/audits, was documented in NISTIR 6989 in 2003 [119]. The
abstract of this report is provided here.
NIST IR 6989 (2003): [119]
“Representatives of the Nuclear Utilities Industry and the laboratory accreditation community met to discuss
Nuclear Regulatory Commission (NRC) recognition of the use of accredited commercial calibration
laboratories as suppliers of commercial grade calibration services to the regulated utilities.
The purpose of the meeting was to map out a strategy for approaching the NRC concerning the issue of
calibration laboratory accreditation and the National Cooperation for Laboratory Accreditation (NACLA).
The intent is to gain NRC endorsement of laboratory accreditation based on ISO/IEC 17025 as a means to
qualify calibration service suppliers for nuclear work. In support of this effort, a comparison between the
requirements of ISO/IEC 17025: General requirements for the competence of testing and calibration
laboratories and the audit requirements of the Nuclear Utilities Procurement Issues Committee (NUPIC)
Commercial Grade Survey Checklist for Calibration Services was prepared.”

The Bellefonte Nuclear Generating Station (BLN), regulated by the NRC, states:
NRC – Bellefonte (BLN) Nuclear Generating Station – Combined License (COL) – Final Safety Analysis
Report (FSAR) – Chapter 17, Quality Assurance, Section 17.5.4.7 (2009): [120]
…As discussed in the QAPD, the applicant also remains responsible for ensuring that the items or services
are suitable for the intended application and for documenting the evaluation that supports this conclusion...
• SRP Section 17.5 paragraph II.L.8 establishes provisions for the procurement of commercial-grade
calibration services for safety-related applications. As an exception to these provisions, the QAPD proposes
that procurement source evaluations and selection measures not be required, provided that all of the following
conditions are met…
-Purchase documents require reporting as-found calibration data when calibrated items are found to
be out of tolerance.
-A documented review of the supplier’s accreditation will be performed and will include a verification of
the following:
•The calibration laboratory holds a domestic accreditation by any one of the following accrediting bodies,
which are recognized by the International Laboratory Accreditation Cooperation (ILAC) Mutual Recognition
Arrangement (MRA):
-NVLAP administered by NIST
-American Association for Laboratory Accreditation (A2LA),
-ACLASS Accreditation Services (ACLASS),
-International Accreditation Service (IAS),
-Laboratory Accreditation Bureau (L-A-B).
•The accreditation encompasses ANS/ISO/IEC 17025, ‘General Requirements for the Competence of Testing
and Calibration Laboratories.’
•The published scope of accreditation for the calibration laboratory covers the necessary measurement
parameters, ranges, and uncertainties.
NRC staff evaluated and found to be acceptable the NVLAP and A2LA accreditation programs
(Ref. 17.5-6). The staff subsequently determined that the accreditation programs of ACLASS,
L-A-B, and IAS are also recognized by the ILAC MRA and are therefore acceptable (Ref. 7, 8,
and 9).
In addition to acceptance of NVLAP and A2LA accredited calibrations, the NRC has authored
letters addressed to IAS [121], L-A-B [122], ACLASS/ANAB [123], and PJLA [124], attesting
to the acceptance of calibrations from laboratories accredited by these AB’s, which are all
signatories to the ILAC MRA or Mutual Recognition Arrangement [125] [126] [127] [128].
Reports in 2010 had conveyed some confusion on behalf of calibration suppliers as to the NRC’s
requirements [129]. However, the most recent status of the NRC’s program of accepting
accredited commercial calibrations is found in the March 2015 version of the Nuclear Energy
Institute’s NEI 14-05A, “Guidelines for the Use of Accreditation in Lieu of Commercial Grade
Surveys for Procurement of Laboratory Calibration and Test Services [130]”. A chronology of
some historical standards and guidelines, prescribing the DOE & NRC requirements for the
calibration and control of measuring and test equipment throughout history, is given in Table 3.

Table 3A. DOE and NRC Calibration Standards, Guidelines, and Requirements
YEAR DOCUMENT NUMBER DOCUMENT OR SECTION TITLE FOR M&TE CALIBRATION
1969 RDT F3-2T Calibration System Requirements [AEC: Atomic Energy Commission]
1970 AEC 10 CFR 50 Appendix B, XII Control of Measuring and Test Equipment [Basic Requirement 12]
1971 ANSI N45.2 1971 Section 13: Control of Measuring and Test Equipment
Safety Guide 28: Quality Assurance Program Requirements (Design and Construction) [Endorsed ANSI N45.2-
1972 AEC/NRC Regulatory Guide 1.28 1971, including Section 13 on Control of Measuring and Test Equipment]
1973 RDT F3-2T Calibration System Requirements [AEC: Atomic Energy Commission – Feb 1973]
IEEE STD 498-1975 IEEE Standard Supplementary Requirements for the Calibration and Control of Measuring and Test Equipment
1975 Used in the Construction and Maintenance of Nuclear Power Generating Stations (used with ANSI N45.2)
ANSI N45.2.16
1976 ANSI N18.7-1976 [ANS 3.2] Administrative Controls and Quality Assurance for the Operational Phase of Nuclear Power Plants.
Section 8: Procedures for Control of Measuring and Test Equipment for Surveillance Tests, Procedures, and
1977 NRC Regulatory Guide 1.33 Calibrations.
1977 ANSI/ASME N45.2.16 Maintenance of Nuclear Facility [Endorsed ANSI N18.7-1976/ANS-3.2]
1978 ANSI N46.2-1978 Quality Assurance Program Requirements for Post Reactor Nuclear Fuel Cycle Facilities
1979 ANSI/ASME NQA-1-1979 Basic Requirement 12, Control of Measuring and Test Equipment
1979 CONST-QAPP 12 Quality Assurance Project Plan: Control of Measuring and Test Equipment
1979 Regulatory Guide 1.28 (Rev.2) Endorsed ANSI/ASME N45.2-1977 with supplemental provisions
IEEE Standard Requirements for the Calibration and Control of Measuring and Test Equipment Used in the
1980 IEEE 498-1980 Construction and Maintenance of Nuclear Power Generating Stations
1981 DOE O 5700.6 Quality Assurance [Note: No mention of calibration requirements]
1981 DOE O 5700.6A Quality Assurance; Attachment 1, Basic Premise 3 – calibration
1981 NUREG-0800 (SRP 17.1) Standard Review Plan, section 17.1.12 Control of Measuring and Test Equipment
1981 NUREG-0800 (SRP 17.2) Standard Review Plan, section 17.2.12, Activities related to Control of Measuring and Test Equipment
Procedures Evaluation Checklist for Maintenance, Test and Calibration Procedures Used in Nuclear Power
1982 NUREG/CR-1369, Rev.1 Plants
Requirement No. 12, Control of Measuring and Test Equipment
1983 ANSI/ASME NQA-1-1983 Supplement 12 S-1, Supplementary Requirements for Control of Measuring and Test Equipment
1983 ASME NQA-2-1983 Quality Assurance Requirements for Nuclear Facility Applications
1985 Regulatory Guide 1.28 (Rev.3) [Endorsed ASME NQA-1-1983 via NQA-1a-1983 addenda]
IEEE Standard Requirements for the Calibration and Control of Measuring and Test Equipment Used in Nuclear
1985 ANSI/IEEE 498-1985 Facilities [title change from 1980 version]
1986 DOE O 5700.6B Quality Assurance
Nuclear Quality Assurance. Requirement No. 12, Control of Measuring and Test Equipment
1986 ASME NQA-1-1986 Supplement 12 S-1, Supplementary Requirements for Control of Measuring and Test Equipment
1989 ASME NQA-1-1989 Nuclear Quality Assurance. Requirement No. 12, Control of Measuring and Test Equipment
1989 ASME NQA-3-1989 Basic Requirement 12, Control of Measuring and Test Equipment
1990 NUREG-0800 (SRP 17.3) Standard Review Plan, Section 17.3.B.9, Measuring and Test Equipment Control
1990 DOE 4330.4A Maintenance Management Program, Control and Calibration of Measuring and Test Equipment
1991 DOE O 5700.6C Quality Assurance; section 9.b.2.a “Criterion 5” and section 9.b.2.d “Criterion 8”
1991 NCSL RP-10 (Rev.1) Establishment and Operation of an Electrical Utility Metrology Laboratory
IEEE Standard Requirements for the Calibration and Control of Measuring and Test Equipment Used in Nuclear
1991 IEEE 498-1990 Facilities

Table 3B. DOE and NRC Calibration Standards, Guidelines, and Requirements
YEAR DOCUMENT NUMBER DOCUMENT OR SECTION TITLE FOR M&TE CALIBRATION

1992 NRC Inspection Manual Inspection Procedure 35750, QA Program – Measuring and Test Equipment
Guideline to Good Practices for Control and Calibration of M&TE at DOE Nuclear Facilities [Replaced in 2001
1993 DOE-STD-1054-93 by DOE G 433.1-1, section 4.11, Control and Calibration of Measuring and Test Equipment]
Basic Requirement 12, Control of Measuring and Test Equipment [Incorporated NQA-2 & NQA-3. Subpart
1994 ASME NQA-1-1994 2.16 endorses IEEE 498-1985]
1994 DOE RW-0333P (Rev. 1) Quality Assurance Requirements and Description. Section 12.0, Control of Measuring and Test Equipment
1994 DOE 4330.4B Section 3.5.4 and Chapter 12 –Control and Calibration of Measuring and Test Equipment
1996 DOE O 430.1 Cancels DOE 4330.4B – Maintenance Management Program.
1998 DOE O 414.1 Quality Assurance: Section 4.b.2.a.4 “Criterion 5” and 4.b.2.d.2. “Criterion 8” [Supersedes 5700.6C-1991]
1999 DOE O 414.1A Quality Assurance: Section 4.b.2.a.4 “Criterion 5” and 4.b.2.d.2. “Criterion 8”
Undated NUPIC Document 28 (Rev.1) Commercial Grade Calibration Services Checklist [Publication date & revision history unknown]
2000 NUPIC Document 29 (Rev.1) Commercial Grade Calibration Services Implementation Guidelines
2001 DOE O 433.1 Attachment 1, section 3(k), Control and Calibration of Measuring and Test Equipment
2001 DOE G 433.1-1 Section 4.11, Control and Calibration of Measuring and Test Equipment [Replaces DOE-STD-1054-93]
2003 NISTIR 6989 Comparison of ISO/IEC 17025 with the NUPIC Audit Checklist
2004 ASME NQA-1-2004 (w/ 2007 Addendum)
2004 NCSL RP-10, Rev.2 Establishment and Operation of an Electrical Utility Metrology Laboratory
2005 DOE O 414.1B Section 4.a.3, 4.b.5.d “Criterion 5” and 4.b.8.b “Criterion 8”
2005 DOE O 414.1C Section 4.a.3, 4.b.5.d “Criterion 5” and 4.b.8.b “Criterion 8”
Management System for Facilities and Activities: Safety Requirements. Topic 5.15 [activities for inspection,
2006 IAEA GS-R-3:2006 testing, verification and validation]
Application of the Management System for Facilities and Activities: Safety Guide. Topic 5 [Control of
2006 IAEA GS-G-3.1:2006 Products]
2006 NUPIC Document 29 (Rev.2) Commercial Grade Calibration Services Implementation Guidelines
Standard Review Plan, Section 17.5 (IIL) Control of Measuring and Test Equipment (10CFR50, Appendix B
2007 NUREG-0800 (SRP 17.5) Criterion XII)
2007 DOE O 433.1A Attachment 1, section 2.b.3.k, Control and Calibration of Measuring and Test Equipment
Application of the Management System for Nuclear Facilities: Draft Safety Guide. Topic 5.15 to 5.19 [Control
2007 IAEA DS 349:2007 of Products]
ASME NQA-1-2008 w/NQA-1a-
2008 Basic Requirement 12, Control of Measuring and Test Equipment
2009 Addenda
2008 DOE RW-0333P (Rev.20) Section 12.0, Control of Measuring and Test Equipment
2010 DOE O 433.1B Attachment 2, section 2 (j) Maintenance Tool and Equipment Control
2010 Regulatory Guide 1.28 (Rev.4) [Endorsed ASME NQA-1-2008 w/NQA-1a-2009 addenda]
2011 DOE O 414.1D Attachment 2: Section 5.d “Criterion 5” and 8.b “Criterion 8”
Quality Assurance Applications for Nuclear Facility Applications; Basic Requirement 12, Control of Measuring
2012 ASME NQA-1-2012 and Test Equipment
2012 ANSI/ANS 3.2-2012 Section 3.5.3.15, Calibration and Test Procedures; Section 3.12, Control of Measuring and Test Equipment
2015 NUREG-0800 (SRP 17.5) Standard Review Plan, Section 17.5 (IIL) Control of Measuring and Test Equipment (Criterion XII)
Guidelines for the Use of Accreditation in Lieu of Commercial Grade Surveys for Procurement of Laboratory
2015 NEI 14-05A Calibration and Test Services.

6.3 DOD: Department of Defense
Several branches of the Department of Defense rely on ISO 17025 and/or Z540.3 to augment
their calibration programs. The following are several examples of such use.
Air Force: AFMETCAL (AFLCMC / WNM) Air Force Metrology and Calibration Traceability AFD-130625
(2013): [131]
“The AFMETCAL program adheres to the general requirements of ISO 17025.
Calibration Services from Non-USAF Laboratories:
Some USAF owned items cannot be calibrated within the USAF calibration laboratory structure. These items
represent approximately 1% of the USAF calibration workload. In order to obtain calibration support for these
items, it is necessary to send them to Non-USAF calibration laboratories...
2. The Non-USAF laboratory must comply with the general requirements of ISO 17025.
3. The Non-USAF laboratory must provide information on their quality system demonstrating their compliance
with the general requirements of ISO 17025. This may include information regarding accreditation or audits
performed by other parties.
6. Information obtained regarding the Non-USAF laboratory's quality system, its compliance with the general
requirements of ISO 17025, and a copy of the calibration report must be retained by the organization obtaining
the calibration service until the calibration expiration date.”
Army: AR 750-43: Army Test, Measurement, & Diagnostic Equipment (2014): [132]
“6-7 Commercial Contractor Support
b. All contracts with a commercial laboratory for calibration services will specify that the commercial laboratory
adhere to International Organization for Standards (ISO)/IEC 17025:2005 (or later) for all measurement
parameters required for the calibration. Commercial calibration laboratories that are accredited to ISO/IEC
17025 and recognized as such by the International Laboratory Accreditation Cooperation, are not subject to
paragraph 6–17 of this regulation. Nonaccredited commercial laboratories will adhere to paragraph 6–17.”
Navy: NACLA Navy Partnership – Naval Surface Warfare Center, Corona Division: [133] [134]
“…the Measurement Science Department of the Naval Surface Warfare Center, Corona Division, is requesting
that NACLA investigate the establishment of a NACLA Recognition Program that would provide a conformity
assessment process for assessment bodies (including but not limited to Accreditation Bodies) that wish to
assess organizations to the requirements of ANSI/NCSL Z540.3 – 2006, Requirements for the Calibration of
Measuring and Test Equipment. (Referred herein as the National Standard).
This National Standard has great potential in reducing “risk” associated with calibration measurement values
in the many calibration systems Suppliers employ in producing their end product. The benefits associated with
reducing risk are many and varied, depending on the industry… Examples of Industries that will benefit from
implementing Z540.3 include, but are not limited to:
• Packaging
• Machining
• Transportation
• Healthcare
• Energy
• Manufacturing
• Chemistry
The goal is to establish a process that will promote uniformity in assessment to the supplemental requirements
of sub-clause 5.3.

Since the Navy published the preceding account, NACLA did indeed develop a program in 2010
for assessment of accreditation bodies that seek to provide laboratory accreditation to Z540.3
[135]. Several accreditation bodies including A2LA [136], L-A-B [137], ANAB [138], IAS
[139], and PJLA [140] now offer calibration laboratory accreditation to Z540.3 as a supplement
to ISO 17025 accreditation.
6.4 Automotive Industry

ISO/TS 16949 [141] is an international technical specification that prescribes quality
management system requirements for the automotive industry. It replaces the QS9000 standard
developed earlier in 1994 by the U.S. automakers. Although there is no specific government
regulatory requirement in the U.S. for automotive manufacturers or their suppliers to be certified
to ISO/TS 16949, vehicle assembly plants and supply chain manufacturers have been encouraged
to obtain certification to this standard; it enjoys ubiquitous implementation and has become a de
facto standard for this sector. Developed by the International Automotive Task Force (IATF), it
was approved and published by ISO in 2009, building upon the requirements in ISO 9001 and
adding language for calibration pertaining to ISO 17025.
ISO TS 16949: Quality Management Systems — Particular Requirements for the Application of ISO
9001:2008 for Automotive Production and Relevant Service Part Organizations (2009): [141]
“7.6.2 Calibration/verification records
Records of the calibration/verification activity for all gauges, measuring and test equipment, needed to provide
evidence of conformity of product to determined requirements, including employee- and customer-owned
equipment, shall include
 equipment identification, including the measurement standard against which the equipment is
calibrated,
 revisions following engineering changes,
 any out-of-specification readings as received for calibration/verification,
 an assessment of the impact of out-of-specification condition,
 statements of conformity to specification after calibration/verification, and notification to the customer if
suspect product or material has been shipped.
7.6.3.1 Internal laboratory…
NOTE Accreditation to ISO/IEC 17025 may be used to demonstrate the organization's in-house laboratory
conformity to this requirement but is not mandatory.
7.6.3.2 External laboratory
External/commercial/independent laboratory facilities used for inspection, test or calibration services
by the organization shall have a defined laboratory scope that includes the capability to perform the required
inspection, test or calibration, and either
 there shall be evidence that the external laboratory is acceptable to the customer, or
 the laboratory shall be accredited to ISO/IEC 17025 or national equivalent.
NOTE 1 Such evidence may be demonstrated by customer assessment, for example, or by customer-
approved second party assessment that the laboratory meets the intent of ISO/IEC 17025 or national
equivalent.
NOTE 2 When a qualified laboratory is not available for a given piece of equipment, calibration services may
be performed by the equipment manufacturer. In such cases, the organization should ensure that the
requirements listed in 7.6.3.1 have been met.”
Similar accreditation requirements also existed in section 4.11.2.b.1 of QS-9000 in 1998 [141A].

6.5 FAA – Federal Aviation Administration
FAA calibration requirements for test and inspection equipment and tools are codified in 14 CFR
§145.109 [11]. The FAA requires the following:
FAA: 14 CFR §145.109 – Equipment, Materials, and Data Requirements. [11]
“(a) Except as otherwise prescribed by the FAA, a certificated repair station must have the equipment, tools,
and materials necessary to perform the maintenance, preventive maintenance, or alterations under its repair
station certificate and operations specifications in accordance with part 43. The equipment, tools, and material
must be located on the premises and under the repair station's control when the work is being done.
(b) A certificated repair station must ensure all test and inspection equipment and tools used to make
airworthiness determinations on articles are calibrated to a standard acceptable to the FAA…”
FAA policy pertaining to this requirement in 14 CFR §145.109 is given below.
FAA Order 8900.1, Vol 6, Chap 9 Part 145 Policy; Flight Standards Information System – FSIMS (2014): [142]
“1) The ASI [Aviation Safety Inspector] should verify:
a) The repair station is calibrating MTE [Measuring & Test Equipment] per intervals, procedures, and the
system described in the RSM [Repair Station Manual] or QCM [Quality Control Manual].
b) All MTE are calibrated and traceable to a standard acceptable to the Federal Aviation Administration
(FAA), to include those recommended by the manufacturer, and the National Institute of Standards and
Technology (NIST) or other national authority.
NOTE: The part 145 rule states that tooling used to make airworthiness determinations must be calibrated to
a standard acceptable to the FAA.
Those standards may be derived from the NIST, to a standard provided by the equipment manufacturer, or
other recognized standards. The International Bureau of Weights and Measures (BIPM) is a recognized
authority that maintains a global list of National Metrology Institutes (NMI). The BIPM Web site lists the NMI
signatory countries that participate in the International Committee for Weights and Measures (CIPM). The
CIPM Mutual Recognition Arrangement (MRA) signatories are acceptable to the FAA and can be found at
http://www.bipm.org.
There are many accreditation bodies that provide third-party laboratory accreditation. The International
Laboratory Accreditation Cooperation (ILAC) establishes a global network for accreditation of laboratory and
testing facilities. Signatories to the ILAC MRA are in full conformance with the standards of International
Organization for Standardization (ISO)/International Electrotechnical Commission (IEC) 17011. ILAC MRA
signatories are acceptable to the FAA and can be found at http://ww.ilac.org.
Accredited laboratories have already established traceability through the assessment and accreditation
process under ISO/IEC 17025. No further documentation is required once traceability is confirmed to a
recognized accredited laboratory. Additionally, for foreign equipment, the standard of the country of
manufacture may be used if acceptable to the Administrator.”

Additional FAA calibration requirements are documented in Order 6200.4G [143].
FAA Order 6200.4G – National Test Equipment Program Management (2010): [143]
“11. Calibration of Test Equipment…

j. All calibration service providers will minimally comply with ANSI/NCSL Z540-1, Calibration Laboratories and
Measuring and Test Equipment General Requirements for the Competence of Testing and Calibration
Laboratories or ISO/IEC 17025 (International Organization for Standardization / Calibration, Measurement
Gages and Tests Laboratories) [sic].
These are the guides used by the FAA for calibration laboratory organization, management, quality system,
personnel, environment, standards/equipment, reference materials, procedures, traceability, certification,
labeling, records, and reporting.”
Similar to NASA’s practice of requiring Z540.3 compliance in solicitations or RFP’s when

selecting a contractor to perform calibration work for the government, the FAA has engaged in
an analogous practice of specifying calibration requirements and standards when issuing
Performance Work Statements. The following is an excerpt from the solicitation for the
calibration contract at the FAA Logistics Center at Mike Monroe Aeronautical Center [144].
FAA Performance Work Statement – Mike Monroney Aeronautical Center – FAA Logistic Center (2008): [144]
FAA Test, Measurement & Diagnostic Equipment Calibration Services.
3. PURPOSE
“1.1.1 The calibration service is the key to ensuring documented accreditation and certification for all
government owned TMDE [Test Measurement & Diagnostic Equipment], and provides the government a
certification of calibration attesting each item performs within specified standards of tolerance. It is not possible
to separate GTMDE used for critical systems from GTMDE used for advisory systems, or to determine any
FAA site as more critical than another to the safe operation of the NAS [National Air Space System]. All
TMDE therefore requires certification to enable certification of critical FAA operating systems. This function is
determined as essential to the safety of human life… It is the FAA’s intent to enter into a performance based,
firm fixed price agreement for the services required. Offerors must develop and provide cost effective
solutions, with the opportunity to propose innovative alternatives that meet the below objectives:
 Provide calibration and certification of calibration attesting each individual item of TMDE
performs within specified standards of tolerance.
 Ensure calibration accreditation can be tracked for each piece of TMDE used to achieve
certification of critical NAS operating systems.
 Ensure Calibrations are performed on time in accordance with calibration due dates.
 Ensure Calibration services meet quality standards comparable to or better than best
commercial, industry practices…
Calibration Documentation, Compliance, and Accreditation Report Review: FAA shall perform random
audits/reviews of:
The Contractors’ ISO 9001 Surveillance Reports including the nonconformance reports. If applicable, the FAA
may review the Contractor’s ISO/IEC 17025 accreditation report and associated documentation.
6.3.3 WORKING STANDARDS ACCURACY
6.3.3.1 All Working Standards utilized by the Contractor in the calibration process shall be more accurate than
the TMDE being calibrated. Accuracy ratios (Working Standards to TMDE being calibrated) shall be
maintained at 4:1 as a minimum except in rare cases where the Contracting Officer’s Technical
Representative (COTR) approves less than a 4:1 ratio”.

Many aircraft rely on accurate measurement data from a variety of sensors being fed to flight
computers. Autopilots and fly-by-wire systems employ these measurements to make real-time
adjustments to the flight control system. Inaccurate calibration of these sensors can result in
significant consequences. For example, on February 25th, 2008, a B-2 bomber suffered a serious crash
at Andersen Air Force Base due to mis-calibrated Port Transducer Units (PTU’s) in the Air Data
System. The crash resulted in injuries to the co-pilot and the total loss of a $1.4 billion aircraft [145].
7. The National Technology Transfer and Advancement Act of 1995 (NTTAA)

For many decades, there has been a movement within federal agencies to incorporate voluntary
consensus standards into their programs and regulatory affairs, in lieu of government-unique standards.
For calibration requirements, this has been evidenced by NASA, the Department of Energy, the
Nuclear Regulatory Commission, the Department of Defense, the Federal Aviation Administration,
and many others. Agencies have come to rely on Z540.3 and/or ISO 17025 in support of their
calibration and metrology programs. For some, this was due in part to the National Technology
Transfer and Advancement Act of 1995 (NTTAA) [146]. For the Department of Defense, this was also
due to the issuance of what has become known as the “Perry Memo” in 1994 [147]. The Perry Memo
was largely responsible for the cancellation of a significant number of military standards in the mid
1990’s (see bottom of Table 1B). In many cases, these “Mil-Specs” were replaced by voluntary
consensus standards (e.g. MIL-STD-45662A was replaced by Z540.1 in 1995). This section provides
background information on government use of consensus standards as they relate to the NTTAA.
W. Perry – Department of Defense “Perry Memo” (1994): [147]
SUBJECT: Specifications & Standards - A New Way of Doing Business
“…the Department of Defense must increase access to commercial state-of-the-art technology and must
facilitate the adoption by its suppliers of business processes characteristic of world class suppliers.
I have repeatedly stated that moving to greater use of performance and commercial specifications and
standards is one of the most important actions that DoD must take to ensure we are able to meet our military,
economic, and policy objectives in the future. Moreover, the Vice President's National Performance Review
recommends that agencies avoid government-unique requirements and rely more on the commercial
marketplace. To accomplish this objective, the Deputy Under Secretary of Defense (Acquisition Reform)
chartered a Process Action Team to develop a strategy and a specific plan of action to decrease reliance, to
the maximum extent practicable, on military specifications and standards…
I wholeheartedly accept the Team's report and approve the report's primary recommendation to use
performance and commercial specifications and standards in lieu of military specifications and standards…
Waivers for the use of military specifications and standards must be approved…
…encourage contractors to propose non-government standards and industry-wide practices that meet the
intent of the military specifications and standards.
These standards… shall be considered as alternatives to those military specifications and standards…
Use of Non-Government Standards: …form partnerships with industry associations to develop non-
government standards for replacement of military standards….
Reducing Oversight: …reduce direct Government oversight by substituting process controls and non-
government standards in place of development and/or production testing and inspection and military-unique
quality assurance systems.
Education and Training: The Under Secretary of Defense (Acquisition and Technology) shall ensure that
training and education programs throughout the Department are revised to incorporate specifications and
standards reform. The Process Action Team's report… are a solid beginning that will increase the use of
performance and commercial specifications and standards. I encourage you and your leadership teams to be
active participants in establishing the environment essential for implementing this cultural change”.

Similar to the Perry Memo for the Department of Defense, the NTTAA set forth requirements for all
federal agencies regarding the use of voluntary consensus standards.
National Technology Transfer and Advancement Act (NTTAA) of 1995: [146]

Public Law 104-113
“Utilization of Consensus Technical Standards by Federal Agencies;

(1) In general.--Except as provided in paragraph (3) of this subsection, all Federal agencies and departments
shall use technical standards that are developed or adopted by voluntary consensus standards bodies,
using such technical standards as a means to carry out policy objectives or activities determined by the
agencies and departments.
(2) Consultation; participation.--In carrying out paragraph (1) of this subsection, Federal agencies and
departments shall consult with voluntary, private sector, consensus standards bodies and shall, when such
participation is in the public interest and is compatible with agency and departmental missions, authorities,
priorities, and budget resources, participate with such bodies in the development of technical standards.
(3) Exception.--If compliance with paragraph (1) of this subsection is inconsistent with applicable law or
otherwise impractical, a Federal agency or department may elect to use technical standards that are not
developed or adopted by voluntary consensus standards bodies if the head of each such agency or
department transmits to the Office of Management and Budget an explanation of the reasons for using
such standards. Each year, beginning with fiscal year 1997, the Office of Management and Budget shall
transmit to Congress and its committees a report summarizing all explanations received in the preceding
year under this paragraph.
(4) Definition of technical standards.--As used in this subsection, the term ’technical standards’ means
performance-based or design-specific technical specifications and related management systems practices”
The NTTAA was based on the White House Office of Management and Budget’s OMB Circular A-
119, which governs the use of voluntary consensus standards by federal agencies. The 1998 revision
of Circular A-119 [148] stated:
OMB Circular A-119 (1998): [148]

“Standards developed by voluntary consensus standards bodies are often appropriate for use in achieving
federal policy objectives and in conducting federal activities, including procurement and regulation. The
policies of OMB Circular A-119 are intended to:
(1) encourage federal agencies to benefit from the expertise of the private sector,
(2) promote federal agency participation in such bodies to ensure creation of standards that are useable by
federal agencies,
(3) reduce reliance on government-unique standards where an existing voluntary standard would suffice.
…the policy of the federal government, in its procurement and regulatory activities, is to:
(1) rely on voluntary standards, both domestic and international, whenever feasible and consistent with law
and regulation,
(2) participate in voluntary standards bodies when such participation is in the public interest and is compatible
with agencies' missions, authorities, priorities, and budget resources,
(3) coordinate agency participation in voluntary standards bodies so that . . . the most effective use is made of
agency resources . . . and that the views expressed by such representatives are in the public interest and do
not conflict with the interests and established views of the agencies."

The most recent 2016 revision to OMB Circular A-119 [149] states:
OMB Circular A-119 (2016): [149]
“…this Circular directs agencies to use standards developed or adopted by voluntary consensus standards
bodies rather than government-unique standards, except where inconsistent with applicable law or otherwise
impractical.
This Circular… describes procedures for satisfying the reporting requirements of the NTTAA. The policies in
this Circular are intended to minimize the reliance by agencies on government-unique standards. The Circular
also provides policy guidance to agencies on the use of conformity assessment in procurement, regulatory,
and program activities. This Circular replaces Office of Management and Budget (OMB) Circular No. A-119,
dated February 10, 1998. Many voluntary consensus standards are appropriate or adaptable for the Federal
government's purposes…
Conformity assessment includes sampling and testing, inspection, supplier’s declaration of conformity,
certification, and management system assessment and registration. Conformity assessment also includes
accreditation of the competence of those activities.
This Circular applies to all agencies and agency representatives who use standards or conformity
assessment and/or participate in the development of standards. “Agency” means any… establishment of the
Federal government. It also includes any regulatory commission or board…
Consistent with Section 12 (d)(1) of the NTTAA, all Federal agencies must use voluntary consensus standards
in lieu of government-unique standards in their procurement and regulatory activities, except where
inconsistent with law or otherwise impractical. In these circumstances, your agency must submit a report
describing the reason(s) for its use of government-unique standards in lieu of voluntary consensus standards
as explained in Sections 9-11.
In evaluating whether to use a standard… an agency should consider the following factors…(f) the ongoing
use of the standard by other agencies for the same or a similar requirement, the use of which in a particular
instance would increase consistency across the Federal government… (h) the prevalence of the use of the
standard in the national and international marketplaces…
As a general matter, standards being considered for use in regulation that specify nomenclature, basic
reference units, or methods of measurement or testing, and that are primarily empirical in their formulation, will
ordinarily warrant less scrutiny by an agency than standards that embody factors that are less objective…
In addition, the United States is obligated under the TBT [Technical Barriers to Trade] Agreement to use
relevant international standards, except where such standards would be an ineffective or inappropriate means
to fulfill the legitimate objective pursued. In particular, the TBT Agreement, Article 2.4, provides that: ‘Where
technical regulations are required and relevant international standards exist or their completion is imminent,
[WTO] Members shall use them, or the relevant parts of them, as a basis for their technical regulations…’
…agencies should consider using voluntary consensus standards, as described in this Circular, to achieve
their regulatory, procurement, and program needs, including for test methods, sampling procedures, and
protocols, if applicable. It may be appropriate for the agency to allow the use of multiple standards in order,
for example, to permit greater flexibility…
…agencies should recognize the possible contribution of private sector conformity assessment activities.
When properly conducted, conformity assessments conducted by private sector conformity assessment
bodies can increase productivity and efficiency in government and industry, expand opportunities for
international trade, conserve resources, improve health and safety, and protect the environment.
Working closely with NIST and OMB, agencies are encouraged to identify their conformity assessment needs
in such areas as regulatory compliance and…to assess whether the use of private sector conformity
assessment mechanisms in lieu of or in conjunction with government conformity assessment procedures
would be beneficial…”
Prior to the 2016 revision of Circular A-119, the Office of Management and Budget stated that:
U.S. Office of Management and Budget (OMB) (2014): [150]
“For two decades, it has been the policy of the United States Government to support the development and use
of efficient and effective standards and conformity assessment approaches that, when adopted by Federal
agencies, can address important regulatory, procurement, and policy objectives, such as increasing the net
benefits of Federal regulation. Promoting and using high-quality standards and standardization systems in turn
supports the broader goals of enhancing economic growth, innovation, and competition and of facilitating
international trade by avoiding the creation of unnecessary obstacles to trade.”
Encouragement of federal agencies to utilize voluntary consensus standards has been evident for nearly
30 years. The Administrative Council of the United States (ACUS) is an independent agency charged
with providing recommendations for improving federal agency procedures. In 1978, ACUS stated:
ACUS Recommendation 78-4 (1978): [151]
“Federal Agency Interaction with Private Standard-Setting Organizations in Health & Safety Regulation
(a ) Many federal agencies have authority to issue mandatory health or safety regulations relating to products,
materials, processes, practices or services that may be the subjects of voluntary standards prepared by
non-governmental organizations. Non-governmental standards, though not legally enforceable, have in
fact gained wide acceptance and a high degree of observance… Standards developed by private
organizations… or under the Department of Commerce voluntary standards program, are frequently
referred to as "voluntary consensus standards," and are the subject of this recommendation…
(c) Members of technical committees that develop voluntary consensus standards often have a wealth of
technical knowledge and expertise that agency staffs do not possess…
(f) The recommendation that follows is limited to agency interaction with standards-developing organizations
and use of voluntary consensus standards in the context of regulation of health or safety. The
recommendation may nevertheless be significant in relation to setting standards for other purposes: For
example, in conservation of energy and resources, environmental issues, and formulation of test methods
and definitions…
4. Each agency should, as a matter of general policy, regularly review standards or revisions proposed by
technical committees active in the areas of regulatory concern of the agency…
6. Agencies with authority to issue health or safety regulations should consider the use of existing relevant
voluntary consensus standards in developing mandatory standards. Voluntary consensus standards
should be considered with due caution and on a case-by-case basis. Ordinarily, standards which embody
judgmental factors should receive greater scrutiny when being considered by agencies for adoption into
regulations than standards which specify nomenclature, basic reference units, or methods of
measurement or testing, and which are primarily empirical in their formulation.
(7a) If the voluntary consensus standard adequately addresses the questions of health or safety and is being
substantially complied with by the affected industry, the agency may decide to take no further regulatory
steps, or, alternatively, to adopt the standard into its regulations (see paragraph (f) below), and direct its
primary regulatory efforts elsewhere…
(7e) Agencies should consider the "regulatory guide" approach as a means of effectively making use of
voluntary consensus standards. A "regulatory guide" is a formal declaration by the agency that
compliance with designated portions, or all, of a voluntary consensus standard will be considered an
acceptable method of compliance with a general mandatory standard appearing in either the governing
statute or the agency's regulations. When taking this approach, the agency should suitably publicize its
decision and reasons therefor.
(f) The agency may adopt a voluntary standard into its regulations either by placing the text of the standard
in the regulations, or, preferably, by incorporating the standard by reference pursuant to 1 CFR part 51.”

Section (f) in the preceding ACUS Recommendation 78-4 references a concept known as
“incorporation by reference”, whereby voluntary consensus standards can be directly referenced
within the text of the CFR. ACUS recommendation 2011-5 [152] explains this option, now codified in
Title 1 of the Code of Federal Regulations Section 51.1 [153]. The ACUS has stated:
ACUS Recommendation 2011-5, Incorporation by Reference (2011): [152]
“Incorporation by reference allows agencies to comply with the requirement of publishing rules in the Federal
Register to be codified in the Code of Federal Regulations (CFR) by referring to material published elsewhere.
The practice is first and foremost intended to—and in fact does—substantially reduce the size of the CFR. But
it also furthers important, substantive regulatory policies, enabling agencies to draw on the expertise and
resources of private sector standard developers to serve the public interest. Incorporation by reference allows
agencies to give effect to a strong federal policy, embodied in the National Technology Transfer and
Advancement Act of 1995 and OMB Circular A-119, in favor of agency use of voluntary consensus standards.
This federal policy benefits the public, private industry, and standard developers”.
Examples of federal agencies incorporating ISO 17025 ‘by reference” into the Code of Federal
Regulations are provided below.
46 CFR 162.060-36 – Quality Assurance Project Plan (QAPP) Requirements [154]

Coast Guard / Department of Homeland Security
“The approval testing and evaluation process must contain a rigorous Quality Assurance and Quality Control
program consisting of a QAPP developed in accordance with ISO/IEC 17025:2005(E), as amended ISO/IEC
17025:2005/Cor.1:2006(E) (incorporated by reference, see § 162.060–5). The independent laboratory
performing approval tests and evaluations is responsible for ensuring the appropriate Quality Assurance and
Quality Control procedures are implemented.”
10 CFR 429.110 – Enforcement Testing [155]

Department of Energy
“(3) Testing will be conducted at a lab accredited to the International Organization for Standardization (ISO) /
International Electrotechnical Commission (IEC), “General requirements for the competence of testing and
calibration laboratories,” ISO/IEC 17025:2005(E) (incorporated by reference; see§ 429.4). If testing cannot be
completed at an independent lab, DOE, at its discretion, may allow enforcement testing at a manufacturer's
lab, so long as the lab is accredited to ISO/IEC 17025:2005(E) and DOE representatives witness the testing.”
47 CFR 2.948 – Description of Measurement Facilities [156]

Federal Communications Commission
“(d) A laboratory that has been accredited with a scope covering the required measurements shall be deemed
competent to test and submit test data for equipment subject to verification, Declaration of Conformity, and
certification. Such a laboratory shall be accredited by an approved accreditation organization based on the
International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) Standard
17025, “General Requirements for the Competence of Calibration and Testing Laboratories.”

The definition of a “standard” or “technical standard” in OMB Circular A-119 is “a common and
repeated use of rules, conditions, guidelines or characteristics for products or related processes and
production methods, and related management systems practices.” Standards which are not considered
voluntary consensus standards are, “government-unique standards, which are developed by the
government for its own uses” and “standards mandated by law such as those contained in the United
States Pharmacopeia and the National Formulary…”
Regarding the use of voluntary consensus standards for quality systems in the healthcare industry, the
following was published by the FDA upon the revision of the cGMPs in 1997:
FDA Final Rule cGMP Quality System Regulation Preamble (1997): [157]
“This revision follows the suggestion underlying many comments on specific provisions that FDA generally
harmonize the CGMP requirements and terminology with international standards. ISO 9001:1994, ISO/CD
13485, and EN 46001 employ this terminology to describe the CGMP requirements. In addition, this title
[Quality System Regulation] accurately describes the sum of the requirements, which now include the CGMP
requirements for design, purchasing, and servicing controls. CGMP requirements now cover a full quality
system. FDA notes that the principles embodied in this quality system regulation have been accepted
worldwide as a means of ensuring that acceptable products are produced. While the regulation has been
harmonized with the medical device requirements in Europe, Australia, and Japan, as well as the
requirements proposed by Canada, it is anticipated that other countries will adopt similar requirements in the
near future.
FDA, however, did not adopt ISO 9001:1994 verbatim for two reasons. First, there were complications in
dealing with the issue of copyrights and, second, FDA along with health agencies of other governments does
not believe that for medical devices ISO 9001:1994 alone is sufficient to adequately protect the public health.
Therefore, FDA has worked closely with the GHTF [Global Harmonization Task Force] and TC 210 to develop
a regulation which is consistent with both ISO 9001:1994 and ISO/CD 13485. FDA made several suggestions
to TC 210 on the drafts of the ISO/CD 13485 document in order to minimize differences and move closer to
harmonization. In some cases, FDA has explicitly stated requirements that many experts believe are inherent
in ISO 9001:1994. Through the many years of experience enforcing and evaluating compliance with the
original CGMP regulation, FDA has found that it is necessary to clearly spell out its expectations. This
difference in approach does not represent any fundamentally different requirements that would hinder global
harmonization. In fact, numerous comments expressed their approval and satisfaction with FDA's effort to
harmonize the quality system requirements with those of ISO 9001:1994 and ISO/CD 13485…
FDA supports the international harmonization of standards and regulations governing medical devices and the
eventual mutual recognition of CGMP inspections between major device markets. While full achievement of
this goal is still in the future, the harmonization of quality standards is an important first step. FDA believes in
a step wise approach toward harmonization and eventual mutual recognition”.
The National Institute of Standards and Technology (NIST), is charged with providing an annual
report to the OMB summarizing the various federal agencies participation in the NTTAA. Excerpts
from several of these annual reports are provided here and contain information relevant to FDA, NRC
and other federal agency programs.

th
NIST 9 Annual Report on Federal Agency Use of Voluntary Consensus Standards and Conformity
Assessment (2005): [158]
"The Department of Health and Human Services HHS reported several activities including:
FDA’s participation in ongoing conformity assessment activities such as the ANSI Accreditation and
International Conformity Assessment, as well as American Society for Testing and Materials (ASTM)
Committee E-36 on Conformity Assessment allows FDA to ensure that its needs are met while utilizing
existing recognition and accreditation criteria.
FDA’s Center for Devices and Radiological Health allows a medical device manufacturer to submit a
Declaration of Conformity to a “recognized standard” as described in ISO/IEC Guide 22 in its standards
recognition program and has developed an MRA with the European Union on mutual recognition of each
other's conformity assessment procedures related to manufacture and marketing of medical devices. This
reduces costs for manufacturers and decreases the time to market for approved products.
The FDA Office of Regulatory Affairs (FDA/ORA) actively participates in the National Cooperation for
Laboratory Accreditation (NACLA), serving as a member of the NACLA Executive Board of Directors and
participating in the NACLA Recognition Committee for Accrediting Bodies who apply for mutual recognition.
This participation may form the basis for the future accreditation [to ISO 17025] of FDA laboratories.
With the idea of enhancing international credibility and recognition, FDA’s Center for Food Safety and Applied
Nutrition (CFSAN) is moving towards ISO [17025] accreditation of its own laboratories that perform regulatory
work…
The Department of Commerce reported that the Nuclear Regulatory Commission (NRC) now accepts
accreditation [to ISO 17025] by qualified laboratory accreditation bodies as an acceptable alternative so a
supplier audit, commercial-grade survey, or in-process surveillance for the qualification of commercial grade
calibration service suppliers. This reduces resource burdens on industry and eliminates costs related to
redundant audits. The Department also reported that:
NIST provides technical support for the Inter-American Accreditation Cooperation (IAAC). Such
arrangements/agreements are designed to harmonize conformity assessment practices and promote the
global acceptance of conformity assessment results from qualified bodies to minimize the need for and cost of
redundant conformity assessment activities.
Federal agencies continue to experience personnel turnover at all organizational levels due to reorganizations,
accelerated or early retirements, and normal attrition. These changes make it difficult for Federal agencies to
retain high-level managers who understand the importance of standards and who visibly support standards-
related activities. Staff turnover has also caused a decrease in “institutional memory” of standards policies,
responsibilities, and practices. To address this issue, NIST recently developed and is now providing training
for Federal employees who are engaged in developing standards and using them in regulation or procurement
actions. NIST is creating a handbook for Standards Executives so that they will have readily available the
information needed to make decisions about the use of standards.
Sustained high-level Federal agency leadership has been identified as the primary driver of successful NTTAA
implementation. Top agency leaders have the ability to direct policy and resources in ways that bring about
other desirable outcomes, such as increased Federal participation and collaboration with the private sector.
Ensuring that agency Standards Executives have the tools at hand to show how standards and the standards
making process contribute to their agency’s mission is a continuing priority”.

NISTIR 7413: Tenth (2006) Annual Report on Federal Agency Use of Voluntary Consensus Standards and
Conformity Assessment - Addendum (2006): [159]
“Standards may become part of conformance activities as they may provide an acceptable approach to be in
compliance with applicable laws and regulations. Also, FDA laboratories which conduct official product testing
are, or are in the process of becoming, ISO/IEC 17025 accredited. They have conducted staff training, are in
the process of writing a Laboratory Quality Assurance Manual centrally documenting Center policies and
procedures related to the official testing of regulated biological products, are implementing a quality
management software tool to assist in the effort, under direction of quality assurance managers hired to
coordinate the implementation of an ISO 17025-based quality system”
NISTIR 7503: Eleventh (2007) Annual Report on Federal Agency Use of Voluntary Consensus Standards and
Conformity Assessment – Addendum (2008): [160]
Establishment and use of standards result in benefits to FDA that include: standards that can assist reviewers
with assessment of product applications; international standards that can be used by multiple regulatory
regions, following our legal mandate to facilitate harmonization on an international level; standards that often
result in better utilization of limited internal resources; and direct participation by various stakeholders in
development of standards that results in a consensus among users, manufacturers and government
regulators on safety and effective use of regulated products.
NISTIR 7598: Twelfth (2008) Annual Report on Federal Agency Use of Voluntary Consensus Standards and
Conformity Assessment – Addendum (2009): [161]
“The FD&C Act requires the Center for Devices and Radiological Health to annually publish a list of voluntary
consensus standards ’recognized’ by the Agency for the use of manufacturers and others in meeting the
regulatory requirements of the FDA. The list of recognized Standards and several applicable guidance
documents are available at http://www.fda.gov/cdrh/stdsprog.html.”
As stated in the 12th annual NIST report addendum above, the FDA/CDRH maintains a database of
recognized consensus standards [84] [85]. For example, the list below indicates specific calibration
procedures which are formally recognized by the FDA. However, ISO 17025 and ANSI/NCSL
Z540.3 for calibration programs are not included in the FDA database of recognized standards.
FDA Standard
Standard Designation Specialty Task FR Publication
Recognition Developing Title of Standard
Number and Date Group Date
Number Organization
S3.7-1997 (Reaffirmed 2003) Method for Coupler Calibration of

4-164 ANSI Dental/ENT 3/16/2012
(Reaffirmed 2008) Earphones
Calibration and usage of ionization
12-61 IEC 61145 (1992-05) Radiology 5/3/1999
chamber systems for assay of radionuclides
Calibration and Usage of "Dose Calibrator"
12-141 IEEE N42.13-2004 Ionization Chambers for the Assay of Radiology 3/31/2006
Radionuclides
Ultrasonics -- Hydrophones -- Part 2:
Calibration for ultrasonic fields up to 40
12-278 IEC 62127-2 Edition 1.0 2007-08 Radiology 7/9/2014
MHz [Including: Technical Corrigendum
1:2008 and Amendment 1:2013]
51261 Second Edition 2013- Practice for calibration of routine
14-381 ASTM ISO Sterility 8/5/2013
04-15 dosimetry systems for radiation processing

Likewise, and as previously stated herein, ISO 17025 is also not currently listed as a “recognized
standard” by the IMDRF N15 document, “Final Report: List of International Standards Recognized by
IMDRF Management Committee Members as of March 2014” [87] [88].
The FDA/CDRH has published guidance on the use of consensus standards [86]. This guidance states,
“CDRH believes that conformance with recognized consensus standards can support a reasonable
assurance of safety and/or effectiveness for many applicable aspects of medical devices”. Although
ISO 17025 is not currently listed as a recognized consensus standard, the FDA has expressed positive
statements towards laboratory accreditation to ISO 17025. However, this dialogue was directed more
toward testing laboratories, rather than calibration laboratories [162].
FDA Guidance for Industry – Submission of Laboratory Packages by Accredited Laboratories (2009): [162]
“Since the time we issued the proposed rule, significant changes in laboratory accreditation have occurred.
For example, when we drafted the proposed rule, there was a trend towards the use of the International
Organization for Standardization (ISO) standard ISO 17025, "General Requirements for the Competence of
Testing and Calibration Laboratories," but no firm consensus (see 69 FR at 23461). Additionally, when we
drafted the proposed rule, parties disagreed as to the value of laboratory accreditation, and FDA’s own
laboratories were not accredited. Today, there is widespread agreement on ISO 17025, the laboratory industry
favors accreditation, and FDA's own laboratories are accredited. Moreover, the Administration’s Strategic
Framework expressly seeks better ways to ensure compliance with safety standards, while the Action Plan
and the GAO testimony reflect greater support for the use of accredited laboratories. Thus, given these and
other developments and the Action Plan's recommendation to issue guidance, we have decided to issue this
guidance document instead of proceeding with a final rule at this time… Rigorous accreditation standards give
us more confidence that accredited laboratories have the technical capability and trained personnel to perform
the specific methods for which they are accredited.”
As stated above, the FDA’s Office of Regulatory Affairs (ORA) laboratories are accredited to ISO
17025. The quality policy of the ORA laboratories states that, “ORA laboratories are committed to
laboratory accreditation according to the requirements of ISO/IEC 17025” [163].
The FDA has documented the historical precedent leading up to the decision to pursue accreditation,
including legal testimony regarding admissibility of scientific laboratory evidence in litigation [164].
FDA: Allergenic Products Advisory Committee Meeting Background Document: ISO 17025 Accreditation of the
Laboratory of Immunobiochemistry (2011): [164]
“Following these media awareness events, the FDA Senior Science Council recommended in 1998 that FDA
develop quality systems compliant to ISO 17025 (then known as ISO Guide 25) for all official testing activities.
ISO 17025 is also known as “General Requirements for the Competence of Testing and Calibration
Laboratories,” and merges the requirements for technical competence in testing and calibration with
requirements for quality systems. ISO 17025 is the main standard used by testing laboratories and is broken
into two main sections: management and technical requirements…
In response to the Senior Science Council’s recommendations, the Center for Biologics Evaluation and
Review (CBER) evaluated existing gaps in CBER testing and lot release activities, purchased and
implemented quality system software, developed the quality system policy manual, and provided basic ISO
17025 training for select laboratory personnel. To facilitate the goal of ISO 17025 accreditation, CBER created
the Division of Product Quality in 2006, and then decided that official testing be transferred to DPQ, thus
satisfying the goal that the testing would be performed in an accredited laboratory according the standards of
ISO 17025”.

The steps taken by the FDA/CBER Laboratory of Immunobiochemistry (LIB) to become accredited to
ISO 17025 were documented in 2011 by Menzies [165, 166]. CBER’s Laboratory Quality System
(LQS) officially received accreditation to ISO 17025 in October 2010 [167]. In 2012, the FDA’s
Division of Federal-State Relations (DFSR) announced a Cooperative Agreement Program for the ISO
17025 accreditation of state food testing Laboratories (U18) [168] [169]. This program provides
monetary grants for laboratories seeking ISO 17025 accreditation or to maintain existing accreditation.
The FDA has also stated, “ISO/IEC 17025 is one of the most important standards for calibration and
testing laboratories… Laboratory customers, regulatory authorities and accreditation bodies may also
use it in confirming or recognizing the competence of laboratories [169A]. ”
FDA: ISO/IEC 17025:2005 Accreditation for State Food Testing Laboratories (2012): [169]
“The intended outcome of this FOA [Funding Opportunity Announcement] is for microbiological and
chemical food analyses performed on behalf of State manufactured food regulatory programs to be
conducted within the scope of an ISO/IEC 17025:2005 accredited laboratory and the goal of achieving a
nationally integrated food safety system to be further advanced. This will be accomplished by preparing
the primary food testing laboratories for State manufactured food regulatory programs to achieve and
maintain ISO/IEC 17025:2005 laboratory accreditation. Currently accredited laboratories will also be
prepared for accreditation enhancements. Increased laboratory analyses from ISO/IEC 17025:2005
accredited labs, as would be accomplished through this cooperative agreement, will in effect serve to
increase the analytical capacity for FDA and enhance efforts to protect the food supply.”
The FDA has also recently issued a Final Rule on the Food Safety and Modernization Act (FSMA)
regarding accredited third party certification [170].
FDA: Accreditation of Third-Party Certification Bodies To Conduct Food Safety Audits and To Issue
Certifications (2015): [170]
“At our own initiative, we are removing the requirement to use a laboratory consistent with section 422 of the
FD&C Act and inserting a requirement in § 1.651(b)(3) to use a laboratory accredited under ISO/IEC
17025:2005 or another laboratory accreditation standard that provides at least a similar level of assurance in
the validity and reliability of sampling methodologies, analytical methodologies, and analytical results.”
Regarding the FSMA, the American Council of Independent Laboratories (ACIL) has stated [171],
ACIL Statement on the Food Safety Modernization Act (2011): [171]
“ACIL has advocated for years the FDA establish uniform requirements for private laboratory submission…
…ACIL has consistently and repeatedly urged FDA to adopt and recognize International Standards
Organization (ISO) accreditation [ISO 17025] as the primary basis for qualification of laboratories to submit
analytical data to FDA for any purpose. We are pleased that FSMA addressed these issues and look forward
to working with FDA on their implementation.”
Recognition and use of ISO 17025 by federal agencies, as well as utilization of the associated network
of private sector accreditation and certification bodies, is consistent with ACUS Recommendation
2012-7 [172] (below) and is addressed in an ACUS report by McAllister [173].

ACUS Recommendation 2012-7 (2012): [172]
“Federal agencies in diverse areas of regulation have developed third-party programs to assess whether
regulated entities are in compliance with regulatory standards and other requirements. Through these
programs, third parties are charged with assessing the safety of imported food, children’s products, medical
devices, cell phones and other telecommunications equipment, and electrical equipment used in workplaces.
In these regulatory third-party programs, regulated entities generally contract with third parties to carry out
product testing, facility inspections, and other regulatory compliance assessment activities in the place of
regulatory agencies. Regulatory agencies take on new roles in coordinating and overseeing these third-party
actors.
In some areas of regulation, Congress has directed federal agencies to develop a third-party program; in
others, regulatory agencies have developed programs under existing statutory authority. Several broad
reasons support the growing use of third-party programs in federal regulation. In many areas, federal
regulatory agencies are faced with assuring the compliance of an increasing number of entities and products
without a corresponding growth in agency resources. Third-party programs may leverage private resources
and expertise in ways that make regulation more effective and less costly. In comparison with other regulatory
approaches, third-party programs may also enable more frequent compliance assessment and more complete
and reliable compliance data. Because agencies can authorize third parties located in other countries to
undertake assessment activities, third-party programs may be particularly effective when regulated products or
processes are international in scope. Regulatory third-party programs raise a host of important questions.
Representing a partial privatization of the public function of implementing and enforcing regulatory law, they
are a form of “public-private governance,” in which private actors play roles that are traditionally viewed as
governmental in nature.
Frequently, regulatory third-party programs use the practices and terminology of an international conformity
assessment framework that has been developed by private-sector standards organizations. “Conformity
assessment” is defined in international standards as the “demonstration that specified requirements relating to
a product, process, system, person, or body are fulfilled.” International standards also set forth how the
organizations that conduct conformity assessment – “conformity assessment bodies,” which are usually
private organizations – should operate.
International standards have been developed for various types of conformity assessment bodies, including
testing bodies, certification bodies, and inspection bodies. Recognizing the assessment of regulatory
compliance as a form of conformity assessment, many federal agencies that have established third-party
programs have relied on conformity assessment standards and bodies. Agencies may require, for example,
that third parties that certify conformity with regulatory requirements operate in accordance with the
international standards for certification bodies. Federal agencies may also require that the third parties be
accredited by accreditation bodies that operate in accordance with international accreditation standards.
Accreditation bodies are established in many countries, and they may be either private or governmental.
Agencies that establish third-party programs generally cannot or do not delegate their regulatory authority to
conformity assessment bodies. Rather, agencies authorize conformity assessment bodies to perform certain
technical tasks to assess conformity, and regulatory agencies rely on these assessments in their own
enforcement of regulatory requirements.
The goal is to leverage expertise and resources in the private sector to serve regulatory objectives. A key
resource for agencies considering a regulatory third-party program is the National Institute of Standards and
Technology (NIST), which has the responsibility under the National Technology Transfer and Advancement
Act of 1995 to coordinate government conformity assessment activities with similar activities of private-sector
entities, with the goal of avoiding unnecessary duplication and complexity. Following Office and Management
and Budget (OMB) Circular A-119, NIST published guidance for federal agencies on conformity assessment
activities in 2000”.

Such recommendations were preceded by efforts of NCSLI in 2001 as documented by Harris [36A].
Harris, NIST SP-986. NIST Workshop on Conformity Assessment for a Changing Government (2001): [36A]
“Collaboration
▪ Met with Jim Turner, House Majority Counsel to Committee on Science, Commerce, Technology
July 27, 2001.
- Laboratory accreditation issues; lack of uniformity in acceptance of accreditation by Federal agencies,
effectiveness of NCSLI in representing interests of members
- Possible proposal to OMB [White House Office of Management and Budget]
Suggestions
Government agencies should recognize competence of laboratories accredited to international quality
standards
- Versus additional audits and requirements
- Compare ISO/IEC 17025 and NRC 10 CFR 50, find out if there are gaps, then address the gaps by
17025 updates
- Environmental, chemical, biological
Collaboration among Federal & accredited labs to provide traceable calibration services”.
The recent implementation of the FDA’s Medical Device Single Audit program (MDSAP) pilot
program and the FSMA are exemplary applications of using third party programs to assess regulatory
compliance. Additional benefit could further be realized from formal recognition of ISO 17025
accredited calibration laboratories during FDA audits, to include recognition as part of the MDSAP
program. This would be analogous to the current recognition of such accredited calibration
laboratories by the DOD, DOE, NRC, FAA, the automotive industry, and other federal agencies.
Perhaps the most direct indication of FDA support and recognition of ISO 17025 accredited
calibration laboratories was provided in March 2011 [173A] when addressing comments received in
response to a proposed FDA rule regarding temperature indicating devices for particular food
processing and packaging requirements.
FDA, Response to Comment 11: Docket FDA-2007-N-0265 (2011): [173A]
“One comment expressed concern about… documentation of accuracy of temperature-indicating devices and
reference devices. The comment suggested that the final rule should instead require documentation that
conforms to the standards established by the American National Standards Institute, National Conference of
Standards Laboratories (ANSI/NCSL) or the International Organization for Standardization, International
Electrotechnical Commission (ISO/IEC) for accrediting calibration laboratories. The comment stated that the
laboratory accreditation standards indicate acceptable reporting practices. The comment acknowledged that
the standards may be too prescriptive for food processors who perform their own calibrations.
(Response) We do not agree that the regulation should require the documentation of accuracy of temperature-
indicating devices and reference devices to conform to the standards specified in the comment for accrediting
calibration laboratories. Although FDA supports use of accredited calibration laboratories and recognizes that
the laboratories must maintain certain documentation for the accreditation, the records required by this final
rule are appropriately limited to those necessary to document that the temperature-indicating device was
tested for accuracy at sufficient frequency to ensure accuracy during processing. As acknowledged by the
comment, a requirement for processors to adhere to accreditation standards would impose an unnecessary
burden on those who successfully perform their own calibrations but are not accredited by ANSI/NCSL or
ISO/IEC”.

The FDA also commented on the use of measurement uncertainty and test accuracy/uncertainty ratios.
FDA, Response to Comment 9: Docket FDA-2007-N-0265 (2011): [173A]
“One comment expressed concern that the proposed rule did not mention measurement uncertainties or test
accuracy ratio, which are essential parameters for assuring an accurate calibration that are specified in
standards issued by the American National Standards Institute (ANSI) and the International Organization for
Standardization (ISO) for certification of calibration laboratories. The comment stated that the ANSI and ISO
standards provide a limit for measurement uncertainty and establish a minimum test accuracy ratio that is
commonly used by calibration facilities. According to the comment, although the proposed rule requires use of
a calibrated accurate reference device, the lack of specific calibration parameters may lead to inaccurate
calibrations for temperature indicating devices.
(Response) …We do not agree that the regulations should specify calibration parameters, such as those
relating to measurement uncertainties or test accuracy ratio, or require use of specific calibration standards,
such as the ANSI and ISO standards suggested by the comment.
Metrology authorities, in addition to ANSI and ISO, issue calibration standards, which may be revised or
replaced. It would be impractical for FDA to maintain in the regulations a current list of acceptable calibration
standards.”
However, as stated previously, certain branches of the FDA do indeed maintain a list of recognized
consensus standards [84] [85] [86], as does the IMDRF [87] [88]. Both the NTTAA [146] and the
Food and Drug Modernization Act encourage the use of such standards.
Z540 has existed for 22 years (1994 to 2016) with a single revision in 2006 when Z540.3 replaced
Z540.1. Likewise, ISO 17025 has existed for 17 years (1999 to 2016) with a single revision in 2005.
While hundreds of documentary calibration standards exist for individual equipment types (i.e.,
calibration procedures/methods), Z540.3 and ISO 17205 are the two standards for calibration systems.
Revision and/or replacement of these standards has been relatively infrequent for nearly two decades.
Z540.3 addresses how measurement uncertainty and test uncertainty ratio (TUR) can influence false
accept/reject decisions for all calibration and measurement or test processes, resulting in incorrect in-
or-out of tolerance decisions (see also appendices A & B). The U.S. Pharmacopeia (USP) has also
recognized the importance ISO 17025 and of test accuracy (or uncertainty) ratios [173B] [173C]
[173D] [174E], regarding their effect on the risk of making false accept and false reject decisions.
USP Subcommittee on Certified Reference Materials [RM] (2007): [173B]
“The impact of an RM uncertainty, unless it is negligible, is that it will alter the likelihood that a quality control
laboratory will make an incorrect administrative decision – passing an item that does not meet its acceptance
criteria (consumer risk) or failing to pass an item that does (producer risk)…
A criterion… can be considered… in terms of the relationship between the expanded uncertainty and the
acceptance criteria… This ratio is sometimes referred to as the test accuracy ratio (TAR) or test uncertainty
ratio... The choice of 4:1 is commonly used and was part of MIL-STD 45662A… …ASME [B89.7.3.1-2001]...
suggests a range of 10:1 to 3:1, and 4:1 and 3:1 have been more commonly used in recent years. Based on
these considerations, a choice of 4:1 seems to be a reasonable default choice and is the TAR used by the
European Pharmacopeia (U. Rose, written communication, June 2007)”.
"A TAR of 4:1 is most commonly used and is USP’s choice” [173C].
Along with historical instrument reliability data (EOPR) [37] [38] [39] [39A] [A96], few aspects of a
calibration program or measurement science are more important than test uncertainty ratio for ensuring
that measuring instruments are “suitable for their intended use and capable of providing valid results”.
8. Summary and Conclusion
The formal origins of modern quality system documentary standards were born in the military and
aerospace industries of the 1950’s, of which, calibration systems were an integral part. Over time,
these military quality standards evolved into ISO 9001 with a multitude of organizations choosing to
become registered. However, specific language prescribing technical competency requirements for
calibration was not included in ISO 9001 (or similar standards such as ISO 13485). When the FDA
introduced equipment calibration requirements into the Good Manufacturing Practices in 1978
(codified in 21 CFR 820.61), the calibration requirements were based to some extent on MIL-C-45662
from 1960; however, no accuracy ratio or uncertainty requirements for decision rules were specified.
The FDA was a member of the NCSL TQM committee and a contributor to Z540.1, which replaced
MIL-STD-45662A; both standards required a 4:1 test accuracy ratio. Since then, standards for
calibration systems have evolved considerably, the latest of which (Z540.3) is a direct descendent of
these earlier standards and focuses on limiting the probability of incorrect acceptance decisions.
Companies registered to ISO 9001 or ISO 13485 are not necessarily required to demonstrate that
measuring equipment has been calibrated in a technically competent manner or that the risks of
erroneously accepting non-conforming instruments have been adequately controlled. Essentially, only
basic documentation requirements for calibrated equipment are required by ISO 9001, ISO 13485, and
21 CFR §820.72. This is contrasted with other federal agencies or regulated industries which employ
ISO 17025 and/or Z540.3 as guidance or requirements documents to ensure technical competence of
calibrations. Thus, there exists a significant disparity in the implementation of voluntary consensus
standards for calibration requirements among federal (or federally-regulated) entities, many of which
are engaged in critical operations which may affect the health and welfare of the general public.
The National Technology Transfer and Advancement Act of 1995 was instituted to encourage federal
agencies to adopt voluntary consensus standards for use in carrying out their responsibilities and
regulatory functions. Federal agencies are required to use such standards in lieu of “government
unique” standards, where such use is consistent with the agencies’ mission and objectives. Adoption
of such voluntary consensus standards harnesses the expertise of the private sector to aid in carrying
out agency objectives. The DOD, NASA, the DOE, the Nuclear Regulatory Commission, the Federal
Aviation Administration, and the automotive industry are examples of federal agencies, or industries
regulated by them, where ISO 17025 and/or Z540.3 have been officially recognized and adopted for
use in demonstrating that equipment has been calibrated in a technically competent manner. Not all
federal agencies have utilized these standards, as guidance documents or otherwise, and some agencies
rely on government unique standards to prescribe equipment calibration requirements.
Many federal agencies formally recognize calibrations which have been performed by a laboratory
accredited to ISO 17025 or Z540.3. Accreditation bodies which are signatories to the ILAC MRA
enjoy a wide degree of acceptance. Global recognition of accreditation bodies is facilitated via peer
review by other MRA signatories to ensure that all members are competent to accredit calibration
laboratories. Laboratory accreditation is a mature system, supported by an infrastructure of ILAC
policy documents on important issues such as metrological traceability, measurement uncertainty,
calibration measurement capability, decision rules, etc. These policies leverage the significant
contributions and expertise of groups such as the Joint Committee on Guides in Metrology (JCGM),
consisting of eight prominent scientific organizations representing diverse areas of chemistry, physics,
legal metrology, clinical chemistry, laboratory medicine, etc. The question could be posed, “If critical
government/regulated industries do not use these standards for guidance, or do not formally recognize
accredited calibration laboratories, then who should?”
1950
MIL-STD-120
1955
1963
Army: AR 750-25
FDA 21 CFR
Government Standards
Air Force: AFR 74-2 & BU-520
§133.4 (GMP)
Navy: SLIM & SecNav 4355.11 No Cal Requirements
Navy: Tech Memo 63-106
(for calibration)
1978
1970 1960
FDA 21 CFR
NATO AQAP-6 MIL-C-45662 §820.61 (GMP)
NATO AQAP-7 MIL-HDBK-52
1997
1973 1976
DEFSTAN 05-26 FDA 21 CFR §820.72

DEFSTAN 05-32 Inspection, Measuring, and Test
DEFSTAN 05-27 Equipment
1979 ~1977 1971

[1] NCSL RP-2
BS 5781-1 ISO CERTICO
BS 5781-2 (Now CASCO) NCSL RP-4
Voluntary Consensus Standards

NTTAA (1995)
(Quality Standards)
Guidelines NCSL RP-11

ISO 9000 Series
“…all Federal agencies and 1987

ISO 10012:2003
Measurement departments shall use technical
Management Systems: standards that are developed or ANSI/ASQC M1
(for calibration)
Requirements for
Measurement Processes adopted by voluntary consensus
and Measuring Equipment standards bodies…”
Public1978
Law 104-113
ISO Guide 25
ASTM E548 1994
ISO/IEC 17025:2005 UKAS M10 ANSI/NCSL Z540.1

General Requirements for the BS 6460-1 & Z540.1 Handbook
Competence of Testing and
Calibration Laboratories
HB 18.25
BS 7501/7502
EN 45001/45002
ANSI/NCSL Z540.3-2006
Requirements for the Calibration of
Measuring and Test Equipment
Figure 4. Should federal agencies formally recognize voluntary consensus standards for calibration?

This paper has been an attempt to provide relevant background information for federal agencies,
accreditation bodies, auditors, metrologists and quality personnel involved with calibration actives
which may benefit from the use of voluntary consensus standards Z540.3 and/or ISO 17025. Federal
agencies may derive efficiencies from the sharing of information and by maintaining a broad
awareness of inter-agency efforts to achieve common goals. Objective knowledge of calibration
programs among different agencies provides cross-sector insight to solutions implemented outside of
one’s own office of responsibility. Many common challenges may have already been addressed by
committees, colleagues, or counterparts in neighboring government agencies. Generating, managing,
and regulating calibration program requirements from the ground up can be costly and inefficient,
particularly when an elegant framework may already exist. If developed independently and without the
benefit of lateral awareness, these individual programs can become splintered, yielding
incompatibilities without mutual recognition and/or acceptance.
There is a deep and rich history of the development of calibration standards and guidelines which have
found application among regulated and non-regulated industries alike. When consideration is extended
to the implications of the National Technology Transfer Advancement Act, the use of such standards
becomes an attractive method of improving calibration quality while reducing the reliance on
government-unique standards for the prescription of calibration requirements and/or guidance.
Utilizing such consensus standards as guidance documents can mitigate the risk of incorrect decisions
during calibration and subsequent measurement processes. Standards such as Z540.3 provide a
consistent approach for ensuring that measuring processes/instruments are “suitable for their intended
use” by managing false-accept risks and incorrect decisions to acceptably low levels. The U.S.
Pharmacopeia has also embraced a similar philosophy to limiting incorrect measurement decisions via
target uncertainties & TUR. Standards such as ISO 17025 could provide utility in the implementation
of the Medical Device Single Audit Program (MDSAP) by streamlining acceptance of calibrations in
the medical industry on an international/global scale via recognition of accredited laboratories.
Complying with government-unique calibration standards has been a challenge for industry dating
back to the 1960’s. Reliance and recognition of voluntary consensus standards for calibration has
many benefits for both government and industry alike and can provide utility in determining
“suitability for intended use” by mitigating measurement decision risk. Leveraging these standards as
guidance documents, rather than requirements documents, along with formal recognition of accredited
calibration laboratories could foster a degree of harmonization without undue industry burden. In the
21st century “risk-based” environment, the approach to calibration incorporated into Z540.3 is
complementary to this risk-based philosophy. Consensus standards for calibration provide a
framework for making sound decisions throughout the lifecycle of products and instruments, from
design through end-use application.
9. Acknowledgement and Disclosures

The author thanks Nancy Mescher and Tom Waltrich of Baxter Healthcare Corporation for their
support and guidance in a multitude of healthcare metrology issues, many of which have been
discussed in this paper. Much gratitude is extended to Shawn Schmitt at Informa, Marcus McNeely at
Blue Mountain Quality Resources, and Roger Burton of Sandia National Laboratories for permission
to reproduce previously published material in its entirety. Rules, regulations, and guidance discussed
in this paper are provided for reference only, are not authoritative, and may be outdated or withdrawn
at the time of publication. Original source documents should be consulted for use in any official
capacity. While every effort has been made to ensure accuracy of information, it is presented “as-is”
without any implied warranty or suitability. The opinions and inferences provided in this paper are
exclusively those of the author and do not necessarily reflect those of any agency or entity.
10. Bibliography
[1] Code of Federal Regulations, Title 21 – Food and Drugs:
[1A] Part 820 – Quality System Regulation, Subpart G – Production and Process Controls,
Section §820.72 – Inspection, Measuring, and Test Equipment, paragraphs (a/b/1/2)
[1B] Part 211 – Current Good Manufacturing Practice for Finished Pharmaceuticals,
Subpart C- Buildings and Facilities, Section §211.68 – Automatic, Mechanical, and
Electronic Equipment, Paragraph (a)
[1C] Part 211 – Current Good Manufacturing Practice for Finished Pharmaceuticals,
Subpart I- Laboratory Controls, Section §211.160 – General Requirements, Paragraph (b/4)
[1D] Part 606 – Current Good Manufacturing Practice for Blood and Blood Components,
Subpart D – Equipment, Section §606.60 – Equipment, Paragraphs (a/b/c)
[1E] Part 58 – Good Laboratory Practice for Nonclinical Laboratory Studies, Subpart D –
Equipment, Section §58.63 – Maintenance and Calibration of Equipment, Paragraphs (a/b/c)
[1F] Part 106 – Infant Formula Requirements Pertaining to Current Good Manufacturing
Practice, Quality Control Procedures, Quality Factors, Records and Reports, and
Notifications, Subpart B – Current Good Manufacturing Practice, Section §106.30 –
Controls to Prevent Adulteration Caused by Equipment or Utensils, Paragraphs (d 1/2/3)
[2] MIL-STD-120, Military Standard: Gage Inspection. U.S Department of Defense.
Washington D.C. December 1950.
[3] H. Castrup, A Note on the Accuracy Ratio Requirements. Integrated Science Group.
Bakersfield CA. June 2006.
[4] J. Condon, NASA Calibration Practices. Proceedings of the 1966 Standards Laboratory
Conference, National Conference of Standards Laboratories – United States Department of
Commerce – National Bureau of Standards, NBS Miscellaneous Publication 291, pp 119 – 120,
Washington DC, May 1966.
[5] F. Russell, Industry’s View of the 10:1 Ratio-of-Accuracy Requirement. Proceedings of the
1966 Standards Laboratory Conference, National Conference of Standards Laboratories – United
States Department of Commerce – National Bureau of Standards. NBS Special Publication 313,
pp 121 – 123, Washington DC. May 1966.
[6] M. Fruechtenicht, Metrology and Calibration in DOD Quality and Reliability Operations.
Making Valuable Measurements - Proceedings of the 1968 Standards Laboratory Conference,
National Conference of Standards Laboratories. United States Department of Commerce –
National Bureau of Standards, pp 11 – 13, Washington DC. August 1968.
[7] MIL-C-45662A, Military Specification: Calibration System Requirements. U.S Department
of Defense. February 1962.
[8] A. Lowery, Device Good Manufacturing Practices – A Workshop Manual. U.S. Department
of Health and Human Services – Public Health Service – Food and Drug Administration –
Bureau of Medical Devices – Office of Small Manufacturers Assistance. Chapter 4: Buildings,
Equipment and Calibration. October 1982.
[9] W. F. Hooten, Measurement Assurance and Medical Devices. National Conference of
Standards Laboratories. NCSL Newsletter. Vol. 20, No. 4, pp 36 – 37. Boulder CO. December
1980.

[10] 10 CFR 50, Code of Federal Regulations. Title 10 – Energy, Part 50 – Domestic Licensing
of Production and Utilization Facilities, Appendix B – Quality Assurance Criteria for Nuclear
Power Plants and Fuel Reprocessing Plants, Basic Requirement XII – Control of Measuring and
Test Equipment.
[11] 14 CFR §145.109, Code of Federal Regulations. Title 14 – Aeronautics and Space –
Chapter I – Federal Aviation Administration – Department of Transportation, Subchapter H –
Schools and Other Certificated Agencies, Part 145 – Repair Stations, Subpart C – Housing,
Facilities, Equipment, Materials, and Data, §145.109 – Equipment, Materials, and Data
Requirements.
[12] MIL-C-45662 (Ord), Military Specification: Calibration of Standards. U.S Department of
Defense. Washington D.C. March 1960.
[13] MIL-HDBK-52, Military Handbook: Evaluation of Contractor’s Calibration System. U.S
Department of Defense. Washington D.C. July 1964.
[14] NATO AQAP-6, Measurement and Calibration System Requirements for Industry. North
Atlantic Treaty Organization. Brussels Belgium. December 1970.
[15] NATO AQAP-7, Guide for the Evaluation of a Contractor’s Measurement and Calibration
System for Compliance with AQAP-6. North Atlantic Treaty Organization. Brussels Belgium.
October 1978.
[16] DEFSTAN 05-26/1:1973, Measurement and Calibration System Requirements for Industry.
British Ministry of Defence. London England. 1973.
[17] DEFSTAN 05-27/1:1973, Guide for the Evaluation of a Contractor’s Measurement and
Calibration System for Compliance with Def Stan 05-26/2. British Ministry of Defence. London
England. 1973.
[18] BS 5781-1:1979, Measurement and Calibration Systems. Specification for System
Requirements. British Standards Institute. London England. October 1979.
[19] BS 5781-2:1981, Quality Assurance Requirements for Measuring Equipment. Guide to the
Use of BS 5781-1 ‘Specification of System Requirements’. British Standards Institute. London
England. February 1981.
[20] ISO 10012-1:1992, Quality Assurance Requirements for Measuring Equipment – Part 1:
Metrological Confirmation System for Measuring Equipment. International Organization for
Standardization. Geneva Switzerland. January 1992.
[21] ISO 10012-2:1997, Quality Assurance for Measuring Equipment – Part 2: Guidelines for
Control of Measurement Process. International Organization for Standardization. Geneva
Switzerland. January 1997.
[22] ISO 10012:2003, Measuring Management Systems – Requirements for Measurement
Processes and Measuring Equipment. International Organization for Standardization. Geneva
Switzerland. April 2003.
[23] ISO Guide 25-1978(E), Guidelines for Assessing the Technical Competence of Testing
Laboratories, first edition. International Organization for Standardization. Geneva Switzerland.
October 1978.
[24] ILAC/78, Summary of Proceedings – Second International Conference on Recognition of
National Programs for Accreditation of Testing Laboratories. United States Department of
Commerce – Office of Product Standards. Washington D.C. October 1978.
[25] ISO/IEC Guide 25:1990(E), General Requirements for the Competence of Calibration and
Testing Laboratories, third edition. International Organization for Standardization. Geneva
Switzerland. December 1990.
[26] ISO/IEC 17025:1999(E), General Requirements for the Competence of Testing and
Calibration Laboratories, first edition. International Organization for Standardization. Geneva
Switzerland. December 1999.
[27] ISO/IEC 17025:2005(E), General Requirements for the Competence of Testing and
Calibration Laboratories, second edition. International Organization for Standardization.
Geneva Switzerland. May 2005.
[28] ISO/IEC 17025 CD2, General Requirements for the Competence of Testing and Calibration
Laboratories. CASCO WG44 N75, Committee Draft 2. International Organization for
Standardization. Geneva Switzerland. March 2016.
[29] MIL-STD-45662A, Military Standard: Calibration Systems Requirements. U.S Department
of Defense. Washington D.C. August 1988.
[30] MIL-STD-45662, Military Standard: Calibration Systems Requirements. U.S Department of
Defense. Washington D.C. June 1980.
[31] MIL-HDBK-52A, Military Handbook: Evaluation of Contractor’s Calibration System. U.S
Department of Defense. Washington D.C. August 1984.
[32] MIL-STD-45662 Notice 2, Notice of Cancellation. U.S Department of Defense. Washington
D.C. February 1995.
[32A] R. Schumacher, New American National and International Standards on Calibration
Control. Proceedings of the National Conference of Standards Laboratories Workshop and
Symposium. Albuquerque NM. August 1991.
[32B] R. Schumacher, Metrology Standards – In ANSI and ISO. Proceedings of the National
Conference of Standards Laboratories Workshop and Symposium. Dallas TX. July 1995.
[33] T. Held, Metrology and Calibration Journal. Issue No. 13. Abbot Laboratories. Abbott Park
IL. September 1992.
[34] R. Randall, ISO 9000 – Questions & Answers. GE Electronic Services. Buford GA. June
1991, Revised April 1992.
[35] ANSI/NCSL Z540.1-1994, American National Standard for Calibration – Calibration
Laboratories and Measuring and Test Equipment – General Requirements. American National
Standards Institute / National Conference of Standards Laboratories. Washington D.C. / Boulder
CO. July 1994
[36] J. Cigler, NVLAP Calibration Laboratory Program. Proceedings of the Workshop on
Measurement Quality Assurance for Ionizing Radiation. International Atomic Energy Agency
(IAEA) – International Nuclear Information System (INIS) Vol. 27, No. 19. Report No. PNL-
10076. March 1993. (Note: Similar version presented at 1993 NCSL Workshop & Symposium).
[36A] L. Neumann, A History of Laboratory Accreditation in the United States. Cal Lab
Magazine. Vol. 8, No. 3, pp 22 – 25. July/August/September 2001.
[36B] NIST SP-986, Proceedings of the NIST Conformity Assessment for a Changing
Government Workshop. NIST Special Publication 986. U.S. Department of Commerce –

National Institute of Standards and Technology – Standards Services Division. Gaithersburg
MD. May 2003.
[37] P. Reese, J. Harben. Implementing Strategies for Risk Mitigation in the Modern Calibration
Laboratory, Proceedings of the NCSL International Workshop and Symposium, National Harbor
MD. August 2011.
[38] P. Reese, J. Harben. Risk Mitigation Strategies for Compliance Testing, Measure – The
Journal of Measurement Science, NCSL International Vol.7, No.1 pp 38-49. March 2012
[39] S. Mimbs, Using Reliability to Meet Z540.3’s 2 % Rule. Proceedings of the NCSL
International Workshop and Symposium. National Harbor, MD. August 2011.
[39A] NCSLI, Request for Interpretation of ANSI/NCSL Z540.3-2006. NCSLI 174 Accredited
Standards Committee. NCSL International. Boulder CO. 2011.
[40] ISO/IEC Guide 25 Draft V, General Requirements for the Competence of Calibration and
Testing Laboratories, Draft 5 (unpublished). August 1996.
[41] ISO/IEC 17025 CD1, General Requirements for the Competence of Testing and Calibration
Laboratories. CASCO WG44, Committee Draft 1 (unpublished). International Organization for
Standardization. Geneva Switzerland. September 2015.
[42] ISO/IEC Guide 98-4, Uncertainty of Measurement – Part 4: Role of Measurement
Uncertainty in Conformity Assessment. First edition. International Organization for
Standardization. Geneva Switzerland. 2012.
[43] JCGM 106:2012, Evaluation of Measurement Data – The Role of Measurement Uncertainty
in Conformity Assessment. Joint Committee for Guides in Metrology – BIPM, IEC, IFCC, ILAC,
ISO, IUPAC, IUPAP, OIML. October 2012.
[44] ANSI/NCSL Z540.3-2006. Requirements for the Calibration of Measuring and Test
Equipment. American National Standards Institute / NCSL International, 2006.
[45] NCSL Z540.3-2006 Handbook, Handbook for the Application of ANSI/NCSL Z540.3-2006 –
Requirements for the Calibration of Measuring and Test Equipment. NCSL International.
February 2009.
[46] NCSLI RP-18, Recommended Practice – Estimation and Evaluation of Measurement
Decision Risk. NCSL International. Boulder CO. July 2014.
[47] NCSLI RP-21, Recommended Practice for the Assessment of ANSI/NCSL Z540.3-2006 Sub-
clause 5.3. NCSL International. Boulder CO. July 2014.
[48] D. Caldwell, ANSI/NCSL Z540.3:2006: Requirements for the Calibration of Measuring and
Test Equipment. NCSLI Measure – The Journal of Measurement Science. Vol. 1, No. 4, pp 26 –
30. December 2006.
[49] S. Mimbs, Measurement Decision Risk - The Importance of Definitions. Proceedings of the
NCSL International Workshop and Symposium. St. Paul MN. July 2007.
[49A] S. Mimbs, Measurement Decision Risk - The Importance of Definitions. Proceedings of
the Measurement Science Conference. Anaheim CA. March 2008.
[50] ISO, Risk-Based Thinking in ISO 9001:2015. Technical Committee 176, Sub-Committee 2.
Document N1269. International Organization for Standardization. Geneva Switzerland. 2015

[51] S. Weber, A. Hillstrom, Economic Model of Calibration Improvements for Automatic Test
Equipment. NBS Special Publication 673. U.S. Department of Commerce – National Bureau of
Standards. April 1984.
[52] NASA RP-1342. H. Castrup, W. Eicke, J. Hayes, A. Mark, R. Martin, J. Taylor. Metrology
– Calibration and Measurement Processes Guidelines. National Aeronautics and Space
Administration. Jet Propulsion Laboratory. Reference Publication 1342. Pasadena CA. June
1994.
[53] ASME B89.7.3.1-2001, Guidelines for Decision Rules: Considering Measurement
Uncertainty in Determining Conformance to Specifications. American Society of Mechanical
Engineers. New York NY. March 2002.
[54] UKAS LAB34, The Expression of Uncertainty in EMC Testing. United Kingdom
Accreditation Service. Middlesex UK. August 2002.
[55] ISO 10576-1:2003, Statistical Methods – Guidelines for the Evaluation of Conformity With
Specified Requirements – Part 1: General Principles. First edition. International Organization
for Standardization. Geneva Switzerland. March 2003.
[56] ASME B89.7.4.1-2005 (Technical Report), Measurement Uncertainty and Conformance
Testing: Risk Analysis. American Society of Mechanical Engineers. New York NY. February
2006.
[57] Eurachem/CITAC Guide, Use of Uncertainty Information in Compliance Assessment. First
edition. Eurachem / Co-Operation on International Traceability in Analytical Chemistry
(CITAC). Leoben Austria. 2007
[58] Eurolab Cook Book Doc No. 8.0, Determination of Conformance with Specifications or
Limit Values with Particular Reference to Measurement Uncertainties – Possible Strategies.
Brussels Belgium. May 2008.
[59] ILAC-G8:03/2009, Guidelines on the Reporting and Assessment of Compliance with
Specification. International Laboratory Accreditation Cooperation. 2009
[60] NASA HDBK 8739.19-4, Estimation and Evaluation of Measurement Decision Risk: NASA
Measurement Quality Assurance Handbook – ANNEX 4. National Aeronautics and Space
Administration. Washington D.C. July 2010.
[61] VDA-5, Capability of Measurement Processes, Quality Management in the Automobile
Industry – Volume 5. Verband der Automobilindustrie (German Association of the Automobile
Industry). July 2011
[62] AIAG MSA-4, Measurement System Analysis – Reference Manual. Fourth edition. ISBN
978-1-60-534211-5. Chrysler Group LLC, Ford Motor Company, General Motors Corporation.
Automotive Industry Action Group. Southfield MI. June 2010.
[63] UKAS:M3003, The Expression of Uncertainty and Confidence in Measurement, Appendix
M – Assessment of Compliance With Specification. Third edition. United Kingdom Accreditation
Service. Middlesex UK. November 2012.
[64] ISO 22514-7:2012, Statistical Methods in Process Management – Capability and
Performance – Part 7: Capability of Measurement Processes. First edition. International
Organization for Standardization. Geneva Switzerland. March 2003.

[64A] Minitab 17 Support, A Type 1 Gage Study Assesses the Capability of a Measurement
Process. Minitab 17 Support. Topic Library – Quality Tools – Measurement System Analysis –
Other Gage Studies and Measures – Type 1 Gage Study. Minitab Inc. Retrieved May 2016.
[65] ISO-14253-1:2013(E), Geometrical Product Specifications (GPS) - Inspection by
Measurement of Workpieces and Measuring Equipment - Part 1: Decision Rules for Proving
Conformance or Non-conformance with Specifications. International Organization for
Standardization. Geneva Switzerland. 2013.
[66] WADA TD2014DL, Decision Limits for the Conformity Quantification of Threshold
Substances. Technical Document TD2014DL, Version 1.0. World Anti-Doping Agency. Quebec
Canada. September 2014.
[67] FDA, Guide to Inspections of Medical Device Manufacturers. Section 8, pg 19, Inspection
Measuring and Test Equipment 21 CFR 820.72. U.S. Department of Health and Human Services.
Food and Drug Administration. Silver Spring MD. December 1997.
[68] USP <1058>, Analytical Instrument Qualification. U.S. Pharmacopeia National Formulary.
USP 39, NF 34. General Chapter <1058>. United States Pharmacopeial Convention. Rockville
MD. May 2016.
(hyperlink is for previous edition).
[69] ISPE GAMP-5, Good Practice Guide – A Risk-Based Approach to GxP Compliant
Laboratory Computerized Systems. Second edition. ISBN 978-1-936379-49-1. International
Society for Pharmaceutical Engineering (ISPE). Community of Practice (COP) Technical
Subcommittee on Good Automated Manufacturing Practice (GAMP). Tampa FL. 2012.
[70] L.Vuolo-Schuessler, M. Newton, P. Smith, C. Burgess, R. McDowall, Harmonizing USP
<1058> and GAMP for Analytical Instrument Qualification. Pharmaceutical Engineering.
International Society for Pharmaceutical Engineering (ISPE). Vol. 34, No. 1, pp 1 – 9.
January/February 2014.
[71] FDA, Code of Federal Regulations, Title 21 – Food and Drugs, Part 133 Drugs; Current
Good Manufacturing Practice in Manufacture, Processing, Packing, or Holding. Section §133.4
– Equipment. Office of the Federal Register, National Archives and Records Service, General
Services Administration. Washington DC. January 1965.
[72] FDA, Code of Federal Regulations, Title 21 – Food and Drugs: Subchapter H – Medical
Devices. Part 820 – Good Manufacturing Practice for Medical Devices. Subpart D – Equipment,
Section §820.61 – Measurement Equipment. Federal Register, 43 FR 31 508. July 1978.
(hyperlink is for a subsequent edition).
[73] AAMI, Quality System Compendium, GMP Requirements & Industry Practice. Chapter 15
– Inspection, Measuring, and Test Equipment (Subpart G). First edition. Association for the
Advancement of Medical Instrumentation (AAMI). ISBN 1-57020-076-9. Arlington VA. 1998
[74] ANSI/ASQC M1-1987, American National Standard for Calibration Systems. American
Society for Quality Control. Milwaukee WI. June 1987. (Reaffirmed in February 1996 as
ANSI/ASQC M1-1997).
[75] FDA, Guidance for Industry PAT – A Framework for Innovative Pharmaceutical
Development, Manufacturing, and Quality Assurance. U.S. Department of Health and Human
Services. Food and Drug Administration. Center for Drug Evaluation and Research. Center for
Veterinary Medicine. Office of Regulatory Affairs. Rockville MD. September 2004.
[76] ISPE/GAMP, Good Practice Guide: A Risk-Based Approach to Calibration Management.
Second edition. ISBN 978-1-936379-00-2. International Society for Pharmaceutical Engineering
(ISPE) – Community of Practice (COP) Technical Subcommittee on Good Automated
Manufacturing Practice (GAMP). Tampa FL. 2010.
[77] NCSL RP-6, Recommended Practice for Calibration Quality Systems for the Healthcare
Industries. NCSL International. ISBN 978-1-58464-069-1. Boulder CO. 2015.
[78] M. McNeely, An Overview of the NCSLI 151 Healthcare Metrology Committee.
Proceedings of the NCSL International Workshop and Symposium. Nashville TN. July 2013.
[79] FDA, Code of Federal Regulations, Title 21 – Food and Drugs: Chapter I – Food and Drug
Administration, Department of Health and Human Services, Subchapter A – General, Part 10,
Administrative Practices and Procedures, Subpart B – General Administrative Procedures,
Section §10.115 – Good Guidance Practices. Federal Register 65 FR 56477. September 2000.
[80] P. James, Comments to the Proposed Rule: Medical Devices; Refurbishers, Servicers, and
‘As Is’ Remarketers of Medical Devices; Review and Revision of Compliance Policy Guides and
Regulatory Requirements; Request for Comments and Information. FDA Docket No. 97N-0477.
Food and Drug Administration. February 1998.
[81] GHTF.SG3.N99-8, Guidance on Quality Systems for the Design and Manufacture of
Medical Devices. Section 4.11 – Control of Inspection, Measuring, and Test Equipment. pg 31.
Global Harmonization Task Force (GHTF), Study Group 3. June 1999
[82] ISO 13485:2016, Medical Devices – Quality Management Systems – Requirements for
Regulatory Purposes. International Organization for Standardization. Geneva Switzerland.
March 2016.
[82A] IAF B5, The IAF Initiative for Accredited Certification to ISO 13485 – Medical Devices.
International Accreditation Forum. Quebec Canada. January 2012.
[83] MDSAP G0002.1.003, Medical Device Single Audit Program Companion Document.
Chapter 6 – Process: Production and Service Controls. Audit Task 13 – Maintenance and
Calibration, pp 69 - 70. International Medical Device Regulators Forum (undated).
[84] FDA, Recognized Consensus Standards - Databases. U.S. Department of Health and
Human Services, Food and Drug Administration. CDRH Standards Program. Rockville MD.
May 2016.
[85] FDA, Federal Register Documents, Modifications to the Initial List of Recognized
Standards.. U.S. Department of Health and Human Services, Food and Drug Administration.
Rockville MD (continuously updated).
[86] FDA, Guidance for Industry and Staff – Recognition and Use of Consensus Standards. U.S.
Department of Health and Human Services, Food and Drug Administration, Center for Devices
and Radiological Health, Standards Management Staff, Office of Science and Engineering
Laboratories. September 2007.
[87] IMDRF N15, Final Report: List of International Standards Recognized by IMDRF
Members as of March 2014. IMDRF Standards Working Group Document N15. International
Medical Device Regulators Forum. September 2014.
[88] IMDRF N15 (USA list), Final Report: List of International Standards Recognized by
IMDRF Members as of March 2014. IMDRF Standards Working Group Document N15.
International Medical Device Regulators Forum. September 2014.

[89] ISO/TR 14969:2004, Medical Devices – Quality Management Systems – Guidance on the
Application of ISO 13485:2003. Section 7.6 – Control of Monitoring and Measuring Devices.
International Organization for Standardization. Geneva Switzerland. 2004.
[90] C. Grim, The Cost of Inaccurate Blood Pressure Measurement. Summary Report –
National High Blood Pressure Education Program (NHBPEP)/National Heart, Lung, and Blood
Pressure Institute (NHLBI) and American Heart Association (AHA) Working Meeting on Blood
Pressure Measurement. pp 30 – 31. National Institutes of Health. Bethesda MD. April 2002.
[91] M. Lozy, Simple Computation of a Bivariate Normal Integral Arising From a Problem of
Misclassification with Applications to the Diagnosis of Hypertension. Communications in
Statistics – Theory and Methods, Vol. 11, No. 19, pp 2195-2205. 1982.
[91A] C. Friedman-Gerlicz, I. Lilly, Misclassification Rates in Hypertension Diagnosis Due to
Measurement Errors. Society for Industrial and Applied Mathematics (SIAM) , Undergraduate
Research Online (SIURO). Vol. 2, No. 2. August 2010.
[92] R. Greaves, Common Reference Intervals. 4th Vietnam Chemical Pathology Course. Roche
Diagnostics. Ho Chi Minh City, Vietnam. July 2012.
[93] R. Forsman, Why is the Laboratory an Afterthought for Managed Care Organizations?
Clinical Chemistry, Vol. 42, No. 5, pp 813-816. May 1996.
[94] I. Kuselman, A. Weisman, W. Wegscheider, Traceability Without Uncertainty: Current
Situation in the Pharmaceutical Industry. Accreditation and Quality Assurance. Vol. 8, No. 11,
pp 530-531. November 2003.
[95] E. Simpson, FDA Standards on Calibrating Lab Equipment Are Not So Tough”.
Engineering.com – Designer Edge. January 2015.
[96] U.S. Department of Health and Human Services, Food and Drug Administration, Form 483
Observation #11 – Control of Inspection, Measuring, and Test Equipment, Commander S.
Creighton, Consumer Safety Officer. Issued to St. Jude Medical IESD, Sylmar CA. October 17,
2012.
 Response (November 7, 2012) – Observation #11
 Response (March 13, 2013) – Observation #11
[97] JCGM 200:2012 (ISO/IEC Guide 99-12:2007). International Vocabulary of Metrology —
Basic and General Concepts and Associated Terms (VIM). Joint Committee for Guides in
Metrology - Working Group 2, 3rd Edition. 2008.
[98] P. Reese, Instrument Adjustment Policies. Proceedings of the 2015 NCSL International
Workshop and Symposium. Grapevine TX. July 2015.
[99] J. Bucher, Measure for Measure - Out of Sync. The Standard. American Society for Quality
(ASQ) Measurement Quality Division. Vol. 27, No. 2, pp 21-22. June 2013.
[99A] The preceding paper was also published in ASQ Quality Progress, pp 52-53. March 2010.
[100] J. Bucher, Debunking The Two Great Myths About Calibration: Traceability to NIST: If
You Cannot Adjust, You Cannot Calibrate. Proceedings of the NCSL International Workshop
and Symposium, National Harbor MD. Aug 2011.
[101] J. Bucher., Where Does It Say That? Clearing Up the FDA’s Calibration Requirements.
American Society for Quality, Measurement Quality Division, The Standard, Vol. 27 No. 2, PDF
pp 31-32. June 2013.

[102] S. Schmitt, Pay Close Attention to Equipment Calibration Certificates, FDA Says. The
Silver Sheet. Article # 09140061001. Informa Business Intelligence, Inc. June 2010.
[103] NASA NPD 8730.1C, Nasa Policy Directive, Metrology and Calibration. National
Aeronautics and Space Administration. Washington D.C. June 2011.
[104] NASA Solicitation No. NNK07189729R, Institutional Services Contract (ISC). National
Aeronautics and Space Administration. Kennedy Space Center FL. October 2008.
[105] NASA Solicitation No. NNJ15532524R, Calibration and Metrology Services (CAMS) III
Contract. National Aeronautics and Space Administration – Johnson Space Center. Houston TX.
July 2015.
[106] NASA Solicitation No. NNC14ZC013R, Technical Information Administration and
Logistics Services (TIALS) Contract. National Aeronautics and Space Administration – Glenn
Research Center. Cleveland OH. September 2014.
[107] ANAB AC-1494, Certificate of Accreditation. Measurement Standards and Calibration
Laboratory (MSCL), NASA – Johnson Space Center, Houston TX. ANSI-ASQ National
Accreditation Board. December 2015.
[108] Sandia, Sandia Primary Standards Laboratory, DOE/AL Approve Use of Accredited
Commercial Calibration Laboratories. Sandia National Laboratories. Sandia Lab News. Vol. 53,
No. 21, pg 9. Albuquerque NM. October 2001.
[109] R. Pettit, Laboratory Accreditation. Report Sand-98-1739C, Conf-980822. Sandia National
Laboratories – Primary Standards Laboratory. Albuquerque NM. August 1998.
[110] ANSI/ASME NQA-1-1986, Nuclear Quality Assurance Program – Supplement 12 S-1,
Supplementary Requirements for Control of Measuring and Test Equipment. American National
Standards Institute / American Society of Mechanical Engineers. New York NY. 1986.
[111] DOE, NQA-1: An Overview for Federal Project Directors. U.S. Department of Energy.
Undated.
[112] IEEE STD 498-1975, IEEE Standard Supplementary Requirements for the Calibration and
Control of Measuring Equipment Used in the Construction and Maintenance of Nuclear Power
Generating Stations. Institute of Electrical and Electronics Engineers. New York NY. July 1975.
[113] IEEE STD 498-1980, IEEE Standard Requirements for the Calibration and Control of
Measuring Equipment Used in the Construction and Maintenance of Nuclear Power Generating
Stations. Institute of Electrical and Electronics Engineers. New York NY. October 1980.
Measuring Equipment Used in Nuclear Facilities. Institute of Electrical and Electronics
Engineers. New York NY. June 1985.
Measuring Equipment Used in Nuclear Facilities. Institute of Electrical and Electronics
Engineers. New York NY. January 1991.
[116] NRC 35705, NRC Inspection Manual: Inspection Procedure 35750, QA Program
Measuring and Test Equipment. LPEB Nuclear Regulatory Commission. Washington D.C. April
1992.
[117] NUPIC Doc No. 28, Commercial Grade Calibration Service Checklist. Nuclear
Procurement Issues Committee. Revision 2. Undated.

[118] NUPIC Doc No. 29, Commercial Grade Calibration Service Checklist Implementation
Guidelines. Nuclear Procurement Issues Committee. Revision 2. August 2006.
[119] NIST IR-6989, Comparison of ISO/IEC 17025 with the NUPIC Audit Checklist.
Department of Commerce – National Institute of Standards and Technology Interagency/Internal
Report. Gaithersburg MD. May 2003.
[120] BLN – COL – FSAR, Final Safety Analysis Report – Chapter 17, Quality Assurance,
Section 17.5.4.7. Bellefonte Nuclear Generating Station (BLN) – Combined License (COL).
Nuclear Regulatory Commission. Hollywood AL. 2009.
[121] P. Hiland, Letter from NRC to P. McCullen of International Accreditation Service Inc.
confirming acceptance of IAS accredited calibrations for commercial-grade calibration
suppliers. Nuclear Regulatory Commission – Office of Nuclear Reactor Regulation – Division of
Engineering. Washington D.C. May 2008.
[122] P. Hiland, Letter from NRC to R. Leonard of Laboratory Accreditation Bureau confirming
acceptance of L-A-B accredited calibrations for commercial-grade calibration suppliers.
Nuclear Regulatory Commission – Office of Nuclear Reactor Regulation – Division of
Engineering. Washington D.C. April 2008.
[123] P. Hiland, Letter from NRC to K. Greenaway of Assured Calibration and Laboratory
Accreditation Select Services confirming acceptance of ACLASS [now ANAB] accredited
calibrations for commercial-grade calibration suppliers. Nuclear Regulatory Commission –
Office of Nuclear Reactor Regulation – Division of Engineering. Washington D.C. December
2007.
[124] P. Hiland, Letter from NRC to T. Szerszen of Perry Johnson Laboratory Accreditation Inc.
confirming acceptance of PJLA accredited calibrations for commercial-grade calibration
suppliers. Nuclear Regulatory Commission – Office of Nuclear Reactor Regulation – Division of
Engineering. Washington D.C. January 2010.
[125] ILAC MRA, The ILAC Mutual Recognition Arrangement. International Laboratory
Accreditation Cooperation. New South Wales Australia. 2016.
[126] ILAC P4:02/2016, ILAC Mutual Recognition Arrangement: Policy and Management.
International Laboratory Accreditation Cooperation. New South Wales Australia. 2016.
[127] ILAC P5:02/2016, ILAC Mutual Recognition Arrangement: Scope and Obligations.
International Laboratory Accreditation Cooperation. New South Wales Australia. 2016.
[128] ILAC P8:02/2016, ILAC Mutual Recognition Arrangement: Supplementary Requirements
and Guidelines for the Use of Accreditation Symbols and for Claims of Accreditation Status by
Accredited Laboratories and Inspection Bodies. International Laboratory Accreditation
Cooperation. New South Wales Australia. 2016.
[129] NCSLI, Nuclear Power Industry Calibration Suppliers Confused About Requirements.
NCSLI Measure – The Journal of Measurement Science. Vol. 5, No. 4, pp 14 – 17. NCSL
International. Boulder CO. December 2010.
[130] NEI 14-05A, Guidelines for the Use of Accreditation in Lieu of Commercial Grade
Surveys for Procurement of Laboratory Calibration and Test Services.” Nuclear Energy
Institute. Revision 0. Washington DC. March 2015.
[131] AFD-130625-019, AFMETCAL (AFLCMC/WNM) Air Force Metrology and Calibration
Traceability. U.S. Air Force, Wright Patterson Air Force Base OH. June 2013

[132] AR 750-43, Maintenance of Supplies and Equipment: Army Test, Measurement, and
Diagnostic Equipment. Army Regulation 750-43, Headquarters – Department of the Army.
Washington DC. January 2014.
[133] R. Fritzsche, NACLA Navy Partnership. Naval Surface Warfare Center – Corona Division.
Measurement Science Department. Corona CA. Undated.
[134] R. Fritzsche, NACLA Value – Navy METCAL Perspective. Naval Surface Warfare Center –
Corona Division. Measurement Science Department. Corona CA. Undated.
[135] NACLA, Accreditation Body Evaluation Procedure for ANSI/NCSL Z540.3-2006
subclause 5.3. Rev A. National Cooperation for Laboratory Accreditation. Washington DC.
April 2010.
[136] A2LA R205a, Annex to Specific Requirements: ANSI/NCSLI Z540.3-2006. American
Association Laboratory Accreditation. Frederick MD. June 2010.
[137] L-A-B LABPR-401.1, METCAP Program Requirements Metrology and Calibration
Accreditation Program ANSI/NCSL Z540.3-2006 subclause 5.3. Rev 3. Laboratory Accreditation
Bureau. Fort Wayne IN. May 2011.
[138] ANAB CL 2043, Z540.3 Checklist. ANSI-ASQ National Accreditation Board (formerly
ACLASS). Milwaukee WI. Undated.
[139] IAS CL/014, Policy Guide on Calibration, Traceability, and Measurement Uncertainty for
Calibration Laboratories. International Accreditation Service, Inc. Brea CA. January 2016.
[140] PJLA LF-1, Application / Questionnaire. Rev 1.10. Perry Johnson Laboratory
Accreditation, Inc. Troy MI. November 2015.
[141] ISO/TS 16949:2009(E), Quality Management Systems — Particular Requirements for the
Application of ISO 9001:2008 for Automotive Production and Relevant Service Part
Organizations. Third edition. International Organization for Standardization. June 2009.
[141A] QS 9000, Quality System Requirements. Chrysler Corporation, Ford Motor Company,
General Motors Corporation. Third edition. Auburn Hills / Allen Park / Flint MI. March 1998.
[142] FAA 8900.1, Flight Standards Information System (FSIMS). Volume 6 – Surveillance,
Chapter 9 – Part 145 Inspections, Section 9 – Safety Assurance System: Inspect a Part 145
Repair Station’s Tools and Equipment, Paragraph 6-1800 – Procedures. Chg 329. Federal
Aviation Administration. August 2014.
[143] FAA Order 6200.4G, National Test Equipment Program Management. Section 11 –
Calibration of Test Equipment. U.S. Department of Transportation – Federal Aviation
Administration. Washington D.C. March 2010.
[144] FAA, Performance Work Statement – FAA Test, Measurement & Diagnostic Equipment
Calibration Services. Mike Monroney Aeronautical Center – FAA Logistics Center. Federal
Aviation Administration. Oklahoma City OK. January 2008.
[145] NASA, System Failure Case Studies – The Unknown Known. Vol. 3, No. 1. National
Aeronautics and Space Administration – NASA Safety Center (NSC). Cleveland OH. January
2009.
[146] NTTAA, National Technology Transfer and Advancement Act of 1995. One Hundred
Fourth Congress of the United States of America. H.R. 2196. Public Law 104-113. Washington
D.C. March 1996.

[147] W. Perry, Specifications & Standards - A New Way of Doing Business (The Perry Memo).
U.S. Secretary of Defense, White House National Security Council, Department of Defense.
Washington DC. June 1994.
[148] OMB Circular A-119, Federal Participation in the Development and Use of Voluntary
Consensus Standards and in Conformance Assessment Activities. U.S. White House (Executive
Office of the President) – Office of Management and Budget. Washington DC. February 1998
(obsolete).
[149] OMB Circular A-119, Federal Participation in the Development and Use of Voluntary
Consensus Standards and in Conformance Assessment Activities. U.S. White House – (Executive
Office of the President) Office of Management and Budget. Washington DC. 2016 (revised).
[150] OMB, Proposed Revisions to Circular A-119 – Public Comments. U.S. White House –
(Executive Office of the President) Office of Management and Budget. Washington DC. 2014.
[151] ACUS 78-4, Federal Agency Interaction with Private Stand-Setting Organizations in
Health and Safety Regulation. Recommendation 78-4. Administrative Conference of the United
States (ACUS). Adopted December 14-15, 1978.
[152] ACUS 2011-5, Incorporation by Reference. Recommendation 2011-5. Administrative
Conference of the United States (ACUS). Adopted December 8, 2011.
[153] 1 CFR §51.1, Code of Federal Regulations. Title 1– General Provisions, Chapter II –
Office of the Federal Register, Part 51 – Incorporation by Reference, Section 51.1 – Policy.
[154] 46 CFR §162.060-36, Code of Federal Regulations. Title 46 – Coast Guard, Department
of Homeland Security, Subchapter Q – Equipment, Construction, and Materials: Specifications
and Approval, Part 162 – Engineering Equipment, Subpart 160.060 – Ballast Water Management
Systems, §162.160.36 – Quality Assurance Project Plan (QAPP) Requirements.
[155] 10 CFR §429.110, Code of Federal Regulations. Title 10 – Energy, Chapter II –
Department of Energy, Subchapter D – Energy Conservation, Part 429 – Certification,
Compliance, and Enforcement for Consumer Products and Commercial and Industrial
Equipment, Subpart C – Enforcement, §429.110 – Enforcement Testing, paragraph (3).
[156] 47 CFR §2.948, Code of Federal Regulations. Title 47 – Telecommunication, Chapter I –
Federal Communications Commission, Subchapter A – General, Part 2 – Frequency Allocations
and Radio Treaty Matters; General Rules and Regulations, Subpart J – Equipment Authorization
Procedures, §2.948 – Description of Measurement Facilities, Paragraph (d).
[157] FDA, Medical Devices; Current Good Manufacturing Practice (CGMP) Final Rule;
Quality System Regulation. Federal Register. Vol. 61, No. 195, pp 52601 – 52662. Washington
D.C. October 1996.
[158] NIST 9th NTTAA Report, Ninth Annual Report of Federal Agency Use of Voluntary
Consensus Standards and Conformity Assessment. U.S. Department of Commerce – National
Institute of Standards and Technology. Washington D.C. December 2006.
[159] NIST IR7413, Addendum to the Tenth (2006) Annual Report of Federal Agency Use of
Voluntary Consensus Standards and Conformity Assessment. U.S. Department of Commerce –
National Institute of Standards and Technology. Washington D.C. April 2007.
[160] NIST IR7503, Addendum to the Eleventh (2007) Annual Report of Federal Agency Use of
National Institute of Standards and Technology. Washington D.C. May 2008.

[161] NIST IR7598, Addendum to the Twelfth (2008) Annual Report of Federal Agency Use of
National Institute of Standards and Technology. Washington D.C. May 2009.
[162] FDA, Guidance for Industry – Submission of Laboratory Packages by Accredited
Laboratories. (draft). U.S. Department of Health and Human Services – Food and Drug
Administration – Office of Regulatory Affairs and Office of Policy, Planning and Preparedness.
Silver Spring MD. January 2009.
[163] FDA, Laboratory Manual of Quality Policies – Volume 1: ORA Laboratory Manual of
Quality Policies. Document No. ORA-LAB.QM, Version 2.0, Effective Date 10-01-03, Revised
05-02-14. U.S. Department of Health and Human Services – Food and Drug Administration –
Office of Regulatory Affairs. Silver Spring MD. May 2014.
[164] FDA: Allergenic Products Advisory Committee Meeting Background Document: ISO
17025 Accreditation of the Laboratory of Immunobiochemistry. U.S. Department of Health and
Human Services – Food and Drug Administration. Silver Spring MD. May 2011.
[165] S. Menzies, Laboratory of Immunobiochemistry and ISO 17025 Accreditation.
(Presentation). U.S. Department of Health and Human Services, Food and Drug Administration.
Center for Biologics Evaluation and Research (CBER). Laboratory of Immunobiochemistry.
Allergenic Products Advisory Committee Meeting. Bethesda MD, May 2011.
[166] S. Menzies, Laboratory of Immunobiochemistry and ISO 17025 Accreditation. (Meeting
minutes). U.S. Department of Health and Human Services, Food and Drug Administration.
Center for Biologics Evaluation and Research (CBER). Laboratory of Immunobiochemistry.
Allergenic Products Advisory Committee Meeting. Bethesda MD, May 2011
[167] FDA, Advancing Regulatory Science: CBER Receives Accreditation for Laboratory
Testing of Influenza Vaccines and Evaluation of Blood Donor Screening Kits. U.S. Department
of Health and Human Services, Food and Drug Administration. Center for Biologics Evaluation
and Research (CBER). Silver Spring MD. Undated.
[168] FDA, ISO/IEC 17025:2005 Accreditation for State Food Testing Laboratories (U18)
Cooperative Agreement Program.. U.S. Department of Health and Human Services, Food and
Drug Administration, Division of Federal-State Relations (DFSR). Silver Spring MD. 2012.
[169] FDA, ISO/IEC 17025:2005 Accreditation for State Food Testing Laboratories (U18).
Funding Opportunity Announcement (FOA) RFA-FD-12-008. U.S. Department of Health and
Human Services, Food and Drug Administration, Division of Federal-State Relations (DFSR).
Silver Spring MD. 2012.
[169A] FDA, ISO/IEC 17025:2005 CAP (Cooperative Agreement Program). U.S. Department of
Health and Human Services, Food and Drug Administration, Office of Partnerships. Silver
Spring MD. 2012.
[170] FDA: Accreditation of Third-Party Certification Bodies To Conduct Food Safety Audits
and To Issue Certifications. Federal Register. Vol. 80, No. 228, pp7450 – 74667. Washington
D.C. November 2015.
[171] ACIL, Statement on the Food Safety Modernization Act. American Council of Independent
Laboratories. Washington D.C. 2011
[172] ACUS 2012-7, Agency Use of Third-Party Programs to Assess Regulatory Compliance.
Administrative Conference of the United States. Recommendation 2012-7. Adopted December 6,
2012.
[173] L. McAllister, Third-Party Programs to Assess Regulatory Compliance. University of San
Diego School of Law (Report prepared for ACUS). San Diego CA. October 2012.
Note: Hyperlink is to September 2012 “draft” report.
[173A] FDA, Docket No. FDA-2007-N-0265 (Formerly Docket No. 2007P-0026). Final Rule –
Temperature-Indicating Devices: Thermally Processed Low-Acid Foods Packaged in
Hermetically Sealed Containers. Comment 9 and Comment 11. Federal Register. Vol. 76, No.
42, pp 11896. U.S. Department of Health and Human Services – Food and Drug Administration.
March 2011.
[173B] R. Manning, S. Lane, S. Dressman, W. Hauk, R. Williams, The Application of
Uncertainty to USP’s Compendial Reference Standards Program: Certified Reference Materials.
Pharmacopeial Forum. Vol. 33, No. 6, pp 2 – 14. November – December 2007.
[173C] S. Dressman, W. Koch, R. Manning, S. Lane, W. Hauk, R. Williams, The Application of
Uncertainty to USP’s Compendial Reference Standards Program: Certified Reference Materials.
CITAC News. pp 22 – 24. February 2009.
[173D] M. Weitzel, How to Monitor Reference Substances for Continuous Fitness for Purpose:
Novel Approaches or Solutions to Challenges? USP Expert Committee Member Presentation.
Pharmaceutical Reference Standards 11th International Symposium – Session 2b. European
Directorate for the Quality of Medicines & Healthcare (EDQM), Council of Europe. Strasbourg
France. September 2012.
[173E] M. Weitzel, Using Design of Experiment Studies and QbD for Robustness Testing. IVT
Lab Compliance Week. Philadelphia PA. December 2013.
[174] NASA NPC 200-2, Quality Program Provision for Space System Contractors. National
Aeronautics and Space Administration. Section 9 – Inspection Measuring and Test Equipment.
Washington D.C. April 1962.
[175] NASA NHB 5300.2, Apollo Metrology Requirements Manual. NASA Handbook. Apollo
Documentation No. RA-006-008-1. National Aeronautics and Space Administration.
Washington D.C. December 1965.
[176] NASA NHB 5300.4(1B), Quality Program Provisions for Aeronautical and Space System
Contractors. NASA Handbook – Reliability and Quality Assurance Publication N93-72440.
(Formerly NPC 200-2). National Aeronautics and Space Administration. Washington D.C. April
1969.
[177] MIL-HDBK-52B, Military Handbook: Evaluation of Contractor’s Calibration System. U.S
[178] MIL-PRF-38793A, Military Performance Specification: Manuals, Technical: Calibration
Procedures, Preparation of. U.S. Department of Defense. Washington D.C. March 1993.
[179] DOE-STD-1054-93, Guideline to Good Practices for Control and Calibration of
Measuring and Test Equipment (M&TE) at DOE Nuclear Facilities. U.S. Department of Energy.
Washington D.C. March 1993.
[180] DOE 4330.4B, Maintenance Management Program, Section 12 – Control and Calibration
of Measuring and Test Equipment. U.S. Department of Energy. Washington D.C. February 1994
[181] NASA KHB 5330.9A, Metrology & Calibration Handbook. National Aeronautics and
Space Administration. Kennedy Space Center FL. October 1994

[181A] NCSL Z540.1 Handbook, Handbook for the Interpretation and Application of
ANSI/NCSL Z540-1-1994. First edition. NCSL International. Boulder CO. October 1995.
[182] DOE RW-0333P, Quality Assurance Requirements and Description. Section 12.0 –
Control of Measuring and Test Equipment. Rev.5. U.S. Department of Energy – Office of
Civilian Radioactive Waste Management. Washington D.C. October 1995.
[183] MIL-HDBK-1839, Handbook – Calibration and Measurement Requirements. U.S
[184] MIL-HDBK-1839A, Handbook – Calibration and Measurement Requirements. U.S
Department of Defense. Washington D.C. November 2000.
[185] NASA NSTS 5300.4(1D-2), Space Shuttle: Safety, Reliability, Maintainability, and
Quality Provisions for the Space Shuttle Program. Change No. 9 (June 2006). Section 1D507 –
Metrology. Formerly NHB 5300.4(1D-2). National Aeronautics and Space Administration –
Lyndon B. Johnson Space Center. Houston TX. September 1997 (Baseline).
Note: Hyperlink is to older version (Chg No. 2 - 2000)
[186] DOE RW-0333P, Quality Assurance Requirements and Description. Section 12.0 –
Control of Measuring and Test Equipment. Rev.21. U.S. Department of Energy – Office of
Civilian Radioactive Waste Management. Washington D.C. January 2009.
[187] MIL-STD-1839D. Standard Practice – Calibration and Measurement Requirements. U.S.
Department of Defense. Washington D.C. May 2010.
[188] T.O. 00-20-14, Air Force Metrology and Calibration Program. Technical Manual. U.S.
Department of Defense. Secretary of the Air Force. Washington D.C. September 2011.
[189] L-A-B LABAG-502, Accreditation Guidance. (2011):
[190] A2LA P102, A2LA Policy on Metrological Traceability. American Association for
Laboratory Accreditation. Frederick MD. May 2015.
[191] A2LA R205, Specific Requirements: Calibration Laboratory Accreditation Program.
American Association for Laboratory Accreditation. Frederick MD. June 2015.
[192] IAS AC204 CL/009, Accreditation Criteria for Calibration Laboratories. International
Accreditation Service, Inc. Brea CA. February 2016.
[193] ANAB CL 2043 – Z540.3 Checklist ANSI-ASQ National Accreditation Board (previously
ACLASS). Milwaukee WI. 2016
[194] ANAB MA 2002: ISO/IEC 17025 Accreditation Requirements for Calibration
Laboratories. ANSI-ASQ National Accreditation Board (previously ACLASS). Milwaukee WI.
January 2016.
[195] IAS CL/014, Policy Guide on Calibration, Traceability, and Measurement Uncertainty for
Calibration Laboratories. International Accreditation Service, Inc. Brea CA. February 2016.
[196] IAEA STR-368, International Target Values 2010 for Measurement Uncertainties in
Safeguarding Nuclear Materials. International Atomic Energy Agency. Vienna Austria.
November 2010.
[197] N. Majcen, I. Skubic, P. DeBièvre, Target Measurement Uncertainties for Results of
Chemical Measurements in Slovenia – Part 1: Legislation Related to Quality of Food.
Accreditation and Quality Assurance. Vol. 9, No. 1, pp 106 – 111. January 2004.

[198] P. DeBièvre, Target Measurement Uncertainties I: Are They Coming? Accreditation and
Quality Assurance. Vol. 9, No. 10, pp 589 – 590. September 2004.
[199] P. DeBièvre, Target Measurement Uncertainties II: Are They Coming? Accreditation and
Quality Assurance. Vol. 9, No. 11, pp 651 – 652. October 2004.
[200] R. DaSilva, A. Williams, Setting the Target (Maximum Admissible) Measurement
Uncertainty for Different Analytical Goals. Poster Presentation – Pittsburgh Conference on
Analytical Chemistry and Applied Spectroscopy (Pittcon). New Orleans LA. October 2015.
[201] A2LA P903, Policy on Estimating Measurement Uncertainty for ISO 15189 Testing
Laboratories. March 2014.
[202] Eurachem Guide – V. Barwick, E. Prichard, Terminology in Analytical Measurement – An
Introduction to VIM 3. First Edition. ISBN 978-0-948926-29-7
[203] RDT F3-2T, RDT Standard - Calibration System Requirements. United States Atomic
Energy Commission (AEC) – Division of Reactor Development and Technology (RDT).
Germantown MD. February 1969.

Appendix A
(Additional References on Probability/Risk of Incorrect Decisions Due to Measurement Uncertainty)
See also preceding bibliographical references [2 – 7 , 42 – 47, 49, 51 – 66, and 173B-E]
1950’s
[A1] A. Eagle, A Method for Handling Errors in Testing and Measuring. Industrial Quality
Control, Vol. 10, No. 5, pp 10 – 15. March 1954.
[A2] F. Grubbs, H. Coon, On Setting Test Limits Relative to Specification Limits. Industrial
Quality Control, Vol. 10, No. 5, pp 15 – 20. March 1954.
[A3] J. Wiesen, C. Clark, Determining Allowable Test Set Errors. Sandia Corporation,
Technical Memorandum 201-54-51. Albuquerque NM. September 1954.
[A4] L. Mandel, Grading With a Gauge Subject to Random Output Fluctuations. Journal of the
Royal Statistical Society, Series B (Methodological). Vol. 16, No. 1, pp 118-130. 1954.
[A5] J. Hayes, Factors Affecting Measurement Reliability. Technical Memorandum No. 63-106.
U.S. Naval Ordnance Laboratory – Measurements Reliability Branch – Production Quality
Division – Missile Evaluation Department. Corona CA. October 1955.
[A6] F. Tingey, J. Merrill, Minimum Risk Specification Limits. Atomic Energy Commission
(AEC) Research and Development Report IDO-16396. Phillips Petroleum Company – Atomic
Energy Division. Idaho Falls ID. July 1957.
[A7] H. David, E. Fay, J. Walsh, Acceptance Inspection by Variables When the Measurements
are Subject to Error. Annals of the Institute of Statistical Mathematics. Vol. 10, No. 2, pp 107-
129. June 1959.
[A8] J. Wiesen, D. Owen, A Method of Computing Bivariate Normal Probabilities with an
Application to Handling Errors in Testing and Measuring, Bell System Technical Journal,
Volume 38, No. 2, pp 553-572. March 1959.
[A9] F. Tingey, J. Merrill, Minimum Risk Specification Limits. Journal of the American
Statistical Association, Vol. 54, No. 285, pp 260 – 275. March 1959.
[A10] J. Gant, Let’s Take the Guesswork out of Inspection! American Machinist. Vol 103, pp
117-122. March 9, 1959.
1960’s
[A11] R. Traver, Measuring Equipment Repeatability – The Rubber Ruler. Transactions of the
Annual Convention of the American Society for Quality Control (ASQC). pp 25 – 32.
Milwaukee WI. May 1962. (reprinted in Quality Engineering Vol. 5, No. 1, 1992, pp 181-190).
[A11A] T. Diviney, N. David, A Note on the Relationship Between Measurement Error and
Product Acceptance. The Journal of Industrial Engineering. Vol 14, No. 4, pp 218-219. August
1963.
[A11B] H. Singh, Producer and Consumer Risks for Asymmetrical Test and Specification Limits.
Journal of the American Statistical Association. Vol. 61, No. 314. 1966.
[A12] H. Singh, Producer and Consumer Risks in Non-Normal Populations. Technometrics.
Vol. 8, No. 2, pp 335-343. May 1966.
[A13] A. Marshall, I. Olkin. A General Approach to Some Screening and Classification
Problems. Journal of the Royal Statistical Society – Series B, Methodological. Vol. 30, No. 3, pp
407 – 443. 1968.
Appendix A
1970’s
[A14] T. Mukaihata, Administration of a Standards Laboratory, Section 3 – Special Problems of
Technical and Administrative Nature. Proceedings of the Measurement Science Conference. San
Luis Obispo CA. December 1976.
[A15] ASTM STP-616, B. Marguglio, Quality Systems in the Nuclear Industry, Chapter 14 –
Measurement Control, Section – Multiple (4 times to 10 times rule). Special Technical
Publication No. 616, pg 344. American Society for Testing and Materials. ISBN-13: 978-0-8031-
0197-5. Philadelphia PA. 1977
[A16] H. Castrup, Evaluation of Customer and Manufacturer Risk vs. Acceptance Test
Instrument In-Tolerance Level. TRW Technical Report No. 99900-7871-RU-00. Redondo Beach
CA. April 1978.
[A17] NCSL, Uncertainty, Accuracy, and Accuracy Ratios. NCSL Newsletter, National
Conference of Standards Laboratories. Vol. 18, No. 3, pp 22 – 24. Boulder CO. December 1978.
[A17A] K. Kuskey, Cost-Benefit Model for Analysis of Calibration System Designs in the Case
of Random-Walk Equipment Behavior. SAI Comsystems Technical Report. Prepared for U.S.
Navy Metrology Engineering Center. Contract N000123-76-C-0589. January 1979.
[A18] M. Lay, There are No Magic Accuracy Ratios in Metrology and Calibration. NCSL
Newsletter. Vol. 19, No. 1. pp 26 – 27. National Conference of Standards Laboratories. Boulder
CO. March 1979.
[A19] J. Ferling, A Calibration Interval Analysis Model – Expected Costs Due to Wrong Test
Decisions as a Function of the Calibration Interval. SAI Comsystems Corp. Prepared for U.S.
Navy Metrology Engineering Center (Contract No. N00123-76-C-0589). Corona CA. April 1979
[A20] C. Koop, Accuracy Ratio Requirements. NCSL Newsletter. Vol. 19, No. 1. pp 28 – 29.
National Conference of Standards Laboratories. Boulder CO. March 1979.
[A21] J. Ferling, Accuracy Ratios Related to Consumer’s and Producer’s Risks. NCSL
Newsletter. Vol. 19, No. 1. pp 37 – 39. National Conference of Standards Laboratories. Boulder
CO. March 1979.
[A22] R. Schumacher, Another Viewpoint on Accuracy Ratios. NCSL Newsletter. Vol. 19, No. 2
(mislabeled No. 1). pp 32 – 34. National Conference of Standards Laboratories. Boulder CO.
June 1979.
1980’s
[A23] H. Castrup, Evaluation of Customer and Manufacturer Risk vs. Acceptance Test
Instrument In-Tolerance Level. Proceedings of the National Conference of Standards
Laboratories. Gaithersburg MD. September 1980.
[A24] G. Hahn, Removing Measurement Error in Assessing Conformance to Specifications.
Journal of Quality Technology. Vol. 14, No. 3, pp117 – 121. July 1982.
[A24A] M. Lotti, Measurement Variability and Product Acceptance. Quality Progress. Vol. 15,
No. 9, pp 38 – 41. September 1982.
[A25] R. Mee, D. Owen, A Simple Approximation for Bivariate Normal Probabilities. Technical
Report No. 169. Department of Statistics, Office of Naval Research, Contract No. N00014-76-C-
0613. Dallas TX. December 1982.
Appendix A
[A26] K. Kuskey, New Capabilities for Analyzing METCAL Decisions. Proceedings of the
Measurement Science Conference. Long Beach CA. January 1984.
[A27] J. Ferling, The Role of Accuracy Ratios in Test and Measurement Processes. Proceedings
of the Measurement Science Conference. Long Beach CA. January 1984.
[A28] R. Mee, D. Owen, J. Shyu, Confidence Bounds for Misclassification Probabilities Based
on Data Subject to Measurement Error. Journal of Quality Technology. Vol. 18, No. 1, pp 29 –
40. January 1986.
[A29] D. Jackson, Instrument Intercomparisons and Calibration. Proceedings of the
Measurement Science Conference. Anaheim CA. 1987.
[A30] J. Hayes, Decisions That Influence Quality. Proceedings of the Measurement Science
Conference. Anaheim CA. 1988.
[A31] R. Schumacher, MIL-STD-45662: Conflicting Positions on Out-Of-Tolerance
Measurements. Proceedings of the Measurement Science Conference. Anaheim CA. 1988.
[A32] F. Capell, From 4:1 to SPC, Proceedings of the National Conference of Standards
Laboratories Workshop and Symposium. Washington DC. August 1988.
[A33] F. Capell, More on ‘From 4:1 to SPC’, Proceedings of the National Conference of
Standards Laboratories Workshop and Symposium. pp 8-1 to 8-15. Denver CO. July 1989.
[A34] B. Gunter, The Use and Abuse of Cpk: Part 4. Quality Progress. Vol. 22, No. 7, pp 86 –
87. July 1989.
[A35] H. Castrup, Calibration Requirements Analysis System. Proceedings of the NCSL
Workshop and Symposium. National Conference of Standards Laboratories. Denver CO. July
1989.
[A36] J. Ferling, D. Caldwell, Analytic Modeling for Electronic Test Equipment Adjustment
Policies, Proceedings of the NCSL Workshop and Symposium. National Conference of
Standards Laboratories. Denver CO. July 1989.
[A36A] J. Ferling, On the Determination of Optimum Tolerance Test Limits Under Measurement
Uncertainty Conditions. Proceedings of the 35th International Instrumentation Symposium.
Orlando FL. 1989.
1990’s
[A37] T. Mukaihata, The Accuracy Ratio Dilemma. Presentation at NCSL Region 8 San Diego
Section Meeting. National Conference of Standards Laboratories. Montgomery Field CA. May
1990.
[A38] S. Navard, Evaluating the Effect of Accuracy Ratios on the Percent of Calibrations Which
Are Out of Tolerance. National Aeronautics and Space Administration (NASA) – Johnson Space
Center. Final Report N91-27107. Contract No. NGT 44-055-803. Houston TX. August 1990.
[A38A] P. Mullenix, The Capability of Capability Indices With an Application to Guardbanding
in a Test Environment. IEEE Proceedings of the International Test Conference. pp 907 – 915.
ISBN 0-8186-9064-X. Washington D.C. September 1990.
[A38B] R. Easterling, M. Johnson, T. Bement, C. Nachtsheim, Statistical Tolerancing Based on
Consumer’s Risk Considerations. 34th Annual Fall Technical Conference the American Society
for Quality (Chemical and Process Industries Division and Statistics Division). October 1990.
Appendix A
[A39] F. Capell, From 4:1 to SPC, International Society of Automation. ISA Transactions. Vol.
29, No. 4. November 1990.
[A40] F. Capell, How Good Is Your TUR? Evaluation Engineering. pp 81-84. January 1991.
[A41] H. Castrup, Analytical Metrology SPC Methods for ATE Implementation. Proceedings of
the National Conference of Standards Laboratories Workshop and Symposium. Albuquerque
NM. August 1991.
[A42] B. Hutchinson, Setting Guardband Test Limits to Satisfy MIL-STD-45662A Requirements.
Proceedings of the National Conference of Standards Laboratories Workshop and Symposium.
Albuquerque NM. August 1991.
[A42A] J. Hayes, Calibration and Maintenance of Test and Measuring Equipment. Encyclopedia
of Applied Physics. Vol. 3, No. 11. VCH Publishers, Inc. ISBN 1-56081-062-9. New York NY.
1992.
[A42B] R. Schumacher, Decision Making and Customizing Decision Limits in Measurement
Assurance. Proceedings of the National Conference of Standards Laboratories Workshop and
Symposium. pp 301-310. 1992.
[A42C] D. Wheeler, Problems with Gauge R&R Studies. Transactions of the ASQ Annual
Quality Congress. Vol. 46, No.0, pp 179-185. Nashville TN. May 1992. (reprinted in Quality
Digest, January 2011).
[A43] D. Deaver, How to Maintain Your Confidence in a World of Declining Test Uncertainty
Ratios. Proceedings of the NCSL Workshop and Symposium, National Conference of Standards
Laboratories. Albuquerque NM, July 1993.
[A43A] R. Williams, C. Hawkins, The Economics of Guardband Placement. Proceedings of the
IEEE International Test Conference (ITC). ISBN 0-7803-1430-1. Baltimore MD. October 1993.
[A43B] W. Albers, W. Kallenberg, G. Otten. Accurate Test Limits With Estimated Parameters.
Technometrics. Vol. 36, No. 1, pp 92-101. February 1994.
[A44] D. Deaver, Guardbanding With Confidence. Proceedings of the NCSL Workshop and
Symposium, National Conference of Standards Laboratories. Chicago IL, August 1994.
[A44A] W. Albers, W. Kallenberg, G. Otten, Setting Test Limits Under Prescribed Consumer
Loss. Metrika. Vol. 41, No. 1, pp 163 – 181. December 1994.
[A45] H. Castrup, Uncertainty Analysis for Risk Management. Proceedings of the Measurement
Science Conference. Anaheim CA. January 1995.
[A46] H. Castrup, Analyzing Uncertainty for Risk Management. Proceedings of the ASQC
Annual Quality Congress. Vol. 49, No. 0, Cincinnati OH. May 1995.
[A47] D. Deaver, Using Guardbands to Justify TURs Less Than 4:1. Proceedings of the ASQC
Annual Quality Congress, Vol. 49, No. 0, pp 136-141. Cincinnati, OH. May 1995.
[A48] D. Deaver, Managing Calibration Confidence in the Real World. Proceedings of the
NCSL Workshop and Symposium, National Conference of Standards Laboratories. Dallas TX.
July 1995.
[A48A] H. Castrup, Uncertainty Analysis and Parameter Tolerancing. Proceedings of the
NCSL Workshop and Symposium, National Conference of Standards Laboratories. Dallas TX.
July 1995.
Appendix A
[A48B] D. Fitzsimmons, Calibration Family Risk Approach for 4:1 Equivalence – The Meaning
of Calibration Uncertainty and the Role of Uncertainty in Standards. Proceedings of the NCSL
Workshop and Symposium, National Conference of Standards Laboratories. Dallas TX. July
1995.
[A48C] DIN EN 50222:1996-03 (VDE 0847-222), Standard for the Evaluation of Measurement
Results Taking Measurement Uncertainty Into Account. (withdrawn). Germany. March 1996.
[A49] D. McCarville, D. Montgomery. Optimal Guard Bands for Gauges in Series. Quality
Engineering, Vol 9. No. 2, pp 167-177. 1996-1997.
[A50] W. Albers, G. Arts, W. Kallenberg, Accurate Test Limits Under Prescribed Consumer
Risk. Statistics & Probability Letters. Vol. 34, No. 2, pp 141-149. June 1997.
[A51] M. Fecteau, Test Uncertainty Ratio and Intelligent Instrumentation: From 4:1 to SPC to
IPC in the Army Measurement System. Proceedings of the National Conference of Standards
Laboratories Workshop and Symposium, Atlanta GA. July 1997.
[A52] J. Song, The Guidelines for Expressing Measurement Uncertainties and the 4:1 Test
Uncertainty Ratio (TUR). Proceedings of the National Conference of Standards Laboratories
Workshop and Symposium. Atlanta GA. July 1997.
[A53] D. Deaver, Guardbanding and the World of ISO Guide 25 – Is There Only One Way?
Proceedings of the NCSL Workshop and Symposium, National Conference of Standards
Laboratories. Albuquerque NM, July 1998.
[A53A] S. Phillips, W. Estler, M. Levenson, K. Eberhardt, Calculation of Measurement
Uncertainty Using Prior Information.
[A54] R. Nicholas. Measurement Decision Risk Simplified. Proceedings of the Measurement
[A55] W. Wong. What TUR do You Really Need? Putting Statistical Theory Into Practice.
Proceedings of the Measurement Science Conference. Anaheim CA. January 1999.
[A56] R. Nicholas. Risk-Based Guardbanding for Unilateral Uncertainties. Proceedings of the
NCSL Workshop and Symposium. National Conference of Standards Laboratories. Charlotte
NC. July 1999.
[A57] W. Schultz, K. Sommer, Uncertainty of Measurement and Error Limits in Legal
Metrology. International Organization for Legal Metrology. OIML Bulletin Vol. 40, No. 4, pp 5-
15. October 1999.
[A57A] T. Estler, Measurement as Inference: Fundamental Ideas, Section 7.4 – Industrial
Inspections III: Accept/Reject Decisions. CIRP Annals – Manufacturing Technology. Vol. 48,
No. 2, pp 611 – 631. December 1999.
[A58] D. Mader, J. Prins, R. Lampe, The Economic Impact of Measurement Error. Quality
Engineering. Vol. 11, No. 4. Pp 563 – 574. 1999.
[A59] I. Lira, A Bayesian Approach to the Consumer’s and Producer’s Risk in Measurement.
Metrologia. Vol. 36, No. 5, pp 397 – 402. 1999.
2000’s
[A60] K. Sommer, M. Kochsiek, W. Schulz, Error Limits and Measurement Uncertainty in
Legal Metrology. XVI IMEKO World Congress. Vienna Austria. September 2000.
Appendix A
[A61] IEC CISPR/A/291/CDV, Accounting for Measurement Uncertainties When Determining

Compliance With a Limit. International Electrotechnical Commission / International Special
Committee on Radio Interference. Geneva Switzerland. December 2000.
[A62] N. Doganaksoy, Assessment of Impact of Measurement Variability in the Presence of
Multiple Sources of Product Variability. Quality Engineering. Vol. 13, No. 2, pp 83 – 89. 2000
[A62A] H. Castrup, Risk Based Control Limits. Proceedings of the Measurement Science
Conference. Anaheim CA. January 2001.
[A62B] R. Boyles, Gauge Capability for Pass-Fail Inspection. Technometrics. Vol. 43, No. 2,
pp 223 – 229. May 2001.
[A63] T. Skwirczyṅski, Uncertainty of the Calibrating Instrument, Confidence in the
Measurement Process and the Relation Between Them. OIML Bulletin, Vol. XLII, No. 3, pp 5-
10. July 2001.
[A63A] G. Arts, W. Albers, W. Kallenberg, Simultaneous Inspection of Several Product
Characteristics. Metrika. Vol. 54, No. 1, pp 19 – 41. August 2001.
[A64] P. Carbone, D. Macii, D. Petri, Measurement Uncertainty and Metrological Confirmation
in Documented Quality Systems. 11th IMEKO TC-4 Symposium – Trends in Electrical
Measurement and Instrumentation. Lisbon Portugal. September 2001.
[A65] P. Stein, Meeting Specifications – Compensate for Measurement Error in Test Results.
American Society for Quality (ASQ) – Quality Progress. Milwaukee WI. March 2002.
[A66] R. Burdick, Recent Extensions in Gauge Capability Studies. Quality and Productivity
Research Conference. American Statistical Association. University of Arizona. Tempe AZ. June
2002.
[A67] P. Stein, Statistical Issues in Measurement, Item #S1010. Special Publication – ASQ
Statistics Division. American Society for Quality. Milwaukee WI. Fall 2002.
[A68] D. Jackson, Measurement Uncertainty and Risk for Navy Calibrations. Proceedings of the
NCSL International Workshop and Symposium. San Diego CA. August 2002.
[A68A] I. Lira, Evaluating The Measurement Uncertainty – Fundamentals and Practical
Guidance. Section 4.3 – The Consumer’s and Producer’s Risks, Section 4.4 – The Acceptance
Interval: Guardbanding. pp 109 – 111. Section 6.7 – Evaluation of the Producer’s and
Consumer’s Risks. pp 192 – 200. Institute of Physics – Series in Measurement Science and
Technology. ISBN 0 7503 0840 0. IOP Publishing Ltd. Bristol UK and Philadelphia PA. 2002.
[A69] P. Carbone, D. Macii, D. Petri, Measurement Uncertainty and Metrological Confirmation
in Quality-Oriented Organizations. Measurement. Vol. 34, pp 263-272. 2003
[A70] D. Macii, P. Carbone, D. Petri, Management of Measurement Uncertainty for Effective
Statistical Control. IEEE Transactions on Instrumentation and Measurement. Vol. 52, No. 5.
October 2003.
[A70A] R. Burdick, C. Borror, D. Montgomery, A Review of Methods for Measurement System
Capability Analysis. 47th Annual Fall Technical Conference of the Chemical and Process
Industries Division and Statistics Division of the American Society for Quality and the Section
on Physical Engineering Sciences of the American Statistical Association. El Paso TX. October
2003.

Appendix A
[A71] H. Kӓllgren, M. Lauwaars, B. Magnusson, L. Pendrill, P. Taylor, Role of Measurement

Uncertainty in Conformity Assessment in Legal Metrology and Trade. Accreditation and Quality
Assurance. Vol. 8, No. 11, pp 541-547. November 2003.
[A72] J. Sheppard, Accounting for False Indication in a Bayesian Diagnostics Framework.
Proceedings of Autotestcon 2003 – IEEE Systems Readiness Technology Conference. Future
Sustainment for Military and Aerospace. Cat No. 03CH37447. pp 273 – 279. ISBN 0-7803-
7837-7. September 2003.
[A73] P. Carbone, D. Macii, D. Petri, Management of Measurement Uncertainty for Effective
Statistical Control. Technical Report No. DIT-04-053. University of Trento. Department of
Information and Communication Technology. Trento Italy. May 2004.
[A74] D. Shah, The Uncertain State of Uncertainty in Industry. Proceedings of the ASPE,
Uncertainty Analysis in Measurement and Design. American Society for Precision Engineering.
University Park PA. June 2004.
[A75] R. Nicholas, L. Anderson. Guardbanding Using Automated Software. Proceedings of the
NCSL International Workshop and Symposium. Salt Lake City UT. July 2004.
[A76] L. Rodriguez, A. Arroyo, O. Ramirez, Implementation of a Methodology for Quantifying
the Economic Effect of Measurement Errors. Proceedings of the NCSL International Workshop
and Symposium. Salt Lake City UT. July 2004.
[A77] K. Sommer, M. Kochsiek, Modeling and Uncertainty Evaluation in Calibration and
Conformance Testing. Proceedings of the NCSL International Workshop and Symposium. Salt
Lake City UT. July 2004.
[A78] S. Phillips, Measurement Uncertainty and Traceability Issues: A Standards Activity
Update. Proceedings of ASPE Spring Topical Meeting on Uncertainty Analysis in Measurement
and Design. July 2004.
[A79] AFNOR FD x07-022, Metrology and Applications of Statistics - Use of Measurement
Uncertainties: Presentation of Some Cases and Customary Practices. Section 6 – Using the
Uncertainty in Conformity Assessment: Evaluation of the Associated Risk. pp 16 – 21. AFNOR
Groupe. Association Française de Normalisation (French Association for Standardization).
December 2004.
[A80] D. Jackson, Measurement Decision Risk and Methods. Proceedings of the Measurement
[A81] K. Bennett, H. Zion, Metrology Concepts: Understanding Test Uncertainty Ratio (TUR).
Transcat White Paper, P/N 7-05-879. Rochester NY. May 2005.
[A82] R. Burdick, Y. Park, D. Montgomery, C. Borror, Confidence Intervals for
Misclassification Rates in a Gauge R&R Study. Journal of Quality Technology. Vol. 37, No. 4,
pp 294 – 303 . October 2005.
[A83] D. Jackson, Measurement Decision Risk Analysis Methods with Multiple Test Points.
Proceedings of the Measurement Science Conference. Anaheim CA. January 2006.
[A84] J. Sheppard, M. Kaufman, A Bayesian Approach to Diagnosis and Prognosis Using Built-
In Test. IEEE Transactions on Instrumentation and Measurement. Vol. 54, No. 3, pp 1003 –
1018. June 2005.

Appendix A
[A85] R. Burdick, C. Borror, D. Montgomery, Design and Analysis of Gauge R&R Studies:
Making Decisions with Confidence Intervals in Random Mixed ANOVA Models. Section 1.6 –
Misclassification Rates. ASA-SIAM Series on Statistics and Applied Mathematics. Society for
Industrial and Applied Mathematics. ISBN 978-0-89871-588-0. 2005.
[A85A] G. Rossi, F. Crenna, A Probabilistic Approach to Measurement-Based Decisions.
Measurement. Vol. 39, No. 2, pp 101-119. February 2006.
[A86] WELMEC, Elements for Deciding the Appropriate Level of Confidence in Regulated
Measurements. (Accuracy Classes, MPE In-service, Non-conformity, principles of Uncertainty).
Section 6 – Measurement Uncertainty and Decision-Making. WELMEC 4.2, Issue 1. European
Co-operation in Legal Metrology. Overveen Netherlands. June 2006.
[A87] D. Jackson, Measurement Risk Analysis Methods as Applied to Guardbands. Proceedings
of the NCSLI International Workshop and Symposium. Nashville TN. August 2006.
[A88] R. Nicholas. The Significance on Consumer and Producer Risk for Uncorrected Product
and Measurement Biases. NCSL International Workshop and Symposium. Nashville TN.
August 2006.
[A89] L. Pendrill, Optimised Decision-Making in Conformity Assessment, Proceedings of the
NCSLI International Workshop and Symposium. Nashville TN. August 2006.
[A89A] D. Wheeler, EMP III, Evaluating the Measurement Process & Using Imperfect Data,
Chapter 14 – The Basis for Manufacturing Specifications, Section 14.3 The Bivariate Normal
Model, Section 14.4 The Probability of Conforming Product, pp 181 – 193. SPC Press.
Knoxville TN. October 2006.
[A89B] D. Wheeler, An Honest Gauge R&R Study. ASQ/ASA Fall Technical Conference.
Manuscript No. 189 (January 2009 Revision). American Society for Quality / American
Statistical Association. Columbus OH. October 2006.
[A90] L. Pendrill, Optimised Measurement Uncertainty and Decision-Making when Sampling by
Variables or by Attribute. Measurement. Vol. 39, No. 9, pp 829-840. November 2006.
[A91] A. Forbes, Measurement Uncertainty and Optimized Conformance Assessment.
Measurement. Vol. 39, No. 9, pp 808 – 814. November 2006.
[A91A] H. Castrup, S. Castrup, Analytical Metrology Handbook, Part 3 – Analyzing
Measurement Decision Risk. First edition. Integrated Sciences Group. Bakersfield CA. 2008.
[A92] H. Castrup, Bayesian Risk Analysis. Technical Document. Integrated Sciences Group.
Bakersfield CA. April 2007.
[A93] Y. Kim, B. Cho, N. Kim, Economic Design of Inspection Procedures Using Guard Band
When Measurement Errors are Present. Applied Mathematical Modeling, Vol. 31, No. 5. pp
805-816. May 2007.
[A94] L. Pendrill, Optimised Measurement Uncertainty and Decision-Making in Conformity
Assessment, NCSLI Measure – The Journal of Measurement Science. Vol. 2, No. 2. June 2007.
[A95] M. Dobbert, Understanding Measurement Risk. Proceedings of the NCSL International
Workshop and Symposium. St. Paul MN. July 2007.
[A96] H. Castrup, Risk Analysis Methods for Complying with Z540.3. Proceedings of the NCSL
International Workshop and Symposium. St. Paul MN. July 2007.

Appendix A
[A96A] TAF-CNLA-G-04, Method of Stating Test and Calibration Results and Compliance
With Specification. Taiwan Accreditation Foundation. Taiwan ROC. July 2007.
[A97] OMCL PA/PH/OMCL (05) 49 DEF CORR, Uncertainty of Measurement – Part 1.
General OMCL Policy for Implementation of Measurement Uncertainty in Compliance Testing.
European Directorate for the Quality of Medicines & Healthcare. December 2007.
[A97A] Z. Kosztyán, T. Csizmadia, C. Hegedűs, Z. Kovacs, Treating Measurement Uncertainty
in Complete Conformity Control System. Conference: Innovations and Advances in Computer
Sciences and Engineering. Vol.1 of Proceedings of the 2008 International Conference on
Systems, Computing Sciences and Software Engineering. Bridgeport CT. December 2007.
[A98] F. Wang, S. Yang, Applying Bootstrap Method to the Types I-II Errors in the
Measurement System. Quality and Reliability Engineering International. Vol. 24, No. 1, pp 83 –
97. February 2008.
[A99] D. Caldwell, D. Jackson. Measurement Risk Analysis Definitions as Applied to Z540.3.
Proceedings of the Measurement Science Conference. Anaheim CA. March 2008.
[A100] R. Nicholas. How to Build Your Own Unilateral Consumer Risk Calculation Tool,
Including the Treatment of Biases. Proceedings of the Measurement Science Conference.
Anaheim CA. March 2008.
[A101] H. Castrup, Applying Measurement Science to Ensure End Item Performance.
Proceedings of the Measurement Science Conference. Anaheim CA. March 2008.
[A102] R. Walker. Evaluation of Out-of-Tolerance Risk in Measuring and Test Equipment.
Proceedings of the NCSL International Workshop and Symposium. Orlando FL. August 2008
[A103] M. Dobbert, A Guard-Band Strategy for Managing False-Accept Risk. Proceedings of
the NCSL International Workshop and Symposium. Orlando FL. August 2008.
[A103A] M. Cox, G. Rossi, P. Harris, A. Forbes, A Probabilistic Approach to the Analysis of
Measurement Processes. Metrologia. Vol. 45, No. 5. August 2008.
[A104] Transcat, Risk Management: The Critical Role of Measurement and Calibration.
Transcat White Paper, P/N TC918 10-08. Rochester NY. October 2008.
[A105] K. Weißensee, O. Kühn, G. Linß, K. Sommer. Risk of Erroneously Deciding Conformity
of Measuring Instruments. Accreditation and Quality Assurance. Vol. 13, No. 11, pp 663 – 669.
November 2008.
[A106] L. Pendrill, Review of Symposium ‘New Developments in Measurement Uncertainty in
Chemical Analysis’. Accreditation and Quality Assurance. Vol. 13, No. 11, pp 671 – 674,
November 2008.
[A107] A. Williams. Compliance With Specifications. Accreditation and Quality Assurance.
Vol. 13, pp 617 – 618, November 2008.
[A108] A. Williams, Principles of the EURACHEM/CITAC Guide ‘Use of Uncertainty
Information in Compliance Assessment’. Accreditation and Quality Assurance. Vol. 13, No. 11,
pp 633 – 638. November 2008.
[A109] M. Czaske, Usage of the Uncertainty of Measurement by Accredited Calibration
Laboratories When Stating Compliance. Accreditation and Quality Assurance. Vol. 13, No. 11,
pp 645-651. November 2008.

Appendix A
[A110] E. Desimoni, B. Brunetti, A. Clerici, About Considering Both False Negative and False-
Positive Errors When Assessing Compliance and Non-compliance with Reference Values given
in Compositional Specifications and Statutory Limits. Accreditation and Quality Assurance. Vol.
13, No. 11, pp 653 – 662. November 2008.
[A111] M. Dobbert, A Guard-Band Strategy for Managing False-Accept Risk. NCSLI Measure –
The Journal of Measurement Science. Vol. 3, No. 4, pp 44-48. December 2008.
[A112] D. Macii, D. Petri, Guidelines to Manage Uncertainty in Conformance Testing
Procedures. IEEE Transactions on Instrumentation and Measurement. Vol. 58, No. 1. January
2009.
[A112A] I. Lira, Comment on ‘A Probabilistic Approach to the Analysis of Measurement
Processes’. Metrologia. Vol. 46, No. 1. January 2009.
[A113] M. Dobbert, R. Stern, A Pragmatic Method for Pass/Fail Conformance Reporting that
Complies with ANSI Z540.3, ISO 17025, and ILAC-G8. Proceedings of the NCSL International
Workshop and Symposium. San Antonio TX. July 2009.
[A114] H. Castrup, An Examination of Measurement Decision Risk and Other Measurement
Quality Metrics. Proceedings of the NCSL International Workshop and Symposium. San
Antonio TX. July 2009.
[A115] D. Huang, S. Dwyer, Test Instrument Reliability Perspectives and Practices: Part 1:
Interpreted Within System Reliability Framework. Proceedings of the NCSL International
Workshop and Symposium. San Antonio TX. July 2009.
[A116] S. Khanam, Test Uncertainty Ratio (TUR) and Test Uncertainty. Ph.D Dissertation in
Mechanical Engineering. University of North Carolina at Charlotte. Charlotte NC. 2009
[A116A] APLAC TC-004, Method of Stating Test and Calibration Results and Compliance With
Specifications. Asia Pacific Laboratory Accreditation Cooperation. October 2009.
[A117] E. Desimoni, B. Brunetti, Uncertainty of Measurement and Conformity Assessment: A
Review. Analytical and Bioanalytical Chemistry. Vol. 400, No. 6, pp 1729 – 1741. June 2011.
[A118] J. Hurll, Assessment of Compliance With Specifications. Cal Lab Magazine. Vol. 15, No.
4, pp 34-37. Oct-Nov-Dec 2009.
[A119] R. Fritzsche, Metrology and Calibration (METCAL) Overview. Naval Surface Warfare
Center, Corona Division – Measurement Science Department. Corona CA. Undated.
2010’s
[A120] D. Deaver, J. Somppi, A Study of and Recommendations for Applying the False
Acceptance Risk Specification of Z540.3. Proceedings of the Measurement Science Conference.
[A121] M. Dobbert, R. Stern, A Pragmatic Method for Pass/Fail Conformance Reporting that
Complies with ANSI Z540.3, ISO 17025, and ILAC-G8. NCSLI Measure – The Journal of
Measurement Science. Vol. 5, No. 1. March 2010.
[A122] D. Jackson, Threshold Analysis. Proceedings of the Measurement Science Conference.
[A123] W. Hinrichs, The Impact of Measurement Uncertainty on the Producer’s and User’s
Risks, on Classification and Conformity Assessment: An Example Based on Tests on Some
Appendix A
Construction Products. Accreditation and Quality Assurance. Vol. 15, No. 5, pp 289-296. May
2010.
[A124] M. Yiu. Calibration is Changing. EDN Network. June 2010.
[A125] S. Mimbs, Conformance Testing: Measurement Decision Rules. Proceedings of the
NCSL International Workshop and Symposium. Providence RI. July 2010.
[A126] L. Lewis, Managing Risk Using the Guard Band Method 2 % Bayesian Analysis.
Proceedings of the NCSL International Workshop and Symposium. Providence RI. July 2010.
[A127] S. Beckert, E. Paladini, L. Valentina, Analysis of the Acceptable Deviation for the
Measurement Process Established by the MSA and the VDA Manuals. Proceedings of the World
Congress on Engineering. Vol III. London England. July 2010.
[A128] J. Salsbury, Uncertainty Management Versus Risk Management in Calibration. NCSLI
Measure – The Journal of Measurement Science. Vol. 5, No. 3. September 2010.
[A129] J. DeMast, T. Erdmann, W. Van Wieringen, Measurement System Analysis for Binary
Inspection. Continuous Versus Dichotomous Measurands. Journal of Quality Technology.
American Society for Quality. Vol. 43, No. 2. April 2011.
[A130] Y. Zhu, M. Li, C. Young, M. Xie, E. Elsayed, Impact of Measurement Error on
Container Inspection Policies at Port-of-Entry. Annals of Operations Research. Vol. 187, No. 1,
pp 22-43. July 2011.
[A131] D. Jackson Enhanced Equipment Calibration Tradeoff Analysis. Proceedings of the
NCSL International Workshop and Symposium. National Harbor MD. August 2011.
[A132] H. Alexander, How to Satisfy the Requirements of ISO/IEC 17025:2005 When Reporting
Calibration or Test Results and Making a Statement of Compliance with a Metrological
Specification from the Perspective of the Accrediting Body. Proceedings of the NCSL
International Workshop and Symposium. National Harbor MD. August 2011.
[A133] ASTM E2782, Standard Guide for Measurement System Analysis (MSA), Section 9 –
Measurement System Performance.; Section 10.7 – Probability of Acceptance Given
Measurement Value; Section 10.8 – Probability of Gage Acceptance Given True Measurement.
ASTM International. West Conshohocken. 2011 (editorial corrections, May 2014).
[A134] S. Rishi, Guard-Banding Methods – An Overview. Proceedings of AdMET, Paper No.
UM001. London England. 2012.
[A135] C. Grachanen, Are TUR’s a Thing of The Past? The Standard. American Society for
Quality (ASQ). Vol. 26, No. 1, pp 4 – 5. March 2012.
[A135A] J. Salsbury, E. Morse, Measurement Uncertainty in the Performance Verification of
Indicating Measuring Instruments. Precision Engineering. Vol. 36, No. 2, pp 218 – 228. April
2012.
[A135B] D. Montgomery, Introduction to Statistical Quality Control, Chapter 8 – Process and
Measurement System Capability Analysis. Section 8.7.4 – False Defectives and Passed
Defectives. 7th edition. John Wiley & Sons, Inc. June 2012.
[A136] M. Dobbert, Understanding Measurement Risk. Agilent Technologies (now Keysight
Technologies). White Paper 5991-1265EN. Santa Rosa CA. October 2012.

Appendix A
[A137] M. Dobbert, A Guard-Band Strategy for Managing False-Accept Risk. Agilent

Technologies (now Keysight Technologies). White Paper 5991-1267EN. Santa Rosa CA.
October 2012.
[A138] M. Weitzel, W. Johnson, Using Target Measurement Uncertainty to Determine Fitness
for Purpose. Accreditation and Quality Assurance. Vol. 15, No. 5, pp 491 – 495. October 2012.
[A139] T. Wendle, Too Much Calibration? Agilent Technologies (now Keysight Technologies).
White Paper 5991-1311EN. Santa Rosa CA. November 2012.
[A140] SureCal TN-201203, Interpreting Acceptance Criteria as Reported in SureCal. Northrup
Grumman Systems Corporation. December 2012.
[A141] ISO/TR 14253-6:2012, Geometrical Product Specifications (GPS) — Inspection by
Measurement of Workpieces and Measuring Equipment — Part 6: Generalized Decision Rules
for the Acceptance and Rejection of Instruments and Workpieces. International Organization for
Standardization. Geneva Switzerland. 2012
[A142] B. Stern, How to Comply with ANSI Z540.3. Webcast. Agilent Technologies (now
Keysight Technologies). Santa Rosa CA. February 2013.
[A143] W. Estler, D. Hibbert, JCGM 106:2012 – A New Guidance Document on Measurement
Uncertainty and Conformity Assessment. International Organization of Legal Metrology. OIML
Bulletin Vol. LIV, No. 2. pp 14 – 16. April 2013.
[A143A] S. Stamm, A Comparison of Gauge Repeatability and Reproducibility Methods. PhD
Dissertation. Indiana State University. Terre Haute IN. May 2013.
[A144] M. Bar, C. Elster, S. Heidenreich, C. Matthews, L. Pendrill, L. Wright, Novel
Mathematical and Statistical Approaches to Uncertainty Evaluation: Introducing a New EMRP
Research Project, Section 4 – Conformity Assessment and Decision Making. Conference Paper,
16th International Congress of Metrology. Paris France. October 2013.
[A145] ASTM E29-13, Standard Practice for Using Significant Digits in Test Data to Determine
Conformance With Specifications. ASTM International. Philadelphia PA. 2013
[A146] OIML TC3/SC5/p2/N002/2CD, The Role of Measurement Uncertainty in Conformity
Assessment Decisions in Legal Metrology. International Organization of Legal Metrology
(OIML) Committee Draft 2. February 2014.
[A147] H. Mezouara, L. Dlimi, A. Salih, M. Afechcar. Determination of the Optimal
Guardbanding to Ensure Acceptable Risk Decision in the Declaration of Conformity. Journal of
Engineering Research and Applications. Vol. 4, No. 6, pp 16-20. June 2014.
[A148] C. Grachanen, Understanding Test Accuracy and Test Uncertainty Ratios. Quality
Progress. American Society for Quality (ASQ). Vol. 47, No. 7, pp 44 – 45. July 2014.
[A149] R. Long, Taking My Uncertainty Into Account… Proceedings of the NCSL International
Workshop and Symposium. Orlando, FL. July 2014.
[A149A] J. Harben, Descriptive Implementation of Prior Knowledge. Proceedings of the NCSL
International Workshop and Symposium. Orlando, FL. July 2014.
[A150] R. Long, Taking My Uncertainty Into Account… I Have My Uncertainties, Now What?
Presentation at the NCSL International Workshop and Symposium. Orlando, FL. July 2014.

Appendix A
[A150A] N. Finstrom, J. Strief, T. Haddad, E. Maass, K. Hulting, A Comparison of Measurement

System Analysis Metrics: Part 1 of 2. iSixSigma. July 2014
[A150B] N. Finstrom, J. Strief, T. Haddad, E. Maass, K. Hulting, A Comparison of Measurement
System Analysis Metrics: Part 2 of 2. iSixSigma. July 2014
[A151] L. Pendrill, Using Measurement Uncertainty in Decision-Making and Conformity
Assessment. Metrologia. Vol. 51, No. 4, pp 206-218. August 2014.
[A151A] C. Hegedűs, Risk-Based Consideration of Measurement Uncertainty in Decisions.
Pannon Management Review. Vol.3, No. 3. Pp 73 – 96. September 2014.
[A152] C. Burgess, Using the Guard Band to Determine a Risk-Based Specification.
Pharmaceutical Technology. Vol. 38, No. 10. October 2014.
[A153] ASME B89.7.2-2014, Dimensional Measurement Plan, Appendix D - Probabilities of
Pass and Fail Errors. American Society of Mechanical Engineers. New York NY. December
2014.
[A153A] C. Hegedűs, Support of Risk-Based Decisions in Conformity and Control Considering
Measurement Uncertainty. PhD dissertation. University of Pannonia. Veszprém Hungary. 2014.
[A154] C. Shakarji, S. Phillips, Understanding the Ramifications of Important Proposed
Changes to Decision Rules in Three Standards Within ISO and ASME. Presentation at the NCSL
International Workshop and Symposium. Grapevine TX. July 2015.
[A155] J. Sims, Process Accuracy Ratio vs Process Uncertainty Ratio – Risk Mitigation:
Calibration and the Customer’s Process. Transcat white-paper. Rochester NY. 2014.
[A155A] M. Kuster, Resolving Resolution Uncertainty. Measure – The Journal of Measurement
Science. NCSL International. Vol. 10, No. 3, September 2015.
[A156] A. Dregelyi-Kiss, Performance Verification of Dimensional Measuring Instruments in
Automotive Industry. Proceedings of First International Academy for Quality World Quality
Forum (IAQ-WQF). Budapest Hungary. October 2015.
[A157] ISO 14253-5:2015, Geometrical Product Specifications (GPS) — Inspection by
Measurement of Workpieces and Measuring Equipment — Part 5: Uncertainty in Verification
Testing of Indicating Measuring Instruments. International Organization for Standardization.
Geneva Switzerland. 2015.
[A158] Eurachem/CITAC STMU, Guide on Setting and Using Target Uncertainty in Chemical
Measurement. Eurachem and the Cooperation on International Traceability in Analytical
Chemistry. ISBN 978-989-98723-7-0. First Edition. 2015.
[A159] C. Burgess, P. Curry, D. LeBlond, G. Gratzl, E. Kovacs, G. Martin, R McGregor, P.
Netthercote, H. Pappa, J. Weitzel. Fitness for Use: Decision Rules and Target Measurement
Uncertainty (Stimuli to the Revision Process). U. S. Pharmacopeial Convention (USP). Vol. 42,
No. 2. February 2016.
[A160] D. Jackson, Equipment Level Risk Assessment. Proceedings of the Measurement Science
Conference. Anaheim CA. March 2016.
[A161] R. Stern, J. Harben, Conformance Decision Rules to Support ISO/IEC CD2 17025 Under
Revision. Proceedings of the NCSL International Workshop and Symposium. St. Paul MN. July
2016.

Appendix A
[A162] I. Lin, The Influence of Measurement Uncertainty in Conformity Assessment. NCSL

International Workshop and Symposium. Conference Poster Session. St. Paul MN. July 2016.
[A163] J. Salsbury, Understanding the Test Measurand and the Profound Impact on Calibration,
Verification, and Uncertainty. Proceedings of the NCSL International Workshop and
Symposium. St. Paul MN. July 2016.

Appendix B
(Chronological Summary of Some Accuracy or Uncertainty Ratio Requirements)
MIL-STD-120 (1950): [2]
“2.3.3 In general practice, gage tolerances should not exceed 10 percent of the tolerance of the part to be
gaged [10:1 TAR].
2.3.6 Disputed Rejections. Any part which is so close to either rejection limit as to be improperly rejected [false
reject] either as a result of tolerance or wear of the inspection gages, will be reinspected…Any observational
errors in the inspection must, however, be in the direction of safety rather than in the direction of danger of
acceptance of improper parts [false accept].
6.2.5 Gages in use must be checked periodically to insure that they have not worn to the extent that they
accept defective material or reject acceptable material.
8.3.4.3 When the dimensions of gages are very close to the specified limits… the gage inspector will often be
required to exercise considerable judgment in deciding whether or not the gage should be accepted.
For example, a plain “go” plug gage is given a plus tolerance. If the plus tolerance should be a little greater
than that specified, the plug gage will tend to reject a few more components than a properly made plug gage
[false-reject].
On the other hand, if a plain “Go” plug gage should have a diameter less than nominal, in other words, if it is
undersize, the gage could pass unsatisfactory components [false-accept].
8.3.5.2 The measuring instruments used should always be suitable for the tolerance specified on the gage
drawings… In general, the accuracy of the measuring instruments should be less than 20 percent [5:1 TAR] of
the tolerance on the gage being inspected. A measuring instrument which has an accuracy of 10 percent
[10:1 TAR] of the gage tolerance should be used whenever such an instrument is available provided that its
use does not involve an excessive expenditure of time. When the acceptability of gage is questionable
because it is near the tolerance limit, the gage may be re-inspected by more accurate instruments.”
NAVY: Tech Memo 63-106 (1955): [A5]

“The value of a program aimed at improving and ensuring the reliability… is degraded if the quality and proof
tests to which… components and systems are subjected, are not in themselves reliable.
Considerable effort has been expended on these relationships by Mr. Alan R. Eagle and Dr. Frank E. Grubbs.
Their work has resulted in a mathematical expression for the relationship between design [specification] and
test tolerance [guard band] and allowable instrumentation error [measurement uncertainty].
…to ensure… a specific testing risk or reliability can be attained… establish a reasonable ratio between A.I.E.
[allowable instrumentation error – or measurement uncertainty] and the Design Tolerance [specification of the
item being tested] based on known accuracy limitation of the measuring instrument available.”
MIL-C-45662 (1960): [12]

“3.1.2 Adequacy of Standards: Standards established by the contractor for this calibration system shall be
those authorized or recognized as having capabilities for accuracy, stability, and range required for the
intended use.
3.1.5 Calibration Procedures. Written procedures shall be prepared and utilized for calibration of all standards
used to control the accuracy of the measuring and test equipment involved in establishing product
conformance. The procedure shall require that calibration be performed by the comparison with higher level
standard utilizing a ten to one (10:1) accuracy ratio objective. Cross checking between equal or near equal
standard shall not be acceptable except for those cases where the state of the art does not provide a higher
ratio”.

Appendix B
NASA NPC 200-2 (1962): [174]

“9.3(b) Within state-of-the art limitations, the standards used for calibration of inspection, measuring, and test
equipment shall have a tolerance no greater than 10 % of the allowable tolerance for the equipment being
calibrated.
9.4 Evaluation. …evaluations shall determine the accuracy of the inspection, measuring, and test equipment
for determining the quality of the articles. Evaluations shall determine the amount of the specified product
tolerance that will be occupied by the inspection, measuring, and test equipment tolerances and inaccuracies.
“…tests shall provide assurance that the indicated measurements [errors] are those of the article being
measured and not of the test equipment itself.”
MIL-HDBK-52 (1964): [13]

“3.2.1(c) “…assure that measuring and test equipment used in product inspection is calibrated against
reference standards having values of 4 to 10 times their accuracy.
Also, the reference standards used in the contractor's system should be calibrated against higher level
standards having recommended values of 4 to 10 times the accuracy of the reference standards.
A value of less than 4 times the accuracy level may be acceptable to the government representative for those
requirements which approach the limits of knowledge for given characteristic ."
NASA NHB 5300.2 (1965): [175]

“3.3 Selection. The selection of standards and ‘equipment’ will be based on the following criteria:
a. The accuracy, stability, and range of standard and ‘equipment’ will be in conformance with end-use
requirements. Within state-of-the-art limitations, the standard used for calibration of inspection, measuring
and test equipment will have a tolerance no greater than 10 per cent of the allowable tolerance of the
‘equipment’ being calibrated, i.e., I the item to be calibrated has a tolerance of ±1.0 per cent, the maximum
allowable tolerance of the standard should be ±0.1 per cent. If the state-of-the-art prevents the attainment of
this 10 to 1 ratio, authorization must be secured prior to use.”
NASA NHB-5300.4-1B (1969): [176]

Formerly NPC 200-2
“1B903. Article or Material Measurement Process. Random and systematic errors in any article or material
measurement process shall not exceed 10 % of the tolerance of the article or material characteristic being
measured. Authorization for exemption shall be required from the procuring NASA installation.
1B904. Calibration and Measurement Processes. Random and systematic errors in any calibration
measurement process shall not exceed 25 % of the tolerance of the parameter being measured. Authorization
for exemption shall be required from the procuring NASA installation”.
RDT F3-2T (1969): [203]
“3.2 (a) Standards established by the manufacturer for calibrating the measuring and test equipment used to
control and verify product quality shall be accurate and stable over the range of calibration necessary to
assure the product requirements are met and that any deviation from product requirements are accurately
known...(d) The procedure shall require that calibration be performed by comparison with higher accuracy
level standards.”

Appendix B
IEEE STD-498-1975 (1975): [112]
“3.1 Adequacy of Reference Standards: Reference standards used for calibrating M&TE shall have an
accuracy level, acceptable calibration ranges, and precisions that are equal to or better than those required of
M&TE. The accuracies of the M&TE and the reference standard should be chosen such that the equipment
being calibrated can be calibrated and maintained within the required tolerances.”
NATO AQAP-6 (1976): [14]

“204. All measurements, whether made for purposes of calibration or product characteristic assessment, shall
take into account the total error in the measurement process attributable to the standard, or measuring
equipment, and as appropriate those contributed by personnel, procedure and environment.
206 …measuring equipment and standards shall be the subject of records which include… a statement of the
cumulative effects of errors on the data obtained in the calibration…
214. The cumulative effect of the errors in each successive component of a calibration chain shall be taken
into account for each standard or measure equipment calibrated. Corrective action shall be taken when the
total error is such as to compromise significantly the ability to make measurements within the designated
limits. The basis for the calculation of the cumulative error shall be recorded”.
NATO AQAP-7 (1978): [15]

“201 …use measuring devices and techniques of a suitably higher degree of accuracy than the tolerance of
the product parameters.
203… ensure that standard and measuring equipment… are of the accuracy, stability, and range appropriate
to the intended applications.
A significant factor in planning for… measurement.., is the… determination of what measurement equipment
will be required, what its accuracy must be…
204 Error is inherent in all measurement processes, whether performed for purposes of calibration, or
product characteristic measurement. Valid estimates of the amount of error can be determined and must be
taken into account, as appropriate, not only in the use of standards in subsequent calibrations, but also in the
use of measuring equipment when making decisions on whether measured values of product characteristics
fall within specified limits”.
IEEE STD-498-1980 Section 4.1 (1980): [113] and IEEE STD 498-1985 Section 5.1 (1985): [114]
“Adequacy of Reference Standards: Reference standards used for calibrating M&TE shall have an accuracy
level, acceptable calibration ranges, and precisions that are equal to or better than those required of M&TE.
The accuracies of the M&TE and the reference standard should be chosen such that the equipment being
calibrated can be calibrated and maintained within the required tolerances.
In general, the inaccuracy of the reference standard shall contribute no more than one fourth of the allowable
measuring and test tolerance. However, when the actual inaccuracy of the measuring and test equipment is
less than on fourth of the plant equipment tolerance, the requirement of one fourth the tolerance between the
reference standard and measuring at test equipment may not be necessary. The rational for deviating from
these requirement must be justified and documented.”

Appendix B
MIL-HDBK-52A (1984): [31]
“The standards used for calibrating M&TE shall have capabilities for accuracy, stability, range and resolution
required for the intended use. Accuracy ratios may be used for determining adequacy. The accuracy ratio may
be high (4:1, 10:1, or higher) or low (3:1, or 2:1). The exact ratio depends on state-of-the-art limitations, and
the contactor’s measurement requirements. The Government representative must verify that standards used
by the contractor are capable of calibrating the M&TE for the intended use. The accuracy of the standards
must at least be equal to the tolerance required (i.e., 1:1) but in most cases should be greater. A 1:1
comparison is permitted only when state-of-the-art limitations preclude a higher accuracy ratio. Normally,
when only a 1:1 accuracy ratio can be achieved, any out of tolerance condition of the M&TE will be
significant.”
ANSI/ASQC M1-1987 (1987): [74]

(Reaffirmed in 1996)
“2.5 Accuracy Ratio: The Ratio of the tolerance of the instrument being calibrated to the uncertainty of the
standard.
4.3 Calibration ensembles shall have accuracies sufficient for the purpose of the calibration performed with
them.
4.3.1 Calibration Uncertainties. The uncertainty of calibration ensembles shall be taken into account by
increasing the limits of uncertainty of the calibration or by decreasing the acceptance/rejection limits of
tolerance of the calibration by an amount at least equal to the uncertainty of the calibration ensemble, unless
such uncertainty is acceptable or negligible with respect to the required uncertainty limits.
4.3.2 Accuracy Ratios. When the use of a fixed accuracy ratio is determined to be acceptable to establish a
sufficient level of measurement accuracy between the ensemble to be calibrated and the calibration ensemble,
the ratio and rational for its selection shall be clearly specified.
Comments: The specification in this standard of a minimum accuracy ratio is considered invalid because the
acceptable risk associated with a measurement process may vary with each process, thereby suggesting
varying accuracy ratios.
Appendix. In the application of the more traditional approach to calibration involving accuracy ratios, one
typically find requirements to the effect that any instrument or standard used to calibrate another device must
have an accuracy 10 times better than that of the device being calibrated. As an example, a device having a
±0.5 percent tolerance might be calibrated using a standard having an uncertainty of ±0.05 percent or better,
and in turn might be calibrated using an item having an uncertainty of ±0.005 percent or better… In such a
process, one makes no attempt to determine precisely the actual errors introduced at each step. It is
assumed that because of the built-in ‘safety factor’ of 10 at each level, the calibration hierarchy will produce
devices of adequate accuracy. In most cases, experience shows that this works quite well, provided devices
having the necessary accuracy are available and providing the waste of information in tolerable.”
MIL-STD-45662A (1988): [29]

“5.2 Measurement standards used… for calibrating M&TE and other measurement standards shall be
traceable and shall have the accuracy, stability, range and resolution required for the intended use. Unless
otherwise specified in contract requirements, the collective uncertainty of the measurement standards shall not
exceed 25 percent of the acceptable tolerance for each characteristic being calibrated. The… calibration
system description may include provisions for deviating for the uncertainty requirements, provided the
adequacy of the calibration is not degraded. All deviations must be documented.”

Appendix B
MIL-HDBK-52B (1989): [177]

“5.2 Adequacy of measurement standards. Measurement standards used by the contractor for calibrating
M&TE and other measurement standards shall be traceable and shall have the accuracy, stability, range and
resolution required for the intended use. Unless otherwise specified in the contract requirements, the collective
uncertainty of the measurement standards shall not exceed 25 percent of the acceptable tolerance for each
characteristic being calibrated. The contractor's calibration uncertainty requirements provided t h e adequacy
of t h e calibration is not degraded. All deviations shall be documented.
Concept:
The two principal and essential requirements for measurement standards are traceability and accuracy. The
traceability requirement is as defined in paragraph 3.5. The accuracy requirement is resolved by comparing
the accuracy attributes of the measurement standard (expressed as uncertainty) to the tolerances of the
characteristics being calibrated. Since it is recognized that it is not possible to have a perfect measurement
standard (i.e., one without any error or uncertainty), some amount of uncertainty is allowable. As stated, the
MIL-STD provides for an uncertainty limit of 25% of the tolerance of the characteristic being calibrated unless
otherwise specified in the contract. This is commonly called a 4:1 test accuracy ratio. When test accuracy
ratios of 4:1 or greater are maintained, the error contributed by the measurement standards is considered
negligible. However, when lesser ratios are used due to state-of-the-art or other limitations, the relatively larger
error of the measurement standards increases the risks of erroneous acceptance or rejection of the instrument
being calibrated. These conditions could ultimately lead to increased acceptance of
bad material by the Government or rejection of good material by the contractor .
Application:
The adequacy of a measurement standard for its intended use is determined by comparing its accuracy,
stability, range and resolution to the expected values and tolerances of the instrument’s characteristic being
calibrated. The collective uncertainty of the measurement standards used to perform the calibration is
typically a combination of the inherent accuracy, including stability and resolution, of each standard comprising
the calibration system. For purpose of compliance to paragraph 5.2 of the MIL-STD, collective uncertainty of
the measurement standards does not include other possible sources of errors in the use of the standards,
such as technician or procedure. Methods for combining uncertainties of the characteristics of more than one
measurement standard include simple arithmetic addition, root sum of squares (RSS), use of partial
derivatives, distribution analysis, or some combination of these or other methods. The contractor may use any
or all applicable methods but should describe and justify the methodology used in his calibration system
description. If there are questions concerning the contractor's methods and conclusions in developing
collective uncertainty estimates, the Government representative should request assistance from the Metrology
Engineering Center of the cognizant DOD agency.
To verify the adequacy of a measurement standard or group of measurement standards for its intended use,
the Government representative should review the calibration procedure in which the measurement standards
are used. Since paragraph5 .5 of the MIL-STD requires that the calibration procedure contain the required
parameter, range and accuracy of each measurement standard and the acceptable tolerance of the instrument
characteristic being calibrated, comparison of the uncertainty values should reflect a minimum 4:1 test
accuracy ratio. Such comparisons should always be in terms of measurement units( e.g., volts, ohms, etc.)
and not percentages or parts per million (ppm) unless percentages and ppm are the measurement units (e.g.,
voltage divider linearity). The reason for using units of measurement versus percentages is demonstrated in
the following example. If a ±1% of full scale 0-15 psig working pressure gage were to be calibrated against
±0.1% of full scale 0-100 psig precision pressure gage, it would appear that there is a 10:1 test accuracy ratio
if only percentage uncertainties were compared. However, since the percentage uncertainties for both test and
precision gages apply to the full-scale values, in terms of measurement units the calibration tolerance of the
test gage is ±0.15 psig (±1% of 15 psig) and the measurement uncertainty of
the-precision pressure gage is ±0.1 psig (±0.1% of 100 psig). The true test accuracy ratio is then 1.5:1 rather
than 10:1 and the precision gage does not meet the minimum test accuracy ratio requirement.”

Appendix B
MIL-HDBK-52B (1989): Continued.

“When a minimum 4:1 test accuracy ratio is not achieved or, stated differently, when the collective uncertainty
of the measurement standards exceeds 25% of the acceptable tolerance of the characteristic being calibrated,
the contractor should document such deviations and demonstrate how the adequacy of the calibration is not
degraded. (The calibration would be degraded if there were and adverse impact to the material or services
being delivered to the Government.)
Using an example of a measurement standard calibrating M&TE, the following are approaches that maybe
taken by the contractor when a minimum 4:1 test accuracy ratio is not achieved:
a. The M&TE being calibrated could be derated (test tolerances increased). This option is appropriate when all
applications of the instrument are known and the derated tolerances are consistent with the known usage
and/or contractual requirements, The M&TE should be labeled as to the specific
derating, In effect, this approach may increase the test accuracy ratio to 4:1 or greater for the specific
calibration.
b. The M&TE test tolerances could be reduced to compensate for the uncertainty of the measurement
standards [i.e. guardbanding]. While this would result in an increase of erroneous out-of-tolerance findings for
the M&TE during the calibration process, it should not have an adverse impact on the material to be delivered
to the Government. This approach should be used sparingly since the increased out-of-tolerance results
during the calibration may result in excessive repairs and recalibrations.
The contractor's calibration system could also provide for instances where test accuracy ratio is not a
significant factor. This could arise when the value and uncertainty of the known measurement standard is
essentially transferred to the M&TE unknown measurement standard through a relatively simple
intercomparison process. Other typical but more complex methods for obviating the test accuracy ratio
requirement are through Measurement Assurance Programs (MAPS) and use of statistical process control
(SPC) or statistical quality control in the measurement system.
If the contractor uses any of these or similar methods, the Government representative should assure that the
deviation from the test accuracy ratio requirement is documented, that the rationale and methodology
supporting the deviation are adequate and the adequacy of the calibration is not degraded. If there are
questions concerning the acceptability of the proposed deviation, the Government representative should
request assistance from the Metrology Engineering Center of the cognizant DOE) agency.
To be acceptable for use in the contractor's calibration system, measurement standards must also be
traceable as defined in paragraph 3.5 of the MIL-STD, and supported by an appropriate certificate or report of
calibration. Paragraph 5.8 of this handbook provides further details concerning traceability
requirements for measurement standards.
In addition to traceability and accuracy requirements, the Government representative should confirm that the
contractor reviews calibration history of the measurement standards to ensure that appropriate reliability in
terms of calibration interval is maintained. (See paragraph 5.4 of the MIL-STD.)”

Appendix B
IEEE STD-498-1990 (1990): [115]
“1.2 Applicability. Selection of measuring and test equipment shall be controlled to assure that such items are
of the proper type, range, accuracy, resolution, and tolerance to accomplish the function of determining
conformance to specified requirements.
5.1 Adequacy of Reference Standards: Reference standards used for calibrating measuring and test
equipment shall have calibrated ranges, precisions and accuracies such that the measuring and test
equipment and ultimately the facility equipment can be and maintained within the required tolerances.
In general, the inaccuracy of the reference standard shall contribute no more than one fourth of the allowable
measuring and test tolerance. That is, reference standards utilized to calibrate measuring and test equipment
shall have a minimum accuracy four times greater than the measuring and test equipment being calibrated.
The accuracy requirement is satisfied with M&TE is calibrated by a reference standard with a minimum 4:1
higher accuracy ratio, and when that reference standard is traceable through a series of calibrations, using
reference standards also of 4:1 greater accuracy, back to the National Institute of Standards and Technology
(NIST) or other appropriate governing laboratory standard. This is depicted in Fig 1.
If it is impractical to maintain these accuracy ratios, the rational for deviating from these requirements shall be
justified, documented, and authorized by responsible management”.
ISO 10012-1 (1992): [20]

“The error attributable to calibration should be as small as possible. In most areas of measurement, it should
be no more than one third and preferably one tenth of the permissible error of the confirmed equipment when
in use”.
MIL-PRF-38793A (1993): [178]
“3.2.3.2 A measurement system shall consist of all recommended calibration equipment of combinations
thereof and have a uncertainty equal to or better than one-fourth of the uncertainty of the test instrument for
each parameter tested, i.e. a test accuracy ratio (TAR) (see 6.4.5) equal to or better than four to one.
Measurement systems of better TAR may be used for reasons of equipment availability, reduced complexity,
or reduced calibration time. If a TAR equal to or better than four-to-one is not feasible because of state-of-the-
art or other technical considerations, the best TAR available shall be used. In such cases, the actual TAR
shall be stated in the procedure. Where several items of equipment are applicable, preference shall be given
to the equipment affording the simplest, quickest, and most efficient test method.
6.4.5 Test Accuracy Ratio: The maximum permitted error of the unit to be measured or calibrated, divided by
the maximum known error of the measuring or generating device used to perform the measurement. For
example, if it is required that a system or equipment output parameter be accurate to 8 % (maximum permitted
error) and the known accuracy (maximum known error) of the measuring device used to measure the output
parameter is 2 %, then the TAR is 4.”
DOE-STD-1054-93 (1993): [179]
“3.4.4.1 M&TE should be calibrated using reference standards (secondary or working) whose calibration has a
known valid relationship to nationally recognized standards or accepted values of natural physical constants…
The reference standard used should have an accuracy at least four times greater than the device under test.
If this accuracy ratio cannot be met, analysis of the error should be estimated to provide a valid uncertainty of
the calibration process.”

Appendix B
DOE 4330.4B (1994): [180]

“12.1 … The M&TE selected for use should have the precision necessary to ensure that facility
instrumentation and equipment will operate within design accuracy requirements and be durable enough for
their intended application.
12.3.2 (b) Calibration Procedures. …The ‘ratio of accuracy’ of the standard to the M&TE being calibrated
should be as high as reasonably achievable and consistent with national standards. These procedures
should… contain the following: …accuracy of calibration standards to be used; calibration standards to be
used…”
Calibration Standards:
NASA KHB 5330.9A (1994): [181]
“2.10 (i) Select calibration standards to attain a minimum calibration accuracy ratio of 4:1. When state of the
art limitations prevent attainment of the minimum accuracy ratio, the measurement uncertainty must be
reported as part of the calibration data record. Users of equipment calibrated to less than 4:1 accuracy ratios
must account for the reported instrument uncertainty when establishing end-item test pass/fail criteria.”
ANSI/NCSL Z540.1-1994 (1994): [35]

“10.2 (b) The laboratory shall ensure that the calibration uncertainties are sufficiently small so that the
adequacy of the measurement is not affected. Well defined and documented measurement assurance
techniques or uncertainty analysis may be used to verify the adequacy of a measurement process.
If such techniques or analyses are not used, then the collective uncertainty of the measurement standards
shall not exceed 25% of the acceptable tolerance (e.g. manufacturer’s specification) for each characteristic of
the measuring and test equipment being calibrated”.
NCSL Z540.1 Handbook (1995): [181]

“10.2(b) The standards selected and their associated uncertainties or tolerances should be adequate for the
calibration/verification being performed.
Example(s):
a) An appropriate method or measurement technique is one that meets the needs of the
calibration/verification being performed. (You don’t need to use a gage block to calibrate a yardstick
and you can’t use a yardstick to calibrate a gage block).
b) … When using the standards prescribed by the procedure, it is recommended that the uncertainties
and related Test Accuracy Ratio (TAR) be reviewed.
c) …As a default alternative to doing an uncertainty analysis, a laboratory may rely on a Test Accuracy
Ratio (TAR) of 4:1. A TAR of 4:1 means that the tolerance of the parameter (specification) being
tested is equal to or greater than four times the combination of the uncertainties of all the
measurement standards employed in the test. If it is determined that the TAR is less than 4:1, then
one of the following method may be used: uncertainty analysis as described above, guard-banding,
widening the specification, or another appropriate method”.

Appendix B
DOE RW-0333P Rev.5 (1995): [182]
“12.2.1 (B) Calibration standards shall have a greater accuracy than the required accuracy of the measuring
and test equipment being calibrated.
1. If calibration standards with greater accuracy than required of the measuring and test equipment being
calibrated do not exists or are unavailable, calibration standards with accuracy equal to the required calibration
accuracy may be used if they can be shown to be adequate to the requirements.
2. The basis for the calibration acceptance shall be documented and authorized by responsible management.
The level of management authorized to perform this function shall be identified”.
MIL-HDBK-1839 (1996): [183]
“5.3.3 Test accuracy ratio (TAR): Unless otherwise specified, the recommended TMDE shall be capable of
measuring or generating to a higher accuracy than the measurement parameters being supported. A
minimum TAR of 4 to 1 is required. The actual TAR shall be documented.
a. If a TAR of 4 to 1 cannot be achieved, the contractor shall analyze the measurement requirements
and justify the lesser TAR.
b. A minimum TAR of 4 to 1 is required when an actual test is in conducted to characterize performance
of operational equipment or to caliber other TMDE.
c. A TAR of 4 to 1 is not required when the TMDE only provides input stimuli which is not used to
characterize performance of the operational equipment of other TMDE. In this case, the TAR does
not need to be greater that 1 to 1”.

Appendix B
MIL-HDBK-1839A (2000): [184]
“3.5 Test Uncertainty Ratio: The total uncertainty of the unit to be measured or calibrated divided by the total
uncertainty of the measuring or generating device used to perform the measurement. For example, if it is
required that a system or equipment output parameter’s uncertainty is 8% and the uncertainty of the
measuring device used to measure the output parameter is 2%, then the TUR is 8 to 2 or 4 to 1.
Interpretive guidance: The desired TUR is 4 to 1. TUR is the calculated result of dividing the uncertainty or
tolerance of a measurement/input requirement by the uncertainty/tolerance of the equipment satisfying the
requirement. TURs are generally calculated for each operational system test requirement that is supported by
SE [support equipment] and for SE supported (calibrated) by SE. The maximum permitted uncertainty should
reflect the actual use requirement of the unit being measured/calibrated for the intended use. Tolerances used
in TUR calculations will be in the same measurement units. Uncertainties expressed in logarithmic units such
as decibels should be considered. Decibels should first be converted to linear units before computing TUR.
For example, suppose the system uncertainty and the measurement uncertainty is 4db and 1db respectively.
The db ratio may be 4 to 1 but the linear ratio may be only 2 to 1 or a 3db gain is generally double in value
when dealing with power measurements. Caution should be used when uncertainties are stated as
percentages because they often are not related (i.e. percentage of full scale versus percentage of reading).
Tolerances expressed as ‘minimum’, ‘maximum’, ‘less than’, ‘more than’, ‘greater than’ and ‘less than’, do not
allow TURs to be calculated, thus using them is not desired. However, when no other option exists, the design
activity should specify the recommended uncertainty of the supporting equipment (see 5.4.3).
5.4.3 Test Uncertainty Ratio: The recommended TMDE shall be capable of measuring or generating to a
higher accuracy than the measurement parameters being supported. Unless otherwise specified, a minimum
TUR of 4 to 1 is desired. The actual TUR shall be documented.
Interpretive guidance: TUR is a recognized mechanism for establishing the criteria between equipment listed
in proximity categories in the [CMRS Calibration and Measurement Requirements Summary]. Actually, use of
the TUR is an alternative to performing a more difficult uncertainty analysis as described in ANSI/NCSL Z540-
2-1997. The TUR does not relate directly to uncertainty analysis. However, it can be linked to the probability of
making erroneous test decisions. Another important point, design activities often select SE that is more
accurate than required for the application due to availability, future planning, cost considerations, etc. That is
perfectly acceptable but in the application, as defined by the CMRS, the specifications should be derated to
the 4 to 1 TUR so as not to create unnecessarily accurate and costly support requirements for the SE.
Creation of unnecessarily over/under specified problems are minimized when the design activity emphasizes
concurrent engineering processes where the logistics support planning and design engineering are time
related and maintain a close liaison. The importance of not over specifying a piece of support equipment
cannot be over emphasized as substantial unnecessary costs can be incurred by designs using overly
stringent specifications.
a. If a TUR of 4 to 1, or the specified TUR, cannot be achieved, the design activity shall analyze the
measurement requirements and provide documented justification for the lesser TUR.
Interpretive guidance: It is highly recommended that if the CMRS preparing activity contact their designated
service CMRS review agent to discuss any shortcomings in meeting specified TUR prior to submittal.
b. A TUR of 4 to 1 is not required when the TMDE only provides input stimuli which are not used to
characterize performance of the operational equipment or other TMDE. In this case, a minimum TUR of 1 to 1
is acceptable.
Interpretive guidance: A TUR of 1 to 1 is acceptable between the system and an item of support equipment
only when the system requires an input, stimulus, or applied value for establishing test conditions and the
system itself does not measure the applied parameters or characterize its own performance. Examples of
acceptable 1 to 1 TURs are input/supplied torque applications, electrical stimuli signals, and pressure inputs. A
minimum TUR of 1 to 1 only applies between the system and the SE. It does not apply between SE and
calibration equipment. A 4 to 1 TUR is desired when the system measures an applied/input parameter and the
results determine pass/fail status or the systems performance is characterized.”

Appendix B
ASME B89.7.3.1-2001 (2001): [53]

“4.1 Simple Acceptance and Rejection Using an N:1 Decision Rule.
This is the most common form of acceptance and rejection used in industry and is the descendant of MIL-STD
45662A. Simple acceptance means that product conformance is verified if the measurement result lies in the
specification zone and rejection is verified otherwise (see Fig. 2), provided that the magnitude of the
measurement uncertainty interval is no larger than the fraction 1/N of the specification zone.
In recent years, as tolerances have become increasingly tighter, the well-known ten-to-one ratio has
transitioned to a more commonly used ratio of four-to-one (see MILSTD 45662A) or even three-to-one (see
International Standard 10012-1). A 4:1 decision rule means the uncertainty interval associated with the
measurement result should be no larger than one-fourth of the allowable product variation, which requires the
expanded uncertainty, U, to be no larger than one-eighth of the specification zone. Once the uncertainty
requirement is satisfied, then the product is accepted if the measurement result lies within the specification
zone and rejected otherwise”.
ISO 17025:2005 (2005): [27]

1st edition published 1999
“5.10.3.1 Test Reports should include …where relevant, a statement of compliance/non-compliance with
requirements or specifications. …uncertainty is needed in test reports when…it affects compliance to a
specification limit.
5.10.4.2 When statements of compliance are made, the uncertainty of measurement shall be taken into
account.”
NASA NSTS 5300.4(1D-2) w/ Change No. 9 (2006): [185]

Note: Hyperlink is to older version (Chg No. 2 - 2000)
1D507
“4. Article or Material Measurement Processes. The expanded uncertainty in any article or material
measurement process shall not exceed 10 percent of the tolerance of the article or material characteristic
being measured…
5. Calibration Measurement Process. ANSI/NCSL Z540.1-1994 (R2002), ‘Calibration Laboratories and

Measuring and Test Equipment – General Requirements’ shall be the standard for calibration processes. Per
this standard, uncertainty analysis may be used to verify the adequacy of a calibration Measurement
processes… If this technique is not used, the expanded uncertainty in any calibration measurement process
shall not exceed 25 percent of the tolerance of the parameter being measured”.

Appendix B
DOE RW-0333P Rev.21 (2009): [186]
“12.2.1 (B) Calibration standards shall have a greater accuracy than the required accuracy of the measuring
and test equipment being calibrated.
1. If calibration standards with greater accuracy than required of the measuring and test equipment being
calibrated do not exists or are unavailable, calibration standards with accuracy equal to the required calibration
accuracy may be used if they can be shown to be adequate to the requirements.
2. The basis for the calibration acceptance shall be documented and authorized by responsible management.
The level of management authorized to perform this function shall be identified.
(C) Calibration standards used for the calibration of M&TE shall have an accuracy of at least four times the
required accuracy of the equipment being calibrated or, when this is not possible, shall have an accuracy that
ensures the equipment being calibrated will be within required tolerance.”
MIL-STD-1839D (2010): [187]
“3.1.13 Test Accuracy Ratio. A simple comparison between the accuracy of the Unit Under Test (UUT) and the
accuracy of the calibration standard. However, this ratio does not consider other potential sources of error in
the calibration process. For example, if it is required that an equipment output parameter’s accuracy is ±8%
and the accuracy of the single measuring device used to measure the output parameter is ±2%, the TAR is 8
to 2, or 4 to 1.
3.1.16 Test Uncertainty Ratio. The comparison between the accuracy of the Unit Under Test and the
estimated calibration uncertainty is known as a Test Uncertainty Ratio (TUR). This ratio is more reliable
because is accounts for possible sources of error in the calibration process that the TAR does not. This value
also takes into account issues such as temperature, humidity, Type A (statistical analysis including actual and
random measurement results), and Type B (anything not a Type A) uncertainties.
5.4.3 Test Uncertainty Ratio (TUR). The recommended TMDE shall be capable of measuring or generating to
a higher accuracy than the measurement parameters being supported. Unless otherwise specified, a minimum
TUR of 4 to 1 is desired. The actual TUR shall be documented. If a TUR of 4 to 1, or the specified TUR,
cannot be achieved, the design activity shall analyze the measurement requirements and provided
documented justification for the lesser TUR. A TUR of 4 to 1 is not required when the TMDE only provides
input stimuli which are not used to characterize performance of the operational equipment of other TMDE. In
this case, a minimum TUR of 1 to 1 is acceptable.”

Appendix B
Air Force T.O. 00-20-14 (2011): [188]
“1.4.28 TAR/TUR. Test Accuracy Ratio/Test Uncertainty Ratio. The ratio of the uncertainty (accuracy) of the TI
[Test Instrument or Unit-Under-Test] to the uncertainty (accuracy) of the standard (for example 4:1).
TAR and TUR are equivalent in the Air Force Metrology and Calibration Program, as the term Accuracy has
been used for the (more correct) term Uncertainty in Air Force Calibration procedures.
3.1.7 AFMETCAL shall ensure: …The calibration uncertainties are sufficiently small so that the adequacy of
the measurement is not affected. Well-defined and documented measurement assurance techniques or
uncertainty analyses may be used to verify the adequacy of a measurement process. If such techniques or
analyses are not used, then the collective uncertainty of the measurement standards shall not exceed 25
percent (4:1) of the acceptable tolerance for each characteristic of the measuring and test equipment being
calibrated or verified”
L-A-B LABAG-502 Accreditation Guidance. (2011): [189]

“3.9. TAR – Test Accuracy Ratio, a ratio of the collective uncertainties of the equipment used to the accuracy
specification(s) of the unit under test.
3.9.1. Test Accuracy Ratio (incorrect), a ratio of the collective accuracies of the equipment used to the
accuracy specification(s) of the unit under test. (This incorrect definition was commonly used due to
the use of the term “Accuracy Ratio”)
3.10. TUR – Test Uncertainty Ratio, a ratio of the collective uncertainties of the equipment used to the
measurement uncertainties of the unit under test.
A technical review of the calibration certificate’s content will evaluate the following;
* the measurement uncertainty is sufficiently low for your application,
* the data’s impact on the laboratory’s work (data beyond prescribed limits such as manufacturer’s
specifications or calibration tolerances, requires an investigation to determine if there is non-conforming work).
* ISO/IEC 17025:2005 Section 5.10.4.1c states that calibration certificates shall include “the uncertainty of
measurement and/or a statement of compliance with an identified metrological specification or clause thereof”.
* a calibration certificate may make a statement of compliance and contain a statement referring to a TAR or
TUR of 4:1 or greater. Neither a TAR nor a TUR is acceptable as a measurement uncertainty for a unit under
test.
NOTE: If a laboratory can calculate a proper TUR, then a measurement uncertainty for the unit under test has
been calculated and should be available upon request”.
A2LA P102 Policy on Measurement Traceability (2015): [190]
“It is often the case that a calibration certificate will contain the statement “in tolerance”, or words to that effect,
along with a statement to the effect that the measurement uncertainty does not exceed a certain fraction of the
tolerance. Such fractions are often called “test uncertainty ratios”, TURs for short. Uncertainty statements
phrased in terms of TURs are implicit statements of the uncertainty: knowing the tolerance ratio allows one to
determine the largest possible value of the measurement uncertainty. Implicit statements of uncertainty are
acceptable on accredited calibration certificates as long as the measurement uncertainty and the
measurement results are also provided”.

Appendix B
A2LA R205 Specific Requirements: Calibration Laboratory Accreditation Program (2015): [191]
“6.5 Statements of Compliance

Laboratories are permitted to issue certificates with a statement of compliance (e.g., conformance to a
specification) relating to the metrological aspects of specifications. In such cases the laboratory shall ensure
that:
1) The specification is a national or international standard or one that has been agreed to or defined by
the customer;
2) The measurement needed to determine conformance are within the accredited scope of the
laboratory;
3) When parameters are verified to be within specified tolerance, the associated uncertainty of the
measurement result is properly taken into account with respect to the tolerance by a documented
procedure or policy implemented by the laboratory that defines the decision rules used by the
laboratory for declaring in or out of tolerance conditions;
4) The laboratory shall ensure the decision rule used meets the needs of the customer;
5) When parameters are verified to be within specified tolerance, the calibration laboratory shall
determine the uncertainty and take that uncertainty into account when issuing the statement of
compliance. In addition, no claim of compliance shall be made when the measurement uncertainty is
larger than the tolerance being evaluated.
An exception can be made only in cases where the laboratory indicates in the contract with the client
that the calibration results will be reported without factoring in the effect of uncertainty on the
assessment of compliance, and the client agrees to the contract.
In this case, the uncertainty can be excluded when making that statement of compliance on the
calibration certificate. In effect, both parties share the risk that the results may or may not meet the
specification since the uncertainty was not included when the results were determined. In these cases,
the measurement uncertainty shall still be determined and shall be reported on the calibration
certificate.
6) In accordance with ILAC P14:01/2013 ILAC Policy for Uncertainty in Calibration section 6.1, the
calibration laboratory shall retain documentary evidence of the measured quantity value and the
uncertainty of measurement as specified in ISO/IEC 17025 clauses 5.10.4.2 and 4.13, and shall
provide such evidence upon request.
7) The certificate relates only to metrological quantities and states which clauses of the specification are
verified to have been met”.
IAS AC204 CL/009 (2016): [192]

“5.4 As applicable, assessments may include review of the 4:1 Test Uncertainty Ratio (TUR) requirement for
each technical demonstration”.
ANAB (previously ACLASS) CL 2043 – Z540.3 Checklist (2016): [193]

“In cases where it is not practicable to estimate PFA, does the laboratory ensure that the Test Uncertainty
Ratio (TUR) is equal to or greater than 4:1?”

Appendix B
ANAB (previously ACLASS) MA 2002: ISO/IEC 17025 Accreditation Requirements for Calibration Laboratories
(2016): [194]
“In some instances, a calibration certificate may contain the statement “in tolerance,” along with a statement
that the measurement uncertainty does not exceed a certain fraction of the tolerance. These fractions are
often called test accuracy ratios (TARs) or test uncertainty ratios (TURs). Uncertainty statements phrased in
terms of TARs or TURs are not acceptable for demonstrating measurement traceability or as an alternative to
calculated MU [measurement uncertainty].”
IAS CL/014 Policy Guide on Calibration, Traceability, and Measurement Uncertainty for Calibration
Laboratories. (2016): [195]
“Calibration certificates and/or reports held by or issued by IAS accredited calibration laboratories must meet
the requirements of ANSI/ISO/IEC Standard 17025:2005, Clause 5.10 (sub-clauses 5.10.2 and 5.10.4 are
required, other sub-clauses as appropriate).
Calibration certificates must include appropriate statements of uncertainty… All calibration certificates issued
under the laboratory’s accreditation, whether or not indicating compliance to the 4:1 rule or other accepted
metrological specification, must include the IAS logo and the uncertainty of the calibration...
The American National Standard for calibration, ANSI/NCSL Z540-1-1994, known as Z540-1, has been
formally retired effective in July 2007, although the standard continues to be used. The standard was replaced
by ANSI/NCSL Z540.3-2006, known as Z540.3. There are significant differences between the two documents.
The two most obvious differences are the alignment of Z540.3 to be consistent with the requirements of
ANSI/ISO/IEC ISO 17025:2005 in Section 5.3…
Z540-1 allowed the use of a Test Accuracy Ratio (TAR), with a minimum ratio of 4:1. Z540.3 allows the use of
a Test Uncertainty Ratio (TUR), again with a minimum of 4:1. This provision is known as the 4:1 Rule. The
laboratory may claim compliance to the 4:1 rule, providing the laboratory maintains evidence, including
information regarding calibration of the laboratory’s measuring equipment and measurement uncertainty
calculations, that it can consistently maintain the 4:1 ratio for any calibration where compliance with the 4:1
rule is stated. The evidence must be available for review by IAS upon request. If the 4:1 cannot be
maintained, the laboratory must report either specific uncertainties or the actual TUR for any affected
readings, as appropriate for the specific type of calibration…
It is understood that some laboratory may have a need to be verified as compliant to the requirements of
ANSI/NCSL Z540.3-2006, either solely for Section 5.3, or for the entire Standard. This will require additional
assessment time and the laboratory must provide additional documentation as required by the Standard”.

Appendix C

Appendix C

Appendix C

Appendix C

Appendix D
FDA Good Guidance Practices (excerpts from 21 CFR §10.115) [79]
“(a) What are good guidance practices?

Good guidance practices (GGP's) are FDA's policies and procedures for developing, issuing, and
using guidance documents.
(b) What is a guidance document?

(1) Guidance documents are documents prepared for FDA staff, applicants/sponsors, and the
public that describe the agency's interpretation of or policy on a regulatory issue.
(2) Guidance documents include, but are not limited to, documents that relate to: The design,
production, labeling, promotion, manufacturing, and testing of regulated products; the processing,
content, and evaluation or approval of submissions; and inspection and enforcement policies...
(d) Are you or FDA required to follow a guidance document?

(1) No. Guidance documents do not establish legally enforceable rights or responsibilities. They
do not legally bind the public or FDA...
(2) You may choose to use an approach other than the one set forth in a guidance document...
(e) Can FDA use means other than a guidance document to communicate new agency policy or a
new regulatory approach to a broad public audience?
The agency may not use documents or other means of communication that are excluded from the
definition of guidance document to informally communicate new or different regulatory expectations
to a broad public audience for the first time. These GGP's must be followed whenever regulatory
expectations that are not readily apparent from the statute or regulations are first communicated to a
broad public audience.
(f) How can you participate in the development and issuance of guidance documents?
(1) You can provide input on guidance documents that FDA is developing under the procedures
described in paragraph (g) of this section.
(2) You can suggest areas for guidance document development. Your suggestions should address
why a guidance document is necessary.
(3) You can submit drafts of proposed guidance documents for FDA to consider. When you do so,
you should mark the document ‘Guidance Document Submission’ and submit it to Division of
Dockets Management (HFA-305), 5630 Fishers Lane, rm. 1061, Rockville, MD 20852...”

Appendix E
(S. Schmitt – Silver Sheet Article; June 2010)
Pay Close Attention To Equipment Calibration Certificates, FDA Says

By Shawn M. Schmitt
[Editor's note: This article originally ran in the medical device industry publication The Silver Sheet in June
2010. The Silver Sheet was incorporated into The Gray Sheet in April 2015. Please visit TheGraySheet.com for
in-depth regulatory and quality news, analysis, podcasts and features.]
Too many device firms aren’t double-checking the quality of finished products after determining that
manufacturing equipment was out of calibration, an FDA official says.
“A large majority of manufacturers contract out their calibration activities. Because of this, what we typically
see is that companies are blindly accepting calibration certificates from test houses,” FDA/GMP expert Kim
Trautman told “The Silver Sheet.”
Test houses are third parties used by firms to ensure that equipment used in the manufacture or the inspection
and testing of their devices is properly calibrated and maintained.
Calibration certificates are sent to firms describing the condition of the equipment as it was received by the test
house, as well as its condition when it was returned to the manufacturer.
“I have seen situations where a test house has sent equipment back to a manufacturer and the certificate said,
‘Yes, this piece of equipment was way out of calibration. It was XYZ, and we have now recalculated it,’”
Trautman said.
“So my question to the manufacturer is, what did you do with that information? Because you now know that
there is finished product that was sold during this period of time when the instrument [used to manufacture it]
was way out of calibration,” she said.
At that point, a firm should investigate to ensure that the equipment in question did not have an adverse effect
on devices already manufactured, Trautman said.
In addition to equipment condition, test-house certificates also specify the calibration method that was used.
Those test methods typically tie to national and international standards from the International Organization for
Standardization (ISO), the American National Standards Institute (ANSI), the American Association for
Laboratory Accreditation (A2LA), or another accrediting body.
FDA requires equipment testing and calibration under Quality System Regulation Sec. 820.72, “Inspection,
Measuring and Test Equipment,” as well as Sec. 820.70(g), “Production and Process Controls; Equipment.”
Manufacturing equipment, as well as instruments used for inspection and testing, can range from a simple
ruler, to widely used voltage meters, to complex custom-built test equipment that contains software – all of
which “require accuracy and precision,” Trautman said.
“All equipment has to be appropriate for its purpose, and it has to be maintained, adjusted and cleaned,” she
said.
Companies may calibrate equipment themselves; however, most firms use a combination of in- and out-of-
house testing.

Appendix E
Firms also may have test-house representatives calibrate equipment at their manufacturing facilities if the
equipment is too large to move or if the firms don’t have the expertise to conduct calibration activities on their
own.
Tracie Capozzio, director of quality assurance & regulatory compliance for NeuroWave Systems in Cleveland
Heights, Ohio, says regulatory agencies, including FDA, are checking more thoroughly to see if manufacturers
are examining whether finished devices may have been tainted by out-of-calibration equipment.
During an agency inspection, “they’re usually looking for pieces of equipment that have come back out of
calibration, and what you’ve done when they’ve come back out of calibration – whether or not you have
evaluated the equipment that was used and how you went about identifying which product was processed with
that equipment, and what the impact was,” Capozzio said.
NeuroWave’s “procedures are written so that any product that was made during the time period between the
last calibration, which was good, and the current calibration – if it was received at the calibration site out of
calibration – is reviewed for any adverse impacts,” she said.
Regulatory bodies, including FDA, are “focused a bit more now on inspection and test equipment than in the
past, and you’re starting to see more regulatory activity related to equipment,” Capozzio said.
FDA: Many Firms Don’t Look At Certificates
Another problem FDA sees is that many companies are not adequately reviewing test-house certificates, and
some do not look at the certificates at all.
“Ninety percent of the time when there are problems [with manufacturing equipment] it’s because someone
hasn’t even paid attention to the certificates that the calibration test house has given them,” Trautman said.
“The manufacturers that get into these problems are those that just take the certificates, file them and don’t
really look at them and utilize the information that the test house is giving them,” she said.
“Kim Trautman is probably correct that 90 percent of companies file away the certificates, assuming that
everything is good, but you can’t assume that,” NeuroWave’s Capozzio told “The Silver Sheet.”
“It’s very easy to get a certificate from a certification house and just assume that they’re the experts, and when
they give you a certificate, that means the equipment is good,” she said.
“That assumption isn’t 100-percent false, though, because when the test house gives the equipment back and
they give you a certificate, the equipment should be good at that point,” Capozzio said. “However, you have to
be cognizant enough to know what the equipment was like when the test house received it.”
NeuroWave recently received a CE mark for its first product, the NeuroSENSE Monitor, which monitors
patient brain activity levels to help guide anesthesia and sedation in operating rooms and intensive care units.
Although the device is not yet on the market, NeuroWave has produced valuation units. It uses several pieces
of manufacturing equipment that it must calibrate, including signal generators, high-voltage probes, digital
calipers, oscilloscopes and multimeters.
Most of the firm’s equipment is sent to test houses for calibration, although some, such as electrostatic
discharge equipment and an in-house built biosignal playback unit, is tested by NeuroWave itself.

Appendix E
The manufacturer’s biosignal playback unit “is not something we can send out for calibration because there is
no standard known to a test house that it could be calibrated against,” Capozzio said.
“In-house built equipment often requires special or unique maintenance and calibration instructions that can
only be developed and implemented by the manufacturer of the equipment,” she said.
At NeuroWave, “the test-house certificates that come back to us with equipment are reviewed, and after
review, the person who is in charge of the calibration and maintenance program is responsible for looking at
the different specifications that are critical for our product,” she said.
Next, the certificates are initialed, indicating that they were reviewed for accuracy.
“That is definitely one of the areas that regulatory agencies such as FDA look at during an inspection,”
Capozzio said. “So we have it built into our system to do that every time the certificates come back in.”
Improper Review Of Certificates Means

Test-House Mistakes May Be Missed
The key risk in not reviewing calibration certificates, of course, is that the manufacturer won’t notice that a test
house has made a mistake – and that devices that have already gone out the door are out of spec.
“You may find that you’re getting nonconformances kicked back, and your failure investigation shows that
somehow this testing is either not being done right or not being done with the right equipment,” FDA’s
Trautman said.
“The investigation might lead you back to a piece of test equipment, and then possibly it might lead you back
to a calibration test house not doing something right,” she said.
Test-house errors indeed happen, admits Pete Sweetnam, VP of test house RS Calibration Services in
Pleasanton, Calif.
“Mistakes can take place, especially when you’re moving equipment back and forth,” he told “The Silver
Sheet.” “We try to do the majority of our calibrations through a pick-up and drop-off system to avoid UPS and
FedEx if we can. Even during the physical handling and movement of equipment, things can get damaged or
become out of adjustment again.
“There are certain pieces of equipment where there is a high tendency for that to happen, so we recommend
that it not be moved, and that’s why we’ll go onsite and do the work,” added Sweetnam, whose test house is
accredited to ISO, ANSI, A2LA and other organizations.
Although FDA ideally wants manufacturers to adequately review calibration certificates, Trautman
acknowledges that calibration errors may still slip through.
“You can never prevent everything bad from happening,” she said.
However, having a robust failure investigation program in place may mitigate any problems brought on by
equipment improperly calibrated by a test house and not immediately detected when the equipment is accepted
at the company, Trautman said.
“In a circumstance where there is a product failure and a manufacturer can get down to a root cause [and
determine that equipment was calibrated incorrectly], we’re not going to ding the firm during an investigation

Appendix E
for a calibration violation or a purchasing control violation, provided they have a robust failure system in place
that they were following,” she said.
“What’s more important to FDA is, what are you going to do about it now?” Trautman continued. “What are
you going to do to fix the issue? Are you going to switch test houses? Are you going to increase the frequency
of calibration? But we’re not going to say during an investigation, ‘Oh, this test house is terrible.’”
CAPA Not Always Necessary In

Cases Of Improper Calibration
If a manufacturer finds that a piece of equipment was improperly calibrated by a test house, it should not
automatically open a corrective and preventive action (CAPA), Trautman says.
“CAPA is an escalating activity, but clearly there needs to be some elevation to at least explore or evaluate a
correction,” she said. “Are you going to go back to your test house? Are you going to talk to this supplier
about what happened?”
Calibration test houses must be controlled under QSR Sec. 820.50, “Purchasing Controls,” because they supply
a service.
“If it’s a one-off and you can handle it through a nonconformance investigation, then that is OK,” Trautman
said. “It needs to be handled, but how the manufacturer handles it is important, whether they do it via a
nonconforming report or a particular correction.”
However, “if it is elevated to a CAPA, then it says to me that something a little bit bigger systemically needs to
happen with feedback to this particular supplier.”
When NeuroWave notices trouble with poorly calibrated equipment, it first opens a nonconformance report.
“It would depend on how broad the problem was and how many pieces of equipment may be affected,”
Capozzio said.
“If it’s isolated to one piece of equipment, we may not move it into a CAPA because our actions through our
nonconformance process are very similar to what we do through CAPA,” she explained. “Unless it’s systemic
in nature, we may not advance it to a CAPA unless there is some reason to do so.”
Test-House Errors Are Manufacturer’s Responsibility
Further, Trautman reminds manufacturers that the buck stops with them if a test house makes an error on a
calibration certificate.
“If that certificate doesn’t contain a piece of information that is required by the Quality System Regulation, the
manufacturer can’t use the excuse that ‘This is all the calibration test house people gave me,’” she said.
“It’s the manufacturer’s responsibility to make sure that they have the control, through supplier controls and
contracts, to get the information that is necessary for that finished device manufacturer to meet regulatory
requirements,” Trautman said.
If firms run into recurring problems, they will have to decide whether they should use a different test house.
“If I’m a manufacturer and it’s a problem that the test house continues to give me documentation that doesn’t
meet my requirements, then yes, I’m going to probably look into switching test houses,” Trautman said.

Appendix E
“It depends on the scope of the issue and how willing that test house is to rectify whatever issues I’m having
with them,” she said. “After all, there are other test houses that can do a fair majority of these types of
calibrations.”
How Often Should Equipment Be Calibrated?
There are several factors that help determine how often manufacturing equipment and instruments used for
inspection and testing should be calibrated.
“When companies buy manufacturing equipment, they come with instruction manuals that talk about the
recommended frequency of calibration,” FDA’s Trautman said. “Most manufacturers will start off with that
recommendation unless they have historical data that tells them they can lengthen or shorten” the time between
calibrations.
“Suppose this is the first time that I’m using particular scales for weighing, and the operation manual
recommends that the scales be calibrated every nine months,” she said. “As a manufacturer, I’m going to start
off my calibration of this particular piece of equipment to have it calibrated every nine months.
“Then one of three things can happen,” Trautman continued. “One, that frequency may be perfect. Two,
because I use the scale a lot, I may need to shorten the calibration time to six months. Or three, because I don’t
use the scale all that often, I probably can lengthen the calibration time to one year. But I have to justify those
changes in my calibration procedures.”
According to QSR Sec. 820.72(b), calibration procedures must include “specific directions and limits for
accuracy and precision.”
Further, Sec. 820.72(b)(2) notes that calibration records should include “equipment identification, calibration
dates, the individual performing each calibration and the next calibration date.”
“The obvious question for FDA investigators is, how is the frequency determined?” Trautman said. “Ninety-
nine percent of manufacturers will just pull out an operations manual that came along with the equipment and
say, ‘This is what was suggested.’
“If it’s not the same timeframe as in the procedures, then the investigator might ask why,” she added. “If there
is a documented specification as to why the calibration of a particular piece of equipment can be less frequent
or more frequent, then the manufacturer is probably fine as long as that has been documented.”
Karen St. Onge, director of corporate quality assurance at NxStage Medical in Lawrence, Mass., cautions firms
that at a minimum they should follow the equipment’s manufacturer-recommended calibration intervals.
She said companies should consider what the equipment is used for when determining when it should be
tested.
“You may consider a much shorter calibration interval depending on the level of risk or the intended use of
that piece of equipment,” says St. Onge, whose firm manufactures home hemodialysis devices and other
products.
For example, if a piece of equipment “is being used for acceptance criteria – it’s measuring some criterion that
is being used to accept the product – then you should not have a calibration cycle longer than 12 months, and
in some cases it could be even shorter,” she said.

Appendix E
“You have to assess,” St. Onge said. “If after a year you go in and look at that same piece of equipment and
find out that that attribute is out of calibration, then you have to look at a year’s worth of product that you
accepted and determine whether or not there is any impact [to devices] due to that piece of equipment being
out of calibration.
“The bottom line is that it’s not a one-size-fits-all situation,” she continued. “There is a business risk involved.
You have to look at all equipment individually when determining the appropriate calibration interval.”
St. Onge recommends that device manufacturers determine one month in advance which equipment needs to
be calibrated.
“You need to run things in advance to make sure that it gets calibrated before its due date. So at the beginning
of June we are looking at what is due next month in July,” St. Onge told “The Silver Sheet.” “We are already
starting to pull equipment that is due for testing in July. We’re going through and determining what we need to
do to get it calibrated.”
For equipment that is no longer used, St. Onge suggests that companies conduct and document one last
calibration.
“If the process changes and you are no longer going to use a specific piece of equipment, part of the closure to
put that piece of equipment out to pasture is to do a final calibration,” she said.
When that happens, “you have closed the book on that piece of equipment, knowing that the entire time it was
in use during your process that it was within its calibration,” St. Onge said.
That is “important, because if it were to come back as being out of calibration, you will need to do an
assessment of what impact, if any, that had on how you were using that equipment,” she said.
According to Trautman, “a red flag is raised when a manufacturer has a calibration system that is set up to
calibrate every single thing all the time, once a year, because that’s rather arbitrary.
“Maybe all of their instruments really do have a one-year recommended calibration time,” she added.
“However, I see too many programs that have everything calibrated once a year, and then you sometimes need
to go back and wonder, what did the operation manual for this instrument say? Or, do they even still have the
manual?”
FDA Investigators Check Calibration Stickers
Most calibration citations found on FDA-483 inspection forms come about because manufacturers do not
calibrate equipment when they said they would.
Firms “will have a calibration schedule set up and there will be a sticker on an instrument, and the investigator
will go in and pick up the instrument and they’ll find that it was supposed to be recalibrated last month, but it
wasn’t,” FDA’s Trautman said.
“The sticker will say, ‘Calibration due on X date,’ and the investigator will walk in and find that it is past that
date,” she said. “There is really no excuse for these problems, because this isn’t tough technology. These aren’t
things that are hard to handle.
“This is just about manufacturers being diligent and following through, and keeping track, and keeping on top
of their equipment,” Trautman noted.

Appendix E
Even the tiniest pieces of equipment should be labeled with a calibration sticker, NxStage’s St. Onge said.
“If you have a set of pin gauges, then you may not be able to necessarily label each pin gauge, but they
generally would come in a box, so you would label the box that they’re in with an equipment number,” she
said. “Any equipment that requires calibration generally must bear a calibration sticker.”
St. Onge urges manufacturers to check periodically that calibration stickers on equipment are kept up to date.
“My experience has been that in walking through a manufacturing area or an engineering area where
equipment is in use, most auditors or investigators will look at different equipment that is in use and make note
of their identification numbers,” which are included on calibration stickers, she said.
Identification numbers make it possible for an investigator to more easily find calibration documentation and
records related to a particular piece of equipment.
“They will always look to see if equipment is marked as ‘calibration due,’ and whether it is within its
calibration date,” St. Onge said. “Often they will go back and ask to see some examples of the calibration that
was performed, whether it’s in the form of a certificate from an outside test house or documentation that
something was calibrated in-house.
“They will look at the results and look at the certificate and make sure that the certificate has appropriate
information on it and that the standards that were used are noted and are identified as traceable to national or
international standards,” as required by QSR Sec. 820.72(b)(1), “Calibration Standards,” she added.
NIST is the National Institute of Standards and Technology, an agency of the U.S. Department of Commerce.
NIST is the regulatory overseer of calibration test houses that works to ensure that measuring and test
standards are reliable.
Instruction Manual Storage
Instruction manuals must be kept as part of calibration records.
“Everyone has a manual for a refrigerator, or a manual from a washer and dryer in some drawer or some file
someplace,” FDA’s Trautman said. “Just like all of us at home, manufacturers have that kind of setup for all of
their equipment as well.”
That is how it is performed at Hitachi Chemical Diagnostics, a company that manufactures tests to measure a
patient’s reaction to allergens.
Instruction manuals “are not part of our standard operating procedures, because they would make the SOPs too
big. So they are kept in a separate file,” says Emi Zychlinsky, VP of research and development for the
Mountain View, Calif., firm.
“We basically just have a file where we just go by equipment name, serial number and part number,” she said.
“This is not a big deal for us because we don’t have that much equipment.”
The company has “manufacturing equipment that we use to manufacture our reagents, and then we have
manufacturing equipment we use for bottling and mixing reagents,” Zychlinsky told “The Silver Sheet.”
The firm also has equipment for general manufacturing purposes, such as centrifuges, cold rooms, pipettes and
balances.

Appendix E
After initial purchase of the equipment, Hitachi Chemical rarely looks at the instruction manuals.
“If we have a problem, we usually call the manufacturer,” Zychlinsky said.
NeuroWave handles its instruction manuals in a similar fashion.
“Each piece of equipment has an equipment folder, and the equipment folder is going to have all of the
calibration certificates, all of the maintenance protocols and maintenance test results, and the user manuals or
any instructions that came with it,” Capozzio said.
She notes that equipment folders also contain new equipment forms, which are filled out when equipment is
purchased.
“The purpose for the new equipment form is to identify the equipment and its unique identification number,
and to set the calibration and maintenance frequency, as well as the methods we’ll use for the calibration and
maintenance – whether it will be sent out to a calibration house or whether it’s something that we would do
internally,” Capozzio said.
Unique identification numbers are created by NeuroWave as a way to track its equipment.
Manufacturers also may mine information from instruction manuals and incorporate that information into
calibration documentation or records.
“Maybe a particular equipment-maker is prolific, or the manual has six different languages, and the
manufacturer doesn’t want to keep an inch-thick manual,” Trautman said.
If that is the case, “then manufacturers can extract the important information from that operation manual into
some other documentation to say, ‘The original owner’s manual says this,’ so it’s clear where that information
came from,” she said.
“We have done that at our company,” Zychlinsky said. Most instruction manuals “come in 10 languages now.
So if we need to, we extract from the manual’s recommendations and put them into an SOP. But we don’t do
that for every piece of equipment. It just depends.”
Keep Sharp Eye On Calibration Test Houses – Device Manufacturers Aren’t Their Only
Customers
Calibration test houses, which are widely used by manufacturers, offer convenience when it comes to testing
equipment. However, firms must keep a sharp eye on them just as they would any other supplier.
“Most of the time manufacturers send equipment to a calibration test house or they have a calibration test
house come in. I’m not saying that a firm has to. They don’t,” FDA’s Trautman said.
“There are ways that manufacturers can calibrate their own equipment, but a lot firms, because they have to tie
[calibration] to a national or international standard, send their equipment to test houses that already have all of
those national and international standards set up,” she said.
For example, “let’s consider weights,” Trautman said. “There are weights that tell you that something is
exactly 10 grams. So instead of a manufacturer going out and buying a whole set of calibrated weights to
calibrate particular measuring equipment, they send it to a test house instead.”

Appendix E
Most test houses calibrate equipment for other manufacturing sectors in addition to the medical device
industry.
At RS Calibration Services, about 40 device manufacturers are clients, which accounts for about 30 percent of
the firm’s business. Other industries served by the company include biomedical, pharmaceutical and
micromachining, among others.
“We work in what we call ‘the FDA environment,’” RS Calibration VP Sweetnam said. “A very high
percentage of our customers are being monitored and controlled by the FDA, so our role is to make sure that
all of our customers’ equipment and documentation, as it pertains to calibration, is in compliance with FDA
requirements.”
Another test house, Accutek Testing Laboratory in Cincinnati, derives approximately 55 percent of its business
from device manufacturers.
However, it also services the aerospace and defense sectors, company President John McCoy told “The Silver
Sheet.”
Because they are only one client out of many, it’s important for a device manufacturer to ensure that it receives
proper equipment calibrations and documentation that meet FDA regulatory requirements.
“Calibration test houses do calibrations for a whole wide range of industry sectors, some regulated, some
clearly not,” Trautman said. “We are obviously a regulated sector, and in the Quality System Regulation there
are some very specific requirements for calibration and specific requirements for what a calibration record has
to have.
“There are some very particular requirements that every calibration test house might not put on a certificate
that goes back to the manufacturer,” she added. “So this is where, under purchasing and supplier controls,
firms need to establish quality requirements and documentation requirements up front with the supplier of this
service, since the manufacturer knows the type of information that is required in their calibration records.”
Hitachi Chemical’s Zychlinsky said her firm does some in-house calibration, but much of the work is
outsourced to test houses, including Rainin, Alpha Omega Instruments and USHIO.
All three of those companies not only calibrate instruments, but manufacture equipment as well.
Whether or not Hitachi Chemical sends equipment to test houses for calibration “depends on the capabilities
we have here,” Zychlinsky said. “If we have enough engineering capabilities to do it here, we might choose to
do that, but if not, we go to the outside. We’re a small company, so going outside a lot of times is better.”
When it comes to selecting a test house to place on its approved supplier list, Hitachi Chemical “looks for
certification, and we look to make sure that they comply with the Quality System Regulation and with the
requirements of ISO 13485,” Zychlinsky said.
ISO 13485 is the international quality system standard for medical device manufacturing used in Canada,
Australia and the European Union, among others. Its requirements for device manufacturers are similar – but
not identical – to the QSR.
Researching Your Calibration Test House
FDA recommends that manufacturers research test houses as part of their supplier control activities to ensure
that equipment will be properly calibrated.

Appendix E
“There are criteria and accreditations and so forth for test houses, so firms should ask them these questions:
Are you accredited? What standards do you use?” FDA’s Trautman said.
“That is part of the assessment of the supplier,” she said. After receiving that information, a device
manufacturer “should then be able to say, ‘OK, this test house is certified by ISO or ANSI and is traceable to
NIST, so there is some assurance.’”
Trautman notes, however, that most manufacturers don’t typically audit test houses in the way they would a
component supplier.
“But that’s not to say that they couldn’t audit them,” she said. Manufacturers “absolutely can if they want to.
However, because inspectional test houses have other governing bodies that are looking at them, most firms
rely on national and international accreditations.
“Most manufacturers will use accreditations as their assurances,” Trautman continued. “Firms are going to
want a test house that is at least tied to NIST.”
NeuroWave’s test houses are all accredited through ISO, which gives the company a higher level of confidence
that they have the quality management systems in place to ensure that equipment will be properly calibrated.
“Because they’re ISO certified it doesn’t require an onsite visit, unless our risk evaluation determines
otherwise,” Capozzio said. “But we do send out what we call a ‘request for information,’ which is very similar
to a self-assessment. It goes through and asks them a lot of the general questions we would look for during an
onsite audit.
“However, if we have not done a physical visit to the calibration house, that doesn’t mean we wouldn’t in the
future,” she added. “But at this point we haven’t.”
NxStage’s St. Onge recommends that firms evaluate a test house in the same manner that they would any
supplier that could impact the quality of finished device products. This means that test houses should be
included on a company’s approved supplier list.
“I have physically audited test houses even though they were accredited and had applicable ISO certifications,”
she said.
“However, I don’t audit the manufacturer of a piece of equipment,” St. Onge said. “If I’m going to use the
original manufacturer [of the equipment] to calibrate it, I’m going to assume that they’re going to calibrate it
correctly since they’re the manufacturer.”
Hitachi Chemical also is cautious.
“We don’t go solely by a test house’s accreditation,” Zychlinsky said. “We evaluate and monitor service
suppliers exactly the same way as we do with component suppliers.
“We qualify them first, and then we monitor them on a regular basis,” she noted. “And then we audit them
again as per our procedures.”
*Follow Shawn M. Schmitt @shawnmschmitt
Used with permission. © Informa Business Intelligence, Inc., an Informa company. All rights reserved.

Appendix F
(FDA Medical Device Quality Systems Manual – Chapter 7: withdrawn on 12/12/2013)
Medical Device Quality Systems Manual – A Small Entity Compliance Guide

Chapter 7 – Equipment and Calibration*
First Edition, Dec 1996
Note: Withdrawn by the FDA on 12/12/2013
U.S. Department of Health and Human Services (HHS)

Food and Drug Administration (FDA)
Centers for Devices and Radiological Health (CDRH)
Division of Small Manufacturers Assistance (DSMA)
Office of Health and Industry Programs
Andrew Lowery, Judy Strojny, Joseph Puleo
HHS Publication 97-4179 (1996) Supersedes:

1991: Medical Device Good Manufacturing Practices Manual. Fifth Edition (HHS Publication FDA 91-4179)
U.S. Department of Health and Human Services. Public Health Service. Food and Drug Administration. Center for Devices and Radiological Health. Division of
Small Manufacturers Assistance. Office of Training and Assistance. Andrew Lowery, Joseph Puleo. ISBN 0-16-035844-2. August, 1991.
NOTE: Refer to Chapter 5: Equipment and Calibration
1987: Device Good Manufacturing Practices Manual. Fourth Edition (HHS Publication FDA 87-4179)
Small Manufacturers Assistance. Office of Training and Assistance. Andrew Lowery, Richard J. Rivera. November, 1987.
NOTE: Refer to Chapter 5: Equipment and Calibration
1984: Device Good Manufacturing Practices Manual. Third Edition (HHS Publication FDA 85-4179)
Small Manufacturers Assistance. Office of Training and Assistance. Andrew Lowery, Richard J. Rivera. November, 1984.
NOTE: Refer to Chapter 4: Buildings, Equipment and Calibration
Note: No information is readily available on the First and Second editions of the GMP Manual, which presumably preceded the
1984 Third edition. However, the following publications are available:
1982: Device Good Manufacturing Practices – A Workshop Manual

U.S. Department of Health and Human Services. Public Health Service. Food and Drug Administration. Bureau of Medical Devices. Office of Small Manufacturers
Assistance. Andrew Lowery. Fred Hooten. October, 1982.
NOTE: Refer to Section 4: Buildings, Equipment and Calibration
1979: Device Good Manufacturing Practices – A Quality Audit Program for Industry
U.S. Department of Health, Education, and Welfare. Public Health Service. Food and Drug Administration. Bureau of Medical Devices. Division of Compliance
Programs. September, 1979.
NOTE: Refer to pages 88 through 92 for calibration guidance
1978: Device Good Manufacturing Practices – An Industry Self-Inspection Program

U.S. Department of Health, Education, and Welfare. Public Health Service. Food and Drug Administration. Bureau of Medical Devices. Division of Compliance
Programs. February, 1978.
NOTE: Refer to pages 30 through 34 for calibration guidance
Note: FDA GMP/QSR guidance documents, subsequent to FDA 97-4179 (1997), do not include specific chapters on Equipment
Calibration. Such documents include:
 Guide to Inspections of Quality Systems – Quality System Inspection Techniques (QSIT)
 Compliance Program Guidance Manual for Inspection of Medical Device Manufacturers (CP 7382.845)
 Compliance Policy Guides (CPG) for devices (Sub Chapter 300)
Note: The information contained in this Appendix is a work of the United States Government, Department of Health and Human
Service, and is therefore not subject to copyright under the provisions of Title 17 of United States Code, Chapter1, Section 105
of The Copyright Act. Appreciation is extended to those who authored the content of the work as identified above.
*Note: Sections of Chapter 7 which precede “Measuring Equipment and Calibration” have been omitted in this Appendix.

Appendix F
Chapter 7: MEASURING EQUIPMENT CALIBRATION
The QS regulation is intended to help assure that devices will be safe, effective, and in compliance
with the FD&C Act. To support this goal, each medical device manufacturer should develop and
implement a quality system that assures, with a high degree of confidence, that all finished devices
meet the company's device master record specifications. These specifications should, in turn, reflect
the company quality claims. Section 501(c) of the FD&C Act states a device shall be deemed to be
adulterated if its strength differs from, or its purity or quality falls below, that which it purports
(claims). Such assurance is obtained by many activities including the measurement of component,
device, and process parameters during design and production. These measurements shall be made
with appropriate and calibrated equipment as required by 820.72.
Each manufacturer should assure that production equipment and quality assurance measurement
equipment, including mechanical, electronic, automated, chemical, or other equipment, are:
• suitable for the intended use in the design, manufacture, and testing of components, in-process
devices and finished devices;
• capable of producing valid results;
• operated by trained employees; and
• properly calibrated versus a suitable standard.
To succeed, the quality system shall include a calibration program that is at least as stringent as that
required by the QS regulation (820.72). The intent of the GMP calibration requirements is to assure
adequate and continuous performance of measurement equipment with respect to accuracy, precision,
etc. The calibration program implemented by a company may be as simple or as sophisticated as
required for the measurements to be made. Some instruments need only be checked to see that their
performance is within specified limits, while others may require extensive calibration to a
specification.
Manufacturers should determine which measurements are necessary to assure that finished
devices meet approved device master record specifications, and assure these measuring instruments
are included in a calibration program. Measurement equipment should be identified by label, tag,
color code, etc., when located in the same areas as instruments that are not part of the calibration
system. Identification can assure that proper equipment is employed to verify and determine
compliance to specification of a device component, in-process device, or finished device.
Sometimes equipment used only for monitoring a parameter need not be calibrated but should be
identified (e.g., for monitoring). A monitoring function might be to indicate if a voltage or other
parameter exists, but the exact value is not important.
Calibration Requirements
The QS regulation requires in section 820.72(b) that equipment be calibrated according to written
procedures that include specific directions and limits for accuracy and precision. Figure 5.1 illustrates
bias, precision, and accuracy.

Appendix F
Precision has no unit of measure and only indicates a relative degree of repeatability, i.e., how
closely the values within a series of replicate measurements agree with each other. Repeatability is
the result of resolution and stability.
Bias is a measure of how closely the mean value in a series of replicate measurements approaches the
true value. The mean value is that number attained by dividing the sum of the individual values in a
series by the total number of individual values.
Accuracy is the measure of an instrument's capability to approach a true or absolute value.
Accuracy is a function of precision and bias. Because different manufacturers have different
accuracy requirements, each manufacturer should decide the level of accuracy required for each
measurement and provide equipment to achieve that accuracy.
Figure 5.1 Bias, Precision and Accuracy

Appendix F
Proper and periodic calibration will assure that the selected equipment continues to have the desired
accuracy. GMP calibration requirements are:
• routine calibration according to written procedures;

• documentation of the calibration of each piece of equipment requiring calibration;
• specification of accuracy and precision limits;
• training of calibration personnel;
• use of standards traceable to the National Institute of Standards and Technology (NIST), other
recognizable standards, or when necessary, in-house standards; and
• provisions for remedial action to evaluate whether there was any adverse effect on the device’s
quality.
Remedial action includes recalibration and evaluation of the impact of out-of-tolerance
measurements:
• on the device design or process validation parameters or data;
• on the quality of existing components, in-process, or finished devices; and
• appropriate corrective action.
Equipment Selection
The manufacturer should establish and maintain procedures to ensure that purchased and otherwise
received equipment and associated supplies conform to specified requirements (820.50). The
purchase of stable and accurate measuring equipment can reduce the frequency of calibration and
increase confidence in the company's metrology program. Where economically feasible, equipment
with more accuracy than needed for various measurements can be used longer without recalibration
than equipment that marginally meets the desired accuracy requirements. Delicate instruments,
however, that are "pushing the state-of-the-art" should not be used for routine measurements unless
no other approach is feasible.
Procedures
There are a number of sources of information from which calibration procedures can be developed.
Instrumentation manufacturers often include calibration instructions with their instruction manuals.
Although these instructions alone are not adequate to meet the QS requirements for a calibration
procedure, they usually can be used for the actual calibration process. In some cases, voluntary
standards exist such as those by the American Society for Testing and Materials (ASTM), the
American National Standards Institute (ANSI), and the Institute of Electrical and Electronic
Engineers (IEEE).
Information contained in calibration procedures should be adequate to enable qualified personnel to

properly perform the calibrations. An example of a calibration procedure for mechanical measuring
tools appears at the end of this chapter.
A typical equipment calibration procedure includes:
• purpose and scope;

• frequency of calibration;
• equipment and standards required;
• limits for accuracy and precision;

Appendix F
• preliminary examinations and operations;

• calibration process description;
• remedial action for product; and
• documentation requirements.
Management of Metrology
Managers and administrators should understand the scope, significance, and complexity of a
metrology program in order to effectively administer it.
The selection and training of competent calibration personnel is an important consideration in

establishing an effective metrology program. Personnel involved in calibration should ideally possess
the following qualities:
• technical education and experience in the area of job assignment;
• basic knowledge of metrology and calibration concepts;
• an understanding of basic principles of measurement disciplines, data processing steps, and
acceptance requirements;
• knowledge of the overall calibration program;
• ability to follow instructions regarding the maintenance and use of measurement equipment and
standards; and
• mental attitude which results in safe, careful, and exacting execution of his or her duties.
Calibration Records
Calibration of each piece of equipment shall be documented to include:
• equipment identification,
• the calibration date,
• the calibrator, and
• the date the next calibration is due.
Many manufacturers use a system where each device has a decal or tag which contains the date of
calibration, by whom calibrated, and date the next calibration is due. Examples of such decals are
shown on the next page.
These decals are examples of the types commonly used to identify the status of measuring
instruments and tools. They are available as catalog items or a manufacturer may use its own artwork
to purchase decals with specialized wording.
Typical calibration decals have a

write-on surface. A tough paper or
cloth stock and a pressure sensitive
adhesive are used for easy
application and removal of the
decal. “Due” is the blank for the
date when recalibration is due.

Appendix F
Calibration Identification Number or

its equivalent is usually the
minimum information that may be
on equipment. This information
allows the manufacturer to read by
finding the associated calibration
record\card\file.
Measuring equipment that is not
calibrated or otherwise unsuitable
for use should be placed in a
quarantine area or labeled with a
"calibration void" decal.
A seal or protective cover for

exposed, recessed calibration
controls on instruments. The
calibration control cannot be
adjusted without breaking the seal or
removing the instrument case.
A decal to be applied to
measurement or monitoring
instruments not intended for use in
determining conformance to device
master record specifications with
respect to testing, manufacturing,
environmental control, etc.
Calibration information is entered onto cards or forms, one for each piece of equipment, or entered
into a computerized data system. Most data systems include the calibration date, by whom calibrated,
date recalibration is due, the reason for the calibration, comments, address of the manufacturer and
calibration laboratory, equipment specifications, serial number, use, etc. An example of a typical card
used to record calibration information follows.

Appendix F
Schedules
Measuring instruments should be calibrated at periodic intervals established on the basis of stability,
purpose, and degree of usage of the equipment. Intervals between calibrations should be shortened as
required to assure prescribed accuracy as evidenced by the results of preceding calibrations. Intervals
should be lengthened only when the results of previous calibrations indicate that such action will not
adversely affect the accuracy of the system, i.e., the quality of the finished product.

Appendix F
A manufacturer should use a suitable method to remind employees that recalibration is due. For
small manufacturers, calibration decals on the measuring equipment may be sufficient because
recalibration can be tracked by scanning the decals for the recalibration date. For other
manufacturers, a computerized system, calibration cycle cards, tickler file, or the like may be used.
Calibration cycle cards are maintained in a 12-month (12-section) tickler file. There is one card per
item of measuring equipment. The cards in the section of the file for the current month are pulled and
all of the equipment listed is calibrated. For example, in a 6-month calibration cycle, when an
instrument is calibrated in May, the card is moved from the May section to the November section of
the file. When the file is checked in November, the cycle card will be there to remind the
manufacturer that calibration is due. The process is repeated until an event such as instrument wear-
out occurs and the respective cycle card is removed from the file.
Cycle cards are used where a manufacturer has many instruments to be calibrated. It would be rather
difficult to keep track of the calibration of a large number of instruments by reviewing calibration
record cards or scanning the decal on each instrument. It is easier to use a cycle card file. A cycle
card file or equivalent also should be used if the calibration records are filed by type of instrument or
manufacturer rather than due date. A typical cycle card follows. The "calibration card number" blank
refers to the calibration record card for the same item of equipment.
Standards
Where practical, the QS regulation requires that standards used to calibrate equipment be
traceable to the National Institute of Standards and Technology (NIST), or other recognized national
or international standards. Traceability also can be achieved through a contract calibration laboratory
which in turn uses NIST services.
The meaning of traceability to NIST is not always self-evident. Two general methods commonly
used to establish and maintain traceability to NIST are:
• NIST calibration of standards or instruments: When this method is used, private standards are
physically sent to NIST for calibration and returned.
• Standard Reference Materials (SRM's): NIST provides reference materials to be used in a user's
calibration program. These SRM's are widely used in the chemical, biological, medical, and
environmental fields.

Appendix F
Information can be obtained from the "Catalog of NIST Standard Reference Materials,” available
free from the National Institute of Standards and Technology, Office of Standard Reference
Materials, Gaithersburg, MD 20899, phone: (301)975-2016.
When in-house standards are used, they should be fully described in the device master record or
quality system record. Independent or in-house standards should be given appropriate care and
maintenance and should be used according to a written procedure as is required for other calibration
activities. FDA recommends that at least two in-house standards be maintained – one for routine use
and one for a back-up.
Calibration Environment
As appropriate, environmental controls should be established and monitored to assure that measuring
instruments are calibrated and used in an environment that will not adversely effect [sic] the accuracy
required. Consideration should be given to the effects of temperature, humidity, vibration, and
cleanliness when purchasing, using, calibrating, and storing instruments.
AUDIT OF CALIBRATION SYSTEM

The calibration program shall be included in the quality system audits required by the QS regulation.
These audits should determine the continuing adequacy of the calibration program and assess
compliance with the program.
Many manufacturers use contract calibration laboratories to calibrate their measurement and test
equipment. If this is the case, FDA views the contract laboratory as an extension of the
manufacturer's GMP program or quality system. Normally FDA does not inspect contract laboratory
facilities, but it does expect the manufacturer to assess the contract lab to verify that proper
procedures are being used. Generally, the manufacturer of the finished device is responsible for
assuring the device is manufactured under an acceptable quality system.
When a medical device manufacturer uses a contract calibration laboratory, FDA expects the
manufacturer to have evidence that the equipment was calibrated according to the GMP
requirements. The device manufacturer can do this by:
• requiring and receiving certification that the equipment was calibrated under controlled conditions
using traceable standards;
• maintaining an adequate calibration schedule;
• maintaining records of calibration; or
• periodically auditing the contractor to assure appropriate and adequate GMP procedures are being
followed. For example, the contractor should have:
• written calibration procedures;
• records of calibration;
• trained calibration personnel; and
• standards traceable to NIST or other independent reproducible standards.
Certification notes and data should include accuracy of equipment when received by the lab to
facilitate remedial action by the finished device manufacturer, if necessary. Certification should also
include accuracy after calibration, standards used, and environmental conditions under which the
equipment was calibrated. The certification should be signed and dated by a responsible employee of
the contract lab.

Appendix F
If in-house standards are used by a contractor to calibrate device-related measuring equipment, these
standards shall be documented, used, and maintained the same as other standards.
INTEGRATING MEASUREMENTS INTO THE QA SYSTEM

Proper and controlled calibration can contribute to overall quality by assuring that device design and
process parameters are accurately measured and that unacceptable items are not accepted, and
acceptable items are not rejected as a result of measurements. If the appropriate product-quality
parameters are not checked, however, calibrated equipment will have little impact on assuring
quality.
A good quality system shall include calibration activities. However, proper calibration will be of
little use unless the applications of the measurement equipment are properly developed and qualified
during the preproduction development of inspection test methods and procedures. As stated,
effectiveness depends on the participation and influence of QA and production management at the
preproduction stage. Calibration of equipment cannot correct poor design of products nor can it
compensate for poor applications of equipment and techniques. It is the continued use of a complete,
integrated quality system, which assures that safe and effective devices are produced.
EXHIBITS
Examples of calibration cards, decals, and cycle cards were presented above in the text. Examples of
a device cleaning procedure and a calibration procedure follow. Manufacturers may use these as
presented if they match the manufacturers operations; or may modify them to meet specific
requirements.
P.C. Board Cleaning [Omitted herein]
Calibration Procedures for Mechanical Measuring Tools
This is a calibration procedure for mechanical measuring tools. In actual use, the initial accuracy of
each tool is checked using the procedure and is recorded. Thereafter, each tool is recalibrated
(checked) versus the initial accuracy. Of course, the initial accuracy should meet or exceed the
requirements of the measurements to be made with the tool. Precision is checked by making several
measurements at various points on the tool's measuring face (surface).

Appendix F
Sheet 1 of 1
TITLE: Calibration Procedures for Mechanical Measuring Tools No.________Rev.___________
ECN Notes __________________________________________________________________

______________________________________________________________________________
Drafted by______________________________App.________________Date____________
PURPOSE:
This procedure establishes a standard method for the calibration and maintenance of mechanical
measuring tools such as micrometers, calipers, etc.
SCOPE:
All measuring tools used to set specifications or measure conformance to specifications, such as
micrometers, calipers, etc., will be included in the calibration program. Each tool will be assigned a
number and checked every six months for accuracy. If you suspect a tool is damaged or out of calibration,
it should be removed from service and brought to the Quality Control Lab (QC) for checking. To enter a
tool in the program, take it to QC where a number will be assigned and initial accuracy checked and
recorded.
PROCEDURE:
1. Each measuring tool shall be kept clean and maintained in a protective container. As needed, all
threads and slides shall be lubricated with a fine tool oil to assure free movement.
2. The calibration shall be done by a comparison to standard gage blocks traceable to the National
Institute of Standards and Technology standard with an accuracy 3 to 10 times greater than that of the
measuring tool.
3. The comparisons shall be made at different points along the measuring range of the tool. The gage
blocks used shall be picked at random to assure that the measuring tool is not checked at the same points
on each calibration cycle. When a measurement is made, move the gage blocks from one side of the tool's
measuring face to the other on an X/Y axis to assure no wear or taper exists on the measuring faces.
4. Measurement tools not intended for testing or manufacturing do not require calibration in accordance
with the QS regulation. These tools should be kept out of manufacturing or labeled to avoid inadvertent
use. Otherwise, they should be entered in this calibration program.
5. After calibration, the date of calibration and the next due date of calibration shall be recorded on the
Calibration Form No._______. Any adjustments and/or repairs to be recorded. The form is placed in the
tickler file according to the next calibration date.
6. If a tool is found to be out of calibration, the QC lab will immediately pass the out-of-calibration
information to the appropriate supervisor in the department where the tool is used. The Department and
QC management will take appropriate remedial action for affected in-process or finished devices.

Baxter P.reese Calibration in Regulated Industries Federal Agency Use ISO-17025 ANSI Z540.3 28

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Baxter P.reese Calibration in Regulated Industries Federal Agency Use ISO-17025 ANSI Z540.3 28

Uploaded by

Copyright:

Available Formats

Calibration in Regulated Industries:

Federal Agency use of ANSI Z540.3 and ISO 170251

2016 NCSL International Workshop & Symposium

 National Aeronautics and Space Administration (NASA)

 Government-unique standards (e.g. FDA, DoE, DoD, NASA, NRC, etc.)

2016 NCSL International Workshop & Symposium

2016 NCSL International Workshop & Symposium

2016 NCSL International Workshop & Symposium

Government Standards & Guides

1973 1976 1997

DEFSTAN 05-26 FDA 21 CFR

1979 1976 1971

Voluntary Consensus Standards & Guides

UKAS M10 ANSI/NCSL Z540.1

* Asterisk designates active voluntary consensus standards for calibration.

2016 NCSL International Workshop & Symposium

MIL-STD-120 (1950) Military Standard, Gage Inspection

Government-Unique Quality & Calibration System Standards

Allied Quality Assurance Publications (NATO) 1968 - 1972

British Ministry of Defence Quality Standards (MoD) 1973 - 1976

Calibration System Voluntary Consensus Standards

 BS:5179-2 Part 2 – Comprehensive Inspection System Measurement and

Figure 2. Divergence of ISO quality and calibration system standards

2016 NCSL International Workshop & Symposium

Russell: NCSL (1966)

Fruechtenicht: NCSL (1968): [6]

2016 NCSL International Workshop & Symposium

2016 NCSL International Workshop & Symposium

MIL-STD-45662A (1988): [29]

 1960: MIL-C-45662: Required a 10:1 TAR [12]

2016 NCSL International Workshop & Symposium

T. Held – Abbott Laboratories (1992): [33]

2. Promote the establishment of a nationally recognized calibration laboratory accreditation system.

2016 NCSL International Workshop & Symposium

T. Held – Abbott Laboratories (1992): [33]

J. Cigler – NVLAP Calibration Laboratory Program (1993): [36]

Schumacher, NCSL (1995): [32B]

2016 NCSL International Workshop & Symposium

4. The 2000’s: ISO 17025 and ANSI/NCSL Z540.3

2016 NCSL International Workshop & Symposium

ISO Guide 25 Draft V (1996): [40]

ISO 17025:2005 (2005): [27]

2016 NCSL International Workshop & Symposium

NOTE For further information see ISO/IEC Guide 98-4”.

ANSI/NCSL Z540.3 (2006): [44]

2016 NCSL International Workshop & Symposium

2016 NCSL International Workshop & Symposium

FDA, Guide to Inspections of Medical Device Manufacturers (1997): [67]

2016 NCSL International Workshop & Symposium

2016 NCSL International Workshop & Symposium

ANSI/NCSL Z540.3-2006 (R2013): [44]

2016 NCSL International Workshop & Symposium

Eurachem Guide on Terminology in Analytical Measurement (2011): [202]

2016 NCSL International Workshop & Symposium

Weitzel & Johnson - USP: (2012) [A138]

2016 NCSL International Workshop & Symposium

The ANSI/ASQC standard, ‘Calibration Systems’ [ANSI/ASQC M1-1987], discusses quantification of

2016 NCSL International Workshop & Symposium

ISPE/GAMP Risk Based Approach to Calibration Management (2010): [76]

“5.5.5 Introduction to the Application of Measurement Uncertainty

5.5.7 Review of Third Party Calibration Certificates for Reference Standards

Terms which may be used by pharmaceutical organizations include:

2016 NCSL International Workshop & Symposium

“5.3 Adequacy of Measuring and Test Equipment (M&TE)