Trs1025-Annex13 WHO Technical Report 2020

Annex 13
WHO guideline on the implementation of quality

management systems for national regulatory authorities
Abbreviations 274
1. Background 274
2. Objectives 275
3. Scope 276
4. Glossary 278
5. Quality management system requirements for national
regulatory authorities 279
5.1 Quality management system concepts 279
5.2 Quality management system requirements 282
6. Quality management system implementation methodology 304
6.1 Supporting factors for quality management system implementation 304
6.2 Situational analysis of quality management system implementation status
in the national regulatory authority 305
6.3 Gap analysis for developing a roadmap for quality management system
implementation 307
6.4 Quality management system development and implementation roadmap 317
6.5 Activity plan for quality management system implementation 318
References 318
Appendix 1 References to the WHO Global Benchmarking Tool, revision VI 320
Appendix 2 Activity plan for quality management system implementation 323
273
WHO Expert Committee on Specifications for Pharmaceutical Preparations Fifty-fourth report
Abbreviations
CREAM clear, relevant, economic, adequate and monitorable
EDQM European Directorate for the Quality of Medicines and
HealthCare
GBT WHO Global Benchmarking Tool (4)
GRP good regulatory practices
ICT information and communication technology
IT information technology
ISO International Organization for Standardization
M&E monitoring and evaluation
NRA national regulatory authority
PDCA plan–do–check–act
QA quality assurance
QMS quality management system
RCA root cause analysis
SMART specific, measurable, attainable, realistic and time-bound
SWOT strengths, weaknesses, opportunities and threats
WHO World Health Organization

1. Background
WHO Technical Report Series, No. 1025, 2020
Implementation of the Thirteenth World Health Organization (WHO) General

Programme of Work (2019–2023) (1), as adopted by the Seventy-first World
Health Assembly (2018), and the WHO Leadership priorities (2), has attracted
much international public health attention to the theme of universal health
coverage and to increased access to safe and effective medical products.
Several World Health Assembly resolutions, including WHA67.20 (2014)
(3), mandate WHO to provide support to its Member States in strengthening
national regulatory systems for medical products. It recognizes that “effective
regulatory systems are an essential component of health system strengthening
and contribute to better public health outcomes, that regulators are an essential
part of the health workforce, and that inefficient regulatory systems themselves
can be a barrier to access to safe, effective and quality medical products” (3).
274
Annex 13
Accordingly, to facilitate access to these products, WHO’s vision is for all Member
States to have an effective regulatory system that ensures medical products
and other health technologies in the market meet internationally recognized
standards of quality, safety and efficacy.
National regulatory authorities (NRAs) are responsible for ensuring the
safety, quality and efficacy of medical products within their respective Member
States; demonstrating that the services they provide consistently meet legal and
regulatory requirements; delivering effective and efficient services; evaluating
performance; and making improvements. An effective quality management
system (QMS) can help to ensure that the products or services an NRA provides
consistently meet statutory and regulatory standards and meet customers’
expectations. A QMS provides opportunities to enhance customer satisfaction;
address context-associated risks and opportunities for continued improvement;
demonstrate conformity to specific QMS requirements; and assure the quality,
safety and efficacy of medical products.
In 2015, WHO developed and launched the WHO Global Benchmarking
Tool (GBT) (4). This tool assists WHO and regulators worldwide in evaluating
the maturity and performance of regulatory systems and related functions. The
GBT includes one indicator that assesses the NRA’s level of development with
respect to a QMS (4). Benchmarking results of low- and middle-income countries
indicate that the majority of NRAs need to establish and implement a QMS or,
if already established, enhance and maintain the QMS.
QMS implementation is challenging for NRAs, owing to the diversity
of NRA legal mandates and organizational structures; the different levels of
NRA development; and the number of regulatory functions that need to be
implemented. WHO has developed this guideline to respond to requests by
Member States for an international guideline on implementation of QMSs
by NRAs.
2. Objectives
The aim of this guideline is to assist NRAs to develop, implement and
improve their QMSs, based on principles from International Organization for
Standardization (ISO) document ISO 9001 standard requirements (5). It
provides recommendations on what NRAs should implement and maintain
under each QMS to effectively and efficiently support the execution of NRA
functions as mandated by national laws and regulations. The guideline is
expected to promote consistency in regulatory practices within and across
NRAs, to facilitate harmonization, mutual reliance and recognition mechanisms
among Member States.
Therefore, the objectives of the guideline are as listed next.
275
1. Describe principles for implementing a QMS to support planning,

execution, monitoring and evaluation (M&E) of the performance
of all applicable functions and activities of an NRA.
2. Provide requirements for the QMS to support and facilitate
systematic linkages and integration of different processes and systems
of the regulatory functions and activities within an NRA.
3. Provide requirements that NRAs should consider for evaluating
the performance of the QMS and measures that the NRA should
implement for continually improving the QMS.
3. Scope
This is an overarching guideline that should be applied across all regulatory
functions and activities, including registration and marketing authorization;
vigilance; market surveillance and control; licensing establishments; regulatory
inspections; laboratory access and testing; clinical trials oversight; national lot
release; and others, as applicable to the implementing NRA. The guideline should
be implemented to cover all types and categories of medical products and other
health technologies under the responsibility of an implementing NRA. All other
existing or future guidelines for QMSs for specific regulatory functions will
complement this guideline.
The guideline can also be used for other regulatory activities that are
mandated by the national laws and regulations to ensure public health safety, by
assuring the quality, safety and effectiveness of medical products. This extends to
areas of medical products such as pricing, professional training and regulation, as
well as to other areas within the legislative mandates and functions of the NRA.
This guideline on QMS implementation can also be used for all models
of NRA. NRAs can be legally, organizationally and operationally structured as
follows:
■■ discrete: two or more institutions involved in partial or full

enforcement of national laws and regulations for medical products
in a country (e.g. one institution with legal mandates to enforce
marketing authorizations and another one within the same country
for licensing establishments’ regulatory function);
■■ decentralized: one NRA with full legal mandates to enforce national
laws and regulation of medical products within the country. A
legally defined amount of enforcement, authority and operations
is executed in localized zones or geopolitical zones of the country,
while the rest is enforced at country level. This model exists in
Member States with a federal governance system where laws and
regulations are enforced at state/province and national levels; and
276
Annex 13
■■ centralized: one NRA with full legal mandates to enforce national

laws and regulation of medical products within the country. The
enforcement, authority and operations are executed, managed and
controlled centrally for all applicable regulatory functions and
activities.
The guideline is equally applicable to small, medium and large NRAs,
as the principles and intended results of a QMS remain the same, regardless of
the complexity of NRA. Therefore, this guideline describes the requirements that
should be implemented; the MS and respective NRAs reserve the right to decide
on how to address these requirements within the existing contexts and provisions
of the laws. The guideline can be utilized by institutions that are responsible for
single or multiple specific regulatory functions related to medical products.
Although the use of this guideline is voluntary, NRAs are encouraged to
use it to facilitate implementation of their QMS. The implemented QMS should
be demonstrated by documented evidence to have systematic processes that are
controlled, maintained and evaluated for continuous improvement. NRAs are
free to use any appropriate national or international standard or guideline on
QMSs as a basis for the implementation.
Where different units within the NRA have already implemented a QMS
for specific regulatory functions (such as laboratory testing and/or regulatory
inspection), this guideline could be used by the NRA for those functions and
processes that have not been addressed by the management systems already
implemented. This is to avoid duplications and overlaps of management systems.
It is expected that NRAs will gradually integrate all existing management systems
within the overall QMS of the NRA. The implementing NRA could determine
the extent to which this guideline should be implemented, without omitting any
of its processes and activities that are mandated by national laws and regulations.
Effective implementation of this guideline will not lead to any
WHO certifications and WHO will not conduct any audits for verification
of implementation of a QMS. However, as part of the regulatory systems
strengthening programme, WHO will conduct the benchmarking of the Member
State’s regulatory system and functions, including QMS-related processes, using
the GBT (4) to determine the strengths and gaps, if any, for capacity-building
and continuous improvement. This guideline should be implemented to cover
regulatory functions that are part of the GBT, and other functions and activities
of the NRA that are addressed by national laws and regulations but that are not
part of the GBT. References to GBT revision VI (4) provides a linkage between
GBT indicators and the relevant sections of this guideline (see Appendix 1).
The QMS using this guideline should be implemented on the foundation
of the principles and recommendations on Good regulatory practices (GRP) (6).
The implementation of the QMS should ensure that the GRP are integrated to the
277
extent possible without affecting the effectiveness and efficiency of the NRA to
execute its functions.
4. Glossary
The definitions given below apply to the terms used in this guideline that are
not defined in existing WHO terms and definitions databases. They may have
different meanings in other contexts.
competence. Knowledge, skills and attitude required for successful work
performance.
correction. Any action that is taken to eliminate a nonconformity. However,
corrections do not address causes.
corrective actions. Steps that are taken to eliminate the causes of existing
nonconformities in order to prevent recurrence. The corrective action process
tries to ensure that existing nonconformities and potentially undesirable
situations do not happen again.
customer. A person or organization that could or does receive a product or a
service that is intended for or required by this person or organization. Customers
of an NRA include individuals or parties who receive or could receive and
use products and services that are provided and offered by the NRA. These
parties include the general public, patients, manufacturers, distributors, health
practitioners, researchers, the ministry of health and other individuals and
institutions that rely on the NRA’s products and services to make public health
decisions.
customer satisfaction. A customer’s perception of the degree to which the
customer’s expectations have been fulfilled. This relates to the expectations
that different parties have of the NRA. The expectations include assurance
that safe, efficacious and high-quality medical products will be available under
the NRA mandate to regulate, and that the NRA will provide other products
such as guidelines, public reports and related regulatory services that meet the
expectations of different types of customers.
internal audit. An examination and assessment of all or part of a QMS with the
specific purpose of improvement. An internal audit should be conducted by an
independent (i.e. of the function to be audited) team of competent auditors as
designated by the management for this purpose.
process. A set of interrelated or interacting activities that use inputs to deliver an
intended result. In the context of NRAs, the production and service provision
processes should coincide with basic regulatory functions.
278
Annex 13
product. Output of an organization that can be produced without any transaction

taking place between the organization and the customer. They are also called
regulatory products in this guideline. Products of NRAs relate to the tangible
items that the NRA produces for its customers. These items include regulatory
guidelines; public health notices; guidance notes; alerts; databases; mobile phone
applications; reports; and other materials that are intended to provide regulatory
information and communications to customers. Before their production, some
of these products may require lengthy consultations for designing them.
quality. The total set of characteristics of an entity that affect its ability to satisfy
stated and implied needs and to ensure the consistent and reliable performance
of services or products in conformity with specified requirements.
quality management system. An appropriate infrastructure, encompassing the
organizational structure, procedures, processes, resources and systematic actions
necessary to ensure adequate confidence that a product or service will satisfy
given requirements for quality.
quality policy. A brief statement that describes the organization’s purpose,
overall intentions and strategic direction; provides a framework for quality
objectives; and includes a commitment to meet applicable requirements.
senior (top) management. Person(s) who direct and control an organization
at the highest levels and who have the authority and responsibility to mobilize
resources within the organization. In NRAs, the terms “senior management” or
“top management” can be used interchangeably.
services. Output of an organization with at least one activity necessarily
performed between the organization and the customer. Services of NRAs are also
called regulatory services in this guideline. This includes, for example, activities
such as evaluation of applications for market authorizations, inspections of
facilities, testing of health product samples, etc.
5. Quality management system requirements

for national regulatory authorities
5.1 Quality management system concepts
NRAs should implement a QMS that is supported by the process approach
concept, plan–do–check–act (PDCA) cycle and risk-based thinking. NRAs
should ensure that the implemented QMS meets its needs without making it
unnecessarily complex, to avoid a negative effect on the NRA’s effectiveness and
efficiency. The QMS should be simple, fit for purpose and understandable.
279
The PDCA cycle requires NRAs to carry out planning, performing

(implementing), checking (evaluating) and acting (to improve) processes in
the QMS. The applied PDCA cycle covering the chapters in this guideline is
provided in Fig. A13.1. The ISO 9001 standard (5) provides a brief description
of the PDCA as follows:
■■ Plan: establish the objectives of the system and its processes,
obtain the resources needed to deliver results in accordance with
customers’ requirements and the NRA’s policies, and identify and
address risks and opportunities;
■■ Do: implement what was planned;
■■ Check: monitor and, where applicable, measure processes and
the resulting products and services against policies, objectives,
requirements and planned activities and report the results;
■■ Act: take actions to improve performance, as necessary.
Fig. 1
Applied PDCA cycle
NRA: national regulatory authority; PDCA: plan–do–check–act;
QMS: quality management system.
280
Annex 13
The scope of the QMS and context of the NRA are placed in the middle,
to provide the limitations to which the QMS should be implemented.
Leadership and management are centrally indicated, as they are
important requirements for effective QMS implementation. Top management
should commit and support all QMS processes, from planning up to acting for
continuous improvement.
Document and data management are centrally indicated, because they
should be part of every step of the PDCA cycle, in the form of procedures, forms
and records that facilitate the consistent implementation of QMS processes and
record retention.
Applying risk-based approaches (included in planning stages) enables
NRAs to identify factors that could cause QMS processes to deviate or that
could prevent the planned results from being achieved; to put in place proactive
measures and controls to minimize the impact of negative effects; and to leverage
opportunities as they arise. Risk-based thinking is applicable and should be
implemented throughout the PDCA cycle.
NRAs should implement a QMS that identifies and integrates other
management system standards that are applicable to the processes. The
management systems that are for specific areas and processes should be
documented. The NRA should ensure that the management systems do not
create duplications, overlaps or inconsistencies within the overall QMS. While
other WHO guidelines have been implemented for management systems of
specific regulatory functions such as inspections and quality control testing, the
overall QMS should be consistently implemented throughout the organization
across different regulatory functions and other supporting areas.
QMSs are influenced by the different policies, objectives, diverse work
methods, resource availability and administrative practices specific to each
NRA. NRAs are free to decide the mode and routes to use when implementing
this guideline, as long as the implemented QMS yields effective, consistent,
transparent and reliable results in the regulation of medical products.
The QMS requirements that are described in this guideline are based
upon the quality management principles presented next, as provided in ISO
9000 (7).
■■ Customer focus: the primary focus of a QMS is to meet customer

requirements and to strive to exceed customer expectations. In
this guideline, customer focus means meeting the needs and
expectations of the public, patients, health-care practictioners,
manufacturers, researchers and procurers, by providing regulatory
products and services that assure access to high-quality, safe,
effective and affordable medical products and health technologies.
281
■■ Leadership: leaders at all levels establish unity of purpose and

direction and create conditions in which people are engaged in
achieving the NRA’s planned objectives.
■■ Engagement (involvement) of people: competent, motivated,
empowered and engaged people at all levels throughout the
organization are essential to enhance the organization’s capability to
create and deliver valued services.
■■ Process approach: consistent and predictable results are achieved
more effectively and efficiently when activities are understood
and managed as interrelated processes that function as a coherent
system. This is critical, as it avoids having systems that are based on
individuals within the NRA.
■■ Improvement: successful organizations have an ongoing focus on
improvement. The NRA should ensure that it strives continously to
improve its processes, products and services and the QMS.
■■ Evidence-based decision-making: decisions based on the analysis
and evaluation of data and information are more likely to produce
the desired results. This requires NRAs to implement measures for
monitoring, analysing and evaluating the collected data, to assess
whether the processes are delivering the desired results.
■■ Relationship management: for sustained success, organizations
should manage their relationships with relevant interested parties.
Implementing an effective QMS requires the NRA to ensure that its
relationships are managed strategically for continous operations.
The relationships include management of contractual agreements
for activities subcontracted to individuals and institutions. The
areas with subcontract agreements would either be technical or
administrative and, if not managed properly, may have negative

effects on the effective implementation of the QMS.
5.2 Quality management system requirements

The QMS requirements described in the subsequent subsections describe what
NRAs should implement as part of their overall QMS. Table A13.1 provides a
summary and focus for each subsection.
282
Annex 13
Table A13.1
Summary of quality management system requirements for each subsection
Subsection Summary of requirements for implementation of a quality

management system
5.2.1 Introduction The requirements of this subsection focus on the NRA
describing and documenting the setup of its QMS. The setup
includes:
• legislative mandates and scope (functions) of the NRA;
• standards and guidelines used in QMS implementation;
• QMS implementation history;
• integration (as applicable) with other management and
software systems for personnel performance appraisals,
finances and accounting, environment, occupational health
and safety, workflow, customer relationship management,
ministry of health policies and strategic action plans;
• identification of the functions and processes that are
already covered by other QMSs.
5.2.2 Scope of the This subsection describes the requirements for NRAs
quality management to document the processes that are covered by the
system implemented QMS. All processes and activities that are done
by the NRA as mandated by national laws and regulations
should be included in the QMS. Implementation can be done
at once or in phases.
5.2.3 Organizational The focus of this subsection is to provide guidance regarding
context of the what should be indicated when describing and documenting
national regulatory the setup of the NRA with its regulatory system, functions
authority and activities within the QMS. This extends to the model
type (discrete, decentralized or centralized) and to the
relationships with other institutions providing regulatory
services for medical products and other health technologies.
The context should also specify what to implement in the
QMS to support the NRA in handling and managing internal
and external issues within its regulatory mandates and
functions, as well as meeting the needs and expectations of
interested parties (i.e. customers and stakeholders).
5.2.4 Leadership, This subsection describes requirements for what should be
management and expected from top management for effective implementation
organization of the QMS. It also includes the roles, responsibilities and
authorities that should be part of the implemented QMS.
283
Table A13.1 continued

Subsection Summary of requirements for implementation of a quality
management system
5.2.5 Document and This subsection provides requirements that should be
data management applicable to internally generated documents and to
those of external origins, including data. The requirements
include development, review, approval, distribution, version
and access control, storage, retrieval and disposition of
documents.
5.2.6 Planning This subsection focuses on planning requirements
for achieving the set objectives, managing risks and
opportunities across the NRA, and planning changes to the
QMS for continuous improvement.
5.2.7 Support and This subsection includes requirements for input resources
resources (technical and non-technical), personnel and infrastructure
needed for effective implementation of the QMS.
5.2.8 Operation This subsection addresses the requirements for QMS
implementation in core processes and activities that are
within the mandates of the NRA. It also provides guidance on
documenting operational linkages of processes and systems
for effective and efficient QMS implementation.
5.2.9 Performance This subsection provides methodologies and
evaluation recommendations regarding what should be implemented
by the NRA to facilitate accurate, objective and efficient
performance monitoring, analysis and evaluation of
operations indicators, QMS effectiveness, resources and
customer satisfaction.
5.2.10 Improvement This subsection presents requirements for NRAs to implement
in the QMS to support continuous improvements based on

collected, analysed and evaluated data.
5.2.1 Introduction
NRAs should have documented, available and accessible legislative laws and
regulatory policies on medical products that describe the regulatory functions
and activities that should be included in the QMS.
NRAs should list and maintain current versions and copies of national,
regional and international management system standards and guidelines that are
used for the QMS implementation.
NRAs should document the history and evolution of their QMS, to
demonstrate the controls and management of changes in the system to ensure
284
Annex 13
that it is effective for the institution. The evolution and changes should be justified
and related to any changes to the adopted national, regional and/or international
management system standards and guidelines (e.g. the case of the Compendium
of quality management system (QMS) technical documents for harmonization of
medicine regulation in the East African Community (8)).
NRAs should ensure that all existing and already implemented
management systems are integrated in the QMS. The integration should ensure
that there are systematic, adequate and appropriate linkages between the overall
QMS and the management systems for specific technical or administrative
functions. The QMS should be integrated into the regulatory processes, to ensure
that it helps the NRA to achieve its legal mandates and functions.
The NRA should identify the functions and processes that are already
covered by other QMSs. This should be done to identify gaps and align specific
management systems with the overall QMS of the NRA as much as possible, to
ensure consistency and facilitate effective performance M&E. The management
systems for specific technical and administrative functions and processes are
described in the next subsections.
5.2.2 Scope of the quality management system

This guideline aims to provide guidance on implementing a sustainable and
effective QMS based on adapted ISO 9001 standard (5) requirements, to
address the needs of NRAs with respect to all regulatory functions, including
administrative and supporting processes. The scope of the QMS should include
functions, processes and the facilities where they are undertaken.
NRAs should ensure that the implemented QMS provides a clear
statement of scope that specifies the functions and processes that are covered as
mandated by the national legal framework. The scope should include all applicable
regulatory functions that are provided in the current version of the GBT (4). In
addition, the QMS should also cover all additional technical and administrative
functions and processes that are part of the NRA’s routine operations.
Where there is more than one institution that is partially or fully involved
in the regulatory activities of medical products of a country, the QMS for each
institution should support consistency, effectiveness, efficiency and systematic
collaboration, to improve and strengthen coordination between institutions.
The QMS should also include the technical and administrative functions and
processes that are interrelated and interdependent for the effective undertaking
of the affected regulatory function(s). The QMS should be clear on the scope for
each involved institution, to ensure that there are neither gaps nor overlaps of the
processes and activities.
When a specific unit of an NRA has implemented a QMS (e.g. a quality
control laboratory, inspectorate or province/zone/state), the scope should be
285
clear on the inclusions and exclusions (with justifications as applicable), without

weakening operational linkages and interdependencies for timely and effective
regulatory decision-making.
The scope statement of the implemented QMS should be documented
and supported by a relevant national legal framework and by current best
practices of the affected functions and processes.
5.2.3 Organizational context of the national regulatory authority

5.2.3.1 Understanding the national regulatory authority organization and its context
The NRA should demonstrate that it understands its organizational and
operational context within the country’s regulatory framework as part of national
health system. The organization should understand the context under which it
provides the regulatory products and services, which may be through using a
discrete, decentralized or centralized type of organizational structure. This
understanding facilitates the identification and management of internal and
external issues relevant to its ability to achieve the objectives as defined in the
NRA’s strategic plans.
The NRA should document the context in which it exists and in which it
has been given the legal mandate to perform regulatory functions that are within
the scope of the QMS. The context should indicate the limitations of the NRA
and the relationships with other institutions that are part of its routine operations.
The documented context of the NRA should clearly indicate the
technical and administrative areas that are not exclusively under the control and
management of the NRA. This could include areas such as personnel recruitment,
management of finances, procurement, and management of equipment and
infrastructure.
Determination and documentation of internal and external issues should
be integrated in the regulatory processes of the NRA, based on the needs and
expectations of customers and stakeholders. The determination of internal and

external issues should also be linked to the development of a strategic plan to
ensure that the implemented QMS helps the NRA achieve the objectives.
Internal and external issues can change (e.g. changes to laws or
regulations, or procurement, changes in national labour laws, or changes to
professional practice regulations), and therefore they should be monitored and
reviewed. The NRA should conduct and document reviews of its organizational
context at planned intervals, and whenever there are changes to the legal
framework or when there are organizational or structural changes.
NRAs should understand the context, as well as the internal and external
issues that provide the foundation and inputs for determining a strategic plan;
scope of the QMS; quality policy; quality objectives; and related risks and
opportunities.
286
Annex 13
NRAs may use national legal provisions to identify different types of

interested parties (customers and other stakeholders) for the regulatory products
and services that are provided. Where customers and other stakeholders are
defined in the national laws and regulations, this would be sufficient, as long as
all outputs and services provided by the NRA are addressed. This identification
helps NRAs to separate other stakeholders from customers who should also be
the focus of the QMS. NRAs should focus on all interested parties that can affect
its ability to achieve the quality objectives. In addition, these interested parties
should be categorized, along with their respective needs and expectations of the
NRA that the implemented QMS is designed to support.
NRAs should have a robust and defined system in place to monitor,
review and document the relevant requirements of interested parties at planned
intervals.
5.2.3.2 Quality management system processes

NRAs should ensure that inputs and resources that are required to perform the
processes and functions covered in the QMS scope, with expected outputs, are
determined, documented and provided. NRAs should document the sequences
and interactions of regulatory processes, together with related measures and
criteria for their control (e.g. key performance indicators). The level and type
of controls that are applied to the regulatory processes should be determined
and documented with a risk-based approach and should utilize the available
opportunities. The QMS should provide procedures for evaluating the processes
and allow for the implementation of corrective actions under a controlled and
managed change process. This should facilitate continuous improvements of the
processes and the entire QMS.
The QMS should be integrated in the regulatory functions and
processes, to ensure that the personnel who are assigned with responsibilities
and authorities in performing regulatory and administrative activities have the
required competencies.
5.2.4 Leadership, management and organization

5.2.4.1 Commitment of top management
Top management of the NRA should demonstrate leadership and commitment
towards the effective implementation and sustainability of the QMS within
the national legal framework, through continual identification of the needs
and expectations of its customers. The following are responsibilities of top
management with respect to a QMS:
■■ providing needed resources for the implementation of an effective
QMS that is consistently implemented across the NRA units,
functions and processes;
287
■■ integrating QMS requirements into the regulatory processes and

aligning the quality policy and quality objectives with the strategic
plans of NRAs;
■■ implementing a QMS that incorporates a risk-based approach and
that is based on functions and processes rather than being built
around individual personnel or specific activities;
■■ communicating the importance of the QMS, to maintain consistency
in NRA functions and to improve the effectiveness and efficiency of
the QMS;
■■ engaging, supervising and supporting all NRA personnel to
contribute to the implementation and effectiveness of the QMS and
to ensure that the NRA achieves the intended expectations; and
■■ reviewing the performance of the QMS and promoting
improvements.
Top management should ensure that the risks and opportunities that
can affect the ability of the NRA to provide products and services of the quality
expected by customers are determined and documented. Top management
should also ensure that the NRA implements measures to enhance customer
satisfaction. To increase customer satisfaction, innovation and best practices may
be introduced into the NRA’s processes, with the appropriate determination of
related risks and practicality.
5.2.4.2 Quality policy

Top management of NRAs should establish, implement and maintain a
documented quality policy that contains actionable and practical statements that:
■■ take into consideration the organizational context and strategic
directions and plans and provide a framework for setting quality

objectives;
■■ include a commitment to comply with applicable national legislation,
as well as regional and global regulatory requirements and best
practices;
■■ include a commitment to continual improvement of the QMS; and
■■ include a commitment to adopt and implement GRP, as provided in
the WHO good regulatory practices: guideline for national regulatory
authorities for medical products (6).
Top management should communicate the quality policy to all NRA
personnel and ensure that the personnel have read, understood and applied it
within their respective activities. Where applicable and appropriate, controlled
288
Annex 13
copies of the quality policy should be available to customers and stakeholders,

through established document control procedures as per QMS requirements.
5.2.4.3 Roles, responsibilities and authorities

To effectively implement the QMS, top management should assign and
document roles, responsibilities and authorities, and should ensure that this
information is communicated and understood within the NRA. Depending
on the organizational context of the NRA and on the scope and complexity
of the QMS, top management should assign the following responsibilities and
authorities to one or more job function:
■■ ensuring that the QMS of the NRA conforms to the requirements of
the adopted standards and guidelines;
■■ ensuring that the integrated QMS and regulatory processes are
delivering their intended results as per action and strategic plans of
the NRA;
■■ monitoring and reporting on the performance of the QMS and
proposing opportunities for improvements to top management;
■■ ensuring the promotion of customer focus throughout the NRA,
while assuring the quality, safety and efficacy/effectiveness of health
products; and
■■ ensuring that the integrity of the QMS is maintained when changes
(e.g. legislative, process, organizational or structural) to the QMS are
planned and implemented.
Top management should ensure that the job function(s) to be assigned
the above responsibilities and authorities have the necessary competencies and
have direct access to and are accountable to top management.
5.2.5 Document and data management

NRAs should have the guidelines, policies and procedures that are necessary for
the effective implementation of the QMS within the legislative provisions.
QMS documents should include, but are not limited to, internally and
externally generated hard copy and/or soft copy formats of regulations, drawings,
policies, guidelines, strategic plans, action/work plans, manuals, procedures,
registers, logbooks, databases, spreadsheets, templates and forms, codes of ethics
and professional conduct, inventories, checklists and all other documents that are
used in the technical and administrative activities of NRAs.
QMS documents should include internally and externally generated
evidential documents (e.g. records, files and reports) in hard copy and/or soft
copy formats, which are retained by NRAs.
289
NRAs should consider all published materials, either on intranets or

websites or in newsletters and other forms of publication, to be part of QMS
documents and covered by the requirements of this guideline.
Within the QMS, NRAs should implement policies and procedures
for identifying, describing, formatting, reviewing, approving, controlling
(e.g. distribution, access and version control, retrieval and use), retaining and
disposing of internally generated documents. QMS documents of external origin
(e.g. regulations, standards, pharmacopoeia and WHO guidelines) should be
subjected to the same requirements as for those that are internally generated, to
the extent possible and practical, depending on the nature and intended use.
Where information technology (IT) is utilized to optimize regulatory
processes for technical and administrative functions, NRAs should ensure that
the system templates, forms and software that are used are identified, reviewed,
approved, controlled and maintained under the same QMS policies and
procedures that apply for other documents.
NRAs should implement a data management, protection (i.e.
confidentiality, loss and integrity) and retention policy/procedure, to define
clearly the types and categories of collected, analysed, evaluated and retained
data. The policy/procedure should provide clear requirements for the format,
medium and duration of retention for data and documents. In addition, there
should be a policy/procedure for NRAs covering the maintenance and retention
of all documents and data.
5.2.6 Planning
5.2.6.1 Quality management system planning
NRAs should plan and document how they will meet the needs and expectations
of their customers and stakeholders, as stipulated in the national laws and
regulations. The plan should include all technical and administrative functions,
processes and activities of the NRA and their respective objectives.
5.2.6.2 Action to address risks and opportunities

When planning for the QMS, the NRA should consider the issues (internal
and external) and requirements of the stakeholders and determine the risks
and opportunities that need to be addressed in the context of the organization.
The NRA should plan actions to address these risks and opportunities, with
assigned roles, responsibilities and authorities. The planned actions should
include a framework for monitoring and evaluating the effectiveness of the
actions taken. The NRA can choose the methods of risk management that suit its
needs. Depending on the size, complexity and regulatory functions of the NRA,
290
Annex 13
principles can be based on the WHO guidelines on quality risk management (9)
and the ISO 31000 standard (10).
5.2.6.3 Quality objectives and planning to achieve them

NRAs should establish quality objectives for relevant regulatory and
administrative functions, for all levels and sections of the NRA and for all
processes needed for the QMS. Where quality objectives are established for
multiple levels within the NRA (e.g. directorate, department, unit or zone),
the objectives should be consistent, to ensure that all levels contribute towards
achieving the overall expectations from legal mandates and of customers. The
quality objectives should be integrated with regulatory objectives, to ensure
that the QMS supports the consistency, effectiveness and efficiency of the NRA.
Quality objectives of the NRA should be consistent with its quality
policy; should contribute to customer satisfaction; and should be relevant to the
regulatory products and services as mandated by the national legal framework.
To the extent possible, quality objectives should be specific, measurable,
attainable, realistic and time-bound (SMART). The QMS should provide the
measures for the NRA to communicate the objectives to designated audiences
within the NRA and the means to monitor and update the objectives.
The QMS should include a plan to ensure that the set objectives will be
met. The planning exercise includes determining the actions that will need to be
taken; the resources that will be required (e.g. human and financial to purchase
equipment and the required supplies); the responsibilities that will be assigned to
staff for specific tasks; the timelines that will be defined for completion of each
step; and the means that will be used for monitoring and evaluating whether or
not the objectives have been achieved.
5.2.6.4 Planning of changes

The NRA should plan for changes to the QMS. The purpose of planning changes
is to maintain the integrity of the QMS and ensure the NRA’s ability to provide
conforming regulatory products and services during any changes. For any
change, the NRA should consider the availability of resources and necessary
allocation or reallocation of responsibilities. This could be done by implementing
an effective change management process within the QMS.
The need for changes can result from changing needs of customers and
other relevant interested parties, for example, new products to be evaluated to
grant market authorization; availability of new information and communication
technologies (ICTs) for a service or process; a move to outsourcing of important
processes; departure of persons in key roles (e.g. due to retirement or job change);
or a move to online service provision.
291
5.2.7 Support and resources

5.2.7.1 Resources
NRAs should determine, document and provide the resources that are needed
to establish, implement, maintain and continuously improve the QMS. The
determination should be done within the organizational context of the NRA
and the scope of the legal mandates on the functions and activities. NRAs should
also determine and document those technical and administrative resources that
need to be provided by external providers (companies and individual experts).
5.2.7.2 Personnel
NRAs should determine, provide and document the personnel and their required
minimum competencies necessary for effective implementation of the QMS
and for effective operation and control of its processes. A focal person or lead
may be appointed to coordinate and monitor the implementation of the QMS at
appropriate levels and functions.
The competencies of all personnel should include a combination of
appropriate education, professional training, experience and behavioural attitude,
as deemed necessary by the NRA. Where the assigned personnel with defined
responsibilities and authorities do not have all the competencies, training plans
should be developed and implemented, with appropriate evaluation criteria for
acquired competencies.
For the purposes of consistency of the QMS, NRA training plans for the
rest of the technical and administrative personnel and functions should be based
on the competency framework or matrix and/or performance appraisal system
coordinated by human resources departments.
Records of evidence of an employee’s competence, including educational
diplomas or degrees; completion of training certificates; resumés; performance
reviews; licences; and other documents should be retained.
The competency framework or matrix should be used in assigning

official and non-official job function hierarchies and relationships (e.g. junior
officer, senior officer or head of unit). The framework should also include the
procedure for designation or qualification of technical officers (e.g. senior or
lead assessor, senior or lead inspector, senior or lead analyst); these should be
supported by requalification procedures.
5.2.7.3 Infrastructure and work environment

NRAs should determine, provide and maintain a documented list of
infrastructures needed for technical and administrative processes in the
execution of the legal mandates. Lists of the following should be maintained to
allow for identification, location, type, quantities, versions, operational status
292
Annex 13
(i.e. in use versus not in use) and plans for qualification, validation, calibration
and maintenance (as applicable):
■■ buildings and associated utilities;
■■ technical (e.g. inspection and testing equipment) and administrative
equipment (e.g. servers, computers, and printers), including
hardware and software;
■■ transportation and logistical resources; and
■■ ICT.
NRAs should determine, provide and maintain the human and physical
factors of the work environment necessary for the operation of technical and
administrative processes and activities within the context of the organizational
structure and national legislation. To the extent that it is practical, the
environment should address social, psychological and physical (i.e. workspace)
conditions to promote work–life balance. Depending on the activities of the
NRA, applicable occupational, health and safety policies and procedures should
be considered for implementation, as provided in ISO 45001 (11).
The NRA and its units should implement and document a policy and
procedure on the management of waste that is generated. The waste management
should be conducted within the recommendations and applicable requirements
of the current version of ISO 14001 (12).
5.2.7.4 Monitoring and measuring resources and equipment

NRAs should determine and document a list of monitoring and measuring
resources and equipment used, to ensure that the regulatory products and
services meet the expected requirements. The equipment should be suitable for
the measurement activity to be undertaken, and maintained to ensure continued
fitness.
For the equipment, including software, that is used in technical
measurements (e.g. inspection and laboratory equipment), NRAs should ensure
that the results obtained from such equipment are valid and that the calibration
of equipment is traceable to national or international measurement standards.
The calibrated equipment should be identified with its calibration status and
safeguarded from adjustments, damage or deterioration.
In the event of measuring equipment being found to be out of calibration,
NRAs should evaluate and document the validity of previous measurement
results obtained from the equipment, and take appropriate actions.
5.2.7.5 Organizational knowledge management and awareness

The NRAs should consider how to determine and manage the organizational
knowledge required to meet the NRA’s present and future needs. Individuals
293
and their experience are the foundation of organizational knowledge. Capturing

their experience and knowledge can generate synergies leading to the creation
of new or updated organizational knowledge. In determining, maintaining and
making organizational knowledge available, NRAs can benefit by (i) learning
from failures and successes; (ii) gathering knowledge from stakeholders, experts
and partners; and (iii) capturing existing internal knowledge.
The tools for maintenance and distribution of organizational knowledge
can include the intranet, libraries, awareness sessions, newsletters and others.
NRAs should ensure that all personnel (both full-time and part-time)
have read and understood the quality policy and the quality objectives that
are relevant to their level in the organization. This should be documented to
verify that personnel understood their contributions to the effectiveness of the
QMS and the benefits of improved performance. NRA personnel should be
aware of the implications of not following policies and procedures established
under the QMS, for example, the release to customers of non-conforming
regulatory products.
5.2.7.6 Internal and external communication

NRAs should determine, implement and document internal and external
communication policies and procedures within the QMS. The policy should
clearly describe “what” to communicate, and define responsibilities and
authorities for communication to the assigned competent personnel. Depending
on the context, nature and intent of the communication, the policy should
describe the level, audience and frequency of the communication, including
the format and medium (e.g. verbal, letter, mail, website or intranet). Social
media and mobile applications are additional tools for communicating with
interested parties.
The communication policy and procedure should be implemented
within the legal framework of the NRA and related national (governmental)
procedures and practices.
5.2.8 Operation
NRAs should ensure that planning of technical and administrative processes is
done effectively, as provided under Section 5.2.6 for all operations within the
scope of the QMS.
5.2.8.1 Customer communication and review of the regulatory

products and services requirements
NRAs should ensure that there is a process of consistent communication with
customers and stakeholders to collect their feedback, inputs and other inquiries
that may be useful for reviewing the pertinence of the offered regulatory products
294
Annex 13
and services. The details of the regulatory products and services offered, including
contingency requirements (such as those applied during natural disasters or
epidemics), should be publicised (e.g. through the NRA website, pre-submission
meetings, or scientific advice), so that customers are aware of the requirements
for submissions to the NRA relating to regulatory products and services.
NRAs should ensure that the requirements and expectations for the
products and services are determined and defined within the applicable national
laws and regulations. To promote public transparency and accountability, the
product and service requirements may include fee schedules and delivery
timelines for product market authorizations, licences, reference standards (e.g.
pharmacopoeia), permits and certificates. This information may be included in
the national guidelines and guidance notes and should be publicly available to
customers and stakeholders.
NRAs should ensure through a review process that requests for
services received from customers are complete and in conformity with service
requirements. A checklist used for such reviews should be documented. When
there is a difference between the requirements for products or services as
requested by the customer and the requirements prescribed by the NRA, this
should be communicated to the customer and resolved before processing the
request. Any verbal request or change in the requirements, either by the NRA or
by the customer, should be confirmed before services are processed.
When the requirements for products and services are changed for any
reason, NRAs should take measures to inform all relevant interested parties.
They should retain evidence of the results of the revisions to the requirements of
products and services, and any new requirements for the products and services
that are provided.
5.2.8.2 Design and development of new products and services

When NRAs plan on implementing new regulatory function(s) due to revision
of the national legal framework, or wish to introduce new regulatory products
and/or services (such as through mobile phone application), the following
process steps should be followed:
1. Determine and document the process(es) that will form part of
the new function, including the stages, steps and control measures
needed through implementation roadmaps or projects. The
determination should include expected reviews, verifications
and validations that the processes are sufficiently robust for the
intended function. NRAs should also determine and document
the competencies, responsibilities and authorities of the project
development team. Where the NRA would not be able to provide
all the required resources, the NRA should document those
295
resources that will be externally sourced. NRAs should determine

the need to involve customers, stakeholders and internal personnel
to ensure that key inputs are collected. NRAs should also assess
whether any of the existing requirements (e.g. timelines or schedule
of fees) are applicable to the new regulatory function, or whether
there is a need to establish additional ones. All documents used
and generated out of these roadmaps should be retained in an
appropriate format and medium.
2. Once the implementation roadmap has been completed, NRAs
should determine and document the inputs, such as performance
indicators, national legal requirements for compliance, and
codes of ethics and professional conduct, as well as the potential
consequences of failure, using a risk-based approach.
3. As defined in the implementation roadmaps, intermediate reviews
(where practical and possible), verification steps (i.e. comparing
the new application/process with a similar proven application/
process) and validation exercises (i.e. testing under intended
user conditions) should be conducted by NRAs, to ensure that
the resulting function or product meets the requirements for the
intended use.
4. The expected outputs of the design and development process
will be in the form of standard operating procedures or service
provision manuals that give the information necessary for all the
processes required to provide intended products and services,
including information to be provided by the customers.
5. Where changes are to be made in the new application or to the
developed products or process(s), these changes will be identified,
reviewed and controlled. A risk-based change management
procedure should be documented and implemented.
5.2.8.3 Externally provided products and services

NRAs should ensure that externally provided products and services (e.g.
subcontracted ICT support, purchased reference standards, or subcontracted
quality control laboratory testing) required for technical and administrative
functions and activities of the NRA, conform to the QMS requirements. Where
national laws and regulations exist for managing the use of public NRA funds in
procurement, for example, a national public procurement act, with procedures
based on amount thresholds for either single sourcing or open/closed bid
competitions and decision levels (i.e. director-general, council, or board level),
the QMS should not duplicate any procedures that are provided for public
296
Annex 13
procurements. However, the NRA should ensure that the public procurement
procedure conforms to the requirements described in the subsequent paragraphs
of this subsection and should close gaps, if any. The NRA should also implement
these requirements when it performs direct procurement.
NRAs should ensure that competence criteria are defined, documented
and implemented for the evaluation, selection, performance monitoring and
re‑evaluation of external providers and suppliers (e.g. NRAs having documented,
well-defined and transparent criteria for the selection and performance
monitoring of external non-staff experts).
When NRAs must perform in-house prequalification of providers,
there should be a documented procedure and policy on the competence criteria
for evaluation, selection, performance monitoring, and requalification. The
prequalification and requalification should focus on the competence of the
individual persons and the institution or company to provide the products and/
or services that meet applicable QMS requirements.
NRAs should implement measures for ensuring that the externally
provided products and services do not adversely affect the organization’s image
and ability to consistently deliver the products and services to the customers.
The NRA should determine which specific controls are to be implemented
for an external provider, and for incoming products and services provided by
them. Control activities that may be considered include inspections, certificates
of analysis or testing, second party audits, evaluation of statistical data and key
performance indicators.
The NRA should clearly communicate the requirements and controls to
be applied to the external provider, and both parties should agree as to what is
required. This understanding of requirements is usually reflected in a technical
service agreement, or through a purchase order or contract. The NRA should
ensure that the requirements communicated to the external providers are
complete and clear and address any potential issues.
5.2.8.4 Service provision

NRAs should carry out their technical and administrative functions for
processing requests for services under controlled conditions. The controlled
conditions should include, as applicable, the points listed next:
■■ use of guidelines, policies and procedures that provide the
requirements for the regulatory products and services, including
those for performance of activities;
■■ NRAs should document and implement measures for reviewing
(peer-review or quality assurance [QA] review), approving and
releasing the output of intermediate processes, to ensure that
297
there are adequate controls for those activities that are involved
in providing conforming products and services. For this purpose,
the following guidelines should be considered for adoption and
implementation, as applicable and to the extent necessary:
–– for technical processes involving review of application
documents for marketing authorization for the assurance of
quality, safety and efficacy, procedures and recommendations
this includes, among others: Good review practices: guidelines for
national and regional regulatory authorities (13); Regulation and
licensing of biological products in countries with newly developing
regulatory authorities (14) and WHO guidelines on evaluation of
similar biotherapeutic products (SBPs) (15);
–– where the NRA has a unit responsible for good practices
inspections, recommendations and technical requirements:
WHO Quality management systems requirements for national
inspectorates (16);
■■ as required, monitoring and measuring resources and equipment
should be available and in use, to ensure that the processes are
effective and controlled. Where measuring equipment must be
used in providing regulatory services of the NRA laboratory,
technical requirements and recommendations from the following
guidelines should be considered for adoption and implementation,
as applicable and to the extent necessary:
–– WHO good practices for pharmaceutical quality control
laboratories (17) for physicochemical testing and WHO good
practices for pharmaceutical microbiology laboratories (18) for
microbiological testing. These two WHO guidelines can be
supported and complemented with the current ISO/IEC 17025
standard (19) and the European Directorate for the Quality

of Medicines and HealthCare (EDQM) Quality management
documents (20);
■■ NRAs should ensure that the provided infrastructure and working
environment are suitable for the operation of both technical and
administrative processes and activities and for the performance of
applicable regulatory functions; and
■■ NRAs should ensure that the appointment of personnel is based
on the required competencies and qualifications and is described
and documented in respective units. This should include the
implementation of control measures to avoid or reduce human
errors through peer- and QA reviews.
298
Annex 13
NRAs should document and implement policies and procedures on

the unique identification and traceability of released regulatory products and
services. As far as practical and possible, these should also be supported by
systematic measures to facilitate traceability of the products and services to the
equipment, software, personnel and location used by the NRA.
5.2.8.5 Property belonging to customers or external providers

NRAs should implement measures to verify, protect and safeguard properties
that belong to customers and stakeholders, including providers, and avoid their
loss, damage and any effects that would make them unsuitable for use. This can
include properties, for example, that may have been seized and quarantined or
used as input for making regulatory decisions. Examples of property include
marketing authorization product dossiers, quarantined products, samples for
testing, intellectual property and personal data.
The NRAs should determine those products and services (e.g. seized
drugs, drug samples collected for analysis, vaccines under release, licences,
market authorizations, permits or certificates to be issued) that can deteriorate
or degrade, and implement appropriate storage conditions.
5.2.8.6 Release and compliance control of products and services

NRAs should document and implement practical procedures on the release
of regulatory products and services through all stages up to and including
the customer. The release process includes defining the responsibilities and
authorities of the involved job functions. These processes should provide an
internal QA procedure to ensure that the released products and services comply
with all planned requirements.
NRAs should document and implement procedures for control
of nonconformances and deviations that are observed or reported. If the
nonconformity is discovered after the product has been delivered to the
customer, the NRA should take appropriate actions to prevent unintended
use or undesired consequences, and take measures such as issuing a recall or
suspension. The QMS should not duplicate any existing procedures in technical
units, such as a laboratory or inspectorate.
5.2.9 Performance evaluation

5.2.9.1 Monitoring and measurement
NRAs should conduct monitoring, measurement, analysis and evaluation of
all planned technical and administrative activities, to determine whether the
intended results, as defined in action plans, workplans or strategic plans, are
being achieved. NRAs should define what needs to be monitored and measured
299
(e.g. characteristics of processes, products, services and potential risks) and the
methods to be used for monitoring, measurement, analysis and evaluation of
the performance and effectiveness of the QMS. The monitoring, measurement,
analysis and evaluation of the NRA performance should be linked to the
planned key performance indicators (or simply indicators), as applicable. The
establishment and implementation of the indicators should be as practical
as possible, to ensure that value is added through monitoring, measurement,
analysis and evaluation activities. Therefore, the indicators or key performance
indicators should have clear, relevant, economic, adequate and monitorable
(CREAM) attributes. NRAs should determine and document the frequency of
M&E of the indicators, from the implemented action and activity plans, as well
as from the performance and evaluation of the QMS. NRAs should ensure that
the M&E framework is consistent across different units, levels and functions of
the organization. The framework should be documented and aligned with the
relevant quality objectives (strategic objectives) of the NRA.
5.2.9.2 Monitoring of customer satisfaction

The NRA should develop methods to systematically seek feedback from a selected
population of customers, or from every customer at planned intervals. Means to
obtain feedback is provided by social and published media such as websites and
message boards, opinion surveys and compliments, suggestions or complaints.
The NRAs should determine the degree of customer satisfaction after the results
of feedback are analysed and evaluated and then act based on this information.
NRAs should document, implement and publish comprehensive policies
and procedures on handling of complaints, in order to provide guidance to
customers and stakeholders on complaint submission, investigation, resolution,
appeal and communication within the national legislations. The procedures
should define the roles and responsibilities of a complainant and the NRA and
specify timelines to effectively manage complaints related to regulatory products

and services.
5.2.9.3 Analysis and evaluation

NRAs should analyse and evaluate monitoring and measurement data and
information, to determine summary performance results of the following:
■■ compliance of regulatory products (e.g. guidelines and software

applications) to quality and validity requirements;
■■ compliance of regulatory services to quality and timeline
commitments and requirements;
■■ degree of customer satisfaction;
300
Annex 13
■■ performance and effectiveness of the QMS for the overall NRA

and/or the QMS for NRA units or functions and the need for
improvements to the QMS;
■■ level of implementation of action or activity plans and strategic
plans at the time of reporting;
■■ effectiveness of the actions taken to address risks and opportunities
(such as strengths, weaknesses, opportunities and threats (SWOT)
analysis); and
■■ performance of external providers (including external technical
experts).
5.2.9.4 Internal audit

NRAs should plan and conduct internal audits (at least once a year), to
verify compliance to the QMS requirements across the organization and to
verify that the QMS is effectively implemented and maintained. An internal
audit programme should have defined planning requirements, frequencies,
methodologies, responsibilities, competencies and reporting. Each internal audit
programme should take into consideration the importance and associated risks
of the processes to be audited, the internal and external changes affecting the
NRA, and the results of previous audits, in order to:
■■ define the audit requirements for the criteria (QMS requirements)

of compliance; scope (functions and departments to be audited);
and methodology (interviews, examination of records, results, and
trends) for each audit. The criteria for compliance may add and
implement a scale for reporting observations (critical, major and
minor), which should be clearly and objectively defined within the
internal audit programme;
■■ select appropriately trained, qualified and competent auditors who
can conduct the audit objectively and impartially. The impartiality
can be achieved by employing auditors that audit those processes in
which they are not involved while serving in the NRA;
■■ ensure that the internal audit reports are submitted to top
management for actions;
■■ take appropriate corrections and corrective actions without delay
and within timelines defined by top management. Where corrective
actions are delayed due to unavailability of required resources,
appropriate risk management plans should be implemented and
documented; and
301
■■ retain records of internal audit programmes and internal audit

reports, including records of corrections and corrective actions.
Further technical guidance on managing and performing internal audits
can be adopted from the current version of ISO 19011 (21).
5.2.9.5 Management review

Top management of NRAs should review the QMS at planned intervals (i.e. at
least once a year), to ensure its suitability, adequacy, effectiveness and alignment
with the strategic direction of the organization as per strategic plans. Ideally,
top management should review the QMS alongside the review of the NRA’s
regulatory plans (activity, action or strategic plans). This will ensure that the
QMS remains integrated into regulatory processes effectively.
QMS reviews should consider inputs as provided in Table A13.2, with
the listed expectations of the outputs to be presented in the minutes of the
meeting (report).
Table A13.2
Inputs and outputs for review meetings
Inputs (to be reviewed) Outputs

Status of actions from previous reviews • Decisions and actions
Changes in internal and external issues that are related to opportunities for
relevant to the QMS improvements
• Decisions and actions related
Information on the performance and to changes required to the QMS
effectiveness of the QMS, including trends in: • Actions on additional resources
• customer satisfaction and feedback from needed to implement
stakeholders; improvement initiatives and
• the extent to which quality objectives have suggested changes in the

been achieved; QMS and in other areas where
• performance on compliance to commitments resources (including human
and requirements for regulatory products and resources) are not adequate
services; • Actions to implement for
• nonconformances and deviations, and the achievement of quality
status of implemented corrective actions; objectives
• results of M&E of indicators/key performance • Responsibilities for follow-up of
indicators; actions on the decisions taken
• results of internal and external audits; and in the meeting
• performance of external providers (including
external technical experts)
Adequacy of resources (financial, human,
equipment and infrastructure)
302
Annex 13

Inputs (to be reviewed) Outputs
Effectiveness of the actions taken to address • Management review meeting
risks and opportunities (such as SWOT or similar minutes to be retained
analysis) as records or reports and
Opportunities for improvements of the QMS communicated appropriately
to internal and external
customers and stakeholders as
per NRA communication policy
M&E: monitoring and evaluation; NRA: national regulatory authority; QMS: quality management system;
SWOT: strengths, weaknesses, opportunities and threats.
Management review agenda (inputs) and meeting minutes should

be retained as records or reports and communicated appropriately to internal
stakeholders as per NRA communication policy.
5.2.10 Improvement
There are different methods to conduct improvement, such as correcting existing
nonconformities and deviations and taking actions to prevent recurrence, or
conducting ongoing, small-step improvement activities based on opportunities
identified either through risk analyses or breakthrough projects. These
improvement activities can lead to innovation, revision and/or improvement of
existing processes, or to the implementation of new processes.
NRAs should implement and document measures to record and react to
nonconformances and deviations by taking actions to control and correct them,
including with related plans for managing related activities, if any. In addition,
NRAs should conduct a root cause analysis (RCA) and evaluate the need to
act in order to avoid recurrence of the nonconformances and deviations in the
affected area, as well as in any similar processes in the organization in which
such nonconformances or deviations could occur. The steps involved in this
process are:
■■ reviewing and analysing the nonconformance or deviation;
■■ determining, to the extent possible, the cause(s) of the
nonconformance or deviation; and
■■ determining whether similar nonconformances exist or could
potentially occur within the affected unit or function and/or other
units of the NRA or functions that have similar processes.
After implementing the corrective action, NRAs should review and
document the effectiveness of the corrective action taken through practical
303
means, including during future internal audits that look for a recurrence of the
same nonconformity. The results of the RCA and the implemented corrective
actions should be used to update the risk and opportunity planning, as applicable.
Where corrective actions lead to changes to the process(es), NRAs should plan
for similar changes to the QMS, supported by a defined change management
plan. NRAs should define the communication of reports on nonconformances
and corrective actions to internal and external customers, as defined in the
communication policy/procedures.
For technical and administrative processes, NRAs should ensure
that the handling of nonconformances, deviations and corrective actions is
consistent across the entire organization. Nonconformances and deviations that
are related to professional misconduct of NRA personnel should be handled
in accordance with conditions of employment and service, including related
national legal provisions.
Improvement can include actions to reduce process variation; increase
the consistency of process outputs, products and services; and improve process
capability. This should be done to enhance the NRA’s performance and give
benefits to its customers and stakeholders. The results from performance
monitoring and evaluation and management reviews should be used to decide
which continual improvement actions should be implemented and what
resources and support should be provided for their implementation by top
management.
6. Quality management system

implementation methodology
6.1 Supporting factors for quality management
system implementation
Full commitment of the head of the NRA and the heads of technical, support
and administrative units (i.e. top management) is necessary for effective
implementation and maintenance of the QMS in NRAs. This commitment
should be supported by demonstrating leadership, management, commitment
and customer focus through all stages of the implementation of the QMS.
The QMS should be designed to be integrated in regulatory processes (i.e. not
standalone); supported with adequate resources (human, financial, equipment
and infrastructure); and created to be simple enough to remain manageable with
the available resources, while being effective enough to support consistency,
effectiveness and efficiency.
Potential mechanisms that can support QMS implementation include:
■■ establishing effective coordination and communication mechanisms;
304
Annex 13
■■ receiving high-level support from top management for QMS

implementation;
■■ establishing high-level ownership and commitment by top
management for QMS implementation and maintenance;
■■ including QMS implementation roadmaps in NRA strategic plans
by top management when submitting to an oversight body (council,
board, committee or ministry of health) for approval, as applicable;
■■ including QMS implementation by the NRA in the national health
strategic plans;
■■ including responsibilities and authorities for contributing to the QMS
in every staff job description and human resources performance
appraisal;
■■ creating and implementing training plans for QMS personnel, based
on NRA competence frameworks;
■■ engaging all customers and stakeholders for communication and
awareness;
■■ implementing applicable ICT tools for internal and external
implementation of QMS and communication of quality policy
awareness;
■■ embedding assigned QMS personnel within regulatory processes,
with the dual responsibilities of regulatory job functions and QMS
responsibilities to support and maintain the QMS in the respective
regulatory unit.
6.2 Situational analysis of quality management system

implementation status in the national regulatory authority
Regardless of the size of NRA, the scope of regulatory functions and the
NRA organizational model (i.e. discrete, decentralized or centralized), the
recommendations in Table A13.3 for gap and situational analyses should be
considered when implementing QMS and when planning for continuous
improvement of a QMS that is already implemented. NRAs should first identify
existing gaps and determine the level of implementation of the QMS, with the
use of Appendix 2, Table A13.3 and self-benchmarking results. Table A13.3
categorizes the key aspects relevant to the different stages of the QMS, as
listed next.
■■ Non-existing QMS: NRAs should focus on ensuring that processes
and activities are performed consistently, regardless of the
personnel or location of execution. This may be covered for certain
areas with automated systems (such as laboratory information
305
management systems for laboratories or e-performance appraisals

for human resources). NRAs should prioritize development and
implementation of procedures for areas based on the related risks
with respect to the products and services; the affected quality
objectives; and the availability of resources for maintenance of the
procedures. This means that not every area should be prioritized
at the same time for QMS development and implementation (for
NRAs without an implemented QMS).
■■ Existing QMS without implementation: the focus at this stage
should be on ensuring that consistent procedures are developed
and implemented for the QMS to support regulatory processes
effectively. Careful consideration should be given at this stage
to objectively addressing the activities for gap identification
and validation; these steps would also be useful for NRAs that
have already developed and implemented a QMS. NRAs should
ensure that the person(s) identifying the gaps have the necessary
competence and that top management fully supports the process.
The review should be done to cover all areas in which the QMS
has been implemented, and the scope should be limited to records,
reports or other means of verification that procedures have been
implemented and are being used to the full extent as intended.
The outcome of this review should be a root cause analysis with
proposed measures to implement; these measures should take into
consideration the availability of resources and associated risks of
delayed implementation.
■■ Ineffective implementation of QMS: addressing this stage is
considered useful once the first two stages are addressed for the
respective processes and activities. This stage focuses on the main

objectives of the QMS, namely, to ensure that the NRA is being
effective in supporting regulatory processes and activities; providing
regulatory products and services; and achieving strategic objectives.
Therefore, it is important that the QMS is uncomplicated/
unsophisticated and manageable enough in its implementation
and maintenance to avoid diverting NRA time and resources on
the QMS instead of delivering regulatory products and services to
the customers as provided by national legal mandates. Increasing
the effectiveness and efficiency of the QMS may also involve the
adoption and implementation of ICT to remove human errors while
promoting consistency; reducing time for implementation and
recording; and providing long-term cost reductions.
306
Annex 13
6.3 Gap analysis for developing a roadmap for quality

management system implementation
The information in Table A13.3 can be used to identify gaps and define activities
to be done for QMS implementation, based on the recommendations of this
guideline. The planning, prioritization and implementation should be as
practical as possible and be determined by the NRA, taking into consideration
the availability of resources and priorities for the provision of regulatory products
and services.
307
308
Table A13.3

Gap analysis
Guideline Existing system Stage 1 (non-existing QMS) Stage 2 (existing QMS Stage 3 (ineffective
section without implementation) implementation of QMS)
Needed documents for Implemented evidence Effectiveness and

consistency (by records, reports) efficiency
5.2.1 Linking and integration of NRAs should perform an Where consistency and NRAs should conduct
Introduction overall QMS to quality systems organization-wide review for operational interfaces have reviews to identify gaps
and (automated) software for: consistency of practice by different been implemented and in the effectiveness and
• registration and market staff using the same processes and supported under the QMS, efficiency of the QMS
authorization the existing system. This review NRAs should conduct reviews interventions with respect
• laboratory inspections and can be used identify a consistency to identify gaps in the level to the achievement of
licensing gap for QMS intervention and of implementation of the the intended objectives
• vigilance document development. Once the QMS documents. This should based on evidence from
• market surveillance and reviews are completed and gaps be evaluated by reviewing stage 2 outputs. When
control established, reviews should be records and reports gaps have been identified,
• clinical trials oversight done to determine whether the generated from the systems NRAs should revise the
• lot release existing systems and/or software and/or software, to establish QMS implementation
• environmental (waste) have operational interfaces consistency and operational documents to ensure that
management between one another when they linkages for the same they are effective and
• occupational health and all contribute towards achieving objectives. Where gaps are efficient in contributing
safety the same objective; these reviews found to exist, NRAs should towards the achievement
• finance and accounting can help to identify operational perform RCA and implement of the objectives.
• e-procurement gaps in interfaces. The QMS should changes as appropriate to
• planning, monitoring and be used to link the processes and stage 1 QMS interventions.
evaluation activities between systems and/or
• human resource perfor software by providing documents.
mance appraisal, training
and staff/talent retention
• others

5.2.2 Scope of Documented statement To identify gaps in the scope of NRAs should review the When identifying gaps for
the QMS defining the scope of the QMS, NRAs should review level of implementation QMS revision, NRAs should
the regulatory functions, for the existence of consistent of the QMS across all units review the effectiveness
physical locations, processes, documented scope statements, (including administrative) and efficiency of the scope
regulatory products and which includes all areas, locations and locations, to identify of the QMS in facilitating
services of the NRA and processes. gaps in the implementation the provision of required
of the scope. products and services.
5.2.3 Adequate description NRAs should review and identify NRAs should review the NRAs should review the
Organizational and mandates of NRAs in gaps in the consistency of planning reports and contribution of QMS
context of the terms of: how issues for planning are records from different documents in making the
NRA • ability to define internal determined among different units, to identify gaps in planning more effective
and external issues units of the organization. QMS implementation of QMS and efficient to identify
and customers and documents should be developed documents. Where gaps gaps. Identified gaps
stakeholders and implemented to establish are identified, RCA should should be addressed by
consistency. be performed to ensure implementing changes to
that procedures are QMS documents.
implemented.
Annex 13
309
310


5.2.4 Adequate description NRAs should review the NRAs should review the NRAs should review
Leadership, and mandates of NRAs in consistency of supervisory and level of implementation of the effectiveness
management terms of: reporting structures, consistency existing QMS procedures, and efficiency of the
and • ability to develop and in developing and implementing to ensure consistency in procedures in supporting
organization implement organizational quality objectives, and consistency organizational structures, leadership, management
structure of assigned QMS responsibilities job titles, reporting and organization
• ability to develop and and authorities across units lines, quality policies, processes to identify
implement quality policy and locations, to identify gaps QMS responsibilities and gaps in the existing QMS.
and customer-focused in leadership, management or authorities across the units Procedures should be
initiatives organization. QMS procedures and locations, to identify revised to ensure that they
should be implemented to ensure gaps in implementation of are effective and efficient
• ability to assign QMS
that leadership, management procedures. RCA should be in supporting the NRA
responsibilities and
and organization processes done to determine changes and all its units in having
authorities to personnel
are carried out consistently in that would improve levels leadership, management
implementation of the QMS of implementation of QMS and organization that
procedures. is able to deliver on the
regulatory products and
services.

5.2.5 Documents under the NRAs should identify gaps by NRAs should review NRAs should review
Document QMS that are internally reviewing the consistency in the the records in units and the effectiveness
and data generated or from external development, review, approval, locations, to identify gaps and efficiency of the
management origins for: version and access control, in the implementation of procedures in identifying
• regulations distribution, storage, retrieval existing procedures for gaps in the management
• guidelines and disposition of documents, management of documents. of documents. Procedures
as applicable across all units and RCA should be done to should be revised to
• Policies
locations of the organization. determine measures to ensure that they are more
• notes and guidance
Where gaps exist, procedures promote implementation effective and efficient
• procedures (SOPs/work should be implemented to ensure of existing procedures. in the management of
instructions) that documents are managed NRA documents. NRAs
• lists, registers, logbooks consistently across all units and can consider the use of
• databases and locations of the NRA. IT in the management of
spreadsheets documents, depending
• templates, forms on the availability of
• application documents resources, size of NRA and
(dossiers, files) complexity of documents
• financial, accounting, to be managed.
procurement and HR
records
• reports, letters, emails,
permits, licences,
Annex 13
certificates, others
311
312


5.2.6 Planning Linking and integration of NRAs should review the NRAs should review the To identify gaps with
planning in quality systems consistency in the planning, level of implementation of existing procedures,
and (automated) software monitoring and evaluation procedures for consistency NRAs should review the
for objectives in: of technical, administrative in planning, monitoring effectiveness and efficiency
• technical activities and support activities, with and evaluating of technical of the QMS procedures
• support and associated risk and change and support activities. in support of planning,
administrative activities management plans to identify To identify gaps for QMS monitoring and evaluating
gaps across all units and revision, the review should of activities, risks and
• M&E
locations. QMS procedures evaluate the consistency changes. QMS procedures
should be implemented to in implementation of risk should be revised or
ensure that all planning, and change management replaced with automated
monitoring and evaluating of plans, based on existing systems, based on the
technical and support activities records and reports. RCA complexity and size of
are done consistently and should be done to ensure the NRA and its planning
with related risk and change that procedures are activities.
management plans. implemented.

5.2.7 Support Adequate and quality NRAs should review the NRAs should review the To identify gaps for QMS
and resources resources for: consistency in the allocation records to identify gaps in revisions, NRAs should
• personnel and of personnel, training in QMS, levels of implementation of review the effectiveness
competencies knowledge sharing, use of ICT existing procedures for QMS and efficiency of the
• organizational knowledge and communication of QMS personnel, competencies, procedures in ensuring
management requirements to identify gaps knowledge management, that there are adequate
for QMS implementation. ICT, work environment and and quality personnel,
• ICT
Procedures should be communication across QMS competencies,
• work environment
implemented to ensure all units and locations. knowledge management,
• communication and consistency across all units RCA should be done to ICT, workspace,
awareness and locations in allocation of ensure procedures are communication and
personnel, training of staff in implemented. awareness of QMS
QMS implementation, use of implementation.
intranets and other ICT tools Procedures should be
and communication. revised to ensure that
they are effective and
increase efficiency in their
implementations.
Annex 13
313
314


5.2.8 Process approach focused To identify gaps for QMS NRAs should review the NRAs should review
Operation on the regulatory products implementation, NRAs should records from technical and and identify gaps in the
and services and on NRA review the consistency in administrative units and effectiveness and efficiency
quality objectives the conduct of technical and locations, to identify gaps of the implemented
administrative activities in in the implementation of procedures and quality
providing products, services existing procedures. RCA systems in facilitating
and operational interfaces or should be performed, the provision of products
linkages among processes that and measures should be and services that meet
contribute to the same product, put in place to ensure requirements and support
service or quality objective. full implementation the achievement of the
Where gaps exist, procedures of procedures across all objectives. Procedures
should be implemented affected units and and systems should be
to ensure consistency and locations. revised to ensure that they
operational linkages of are effective and increase
processes. efficiency in the processes
for providing products and
services, and for supporting
the achievement of NRA
objectives.

5.2.9 M&E framework NRAs should review and NRAs should review NRAs should review
Performance with performance determine gaps in consistency and identify gaps in the and identify gaps in the
evaluation indicators for: in the M&E activities across all level of implementation effectiveness and efficiency
• products and services units and locations for QMS of existing procedures of the implemented QMS
requirements implementation. Where gaps and systems of the QMS procedures and systems
• quality objectives in consistency are identified, for the M&E framework. used for M&E. These
• customer complaints procedures should be RCA should be done to procedures and systems
implemented to ensure that all inform revised measures should be evaluated to
• QMS
M&E of performance indicators is for the implementation of ensure that their output
• resources (human,
done consistently across different procedures and systems provides evidence that
financial, ICT, equipment
units and locations of NRAs. across all affected NRA is useful for planning of
and infrastructure)
units and locations. continuous improvements.
• risk and opportunity Where gaps exist,
management NRAs should revise the
procedures and systems
to ensure that they
are more effective and
efficient in supporting
M&E of performance
indicators across all units
and locations of the
organizations.
Annex 13
315
316


5.2.10 Evidence-based NRAs should review and identify NRAs should review and NRAs should review
Improvement improvements gaps in the consistency of identify gaps in levels of and identify gaps
handling and prioritization of implementation of QMS in the effectiveness
improvements across the entire procedures for handling and and efficiency of the
organization. Where there are implementing improvements procedures in facilitating
inconsistencies, procedures across all units and locations timely implementation
should be implemented to of the organization. Where of improvements.
ensure that all proposals for gaps are identified, RCA Procedures should be
improvements are submitted should be done with revised to ensure that
with evidence and evaluated revised measures for the they are more effective
with respect to priorities implementation of the and efficient in facilitating
and availability of resources. procedures. timely implementation of
Procedures for improvement improvements.
should define responsibilities
and authorities for handling,
planning and implementation
of improvements.
HR: human resources; ICT: information and communication technology; IT: information technology; M&E: monitoring and evaluation; NRA: national regulatory authority;
QMS: quality management system; RCA: root cause analysis; SOP: standard operating procedure.
Annex 13
6.4 Quality management system development

and implementation roadmap
The QMS roadmap for NRAs will depend on the respective stages of
implementation. The roadmap will be used to identify activities to be done;
required resources; competencies of personnel; responsibilities and authorities;
timelines (time frame); and prioritization based on the needs of the NRA with
respect to the regulatory products and services as mandated by national laws
and regulations. Table A13.4 summarizes the steps in the development and
implementation roadmap for QMS.
Table A13.4
Development of quality management system implementation roadmap
Steps Activity Responsible

1 Assign resources (personnel, financial, equipment and Top management
infrastructure).
2 Use Table A13.3 and results from self-benchmarking Assigned staff/
to determine the status of the QMS and submit report consultant
to top management, noting activities and areas that
require actions.
3 Prioritize activities based on availability of resources Top management
(internal and external); risks of non-implementation;
and regulatory products and services, as mandated by
national laws and regulations.
4 Allocate responsibilities and authorities with timelines Top management
for development, review, approval, implementation, and assigned staff/
and monitoring and evaluation of prioritized QMS consultant
requirements.
5 Validate the prioritization of QMS requirements, Top management
timelines, responsibilities and authorities with NRA and assigned staff/
staff, through collection of input and feedback to consultant
promote ownership of QMS implementation.
6 Consolidate the feedback and input into an activity/ Assigned staff/
action plan, as a roadmap for QMS implementation for consultant
the NRA.
7 Integrate the QMS roadmap (activity/action plan) into Top management
the NRA organizational activity/action plans, the NRA
strategic plans, and the ministry of health strategic
plan/policy, as applicable.
NRA: national regulatory authority; QMS: quality management system.
317
6.5 Activity plan for quality management

system implementation
Appendix 1 provides a typical action plan for the systematic development and
implementation of a QMS. The plan provides a linkage between the section/
subsections of this guideline and includes examples of documents and records
to be established to demonstrate adequate implementation of the QMS.
References
1. Thirteenth General Programme of Work 2019–2023. Promote health, keep the world safe, serve
the vulnerable. Geneva: World Health Organization; 2019 (https://apps.who.int/iris/bitstream/
handle/10665/324775/WHO-PRP-18.1-eng.pdf, accessed 4 November 2019).
2. World Health Organization. Leadership priorities (https://www.who.int/about/resources_planning/
WHO_GPW12_leadership_priorities.pdf, accessed 12 November 2019).
3. Resolution WHA67.20. Regulatory system strengthening for medical products. In: Sixty-seventh
World Health Assembly, Geneva, 19–23 May 2014. Geneva: World Health Organization; 2014
(http://apps.who.int/medicinedocs/documents/s21456en/s21456en.pdf, accessed 12 November
2019).
4. World Health Organization. Essential medicines and health products. WHO Global Benchmarking
Tool (GBT) for evaluation of national regulatory systems (https://www.who.int/medicines/
regulation/benchmarking_tool/en/, accessed 15 November 2019).
5. ISO 9001:2015(en). Quality management systems – requirements (https://www.iso.org/obp/
ui/#iso:std:iso:9001:ed-5:v1:en, accessed 5 November 2019).
6. Good regulatory practices: guidelines for national regulatory authorities for medical products.
Working document QAS/16.686. Geneva: World Health Organization; 2016 (https://www.who.int/
medicines/areas/quality_safety/quality_assurance/GoodRegulatory_PracticesPublicConsult.pdf,
accessed 12 November 2019).
7. ISO 9000:2015(en). Quality management systems – fundamentals and vocabulary (https://www.
iso.org/obp/ui/#iso:std:iso:9000:ed-4:v1:en, accessed 12 November 2019).
8. Compendium of quality management system (QMS) technical documents for harmonization of
medicine regulation in the East African Community. Arusha: East African Community Secretariat;
2014 (EAC/TF-MED/QMS/FD/COM/N1R0; http://apps.who.int/medicinedocs/documents/
s22311en/s22311en.pdf, accessed 12 November 2019).
9. WHO guidelines on quality risk management. In: WHO Expert Committee on Specifications for
Pharmaceutical Preparations: forty-seventh report: Annex 2. Geneva: World Health Organization;
2013 (WHO Technical Report Series, No. 981; https://www.who.int/medicines/areas/quality_
safety/quality_assurance/Annex2TRS-981.pdf, accessed 12 November 2019).
10. ISO 31000:2018(en). Risk management – guidelines (https://www.iso.org/obp/ui/#iso:std:
iso:31000:ed-2:v1:en, accessed 12 November 2019).
11. ISO 45001:2018(en) Occupational health and safety management systems – requirements
with guidance for use (https://www.iso.org/obp/ui/#iso:std:iso:45001:ed-1:v1:en, accessed 12
November 2019).
12. ISO 14001:2015(en). Environmental management systems – requirement with guidance for use
(https://www.iso.org/obp/ui/#iso:std:iso:14001:ed-3:v1:en, accessed 12 November 2019).
318
Annex 13
13. Good review practices: guidelines for national and regional regulatory authorities. In: WHO
Expert Committee on Specifications for Pharmaceutical Preparations: forty-ninth report: Annex 9.
Geneva: World Health Organization; 2015 (WHO Technical Report Series, No. 992; https://www.
who.int/medicines/areas/quality_safety/quality_assurance/Annex9-TRS992.pdf?ua=1, accessed
12 November 2019).
14. Regulation and licensing of biological products in countries with newly developing regulatory
authorities. In: WHO Expert Committee on Biological Standardization: forty-fifth report: Annex 1.
Geneva: World Health Organization; 1995 (WHO Technical Report Series, No. 858; https://www.
who.int/biologicals/publications/trs/areas/vaccines/regulatory/WHO_TRS_858_A1.pdf?ua=1,
accessed 12 November 2019).
15. WHO guidelines on evaluation of similar biotherapeutic products (SBPs). In: WHO Expert
Committee on Biological Standardization: sixtieth report: Annex 2. Geneva: World Health
Organization; 2013 (WHO Technical Report Series, No. 977; https://www.who.int/biologicals/
publications/trs/areas/biological_therapeutics/TRS_977_Annex_2.pdf?ua=1, accessed 12
November 2019).
16. Quality management systems requirements for national inspectorates. In: WHO Expert
Committee on Specifications for Pharmaceutical Preparations: fifty-fourth report: Annex 5.
Geneva: World Health Organization; 2020 (WHO Technical Report Series, No. 1025xxx ).
17. WHO good practices for pharmaceutical quality control laboratories. In: WHO Expert Committee
on Specifications for Pharmaceutical Preparations: forty-fourth report: Annex 1. Geneva: World
Health Organization; 2010 (WHO Technical Report Series, No. 957; https://www.who.int/
medicines/areas/quality_safety/quality_assurance/GoodpracticesPharmaceuticalQualityControl
LaboratoriesTRS957Annex1.pdf, accessed 12 November 2019).
18. WHO good practices for pharmaceutical microbiology laboratories. In: WHO Expert Committee
on Specifications for Pharmaceutical Preparations: forty-fifth report: Annex 2. Geneva: World
Health Organization; 2011 (WHO Technical Report Series, No. 961; http://apps.who.int/
medicinedocs/documents/s18682en/s18682en.pdf, accessed 12 November 2019).
19. ISO/IEC 17025. General requirements for the competence of testing and calibration laboratories.
Geneva: International Organisation for Standardisation; 2017 (https://www.iso.org/files/live/
sites/isoorg/files/store/en/PUB100424.pdf, accessed 12 November 2019).
20. Council of Europe. European Directorate for the Quality of Medicines and HealthCare. Quality
management (QM) documents (https://www.edqm.eu/en/quality-management-guidelines-86.
html, accessed 12 November 2019).
21. ISO 19011:2018(en). Guidelines for auditing management systems (https://www.iso.org/obp/
ui/#iso:std:iso:19011:ed-3:v1:en, accessed 13 November 2019).
319
Appendix 1
References to the WHO Global Benchmarking Tool,
revision VI
The WHO Global Benchmarking Tool (GBT) (4) is used to assess the level of
implementation of a quality management system (QMS) in a national regulatory
authority (NRA). The QMS indicator, RS05, consists of 14 subindicators that
are used to identify the degree of QMS implementation and the existing gaps
across the NRA.
Subsection in GBT VI – QMS Related GBT VI subindicators (4)

QMS guideline subindicators (4)
5.2.1 Introduction RS05.06
5.2.2 Scope of the RS05.02 RS01.01, RS01.02
QMS VL01.01
MA01.01
MC01.01
LI01.01
RI01.01
LT01.01
CT01.01
LR01.01
5.2.3 Organizational RS05.06, RS05.08 RS02.04, RS03.04, RS07.04
context of the NRA
5.2.4 Leadership, RS05.01, RS05.02, RS02.01, RS04.01

management and RS05.03, RS05.04 VL02.01, VL03.02
organization MA02.01, MA03.02
MC02.01, MC03.02
LI02.01, LI03.02
RI02.01, RI03.02
LT02.01, LT04.02
CT02.01, CT03.02, CT04.04
LR03.02
320
Annex 13
Table continued
Subsection in QMS GBT VI – QMS Related GBT VI subindicators (4)
guideline subindicators (4)
5.2.5 Document RS05.07 RS01.04, RS01.05, RS01.08, RS09.06, RS09.08
and data VL03.04, VL04.01, VL04.02, VL04.03
management MA03.04, MA04.01, MA04.02, MA04.03,
MA04.10, MA05.02, MA06.01
MC03.04, MC04.01, MC04.02, MC04.03,
MC04.05, MC04.07, MC04.08, MC05.01,
MC05.02
LI03.04, LI04.01, LI05.01, LI06.01
RI03.04, RI04.01, RI04.02, RI04.04, RI04.05,
RI04.06, RI05.01, RI05.02
LT03.02, LT03.04, LT04.04, LT06.02, LT06.03,
LT08.01
CT03.04, CT04.05, CT04.06, CT04.07,
CT06.01
LR01.02, LR03.04, LR04.03
5.2.6 Planning RS05.02 RS03.03, RS04.05
VL04.04, VL04.08
MA01.12, MA04.06, MA04.07, MA06.02
MC04.04, MC05.03
LI04.03, LI05.02
RI04.03, RI05.05
LT03.01, LT08.04
CT06.02, CT06.04
LR06.04
5.2.7 Support and RS05.04, RS05.14 RS02.02, RS06.01, RS06.02, RS08.01, RS08.02,
resources RS08.03, RS09.03, RS09.07, RS09.09
VL02.02, VL03.01, VL03.02, VL03.03, VL06.01,
VL06.02, VL06.03
MA02.02, MA03.01, MA03.03, MA05.01,
MA05.03, MA05.04
MC02.02, MC03.01, MC03.03, MC06.01,
MC06.02, MC06.03
LI02.02, LI03.01, LI03.03, LI06.02
RI02.02, RI03.01, RI03.03, RI06.01, RI06.02,
RI06.03, RI06.04
LT03.03, LT04.01, LT04.03, LT05.01, LT05.02,
LT06.05, LT07.01, LT09.01, LT09.02, LT09.03
CT02.02, CT03.01, CT03.03, CT05.02
LR02.02, LR03.01, LR03.03, LR05.01, LR05.02,
LR06.01
321
Table continued
Subsection in QMS GBT VI – QMS Related GBT VI subindicators (4)
guideline subindicators (4)
5.2.8 Operation RS05.06, RS05.09 RS02.03, RS04.02, RS04.03, RS06.03,
RS06.04, RS09.05, RS09.07
VL04.05, VL04.06, VL04.07
MA01.09, MA01.10, MA01.11, MA01.13,
MA04.05, MA04.08, MA04.09, MA04.10
MC01.06, MC01.07
LI01.04, LI04.02, LI04.04
RI01.04, RI05.03
LT02.02, LT06.01, LT06.04, LT10.01
CT01.09, CT01.10, CT04.01, CT04.02,
CT04.03, CT05.01
LR04.01, LR04.02
5.2.9 Performance RS05.10, RS05.11, RS01.09, RS10.01, RS10.02
evaluation RS05.12, RS05.13 VL05.02
MA04.06, MA06.02
MC04.06, MC05.03
LI05.02
RI05.04, RI05.05
LT08.02, LT08.03, LT08.04
CT06.02, CT06.04
LR06.02, LR06.04
5.2.10 Improvement RS05.05 LR06.02
GBT: WHO Global Benchmarking Tool (4); NRA: national regulatory authority; QMS: quality management system.
322
Appendix 2
Activity plan for quality management system implementation
Step Activity Subsection Recommendations of documents Responsibility
of QMS and records to be established within the NRA
guideline
1. Appointment of QMS focal 5.2.4.3 Official letters of appointment with defined Head of the NRA
person(s) or lead(s) responsibilities and authorities in the QMS
2. QMS focal person(s) 5.2.7.2 • Competency matrix for QMS focal person(s)/ Top management
understands the QMS lead(s)
requirements • Training plans for competency gaps in QMS
implementation
• Training records of QMS focal person(s)/lead(s)
• Training/orientation records in development
and implementation of QMS documents (quality
manual, standard operating procedures and/or
forms and templates)
3. QMS focal person(s)/lead(s) 6.1 • Documented gap or situation analysis report • Top management
conducts a gap analysis of • Documented roadmap with resources, timelines • QMS focal
the current system based on and responsibilities (part of NRA strategic and person(s)/ lead(s)
Tables A13.3 and A13.4 of action plans)
the guideline and develops
Annex 13
a QMS action plan (as part of
the strategic plan)
323
324

guideline
4. QMS focal person(s)/lead(s) 5.2.7.5 Accessible and available QMS orientation and QMS focal
conduct(s) orientation awareness sessions records and materials in person(s)/ lead(s)
and awareness sessions appropriate format
for NRA employees on
QMS development and
implementation (with roles and
responsibilities)
5. • Establishment of NRA current 5.2.3.1 • Documented official organizational chart covering • Top management
context (SWOT analysis), if NRA governance and top management and • QMS focal
already available internal and external operational relationships person(s)/ lead(s)
• Determination of the • Documented description of internal and external
comprehensiveness of the issues, including SWOT analysis of the NRA (with
legal provisions (Acts and defined customers and stakeholders based on
regulations) in describing legal provisions)
interested parties relevant to • Documented description of internal and external
the QMS customers and stakeholders, with their respective
• Identification of QMS needs and expectations (if not adequately
processes, sequences, linkages described in the national legislations)
and interdependencies • Documented statement of scope for the QMS
• Determination of the scope of • Documented flowcharts, process maps and their
the QMS and relationships of operational linkages for all processes under the
its processes scope of the QMS, with related products and
services
guideline
6. Documentation of a quality 5.2.4.2 Documented, accessible (publicly) and available • Top management
policy within the context and quality policy understood by NRA staff • QMS focal
strategic direction of the NRA person(s)/ lead(s)
7. Use information from step 5 5.2.6.2 • Documented and controlled registry of assessed • Top management
above, as input, to determine and categorized risks and opportunities (from • QMS focal
risks and opportunities and SWOT analysis) person(s)/ lead(s)
develop risk and opportunity • Risk and opportunity responsibility matrix (based
management plans on responsible, accountable, consulted and
informed [RACI] principles)
8. Development and 5.2.4.2 Documented quality objectives (and their short- Top management
documentation of SMART and long-term targets), resources, responsibilities
quality objectives, including (ideally in NRA’s strategic plan) and M&E indicators
a plan for M&E with related
required resources
Annex 13
325
326

guideline
9. Development of new or 5.2.7.4 Documented and implemented procedures for: • Top management
harmonization with existing • staff recruitment (based on a defined competency • QMS focal
procedures for control of framework for different levels and positions), person(s)/ lead(s)
measuring equipment, training and retraining based on established gaps
organizational knowledge as per organizational competency framework
management, personnel
• management and maintenance of measuring
training and communication
equipment, as applicable in making regulatory
decisions (laboratory and/or inspection
equipment)
• management of organizational knowledge (e.g.
retirements, resignations and new knowledge
acquisition)
• management of internal and external
communication of regulatory decisions, products,
services and other engagements with customers
and stakeholders
• use of IT in technical and administrative processes,
including management of templates used in the
software or equipment or in other procedures
needed to manage resources as described in the
guideline
guideline
10. Development of new or 5.2.8 Documented and implemented procedures for all • Top management
harmonization with existing applicable technical and administrative processes • QMS focal
procedures for all processes in within the NRA and that contain the appropriate person(s)/ lead(s)
technical and administrative level of detail based on the complexity of the
units of the NRA processes and associated risks. The procedures
should address all activities that are involved in
provision of products and services as mandated by
national legislations
11. Development of procedures 5.2.9 • Documented and implemented procedures for • Top management
for monitoring of customer customer complaints and satisfaction, along with • QMS focal
satisfaction, internal audit, publications of guidance to the public on the person(s)/ lead(s)
management review and procedures and communication
complaints handling, and put • Documented and implemented internal audit
them into practice programmes
• Documented and implemented regular reviews
of QMS implementation and performance by top
management
12. Development of procedures for 5.2.10 Documented and implemented procedures for • Top management
corrections, corrective actions corrective actions and change management, • QMS focal
and improvements, and put along with a link for updating risk and opportunity person(s)/ lead(s)
them into practice management plans
Annex 13
IT: information technology; M&E: monitoring and evaluation; NRA: national regulatory authority; QMS: quality management system; SMART: specific, measurable, attainable,
realistic and time-bound; SWOT: strengths, weaknesses, opportunities and threats.
327

Trs1025-Annex13 WHO Technical Report 2020

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Trs1025-Annex13 WHO Technical Report 2020

Uploaded by

Copyright:

Available Formats

Annex 13

WHO guideline on the implementation of quality

Implementation of the Thirteenth World Health Organization (WHO) General

1. Describe principles for implementing a QMS to support planning,

■■ discrete: two or more institutions involved in partial or full

■■ centralized: one NRA with full legal mandates to enforce national

product. Output of an organization that can be produced without any transaction

5. Quality management system requirements

The PDCA cycle requires NRAs to carry out planning, performing

■■ Customer focus: the primary focus of a QMS is to meet customer

■■ Leadership: leaders at all levels establish unity of purpose and

administrative and, if not managed properly, may have negative

5.2 Quality management system requirements

Subsection Summary of requirements for implementation of a quality

Table A13.1 continued

in the QMS to support continuous improvements based on

5.2.2 Scope of the quality management system

clear on the inclusions and exclusions (with justifications as applicable), without

5.2.3 Organizational context of the national regulatory authority

expectations of customers and stakeholders. The determination of internal and

NRAs may use national legal provisions to identify different types of

5.2.3.2 Quality management system processes

5.2.4 Leadership, management and organization

■■ integrating QMS requirements into the regulatory processes and

5.2.4.2 Quality policy

directions and plans and provide a framework for setting quality

copies of the quality policy should be available to customers and stakeholders,

5.2.4.3 Roles, responsibilities and authorities

5.2.5 Document and data management

NRAs should consider all published materials, either on intranets or

processes and activities of the NRA and their respective objectives.

5.2.6.2 Action to address risks and opportunities

5.2.6.3 Quality objectives and planning to achieve them

5.2.6.4 Planning of changes

5.2.7 Support and resources

The competency framework or matrix should be used in assigning

5.2.7.3 Infrastructure and work environment

5.2.7.4 Monitoring and measuring resources and equipment

5.2.7.5 Organizational knowledge management and awareness

and their experience are the foundation of organizational knowledge. Capturing

5.2.7.6 Internal and external communication

procedures and practices.

5.2.8.1 Customer communication and review of the regulatory

5.2.8.2 Design and development of new products and services

resources that will be externally sourced. NRAs should determine

procedure should be documented and implemented.

5.2.8.3 Externally provided products and services

5.2.8.4 Service provision

standard (19) and the European Directorate for the Quality

NRAs should document and implement policies and procedures on

5.2.8.5 Property belonging to customers or external providers

5.2.8.6 Release and compliance control of products and services

5.2.9 Performance evaluation

5.2.9.2 Monitoring of customer satisfaction

specify timelines to effectively manage complaints related to regulatory products

5.2.9.3 Analysis and evaluation

■■ compliance of regulatory products (e.g. guidelines and software

■■ performance and effectiveness of the QMS for the overall NRA

5.2.9.4 Internal audit

■■ define the audit requirements for the criteria (QMS requirements)

■■ retain records of internal audit programmes and internal audit

5.2.9.5 Management review

Inputs (to be reviewed) Outputs

• the extent to which quality objectives have suggested changes in the