0% found this document useful (0 votes)

86 views350 pages

WHO Global Benchmarking Tool (GBT) : For Evaluation of National Regulatory System of Medical Products

Uploaded by

Gerardo Soto

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

86 views350 pages

WHO Global Benchmarking Tool (GBT) : For Evaluation of National Regulatory System of Medical Products

Uploaded by

Gerardo Soto

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 350

WHO Global

Benchmarking
Tool (GBT)
for Evaluation of National Regulatory
System of Medical Products
Revision VI
WHO Global Benchmarking Tool (GBT)
for Evaluation of National Regulatory
System of Medical Products
Revision VI
WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory systems of medical products,
revision VI

ISBN 978-92-4-002024-5 (electronic version)

ISBN 978-92-4-002025-2 (print version)

© World Health Organization 2021

Some rights reserved. This work is available under the Creative Commons Attribution-NonCommercial-
ShareAlike 3.0 IGO licence (CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/igo).

Under the terms of this licence, you may copy, redistribute and adapt the work for non-commercial purposes,
provided the work is appropriately cited, as indicated below. In any use of this work, there should be no
suggestion that WHO endorses any specific organization, products or services. The use of the WHO logo is
not permitted. If you adapt the work, then you must license your work under the same or equivalent Creative
Commons licence. If you create a translation of this work, you should add the following disclaimer along with
the suggested citation: “This translation was not created by the World Health Organization (WHO). WHO is not
responsible for the content or accuracy of this translation. The original English edition shall be the binding and
authentic edition”.

Any mediation relating to disputes arising under the licence shall be conducted in accordance with the
mediation rules of the World Intellectual Property Organization (http://www.wipo.int/amc/en/mediation/rules/).

Suggested citation. WHO Global Benchmarking Tool (GBT) for evaluation of national regulatory systems of
medical products, revision VI. Geneva: World Health Organization; 2021. Licence: CC BY-NC-SA 3.0 IGO.

Cataloguing-in-Publication (CIP) data. CIP data are available at http://apps.who.int/iris.

Sales, rights and licensing. To purchase WHO publications, see http://apps.who.int/bookorders. To submit
requests for commercial use and queries on rights and licensing, see http://www.who.int/about/licensing.

Third-party materials. If you wish to reuse material from this work that is attributed to a third party, such as
tables, figures or images, it is your responsibility to determine whether permission is needed for that reuse and
to obtain permission from the copyright holder. The risk of claims resulting from infringement of any third-party-
owned component in the work rests solely with the user.

General disclaimers. The designations employed and the presentation of the material in this publication do not
imply the expression of any opinion whatsoever on the part of WHO concerning the legal status of any country,
territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Dotted and
dashed lines on maps represent approximate border lines for which there may not yet be full agreement.

The mention of specific companies or of certain manufacturers’ products does not imply that they are endorsed
or recommended by WHO in preference to others of a similar nature that are not mentioned. Errors and
omissions excepted, the names of proprietary products are distinguished by initial capital letters.

All reasonable precautions have been taken by WHO to verify the information contained in this publication.
However, the published material is being distributed without warranty of any kind, either expressed or implied.
The responsibility for the interpretation and use of the material lies with the reader. In no event shall WHO be
liable for damages arising from its use.
WHO Global Benchmarking Tool (GBT)

Contents
Introduction................................................................................................................................................iv
Acknowledgements....................................................................................................................................vi

01 National Regulatory System (RS).............................................................................................. 1

02 Registration and Marketing Authorization (MA)...................................................................... 73
03 Vigilance (Vl)........................................................................................................................ 115
04 Market Surveillance and Control (MC)................................................................................... 147
05 Licensing Establishments (LI)............................................................................................... 179
06 Regulatory Inspection (RI)..................................................................................................... 203
07 Laboratory Testing (LT)......................................................................................................... 241
08 Clinical Trials Oversight (CT)................................................................................................. 273
09 NRA Lot Release (LR)............................................................................................................ 309

Glossary and Definitions......................................................................................................................... 330

iii
WHO Global Benchmarking Tool (GBT)

Introduction
The World Health Organization (WHO) considers medical products and other health technologies one of the six
building blocks of health systems. Unlike many other commodities, however, end users and health care workers
are typically not in a position to judge the quality of medical products. It is therefore essential that the interests
and safety of the public be entrusted to a regulatory body responsible for ensuring the quality, safety, and
efficacy of medical products throughout the product life cycle.

Regulatory systems play a key role in assuring the quality, safety, and efficacy of medical products. Effective,
efficient and transparent regulatory systems are consequently an essential component of overall health
systems and contribute to desired public health outcomes and to innovation and investment. In contrast,
inefficient regulatory systems can be a barrier to access of safe, effective and quality medical products.

WHO has played a pivotal role in supporting countries to strengthen their regulatory systems, a role implicitly
mandated by the WHO constitution and elaborated explicitly in various World Health Assembly (WHA)
resolutions. Most notable of these is Resolution WHA 67.20 that addresses strengthening regulatory systems
for medical products adopted in May 2014 .

The benchmarking of regulatory systems referred to in Resolution WHA 67.20 implies a structured and
documented process by which Member States (MSs) can identify and address gaps with the goal of reaching a
level of regulatory oversight commensurate with a stable, well-functioning and integrated regulatory system.

The use of the WHO global benchmarking tool (GBT) is the primary means by which WHO assesses regulatory
systems for the regulation of medical products. The tool and benchmarking methodology enable WHO and
regulatory authorities to identify areas of strength as well as areas for improvement; facilitate the formulation
of an institutional development plan (IDP) to build upon strengths and address identified gaps; to aid in the
prioritization of investments in IDP implementation; and to help monitor progress.

WHO began assessing regulatory systems in 1997 using a set of indicators designed to assess the regulatory
program for vaccines. Since that time, several tools and revisions have been introduced, and the regulatory
systems of over 150 countries have been benchmarked.

The development of a unified WHO GBT for the assessment of medicine and vaccine programmes began in
2013 following a mapping of benchmarking tools internal and external to WHO with a view to ensuring policy
coherence, maximizing regulatory outcomes and reducing burden on regulatory authorities.

The benchmarking policy and methodology were the subject of two international consultations in January and
December 2015, respectively. The GBT revision VI was also the subject of a public consultation and extensive
expert consultation in 2018. Importantly, GBT revision VI was integrated with the medical regulatory systems
assessment tools developed by the WHO Regional Office for the Americas/Pan American Health Organization
(AMR/PAHO).

The GBT replaces all tools previously used by WHO, representing the first truly ‘global’ tool for benchmarking
regulatory systems. The GBT is designed to evaluate the overarching regulatory framework and the component
regulatory functions e.g. clinical trial oversight through a series of sub-indicators that may also be grouped and
examined according to nine cross-cutting categories or themes, for example, quality and risk management
system. Fact sheets have been developed for each sub-indicator to guide the benchmarking team and ensure
consistency in the evaluation, documentation and rating of the sub-indicator.

The GBT also incorporates the concept of ‘maturity level’ or ML (adapted from ISO 9004), allowing WHO
and regulatory authorities to assess the overall ‘maturity’ of the regulatory system on a scale of 1 (existence
of some elements of regulatory system) to 4 (operating at advanced level of performance and continuous
improvement).

iv
WHO Global Benchmarking Tool (GBT)

A benchmarking manual is available to help users to understand and apply the policy, methodology and process
to be followed and completed for benchmarking of regulatory system of medical products.

The GBT revision VI (for medicines and vaccines) is available in four UN official languages of English, French,
Spanish and Russian.

The GBT is supported by a computerized platform to facilitate the benchmarking, including the calculation of
maturity levels. The computerized GBT (cGBT) is available, upon request, to Member States and organizations
working with WHO under the Coalition of Interested Parties (CIP).

All queries related to GBT should be sent to WHO Regulatory Systems Strengthening Team at nra_admin@who.int.

v
WHO Global Benchmarking Tool (GBT)

Acknowledgements
The World Health Organization (WHO) would like to thank all Member States, the donor community, technical
partners, entities, individuals, and staff who over the years contributed to the establishment and maintenance
of the global benchmarking tool including its fact sheets, and who continue to do so, thereby ensuring the
continued success ad sustainability of the WHO regulatory system strengthening programme. We are indebted
to you all. Without your support this work would not have been possible.

vi
WHO Global Benchmarking Tool (GBT)

01
National Regulatory
System (RS): Indicators
and Fact Sheets

WHO Global
Benchmarking
Tool (GBT)
for Evaluation of
National Regulatory
System of Medical
Products

Page 1 of 340
WHO Global Benchmarking Tool (GBT)

01. National Regulatory System (RS):

Indicators and Fact Sheets

Function:
01 — NATIONAL REGULATORY SYSTEM (RS)
01

National Regulatory System (RS): Indicators and Fact Sheets

Description: The National Regulatory System provides the framework that supports the World
Health Organization (WHO) recommended regulatory functions. The National
Regulatory Authority (NRA) is the institution in charge of assuring the quality, safety,
and efficacy of medical products as well as ensuring the relevance and accuracy of
product information. A sustainable, well-functioning regulatory system will ensure an
independent and competent oversight of medical products.

Indicator: RS01 Legal provisions, regulations and guidelines required to define regulatory
framework of national regulatory system (RS)

Objective: The objective of this indicator is to ensure that the legal basis defining the regulatory
framework for the national regulatory system exists. The assessor should identify
how the different pieces of the legislation are drafted and to know which organizations
and institutions are consulted during this process, including the public, industry, non-
governmental organizations and other interested parties.
The assessor should identify the cases where the relevant legal provisions have been
defined but the regulations have not been enacted and published, which may lead to legal
uncertainty, misunderstanding or misinterpretation. The regulatory system functions
should be supported by appropriate and promulgated legislation.
02
Category: 01. Legal provisions, regulations and guidelines
03
Sub Indicator: RS01.01: Legal provision and regulations define the medical products that should
be regulated. 04

Maturity Level: 1 05

Scope: 1. Medicines 06
2. Vaccines
07
Description: The assessor should identify within the existing legislation and institutional regulations,
the scope of regulatory activities and products that should be regulated. Existing 08
definitions for regulated medical products (e.g., medicines, biological products, and
medical devices) should be used. It is not necessary to have a single (standalone)
09
drug law; however, a promulgated and enforced law should exist. If the base laws and
regulations refer to the need for complementary regulation, it is important to access
that information.

Objective: The objective of this sub-indicator is to ensure the existence of legislation and
institutional regulations that define the products that should be regulated. It is
important to set up the scope and mandate of the regulatory agency in charge of
regulating medical products in the country.

Requirement: Scope of regulated medical products

Evidence to review: The assessor should request for and review:

1. Promulgated legal provisions and regulations that define the medical products that
should be regulated.

References: 1. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations:

Page 2 of 340
WHO Global Benchmarking Tool (GBT)

thirty-fifth report. World Health Organization; 1999: Annex 8 (WHO Technical Report Series,
No. 885)., (1), (http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
2. Guidelines for national authorities on quality assurance for biological products. In:
WHO Expert Committee on Biological Standardization: forty-second report. World Health
Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2), (http://www.who.
int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_A2.pdf)
3. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization:
forty-fifth report. World Health Organization; 1995: Annex 1 (WHO Technical Report 01
Series, No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_TRS_858_

National Regulatory System (RS): Indicators and Fact Sheets

A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001.,
(116), (http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There are no legal provisions or regulations defining the
medical products that should be regulated.
 ONGOING IMPLEMENTATION (OI): There are some legal provisions and regulations or
although they do not exist, demonstrable steps have been taken towards developing them.
 PARTIALLY IMPLEMENTED (PI): The legal provisions and regulations defining the
medical products that should be regulated were recently developed as draft but not
yet promulgated and enforced.
 IMPLEMENTED (I): The legal provisions and regulations defining the medical products
are promulgated and enforced.

Limitations and remarks:  In a short time, it may not be possible for the assessor to review all aspects that this
indicator includes. Preferably the country profile or other documents that provide
a good description of the regulatory landscape in the country should be studied
beforehand. 02
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 03

Sub Indicator: RS01.02: Legal provision and regulations define the institutions that are involved 04
as part of the regulatory system, as well as their mandates, functions, roles,
responsibilities and enforcement powers. 05

Maturity Level: 1 06

Scope: 1. Medicines 07
2. Vaccines
08
Description: The assessor should identify the legislation and regulations that define all institutions
involved in medical product regulation including the relevant regulatory functions for
09
each institution. The regulatory functions covered will depend on the scope of the
regulatory system. The regulations should provide a clear mandate and define functions,
roles, responsibilities and enforcement powers for each regulatory function in order to
avoid overlapping of the respective empowerments. In the case of an assessment of a
framework with multiple regulatory authorities, the sub-indicator should be fulfilled by each
regulatory authority as well as any regional institution or authority. If different organizations
at different level of the state are involved the assessor should review how the linkage
among the organizations is defined in the law.

Objective: The objective of this sub-indicator is to ensure that the legislation and institutional
regulations provide clear mandates, functions, roles, responsibilities and enforcement
powers for each section in order to avoid overlap of the respective empowerments. The
same principle is applied if more than one institution is involved, or if different entities at
different levels of the state are involved.

Requirement: Institutions involved in the regulatory system

Page 3 of 340
WHO Global Benchmarking Tool (GBT)

Evidence to review: The assessor should request for and review:

1. List of agencies or authorities participating in the regulation of medical products, and the
mandate for each.
2. Promulgated legal provisions and regulations establishing the regulatory system and
defining the mandate for each of the institutions involved in the regulatory system.
3. Duties, responsibilities, level of competencies, mandates and enforcement powers
established for each institution.
4. Documents that demonstrate that the role and scope of each institution is clear to all
involved, especially when there are complementary or shared responsibilities. 01

National Regulatory System (RS): Indicators and Fact Sheets

References: 1. Guidelines for national authorities on quality assurance for biological products. In:
WHO Expert Committee on Biological Standardization: forty-second report. World
Health Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2), (http://
www.who.int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_
A2.pdf)
2. National Drug Regulatory Legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications on Pharmaceutical Preparations:
thirty- fifth report. World Health Organization; 1999: Annex 8 (WHO Technical Report Series,
No. 885)., (78), (http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
3. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization:
forty-fifth report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical
Report Series, No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_
TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001.,
(116), (http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There are no legal provisions or regulations defining the 02
institutions involved as part of the regulatory system.
 ONGOING IMPLEMENTATION (OI): There might be legal provisions and regulations 03
defining the institutions involved as part of the regulatory system for some
institutions but not all, or although they do not exist, demonstrable steps have been 04
taken towards developing them.
 PARTIALLY IMPLEMENTED (PI): The legal provisions and regulations defining the 05
institutions involved as part of the regulatory system were recently developed as draft
but not yet published officially. 06
 IMPLEMENTED (I): The legal provisions and regulations defining the institutions
involved as part of the regulatory system are published and available. 07

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will 08
always apply for all benchmarked NRAs).
09
Sub Indicator: RS01.03: When more than one institution or authority is involved in regulatory
oversight, the regulations should define administrative arrangements and the
channels of communication and coordination.

Maturity Level: 2

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the availability and implementation of regulations defining
channels of coordination among different institutions when the National Regulatory
Authority (NRA) is represented by more than one institution or authority. The assessor
should verify that there are well-defined administrative mechanisms for clear coordination
among the different institutions in order to avoid overlap of the respective empowerments.
If different organizations at different levels of the state are involved, the assessor should
review how the exchange of information is established and implemented.

Page 4 of 340
WHO Global Benchmarking Tool (GBT)

Objective: The objective of this sub-indicator is to ensure that the regulations define mechanisms
for communication, coordination and management of regulatory activities and processes
when more than one institution or authority is involved in the national regulatory system. If
different entities at different levels of the state are involved, the same principle is applied.

Requirement: Defined channels of coordination and an administrative mechanism

Evidence to review: The assessor should request for and review:

1. List of agencies or authorities participating in the regulation of medical products and the 01
mandate for each.

National Regulatory System (RS): Indicators and Fact Sheets

2. Published regulations defining the channels of coordination and an administrative
mechanism when more than one institution or authority is involved in regulatory activities.
3. Record of communications and other related evidence that shows coordination and
proper exchange of information.
4. Identified focal points for the purpose of coordination and exchange of information.

References:

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There are no regulations defining the administrative
arrangements and the channels of communication and coordination when more than
one institution or authority is involved in regulatory activities.
 ONGOING IMPLEMENTATION (OI): There might be regulations defining the
administrative arrangements and the channels of communication and coordination
for some institutions or authorities but not for all, or although they do not exist,
demonstrable steps have been taken towards developing them.
 PARTIALLY IMPLEMENTED (PI): The regulations defining the administrative
arrangements and the channels of communication and coordination were recently
developed as draft but not yet published officially. 02
 IMPLEMENTED (I): The regulations defining the administrative arrangements and
the channels of communication and coordination, when more than one institution or 03
authority is involved in regulatory activities, are published and available.
04
Limitations and remarks:
05
Sub Indicator: RS01.04: All regulatory entities (central and decentralized ones) follow
non- contradictory regulations, standards, guidelines and procedures. 06

Maturity Level: 3 07

Scope: 1. Medicines 08
2. Vaccines
09
Description: The assessor should understand, for the different administrative levels, the policy and
administrative organization for each of the organizational structures, as well as their
responsibilities and mandates. The assessor should verify if legislation defines whether
activities are centralized or whether they have been decentralized to other agencies or
authorities (states, provinces, others). If decentralized, assessor should verify that they
follow non-contradictory regulations, standards, guidelines, and procedures. Assessor
should check the availability and implementation of such non-contradictory regulations,
standards, guidelines and procedures in all levels of the regulatory bodies. In addition,
there should be a mechanism in place to manage the possible changes to any related
regulations, standards, guidelines and procedures. Guidelines for regulatory activities
should be developed and/or adopted and regularly updated. All regulatory bodies should
apply these guidelines.

Objective: The objective of this indicator is to ensure that activities conducted by decentralized entities
follow updated regulations, standards, guidelines, and procedures that are in agreement with
the central authority of the country. This ensures that regardless of the entity or place where

Page 5 of 340
WHO Global Benchmarking Tool (GBT)

regulatory activities are taking place, the same principles and standards will be followed by
all relevant parties. It also allows those regulatory bodies to have clarity on their roles and
responsibilities.

Requirement: Updated and non-contradictory regulations, standards, guidelines and procedures for all
regulatory bodies.

Evidence to review: The assessor should request for and review:

1. List of all regulatory bodies (central and decentralized ones) participating in the 01
regulation of medical products and the mandate for each.

National Regulatory System (RS): Indicators and Fact Sheets

2. The legislation and regulations that establish that all regulatory bodies (central and
decentralized ones) should follow the same regulations, standards, guidelines, and
procedures.
3. The identity of the individual or group responsible for monitoring of adherence to these
principles (item2)
4. Documented evidence that procedures and mechanisms are in place to manage
changes and inform all involved parties.
5. Documented evidence that procedures and mechanisms are in place to ensure
guidelines are developed, regularly updated and available to all regulatory bodies.

Framework: Structure/Foundation/Input 06

Rating Scale:  NOT IMPLEMENTED (NI): There are no legal provisions or regulations defining the 07
regulatory bodies (central and decentralized ones) involved as part of the regulatory
system. 08
 ONGOING IMPLEMENTATION (OI): There might be legal provision and regulations
defining the regulatory bodies (central and decentralized ones) involved as part of
09
the regulatory system for some institutions but not all, or although they do not exist,
demonstrable steps have been taken towards developing them.
 PARTIALLY IMPLEMENTED (PI): The legal provisions and regulations defining the
regulatory bodies (central and decentralized ones) involved as part of the regulatory
system were recently developed as draft but not yet published officially.
 IMPLEMENTED (I): The legal provisions and regulations defining the regulatory bodies
(central and decentralized ones) involved as parts of the regulatory system are
published and available.

Limitations and remarks:

Sub Indicator: RS01.05: Legal provisions and relevant regulations to take actions on recall,
suspension, withdrawal and/or destruction of substandard and falsified (SF)
medical products.

Maturity Level: 1

Page 6 of 340
WHO Global Benchmarking Tool (GBT)

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the availability of legal provisions and regulations supporting
regulatory enforcement actions related to SF medical products, including recall,
suspension, withdrawal and/or destruction. In addition, the assessor should ensure that
the NRA has authority to withhold or withdraw products from the market as well as from
use in clinical trials.
01
Objective: The objective of this indicator is to ensure that adequate provisions and mechanisms

National Regulatory System (RS): Indicators and Fact Sheets

exist and have been enacted to handle actions on recall, suspension, withdrawal and/or
destruction of SF medical products. This should include their destruction and a requirement
for the manufacturers, distributors, or marketing authorization holders to participate in the
recall of unsafe, defective or inappropriately labelled products. Possible actions include the
suspension of a marketing authorization, the recall of certain batches, issuance of warnings
in national drugs bulletins, or issuance of a separate warning sent out to a list of institutions
and key persons dealing with handling of medical products. NRA should have authority to
withdraw products from the market and from clinical development or clinical trials.

Requirement: Legal provisions and/or regulations for regulatory actions against SF medical products.

Evidence to review: The assessor should request for and review:

1. Promulgated and enforced legal provisions and regulations to take actions on recall,
suspension, withdrawal and/or destruction of SF medical products.
2. Guidance for manufacturers, importers, exporters and distributors on how to handle a
recall including product destruction.
3. Procedures for handling quality defects (e.g., notifications of suspected quality defects
or batch recalls by the company) including, focal points, defined responsibilities, deadlines
and records.
4. List of recalls performed and documented reports. 02

References: 03

Framework: Structure/Foundation/Input 04

Rating Scale:  NOT IMPLEMENTED (NI): There are no legal provisions or regulations to take actions 05
on recall, suspension, withdrawal and/or destruction of SF medical products.
 ONGOING IMPLEMENTATION (OI): There might be some legal provisions and 06
regulations to take actions on recall, suspension, withdrawal and/or destruction of
some SF medical products but not all, or although they do not exist, demonstrable 07
steps have been taken towards developing them.
 PARTIALLY IMPLEMENTED (PI): The legal provisions and regulations to take actions 08
on recall, suspension, withdrawal and/or destruction of SF medical products were
recently developed as draft but not yet published officially.
09
 IMPLEMENTED (I): The legal provisions and regulations to take actions on recall,
suspension, withdrawal and/or destruction of SF medical products are published and
available.

Limitations and remarks:  Market Surveillance and Control (MC) function

 Regulatory Inspection (RI) function
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: RS01.06: Legal provisions and regulations define requirements of transparency
and dissemination of information to the public and relevant stakeholders.

Maturity Level: 2

Scope: 1. Medicines
2. Vaccines

Page 7 of 340
WHO Global Benchmarking Tool (GBT)

Description: The assessor should check whether the legal basis requires the mandated institutions
to provide information to public and relevant stakeholders. The assessor should
verify whether legislation and regulations are publicly available, and that channels of
communication exist for this purpose.

Objective: The objective of this sub-indicator is to ensure that legislation, regulations and other
relevant information are publicly available. This public availability will ensure that those
governed by the regulations, as well as society in general, can be aware of the information
and utilize it as appropriate. In order to perform adequately and in a timely manner, the 01
NRA should have proper channels of communication in order to respond to the needs,

National Regulatory System (RS): Indicators and Fact Sheets

obligations and priorities.

Requirement: Legal provisions for transparency and dissemination of information.

Evidence to review: The assessor should request for and review:

1. Laws, regulations, other regulatory provisions, methodologies, and procedures
establishing that legislation and regulations are to be made public.
2. Public information from websites or in official bulletins or other legal publications.
3. Evidence for the availability and use of electronic or other platforms or means that allow
external communications regarding the regulations.
4. Direct forms of dissemination such as letters and emails.
5. Documentation of feedback after dissemination and publication (e.g., acknowledgments
of receipt, complaints, or questions).

References: National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations:
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical
Report Series, No. 885) (1) (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf)
2. Guidelines for national authorities on quality assurance for biological products. In: 02
WHO Expert Committee on Biological Standardization: forty-second report. WorldHealth
Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2), (http://www.who. 03
int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_A2.pdf)
3. Regulation and licensing of biological products in countries with newly developing 04
regulatory authorities. In: WHO Expert Committee on Biological Standardization:
forty-fifth report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical 05
Report Series, No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_
TRS_858_A1.pdf) 06
4. How to develop and implement a national drug policy, Second edition. WHO, 2001., (116),
(http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf) 07
5. Good governance for medicines: Model framework. Updated version 2014. WHO (110)
(https://www.who.int/medicines/areas/governance/ggm_modelframe_updated/en/) 08

Framework: Structure/Foundation/Input
09

Rating Scale:  NOT IMPLEMENTED (NI): There are no legal provisions or regulations to define
requirements of transparency and dissemination of information to the public and
relevant stakeholders.
 ONGOING IMPLEMENTATION (OI): The legal provisions and regulations to define
requirements for transparency and dissemination of information to the public and
relevant stakeholders were recently developed as draft but not yet published officially
and implemented.
 PARTIALLY IMPLEMENTED (PI): There are legal provisions and regulation to define
requirements for transparency and dissemination of information to the public and
relevant stakeholders, and these were recently published and/or implemented.
 IMPLEMENTED (I): The legal provisions and regulations to define requirements
of transparency and dissemination of information to the public and relevant
stakeholders are published and available.

Limitations and remarks:

Page 8 of 340
WHO Global Benchmarking Tool (GBT)

Sub Indicator: RS01.07: Development of the regulations involves the NRA responsible for their
implementation and enforcement.

Maturity Level: 1

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that legislation and regulations ensure that the 01
responsibilities, qualifications, rights and roles of each party are defined and recognized

National Regulatory System (RS): Indicators and Fact Sheets

(including those of medical practitioners, pharmacists and the NRA). They also create
the legal basis enabling the regulatory control of activities such as drug manufacture,
import, export, marketing, prescribing, dispensing and distribution. The legal provisions
also should establish the legal basis for enforcement of such laws and regulations. The
purpose of the legislation is therefore the same as that of the drug policy: to ensure that
only safe, effective, quality drugs are produced, imported and distributed, and that these
drugs are made available, managed effectively, and used appropriately. When writing
the law, the practicalities of future enforcement should be kept in mind. If there are no
strategies, facilities and resources for implementation and enforcement, legislation on
its own will achieve nothing. A law with modest aims and objectives that is properly
enforced is preferable to a more comprehensive one that cannot be implemented.
As a good practice, development of the legal provisions should involve the regulatory
authority responsible for their implementation and enforcement. In addition, the
assessor should check if suitable timelines for the implementation of newly adopted
regulations and guidelines are practiced. The suggested timeline for adoption of new
internationally recognized guidelines (e.g., guidelines from World Health Organization
(WHO) or International Conference on Harmonization) ranges from 6 to 12 months;
however, it is recognized that guideline implementation, might be extended over a
longer period.
02
Objective: The objective of this sub-indicator is to ensure that the development of the legal
provisions involves the regulatory authority responsible for their implementation and 03
enforcement. A well-formulated enforcement strategy is one that provides correct
incentives for the regulated subjects as well as appropriate guidelines for enforcement. 04
Furthermore, the enforcement strategy should be designed to minimize both the
monitoring effort and the costs for the regulated subjects and the public sector. The 05
importance of the enforcement phase is to ensure the quality and effectiveness of
regulatory policy and delivery, to reduce the overall level of regulatory burdens imposed 06
on businesses and citizens, and to increase incentives for compliance by private actors.
07
Requirement: Involvement of the regulatory system in the development of regulations
08
Evidence to review: The assessor should request for and review:
1. Legal provisions supporting different regulatory activities;
09
2. Records of process.

Page 9 of 340
WHO Global Benchmarking Tool (GBT)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence that the development of regulations
involve the regulatory authority responsible for their implementation and enforcement.
 ONGOING IMPLEMENTATION (OI): There is no evidence but demonstrable steps have
been taken towards this.
 PARTIALLY IMPLEMENTED (PI): The regulatory authority responsible for its
implementation has developed a draft but it has not yet been promulgated and
enforced. 01
 IMPLEMENTED (I): The regulations are developed with the involvement of the

National Regulatory System (RS): Indicators and Fact Sheets

regulatory authority responsible for their implementation and enforcement.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: RS01.08: The NRA consults or involves specific sectors of the civil society
(such as non-governmental organizations (NGOs) representing health
professionals, industry, consumers and patients) during the development
or adoption of regulations and guideline.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the development and application of a public and civil society
consultation process prior to issuance of guidelines related to different regulatory
activities. Specifically, assessor should:
1. Check whether the current and applicable legal and methodological bases for
thedevelopment of regulations in the country require the participation of the NRA, 02
industry, representatives of the health sector, patients, and NGOs.
2. Check whether a procedure is established to officially circulate or publish drafts in 03
consultation with the public, the NRA, industry, representatives of the health sector,
patients, and NGOs. 04
3. Assess evidence of how existing regulations were drafted, preferably those that were
approved more than one year ago. Evaluate the extent to which they were circulated, and 05
input was received from the NRA, industry, representatives of the health sector, patients,
and NGOs. 06
4. Assess evidence that the time provided for consultation with the public, the NRA,
industry, representatives of the health sector, patients, and NGOs was sufficient for 07
review and response.
5, Assess evidence for the existence of suitable timelines for the implementation 08
of newly adopted regulations and guidelines. Preferably, these timelines should be
specified in the public consultation process. The suggested timeline for adoption of
09
new internationally recognized guidelines (e.g. WHO and International Conference on
Harmonization guidelines) ranges from 6 to 12 months; however, it is recognized that the
guideline implementation might be extended over a longer period.

Objective: The objective of this indicator is to confirm that the regulations are developed through
an interactive and participative process involving, among others, the NRA, industry,
representatives of the health sector, patients, and NGOs. Assessor should note that the
participants may vary according to the draft regulation or guideline under consideration;
however, the consultation process ideally should be open to the public in accordance with
good regulatory practices (GRP).
As part of an open and transparent public policy, the development of regulations demands
the participation of the general public as well as those regulated. From the earliest stages
of developing a regulatory provision, consideration must be given to the viewpoints of those
who must follow the standards that are set, those who are impacted by them, and those
in general society. If regulations are approved without input from the key stakeholders
involved, it is doubtful that they will be fully understood or successfully implemented.

Page 10 of 340
WHO Global Benchmarking Tool (GBT)

Requirement: Consults or involves specific sectors for the development or adoption of the guidelines

Evidence to review: The assessor should request for and review:

1. Procedures for public and civil society consultation along with published drafts of
guidelines;
2. Laws, regulations, guidelines, methodologies, or procedures establishing how
regulations are prepared, who is invited to participate in the process, and who is consulted
before approval;
3. Institutional records on the preparation of regulations and consultations carried out. 01

National Regulatory System (RS): Indicators and Fact Sheets

Framework: “Structure/Foundation/Input”

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence the NRA consults or involves specific
sectors of the civil society (such as NGOs representing health professionals, industry,
consumers and patients) during the development or adoption of regulations and 02
guidelines.
 ONGOING IMPLEMENTATION (OI): There is no evidence but demonstrable steps have 03
been taken towards this.
 PARTIALLY IMPLEMENTED (PI): The regulatory authority responsible for its 04
implementation has developed a draft but it has not yet been promulgated and enforced.
 IMPLEMENTED (I): There is documented evidence the NRA consults or involves 05
specific sectors of the civil society (such as NGOs representing health professionals,
industry, consumers and patients) during the development or adoption of regulations 06
and guidelines.
07
Limitations and remarks: The participation of all parties identified will vary depending on the type of regulation.
08


The assessor should be mindful of why some may be excluded from the process for
certain regulations.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
09
always apply for all benchmarked NRAs).

Sub Indicator: RS01.09: A guideline on complaints and appeals against regulatory decisions
is available to the public.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should review if the possibility to appeal regulatory decisions, including
those related to enforcement and inspection activities, is a fundamental aspect of the
law. An efficient and effective enforcement system must provide readily accessible and
transparent means for receiving complaints and appeals against regulatory decisions.
The NRA should have implemented guidelines on review, evaluation, communication of
complaints and appeals against regulatory decisions.

Page 11 of 340
WHO Global Benchmarking Tool (GBT)

Objective: The objective of this sub-indicator is to ensure that a mechanism exists to appeal
regulatory decisions.

Requirement: Complaints and appeals against regulatory decisions

Evidence to review: The assessor should request for and review:

1. Guidelines for complaints and appeals related to different regulatory decisions.

References: 1. National drug regulatory legislation: guiding principles for small drug regulatory 01
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations:

National Regulatory System (RS): Indicators and Fact Sheets

thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical
Report Series, No. 885) (1) (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf)
2. Guidelines for national authorities on quality assurance for biological products. In:
WHO Expert Committee on Biological Standardization: forty-second report. World Health
Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2), (http://www.who.
int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_A2.pdf)
3. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization: forty-fifth
report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical Report Series,
No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001., (116),
(http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There is no guideline available on complaints and appeals
against regulatory decisions.
 ONGOING IMPLEMENTATION (OI): There is no evidence of a guideline on complaints
and appeals against regulatory decisions, but demonstrable steps have been taken 02
towards its development.
 PARTIALLY IMPLEMENTED (PI): There is a draft guideline on complaints and appeals 03
against regulatory decisions but it is not yet implemented.
 IMPLEMENTED (I): There is a documented and implemented guideline on complaints 04
and appeals against regulatory decisions.
05
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 06

Indicator: RS02 Arrangement for effective organization and good governance. 07

08
Objective: The objective of this indicator is to ensure that arrangements for effective organization and
good governance are in place. Good governance is an essential factor for economic growth
09
and sustainable development at all levels and within all sectors of society. The term Good
Governance is increasingly used to emphasize the need for governance to operate with due
regard for the rule of law and especially in a manner that is free of corruption. There is also
growing consensus on the major characteristics of good governance. Good governance is
participatory, consensus-oriented, accountable, transparent, responsive, effective, efficient,
equitable, inclusive, and follows the rule of law.

Category: 02. Organization and governance

Sub Indicator: RS02.01: The structure and line of authority among, and within, all institutions that
participate in the regulatory system is defined, documented and implemented.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Page 12 of 340
WHO Global Benchmarking Tool (GBT)

Description: The assessor should verify that legislation or institutional regulation defines the structure
and line of authority among, and within, all institutions that participate in the regulatory
system.
If more than one institution is involved, the legislation or institutional regulation should
provide for clear coordination and avoid overlap of the regulatory activities. If different
organizations at different level of the state are involved, the assessor should review
the linkage among the organizations and evaluate how the exchange of information is
established and implemented.
01
Objective: The objective of this sub-indicator is to ensure that the structure and line of

National Regulatory System (RS): Indicators and Fact Sheets

authority among, and within, all institutions that participate in the regulatory system
is implemented to ensure coordination and to avoid overlap of the respective
empowerments.

Requirement: Structure and line of authority

Evidence to review: The assessor should request for and review:

1. Institutional founding documents along with evidence of implementation through
procedures and respective records;
2. Organizational chart of NRA.
3. Procedures and human resources record forms of members and staff of the authority,
their terms of reference and duration of office.

(https://www.who.int/medicines/areas/governance/ggm_modelframe_updated/en/)

Framework: Structure/Foundation/Input
09

Rating Scale:  NOT IMPLEMENTED (NI): The structure and line of authority among, and within, all
institutions that participate in the regulatory system is not defined, documented and
implemented.
 ONGOING IMPLEMENTATION (OI): There is no evidence that the structure and line of
authority among, and within, all institutions that participate in the regulatory system
are defined, documented and implemented but demonstrable steps have been taken
towards its development.
 PARTIALLY IMPLEMENTED (PI): There is a draft defining the structure and line of
authority among, and within, all institutions that participate in the regulatory system
but it is not yet implemented.
 IMPLEMENTED (I): The structure and line of authority among, and within, all institutions
that participate in the regulatory is defined, documented and implemented.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Page 13 of 340
WHO Global Benchmarking Tool (GBT)

Sub Indicator: RS02.02: Channels of communication and decision-making are clearly established
among the structures, institutions, and departments forming the NRA.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the existence and the actual use of communication and 01
decision makings channels among different structures, institutions, and departments

National Regulatory System (RS): Indicators and Fact Sheets

forming the NRA.

Objective: The objective of this sub-indicator is to ensure that channels of communication

and decision-making are clearly established among the structures, institutions, and
departments forming the NRA. Supporting documentation should include the definitions
for roles and responsibilities for each section, mechanisms for communication and
coordination, and protocols for information and resource sharing. Collaborative
agreements can assist in avoiding overlap and the duplication of efforts.

Requirement: Communication and decision-making channels

Evidence to review: The assessor should request for and review:

1. Documentation for communication channels and decision making-process among the
structures, institutions, and departments;
2. Evidence of implementation through procedures and respective records.

References: 1. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations:
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical
Report Series, No. 885) (1) (http://apps.who.int/medicinedocs/documents/s21964en/ 02
s21964en.pdf)
2. Guidelines for national authorities on quality assurance for biological products. In: 03
WHO Expert Committee on Biological Standardization: forty-second report. World Health
Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2), (http://www.who. 04
int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_A2.pdf)
3. Regulation and licensing of biological products in countries with newly developing 05
regulatory authorities. In: WHO Expert Committee on Biological Standardization: forty-fifth
report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical Report Series, 06
No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001., (116), 07
(http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)
08
Framework: Structure/Foundation/Input
09
Rating Scale:  NOT IMPLEMENTED (NI): Channels of communication and decision-making are not
clearly established among the structures, institutions, and departments that form the
NRA.
 ONGOING IMPLEMENTATION (OI): There is no evidence that channels of
communication and decision-making are established among the structures,
institutions, and departments that form the NRA, but demonstrable steps have been
taken towards their development.
 PARTIALLY IMPLEMENTED (PI): There is a draft defining channels of communication
and decision-making among the structures, institutions, and departments that form
the NRA, but it is not yet implemented.
 IMPLEMENTED (I): Channels of communication and decision-making among the
structures, institutions, and departments that form the NRA are clearly defined,
documented and implemented.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Page 14 of 340
WHO Global Benchmarking Tool (GBT)

Sub Indicator: RS02.03: Scientific and advisory committees exist to advise the NRA on topics
of scientific and regulatory interest and on future objectives and strategies.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that: 01

1. there are legal provisions granting the NRA authority to convene outside experts and

National Regulatory System (RS): Indicators and Fact Sheets

committees;
2. the scope of existing legal provisions authorizes the NRA to determine:
a. the functions of the outside experts and committees;
b. their role in the decision-making process;
c. the circumstances under which they should be convened;
3. the NRA uses this authority to convene outside experts and committees

Objective: The objective of this indicator is to ensure that the legal basis exists to convene outside
experts and expert committees, to define their functions, (in particular, their role in the NRA
decision-making process) and to define the circumstances under which they should be
convened.
Science and technology are evolving at a dizzying pace. Therefore, issues must be
handled by multidisciplinary teams and it is impossible for the NRAs to have all the
expertise needed to handle all issues. For this reason, it is highly recommended to
have external experts and committees of experts available to help with the regulatory
decision-making process. Nevertheless, advisory and scientific committees should not
be considered as mandatory for the regulatory work. In this context, it should be noted
that advisory and scientific committees add financial and administrative burden on the
regulatory system. Legislation should give the NRA the authority to convene them in well-
defined circumstances. 02

Requirement: Scientific/advisory committees 03

Evidence to review: The assessor should request for and review: 04

1. Laws, regulations, decrees, agreements, and other legal provisions conferring authority
on the NRA to convene outside experts and expert committees, to define their functions, 05
especially in the decision-making process, and to define the circumstances under which
they should be convened. 06
2. Records, proceedings, and any other documents showing that the NRA makes use of
the authority granted by the legislation. 07

References: 1. National drug regulatory legislation: guiding principles for small drug regulatory 08
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations:
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical
09
Report Series, No. 885) (1) (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf)
2. Guidelines for national authorities on quality assurance for biological products. In: WHO
Expert Committee on Biological Standardization: forty-second report. World
Health Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2), (http://
www.who.int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_
A2.pdf)
3. Regulation and licensing of biological products in countries with newly
developing regulatory authorities. In: WHO Expert Committee on Biological Standardization:
forty-fifth report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical
Report Series, No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_
TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001., (116),
(http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)

Framework: Structure/Foundation/Input

Page 15 of 340
WHO Global Benchmarking Tool (GBT)

Rating Scale:  NOT IMPLEMENTED (NI): Scientific and advisory committees to advise the NRA on
topics of scientific and regulatory interest and future objectives and strategies are not
used or documented.
 ONGOING IMPLEMENTATION (OI): There is no evidence that scientific and advisory
committees to advise the NRA on topics of scientific and regulatory interest and
future objectives and strategies are used but demonstrable steps have been taken
towards their implementation.
 PARTIALLY IMPLEMENTED (PI): There is a draft defining scientific and advisory
committees to advise the NRA on topics of scientific and regulatory interest and 01
future objectives and strategies, but it is not yet implemented.

National Regulatory System (RS): Indicators and Fact Sheets

 IMPLEMENTED (I): Scientific and advisory committees to advise the NRA on topics of
scientific and regulatory interest and future objectives and strategies are used and
documented.

Limitations and remarks: Advisory and scientific committees, formed or external or internal experts should not be
considered as mandatory for the regulatory work. Therefore, if there is a duly documented
political or legal decision against the participation of outside experts and committees
(e.g., due to their administrative and/or financial burden), this indicator should be rated
Not Applicable. However, in this case, which is exceptional indeed, the assessor should
verify the availability of internal personnel with the expertise needed to take over the
required regulatory activities without the need, for internal or external, expert committees.
Availability of terms of reference for experts and expert committees would provide
additional guidance to the assessor.

Sub Indicator: RS02.04: Independence of NRA from researchers, manufacturers, distributors and
wholesalers, as well as from the procurement system.

Maturity Level: 2

Scope: 1. Medicines 02
2. Vaccines
03
Description: The assessor should identify documented evidence that demonstrates the independence
of NRA decision-making from researchers, manufacturers, distributors, wholesalers as 04
well as from procurement institutions involved in acquiring different medical products.
For example, the control laboratory of a manufacturer must not perform the quality 05
testing on behalf of the NRA when quality testing is deemed necessary by the NRA (e.g.
for the purpose of post-marketing surveillance or NRA lot release, if applicable). Another 06
example is that the decision-making bodies should not include or be influenced by experts
who represent institutions interested in marketing of medical products. If the Ministry 07
of Health or other governmental authority is responsible for procuring medical products
in the country, documented evidence should be provided that NRA decision- making is 08
independent from the organization or office that is responsible for procuring the products.
Aspects to consider when assessing whether the objectives of the indicator have been
09
met:
1. Determine whether the duties assigned to the NRA include research, manufacture, or
distribution of medical products;
2. Examine the hierarchical level of the NRA and verify that the NRA is independent of
those involved in research, manufacturing, and distribution of product.

Objective: The objective of this indicator is to ensure that, consistently over time, the NRA operates
independently of researchers, producers, distributors and other regulated parties.
In order to discharge its duties fairly, the NRA must be independent of those regulated
entities. Thus, the NRA may not be engaged in the activities that it regulates and may not
be at a hierarchal level that is subordinate to those institutions that perform regulated
activities.
In some countries, the NRA historically has been responsible for manufacturing vaccines,
which placed the NRA in the position of being judge and party to the matters before it, and
thus compromising its independence and impartiality.
Independence of the NRA decision-making process from influence by institutions,

Page 16 of 340
WHO Global Benchmarking Tool (GBT)

societies, and industries which may have direct or indirect interest in the NRA decisions
is one of the key elements regarding the safe use of medical products and protection of
public health.

Requirement: Independence of NRA

Evidence to review: The assessor should request for and review:

1. Organizational chart of the national regulatory system;
2. Functioning organizational chart of the NRA; 01
3. Documents defining mission and functions of the organizations within the national

National Regulatory System (RS): Indicators and Fact Sheets

regulatory system.
References: 1. National drug regulatory legislation: guiding principles for small drug regulatory authorities.
In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: thirty-fifth
report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical Report Series, No.
885) (1) (http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
2. Guidelines for national authorities on quality assurance for biological products. In:
WHO Expert Committee on Biological Standardization: forty-second report. World Health
Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2), (http://www.who.
int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_A2.pdf)
3. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization: forty-fifth
report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical Report Series,
No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001., (116),
(http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence that the NRA is independent from
researchers, manufacturers, distributors and wholesalers, as well as from the 02
procurement system.
 ONGOING IMPLEMENTATION (OI): There is no evidence that the NRA is independent 03
from researchers, manufacturers, distributors and wholesalers, as well as from the
procurement system; however, demonstrable steps toward this have been taken. 04
 PARTIALLY IMPLEMENTED (PI): There is evidence that some steps have been taken to
establish the independence of NRA from researchers, manufacturers, distributors and 05
wholesalers, as well as the procurement system, but it is not yet implemented.
 IMPLEMENTED (I): There is documented evidence the NRA is independent from 06
researchers, manufacturers, distributors and wholesalers, as well as from the
procurement system. 07

Limitations and remarks:  In case the manufacturer is part of the structure of the National Regulatory System, 08
this must be taken into consideration when evaluating the independency.
It is important to maintain good collaboration and communication between industry
09
and academia while maintaining the independence of the regulatory system.
Regulatory authorities (including national control laboratories) may be involved in
scientific research activities. However, these research activities should not entail any
 conflict of interest with respect to regulatory oversight. Internal researchers with no
conflicts of interest are not meant to be addressed by this sub-indicator and scoring
the sub-indicator as “not implemented” should be excluded (unless justified for some
other reason). On the other hand, if the regulatory authority is involved in research
activities which conflict with the mandate to regulate medical products, the sub-
indicator should be scored as “not implemented”.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Indicator: RS03 Strategic plan with clarified objective in place.

Objective: The objective of this indicator is to ensure that the NRA has implemented and updated

Page 17 of 340
WHO Global Benchmarking Tool (GBT)

its strategic plan, including general objectives that are broken down into specific targets
for various regulatory functions, and that are provided with defined time periods for
implementation. The NRA should also ensure that adherence to the strategic plan is
monitored periodically. Based on its mission and vision, and in accordance with the
government’s policy, the NRA should adopt a coherent development strategy, update
the strategy periodically, and monitor its implementation. This requires the availability
of a detailed institutional development plan, including targets and deadlines, which
complements the NRA strategy as stated in the NRA strategic plan.
Setting a NRA strategy involves a complex process of development, implementation 01
and monitoring. First, the strategy development process results in the formulation of the

National Regulatory System (RS): Indicators and Fact Sheets

strategic plan. Second, activities aimed at achieving strategic objectives are implemented
by the various entities of the NRA. Finally, the effect of these activities is monitored, and
the regulatory work is adjusted if necessary. Throughout the process, careful planning and
the involvement of the related parties are needed; at all times, the political dynamics have
to be considered.
A strategic plan without an implementation plan (also called institutional development
plan) remains a dead document. Careful planning of the implementation steps and
activities necessary to arrive at the expected outcome is important throughout the
process. As evident, there are various types of plans. The first is probably the strategic
plan (which may come under a higher national drug and medicine policy) to develop
the strategic objectives themselves. After the strategic plan has been adopted, an
implementation plan (also called institutional development plan or master plan) is
needed; this implementation plan typically covers a 3 to 5-year period. This plan details
the various activities for each of the strategic objectives. The implementation plan spells
out what needs to be done and who is responsible, and then estimates the budget and
proposes a time frame. If resources are insufficient without external input, a set of priority
activities should be identified that can be executed within the existing resources. The
implementation plan can be broken down into individual annual work plans for the NRA as
a whole, or for the various units and departments.
02
Category: 03. Policy and strategic planning
03
Sub Indicator: RS03.01: A national drug policy, aligned with health policy, exists
and is implemented. 04

Maturity Level: 4 05

Scope: 1. Medicines 06
2. Vaccines
07
Description: The assessor should verify the existence of national drug policy. A national drug policy,
presented and printed as an official government statement, is important because it acts 08
as a formal record of aspirations, aims, decisions and commitments. Without such a
formal policy document, there may be no general overview of what is needed. As a result,
09
some government measures may conflict with others because the various goals and
responsibilities are not clearly defined and understood. The policy document should be
developed through a systematic process that incudes consultation with all interested
parties. In this process the objectives must be defined, priorities must be set, strategies
must be developed, and commitments must be built. A national drug policy cannot
bedeveloped in a vacuum; it must fit within the framework of a particular health care
system, a national health policy and, perhaps, a programme of health sector reform.
The goals of the national drug policy should always be consistent with broader health
objectives, and policy implementation should help to achieve those broader objectives.

Objective: The objective of this sub-indicator is to ensure that a national drug policy aligned with
the health policy exists and it is implemented. A national drug policy is a commitment to
a goal and a guide for action. It defines and prioritizes the medium to long-term goals set
by the government for the pharmaceutical sector and identifies the main strategies for
attaining them. It provides a framework within which the activities of the pharmaceutical
sector can be coordinated. The policy covers both the public and the private sectors and

Page 18 of 340
WHO Global Benchmarking Tool (GBT)

involves all the main actors in the pharmaceutical field. A national drug policy is needed for
many reasons. The most important are: to present a formal record of values, aspirations,
aims, decisions and medium to long-term government commitments; to define the
national goals and objectives for the pharmaceutical sector; to set priorities; to identify
the strategies needed to meet those objectives; to identify the various actors responsible
for implementing the main components of the policy; and to create a forum for national
discussions on
these issues.
01
Requirement: National drug policy/national medicines policy

National Regulatory System (RS): Indicators and Fact Sheets

Evidence to review: The assessor should request for and review:
1. National drug policy;
2. National health policy;
3. Evidence for implementation of the drug policy.

Framework: Structure/Foundation/Input 04

Rating Scale:  NOT IMPLEMENTED (NI): A national drug policy, aligned with health policy is not 05
defined, documented and implemented.
 ONGOING IMPLEMENTATION (OI): There is no evidence that a national drug policy, 06
aligned with health policy, exists and is implemented but demonstrable steps have
been taken towards its development. 07
PARTIALLY IMPLEMENTED (PI): There is a draft defining a national drug policy, aligned
08

with health policy but it is not yet implemented.
IMPLEMENTED (I): A national drug policy, aligned with health policy is defined,
09

documented and implemented.

Limitations and remarks: National drug policy is the term broadly used, however national medicines policy can also
be found. In some countries the national drug policy can go to the cabinet or parliament for
endorsement. In others, the policy will remain an administrative document that serves as a
basis for implementation plans and changes in the law and regulations. In some countries
the entire national drug policy document has become law. Although this is a powerful
demonstration of political commitment, it can also cause problems, as future adjustments
to the policy may become difficult. It is therefore recommended that only certain enabling
components of the policy are incorporated into law, without too many operational details.
Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: RS03.02: The NRA has established and declared its vision, mission
and strategic priorities.

Maturity Level: 3

Page 19 of 340
WHO Global Benchmarking Tool (GBT)

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify:

1. The existence of legal provisions establishing that the NRA must have a Board,
Management Committee, or administrative leadership body.
2. That the functions of the Board, Management Committee or administrative body include
the establishment and/or adoption of strategic development plans.
3. That such an administrative body exists in practice, that the administrative body is 01
functioning, and that it approves and monitors the institution’s strategic development

National Regulatory System (RS): Indicators and Fact Sheets

plans.

Objective: The objective of this sub-indicator is to ensure that the NRA has established and
declared its vision, mission and strategic priorities. The general objectives of the
national drug policy are access, quality and objective use. The policy should balance
the various goals and objectives, and thus create a complete and consistent entity.
For example, one objective is to provide access to assured source of medicines.
Additionally, strategic development plans set out the means to achieve short, medium,
and long-term goals, and to channel resources and activities toward achievement of
same.

Requirement: Vision, mission and strategic priorities

Evidence to review: The assessor should request for and review:

1. Published or unpublished legal provisions on the leadership levels within the NRA;
2. Records, for the last three-time periods, of the proceedings of the administrative body
that is responsible for adopting the institution’s strategic development plans;
3. Organizational chart or other NRA document describing the Board or other
administrative body.
02
References: 1. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical 03
Preparations: thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8
(WHO Technical Report Series, No. 885) (1) (http://apps.who.int/medicinedocs/ 04
documents/s21964en/s21964en.pdf)
2. Guidelines for national authorities on quality assurance for biological products. In: 05
WHO Expert Committee on Biological Standardization: forty-second report. World
Health Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2), (http:// 06
www.who.int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_
A2.pdf) 07
3. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization: 08
forty-fifth report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical
Report Series, No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_
09
TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001.,
(116), (http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): The NRA has not established and declared its vision,
mission and strategic priorities.
 ONGOING IMPLEMENTATION (OI): There is no evidence that the NRA has established
and declared its vision, mission and strategic priorities but demonstrable steps have
been taken towards its development.
 PARTIALLY IMPLEMENTED (PI): There is a draft document defining the NRA vision,
mission and strategic priorities.
 IMPLEMENTED (I): There is documented evidence that the NRA has established and
declared its vision, mission and strategic priorities.

Page 20 of 340
WHO Global Benchmarking Tool (GBT)

Limitations and remarks:  The terminology for collegial administrative bodies and institutional strategic
development plans may vary from country to country.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: RS03.03: A plan for achieving strategic objectives is developed, implemented
and regularly updated.

Maturity Level: 3 01

National Regulatory System (RS): Indicators and Fact Sheets

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that:

1. There are institutional strategic development plans and that they are updated (check last
three versions or past three years).
2. The plan contains general and specific objectives, includes all basic regulatory functions,
and establishes deadlines for carrying out actions or achieving expected results.
3. There is periodic monitoring of compliance with the plan.

Objective: The objective of this indicator is to ensure that the NRA has implemented and updated its
development plan, including the plan’s general objectives that define specific targets and
time periods for various regulatory functions. The NRA should also ensure that adherence
to the development plan is monitored periodically.
Based on its mission and vision, and in accordance with the government’s policy, the NRA
should adopt a coherent development strategy that is periodically updated and should then
monitor its implementation. This requires the existence of an institutional development
plan, as well as a strategy, including targets and deadlines, for following the plan.

Requirement: Plan for achieving strategic objectives 02

Evidence to review: The assessor should request for and review: 03

1. The NRA’s institutional development plans and their updates;
2. The NRA’s annual plans; 04
3. Documents that describe the general and specific objectives and measurement
indicators; 05
4. periodic reports on compliance with the plans and objectives (which should be
completed on at least an annual basis). 06

References: 1. National drug regulatory legislation: guiding principles for small drug regulatory 07
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations:
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical 08
Report Series, No. 885) (1) (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf)
09
2. Guidelines for national authorities on quality assurance for biological products.
In: WHO Expert Committee on Biological Standardization: forty-second report. World
Health Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2), (http://www.
who.int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_A2.pdf)
3. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization: forty-fifth
report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical Report Series,
No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001., (116),
(http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): The NRA has not developed, implemented or regularly
updated a plan for achieving strategic objectives.
 ONGOING IMPLEMENTATION (OI): There is no evidence that the NRA has developed,

Page 21 of 340
WHO Global Benchmarking Tool (GBT)

implemented or regularly updated a plan for achieving strategic objectives, but

demonstrable steps have been taken towards its development.
 PARTIALLY IMPLEMENTED (PI): There is a draft document defining a plan for
achieving strategic objectives.
 IMPLEMENTED (I): There is documented evidence that the NRA has developed,
implemented and regularly updated a plan for achieving strategic objectives.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 01

National Regulatory System (RS): Indicators and Fact Sheets

Sub Indicator: RS03.04: Documented policies, procedures and mechanisms, including written criteria,
are established for recognition and reliance on decisions of other NRAs (if applicable).

Maturity Level: 2

Scope: 1. Medicines
2. Vaccines

Description: The assessor should identify if there are provisions or written criteria for recognition and
reliance on other NRAs. The assessor should check the criteria, provisions, procedures,
and decision-making processes related to this recognition process, along with a list of
recognized NRAs.

Objective: The objective of this sub-indicator is to ensure there are documented policies, procedures
and mechanisms in place for recognition and reliance on other NRAs. For example,
some countries will not have the mechanisms and resources that enable them to assess
the medicines applications with their own expertise; such countries may use a system
or mechanism of recognizing decisions of other NRAs. In that case the country may
recognize certificates, reports or decisions from the foreign NRA (e.g. in the country
of origin of the product) or from the WHO prequalification process. Other examples 02
may include the functions such as regulatory inspections, clinical trials oversight, and
lot release. If this is the case, the assessor should verify that written provisions and 03
procedures are established that define who is allowed to recognize decisions, reports or
certificates from other authorities. For this purpose, the NRA should create a list of the 04
NRAs that may be recognized.
05
Requirement: Recognition and reliance on other NRA’s decisions
06
Evidence to review: The assessor should request for and review:
1. Documentation for the mechanisms and procedures for recognition and reliance on 07
other NRA’s decisions;
2. Documentation for the policy that defines the criteria for recognition of foreign NRA 08
inspections or regulatory decisions;
3. A list of recognized NRAs and supporting records.
09

Page 22 of 340
WHO Global Benchmarking Tool (GBT)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): The NRA has not established documented policies,
procedures and mechanisms with written criteria for recognition and reliance on other
NRA’s decisions (if applicable).
 ONGOING IMPLEMENTATION (OI): There is no evidence that the NRA has established
documented policies, procedures and mechanisms with written criteria for
recognition and reliance on other NRA’s decisions (if applicable) but demonstrable
steps have been taken towards its development. 01
 PARTIALLY IMPLEMENTED (PI): There is a draft document defining documented

National Regulatory System (RS): Indicators and Fact Sheets

policies, procedures and mechanisms with written criteria for recognition and reliance
on other NRA’s decisions (if applicable)
 IMPLEMENTED (I): There is documented evidence that the NRA has developed,
implemented or regularly updated documented policies, procedures and mechanisms,
with written criteria, for recognition and reliance on other NRA’s decisions (if
applicable)

Limitations and remarks: If the country has enough resources to conduct all regulatory oversight with internal
resources, this sub-indicator is not applicable.

Sub Indicator: RS03.05: The NRA is promoting good regulatory practices (GRPs).

Maturity Level: 4

Scope: 1. Medicines
2. Vaccines

Description: The assessor should evaluate if the NRA promotes, establishes and applies the principles
of GRP to the regulation of medical products in different functions and areas. 02
The principles on which regulatory systems may be established and by which they may
be evaluated are legality, impartiality, consistency, proportionality, flexibility, effectiveness, 03
efficiency, clarity and transparency.
04
Objective: The objective of this sub-indicator is to ensure the NRA applies the principles of GRP to
the regulation of medical products. GRPs are built on a foundation of transparency, good 05
governance and sound policy-making. Public confidence in health products depends on
confidence in the integrity of regulatory oversight. In order to meet their objectives of 06
promoting and protecting public health, national regulatory authorities need to carry out
their functions based on this set of principles. 07
GRPs help to ensure that national regulatory systems, and international regulatory
cooperation programmes, remain relevant, current and flexible as technology evolves and 08
unforeseen needs and emergencies occur.
09
Requirement: GRP

Evidence to review: The assessor should request for and review:

1. Policies and regulations outlining the application of the principles of GRPs in the
oversight of medical products;
2. Documents that provide examples of the application of the principles of GRPs.

Page 23 of 340
WHO Global Benchmarking Tool (GBT)

regulatory authorities. In: WHO Expert Committee on Biological Standardization: forty-fifth

report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical Report Series,
No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001., (116),
(http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)
5. Good regulatory practices: guidelines for national regulatory authorities for medical
products., (112), (http://digicollection.org/whoqapharm/p/about/) and (http://apps.who.int/
medicinedocs/en/)
01
Framework: “Structure/Foundation/Input”

National Regulatory System (RS): Indicators and Fact Sheets

Rating Scale:  NOT IMPLEMENTED (NI): The NRA is not promoting the principles of GRP.
 ONGOING IMPLEMENTATION (OI): There is no documented evidence the NRA is
promoting the principles of GRP, but demonstrable steps have been taken towards its
development.
 PARTIALLY IMPLEMENTED (PI): There is a draft document defining the principles of
GRP.
 IMPLEMENTED (I): There is documented evidence that the NRA is promoting and
applying accepted principles of GRP.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Indicator: RS04 Regulatory system is supported with leadership

and crisis management plans.

Objective: The objective of this indicator is to ensure that national regulatory authorities have the
appropriate tools and institutional framework for co-coordinated action for management of
crises.
Getting a clear operational picture of the development of the crisis is the basis for decision- 02
making both at operational and strategic levels. What happened, how many people are or
might be affected, what are the issues at stake, how might the crisis develop, and what 03
are the means in the operational field are a series of questions that leadership needs for
making decisions. The complexities of modern crises often require the involvement of 04
many actors, above and beyond emergency services. This approach demands effective
coordination for a successful outcome. The need for coordination also raises significant 05
public governance challenges, as crisis management functions are often exercised at sub-
national levels but coordinated at the centers of governments. The capacity to coordinate 06
crisis management is a fundamental element of good governance, as it tests the
government’s capacity to provide, in a timely manner, the appropriate responses required 07
to protect citizens and businesses and to mitigate the impact of disasters. Crisis managers
need to adapt their approaches to deal with a variety of different stakeholders that may 08
have different interests, priorities, and values.
09
Category: 04. Leadership and crisis management

Sub Indicator: RS04.01: Leadership ensures that the strategic priorities and objectives
are well known and communicated throughout the NRA.

Maturity Level: 4

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that the strategic priorities and objectives are defined and
implemented. In the case of crisis management, harmonized systems and situation
reports from all active operational entities should be gathered at a centralized location in
order to inform the crisis cell (i.e., the group composed of high level officials designated
to be in charge and to respond, handle and manage crisis situations). Information and
communication systems, as well as standard reporting protocols among the emergency

Page 24 of 340
WHO Global Benchmarking Tool (GBT)

response network, enable easier analysis and raise awareness of the situation. Ongoing
work to address the challenge of public health emergencies has increasingly recognized
the important role that public communication plays in their effective management.

Objective: The objective of this sub-indicator is to confirm that strategic priorities and objectives
are issued by senior management or the head of the institution. In addition, the assessor
should
verify that a development plan exists, and its implementation is supported by the senior
management. 01

National Regulatory System (RS): Indicators and Fact Sheets

Requirement: Strategic priorities and objectives communication

Evidence to review: The assessor should request for and review:

1. Documents that verify that strategic priorities and objectives are available, distributed to
different organizational levels and understood by the staff;
2. Records of relevant communications.

References: 1. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations:
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical
Report Series, No. 885) (1) (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf)
2. Guidelines for national authorities on quality assurance for biological products. In:
WHO Expert Committee on Biological Standardization: forty-second report. World
Health Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2),
(http://www.who.int/biologicals/publications/trs/areas/biological_products/WHO_
TRS_822_A2.pdf) 3. Regulation and licensing of biological products in countries with
newly developing regulatory authorities. In: WHO Expert Committee on Biological
Standardization: forty-fifth report. Geneva: World Health Organization; 1995: Annex 1 (WHO
Technical Report Series, No. 858), (3), (http://www.who.int/bloodproducts/publications/ 02
WHO_TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001., (116), 03
(http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)
04
Framework: Structure/Foundation/Input
05
Rating Scale:  NOT IMPLEMENTED (NI): The strategic priorities and objectives are neither known nor
communicated throughout the NRA. 06
 ONGOING IMPLEMENTATION (OI): There is no evidence that strategic priorities and
objectives are known and communicated throughout the NRA, but demonstrable 07
steps have been taken towards its development.
 PARTIALLY IMPLEMENTED (PI): There is a draft document defining the strategic
08
priorities and objectives and how to communicate them through the NRA.
 IMPLEMENTED (I): There is documented evidence that the strategic priorities and
09
objectives are known and communicated throughout the NRA.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: RS04.02: A rapid alert system to for managing the threats by SF medical products
and for recalling these products from the market.

Maturity Level: 2

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the existence of a nationwide rapid alert system to manage
and react to SF medical products whenever and wherever identified. Such rapid alerts may
or may not include an order to recall the concerned product or batch; however, the option

Page 25 of 340
WHO Global Benchmarking Tool (GBT)

for either limited or nationwide recalls should be available. In addition to product recalls,
other possible actions in association with the rapid alert include, but are not limited to,
suspension of distribution, quarantine in place, and importation alert. Furthermore, the
rapid alert (with or without recall) may be associated with suspension or withdrawal of the
marketing authorization for the related product. In addition, the rapid alert system may
entail different levels of implementation (and accordingly communication), for example,
national level, state level, district level, main store, and others throughout the organizations
involved in the distribution channel. The assessor should also review the related roles and
responsibilities, points of contact, communication channels, and decision trees, including 01
representative examples of rapid alerts issued over the last few years.

National Regulatory System (RS): Indicators and Fact Sheets

Objective: The objective of this sub-indicator is to confirm the existence of a rapid alert system
that is able to manage threats posed by SF medical products and ensure a timely and
proportionate response.
The existence of SF medical products is an unacceptable risk to public health because
they have the potential to harm patients and undermine confidence in medical products,
healthcare professionals and health systems.

Requirement: Management system of SF medical products

Evidence to review: The assessor should request for and review:

1. Regulations and documented procedures that address the management of threats by
SF medical products and recalls of such products from the market;
2. Number of complaints received;
3. Number of recalled medical products;
4. Documentation that information is publicly available, that the media used (web
page, official gazette or other NRA bulletin) are appropriate, and that the information is
maintained and updated on a regular basis;
5. Guidance given to manufacturers, importers, exporters and distributors on management
of recalls, including the destruction of product; 02
6. Procedures and related records for handling quality defects (e.g., notifications of
suspected quality defects and company batch recalls) 03
7. List of recalls performed and documented reports;
8. Procedures and records of the rapid alert system including responsibilities, points of 04
contact, communication channels and decision trees.
05
References: 1. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: 06
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical
Report Series, No. 885) (1) (http://apps.who.int/medicinedocs/documents/s21964en/ 07
s21964en.pdf)
2. Guidelines for national authorities on quality assurance for biological products. In: 08
WHO Expert Committee on Biological Standardization: forty-second report. World
Health Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2),
09
(http://www.who.int/biologicals/publications/trs/areas/biological_products/WHO_
TRS_822_A2.pdf)
3. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization: forty-fifth
report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical Report Series,
No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001.,
(116), (http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence of a rapid alert system to react to and
to manage threats by SF medical products.
 ONGOING IMPLEMENTATION (OI): There is no evidence of a rapid alert system to
react to and to manage threats by SF medical products; however, demonstrable steps
have been taken towards its development.

Page 26 of 340
WHO Global Benchmarking Tool (GBT)

 PARTIALLY IMPLEMENTED (PI): There is a draft document defining a rapid alert

system to react to and to manage threats by SF medical products.
 IMPLEMENTED (I): There is documented evidence of a rapid alert system to react to
and to manage threats by SF medical products.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: RS04.03: A rapid alert and recall system based on documented communication to the 01
appropriate level of the distribution channel and with a feedback mechanism.

National Regulatory System (RS): Indicators and Fact Sheets

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify how the alert and recall system works, focusing on
communications to the appropriate level across the distribution channel along with the
confirmatory feedback mechanism.
In a timely process, the decision taken by the NRA should be shared appropriately and
should state the severity of the defect. Communications should be given to the entire
supply chain and to all stakeholders using the fastest mode of communication available,
which may include email, telephone, fax, text or other appropriate means. Immediately
after the decision to recall has been made, the licensee, or representative of licensee
where the product is marketed, should inform the concerned regulatory authorities in the
country in which any affected product batches were distributed. Further actions on recall
should be undertaken according to the recall class and a feedback mechanism should be
established.

Objective: The objective of this sub-indicator is to ensure a system is in place to recall from the 02
market, promptly and effectively, products known or suspected to be defective. Recall is an
action taken to withdraw and remove the medical products from distribution or use. The 03
process should include corrective actions taken for deficiencies reported on quality, safety
or efficacy. Recall operations should be capable of being initiated promptly down to the 04
required level in the distribution channel and a feedback mechanism should be in place.
05
Requirement: A rapid alert and recall system
06
Evidence to review: The assessor should request for and review:
1. Rapid alert and recall procedures along with the associated records and 07
communications;
2. List of recalls, examples of recalls and recall notifications. 08

References: 1. National drug regulatory legislation: guiding principles for small drug regulatory
09
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations:
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical
Report Series, No. 885) (1) (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf)
2. Guidelines for national authorities on quality assurance for biological products. In:
WHO Expert Committee on Biological Standardization: forty-second report. World Health
Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2), (http://www.who.
int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_A2.pdf)
3. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization: forty-fifth
report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical Report Series,
No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001.,
(116), (http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)

Framework: Process

Page 27 of 340
WHO Global Benchmarking Tool (GBT)

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence of a rapid alert and recall system
based on documented communication to the appropriate level of the distribution
channel and with a feedback mechanism.
 ONGOING IMPLEMENTATION (OI): There is no evidence of a rapid alert and
recall system based on documented communication to the appropriate level of
the distribution channel, but demonstrable steps have been taken towards its
development.
 PARTIALLY IMPLEMENTED (PI): There is a draft document defining a rapid alert and
recall system based on documented communication to the appropriate level of the 01
distribution channel.

National Regulatory System (RS): Indicators and Fact Sheets

 IMPLEMENTED (I): There is documented evidence of a rapid alert and recall system
based on documented communication to the appropriate level of the distribution
channel and with a feedback mechanism.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: RS04.04: Recall system based on documented confirmation that appropriate,
batch-traceable action and/or destruction has been undertaken when necessary.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the existence of documentation which covers the regulatory
decisions related to product recalls, and which can provide sufficient evidence that proper
actions have been taken. There should be written procedures describing the actions to be
taken, including the need to consider a recall in the case of a complaint concerning a possible
product defect. If a product defect is discovered or suspected in a batch, consideration 02
should be given to whether other batches should be checked to determine whether they
have been also affected. In particular, other batches that may contain reprocessed product 03
from the defective batch should be investigated. Where necessary, appropriate follow-up
action, possibly including product recall, should be taken after investigation and evaluation 04
of the complaint. All decisions made, and all measures taken as a result of a complaint or
detection of unqualified products should be recorded and referenced to the corresponding 05
batch records. Complaints records should be regularly reviewed for any indication of specific
or recurring problems that require attention and that might justify the recall of marketed 06
products. The competent authorities should be informed if a manufacturer or other
relevant stakeholder is considering action following possibly faulty manufacture, product 07
deterioration, or detection of any other serious quality problems with a product.
08
Objective: The objective of this sub-indicator is to ensure that a recall system exists, and the progress
of the recall process is monitored and recorded. Records include the disposition of the
09
batches and/or destruction when necessary.

Requirement: Recall system based on documented confirmation

Evidence to review: The assessor should request for and review:

1. Documented evidence of the fate of recalled batches;
2. Documentation of reconciliation and recalls success rate (i.e. comparison of actual
recalled units versus distributed ones within a certain timeframe).

Page 28 of 340
WHO Global Benchmarking Tool (GBT)

www.who.int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_
A2.pdf)
3. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization: forty-fifth
report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical Report Series,
No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001.,
(116), (http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)
5. WHO good manufacturing practices for pharmaceutical products: main principles (51) 01
(http://digicollection.org/whoqapharm/p/about/) and (http://apps.who.int/medicinedocs/en/)

National Regulatory System (RS): Indicators and Fact Sheets

Framework: “Structure/Foundation/Input”

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence of a recall system based on
documented confirmation that appropriate, batch-traceable action and/or destruction
has been undertaken when necessary.
 ONGOING IMPLEMENTATION (OI):There is no evidence of a recall system based on
documented confirmation that appropriate, batch-traceable action and/or destruction
has been undertaken when necessary; however, demonstrable steps have been taken
towards its development.
 PARTIALLY IMPLEMENTED (PI): There is a draft document defining a recall system
based on documented confirmation that appropriate, batch-traceable action and/or
destruction has been undertaken when necessary.
 IMPLEMENTED (I): There is documented evidence of a recall system based on
documented confirmation that appropriate, batch-traceable action and/or destruction
has been undertaken when necessary.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).
02
Sub Indicator: RS04.05: Written criteria to cover circumstances in which the routine regulatory
processes may not have to be followed in relation to crises and emergencies linked 03
to a risk management plan.
04
Maturity Level: 3
05
Scope: 1. Medicines
2. Vaccines 06

Description: The assessor should verify the existence of a risk management plan ready to be 07
implemented in circumstances in which the routine regulatory processes may not be
followed when responding to crises and emergencies (e.g. outbreaks, force majeure, 08
and medical product shortages). In addition, the criteria should be defined in a written
document to describe management of crises and emergencies.
09
A crises and emergencies plan is a course of action developed to mitigate the damage
caused by those events that could endanger an organization’s ability to function. Such
a plan should include measures that provide for the safety of personnel and, if possible,
property and facilities. It should also include provisions to assess the severity of an
incident and to implement steps to eliminate the problem. Development of the plan begins
with a vulnerability assessment. The results of the assessment will show:
1. The likelihood of a situation occurring;
2. The means that are available to stop or prevent the situation;
3. The potential responses to a given situation.
From this analysis, appropriate crisis and emergency management procedures can be
established. At the planning stage, it is important to employ a multidisciplinary team
approach. Communication, training and periodic drills will ensure adequate performance if
the plan must be carried out.

Objective: The objective of this sub-indicator is to ensure that procedures related to NRA response
to relevant crises and emergencies have been developed, documented and tested prior

Page 29 of 340
WHO Global Benchmarking Tool (GBT)

to the occurrence of an event. Such procedures should be developed according to a risk

management plan.
An important element of any system for the prevention of major incidents is the
establishment of an emergency plan. Crises and emergencies plans seek to minimize
the consequences of a crisis or emergency and require the timely application of defined
procedures by people with adequate training and resources.

Requirement: Crises and Emergencies Plan

01
Evidence to review: The assessor should request for and review:

National Regulatory System (RS): Indicators and Fact Sheets

1. Regulatory risk management plan to cover circumstances in which the routine
regulatory
processes may not be followed when responding to crises and emergencies;
2. Written documents that describe the process;
3. Any records of crises management.

References: 1. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations:
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical
Report Series, No. 885) (1) (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf)
2. Guidelines for national authorities on quality assurance for biological products. In:
WHO Expert Committee on Biological Standardization: forty-second report. World
Health Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2), (http://
www.who.int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_
A2.pdf)
3. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization:
forty-fifth report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical
Report Series, No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_ 02
TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001., (116), 03
(http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)
04
Framework: Structure/Foundation/Input
05
Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence of written criteria to cover
circumstances in which the routine regulatory processes may not be followed in 06
relation to crises and emergencies.
 ONGOING IMPLEMENTATION (OI): There is no evidence of written criteria to cover 07
circumstances in which the routine regulatory processes may not have to be followed
in relation to crises and emergencies, but demonstrable steps have been taken 08
towards its development.
 PARTIALLY IMPLEMENTED (PI): There is a draft document defining written criteria to
09
cover circumstances in which the routine regulatory processes may not have to be
followed in relation to crises and emergencies.
 IMPLEMENTED (I): There is documented evidence of written criteria to cover
circumstances in which the routine regulatory processes may not have to be followed
in relation to crises and emergencies.

Limitations and remarks: If routine regulatory processes are mandatory in all circumstance based on the regulation,
the assessor should review how the country can handle an emergency condition in a timely
manner, for example, by providing additional staff and resources that allow regulatory
activities to be conducted under emergency conditions.

Indicator: RS05 Quality management systems (QMS) including the risk management
principles are applied and realized.

Objective: The objective of this indicator is to confirm whether the NRA has implemented the main
principles of a Quality Management System (QMS) which includes the application of risk

Page 30 of 340
WHO Global Benchmarking Tool (GBT)

management principles for all regulatory functions and which has been updated to include
the required documentation.
QMS is a valuable tool that helps NRAs to achieve greater credibility for their decisions and
greater stability in their operations, to include systematic planning, control, and improved
quality in all processes throughout all regulatory functions, and to ensure a comprehensive
approach for all.

Category: 05. Quality and risk management system

01
Sub Indicator: RS05.01: Top management demonstrates commitment and leadership to develop

National Regulatory System (RS): Indicators and Fact Sheets

and implement quality management system (QMS).

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The commitment and leadership with respect to QMS is demonstrated by:
1. taking accountability for the effectiveness of the QMS;
2. ensuring that the quality policy and quality objectives are established for the QMS
and that they are compatible with the visions, mandates and strategic directions of the
organization;
3. ensuring the integration of the QMS requirements into the organization’s business processes;
4. promoting the use of the process approach and risk-based thinking;
5. ensuring that the resources needed for the QMS are available;
6. communicating the importance of effective quality management and of conformance to
the QMS requirements;
7. ensuring that the QMS achieves its intended results;
8. engaging, directing and supporting persons to contribute to the effectiveness of the QMS;
9. promoting continuous QMS improvement; 02
10. supporting other relevant management roles to demonstrate their leadership as it
applies to their areas of responsibility. 03

Objective: The objective of this sub-indicator is to confirm the commitment and leadership from 04
top management that is required for the proper development, implementation and
maintenance of the QMS. 05

Requirement: Leadership and commitment 06

Evidence to review: The assessor should request for and review: 07

1. Documentation of the resources (financial, human, infrastructure, and equipment)
allocated for the development, implementation and maintenance of QMS; 08
2. Communication aids that convey to all staff levels the importance of effective QMS
3. Documentation for top management involvement in the monitoring, review and
09
continuous improvement of QMS.

References: 1. Quality management systems — Requirements. International Standard ISO 9001:2015.

International Organization for Standardization, 2015 sub-clause 5.1.1., (4), (https://www.
iso.org/iso-9001-quality-management.html)

Framework: Structure/Foundation/Input

Rating Scale:  ONGOING IMPLEMENTATION (OI): There is no evidence that top management
demonstrates commitment and leadership to develop and implement QMS, but
demonstrable steps have been taken in this direction.
 PARTIALLY IMPLEMENTED (PI): Some resources (financial, human, infrastructure
and equipment) have been allocated for the development, implementation, and
maintenance of QMS.
 IMPLEMENTED (I): There is documented evidence that top management demonstrate
commitment and leadership to develop and implement QMS.

Page 31 of 340
WHO Global Benchmarking Tool (GBT)

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: RS05.02: Quality policy, objectives, scope and action plans for establishment
of the QMS are in place and communicated to all levels.

Maturity Level: 3

Scope: 1. Medicines 01
2. Vaccines

National Regulatory System (RS): Indicators and Fact Sheets

Description: The assessor should verify that top management has established, implemented and
maintained a quality policy that:
1. is appropriate to the purpose and context of the NRA and supports its strategic
direction;
2. provides a framework for setting quality objectives;
3. includes a commitment to satisfy applicable requirements;
4. includes a commitment to continued improvement of the QMS;
5. is communicated to all levels across the organization.
The assessor should verify that the quality policy:
1. Is available and is maintained as documented information;
2. is communicated, understood and applied within the NRA;
3. is available to relevant interested parties, as appropriate.
The assessor should verify that the NRA has established quality objectives for
functions, levels and processes needed for the QMS.
The quality objectives should:
1. be consistent with the quality policy;
2. be measurable;
3. take into account applicable requirements;
4. be relevant for conformity of products and services and for enhancement of 02
customer satisfaction;
5. be monitored; 03
6. be communicated;
7. be updated as appropriate. 04
The NRA should maintain documented information on the quality objectives. When NRA
is planning how to achieve its quality objectives, the NRA should determine: 05
1. what will be done;
2. what resources will be required; 06
3. who will be responsible;
4. when it will be completed; 07
5. how the results will be evaluated.
08
Objective: The objective of this sub-indicator is to confirm that a policy statement to support QMS
is issued by senior management or the head of institution, that a development plan
09
exists, and that implementation of the plan is supported by senior management. The
quality policy should be communicated to all levels across the organization.

Requirement: Quality policy, objectives, scope, and action plans

Evidence to review: The assessor should request for and review:

1. Evidence that the quality policy is available, distributed to different levels of the
organization and understood by the staff;
2. Evidence that quality objectives are documented and communicated to all levels;
3. Evidence that there is agreement on the scope of QMS application;
4. Evidence that the action plan for QMS establishment and maintenance is available
and regularly updated.

References: 1. Quality management systems — Requirements. International Standard ISO 9001:2015.

International Organization for Standardization, 2015 sub-clause 5.2.1, 5.2.2 , 6.2., (4),
(https://www.iso.org/iso-9001-quality-management.html)

Page 32 of 340
WHO Global Benchmarking Tool (GBT)

Framework: “Structure/Foundation/Input”

Rating Scale:  NOT IMPLEMENTED (NI): The NRA has no quality policy for the establishment of the
QMS.
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps toward the quality policy
for the establishment of a QMS and there is documented evidence of this.
 PARTIALLY IMPLEMENTED (PI): The quality policy for the establishment of the QMS
was recently established, a quality system policy was implemented less than a year
ago, or there is a QMS, but it does not yet cover all regulatory functions. 01
 IMPLEMENTED (I): The NRA has all of the aforementioned elements, and consistently

National Regulatory System (RS): Indicators and Fact Sheets

maintains documentation of the results of related activities over time.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: RS05.03: Organizational chart, with roles and responsibilities

to establish the QMS are defined and in place.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that top management assign the responsibility and authority
for:
1. ensuring that the QMS conforms to the requirements of international standards;
2. ensuring that the processes are delivering their intended outputs;
3. reporting on the performance of the QMS and on opportunities for improvement, in
particular to top management;
4. ensuring the promotion of customer focus throughout the NRA; 02
5. ensuring that the integrity of the QMS is maintained when changes to the QMS are
planned and implemented. 03

Objective: The objective of this sub-indicator is to confirm that top management ensures that 04
responsibilities and authorities for relevant roles are assigned, communicated, understood,
and followed within the NRA in order to establish and maintain the QMS. 05

Requirement: Organizational roles, responsibilities and authorities 06

Evidence to review: The assessor should identify the entities that are in place to establish, implement or 07
maintain the QMS. The different responsibilities should be identified and assigned. So, the
assessor should check and evaluate that the: 08
1. Organizational chart is documented and authorized;
2. Responsibilities & authorities for processes are assigned;
09
3. Job descriptions that include roles, responsibilities, and authorities for each block in the
organizational chart have been written and approved

References: Quality management systems — Requirements. International Standard ISO 9001:2015.

International Organization for Standardization, 2015 sub-clause 5.3., (4), (https://www.iso.
org/iso-9001-quality-management.html)

Framework: “Structure/Foundation/Input”

Rating Scale:  NOT IMPLEMENTED (NI): There are no defined roles and responsibilities (with or
without organizational chart) to establish the QMS.
 ONGOING IMPLEMENTATION (OI): There is no organizational chart, with defined roles
and responsibilities to establish the QMS, but demonstrable steps have been taken
towards its development.
 PARTIALLY IMPLEMENTED (PI): The organizational chart, with defined roles and
responsibilities to establish the QMS, was recently developed.

Page 33 of 340
WHO Global Benchmarking Tool (GBT)

 IMPLEMENTED (I): There is an organizational chart, with defined roles and

responsibilities, and the QMS is established.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: RS05.04: Enough competent staff is assigned to develop,

implement and maintain the QMS.
01
Maturity Level: 3

National Regulatory System (RS): Indicators and Fact Sheets

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify whether the NRA:

1. Has determined the adequate number of staff to implement and maintain QMS;
2. Has determined the necessary competence of NRA personnel doing work that affects
the performance and effectiveness of the QMS;
3. Has ensured that these personnel are competent on the basis of appropriate education,
training, skills and experience;
4. Has taken actions to acquire the necessary competence, and evaluate the effectiveness
of the actions taken, where applicable;
5. Has retained appropriate documented information as evidence of competence.

Objective: The objective of this sub-indicator is to ensure the NRA provides the human resources
(i.e., with respect to number and competence of personnel) necessary for the effective
implementation of its QMS and for the operation and control of its processes. Necessary
competence for personnel performing work significantly influences the work output and
outcome.
02
Requirement: Human resources and competency
03
Evidence to review: The assessor should evaluate and verify if adequate competent (i.e., with respect to
education, training, skills and experience) human resources has been designated and 04
assigned to develop, implement and maintain QMS. The assessor should check and
evaluate the following: 05
1. Criteria for the necessary personnel competence (e.g., competency matrix for each
function); 06
2. Staff training plan (addressing recruitment, on-the-job training and other actions to
achieve the necessary competencies); 07
3. Appropriate training records and system for retention of these records.
08
References: 1. Quality management systems — Requirements. International Standard ISO 9001:2015.
International Organization for Standardization, 2015 sub-clause 7.1.2, 7.2., (4), (https://
09
www.iso.org/iso-9001-quality-management.html)

Framework: “Structure/Foundation/Input”

Rating Scale:  NOT IMPLEMENTED (NI): The NRA does not have adequate competent staff
(education, training, skills and experience) to develop, implement and maintain the
QMS.
 ONGOING IMPLEMENTATION (OI): The NRA has recently developed the
plan to recruit adequate competent staff; however, the plan has not been
implemented.
 PARTIALLY IMPLEMENTED (PI): The NRA recently recruited the adequate number of
the staff to develop, implement and maintain the QMS and their training programmes
or competency building activities are ongoing.
 IMPLEMENTED (I): The NRA has had a sustained number of competent staff
(education, training, skills and experience) assigned to develop, implement and
maintain the QMS.

Page 34 of 340
WHO Global Benchmarking Tool (GBT)

Limitations and remarks:  Applicable actions can include, for example, the provision of training to, the
mentoring of, or the reassignment of currently employed personnel, as well as
the hiring or).
 contracting of competent personal. Regarding effectiveness of training please
refer to sub-indicator RS05.14
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator
will always apply for all benchmarked NRAs

Sub Indicator: RS05.05: The NRA establishes mechanisms to continually improve the QMS. 01

National Regulatory System (RS): Indicators and Fact Sheets

Maturity Level: 4

Scope: 1. Medicines
2. Vaccines

Description: Continuous improvement is a constant theme of a QMS. QMS continuous improvement

can be achieved through different complementary approaches including, for example,
management review, customer feedback, and process and product evaluation. The
NRA should continually work to improve the adequacy and effectiveness of the QMS.
The NRA should consider the results of analyses and evaluations, and the outputs from
management reviews, to determine if there are needs or opportunities that should be
addressed as part of continual improvement.

Objective: The objective of this sub-indicator is to ensure that the NRA establishes, implements,
maintains and continually improves the QMS. The NRA should identify the processes
needed for developing and maintaining the QMS and should adopt the mechanisms
necessary to ensure continued improvement.

Requirement: Continual improvement 02

03
Evidence to review: The assessor should evaluate and verify whether the NRA has established, documented
and implemented a QMS, and has worked continually to improve its effectiveness 04
through management review.
The assessor should check and evaluate the following: 05
1. mechanisms of QMS monitoring and evaluation and ways for figuring out and
addressing identified gaps; 06
2. availability of resources needed for QMS continual improvement;
3. availability of documentation related to addressing risks, opportunities and outputs 07
from management review;
4. availability of documents that describe processes for QMS and regulatory system 08
evaluation and that identify processes that require optimization;
5. availability of documents to improve the processes and address QMS gaps.
09

References: 1. Quality management systems — Requirements. International Standard ISO 9001:2015.

International Organization for Standardization, 2015 sub-clause 10.3., (4), (https://www.iso.
org/iso-9001-quality-management.html)

Framework: “Structure/Foundation/Input”

Rating Scale:  NOT IMPLEMENTED (NI): The NRA has no QMS.

 ONGOING IMPLEMENTATION (OI): The NRA is taking steps toward the establishment
of a QMS and there is documented evidence of this.
 PARTIALLY IMPLEMENTED (PI): The NRA’s QMS and its system of documentation
were recently established, one of the systems was implemented less than a year ago,
or there is a QMS, but it does not yet cover all regulatory functions.
 IMPLEMENTED (I): The NRA has established mechanisms to continually improve the
QMS and consistently maintains documentation of the results of related activities
over time.

Page 35 of 340
WHO Global Benchmarking Tool (GBT)

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: RS05.06: The NRA has identified its regulatory processes, determined their interactions
and defined the methods needed to control these processes.

Maturity Level: 4

Scope: 1. Medicines 01
2. Vaccines

National Regulatory System (RS): Indicators and Fact Sheets

Description: The assessor should verify that the NRA determines the processes needed for the QMS
and their application. The NRA should:
1. determine the regulatory inputs required and the outputs expected;
2. determine the sequence (i.e. regulatory workflow including all related steps assigned to
each unit within NRA and those assigned to other stakeholders e.g. industry) and
interactions among these processes;
3. determine and apply the criteria and methods (including monitoring, measurements and
related performance indicators) needed to ensure the effective operation and control of
these processes;
4. determine the resources needed for these processes and ensure their availability;
5. assign the responsibilities and authorities for these processes;
6. address the risks and opportunities;
7. evaluate these processes and implement any changes needed to ensure that these
processes achieve their intended results;
8. improve the processes and the QMS.
To the extent necessary, the NRA should:
1. maintain documented information to support the operation of its processes;
2. retain documented information to provide confidence that the processes are being
carried out as planned. 02
The NRA should apply controls to the design and development process to ensure that:
1. the results to be achieved are defined; 03
2. reviews are conducted to evaluate the ability of the results of design and development
actions to meet requirements; 04
3. verification activities are conducted to ensure that the design and development of the
regulatory outputs (e.g. approved marketing authorizations) meet the input requirements 05
(e.g., marketing authorization standards and regulations);
4. validation activities are conducted to ensure that the resulting products and services 06
(e.g. laboratory testing) meet the requirements for the specified application or intended use
(e.g. pharmacopeial standards); 07
5. necessary actions are taken on problems determined during the reviews or verification
and validation activities; 08
6. documented information of these activities is retained.
The NRA should implement production and service provisions under controlled conditions.
09
Controlled conditions should include, as applicable:
1. the availability of documented information that define:
a) the characteristics of the products to be produced, the services to be provided, or the
activities to be performed;
b) the results to be achieved;
2. the availability and use of suitable monitoring and measuring resources;
3. the implementation of monitoring and measurement activities at appropriate stages to
verify that criteria for control of processes or outputs, and acceptance criteria for products
and services, have been met;
4. the use of suitable infrastructure and environment for the operation of processes;
5. the appointment of competent personnel with any required qualifications;
6. the validation, and periodic revalidation, of the ability to achieve planned results of the
processes for production and service provision, where the resulting output cannot be
verified by subsequent monitoring or measurement;
7. the implementation of actions to prevent human error;
8. the implementation of release, delivery and post-delivery activities.

Page 36 of 340
WHO Global Benchmarking Tool (GBT)

Design and development reviews, verifications and validations have distinct purposes.
They can be conducted separately or in any combination, as is suitable for the products
and services of the organization.

Objective: The objective of this sub-indicator is to confirm the organization has established,
implemented and maintained a design and development process that is appropriate to
ensure the subsequent provision of products and services. The NRA should establish
methods and tools to monitor proper implementation.
01
Requirement: Operation

National Regulatory System (RS): Indicators and Fact Sheets

Evidence to review: The assessor should request and review evidence for:
1. availability of processes list;
2. availability of different regulatory processes flowcharts including process controls and
interactions with other processes.

References: 1. Quality management systems — Requirements. International Standard ISO 9001:2015.

International Organization for Standardization, 2015 sub-clause 4.4, 8.3, 8.5.1., (4), (https://
www.iso.org/iso-9001-quality-management.html)

Framework: “Structure/Foundation/Input”

Rating Scale:  NOT IMPLEMENTED (NI): The NRA has not identified its regulatory processes.
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps toward the establishment
of its regulatory processes and there is documented evidence of this.
 PARTIALLY IMPLEMENTED (PI): The NRA has recently identified its regulatory processes,
determined their interactions and defined the methods needed to control these processes.
 IMPLEMENTED (I): The NRA has identified its regulatory processes and determined
their interactions, and there is documented evidence that the regulatory processes
are under control. 02

Limitations and remarks:  The implementation of this requirement can be evaluated with more detail under each 03
function.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will 04
always apply for all benchmarked NRAs).
05
Sub Indicator: RS05.07: Requirements for documentation management as well as traceability
of regulatory activities are established. 06

Maturity Level: 2 07

Scope: 1. Medicines 08
2. Vaccines
09
Description: The assessor should verify the NRA’s QMS includes:
1. documented information required by the applicable International Standard;
2. documented information determined by the organization to be necessary for the
effectiveness of the QMS.
The assessor should check that the NRA, when creating and updating documented
information, ensures that:
1. identifications and descriptions (e.g. a title, date, author, or reference number) are
appropriate;
2. format (e.g., language, software version, and graphics) and media (e.g. paper or
electronic) are appropriate;
3. reviews and approvals are suitable and adequate.
The assessor should verify that documented information required by the QMS and by the
applicable International Standard should be controlled to ensure that information is:
1. available and suitable for use, where and when it is needed;
2. adequately protected (e.g. from loss of confidentiality, improper use, or loss of integrity).
With respect to the control of documented information, the assessor should check that the

Page 37 of 340
WHO Global Benchmarking Tool (GBT)

NRA addresses the following activities, as applicable:

1. distribution, access, retrieval and use;
2. storage and preservation, including preservation of legibility;
3. control of changes (e.g. version control);
4. retention and disposition.
The assessor should confirm that documented information of external origin that the NRA
considers to be necessary for QMS planning and operation is appropriately identified and
controlled.
The assessor should verify that documented information retained as evidence of 01
conformity is protected from unintended alterations. Procedures should identify those

National Regulatory System (RS): Indicators and Fact Sheets

personnel who have permission and authority to view and change the documented
information, and those personnel who have permission only to view the documented
information.

Objective: The objective of this sub-indicator is to ensure the NRA has a suitable document control
procedure for the organization, identification, storage, protection, retrieval, retention, and
disposition of documents (i.e., procedures and records).

Requirement: Documented information

Evidence to review: The assessor should evaluate and verify this requirement by reviewing:
1. Record control procedures;
2. Record control system, including identification of records, safeguard of records, retrieval
of records, retention of records for defined periods, and destruction of obsolete records;
3. Mechanisms for ensuring traceability of different regulatory activities;
4. Examples of records;
5. Records access, archiving, retention, and disposition.

References: 1. Quality management systems — Requirements. International Standard ISO 9001:2015.

International Organization for Standardization, 2015 sub-clause 7.5., (4), (https://www.iso. 02
org/iso-9001-quality-management.html)
03
Framework: “Structure/Foundation/Input”
04
Rating Scale:  NOT IMPLEMENTED (NI): The NRA has no requirements for documentation
management or cannot trace its regulatory activities and decisions. 05
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps toward the establishment
of its requirements for documentation management and traceability of regulatory 06
activities; however, there is no documented evidence for this.
 PARTIALLY IMPLEMENTED (PI): The NRA has recently identified its requirements for 07
documentation management and traceability of regulatory activities.
 IMPLEMENTED (I): The NRA has identified its requirements for documentation 08
management and traceability of regulatory activities and there is documented
evidence of the same
09
.
Limitations and remarks: The extent of documented information for a QMS can differ from one organization to
another due to:
1. the size of NRA and the scope of its mandate with respect to types of activities,
processes, products and services;
2. the complexity of processes and their interactions;
3. the competence of personnel;
4. technical resources and technologies available.
Records must remain legible and identifiable throughout their retention period.
Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: RS05.08: External and internal issues including relevant potential risks are
defined and assessed periodically for proper risk mitigation.

Maturity Level: 4

Page 38 of 340
WHO Global Benchmarking Tool (GBT)

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that the NRA monitors and reviews information about external
and internal issues. Issues can include positive and negative factors or conditions for
consideration. Understanding the external context can be facilitated by considering
issues arising from legal, technological, competitive, market, cultural, social and economic
environments, whether international, national, regional or local. Understanding the internal
context can be facilitated by considering issues related to values, culture, knowledge and 01
performance of the organization.

National Regulatory System (RS): Indicators and Fact Sheets

The assessor should confirm the NRA considers the issues and requirements referred
to the organization when planning for the QMS. The risks and opportunities need to be
addressed in order to:
1. give assurance that the QMS can achieve its intended result(s); nization or its
customers. 2. enhance desirable effects;
3. prevent, or reduce, undesired effects;
4. achieve improvement.
The assessor should verify the NRA has planned:
1. actions to address these risks and opportunities;
2. steps to:
a) integrate and implement the actions into its QMS processes;
b) evaluate the effectiveness of these actions.
The assessor should check that actions taken to address risks and opportunities are
proportionate to the potential impact on the conformity of products and services.
Options to address risks can include avoiding risks, taking risks in order to pursue an
opportunity, eliminating the risk source, changing the likelihood or consequences of the
risk, sharing the risk, or retaining risk by informed decision.
Opportunities can lead to the adoption of new practices, launching new products,
addressing new customers, building partnerships, and using new technology and other
desirable and viable possibilities to address the needs of the organization or its customers. 02

Objective: The objective of this sub-indicator is to ensure the organization determines external and 03
internal issues that are relevant to its purpose and strategic direction and that may affect
its ability to achieve the intended QMS result. When planning for the QMS, the NRA should 04
consider the issues and requirements and determine the risks and opportunities that need
to be addressed in the context of the organization. The NRA should identify issues that 05
may eventually impact the regulatory performance in a negative way and develop actions
and steps needed to implement the mitigation strategy. Such strategies can provide 06
assurance that the QMS can achieve its intended results.
07
Requirement: Actions to address risk and opportunities
08
Evidence to review: The assessor should evaluate and verify the integration of this requirement by reviewing:
1. Evidence for identification of potential risks for each process, and for creation of a list
09
of potential risks;
2. Methodology and SOPs for risk analysis with examples of some potential risks (e.g.,
calculations of likelihood and severity and creation of risk index);
3. Evidence for availability of risk mitigation action plan or strategy to control the potential
risks based on its risk assessment.

References: 1. Quality management systems — Requirements. International Standard ISO 9001:2015.

International Organization for Standardization , 2015 sub-clause 6.1, 4.1., (4), (https://www.
iso.org/iso-9001-quality-management.html)

Framework: “Structure/Foundation/Input”

Rating Scale:  NOT IMPLEMENTED (NI): The NRA has not identified or prioritized potential risks or
defined essential processes for risk mitigation.
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps towards the identification
and prioritization of potential risks and defining essential processes for risk mitigation,

Page 39 of 340
WHO Global Benchmarking Tool (GBT)

and there is documented evidence of this.

 PARTIALLY IMPLEMENTED (PI): The NRA has recently identified and prioritized
potential risks, defined essential processes and developed SOPs for risk analysis and
risk mitigation plan.
 IMPLEMENTED (I): The NRA has identified and prioritized potential risks,
defined essential processes, and implemented SOPs for risk analysis. There is a
mitigation action plan or strategy to control the potential risks based on its risk
assessment.
01
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will

National Regulatory System (RS): Indicators and Fact Sheets

always apply for all benchmarked NRAs).

Sub Indicator: RS05.09: The externally provided products and services relevant to regulatory
activities are controlled through established mechanisms.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that the organization has determined the controls to be applied
to externally provided processes, products and services when:
1. products and services from external providers are intended for incorporation into the
organization’s own products and services;
2. products and services are provided directly to the customers by external providers on
behalf of the organization;
3. a process, or part of a process, is provided by an external provider as a result of a
decision by the NRA.
The NRA should retain documented information of these activities and any necessary 02
actions arising from the evaluations.
The NRA should ensure that externally provided processes, products and services do not 03
adversely affect the NRA’s ability to consistently deliver conforming products and services
to its customers. 04
The assessor should verify the NRA should:
1. ensure that externally provided processes remain within the control of its QMS; 05
2. define both the controls that it intends to apply to an external provider and those that it
intends to apply to the resulting output; 06
3. take into consideration:
a) the potential impact of the externally provided processes, products and services on 07
the organization’s ability to consistently meet customer and applicable statutory and
regulatory requirements; 08
b) the effectiveness of the controls applied by the external provider;
4. determine the verification steps, or other activities, necessary to ensure that the
09
externally provided processes, products and services meet requirements.
The NRA should ensure the adequacy of requirements prior to their communication to the
external provider.
The NRA should communicate to external providers its requirements for:
1. the processes, products and services to be provided;
2. the approval of:
a) products and services;
b) methods, processes and equipment;
c) the release of products and services;
3. competence, including any required personnel qualifications, and the need to establish
terms of reference addressing confidentiality and potential conflicts of interest (as per
RS06.04 and RS09.06);
4. the external providers’ interactions with the NRA;
5. the controls and monitoring of the external providers’ performance that will be applied by
the NRA;
6. verification or validation activities.

Page 40 of 340
WHO Global Benchmarking Tool (GBT)

Objective: The objective of this sub-indicator is to confirm the NRA ensures that externally provided
processes, products and services conform to requirements. The NRA should determine
and apply criteria for the evaluation, selection, performance monitoring, and re- evaluation
of external providers. These criteria should be based on their ability to provide processes
or products and services in accordance with established requirements. An example of
externally provided services is outsourced laboratory testing which should ideally be
controlled through a quality agreement (with defined roles and responsibilities) between
the responsible unit within the NRA (contract giver) and the external laboratory (contract
acceptor). Another example of externally provided product is the information provided 01
by the regulated sector. In the latter example, it should be noted that some of the main

National Regulatory System (RS): Indicators and Fact Sheets

outputs from an NRA include those that establish parameters, rules and services for
regulated industry, and thus these actors from the regulated sector can be considered
both as suppliers and customers of the NRA services. This depends on the direction of the
workflow. In all cases, as part of the QMS establishment within the NRA, services, products
and customers should be well-defined and controlled.

Requirement: Control of externally provided process, products and services

Evidence to review: The assessor should evaluate and verify this requirement by reviewing:
1. List of externally provided products and services (e.g., tests, audits, or assessments)
that are relevant to regulatory activities;
2. Availability of criteria for selection, evaluation, and re-evaluation of suppliers and
outsourcing organizations;
3. Availability of list of approved suppliers and outsourcing organizations;
4. Availability of evaluation records of suppliers and outsourced organizations.

References: 1. Quality management systems — Requirements. International Standard ISO 9001:2015.

International Organization for Standardization, 2015 sub-clause 8.4., (4), (https://www.iso.
org/iso-9001-quality-management.html)
02
Framework: “Structure/Foundation/Input”
03
Rating Scale:  NOT IMPLEMENTED (NI): The NRA does not have a list of regulatory-relevant
externally-provided products and services. 04
 ONGOING IMPLEMENTATION (OI): The NRA does not have a list of regulatory-relevant
externally-provided products and services, but there are some demonstrable steps. 05
 PARTIALLY IMPLEMENTED (PI): The NRA has recently identified the list of regulatory-
relevant externally-provided products and services. 06
 IMPLEMENTED (I): The NRA has a list of regulatory-relevant externally-provided
products and services, has defined criteria for selection, evaluation, and re-evaluation 07
of suppliers and outsourcing organizations, and has records of evaluations.
08
Limitations and remarks:
09
Sub Indicator: RS05.10: A mechanism to evaluate the satisfaction of internal and external customers
and other interested parties is in place for system improvement.

Maturity Level: 4

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that the NRA monitors customers’ perceptions of the degree
to which their needs and expectations have been fulfilled. The NRA should determine the
methods for obtaining, monitoring and reviewing this information. Mapping or identifying
the customers and other stakeholder is essential for the proper implementation of this sub-
indicator.
Examples of monitoring customer perceptions can include customer surveys, customer
feedback on delivered products and services, meetings with customers, market-share
analysis, compliments, and warranty claims.

Page 41 of 340
WHO Global Benchmarking Tool (GBT)

Objective: The objective of this sub-indicator is to ensure the organization implements an effective
customer satisfaction process to record and evaluate outcomes. Effective arrangements
for communicating customer feedback and complaints should be in place.

Requirement: Customer satisfaction

Evidence to review: The assessor should evaluate and verify the implantation of this requirement by
reviewing:
1. Results of customer satisfaction measurement by means of surveys, complaints 01
analysis and claims analysis;

National Regulatory System (RS): Indicators and Fact Sheets

2. Actions taken with each unsatisfied customer (complaints or claims).

References: 1. Quality management systems — Requirements. International Standard ISO

9001:2015. International Organization for Standardization, 2015 sub-clause 9.1.2., (4),
(https://www.iso.org/iso-9001-quality-management.html)

Framework: “Output”.

Rating Scale:  NOT IMPLEMENTED (NI): The NRA does not perform customer satisfaction
measurement by means of surveys, complaints analysis and claims analysis.
 ONGOING IMPLEMENTATION (OI): The NRA does not perform customer
satisfaction measurement by means of surveys, complaints analysis and claims
analysis, but there are some demonstrable steps taken towards this end.
 PARTIALLY IMPLEMENTED (PI): The NRA has recently implemented customer
satisfaction measurement by means of surveys, complaints analysis and claims
analysis.
 MPLEMENTED (I): There is documented evidence that the NRA performs
customer satisfaction measurement by means of surveys, complaints analysis
and claims analysis and that actions are taken to deal with unsatisfied
customers. 02

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will 03
always apply for all benchmarked NRAs).
04
Sub Indicator: RS05.11: Internal and external audits of the QMS are established and conducted
at planned intervals. 05

Maturity Level: 3 06

Scope: 1. Medicines 07
2. Vaccines
08
Description: The assessor should verify that the NRA:
1. plans, establishes, implements and maintains an audit programme which specifies the
09
frequency, methods, responsibilities, planning, and reporting requirements, and which takes
into consideration the importance of the processes concerned, changes affecting the NRA,
and the results of previous audits;
2. defines the audit criteria and scope for each audit;
3. selects auditors and conducts audits to ensure objectivity and impartiality of the audit
process;
4. ensures that the results of the audits are reported to relevant management;
5. takes appropriate correction and corrective actions without undue delay;
6. retains documented information as evidence of the implementation of the audit
programme and the audit results.

Objective: The objective of this sub-indicator is to ensure that internal and external audits are
planned at regular intervals to provide information on whether the QMS conforms to the
organization’s own requirements and is effectively implemented and maintained.

Requirement: Internal audit

Page 42 of 340
WHO Global Benchmarking Tool (GBT)

Evidence to review: The assessor should evaluate and verify implementation of auditing system by reviewing:
1. Internal audit procedures;
2. Audit schedules;
3. Records of auditors’ qualifications and competence;
4. Audit records;
5. Root cause analyses and corrective actions to prevent recurrence of non-conformities.

References: 1. Quality management systems — Requirements. International Standard ISO 9001:2015.

International Organization for Standardization, 2015 sub-clause 9.2 (4) (https://www.iso. 01
org/iso-9001-quality-management.html)

National Regulatory System (RS): Indicators and Fact Sheets

2. ISO 19011: 2018 Guidelines for auditing management systems., International
Organization for Standardization., (117), (https://www.iso.org/standard/70017.html)

Framework: “Process”

Rating Scale:  NOT IMPLEMENTED (NI): The NRA has not established internal and external audits of
the QMS at planned intervals.
 ONGOING IMPLEMENTATION (OI): The NRA has not established internal and external
audits of the QMS at planned intervals, but some demonstrable steps have been taken.
 PARTIALLY IMPLEMENTED (PI): The NRA has recently established internal and
external audits of the QMS at planned intervals but these are not yet implemented.
 IMPLEMENTED (I): There is documented evidence that the NRA has established and
implemented internal and external audits of the QMS at planned intervals

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: RS05.12: Corrections, corrective actions, and other actions for risk mitigation
and overall improvement, are implemented and documented and their
effectiveness is verified. 02

Maturity Level: 4 03

Scope: 1. Medicines 04
2. Vaccines
05
Description: The assessor should verify that when a nonconformity occurs, including any arising from
complaints, the NRA should: 06
1. react to the nonconformity and, as applicable:
a) take action to control and correct it; 07
b) deal with the consequences;
2. evaluate the need for action to eliminate the cause of the nonconformity, in order that it 08
does not recur or occur elsewhere, by:
a) reviewing and analyzing the nonconformity;
09
b) determining the causes of the nonconformity;
c) determining if similar nonconformities exist or could potentially occur;
3. implement any action needed;
4. review the effectiveness of any corrective action taken;
5. Update, if necessary, risks and opportunities determined during planning;
6. make any required changes to the QMS.
Corrective actions should be appropriate taking into consideration the effects of the
nonconformities encountered.
The NRA should retain documented information as evidence of:
1. the nature of the nonconformities and any subsequent actions taken;
2. the results of any corrective action

Objective: The objective of this sub-indicator is to ensure the NRA determines and selects
opportunities for improvement and implements any necessary actions to meet
customer requirements and enhance customer satisfaction. These should include
improving products and services to meet requirements and to address future needs and

Page 43 of 340
WHO Global Benchmarking Tool (GBT)

expectations, as well as correcting, preventing or reducing undesired effects. The goal

of these actions is to improve the performance and effectiveness of the QMS. Examples
of improvement can include correction, corrective action, continual improvement,
breakthrough change, innovation and re-organization.

Requirement: Non-conformity, correction and corrective action

Evidence to review: The assessor should evaluate and verify the status of corrective actions and actions for
risk mitigation and for overall improvement, as well as the implementation of these actions 01
and documentation of their effectiveness. The assessor should request for and review:

National Regulatory System (RS): Indicators and Fact Sheets

1. Evidence of implementing corrections, corrective actions & preventive actions;
2. Corrections and corrective action procedures with example of records;
3. Preventive action procedures with examples of records, if any.

References: 1. Quality management systems — Requirements. International Standard ISO 9001:2015.

International Organization for Standardization, 2015 sub-clause 10.2., (4), (https://www.iso.
org/iso-9001-quality-management.html)

Framework: “Process”

Rating Scale:  NOT IMPLEMENTED (NI): The NRA has not implemented corrections, corrective
actions, and other actions to mitigate risks.
 ONGOING IMPLEMENTATION (OI): The NRA has not implemented corrections,
corrective actions, and other actions to mitigate risks, but some demonstrable steps
have been taken.
 PARTIALLY IMPLEMENTED (PI): The NRA has recently implemented corrections,
corrective actions, and other actions to mitigate risks but these are not
documented.
 IMPLEMENTED (I): The NRA has implemented and documented corrections,
corrective actions, and other actions for risk mitigation and overall improvement. 02
The effectiveness of these actions is verified.
03
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 04

Sub Indicator: RS05.13: Top management reviews and documents the organization’s QMS 05
at planned intervals (i.e., management review).
06
Maturity Level: 4
07
Scope: 1. Medicines
2. Vaccines 08

Description: The assessor should verify that management reviews should be planned and carried out
09
taking into consideration:
1. the status of actions from previous management reviews;
2. changes in external and internal issues that are relevant to the QMS;
3. information on the performance and effectiveness of the QMS, including trends in:
a. customer satisfaction and feedback from relevant interested parties;
b. the extent to which quality objectives have been met;
c. process performance and conformity of products and services;
d. nonconformities and corrections, corrective actions and preventive actions;
e. monitoring and measurement results;
f. audit results;
g. the performance of external providers;
4. the adequacy of resources;
5. the effectiveness of actions taken to address risks and opportunities;
6. opportunities for improvement.
The outputs of the management review should include decisions and actions related to:
1. opportunities for improvement;

Page 44 of 340
WHO Global Benchmarking Tool (GBT)

2. any need for changes to the QMS;

3. resources need.
The organization should retain documented information as evidence of the results of
management reviews.

Objective: The objective of this sub-indicator is to ensure that top management reviews the
organization’s QMS at planned intervals, to ensure its continued suitability, adequacy,
effectiveness and alignment with the strategic direction of the organization.
01
Requirement: Management review

National Regulatory System (RS): Indicators and Fact Sheets

Evidence to review: The assessor should evaluate and verify that top management monitors the QMS at
planned intervals by review of the following:
1. Management review procedures;
actions),
2. Management review periodicity;
3. Management review records;
4. Management review inputs (e.g., audit results and customer satisfaction reports)
as well as outputs (e.g., decisions, corrections, corrective actions and preventive

References: 1. Quality management systems — Requirements. International Standard ISO 9001:2015.

International Organization for Standardization, 2015 sub-clause 9.3., (4), (https://www.iso.
org/iso-9001-quality-management.html)

Framework: “Structure/Foundation/Input”

Rating Scale:  NOT IMPLEMENTED (NI): The NRA top management does not review and document
the organization’s QMS at planned intervals (i.e., management review).
 ONGOING IMPLEMENTATION (OI): The NRA top management does not review and
document the organization’s QMS at planned intervals (i.e., management review), but 02
some demonstrable steps have been taken.
 PARTIALLY IMPLEMENTED (PI): The NRA has recently implemented management 03
reviews at planned intervals, but these are not documented.
 IMPLEMENTED (I): The NRA top management reviews and documents the 04
organization’s QMS at planned intervals (i.e., management review).
05
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 06

Sub Indicator: RS05.14: A mechanism is established to evaluate and demonstrate 07

the effectiveness of training activities.
08
Maturity Level: 4
09
Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the NRA reviews and evaluates whether planned training
meets their objectives. In addition, NRA should evaluate the learning effectiveness of
trained staff.

Objective: The objective of this sub-indicator is to ensure the NRA has established a mechanism
to evaluate and demonstrate the effectiveness of training activities. For the purpose of
this sub-indicator, training is not confined to QMS training activities but extends to all
training activities (e.g., technical, management, and soft (i.e., interpersonal)) skills.
Training is one of the key elements for building and developing personnel competencies.
In order to ensure proper conduct of training programs, training evaluations must be
performed. Training evaluations may cover single or multiple trainings, with or without
focus on individual trainees. Pre and/or post training examination is one form of
evaluation of individual training effectiveness. Other evaluation mechanisms include, for

Page 45 of 340
WHO Global Benchmarking Tool (GBT)

example, evaluation of the individual and overall performance of trainees, and evaluation of
behavioral changes in the trainees. Training effectiveness evaluation is essential
for ensuring that training objectives are met, and that participating staff has benefited
from it.

Requirement: Competency and training

Evidence to review: The assessor should evaluate and verify the effectiveness of training by reviewing:
1. Standard operating procedures (SOPs) for evaluation of the effectiveness of the training 01
and actions taken;

National Regulatory System (RS): Indicators and Fact Sheets

2. Samples of training evaluation records and analyses.

References: 1. Quality management systems — Requirements. International Standard ISO 9001:2015.

International Organization for Standardization, 2015 sub-clause 7., (4), (https://www.iso.
org/iso-9001-quality-management.html)

Framework: “Output”

Rating Scale:  NOT IMPLEMENTED (NI): There is no mechanism established to evaluate and
demonstrate the effectiveness of training activities.
 ONGOING IMPLEMENTATION (OI): Records to evaluate and demonstrate the
effectiveness of training activities have not been implemented; however some
demonstrable steps have been taken.
 PARTIALLY IMPLEMENTED (PI): Documentation to evaluate and demonstrate the
effectiveness of training activities has been recently implemented.
 IMPLEMENTED (I): Records to evaluate and demonstrate the effectiveness of training
activities are available and maintained over time.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 02

Indicator: RS06 Human resources to perform regulatory activities. 03

04
Objective: The objective of this indicator is to ensure that all entities within an NRA are adequately
resourced with a trained, experienced and skilled workforce that is empowered to fully 05
perform their regulatory functions. This will ensure that regulatory processes and activities
are performed in accordance with international best practices. 06
The government should take responsibility for planning and overseeing the development of
the necessary human resources. The strategies chosen should realistically reflect the needs 07
and capacity of the country, and an adequate budget should be allocated. Consideration of
the following aspects will help to ensure the development of a human resources policy that 08
is supportive of the national drug policy and national regulatory system implementation.
Planning should be initiated at an early stage and should consider short, medium and
09
longer-term needs. A quantitative analysis of the human resources needed (including a
realistic estimate of the attrition rate) may help to set priorities. Financial planning should
match the financial resources with priority needs. Good planning and appropriate lead times
will help to ensure that sufficient number of competent staff is available. Plans should
include a career development policy and measures to retain staff in the service.
The assessor should evaluate this general approach for the NRA by reviewing the human
resources chapters of each regulatory functions and the education, skills, experience and
training of the staff employed to perform the regulatory activities

Category: 06. Resources (HR, FR, infrastructure and equipment)

Sub Indicator: RS06.01: The NRA has the power to select and recruit its own staff following
documented procedures based on its own written criteria (i.e., education, training,
skills and experience).

Maturity Level: 4

Page 46 of 340
WHO Global Benchmarking Tool (GBT)

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the role and responsibilities of the NRA in the selection of the
personnel in its workforce. Assessor should verify that the NRA is involved in the framing
of the recruitment criteria as per the requirements of the job or function.
Key policy issues include:
1. government responsibility for planning and overseeing the development and training of
the human resources needed; 01
2. definitions for minimum education, training, skills and experience requirements for each

National Regulatory System (RS): Indicators and Fact Sheets

category of staff;
4. career planning and team building in government service;
5. the need for external assistance (i.e., national and international).

Objective: The objective of this sub-indicator is to ensure that human resources development policies
and strategies ensure that sufficient competent and motivated personnel are available
to implement the components of the national regulatory system. Lack of motivation or
appropriate expertise has been a decisive factor in failures to implement an effective and
efficient national regulatory system.

Requirement: Autonomy to select and recruit its own staff

Evidence to review: The assessor should request for and review:

1. Regulations that give power to NRA to select and hire their own staff;
2. Documented recruitment procedures that are based on its own written criteria (i.e.,
education, training, skills and experience).

Framework: “Structure/Foundation/Input”

Rating Scale:  NOT IMPLEMENTED (NI): The NRA has no power to select and recruit its own staff.
 ONGOING IMPLEMENTATION (OI): There is no regulation that gives NRA the
power to select and recruit its own staff however some demonstrable steps have
been taken.
 PARTIALLY IMPLEMENTED (PI): Documented procedures for recruitment based
on NRA’s own written criteria (i.e., education, training, skills and experience) have
been recently drafted but not yet implemented.
 IMPLEMENTED (I): The NRA has the power to select and recruit its own staff and
there are documented procedures and records based on its own written criteria
(i.e., education, training, skills and experience).

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Page 47 of 340
WHO Global Benchmarking Tool (GBT)

Sub Indicator: RS06.02: A periodic staff appraisal system is established to review performance and
competencies, to identify training needs, and to agree on performance targets.

Maturity Level: 4

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that a periodic staff appraisal system is established to review 01
performance and competencies, to identify training needs and to agree on performance

National Regulatory System (RS): Indicators and Fact Sheets

targets. For each category of personnel, the nature and extent of their involvement, as well
as their roles and responsibilities should be clearly defined. This will make it possible to
decide on the discipline and level of training required for each category. There should be a
number of minimum educational and training requirements for each category. In addition,
there is need to evaluate the competency and performance of staff members at defined
intervals, to ensure the adequacy of expertise needed to perform regulatory functions.

Objective: The practical and theoretical training of human resources involved in drug regulation,
as well as the continuous updating of their skills, are key components of good NRA
performance. This provides consistency and ensures that the agency’s actions will be
based on the latest scientific developments.

Requirement: Assessment of human resource capability (e.g., number proportionate to tasks, possession
of appropriate skills and education, identification of training needs) along with periodic
staff appraisal for performance evaluation
Evidence to review: The assessor should request for and review:
1. SOP or any written procedure to define periodic staff appraisal system, interval of
evaluation and performance targets for each function;
2. List of personnel trained in the last three years;
3. Records for re- evaluation of staff. 02

References: 1. National drug regulatory legislation: guiding principles for small drug regulatory 03
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations:
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical 04
Report Series, No. 885) (1) (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf) 05
2. Guidelines for national authorities on quality assurance for biological products.
In: WHO Expert Committee on Biological Standardization: forty-second report. World 06
Health Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2), (http://
www.who.int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_ 07
A2.pdf)
3. Regulation and licensing of biological products in countries with newly developing 08
regulatory authorities. In: WHO Expert Committee on Biological Standardization: forty-fifth
report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical Report Series,
09
No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001., (116),
(http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)

Framework: “Process”

Rating Scale:  NOT IMPLEMENTED (NI): There is no SOP or any written procedure to define a
periodic staff appraisal system.
 ONGOING IMPLEMENTATION (OI): There is no SOP or any written procedure to define
a periodic staff appraisal system; however some demonstrable steps have been
taken.
 PARTIALLY IMPLEMENTED (PI): The SOP to define periodic staff appraisal system has
been recently drafted but is not yet implemented.
 IMPLEMENTED (I): There is a documented and implemented procedure to define
periodic staff appraisal system, interval of evaluation and performance targets for
each regulatory function.

Page 48 of 340
WHO Global Benchmarking Tool (GBT)

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: RS06.03: A documented policy or procedure for the appointment and recruitment
of external experts is available.

Maturity Level: 3

Scope: 1. Medicines 01
2. Vaccines

National Regulatory System (RS): Indicators and Fact Sheets

Description: The assessor should verify that there is a policy for appointment of external experts,
evaluate whether the selection process uses an unbiased selection panel, and assess
whether the process is transparent. The results of such selection should be made public.
When assessing whether the objectives of the indicator have been met, assessor should
consider
1. whether the NRA has a documented policy for the recruitment and selection of outside
experts;
2. if there is no such policy, how the NRA demonstrates the ability to base its decisions on
current scientific developments;
3. whether the policy is applied.

Objective: The objective of this sub- indicator is to ensure the NRA has a documented policy
or procedure for the appointment and recruitment of external experts. This policy or
procedure should have defined criteria for the selection of experts. Selection process
should be conducted by a competent panel and decisions should be made public.
Outside experts and expert committees are mechanisms that allow the NRA access to
expertise on a particular topic. Such a practice is recommended in light of the rapid pace
of progress in science and technology, which means that no NRA can have staff with
sufficient first-hand, up-to-date knowledge in all fields. External experts are an extension 02
of the NRA’s staff, and therefore must meet the same requirements as in- house staff,
especially given the implicit risks because they are not an internal and integral part of the 03
NRA, and thus are not subject to the same work-related laws and regulations. Therefore,
it is good regulatory practice to establish a policy with defined criteria for recruitment of 04
experts, creation of selection panels, and publication of final decisions.
05
Requirement: Recruitment of external experts
06
Evidence to review: The assessor should request for and review:
1. Public information from the internet, bulletins, publications, or other official documents; 07
2. Supporting documentation, including notices for external experts, records of selection
meetings, appointments, and contracts. 08

References: 1. National drug regulatory legislation: guiding principles for small drug regulatory
09
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations:
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical
Report Series, No. 885) (1) (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf)
2. Guidelines for national authorities on quality assurance for biological products. In:
WHO Expert Committee on Biological Standardization: forty-second report. World Health
Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2), (http://www.who.
int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_A2.pdf)
3. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization: forty-fifth
report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical Report Series,
No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001., (116),
(http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)

Framework: “Process”

Page 49 of 340
WHO Global Benchmarking Tool (GBT)

Rating Scale:  NOT IMPLEMENTED (NI): There is no policy for the appointment, recruitment and
selection of external experts.
 ONGOING IMPLEMENTATION (OI): A circulated draft of such a policy is available, or
there is evidence that proposals have been made.
 PARTIALLY IMPLEMENTED (PI): The policy was approved recently (less than one year
ago), and these concepts were only recently incorporated into experts’ contracts or
other documents.
 IMPLEMENTED (I): The NRA has and consistently applies a policy for the recruitment
and selection of external experts. 01

National Regulatory System (RS): Indicators and Fact Sheets

Limitations and remarks:

Sub Indicator: RS06.04: Documented mechanism to handle potential conflicts of interest for internal
and external experts and committee members, to gather declarations of interest and to
guarantee the update of these declarations for all regulatory functions.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should review the procedure for declaration of interests by experts and verify
whether the same criteria are applied in case of internal or external experts and committee
members for all regulatory functions.
Assessor should consider the following aspects when assessing whether the objectives of
the indicator have been met:
1. whether there is a conduct policy for external experts and expert committees;
2. if there is such a policy, whether it establishes terms of reference regarding
confidentiality and potential conflicts of interest;
3. whether the policy is applied. 02

Objective: The objective of this sub-indicator is to ensure that there is a mechanism in place 03
to manage potential conflicts of interest for internal and external experts and expert
committees. 04
Outside experts and committees on which they serve are an extension of the NRA’s work;
therefore, they must meet the same criteria as those applied to staff involved in regulatory 05
functions. Since they do not work at the NRA and generally perform other tasks, potentially
with and for those regulated, there is a risk of conflicts of interest that are different from 06
those faced by civil servants. For this reason, the NRA must apply a special conflict of
interest policy when hiring outside experts. 07

Requirement: Managing potential conflicts of interest of internal and external experts and committee 08
members
09
Evidence to review: The assessor should request for and review:
1. Public information in the internet, bulletins, or other official documents;
2. Code of conduct for experts;
3. Contracts for external experts;
4. Expert committee rules and special exclusions;
5. Examples of declarations of interest, as well as examples of deferrals due to potential
conflicts of interest

Page 50 of 340
WHO Global Benchmarking Tool (GBT)

int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_A2.pdf)
3. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization: forty-fifth
report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical Report Series,
No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001.,
(116), (http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)

Framework: “Process” 01

National Regulatory System (RS): Indicators and Fact Sheets

Rating Scale:  NOT IMPLEMENTED (NI): The NRA has no internal mechanism to handle conflicts
of interest for internal and external experts and committee members.
 ONGOING IMPLEMENTATION (OI): There is documented evidence that steps have
been taken to develop a rule or policy and internal mechanisms and procedures
to handle conflicts of interest, but they are not available or have not yet been
implemented.
 PARTIALLY IMPLEMENTED (PI): A policy and mechanism to handle potential
conflicts of interest were recently established (less than one year ago), but there
is not sufficient, robust evidence of the NRA’s work in this area.
 IMPLEMENTED (I): The NRA has policies, mechanisms, and procedures to
handle potential conflicts of interest for internal and external experts and
committee members, and they are periodically updated and applied consistently
over time.

Limitations and remarks:

Indicator: RS07 Financial resources to perform regulatory activities.

Objective: The objective of this indicator is to ensure that whether or not a public system operates 02
with cost-recovery, decentralized medical products budgets or other systems, the system
should be well-designed and provided with resources sufficient to run properly. Financial 03
planning should match the financial resources with priority needs.
The assessor should identify if the NRA has sustainable funding. The assessor should find 04
out if the NRA is funded by the Government, by the fees collected for the services provided
or by donor sources. The availability of an adequate budget is essential to provide salaries 05
that will attract personnel with the required training and experience, as well as the facilities
and infrastructure needed. The NRA should have the authority to collect and utilize 06
internally the funds that it generates.
07
Category: 06. Resources (HR, FR, infrastructure and equipment)
08
Sub Indicator: RS07.01: Sources of funding are established for the NRA and affiliated institutions
to carry out all regulatory functions.
09

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that sources of funding needed to carry out all functions
are available for the NRA and the affiliated institutions. The process or mechanism for
obtaining funding should be reviewed.
Assessor should evaluate the following when assessing whether the objectives of this
indicator have been met:
1. Identify the sources of funding for the NRA (e.g., the government, fees collected for
services provided by the NRA, grants, or a combination of these or other sources).
2. Determine whether the NRA has annual budgets. If so, determine whether they
cover execution of all regulatory functions. Budgets for at least two years should
be reviewed.

Page 51 of 340
WHO Global Benchmarking Tool (GBT)

Objective: The objective of this sub-indicator is to confirm that the NRA has sources of funding that
allow it to perform its regulatory functions under stable conditions.
For an NRA to maintain adequate performance, the agency must have sustainable funding.
The budget must allow it to operate satisfactorily and perform all regulatory and oversight
duties with sufficient specialized personnel.

Requirement: Existence of fee system or an equivalent mechanism for stable funding.

Evidence to review: The assessor should request for and review: 01

1. Applicable regulations;

National Regulatory System (RS): Indicators and Fact Sheets

2. Annual NRA budget for at least two years;
3. Funding agreements;
4. NRA annual reports on activities and budgets.

References: 1. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations:
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical
Report Series, No. 885) (1) (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf)
2. Guidelines for national authorities on quality assurance for biological products. In:
WHO Expert Committee on Biological Standardization: forty-second report. World Health
Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2), (http://www.who.
int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_A2.pdf)
3. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization:
forty-fifth report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical
Report Series, No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_
TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001.,
(116), (http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf) 02
5. Quality management systems — Requirements. International Standard ISO 9001:2015.
International Organization for Standardization, 2015, (4), (https://www.iso.org/iso-9001- 03
quality-management.html)
04
Framework: “Structure/Foundation/Input”
05
Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence of stable sources of funding, or no
budget is established for performing all regulatory functions. 06
 ONGOING IMPLEMENTATION (OI): There is evidence of steps being taken to
systematize the sources of funding for the NRA and allocate a budget periodically. 07
 PARTIALLY IMPLEMENTED (PI): There are sources of funding, but they have not been
stable during the past three years. The NRA stated that certain functions or indicators 08
(i.e., under this or other modules) cannot be implemented due to budget constraints.
 IMPLEMENTED (I): The NRA has defined sources of funding and has been executing a
09
budget for the performance of all its regulatory functions over time.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: RS07.02: The amounts collected for fees, taxes, tariffs or dues payable
for the services provided are defined and publicly available.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that the amounts of fees, taxes, tariffs or dues payable for the
services provided are defined and that this information is publicly available. Fees, taxes,
tariffs or dues are a financial charge imposed upon a taxpayer (an individual or legal entity).

Page 52 of 340
WHO Global Benchmarking Tool (GBT)

Objective: The objective of this sub-indicator is to ensure that for purposes of transparency, the NRA
or the responsible authority in the country should define the amounts of fees, taxes, tariffs
or dues payable for the services provided by the NRA and should make this information
available to the public.

Requirement: Fees, taxes, tariffs

Evidence to review: The assessor should request for and review:

1. Documentation showing fees, taxes, tariffs or dues payable for the services provided; 01
2. Documentation demonstrating public availability of this information.

National Regulatory System (RS): Indicators and Fact Sheets

Framework: “Structure/Foundation/Input”
02
Rating Scale:  NOT IMPLEMENTED (NI): There is no public list, although fees are charged for
services. 03
 ONGOING IMPLEMENTATION (OI): The list is awaiting approval from the governing
institutions, and there is evidence that such a proposal has been submitted. 04
 PARTIALLY IMPLEMENTED (PI): The list was recently published (i.e., less than a year
ago), or a list was published more than a year ago but it does not include all services. 05
 IMPLEMENTED (I): The NRA has the authority to charge for services, and there is a list
with the rates of all NRA services that require payment. This rate list is consistently 06
applied, is updated periodically, and was published digitally or printed less than a year
ago. 07

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will 08
always apply for all benchmarked NRAs).
09
Sub Indicator: RS07.03: There are provisions relating to reduction or exemption of dues, taxes,
tariffs or fees in defined situations for public health interest.

Maturity Level: 4

Scope: 1. Medicines
2. Vaccines
Description: The assessor should check that there are provisions in the NRA to waive or reduce the fees
and taxes in cases of public health interest, for example, to guarantee the availability of
medical products. The NRA should define the criteria for specific circumstances, as well as
necessary actions.

Objective: The objective of this sub-indicator is to ensure that provisions exist to facilitate and
improve access to medical products and to guarantee the availability of drugs in defined
situations (e.g., shortages, emergencies, or outbreaks) through reductions or exemptions in
dues, taxes, tariffs or fees. This information should be publicly available.

Page 53 of 340
WHO Global Benchmarking Tool (GBT)

Requirement: Reduction or exemption of dues, taxes, tariffs or fees

Evidence to review: The assessor should request for and review:

1. The provisions in the NRA to waive or reduce the fees and taxes in emergency situations;
2. Examples of tax exemptions or reductions under emergency conditions or other relevant
situations.

National Regulatory System (RS): Indicators and Fact Sheets

885) (1) (http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
2. Guidelines for national authorities on quality assurance for biological products. In:
WHO Expert Committee on Biological Standardization: forty-second report. World Health
Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2), (http://www.who.
int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_A2.pdf)
3. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization: forty-fifth
report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical Report Series,
No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001., (116),
(http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)

Framework: “Structure/Foundation/Input”

Rating Scale:  NOT IMPLEMENTED (NI): There are no provisions relating to reduction or exemption
of dues, taxes, tariffs or fees in defined situations for public health interest.
 ONGOING IMPLEMENTATION (OI): The provisions are awaiting approval from the
governing institutions, and there is evidence that such a proposal has been submitted.
 PARTIALLY IMPLEMENTED (PI): The provisions were recently published (i.e., less than
a year ago), but there are no examples of tax exemption or reduction to show. 02
 IMPLEMENTED (I): There are provisions relating to reduction or exemption of dues,
taxes, tariffs or fees in defined situations for public health interest and there are 03
examples of their application.
04
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 05

Sub Indicator: RS07.04: The NRA has authority to manage the funds allocated 06
and/or generated internally.
07
Maturity Level: 4
08
Scope: 1. Medicines
2. Vaccines
09

Description: The assessor should verify that the NRA has the authority to manage the funds allocated
and/or generated internally. The assessor also should determine the percentage of these
funds that are used by the NRA for its functions.

Objective: The objective of this sub-indicator is to ensure the NRA has the authority to manage
allocated or internally-generated funds to finance regulatory activities. This authority plays
an important role in enhancing the autonomy and independence of the NRA.

Requirement: Authority to manage the funds allocated and/or generated internally.

Evidence to review: The assessor should request for and review

1. Provisions that give authority to NRA to manage allocated funds and to collect and use
the funds generated internally;
2. Mechanisms or processes in place to manage these funds;
3. Records of funds that are generated by NRA and the percentage that are used internally.

Page 54 of 340
WHO Global Benchmarking Tool (GBT)

National Regulatory System (RS): Indicators and Fact Sheets

regulatory authorities. In: WHO Expert Committee on Biological Standardization: forty-fifth
report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical Report Series,
No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001., (116),
(http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)

Framework: “Structure/Foundation/Input”

Rating Scale:  NOT IMPLEMENTED (NI): The National Regulatory Authority does not have the
authority to manage the allocated or internally generated funds.
 ONGOING IMPLEMENTATION (OI): There is evidence of steps being taken to enable
the NRA to manage the allocated or internally generated budget.
 PARTIALLY IMPLEMENTED (PI): The NRA is recently able to manage allocated or
internally generated budget or the NRA, with some constraints or limitations, is
having the authority to manage allocated or internally generated budget.
 IMPLEMENTED (I): The NRA have the full authority to manage the allocated or
internally generated funds.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will 02
always apply for all benchmarked NRAs).
03
Sub Indicator: RS07.05: The NRA periodically publicizes its budget.
04

Maturity Level: 4 05

Scope: 1. Medicines 06
2. Vaccines
Description: The assessor should check that the NRA publicizes its budget periodically (i.e., 07
at least once per year). The assessor should review publically available budgets, both
an overall budget as well as a budget broken down by regulatory function and/or 08
medical products.
09
Objective: The objective of this sub-indicator is to ensure that the NRA periodically publicizes its
budget to demonstrate transparency, prevent corruption, and build trust between the
public and the NRA.

Requirement: Periodically publicize its budget

Evidence to review: The assessor should request for and review:

1. Provisions requiring the NRA to publicize its budget periodically;
2. Records of published budgets.

Page 55 of 340
WHO Global Benchmarking Tool (GBT)

WHO Expert Committee on Biological Standardization: forty-second report. World

Health Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2), (http://
www.who.int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_
A2.pdf)
3. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization:
forty-fifth report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical
Report Series, No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_
TRS_858_A1.pdf) 01
4. How to develop and implement a national drug policy, Second edition. WHO, 2001.,

National Regulatory System (RS): Indicators and Fact Sheets

(116), (http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)

Framework: “Output”.

Rating Scale:  NOT IMPLEMENTED (NI): There is no provision stating the NRA periodically publicizes
its budget.
 ONGOING IMPLEMENTATION (OI): There is evidence of steps being taken to
systematize the publication of the NRA budget.
 PARTIALLY IMPLEMENTED (PI): The provisions were recently published (i.e., less than
a year ago), but there are no examples of NRA budget publication.
 IMPLEMENTED (I): There are provisions stating the NRA periodically publicizes its
budget and there are records of published budgets available for two years.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Indicator: RS08 Infrastructure and equipment to perform regulatory activities.

Objective: The objective of this indicator is to ensure that resources and infrastructure are available 02
to the national regulatory authority in order to perform regulatory activities adequately. A
well-resourced authority backed with adequate drug legislation and regulations is needed 03
to ensure the delivery of regulatory activities.
04
Category: 06. Resources (HR, FR, infrastructure and equipment)
05
Sub Indicator: RS08.01: The workspace and work environment provided for performing
the regulatory activities are adequate. 06

Maturity Level: 2 07

Scope: 1. Medicines 08
2. Vaccines
09
Description: The assessor should verify that “work environment” under which work is performed
(including physical, environmental and other factors) is adequate to perform regulatory
activities. Infrastructure includes, as applicable:
1. buildings, workspaces and associated utilities;
2. lighting and ventilation in the workspaces;

Objective: The objective of this sub-indicator is to ensure the NRA provides and maintains the
minimum infrastructure needed to achieve conformity to service requirements and
conduct regulatory activities. The organization should determine and manage the work
environment (workspace) and the environmental conditions needed for employees to
perform their daily activities.

Requirement: Workspace and work environment

Evidence to review: The assessor should request for and review:

1. Number of staff and the number of offices to which they are assigned;

Page 56 of 340
WHO Global Benchmarking Tool (GBT)

2. Quality of workspace;
3. Lighting and ventilation in the workspace;
4. Supporting services (e.g., transportation and communication systems).

National Regulatory System (RS): Indicators and Fact Sheets

WHO Expert Committee on Biological Standardization: forty-second report. World Health
Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2), (http://www.who.
int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_A2.pdf)
3. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization: forty-fifth
report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical Report Series,
No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001., (116),
(http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)
5. Quality management systems — Requirements. International Standard ISO 9001:2015.
International Organization for Standardization, 2015 (4) (https://www.iso.org/iso-9001-
quality-management.html)

Framework: “Structure/Foundation/Input”

Rating Scale:  NOT IMPLEMENTED (NI): There is inadequate work space and work environment to
perform regulatory activities.
 ONGOING IMPLEMENTATION (OI): There is evidence of steps being taken to ensure
adequate work space and work environment to perform all regulatory activities.
 PARTIALLY IMPLEMENTED (PI): There is adequate work space and work environment 02
in some areas but not for all regulatory functions.
 IMPLEMENTED (I): The NRA provides adequate work space and work environment for 03
all regulatory activities.
04
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 05

Sub Indicator: RS08.02: The workspace and work environment provided for performing 06
the regulatory activities includes essential requirements.
07
Maturity Level: 3
08
Scope: 1. Medicines
2. Vaccines
09

Description: The assessor should verify the “work environment” including the essential requirements
to perform regulatory activities. The assessor should verify that there is enough space for
secure storage of documents and an easily accessible archive. In addition, some specific
regulatory activities require specific workspace (e.g., laboratory facilities). The archival of
the documents should be verified.
Infrastructure includes, as applicable:
1. process equipment (both hardware and software);
2. supporting services (servers and information systems);
3. archives;
4. workspace for specific regulatory activities, e.g., laboratories.

Objective: The objective of this sub-indicator is to ensure the NRA provides the work environment
needed to conduct regulatory activities in an efficient manner. The organization should
determine and manage the work environment needed to achieve conformity to product
requirements.

Page 57 of 340
WHO Global Benchmarking Tool (GBT)

Requirement: Workspace and work environment

Evidence to review: The assessor should request for and review:

1. Workspace and security access for storage of documents (paper and electronic);
2. Archives;
3. List of needed equipment, as well as availability of this equipment (e.g., process
equipment and supporting services)

References: 1. National drug regulatory legislation: guiding principles for small drug regulatory 01
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations:

National Regulatory System (RS): Indicators and Fact Sheets

thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical
Report Series, No. 885) (1) (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf)
2. Guidelines for national authorities on quality assurance for biological products. In:
WHO Expert Committee on Biological Standardization: forty-second report. World Health
Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2), (http://www.who.
int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_A2.pdf)
3. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization:forty-fifth
report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical Report Series,
No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001.,
(116), (http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)
5. Quality management systems — Requirements. International Standard ISO 9001:2015.
International Organization for Standardization, 2015 (4) (https://www.iso.org/iso-9001-
quality-management.html)

Framework: “Structure/Foundation/Input”

Rating Scale:  NOT IMPLEMENTED (NI): The workspace or work environment provided for 02
performing the regulatory activities does not include essential requirements as
indicated in this fact sheet. 03
 ONGOING IMPLEMENTATION (OI): A plan exists for the the workspace and work
environment provided for performing the regulatory activities to meet the essential 04
requirement as indicated in this fact sheet.
 PARTIALLY IMPLEMENTED (PI): There is evidence of steps and actions taken for the 05
the workspace and work environment provided for performing the regulatory activities
to meet the essential requirement as indicated in this fact sheet. 06
 IMPLEMENTED (I): The workspace and work environment provided for performing the
regulatory activities are verified to include essential requirements as indicated in this 07
fact sheet
08
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).
09

Sub Indicator: RS08.03: The equipment provided for performing the regulatory activities
is adequate.

Maturity Level: 4

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the equipment provided for the performance of regulatory
activities is adequate to perform these activities (i.e., taking into consideration the types of
activities to be performed and the number of staff allocated).

Objective: The objective of this sub-indicator is to ensure the NRA provides and maintains adequate
equipment as part of the infrastructure needed to achieve conformity to service
requirements and to perform regulatory activities.

Page 58 of 340
WHO Global Benchmarking Tool (GBT)

Requirement: Equipment provided for performing

Evidence to review: The assessor should request for and review:

1. List of needed equipment (e.g., computers and printers) and its availability;
2. List of functional equipment and its maintenance.

National Regulatory System (RS): Indicators and Fact Sheets

s21964en.pdf)
2. Guidelines for national authorities on quality assurance for biological products. In:
WHO Expert Committee on Biological Standardization: forty-second report. World
Health Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2), (http://
www.who.int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_
A2.pdf)
3. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization:
forty-fifth report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical
Report Series, No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_
TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001.,
(116), (http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)
5. Quality management systems — Requirements. International Standard ISO 9001:2015.
International Organization for Standardization, 2015 (4) (https://www.iso.org/iso-9001-
quality-management.html)

Framework: “Structure/Foundation/Input”

Rating Scale:  NOT IMPLEMENTED (NI): There is insufficient equipment to perform regulatory 02
activities.
 ONGOING IMPLEMENTATION (OI): There is evidence of steps being taken to ensure 03
adequate equipment to perform all regulatory activities.
 PARTIALLY IMPLEMENTED (PI): There is adequate equipment available in some areas 04
but not enough to cover all regulatory functions.
 IMPLEMENTED (I): The NRA provides adequate equipment for all regulatory 05
activities.
06
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 07

Indicator: RS09 Mechanisms exist to promote transparency, accountability 08

and communication.
09
Objective: The objective of this indicator is to ensure that NRA’s general practices and policies
dealing with transparency issues are established in procedures and outcomes. On
this matter the assessor could undertake interviews outside the NRA, for example,
with stakeholders’ representatives, to collect information on how the NRA is perceived
or operates. Information is a key pre-requisite for communication. Adequate and
effective communication plays a fundamental role for exchanging information,
informing relevant stakeholders, and preventing misunderstanding. When regularly
communicating both internally and externally, organizations remain more transparent.
Accountability is the process through which an organization makes a commitment
to respond to and balance the needs of stakeholders in its decision-making processes
and activities. The assessor should verify that the information and documents are
updated for all sub-indicators.
Documented evidence to be studied:
1. Reporting and communication policy;
2. Annual or periodic reports;
3. NRA web page or website.

Page 59 of 340
WHO Global Benchmarking Tool (GBT)

Category: 08. Transparency, accountability and communication

Sub Indicator: RS09.01: The NRA participates in regional and/or global networks to promote
convergence and harmonization efforts and expand its collaboration in the
regulatory field.

Maturity Level: 4

Scope: 1. Medicines 01
2. Vaccines

National Regulatory System (RS): Indicators and Fact Sheets

Description: The assessor should check the global, regional and/or sub-regional networks joined by
the relevant NRA. Only those networks which promote convergence, harmonization,
collaboration and reliance among the NRAs or related institutions (e.g. pharmacovigilance
centers) should be considered. Active participation in meetings, seminars, committees,
and adoption of decisions and recommendations by the networks should be verified.

Objective: The objective of this indicator is to ensure the participation of the NRA in regional or global
networks. These networks can serve as a forum for sharing information, exchanging
experiences on technical issues and best practices, and facilitating assistance between
NRAs. They can contribute to efforts to promote convergence of international regulatory
requirements and practices, as well as harmonization efforts.

Requirement: Regional and/or global networks to promote convergence and harmonization efforts in
regulatory requirements and practices

Evidence to review: The assessor should request for and review:

1. Global, regional and/or sub-regional network memberships and agreements;
2. Minutes of meetings of these networks, documentation of high level decisions (e.g.
guidelines adaptation), and records of communications. 02

References: 1. National drug regulatory legislation: guiding principles for small drug regulatory 03
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations:
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical 04
Report Series, No. 885) (1) (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf) 05
2. Guidelines for national authorities on quality assurance for biological products. In:
WHO Expert Committee on Biological Standardization: forty-second report. World Health 06
Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2), (http://www.who.
int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_A2.pdf) 07
3. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization: 08
forty-fifth report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical
Report Series, No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_
09
TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001.,
(116), (http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)

Framework: “Structure/Foundation/Input”

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence that the NRA participates in a regional
and/or global network.
 ONGOING IMPLEMENTATION (OI): There is no evidence that the NRA participates in
regional and/or global networks but demonstrable steps have been taken towards
this.
 PARTIALLY IMPLEMENTED (PI): The NRA has participated in some meetings,
seminars, or committees but there is no evidence of adoption of decisions or
recommendation by the networks.
 IMPLEMENTED (I): There is documented evidence of NRA participation in regional
and/or global networks.

Page 60 of 340
WHO Global Benchmarking Tool (GBT)

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: RS09.02: The information on laws, regulations guidelines and procedures
is publicly available and is kept duly updated.

Maturity Level: 3

Scope: 1. Medicines 01
2. Vaccines

National Regulatory System (RS): Indicators and Fact Sheets

Description: The assessor should verify that all relevant laws, regulations, guidelines and procedures
are publicly available and duly updated. Some other information related to internal
procedures and decision making may be for internal use only. Assessor should verify
the availability of document control mechanisms and procedures for updating the
documents at a regular frequency.
The assessor should verify that there is public access to non-classified information
unless otherwise justified for well-specified reasons as provided by law (such as
protection of privacy or ensuring fairness of procurement procedures).

Objective: The objective of this sub-indicator is to ensure that medical products laws, regulations,
guidelines and procedures are available and accessible to stakeholders and the general
public. Openness and transparency are key ingredients for building the accountability
and trust which are necessary for the functioning of organization and for good
governance.

Requirement: Laws, regulations, guidelines and procedures are publicly available.

Evidence to review: The assessor should request for and review:

1. Publicly available laws, regulations, guidelines and procedures; 02
2. Mechanisms in place for information sharing (e.g., website or other public
communications); 03
3. Written procedures describing the process.
04
References: 1. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical 05
Preparations: thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO
Technical Report Series, No. 885) (1) (http://apps.who.int/medicinedocs/documents/ 06
s21964en/s21964en.pdf)
2. Guidelines for national authorities on quality assurance for biological products. In: 07
WHO Expert Committee on Biological Standardization: forty-second report. World Health
Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2), (http://www. 08
who.int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_A2.pdf)
3. Regulation and licensing of biological products in countries with newly developing
09
regulatory authorities. In: WHO Expert Committee on Biological Standardization:
forty-fifth report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical
Report Series, No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_
TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001.,
(116), (http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)
5. Good governance for medicines: Model framework. Updated version 2014. WHO, (110),
(https://www.who.int/medicines/areas/governance/ggm_modelframe_updated/en/)

Framework: “Structure/Foundation/Input”

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence that the information on laws,
regulations guidelines and procedures is publicly available and is kept duly updated.
 ONGOING IMPLEMENTATION (OI): The information on laws, regulations guidelines
and procedures is not publicly available or kept duly updated but demonstrable steps
have been taken towards this.

Page 61 of 340
WHO Global Benchmarking Tool (GBT)

 PARTIALLY IMPLEMENTED (PI): There is some information publicly available but not
from all types of documents, or information is not kept duly updated.
 IMPLEMENTED (I): The information on laws, regulations guidelines and procedures is
publicly available and is kept duly updated.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: RS09.03: Information on decisions related to regulatory activities is available 01

to the public.

National Regulatory System (RS): Indicators and Fact Sheets

Maturity Level: 4

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that all relevant information and summary reports that serve
as the bases for regulatory decisions for different functions are available and that final
decisions are communicated to the public.

Objective: The objective of this sub-indicator is to ensure that relevant documents that support
positive or negative NRA decisions are publicly available, and that final decisions are
communicated to the public. Evidence-based decisions and honesty regarding the facts
and principles considered in the regulatory decision-making process greatly enhance
public trust.
A basic component of transparency in the work of NRAs is based on whether its decisions
are published, and whether the relevant information is quickly and easily accessible
to those regulated and to the general public. Regulatory decisions include approvals
or denials; unless prohibited by specific legal provisions, decisions to not approve (i.e.,
denials) should also be published. 02

Requirement: Decisions are accessible to the public 03

Evidence to review: The assessor should request for and review: 04

1. Provisions or guidelines describing the public communication process;
2. List of publicly-available final decisions and supporting documents and reports related 05
to different regulatory processes (e.g., marketing authorizations, clinical trials, inspections,
market surveillance, pharmacovigilance, and licensing); 06
3. Relevant documents from website or other mechanisms used to publish the
information. 07

References: 1. National drug regulatory legislation: guiding principles for small drug regulatory 08
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations:
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical
09
Report Series, No. 885) (1) (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf)
2. Guidelines for national authorities on quality assurance for biological products. In:
WHO Expert Committee on Biological Standardization: forty-second report. World Health
Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2), (http://www.who.
int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_A2.pdf)
3. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization: forty-fifth
report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical Report Series,
No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001., (116),
(http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)
5. Good governance for medicines: Model framework. Updated version 2014. WHO (110)
(https://www.who.int/medicines/areas/governance/ggm_modelframe_updated/en/)

Framework: “Output”

Page 62 of 340
WHO Global Benchmarking Tool (GBT)

Rating Scale:  NOT IMPLEMENTED (NI): No regulatory decisions are published.

 ONGOING IMPLEMENTATION (OI): There is evidence that a process is underway for
the publication of regulatory decisions.
 PARTIALLY IMPLEMENTED (PI): Regulatory decisions began to be published recently
(i.e., less than a year ago).
 IMPLEMENTED (I): The NRA systematically publishes the positive and negative
regulatory decisions.

Limitations and remarks: In some countries, legislation may not allow the publication of negative regulatory 01
decisions. In this case, the assessment of this sub-indicator should be limited to the

National Regulatory System (RS): Indicators and Fact Sheets

publication of positive regulatory decisions (e.g. marketing authorizations granted or Good
Manufacturing Practice compliance assessments).
Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: RS09.04: Information on marketed medical products, authorized companies

and licensed facilities is publicly available.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that lists identifying all authorized products and all licenses
issued to manufacturers, importers, wholesalers and other institutions engaged in
manufacture, control, distribution, or marketing of medical products are published and
publicly available. Experience in countries has demonstrated that informing consumers
and public interest groups on regulation of medicines enhances political support for the
regulatory process, promotes NRA transparency and accountability, and shields them
from negative external influence. 02

Objective: The objective of this sub-indicator is to ensure that information on NRA decisions is 03
available to the public. Such transparency can help to reduce conflicts of interest and
corruption. In addition, public availability of a list of approved products may prevent use of 04
any unregistered products.
05
Requirement: Transparency
06
Evidence to review: The assessor should request for and review:
1. Guidelines which describe the information that should be publicly available; 07
2. Publicly available information on marketed products and authorized companies and
facilities. 08

References: 1. National drug regulatory legislation: guiding principles for small drug regulatory
09
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations:
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical
Report Series, No. 885) (1) (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf)
2. Guidelines for national authorities on quality assurance for biological products.
In: WHO Expert Committee on Biological Standardization: forty-second report. World
Health Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2),
(http://www.who.int/biologicals/publications/trs/areas/biological_products/WHO_
TRS_822_A2.pdf)
3. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization:
forty-fifth report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical
Report Series, No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_
TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001.,
(116), (http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)

Page 63 of 340
WHO Global Benchmarking Tool (GBT)

Framework: “Output”.

Rating Scale:  NOT IMPLEMENTED (NI): No information on marketed medical products,

authorized companies and licensed facilities is published.
 ONGOING IMPLEMENTATION (OI): There is evidence that a process is underway
for the publication of some regulatory decisions on marketed medical products,
authorized companies and licensed facilities.
 PARTIALLY IMPLEMENTED (PI): Regulatory decisions on marketed medical products,
authorized companies and licensed facilities began to be published recently (i.e., less 01
than a year ago).

National Regulatory System (RS): Indicators and Fact Sheets

 IMPLEMENTED (I): The NRA systematically publishes the positive and
negative regulatory decisions on marketed medical products, authorized
companies and licensed facilities and there is evidence available for more than
2 years.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: RS09.05: All publicly available information is periodically reviewed

and maintained.

Maturity Level: 4

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that public information is reviewed on regular basis so that the
latest updates are currently available to the public.

Objective: The objective of this sub-indicator is to ensure the NRA considers and defines specific 02
intervals for reviewing and maintaining the publicly available information. As part of this
process, the NRA should consider the risks linked to different categories of information. 03

Requirement: Publicly available information is reviewed and maintained. 04

Evidence to review: The assessor should request for and review: 05

1. A guideline for publication of information that describes the types of information to
be published, specifies the format for publication, identifies the persons responsible, and 06
defines the frequency for updating;
2. Records documenting that these procedures are followed. 07

References: 1. National drug regulatory legislation: guiding principles for small drug regulatory 08
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations:
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical
09
Report Series, No. 885) (1) (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf)
2. Guidelines for national authorities on quality assurance for biological products. In:
WHO Expert Committee on Biological Standardization: forty-second report. World
Health Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2), (http://
www.who.int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_
A2.pdf)
3. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization:
forty-fifth report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical
Report Series, No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_
TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001.,
(116), (http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)

Framework: “Output”.

Page 64 of 340
WHO Global Benchmarking Tool (GBT)

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence that publicly available information is
periodically reviewed and maintained.
 ONGOING IMPLEMENTATION (OI): The publicly available information is not
periodically reviewed and maintained but demonstrable steps have been taken
towards this.
 PARTIALLY IMPLEMENTED (PI): There is some information publicly available but not
from all types of documents or information is not kept duly updated.
 IMPLEMENTED (I): The publicly available information is periodically reviewed and
maintained. 01

National Regulatory System (RS): Indicators and Fact Sheets

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: RS09.06: Appropriate mechanisms exist for management of confidential

information.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should identify the written procedures for the management of confidential
information for internal staff, external experts and members of advisory committees. A
written document and established mechanisms should explain which information should
be kept confidential and how this should be done. The mechanisms should be explicit
regarding the means used by the NRA to keep confidentiality, including administrative,
physical and technical controls.
It is very important to define how information can be shared within and outside
the NRA.
02
Objective: The objective of this sub-indicator is to ensure that the NRA has appropriate mechanisms
and procedures for management of confidential information. 03
Confidential Information refers to any documents, materials or information (whether
in written, oral, electronic or any other format) which are of a confidential, proprietary, 04
personal or non-public nature. Such confidential information may belong to either (i) the
NRA or (ii) any third party collaborating with the NRA or to whom the NRA owes a duty 05
of confidentiality (including, but not limited to, manufacturers, marketing authorization
holders and clinical investigators). Confidential information must be protected from 06
unauthorized access to safeguard the privacy or security of an individual or organization.
07
Requirement: Management of confidential information
08
Evidence to review: The assessor should request for and review:
1. Written documents that describe management of confidential information;
09
2. Documentation that there is adequate infrastructure to keep the confidential information,
e.g., locked rooms or secure computerized systems.

Page 65 of 340
WHO Global Benchmarking Tool (GBT)

Report Series, No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_

TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001., (116),
(http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)
5. Good governance for medicines: Model framework. Updated version 2014. WHO (110)
(https://www.who.int/medicines/areas/governance/ggm_modelframe_updated/en/)

Framework: “Structure/Foundation/Input”
01
Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence that the NRA has an appropriate

National Regulatory System (RS): Indicators and Fact Sheets

mechanism to manage confidential information.
 ONGOING IMPLEMENTATION (OI): There is no evidence that the NRA has an
appropriate mechanism to manage confidential information but demonstrable steps
have been taken towards this.
 PARTIALLY IMPLEMENTED (PI): The NRA has developed a draft document to manage
confidential information but it has not yet been implemented.
 IMPLEMENTED (I): There is documented evidence that the NRA has an appropriate
mechanism to manage confidential information.

Limitations and remarks:  Confidential information means any documents, materials or information
(whether in written, oral, electronic or any other format) which are of a
confidential, proprietary, personal or non-public nature. Such confidential
information may belong to either (i) the NRA or (ii) any third party collaborating
with the NRA or to whom the NRA owes a duty of confidentiality (including, but
not limited to, manufacturers).
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-
indicator will always apply for all benchmarked NRAs).

Sub Indicator: RS09.07: A code of conduct, which includes management of conflicts of interest,
is published and enforced for internal and external staff, including members of the 02
advisory committees.
03
Maturity Level: 3
04
Scope: 1. Medicines
2. Vaccines 05

Description: The assessor should verify the existence of a written, published and enforced code 06
of conduct for internal staff and for external experts and members of advisory
committees. The main issues are related to the independence of all experts involved in 07
the regulatory processes and to the management and prevention of potential conflicts
of interests. 08

Objective: The objective of this sub-indicator is to ensure the NRA has established codes of
09
conduct based on professionalism, institutional values, and ethical principles.
This is a mechanism to prevent unethical behavior by public servants and external
staff (including members of the advisory committees) in the performance of their
duties. Studies by Transparency International indicate that governments which have
established a framework for good governance, based on codes of conduct, ethical
principles and other key components of an integrity system, are listed among the
countries with very low levels of corruption. There is a logical and consistent link
between values, principles and a code of conduct. A code of conduct attempts to
articulate in concrete terms the application of ethical principles. The normative base for
the code of conduct is the framework of institutional values.

Requirement: Code of conduct

Evidence to review: The assessor should request for and review:

1. A guideline describing the code of conduct for the NRA;
2. Records of signed conflict of interest forms.

Page 66 of 340
WHO Global Benchmarking Tool (GBT)

National Regulatory System (RS): Indicators and Fact Sheets

3. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization:
forty-fifth report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical
Report Series, No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_
TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001.,
(116), (http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)
5. Good governance for medicines: Model framework. Updated version 2014. WHO (110)
(https://www.who.int/medicines/areas/governance/ggm_modelframe_updated/en/)

Framework: “Structure/Foundation/Input”

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence that a code of conduct, including
management of conflicts of interest, is published and enforced for internal and
external staff, including members of the advisory committees.
 ONGOING IMPLEMENTATION (OI): There is no evidence that a code of conduct,
including management of conflicts of interest, is published and enforced; however,
demonstrable steps have been taken towards this.
 PARTIALLY IMPLEMENTED (PI): The NRA has developed a draft document but it has
not yet been implemented. 02
 IMPLEMENTED (I): There is documented evidence that a code of conduct, including
management of conflicts of interest, is published and enforced. 03

Maturity Level: 4 07

Scope: 1. Medicines 08
2. Vaccines
09
Description: The assessor should verify that the NRA makes appropriate use of automation and
effectively manages its databases to process information and generate good quality and
readily accessible reports.
“E-government” refers to the use of information and communication technologies to
promote more efficient and effective government, facilitate more accessible government
services, allow greater public access to information, and make government more
accountable to citizens. The assessor should verify the utilization of electronic systems
and workstations by the NRA to handle these activities, which may be repetitive in nature.
In general, e-governance would be required to be checked.

Objective: The objective of this sub-indicator is to ensure the NRA manages computerized systems
for information processing, records management, and data analysis. This sub-indicator
measures the NRA’s handling of automated data management systems, use of safeguards
to protect data integrity, and development of integrated networks. Under this indicator, the
degree to which these objectives are achieved is also assessed. E-governance can play a
facilitating role.

Page 67 of 340
WHO Global Benchmarking Tool (GBT)

Requirement: Computerized systems to automate repetitive activities

Evidence to review: The assessor should request for and review:

1. Evidence for availability of electronic systems and workstations;
2. Evidence for availability of required software for conducting regulatory activities;
3. List of processes that use computerized systems;
4. Databases in use or under in development within the NRA;
5. Databases and systems in use that can be consulted by those regulated and the
general public. 01

National Regulatory System (RS): Indicators and Fact Sheets

Framework: “Structure/Foundation/Input”
02
Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence that the NRA uses computerized
systems for information processing, records management, and data analysis. 03
 ONGOING IMPLEMENTATION (OI): There is no evidence that the NRA uses
computerized systems for information processing, records management, and data 04
analysis, but demonstrable steps have been taken towards this.
 PARTIALLY IMPLEMENTED (PI): The NRA has started to use computerized systems 05
for information processing, records management, and data analysis but this has not
been applied in all regulatory processes. 06
 IMPLEMENTED (I): There is documented evidence that the NRA uses computerized
systems for information processing, records management, and data analysis. 07

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will 08
always apply for all benchmarked NRAs).
09
Sub Indicator: RS09.09: The NRA has its own web page with timely information that gives
public access to related legal provisions, guidelines and decisions.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should check that the NRA web page serves as a platform for sharing
timely information and for providing public access to legal provisions, guidelines and
decisions.

Objective: The objective of this sub-indicator is to ensure the NRA’s website plays an active role in
effective and timely communication of information required to be publicized.

Requirement: Website

Page 68 of 340
WHO Global Benchmarking Tool (GBT)

Evidence to review: The assessor should request for and review:

1. NRA website and information available on it;
2. NRA communication policy;
3. Written documents describing how frequently information provided in the website
should be reviewed and updated.

National Regulatory System (RS): Indicators and Fact Sheets

s21964en.pdf)
2. Guidelines for national authorities on quality assurance for biological products. In:
WHO Expert Committee on Biological Standardization: forty-second report. World Health
Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2), (http://www.who.
int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_A2.pdf)
3. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization: forty-fifth
report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical Report Series,
No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001., (116),
(http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)

Framework: “Structure/Foundation/Input”

Rating Scale:  NOT IMPLEMENTED (NI): The NRA does not have a website with timely information
that gives public access to related legal provisions, guidelines and decisions.
 ONGOING IMPLEMENTATION (OI):The NRA does not have a website but
demonstrable steps have been taken towards this.
 PARTIALLY IMPLEMENTED (PI): The NRA has started to use its website but the
information posted is not up to date. 02
 IMPLEMENTED (I): The NRA has a website with timely information that gives public
access to related legal provisions, guidelines and decisions. 03

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will 04
always apply for all benchmarked NRAs).
05
Indicator: RS10 Mechanism in place to monitor regulatory performance and output.
06

Objective: The objective of this indicator is to ensure monitoring of regulatory performance through 07
the establishment of key performance indicators. Monitoring of regulatory performance
plays an important role when taking the appropriate actions and when adjusting the 08
process to ensure that the regulatory activity conforms to its specifications.
In addition, the NRA should monitor and review the performance of the contracted entities
09
(in the case of outsourced activities) including the implementation and effectiveness of
any needed improvements.

Category: 09. Monitoring progress and assessing outcomes and impact

Sub Indicator: RS10.01: Requirements established to monitor, supervise and review the performance
of the NRA and affiliated institutions using key performance indicators (KPIs).

Maturity Level: 4

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the existence and implementation of requirements and
procedures for monitoring, supervising, and reviewing the performance of the NRA
and affiliated institutions. Monitoring and evaluating NRA performance are essential

Page 69 of 340
WHO Global Benchmarking Tool (GBT)

components of a regulatory framework, and the provisions necessary to achieve this need
to be included in the regulatory oversight of medical products. Key issues include: explicit
government commitment to the principles of monitoring and evaluation; monitoring of the
pharmaceutical sector through regular indicator-based surveys; and independent external
evaluation of the impact of the regulatory oversight.
The assessor should confirm the establishment of a set of KPIs for monitoring progress
in meeting the objectives of the strategic plan (or equivalent) and the institutional
development plan.
The assessor should check whether the outcome of monitoring and measuring regulatory 01
performance has resulted in corrective actions, preventive actions, and/or improvements in

National Regulatory System (RS): Indicators and Fact Sheets

the regulatory activities.

Objective: The objective of this sub-indicator is to ensure the NRA and affiliated institutions have
established requirements to monitor, supervise and review its performance. Establishment
of KPIs is a very effective approach for measuring and monitoring progress. A KPI is a
measurable value that evaluates how effectively the NRA is achieving the strategic plan
and the institutional development plan. Organizations use KPIs to evaluate their success at
reaching targets.
In the case of outsourced activities, the NRA should monitor and review the performance
of the contract acceptor including the implementation of any needed improvements and
the assessment of their effectiveness.

Requirement: Monitor, supervise and review the key regulatory performance indicators used for
monitoring progress

Evidence to review: The assessor should request for and review:

1. Written documents that describe the monitoring, supervision and review of different
regulatory activities;
2. Supporting records, including corrective actions and preventive actions, when applicable;
3. Documentation of KPIs; 02
4. Records of surveys conducted, and indicators reported;
5. KPI evaluation reports, including statistical data and trend analyses in regard to KPIs, 03
where applicable.
04
References: 1. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: 05
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical
Report Series, No. 885) (1) (http://apps.who.int/medicinedocs/documents/s21964en/ 06
s21964en.pdf)
2. Guidelines for national authorities on quality assurance for biological products. In: 07
WHO Expert Committee on Biological Standardization: forty-second report. World Health
Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2), (http://www.who. 08
int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_A2.pdf)
3. Regulation and licensing of biological products in countries with newly developing
09
regulatory authorities. In: WHO Expert Committee on Biological Standardization: forty-fifth
report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical Report Series,
No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001.,
(116), (http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)

Framework: “Process”

Rating Scale:  NOT IMPLEMENTED (NI): There are no established requirements to monitor, supervise
and review the performance of the NRA and affiliated institutions using KPIs.
 ONGOING IMPLEMENTATION (OI): There are no established requirements to monitor,
supervise and review the performance of the NRA and affiliated institutions using
KPIs, but demonstrable steps have been taken towards this.
 PARTIALLY IMPLEMENTED (PI): There are some established requirements to monitor,
supervise and review the performance of the NRA and affiliated institutions but KPIs
have not been established.

Page 70 of 340
WHO Global Benchmarking Tool (GBT)

 IMPLEMENTED (I): There are established requirements to monitor, supervise and

review the performance of the NRA and affiliated institutions using KPIs.

Limitations and remarks:  A regulatory impact analysis is a valuable tool for systematic assessment of the
expected effects of regulatory proposals. It is aimed mainly at assisting decision
makers in their consideration of the recommended proposal.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator
will always apply for all benchmarked NRAs).
01
Sub Indicator: RS10.02: Reports on the regulatory activities and on the progression and status

National Regulatory System (RS): Indicators and Fact Sheets

of resources are available at regular intervals.

Maturity Level: 4

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that reviews are planned and carried out at regular intervals,
and that these reviews take into consideration:
1. NRA performance and effectiveness of medical products regulatory oversight;
2. opportunities for improvement;
3. adequacy of resources.
The organization should retain documented evidence of results of management reviews.

Objective: The objective of this sub-indicator is to ensure that top management reviews the NRA
performance, the areas for improvement, and the status of resources at planned intervals.
The purpose is to ensure continued effectiveness of medical products regulatory oversight
and alignment with the organization’s strategic direction. In addition, the review should
evaluate whether required resources are available for conducting the regulatory activities.
02
Requirement: Reports at regular intervals on the regulatory activities
03
Evidence to review: The assessor should request for and review:
1. Management review procedures (i.e., SOPs or other written documents); 04
2. Records documenting the conduct of management reviews, including periodicity;
3. Annual reports; 05
4. Documentation of the inputs to (e.g., audit results and customer satisfaction surveys)
and outputs from (e.g., decisions, corrective actions, and preventive actions) these periodic 06
reviews.
07
References: 1. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: 08
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical
Report Series, No. 885) (1) (http://apps.who.int/medicinedocs/documents/s21964en/
09
s21964en.pdf)
2. Guidelines for national authorities on quality assurance for biological products. In:
WHO Expert Committee on Biological Standardization: forty-second report. World Health
Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2), (http://www.who.
int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_A2.pdf)
3. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization: forty-fifth
report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical Report Series,
No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_TRS_858_A1.pdf)
4. How to develop and implement a national drug policy, Second edition. WHO, 2001., (116),
(http://apps.who.int/medicinedocs/pdf/s2283e/s2283e.pdf)

Framework: “Output”.

Rating Scale:  NOT IMPLEMENTED (NI): The NRA does not have reports at regular intervals on the
NRA performance, areas for improvement and status of resources.

Page 71 of 340
WHO Global Benchmarking Tool (GBT)

 ONGOING IMPLEMENTATION (OI): The NRA does not have reports of such activities
but demonstrable steps have been taken towards this.
 PARTIALLY IMPLEMENTED (PI): The NRA has started to report on the NRA
performance, areas for improvement and status of resources, but the activity is not
conducted regularly.
 IMPLEMENTED (I): Reports at regular intervals on the regulatory activities,
progression and status of resources are available.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will 01
always apply for all benchmarked NRAs).

National Regulatory System (RS): Indicators and Fact Sheets

02
03
04
05
06
07
08
09

Page 72 of 340
WHO Global Benchmarking Tool (GBT)

02 01

National Regulatory System (RS): Indicators and Fact Sheets

Registration and Marketing
Authorization (MA):
Indicators and Fact Sheets

WHO Global
Benchmarking 02
Tool (GBT) 03
for Evaluation of
04
National Regulatory
System of Medical 05
Products 06
07
08
09

Page 73 of 340
WHO Global Benchmarking Tool (GBT)

02. Registration and Marketing Authorization

(MA): Indicators and Fact Sheets

Function:
02— REGISTRATION AND MARKETING AUTHORIZATION (MA)
01
Description: The issuance of marketing authorizations (also referred as product licensing or 02
registration) is critical to any National Regulatory Authority (NRA). Marketing

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

authorization (MA) refers to a procedure for approval of a medical product for
marketing after it has undergone a process of evaluation to determine the safety,
efficacy and quality of the product and the appropriateness of the product information.
The objective of this regulatory function is to provide a system which ensures that
only medical products which have been duly authorized by the NRA are allowed to
be manufactured, imported, distributed, sold or supplied to end-users. The process
of assessment for MA includes the review of data on quality, safety and efficacy
submitted by the applicant. The same standards should be applied to imported
and locally manufactured medical products. Nevertheless, the evaluation of the
complex data used to support market authorization of new or novel medical
products may require specialized resources and experience not available in the
NRA; therefore, countries with limited resources may wish to give priority to well-
established products.

In the evaluation of well-established products, the NRA may elect to prepare its
own report, rely on evaluation reports prepared by other national authorities, rely on
decisions made by another NRA, or use a combination of these approaches. Good
Manufacturing Practices (GMP) inspections or certifications should be part of the
MA requirements. A legal provision should exist that allows the NRA to grant MA of
either unlimited or limited duration. In the earlier case, measures should be in place
to ensure continued adherence of the medical product to quality, safety, and efficacy
standards (e.g. through a vigilance system). In the latter case, MA should be renewed
at a predefined time interval.
03
Mechanisms should exist to ensure that information on MA applications,
including authorized, suspended, rejected or completed applications, is published 04
to promote transparency and information sharing among stakeholders.
Regular publication of MA decisions is helpful to procurement and distribution 05
networks and to other NRAs. Mechanisms and procedures should be in place
to monitor and ensure that all activities within the MA function are checked to 06
reduce errors and to reasonably ensure that the processes are consistent and will
provide assurance of high quality outputs. Such approaches would lead to 07
consistency in the performance of the MA regulatory function and to reliability
of the regulatory outputs. 08
09
Indicator: MA01 Legal provisions, regulations and guidelines required to define regulatory
framework of registration and/or marketing authorization.

Objective: The objective of this indicator is to ensure the existence of the legal provisions,
regulations and guidelines which form the backbone of the regulatory framework.
Such mandates will support the MA function by ensuring that medical products for
human use in a particular jurisdiction have been assessed by an NRA and found to
be of adequate quality, safety and efficacy (or effectiveness) before being registered
or granted MA and placed in the market. The legal provisions empower the NRA
with the legal mandate to register and grant MA and also to penalize those who
market medical products without the proper health registration or MA.

Category: Legal provisions, regulations and guidelines

Page 74 of 340
WHO Global Benchmarking Tool (GBT)

Sub Indicator: MA01.01: There are legal provisions that require the receipt of a registration
or marketing authorization (MA) before placing the product on the market.

Maturity Level: 1

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that legal provisions requiring registration or MA before 01
placing a medical product on the market exist and are enacted and implemented. The
legal provisions should clearly mandate that all medical products require registration 02
or MA before they are placed on the market. These provisions should provide a list of

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

medical product classes (e.g. drugs, vaccines, or medical devices) that require registration
or MA before they are marketed or sold. The legal provisions should specify the applicable
fines, charges, penalties, or sanctions in the event of non-compliance. The assessor
should note that some medical products may be imported and marketed before
registration or MA in special circumstances such as during an emergency or in response
to an urgent public health concern. The National Regulatory Authority (NRA) should have
systems in place for the active monitoring of such medical products. The assessor may
refer to the legal provisions related to emergency situations before applying the scoring

Objective: The objective of this sub-indicator is to ensure legal provisions are in place that require
that a medical product must have been registered or granted MA by the NRA before it is
marketed or placed on the market

Requirement: Legal provisions for medical product registration or MA

Evidence to review: The assessor should ask for and review:

1) Legal provisions that mandate that all medical products require registration or MA
before they are placed on the market. The legal provisions should also specify the
applicable fines, charges, penalties, and sanctions in the event of non-compliance.
2) Evidence that the legal provisions have been published and implemented;
3) Evidence that the legal provisions are applicable to all medical products.
4) Evidence that registrations or MAs have been issued. Note that this evidence may be the
registration or MA numbers or the registration or MA certificates
03
References: 1. Guiding principles for small national drug regulatory authorities. In: WHO Expert
Committee on Specifications for Pharmaceutical Preparations: thirty-first report. Geneva: 04
World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No. 790), (5),
(http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf) 05
2. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization:
06
forty-fifth report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical
Report Series, No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_
07
TRS_858_A1.pdf)
08
Framework: Structure/Foundation/Input
09
Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence of legal provisions to hold a
registration or MA before placing the product on the market.
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted legal provisions to
hold a registration or MA but it has not yet been followed.
 PARTIALLY IMPLEMENTED (PI): The NRA has legal provisions to hold a registration or
MA and has been applying it for less than two years.
 IMPLEMENTED (I): The NRA has legal provisions to hold a registration or MA before
placing a product on the market and can demonstrate a proven track record of
applying them.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Page 75 of 340
WHO Global Benchmarking Tool (GBT)

Sub Indicator: MA01.02: There are legal provisions that require the NRA to withhold, suspend,
withdraw or cancel an MA if there are concerns regarding quality, safety or
efficacy issues.

ML 1

Scope: 1. Medicines
2. Vaccines
01
Description: The assessor should verify that legal provisions are in place that address the withholding,
suspending, withdrawing or cancelling of a registration or MA in the event that quality, 02
safety or efficacy issues are uncovered. Such legal provisions should be enacted and

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

implemented. The legal provisions also should specify requirements on when and how to
withhold, suspend, withdraw or cancel a registration or MA.

Objective: The objective of this sub-indicator is to ensure that legal provisions exist mandating
the NRA to withhold, suspend, withdraw or cancel registration or MA in the event of
adverse findings related to the quality, safety or efficacy of medical products. The NRA is
empowered by legal provisions to withhold, suspend, withdraw or cancel a registration or
MA if the requirements set forth for registration or MA are no longer being met. Details
on reasons for these decisions should be notified to MA holders. These legal provisions
should provide for the sanctioning of those who market medical products without the
proper registration or MA.

Requirement: Legal provisions on withholding, suspending, withdrawing or cancelling an MA

Evidence to review: The assessor should ask for and review:

1. Legal provisions that mandate the NRA to withhold, suspend, withdraw or cancel a
registration or MA in the event of adverse findings related to the quality, safety and efficacy
of the medical products.
2. Regulations and guidelines presenting details on when and how to withhold, suspend,
withdraw or cancel registration or MA,
3. Evidence that the legal provisions have been implemented, and if possible, examples of
implementation.
4. Evidence that stakeholders (e.g., applicants, wholesalers, and retailers) are aware of the
legal provisions. 03

References: 1. Guidelines for national authorities on quality assurance for biological products. In: WHO 04
Expert Committee on Biological Standardization: forty-second report. Geneva. World Health
Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822) (2) (http://www.who. 05
int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_A2.pdf )
2. Guiding principles for small national drug regulatory authorities. In: WHO Expert
06
Committee on Specifications for Pharmaceutical Preparations: thirty-first report. Geneva:
World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No. 790), (5),
07
(http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
08
Framework: Structure/Foundation/Input
09
Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence of legal provisions to withhold, suspend,
withdraw or cancel a MA in case there are concerns on quality, safety or efficacy issues.
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted the legal provisions
but they have not yet been followed.
 PARTIALLY IMPLEMENTED (PI): The NRA has legal provisions and has been applying
them for less than two years.
 IMPLEMENTED (I): The NRA has legal provisions to withhold, suspend, withdraw or
cancel a MA in case there are concerns on quality, safety or efficacy issues and can
demonstrate proven track record of applying them

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Page 76 of 340
WHO Global Benchmarking Tool (GBT)

Sub Indicator: MA01.03: There are legal provisions that require demonstration of the product
quality, safety and efficacy prior to registration or MA.

Maturity Level: 1

Scope: 1. Medicines
2. Vaccines
01
Description: The assessor should verify that legal provisions are in place that stipulate that
the demonstration of acceptable quality, safety and efficacy profiles, including 02
demonstration of bioequivalence/bioavailability (BE/BA) when deemed necessary for

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

multisource generic medicines, is a pre-requisite to registration or MA. These legal
provisions should be enacted and implemented. Assessors should note that in the case
of multisource generic medicines, BE/BA studies would be required based on type of
dosage form, method of administration, and health risk criteria determined by the NRA.
The legal provisions should empower the NRA to request satisfactory information on
the quality, safety and efficacy of medical products seeking registration or MA. The
submission should follow the either the common technical document format or another
internationally-accepted format. The provisions should be supported by regulations
and guidelines that provide guidance to applicants on the required quality, safety and
efficacy documentation. All the necessary legal documents should be published, easily
accessible, and enforceable.

Objective: The objective of this sub-indicator is to ensure that legal provisions exist mandating the
NRA to request a demonstration (technically and scientifically) that the medical product
meets all quality, safety and efficacy regulatory requirements prior to MA. The NRA should
require that applicants comply fully with the legal requirements for product registration or
MA.

Requirement: Quality, safety and efficacy prerequisites for registration or MA

Evidence to review: The assessor should ask for and review:

1. Legal provisions that empower the NRA to request satisfactory information on
the quality, safety and efficacy of medical products seeking registration or MA. This
information would include the demonstration of BE/BA when it is deemed necessary for 03
multisource generic medicines.
2. Evidence that legal provisions are published and easily accessible to applicants and 04
other stakeholders.
3. Evidence that systems and structures are in place to enforce implementation of the legal 05
provision.
4. Evidence that the legal provisions have been implemented.
06
5) Evidence that MA application guidance documents provide direction on the quality,
safety and efficacy aspects of the legal provisions.
07
08
References: 1. Guidelines for national authorities on quality assurance for biological products. In:
WHO Expert Committee on Biological Standardization: forty-second report. Geneva.
09
World Health Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822) (2)
(http://www.who.int/biologicals/publications/trs/areas/biological_products/WHO_
TRS_822_A2.pdf )
2. Guiding principles for small national drug regulatory authorities. In: WHO Expert
Committee on Specifications for Pharmaceutical Preparations: thirty-first report. Geneva:
World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No. 790), (5),
(http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence of existence of legal provisions that
require demonstration of the product quality, safety and efficacy prior to registration
or MA.

Page 77 of 340
WHO Global Benchmarking Tool (GBT)

 ONGOING IMPLEMENTATION (OI): The NRA has recently established these legal
provisions, but they have not yet been implemented.
 PARTIALLY IMPLEMENTED (PI): The NRA has legal provisions and has been applying
them for less than two years.
 IMPLEMENTED (I): The NRA has legal provisions that require demonstration of the
product quality, safety and efficacy prior to registration or MA and can demonstrate
a proven track record of applying them.

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

of the MA and requiring periodic reviews of MAs (i.e. renewals).

Maturity Level: 2

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that the legal provisions or regulations specifying the validity
period for registrations or MAs (and their renewals) should exist and be enacted and
implemented. The legal provisions or regulations should mandate the NRA to register
or grant MA with limited validity and also require that the registration or MA be renewed
before the medical product is re-introduced onto the market. The legal provisions and
regulations should specify the appropriate validity periods of the initial registration or MA
and subsequent renewals of registration or MA. The legal provisions also should provide
information on the requirements for renewing the registrations or MAs. Ideally, the validity
period for MA should not be more than five years.

Objective: The objective of this sub-indicator is to ensure that legal provisions or regulations exist that
mandate the NRA to register or grant MA with limited validity and that require the registration
or MA to be renewed before the medical product is re-introduced onto the market.

Requirement: Registration or MA validity period and registration or MA renewal

Evidence to review: The assessor should ask for and review: 03

1. Legal provisions or regulations that mandate the NRA to register or grant MA with
limited validity and that require the registration or MA to be renewed before the medical 04
product is re-introduced onto the market
2. Legal provisions or regulations that specify the validity periods for the initial registration 05
or MA and subsequent registration or MA renewals.
3. Regulations and guidelines that provide the requirements for renewing registrations
06
or MAs.
07
References: 1. Guiding principles for small national drug regulatory authorities. In: WHO Expert
08
Committee on Specifications for Pharmaceutical Preparations: thirty-first report. Geneva:
World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No. 790), (5),
09
(http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There are no legal provisions or regulations regarding the
limited duration of the validity of the MA and for periodic reviews to MAs
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted these legal
provisions, but they have not yet been implemented.
 PARTIALLY IMPLEMENTED (PI): The NRA has legal provisions and has been applying
them for less than two years.
 IMPLEMENTED (I): The NRA has legal provisions or regulations regarding the
limited duration of the validity of the MA and for periodic reviews to MAs and can
demonstrate a proven track record of applying them.

Page 78 of 340
WHO Global Benchmarking Tool (GBT)

Limitations and remarks:  The assessor should note that some NRAs have regulations that mandate them
to grant a “One Time” registration to a medical product due to active post-market
surveillance
 (PMS) (i.e., market surveillance and vigilance) activities. The assessor may review the
PMS portfolio of the NRA for more information; if this applies, the scoring should be
“Not Applicable”.

Sub Indicator: MA01.05: There are regulations or guidelines for the definitions, types and the scope
of variations along with the required documentation for these variations. 01
Maturity Level: 3 02

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that the regulations or guidelines with respect to
the definitions, types and the scope of variations as well as the documentation
requirements for these variations should exist and be enacted and implemented. The
regulations or guidelines should provide details of the corresponding documentation
requirements for each type or class of variation as well as guidance on the scope of
evaluation required. Ideally, these guidelines should follow international best practices
such as those from World Health Organization (WHO) or International Council on
Harmonization (ICH).

Objective: The objective of this sub-indicator is to ensure that regulations or guidelines exist
to provide definitions, types and the scope of variations, as well as documentation
requirements for each variation. Scientific and technological advancements or new
developments and approaches that may affect the requirements that permitted the
original product registration or MA should be communicated to the NRA in support to
the variation or post-approval changes application.

Requirement: Regulations/ guidelines on variations

Evidence to review: The assessor should ask for and review:

1. Regulations or guidelines that document the definitions, types and scope of
variations, as well as the corresponding documentation requirements. 03
2. Evidence that the regulations or guidelines are published, easily accessible,
implemented and known to the applicant, and that these regulations and guidelines 04
effectively communicate requirements.
3. Examples of the types of variations documented and the scope of evaluations 05
required.
4. Documentation supporting the appropriateness of the corresponding fees (if any).
06
5. Processes, procedures and applicable timelines for submitting variations to the NRA
for review and for receiving feedback
07
08
References: 1. Procedure for Assessing the Acceptability, in Principle, of Vaccines for Purchase by
United Nations Agencies, (77), (http://apps.who.int/medicinedocs/en/q/ and http://
09
digicollection.org/whoqapharm/)
2. Collaborative procedure between the World Health Organization (WHO)
Prequalification Team and interested national regulatory authorities in the assessment
and accelerated national registration of WHO-prequalified pharmaceutical products and
vaccines, (56), (http://apps.who.int/medicinedocs/en/q/ and http://digicollection.org/
whoqapharm/)
3. Guidelines on procedures and data requirements for changes to approved vaccines, (50),
(http://apps.who.int/medicinedocs/en and https://www.who.int/biologicals/vaccines/en/)
4. Multisource (generic) pharmaceutical products: guidelines on registration requirements
to establish interchangeability, (12), (http://digicollection.org/whoqapharm/p/about and
http://apps.who.int/medicinedocs/en)

Framework: Structure/Foundation/Input

Page 79 of 340
WHO Global Benchmarking Tool (GBT)

Rating Scale:  NOT IMPLEMENTED (NI): There are no regulations or guidelines for the definitions,
types and the scope of variations along with the required documentation.
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted these regulations or
guidelines but they have not yet been implemented.
 PARTIALLY IMPLEMENTED (PI): The NRA has regulations or guidelines and has been
applying them for less than two years.
 IMPLEMENTED (I): The NRA has regulations or guidelines for the definitions, types
and the scope of variations along with the required documentation and required
evidence of implementation. 01
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will 02
always apply for all benchmarked NRAs).

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

Sub Indicator: MA01.06: There are legal provisions to cover circumstances under which the
routine MA procedures may not be followed (e.g., for public health interest).

Maturity Level: 1

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that legal provisions, which describe the circumstances
under which the routine medical product registration or MA procedures may not be
followed, should exist and be enacted and implemented. The legal provisions should
clearly describe the instances or situations when the normal or routine procedures
should not be applied. The legal provisions also should provide guidance on the scope
of evaluation and on the documentation required in the event that the non- routine
approach is used for registration or MA. In addition, if the conditional authorization is
time limited, the assessor should verify that the duration of validity is defined in this
regulation.

Objective: The objective of this sub-indicator is to ensure that legal provisions exist to allow the
NRA to use a non-routine approach or procedure for registration or MA of medical
products under exceptional circumstances in the interest of public health (e.g.,
emergencies or epidemics).
03
Requirement: Exemptions to routine MA procedures
04
Evidence to review: The assessor should ask for and review:
1. Legal provisions that define circumstances or instances under which the routine MA 05
procedures may not be followed.
2. Regulations or guidelines that give clarity on the regulatory requirements for granting
06
MA to medical products through a route other than the routine MA procedure.
3. Records of medical products, if any, that have been authorized by this route.
07
08
References: 1. Guiding principles for small national drug regulatory authorities. In: WHO Expert
Committee on Specifications for Pharmaceutical Preparations: thirty-first report. Geneva:
09
World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No. 790), (5),
(http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
2. WHO Emergency response framework, second edition. Geneva. World
Health Organization, 2017 (140) (http://apps.who.int/iris/bitstream/hand
le/10665/258604/9789241512299-eng.pdf?sequence=1)
3. Emergency Use Assessment and Listing Procedure (EUAL) for Candidate Medicines
for Use in the Context of a Public Health Emergency, (121), (http://apps.who.int/
medicinedocs/en/)
4. Emergency Use Assessment and Listing Procedure (EUAL) for Candidate Vaccines
for Use in the Context of a Public Health Emergency, (122), (http://apps.who.int/
medicinedocs/en/)

Framework: Structure/Foundation/Input

Page 80 of 340
WHO Global Benchmarking Tool (GBT)

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence of existence of legal provisions to
cover circumstances in which the routine MA procedures may not be followed.
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted these legal
provisions but they have not yet been followed.
 PARTIALLY IMPLEMENTED (PI): The NRA has legal provisions and has been applying
them for less than two years.
 IMPLEMENTED (I): The NRA has legal provisions to cover circumstances in which
the routine MA procedures may not be followed and has applied them for all medical
products that are used under this condition. 01
Limitations and remarks:  It is proposed that the NRA should establish a provision to clarify the role and 02
responsibilities of all involved stakeholders who are involved in the emergency. The

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

establishment of an advisory committee is suggested to make decisions and to share
the responsibilities during an emergency situation.
 Non-routine MA procedures are those applicable only under specific
circumstances (e.g., emergencies with public health impact) and aimed at
reducing the time needed for registration or granting MA. The terminology may
vary according the national legislation. Examples of such procedures include
different types of collaborative and facilitated registration and MA procedures
(e.g., WHO collaborative procedure or stringent regulatory authority facilitated
registration procedure).
The collaborative procedure is a procedure for collaboration between the
WHO Prequalification Team and interested national regulatory authorities for
the assessment and accelerated national registration of WHO-prequalified
pharmaceutical products and vaccines.

Sub Indicator: MA01.07: There are legal provisions or regulations that define regulatory
requirements to approve donation of medical products.

Maturity Level: 1

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that legal provisions or regulations, which define the
circumstances under which medical products can be received through donation, exist 03
and are implemented. The legal provisions should provide guidance regarding required
information and documentation that should be reviewed before approval of donated 04
medical products. The provisions should also provide guidance on the scope of evaluation
as well as the documentation required to show quality, safety and efficacy of donated 05
medical products.
06
Objective: The objective of this sub-indicator is to ensure that legal provisions or regulations exist and
are implemented to mandate the NRA to have adequate regulatory oversight on medical
07
products received through donation.
08
Requirement: MA procedures for approval of donated medical products
09
Evidence to review: The assessor should ask for and review:
1. Legal provisions that mandate the NRA to have adequate regulatory oversight on
medical products received through donation.
2. Regulations or guidelines giving clarity on the regulatory requirements for use of medical
products that are received through donation.
3. Supporting records for any application of these provisions.

References: 1. GPV Policy statement. Vaccine donations. Geneva. World Health Organization (WHO/
VSQ/97.03), (111), (https://www.who.int/immunization/hpv/plan/who_unicef_joint_
statement_on_vaccine_donations_who_unicef_2011.pdf)
2. Guidelines for Medicine Donations - Revised 2010. (First Edition 1996, Second Edition
1999, Third Edition 2011), (141), (http://apps.who.int/medicinedocs/en/)

Page 81 of 340
WHO Global Benchmarking Tool (GBT)

3. Improvements for International Medicine Donations: a Review of the World Health

Organization Guidelines for Medicine Donations, 3rd Edition, (142), (http://apps.who.int/
medicinedocs/en/)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence for existence of legal provisions or
regulations that define regulatory requirements for use of donated medical products
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted these legal 01
provisions or regulations but they have not yet been followed.
 PARTIALLY IMPLEMENTED (PI): The NRA has these legal provisions and has been 02
applying them for less than two years.

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

 IMPLEMENTED (I): The NRA has legal provisions or regulations that define the
regulatory requirements for use of donated medical products and these have been
applied for all medical products received through donation.

Limitations and remarks: In exceptional cases in the interest of public health (e.g., emergencies or epidemics), use
of a non-routine approach or procedure should be considered to permit use of a donated
medical product or to grant MA to a donated medical product. If the NRA use the same
regulatory approach for review of donated products as it uses for review of other medical
products, this sub indicator can score as not applicable.

Sub Indicator: MA01.08: Legal provisions or regulations allow the NRA to recognize and/or
rely on MA-relevant decisions, reports or information from other NRAs or regional
and international bodies.

Maturity Level: 1

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that the legal provisions or regulations allowing the NRAs to
recognize and/or use relevant MA decisions, reports or information from other NRAs or
regional and international bodies should exist and be enacted and implemented. The
legal provisions or regulations should provide clarity on the modalities, processes
and procedures to employ to recognize and/or use relevant MA decisions, reports or 03
information from other NRAs or regional and international bodies. The legal provisions also
should provide clarity on the conditions (i.e., when and how) under which it is acceptable 04
to recognize and/or use relevant MA decisions, reports or information from other NRAs,
or regional and international bodies. The legal provisions or regulations should also 05
provide clear directives on the scope or extent of recognition and/or use of MA decisions,
reports or information from other NRAs, or regional and international bodies (e.g., WHO
06
prequalification).
07
Objective: The objective of this sub-indicator is to verify that legal provisions or regulations exist
08
to allow the NRA to establish a systematic framework to recognize and/or use relevant
MA decisions, reports or information from other NRAs, or regional and international
09
bodies.

Requirement: Recognition and/or reliance of MA application decisions and information

Evidence to review: The assessor should ask for and review:

1. Legal provisions including guidelines that permit the NRA to recognize and/or
use relevant MA decisions, reports or information from other NRAs or regional and
international bodies
2. Documentation defining the scope and extent of recognition and /or use of relevant MA
decisions, reports or information from other NRAs, or regional and international bodies.
3. Documentation providing examples of instances or situations that have permitted
recognition and/or use of relevant MA decisions, reports or information from other NRAs or
regional and international bodies

Page 82 of 340
WHO Global Benchmarking Tool (GBT)

4. List of NRAs or regional and international bodies whose relevant decisions, reports or
information may be used to influence a MA application decision.
5. Evidence that the listed NRAs or regional and international bodies are aware that
their relevant MA decisions, reports or information may be used (i.e., aware of the legal
provision).

References: 1. Good regulatory practices: guidelines for national regulatory authorities for medical
products, (112), (http://digicollection.org/whoqapharm/p/about and http://apps.who.int/
medicinedocs/en) 01
2. WHO general guidance on variations to multisource pharmaceutical products, (143),
(http://digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en) 02
3. Guidelines on procedures and data requirements for changes to approved vaccines, (50),

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

(http://apps.who.int/medicinedocs/en and https://www.who.int/biologicals/vaccines/en/)
4. Guidelines on procedures and data requirements for changes to approved biotherapeutic
products, (144), (http://apps.who.int/medicinedocs/en/)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence for existence of legal provisions or
regulations that allow the NRA to recognize and/or use MA-relevant decisions, reports
or information from other NRAs or regional and international bodies
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted these legal
provisions or regulations but they have not yet been followed.
 PARTIALLY IMPLEMENTED (PI): The NRA has these legal provisions or regulations
and has been applying them to some supported resources for less than two years.
 IMPLEMENTED (I): The NRA has legal provisions or regulations allowing the NRA
to recognize and/or use MA-relevant decisions, reports or information from other
NRAs or regional and international bodies and is able to provide evidence supporting
implementation for a list of NRAs or regional and international bodies.

Limitations and remarks: If the NRA has adequate resources, this sub indicator can be considered not applicable;
however, the NRA should have the legal bases for any reliance that does exist.

Sub Indicator: MA01.09: Specific guidelines on the quality, nonclinical and clinical aspects
are established and implemented.
03
Maturity Level: 3
04
Scope: 1. Medicines
2. Vaccines 05

Description: The assessor should verify that specific guidelines on quality, nonclinical, and clinical
06
requirements (as well as guidelines for BE/BA studies in the case of multisource generic
medicines) for registering or granting MA should be established and implemented. The
07
guidelines should be detailed and should provide the specific regulatory requirementsfor
08
each section of the quality, nonclinical, and clinical dossier. The guidelines should be
specific for the type of application and the medical product. In some cases, the specific
09
guidelines may consist of documents that were not prepared by the NRA but instead were
adopted in accordance with proper process.

Objective: The objective of this sub-indicator is to verify that specific guidelines exist on the quality,
nonclinical and clinical requirements for registering or granting MA, and to verify that they
have been implemented.

Requirement: Requirement on quality, nonclinical and clinical aspects

Evidence to review: The assessor should ask for and review:

1. Guidelines detailing the specific regulatory requirements for the quality, nonclinical and
clinical aspects (as well as guidelines for BE/BA studies in the case of multisource generic
medicines) of the MA dossier. The requirements, including the content and format, should

Page 83 of 340
WHO Global Benchmarking Tool (GBT)

comply with updated international best practices such as those from WHO or ICH.
2. Evidence that the guidelines are specific for the quality, nonclinical, and clinical aspects
of the MA application dossier and for the type of application and pharmaceutical product.
3. Evidence that the guidelines are published and implemented.
4. Examples of MA application submissions that are in compliance with the published
guidelines.
5. Evidence that documents submitted were appropriate for the type of product and type of
application (e.g., new drug application, multisource drug, or product of biological origin or
chemical synthesis). 01
6. Specific guidelines related to
a. product labelling and packaging, 02
b. package inserts,

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

c. summary of product characteristics (SPC) information pamphlet for professionals or
equivalent
d. information pamphlet for patients.
7. Copies of all quality, safety, efficacy reports for recently approved MA applications to
determine whether reviews were done thoroughly and based on guidelines requirements.

References: 1. Quality assurance of pharmaceuticals WHO guidelines, good practices, related

regulatory guidance and GXP training materials, (79), (http://digicollection.org/
whoqapharm/)
2. Guidelines on the quality, safety and efficacy of biotherapeutic protein products prepared
by recombinant DNA technology, (54), (https://www.who.int/biologicals/WHO_ECBS/en/
and https://www.who.int/biologicals/vaccines/en/)
3. WHO good manufacturing practices for pharmaceutical products: main principles, (51),
(http://digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en)
4. Multisource (generic) pharmaceutical products: guidelines on registration requirements
to establish interchangeability, (12), (http://digicollection.org/whoqapharm/p/about and
http://apps.who.int/medicinedocs/en)
5. Guidelines for national authorities on quality assurance for biological products. In:
WHO Expert Committee on Biological Standardization: forty-second report. Geneva.
World Health Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822),
(2), (http://www.who.int/biologicals/publications/trs/areas/biological_products/WHO_
TRS_822_A2.pdf )

Framework: Structure/Foundation/Input 03

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence for the existence of specific guidelines 04
on the quality, nonclinical, and clinical aspects.
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted these guidelines but 05
they have not yet been followed.
 PARTIALLY IMPLEMENTED (PI): The NRA has these guidelines and has been applying
06
them for less than two years.
 IMPLEMENTED (I): The NRA has established and implemented specific guidelines on
07
the quality, nonclinical, and clinical aspects.
08
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
09
always apply for all benchmarked NRAs).

Sub Indicator: MA01.10: There are guidelines on the format and content for submission
of MA applications that are consistent with the WHO or other internationally
accepted standards.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that guidelines on the format and content of a MA submission
exist and that they are consistent with international best practices, such as those

Page 84 of 340
WHO Global Benchmarking Tool (GBT)

proposed or recommended by WHO or other internationally established agencies such

as ICH. The guidelines should provide clarity on the format and content on all sections
of the MA application submission and should provide specific guidance for each medical
product stream. The guideline also should describe the processes and procedures for
submitting the application to the NRA. Additionally, the guidelines should be established
and implemented.

Objective: The objective of this sub-indicator is to verify that guidelines exist for the format and
content of a MA application submitted to the NRA; these guidelines should consider the 01
nature of products and the different medical product streams. The content and format of
submitted applications should be consistent with the recommendations from the WHO or 02
other internationally accepted agencies.

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

Requirement: Format and content for MA application

Evidence to review: The assessor should ask for and review:

1. Guidelines providing clarity on the format and content of the MA application submission,
as well as the processes and procedures for submitting the application to the NRA.
2. Evidence that the guidelines are consistent with international best practices, such as
those proposed by the WHO or other internationally accepted agencies.
3. Copies of the checklist used to ensure that MA submissions actually comply with the
content of the guidelines before a MA application is acknowledged by the NRA.

References: 1. Multisource (generic) pharmaceutical products: guidelines on registration requirements

to establish interchangeability, (12), (http://digicollection.org/whoqapharm/p/about and
http://apps.who.int/medicinedocs/en)
2. WHO good manufacturing practices for pharmaceutical products: main principles, (51),
(http://digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en)
3. Guidelines on the quality, safety and efficacy of biotherapeutic protein products prepared
by recombinant DNA technology, (54), (https://www.who.int/biologicals/WHO_ECBS/en/
and https://www.who.int/biologicals/vaccines/en/)
4. Open this document and view its content Guidelines on Submission of Documentation
for a Multisource (Generic) Finished Product. General Format: Preparation of Product
Dossiers in Common Technical Document Format, (65), (http://apps.who.int/
medicinedocs/en/q/ and http://digicollection.org/whoqapharm/)
5. Guidelines on submission of documentation for a multisource (generic) finished 03
pharmaceutical product: quality part, (52), (http://digicollection.org/whoqapharm/p/about
and http://apps.who.int/medicinedocs/en) 04
6. International Conference on Harmonization, ICH tripartite guidelines. The Common
Technical document for registration of pharmaceuticals for human use: quality. M4Q (R1). 05
Quality overall summary of Module 2. Module 3: Quality; September 2002, (118), (https://
www.ich.org/products/ctd/ctdsingle/article/m4qr1-quality.html)
06

Framework: Structure/Foundation/Input
07
08
Rating Scale:  NOT IMPLEMENTED (NI): There are no guidelines for the format and content of
submitted MA applications.
09
 ONGOING IMPLEMENTATION (OI): The NRA has recently developed or drafted these
guidelines, but they have not yet been followed.
 PARTIALLY IMPLEMENTED (PI): The NRA has developed or adopted these guidelines
and has been applying them for less than two years, or these guidelines have not
been applied for all marketed medical products.
 IMPLEMENTED (I): The NRA has guidelines for the format and content of submitted
MA applications that are consistent with WHO or other internationally accepted
standards and that specify required documentation. This process is implemented and
followed for all marketed medical products.

Limitations and remarks:  In case of benchmarking for more than one medical product stream, the assessor
should consider and review different guidelines, as applicable. In some cases,
instructions for dossier compilation may be part of the main (technical) guideline.

Page 85 of 340
WHO Global Benchmarking Tool (GBT)

 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: MA01.11: There are guidelines for MA holders that define the types and scope of
variations, the format and content to be used for documenting the variations, and the
identification of those variations that require prior approval or notification.

Maturity Level: 3
01
Scope: 1. Medicines
2. Vaccines 02

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

Description: The assessor should verify the existence, enactment and implementation of guidelines
for MA holders that provide guidance on the types and scope of variations, the format
and content to be used for documenting the variations, and the identification of the
variations that require prior approval or notification. The guidelines should include
directives on changes made to leaflets, package inserts, labelling (i.e., container and
package), and SPCs. They should also clearly communicate to the MA holder the
variations that require NRA approval before implementation, those that require only
notifying the NRA before or during implementation, and those that do not require
notifying the NRA.

Objective: The objective of this sub-indicator is to ensure that guidelines exist that provide
guidance to MA holders on the types and scope of variations, the format and the
content to be used for documenting the variations, and the identification of those
variations that require prior approval or notification.

Requirement: Variation documents for MA holders

Evidence to review: The assessor should ask for and review:

1. Guidelines that provide guidance to MA holders on the types and scope of variations,
the format and content to be used for documenting the variations, and the identification
of those variations that require prior approval or notification. The assessor should note
that the guidelines should include changes to information contained in leaflets, package
inserts, labelling (i.e., package and containers), and SPCs that have been previously
approved. 03
2. Guidelines that communicate to the MA holder the variations that require approval
before implementation, those that require only notifying the NRA before or during 04
implementation, and those that do not require notifying the NRA. The assessor should
review examples of submitted and processed variation applications and evaluate them 05
for conformance to the guidelines.
06
References: 1. Guiding principles for small national drug regulatory authorities. In: WHO Expert
Committee on Specifications for Pharmaceutical Preparations: thirty-first report. Geneva:
07
World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No. 790), (5),
08
(http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
2. Regulation and licensing of biological products in countries with newly developing
09
regulatory authorities. In: WHO Expert Committee on Biological Standardization:
forty-fifth report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical
Report Series, No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_
TRS_858_A1.pdf)
3. Guidelines on procedures and data requirements for changes to approved biotherapeutic
products, (144), (http://apps.who.int/medicinedocs/en/)
4. WHO guidelines on variations to a prequalified product, (145), (http://apps.who.int/
medicinedocs/en/q/ and http://digicollection.org/whoqapharm/)
5. Guidelines on procedures and data requirements for changes to approved vaccines,
(50), (http://apps.who.int/medicinedocs/en and https://www.who.int/biologicals/vaccines/
en/ )

Framework: Structure/Foundation/Input

Page 86 of 340
WHO Global Benchmarking Tool (GBT)

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence for existence and implementation of
guidelines for MA holders that define the types and scope of variations, the format
and the documentation required for each variation, and the identification of the
variations that require prior approval.
 ONGOING IMPLEMENTATION (OI): The NRA has recently developed or drafted these
guidelines, but they have not yet been followed.
 PARTIALLY IMPLEMENTED (PI): The NRA has developed or adopted these guidelines
and has been applying them for less than two years or they have not been applied to
all marketed medical products. 01
 IMPLEMENTED (I): The NRA has guidelines for MA holders that define the types and
scope of variations, the format and the documentation required for each variation, 02
and the identification of the variations that require prior approval, and they have been

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

implemented and documented for all marketed medical products

Limitations and remarks:  In case of benchmarking of more than one medical product stream, the assessor
should consider and review different guidelines as applicable.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: MA01.12: There are established guidelines that cover circumstances under which
the routine MA procedures may not be followed (e.g., for public- health interest).

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that guidelines that provide guidance to the NRA on how to
manage MA applications using a non-routine MA application procedure during public
health emergencies are established and implemented. The guidelines should provide
guidance on the critical requirements that need to be reviewed by the NRA (e.g. label
indication, mode of usage or application, storage conditions, and Good Manufacturing
Practice (GMP) certificates and reports), and on the information that should be
documented and monitored (e.g., location of deployment, quantities to be deployed, and
identity of persons to receive and manage deployment) by the NRA before registration
of the medical products. The guidelines should be supported with Standard Operating 03
Procedures (SOPs) that provide guidance on the assessment and evaluation process and
on the scope of evaluation needed. The assessor should verify that the NRA will have 04
access to experts (e.g., advisory committees) to support the NRA during the decision-
making process. Guidance on the application processing flow and corresponding timelines 05
should be available.
06
Objective: The objective of this sub-indicator is to ensure that guidelines exist to provide guidance
to the NRA on how to process MA applications using a non-routine MA approach or
07
procedure in exceptional cases and in the interest of public health (e.g. emergencies or
08
epidemics).
09
Requirement: Guidance on non-routine MA procedures

Evidence to review: The assessor should ask for and review:

1. Guidelines that provide guidance to the NRA on the application of a non-routine MA
procedure in emergency situations, as well as the corresponding SOPs or supporting
documents.
2. A sample application that was processed using the guidelines. The assessor should
verify that the information required by the guidelines was submitted and duly evaluated,
and that the evaluation process flow and timelines were respected.
3. Evidence of access to experts or advisory committee in case of emergency

References: 1. Guidance on development and implementation of a national deployment and

vaccination plan for pandemic influenza vaccines. Geneva: World Health Organization;

Page 87 of 340
WHO Global Benchmarking Tool (GBT)

2012, (76), (http://apps.who.int/iris/bitstream/handle/10665/75246/9789241503990_eng.

pdf?sequence=1)
2. WHO Emergency response framework, second edition. Geneva. World
Health Organization, 2017, (140), (http://apps.who.int/iris/bitstream/hand
le/10665/258604/9789241512299-eng.pdf?sequence=1)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There are no guidelines that cover circumstances 01
under which the routine MA procedures may not be followed.
 ONGOING IMPLEMENTATION (OI): The NRA has recently developed or drafted 02
these guidelines but they have not yet been followed.

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

 PARTIALLY IMPLEMENTED (PI): The NRA has developed or officially adopted
these guidelines but has not been applied them to all medical products that were
approved by non-routine regulatory procedures.
 IMPLEMENTED (I): The NRA has guidelines that cover circumstances under
which the routine MA procedures may not be followed and has applied them for
all medical products that have been approved under this condition.

Limitations and remarks: There is need to verify that these guidelines have been applied in emergency situations
and that all relevant records are available. If there were not any emergency situations after
development of the guidelines, the assessor should verify that all required resources are
available.

Sub Indicator: MA01.13: There are guidelines on the content of product information leaflets,
SPC-like information, and product packaging and labelling.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the existence and implementation of guidelines that
provide guidance on the MA application requirements with respect to the content of
the product information leaflets, SPC-like information, and product packaging and 03
labelling. These guidelines should also provide clarity on the information to be included
in the package inserts (i.e., inserts and patient information leaflets), SPC-like information 04
(i.e., information leaflets for professionals or equivalent) and labelling of container-
packaging. 05

Objective: The objective of this sub-indicator is to ensure that guidelines exist that provide
06
guidance on the MA application requirements with respect to the content of the product
information leaflets, SPC-like information, and product packaging and labelling.
07
08
Requirement: Guidance on content of product information leaflets, SPC-like information, and product
packaging and labelling
09
Evidence to review: The assessor should ask for and review:
1. Guidelines that provide guidance on the MA application requirements with respect
to the content of the product information leaflets, SPC-like information, and product
packaging and labelling.
2. Relevant supporting documentation including forms and records (e.g., samples of
mock-ups and published product information).

References: 1. Guidelines on packaging for pharmaceutical products, World Health Organization

(WHO), (80), (http://apps.who.int/medicinedocs/en/q/ and http://digicollection.org/
whoqapharm/)

Framework: Structure/Foundation/Input

Page 88 of 340
WHO Global Benchmarking Tool (GBT)

Rating Scale:  NOT IMPLEMENTED (NI): There are no guidelines on the content of product
information leaflets, SPC-like information, and product packaging and labelling
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted these guidelines but
they have not yet been followed
 PARTIALLY IMPLEMENTED (PI): The NRA has developed or adopted these guidelines
and has been applying them for less than two years or has not fully followed them for
all marketed medical products.
 IMPLEMENTED (I): The NRA has guidelines on the content of product information
leaflets, SPC-like information, and product packaging and labelling and they are 01
followed for all marketed medical products.
02
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

always apply for all benchmarked NRAs).

Indicator: MA02 Arrangement for effective organization and good governance.

Objective: The objective of this indicator is to ensure that there is a legal basis and relevant
administrative process supporting the organization and governance of the authority that is
responsible for registration and MA. These provisions should be documented and should
provide for the smooth coordination, collaboration and exchange of information inside
and outside the regulatory authority. The arrangement should establish and define the
roles and responsibilities inside and outside the NRA for registration and MA activities.
Provisions should ensure that the NRA responsible for registration and MA has complete
control or access to all relevant submitted information related to produced, procured or
received medical products. Relevant information includes, for example, GMP inspection
reports, analyses of laborat
ory data, collected vigilance data, applications for registration and MA, the number and
identity of medical products registered and granted MA, activities of ongoing registrations
and MAs, new directives, registrations for newly registered products, withdrawn or
suspended registrations and MAs, and rejected applications.
The objective of this indicator is to establish that structures are in place at the
organizational and governance levels to promote effective intra- and inter-NRA
relationships and efficient management of information traffic.

Category: Organization and governance 03

Sub Indicator: MA02.01: There is a defined structure with clear responsibilities to conduct 04
registration or MA activities.
05
Maturity Level: 2
06
Scope: 1. Medicines
2. Vaccines
07
08
Description: The assessor should verify that defined structures with clear responsibilities to
conduct registration or MA activities have been established and implemented. The
09
organizational structure should identify the entities within the NRA that have the
mandate for registration or MA activities. The structure should be supported with
regulations or guidelines mandating such entities to conduct registration or MA
activities and defining the scope and extent of their roles and responsibilities within the
NRA in relation to registration or MA activities. The guidance should also clarify lines
of reporting in these structures. The organizational structure of the entities should be
defined with respect to relationships and ranks, as well as information exchange and
flow both inside and outside (e.g., officers at the port of entry or officers in the PMS unit)
of the entities.

Objective: The objective of this sub-indicator is to ensure that there are structures in place, with
clearly defined roles and responsibilities for each structural and governance level, to carry
out registration or MA activities that ensure effectiveness, efficiency and consistency.

Page 89 of 340
WHO Global Benchmarking Tool (GBT)

Requirement: Structure and responsibilities

Evidence to review: The assessor should ask for and review:

1. Documents that define the roles and responsibilities of the entities responsible for
registration or MA activities, and that describe the placement of the entities on the
organizational chart in relation to other entities involved in registration or MA–related
activities.
2. Regulations or guidelines empowering such entities or authorities with the responsibility
to conduct registration or MA activities. 01
3. Documentation of the organizational structure of the entities
02
References: 1. National drug regulatory legislation: guiding principles for small drug regulatory

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations:
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical
Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf)
2. Multisource (generic) pharmaceutical products: guidelines on registration requirements
to establish interchangeability, (12), (http://digicollection.org/whoqapharm/p/about and
http://apps.who.int/medicinedocs/en)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence for a defined structure with clear
responsibilities to conduct registration or MA activities.
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted a defined structure
with clear responsibilities to conduct registration or MA activities, but it has not yet
been implemented.
 PARTIALLY IMPLEMENTED (PI): The NRA has a defined and established structure
with clear responsibilities to conduct registration or MA activities; however there is
the need to improve this function by providing the additional resources and support
needed for effective implementation and coordination with other entities.
 IMPLEMENTED (I): The NRA has a defined structure with clear responsibilities
to conduct registration or MA activities that is fully supported with the required
resources.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will 03
always apply for all benchmarked NRAs).
04
Sub Indicator: MA02.02: Documented and implemented procedures exist to ensure involvement
and communication with all relevant regulatory entities as necessary. 05

Maturity Level: 3
06

Scope: 1. Medicines
07
2. Vaccines
08
Description: The assessor should verify the existence and implementation of documented procedures
09
promoting involvement and communication between all relevant entities inside the NRA.
These procedures should be supported with adequate documentation. The procedures
should provide clear directives on the regulatory divisions and provide guidance on how
to implement the operational procedures to ensure the involvement and communication
between relevant entities within the NRA (e.g., the dossier assessment entity, quality
control laboratory, vigilance and PMS entities, and inspectorate). The procedures should
be clear on the scope and extent of the roles and responsibilities of each relevant entity.
In addition, the procedures should be supported by an information and documentation
transfer policy that ensures that directives, information, and other documentation reach
the intended recipients and that feedback is received. The procedures should be captured
in the form of SOPs and guidelines which are approved and authorized to be implemented.
In case of exchange of confidential information, appropriate measures should be
considered; such measures may include confidentiality agreements.

Page 90 of 340
WHO Global Benchmarking Tool (GBT)

Objective: The objective of this sub-indicator is to ensure there are documented procedures,
mechanisms, and structures in place to ensure proper relationships between and within
entities within the NRA (e.g., dossier assessment entity, quality control laboratory, vigilance
and PMS entities and inspectorate) and to ensure effective and efficient exchange of
information among these entities.

Requirement: Procedures for effective communication

Evidence to review: The assessor should ask for and review: 01

1. The registration or marketing application guidelines that capture the duties, roles
and responsibilities of those entities inside the NRA that are involved in registration or 02
marketing activities.

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

2. Evidence that the guidelines and related documents are known to the relevant
organizations, divisions, institutions, and/or departments.
3. Evidence that the documented procedures have been implemented, and that the impact
of the implementation has been evaluated.
4. SOPs that guide and inform effective communication and collaboration within and
among relevant entities inside the NRA (e.g., dossier assessment entity, quality control
laboratory, vigilance entity, and inspectorate).
5. Documentation for the feedback mechanism in the information and
documentation transfer policy that covers information exchange among all relevant
regulatory divisions.

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence of documented procedures to ensure
involvement and communication with all relevant regulatory entities as necessary.
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted procedures to
ensure involvement and communication, but they have not yet been followed. 03
 PARTIALLY IMPLEMENTED (PI): The relevant entities inside the NRA established the
procedures to ensure involvement and communication, but these either have not 04
been fully incorporated or they are not consistent with relevant SOPs.
 IMPLEMENTED (I): The NRA has procedures to ensure involvement and 05
communication with all relevant regulatory entities as necessary, including required
documentation and records.
06

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
07
always apply for all benchmarked NRAs).
08
Indicator: MA03 Human resources to perform registration and marketing
09
authorization activities.

Objective: The objective of this indicator is to ensure that all entities within an NRA are adequately
resourced with a trained, experienced and skilled workforce, and are empowered to
fully perform the function of registering and granting MA for medical products. This will
ensure that medical products registration and MA processes or activities are performed in
accordance with international best practices.
The objective of this indicator is to evaluate the human resource capacity of the entities
with respect to the number of personnel, the skills and experience of the personnel, and
the overall composition of the workforce, with the goal of evaluating whether the workforce
possesses the specific expertise required to perform the registration and MA function.

Category: Resources (HR, FR, infrastructure and equipment)

Page 91 of 340
WHO Global Benchmarking Tool (GBT)

Sub Indicator: MA03.01: Sufficient competent staff (i.e., education, training, skills and experience)
are assigned to perform MA or registration activities.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that the human resources assigned to perform MA or 01
registration activities should be sufficient with respect to numbers and competent
with respect to the requisite skills, education, experience and training. There should be 02
technical documents and SOPs that provide guidance on the required background for MA

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

or registration activities and that consider the requirements for educational background,
competencies, skills, experience, and training.
The assessor should verify that the NRA estimated the number of staff required to
effectively and efficiently perform the MA or registration function and that the NRA actually
recruited that number. In addition, the assessor should verify that these competency
requirements are well-established and maintained by the NRA. Metrics and statistics
on the different activities performed as well as performance indicators can be used for
estimating the adequacy of the number of the assigned staff. The assessor should also
verify that the competency of the assigned staff is built, maintained and improved through
recruitment as well as continuous on-the-job training.

Objective: The objective of this sub-indicator is to ensure the existing human resources for MA or
registration is sufficient, in terms of numbers, experience, and specific competencies, to
perform all the activities along the entire MA or registration chain.

Requirement: Sufficient number of competent human resources in charge of MA or registration activities.

Evidence to review: The assessor should ask for and review:

1. Evidence that the number of staff members involved in each of the documented
activities along the entire MA or registration process flow is adequate.
2. Evidence that the systems and structures are in place to ensure appropriate placement
of staff with respect to competence and skills.
3. Evidence that the system and structures have been implemented. The documentation
should include the records to verify that the staff competence is appropriate for the job 03
requirements.
4. Evidence that the professional profiles of the human resources engaged in MA or 04
registration activities are appropriate with respect to education, skills, and expertise, to
perform a particular function along the MA or registration chain. Documentation should 05
include a list of the requisite skills and training for each position.
5. Recruitment plan.
06

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): The NRA does not have enough competent staff (i.e.,
education, training, skills and experience) to perform MA or registration activities
 ONGOING IMPLEMENTATION (OI): The NRA has recently developed a plan to recruit
adequate competent staff; however, the plan has not been implemented.
 PARTIALLY IMPLEMENTED (PI): The NRA has initiated the implementation of
the human resources development plan; however, there is need to complete the
competency profile.

Page 92 of 340
WHO Global Benchmarking Tool (GBT)

 IMPLEMENTED (I): The NRA has a sustained number of competent staff (i.e.,
education, training, skills and experience) assigned to perform MA or registration
activities.

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

of MA or registration activities are established and updated in the respective
job descriptions.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that procedures are in place to maintain a current and updated
structure for managing job descriptions for personnel participating in MA or registration
activities. In addition, job descriptions should address current staff duties, responsibilities
and the requisite competencies. A job description with this format and content should be
established and implemented for all staff. The management of job descriptions should be
supported by a guidance document that provides direction on when and how to update
the information, and where the information should be kept for easy access. The guidance
document should present the appropriate duties and responsibilities that are assigned
to each member of the organization involved in MA or registration activities. Thus, the
professional profiles of staff are reflected in their respective roles and responsibilities within
the NRA. There should be procedures to guide responsible persons to document that
duties, functions and responsibilities are revised and kept up to date. In addition, procedures
should be available to guide the keeping and documenting of up to date work schedules and
enforcing the implementation of the documented guidelines and procedures.

Objective: The objective of this sub-indicator is to ensure that duties and responsibilities of the staff 03
are clear and well defined, that job descriptions are kept up to date with current duties,
functions and responsibilities, and that these activities are adequately documented. 04

Requirement: Duties, roles and responsibilities of the staff relevant to MA or registration activities. 05

Evidence to review: The assessor should ask for and review:

06
1. Procedure and guidelines that guide placement of staff members within the NRA;
2. The professional profiles of staff (i.e., job descriptions) and documentation that they are
07
related to their current roles and duties;
08
3. The professional profiles of the external experts and documentation that the profiles
provide a composition that is complete and consistent with that prescribed in the legal
09
provisions;
4. Procedures to guide the documentation of up to date duties and work schedules, and to
enforce the implementation of the documented guidelines and procedures;
5. Job descriptions for designated staff.

Page 93 of 340
WHO Global Benchmarking Tool (GBT)

3. Guiding principles for small national drug regulatory authorities. In: WHO Expert
Committee on Specifications for Pharmaceutical Preparations: thirty-first report.
Geneva: World Health Organization; 1990: Annex 6 (WHO Technical Report Series,
No. 790), (5), (http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
4. Effective drug regulation: A multicounty study. Geneva: World Health Organization;
2002, (7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)

Framework: Structure/Foundation/Input
01
Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence of defined or established duties,
functions, responsibilities, respective job descriptions and necessary required 02
competencies.

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted or developed the
role and responsibilities document but it has not yet been implemented.
 PARTIALLY IMPLEMENTED (PI): The NRA has initiated implementation of this
requirement but it has not been defined or followed for all staff or the roles
and responsibilities documents, including staff job descriptions, are not up to
date.
 IMPLEMENTED (I): The NRA has defined and established all required duties,
functions, and responsibilities, and respective job descriptions are up-to-date.

Limitations and remarks:  The issue of overlapping responsibilities among members of the MA assessment
teams is very common in some NRAs. Assessor should recognize the
arrangement and assess on the basis of the adequacy of competencies of the
official for each of the different functions to which they have been assigned.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-
indicator will always apply for all benchmarked NRAs).

Sub Indicator: MA03.03: Training plan developed, implemented and updated at least once a year
for staff in charge of MA or registration activities.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines
03
Description: The assessor should verify that training plans are developed, implemented and
updated at least once every year to reflect the current situation by considering 04
education and experience of the staff. The training plan should be complemented with
guidelines or similar documents that guide the development and implementation of 05
training plans. The assessor should ensure that induction training for new staff as
well as continued on- the-job for staff is planned and implemented. There should be
06
procedures to approve the training plan and the budget allocated for implementing and
updating the training plan. The plans should present clearly defined training goals and
07
should include training in certain topics and skills to address identified deficiencies.
08
Learning objectives, training methods and activities, evidence of learning, and evaluation
and assessment of training should be documented. This documentation should
09
confirm that the learning objectives were achieved and were designed to address
weaknesses within the entities. Procedures should be in place to ensure that a training
plan is developed, implemented and updated at least once every year. The assessor
should verify that there is a system in place for monitoring the implementation and
effectiveness of the training plan and for documenting the skills acquired in training
activities for internal and external experts.

Objective: The objective of this sub-indicator is to ensure that a training plan for staff exists, and
that it is implemented and updated annually. Through the training plan, NRA can be
sure that competency of staff in charge of MA or registration activities is maintained
and enhanced.

Requirement: Implementation of training plan

Page 94 of 340
WHO Global Benchmarking Tool (GBT)

Evidence to review: The assessor should ask for and review:

1. Guidelines for development, implementation and annual update (i.e., at least once per
year) of the training plan. Guidelines should also provide for a mechanism to measure
effectiveness of training.
2. Documentation for the system or structures used to approve the training plan and to
evaluate the adequacy of the budget allocated to the training activities.
3. The current or existing staff training plan (or matrix) for staff. The assessor should
assess this in in relation to the respective individual job descriptions.
4. SOP for developing and maintaining the training plan. 01
5. Evidence that the NRA has investigated and identified training needs.
6. List of trainings performed. 02
7. Example records for training activities.

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

References: 1. Effective drug regulation: A multicounty study. Geneva: World Health Organization; 2002,
(7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)
2. Guidelines for national authorities on quality assurance for biological products. In: WHO
Expert Committee on Biological Standardization: forty-second report. Geneva. World Health
Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2), (http://www.who.
int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_A2.pdf )

Framework: Process

Limitations and remarks:  Training plans must be updated regularly; ideally on an annual basis, but not less
frequently that once every two years.
 Some regulatory functions may include many training activities that are not 03
incorporated in the institutional training programme. Such training normally is offered
by invitation. In this case, the assessor should recognize reports from non-routine 04
MA-relevant training not included in the NRA training plan.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will 05
always apply for all benchmarked NRAs).
06
Sub Indicator: MA03.04: The NRA generates and maintains records of staff training activities
and training effectiveness verification.
07
08
Maturity Level: 3
09
Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that records of staff training that is performed or organized by
the NRA are generated, maintained, regularly updated. This activity should be supported by
guidelines that direct the NRA to generate and maintain records of staff training activities.
Procedures should be in place to document and propose staff training needs and to
allocate a budget for continuous staff capacity building and development. The assessor
should check that there is an evaluation or assessment mechanism to verify the quality
of learning, and to confirm that learning objectives are achieved. Documentation should
include an inventory (i.e., soft and/or hard) system that records all impactful and non-
impactful trainings and identifies all staff members who participated. A system to measure
or estimate impact of trainings should be established.

Page 95 of 340
WHO Global Benchmarking Tool (GBT)

Requirement: Training records

Evidence to review: The assessor should ask for and review: 01

1. Guidelines or similar documents that guide the NRA to generate and maintain records of
staff training activities; 02
2. Evaluations of training effectiveness;

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

3. The training inventory, and procedures for completing the inventory;
4. Examples of archived records of staff training, and procedures for the archiving system.

Framework: Output

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence that the NRA generates and maintains
records of staff training activities.
 ONGOING IMPLEMENTATION (OI): The NRA has recently initiated plans to generate,
document and keep records of staff training activities, however they are not yet
followed.
 PARTIALLY IMPLEMENTED (PI): The NRA has recently initiated plans to generate,
document and keep records of staff training activities but they are not fully followed
for all training activities or they have been established for less than two years.
 IMPLEMENTED (I): The NRA generates and maintains records of staff training
activities.

Limitations and remarks:  The assessor should note that some NRAs out-source training including staff 03
capacity development activities. In this case the assessor should request the
identity of the provider, as well as evaluations of the provider. The assessor may 04
request the professional profiles of tutors or resource persons used to offer
training. The assessor may also request records covering the archiving systems 05
in place.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-
06
indicator will always apply for all benchmarked NRAs).
07
Indicator: MA04 Procedures established and implemented to perform registration
08
and/or marketing authorization.
09
Objective: The objective of this indicator is to ensure that procedures have been established and
implemented to effectively perform registration and MA activities. The NRA should have
documented procedures for registration and MA activities and these procedures should
be implemented. The documented procedures should be efficiently designed to assist in
the preparation of the registration and MA application dossier and in the documentation
of the receipt and evaluation of the dossier. The procedures should present an overview
of the activities to be carried out, the steps to be followed, the resources required, and the
processes to be implemented. These activities include, but are not limited to, receipt of
registration applications, acknowledgement of registration applications, assessment of
the various components of the applications, generation of recommendations following
the assessment process, issuance of a decision (i.e., approval, deferral, or rejection) for
registration applications, issuance of registration and MA numbers or certificates, and
publication and public sharing of Summary of Product Characteristics-like information

Page 96 of 340
WHO Global Benchmarking Tool (GBT)

and technical evaluation summary reports. The established procedures will give guidance
on how registration and MA applications should be handled to ensure efficiency and
effectiveness of the process.

Category: Regulatory process

Sub Indicator: MA04.01: Documented procedures and tools are implemented for the assessment
of the different parts of the application (i.e., quality, and efficacy) and for the
assessment of specific requirements applicable to specific classes of medical products. 01
Maturity Level: 3 02

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that documented procedures and tools are developed, made
available and implemented for the assessment of the different parts of the registration or
MA applications and for assessment of particular requirements applicable to a specific
class of medical products. The benefit-risk balance of the medical product under review
for registration or MA is the core consideration for regulatory authorities. The assessment
procedures should be designed to be adaptable to all applications regardless of the source
and product class. The procedures should provide guidance on the different parts of the MA
application processing flow, including receipt, screening, review, and decision-making. The
procedures for the assessment of registration or MA should be supported by adequate SOPs
or similar documents. The scope and extent of the screening and review processes should
be clearly documented for routine and non- routine MA applications. The assessor should
note that there may be other registration or MA routes in special cases, e.g., for prequalified
products or during emergencies when the normal procedures and tools may not apply.

Objective: The objective of this sub-indicator is to ensure that documented procedures and tools exist
for the assessment of the different parts of the registration or MA application and for the
assessment of particular requirements applicable to a specific class of medical products.
These procedures and tools should be implemented during the assessment process to
avoid discrimination or bias.

Requirement: Documented procedures and tool implemented 03

Evidence to review: The assessor should ask for and review: 04

1. Documented procedures and tools, including SOPs, that are developed and used in the
assessment of different parts of registration or MA applications or in the assessment of 05
particular requirements applicable to a specific class of medical products
2. SOPs used for different parts of MA applications, such as receipt, screening, review and
06
decision-making.
3. Evidence of use these SOPs, for example, by checking whether major milestones and
07
decision points are recorded for a selected recent MA.
08
4. Documents addressing the scope and extent of the assessment process with respect to
MA applications received and processed through the routine and non-routine routes.
09
5. Documents addressing the registration application backlog and that relate the backlog
to the timelines for the review process and to the number of competent officers involved in
the review process

References: 1. Regulation of vaccines: building on existing drug regulatory authorities. Geneva: World
Health Organization; 1999 (WHO/V&B/99.10), (6), (http://apps.who.int/iris/bitstream/
handle/10665/65968/WHO_V-B_99.10_eng.pdf?sequence=1)
2. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization: forty-fifth
report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical Report Series,
No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_TRS_858_A1.pdf)
3. Quality assurance of pharmaceuticals WHO guidelines, good practices, related regulatory
guidance and GXP training materials, (79), (http://digicollection.org/whoqapharm/)

Page 97 of 340
WHO Global Benchmarking Tool (GBT)

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): There are no documented procedures or tools for the
assessment of the different parts of the application (i.e., quality, safety & efficacy)
and for the assessment of particular requirements applicable to specific classes of
medical products.
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted or adopted the
documented procedures and tools for the assessment of MA but they has not yet
been followed. 01
 PARTIALLY IMPLEMENTED (PI): The NRA has developed or adopted documented
procedures and tools for the assessment of MA and has been applying it for less than 02
two years, or has not considered them for all marketed products.

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

 IMPLEMENTED (I): The NRA has documented procedures and tools for the
assessment of different parts of the application (i.e., quality, safety & efficacy) and for
the assessment of particular requirements applicable to specific classes of medical
products and has applied them for all reviewed and marketed medical products.

Limitations and remarks:  There may not be adequate time during the on-site assessment to review and
compare the scope, extent, and consistency of the assessment processes for two
or more MAs. This may be done off site though review of point 4 (scope and extent
of the assessment process) and will likely require expert assessors (e.g., dossier
reviewers) from stringent regulatory authority or pre-qualification assigned reviewers.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: MA04.02: Documented procedures have been implemented to renew
and/or to periodically review the MAs granted.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that documented procedures and tools to renew and /
or periodic review medical product registrations or MAs have been established and
implemented. Similar to the initial registration and MA, the benefit-risk balance of the 03
medical product is the core consideration for marketing re-authorization by regulatory
authorities. The procedures should be supported with guidance on the requirements to 04
be reviewed, and on the SOPs to be implemented at the various stages of the renewal
or periodic review process. Guidance should be available for each stage of the renewal 05
process, including, for example, receipt of application, screening, reviewing, and decision-
making. The process flow should be a documented and implemented.
06

Objective: The objective of this sub-indicator is to ensure that documented procedures and tools
07
exist and are implemented for the renewal and /or periodical review of registrations or MAs
08
previously granted. These procedures will ensure consistency and effectiveness of the
renewal process.
09
Requirement: Registration or MA renewal

Evidence to review: The assessor should ask for and review:

1. Document listing legally-approved validity periods for authorized products.
2. Documented procedures and tools, including SOPs and other technical documents
covering the stages of the renewal process, e.g., receiving, screening, reviewing, decision-
making, and issuing of registration numbers or certificates.
3. Procedures describing process flow with responsible divisions and timelines

References: 1. Quality assurance of pharmaceuticals WHO guidelines, good practices, related

regulatory guidance and GXP training materials, (79), (http://digicollection.org/
whoqapharm/)

Page 98 of 340
WHO Global Benchmarking Tool (GBT)

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence of documented procedures to renew
and/or to periodically review the MAs granted.
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted or adopted
documented procedures to renew and/or to periodically review the MAs granted, but
they have not yet been applied.
 PARTIALLY IMPLEMENTED (PI): The NRA has developed or adopted documented
procedures to renew and/or to periodically review the MAs granted and has been 01
applying them for less than five years or has not developed them yet for all marketed
medical products. 02
 IMPLEMENTED (I): The NRA has documented procedures to renew and/or

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

periodically review the MAs granted and has applied them for all marketed products.

Limitations and remarks: The assessor should note that not all NRAs apply the registration and/ or MA renewal
or periodic review system. Some NRA’s maintain a stringent and functional vigilance
and market surveillance system or structure with a demonstrated ability to perform its
functions. In this case, the assessor should score this sub indicator as not applicable.

Sub Indicator: MA04.03: Documented procedures are implemented for assessing applications
for variations of MAs.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that documented procedures and SOPs that will guide the
assessment of the various classes and types of variations for the various product classes
are established and implemented. These documents should contain procedures for
receiving and processing the various classes and types of variation applications.

Objective: The objective of this sub-indicator is to ensure that documented procedures to assess
variations to registrations or MAs exist and are implemented.

Requirement: Documented and implemented variations assessment procedures 03

Evidence to review: The assessor should ask for and review. 04

1. Procedures and SOPs that guide the assessment of the various classes of variations.
The documented procedures should be medical product class specific. 05
2. Procedures for receiving and processing variation applications, and evidence that the
prescribed procedures been implemented.
06
3. A sample application that was received and reviewed by the NRA. The assessor should
review for compliance with documented procedures.
07
08
References: 1. Quality assurance of pharmaceuticals WHO guidelines, good practices, related
regulatory guidance and GXP training materials, (79), (http://digicollection.org/
09
whoqapharm/)

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence of documented procedures for
assessing the applications for variations of MAs.
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted or adopted
documented procedures for assessing the applications for variations of Mas, but they
have not yet been followed.
 PARTIALLY IMPLEMENTED (PI): The NRA has developed or adopted documented
procedures for assessing the applications for variations of MAs and has been
applying them for less than two years or all submitted variations were not reviewed
based on these procedures.

Page 99 of 340
WHO Global Benchmarking Tool (GBT)

 IMPLEMENTED (I): The NRA has documented procedures for assessing the
applications for variations of MAs and has implemented them for all marketed
medical products.

Limitations and remarks:  This review may also require an expert dossier reviewer. The assessor should note
that NRA may have different requirements and guidelines for different medical
product streams.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 01
Sub Indicator: MA04.04: The same criteria apply for assessing applications regardless of the origin 02
of or destination for the medical products (e.g., domestic, foreign, public sector, or

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

private sector).

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that the criteria applied to assess registration or MA
applications are the same regardless of the origin or source of the medical product and the
product application, and regardless of the location where the product will be used. A set of
criteria for assessment should be documented and implemented for the various classes
of medical product applications. The criteria should be documented in the form of SOPs.
Guidance in the form of SOPs should be documented and implemented.

Objective: The objective of this sub-indicator is to ensure the criteria used for assessing registration
or MA applications for medical products is the same, regardless of the source or origin of
the product or the destination for use of the product.

Requirement: Same criteria for assessing registration or MA applications regardless of applicant

Evidence to review: The assessor should ask for and review:

1. Evidence that SOPs or similar documents are in place to ensure that the same set of
criteria is used to assess registration or MA applications regardless of source, origin or
destination of the medical product. 03
2. Evidence that the staff members are aware of the single set of criteria and know
how to apply the criteria in their assessment activities. 3. Evidence that the criteria are 04
documented, published and implemented.
4. Sample registration or MA applications from different sources. The assessor 05
should review these for the purpose of verifying consistent compliance with specified
procedures. The assessor also should compare the procedures and determine whether
06
there are any discrepancies in terms of scientific review and assessment. Sample
MA applications of medical products coming from different sources (e.g. domestic,
07
imported, public, or private,) along with their assessment reports, should be checked
08
by the assessor.
09
References: 1. Multisource (generic) pharmaceutical products: guidelines on registration requirements
to establish interchangeability, (12), (http://digicollection.org/whoqapharm/p/about and
http://apps.who.int/medicinedocs/en)
2. WHO good manufacturing practices for pharmaceutical products: main principles,
(51), (http://digicollection.org/whoqapharm/p/about and http://apps.who.int/
medicinedocs/en)
3. Effective drug regulation: A multicounty study. Geneva: World Health Organization;
2002, (7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): The NRA does not apply the same criteria for assessing all
applications (e.g., domestic, foreign, public sector, or private sector)

Page 100 of 340

WHO Global Benchmarking Tool (GBT)

 ONGOING IMPLEMENTATION (OI): The NRA has recently defined or drafted the
same criteria or policy for assessing all applications but they have not yet been
applied.
 PARTIALLY IMPLEMENTED (PI): The NRA has defined the same criteria or
policy for assessing all applications and has been applying them for assessing
applications for less than two years or has not applied them for all currently
market medical products.
 IMPLEMENTED (I): The NRA has the same criteria or policy for assessing
applications regardless of the origin or destination of the medical products (e.g. 01
domestic, foreign, public sector, or private sector) and has applied them for all
marketed medical products. 02

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

Limitations and remarks: There may not be adequate time during the on-site assessment to review and compare
the scope, extent, and consistency of the assessment processes for two or more MAs.
This may be done off site though review of the evidence gathered for this sub-indicator.
The assessor should consider that in non-producing countries, there would be no
records for a domestic product that could be used when evaluating the implementation
of this requirement.
Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: MA04.05: An advisory or scientific committee, including external experts is involved
in the review of MA applications (as needed).

Maturity Level: 4

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that the use of advisory or scientific committees, including
external experts, in the review of registration or MA applications is documented and
supported by adequate regulations. The objectives and functions of the committee, as
well as the professional profile of the committee members, should be available. The
composition of the committee should be guided by appropriate regulations. For each
member of the committee, there should be clearly defined roles and responsibilities in the
review of registrations or MAs. Terms of reference for each member of the committee and 03
each external expert should be prepared and documented.
04
Objective: The objective of this sub-indicator is to ensure that an advisory or scientific committee,
including external experts, is involved in the review of the registration or MA applications. 05

Requirement: Advisory or scientific committee, including external experts

Evidence to review: The assessor should ask for and review:

07
1. Documents defining the composition of the advisory or scientific committee, and
08
evidence that the composition is in line with the directives prescribed in the regulations.
2. Documents defining the objectives and functions of the committee and its individual
09
members, as well as that of the external experts.
3. The terms of reference for committee and its members as well as that of the external
experts.
4. Documents defining the scope and extent of the advisory or scientific contributions to
the registration or MA activities of the NRA
5. SOPs defining the review activities of the advisory or scientific committee and its
programme of work.
6. Meeting minutes and related documents that provide evidence for the activities that
result in registration or granting MA.

References:

Framework: Process

Page 101 of 340

WHO Global Benchmarking Tool (GBT)

Rating Scale:  NOT IMPLEMENTED (NI): The NRA does not have access to an advisory or
scientific committee, including external experts, for assisting in review of MA
applications or providing scientific advice as necessary..
 ONGOING IMPLEMENTATION (OI): The NRA has recently defined the criteria and
composition of the advisory or scientific committee, but the advisory or scientific
committee has not yet been established.
 PARTIALLY IMPLEMENTED (PI): The NRA has established the advisory or
scientific committee, but less than two years ago.
 IMPLEMENTED (I): The NRA has access to an advisory or scientific committee, 01
including external experts, in the review of MA applications; this is supported by
adequate documentation and records 02

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

Limitations and remarks: The assessor should note that some NRAs may not involve an advisory or scientific
committee, including external experts, in the review of the submitted registration or MA
applications, due to the availability of internal experts with the requisite competencies.
Some NRAs may refer to the committee or external experts only certain applications
that require their specialist attention and expert advice. In this case the scoring is not
applicable.

Sub Indicator: MA04.06: Timelines for the assessment of the applications are defined and an
internal tracking system has been established to monitor adherence to the targeted
time frames.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that documented timelines and internal timeline
tracking systems that guide the registration or MA application process exist and
are implemented. These should be supported by guidelines that stipulate that MA
applications should be processed and assessed per a published timeline and that the
published timeline should be monitored internally for compliance. The guidelines should
be supported with SOPs that provide guidance on how to establish the timelines and
how to monitor the established timelines. The guidelines should be designed to adapt to
the various classes of medical products registration or MA applications (e.g., innovative 03
medicines, generic medicines, new molecular entities, in vitro devices, and vaccines).
Timelines should be designed to address each stage of the registration or MA application 04
process flow. The system should be developed to monitor the various timelines for the
various registration or MA application assessment procedures (i.e., the routine and non- 05
routine procedures). The assessor should note that some NRAs will have varied timelines
for the different registration or MA assessment routes, as well as varied internal tracking
06
systems to ensure adherence to the timelines. In this case, the assessor should review
the timelines for each route and their corresponding tracking systems.
07
08
Objective: The objective of this sub-indicator is to ensure that timelines for the assessment of
registration or MA applications exist, and that the timelines are monitored by internal
09
tracking systems to ensure compliance.

Requirement: Timelines and internal control mechanisms

Evidence to review: The assessor to ask for and review:

1. The documented registration application process flow and corresponding timelines
for the various registration or MA application routes and product classes.
2. Documentation for the internal tracking systems used for monitoring adherence to
the assessment timelines.
3. Evidence that the timelines are published, easily accessible and known to all
stakeholders, including the applicants and reviewers.
4. Evidence that these timelines have been implemented and are being adhered to
during the assessment process.

Page 102 of 340

WHO Global Benchmarking Tool (GBT)

5. Evidence that a dedicated unit or office is responsible for instituting, implementing

and monitoring the assessment timelines tracking system for the purposes for
compliance.
6. Evidence that corrective measures are available in the event of non-compliance.

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

Rating Scale:  NOT IMPLEMENTED (NI): There are no timelines for the assessment of the
applications and no internal tracking system to monitor the targeted time frames
 ONGOING IMPLEMENTATION (OI): The NRA has recently defined timelines for the
assessment of the applications but an internal tracking system to monitor the
targeted time frames has not been implemented.
 PARTIALLY IMPLEMENTED (PI): The NRA has defined the timelines for the
assessment of the applications and it has been applying an internal tracking system
to monitor the targeted time frames for less than two years.
 IMPLEMENTED (I): The NRA has timelines for the assessment of the applications
and an internal tracking system to monitor the targeted time frames has been
implemented

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: MA04.07: There are documented mechanisms to handle non-routine registration
or MA requirements in special situations (e.g., public-health interest).

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines
03
Description: The assessor should verify the existence and implementation of regulations and guidelines
allowing the use of a non-routine mechanism that is supported with specific registration 04
requirements. The regulation also should be supported with guidelines and SOPs that
clearly specify the registration requirements and the instances (e.g., products for use in 05
public health emergencies, or orphan medicinal products) in which an application would
be considered for a specific pathway in the interest of public health. In addition, the
06
registration or MA application process flow and their corresponding specific timelines
should be provided, published and implemented. SOPs for the assessment process should
07
be developed to reflect the process flow and the specific requirements.
08
Objective: The objective of this sub-indicator is to ensure that documented mechanisms for specific
09
registration requirements exist and are implemented when necessary in special situations
(e.g., public-health interest).

Requirement: Non-routine registration or MA applications

Evidence to review: The assessor should ask for and review:

1. Regulations related to the establishment and implementation of a mechanism, with
specific registration requirements, that can be used, in special situations, for assessing MA
applications and for granting MA.
2. Guidelines on the instances and special situations in which this system applies,
and guidelines on the specific registration requirements that should be applied. These
requirements should be based on the nature of the products (e.g., new, biological, or
generic products).

Page 103 of 340

WHO Global Benchmarking Tool (GBT)

3. An example of an application that was processed through this route. The documents
should be reviewed to determine if content and format is consistent with the prescribed
requirements and if the process adhered to published timelines.
4. Documents describing the registration or MA application process flow and the
corresponding timelines. The timelines should demonstrate timely review relative to the
normal route or process.
5. SOPs that provide guidance for the assessment process.

References: 1. Regulation of vaccines: building on existing drug regulatory authorities. Geneva: World 01
Health Organization; 1999 (WHO/V&B/99.10), (6), (http://apps.who.int/iris/bitstream/
handle/10665/65968/WHO_V-B_99.10_eng.pdf?sequence=1). 02
2. Regulation and licensing of biological products in countries with newly developing

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

regulatory authorities. In: WHO Expert Committee on Biological Standardization: forty-fifth
report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical Report Series,
No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_TRS_858_A1.pdf)
3. Guidelines on evaluation of similar biotherapeutic products (SBPs), (70), (http://apps.
who.int/medicinedocs/en/q/ )

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): There is no documented mechanism, with specific
registration requirements, for special situations (e.g., public-health interest)
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted or adopted a
documented mechanism, with specific registration requirements, for special
situations (e.g. public-health interest) but it has not yet been applied.
 PARTIALLY IMPLEMENTED (PI): The NRA has developed or adopted a documented
mechanism, with specific registration requirements, for special situations (e.g. public-
health interest), and has been applying this for less than two years or in fewer than
three cases.
 IMPLEMENTED (I): The NRA has a documented and implemented non-routine
mechanism, with specific registration requirements, for special situations (e.g. public-
health interest)

Limitations and remarks: In case that there has been no emergency that required the application of this mechanism,
the assessor should verify the implementation of this sub-indicator by considering the
availability of adequate resources and procedures for implementation of the non-routine 03
mechanism.
04
Sub Indicator: MA04.08: SPC-like, labelling and packaging information are approved
by the NRA as part of the MA procedure. 05

Maturity Level: 3
06

Scope: 1. Medicines
07
2. Vaccines
08
Description: The assessor should verify the existence of regulations or guidelines that state that product
09
information (i.e., SPC-like, packaging and labelling information) should comply with the
regulations that cover product information for medical products. The information should
be assessed as part of the registration or MA application package assessment process.
During this review process, the product information must be found to be appropriate and
satisfactory. The regulation should be supported by specific guidelines or SOPs that guide
the assessment of product information (i.e., SPC-like information, packaging information,
and product labels). The product information should be updated periodically.

Objective: The objective of this sub-indicator is to ensure that product information, including SPC-
like, packing and labelling information, are approved by the NRA as part of the registration
or MA application process. The NRA should have clear requirements regarding medical
products information that should be provided in the SPC-like, packaging and labelling
information.

Page 104 of 340

WHO Global Benchmarking Tool (GBT)

Requirement: SPC-like, packaging, and labelling information

Evidence to review: The assessor should ask for and review:

1. Regulations or guidelines stating that product information (i.e., SPC-like, packaging,
and labelling information) should comply with the regulations provided to cover product
information on medical products.
2. Guidelines or SOPs for assessing product information on SPC-like information,
packaging, and product labels.
3. Medical products registration or MA application assessment reports. 01
4. Documents describing the registration or MA application processing flow for the stage
or phase of the process in which review of the product information (i.e., SPC-like, packaging 02
and labelling information) is conducted. .

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

References: 1. Guidelines on packaging for pharmaceutical products, World Health Organization (WHO),
(80), (http://apps.who.int/medicinedocs/en/q/ and http://digicollection.org/whoqapharm/)
2. 13-Ethical criteria for medicinal drug promotion. WHO, 1998, (13), (http://apps.who.int/
medicinedocs/documents/whozip08e/whozip08e.pdf)

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): There are no requirements for approving SPC-like, labelling
and packaging information by the NRA as part of the MA procedure.
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted or adopted the
requirements to comply with the regulations provided to cover product information
on medical products, such as SPC-like, labelling and packaging information, but they
have not yet been followed.
 PARTIALLY IMPLEMENTED (PI): The NRA has defined the requirements to comply
with the regulations provided to cover product information on medical products,
such as SPC-like, labelling and packaging information, and has been applying them
for less than two year or they have not been considered for all marketed medical
products.
 IMPLEMENTED (I): The NRA is approving SPC-like, labelling and packaging
information as part of the MA procedure for all reviewed medical products.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 03

Sub Indicator: MA04.09: GMP inspection report and/or certification is considered as part 04
of the MA process.
05
Maturity Level: 3
06
Scope: 1. Medicines
2. Vaccines
07
08
Description: The assessor should verify that regulations or guidelines are established and implemented
that specify the type, content and format of GMP documentation that should be submitted
09
to the NRA by another NRA (i.e., in case of imported medical products), or that describes
the processes and procedures through which an on-site inspection or verification will be
conducted by the NRA. The guidelines should be supported by supplementary guidelines
or SOPs that provide guidance for use during the on-site inspection or verification visits
or during assessment of the submitted GMP documentation. The assessor should
verify the existence and implementation of procedures and SOPs that ensure that the
GMP inspections or GMP documentation verifications are part of the medical products
registration or MA process and that ensure that GMP requirements are established,
implemented, published, and easily accessible as part of the registration or MA application
process. The assessor should note that, in some NRAs, the GMP on-site visits and
inspections may be carried out by consultants hired by the NRA to perform the task rather
that by NRA staff. In such instances, if the competencies of the hired consultants are
appropriate, the scoring will apply.

Page 105 of 340

WHO Global Benchmarking Tool (GBT)

Objective: The objective of this sub-indicator is to ensure that GMP inspections or certifications
are part of the registration or MA requirements, and that the NRA has a legal provision
to request the manufacturer to demonstrate adherence to GMP in the manufacture
of the medical products. This can be done either through on-site verification or inspection
by the NRA or through a satisfactory assessment of GMP documentation issued by
other NRAs.

Requirement: GMP inspection report or certificate as part of MA process

01
Evidence to review: The assessor should ask for and review:
1. Legal provisions that clearly state that compliance with the principles of GMP is part of 02
the medical products registration or MA application requirements.

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

2. Legal provisions that clearly describe the format and content of the GMP requirements,
i.e., either through a satisfactory on-site verification or inspection by the NRA or through a
satisfactory assessment of GMP documentation issued by another NRA.
3. Guidelines and SOPs that provide guidance to be used during the on-site inspection
or verification visits or during assessment of the submitted GMP documentation. These
should be reviewed for completeness.
4. Guidelines or regulations that specify the type, content and format of GMP
documentation that should be submitted to the NRA by another NRA (i.e., for the purpose
of GMP verification), or that describes the processes and procedure through which an on-
site inspection or verification will be conducted by the NRA.
5. Procedures that ensure that the GMP inspections or the verifications of submitted GMP
documentation are part of the authorization process, as well as procedures that ensure
that GMP requirements are implemented as part of the MA application process.
6. Evidence that a GMP report or certification is a requirement for all registration or MA
applications and that a risk-based routine inspection is required periodically.
7. A record of registration and MA application submissions. The should be evaluated for
the purposes of verification and compliance with the guidelines.

References: 1. Guidance on Good Manufacturing Practices (GMP): inspection report, (32), (http://
digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en)
2. WHO good manufacturing practices for pharmaceutical products: main principles, (51),
(http://digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en)

Framework: Structure/Foundation/Input & Process 03

Rating Scale:  NOT IMPLEMENTED (NI): A GMP inspection report or certification is not required as 04
part of the MA process
 ONGOING IMPLEMENTATION (OI): The NRA has recently defined GMP inspection 05
report or certification as a requirement for MA authorization, but it has not yet been
followed.
06
 PARTIALLY IMPLEMENTED (PI): The NRA has required a GMP inspection report or
certification as a requirement for MA authorization and has been applying this for less
07
than two years.
08
 IMPLEMENTED (I): GMP inspection report or certification is required by the NRA as
part of the MA process, and has been considered in all received MA applications.
09
Limitations and remarks:  The assessor should verify that all requirements are same regardless of the source of
products.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: MA04.10: The regulations and guidelines for good review practices (GRevPs) are
developed or recognized and implemented.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Page 106 of 340

WHO Global Benchmarking Tool (GBT)

Description: The assessor should verify that procedures and documentation on GRevPs are established
and implemented. The procedures should be supported by an operation manual (i.e., with
documented and approved SOPs for the screening and review of the various classes of
medical products and their corresponding routine or non-routine application routes) that
guides the screening and review of registration or MA applications. The procedures should
include quality control and quality assurance systems or structures that ensure that actual
application assessment and review is conducted in accordance with approved SOPs. The
procedures should provide guidance on how to handle medical product application non-
conformities. 01
Objective: The objective of this indicator is to ensure that regulations and guidelines exist for the 02
NRA to review and evaluate the different components of registration or MA applications;

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

to review and evaluate the specific requirements of each product class; and to register
or grant MAs or to reject registrations following documented procedures for the different
parts of the application. These procedures will provide consistency in the approval or
rejection of applications through the application of defined requirements and criteria
such that the resultant decision would be seen to be based on submitted documentation
irrespective of the reviewer.

Requirement: Guidelines and SOP for GRevPs

Evidence to review: The assessor should ask for and review:

1. Regulations and guidelines that provide guidance on principles of GRevP.
2. SOPs for assessment of registration or MA applications that guide the reviewer during
the screening and review of the different components of the various classes of medical
products.
3. Evidence that the same set of principles is applied during the screening and review
of MA applications submitted through the non-routine procedure. The assessor should
request SOPs that provide guidance on how to handle actual review of applications
through the non-routine procedure.
4. Evidence that procedures are in place to ensure that actual application assessments
and reviews are conducted in accordance with the principles of GRevP and that the
assessment reports are peer-reviewed.
5. An example of an assessment report for a particular product class. The assessor
should evaluate this for compliance to the corresponding GRevP SOPs and related
documents. 03

References: 1. Good review practices: guidelines for national and regional regulatory authorities, (49), 04
(http://digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en)
05
Framework: Process
06
Rating Scale:  NOT IMPLEMENTED (NI): There are no regulations or guidelines for GRevPs
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted or adopted GRevPs
07
but they have not yet been followed.
08
 PARTIALLY IMPLEMENTED (PI): The NRA has developed or adopted GRevPs and has
been applying them for less than two years or the principles are not yet reflected in
09
other relevant SOPs and guideline for MA activities.
 IMPLEMENTED (I): The NRA has regulations or guidelines for GRevPs and is
following them in all relevant activities.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Indicator: MA05 Mechanism exists to promote transparency, accountability

and communication.

Objective: The objective of this indicator is to ensure that mechanisms are in place to ensure that
information on registration and MA applications, including authorized, suspended, rejected
and completed applications, are published to promote transparency and information

Page 107 of 340

WHO Global Benchmarking Tool (GBT)

sharing among stakeholders and potential stakeholder (e.g., retailers and patients). This
approach will build trust among stakeholders. A mechanism for appeal of MA should be
described.

Category: Transparency, accountability and communication

Sub Indicator: MA05.01: Web site or other official publication with SPC-like information
is available and regularly updated.
01
Maturity Level: 3
02
Scope: 1. Medicines

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

2. Vaccines

Description: The assessor should verify that a website or other official publication where SPC-like
information is published is in existence and regularly updated. Guidelines and SOPs should
be documented and implemented to provide guidance on preparation, content and format
of the SPC-like information that is to be published, as well as guidance on the procedures
and processes to be used for the regular updates (i.e., frequency and processes). The
website to host the information should be clearly documented.

Objective: The objective of this sub-indicator is to ensure the NRA has the legal provisions or
regulations to generate and publish SPC-like information, and to ensure that the practice is
implemented.

Requirement: SPC-like information available

Evidence to review: The assessor should ask for and review:

1. Evidence that the SPC-like information document is published, and if published, that the
publication is in the appropriate format per the published guidelines.
2. Evidence that the website is regularly updated to reflect the current situation. The
assessor should review the SOP for updating the website.
3. Evidence that the publications are easily accessible, regardless of where the information
is hosted (i.e., NRA website, Ministry of Health (MoH) website, or another website). If on
another website, the assessor should request SOPs that provide guidance on how to
remotely upload new information onto the website and to update existing information. 03

References: 1. National drug regulatory legislation: guiding principles for small drug regulatory 04
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations:
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical 05
Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf)
06

Framework: Output
07
08
Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence of official SPC-like information
publication on website or in another place.
09
 ONGOING IMPLEMENTATION (OI): The NRA has recently defined official SPC-like
information publication, but it has not yet been published.
 PARTIALLY IMPLEMENTED (PI): The NRA has developed and initiated official SPC-like
information publication but the information is not updated regularly or all information
is not yet reflected.
 IMPLEMENTED (I): The NRA has official SPC-like information publication on website
or in another place for all marketed medical products.

Limitations and remarks:  The assessor should note that in some instances, the NRA may not physically host
the website. The website may be hosted by the MoH, a professional body, a standards
authority, or another entity. In all instances the NRA has the responsibility to ensure
that SPC-like information is published, and that the website is regularly updated. The
scoring is applicable.

Page 108 of 340

WHO Global Benchmarking Tool (GBT)

 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: MA05.02: Updated list of all medical products granted MA is regularly published
and publicly available.

Maturity Level: 3

Scope: 1. Medicines 01
2. Vaccines
02
Description: The assessor should verify that a list of medical products registered or granted MA

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

is in existence. This list should be regularly updated and published. The published
information should be supported by SOPs that provide guidance on the procedure
to enter information, the procedure to translate the information into the appropriate
format, and the type of information to enter (e.g., may include, but is not limited to, the
product brand name, International Nonproprietary Name, the registration or MA number,
the registration or MA certificate number, the validity period of the registration or MA,
the active ingredients, manufacturer, and potency or concentration). The host or website
where the list is published should be documented and should be easily accessible to the
general public.

Objective: The objective of this sub-indicator is to ensure that guidelines and regulations exist to
guide and advise the NRA to regularly update and publish the list of medical products that
are registered or granted MA

Requirement: List of all medical products registered or granted MA

Evidence to review: The assessor should ask for and review:

1. The published list of medical products registered or granted MA.
2. Evidence that the information published for the registered medical products is in
compliance with guidelines and SOPs.
3. Evidence that the list is regularly updated. The assessor should verify that the list is
current.
4. SOPs or similar documents for updating the list.
5. Evidence that an internal register (i.e., paper or electronic version) for entering registered 03
medical products exists. If the internal register exists, the assessor should review the
guidelines and SOPs for completing and regularly updating the register. 04
6. Evidence that the published list is accessible. The assessor should also identify the host
organization, e.g., NRA, MoH, or other organization. 05

References: 1. National drug regulatory legislation: guiding principles for small drug regulatory
06
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations:
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical
07
Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/documents/s21964en/
08
s21964en.pdf)
09
Framework: Output

Rating Scale:  NOT IMPLEMENTED (NI): The updated list of all medical products granted MA is not
regularly published or publicly available.
 ONGOING IMPLEMENTATION (OI): The NRA has recently committed to publish the list
of all medical products granted MA, but it has not yet been followed.
 PARTIALLY IMPLEMENTED (PI): The NRA has initiated publication of the list of all
medical products granted MA; however it is not updated regularly, or the list does not
reflect of all medical products granted MA.
 IMPLEMENTED (I): The NRA has publicly published the list of all medical products
granted MA and it is regularly updated.

Limitations and remarks:  For different products medical streams, some NRAs publish different lists. If so, each
list should be easy to access.

Page 109 of 340

WHO Global Benchmarking Tool (GBT)

The assessor should consider that existence of the database of all granted MA
medical products is not adequate to fulfill this requirement (unless the database is
publicly available), since the purpose is transparency.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: MA05.03: A summary technical evaluation report for approved registration
MA applications is published and available to the public.
01
Maturity Level: 4
02
Scope: 1. Medicines

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

2. Vaccines

Description: The assessor should verify that the summary technical evaluation reports of those medical
products registration applications that have been approved, are published and made
available to the public, and these actions are supported by a regulation or a guideline.
The content and format of the report should be guided by an SOP to ensure uniformity.
Procedures should be in place to monitor the generation of the reports to ensure that the
content and format conforms to the approved standards, and that the reports are regularly
published and made available to the public.

Objective: The objective of this sub-indicator is to ensure that guidelines or regulations exist to allow
the NRA to publish and make available to the public the summary technical evaluation
reports of approved MA applications. The NRA should actually publish and make available
to the public the summary technical evaluation reports.

Requirement: Summary technical evaluation report (Public assessment report) available to stakeholders

Evidence to review: The assessor should ask for and review:

1. Procedures in place to support and monitor the generation of the reports, to ensure
that the content and format conforms to the approved standards, and to ensure that the
reports are regularly published and made available to the public.
2. A sample of the report for the purpose of evaluating conformance to the published
guidelines and or SOPs. The assessor should verify that the report is generated, published
and made available to the public. 03
3. The website for hosting the reports, and the SOPs that provide guidance to staff on
procedures for uploading new reports and updating existing reports. 04

References: 1. Regulation of vaccines: building on existing drug regulatory authorities. Geneva: World 05
Health Organization; 1999 (WHO/V&B/99.10), (6), (http://apps.who.int/iris/bitstream/
handle/10665/65968/WHO_V-B_99.10_eng.pdf?sequence=1).
06
2. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization:
07
forty-fifth report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical
08
Report Series, No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_
TRS_858_A1.pdf)
09
Framework: Output

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence for publication and public availability
of a summary technical evaluation report for approved registration or MA
applications.
 ONGOING IMPLEMENTATION (OI): The NRA has recently defined this requirement, but
it has not yet been implemented.
 PARTIALLY IMPLEMENTED (PI): The NRA has initiated publication of summary
technical evaluation reports for approved registration or MA applications, but this has
not been applied for all evaluated medical product applications.
 IMPLEMENTED (I): The NRA has published and made available to the public
summary technical evaluation reports for approved registration or MA applications.

Page 110 of 340

WHO Global Benchmarking Tool (GBT)

Limitations and remarks: The assessor should note that some NRAs will not publish the summary of the technical
evaluation report for medical product applications processed under special circumstances,
for example, emergencies, WHO collaborative registration procedures, WHO prequalified
products, Article 58 reviews, or reviews by orphan drug route (i.e., abridged registration
application processing route or pathway).

Sub Indicator: MA05.04: A summary technical evaluation report for deferred or rejected registration
or MA applications is published and available to the public.
01
Maturity Level: 4
02
Scope: 1. Medicines

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

2. Vaccines

Description: The assessor should verify that the summary technical evaluation reports of those medical
products registration applications that have been deferred or rejected is published and
made available to the public, and these actions are supported by a regulation or guideline.
The content and format of the report should be guided by an SOP to ensure uniformity.
Procedures should be in place to monitor the generation of the reports to ensure that the
content and format conforms to the approved standards, and that the reports are regularly
published and made available to the public.

Objective: The objective of this sub-indicator is to ensure that guidelines or regulations exist to allow
the NRA to publish and make available to the public the summary technical evaluation
reports of deferred or rejected MA applications. The NRA should actually publish and make
the summary technical evaluation reports available to the public.

Requirement: Summary technical evaluation report (Public assessment report) available to stakeholders

Evidence to review: The assessor should ask for and review:

1. Procedures in place to support and monitor the generation of the reports, to ensure
that the content and format conforms to the approved standards, and to ensure that the
reports are regularly published and made available to the public.
2. A sample of the report for the purposes of evaluating conformance to the published
guidelines and or SOPs. The assessor should verify that the report is generated, published
and made available to the public. 03
3. The website for hosting the reports, and the SOPs that provide guidance to staff on
procedures for uploading new reports and updating existing reports. 04

References: 1. Regulation of vaccines: building on existing drug regulatory authorities. Geneva: World 05
Health Organization; 1999 (WHO/V&B/99.10), (6), (http://apps.who.int/iris/bitstream/
handle/10665/65968/WHO_V-B_99.10_eng.pdf?sequence=1).
06
2. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization: forty-fifth
07
report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical Report Series,
08
No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_TRS_858_A1.pdf)
09
Framework: Output

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence for publication and public availability
of a summary technical evaluation report for deferred or rejected registration or MA
applications.
 ONGOING IMPLEMENTATION (OI): The NRA has recently defined this requirement, but
it has not yet been implemented.
 PARTIALLY IMPLEMENTED (PI): The NRA has initiated publication of summary
technical evaluation reports for deferred or rejected registration or MA applications,
but this has not been applied for all evaluated medical products applications.
 IMPLEMENTED (I): The NRA has published and made available to the public
summary technical evaluation reports for deferred or rejected registration or MA
applications.

Page 111 of 340

WHO Global Benchmarking Tool (GBT)

Limitations and remarks: Some NRAs may not publish the report of registrations applications that were rejected
or deferred based on the NRA’s legal provisions. In either case, the scoring will not apply.
However, it is recommended that summary technical evaluation reports for all medical
product applications regardless of the evaluation outcome should be published and made
available to the public. In this case, the scoring should be yes.

Indicator: MA06 Mechanism in place to monitor regulatory performance and output.

01
Objective: The objective of this indicator is to ensure that mechanisms and procedures are in place
to monitor and ensure that all activities within the MA functionality are subjected to quality 02
control and other checks to reduce errors and to ensure that the processes and results

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

of the registration application assessment are consistent and able to generate a reliable
output. These monitoring mechanisms lead to consistency in the regulatory performance
of the registration and MA function and to reliability of outputs.

Category: Monitoring progress and assessing outcomes & impact

Sub Indicator: MA06.01: There is a database of all product applications received, approved, rejected,
suspended or withdrawn along with their supporting documentation.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that a database to keep all medical product registration
applications received, approved, rejected, suspended or withdrawn, as well as their
supporting documentation exists, and that the database is continually updated to
reflect the current situation. The operation and management of the database should
be supported by guidelines and SOPs that specify the types of information and
documentation to keep and the format and duration of storage. Personnel with
authorization to access information should be documented and the procedure to
access the information should be defined. The location of the actual database should be
documented. Information on the back-up database should be provided. Plans or efforts to
secure a back-up system should be reviewed if a back-up system is not currently available. 03

Objective: The objective of this sub-indicator is to ensure that regulations or guidelines exist to 04
require the NRA to keep and manage a database of the supporting documentation for all
medical products registration or MA applications received, approved, rejected, suspended 05
or withdrawn. This database should continually be kept up to date. This database would
significantly contribute to consistency in regulatory approach and to expansion of
06
institutional memory.
07
Requirement: Database of all medical products application
08
Evidence to review: The assessor should ask for and review:
09
1. Evidence that a database exists for keeping an up to date list of supporting
documentation and information (e.g., legal information and requirements,
communications, application files, and bases of decision) for all medical products
registration or MA applications received, approved, rejected, suspended or withdrawn.
2. Guidelines and SOPs that define the essential information to keep in the database and
the timelines for retaining this information.
3. Evidence that the database is up to date and current and that it is supported by a back-
up data base.
4. Documents defining the authorization needed to access or to update the information in
the database. Procedures for monitoring, maintaining, and updating the database should
also be reviewed.

References:

Page 112 of 340

WHO Global Benchmarking Tool (GBT)

Framework: Output

Rating Scale:  NOT IMPLEMENTED (NI): There is no database of all product applications
received, approved, rejected, suspended or withdrawn, along with their supporting
documentation.
 ONGOING IMPLEMENTATION (OI): The NRA has recently initiated and drafted the
requirement to create the database but it has not yet been established.
 PARTIALLY IMPLEMENTED (PI): The NRA has established a database of all product
applications received, approved, rejected, suspended or withdrawn along with their 01
supporting documentation but not all applications have been included in the database
or the database is not regularly updated or fully integrated. 02
 IMPLEMENTED (I): The NRA has a database of all product applications received,

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

approved, rejected, suspended or withdrawn along with their supporting
documentation, and the database is kept up-to-date.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: MA06.02: Performance indicators for registration and MA activities are established
and implemented.

Maturity Level: 4

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the existence and implementation of performance indicators
for different activities included under the registration and MA functions.
Specifically, the system should define key performance indicators (KPIs) along the entire
registration and MA activity chain and all indicators should be adequately justified. For the
purpose of clarity and consistency, established KPIs should be supported with guidelines
for monitoring and maintenance of the KPIs. The guidelines in turn should be supported
by SOPs and tools that define the procedures to be used for monitoring and evaluating
theperformance indicators and that define procedures and timelines for reviewing and
revising the indicators.
Established KPIs might be qualitative, quantitative or combination of both. In general, 03
quantitative indicators are preferred to avoid bias or misinterpretation. However, qualitative
indicators are also accepted. Qualitative indicators may or may not include scoring 04
or scaling to render them semi-quantitative and thus more informative. The assessor
should ensure that indicators are measured on a regular basis to monitor progress and 05
advancement.
In addition, the assessor should verify measured indicators are analyzed to identify trends
06
or abnormalities. Justifications for any identified abnormalities should be provided; when
necessary, process optimizations should be introduced to avoid recurrence.
07
08
Objective: The objective of this sub-indicator is to ensure that a system, mechanism, or procedure
exists to require the NRA to establish performance indicators along the entire registration
09
and MA chain. Additionally, the objective is to ensure that KPIs are actually contributing to
monitoring of regulatory performance, to measuring effectiveness of registration and MA
regulatory activities, and to making any necessary adjustments or optimizations.

Requirement: KPIs for registration and MA activities

Evidence to review: The assessor should ask for and review:

1. Documents supporting the system, mechanism, or procedure compelling the NRA to
establish and implement performance indicators along the entire registration and MA
activity chain.
2. Evidence that the performance indicators have been established and implemented,
and that the members of staff involved in the registration and MA function are aware
of the indicators and the guidelines and SOPs used for monitoring and evaluating their
performance.

Page 113 of 340

WHO Global Benchmarking Tool (GBT)

3. The current performance indicators for registration and MA activities

4. Analyses of the measured indicators along with the investigations done to identify
trends or abnormalities.
5. Documentation for follow-up of any observed abnormalities, including justifications
for any identified abnormalities as well as any process optimizations introduced to avoid
recurrence.

References: 1. Quality management -- Quality of an organization -- Guidance to achieve sustained

success, ISO 9004:2018, International Organization for Standardization (ISO), (115), 01
(https://www.iso.org/standard/70397.html)
02
Framework: Structure/Foundation/Input

Registration and Marketing Authorization (MA): Indicators and Fact Sheets

Rating Scale:  NOT IMPLEMENTED (NI): There are no KPIs for registration and MA activities.
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted KPIs for registration
and MA activities but they have not yet been reported.
 PARTIALLY IMPLEMENTED (PI): The NRA has developed KPIs for registration and MA
activities and has been applying them for less than two year or they have not covered
all critical steps.
 IMPLEMENTED (I): The NRA has established and implemented KPIs for registration
and MA activities. The indicators are reviewed regularly, and appropriate actions are
taken, and decisions made.

Limitations and remarks:  When they refer to outcomes, indicators may be ambiguous and difficult to interpret,
as outcomes are the result of many factors that are difficult to disentangle. When
they refer to processes, indicators are often too specific, as they may focus on a
particular intervention or condition or they may quickly become outdated as business
models develop.
 Different methodologies are used to measure the NRAs performance on registration
and MA activities. In this case, the assessor should verify that adequate supporting
documents are available. The assessor should consider that developed performance
indicators should be Specific, Measurable, Achievable, Realistic, and Time- bound (i.e.,
“SMART”).
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).
03
04
05
06
07
08
09

Page 114 of 340

WHO Global Benchmarking Tool (GBT)

03 01

National Regulatory System (RS): Indicators and Fact Sheets

Vigilance (VL):
Indicators and
Fact Sheets

WHO Global
Benchmarking 02
Tool (GBT) 03
for Evaluation of
04
National Regulatory
System of Medical 05
Products 06
07
08
09

Page 115 of 340

WHO Global Benchmarking Tool (GBT)

03. Vigilance (VL): Indicators

and Fact Sheets
Function:
03 — VIGILANCE (VL)
01
Description: Medical products vigilance, defined as the science and activities relating to the detection, 02
assessment, understanding and prevention of adverse effects or any other medical
product-related problems, is extremely important for guaranteeing that safe and effective 03
medical products of high quality are used within the country. Vigilance activities should be

Vigilance (Vl): Indicators and Fact Sheets

established in the countries based on a risk management approach.

A reporting system should be established to monitor the safety of medical products.

One important activity within that function is to monitor and assess side effects and other
product-related safety issues (e.g., adverse drug reactions (ADRs) for medicines, and adverse
events following immunization (AEFI) for vaccines). While common side effects are likely
to be detected during pre-approval clinical trials (phases I. II and III), rare events are more likely
to be observed after the marketing of medical products. Other unexpected events may also
be due to errors and thus could occur at any time during product development and marketing.
Side effects may differ with respect to severity, causes and public health consequences.
Hence, it is advised that each country establish its own vigilance system. A vigilance system,
in general, monitors all kinds of patient harm potentially related to medical products, be it
due to inadequate product quality, inappropriate use (e.g., medication errors) or intrinsic
adverse effects.
04
Serious effects (e.g., AEFI) often lead to public concerns and could erode the confidence
in medical products and the overall regulatory and health systems. If not dealt with 05
adequately, such concerns could have significant negative implications on the public
health. A post-marketing vigilance system of medical products is therefore essential. 06

Networking with other international bodies and regulators is a logical method for 07
acquiring, sharing, and exchanging the relevant information on medical products
safety. This information, in turn, contributes to informed science-based decisions. To 08
facilitate networking and exchange of information with other international bodies and
regulators, harmonization across countries of the vigilance systems and safety reporting 09
requirements in accordance with internationally agreed standards, is expected.

Indicator: VL01 Legal provisions, regulations and guidelines required to define

regulatory framework of vigilance.

Objective: The objective of this indicator is to ensure that vigilance is supported by a comprehensive
set of legal provisions, regulations and guidelines which provide the necessary mandate to
implement all activities related to this regulatory function. The scope and extent of
the vigilance system should be clearly defined in relevant legislation, regulations and guidelines.
There should be a legal basis to establish a reporting system for ADRs and AEFIs and to
provide the authority for the responsible entity to take actions if needed.
The legislation should also provide for adequate and proportional sanctions, penalties
and prosecution of violations of the applicable legislation. (Please refer to regulatory inspection
function for further information on the regulatory enforcement and compliance activities).

Category: 01. Legal provisions, regulations and guidelines

Sub Indicator: VL01.01: Legal provisions for a national vigilance system exist.

Maturity Level: 1

Page 116 of 340

WHO Global Benchmarking Tool (GBT)

Scope: 1. Medicines
2. Vaccines

Description: The assessor should identify and review the applicable legal provisions establishing the
national vigilance system and assess whether these provisions have been enacted. The
assessor should verify that legal provisions mandating vigilance activities in both public as
well as private sectors exist and are actually implemented. In addition, legislation should
support activities related to the vigilance function.
This legislation should define the duties, powers, and responsibilities of the regulatory 01
authority to manage risks associated with the use of medical products, including collection
of data on the safety of the medical products, analysis and investigation of this data, and 02
adoption of regulatory measures by the health authority.
In general, legal provisions should designate the entities responsible for undertaking 03
different activities pertinent to vigilance functions. Such entities may or may not lie within

Vigilance (Vl): Indicators and Fact Sheets

the National Regulatory Authority (NRA). However, in all cases, the entities responsible for
vigilance functions should be independent from all interested parties (e.g., manufacturers,
marketing authorization holders (MAHs), purchasers and supply divisions, including
the expanded programme on immunization (EPI), distributors, and health care
professionals).

Objective: The objective of this sub-indicator is to ensure that legal provisions exist for establishing
the national vigilance system. These legal provisions provide the mandate for the
responsible entities to actually implement their activities. Without such a mandate, the
relevant regulatory activities would be too fragile and could be easily challenged.

Requirement: Legal provisions establishing the national vigilance system

Evidence to review: The assessor should ask for and review: 04

1. Legal provisions establishing the national medical products vigilance system (e.g.,
defining the responsible entities as well as the roles, responsibilities, accountability, and 05
obligations of these entities).
06
References: 1. Safety Monitoring of Medicinal Products: Guidelines for Setting Up and Running a
Pharmacovigilance Centre. UMC, WHO, 2000, (26), (http://apps.who.int/medicinedocs/ 07
en/d/Jh2934e/)
08
Framework: Structure/Foundation/Input
09
Rating Scale:  NOT IMPLEMENTED: There are no legal provisions or regulations to satisfy the
requirement of the sub- indicator.
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps towards establishment
of legal provisions and regulations to satisfy the requirement of the sub-indicator;
however no results yet exist.
 PARTIALLY IMPLEMENTED (PI): Legal provisions and regulations to satisfy the
requirement of the sub- indicator were recently established and are in early
implementation phase so no consolidated results exist yet.
 IMPLEMENTED (I): There are legal provisions and regulations which satisfy the
requirement of the sub-indicator and which are actually implemented and enforced.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: VL01.02: Legal provisions and regulations require the manufacturers and/or
MAHs to set up a vigilance system of their medical products and periodically
report vigilance data to the NRA.

Maturity Level: 1

Scope: 1. Medicines
2. Vaccines

Page 117 of 340

WHO Global Benchmarking Tool (GBT)

Description: The assessor should verify that legal provisions stipulate that manufacturers and/or MAHs
have an organized vigilance system to collect, record, store, maintain, evaluate and analyze
adverse events (AEs) of their medical products in order to monitor their safety profile.
In addition, the legal provisions should require manufacturers and/or MAHs to set up a
vigilance system during the life cycle of their medical products and periodically report such
events In
addition, the assessor should also confirm that the legal provisions allow the NRA to
conduct inspections in order to check the existence of vigilance systems as well as the
implementation of the Good Vigilance Practices at the manufacturer’s and/or MAH’s site. 01
and these data, including “zero” events (i.e., situations where no events are reported), to
the NRA. Reporting of these events should be done according to established requirements 02
and in alignment with international standards. Examples include periodic safety update
reports, and periodic benefit risk evaluation reports. The assessor should verify that the 03
“zero” events have been included systematically in routine periodic reports (i.e., include a

Vigilance (Vl): Indicators and Fact Sheets

confirmation that no reports had been received).
The assessor should also verify that legislation includes the obligation that manufacturers
and/or MAHs inform the NRA of any safety signal of a product as well as any marketing
or regulatory decisions taken in the country of origin or other countries where the product
is marketed. In line with these legal provisions, the assessor should verify that relevant
guidelines include provisions that endorse this obligation and that explain how, when and
what safety issues have to be reported (e.g. recall notifications, reporting of study results
from ongoing or previous studies, and information on supervision of clinical trials). A similar
obligation should also be in force for the MAH for all products marketed in the country.

Objective: The objective of this sub-indicator is to ensure the existence of legal provisions mandating
that the responsible entities require the MAH to establish a vigilance system for their
medical products.
04
Requirement: Legal provisions mandating vigilance system establishment by the MAH
05
Evidence to review: The assessor should ask for and review:
1. Legal provisions mandating the manufacturers and/or MAHs to establish vigilance 06
systems for their medical products.
2. Legal provisions obligating the manufacturers and/or MAHs to report safety data to the 07
NRA.
3. Legal provisions authorizing the NRA to conduct vigilance inspections. 08
4. Regulations and guidelines explaining the obligations of the manufacturers and MAHs
for safety data reporting. 09

References: 1. Safety Monitoring of Medicinal Products: Guidelines for Setting Up and Running a
Pharmacovigilance Centre. UMC, WHO, 2000, (26), (http://apps.who.int/medicinedocs/
en/d/Jh2934e/)
2. The importance of Pharmacovigilance: safety monitoring of medicinal products. World
Health Organization, 2002, (17), (http://apps.who.int/medicinedocs/pdf/s4893e/s4893e.pdf)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED: There are no legal provisions or regulations to satisfy the
requirement of the sub- indicator.
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps towards establishment
of legal provisions and regulations to satisfy the requirement of the sub-indicator;
however no results exist yet.
 PARTIALLY IMPLEMENTED (PI): Legal provisions and regulations to satisfy
the requirement of the sub-indicator were recently established and are in early
implementation phase so no consolidated results exist yet.
 IMPLEMENTED (I): There are legal provisions and regulations which satisfy the
requirement of the sub-indicator and which are actually implemented and enforced.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Page 118 of 340

WHO Global Benchmarking Tool (GBT)

Sub Indicator: VL01.03: Guidelines ensure that distributors, importers, exporters, healthcare
institutions, consumers and other stakeholders are encouraged to report adverse
drug reactions (ADRs) and AEs to the MAH and/or NRA.

Maturity Level: 1

Scope: 1. Medicines
2. Vaccines
01
Description: The assessor should verify the existence of guidelines (including any sort of formal
documentation) for encouragement of different stakeholders (e.g., importers, exporters, 02
distributors, health care professionals and institutions, consumers and patients) to
report ADRs and AEs to the MAH and/or the NRA. The regulation should ideally be widely 03
distributed and communicated to the public community in order to be meaningful and to

Vigilance (Vl): Indicators and Fact Sheets

be implemented.

Objective: The objective of this sub-indicator is to ensure reporting of vigilance events by different
stakeholders as an essential block of the vigilance function which significantly contributes
to its effectiveness and impact.

Requirement: Guidelines encouraging different stakeholders (including importers, exporters, distributors,

health care professionals and institutions, public health programmes, consumers and
patients) to report vigilance events to the MAH and/or the NRA.

Evidence to review: The assessor should ask for and review:

1. Guidelines or other documentation encouraging reporting of vigilance events to the MAH
and/or the NRA.
04
References: 1. Safety Monitoring of Medicinal Products: Guidelines for Setting Up and Running a
Pharmacovigilance Centre. UMC, WHO, 2000, (26), (http://apps.who.int/medicinedocs/ 05
en/d/Jh2934e/)
2. WHO guidelines on safety monitoring of herbal medicines in pharmacovigilance 06
systems. World Health Organization; 2004, (18), (http://apps.who.int/medicinedocs/
documents/s7148e/s7148e.pdf) 07
Framework: Structure/Foundation/Input 08
Rating Scale:  NOT IMPLEMENTED: There are no guidelines to satisfy the requirement of the sub- 09
indicator.
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps towards establishment of
guidelines to satisfy the requirement of the sub-indicator; however no results exist yet.
 PARTIALLY IMPLEMENTED (PI): guidelines to satisfy the requirement of the sub-
indicator were recently established and are in early implementation phase so no
consolidated results exist yet.
 IMPLEMENTED (I): There are guidelines which satisfy the requirement of the sub-
indicator and which are actually implemented and enforced.

Limitations and remarks:  Event: a specific identifiable happening or occurrence, e.g., the taking of a medicine or
the experience of an adverse effect.
 Adverse event (AE): Any untoward medical occurrence that may appear during
treatment with a pharmaceutical product but which does not necessarily have a
causal relationship with the treatment.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: VL01.04: Legal provisions and regulations allow NRA to require manufacturers
and/or MAHs to conduct specific studies on safety and effectiveness under
specific conditions.

Maturity Level: 2

Page 119 of 340

WHO Global Benchmarking Tool (GBT)

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that legal provisions and regulations provide the suitable and
proportionate mandate for the NRA to require the manufacturers and/or MAHs to perform
specific post-authorization safety activities to further characterize the safety profile of
the product. For example, these may include the conduct of additional studies on safety
and effectiveness under specific conditions, establishment of registries, and specific data
analyses at time points post authorization. The assessor should verify that such legislation 01
exists and is endorsed.
02
Objective: The objective of this sub-indicator is to ensure that legal provisions and regulations are
providing the necessary mandate for the NRA to request the manufacturers and/or MAHs 03
to conduct specific safety and effectiveness studies under specific conditions.

Vigilance (Vl): Indicators and Fact Sheets

Requirement: Legal provisions and regulations allowing the NRA to require MAHs to perform specific
post-authorization safety activities to further characterize the safety profile of the
product. These may include, for example, the conduct of additional studies on safety
and effectiveness under specific conditions, establishment of registries, or specific data
analyses at time points post authorization, etc.).

Evidence to review: The assessor should ask for and review:

1. Legal provisions and regulations authorizing the NRA to request specific post-marketing
phase safety and/or effectiveness studies.

References: 26- Safety Monitoring of Medicinal Products: Guidelines for Setting Up and Running a
Pharmacovigilance Centre. UMC, WHO, 2000, (26), (http://apps.who.int/medicinedocs/
en/d/Jh2934e/) 04
Framework: Structure/Foundation/Input 05
Rating Scale:  NOT IMPLEMENTED: There are no legal provisions or regulations to satisfy the 06
requirement of the sub- indicator.
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps towards establishment 07
of legal provisions and regulations to satisfy the requirement of the sub-indicator ;
however no results exist yet. 08
 PARTIALLY IMPLEMENTED (PI): Legal provisions and regulations to satisfy
the requirement of the sub-indicator were recently established and are in early 09
implementation phase so no consolidated results exist yet.
 IMPLEMENTED (I): There are legal provisions and regulations which satisfy the
requirement of the sub-indicator and which are actually implemented and enforced.

Limitations and remarks:  Effectiveness: a measure of the chances or odds (i.e., probability) of a medicine
working positively as expected for patients.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: VL01.05: Legal provisions, regulations and guidelines require manufacturers and/or
MAHs to designate an individual person to be in charge of vigilance system.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that legal provisions require the manufacturers and/or MAHs
to designate an individual to be in charge of all vigilance activities relevant to their medical
products placed on the market. This person should have oversight of the vigilance
system in terms of structure and performance. In addition, the assessor should check
the requirements for designation of that person and the responsibilities assigned to that

Page 120 of 340

WHO Global Benchmarking Tool (GBT)

person. Assessors should note that the qualifications for being designated as an individual
person in charge of medical products vigilance may vary from one country or region to
another; however in all cases the requirements should ensure the competence of the
person responsible for these assigned roles and responsibilities.

Objective: The objective of this sub-indicator is to ensure that legal provisions and regulations
provide the necessary mandate for the NRA to obligate the manufacturers and/or MAHs to
designate an individual person to be in charge of medical products vigilance.
01
Requirement: Legal provisions and regulations mandating the manufacturers and/or MAHs to designate
an individual person in charge of medical products vigilance. 02

Evidence to review: The assessor should ask for and review: 03

1. Legal provisions and regulations obligating the manufacturers and/or MAHs to

Vigilance (Vl): Indicators and Fact Sheets

designate an individual person to be in charge of medical products vigilance.
2. Documents that define the qualifications for being designated as an individual person in
charge of medical products vigilance.
3. Documents that define the responsibilities of the individual person in charge of medical
products vigilance.

References: 1. WHO Pharmacovigilance indicators: A practical manual for the assessment of

pharmacovigilance systems. World Health Organization. 2015, (85), (http://apps.who.int/
medicinedocs/documents/s21970en/s21970en.pdf)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED: There are no legal provisions, regulations or guidelines to satisfy
the requirement of the sub-indicator. 04
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps towards establishment
of legal provisions, regulations and guidelines to satisfy the requirement of the sub- 05
indicator; however no results exist yet.
 PARTIALLY IMPLEMENTED (PI): Legal provisions, regulations and guidelines to 06
satisfy the requirement of the sub-indicator were recently established and are in early
implementation phase so no consolidated results exist yet. 07
 IMPLEMENTED (I): There are legal provisions, regulations and guidelines which
satisfy the requirement of the sub-indicator and which are actually implemented and 08
enforced.
09
Limitations and remarks:  The title of the individual person in charge of medical products vigilance may vary
from one country or region to another. For example, this person may be called a
Qualified Person for Pharmacovigilance in one country or region, while a different title
may be used in another.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: VL01.06: There are guidelines for planning, conducting, monitoring, and reporting
of vigilance activities.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that guidelines exist and are endorsed for different vigilance
activities, including planning, conducting, monitoring and reporting. The assessor should
verify that the guidance is in agreement with other internationally accepted guidance
such as that from the World Health Organization (WHO) or International Conference on
Harmonization. As a general principle, harmonization of the safety regulatory framework
is expected in order to facilitate the exchange of information and experience across NRAs.
When applicable, differences should be identified.

Page 121 of 340

WHO Global Benchmarking Tool (GBT)

The assessor should identify and review these guidelines and make sure they include:
1. Objectives of the vigilance system;
2. List of events (ADRs and adverse events following immunization (AEFIs)) to be reported;
3. For vaccines, case definitions of events to be reported;
4. Clear definitions of terminology relevant for analysis and response (e.g., AE, adverse
reaction, medication error, coincidental error, program error, serious event, and cluster
event);
5. Information on how to report (i.e., who, how, where, when and to whom);
6. All medical products to be included in the reporting system (i.e., reporting is not 01
restricted to certain products or sectors);
7. Procedures for analyzing data; 02
8. Procedures for providing feedback of findings to key players (e.g. reporters of
complaints, informers, parents, caregivers, associations and communities) and for relevant 03
follow-up actions;

Vigilance (Vl): Indicators and Fact Sheets

9. Guidance for investigations and actions to be taken in case of serious events or a cluster
of events
10. Guidance on assessment of the balance between risks and benefits for medical
products;
11. Guidance on crisis prevention and management;
12. Guidance on communication of vigilance information;
13. Definition of the individuals in charge;
14. Guidance on potential vigilance-related differences and particularities among different
medical products (e.g. medicines versus vaccines).

Objective: The objective of this sub-indicator is to ensure that the guidelines reflect the current
thinking of the NRA on different vigilance activities and are facilitating and promoting
Good Vigilance Practices. This significantly contributes to vigilance effectiveness and
consistency. 04
Requirement: Guidelines relevant to vigilance including planning, conducting and reporting 05
activities.
06
Evidence to review: The assessor should ask for and review:
1. Guidelines for MAH and competent authorities on vigilance activities. 07
2. Guidelines for planning, conducting and reporting of vigilance activities. Ideally, these
should include the items listed in the description section of this fact sheet. 08
References: 1. The importance of Pharmacovigilance: safety monitoring of medicinal products. 09
World Health Organization, 2002, (17), (http://apps.who.int/medicinedocs/pdf/s4893e/
s4893e.pdf)
2. WHO Pharmacovigilance indicators: A practical manual for the assessment of
pharmacovigilance systems. World Health Organization. 2015, (85), (http://apps.who.int/
medicinedocs/documents/s21970en/s21970en.pdf)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED: There are no guidelines to satisfy the requirement of the
sub-indicator.
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps towards
establishment of guidelines to satisfy the requirement of the sub-indicator;
however no results exist yet.
 PARTIALLY IMPLEMENTED (PI): Guidelines to satisfy the requirement of the sub-
indicator were recently established and are in early implementation phase so no
consolidated results exist yet.
 IMPLEMENTED (I): There are guidelines which satisfy the requirement of the sub-
indicator and which are actually implemented and enforced.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs)

Page 122 of 340

WHO Global Benchmarking Tool (GBT)

Sub Indicator: VL01.07: Legal provisions and regulations allow recognition and/or reliance on
vigilance-related decisions, reports or information from other countries or regional
or international bodies.

Maturity Level: 1

Scope: 1. Medicines
2. Vaccines
01
Description: The assessor should verify that legal provisions and regulations permit the application of
a recognition and/or reliance regulatory model to the vigilance-related national decisions. 02
Although there is an ultimate need to establish a national vigilance system for medical
products, recognition and/or reliance on vigilance-related information and decisions from 03
other countries or regional or international entities would significantly contribute to the

Vigilance (Vl): Indicators and Fact Sheets

safety of medical products in the country. This would, in turn contribute to the public health.
While recognition considers regulatory decisions from other sources in an automatic
way with minimal sharing of information and justification, reliance depends heavily
on information-sharing, communication, and scientific rationale and reasoning. Both
approaches (i.e., recognition and reliance) might be applied in passive or active modes.
However, the active mode is encouraged and preferred over the passive one.

Objective: The objective of this sub-indicator is to ensure that legal provisions and regulations
are providing the necessary mandate for the NRA to apply recognition and/or reliance
regulatory model.

Requirement: Legal provisions and regulations authorize the NRA to recognize and/or rely on vigilance-
related decisions, reports or information from other countries or regional or international
entities when making national vigilance decisions. 04
Evidence to review: The assessor should ask for and review: 05
1. Legal provisions and regulations relevant to reliance and/or recognition as applied to
vigilance. 06
2. Legal provisions and regulations relevant to information sharing, reliance and/or
recognition as applied to vigilance. 07
References: 08
Framework: Structure/Foundation/Input 09

Limitations and remarks: In some countries, legal provisions and regulations might not support such recognition or
reliance. In this case, the scoring of the sub-indicator should be NOT APPLICABLE provided
that national vigilance system is able to respond to vigilance events elsewhere.

Indicator: VL02 Arrangement for effective organization and good governance.

Objective: The objective of this indicator is to ensure effective organization and good governance
practices at the entities in charge of vigilance activities. Effective organization and practices
in turn contribute to the effective and efficient functioning of the vigilance activities.

Page 123 of 340

WHO Global Benchmarking Tool (GBT)

Category: 02. Organization and governance

Sub Indicator: VL02.01: There is a defined organizational structure with clear responsibilities
to conduct vigilance activities.

Maturity Level: 2

Scope: 1. Medicines
2. Vaccines 01
Description: The assessor should identify the organization set in place to establish, implement or 02
maintain the vigilance regulatory function including the defined organizational structures
that undertake different relevant activities. Responsibilities, duties and roles of these 03
structures should be clearly defined and documented. In case of involvement of more than

Vigilance (Vl): Indicators and Fact Sheets

one entity, the assessor should check the ways and approaches by which coordination
among these structures is managed.

Objective: The objective of this sub-indicator is to ensure effective organization and good governance
of vigilance activities and that these are taken over by defined structures with clear roles
and responsibilities.

Requirement: Roles and responsibilities of the organizational structures in charge of vigilance activities.

Evidence to review: The assessor should ask for and review:

1. Organization chart of the organizations responsible for the implementation of vigilance
activities along with identification of the particular structures implementing the function.
2. Documentation clarifying roles and responsibilities of the structures implementing
vigilance activities. This may include, for example, administrative decrees, terms of 04
reference, and position descriptions.
3. Documentation identifying established mechanisms of coordination (e.g., committees, 05
internal work-sharing, and workflow) among structures, if any, which take part in vigilance
activities. 06
References: 1. National drug regulatory legislation: guiding principles for small drug regulatory 07
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations:
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical 08
Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf) 09

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED: There is no defined structure in charge of vigilance activities.
 ONGOING IMPLEMENTATION (OI): A mandate to establish a structure in
charge of vigilance activities is available; however the structure itself is not yet
established.
 PARTIALLY IMPLEMENTED (PI): Structure in charge of vigilance activities is newly
established and mandated; however the regular work and practice of this structure is
not yet consolidated.
 IMPLEMENTED (I): There is defined structure in charge of vigilance activities with
clear, well-documented roles and responsibilities.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: VL02.02: Documented procedures and mechanisms are implemented to ensure
the involvement, coordination and communication among all stakeholders
relevant to vigilance activities

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Page 124 of 340

WHO Global Benchmarking Tool (GBT)

Description: The assessor should verify that different stakeholders relevant to vigilance regulatory
function are identified. These stakeholders would include not only the different
structures (e.g., organizations, institutions, and departments) implementing
the regulatory function, but also other entities (e.g., manufacturers and MAHs),
governmental agencies (e.g., supply divisions, procurement divisions, and public health
programmes including EPI), international organizations (e.g., WHO or the Uppsala
Monitoring Centre), non-governmental organizations, professional associations,
and customer representative associations. In addition, the assessor should verify
the availability and implementation of agreements, memoranda of understanding 01
(MOUs) or documented procedures ensuring the active involvement, communication
and collaboration among the identified stakeholders relevant to vigilance of medical 02
products.
03
Objective: The objective of this sub-indicator is to ensure that documented procedures and

Vigilance (Vl): Indicators and Fact Sheets

other mechanisms are implemented to ensure the involvement, coordination and
communication among different organizations, institutions, and departments for
appropriate organization and good governance of the function.

Requirement: Agreements, MOUs and procedures for ensuring involvement of and communication
among stakeholders relevant to vigilance regulatory function.

Evidence to review: The assessor should ask for and review:

1. List of stakeholders relevant to vigilance of different medical product streams.
2. Agreements, MOUs and standard operating procedures (SOPs) defining the means of
communication and collaboration among the identified stakeholders.
3. Example records of communication and collaboration demonstrating implementation
of the above-mentioned procedures.
4. Guidelines or SOPs defining procedures for external and internal communications. 04
5. Documentation that regular meetings and other formal or official communication
among above-mentioned stakeholders take place. This documentation should provide 05
evidence for a systematic and well-established communication process.
6. Documentation of paths of communication and reporting. 06
7. Platforms for information sharing and exchange.
07
References: 1. The importance of Pharmacovigilance: safety monitoring of medicinal products.
World Health Organization, 2002, (17), (http://apps.who.int/medicinedocs/pdf/s4893e/ 08
s4893e.pdf)
2. WHO Pharmacovigilance indicators: A practical manual for the assessment of 09
pharmacovigilance systems. World Health Organization. 2015, (85), (http://apps.who.
int/medicinedocs/documents/s21970en/s21970en.pdf)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED: There are no coordination and information exchange
procedures or mechanisms among different stakeholders of the vigilance
function or among the central authority and the decentralized entities.
 ONGOING IMPLEMENTATION (OI): The NRA is developing a coordination and
information exchange mechanism, but it is not yet ready or exchanges are being
conducted without an established methodology.
 PARTIALLY IMPLEMENTED (PI): The NRA recently established a coordination and
information exchange procedure and mechanism that is at the implementation
stage; however, this practice is not yet consolidated.
 IMPLEMENTED (I): There are established, implemented and well-maintained
coordination and information exchange procedures and mechanisms among
different stakeholders of the vigilance function and among the central authority
and the decentralized entities.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Page 125 of 340

WHO Global Benchmarking Tool (GBT)

Indicator: VL03 Human resources to perform vigilance activities.

Objective: The objective of this indicator is to ensure that all entities within a National Regulatory
Authority (NRA) that are concerned with vigilance activities are adequately resourced with
a trained, experienced and skilled workforce, and that this workforce has the authority to
fully perform the assigned responsibilities. This will ensure that vigilance processes and
activities are performed in accordance with international best practices.
This indicator is assessed through the evaluation of the human resource capacity of the 01
entities with respect to the number of personnel, the composition, skills and experience of
the personnel, and the expertise in those specific areas required to perform the vigilance 02
function.
03
Category: 06. Resources (HR, FR, infrastructure and equipment)

Vigilance (Vl): Indicators and Fact Sheets

Sub Indicator: VL03.01: Sufficient competent staff (i.e., education, training, skills and experience)
are assigned to perform vigilance activities

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that the human resources assigned to perform vigilance activities
should be sufficient with respect to numbers and competent with respect to the requisite
skills, education, experience and training. There should be technical documents and SOPs
that provide guidance on the required background for vigilance activities and that consider
the requirements for educational background, competencies, skills, experience, and training. 04
The assessor should verify that the NRA estimated the number of staff required to effectively
and efficiently perform vigilance function and that the NRA actually recruited that number. In 05
addition, the assessor should verify that these competency requirements are well-established
and maintained by the NRA. Metrics and statistics on the different activities performed as 06
well as performance indicators can be used for estimating the adequacy of the number of the
assigned staff. The assessor should also verify that the competency of the assigned staff is 07
built, maintained and improved through recruitment as well as continuous on-the-job training.
08
Objective: The objective of this sub-indicator is to ensure the existing human resources for vigilance
is sufficient, in terms of numbers, experience, and specific competencies, to perform all the 09
activities along the entire vigilance chain.

Requirement: Sufficient number of competent human resources in charge of vigilance activities.

Evidence to review: The assessor should ask for and review:

1. Evidence that the number of staff members involved in each of the documented
activities along the entire vigilance process flow is adequate.
2. Evidence that the systems and structures are in place to ensure appropriate placement
of staff with respect to competence and skills.
3. Evidence that the system and structures have been implemented. The documentation
should include the records to verify that the staff competence is appropriate for the job
requirements.
4. Evidence that the professional profiles of the human resources engaged in vigilance
activities are appropriate with respect to education, skills, and expertise, to perform a
particular function along the vigilance chain. Documentation should include a list of the
requisite skills and training for each position.
5. Recruitment plan.

References: 1. Quality management systems — Requirements. International Standard ISO 9001:2015.

International Organization for Standardization, 2015, (4), (https://www.iso.org/iso-9001-
quality-management.html)

Framework: Structure/Foundation/Input

Page 126 of 340

WHO Global Benchmarking Tool (GBT)

Rating Scale:  NOT IMPLEMENTED: The NRA does not have enough competent staff (i.e., education,
training, skills and experience) to perform vigilance activities
 ONGOING IMPLEMENTATION (OI): The NRA has recently developed a plan to recruit
adequate competent staff; however, the plan has not been implemented.
 PARTIALLY IMPLEMENTED (PI): The NRA has initiated the implementation of
the human resources development plan; however, there is need to complete the
competency profile.
 IMPLEMENTED (I): The NRA has a sustained number of competent staff
(i.e., education, training, skills and experience) assigned to perform vigilance 01
activities.
02
Limitations and remarks:  Assessment of the adequacy and appropriateness of the number of staff members
is quite subjective and should be linked to some process or output indicators. When 03
estimating staff adequacy, the assessor should consider the workload, backlog, and

Vigilance (Vl): Indicators and Fact Sheets

delays in delivery based on established timeframes.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: VL03.02: Duties, functions, and responsibilities of the staff in charge of vigilance
activities are established and updated in the respective job descriptions.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that procedures are in place to maintain a current and
updated structure for managing job descriptions for personnel participating in 04
vigilance activities. In addition, job descriptions should address current staff duties,
responsibilities and the requisite competencies. A job description with this format and 05
content
should be established and implemented for all staff. The management of job 06
descriptions should be supported by a guidance document that provides direction
on when and how to update the information, and where the information should be 07
kept for easy access. The guidance document should present the appropriate duties
and responsibilities that are assigned to each member of the organization involved in 08
vigilance activities.
Thus, the professional profiles of staff are reflected in their respective roles and 09
responsibilities within the NRA. There should be procedures to guide responsible
persons to document that duties, functions and responsibilities are revised and
kept up to date. In addition, procedures should be available to guide the keeping and
documenting of up to date work schedules and enforcing the implementation of the
documented guidelines and procedures.

Objective: The objective of this sub-indicator is to ensure that duties and responsibilities of the staff
are clear and well defined, that job descriptions are kept up to date with current duties,
functions and responsibilities, and that these activities are adequately documented.

Requirement: Duties, roles and responsibilities of the staff relevant to vigilance activities.

Evidence to review: The assessor should ask for and review:

1. Procedure and guidelines that guide placement of staff members within the NRA;
2. The professional profiles of staff (i.e., job descriptions) and documentation that they
are related to their current roles and duties;
3. The professional profiles of the external experts and documentation that the profiles
provide a composition that is complete and consistent with that prescribed in the legal
provisions;
4. Procedures to guide the documentation of up to date duties and work schedules, and
to enforce the implementation of the documented guidelines and procedures;
5. Job descriptions for designated staff.

Page 127 of 340

WHO Global Benchmarking Tool (GBT)

Vigilance (Vl): Indicators and Fact Sheets

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED: There is no evidence of defined or established duties, functions,
responsibilities, respective job descriptions and necessary required competencies.
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted or developed the role
and responsibilities document but it has not yet been implemented.
 PARTIALLY IMPLEMENTED (PI): The NRA has initiated implementation of this
requirement but it has not been defined or followed for all staff or the roles and
responsibilities documents, including staff job descriptions, are not up to date.
 IMPLEMENTED (I): The NRA has defined and established all required duties,
functions, and responsibilities, and respective job descriptions are up-to-date.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will 04
always apply for all benchmarked NRAs).
05
Sub Indicator: VL03.03: Training plan developed, implemented and updated at least once a year for
staff in charge of vigilance activities. 06
Maturity Level: 3 07
Scope: 1. Medicines 08
2. Vaccines
09
Description: The assessor should verify that training plans are developed, implemented and updated
at least once every year to reflect the current situation by considering education and
experience of the staff. The training plan should be complemented with guidelines or
similar documents that guide the development and implementation of training plans. The
assessor should ensure that induction training for new staff as well as continued on- the-
job for staff is planned and implemented. There should be procedures to approve the
training plan and
the budget allocated for implementing and updating the training plan. The plans should
present clearly defined training goals, and should include training in certain topics
and skills to address identified deficiencies. Learning objectives, training methods and
activities, evidence of learning, and evaluation and assessment of training should be
documented. This documentation should confirm that the learning objectives were
achieved and were designed to address weaknesses within the entities. Procedures should
be in place to ensure that a training plan is developed, implemented and updated at least
once every year. The assessor should verify that there is a system in place for monitoring
the implementation and effectiveness of the training plan and for documenting the skills
acquired in training activities for internal and external experts.

Objective: The objective of this sub-indicator is to ensure that a training plan for staff exists,
and that it is implemented and updated annually. Through the training plan, NRA can
be sure that competency of staff in charge of vigilance activities is maintained and
enhanced.

Page 128 of 340

WHO Global Benchmarking Tool (GBT)

Requirement: Implementation of training plan

Evidence to review: The assessor should ask for and review:

1. Guidelines for development, implementation and annual update (i.e., at least once per
year) of the training plan. Guidelines should also provide for a mechanism to measure
effectiveness of training.
2. Documentation for the system or structures used to approve the training plan and to
evaluate the adequacy of the budget allocated to the training activities.
3. The current or existing staff training plan (or matrix) for staff. The assessor should 01
assess this in in relation to the respective individual job descriptions.
4. SOP for developing and maintaining the training plan. 02
5. Evidence that the NRA has investigated and identified training needs.
6. List of trainings performed. 03
7. Example records for training activities.

Vigilance (Vl): Indicators and Fact Sheets

References: 1. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations:
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical
Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf)
2. Quality management systems — Requirements. International Standard ISO 9001:2015.
International Organization for Standardization, 2015, (4), (https://www.iso.org/iso-9001-
quality-management.html)
3. Guiding principles for small national drug regulatory authorities. In: WHO Expert
Committee on Specifications for Pharmaceutical Preparations: thirty-first report. Geneva:
World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No. 790), (5),
(http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
4. Effective drug regulation: A multicounty study. Geneva: World Health Organization; 2002, 04
(7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)
5. Quality systems requirements for national good manufacturing practice inspectorates. 05
In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: thirty-sixth
report. Geneva: World Health Organization; 2002: Annex 8 (WHO Technical Report Series, 06
No. 902), (8), (http://apps.who.int/medicinedocs/documents/s22112en/s22112en.pdf)
07
Framework: Process
08
Rating Scale:  NOT IMPLEMENTED: There is no systematic training program including training plan
(or matrix). 09
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted or developed the
training plan but there is no evidence of implementation.
 PARTIALLY IMPLEMENTED (PI): The NRA has developed and initiated the training plan
implementation. However, the NRA has not fulfilled all required planned training or has
been applying the plan for less than two years.
 IMPLEMENTED (I): The NRA has an updated training plan developed that is supported
by adequate records to demonstrate effective plan implementation, including induction
training for new staff and routine on-the-job training for recruited staff.

Limitations and remarks:  Training plans must be updated regularly; ideally on an annual basis, but not less
frequently that once every two years.
 Some regulatory functions may include many training activities that are not
incorporated in the institutional training programme. Such training normally is offered
by invitation. In this case, the assessor should recognize reports from non-routine
vigilance -relevant training not included in the NRA training plan.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: VL03.04: The NRA generates and maintains records of staff training activities
and training effectiveness verification.

Maturity Level: 3

Page 129 of 340

WHO Global Benchmarking Tool (GBT)

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that records of staff training that is performed or organized
by the NRA are generated, maintained, regularly updated. This activity should be
supported by guidelines that direct the NRA to generate and maintain records of
staff training activities. Procedures should be in place to document and propose staff
training needs and to allocate a budget for continuous staff capacity building and
development. The assessor should check that there is an evaluation or assessment 01
mechanism to verify the quality of learning, and to confirm that learning objectives are
achieved. Documentation should include an inventory (i.e., soft and/or hard) system 02
that records all impactful and non-impactful trainings, and identifies all staff members
who participated. A system to measure or estimate impact of trainings should be 03
established.

Vigilance (Vl): Indicators and Fact Sheets

Requirement: Training records

Evidence to review: The assessor should ask for and review:

1. Guidelines or similar documents that guide the NRA to generate and maintain records
of staff training activities;
2. Evaluations of training effectiveness;
3. The training inventory, and procedures for completing the inventory; 04
4. Examples of archived records of staff training, and procedures for the archiving
system. 05
References: 1. WHO Pharmacovigilance indicators: A practical manual for the assessment of 06
pharmacovigilance systems. World Health Organization. 2015, (85), (http://apps.who.
int/medicinedocs/documents/s21970en/s21970en.pdf) 07
2. WHO guidelines on safety monitoring of herbal medicines in pharmacovigilance
systems. World Health Organization; 2004, (18), (http://apps.who.int/medicinedocs/ 08
documents/s7148e/s7148e.pdf)
3. National drug regulatory legislation: guiding principles for small drug regulatory 09
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical
Preparations: thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO
Technical Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/documents/
s21964en/s21964en.pdf)
4. Quality management systems — Requirements. International Standard ISO
9001:2015. International Organization for Standardization, 2015, (4), (https://www.iso.
org/iso-9001-quality-management.html)
5. Guiding principles for small national drug regulatory authorities. In: WHO Expert
Committee on Specifications for Pharmaceutical Preparations: thirty-first report.
Geneva: World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No.
790), (5), (http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
6. Effective drug regulation: A multicounty study. Geneva: World Health Organization;
2002, (7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)

Framework: Output

Rating Scale:  NOT IMPLEMENTED: There is no evidence that the NRA generates and maintains
records of staff training activities.
 ONGOING IMPLEMENTATION (OI): The NRA has recently initiated plans to
generate, document and keep records of staff training activities, however they are
not yet followed.
 PARTIALLY IMPLEMENTED (PI): The NRA has recently initiated plans to generate,

Page 130 of 340

WHO Global Benchmarking Tool (GBT)

document and keep records of staff training activities but they are not fully
followed for all training activities or they have been established for less than two
years.
 IMPLEMENTED (I): The NRA generates and maintains records of staff training
activities.

Limitations and remarks:  The assessor should note that some NRAs out-source training including staff
capacity development activities. In this case the assessor should request the identity
of the provider, as well as evaluations of the provider. The assessor may request the 01
professional profiles of tutors or resource persons used to offer training. The assessor
may also request records covering the archiving systems in place. 02
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 03

Vigilance (Vl): Indicators and Fact Sheets

Indicator: VL04 Procedures established and implemented to perform
vigilance activities.

Objective: The objective of this indicator is to ensure that required activities of the vigilance
system are implemented through standard procedures and work instructions that
ensure the consistency, effectiveness, efficiency, impartiality and proportionality of the
vigilance system.
The NRA should have established a system which allows the regular review of
safety and effectiveness aspects of the authorized products on the market, including
processes to review and share relevant data between key players and to take
appropriate action when necessary. There should be written procedures that define how
relevant data are routinely shared among key personnel engaged in vigilance activities
and how any actions taken are reviewed for appropriateness. Documented evidence
to be assessed include procedures for review of safety and effectiveness aspects of 04
medical products on the market, for review of reports of notifications, investigations,
data analyses, and committee meetings, for sharing of relevant data among key players, 05
and for taking appropriate actions.
06
Category: 07. Regulatory process
07
Sub Indicator: VL04.01: Vigilance procedures and tools are in place and implemented for collection
and assessment of ADRs and AEs. 08
Maturity Level: 3 09

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the existence and implementation of standard procedures
for different vigilance activities related to collection and assessment of ADRs and AEs
including:
1. Existence of a paper-based or an electronic reporting system for ADR- and AEFI-
related information that is accessible to MAHs, healthcare professionals and patients
2. Systems for detection and receipt of vigilance events and ADR reports that is
complemented with reporting systems (e.g., either active or passive, and either sentinel
or country- or state-wide) that have the satisfactory sensitivity to detect serious events
or clusters of events. For this point, the assessor should check for the number and rates
of reports within defined periods, ADRs following off-label use, and a breakdown of
reports that compares district or regional reporting activities for the different products
and populations that are involved in vigilance reporting.
3. Timely review, analysis, and causality assessment of ADRs and AEs. For this process,
assessor should verify that:
a. in 80% of cases an investigation is initiated within 48 hours following reporting (for
vaccines);
b. preliminary investigation report is available within 1 week (for vaccines);
c. Investigation is of adequate quality, procedures are thorough, and findings are clearly-

Page 131 of 340

WHO Global Benchmarking Tool (GBT)

described; d. conclusions are scientifically justified and supported by findings.

4. Records of reported ADRs and AEFIs (for vaccines) are stored and maintained.
The assessor should verify that the above-mentioned procedures are consistent with
the relevant national guidelines which in turn are in coherence with WHO or other
international accepted guidance. The assessor should review records that verify the
actual implementation of such procedures.
In addition, the assessor should verify that all vigilance data received are analyzed on a
regular basis. The assessor should check the tools NRA is using to collect and analyze
the data (e.g. calculation of incidence rates or the assessment of causality). WHO 01
recommends the following tools:
1. A national database or similar centralized (e.g., regional, sub- regional or global) 02
system, that is compatible with International Conference on Harmonization E2B, to
code, collate, and store data and reports and to analyze vigilance data. 03
2. The adoption of standard case definitions for AEFIs (e.g., Brighton collaboration

Vigilance (Vl): Indicators and Fact Sheets

definitions or national case definitions).

Objective: The objective of this sub-indicator is to ensure consistency of different activities

of the medical products vigilance system through standard procedures which are
complemented by the necessary tools.

Requirement: Procedures and tools necessary for effective implementation of different regulatory
activities.

Evidence to review: The assessor should ask for and review:

1. Procedures and records of reporting systems including reporting forms or platforms
used within the country;
2. Codes and case definitions (for vaccines);
3. Procedures and methods used for causality assessment; 04
4. Numbers and rates of reports within defined period(s), a list of ADRs following off-label
use, and a breakdown of reports that compares district or regional reporting activities for 05
the different products and populations involved in vigilance reporting.
5. Number of reports transmitted to WHO database along with the frequency of 06
submission.
07
References: 1. WHO Pharmacovigilance indicators: A practical manual for the assessment of
pharmacovigilance systems. World Health Organization. 2015, (85), (http://apps.who.int/ 08
medicinedocs/documents/s21970en/s21970en.pdf)
2. WHO guidelines on safety monitoring of herbal medicines in pharmacovigilance 09
systems. World Health Organization; 2004, (18), (http://apps.who.int/medicinedocs/
documents/s7148e/s7148e.pdf)
3. The importance of Pharmacovigilance: safety monitoring of medicinal products. World
Health Organization, 2002, (17), (http://apps.who.int/medicinedocs/pdf/s4893e/s4893e.
pdf)

Framework: Process

Rating Scale:  NOT IMPLEMENTED: No procedures exist for collection and assessment of ADRs
and AEs.
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps towards development
and establishment of procedures for collection and assessment of ADRs and AEs;
however no results yet exist.
 PARTIALLY IMPLEMENTED (PI): Procedures for collection and assessment of ADRs
and AEs were recently established by the NRA and are in early implementation phase
so no consolidated results exist yet.
 IMPLEMENTED (I): Standard procedures exist and are implemented for collection and
assessment of ADRs and AEs.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Page 132 of 340

WHO Global Benchmarking Tool (GBT)

Sub Indicator: VL04.02: Vigilance procedures and tools are in place for investigation, interpretation
of and response to ADRs and AEs.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the existence and implementation of standard procedures for 01
different vigilance activities related to investigation and interpretation of and response to
ADRs and AEs including: 02
1. Investigation and assessment of ADRs and AEs
2. Assessment of risk, analysis and evaluation of vigilance data, and identification of trends 03
3. Processes for signal detection;

Vigilance (Vl): Indicators and Fact Sheets

4. Use of statistical tools to calculate reporting disproportionalities such as Proportional
Reporting Ratio
5. Initiation of appropriate actions at the national or sub-national level when needed. For
this particular process, the assessor should verify that all vigilance events are screened
and triaged by staff that is qualified to assess their impact on public health and to
determine the subsequent steps needed. These steps may include one or more of the
following:
• Addition to national vigilance database
• Follow up or further analysis
• Referral for comprehensive investigation or systematic causality assessment
• Issue of safety alerts and/or batch or product recalls
• Other regulatory decisions including product withdrawal and/or revocation of marketing
authorization.
The assessor should verify that the above mentioned procedures are consistent with the 04
relevant national guidelines, which in turn are in coherence with WHO or other international
accepted guidance. The assessor should review records proving the actual implementation 05
of such procedures.
In addition, the assessor should verify that all vigilance data received are investigated and 06
interpreted on a regular basis and that any and all necessary actions are taken.
07
Objective: The objective of this sub-indicator is to ensure consistency of different activities related to
investigation of, interpretation of, and response to ADRs and AEs. 08
Requirement: Procedures and tools necessary for investigation, interpretation of and response to ADRs 09
and AEs.

Evidence to review: The assessor should ask for and review:

1. Procedures for taking action on recommendations arising from causality assessment;
2. Investigation reports;
3. Documentation of actions taken following reporting of vigilance events (ADRs, AE and
AEFIs);
4. Reports of notifications, data analyses, committee meetings, and other related records;
5. Documentation regarding the number of signals detected from national, regional or
linked international databases, if any;
6. Listing of the AEFIs reported (e.g., from the EPI to the NRA or vigilance center) over the
last two years.

References: 1. WHO Pharmacovigilance indicators: A practical manual for the assessment of

pharmacovigilance systems. World Health Organization. 2015, (85), (http://apps.who.int/
medicinedocs/documents/s21970en/s21970en.pdf)
2. WHO guidelines on safety monitoring of herbal medicines in pharmacovigilance
systems. World Health Organization; 2004, (18), (http://apps.who.int/medicinedocs/
documents/s7148e/s7148e.pdf)
3. The importance of Pharmacovigilance: safety monitoring of medicinal products.
World Health Organization, 2002, (17), (http://apps.who.int/medicinedocs/pdf/s4893e/
s4893e.pdf)

Page 133 of 340

WHO Global Benchmarking Tool (GBT)

Framework: Process

Rating Scale:  NOT IMPLEMENTED: No procedures exist for investigation of, interpretation of and
response to ADRs and AEs.
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps towards development and
establishment of procedures for investigation of, interpretation of and response to
ADRs and AEs; however no results yet exist.
 PARTIALLY IMPLEMENTED (PI): Procedures for investigation of, interpretation of and
response to ADRs and AEs were recently established by the NRA and are in early 01
implementation phase so no consolidated results exist yet.
 IMPLEMENTED (I): Standard procedures exist and are implemented for investigation 02
of, interpretation of and response to ADRs and AEs.
03
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will

Vigilance (Vl): Indicators and Fact Sheets

always apply for all benchmarked NRAs).

Sub Indicator: VL04.03: Standard procedures exist and are implemented for enforcement
of the national vigilance system.

Maturity Level: 4

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the existence and enactment of standard procedures for
different vigilance enforcement activities including, for example, product label changes,
restrictions of use, product withdrawals or suspensions, or implementation of intensive
monitoring. The assessor should confirm that, as part of these procedures, the NRA 04
notifies manufacturers (and vice versa) of significant safety and efficacy issues and that
reporting is kept up to date. 05
Objective: The objective of this sub-indicator is to ensure consistency of the vigilance function 06
through standard procedures for enforcement activities.
07
Requirement: Standard procedures for different vigilance enforcement activities.
08
Evidence to review: The assessor should ask for and review:
1. Standard procedures for enforcement activities relevant to vigilance system; 09
2. Records of vigilance inspections of MAHs and corrective actions and preventive actions
issued over the last two years;
3. Records of enactment of these standard procedures (e.g., decisions communicated to
manufacturers and MAHs).

References: 1. WHO guidelines on safety monitoring of herbal medicines in pharmacovigilance

Framework: Process

Rating Scale:  NOT IMPLEMENTED: No standard procedures exist for enforcement of the national
vigilance system.
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps towards development
and establishment of procedures for enforcement of the national vigilance system;
however no results yet exist.

Page 134 of 340

WHO Global Benchmarking Tool (GBT)

 PARTIALLY IMPLEMENTED (PI): Procedures for enforcement of the national vigilance

system were recently established by the NRA and are in early implementation phase
so no consolidated results exist yet.
 IMPLEMENTED (I): Standard procedures exist and are implemented for enforcement
of the national vigilance system.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).
01
Sub Indicator: VL04.04: Risk approach is considered throughout different vigilance activities,
including timely response to detected signals for risks or benefits. 02

Maturity Level: 3 03

Vigilance (Vl): Indicators and Fact Sheets

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that different vigilance activities are implemented with
consideration of risk approach. In general, the principle of risk proportionality should
be applied in the decision-making process for post-approval activities. This entails risk
assessment, including identification of different risks along with their analysis and
evaluation, risk control, via risk reduction, mitigation or acceptance, and regular risk
review, which may include adoption of any necessary changes in the risk management
process. Risk communication should also be considered across all steps of the risk
management process. Those products, events, or activities of high risk should be
given higher attention when establishing risk mitigation and control strategies and
procedures. 04
Objective: The objective of this sub-indicator is to ensure that a risk approach consideration is 05
contributing to vigilance function efficiency and effectiveness.
06
Requirement: Consideration of risk approach in different vigilance activities including timely response to
detected signals for risks or benefits. 07
Evidence to review: The assessor should ask for and review: 08
1. Procedures for risk approach consideration in different vigilance activities.
2. Examples of risk management process including risk assessment, risk control and risk 09
review.
3. Examples of risk management strategy for identified high risk products, events or
activities.

References: 1. WHO Pharmacovigilance indicators: A practical manual for the assessment of

pharmacovigilance systems. World Health Organization. 2015, (85), (http://apps.who.int/
medicinedocs/documents/s21970en/s21970en.pdf)
2. WHO guidelines on quality risk management, (55), (http://digicollection.org/
whoqapharm/p/about and http://apps.who.int/medicinedocs/en)

Framework: Process

Rating Scale: NOT IMPLEMENTED: Risk approach is not considered throughout different vigilance
activities.
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps towards
consideration of risk approach for different vigilance activities; however, no
results yet exist.
 PARTIALLY IMPLEMENTED (PI): Consideration of risk approach for different
vigilance activities was recently established and is in early implementation phase
so no consolidated results exist yet.
 IMPLEMENTED (I): Risk approach is considered throughout different vigilance
activities.

Page 135 of 340

WHO Global Benchmarking Tool (GBT)

Limitations and remarks:  The risk approach meant herein is that related to NRA. However, employment
of risk approach by manufacturers and MAHs is also of great value and should be
verified by the NRA as part of its implementation and enforcement of
vigilance activities.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: VL04.05: Staff access to information resources relevant to vigilance processes
(e.g., safety information sources and reference materials) is ensured. 01
Maturity Level: 1 02

Scope: 1. Medicines 03
2. Vaccines

Vigilance (Vl): Indicators and Fact Sheets

Description: The assessor should verify that staff responsible for different vigilance activities, including
members of any expert committees, have access to both internal and external sources of
information and reference materials, including literature and publications, to allow them
to make decisions on vigilance events. Information resources include, among others, all
scientific information concerning the use of medicinal products and the outcome of their
use, (i.e., quality, nonclinical and clinical data, including pharmacovigilance and pharmaco-
epidemiological data). Access to market information might be helpful as well (e.g.
medicines utilization statistics).

Objective: The objective of this sub-indicator is to ensure access of the responsible staff to vigilance
relevant information resources, reference materials and literature as an essential
contributing factor for effective performance of the function.
04
Requirement: Access of vigilance staff to relevant information resources
05
Evidence to review: The assessor should ask for and review:
1. Subscriptions to databases of scientific literature with up-to-date studies and 06
information on medical products safety and efficacy;
2. A list of electronic and printed materials consulted during the vigilance event analysis 07
and investigation (e.g., books, international guidelines, or international package inserts),
3. Access to pre-market, preclinical and clinical trial data; 08
4. Access to risk management plan.
5. Access to Periodic Safety Update Reports and Periodic Benefit Risk Evaluation Reports 09
6. Access to renewal dossiers

References: 1. WHO Pharmacovigilance indicators: A practical manual for the assessment of

Framework: Process

Rating Scale:  NOT IMPLEMENTED: Staff does not have access to information resources relevant to
vigilance processes (e.g., safety information sources and reference materials).
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps to guarantee staff access
to information resources relevant to vigilance processes (e.g., safety information
sources and reference materials); however no results exist yet.
 PARTIALLY IMPLEMENTED (PI): Staff access to information resources relevant to
vigilance processes (e.g. safety information sources and reference materials) was
recently established and is in early implementation phase; however, no consolidated
results exist yet.
 IMPLEMENTED (I): Staff has reasonable access to information resources relevant to
vigilance processes (e.g., safety information sources and reference materials).

Page 136 of 340

WHO Global Benchmarking Tool (GBT)

Limitations and remarks:  Efficacy is defined as the ability of a drug to produce the purported effect as
determined by scientific methods.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: VL04.06: The NRA has access to expert committees for review of serious emergent
safety concerns, when needed.

Maturity Level: 3 01
Scope: 1. Medicines 02
2. Vaccines
03
Description: The assessor should verify access of the NRA to a standing advisory committee or other

Vigilance (Vl): Indicators and Fact Sheets

group of experts for the investigation, assessment and analysis of serious vigilance events
(ADRs and AEFIs). The expert members of this committee might be internal or external
to the NRA; however in case of external staff, the appropriate measures should be taken
to ensure that relevant interests are declared and that that confidentiality is maintained.
The assessor should review terms of reference and written standard procedures defining
conditions and processes for the election of the members and for assuring their expertise.
Confidentiality, independence of the experts and prevention and handling of conflicts of
interest should also be regulated. The NRA should have access to any other experts when
needed.
The assessor should also verify NRA access to external expertise in specific medical fields
(e.g., teratology) when needed as part of this sub-indicator.

Objective: The objective of this sub-indicator is to ensure effective investigations, assessments and
analyses of serious vigilance events through the access to an expert committee (unless 04
extensive and sufficient internal expertise is available).
05
Requirement: Expert committee for review of serious vigilance events
06
Evidence to review: The assessor should ask for and review:
1. Terms of reference and standard procedures for the expert committee for review of 07
serious vigilance events.
2. Records of discussions and decisions of the expert committee for review of serious 08
vigilance events over the last two years or the last 5 to 10 meetings.
3. Procedures for convening the expert committee when needed to provide advice and 09
recommendations.
4. Procedures for follow-up and implementation of the expert committee
recommendations, including procedures for re-convening the expert committee as needed
to review progress.

References: 1. WHO guidelines on safety monitoring of herbal medicines in pharmacovigilance

Framework: Process

Rating Scale:  NOT IMPLEMENTED: The NRA does not have access to expert committees for review
of serious emergent safety concerns.
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps towards guaranteeing
access to expert committees for review of serious emergent safety concerns;
however no results exist yet.

Page 137 of 340

WHO Global Benchmarking Tool (GBT)

 PARTIALLY IMPLEMENTED (PI): NRA access to expert committees for review

of serious emergent safety concerns was recently established and is in early
implementation phase, so no consolidated results exist yet.
 IMPLEMENTED (I): The NRA has access to expert committees for review of serious
emergent safety concerns, when needed.

Limitations and remarks: There might be a limitation if the expert committee is not formalized or does not meet
frequently.
01
Sub Indicator: VL04.07: With respect to vigilance data, assessment of the risk-benefit balance
of medical products is regularly conducted. 02

Maturity Level: 4 03

Vigilance (Vl): Indicators and Fact Sheets

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that safety and/or effectiveness of medical products placed
on the market are regularly evaluated through periodic assessment of the risk-benefit
balance of medical products. Towards this end, the assessor should verify the availability
of risk-benefit analysis tools and processes, including descriptions of their application
to regulatory decision making. The assessor should also confirm that vigilance data are
informing such risk-benefits assessments, and that informed, science-based decisions
are taken when necessary. Risk-benefit assessment is an important regulatory tool
that highlights the risk-regulations such as withdrawal from the market or suspension
of license and sale. benefit of therapeutic products. Industry (i.e., the MAH) is primarily
responsible for developing risk-benefit assessments. 04
Product vigilance evaluation involves the ongoing assessment of the risk-benefit of a
product (including, for example, data from adverse reaction reports, clinical trials, meta- 05
analysis, and observational studies). These assessments provide the assurance that
benefits outweigh the risks for a given population during clinical trials and following 06
market authorization. The frequency of this evaluation varies significantly depending on
the maturity of the national vigilance system. In addition, the assessor should verify that 07
after the risk-benefit assessment of the medical products, the NRA is making well-justified
decisions based on scientifically sound evidence and is taking necessary regulatory in 08
response to the risks detected. The goals of the actions are to minimize risk and maintain
a favorable risk-benefit profile. With the current regulatory landscape, these interventions 09
generally fall within the following options:
• Issuance of a risk communication;
• Revision of market authorization (e.g., terms and conditions, naming, labelling, or
packaging);
• Further investigation by the regulator;
• Enforcement of

Objective: The objective of this sub-indicator is to ensure the sustained and effective evaluation of
medical products safety and effectiveness, and the ongoing evaluation of their risk- benefit
balance with full consideration of national and international vigilance data.

Requirement: Risk-benefit assessment of medical products with consideration of vigilance data

Evidence to review: The assessor should ask for and review:

1. Standard procedures relevant to the risk- benefit assessment of medical products;
2. Documented evidence of the consideration of the vigilance data in these processes;
3. Regulatory decisions and actions for maintaining the favorable risk-benefit balance of
the medical products placed on the market.

References:

Framework: Process

Page 138 of 340

WHO Global Benchmarking Tool (GBT)

Rating Scale:  NOT IMPLEMENTED: Assessment of risk-benefit balance of medical products is not
regularly conducted at all.
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps towards assessment of
risk-benefit balance of medical products on regular basis; however no results exist
yet.
 PARTIALLY IMPLEMENTED (PI): Assessment of risk-benefit balance of medical
products is regularly conducted; however consideration of vigilance data is not fully
utilized or consideration of vigilance data for risk-benefit assessment was recently
established and remains in early implementation phase. For these reasons, no 01
consolidated results yet exist.
 IMPLEMENTED (I): Risk-benefit balance of medical products is regularly conducted 02
with full consideration of vigilance data.
03
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will

Vigilance (Vl): Indicators and Fact Sheets

always apply for all benchmarked NRAs).

Sub Indicator: VL04.08: Active vigilance activities, as well as proactive monitoring programmes
(when needed) have been developed and implemented.

Maturity Level: 4

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the existence and implementation of active surveillance
systems as well as proactive vigilance systems for early detection of events. The goal is
to initiate corrective actions and preventive actions to manage use of medical products of
significant public health impact and medical products of potential high risk. 04
Passive surveillance means that no active measures are taken to look for AEs other than
the encouragement of health professionals and others to report safety concerns. Reporting 05
is entirely dependent on the initiative and motivation of the potential reporters. This is the
most common form of vigilance. It is commonly referred to as “spontaneous” or “voluntary” 06
reporting. In some countries, this form of reporting is mandatory. Active safety surveillance,
on the other hand, means that active measures are taken to detect AEs. This is managed 07
by active follow-up after treatment and the events may be detected by asking patients
directly or by screening patient records. Such follow-up is best done prospectively. 08
Active vigilance is sometimes very descriptively referred to as, “hot pursuit”. The most
comprehensive method is cohort event monitoring. Examples of this are the Intensive 09
Medicines Monitoring Programme in New Zealand and Prescription Event Monitoring in
England. Other methods used include the use of registers, record linkage and screening of
laboratory results in medical laboratories.
Active surveillance may entail several activities including signal detection and field
research. One of the most important activities in the vigilance system is the ability to
detect safety issues as quickly as possible. Safety management programs should be then
established. Active surveillance might be applied to all medical products or to selected high
risk medical products.
Proactive approach is mainly related to and based on the assessment of pre-market data
which could contribute to risk management, risk mitigation or risk minimization when
necessary.

Objective: The objective of this sub-indicator is to ensure that vigilance systems are implemented in
active and/or proactive manners, rather than reactively, when needed for protecting public
health.

Requirement: Active vigilance monitoring and surveillance programme.

Evidence to review: The assessor should ask for and review:

1. Standard procedures and mechanisms relevant to proactive vigilance systems;
2. Standard procedures and mechanisms relevant to active vigilance activities;
3. Risk management plans from the manufacturers.

Page 139 of 340

WHO Global Benchmarking Tool (GBT)

References: 1. WHO guidelines on safety monitoring of herbal medicines in pharmacovigilance

systems. World Health Organization; 2004, (18), (http://apps.who.int/medicinedocs/
documents/s7148e/s7148e.pdf)
2. WHO Pharmacovigilance indicators: A practical manual for the assessment of
pharmacovigilance systems. World Health Organization. 2015, (85), (http://apps.who.int/
medicinedocs/documents/s21970en/s21970en.pdf)
3. The importance of Pharmacovigilance: safety monitoring of medicinal products.
World Health Organization, 2002, (17), (http://apps.who.int/medicinedocs/pdf/s4893e/
s4893e.pdf) 01
Framework: Process 02

Rating Scale:  NOT IMPLEMENTED: No active surveillance or proactive vigilance systems are 03
implemented at all.

Vigilance (Vl): Indicators and Fact Sheets

 ONGOING IMPLEMENTATION (OI): The NRA is taking steps towards development of
procedures for active surveillance and/or proactive vigilance systems; however no
results exist yet.
 PARTIALLY IMPLEMENTED (PI): Procedures and mechanisms for active surveillance
and/or proactive vigilance system were recently established by the NRA and are in
early implementation phase so no consolidated results exist yet.
 IMPLEMENTED (I): Active surveillance and/or proactive vigilance system are
implemented based on well-established criteria.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Indicator: VL05 Mechanism in place to monitor regulatory performance and output.

04
Objective: The objective of this indicator is to ensure the effectiveness of the different medical- 05
product related vigilance activities, to measure relevant advancements, and to ensure that
performance indicators are established, implemented and verified on a regular basis. 06
Category: 09. Monitoring progress and assessing outcomes & impact 07
Sub Indicator: VL05.01: Vigilance information is used in timely manner to amend existing regulatory 08
decisions or to issue new regulatory decisions or actions.
09
Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that corrective regulatory decisions and actions (e.g.,
suspensions, recalls, updates of product leaflet, product withdrawals or revocations of
marketing authorization) are based on vigilance findings and are in accordance with
national regulations and WHO recommendations. Furthermore, the assessor should
verify that the regulatory actions related to the vigilance function are implemented and
monitored.

Objective: The objective of this sub-indicator is to ensure that the vigilance function provides the
scientific basis for regulatory decisions and actions and serves to maintain a favorable
risk-benefit balance for medical products placed on the market.

Requirement: Regulatory decisions and actions in response to vigilance information.

Evidence to review: The assessor should ask for and review:

1. List of regulatory decisions based on vigilance events in the last two years.
2. Examples of decisions and actions taken in response to vigilance events in the last two
years.

Page 140 of 340

WHO Global Benchmarking Tool (GBT)

3. Communications and publications of regulation decisions and actions that were

provided to different entities within or outside the NRA.
4. List of investigated, detected or analyzed signals in the last two years.

References: 1. WHO Pharmacovigilance indicators: A practical manual for the assessment of

pharmacovigilance systems. World Health Organization. 2015, (85), (http://apps.who.int/
medicinedocs/documents/s21970en/s21970en.pdf)
2. The importance of Pharmacovigilance: safety monitoring of medicinal products. World
Health Organization, 2002, (17), (http://apps.who.int/medicinedocs/pdf/s4893e/s4893e. 01
pdf)
02
Framework: Output
03
Rating Scale:  NOT IMPLEMENTED: no evidence exists for the confirmation that vigilance

Vigilance (Vl): Indicators and Fact Sheets

information is used to amend or issue regulatory decisions and consequent actions in
timely manner.
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps towards usage of vigilance
information to amend or issue regulatory decisions; however no results exist yet.
 PARTIALLY IMPLEMENTED (PI): The NRA started recently to use vigilance information
to amend or issue regulatory decisions; however, the practice is in the early
implementation stage and not yet consolidated.
 IMPLEMENTED (I): Evidence exists to confirm that vigilance information is used to
amend or issue regulatory decisions and consequent actions in timely manner.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: VL05.02: Performance indicators for vigilance activities are established 04
and implemented.
05
Maturity Level: 4
06
Scope: 1. Medicines
2. Vaccines 07
Description: The assessor should verify the existence and implementation of performance indicators for 08
different activities included under the vigilance function.
Specifically, the system should define key performance indicators (KPIs) along the entire 09
vigilance activity chain and all indicators should be adequately justified. For the purpose
of clarity and consistency, established KPIs should be supported with guidelines for
monitoring and maintenance of the KPIs. The guidelines in turn should be supported by
SOPs and tools that define the procedures to be used for monitoring and evaluating the
necessary, process optimizations should be introduced to avoid recurrence.
performance indicators and that define procedures and timelines for reviewing and revising
the indicators.
Performance indicators for activities included under the medical products vigilance system
should include:
• detection or receipt of vigilance events from within the reporting system;
• timely review, investigation and assessment of vigilance events;
• feedback on the vigilance activities to different stakeholders;
• storage and management of reported vigilance data;
• risk assessment, analysis and evaluation of vigilance data and identification of trends;
• initiation of appropriate actions at the national or sub-national level when needed; and
• risk management and risk communication plans.

Established KPIs might be qualitative, quantitative or combination of both. In general,

quantitative indicators are preferred to avoid bias or misinterpretation. However, qualitative
indicators are also accepted. Qualitative indicators may or may not include scoring or scaling
to render them semi-quantitative and thus more informative. The assessor should ensure that
indicators are measured on a regular basis to monitor progress and advancement.

Page 141 of 340

WHO Global Benchmarking Tool (GBT)

In addition, the assessor should verify measured indicators are analyzed to identify trends
or abnormalities. Justifications for any identified abnormalities should be provided; when
necessary, process optimizations should be introduced to avoid recurrence.

Objective: The objective of this sub-indicator is to ensure that a system, mechanism, or procedure
exists to require the NRA to establish performance indicators along the entire vigilance
chain. Additionally, the objective is to ensure that KPIs are actually contributing to
monitoring of regulatory performance, to measuring effectiveness of vigilance regulatory
activities, and to making any necessary adjustments or optimizations. 01
Requirement: KPIs for medical products vigilance activities 02

Evidence to review: The assessor should ask for and review: 03

1. Documents supporting the system, mechanism, or procedure compelling the NRA to

Vigilance (Vl): Indicators and Fact Sheets

establish and implement performance indicators along the entire vigilance activity chain.
2. Evidence that the performance indicators have been established and implemented, and
that the members of staff involved in the vigilance function are aware of the indicators and
the guidelines and SOPs used for monitoring and evaluating their performance.
3. The current performance indicators for vigilance activities
4. Analyses of the measured indicators along with the investigations done to identify
trends or abnormalities.
5. Documentation for follow-up of any observed abnormalities, including justifications
for any identified abnormalities as well as any process optimizations introduced to avoid
recurrence.

References: 1. WHO Pharmacovigilance indicators: A practical manual for the assessment of

pharmacovigilance systems. World Health Organization. 2015, (85), (http://apps.who.int/
medicinedocs/documents/s21970en/s21970en.pdf) 04
2. The importance of Pharmacovigilance: safety monitoring of medicinal products.
World Health Organization, 2002, (17), (http://apps.who.int/medicinedocs/pdf/s4893e/ 05
s4893e.pdf)
3. World Health Organization. European Observatory on Health Systems and Policies. 06
Performance measurement for health system improvement: experiences, challenges
and Prospects, (125), (http://www.who.int/management/district/performance/ 07
PerformanceMeasurementHealthSystemImprovement2.pdf)
4. World Health Organization. European Observatory on Health Systems and Policies. 08
Health System Performance Comparison: an agenda for policy, information and research,
(126), (http://www.euro.who.int/__data/assets/pdf_file/0009/244836/Health-System- 09
Performance-Comparison.pdf)

Framework: Output

Rating Scale:  NOT IMPLEMENTED: There are no KPIs for vigilance activities.
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted KPIs for vigilance
activities but they have not yet been reported.
 PARTIALLY IMPLEMENTED (PI): The NRA has developed KPIs for vigilance activities
and has been applying them for less than two year or they have not covered all critical
steps.
 IMPLEMENTED (I): The NRA has established and implemented KPIs for vigilance
activities. The indicators are reviewed regularly, and appropriate actions are taken and
decisions made.

Page 142 of 340

WHO Global Benchmarking Tool (GBT)

indicators should be Specific, Measurable, Achievable, Realistic, and Time-bound (i.e.,

“SMART”).
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Indicator: VL06 Mechanism exists to promote transparency, accountability

and communication.

Objective: The objective of this indicator is to ensure communication within the NRA, transparency 01
and outreach to public, regional and international partners, and accountability of the NRA.
Additionally, these contribute to mutual understanding and involvement of all stakeholders 02
that are relevant to the vigilance system and raise the confidence in the regulatory system.
03
Category: 08. Transparency, accountability and communication

Vigilance (Vl): Indicators and Fact Sheets

Sub Indicator: VL06.01: Vigilance activities and relevant feedback are appropriately
communicated to the public.

Maturity Level: 2

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that a proper mechanism is available, enacted and used
to demonstrate that regular feedback and information on the vigilance system, in
particular, serious events and clusters of AEs, are communicated appropriately to the
public (including patients, parents and caregivers). Furthermore, guidance to the public
community on ways and approaches to manage any potential risks, may contribute 04
significantly to risk reduction or elimination. Investigation reports of public concerns and
summaries of these reports should be made available to the public. Regular publications 05
(e.g., bulletins) or awareness sessions would serve the above-mentioned communication
mechanism. 06
In addition, considering the fact that medical professionals and the public community play
a crucial role in the vigilance system, activities for raising their awareness is necessary. The 07
assessor should verify that regular meetings, trainings, educational sessions, educational
materials, and media aids are provided for a wide audience, including undergraduate, 08
postgraduate medical, pharmacy and nursing students, staff and healthcare providers
in public and private sectors, staff from immunization programs, and the general public 09
community.

Objective: The objective of this sub-indicator is to ensure the implementation and effectiveness of
the overall vigilance system, in particular, to ensure that the risk management and risk
communication plans are regularly and systematically implemented (i.e., not on ad hoc
basis).

Requirement: Communication mechanism for the public community on vigilance events, especially
serious ones.

Evidence to review: The assessor should ask for and review:

1. Records of communications to the public community (e.g., social media, newsletters,
and websites);
2. Information bulletins and documentation of public awareness sessions and campaigns;
3. Published alerts, assessments and investigation reports.

References: 1. WHO guidelines on safety monitoring of herbal medicines in pharmacovigilance

Page 143 of 340

WHO Global Benchmarking Tool (GBT)

3. The importance of Pharmacovigilance: safety monitoring of medicinal products. World

Health Organization, 2002, (17), (http://apps.who.int/medicinedocs/pdf/s4893e/s4893e.pdf)

Framework: Output

Rating Scale:  NOT IMPLEMENTED: Vigilance activities and feedback are not communicated to the
public community.
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps towards sharing of
vigilance activities and providing feedback to the public community, but no results yet 01
exist.
 PARTIALLY IMPLEMENTED (PI): The NRA started recently to share vigilance activities 02
and provide feedback to the public community; however, it has only limited experience
or a limited number of documented events. 03
 IMPLEMENTED (I): Vigilance activities and feedback, especially for serious events

Vigilance (Vl): Indicators and Fact Sheets

and clusters of AEs, are appropriately communicated to the public community.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: VL06.02: Mechanism for regular feedback to all stakeholders on vigilance events
exists and is complemented with a risk communication plan.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that a proper mechanism is available, enacted and used to 04
demonstrate that information on vigilance system, especially serious events and clusters
of AEs, are shared among all stakeholders engaged in medical products vigilance. Key 05
players include the NRA or the vigilance center, manufacturers, MAHs, purchasers and
supply divisions (including the EPI in case of vaccines), distributors, and other health care 06
professionals. The assessor should verify that the process for feedback is established,
endorsed and followed down to health facility level. The assessor should note that the 07
regularity of outreach to the stakeholders depends on the availability of data, decisions, or
actions. 08
In addition, the assessor should verify that regular formal or official communications and
meetings take place among above-mentioned key players when dealing with vigilance 09
events. The assessor should check that periodic feedback is offered from the NRA or
the responsible authority to all levels country-wide, especially to medical and health
professionals.
Particularly for vaccines, the NRA should be informed about in-country vaccine safety and
performance information including vaccine preventable diseases surveillance data, EPI
coverage data, information on number of doses shipped and administered, seroprevalence
study reports, outbreak investigation reports, EPI reports on vaccine supply and storage
(i.e., cold chain), special instructions, and recall notifications.

Objective: The objective of this sub-indicator is to ensure the implementation and effectiveness of the
overall vigilance system, particularly the risk management and risk communication plans.

Requirement: Risk communication plan as a part of an overall mechanism for regular feedback to all
stakeholders on vigilance system

Evidence to review: The assessor should ask for and review:

1. Risk communication plan and procedures for communication with different stakeholders
involved in the vigilance system.
2. Examples of shared information among those stakeholders.
3. Records of communication (e.g., social media, newsletters, websites, or publications)
among the NRA and those stakeholders.
4. Records of regular meetings among the vigilance relevant stakeholders.

Page 144 of 340

WHO Global Benchmarking Tool (GBT)

References: 1. WHO guidelines on safety monitoring of herbal medicines in pharmacovigilance

systems. World Health Organization; 2004, (18), (http://apps.who.int/medicinedocs/
documents/s7148e/s7148e.pdf)
2. WHO Pharmacovigilance indicators: A practical manual for the assessment of
pharmacovigilance systems. World Health Organization. 2015, (85), (http://apps.who.int/
medicinedocs/documents/s21970en/s21970en.pdf)
3. The importance of Pharmacovigilance: safety monitoring of medicinal products. World
Health Organization, 2002, (17), (http://apps.who.int/medicinedocs/pdf/s4893e/s4893e.pdf)
01
Framework: Output
02
Rating Scale:  NOT IMPLEMENTED: Feedback on vigilance events and activities are not
communicated to all relevant stakeholders. 03
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps towards providing

Vigilance (Vl): Indicators and Fact Sheets

feedback on vigilance events and activities to all relevant stakeholders, but no risk
communication plan or results yet exist.
 PARTIALLY IMPLEMENTED (PI): The NRA started recently to provide feedback on
vigilance events and activities to all relevant stakeholders; however, the NRA has only
limited experience or it has a limited number of documented events.
 IMPLEMENTED (I): Feedback on vigilance events and activities are communicated
to all relevant stakeholders. These activities are based on a well-developed risk
communication plan.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: VL06.03: Vigilance data and findings are shared with relevant regional
and international partners. 04
Maturity Level: 3 05
Scope: 1. Medicines 06
2. Vaccines
07
Description: The assessor should verify that national vigilance data and findings and conclusions are
shared with relevant regional and international partners (e.g. WHO Collaborating Centers). 08
The assessor should confirm that the national vigilance staff have access to the regional
and international tools (e.g., databases) and should verify that these tools facilitate 09
information sharing and analysis. In addition, the assessor should confirm that there is
two-way collaboration and interaction among the national vigilance system and relevant
regional and international partners.

Objective: The objective of this sub-indicator is to ensure medical products safety and efficacy
through sharing of global concerns and relevant vigilance data and findings among the
national and international communities; such sharing will benefit all stakeholders. This
sharing also fosters regulatory convergence and harmonization.

Requirement: Regional and international sharing of vigilance data and findings.

Evidence to review: The assessor should ask for and review:

1. Proof of membership of the national vigilance system in relevant regional and
international organizations or partnerships.
2. Proof of regular communications and interactions with those regional and international
partners.
3. Proof of engagement of national vigilance system in regional or international meetings,
conferences, or symposia.
4. Publication of vigilance data in scientific publications as a means of communication
with international and regional partners.
5. List of serious ADRs and AEFIs submitted to WHO Collaborating Centers in the last
two years.

Page 145 of 340

WHO Global Benchmarking Tool (GBT)

References: 1. WHO guidelines on safety monitoring of herbal medicines in pharmacovigilance

systems. World Health Organization; 2004, (18), (http://apps.who.int/medicinedocs/
documents/s7148e/s7148e.pdf)
2. WHO Pharmacovigilance indicators: A practical manual for the assessment of
pharmacovigilance systems. World Health Organization. 2015, (85), (http://apps.who.int/
medicinedocs/documents/s21970en/s21970en.pdf)
3. The importance of Pharmacovigilance: safety monitoring of medicinal products. World
Health Organization, 2002, (17), (http://apps.who.int/medicinedocs/pdf/s4893e/s4893e.
pdf) 01
Framework: Output 02

Rating Scale:  NOT IMPLEMENTED: Vigilance data and findings are not shared with relevant regional 03
and international partners.

Vigilance (Vl): Indicators and Fact Sheets

 ONGOING IMPLEMENTATION (OI): The NRA is taking steps towards sharing of
vigilance data and findings with other regional and international partners, but results
do not yet exist.
 PARTIALLY IMPLEMENTED (PI): The NRA started recently to share vigilance data and
findings with other regional and international partners; however, it has only limited
experience or a limited number of documented events.
 IMPLEMENTED (I): Vigilance data and findings are shared with relevant regional and
international partners.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

04
05
06
07
08
09

Page 146 of 340

WHO Global Benchmarking Tool (GBT)

04
Market Surveillance and
02
03
04
01

Market Surveillance and Control (MC): Indicators and Fact Sheets

Control (MC): Indicators
and Fact Sheets

WHO Global
Benchmarking
Tool (GBT)
for Evaluation of
National Regulatory
System of Medical
Products 05
06
07
08
09

Page 147 of 340

WHO Global Benchmarking Tool (GBT)

04. Market Surveillance and Control (MC):

Indicators and Fact Sheets

Function:
04 — MARKET SURVEILLANCE AND CONTROL (MC)
01
Description: Market surveillance and control function plays a crucial role in assuring medical 02
products consumer safety since its objective is to ensure compliance of the products
placed on the market with pre-set criteria for quality, safety and efficacy (i.e., verify 03
compliance with marketing authorization and good practices guidelines). Market
surveillance and control function activities are primarily concerned with four themes: (1) 04
control of import activities, (2) prevention and detection of and response to substandard

Market Surveillance and Control (MC): Indicators and Fact Sheets

and falsified medical products, (3) market surveillance program for monitoring the
quality of medical products throughout the supply chain, and (4) control of promotional,
marketing and advertising activities. The aforementioned activities may or may not be
undertaken by a single entity (e.g., organization, division, or department).

A general limitation to one of the market surveillance and control function activities
exists in countries where advertisement is not allowed or is restricted to specific
medical products (e.g. over-the-counter medical products). However, in all cases,
control of marketing and promotional materials (i.e., in its wider meaning that includes
promotion to healthcare professionals) will always apply.

Indicator: MC01 Legal provisions, regulations and guidelines required to define regulatory
framework of market surveillance and control activities.

Objective: The objective of this indicator is to ensure that market surveillance and control activities
are backed up by a comprehensive set of legal provisions, regulations and guidelines
that provide the necessary mandate to implement all activities related to this regulatory
function.

Category: 01. Legal provisions, regulations and guidelines

05
Sub Indicator: MC01.01: Legal provisions and regulations are in place with respect to import activities
including permanent regulatory intervention at designated entry and exit ports where 06
medical products are being moved.
07
Maturity Level: 1
08
Scope: 1. Medicines
2. Vaccines 09

Description: The assessor should verify that the medical products importation activities should
be conducted in conformity with the mandate promulgated under the relevant legal
provisions, regulations and guidelines as implemented and enforced by the National
Regulatory Authority (NRA). These activities may be conducted in collaboration
with other relevant authorities, e.g. customs. These mandates should require that
transactions relating to importation of consignments of medical products be conducted
by licensed entities and that good storage and distribution practices be followed.
In exceptional cases (e.g., emergency situations for the public health interest), the
NRA may retain the discretionary power to waive some licensing or authorization
requirements for some medical product consignments. Such waivers should be
properly justified and should not compromise patient safety or the risk-benefit balance
of the medical products.
Importation of medical products should be channeled exclusively through designated
air, sea, or land ports. These designated entry and exit ports (i.e., air, sea, or land) should
have permanent capacity for the regulatory intervention that is essential to ensure

Page 148 of 340

WHO Global Benchmarking Tool (GBT)

compliance with and enforcement of the relevant mandates. When justified by the
workload, either full-time or part-time inspectors may be stationed at one or more of
these designated ports.

Objective: The objective of this sub-indicator is to ensure that the NRA has the necessary
mandate, supported by the appropriate legal provisions and regulations, to control
import activities of medical products. This control will, in turn, contribute to a positive
public health impact. In the interest of public health, import activities related to medical
products, unlike that of ordinary commodities, need to be controlled and under the 01
proper oversight of the NRA. Hence, the NRA needs to be supported with the mandate
needed to assume this responsibility. Import activities should then comply with this 02
mandate, including the relevant legal provisions, regulations and guidelines. The
activities should be supported by a regulatory presence at the entry and exit ports to 03
enforce these mandates.
04
Requirement: Legal provisions and regulations relevant to import activities of all medical products to or

Market Surveillance and Control (MC): Indicators and Fact Sheets

from the country.

Evidence to review: The assessor should ask for and review:

1. Legal provisions relevant to medical products import activities.
2. Regulations relevant to medical products import and.
3. Guidelines relevant to medical products import activities including good storage and
good distribution practices.
4. Evidence of permanent regulatory intervention at the entry and exit ports.

References: 1. Guidelines on import procedures for pharmaceutical products, World Health

Organization (WHO), (22), (http://digicollection.org/whoqapharm/p/about and http://apps.
who.int/medicinedocs/en)

Framework: Structure/Foundation/Input.

Rating Scale:  NOT IMPLEMENTED (NI): There are no legal provisions or regulations to satisfy the
requirement of the sub- indicator.
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps towards establishment
of legal provisions and regulations to satisfy the requirement of this sub-indicator;
however no results exist so far.
 PARTIALLY IMPLEMENTED (PI): Legal provisions and regulations to satisfy the 05
requirement of this sub-indicator were recently established, and are in an early
implementation phase, so no documented results exist so far. 06
 IMPLEMENTED (I): There are legal provisions and regulations which satisfy the
requirement of the sub-indicator and which are actually implemented and enforced. 07
Limitations and remarks:  Permanent regulatory intervention at the entry and exit ports may or may not entail 08
physical presence of regulators at these ports. Other acceptable alternatives include,
but are not limited to, close cooperation between regulators and custom authorities 09
with proper documentation and authorization controls or electronic systems.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: MC01.02: Legal provisions and regulations authorize market surveillance and control
activities which include product sampling from different points of the supply chain.

Maturity Level: 1

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the existence and implementation of legal provisions and
regulations which establish a market surveillance program. Such market surveillance
should normally include sampling of different medical products from different points

Page 149 of 340

WHO Global Benchmarking Tool (GBT)

across the supply chain, from the manufacturer, through the distributors and wholesalers,
and up to the last point of sale or dispensing. In addition, the assessor should ensure that
market control activities extend to the internet sales of medical products. It should
be noted that testing is not limited to laboratory testing, but is extended to all types of
testing including innovative technologies.

Objective: The objective of this sub-indicator is to ensure the quality, safety and efficacy of
medical products placed on the market. The ultimate goal of the overall medical
products regulatory system is to ensure public accessibility and affordability of safe, 01
effective and high quality products. Hence, a market surveillance program, combined
with market control activities, is necessary to regularly check the quality of medical 02
products available on the market and to take any necessary regulatory actions.
Market surveillance and control activities should be based on and supported by an 03
appropriate legal mandate.
04
Requirement: Legal provisions, regulations and guidelines relevant to market surveillance and control

Market Surveillance and Control (MC): Indicators and Fact Sheets

activities of all medical products.

Evidence to review: The assessor should ask for and review:

1. Legal provisions relevant to surveillance program which includes sampling and testing
of samples of medical products.
2. Regulations relevant to surveillance program which includes sampling and testing of
samples of medical products.
3. Guidelines relevant to surveillance program which includes sampling and testing of
samples of medical products.
4. Legal provisions, regulations or guidelines relevant to market control of internet sales
of medical products.

References: 1. Guidelines on the conduct of surveys of the quality of medicines, World Health
Organization (WHO), (86), (http://apps.who.int/medicinedocs/en/q/ and http://
digicollection.org/whoqapharm/)
2. WHO guidelines for sampling of pharmaceutical products and related materials, World
Health Organization (WHO), (31), (http://digicollection.org/whoqapharm/p/about and
http://apps.who.int/medicinedocs/en)

Framework: Structure/Foundation/Input.
05
Rating Scale:  NOT IMPLEMENTED (NI): There are no legal provisions or regulations to satisfy the
requirement of the sub- indicator. 06
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps towards establishment
of legal provisions and regulations to satisfy the requirement of this sub-indicator; 07
however no results exist so far.
 PARTIALLY IMPLEMENTED (PI): Legal provisions and regulations to satisfy the 08
requirement of this sub-indicator were recently established, and are in an early
implementation phase, so no documented results yet exist. 09
 IMPLEMENTED (I): There are legal provisions and regulations which satisfy the
requirement of the sub-indicator and which are actually implemented and enforced.

Limitations and remarks: I nternet sales might be prohibited in some countries. In this case, the sub-indicator
would not apply to internet sales provided that some legal provisions or regulations
are in place to prohibit internet sales.
 Sampling activities at manufacturers, distributers and points of sale or dispensing
might be undertaken during a regulatory inspection programme.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: MC01.03: Legal provisions and regulations address the role of NRA in dealing with
substandard or falsified (SF) medical products.

Maturity Level: 1

Page 150 of 340

WHO Global Benchmarking Tool (GBT)

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that legal provisions, regulations and guidelines that address the
role of the NRA in dealing with SF medical products are available and enacted. Examples of
possible actions to be taken include: suspension of a drug’s marketing authorization, recall
of certain batches, a warning in national drug bulletins, or a separate warning sent out to a
list of institutions and key persons that deal with or prescribe pharmaceutical products. The
legislation should provide for adequate and proportional sanctions, penalties and prosecutions 01
for violations of the applicable legislation. The assessor should review the guidance published
for all categories of stakeholders and find out if the guidance covers the scope of the 02
applicable legislation and regulations. Consistency with World Health Organization (WHO)
guidance should be checked as well and any differences should be identified. 03

Objective: The objective of this sub-indicator is to ensure that the NRA is empowered with the proper 04
legal provisions and regulations to work on prevention, detection and response to SF

Market Surveillance and Control (MC): Indicators and Fact Sheets

medical products.

Requirement: Legal provisions, regulations and guidelines addressing products and personnel involved
in SF medical products.

Evidence to review: The assessor should ask for and review:

1. Legal provisions relevant to the role of the NRA in dealing with SF medical products;
2. Regulations relevant to the role of the NRA in dealing with SF medical products;
3. Guidelines relevant to the role of the NRA in dealing with SF medical products.

References: 1. Addressing the barriers to effective monitoring, reporting and containment of spurious/
substandard/falsely-labelled/ falsified/counterfeit medical products. A report prepared
for the Medicines Transparency Alliance, Philippines: 2016, (88), (http://apps.who.int/
medicinedocs/documents/s22331en/s22331en.pdf)
2. SF- Frequently asked questions. Scope, scale and harm. How big is the problem of SF
medical products?, (87), (https://www.who.int/medicines/regulation/ssffc/faq-ssffc_1-10/en/)
3. Substandard and falsified medical products. Fact sheet, (134), (https://www.who.int/en/
news-room/fact-sheets/detail/substandard-and-falsified-medical-products)
4. WHO Member State Mechanism on substandard/spurious/falsely-labelled/falsified/
counterfeit medical products, A70/23, appendix 3, Working Definitions, (128), (https://www.
who.int/medicines/regulation/ssffc/A70_23-en1.pdf) 05
5. Recommendations for health authorities to detect and deal with actions, activities and
behaviors that result in substandard/spurious/falsely-labelled/falsified/counterfeit medical 06
products, A68/33, annex 1, (129), (http://apps.who.int/gb/ebwha/pdf_files/WHA68/
A68_33-en.pdf) 07
6. Guidance on developing a national plan for preventing, detecting and responding to actions,
activities and behaviors that result in SF medical products. WHO (document A70/23), (130), 08
(https://www.who.int/medicines/regulation/ssffc/mechanism/A70_23-en6-14.pdf)
7. WHO Global Surveillance and Monitoring System for substandard and falsified medical 09
products, (133), (https://www.who.int/medicines/regulation/ssffc/publications/gsms-
report-sf/en/)

Framework: Structure/Foundation/Input.

Page 151 of 340

WHO Global Benchmarking Tool (GBT)

Limitations and remarks:  Formal guidelines might not exist, however relevant guidance may be provided in
other types of documentation (e.g., regulations).
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator
will always apply for all benchmarked NRAs).

Sub Indicator: MC01.04: Legal provisions and regulations exist for the control of promotion,
marketing and advertising of medical products to avoid communication of false
or misleading information.
01
Maturity Level: 2
02
Scope: 1. Medicines
2. Vaccines 03

Description: The assessor should verify the availability of legal provisions, regulations and 04
guidelines which set the rules for control of promotional, marketing and advertising

Market Surveillance and Control (MC): Indicators and Fact Sheets

activities of medical products. These rules should ensure that communication of false
or misleading information to health professionals, the public, or any other stakeholders
is avoided.

Objective: The objective of this sub-indicator is to ensure that medical professionals and the public
have access to accurate data about the medical products through different promotional,
marketing and advertising activities. Inappropriate promotion and advertisement of
medicines may contribute to the irrational or incorrect use of medicinal products. The
impact of accurate and science-based promotional activities is related to the existence
of trustworthy and accessible information sources and the level of medical knowledge
of the population. Hence, the control of promotion and advertisement of medical products
is necessary and should be sustained by a legal and regulatory framework.

Requirement: Legal provisions, regulations and guidelines for the control of promotion, marketing
and advertising of medical products.

Evidence to review: The assessor should ask for and review:

1. Legal provisions relevant to control of promotion, marketing and advertising of medical
products.
2. Regulations relevant to control of promotion, marketing and advertising of medical
products. 05
3. Guidelines relevant to control of promotion, marketing and advertising of medical
products. 06
References: 1. Drug Promotion: Push, Promote or Educate? Essential Drugs Monitor, No. 020, 1995, 07
World Health Organization (WHO), (91), (http://apps.who.int/medicinedocs/documents/
s21265en/s21265en.pdf) 08
2. Ethical Criteria for the Promotion, Advertisement, and Publicity of Medicines - PANDRH
Series Technical Document No 12 (Pan American Network for Drug Regulatory 09
Harmonization), (94), (http://apps.who.int/medicinedocs/documents/s22161en/s22161en.pdf)
3. CIOMS/WHO Meeting on Ethical Criteria for Medicinal Drug Promotion. Essential Drugs
Monitor No. 017, 1994, (90), (http://apps.who.int/medicinedocs/documents/s21142en/
s21142en.pdf)
4. Ethical Criteria for Medicinal Drug Promotion, World Health Organization, 1988, (13),
(http://apps.who.int/medicinedocs/documents/whozip08e/whozip08e.pdf)
5. Drug Promotion - What We Know, What We Have Yet to Learn - Reviews of Materials in
the WHO/HAI Database on Drug Promotion, World Health Organization (WHO), (89), (http://
apps.who.int/medicinedocs/pdf/s8109e/s8109e.pdf)
6. Cross-border Advertising, Promotion and Sale of Medical Products Using the Internet.
WHA Resolution; Fifty-First World Health Assembly, WHA51.9, (93), (http://apps.who.int/
medicinedocs/documents/s21471en/s21471en.pdf)
7. Alternative Regulatory Models for Pharmaceutical Promotions Involving Civil Society and
Other Non-Government Stakeholders, Medicines Transparency Alliance, (96), (http://apps.
who.int/medicinedocs/documents/s22327en/s22327en.pdf)

Page 152 of 340

WHO Global Benchmarking Tool (GBT)

8. Educational initiatives for medical and pharmacy students about drug promotion: an
international cross-sectional survey. Geneva. World Health Organization, (92), (http://apps.
who.int/medicinedocs/pdf/s8110e/s8110e.pdf)

Framework: Structure/Foundation/Input.

Market Surveillance and Control (MC): Indicators and Fact Sheets

enforced.

Sub Indicator: MC01.05: Legal provisions and regulations exist for placement of a product’s
unique identification number on its outer packaging.

Maturity Level: 4

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the availability of legal provisions and regulations setting
the rules for placement of a unique identification number on the outer package of each
medical product. Legal provisions and regulations should indicate that an identification
number is assigned to each product that receives marketing authorization.

Objective: The objective of this sub-indicator is to ensure the placement of a unique identification 05
number on each outer packaging in order to facilitate tracking and tracing activities
of medical products throughout the supply chain. These activities, in turn, support the 06
detection and response activities related to SF medical products.
07
Requirement: Legal provisions, regulation and guidelines for the placement of product’s unique
identification number on outer packaging. 08
Evidence to review: The assessor should ask for and review: 09
1. Legal provisions relevant to placement of unique identification number on outer
packaging of medical products.
2. Regulations relevant to placement of unique identification number on outer packaging of
medical products.
3. Guidelines relevant to placement of unique identification number on outer packaging of
medical products.

References: 1. WHO good manufacturing practices for pharmaceutical products: main principles, (51),
(http://digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en)
2. WHO good distribution practices for pharmaceutical products, World Health
Organization (WHO), (27), (http://digicollection.org/whoqapharm/p/about and http://apps.
who.int/medicinedocs/en)
3. Existing technologies and “track and trace” models in use and to be developed by
Member States. Appendix 2. World Health Organization. Executive Board B 138/40, (101),
(http://apps.who.int/gb/ebwha/pdf_files/EB138/B138_40-en.pdf)

Page 153 of 340

WHO Global Benchmarking Tool (GBT)

4. WHO Member State Mechanism on substandard/spurious/falsely-labelled/falsified/

counterfeit medical products, A70/23, appendix 2, Available authentication technologies
for the prevention and detection of SSFFC medical products, (139), (http://apps.who.int/
gb/ebwha/pdf_files/WHA70/A70_23-en.pdf)

Framework: Structure/Foundation/Input.

Rating Scale:  NOT IMPLEMENTED (NI): There are no legal provisions or regulations to satisfy the
requirement of the sub- indicator. 01
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps towards establishment
of legal provisions and regulations to satisfy the requirement of this sub-indicator; 02
however no results exist so far.
 PARTIALLY IMPLEMENTED (PI): Legal provisions and regulations to satisfy 03
the requirement of the sub-indicator were recently established and are in early
implementation phase so no documented results yet exist. 04
 IMPLEMENTED (I): There are legal provisions and regulations which satisfy the

Market Surveillance and Control (MC): Indicators and Fact Sheets

requirement of the sub-indicator and which are actually implemented and enforced.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: MC01.06: Guidelines exist for importers that specify the format and content
of the relevant applications and procedures to receive the necessary authorizations
or permissions.

Maturity Level: 2

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the existence and implementation of guidelines for importers
that specify the format and content of the relevant applications and procedures to receive
the necessary authorizations or permissions.

Objective: The objective of this sub-indicator is to ensure the existence of guidelines for import
activities as a tool for ensuring the quality and effectiveness of NRA’s actions of surveillance
and market control. The guidelines are necessary to establish clearly the rules that importers 05
have to follow to obtain the necessary authorizations or permissions from the NRA.
06
Requirement: Guidelines for the format and content of import authorizations or permissions.
07
Evidence to review: The assessor should ask for and review:
1. Guidelines for applicants on the format and content of import authorizations or 08
permissions
09
References: 1. Guidelines on import procedures for pharmaceutical products, World Health
Organization (WHO), (22), (http://digicollection.org/whoqapharm/p/about and http://apps.
who.int/medicinedocs/en)

Framework: Structure/Foundation/Input.

Rating Scale:  NOT IMPLEMENTED (NI): There are no guidelines to satisfy the requirement of the
sub-indicator.
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps towards establishment of
guidelines to satisfy the requirement of this sub-indicator; however no results exist so far.
 PARTIALLY IMPLEMENTED (PI): Guidelines to satisfy the requirement of the sub-
indicator were recently established and are in early implementation phase so no
documented results yet exist.
 IMPLEMENTED (I): There are guidelines which satisfy the requirement of the sub-
indicator and which are actually implemented and enforced.

Page 154 of 340

WHO Global Benchmarking Tool (GBT)

Sub Indicator: MC01.07: Guidelines exist on the recall, storage and disposal of SF medical
products.

Maturity Level: 2 01
Scope: 1. Medicines 02
2. Vaccines
03
Description: The assessor should verify the existence of national guidelines for relevant entities on
the best practices for effective recall of SF medical products. The guidelines should cover 04
recalls throughout the supply chain up to the desired point (e.g. distributor, wholesaler,

Market Surveillance and Control (MC): Indicators and Fact Sheets

or point of sale or use) so that the detected SF medical products are removed from the
supply chain. In addition, the assessor should check if a guideline on the best practices for
handling recalled SF medical products includes provisions for safe storage and disposal
that provide a high degree of assurance that the detected SF medical products will not
be re-introduced to the supply chain. Furthermore, the guidelines should ideally provide
guidance on the ways by which recall effectiveness is measured in terms of time, recalled
units and other relevant aspects.

Objective: The objective of this sub-indicator is to ensure the existence of guidelines for recall,
storage and disposal of SF medical products. These guidelines should ideally reflect the
NRA thinking and provide guidance to the responsible entity about effective actions that
prevent re-introduction of detected SF medical products into the market.

Requirement: Guidelines for recall, storage and disposal of SF medical products.

Evidence to review: The assessor should ask for and review:

1. Guidelines relevant to recall and safe disposal of SF medical products.

References: 1. WHO good manufacturing practices for pharmaceutical products: main principles, (51),
(http://digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en) 05
2. WHO good distribution practices for pharmaceutical products, World Health
Organization (WHO), (27), (http://digicollection.org/whoqapharm/p/about and http://apps. 06
who.int/medicinedocs/en)
3. Guidelines for Safe Disposal of Unwanted Pharmaceuticals in and after 07
Emergencies, (97), (http://apps.who.int/medicinedocs/pdf/whozip51e/whozip51e.pdf)
08
Framework: Structure/Foundation/Input.
09
Rating Scale:  NOT IMPLEMENTED (NI): There are no guidelines to satisfy the requirement
of the sub-indicator.
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps towards
establishment of guidelines to satisfy the requirement of the sub-indicator however
no results are yet exist.
 PARTIALLY IMPLEMENTED (PI): Guidelines to satisfy the requirement of the
sub-indicator were recently established and are in early implementation phase
so no documented results yet exist.
 IMPLEMENTED (I): There are guidelines which satisfy the requirement of the
sub-indicator and which are actually implemented and enforced.

Page 155 of 340

WHO Global Benchmarking Tool (GBT)

Indicator: MC02 Arrangement for effective organization and good governance.

Objective: The objective of this indicator is to ensure the implementation of effective organization
and good governance practices at the organizational structures in charge of establishment
licensing activities, which in turn contributes to effective and efficient functioning of the
market surveillance and control activities.

Category: 02. Organization and governance 01

Sub Indicator: MC02.01: There is a defined structure, with clear responsibilities, to conduct 02
market surveillance and control activities.
03
Maturity Level: 2
04
Scope: 1. Medicines

Market Surveillance and Control (MC): Indicators and Fact Sheets

2. Vaccines

Description: The assessor should identify the organization with responsibility to establish, implement
and maintain the market surveillance and control regulatory function, including the specific
organizational structures assigned the different relevant activities. Responsibilities, duties
and roles of these structures should be clearly defined and documented. If more than one
organizational structure is involved, the assessor should check the ways and approaches
by which coordination among these structures takes place.

Objective: The objective of this sub-indicator is to ensure effective organization and good governance
of market surveillance and control activities and that these activities are taken over by
defined organizational structures with clear roles and responsibilities.

Requirement: Roles and responsibilities of the structures in charge of market surveillance and control
activities.

Evidence to review: The assessor should ask for and review:

1. Organization chart of the institution responsible for the implementation of market
surveillance and control activities. The chart and related information should identify the
particular structures responsible for implementing the function.
2. Documentation clarifying roles and responsibilities of the structures implementing 05
market surveillance and control activities. These may include administrative decrees,
terms of reference, or other relevant documentation. 06
3. Documentation identifying and establishing mechanisms of coordination (e.g.,
committees, internal work-sharing and workflow) among the structures, if any, that take 07
part in market surveillance and control activities.
08
References: 1. National drug regulatory legislation: guiding principles for small drug regulatory authorities.
In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: thirty-fifth 09
report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical Report Series, No.
885), (1), (http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)

Framework: Structure/Foundation/Input.

Rating Scale:  NOT IMPLEMENTED (NI): There is no defined structure in charge of market
surveillance and control activities.
 ONGOING IMPLEMENTATION (OI): A mandate to establish a structure in charge of
market surveillance and control activities is available; however the structure itself is
not established yet.
 PARTIALLY IMPLEMENTED (PI): A structure in charge of market surveillance and
control activities is newly established and mandated; however the regular work and
practice of this structure is not consolidated yet.
 IMPLEMENTED (I): There is a defined structure in charge of market surveillance and
control activities with clear and well documented roles and responsibilities.

Page 156 of 340

WHO Global Benchmarking Tool (GBT)

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: MC02.02: Documented procedures or mechanisms are implemented to ensure

the involvement and communication among all stakeholders relevant to market
surveillance and control activities.

Maturity Level: 3
01
Scope: 1. Medicines
2. Vaccines 02

Description: The assessor should verify that the different stakeholders relevant to market surveillance 03
and control regulatory function are identified. These stakeholders may include not only
different organizational structures (e.g., organizations, institutions, or departments) that 04
implement the function, but also other entities such as: other governmental agencies

Market Surveillance and Control (MC): Indicators and Fact Sheets

(e.g., laboratories, police departments, and customs and judicial authorities), regional and
international organizations (e.g. WHO), non-governmental organizations, professional
associations, customer representative associations, and industry representatives (e.g.,
manufacturers, distributors, and wholesalers). In addition, the assessor should verify the
availability and implementation of agreements, memoranda of understanding (MOUs) and
other documented procedures that ensure the involvement, communication and
collaboration among the identified stakeholders relevant to market surveillance and
control activities.

Objective: The objective of this sub-indicator is to ensure that documented procedures or other
mechanisms are implemented to ensure the involvement and communication among
different organizations, institutions, and departments, as well as the appropriate
organization and good governance of the function.

Requirement: Agreements, MOUs and procedures for collaboration among stakeholders relevant to
market surveillance and control regulatory function.

Evidence to review: The assessor should ask for and review:

1. List of stakeholders relevant to market surveillance and control.
2. Agreements, MOUs or standard operating procedures (SOPs) that describe the
procedures for communication and collaboration among the identified stakeholders. 05
3. Example records of communications and collaborations that demonstrate
implementation of the above mentioned procedures. 06
4. Guidelines or SOPs that cover external and internal communications.
5. Evidence that demonstrates that there are regular formal meetings or official 07
communications among the above- mentioned key players. These are important to verify
that it is a systematic and well established process. 08
6. Documentation of paths of communication and reporting.
7. Documentation of platforms for information sharing and exchange. 09

Framework: Structure/Foundation/Input.

Rating Scale:  NOT IMPLEMENTED (NI): There are no information exchange procedures or
mechanisms among different stakeholders of the market surveillance and control
function or between the central authority and the decentralized entities.
 ONGOING IMPLEMENTATION (OI): The NRA is developing an information exchange
mechanism, but it is not ready yet or exchanges are being conducted without an
established methodology.

Page 157 of 340

WHO Global Benchmarking Tool (GBT)

 PARTIALLY IMPLEMENTED (PI): The NRA recently established an information

exchange procedure or mechanism and it is at the implementation stage, so this
practice is not consolidated yet.
 IMPLEMENTED (I): There are established, implemented and maintained information
exchange procedures or mechanisms among different stakeholders of the market
surveillance and control function and between the central authority and the
decentralized entities.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will 01
always apply for all benchmarked NRAs).
02
Indicator: MC03 Human resources to perform market surveillance and control activities.
03

Objective: The objective of this indicator is to ensure to that all entities within a National Regulatory 04
Authority (NRA) are adequately resourced with a trained, experienced and skilled workforce

Market Surveillance and Control (MC): Indicators and Fact Sheets

that is empowered to fully perform the market surveillance and control function. This will
ensure that market surveillance and control processes and activities are performed in
accordance with international best practices.
The objective of this indicator is to evaluate the human resource capacity of the entities with
respect to the number of personnel, the skills and experience of the personnel, and the overall
composition the workforce, with the goal of evaluating whether the workforce possesses the
specific expertise required to perform the market surveillance and control function.

Category: 06. Resources (HR, FR, infrastructure and equipment)

Sub Indicator: MC03.01: Sufficient competent staff (i.e., education, training, skills and experience)
are assigned to perform market surveillance and control activities.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that the human resources assigned to perform market
surveillance and control activities should be sufficient with respect to numbers and
competent with respect to the requisite skills, education, experience and training. 05
There should be technical documents and SOPs that provide guidance on the required
background for market surveillance and control activities and that consider the 06
requirements for educational background, competencies, skills, experience, and training.
The assessor should verify that the NRA estimated the number of staff required to 07
effectively and efficiently perform market surveillance and control function and that the
NRA actually recruited that number. In addition, the assessor should verify that these 08
competency requirements are well-established and maintained by the NRA. Metrics and
statistics on the different activities performed as well as performance indicators can 09
be used for estimating the adequacy of the number of the assigned staff. The assessor
should also verify that the competency of the assigned staff is built, maintained and
improved through recruitment as well as continuous on-the-job training.

Objective: The objective of this sub-indicator is to ensure the existing human resources for market
surveillance and control are sufficient, in terms of numbers, experience, and specific
competencies, to perform all the activities along the entire market surveillance and control
chain.

Requirement: Sufficient number of competent human resources in charge of market surveillance and
control activities.

Evidence to review: The assessor should ask for and review:

1. Evidence that the number of staff members involved in each of the documented
activities along the entire market surveillance and control process flow is adequate.

Page 158 of 340

WHO Global Benchmarking Tool (GBT)

2. Evidence that the systems and structures are in place to ensure appropriate placement
of staff with respect to competence and skills.
3. Evidence that the system and structures have been implemented. The documentation
should include the records to verify that the staff competence is appropriate for the job
requirements.
4. Evidence that the professional profiles of the human resources engaged in market
surveillance and control activities are appropriate with respect to education, skills, and
expertise, to perform a particular function along the market surveillance and control chain.
Documentation should include a list of the requisite skills and training for each position. 01
5. Recruitment plan.
02
References: 1. Quality management systems — Requirements. International Standard ISO 9001:2015.
Geneva. International Organization for Standardization, 2015, (4), (https://www.iso.org/iso- 03
9001-quality-management.html)
04
Framework: Structure/Foundation/Input.

Market Surveillance and Control (MC): Indicators and Fact Sheets

Rating Scale:  NOT IMPLEMENTED (NI): The NRA does not have enough competent staff (i.e., education,
training, skills and experience) to perform market surveillance and control activities
 ONGOING IMPLEMENTATION (OI): The NRA has recently developed a plan to recruit
adequate competent staff; however, the plan has not been implemented.
 PARTIALLY IMPLEMENTED (PI): The NRA has initiated the implementation of the human
resources development plan; however, there is need to complete the competency profile.
 IMPLEMENTED (I): The NRA has a sustained number of competent staff (i.e.,
education, training, skills and experience) assigned to perform market surveillance and
control activities.

Limitations and remarks:  Assessment of the adequacy and appropriateness of the number of staff members
is quite subjective and should be linked to some process or output indicators. When
estimating staff adequacy, the assessor should consider the workload, backlog, and
delays in delivery based on established timeframes.
 In some countries, internal staff might not be sufficient; however external staff (e.g.,
fellows or interns) may be involved in the performance of the work.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: MC03.02: Duties, functions, and responsibilities of the staff in charge of market 05
surveillance and control activities are established and updated in the respective
job descriptions 06
Maturity Level: 3 07
Scope: 1. Medicines 08
2. Vaccines
09
Description: The assessor should verify that procedures are in place to maintain a current and updated
structure for managing job descriptions for personnel participating in market surveillance
and control activities. In addition, job descriptions should address current staff duties,
responsibilities and the requisite competencies. A job description with this format and
content should be established and implemented for all staff. The management of job
descriptions should be supported by a guidance document that provides direction on when
and how to update the information, and where the information should be kept for easy
access. The guidance document should present the appropriate duties and responsibilities
that are assigned to each member of the organization involved in market surveillance and
control activities. Thus, the professional profiles of staff are reflected in their respective
roles and responsibilities within the NRA. There should be procedures to guide responsible
persons to document that duties, functions and responsibilities are revised and kept up to
date. In addition, procedures should be available to guide the keeping and documenting of
up to date work schedules and enforcing the implementation of the documented guidelines
and procedures.

Page 159 of 340

WHO Global Benchmarking Tool (GBT)

Requirement: Duties, roles and responsibilities of the staff relevant to market surveillance and control
activities.

Evidence to review: The assessor should ask for and review:

1. Procedures and guidelines that guide placement of staff members within the NRA; 01
2. The professional profiles of staff (i.e., job descriptions) and documentation that they are
related to their current roles and duties; 02
3. The professional profiles of the external experts and documentation that the profiles
provide a composition that is complete and consistent with that prescribed in the legal 03
provisions;
4. Procedures to guide the documentation of up to date duties and work schedules, and to 04
enforce the implementation of the documented guidelines and procedures;

Market Surveillance and Control (MC): Indicators and Fact Sheets

5. Job descriptions for designated staff.

Framework: Structure/Foundation/Input.

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence of defined or established duties,
functions, responsibilities, respective job descriptions and necessary required 05
competencies.
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted or developed the role 06
and responsibilities document but it has not yet been implemented.
 PARTIALLY IMPLEMENTED (PI): The NRA has initiated implementation of this 07
requirement but it has not been defined or followed for all staff or the roles and
responsibilities documents, including staff job descriptions, are not up to date. 08
 IMPLEMENTED (I): The NRA has defined and established all required duties,
functions, and responsibilities, and respective job descriptions are up-to-date. 09

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: MC03.03: Training plan developed, implemented and updated at least once a year
for staff in charge of market surveillance and control activities.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Page 160 of 340

WHO Global Benchmarking Tool (GBT)

similar documents that guide the development and implementation of training plans.
The assessor should ensure that induction training for new staff as well as continued
on- the-job for staff is planned and implemented. There should be procedures to
approve the training plan and the budget allocated for implementing and updating
the training plan. The plans should present clearly defined training goals, and should
include training in certain topics and skills to address identified deficiencies. Learning
objectives, training methods and activities, evidence of learning, and evaluation
and assessment of training should be documented. This documentation should
confirm that the learning objectives were achieved and were designed to address 01
weaknesses within the entities. Procedures should be in place to ensure that a training
plan is developed, implemented and updated at least once every year. The assessor 02
should verify that there is a system in place for monitoring the implementation and
effectiveness of the training plan and for documenting the skills acquired in training 03
activities for internal and external experts.
04
Objective: The objective of this sub-indicator is to ensure that a training plan for staff exists, and

Market Surveillance and Control (MC): Indicators and Fact Sheets

that it is implemented and updated annually. Through the training plan, NRA can be
sure that competency of staff in charge of market surveillance and control activities is
maintained and enhanced.

Requirement: Implementation of training plan

Evidence to review: The assessor should ask for and review:

1. Guidelines for development, implementation and annual update (i.e., at least once per
year) of the training plan. Guidelines should also provide for a mechanism to measure
effectiveness of training.
2. Documentation for the system or structures used to approve the training plan and to
evaluate the adequacy of the budget allocated to the training activities.
3. The current or existing staff training plan (or matrix) for staff. The assessor should
assess this in in relation to the respective individual job descriptions.
4. SOP for developing and maintaining the training plan.
5. Evidence that the NRA has investigated and identified training needs.
6. List of trainings performed.
7. Example records for training activities.

Framework: Process

Page 161 of 340

WHO Global Benchmarking Tool (GBT)

supported by adequate records to demonstrate effective plan implementation,

including induction training for new staff and routine on-the-job training for recruited
staff.

Limitations and remarks:  Training plans must be updated regularly; ideally on an annual basis, but not less
frequently that once every two years.
 Some regulatory functions may include many training activities that are not
incorporated in the institutional training programme. Such training normally is offered
by invitation. In this case, the assessor should recognize reports from non-routine 01
market surveillance and control -relevant training not included in the NRA training plan.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will 02
always apply for all benchmarked NRAs).
03
Sub Indicator: MC03.04: The NRA generates and maintains records of staff training activities
and training effectiveness verification. 04

Market Surveillance and Control (MC): Indicators and Fact Sheets

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that records of staff training that is performed or organized by
the NRA are generated, maintained, regularly updated. This activity should be supported
by guidelines that direct the NRA to generate and maintain records of staff training
activities. Procedures should be in place to document and propose staff training needs
and to allocate a budget for continuous staff capacity building and development. The
assessor should check that there is an evaluation or assessment mechanism to verify the
quality of learning, and to confirm that learning objectives are achieved. Documentation
should include an inventory (i.e., soft and/or hard) system that records all impactful and
non-impactful trainings, and identifies all staff members who participated. A system to
measure or estimate impact of trainings should be established.

Objective: The objective of this sub-indicator is to ensure that training organized by the NRA or
responsible regulatory authority is adequately documented and that the training records are
adequately maintained and kept. Staff training records are considered an integral part of staff
file and are a tool for measuring and tracking staff competency, development and adequacy.
05
Requirement: Training records
06
Evidence to review: The assessor should ask for and review:
1. Guidelines or similar documents that guide the NRA to generate and maintain records of 07
staff training activities;
2. Evaluations of training effectiveness; 08
3. The training inventory, and procedures for completing the inventory;
4. Examples of archived records of staff training, and procedures for the archiving system 09

Page 162 of 340

WHO Global Benchmarking Tool (GBT)

Framework: Output

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence that the NRA generates and maintains
records of staff training activities.
 ONGOING IMPLEMENTATION (OI): The NRA has recently initiated plans to generate,
document and keep records of staff training activities, however they are not yet
followed.
 PARTIALLY IMPLEMENTED (PI): The NRA has recently initiated plans to generate,
document and keep records of staff training activities but they are not fully followed 01
for all training activities or they have been established for less than two years.
 IMPLEMENTED (I): The NRA generates and maintains records of staff training 02
activities.
03
Limitations and remarks: The assessor should note that some NRAs out-source training including staff
04


capacity development activities. In this case the assessor should request the identity
of the provider, as well as evaluations of the provider. The assessor may request the

Market Surveillance and Control (MC): Indicators and Fact Sheets

professional profiles of tutors or resource persons used to offer training. The assessor
may also request records covering the archiving systems in place.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Indicator: MC04 Procedures established and implemented to perform market surveillance

and control

Objective: The objective of this indicator is to ensure that different activities of the market
surveillance and control programme are implemented through standard procedures and
work instructions in order to ensure consistency, effectiveness, efficiency, impartiality and
proportionality of the programme.

Category: 07. Regulatory process

Sub Indicator: MC04.01: Documented and implemented procedures exist to grant the necessary
authorizations or permissions for import activities.

Maturity Level: 3

Scope: 1. Medicines 05
2. Vaccines
06
Description: The assessor should verify the availability of documented standard procedures within the
NRA to receive, review and make a decision on import applications of medical products. 07
The assessor should also verify proper implementation of the procedures through review
of the relevant records and documentation. The records for an application should include 08
documentation of receipt, screening, review, regulatory decision-making, (i.e., approval
or denial), and notification of the applicant. Regulatory decisions should be scientifically 09
justifiable.

Objective: The objective of this sub-indicator is to ensure that the control of import activities of
medical products are an integral part of the overall market surveillance and control
regulatory function. Therefore, procedures to handle and decide on (i.e., approve or deny)
applications relevant to these activities are essential to ensure the consistency and
effectiveness of these regulatory activities.

Requirement: Procedures along with their relevant records for regulatory decisions on import activities

Evidence to review: The assessor should ask for and review:

1. SOPs that guide the handling of applications for medical products import activities.
2. List of import applications over a wide range of time (e.g., 6 months or 1 year).
3. Example records and documentation of receiving, processing and decision-making for
medical products import applications.

Page 163 of 340

WHO Global Benchmarking Tool (GBT)

References: 1. Guidelines on import procedures for pharmaceutical products, World Health

Organization (WHO), (22), (http://digicollection.org/whoqapharm/p/about and http://apps.
who.int/medicinedocs/en)
2. Guidelines on the implementation of the WHO certification scheme on the quality of
pharmaceutical products moving in international commerce. In WHO Expert Committee on
Specifications for Pharmaceutical Preparations: thirty-fourth report. Geneva. World Health
Organization; 1996: Annex 10 (WHO Technical Report Series, No. 863), (29), (http://apps.
who.int/medicinedocs/pdf/s5516e/s5516e.pdf)
01
Framework: Process
02
Rating Scale:  NOT IMPLEMENTED (NI): There are no documented procedures in the NRA for
the prevention of unauthorized import activities, and no procedures to grant the 03
necessary authorizations or permissions for import activities.
 ONGOING IMPLEMENTATION (OI): The NRA has taken some steps to establish 04
procedures to grant the necessary authorizations or permissions for import activities;

Market Surveillance and Control (MC): Indicators and Fact Sheets

however no results exist so far.
 PARTIALLY IMPLEMENTED (PI): Documented procedures to grant the necessary
authorizations or permissions for import activities were recently established; however
the procedures are recently implemented and no results associated with this sub-
indicator are documented yet.
 IMPLEMENTED (I): There are documented procedures to grant the necessary
authorizations or permissions for import activities, and these procedures are actually
implemented.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: MC04.02: Documented and implemented procedures exist for regulation
of promotion and advertisement of medical products

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines
05
Description: The assessor should verify the availability of documented standard procedures within the
NRA to receive, review and make a decision on promotion, marketing and advertisement 06
applications. Assessor should also verify proper implementation of the procedures through
review of the relevant records and documentation. The records for an application should 07
include documentation of receipt, screening, review and regulatory decision-making,
(i.e., approval or denial), and notification of the applicant. Regulatory decisions should be 08
scientifically justifiable.
09
Objective: The objective of this sub-indicator is to ensure that the control of promotion and
advertisement of medical products is an integral part of the overall market surveillance and
control regulatory function. Therefore, procedures to regulate the information that reaches
health professionals, as well as the general public, are an important component to promote
the rational use of these products.

Requirement: Procedures along with the relevant records for engagement of regulatory entities in the
regulation of promotion and advertisement of medical products.

Evidence to review: The assessor should ask for and review:

1. SOPs that guide the handling of applications for promotion and advertisement activities.
2. List of promotion, marketing and advertisement applications received and reviewed over
a wide range of time (e.g., 6 months or 1 year).
3. Example records and documentation of regulatory decisions on promotion and
advertisement applications.

Page 164 of 340

WHO Global Benchmarking Tool (GBT)

References: 1. Ethical Criteria for the Promotion, Advertisement, and Publicity of Medicines -
PANDRH Series Technical Document No 12 (Pan American Network for Drug Regulatory
Harmonization), (94), (http://apps.who.int/medicinedocs/documents/s22161en/s22161en.
pdf)
2. Educational initiatives for medical and pharmacy students about drug promotion: an
international cross-sectional survey. Geneva. World Health Organization, (92), (http://apps.
who.int/medicinedocs/pdf/s8110e/s8110e.pdf)
3. Cross-border Advertising, Promotion and Sale of Medical Products Using the Internet.
WHA Resolution; Fifty-First World Health Assembly, WHA51.9, (93), (http://apps.who.int/ 01
medicinedocs/documents/s21471en/s21471en.pdf)
4. Alternative Regulatory Models for Pharmaceutical Promotions Involving Civil Society and 02
Other Non-Government Stakeholders, Medicines Transparency Alliance, (96), (http://apps.
who.int/medicinedocs/documents/s22327en/s22327en.pdf) 03
5. Ethical Criteria for Medicinal Drug Promotion, World Health Organization, 1988, (13),
(http://apps.who.int/medicinedocs/documents/whozip08e/whozip08e.pdf) 04
6. CIOMS/WHO Meeting on Ethical Criteria for Medicinal Drug Promotion. Essential Drugs

Market Surveillance and Control (MC): Indicators and Fact Sheets

Monitor No. 017, 1994, (90), (http://apps.who.int/medicinedocs/documents/s21142en/
s21142en.pdf)
7. Drug Promotion: Push, Promote or Educate? Essential Drugs Monitor, No. 020, 1995,
World Health Organization (WHO), (91), (http://apps.who.int/medicinedocs/documents/
s21265en/s21265en.pdf)
8. Drug Promotion - What We Know, What We Have Yet to Learn - Reviews of Materials in
the WHO/HAI Database on Drug Promotion, World Health Organization (WHO), (89), (http://
apps.who.int/medicinedocs/pdf/s8109e/s8109e.pdf)

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): There are no documented procedures in the NRA for
regulation of promotion and advertisement of medical products.
 ONGOING IMPLEMENTATION (OI): The NRA has taken some steps to establish
procedures for regulation of promotion and advertisement of medical products;
however no results exist so far.
 PARTIALLY IMPLEMENTED (PI): Documented procedures in the NRA for regulation of
promotion and advertisement of medical products were recently established; however
the procedures are recently implemented and no results associated with this sub-
indicator are documented yet.
 IMPLEMENTED (I): There are documented procedures in the NRA for regulation of 05
promotion and advertisement of medical products, and these procedures are actually
implemented. 06
Limitations and remarks:  Procedures and documentation relating to the approval of promotion and 07
advertisement applications might not exist if the NRA does not have the mandate
to pre-approve such applications. In case no pre-approval system is in place 08
the assessor should verify the existence of an alternative approach (e.g., active
surveillance of promotional materials, rather than proactive approval) which satisfies 09
the adequate control of promotion and advertisement activities.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: MC04.03: Documented and implemented procedures for active monitoring
of the promotion and advertisement of medical products

Maturity Level: 4

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the availability of documented standard procedures within
the NRA to actively monitor promotion and advertisement of medical products. These
procedures should include screening mechanisms to identify false or misleading

Page 165 of 340

WHO Global Benchmarking Tool (GBT)

information contained in promotions and advertisements of medical products in the

market. The monitoring and screening should include promotional materials on the
internet. The assessor should also verify proper implementation of the procedures
through checking of the relevant records and documentation. The records should include
documentation of data collected, screening, review and regulatory decision- making, if any.
Regulatory decisions should be scientifically justifiable.

Objective: The objective of this sub-indicator is to ensure that the control of promotion and
advertisement of medical products is an integral part of the overall market surveillance 01
and control regulatory function. NRAs must have mechanisms in place to actively monitor
these activities and take the necessary regulatory measures in case of misconduct. 02

Requirement: Procedures along with the relevant records of regulatory decisions for monitoring the 03
promotion and advertisement of medical products
04
Evidence to review: The assessor should ask for and review:

Market Surveillance and Control (MC): Indicators and Fact Sheets

1. SOPs that guide the active monitoring of the promotion and advertisement of medical
products.
2. List of data collected from the market regarding the promotion and advertisement of
medical products received over a wide range of time (e.g., 6 months or 1 year)
3. Example records and documentation of regulatory decisions, if any.

References: 1. Ethical Criteria for the Promotion, Advertisement, and Publicity of Medicines - PANDRH
Series Technical Document No 12 (Pan American Network for Drug Regulatory Harmonization),
(94), (http://apps.who.int/medicinedocs/documents/s22161en/s22161en.pdf)
2. Educational initiatives for medical and pharmacy students about drug promotion: an
international cross-sectional survey. Geneva. World Health Organization, (92), (http://apps.
who.int/medicinedocs/pdf/s8110e/s8110e.pdf)
3. Cross-border Advertising, Promotion and Sale of Medical Products Using the Internet.
WHA Resolution; Fifty-First World Health Assembly, WHA51.9, (93), (http://apps.who.int/
medicinedocs/documents/s21471en/s21471en.pdf)
4. Alternative Regulatory Models for Pharmaceutical Promotions Involving Civil Society and
Other Non-Government Stakeholders, Medicines Transparency Alliance, (96), (http://apps.
who.int/medicinedocs/documents/s22327en/s22327en.pdf)
5. Ethical Criteria for Medicinal Drug Promotion, World Health Organization, 1988, (13),
(http://apps.who.int/medicinedocs/documents/whozip08e/whozip08e.pdf)
6. CIOMS/WHO Meeting on Ethical Criteria for Medicinal Drug Promotion. Essential Drugs 05
Monitor No. 017, 1994, (90), (http://apps.who.int/medicinedocs/documents/s21142en/
s21142en.pdf) 06
7. Drug Promotion: Push, Promote or Educate? Essential Drugs Monitor, No. 020, 1995,
World Health Organization (WHO), (91), (http://apps.who.int/medicinedocs/documents/ 07
s21265en/s21265en.pdf)
8. Drug Promotion - What We Know, What We Have Yet to Learn - Reviews of Materials in 08
the WHO/HAI Database on Drug Promotion, World Health Organization (WHO), (89), (http://
apps.who.int/medicinedocs/pdf/s8109e/s8109e.pdf) 09

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): There are no documented procedures in the NRA for active
monitoring of the promotion and advertisement of medical products.
 ONGOING IMPLEMENTATION (OI): The NRA has taken some steps to establish
procedures for active monitoring of the promotion and advertisement of medical
products; however no results exist so far.
 PARTIALLY IMPLEMENTED (PI): Documented procedures in the NRA for active
monitoring of the promotion and advertisement of medical products were recently
established; however the procedures are recently implemented and no results
associated with this sub-indicator are documented yet.
 IMPLEMENTED (I): There are documented procedures in the NRA for active
monitoring of the promotion and advertisement of medical products and these
procedures are actually implemented.

Page 166 of 340

WHO Global Benchmarking Tool (GBT)

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: MC04.04: Documented and implemented procedures exist for risk-based sampling
of medical products from different points of the supply chain.

Maturity Level: 3

Scope: 1. Medicines 01
2. Vaccines
02
Description: The assessor should verify the availability within the NRA of a structured programme,
with documented standard procedures, to carry out a medical products sampling and 03
quality surveillance programme. The programme should be risk based with consideration
of different risk factors (e.g. nature and category of products, location within the territory, 04
manufacturer, product quality history, previous complaints, vigilance data, and any other

Market Surveillance and Control (MC): Indicators and Fact Sheets

relevant information). The quality surveillance programme should be extended to cover
all points across the supply chain with emphasis on the terminal point and last stages of
the chain (e.g., points of sale, handling or administration). The programme should include
laboratory testing of the quality of the medical products. The assessor should also verify
that procedures specify any follow up and regulatory actions to be undertaken in the event
that SF medical products (i.e., products failing the quality surveillance) are detected. Ideally,
the follow-up actions should include regulatory inspection when necessary.

Objective: The objective of this sub-indicator is to ensure that risk based sampling and quality
surveillance of medical products are implemented as a means for confirming quality
of medical products in the market and throughout the supply chain. Additionally, a
surveillance program may significantly contribute to detection of SF medical products.

Requirement: Procedure along with relevant records for risk-based sampling of medical products on
the market.

Evidence to review: The assessor should ask for and review:

1. SOPs that guide medical product sampling and quality surveillance across the supply chain.
2. Examples of relevant records and documentation of surveillance activities, including
product sampling, requests for testing, and testing results.
3. Examples of relevant records and documentation for any follow up or regulatory actions 05
taken in the event that medical product failures are detected during the quality surveillance.
06
References: 1. Report on Post-Market Surveillance of First Line Anti-Tuberculosis Medicines in Kenya.
Kenya Ministry of Public health and sanitation, MSH and USAID. 2012, (98), (http://apps. 07
who.int/medicinedocs/documents/s21945en/s21945en.pdf)
2. Guidelines on the conduct of surveys of the quality of medicines, World Health 08
Organization (WHO), (86), (http://apps.who.int/medicinedocs/en/q/ and http://
digicollection.org/whoqapharm/) 09

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): There are no documented procedures for risk-based
sampling of medical products from different points of the supply chain.
 ONGOING IMPLEMENTATION (OI): The NRA has taken some steps to establish
procedures for risk-based sampling of medical products from different points of the
supply chain; however no results exist so far.
 PARTIALLY IMPLEMENTED (PI): Documented procedures for risk- based sampling of
medical products from different points of the supply chain were recently established;
however the procedures are recently implemented and no results associated with this
sub-indicator are documented yet.
 IMPLEMENTED (I): There are documented procedures for risk-based sampling of
medical products from different points of the supply chain, and these procedures are
actually implemented.

Page 167 of 340

WHO Global Benchmarking Tool (GBT)

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: MC04.05: Documented and implemented procedures exist to enable the public
to report suspected SF medical products.

Maturity Level: 3

Scope: 1. Medicines 01
2. Vaccines
02
Description: The assessor should verify the availability and implementation of documented procedures
for enabling the public to report medical products suspected to be SF. Ideally the reporting 03
procedures should be simple, effective, easy to use and accessible to a majority of the
people. The reporting should be two-way, enabling public reporting to the NRA as well as 04
regulatory feedback to the informants about the outcome of the relevant investigation.

Market Surveillance and Control (MC): Indicators and Fact Sheets

When appropriate, a wider group may be targeted for the feedback (e.g., media,
professional associations, and customer representatives).

Objective: The objective of this sub-indicator is to ensure that the public is involved in the reporting of
suspected SF medical products as an essential approach for detection and prevention of
this problem. In addition, these activities contribute to building the public confidence and
trust in the regulatory system.

Requirement: Procedures along with their relevant records to enable the public to report SF medical
products

Evidence to review: The assessor should ask for and review:

1. SOPs for enabling the public to report suspected SF medical products.
2. Examples of relevant records and documentation, including reporting forms, referral
procedures, and reports of actions taken or regulatory decisions made, if any.
3. Examples of feedback to the informants.

References: 1. SF- Frequently asked questions. Scope, scale and harm. How big is the problem of SF
medical products?, (87), (https://www.who.int/medicines/regulation/ssffc/faq-ssffc_1-10/en/)
2. Addressing the barriers to effective monitoring, reporting and containment of spurious/
substandard/falsely-labelled/ falsified/counterfeit medical products. A report prepared 05
for the Medicines Transparency Alliance, Philippines: 2016, (88), (http://apps.who.int/
medicinedocs/documents/s22331en/s22331en.pdf) 06
3. Recommendations for health authorities to detect and deal with actions, activities and
behaviors that result in substandard/spurious/falsely-labelled/falsified/counterfeit medical 07
products, A68/33, annex 1, (129), (http://apps.who.int/gb/ebwha/pdf_files/WHA68/
A68_33-en.pdf) 08
4. Guidance on developing a national plan for preventing, detecting and responding to actions,
activities and behaviors that result in SF medical products. WHO (document A70/23), (130), 09
(https://www.who.int/medicines/regulation/ssffc/mechanism/A70_23-en6-14.pdf)

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): There are no documented procedures to enable the public
to report suspected SF medical products.
 ONGOING IMPLEMENTATION (OI): The NRA has taken some steps to establish
procedures to enable the public to report suspected SF medical products; however no
results exist so far.
 PARTIALLY IMPLEMENTED (PI): Documented procedures to enable the public to
report suspected SF medical products were recently established; however such
procedures are recently implemented and no results associated with this sub-
indicator are documented yet.
 IMPLEMENTED (I): There are documented procedures to enable the public to report
suspected SF medical products, and these procedures are actually implemented.

Page 168 of 340

WHO Global Benchmarking Tool (GBT)

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: MC04.06: Documented and implemented procedures exist in the NRA to review
any complaints or market reports received.

Maturity Level: 3

01
Scope: 1. Medicines
2. Vaccines 02

Description: The assessor should verify the availability of documented procedures within the NRA to 03
receive, review and respond to any complaints or reports related to medical products.
These may include complaints and reports related to: (1) quality, safety or efficacy of 04
medical products, (2) suspected cases of SF medical products, and (3) misconduct

Market Surveillance and Control (MC): Indicators and Fact Sheets

in promotion or advertising, including via the internet. Complaints and reports could
be related to any medical product or company. The assessor should verify actual
and proper implementation of the procedures through review of the relevant records
and documentations. The records should include documentation of investigations of
complaints and any subsequent regulatory decisions. Complaints and reports may come
from consumers, health professionals, health institutions, and participants in the supply
chain.

Objective: The objective of this sub-indicator is to ensure that complaints and reports, as a major
feedback mechanism about market control activities, are properly received and handled
following well-established procedures in order to ensure consistency and effectiveness in
the responses to these cases.

Requirement: Procedures along with relevant records of regulatory decisions on complaints and reports
received.

Evidence to review: The assessor should ask for and review:

1. SOPs that guide the handling of complaints and reports.
2. List of market complaints and reports over a wide range of time (e.g., 6 months or 1
year).
3. Examples of market complaints and reports along with associated records (e.g., reports 05
of case investigations and any subsequent regulatory decisions).
06
References: 1. WHO good manufacturing practices for pharmaceutical products: main principles,
(51), (http://digicollection.org/whoqapharm/p/about and http://apps.who.int/ 07
medicinedocs/en)
08
Framework: Process
09
Rating Scale:  NOT IMPLEMENTED (NI): There are no documented procedures in the NRA to review
and respond to any complaints or market reports received.
 ONGOING IMPLEMENTATION (OI): The NRA has taken some steps to establish
procedures to review and respond to any complaints or market reports received;
however no results exist so far.
 PARTIALLY IMPLEMENTED (PI): Documented procedures in the NRA to review and
respond to any complaints or market reports received were recently established;
however such procedures are recently implemented and no results associated with
this sub-indicator are documented yet.
 IMPLEMENTED (I): There are documented procedures in the NRA to review and
respond to any complaints or market reports received, and these procedures are
actually implemented.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Page 169 of 340

WHO Global Benchmarking Tool (GBT)

Sub Indicator: MC04.07: Documented and implemented procedures and mechanisms exist
to prevent, detect and respond to SF medical products.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the availability of documented standard procedures and 01
mechanisms within the NRA to prevent, detect and respond to SF products in the market,
including products being distributed via the internet. These procedures and mechanisms 02
should include various approaches. Examples of approaches for detection of SF medical
products might include routine inspections, targeted risk-based surveys, or investigations 03
of reports of complaints, suspicious observations, unexpected adverse events, or whistle-
blower concerns. Similarly measures for prevention of SF medical products might include 04
public education and awareness campaigns, as well as other measures to ensure access

Market Surveillance and Control (MC): Indicators and Fact Sheets

to and affordability of quality medical product. Actions for response to SF medical
products might include rapid alerts, prosecutions, and recalls, quarantines, or withdrawals
of affected products.
The assessor should verify actual and proper implementation of the procedures through
checking of the relevant records and documentation. The records should include relevant
databases, evidence collected, investigation reports (including descriptions of any field
procedures, inspections and intelligence activities that might have been carried out jointly
or with the support of national and international stakeholders) and any follow-up regulatory
decisions.

Objective: The objective of this sub-indicator is to ensure that well established procedures or
mechanisms are implemented for the prevention, detection, and response to SF medical
products.

Requirement: Procedures along with the relevant records for prevention, detection and response to SF
medical products

Evidence to review: The assessor should ask for and review:

1. SOPs and other mechanisms to prevent, detect and respond to SF medical products.
2. Example records and documentation of prevention, detection and response to SF
medical products. 05
References: 1. Addressing the barriers to effective monitoring, reporting and containment of spurious/ 06
substandard/falsely-labelled/ falsified/counterfeit medical products. A report prepared
for the Medicines Transparency Alliance, Philippines: 2016, (88), (http://apps.who.int/ 07
medicinedocs/documents/s22331en/s22331en.pdf)
2. SF- Frequently asked questions. Scope, scale and harm. How big is the problem of SF 08
medical products?, (87), (https://www.who.int/medicines/regulation/ssffc/faq-ssffc_1-10/
en/) 09
3. WHO Member State Mechanism on substandard/spurious/falsely-labelled/falsified/
counterfeit medical products, A70/23, appendix 3, Working Definitions, (128), (https://www.
who.int/medicines/regulation/ssffc/A70_23-en1.pdf)
4. Recommendations for health authorities to detect and deal with actions, activities and
behaviors that result in substandard/spurious/falsely-labelled/falsified/counterfeit medical
products, A68/33, annex 1, (129), (http://apps.who.int/gb/ebwha/pdf_files/WHA68/
A68_33-en.pdf)
5. Guidance on developing a national plan for preventing, detecting and responding to actions,
activities and behaviors that result in SF medical products. WHO (document A70/23), (130),
(https://www.who.int/medicines/regulation/ssffc/mechanism/A70_23-en6-14.pdf)

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): There are no documented procedures or mechanisms, in
the NRA, for prevention, detection and response to SF medical products.

Page 170 of 340

WHO Global Benchmarking Tool (GBT)

 ONGOING IMPLEMENTATION (OI): The NRA has taken some steps to establish
procedures and mechanisms for prevention, detection and response to SF medical
products; however no results exist so far.
 PARTIALLY IMPLEMENTED (PI): Documented procedures and mechanisms for
prevention, detection and response to SF medical products were recently established
in the NRA; however such procedures are recently implemented and no results
associated with this sub-indicator are documented yet.
 IMPLEMENTED (I): There are documented procedures and mechanisms, in the NRA,
for prevention, detection and response to SF medical products, and these procedures 01
are actually implemented.
02
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 03

Sub Indicator: MC04.08: Documented and implemented procedures exist to ensure safe storage 04
and disposal of detected SF medical products.

Market Surveillance and Control (MC): Indicators and Fact Sheets

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the availability of documented standard procedures within
the NRA for safe storage and disposal of detected SF medical products. Safe storage and
disposal provides assurance that detected SF medical products are not re-introduced
to the market and that the best practices were employed to avoid or minimize adverse
effects of this disposal on the people and environment. The assessor should also verify
actual and proper implementation of the procedures through review of the relevant records
and documentation. The records should include proof of storage and disposal along with
documentation for the reconciliation between the recovered quantities versus the
disposed ones.

Objective: The objective of this sub-indicator is to ensure the control of detected SF medical products.
Therefore, a national policy and strategy, combined with procedures to ensure safe storage
and disposal of detected SF medical products, are essential to avoid re-introduction of
these products to the market. 05
Requirement: Procedures along with their relevant records for safe storage and disposal of SF medical 06
products
07
Evidence to review: The assessor should ask for and review:
1. SOPs for storage and disposal of SF medical products. 08
2. Example records and documentation of SF medical products storage and disposal
including reconciliation of quantities (i.e., quantities detected, recovered, or recalled vs. 09
quantities disposed).

References: 1. Guidelines for Safe Disposal of Unwanted Pharmaceuticals in and after Emergencies,
(97), (http://apps.who.int/medicinedocs/pdf/whozip51e/whozip51e.pdf)
2. WHO good manufacturing practices for pharmaceutical products: main principles,
(51), (http://digicollection.org/whoqapharm/p/about and http://apps.who.int/
medicinedocs/en)

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): There are no documented procedures, in the NRA, to
ensure safe storage and disposal of detected SF medical products.
 ONGOING IMPLEMENTATION (OI): The NRA has taken some steps to establish
procedures to ensure safe storage and disposal of detected SF medical products;
however no results exist so far.

Page 171 of 340

WHO Global Benchmarking Tool (GBT)

 PARTIALLY IMPLEMENTED (PI): Documented procedures to ensure safe storage

and disposal of detected SF medical products were recently established; however
such procedures are recently implemented and no results associated with this sub-
indicator are documented yet.
 IMPLEMENTED (I): There are documented procedures to ensure safe storage
and disposal of detected SF medical products, and these procedures are actually
implemented.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will 01
always apply for all benchmarked NRAs).
02
Indicator: MC05 Mechanism in place to monitor regulatory performance and output.
03

Objective: The objective of this indicator is to ensure that mechanisms are in place to track regulatory 04
effectiveness of the market surveillance and control activities, to measure relevant

Market Surveillance and Control (MC): Indicators and Fact Sheets

progress in the programme, and to establish, implement and regularly verify performance
indicators.

Category: 09. Monitoring progress and assessing outcomes & impact

Sub Indicator: MC05.01: Database exists of approved and refused promotional and advertising
materials along with the supporting documentation.

Maturity Level: 4

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the establishment of a database of approved as well as refused
marketing, promotional and advertising materials, along with documentation that supports
the decisions taken. The database should not be limited to listing of approved and rejected
materials. Instead, the database should also include the applications, the supporting
documentation submitted by applicants, and records of the regulatory review, assessment,
and final decision-making for these applications.

Objective: The objective of this sub-indicator is to ensure that a database of applications for 05
marketing, promotional and advertising materials is established for storing, consolidating
and analyzing relevant information. The database also serves as a basis for further follow 06
up and enforcement actions. Thus tracing relevant regulatory activities and decisions
for the sake of institutional memory is ensured. Furthermore, analyses of the data would 07
contribute to optimization of the performance of the regulatory program.
08
Requirement: Database of promotional, marketing and advertising materials applications,
documentations and regulatory decisions 09

Evidence to review: The assessor should ask for and review:

1. Database of applications and supporting documentation of marketing, promotional and
advertising materials.
2. Database of regulatory reviews and assessments of marketing, promotional and
advertising materials.

References:

Framework: Output

Rating Scale:  NOT IMPLEMENTED (NI): There is no database for approved and refused promotional
materials along with their supporting documentation.
 ONGOING IMPLEMENTATION (OI): The NRA is preparing to establish a database, but
there is no evidence of results from such activities.

Page 172 of 340

WHO Global Benchmarking Tool (GBT)

 PARTIALLY IMPLEMENTED (PI): There is evidence that the NRA has the elements
(i.e., procedures, documentation, management systems, and related information.)
and the capacity to perform the processes mentioned in the indicator; however it
has only limited experience or a limited number of documented events.
 IMPLEMENTED (I): There is a database for approved and refused promotional
materials along with their supporting documentation

Limitations and remarks: This sub-indicator is applicable only to the countries where there are pre-approval
activities related to medical products promotion and advertising. In case there is no 01
NRA pre-approval system is in place, the assessor should verify the existence of an
alternative approach which provides adequate control of promotional, marketing and 02
advertising activities.
03
Sub Indicator: MC05.02: Database for product batches that have undergone surveillance along
with their relevant testing results and regulatory actions is established and 04
periodically reviewed.

Market Surveillance and Control (MC): Indicators and Fact Sheets

Maturity Level: 4

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the establishment and maintenance of a database of
product batches that have been included in the market surveillance program (i.e.,
batches that have been sampled or tested). The database should include relevant
findings, test results, and regulatory decisions. Importantly, this database should
include information about SF medical products. The database should not only include a
listing of the relevant information, but also provide data analyses designed to optimize
the surveillance program, identify repeated violations and trends, and guide regulatory
measures to prevent, detect and respond to SF medical products. Also, such database
makes it possible to compile the information needed for similar actions at the global
and international level.

Objective: The objective of this sub-indicator is to ensure the establishment of a database of

product batches that have been included in the surveillance program. The database
should store, consolidate and analyze information from the market surveillance
program. Thus tracing relevant regulatory activities and decisions for the sake of 05
institutional memory is ensured. Furthermore, analyses of the data would contribute to
optimization of the performance of the regulatory program. 06
Requirement: Database of product batches that have been included in the market surveillance program, 07
their relevant testing results and regulatory decisions or actions, if any
08
Evidence to review: The assessor should ask for and review:
1. Database of product batches that have been included in the market surveillance 09
program along with their results and regulatory actions, if any.
2. Updated analyses of the data along with consequent optimization, if any.

References:

Framework: Output

Rating Scale:  NOT IMPLEMENTED (NI): There is no database within for NRA for product batches
that have undergone surveillance along with their regulatory actions.
 ONGOING IMPLEMENTATION (OI): The NRA is preparing to establish a database, but
there is no evidence of results from such activities.
 PARTIALLY IMPLEMENTED (PI): There is evidence that the NRA has the elements (i.e.,
procedures, documentation, management systems, and related information) and the
capacity to perform the processes mentioned in the indicator; however it has only
limited experience or a limited number of documented events.

Page 173 of 340

WHO Global Benchmarking Tool (GBT)

 IMPLEMENTED (I): There is a database within the NRA for product batches that have
undergone surveillance along with their regulatory actions. Also, this database is
regularly updated.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: MC05.03: Performance indicators for market surveillance and control activities
are established and implemented 01
Maturity Level: 4 02

Scope: 1. Medicines 03
2. Vaccines
04
Description: The assessor should verify the existence and implementation of performance indicators

Market Surveillance and Control (MC): Indicators and Fact Sheets

for different activities included under the market surveillance and control functions.
Specifically, the system should define key performance indicators (KPIs) along the entire
market surveillance and control activity chain and all indicators should be adequately
In addition, the assessor should verify measured indicators are analyzed to identify trends
or abnormalities. Justifications for any identified abnormalities should be provided; when
necessary, process optimizations should be introduced to avoid recurrence. justified.
For the purpose of clarity and consistency, established KPIs should be supported with
guidelines for monitoring and maintenance of the KPIs. The guidelines in turn should
be supported by SOPs and tools that define the procedures to be used for monitoring
and evaluating the performance indicators and that define procedures and timelines for
reviewing and revising the indicators.
For market surveillance and control activities, the performance indicators should cover the
four themes: (1) control of import activities, (2) preventing, detecting and responding to SF
medical products, (3) market surveillance program for monitoring the quality of medical
products, and (4) control of promotional, marketing and advertising activities. Examples
of indicators include, but are not limited to: number of product batches sampled as part of
market surveillance program, number of approvals and refusals of promotional materials,
and number of detected SF medical products (with sub- categorization).
Established KPIs might be qualitative, quantitative or combination of both. In general,
quantitative indicators are preferred to avoid bias or misinterpretation. However, qualitative
indicators are also accepted. Qualitative indicators may or may not include scoring 05
or scaling to render them semi-quantitative and thus more informative. The assessor
should ensure that indicators are measured on a regular basis to monitor progress and 06
advancement.
07
Objective: The objective of this sub-indicator is to ensure that a system, mechanism, or procedure
exists to require the NRA to establish performance indicators along the entire market 08
surveillance and control chain. Additionally, the objective is to ensure that KPIs are
actually contributing to monitoring of regulatory performance, to measuring effectiveness 09
of market surveillance and control regulatory activities, and to making any necessary
adjustments or optimizations.

Requirement: KPIs for market surveillance and control activities

Evidence to review: The assessor should ask for and review:

1. Documents supporting the system, mechanism, or procedure compelling the NRA to
establish and implement performance indicators along the entire market surveillance and
control activity chain.
2. Evidence that the performance indicators have been established and implemented, and
that the members of staff involved in the market surveillance and control function are
aware of the indicators and the guidelines and SOPs used for monitoring and evaluating
their performance.
3. The current performance indicators for market surveillance and control activities. KPIs
should cover the four themes listed in Description column.

Page 174 of 340

WHO Global Benchmarking Tool (GBT)

4. Analyses of the measured indicators along with the investigations done to identify
trends or abnormalities.
5. Documentation for follow-up of any observed abnormalities, including justifications for any
identified abnormalities as well as any process optimizations introduced to avoid recurrence.

References: 1. Quality management -- Quality of an organization -- Guidance to achieve sustained

success, ISO 9004:2018, International Organization for Standardization (ISO), (115),
(https://www.iso.org/standard/70397.html)
2. World Health Organization. European Observatory on Health Systems and Policies. 01
Performance measurement for health system improvement: experiences, challenges
and Prospects, (125), (http://www.who.int/management/district/performance/ 02
PerformanceMeasurementHealthSystemImprovement2.pdf)
3. World Health Organization. European Observatory on Health Systems and Policies. 03
Health System Performance Comparison: an agenda for policy, information and research,
(126), (http://www.euro.who.int/__data/assets/pdf_file/0009/244836/Health-System- 04
Performance-Comparison.pdf)

Market Surveillance and Control (MC): Indicators and Fact Sheets

Framework: Output

Rating Scale:  NOT IMPLEMENTED (NI): There are no KPIs for market surveillance and control
activities.
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted KPIs for market
surveillance and control activities but they have not yet been reported.
 PARTIALLY IMPLEMENTED (PI): The NRA has developed KPIs for market surveillance
and control activities and has been applying them for less than two year or they have
not covered all critical steps.
 IMPLEMENTED (I): The NRA has established and implemented KPIs for market
surveillance and control activities. The indicators are reviewed regularly, and
appropriate actions are taken and decisions made.

Limitations and remarks:  When they refer to outcomes, indicators may be ambiguous and difficult to interpret,
as outcomes are the result of many factors that are difficult to disentangle. When
they refer to processes, indicators are often too specific, as they may focus on a
particular intervention or condition or they may quickly become outdated as business
models develop.
 Different methodologies are used to measure the NRAs performance on market
surveillance and control activities. In this case, the assessor should verify that 05
adequate supporting documents are available. The assessor should consider that
developed performance indicators should be Specific, Measurable, Achievable, 06
Realistic, and Time-bound (i.e., “SMART”).
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will 07
always apply for all benchmarked NRAs).
08
Indicator: MC06 Mechanism exists to promote transparency, accountability
and communication. 09

Objective: The objective of this indicator is to ensure that mechanisms are in place that promote
effective communication of market surveillance and control activities and related
information within the NRA, that promote transparency and outreach to the public, and that
establish milestones that encourage accountability of the NRA to its mandate. Additionally
these activities contribute to mutual understanding and involvement of all stakeholders
relevant to market surveillance and control activities. Consequently, confidence in the
regulatory system is raised.

Category: 08. Transparency, accountability and communication

Sub Indicator: MC06.01: Market surveillance and control activities are appropriately
communicated within the NRA.

Maturity Level: 3

Page 175 of 340

WHO Global Benchmarking Tool (GBT)

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that market surveillance and control activities are properly
communicated among different institutions, departments, and entities of the NRA (e.g.
laboratory, inspection and enforcement departments). These internal communications
should be guided by appropriate policies or procedures and supported by mechanisms for
implementation.
01
Objective: The objective of this sub-indicator is to ensure that communication of market surveillance
and control activities within the NRA is well established as a best practice to promote 02
regulatory effectiveness. In addition, such communication increases awareness and
encourages internal understanding and cooperation among all relevant entities and 03
departments within the NRA.
04
Requirement: Communication within the NRA of the market surveillance and control activities

Market Surveillance and Control (MC): Indicators and Fact Sheets

Evidence to review: The assessor should ask for and review:
1. Documentation of guidelines, policies, procedures or mechanisms for communicating
market surveillance and control activities within the NRA.
2. Records of communication (i.e., paper based or electronic) proving communication of
information within the NRA.

References: 1. MDS-3: Managing Access to Medicines and Health Technologies (Third Edition), MSH,
2012, (99), (http://apps.who.int/medicinedocs/documents/s19577en/s19577en.pdf)
2. Guidelines on the conduct of surveys of the quality of medicines, World Health
Organization (WHO), (86), (http://apps.who.int/medicinedocs/en/q/ and http://
digicollection.org/whoqapharm/)
3. Recommendations for health authorities to detect and deal with actions, activities and
behaviors that result in substandard/spurious/falsely-labelled/falsified/counterfeit medical
products, A68/33, annex 1, (129), (http://apps.who.int/gb/ebwha/pdf_files/WHA68/
A68_33-en.pdf)
4. Guidance on developing a national plan for preventing, detecting and responding to
actions, activities and behaviors that result in SF medical products. WHO (document
A70/23), (130), (https://www.who.int/medicines/regulation/ssffc/mechanism/A70_23-
en6-14.pdf)
05
Framework: Output
06
Rating Scale:  NOT IMPLEMENTED (NI): Market surveillance and control activities are not
communicated within the NRA. 07
 ONGOING IMPLEMENTATION (OI): The NRA is preparing to establish mechanisms for
communication of market surveillance and control activities, but there is no evidence 08
of results from such activities.
 PARTIALLY IMPLEMENTED (PI): There is evidence that the NRA has the elements (i.e., 09
procedures, documentation, management systems, and relevant information) and
the capacity to perform the processes mentioned in the indicator; however it has only
limited experience or a limited number of documented events.
 IMPLEMENTED (I): Market surveillance and control activities are appropriately
communicated within the NRA, and documentation is available to support that
communication within the NRA is well-established.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: MC06.02: Findings and regulatory decisions of market surveillance and control
activities are appropriately communicated to all national stakeholders including
the general public.

Maturity Level: 3

Page 176 of 340

WHO Global Benchmarking Tool (GBT)

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that market surveillance and control activities are properly
communicated to all relevant national stakeholders. For this activity, the intended
stakeholders include, but are not limited to, governmental agencies (e.g., police and
judicial authorities), non-governmental organizations, professional associations, customer
representative associations, and the public community.
01
Objective: The objective of this sub-indicator is to ensure that market surveillance and control
activities are actively communicated among all relevant stakeholders. These 02
communications, in turn, ensure regulatory effectiveness through involvement of these
key players. In addition, such communication increases awareness, encourages mutual 03
understanding and cooperation, and builds confidence and trust among all stakeholders,
including public trust in the regulatory system. 04

Market Surveillance and Control (MC): Indicators and Fact Sheets

Requirement: Communication to different stakeholders including the community of market surveillance
and control activities.

Evidence to review: The assessor should ask for and review:

1. Records of communication (i.e., paper based or electronic) proving communication of
market surveillance and control activities with other relevant stakeholders.

References: 1. Terms of reference for the Global Focal Point Network for SF medical products. WHO
(document A69/41), (131), (https://www.who.int/medicines/regulation/ssffc/mechanism/
A69_41-en6-8.pdf)
2. Recommendations for health authorities to detect and deal with actions, activities and
behaviors that result in substandard/spurious/falsely-labelled/falsified/counterfeit medical
products, A68/33, annex 1, (129), (http://apps.who.int/gb/ebwha/pdf_files/WHA68/
A68_33-en.pdf)
3. Guidance on developing a national plan for preventing, detecting and responding to
actions, activities and behaviors that result in SF medical products. WHO (document
A70/23), (130), (https://www.who.int/medicines/regulation/ssffc/mechanism/A70_23-
en6-14.pdf)

Framework: Output
05
Rating Scale:  NOT IMPLEMENTED (NI): Market surveillance and control activities are not
communicated to national stakeholders or the public. 06
 ONGOING IMPLEMENTATION (OI): The NRA is preparing to establish mechanisms for
communication of market surveillance and control activities to national stakeholders 07
and the public, but there is no evidence of results from such activities.
 PARTIALLY IMPLEMENTED (PI): There is evidence that the NRA has the elements (i.e., 08
procedures, documentation, management systems, and other relevant information)
and the capacity to perform the processes mentioned in the indicator; however, it has 09
only limited experience or a limited number of documented events.
 IMPLEMENTED (I): Market surveillance and control activities are communicated to
national stakeholders and the public.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: MC06.03: Findings and regulatory decisions of market surveillance and control
activities of common interest are appropriately communicated and shared with other
countries and regional and international organizations.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Page 177 of 340

WHO Global Benchmarking Tool (GBT)

Description: The assessor should verify that findings and regulatory decisions of market surveillance
and control programme of common interest are shared with other countries, regional
networks and international organizations (e.g. Rapid Alert Notifications and WHO Global
Surveillance and Monitoring System for SF medical products). The assessor should verify
that the relevant information is shared with external partners in a timely manner so that the
information remains valuable and has the potential to minimize adverse events in locations
outside the country.

Objective: The objective of this sub-indicator is to ensure that those findings and regulatory decisions 01
that are related to the market surveillance and control programme and that are of common
interest, are shared with other countries, regional networks and international organizations. 02
Sharing of market surveillance and control information, including decisions, is essential for
ensuring coordinated global efforts for access to medical products of reliable quality. 03

Requirement: Sharing of market surveillance and control information of interest with other countries and 04
regional and international organizations.

Market Surveillance and Control (MC): Indicators and Fact Sheets

Evidence to review: The assessor should ask for and review:
1. Agreements, MOUs and other documentation reflecting cooperation between the NRA
and other foreign entities for the purpose of sharing findings, data and decisions of market
surveillance and control programme.
2. Examples of communication with foreign entities for the purpose of sharing findings,
data and decisions of market surveillance and control programme.

References: 1. Recommendations for health authorities to detect and deal with actions, activities and
behaviors that result in substandard/spurious/falsely-labelled/falsified/counterfeit medical
products, A68/33, annex 1, (129), (http://apps.who.int/gb/ebwha/pdf_files/WHA68/
A68_33-en.pdf)
2. Terms of reference for the Global Focal Point Network for SF medical products. WHO
(document A69/41), (131), (https://www.who.int/medicines/regulation/ssffc/mechanism/
A69_41-en6-8.pdf)
3. Guidance on developing a national plan for preventing, detecting and responding to
actions, activities and behaviors that result in SF medical products. WHO (document
A70/23), (130), (https://www.who.int/medicines/regulation/ssffc/mechanism/A70_23-
en6-14.pdf)

Framework: Output 05
Rating Scale:  NOT IMPLEMENTED (NI): Market surveillance and control activities are not 06
communicated or shared with other countries or regional or international
organizations. 07
 ONGOING IMPLEMENTATION (OI): The NRA is preparing to establish mechanisms for
communication of market surveillance and control activities with other countries and 08
regional and international organizations, but there is no evidence of results from such
activities. 09
 PARTIALLY IMPLEMENTED (PI): There is evidence that the NRA has the elements (i.e.,
procedures, documentation, management systems, and other related information)
and the capacity to perform the processes mentioned in the indicator; however, it has
only limited experience or a limited number of documented events.
 IMPLEMENTED (I): Market surveillance and control activities are communicated and
shared with other countries and regional and international organizations.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Page 178 of 340

WHO Global Benchmarking Tool (GBT)

05 01

National Regulatory System (RS): Indicators and Fact Sheets

Licensing Establishments
(LI): Indicators and
Fact Sheets

WHO Global
Benchmarking 02
Tool (GBT) 03
for Evaluation of
04
National Regulatory
System of Medical 05
Products 06
07
08
09

Page 179 of 340

WHO Global Benchmarking Tool (GBT)

05. Licensing Establishments

(LI): Indicators and Fact Sheets

Function:
05 — LICENSING ESTABLISHMENTS (LI)
01
Description: In order to protect public health, licensing activities are of outstanding importance 02
and are considered fundamental, together with inspections activities, for guaranteeing
the quality, safety and efficacy of medical products used within or exported out of 03
the country. The National Regulatory Authority (NRA) is responsible for coordinating
licensing activities and should be supported by published and readily available 04
legal provisions, regulations and guidelines which ensure that licensing of facilities
throughout the supply chain is based on compliance with Good Practices (GXP) and 05
that the NRA is empowered to issue, suspend or revoke licenses for premises and

Licensing Establishments (LI): Indicators and Fact Sheets

establishments.

Premises, facilities, establishments and companies throughout the supply chain

should possess a license to operate issued by the NRA. These facilities include, but
are not limited to, manufacturers, distributors, wholesalers, importers, exporters and
retailers. The process of issuing licenses should be based on the implementation of and
compliance with quality standards of GXP. An inspection for confirmation of compliance
with GXP is required in order to grant or re-grant a license or approval of a substantial
modification.

An updated list or database of all licensed facilities should be published and publicly
available.

The GXPs considered most relevant for this function are good manufacturing practices
and good distribution practices, including good cold chain management practices.
Good clinical practices and good vigilance practices, which are generally excluded
from this function, are addressed, however, in other functions.

A general limitation to this function occurs when there is no domestic manufacturing

of medical products. In this case, the function cannot apply to manufacturers,
06
because none exist in the country. Nevertheless, the function will always apply to
distribution practices, including wholesaling. Another general limitation applies to
07
those countries that depend on the regulatory inspection function, without licensing, to
ensure compliance to GXPs at the premises, facilities, establishments and companies 08
throughout the supply chain. In the latter case, the whole function might not apply;
however, the assessor of this establishment licensing function should liaise with the 09
assessor of the regulatory inspection function to verify that proper and appropriate
controls are in place despite the absence of any licensing activities in the country.

Indicator: LI01 Legal provisions, regulations and guidelines required to define framework
for licensing activities.

Objective: The objective of this indicator is to ensure that licensing activities are supported by a
comprehensive set of legal provisions, regulations and guidelines which provide the
necessary mandate to implement all activities related to this regulatory function. There
should be a legal basis to establish the licensing system for the facilities throughout
the supply chain and to authorize the responsible entities to take the necessary actions.
The legislation should also provide a mandate that allows for adequate and proportional
sanctions, penalties and prosecutions for violations of the applicable legislation.
(Please refer to regulatory inspection function for further information on the regulatory
enforcement and compliance activities).

Category: 01. Legal provisions, regulations and guidelines

Page 180 of 340

WHO Global Benchmarking Tool (GBT)

Sub Indicator: LI01.01: There are legal provisions for licensing of facilities throughout the supply
chain and based on Good Practices (GXPs) compliance.

Maturity Level: 1

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that legislation establishes fundamental functions in the 01
regulatory authority appointed to license different type of facilities across the supply
chain (e.g., manufacturer, re-packager, re-labeller, distributor, wholesaler and retailer). The 02
assessor should ensure that, for the purpose of being licensed, legal provisions require the
responsible entity to conduct a GXP inspection prior to taking related decision. 03

Objective: The objective of this sub-indicator is to ensure the existence of legislation that 04
requires premises, facilities, establishments and companies throughout the supply
chain (including, but not limited to, manufactures, distributors, wholesalers, importers, 05
exporters and retailers) hold a license to operate that is issued by the National

Licensing Establishments (LI): Indicators and Fact Sheets

Regulatory Authority (NRA).

Requirement: Legal provisions, regulations and guidelines for licensing facilities

Evidence to review: The assessor should ask for and review:

1. Published legal provisions establishing the licensing of facilities throughout the supply
chain;
2. Relevant public information available from the website, official bulletins or other legal
publications;
3. Legal provisions authorizing the NRA to conduct GXP inspections;
4. Guidelines, procedures, forms, and instructions for applicants;
5. Guidance to applicants defining the circumstances that would warrant an application for
a new license or for renewal, expansion, or modification to an existing license.

References: 1. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations:
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical
Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf)
06
2. Guiding principles for small national drug regulatory authorities. In: WHO Expert
Committee on Specifications for Pharmaceutical Preparations: thirty-first report. Geneva: 07
World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No. 790), (5),
(http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf) 08
3. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization: forty-fifth 09
report. Geneva: World Health Organization; 1995: Annex 1 (WHO Technical Report Series,
No. 858), (3), (http://www.who.int/bloodproducts/publications/WHO_TRS_858_A1.pdf)
4. Effective drug regulation: A multicounty study. Geneva: World Health Organization; 2002,
(7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)
5. Guidelines for national authorities on quality assurance for biological products. In: WHO
Expert Committee on Biological Standardization: forty-second report. Geneva. World Health
Organization; 1992: Annex 2 (WHO Technical Report Series, No. 822), (2), (http://www.who.
int/biologicals/publications/trs/areas/biological_products/WHO_TRS_822_A2.pdf)
6. Guidelines on import procedures for pharmaceutical products, World Health
Organization (WHO), (22), (http://digicollection.org/whoqapharm/p/about and http://apps.
who.int/medicinedocs/en)

Framework: Structure/Foundation/Inpu

Page 181 of 340

WHO Global Benchmarking Tool (GBT)

 PARTIALLY IMPLEMENTED (PI): The legal basis was established recently (less
than one year ago) and it is at the implementation stage, so this practice is not yet
consolidated.
 IMPLEMENTED (I): The NRA has legal provisions in place requiring premises,
facilities, establishments and companies throughout the supply chain including,
but not limited to, manufactures, distributors, wholesalers, importers, exporters and
retailers, to hold a license to operate issued by the NRA.

Limitations and remarks:  A general limitation to this function occurs when there is no domestic manufacturing 01
of medical products. In this case, the function cannot apply to manufacturers,
because none exist in the country. Nevertheless, the function will always apply to 02
distribution, including wholesaling, practices. Another general limitation applies
to those countries that depend on the regulatory inspection function, without 03
licensing, to ensure compliance to GXP at the premises, facilities, establishments and
companies throughout the supply chain. In the latter case, the whole function might 04
not apply; however, the assessor of this establishment licensing function should
liaise with the assessor of the regulatory inspection function to verify that proper and 05
appropriate controls are in place despite the absence of any licensing activities in the

Licensing Establishments (LI): Indicators and Fact Sheets

country.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: LI01.02: There are legal provisions to empower the NRA to issue, suspend
or revoke licenses for establishments.

Maturity Level: 1

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify existing legislation establishes a requirement for compliance
with and respect for the quality standards of GXPs on the part of manufacturers of
pharmaceutical products and for the quality standards of Good Distribution Practice
on the part of wholesale companies. Compliance with these quality standards is
essential for ensuring that licensing establishments meet the requirements recommended
by the World Health Organization (WHO) to safeguard the quality, safety, and effectiveness
of their products. The NRA must verify a manufacturer’s adherence to Good Manufacturing
06
Practices before and after production starts. Therefore, it is imperative that the
NRA is empowered by legislation to both issue licenses to such establishments and to 07
suspend them when critical risks associated with lack of compliance with GXP
are identified. 08
The assessor should confirm the existence of laws, decrees and provisions related to
licensing and verify that the legislation empowers the NRA to suspend or revoke licenses 09
for establishments.

Objective: The objective of this sub-indicator is to ensure that legislation empowers the NRA to not
only issue licenses, but also to suspend or revoke licenses for establishments.

Requirement: Legal provisions, regulations and guidelines on issuance, suspension, withdrawal or

cancellation of licenses.

Evidence to review: The assessor should ask for and review:

1. Legal provisions authorizing the NRA to suspend or revoke a license in case
requirements are not met;
2. Relevant public information available from the website, official bulletins, or other legal
publications.

Page 182 of 340

WHO Global Benchmarking Tool (GBT)

In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: thirty-fifth

report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical Report Series, No.
885), (1), (http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
3. Guiding principles for small national drug regulatory authorities. In: WHO Expert
Committee on Specifications for Pharmaceutical Preparations: thirty-first report. Geneva:
World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No. 790), (5),
(http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)

Framework: Structure/Foundation/Input 01
Rating Scale:  NOT IMPLEMENTED (NI): There are no legal provisions empowering the NRA to issue 02
licenses and to suspend or halt production.
 ONGOING IMPLEMENTATION (OI): The NRA is preparing to establish a legal basis for 03
this indicator, but there is no evidence of any results from such activities.
 PARTIALLY IMPLEMENTED (PI): The legal basis was established recently (less 04
than one year ago) and it is at the implementation stage, so this practice is not
yet consolidated; or the NRA carries out these actions, but legal provisions do not 05
explicitly empower it to do so.

Licensing Establishments (LI): Indicators and Fact Sheets

 IMPLEMENTED (I): The NRA has the legal bases and also consistently maintains
documentation of the results of related activities over time. The legal provisions
explicitly indicate that the NRA grants licenses and may revoke them or apply health
sanctions, as the case may be, for any noncompliance it finds. There is evidence that
the NRA is following through on these actions.

Limitations and remarks:  In some cases, license revocation is not the responsibility of the NRA. Instead, other
institutions may have that responsibility (e.g., police authorities). In any case, the
inspectors should have the authority to halt production.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: LI01.03: There are legal provisions that require that the NRA to be informed,
for the purpose of notification or approval, in case post-licensure changes or
variations are made.

Maturity Level: 3

Scope: 1. Medicines
06
2. Vaccines
07
Description: The assessor should verify the existence of legal provisions that require a responsible
person from the establishment inform the NRA for notification or approval of any changes 08
or variations to the conditions under which the initial license was issued. This provision
applies to any change or variation which may affect the quality, safety or efficacy of 09
medical products.
Establishments may make changes at their facilities (e.g. introduce a new manufacturing
line or new product). However, because such changes may create new risk factors that
might impact the quality, safety, or efficacy of the products being manufactured, the NRA
must assess the situation to decide whether to approve such changes and include them in
the corresponding manufacturing license.

Objective: The objective of this sub-indicator is to ensure that legal provisions or regulations
provide the necessary mandate for the NRA to require that a responsible person from the
establishment inform the NRA, for purpose of notification or approval, of any changes or
variations to the conditions under which the initial license was issued.

Requirement: Legal provisions, regulations and guidelines on post-licensure changes or variations

Evidence to review: The assessor should ask for and review:

1. Published legal provisions regarding the obligation to report changes or variations to the
original license;

Page 183 of 340

WHO Global Benchmarking Tool (GBT)

2. Relevant published guidelines, procedures, forms, and instructions for applicants;

3. Evidence of approved and rejected amendments.

References: 1. Effective drug regulation: A multicounty study. Geneva: World Health Organization; 2002,
(7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)
2. National drug regulatory legislation: guiding principles for small drug regulatory authorities.
In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: thirty-fifth
report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical Report Series, No.
885), (1), (http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf) 01
3. Guiding principles for small national drug regulatory authorities. In: WHO Expert
Committee on Specifications for Pharmaceutical Preparations: thirty-first report. Geneva: 02
World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No. 790), (5),
(http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf) 03

Framework: Structure/Foundation/Input 04

Rating Scale:  NOT IMPLEMENTED (NI): There is no legal basis for the indicator. 05
 ONGOING IMPLEMENTATION (OI): The NRA is preparing to establish the legal basis,

Licensing Establishments (LI): Indicators and Fact Sheets

but there is no evidence of results associated with such activities.
 PARTIALLY IMPLEMENTED (PI): The legal basis was established recently, and it is at
the implementation stage, so this practice is not yet consolidated.
 IMPLEMENTED (I): The NRA has such legal bases and also consistently maintains
documentation of the results of related activities over time. There is a legal provision
requiring notification, and there are records that both license holders and the NRA are
adhering to this legal provision.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: LI01.04: There are guidelines on the procedures to apply for a license and on content
and format of the license application.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines
06
Description: The assessor should verify the existence of published guidance that specifies the
content of the application, the forms that are required and the procedures to follow when 07
submitting a license application. In order to ensure proper interpretation and complete
compliance, it is important to verify that: 08
1. the instructions regarding the content and format of license application are clear;
2. the instructions that describe the procedure to follow when submitting an application 09
license are clear;
3. the NRA has evidence, preferably public, that states that the guidelines are to be followed;
4. the requirements state that products at the stage of research and development also are
included;
5. The guidelines clearly establish that the licensing requirements apply to domestic,
foreign, public, and private manufacturers, distributers, wholesalers, importers, exporters
and retailers;
6. the NRA applies these requirements consistently, and has documentary evidence,
preferably public, to demonstrate this.

Furthermore, the assessor should verify the guidelines are available, intelligible and
properly communicated to the target audience. For these communications, a push rather
than a pull approach should be employed.

Objective: The objective of this sub-indicator is to ensure that guidelines for submitting an application
for a license are available to those governed by the regulations. These guidelines should
address the content of the application and its format. Availability of such guidelines will

Page 184 of 340

WHO Global Benchmarking Tool (GBT)

contribute to proficiency in all aspects of GXP covered by the legislation and to proper
interpretation and implementation of the guidelines. Availability of guidelines also will help
the NRA to be consistent in application of this requirement. Assessor should note that the
publication and communication of these guidelines to the target audience is as critical as
the guidelines themselves. For these communications, a push rather than a pull approach
should be employed.

Requirement: Guidelines on content and format of the application and on procedure to follow when
submitting a license application. 01
Evidence to review: The assessor should ask for and review: 02
1. Published guidelines for licensing applications, including instructions or guidelines for
applicants. 03
2. Evidence that administrative instructions exist and that they are all available to
establishments applying for licenses. 04
3. Official requirements for the submission of licensing applications;
4. Documented evidence from the NRA demonstrating that this indicator is implemented (if 05
no publicly-available evidence is available)

Licensing Establishments (LI): Indicators and Fact Sheets

References: 1. Effective drug regulation: A multicounty study. Geneva: World Health Organization; 2002,
(7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)
2. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations:
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical
Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf)
3. Guiding principles for small national drug regulatory authorities. In: WHO Expert
Committee on Specifications for Pharmaceutical Preparations: thirty-first report. Geneva:
World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No. 790), (5),
(http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There are no guidelines for the content and format of the
license application.
 ONGOING IMPLEMENTATION (OI): The NRA is preparing to establish such
guidelines, but there is no evidence of results associated with such activities, or the
06
guidelines are available within the NRA however they are not available, intelligible or
communicated to the target audience. 07
 PARTIALLY IMPLEMENTED (PI): The guidelines were established recently and it is at
the early implementation stage, so this practice is not yet consolidated. 08
 IMPLEMENTED (I): The NRA has guidelines for producers on the content of the
application and its format, and on the procedures to follow when submitting an 09
application for a production license in accordance with WHO standards. The NRA also
consistently maintains documentation of the results of related activities over time.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: LI01.05: There are legal provisions that require manufacturers to inform
the NRA about the appointed qualified and authorized person for the purpose
of acknowledgment or approval.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that legal provisions or regulations exist to obligate the
manufacturer to designate a qualified and authorized person.

Page 185 of 340

WHO Global Benchmarking Tool (GBT)

Objective: The objective of this sub-indicator is to ensure the existence of legal provisions or
regulations that provide necessary mandate to the NRA to obligate the manufacturers to
designate a qualified and authorized person.

Requirement: Legal provisions, regulations and guidelines on the qualified and authorized person

Evidence to review: The assessor should ask for and review:

1. Legal provisions or regulations authorizing the NRA to approve or acknowledge the
qualified and authorized person appointed by the manufacturer; 01
References: 1. Effective drug regulation: A multicounty study. Geneva: World Health Organization; 2002, 02
(7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)
2. National drug regulatory legislation: guiding principles for small drug regulatory 03
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations:
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical 04
Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf) 05
3. Guiding principles for small national drug regulatory authorities. In: WHO Expert

Licensing Establishments (LI): Indicators and Fact Sheets

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There is no legal basis for the indicator.
 ONGOING IMPLEMENTATION (OI): The NRA is preparing to establish a legal basis for
this requirement, but there is no evidence of any results from those preparations.
 PARTIALLY IMPLEMENTED (PI): The legal basis was established recently (less
than one year ago) and it is at the implementation stage, so this practice is not yet
consolidated.
 IMPLEMENTED (I): The NRA has legal provisions in place requiring manufacturers
to designate a qualified and authorized person and to inform the NRA of this for
notification and approval purposes.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).
06
Indicator: LI02 Arrangement for effective organization and good governance.
07
Objective: The objective of this indicator is to ensure the implementation of effective organization 08
and good governance practices at the organizational structures in charge of establishment
licensing activities. 09
Category: 02. Organization and governance

Sub Indicator: LI02.01: There is a defined structure with clear responsibilities to conduct
establishments licensing activities.

Maturity Level: 2

Scope: 1. Medicines
2. Vaccines

Description: The assessor should identify the organization designated to establish, implement
or maintain the licensing regulatory function, as well as the specific organizational
structures taking on the different relevant activities. Responsibilities, duties and roles of
these structures should be clearly defined and documented. If more than one structure
is involved, the assessor should check the ways and approaches by which coordination
between these structures is taking place.

Page 186 of 340

WHO Global Benchmarking Tool (GBT)

Objective: The objective of this sub-indicator is to ensure effective organization and good governance
of establishment licensing activities and to ensure that these licensing activities are taken
over by defined organizational structures with clear roles and responsibilities.

Requirement: Roles and responsibilities of the structures in charge of establishment licensing activities.

Evidence to review: The assessor should ask for and review:

1. Organization chart of the organization responsible for the implementation of licensing
activities along with identification of the specific structures implementing the function. 01
2. Documentation clarifying roles and responsibilities of the organizational structures
implementing licensing activities. This may include administrative decrees, terms of 02
reference, or other relevant documents
03
References: 1. Effective drug regulation: A multicounty study. Geneva: World Health Organization; 2002,
(7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf) 04
2. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: 05
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical

Licensing Establishments (LI): Indicators and Fact Sheets

Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf)
3. Guiding principles for small national drug regulatory authorities. In: WHO Expert
Committee on Specifications for Pharmaceutical Preparations: thirty-first report. Geneva:
World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No. 790), (5),
(http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There is no defined structure in charge of establishment
licensing activities.
 ONGOING IMPLEMENTATION (OI): A mandate to establish a structure in charge of
licensing activities is available however the structure itself is not yet established.
 PARTIALLY IMPLEMENTED (PI): Structure in charge of licensing activities is newly
established and mandated however the regular work and practice of this structure is
not yet consolidated.
 IMPLEMENTED (I): There is a defined structure in charge of establishment licensing
activities with clear and well-documented roles and responsibilities.
06
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 07
Sub Indicator: LI02.02: Documented procedures and mechanisms are implemented to ensure 08
the involvement and communication between all stakeholders relevant to
establishments licensing activities. 09
Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that documented procedures and mechanisms are
implemented to ensure the involvement and communication between different entities
and departments relevant to licensing activities. These entities may be inside or outside
the NRA (e.g., laboratory, inspection and enforcement departments, police authorities, and
professional associations).
In case of a decentralized establishment licensing function, an information exchange
system, mechanism or platform must be established and used so that appropriate
communication between the central and peripheral structures is ensured. As one example,
the decentralized entity can receive requests or guidance from the central authority and
report back to it. The availability of such communication mechanisms will also encourage
consistency among different peripheral structures.

Page 187 of 340

WHO Global Benchmarking Tool (GBT)

Objective: The objective of this sub-indicator is to ensure the existence and implementation of
documented procedures and mechanisms to guide the involvement and communication
among the different entities and departments. These activities will encourage appropriate
organization and good governance of the function.

Requirement: Organization and governance

Evidence to review: The assessor should ask for and review:

1. Guidelines or standard operating procedures (SOPs) that are related to external and 01
internal communications;
2. Documentation of paths of communication and reporting; 02
3. Platforms for information sharing and exchange.
03
References: 1. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: 04
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical
Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/documents/s21964en/ 05
s21964en.pdf)

Licensing Establishments (LI): Indicators and Fact Sheets

2. Guiding principles for small national drug regulatory authorities. In: WHO Expert
Committee on Specifications for Pharmaceutical Preparations: thirty-first report. Geneva:
World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No. 790), (5),
(http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
3. Effective drug regulation: A multicounty study. Geneva: World Health Organization; 2002,
(7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)
4. Quality systems requirements for national good manufacturing practice inspectorates.
In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: thirty-sixth
report. Geneva: World Health Organization; 2002: Annex 8 (WHO Technical Report Series,
No. 902), (8), (http://apps.who.int/medicinedocs/documents/s22112en/s22112en.pdf)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There are no information exchange procedures or
mechanisms among different stakeholders of the licensing function or between the
central authority and the decentralized entities.
 ONGOING IMPLEMENTATION (OI): The NRA is developing an information exchange
mechanism, but it is not yet ready, or exchanges are being conducted without an
established methodology.
06
 PARTIALLY IMPLEMENTED (PI): The NRA recently established an information
exchange procedure and mechanism and it is at the implementation stage, so this 07
practice is not yet consolidated.
 IMPLEMENTED (I): There are established, implemented and maintained information 08
exchange procedures and mechanisms among different stakeholders of the licensing
function and between the central authority and the decentralized entities. 09
Limitations and remarks:  The scope of this sub-indicator is not confined only to routine activities, but also
applies, most importantly, to regulatory decisions and actions.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Indicator: LI03 Human resources to perform licensing activities.

Objective: The objective of this indicator is to ensure that all entities within an NRA are adequately
resourced with a trained, experienced and skilled workforce that is empowered to fully
perform the licensing function. This will ensure that medical products licensing processes
or activities are performed in accordance with international best practices.
The objective of this indicator is to evaluate the human resource capacity of the entities
with respect to the number of personnel, the skills and experience of the personnel, and
the overall composition the workforce, with the goal of evaluating whether the workforce
possesses the specific expertise required to perform the licensing function.

Page 188 of 340

WHO Global Benchmarking Tool (GBT)

Category: Resources (HR, FR, infrastructure and equipment)

Sub Indicator: LI03.01: Sufficient competent staff (i.e., education, training, skills and experience)
are assigned to perform licensing activities.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines 01
Description: The assessor should verify that the human resources assigned to perform licensing 02
activities should be sufficient with respect to numbers and competent with respect to the
requisite skills, education, experience and training. There should be technical documents 03
and SOPs that provide guidance on the required background for licensing activities
and that consider the requirements for educational background, competencies, skills, 04
experience, and training.
The assessor should verify that the NRA estimated the number of staff required to 05
effectively and efficiently perform licensing function and that the NRA actually recruited

Licensing Establishments (LI): Indicators and Fact Sheets

that number. In addition, the assessor should verify that these competency requirements
are well-established and maintained by the NRA. Metrics and statistics on the different
activities performed as well as performance indicators can be used for estimating the
adequacy of the number of the assigned staff. The assessor should also verify that the
competency of the assigned staff is built, maintained and improved through recruitment as
well as continuous on- the-job training.

Objective: The objective of this sub-indicator is to ensure the existing human resources for licensing
is sufficient, in terms of numbers, experience, and specific competencies, to perform all the
activities along the entire licensing chain.

Requirement: Sufficient number of competent human resources in charge of licensing activities.

Evidence to review: The assessor should ask for and review:

1. Evidence that the number of staff members involved in each of the documented
activities along the entire licensing process flow is adequate.
2. Evidence that the systems and structures are in place to ensure appropriate placement
of staff with respect to competence and skills.
3. Evidence that the system and structures have been implemented. The documentation
06
should include the records to verify that the staff competence is appropriate for the job
requirements. 07
4. Evidence that the professional profiles of the human resources engaged in licensing
activities are appropriate with respect to education, skills, and expertise, to perform a 08
particular function along the licensing chain. Documentation should include a list of the
requisite skills and training for each position. 09
5. Recruitment plan.

References: 1. WHO good manufacturing practices for pharmaceutical products: main principles,
(51), (http://digicollection.org/whoqapharm/p/about and http://apps.who.int/
medicinedocs/en)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): The NRA does not have enough competent staff (i.e.,
education, training, skills and experience) to perform licensing activities
 ONGOING IMPLEMENTATION (OI): The NRA has recently developed a plan to recruit
adequate competent staff; however, the plan has not been implemented.
 PARTIALLY IMPLEMENTED (PI): The NRA has initiated the implementation of
the human resources development plan; however, there is need to complete the
competency profile.
 IMPLEMENTED (I): The NRA has a sustained number of competent staff (i.e.,
education, training, skills and experience) assigned to perform licensing activities.

Page 189 of 340

WHO Global Benchmarking Tool (GBT)

Sub Indicator: LI03.02: Duties, functions, and responsibilities of the staff in charge of licensing
activities are established and updated in the respective job descriptions. 01
Maturity Level: 3 02

Scope: 1. Medicines 03
2. Vaccines
04
Description: The assessor should verify that procedures are in place to maintain a current and updated
structure for managing job descriptions for personnel participating in licensing activities. 05
In addition, job descriptions should address current staff duties, responsibilities and

Licensing Establishments (LI): Indicators and Fact Sheets

the requisite competencies. A job description with this format and content should be
established and implemented for all staff. The management of job descriptions should be
supported by a guidance document that provides direction on when and how to update
the information, and where the information should be kept for easy access. The guidance
document should present the appropriate duties and responsibilities that are assigned to
each member of the organization involved in licensing activities. Thus, the professional
profiles of staff are reflected in their respective roles and responsibilities within the NRA.
There should be procedures to guide responsible persons to document that duties,
functions and responsibilities are revised and kept up to date. In addition, procedures
should be available to guide the keeping and documenting of up to date work schedules
and enforcing the implementation of the documented guidelines and procedures.

Requirement: Duties, roles and responsibilities of the staff relevant to licensing activities.

Evidence to review: The assessor should ask for and review:

06
1. Procedure and guidelines that guide placement of staff members within the NRA;
2. The professional profiles of staff (i.e., job descriptions) and documentation that they are 07
related to their current roles and duties;
3. The professional profiles of the external experts and documentation that the profiles 08
provide a composition that is complete and consistent with that prescribed in the legal
provisions; 09
4. Procedures to guide the documentation of up to date duties and work schedules, and to
enforce the implementation of the documented guidelines and procedures;
5. Job descriptions for designated staff.

Page 190 of 340

WHO Global Benchmarking Tool (GBT)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence of defined or established duties,
functions, responsibilities, respective job descriptions and necessary required
competencies.
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted or developed the role
and responsibilities document but it has not yet been implemented.
 PARTIALLY IMPLEMENTED (PI): The NRA has initiated implementation of this
requirement but it has not been defined or followed for all staff or the roles and 01
responsibilities documents, including staff job descriptions, are not up to date.
 IMPLEMENTED (I): The NRA has defined and established all required duties, 02
functions, and responsibilities, and respective job descriptions are up-to-date.
03
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 04

Sub Indicator: LI03.03: Training plan developed, implemented and updated at least once a year 05
for staff in charge of licensing activities.

Licensing Establishments (LI): Indicators and Fact Sheets

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that training plans are developed, implemented and updated
at least once every year to reflect the current situation by considering education and
experience of the staff. The training plan should be complemented with guidelines or
similar documents that guide the development and implementation of training plans. The
assessor should ensure that induction training for new staff as well as continued on- the-
job for staff is planned and implemented. There should be procedures to approve the
training plan and the budget allocated for implementing and updating the training plan. The
plans should
present clearly defined training goals and should include training in certain topics and skills
to address identified deficiencies. Learning objectives, training methods and activities,
evidence of learning, and evaluation and assessment of training should be documented.
This documentation should confirm that the learning objectives were achieved and were
designed to address weaknesses within the entities. Procedures should be in place to
06
ensure that a training plan is developed, implemented and updated at least once every
year. The assessor should verify that there is a system in place for monitoring the 07
implementation and effectiveness of the training plan and for documenting the skills
acquired in training activities for internal and external experts. 08
The assessor should verify that inspectors in charge of Good Manufacturing Practices
inspections associated with manufacturer licensing activities receive a minimum of ten 09
(10) training days per year as recommended by the WHO and Pharmaceutical Inspection
Cooperation Scheme guidelines.

Objective: The objective of this sub-indicator is to ensure that a training plan for staff exists, and that
it is implemented and updated annually. Through the training plan, NRA can be sure that
competency of staff in charge of licensing activities is maintained and enhanced.

Requirement: Implementation of training plan

Evidence to review: The assessor should ask for and review:

Page 191 of 340

WHO Global Benchmarking Tool (GBT)

4. SOP for developing and maintaining the training plan.

5. Evidence that the NRA has investigated and identified training needs.
6. List of trainings performed.
7. Example records for training activities.

Licensing Establishments (LI): Indicators and Fact Sheets

(7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)
5. Quality systems requirements for national good manufacturing practice inspectorates.
In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: thirty-
sixth report. Geneva: World Health Organization; 2002: Annex 8 (WHO Technical
Report Series, No. 902), (8), (http://apps.who.int/medicinedocs/documents/s22112en/
s22112en.pdf)

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): There is no systematic training program including training
plan (or matrix).
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted or developed the
training plan but there is no evidence of implementation.
 PARTIALLY IMPLEMENTED (PI): The NRA has developed and initiated the training plan
implementation. However, the NRA has not fulfilled all required planed training or has
been applying the plan for less than two years.
 IMPLEMENTED (I): The NRA has an updated training plan developed that is
supported by adequate records to demonstrate effective plan implementation,
including induction training for new staff and routine on-the-job training for recruited
06
staff.
07
Limitations and remarks:  Training plans must be updated regularly; ideally on an annual basis, but not less
frequently that once every two years. 08
 Some regulatory functions may include many training activities that are not
incorporated in the institutional training programme. Such training normally is offered 09
by invitation. In this case, the assessor should recognize reports from non-routine
licensing -relevant training not included in the NRA training plan.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: LI03.04: The NRA generates and maintains records of staff training activities
and training effectiveness verification.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Page 192 of 340

WHO Global Benchmarking Tool (GBT)

allocate a budget for continuous staff capacity building and development. The assessor
should check that there is an evaluation or assessment mechanism to verify the quality
of learning, and to confirm that learning objectives are achieved. Documentation should
include an inventory (i.e., soft and/or hard) system that records all impactful and non-
impactful trainings and identifies all staff members who participated. A system to measure
or estimate impact of trainings should be established.

Objective: The objective of this sub-indicator is to ensure that training organized by the NRA or
responsible regulatory authority is adequately documented and that the training records 01
are adequately maintained and kept. Staff training records are considered an integral part
of staff file and are a tool for measuring and tracking staff competency, development and 02
adequacy.
03
Requirement: Training records
04
Evidence to review: The assessor should ask for and review:
1. Guidelines or similar documents that guide the NRA to generate and maintain records 05
of staff training activities;

Licensing Establishments (LI): Indicators and Fact Sheets

2. Evaluations of training effectiveness;
3. The training inventory, and procedures for completing the inventory;
4. Examples of archived records of staff training, and procedures for the archiving
system.

References: 1. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations:
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical
Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf)
2. Quality management systems — Requirements. International Standard ISO 9001:2015.
Geneva. International Organization for Standardization, 2015, (4), (https://www.iso.org/iso-
9001-quality-management.html)
3. Guiding principles for small national drug regulatory authorities. In: WHO Expert
Committee on Specifications for Pharmaceutical Preparations: thirty-first report. Geneva:
World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No. 790), (5),
(http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
4. Effective drug regulation: A multicounty study. Geneva: World Health Organization; 2002,
(7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)
06
Framework: Output 07
Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence that the NRA generates and 08
maintains records of staff training activities.
 ONGOING IMPLEMENTATION (OI): The NRA has recently initiated plans to 09
generate, document and keep records of staff training activities, however they are
not yet followed.
 PARTIALLY IMPLEMENTED (PI): The NRA has recently initiated plans to generate,
document and keep records of staff training activities but they are not fully
followed for all training activities or they have been established for less than two
years.
 IMPLEMENTED (I): The NRA generates and maintains records of staff training
activities

Limitations and remarks:  The assessor should note that some NRAs out-source training including staff
capacity development activities. In this case the assessor should request the
identity of the provider, as well as evaluations of the provider. The assessor may
request the professional profiles of tutors or resource persons used to offer
training. The assessor may also request records covering the archiving systems
in place.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-
indicator will always apply for all benchmarked NRAs).

Page 193 of 340

WHO Global Benchmarking Tool (GBT)

Indicator: LI04 Procedures established and implemented to perform licensing activities.

Objective: The objective of this indicator is to ensure that the NRA, when supported by appropriate
mandates, effective organization and governance, and essential resources, is actually
able to perform the licensing activities. These activities are performed through standard
procedures and work instructions in order to ensure consistency, effectiveness, efficiency,
impartiality and proportionality of the activities.
There should be written procedures that define how relevant data are routinely shared 01
among key personnel engaged in licensing activities and on how any actions taken are
reviewed for appropriateness. Documented evidence to be assessed include procedures 02
for review and sharing of relevant data between key players, for review of reports of
notifications, investigations, data analyses, and committee meetings, and for taking 03
appropriate actions.
04
Category: Regulatory process
05
Sub Indicator: LI04.01: Procedures for assessment of applications for licensing activities,

Licensing Establishments (LI): Indicators and Fact Sheets

including license issuance, renewal, modification or revocation, are established
and documented.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that different phases of licensing activities, including
application receiving, screening, processing, assessing, and decision-making (e.g., issue,
renew, amend, modify, suspend or revoke) are well established and documented. With
particular emphasis on license renewals, amendments and modifications, the relevant
procedures and mechanisms should ideally be based on Quality Risk Management
principles. When applicable, the validity of the establishment license should be stated in
the respective certificate in order to clarify status and to avoid misunderstandings.

Objective: The objective of this sub-indicator is to ensure consistency and standardization of

licensing activities through established and documented procedures and mechanisms that
are complemented with the necessary tools.
06
Requirement: Regulatory process 07
Evidence to review: The assessor should ask for and review: 08
1. SOPs, checklists and records for different licensing activities (e.g. issuance, renewal,
modification, and revocation); 09
2. Evaluation reports provided by the NRA and other documented evidence of decisions
that were based on the procedures.

Page 194 of 340

WHO Global Benchmarking Tool (GBT)

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): There are no procedures for making decisions on license
issuance, renewal, modification, or revocation.
 ONGOING IMPLEMENTATION (OI): The NRA is preparing procedures for making
decisions on licensing activities, but they are not completed so there is no evidence of
results from such activities.
 PARTIALLY IMPLEMENTED (PI): Procedures for different licensing activities are newly
established and at the implementation stage, so this practice is not yet consolidated. 01
 IMPLEMENTED (I): The NRA has procedures for making decisions on license
issuance, renewal, modification, or revocation; the NRA applies the procedures 02
consistently, and reliably maintains documentation of the results of related activities
over time. 03

Limitations and remarks:  In some countries, license renewal might not be applicable. Rather the license is 04
maintained based on regular inspections for compliance with GXPs. In this case,
the requirement to document license renewals will not apply provided the assessor 05
confirms the proper implementation of regular regulatory inspections.

Licensing Establishments (LI): Indicators and Fact Sheets

 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: LI04.02: Inspection is required for granting or re-granting a license or approval
of a substantial modification.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the existence and implementation of standard procedures for
the inspection process that is required before granting or re-granting a license or approval
of a substantial modification. The inspection should be carried out in accordance with the
official guidelines and in accordance with a formal inspection plan.

Objective: The objective of this sub-indicator is to ensure that the inspectorate has established
procedures and other mechanisms to consistently perform inspections before granting
or re-granting a license or an approval of a substantial modification. Also, the objective is
06
to ensure consistency of licensing and inspection procedures and to ensure regulatory
compliance of establishments. These activities will promote the ultimate goal of protecting 07
public health.
08
Requirement: Regulatory process
09
Evidence to review: The assessor should ask for and review:
1. Procedures for the inspection process.
2. SOPs, checklists and records for different licensing activities (e.g. issuance, renewal,
modification, or revocation.
3. Examples of inspection reports done prior to license issuance, renewal, amendment
or revocation.
4. Procedures for the NRA to conduct GXP inspections.

References: 1. Quality systems requirements for national good manufacturing practice inspectorates.
In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: thirty-sixth
report. Geneva: World Health Organization; 2002: Annex 8 (WHO Technical Report Series,
No. 902), (8), (http://apps.who.int/medicinedocs/documents/s22112en/s22112en.pdf)
2. Guiding principles for small national drug regulatory authorities. In: WHO Expert
Committee on Specifications for Pharmaceutical Preparations: thirty-first report. Geneva:
World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No. 790), (5),
(http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
3. Quality management systems — Requirements. International Standard ISO 9001:2015.

Page 195 of 340

WHO Global Benchmarking Tool (GBT)

Geneva. International Organization for Standardization, 2015, (4), (https://www.iso.org/iso-

9001-quality-management.html)
4. Effective drug regulation: A multicounty study. Geneva: World Health Organization; 2002,
(7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)
5. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations:
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical
Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf) 01
Framework: Process 02

Rating Scale:  NOT IMPLEMENTED (NI): License issuances and renewals and approvals of 03
substantial modification are never made in conjunction with GXP inspections.
 ONGOING IMPLEMENTATION (OI): License issuances and renewals and approvals of 04
substantial modification are sometimes made in conjunction with GXP inspections.
 PARTIALLY IMPLEMENTED (PI): License issuances and renewals and approvals of 05
substantial modification have been made in conjunction with GXP inspections for a

Licensing Establishments (LI): Indicators and Fact Sheets

short time (less than one year) so this practice is not yet consolidated.
 IMPLEMENTED (I): License issuances and renewal and approvals of substantial
modification are always made in conjunction with GXP inspections.

Limitations and remarks:  In some cases, a license or approval of a substantial modification may be granted
without inspection. However, these should be seen as exceptional situations and need
to be justified, well described and compatible with guidelines. Also, these activities are
tightly linked to transparency.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs)

Sub Indicator: LI04.03: There are clearly defined timelines for the assessment of applications.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines
06
Description: The assessor should verify that clear timelines have been defined for the different steps
in the licensing process as well as for the different types of license applications, including 07
new applications and applications for license renewals or modifications. These timelines
should be clear and understandable to the target audience. The assessor should review 08
examples of the processing of different applications and ensure that a proper monitoring
system is in place to verify adherence to the stated timelines. 09
Objective: The objective of this sub-indicator is to ensure the existence and implementation of
standard procedures for defining clear timelines for the processing of license applications.

Requirement: Regulatory process

Evidence to review: The assessor should ask for and review:

1. Records of license processing that cover the steps from initial application up to issuance
of final decision. Assessor review should focus on timelines.
2. Records from the timelines tracking system, if available.

References:

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): There are no timelines for different licensing
activities.

Page 196 of 340

WHO Global Benchmarking Tool (GBT)

 ONGOING IMPLEMENTATION (OI): The NRA is drafting timelines for different

licensing activities however these are not yet implemented and no results are
yet achieved.
 PARTIALLY IMPLEMENTED (PI): The NRA recently established timelines for
different licensing activities so this practice is not yet consolidated.
 IMPLEMENTED (I): There are clear timelines, which are tracked, for different
licensing activities.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will 01
always apply for all benchmarked NRAs).
02
Sub Indicator: LI04.04: The same criteria are used for the licensing of domestic, public and private
establishments regardless of ownership. 03

Maturity Level: 3 04

Scope: 1. Medicines 05
2. Vaccines

Licensing Establishments (LI): Indicators and Fact Sheets

Description: The assessor should verify that the conduct of licensing activities and application of the
relevant standards, procedures, and GXPs is consistent and unbiased regardless of the
source of production (e.g., domestic or foreign, public or private sector).

Objective: The objective of this sub-indicator is to ensure consistent and unbiased conduct of
licensing activities and application of relevant standards, procedures and GXPs.

Requirement: Regulatory process

Evidence to review: The assessor should ask for and review:

1. SOPs or similar documents that ensure that the same licensing criteria are used
regardless of ownership of the establishment (e.g., domestic, foreign, public or private).
2. Sample licensing records for establishments from different ownership categories (e.g.,
domestic, foreign, public or private).

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): The NRA does not apply the same licensing criteria
for domestic, public and private establishments, including for products at the
research and development stage.
 ONGOING IMPLEMENTATION (OI): The NRA is preparing to establish procedures
to apply the same criteria, regardless of ownership, for licensing of domestic,
public and private establishments, including for products at the research and
development stage. However, there is no evidence of any results associated with
such activities.

Page 197 of 340

WHO Global Benchmarking Tool (GBT)

 PARTIALLY IMPLEMENTED (PI): The NRA recently established procedures to

apply the same criteria, regardless of ownership, for licensing of domestic,
public and private establishments, including for products at the research and
development stage. These are at the implementation stage, so these practices
are not yet consolidated.
 IMPLEMENTED (I): The NRA applies the same criteria, regardless of ownership, for
licensing of domestic, public and private establishments, including for products at the
research and development stage. The NRA consistently maintains documentation of
the results of related activities over time. 01
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will 02
always apply for all benchmarked NRAs).
03
Indicator: LI05 Mechanism in place to monitor regulatory performance and output.
04

Objective: The objective of this indicator is to ensure the existence and implementation of a system 05
or mechanism for monitoring regulatory performance and output and for using that

Licensing Establishments (LI): Indicators and Fact Sheets

information to estimate the effectiveness and efficiency of the licensing function.

Category: 09. Monitoring progress and assessing outcomes and impact

Sub Indicator: LI05.01: A database is established and regularly updated that includes all licensing
applications received, approved, refused, suspended or withdrawn, along with the
essential documentation for each application.

Maturity Level: 4

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that, within the NRA, there is a database of all the applications
received, approved, refused, suspended or withdrawn. The database should also include
the essential documentation for each application. At a minimum, documentation should
include the applications received and records of assessments done, and regulatory
decisions made.
06
Objective: The objective of this sub-indicator is to ensure consistency and traceability of licensing
activities through the establishment and maintenance of a database of all the applications 07
received, approved, refused, suspended or withdrawn, along with the essential
documentation for each application. 08
Requirement: Monitoring progress and assessing outcomes and impact 09
Evidence to review: The assessor should ask for and review:
1. The NRA internal database of the licensing applications along with the regulatory
decisions made for each application (i.e., approved, rejected, denied, withdrawn, cancelled,
or suspended).

Page 198 of 340

WHO Global Benchmarking Tool (GBT)

4. Quality management systems — Requirements. International Standard ISO 9001:2015.

Geneva. International Organization for Standardization, 2015, (4), (https://www.iso.org/iso-
9001-quality-management.html)

Framework: Output

Rating Scale:  NOT IMPLEMENTED (NI): There is no internal database of licensing applications
along with their regulatory decisions (i.e., approved, rejected, denied, withdrawn,
cancelled, or suspended). 01
 ONGOING IMPLEMENTATION (OI): The NRA is working on the establishment of an
internal database of licensing applications along with their regulatory decisions 02
(i.e., approved, rejected, denied, withdrawn, cancelled, or suspended) however the
database is not yet established. 03
 PARTIALLY IMPLEMENTED (PI): The NRA has recently established an internal
database of licensing applications along with their regulatory decisions (i.e., approved, 04
rejected, denied, withdrawn, cancelled, or suspended), however the outputs and
outcomes of this practice are not yet documented. 05
 IMPLEMENTED (I): There is an implemented internal database of licensing

Licensing Establishments (LI): Indicators and Fact Sheets

applications along with their regulatory decisions (i.e., approved, rejected, denied,
withdrawn, cancelled, or suspended).

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: LI05.02: Performance indicators for licensing activities are established
and implemented

Maturity Level: 4

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the existence and implementation of performance indicators
for different activities included under the licensing functions.
Specifically, the system should define key performance indicators (KPIs) along the entire
licensing activity chain and all indicators should be adequately justified. For the purpose
06
of clarity and consistency, established KPIs should be supported with guidelines for
monitoring and maintenance of the KPIs. The guidelines in turn should be supported 07
by SOPs and tools that define the procedures to be used for monitoring and evaluating
the performance indicators and that define procedures and timelines for reviewing and 08
revising the indicators.
Established KPIs might be qualitative, quantitative or combination of both. In general, 09
quantitative indicators are preferred to avoid bias or misinterpretation. However,
qualitative indicators are also accepted. Qualitative indicators may or may not include
scoring or scaling to render them semi-quantitative and thus more informative. The
assessor should ensure that indicators are measured on a regular basis to monitor
progress and advancement.
In addition, the assessor should verify measured indicators are analyzed to identify
trends or abnormalities. Justifications for any identified abnormalities should be
provided; when necessary, process optimizations should be introduced to avoid
recurrence.

Objective: The objective of this sub-indicator is to ensure that a system, mechanism, or procedure
exists to require the NRA to establish performance indicators along the entire licensing
chain. Additionally, the objective is to ensure that KPIs are actually contributing to
monitoring of regulatory performance, to measuring effectiveness of licensing regulatory
activities, and to making any necessary adjustments or optimizations.

Requirement: KPIs for licensing activities

Page 199 of 340

WHO Global Benchmarking Tool (GBT)

Evidence to review: The assessor should ask for and review:

1. Documents supporting the system, mechanism, or procedure compelling the NRA to
establish and implement performance indicators along the entire licensing activity chain.
2. Evidence that the performance indicators have been established and implemented, and
that the members of staff involved in the licensing function are aware of the indicators and
the guidelines and SOPs used for monitoring and evaluating their performance.
3. The current performance indicators for licensing activities
4. Analyses of the measured indicators along with the investigations done to identify
trends or abnormalities. 01
5. Documentation for follow- up of any observed abnormalities, including justifications for any
identified abnormalities as well as any process optimizations introduced to avoid recurrence. 02

Licensing Establishments (LI): Indicators and Fact Sheets

(7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)
3. Guiding principles for small national drug regulatory authorities. In: WHO Expert
Committee on Specifications for Pharmaceutical Preparations: thirty-first report. Geneva:
World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No. 790), (5),
(http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
4. Quality management systems — Requirements. International Standard ISO 9001:2015.
Geneva. International Organization for Standardization, 2015, (4), (https://www.iso.org/iso-
9001-quality-management.html)
5. Quality management -- Quality of an organization -- Guidance to achieve sustained
success, ISO 9004:2018, International Organization for Standardization (ISO), (115),
(https://www.iso.org/standard/70397.html)

Framework: Output

Rating Scale:  NOT IMPLEMENTED (NI): There are no KPIs for licensing activities.
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted KPIs for licensing
activities but they have not yet been reported.
 PARTIALLY IMPLEMENTED (PI): The NRA has developed KPIs for licensing activities and
has been applying them for less than two year or they have not covered all critical steps.
06
 IMPLEMENTED (I): The NRA has established and implemented KPIs for licensing
activities. The indicators are reviewed regularly, and appropriate actions are taken, and 07
decisions made.
08
Limitations and remarks:  When they refer to outcomes, indicators may be ambiguous and difficult to interpret,
as outcomes are the result of many factors that are difficult to disentangle. When 09
they refer to processes, indicators are often too specific, as they may focus on a
particular intervention or condition or they may quickly become outdated as business
models develop.
 Different methodologies are used to measure the NRAs performance on licensing
activities. In this case, the assessor should verify that adequate supporting documents
are available. The assessor should consider that developed performance indicators
should be Specific, Measurable, Achievable, Realistic, and Time-bound (i.e., “SMART”).
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Indicator: LI06 Mechanism exists to promote transparency, accountability

and communication.

Objective: The objective of this indicator is to ensure that mechanisms are in place that promote
effective communication inside and outside the NRA, that promote transparency and
outreach to the public, and that establish milestones that encourage accountability of
the NRA to its mandate. Additionally, these contribute to mutual understanding and

Page 200 of 340

WHO Global Benchmarking Tool (GBT)

involvement of all stakeholders relevant to licensing activities. Consequently, confidence in

the regulatory system is raised.

Category: 08. Transparency, accountability and communication

Sub Indicator: LI06.01: An updated list or database of all licensing applications, along with the
regulatory decision for each, is regularly published and publicly available.

Maturity Level: 3 01
Scope: 1. Medicines 02
2. Vaccines
03
Description: The assessor should verify the availability of a well-established mechanism or procedure to
promote transparency, accountability and communication of licensing activities. 04
The assessor should check the availability of an updated and complete list or database of
licensed facilities throughout the supply chain, along with the regulatory decision for each 05
application (i.e., whether the license was granted or not). The assessor should determine

Licensing Establishments (LI): Indicators and Fact Sheets

whether the list or database is published and can be accessed by the public. As a minimal
requirement, the list or database of all manufacturing licenses approved should be publicly
available on the NRA website or the official government record of each country. The list or
database should be maintained and regularly updated. This information supports decision-
making on other regulatory activities within the NRA and provides transparency for the
general public on the status of the NRAs licensing activities.
The publication of negative decisions (i.e. applications which were not granted or
approved) is encouraged; however, priority should go to approved applications.

Objective: The objective of this sub-indicator is to build confidence and accountability of the licensing
process via enhanced transparency.

Requirement: Transparency, accountability and communication

Evidence to review: The assessor should ask for and review:

1. Databases of establishments licensed by the NRA containing information on licenses
that were approved, withdrawn, canceled, or suspended;
2. Latest publicly available list of licensed establishments.
06
References: 1. Guidelines on the implementation of the WHO certification scheme on the quality of
pharmaceutical products moving in international commerce. In WHO Expert Committee on 07
Specifications for Pharmaceutical Preparations: thirty-fourth report. Geneva. World Health
Organization; 1996: Annex 10 (WHO Technical Report Series, No. 863), (29), (http://apps. 08
who.int/medicinedocs/pdf/s5516e/s5516e.pdf)
2. Quality systems requirements for national good manufacturing practice inspectorates. 09
In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: thirty-sixth
report. Geneva: World Health Organization; 2002: Annex 8 (WHO Technical Report Series,
No. 902), (8), (http://apps.who.int/medicinedocs/documents/s22112en/s22112en.pdf)
3. Guidelines on import procedures for pharmaceutical products, World Health
Organization (WHO), (22), (http://digicollection.org/whoqapharm/p/about and http://apps.
who.int/medicinedocs/en)

Framework: Output

Rating Scale:  NOT IMPLEMENTED (NI): There is no publicly available list or database of all
establishment licenses along with their regulatory actions.
 ONGOING IMPLEMENTATION (OI): The NRA is preparing to establish a database on
licensed establishments, but there is no evidence of results from such activities.
 PARTIALLY IMPLEMENTED (PI): There is evidence that the NRA has the elements (i.e.,
procedures, documentation, management systems, and other relevant information.)
and the capacity to perform the processes mentioned in the indicator, however, it has
only limited experience or a limited number of documented events.

Page 201 of 340

WHO Global Benchmarking Tool (GBT)

 IMPLEMENTED (I): There is a publicly available list or database of all establishment

licenses along with their regulatory actions.

Limitations and remarks: The legal provisions may not allow publication of the establishments that have been
sanctioned or suspended.

Sub Indicator: LI06.02: Inspection reports or summaries (or excerpts) relevant to licensing
activities are published and publicly available.
01
Maturity Level: 4
02
Scope: 1. Medicines
2. Vaccines 03

Description: For inspections relevant to licensing activities, the assessor should verify that reports of 04
inspections performed are available for the general public. The information which can
be shared with the public varies according to the national legislation. In some countries, 05
full inspection reports, redacted or non-redacted, might be published. In other countries,

Licensing Establishments (LI): Indicators and Fact Sheets

only inspection summaries or excerpts might be publicly available.

Objective: The objective of this sub-indicator is to build confidence and accountability in the licensing
entity via enhanced transparency through publication of the licensing-related inspection data.

Requirement: Transparency, accountability and communication

Evidence to review: The assessor should ask for and review:

1. Procedures and processes for publishing the list of inspections performed;
2. List of inspections performed;
3. Examples of actual published Inspection reports or summaries (or excerpts)

Framework: Output

Rating Scale:  NOT IMPLEMENTED (NI): Licensing relevant inspection reports or summaries (or
excerpts) are not publicly available.
 ONGOING IMPLEMENTATION (OI): The NRA established a procedure or
mechanism by which licensing- relevant inspection reports or summaries (or
excerpts) could be publicly available, however, this is not yet implemented.
 PARTIALLY IMPLEMENTED (PI): Licensing-related inspection reports or
summaries are publicly available, but only upon request (i.e., using a pull rather
than push publication mechanism).
 IMPLEMENTED (I): Licensing relevant inspection report or summaries (or
excerpts) are regularly and consistently published and publicly available.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Page 202 of 340

WHO Global Benchmarking Tool (GBT)

06
Regulatory Inspection
02
03
04
05
01

(RI): Indicators and

Licensing Establishments (LI): Indicators and Fact Sheets

Fact Sheets

WHO Global
Benchmarking
Tool (GBT)
for Evaluation of
National Regulatory
System of Medical
Products 06
07
08
09

Page 203 of 340

WHO Global Benchmarking Tool (GBT)

06. Regulatory Inspection (RI):

Indicators and Fact Sheets

Function:
06 — REGULATORY INSPECTION (RI)
01
Description: Inspection of establishments across the medical product supply chain is an essential 02
regulatory function. The supply chain includes manufacturers, distributors, re-packagers,
re-labelers, importers, agents, traders, wholesalers and retailers of medical products. The 03
purpose of regulatory inspections is to ensure that operations at these establishments
are carried out in accordance with approved standards, norms, and guidelines and are 04
in compliance with the national medical products legislation and regulations. These, in
turn, should be consistent with World Health Organization recommendations and other 05
internationally recognized guidelines. The scope of the function applies to different Good
Practices (GXPs) and is not limited to Good Manufacturing Practices (GMPs). Good 06
Distribution Practices and Good Clinical Practices also come under the scope of this

Regulatory Inspection (RI): Indicators and Fact Sheets

function. Good Vigilance Practices are not addressed in this function but are addressed
under the vigilance function.

National Regulatory Authorities (NRAs) should have the legal mandate to inspect and
enforce GXPs throughout the supply chain, to make decisions concerning the issuance,
suspension or withdrawal of establishment licenses, and to issue authorizations or
certifications for the activities performed by these establishments. Additionally, the NRA
should develop policies, regulatory actions and procedures on the handling of medical
products with suspected quality defects and medical products identified as substandard
and falsified. Commonly, inspectors perform several types of inspections: pre-licensing or
post-licensing, pre-approval or post-approval, announced or unannounced, and domestic
or overseas inspections.

Inspection activities should be conducted based on a risk management approach.

Inspections reveal weaknesses and deficiencies, as well as actual or potential errors in
the production, quality control, storage or distribution of medical products. Therefore,
inspection activities are fundamental for guaranteeing the quality, safety and efficacy of
medical products used by the population. An appeal system that is independent of the 07
body that made the initial decision should be available.

The credibility of the inspection depends on the transparency and clarity of the process,
08
on the absence of conflicts of interest, and on the availability of regulations, directives,
09
guidelines and procedures related to the quality management and assurance system
of the inspectorate. The system also is highly dependent on the technical competence
and integrity of the inspectors. The inspectorate must also ensure confidentiality of the
information obtained in the course of its inspection activities.

Networking with other international bodies and NRAs is an important method for
acquiring, sharing, and exchanging information relevant to the quality and safety of
medical products; in turn, this information contributes to informed science-based
decisions. The inspectorate of the NRA should follow uniform procedures incorporating
quality system principles.

A general limitation for this function exists in countries where no domestic

manufacturing capacities exist. In this case, domestic regulatory inspections based
on GMP are not applicable; however Good Distribution Practices would always apply.
Similarly, Good Clinical Practices inspections might not apply in countries where
no clinical trials are conducted. Even in the absence of a domestic pharmaceutical
manufacturer, a GMP inspection function is needed. GMP compliance is always a
requirement for granting marketing authorization for a medical product. In this case,
GMP compliance can be assured through overseas inspections or desk assessments,

Page 204 of 340

WHO Global Benchmarking Tool (GBT)

which may or may not include verification of documentation. Thus, even if there is no
domestic manufacturer, there will be a need for a function which can ensure the GMP
compliance of foreign pharmaceutical manufacturers either by physical inspection
or desk review (e.g., taking into consideration the GMP certificates from a stringent
regulatory authority).

Indicator: RI01 Legal provisions, regulations and guidelines required to define regulatory
framework of inspection and enforcement.
01
Objective: The objective of this indicator is to ensure that regulatory inspection activities are
supported by a comprehensive set of legal provisions, regulations and guidelines which 02
provide the necessary mandate to implement all activities related to this regulatory
function. Furthermore, national GXP rules should be consistent with internationally 03
recognized and accepted GXP guidelines such as those from the World Health
Organization. 04
The NRA should show evidence that laws, regulations, decrees, agreements, or other
mandatory legal provisions are in place regarding inspections that evaluate compliance 05
with best practices and that provide oversight of activities at the establishments.
The legal provisions also should provide a mandate to inspect the establishments of 06
marketing authorization holders, manufacturers, importers, exporters, and distributors

Regulatory Inspection (RI): Indicators and Fact Sheets

for compliance with national standards and GXP guidelines.

Category: 01. Legal provisions, regulations and guidelines

Sub Indicator: RI01.01: Legal provisions authorize the inspectorate to inspect and enforce
Good Practices (GXPs) throughout the supply chain.

Maturity Level: 1

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that current legislation establishes fundamental
functions in the regulatory authority that is appointed to verify compliance with laws,
regulations, standards and GXPs. These include Good Manufacturing Practices (GMP)
for manufacturers, Good Distribution Practices (GDP) for distributors, wholesalers
and retailers, and Good Clinical Practices (GCP) for sites where clinical studies are
conducted. 07
The assessor should also verify that legislation provides the obligation for
manufacturers and marketing authorization (MA) holders to inform the National 08
Regulatory Authority (NRA) of:
1. any safety signal or any quality defect which could impact patient safety of a 09
marketed product, and
2. any marketing or regulatory decisions made in the country of origin or in another
country where the product is marketed.
In line with these legal provisions, the assessor should review the relevant, approved
guidelines that explain the reporting obligations and that describe how, when and what
quality defects or safety issues have to be reported. Examples of the types of reports
include recall notifications, reports of study results from ongoing or previous studies,
and information on supervision of clinical trials. Similar obligations should also be in
force for the MA holder for all products marketed in the country.
The assessor should ensure the existence and the implementation of a set of
enforcement actions in case of non-compliance. The enforcement actions should
include, but are not limited to recall, suspension, withdrawal, sanction, and prosecution.
The assessor should identify if the law gives the inspectorate the mandate, power and
authority that is adequate to implement administrative measures such as suspending
or stopping production or supply. To ensure the quality of marketed products, such
actions would be required in case of violations of regulatory requirements or detection
of confirmed quality defects (i.e., circumstances potentially leading to issuance of a
rapid alert).

Page 205 of 340

WHO Global Benchmarking Tool (GBT)

Objective: The objective of this sub-indicator is to ensure that laws, regulations, decrees,
agreements, or other mandatory legal provisions have been established that give
authority to the inspectorate to inspect establishments (e.g., MA holders, manufacturers,
importers, exporters, and distributors) to assess compliance with best practices and to
provide oversight of activities. The regulatory authorities should have the mandate to
evaluate compliance with national standards and GXP guidelines throughout the supply
chain.

Requirement: Legal provisions, regulations and guidelines 01

Evidence to review: The assessor should ask for and review: 02
1. Published legal provisions (e.g., laws, acts, executive orders, or regulations)
establishing the mandate for the inspectorate to inspect establishments and enforce 03
regulations;
2. Legal provisions establishing the authority to inspect according to GXP throughout 04
the supply chain.
05
References: 1. Guiding principles for small national drug regulatory authorities. In: WHO Expert
Committee on Specifications for Pharmaceutical Preparations: thirty-first report. 06
Geneva: World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No.

Regulatory Inspection (RI): Indicators and Fact Sheets

790), (5), (http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
2. Quality systems requirements for national good manufacturing practice
inspectorates. In: WHO Expert Committee on Specifications for Pharmaceutical
Preparations: thirty-sixth report. Geneva: World Health Organization; 2002: Annex
8 (WHO Technical Report Series, No. 902), (8), (http://apps.who.int/medicinedocs/
documents/s22112en/s22112en.pdf)
3. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical
Preparations: thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8
(WHO Technical
Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf)
4. Effective drug regulation: A multicounty study. Geneva: World Health Organization;
2002, (7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)
5. Quality management systems — Requirements. International Standard ISO
9001:2015. Geneva. International Organization for Standardization, 2015, (4), (https://
www.iso.org/iso-9001-quality-management.html)
6. PIC/S Pre-Accession Procedure, related documents, PIC/S Audit Checklist, (138), 07
(https://www.picscheme.org/en/pre-accession)
08
Framework: Structure/Foundation/Input
09
Rating Scale:  NOT IMPLEMENTED (NI): There are no legal provisions for mandating the NRA
to inspect and enforce GXP.
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps toward the
establishment of legal provisions to allow for regulatory inspections however no
formal mandate exists for the regulatory inspection function.
 PARTIALLY IMPLEMENTED (PI): The NRA has the legal provisions to conduct the
regulatory inspection activities; however such activities are recently implemented
and results associated with this sub-indicator are not yet documented.
 IMPLEMENTED (I): Legal provisions exist for mandating the NRA to inspect and
enforce GXP.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: RI01.02: Legal provisions allow inspectors to enter facilities throughout the
supply chain at any reasonable time and in any place.

Maturity Level: 1

Page 206 of 340

WHO Global Benchmarking Tool (GBT)

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that the legal provisions provide the mandate, the power and
the authority for the inspectors to enter establishments throughout the supply chain at
any reasonable time and in any place within those facilities.
In this context, “any reasonable time” means any time within normal working hours of
the establishment that is the subject of the inspection. For example, if the establishment
operates during the night, then the legal provisions should ideally authorize the 01
inspectors to have access at that time. On the other side, access to unlicensed or
unauthorized establishments or access to establishments outside normal working 02
hours may not be mandated by the respective legal provisions. In such cases, special
approvals from judicial departments might be needed. 03

Objective: The objective of this sub-indicator is to ensure that laws, regulations, decrees, 04
agreements, or other mandatory legal provisions regarding inspections are actually
providing the necessary mandate to allow inspectors to enter establishments 05
throughout the supply chain at any reasonable time and in any place within those
facilities. Such provisions should apply to regulatory inspections that evaluate 06
compliance with best practices and provide other oversight activities. Additionally, these

Regulatory Inspection (RI): Indicators and Fact Sheets

provisions significantly contribute to efficiency and effectiveness of inspection function.

Requirement: Legal provisions, regulations and guidelines

Evidence to review: The assessor should ask for and review:

1. Published legal provisions (e.g. laws, acts, executive orders, or regulations)
2. Examples of inspection records. Assessor should review inspection activities and
select examples of records of unannounced or “for cause” inspections that were initiated
either as a part of an investigation or in response to a complaint or quality issue.

References: 1. Guiding principles for small national drug regulatory authorities. In: WHO Expert
Committee on Specifications for Pharmaceutical Preparations: thirty-first report.
Geneva: World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No.
790), (5), (http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
2. Quality systems requirements for national good manufacturing practice inspectorates.
In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: thirty-sixth
report. Geneva: World Health Organization; 2002: Annex 8 (WHO Technical Report Series,
No. 902), (8), (http://apps.who.int/medicinedocs/documents/s22112en/s22112en.pdf) 07
3. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: 08
thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO Technical
Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/documents/s21964en/ 09
s21964en.pdf)
4. Effective drug regulation: A multicountry study. Geneva: World Health Organization;
2002, (7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)
5. PIC/S Pre-Accession Procedure, related documents, PIC/S Audit Checklist, (138),
(https://www.picscheme.org/en/pre-accession)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There are no legal provisions allowing the inspectors to
enter establishments throughout the supply chain at any reasonable time and in
any place within the facilities.
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps toward the
establishment of legal provisions to allow inspectors to enter establishments
throughout the supply chain at any reasonable time and in any place within the
facilities; however, no results yet exist.
 PARTIALLY IMPLEMENTED (PI): Legal provisions allowing the inspectors to enter
establishments throughout the supply chain at any reasonable time and in any
place within facilities are recently established; however, the results of this practice
are not yet documented.

Page 207 of 340

WHO Global Benchmarking Tool (GBT)

 IMPLEMENTED (I): There are legal provisions allowing the inspectors to enter
establishments throughout the supply chain at any reasonable time and in any
place within the facilities, and documented evidence exists to demonstrate that it
happens.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: RI01.03: Legal provisions allow inspectors to collect relevant evidence, including 01
samples, during GXP inspections.
02
Maturity Level: 2
03
Scope: 1. Medicines
2. Vaccines 04

Description: The assessor should verify that inspectors have the power and authority to collect 05
any important evidence during their inspection and investigation activities. Examples
of evidence include copies of documents, photos, videos and samples. As part of 06
compliance verification, the designated inspectors should also be able to collect product

Regulatory Inspection (RI): Indicators and Fact Sheets

samples for testing by the national control laboratory at any phase of the production or
supply chain.

Objective: The objective of this sub-indicator is to ensure that the respective laws, regulations,
decrees, or other mandatory legal provisions allow inspectors to collect relevant
evidence, including samples, during GXP inspections. Samples can also be used to
check adulteration which might appear during the supply chain.

Requirement: Legal provisions, regulations and guidelines

Evidence to review: The assessor should ask for and review:

1. Published legal provisions (e.g. laws, acts, executive orders, or regulations)
2. Examples of inspection records. Assessor should review inspection reports and
the evidence (e.g., documents or materials) or samples collected during inspection
activities.
3. Records of laboratory results for samples and decisions based on those results.

References: 1. WHO good practices for pharmaceutical quality control laboratories, (35), (http:// 07
digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en)
2. WHO good manufacturing practices for pharmaceutical products: main principles, 08
(51), (http://digicollection.org/whoqapharm/p/about and http://apps.who.int/
medicinedocs/en) 09
3. Guidelines on heating, ventilation and air-conditioning systems for non-sterile
pharmaceutical products, World Health Organization (WHO), (60), (http://apps.who.int/
medicinedocs/en/q/ and http://digicollection.org/whoqapharm/)
4. WHO good manufacturing practices: water for pharmaceutical use, World Health
Organization (WHO), (58), (http://apps.who.int/medicinedocs/en/q/ and http://
digicollection.org/whoqapharm/)
5. WHO good manufacturing practices for sterile pharmaceutical products, World
Health Organization (WHO), (61), (http://apps.who.int/medicinedocs/en/q/ and http://
digicollection.org/whoqapharm/)
6. WHO guidelines for drafting a site master file, World Health Organization (WHO), (64),
(http://apps.who.int/medicinedocs/en/q/ and http://digicollection.org/whoqapharm/)
7. WHO good manufacturing practices for active pharmaceutical ingredients, World
Health Organization (WHO), (67), (http://apps.who.int/medicinedocs/en/q/ and http://
digicollection.org/whoqapharm/)
8. Guidelines on packaging for pharmaceutical products, World Health Organization
(WHO), (80), (http://apps.who.int/medicinedocs/en/q/ and http://digicollection.org/
whoqapharm/)
9. WHO good distribution practices for pharmaceutical products, World Health

Page 208 of 340

WHO Global Benchmarking Tool (GBT)

Organization (WHO), (27), (http://digicollection.org/whoqapharm/p/about and http://

apps.who.int/medicinedocs/en)
10. Guidelines for good clinical practice (GCP) for trials on pharmaceutical products,
(40), (http://apps.who.int/medicinedocs/en)
11. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical
Preparations: thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO
Technical Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/documents/
s21964en/s21964en.pdf) 01
12. WHO guidelines for sampling of pharmaceutical products and related materials,
World Health Organization (WHO), (31), (http://digicollection.org/whoqapharm/p/about 02
and http://apps.who.int/medicinedocs/en)
13. Quality management systems — Requirements. International Standard ISO 03
9001:2015. Geneva. International Organization for Standardization, 2015, (4), (https://
www.iso.org/iso-9001-quality-management.html) 04
14. Guidance on developing a national plan for preventing, detecting and responding to
actions, activities and behaviors that result in SF medical products. WHO (document 05
A70/23), (130), (https://www.who.int/medicines/regulation/ssffc/mechanism/A70_23-
en6-14.pdf) 06
15. Guiding principles for small national drug regulatory authorities. In: WHO Expert

Regulatory Inspection (RI): Indicators and Fact Sheets

Committee on Specifications for Pharmaceutical Preparations: thirty-first report.
Geneva: World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No.
790), (5), (http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
16. WHO guidelines for sampling of pharmaceutical products and related materials,
World Health Organization (WHO), (31), (http://digicollection.org/whoqapharm/p/about
and http://apps.who.int/medicinedocs/en)
17. PIC/S Pre-Accession Procedure, related documents, PIC/S Audit Checklist, (138),
(https://www.picscheme.org/en/pre-accession)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There are no legal provisions allowing the inspectors to
collect relevant evidence, including samples, during GXP inspections.
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps toward the
establishment of legal provisions to allow inspectors to collect relevant evidence,
including samples, during GXP inspections however this has not yet been
carried out.
 PARTIALLY IMPLEMENTED (PI): There is evidence that the NRA has the legal basis, 07
elements, and capacity to perform the processes mentioned in the indicator, but
the NRA has only limited, recent experience with it. 08
 IMPLEMENTED (I): There are legal provisions allowing the inspectors to
collect relevant evidence, including samples, during GXP inspections and the 09
implementation of this activity is documented.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: RI01.04: Updated national GXP regulations, norms or guidelines are mandatory.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should review the GXP regulations and guidelines which should be updated,
published and available to all stakeholders. Also, the assessor should verify that the
guidance is consistent with World Health Organization (WHO) or other internationally
recognized guidance. If applicable, differences should be identified. The assessor should
verify that the national regulations are mandatory and actually implemented.

Page 209 of 340

WHO Global Benchmarking Tool (GBT)

Objective: The objective of this sub-indicator is to ensure that national GXP regulations, norms and
guidelines are available and their implementation is mandatory for the establishments
subjected to regulatory inspections.

Requirement: Legal provisions, regulations and guidelines

Evidence to review: The assessor should ask for and review:

1. Published GXP regulations, norms and guidelines (e.g. GMP, GDP, GCP, and Good Cold
Chain Management Practices) 01
2. Evidence that the guidelines are implemented and enforceable by the NRA.
02
References: 1. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical 03
Preparations: thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO
Technical Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/documents/ 04
s21964en/s21964en.pdf)
2. Quality management systems — Requirements. International Standard ISO 05
9001:2015. Geneva. International Organization for Standardization, 2015, (4), (https://
www.iso.org/iso-9001-quality-management.html) 06
3. Effective drug regulation: A multicountry study. Geneva: World Health Organization;

Regulatory Inspection (RI): Indicators and Fact Sheets

2002, (7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)
4. Quality systems requirements for national good manufacturing practice
inspectorates. In: WHO Expert Committee on Specifications for Pharmaceutical
Preparations: thirty-sixth report. Geneva: World Health Organization; 2002: Annex
8 (WHO Technical Report Series, No. 902), (8), (http://apps.who.int/medicinedocs/
documents/s22112en/s22112en.pdf)
5. WHO good distribution practices for pharmaceutical products, World Health
Organization (WHO), (27), (http://digicollection.org/whoqapharm/p/about and http://
apps.who.int/medicinedocs/en)
6. WHO good practices for pharmaceutical quality control laboratories, (35), (http://
digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en)
7. WHO good manufacturing practices for pharmaceutical products: main principles,
(51), (http://digicollection.org/whoqapharm/p/about and http://apps.who.int/
medicinedocs/en)
13. Guiding principles for small national drug regulatory authorities. In: WHO Expert
Committee on Specifications for Pharmaceutical Preparations: thirty-first report.
Geneva: World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No.
790), (5), (http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf) 07
14. PIC/S Pre-Accession Procedure, related documents, PIC/S Audit Checklist, (138),
(https://www.picscheme.org/en/pre-accession) 08
8. WHO good manufacturing practices: water for pharmaceutical use, World Health
Organization (WHO), (58), (http://apps.who.int/medicinedocs/en/q/ and http:// 09
digicollection.org/whoqapharm/)
9. Guidelines on heating, ventilation and air-conditioning systems for non-sterile
pharmaceutical products, World Health Organization (WHO), (60), (http://apps.who.int/
medicinedocs/en/q/ and http://digicollection.org/whoqapharm/)
10. WHO good manufacturing practices for sterile pharmaceutical products, World
Health Organization (WHO), (61), (http://apps.who.int/medicinedocs/en/q/ and http://
digicollection.org/whoqapharm/)
11. WHO good manufacturing practices for active pharmaceutical ingredients, World
Health Organization (WHO), (67), (http://apps.who.int/medicinedocs/en/q/ and http://
digicollection.org/whoqapharm/)
12. Guidelines on packaging for pharmaceutical products, World Health Organization
(WHO), (80), (http://apps.who.int/medicinedocs/en/q/ and http://digicollection.org/
whoqapharm/)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): National GXP guidelines do not exist or are not
mandatory.

Page 210 of 340

WHO Global Benchmarking Tool (GBT)

 ONGOING IMPLEMENTATION (OI): The NRA is taking steps towards establishment

of national GXP guidelines however none of these guidelines (e.g., GMP, GDP, and
GCP) is yet established.
 PARTIALLY IMPLEMENTED (PI): Some national GXP guidelines (e.g., GMP, GDP or
GCP) are established while others are not.
 IMPLEMENTED (I): National GXP regulations, norms and guidelines exist and are
mandatory.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will 01
always apply for all benchmarked NRAs).
02
Sub Indicator: RI01.05: Legal provisions and regulations allow the recognition of and/or
reliance on foreign NRA inspections and enforcement actions based on 03
well- defined criteria.
04
Maturity Level: 1
05
Scope: 1. Medicines
2. Vaccines 06

Regulatory Inspection (RI): Indicators and Fact Sheets

Description: The assessor should review the legal provisions and regulations which should be
available and published and verify that the guidance allows for the recognition of and/
or reliance on foreign NRA inspections and enforcement actions based on well- defined
criteria. While the criteria may vary significantly, the assessor must ensure the criteria
are clear and supported by an appropriate rationale.

Objective: The objective of this sub-indicator is to ensure that laws, regulations, or other
mandatory legal provisions are actually providing the mandate for the recognition of
and/or reliance on foreign NRA inspections and enforcement actions based on well-
defined criteria.

Requirement: Legal provisions, regulations and guidelines

Evidence to review: The assessor should ask for and review:

1. Published legal provisions and regulations for recognition and/or reliance.
2. Examples of records demonstrating recognition of and/or reliance on foreign NRA
inspections and enforcement actions.
07
References: 1. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical 08
Preparations: thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO
Technical Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/documents/ 09
s21964en/s21964en.pdf)
2. Quality systems requirements for national good manufacturing practice
inspectorates.
In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: thirty-
sixth report. Geneva: World Health Organization; 2002: Annex 8 (WHO Technical
Report Series, No. 902), (8), (http://apps.who.int/medicinedocs/documents/s22112en/
s22112en.pdf)
3. Guiding principles for small national drug regulatory authorities. In: WHO Expert
Committee on Specifications for Pharmaceutical Preparations: thirty-first report.
Geneva: World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No.
790), (5), (http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
4. Effective drug regulation: A multicountry study. Geneva: World Health Organization;
2002, (7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)
5. Quality management systems — Requirements. International Standard ISO
9001:2015. Geneva. International Organization for Standardization, 2015, (4), (https://
www.iso.org/iso-9001-quality-management.html)

Framework: Structure/Foundation/Input

Page 211 of 340

WHO Global Benchmarking Tool (GBT)

Rating Scale:  NOT IMPLEMENTED (NI): There are no legal provisions or regulations allowing
the recognition of and/or reliance on foreign NRA inspections and enforcement
actions.
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps toward the
establishment of legal provisions and regulations allowing the recognition of and/
or reliance on foreign NRA inspections and enforcement actions however this has
not yet been carried out in practice.
 PARTIALLY IMPLEMENTED (PI): Legal provisions or regulations allowing the
recognition of and/or reliance on foreign NRA inspections and enforcement actions 01
are recently established, but there is only limited, recent experience (less than two
years) with the implementation of the same. 02
 IMPLEMENTED (I): There are legal provisions or regulations allowing the
recognition of and/or reliance on foreign NRA inspections and enforcement 03
actions.
04
Limitations and remarks: In some countries neither recognition nor reliance are permitted. In this case, the
assessor can score the sub-indicator as NOT APPLICABLE if evidence is provided 05
to show effective and efficient regulatory inspection function without recognition or
reliance. 06

Regulatory Inspection (RI): Indicators and Fact Sheets

Indicator: RI02 Arrangement for effective organization and good governance.

Objective: The objective of this indicator is to ensure the implementation of effective organization
and good governance practices at the entities in charge of establishments licensing
activities, which in turn contributes to effective and efficient functioning of the regulatory
inspection activities.

Category: 2. Organization and good governance

Sub Indicator: RI02.01: There is a defined organizational structure with clear responsibilities to
conduct regulatory inspection activities.

Maturity Level: 2

Scope: 1. Medicines
2. Vaccines
07
Description: The assessor should identify the organization designated to establish, implement
or maintain the regulatory inspection function, as well as the specific organizational 08
structures taking on the different relevant activities. Responsibilities, duties and roles of
these structures should be clearly defined and documented. If more than one structure 09
is involved, the assessor should check the ways and approaches by which coordination
among these structures takes place.

Objective: The objective of this sub-indicator is to ensure effective organization and good
governance of regulatory inspection activities and to ensure that these activities are
taken over by defined structures with clear roles and responsibilities.

Requirement: Roles and responsibilities of the structures in charge of establishments licensing

activities.

Evidence to review: The assessor should ask for and review:

1. Organization chart of the organization responsible for the implementation of
inspection activities along with identification of the particular structures implementing
the function.
2. Documentation clarifying roles and responsibilities of the organizational structures
implementing inspection activities. This may include administrative decrees, terms of
reference, and other relevant documents.
3. Documentation identifying established mechanisms of coordination (e.g.,

Page 212 of 340

WHO Global Benchmarking Tool (GBT)

committees, internal and work-sharing) among organizational structures, if any, which

take part in regulatory inspection activities.

Regulatory Inspection (RI): Indicators and Fact Sheets

9001:2015. Geneva. International Organization for Standardization, 2015, (4), (https://
www.iso.org/iso-9001-quality-management.html)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There is no defined organizational structure in charge of
regulatory inspection activities.
 ONGOING IMPLEMENTATION (OI): A mandate to establish a structure in charge of
regulatory inspection activities is available however the structure itself is not yet
established.
 PARTIALLY IMPLEMENTED (PI): A structure in charge of regulatory inspection
activities is newly established and mandated however the regular work and
practice of this structure is not yet consolidated.
 IMPLEMENTED (I): There is a defined organizational structure in charge of
regulatory inspection activities with clear and well-documented roles and
responsibilities.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will 07
always apply for all benchmarked NRAs).
08
Sub Indicator: RI02.02: Documented procedures and mechanisms are implemented to ensure
the involvement and communication among all stakeholders relevant to regulatory 09
inspection activities.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that documented procedures and mechanisms are
implemented to ensure the involvement and communication among different entities
and departments relevant to regulatory inspection activities. These entities may be
inside or outside the NRA
(e.g. National Control Laboratory, NRA departments involved in vigilance or MA activities,
police, customs authorities, judicial offices, professional associations, manufacturers,
and other industrial organizations).
In case of a decentralized establishment licensing function, an information exchange
system, mechanism or platform must be established and used so that appropriate
communication between the central and peripheral structures is ensured. As one
example, the decentralized entity can receive requests or guidance from the central

Page 213 of 340

WHO Global Benchmarking Tool (GBT)

authority and report back to it. The availability of such communication mechanisms will
also encourage consistency among different peripheral structures.

Objective: The objective of this sub-indicator is to ensure the existence and implementation of
documented procedures and mechanisms to guide the involvement and communication
among the different entities and departments. These activities will encourage
appropriate organization and good governance of the function.

Requirement: Agreements, memoranda of understanding and procedures for ensuring involvement of 01

and communication among stakeholders relevant to regulatory inspection function.
02
Evidence to review: The assessor should ask for and review:
1. Guidelines or Standard Operating Procedures (SOPs) that are related to external and 03
internal communications.
2. Examples of records of communication and collaborations providing evidence for 04
implementation of the above-mentioned procedures.
3. Documented evidence for regular formal and official communications and meetings 05
among above mentioned key players. Assessor should verify that these are systematic
and well-established processes. 06
4. Documentation for paths of communication and reporting.

Regulatory Inspection (RI): Indicators and Fact Sheets

5. Platforms for information sharing and exchange.

Framework: Structure/Foundation/Input 09

Rating Scale:  NOT IMPLEMENTED (NI): There are no information exchange procedures or
mechanisms among different stakeholders of the regulatory inspection function or
between the central authority and the decentralized entities.
 ONGOING IMPLEMENTATION (OI): The NRA is developing an information exchange
mechanism, but it is not yet ready, or exchanges are being conducted without an
established methodology.
 PARTIALLY IMPLEMENTED (PI): The NRA recently established an information
exchange procedure or mechanism and it is at the early implementation stage, so
this practice is not yet consolidated.
 IMPLEMENTED (I): There are established, implemented and maintained
information exchange procedures and mechanisms among different stakeholders
of the regulatory inspection function and between the central authority and the
decentralized entities.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Page 214 of 340

WHO Global Benchmarking Tool (GBT)

Indicator: RI03 Human resources to perform regulatory inspection activities.

Objective: The objective of this indicator is to ensure to that all entities within an NRA are
adequately resourced with a trained, experienced and skilled workforce that is
empowered to fully perform the regulatory inspection function. This will ensure that
regulatory inspection activities are performed in accordance with international best
practices.
01
Category: 06. Resources (HR, FR, infrastructure and equipment)
02
Sub Indicator: RI03.01: Sufficient competent staff (i.e., education, training, skills and experience)
are assigned to perform regulatory inspection activities 03

Maturity Level: 3 04

Scope: 1. Medicines 05
2. Vaccines
06
Description: The assessor should verify that the human resources assigned to perform regulatory

Regulatory Inspection (RI): Indicators and Fact Sheets

inspection activities should be sufficient with respect to numbers and competent
with respect to the requisite skills, education, experience and training. There should
be technical documents and SOPs that provide guidance on the required background
for regulatory inspection activities and that consider the requirements for educational
background, competencies, skills, experience, and training.
The assessor should verify that the NRA estimated the number of staff required to
effectively and efficiently perform regulatory inspection function and that the NRA
actually recruited that number. In addition, the assessor should verify that these
competency requirements are well-established and maintained by the NRA. Metrics and
statistics on the different activities performed as well as performance indicators can
be used for estimating the adequacy of the number of the assigned staff. The assessor
should also verify that the competency of the assigned staff is built, maintained and
improved through recruitment as well as continuous on-the-job training.

Objective: The objective of this sub-indicator is to ensure the existing human resources for
regulatory inspection is sufficient, in terms of numbers, experience, and specific
competencies, to perform all the activities along the entire regulatory inspection chain.
07
Requirement: Sufficient number of competent human resources in charge of regulatory inspection
activities. 08

Evidence to review: The assessor should ask for and review: 09

1. Evidence that the number of staff members involved in each of the documented
activities along the entire regulatory inspection process flow is adequate.
2. Evidence that the systems and structures are in place to ensure appropriate
placement of staff with respect to competence and skills.
3. Evidence that the system and structures have been implemented. The documentation
should include the records to verify that the staff competence is appropriate for the job
requirements.
4. Evidence that the professional profiles of the human resources engaged in regulatory
inspection activities are appropriate with respect to education, skills, and expertise, to
perform a particular function along the regulatory inspection chain. Documentation
should include a list of the requisite skills and training for each position.
5. Recruitment plan.

Page 215 of 340

WHO Global Benchmarking Tool (GBT)

2. Quality systems requirements for national good manufacturing practice

inspectorates. In: WHO Expert Committee on Specifications for Pharmaceutical
Preparations: thirty-sixth report. Geneva: World Health Organization; 2002: Annex
8 (WHO Technical Report Series, No. 902), (8), (http://apps.who.int/medicinedocs/
documents/s22112en/s22112en.pdf)
3. Guiding principles for small national drug regulatory authorities. In: WHO Expert
Committee on Specifications for Pharmaceutical Preparations: thirty-first report.
Geneva: World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No.
790), (5), (http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf) 01
4. Effective drug regulation: A multicountry study. Geneva: World Health Organization;
2002, (7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf) 02
5. Quality management systems — Requirements. International Standard ISO
9001:2015. Geneva. International Organization for Standardization, 2015, (4), (https:// 03
www.iso.org/iso-9001-quality-management.html)
6. PIC/S Pre-Accession Procedure, related documents, PIC/S Audit Checklist, (138), 04
(https://www.picscheme.org/en/pre-accession)
05
Framework: Process
06

Regulatory Inspection (RI): Indicators and Fact Sheets

Rating Scale:  NOT IMPLEMENTED (NI): The NRA does not have enough competent staff
(i.e., education, training, skills and experience) to perform regulatory inspection
activities
 ONGOING IMPLEMENTATION (OI): The NRA has recently developed a plan to
recruit adequate competent staff; however, the plan has not been implemented.
 PARTIALLY IMPLEMENTED (PI): The NRA has initiated the implementation of
the human resources development plan; however, there is need to complete the
competency profile.
 IMPLEMENTED (I): The NRA has a sustained number of competent staff
(i.e., education, training, skills and experience) assigned to perform regulatory
inspection activities.

Sub Indicator: RI03.02: Duties, functions, and responsibilities of the staff in charge of regulatory 08
inspection activities are established and updated in the respective job descriptions
09
Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that procedures are in place to maintain a current and
updated structure for managing job descriptions for personnel participating in regulatory
inspection activities. In addition, job descriptions should address current staff duties,
responsibilities and the requisite competencies. A job description with this format and
content should be established and implemented for all staff. When inspection activities
are subcontracted, liability of third party inspectors should be clearly defined in an
agreement or contract. The management of job descriptions should be supported by a
guidance document that provides direction on when and how to update the information,
and where the information should be kept for easy access. The guidance document
should present the appropriate duties and responsibilities that are assigned to each
member of the organization involved in regulatory inspection activities. Thus, the
professional profiles of staff are reflected in their respective roles and responsibilities
within the NRA. There should be procedures to guide responsible persons to document

Page 216 of 340

WHO Global Benchmarking Tool (GBT)

that duties, functions and responsibilities are revised and kept up to date. In addition,
procedures should be available to guide the keeping and documenting of up to date
work schedules and enforcing the implementation of the documented guidelines and
procedures.

Objective: The objective of this sub-indicator is to ensure that duties and responsibilities of
the staff are clear and well defined, that job descriptions are kept up to date with
current duties, functions and responsibilities, and that these activities are adequately
documented. 01
Requirement: Duties, roles and responsibilities of the staff relevant to regulatory inspection activities. 02

Evidence to review: The assessor should ask for and review: 03

1. Procedure and guidelines that guide placement of staff members within the NRA;
2. The professional profiles of staff (i.e., job descriptions) and documentation that they 04
are related to their current roles and duties;
3. The professional profiles of any external experts and documentation that the profiles 05
provide a composition that is complete and consistent with that prescribed in the legal
provisions; 06
4. Procedures to guide the documentation of up to date duties and work schedules, and

Regulatory Inspection (RI): Indicators and Fact Sheets

to enforce the implementation of the documented guidelines and procedures;
5. Job descriptions for designated staff.

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence of defined or established duties,
functions, responsibilities, respective job descriptions and necessary required
competencies.
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted or developed
the role and responsibilities document but it has not yet been implemented.
 PARTIALLY IMPLEMENTED (PI): The NRA has initiated implementation of
this requirement but it has not been defined or followed for all staff or the roles
and responsibilities documents, including staff job descriptions, are not up
to date.
 IMPLEMENTED (I): The NRA has defined and established all required duties,
functions, and responsibilities, and respective job descriptions are up-to-date.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Page 217 of 340

WHO Global Benchmarking Tool (GBT)

Sub Indicator: RI03.03: Training plan developed, implemented and updated at least once a year for
staff in charge of regulatory inspection activities.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that training plans are developed, implemented and updated 01
at least once every year to reflect the current situation by considering education and
experience of the staff. The training plan should be complemented with guidelines or 02
similar documents that guide the development and implementation of training plans.
The assessor should ensure that induction training for new staff as well as continued 03
on- the-job for staff is planned and implemented. There should be procedures to approve
the training plan and the budget allocated for implementing and updating the training 04
plan. The
plans should present clearly defined training goals and should include training in certain 05
topics and skills to address identified deficiencies. Learning objectives, training methods
and activities, evidence of learning, and evaluation and assessment of training should 06
be documented. This documentation should confirm that the learning objectives were

Regulatory Inspection (RI): Indicators and Fact Sheets

achieved and were designed to address weaknesses within the entities. Procedures
should be in place to ensure that a training plan is developed, implemented and
updated at least once every year. The assessor should verify that there is a system in
place for monitoring the implementation and effectiveness of the training plan and for
documenting the skills acquired in training activities for internal and external experts.
The assessor should verify that inspectors in charge of GMP inspections receive
a minimum of ten (10) training days per year as recommended by the WHO and
Pharmaceutical Inspection Cooperation Scheme guidelines. Furthermore, the NRA
should establish the minimum number of days that an inspector must spend on
inspections in order to retain competence.

Objective: The objective of this sub-indicator is to ensure that a training plan for staff exists, and
that it is implemented and updated annually. Through the training plan, NRA can be sure
that competency of staff in charge of regulatory inspection activities is maintained and
enhanced.

Requirement: Implementation of training plan

07
Evidence to review: The assessor should ask for and review:
1. Guidelines for development, implementation and annual update (i.e., at least once per 08
year) of the training plan. Guidelines should also provide for a mechanism to measure
effectiveness of training. 09
2. Documentation for the system or structures used to approve the training plan and to
evaluate the adequacy of the budget allocated to the training activities.
3. The current or existing staff training plan (or matrix) for staff. The assessor should
assess this in in relation to the respective individual job descriptions.
4. SOP for developing and maintaining the training plan.
5. Evidence that the NRA has investigated and identified training needs.
6. List of trainings performed.
7. Example records for training activities.
8. Documentation of the number of training days per inspector as well as average
number of training days for staff.
9. Number of days spent on site for inspections per inspector as well as average number
of days spent on inspection.

Page 218 of 340

WHO Global Benchmarking Tool (GBT)

2. Quality systems requirements for national good manufacturing practice

Regulatory Inspection (RI): Indicators and Fact Sheets

training plan (or matrix).
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted or developed the
training plan but there is no evidence of implementation.
 PARTIALLY IMPLEMENTED (PI): The NRA has developed and initiated the training
plan implementation. However, the NRA has not fulfilled all required planned
training or has been applying the plan for less than two years.
 IMPLEMENTED (I): The NRA has an updated training plan developed that is
supported by adequate records to demonstrate effective plan implementation,
including induction training for new staff and routine on-the-job training for recruited
staff.

Limitations and remarks:  Training plans must be updated regularly; ideally on an annual basis, but not less
frequently that once every two years.
 Some regulatory functions may include many training activities that are not
incorporated in the institutional training programme. Such training normally is
offered by invitation. In this case, the assessor should recognize reports from
non-routine regulatory inspection -relevant training not included in the NRA training
plan. 07
Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator
08


will always apply for all benchmarked NRAs).

Sub Indicator: RI03.04: The NRA generates and maintains records of staff training activities 09
and training effectiveness verification.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that records of staff training that is performed or organized
by the NRA are generated, maintained, regularly updated. This activity should be
supported by guidelines that direct the NRA to generate and maintain records of
staff training activities. Procedures should be in place to document and propose staff
training needs and to allocate a budget for continuous staff capacity building and
development. The assessor should check that there is an evaluation or assessment
mechanism to verify the quality of learning, and to confirm that learning objectives are
achieved. Documentation should include an inventory (i.e., soft and/or hard) system
that records all impactful and non-impactful trainings and identifies all staff members
who participated. A system to measure or estimate impact of trainings should be
established.

Page 219 of 340

WHO Global Benchmarking Tool (GBT)

Requirement: Training records

Evidence to review: The assessor should ask for and review: 01

1. Guidelines or similar documents that guide the NRA to generate and maintain records
of staff training activities; 02
2. Evaluations of training effectiveness;
3. The training inventory, and procedures for completing the inventory; 03
4. Examples of archived records of staff training, and procedures for the archiving
system. 04

References: 1. National drug regulatory legislation: guiding principles for small drug regulatory 05
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical
Preparations: thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO 06
Technical Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/documents/

Regulatory Inspection (RI): Indicators and Fact Sheets

s21964en/s21964en.pdf)
2. Quality systems requirements for national good manufacturing practice
inspectorates. In: WHO Expert Committee on Specifications for Pharmaceutical
Preparations: thirty-sixth report. Geneva: World Health Organization; 2002: Annex
8 (WHO Technical Report Series, No. 902), (8), (http://apps.who.int/medicinedocs/
documents/s22112en/s22112en.pdf)
3. Guiding principles for small national drug regulatory authorities. In: WHO
Expert Committee on Specifications for Pharmaceutical Preparations: thirty-first
report. Geneva: World Health Organization; 1990: Annex 6 (WHO Technical Report
Series, No. 790), (5), (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf)
4. Effective drug regulation: A multicountry study. Geneva: World Health Organization;
2002, (7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)
5. Quality management systems — Requirements. International Standard ISO
9001:2015. Geneva. International Organization for Standardization, 2015, (4), (https://
www.iso.org/iso-9001-quality-management.html)

Framework: Output 07
08
Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence that the NRA generates and
maintains records of staff training activities. 09
 ONGOING IMPLEMENTATION (OI): The NRA has recently initiated plans to
generate, document and keep records of staff training activities, however they are
not yet followed.
 PARTIALLY IMPLEMENTED (PI): The NRA has recently initiated plans to generate,
document and keep records of staff training activities but they are not fully
followed for all training activities or they have been established for less than two
years.
 IMPLEMENTED (I): The NRA generates and maintains records of staff training
activities.

Limitations and remarks:  The assessor should note that some NRAs out-source training including staff
capacity development activities. In this case the assessor should request the
identity of the provider, as well as evaluations of the provider. The assessor may
request the professional profiles of tutors or resource persons used to offer
training. The assessor may also request records covering the archiving systems in
place.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator
will always apply for all benchmarked NRAs).

Page 220 of 340

WHO Global Benchmarking Tool (GBT)

Indicator: RI04 Procedures established and implemented to perform inspection

and enforcement.

Objective: The objective of this indicator is to ensure the NRA or Inspectorate is supported by
appropriate mandates, effective organization and governance, and essential resources.
The regulatory inspection activities should be implemented through standard
procedures and work instructions in order to ensure consistency, effectiveness,
efficiency, impartiality and proportionality of the activities.
The NRA should have established a system which allows the regular review of quality 01
aspects of the authorized products on the market, and which includes processes to
review and share relevant data between key players and to take appropriate action. 02

Category: 07. Regulatory process 03

Sub Indicator: RI04.01: The different inspection activities, including inspection preparation, 04
conduct and reporting, are documented for GXP inspections.
05
Maturity Level: 3
06
Scope: 1. Medicines

Regulatory Inspection (RI): Indicators and Fact Sheets

2. Vaccines

Description: The assessor should verify the existence of procedures for efficiently working on
planning, preparing, conducting, reporting and monitoring of activities for GXP
inspections performed throughout the supply chain. The assessor should also
confirm the existence of documented procedures and resources to enable regulatory
GXP inspections to be carried out in accordance with official guidelines and national
legislation and in accordance with a formal inspection plan. All relevant instructions,
standards, written procedures, worksheets, check lists and reference data should be
maintained up-to-date and be readily available to staff. As part of inspection processes
and procedures, the assessor should verify that inspection reports are prepared in
accordance with applicable national or other requirements.
In addition, the assessor should verify that reports are prepared in approved format and
signed and dated by the relevant inspector. Ideally, the report should follow a standard
format which in, turn, is in compliance with the relevant WHO guidelines or other
internationally recognized and accepted guidelines.

Objective: The objective of this sub-indicator is to ensure that all pre-inspection, inspection and 07
post-inspection activities are well-documented and follow written procedures. These
activities include inspection planning, preparation (e.g., inspection announcement 08
and designation of inspectors), conduct (e.g., opening meeting and evidence
collection), and reporting (e.g., classification of findings, closing meeting, conclusions 09
and recommendations). These procedures enable the regulatory inspections of
manufacturing and distribution operations to be carried out in accordance with official
guidelines and national legislation and in accordance with a formal inspection plan.
Following these procedures will also ensure consistency, effectiveness, impartiality and
proportionality of practices.

Requirement: Regulatory process

Evidence to review: The assessor should ask for and review:

1. SOPs and procedures for planning, conducting, and monitoring of GXP inspections, as
well as records for performance of these activities;
2. GXP inspection reports and documentation of processes for inspection review and
follow-up. The inspection process should be analyzed on a regular basis.;
3. Instructions, worksheets, check lists and reference data relevant to the different
inspection activities.

References: 1. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical

Page 221 of 340

WHO Global Benchmarking Tool (GBT)

Preparations: thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO
Technical Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/documents/
s21964en/s21964en.pdf)
2. Quality systems requirements for national good manufacturing practice
inspectorates. In: WHO Expert Committee on Specifications for Pharmaceutical
Preparations:
thirty-sixth report. Geneva: World Health Organization; 2002: Annex 8 (WHO Technical
Report Series, No. 902), (8), (http://apps.who.int/medicinedocs/documents/s22112en/
s22112en.pdf) 01
3. Guiding principles for small national drug regulatory authorities. In: WHO Expert
Committee on Specifications for Pharmaceutical Preparations: thirty-first report. 02
Geneva: World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No.
790), (5), (http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf) 03
4. Effective drug regulation: A multicountry study. Geneva: World Health Organization;
2002, (7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf) 04
5. Quality management systems — Requirements. International Standard ISO
9001:2015. Geneva. International Organization for Standardization, 2015, (4), (https:// 05
www.iso.org/iso-9001-quality-management.html)
6. PIC/S Pre-Accession Procedure, related documents, PIC/S Audit Checklist, (138), 06
(https://www.picscheme.org/en/pre-accession)

Regulatory Inspection (RI): Indicators and Fact Sheets

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): There is no documentation for the different inspection
activities that evaluate compliance with GXP, including inspection preparation,
conduct and reporting.
 ONGOING IMPLEMENTATION (OI): The NRA is preparing to establish the
procedures along with the associated documentation for the different inspection
activities that check compliance with GXP, including inspection preparation,
conduct and reporting; however these are not yet established.
 PARTIALLY IMPLEMENTED (PI): There is evidence that the NRA has the capacity
to conduct the inspections mentioned in the indicator, with procedures and
documentation in place; however, experience is limited or recent and therefore the
relevant records are limited.
 IMPLEMENTED (I): The NRA has established procedures for the different
inspection activities that check compliance with GXP, including inspection
preparation, conduct and reporting, and the NRA consistently maintains
documentation of the results of related activities over time. 07

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will 08
always apply for all benchmarked NRAs).
09
Sub Indicator: RI04.02: Regulatory inspection follow-up, decision-making (including certification)
and enforcement activities are documented.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should check the availability and implementation of procedures
related to following up after the inspection, making of regulatory decisions (including
certifications) and initiating enforcement actions. There should be written guidance, e.g.,
on how to follow up on identified deficiencies. The assessor should review documented
evidence that actions were initiated when needed, e.g. with regard to product MAs
or establishment licenses. The assessor should review examples of inspection
reports, with special attention to findings and observations from the inspection and
to deficiencies, recommendations, summaries and conclusions. The assessor should
check and verify that relevant regulatory actions were taken to enforce compliance with
GXP.

Page 222 of 340

WHO Global Benchmarking Tool (GBT)

Assessor should review that any corrective or preventive actions taken as a result of
audits (or other reports of non-conformities) are implemented and documented, and
that effectiveness of the actions is verified.
The assessor should also verify the existence of internal procedures or mechanisms to
ensure that GXP certifications (where applicable) are properly issued. Evidence that a
system is in place for GXP certification should also be reviewed.

Objective: The objective of this sub-indicator is to ensure the availability and implementation of
procedures related to following up after the regulatory inspections, making of regulatory 01
decisions (including certifications) and initiating enforcement actions. Assessor should
review evidence that actions were initiated, when needed, e.g., with regard to product 02
MAs or establishment licenses. Such documentation would in turn contribute to
consistency and effectiveness of the regulatory inspection function. 03

Requirement: Regulatory process 04

Evidence to review: The assessor should ask for and review: 05

1. SOPs for GXP follow up
2. SOPs for GXP certification 06
3. SOPs for GXP enforcement

Regulatory Inspection (RI): Indicators and Fact Sheets

4. Examples of GXP certificates
5. Examples of GXP enforcement records.

References: 1. Effective drug regulation: A multicountry study. Geneva: World Health Organization;
2002 (7) (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)
2. Quality management systems — Requirements. International Standard ISO
9001:2015. Geneva. International Organization for Standardization, 2015, (4), (https://
www.iso.org/iso-9001-quality-management.html)
3. Quality systems requirements for national good manufacturing practice
inspectorates. In: WHO Expert Committee on Specifications for Pharmaceutical
Preparations: thirty-sixth report. Geneva: World Health Organization; 2002: Annex
8 (WHO Technical Report Series, No. 902), (8), (http://apps.who.int/medicinedocs/
documents/s22112en/s22112en.pdf)
4. Guiding principles for small national drug regulatory authorities. In: WHO Expert
Committee on Specifications for Pharmaceutical Preparations: thirty-first report.
Geneva: World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No.
790), (5), (http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
5. National drug regulatory legislation: guiding principles for small drug regulatory 07
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical
Preparations: thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO 08
Technical Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/documents/
s21964en/s21964en.pdf) 09
6. PIC/S Pre-Accession Procedure, related documents, PIC/S Audit Checklist, (138),
(https://www.picscheme.org/en/pre-accession)

Framework: Proces

Rating Scale: There is no documentation of the regulatory inspection follow up, decision-making
(including certification) or enforcement activities.
 ONGOING IMPLEMENTATION (OI): The NRA is preparing to establish procedures
along with the associated documentation for regulatory inspection follow up,
decision-making (including certification) and enforcement activities.
 PARTIALLY IMPLEMENTED (PI): There is evidence that the NRA has established
the documentation relevant to regulatory inspection follow up, decision-making
(including certification) and enforcement activities; however NRA experience with
these activities is only recent or limited.
 IMPLEMENTED (I): The NRA demonstrates that all the regulatory inspection follow
up, decision- making (including certification) and enforcement activities are in
place, and that the NRA consistently maintains documentation of the results of
related activities over time.

Page 223 of 340

WHO Global Benchmarking Tool (GBT)

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: RI04.03: Inspection planning is based on quality risk management (QRM).

Maturity Level: 3

Scope: 1. Medicines 01
2. Vaccines
02
Description: The assessor should verify the existence and implementation of QRM throughout the
process of inspection planning. The scope of QRM includes: 03
• Planning of routine GXP inspections by the inspectorates:
• Inspections of Investigational Medicinal Product manufacturers; 04
• Follow-up activities, such as assigning a new risk rating to the site following the
receipt of new information about the site or its products. This normally occurs between 05
inspections. Examples of the types of new information might include information on
quality defects, product recalls, market surveillance or product testing. 06
This review process requires a complete knowledge of the GXP compliance status. New

Regulatory Inspection (RI): Indicators and Fact Sheets

sites should not be rated for their initial inspection in accordance with QRM principles,
because the inspectorate in question probably will not have the necessary and complete
knowledge about the site or products. Thus, QRM should not normally be applied until a
full inspection has occurred.
Important questions to be answered during the process of preparing for an inspection
include:
1. The focus, depth, site and the duration of the inspection;
2. The required number of inspectors to be assigned;
3. Identification of any specific competency or expertise required on the inspection
team.
Applying QRM principles require the collection of relevant data during inspection
preparation, and these data (e.g. vigilance, laboratory, and previous inspection data) are
fundamental for risk-based planning.
The preparation requires the inspectors to consider the following items before making
their recommendations:
1. Specific areas in which deficiencies were identified during the most recent inspection
at the site;
2. specific areas that were not inspected in recent inspections, 07
3. Any new information that may relate to the site, for example, new quality defect
reports, MA variation applications affecting the site, product recall actions, non- 08
conforming results from market surveillance testing, or any other general indicators of
non-compliance (e.g., a failure to implement a MA variation on time). 09
Ideally, a meaningful and robust inspection plan based on QRM should be developed
in conjunction with data from other regulatory departments (e.g. vigilance data, quality
control data, or market surveillance data).

Objective: The objective of this sub-indicator is to ensure that planning of the frequency and scope
of inspections is based on QRM for enhancement of the inspection programme
effectiveness and efficiency. QRM is a methodology based upon the concept of rating
sites on the basis of an estimated risk that they may pose to patients, consumers,
animals and users of medicines. The Inspectorates should prioritize sites for inspection
when planning the frequency and scope of GXP inspections.

Requirement: Regulatory process

Evidence to review: The assessor should ask for and review:

1. Procedures describing methodology for developing annual (i.e., routine) inspection
plans along with examples of completed plans;
2. Documentation of GXP inspection plan review process along with any updates to the
inspection plans (e.g., updates based on access to additional information).

Page 224 of 340

WHO Global Benchmarking Tool (GBT)

References: 1. Quality management systems — Requirements. International Standard ISO

9001:2015. Geneva. International Organization for Standardization, 2015, (4), (https://
www.iso.org/iso-9001-quality-management.html)
2. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical
Preparations: thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8
(WHO Technical Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/
documents/s21964en/s21964en.pdf)
3. Guiding principles for small national drug regulatory authorities. In: WHO Expert 01
Committee on Specifications for Pharmaceutical Preparations: thirty-first report.
Geneva: World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No. 02
790), (5), (http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
4. Effective drug regulation: A multicountry study. Geneva: World Health Organization; 03
2002, (7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)
5. WHO guidelines on quality risk management, (55), (http://digicollection.org/ 04
whoqapharm/p/about and http://apps.who.int/medicinedocs/en)
6. Quality systems requirements for national good manufacturing practice 05
inspectorates. In: WHO Expert Committee on Specifications for Pharmaceutical
Preparations: thirty-sixth report. Geneva: World Health Organization; 2002: Annex 8 06
(WHO Technical Report Series, No. 902), (8), (http://apps.who.int/medicinedocs/

Regulatory Inspection (RI): Indicators and Fact Sheets

documents/s22112en/s22112en.pdf)
7. Quality Risk Management, Q9, International Council for Harmonization (ICH), (123),
(http://www.ich.org/products/guidelines/quality/quality-single/article/quality-risk-
management.html)
8. A recommended model for risk-based inspection planning in the GMP environment,
PIC/S. PI 037-1, Pharmaceutical Inspection Convention / Pharmaceutical Inspection
Cooperation Scheme (PIC/S), (124), (https://www.picscheme.org/layout/document.
php?id=160)
9. PIC/S Pre-Accession Procedure, related documents, PIC/S Audit Checklist, (138),
(https://www.picscheme.org/en/pre-accession)

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence of activities or documentation for
this indicator.
 ONGOING IMPLEMENTATION (OI): The NRA is preparing to establish procedures to
plan inspections based on risk management but no results yet exist.
 PARTIALLY IMPLEMENTED (PI): There is evidence that the NRA has procedures 07
to plan inspections based on risk management, as well as the capacity to perform
the procedures mentioned in the indicator; however NRA experience with this is 08
only recent or limited. If inspection planning has only recently been established, the
rating should be PI. 09
 IMPLEMENTED (I): The NRA demonstrates all the aforementioned elements and
also consistently maintains documentation of the results of related activities
over time.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: RI04.04: Multi-disciplinary teams are used to ensure proper expertise for inspection
of specific medical products.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the existence and implementation of a system for defining
the composition of the inspection team. The composition of the inspection team
should be based on the skills and experience required for the type of inspection

Page 225 of 340

WHO Global Benchmarking Tool (GBT)

being performed. A lead inspector should be designated. This lead inspector has the
responsibility to coordinate the entire inspection process (i.e., from the planning up
to the evaluation of the corrective measures) in active collaboration with the other
members of the inspection team. Assessor should note that a follow-up inspection
would trigger a new inspection process. Sub-contracted personnel or experts may be
employed as part of an inspection team to assist or to advise in a technical capacity;
however, the team should normally be led by a GMP lead inspector. Sub-contracted
personnel should be bound by the requirements of the quality system and there should
be a written contractual agreement between the parties. Organizations, experts or 01
other persons to whom inspection activities are contracted, should be free from any
commercial or financial pressures which might affect their freedom to act. They should 02
follow defined ethics rules to avoid conflict of interests. Senior management of the
Pharmaceutical Inspectorate should ensure that these outside persons are appropriately 03
qualified and experienced and that they are independent of any organizations which they
might be asked to inspect. 04
In case of specialized or highly sophisticated products (e.g. vaccines or advanced
therapy medicinal products), experts may be asked to join the inspection team. However, 05
it is important that designation of the necessary experts be based on QRM in order to
ensure consistency, effectiveness, impartiality and proportionality of inspection process. 06
The number of members of the inspection team is another aspect which should be

Regulatory Inspection (RI): Indicators and Fact Sheets

considered for optimal work of the team throughout the inspection process.

Objective: The objective of this sub-indicator is to ensure that optimal human resources are
available for and involved in regulatory inspections. Optimal composition of inspection
teams contributes to effective and efficient implementation of the function.

Requirement: Regulatory process

Evidence to review: The assessor should ask for and review:

1. Procedures for team inspections
2. Examples of inspection plans, including nominations for inspection teams.
3. Examples of inspection records.

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): The NRA does not use multi-disciplinary teams for
inspection of specific medical products.
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps to use multi-disciplinary
teams for inspection of specific medical products but no results are yet available.
 PARTIALLY IMPLEMENTED (PI): The NRA recently established the practice of

Page 226 of 340

WHO Global Benchmarking Tool (GBT)

using multi-disciplinary teams for inspection of specific medical products however

results of such practice are not yet documented.
 IMPLEMENTED (I): The NRA regularly and consistently uses multi-disciplinary
teams for inspection of specific medical products.

Limitations and remarks:  In order to ensure the access to certain specific expertise, the inspection team may
include experts from the quality control laboratory or reviewers or other personnel
external to the NRA. However, there is still a need for all members of the inspection
team to be qualified as inspectors with at least a minimum of inspection training 01
(e.g., techniques for asking questions or writing observations).
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator 02
will always apply for all benchmarked NRAs).
03
Sub Indicator: RI04.05: Inspection findings and observations are categorized according
to QRM. 04

Maturity Level: 3 05

Scope: 1. Medicines 06
2. Vaccines

Regulatory Inspection (RI): Indicators and Fact Sheets

Description: The assessor should verify the existence and implementation of a quality system
that covers the different inspection processes. The assessor should also verify that
inspection findings and observations are categorized according to QRM following a
systematic process consistent with WHO or other internationally-recognized guidelines
(e.g., observations should be classified as critical, major, or other, according to their
actual or potential impact on the patients and receivers).

Objective: The objective of this sub-indicator is to ensure that QRM considerations include
reporting and are not limited to inspection planning, preparation and conduct. Use
of QRM for the reporting process would significantly contribute to efficiency and
effectiveness of the inspection process, including the categorization of the findings and
observations.

Requirement: Regulatory process

Evidence to review: The assessor should ask for and review:

1. Procedures for categorization of findings, including categorization criteria 07
2. Examples of reviews of inspection reports
3. Examples of findings reviews and categorizations. These should be compared to 08
categorization procedures and criteria.
09
References: 1. Quality systems requirements for national good manufacturing practice
inspectorates. In: WHO Expert Committee on Specifications for Pharmaceutical
Preparations: thirty-sixth report. Geneva: World Health Organization; 2002: Annex
8 (WHO Technical Report Series, No. 902), (8), (http://apps.who.int/medicinedocs/
documents/s22112en/s22112en.pdf)
2. Quality management systems — Requirements. International Standard ISO
9001:2015. Geneva. International Organization for Standardization, 2015, (4), (https://
www.iso.org/iso-9001-quality-management.html)
3. WHO guidelines on quality risk management, (55), (http://digicollection.org/
whoqapharm/p/about and http://apps.who.int/medicinedocs/en)
4. Effective drug regulation: A multicountry study. Geneva: World Health Organization;
2002, (7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)
5. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical
Preparations: thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO
Technical Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/documents/
s21964en/s21964en.pdf)
6. Guiding principles for small national drug regulatory authorities. In: WHO Expert

Page 227 of 340

WHO Global Benchmarking Tool (GBT)

Committee on Specifications for Pharmaceutical Preparations: thirty-first report.

Geneva: World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No.
790), (5), (http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
7. Quality Risk Management, Q9, International Council for Harmonization (ICH), (123),
(http://www.ich.org/products/guidelines/quality/quality-single/article/quality-risk-
management.html)
8. Guidance on Good Manufacturing Practices (GMP): inspection report, World
Health Organization (WHO), (84), (http://apps.who.int/medicinedocs/en/q/ and http://
digicollection.org/whoqapharm/) 01
Framework: Process 02

Rating Scale:  NOT IMPLEMENTED (NI): Inspection findings and observations are not categorized 03
according to QRM.
 ONGOING IMPLEMENTATION (OI): The NRA is taking steps to categorize inspection 04
findings and observations according to QRM, but no results are yet available.
 PARTIALLY IMPLEMENTED (PI): The NRA recently established the practice to 05
categorize findings and observations according to QRM; however results of such
practice are not yet documented. 06
 IMPLEMENTED (I): Inspection findings and observations are categorized according

Regulatory Inspection (RI): Indicators and Fact Sheets

to QRM.

Limitations and remarks:  As per WHO guidelines (WHO Technical Report Series 996, annex 4, Guidance
on good manufacturing practices: inspection report), a “critical” deficiency may
be defined as an observation that has produced, or may result in a significant
risk of producing, a product that is harmful to the user. A “major” deficiency may
be defined as a non-critical observation that: a) has produced or may produce
a product that does not comply with its MA and/or prequalification application
(including variations); b) indicates a major deviation from the GMP guide; c)
indicates a failure to carry out satisfactory procedures for release of batches; d)
indicates a failure of the person responsible for quality assurance or quality control
to fulfill his or her duties; or e) consists of several other deficiencies, none of which
on its own may be major, but which together may represent a major deficiency and
should be explained and reported as such. A deficiency may be classified as “other”
if it cannot be classified as either critical or major but indicates a departure from
GMP. A deficiency may be classified as “other” either because it is judged as minor
or because there is insufficient information to classify it as “major” or “critical”.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator 07
will always apply for all benchmarked NRAs).
08
Sub Indicator: RI04.06: The same criteria are used for the inspection of domestic, foreign,
public and private facilities regardless of the ownership. 09

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should check that the same criteria are applied equally for all
establishments, domestic, foreign, public, or private, regardless of ownership. In this
case, the relevant criteria include regulatory requirements and actual inspection and
enforcement actions. In the evaluation of this sub-indicator, the assessor is asked to
perform a systematic review and comparison of inspection processes as applied to
different types of facilities.

Objective: The objective of this sub-indicator is to ensure that regulatory requirements are applied
equally for all inspections (i.e., domestic, foreign, public, or private) regardless of
ownership.

Requirement: Regulatory process

Page 228 of 340

WHO Global Benchmarking Tool (GBT)

Evidence to review: The assessor should ask for and review:

1. Examples of inspection processes and reports from domestic, foreign, public, or
private facilities. Assessor should review these to confirm that there are no differences
in application of regulatory requirements, regardless of ownership.

Regulatory Inspection (RI): Indicators and Fact Sheets

5. Quality systems requirements for national good manufacturing practice
inspectorates. In: WHO Expert Committee on Specifications for Pharmaceutical
Preparations: thirty-sixth report. Geneva: World Health Organization; 2002: Annex
8 (WHO Technical Report Series, No. 902), (8), (http://apps.who.int/medicinedocs/
documents/s22112en/s22112en.pdf)

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence that the inspectorate uses
the same criteria for the inspection of domestic, foreign, public and private
establishments regardless the ownership.
 ONGOING IMPLEMENTATION (OI): The NRA is working on ensuring that the
same criteria are used for the inspection of domestic, foreign, public and private
establishments regardless the ownership; however, no yet results are yet available.
 PARTIALLY IMPLEMENTED (PI): There is evidence that the NRA uses the same
criteria for the inspection of domestic, foreign, public and private establishments
regardless the ownership; however, it has only limited or recent experience with it.
 IMPLEMENTED (I): There is evidence that the NRA uses the same criteria for the 07
inspection of domestic, foreign, public and private establishments regardless the
ownership; and also consistently maintains documentation of the results of related 08
activities over time.
09
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Indicator: RI05 Mechanism in place to monitor regulatory performance and output.

Objective: The objective of this indicator is to ensure the existence and implementation of a
system or mechanism for monitoring regulatory performance and output and for using
that information to estimate the effectiveness and efficiency of the regulatory inspection
function.

Category: 09. Monitoring progress and assessing outcomes & impact

Sub Indicator: RI05.01: A database is established and regularly updated of all establishments
which may be subject to inspection, along with their relevant regulatory decisions
(certifications and/or enforcement activities).

Maturity Level: 4

Page 229 of 340

WHO Global Benchmarking Tool (GBT)

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the existence of an updated database of all facilities and
premises subject to regulatory inspections throughout the supply chain. Clinical trial
sites should also be included. The database should be comprehensive and should
contain essential information for the inspected facility, as well as information from
previous inspections (e.g., general information, conclusions, and relevant regulatory
decisions such as certification or enforcement actions). Furthermore, the assessor 01
should verify that the database is regularly updated in accordance with well-established
procedures and mechanisms. 02

Objective: The objective of this sub-indicator is to ensure consistency and traceability of regulatory 03
inspection activities through the establishment and maintenance of a database of all
the inspected establishments along with the regulatory decisions associated with each 04
establishment (e.g., certifications and enforcement actions).
05
Requirement: Monitoring progress and assessing outcomes and impact
06
Evidence to review: The assessor should ask for and review:

Regulatory Inspection (RI): Indicators and Fact Sheets

1. Database of inspected facilities showing inventory of facilities along with regulatory
decisions.

Framework: Output

Rating Scale:  NOT IMPLEMENTED (NI): There is no list or database of all establishments that
are subject to inspection along with their regulatory actions (certification and/or
enforcement).
 ONGOING IMPLEMENTATION (OI): The NRA is preparing to establish a database,
but there is no evidence of results from such activities.
 PARTIALLY IMPLEMENTED (PI): There is evidence that the NRA has the elements
(i.e., procedures, documentation, management systems, and other relevant
information) and the capacity to perform the processes mentioned in the indicator,
however, it has only limited experience or a limited number of documented events.
 IMPLEMENTED (I): There is list or database of all establishments that are subject
to inspection along with their regulatory actions (certification and/or enforcement).
Also, this list or database is regularly updated.

Limitations and remarks:  Some facilities may be subject to inspection without being licensed by the NRA
(e.g. clinical trial sites). Although these sites may not be licensed by the NRA,

Page 230 of 340

WHO Global Benchmarking Tool (GBT)

ideally, they are licensed by another governmental institution. Integration and

communication within the government is critical in the latter situation.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator
will always apply for all benchmarked NRAs).

Sub Indicator: RI05.02: Inspection reports are well-archived and easily retrieved.

Maturity Level: 3 01
02
Scope: 1. Medicines
2. Vaccines 03

Description: The assessor should verify that reports of inspections are well-archived and easily 04
retrieved. The assessor should review how all the information collected during the
inspection process is managed and how inspection reports are registered and archived 05
(e.g., in paper or electronic formats).
06
Objective: The objective of this sub-indicator is to ensure that a mechanism or procedure is in

Regulatory Inspection (RI): Indicators and Fact Sheets

place for defining a standard format for inspection reports and for setting up an archival
system which is secure and can be reached only by authorized, competent personnel.
Reports should be easily, securely, effectively and efficiently retrieved.

Requirement: Monitoring progress and assessing outcomes and impact

Evidence to review: The assessor should ask for and review:

1. SOPs for archiving and retrieval of inspection reports
2. Evidence for adequate space and suitability of the archive system
3. Documentation of the time needed to retrieve inspection reports.

References: 1. Effective drug regulation: A multicountry study. Geneva: World Health Organization;
2002, (7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)
2. Quality management systems — Requirements. International Standard ISO
9001:2015. Geneva. International Organization for Standardization, 2015, (4), (https://
www.iso.org/iso-9001-quality-management.html)
3. Quality systems requirements for national good manufacturing practice
inspectorates. In: WHO Expert Committee on Specifications for Pharmaceutical 07
Preparations: thirty-sixth report. Geneva: World Health Organization; 2002: Annex
8 (WHO Technical Report Series, No. 902), (8), (http://apps.who.int/medicinedocs/ 08
documents/s22112en/s22112en.pdf)
4. Guiding principles for small national drug regulatory authorities. In: WHO 09
Expert Committee on Specifications for Pharmaceutical Preparations: thirty-first
report. Geneva: World Health Organization; 1990: Annex 6 (WHO Technical Report
Series, No. 790), (5), (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf)
5. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical
Preparations: thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8
(WHO Technical Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/
documents/s21964en/s21964en.pdf)
6. PIC/S Pre-Accession Procedure, related documents, PIC/S Audit Checklist, (138),
(https://www.picscheme.org/en/pre-accession)

Framework: Output

Rating Scale:  NOT IMPLEMENTED (NI): Inspection reports are not properly archived and not
easily retrieved.
 ONGOING IMPLEMENTATION (OI): The NRA is in the process of establishing an
inspection archival system.

Page 231 of 340

WHO Global Benchmarking Tool (GBT)

 PARTIALLY IMPLEMENTED (PI): Inspection report archival system is recently

established; however, consolidated documentation over time (at least two years) is
not yet available.
 IMPLEMENTED (I): Inspection reports are properly archived and easily retrieved.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: RI05.03: Inspection reports are subjected to a regular and robust review by experts 01
other than the designated inspection team.
02
Maturity Level: 4
03
Scope: 1. Medicines
2. Vaccines 04

Description: The assessor should check the existence of a process for systematic review of each 05
inspection report by a group of experts other than the designated inspectors who
performed the inspection and wrote the report. The reviewing experts or committee 06
may be internal or external provided that the necessary confidentially and declaration

Regulatory Inspection (RI): Indicators and Fact Sheets

of interest measures are followed as indicated in the transparency and communication
indicator. The review process should be technically efficient, thorough and deep.

Objective: The objective of this sub-indicator is to ensure the existence of a systematic review
of each individual inspection report by a group of experts other than the designated
inspectors who performed the inspection and wrote the report. This enhances
objectivity and fairness and reduces bias.

Requirement: Regulatory process

Evidence to review: The assessor should ask for and review:

1. SOPs for review of GXP inspection reports
2. Examples of records of the review of GXP inspection reports.

References: 1. Guiding principles for small national drug regulatory authorities. In: WHO Expert
Committee on Specifications for Pharmaceutical Preparations: thirty-first report.
Geneva: World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No.
790), (5), (http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf) 07
2. Effective drug regulation: A multicountry study. Geneva: World Health Organization;
2002, (7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf) 08
3. Guidance on Good Manufacturing Practices (GMP): inspection report, World
Health Organization (WHO), (84), (http://apps.who.int/medicinedocs/en/q/ and http:// 09
digicollection.org/whoqapharm/)
4. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical
Preparations: thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO
Technical Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/documents/
s21964en/s21964en.pdf)
5. Quality management systems — Requirements. International Standard ISO
9001:2015. Geneva. International Organization for Standardization, 2015, (4), (https://
www.iso.org/iso-9001-quality-management.html)
6. Quality systems requirements for national good manufacturing practice inspectorates.
In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: thirty-sixth
report. Geneva: World Health Organization; 2002: Annex 8 (WHO Technical Report Series,
No. 902), (8), (http://apps.who.int/medicinedocs/documents/s22112en/s22112en.pdf)

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): Inspection reports are not subjected to a regular and
robust review by experts other than the designated inspection team.

Page 232 of 340

WHO Global Benchmarking Tool (GBT)

 ONGOING IMPLEMENTATION (OI): The NRA is in the process of establishing a

regular and robust review by experts other than the designated inspection team;
however, such a mechanism or process is not yet established.
 PARTIALLY IMPLEMENTED (PI): The NRA recently established a process for regular
and robust review by experts other than the designated inspection team; however,
it is at the implementation stage, so this practice is not yet consolidated.
 IMPLEMENTED (I): Inspection reports are subjected to a regular and robust review
by experts other than the designated inspection team.
01
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 02

Sub Indicator: RI05.04: Inspection data and outcomes are systematically evaluated or interpreted. 03
04
Maturity Level: 4
05
Scope: 1. Medicines
2. Vaccines 06

Regulatory Inspection (RI): Indicators and Fact Sheets

Description: The assessor should confirm that inspection reports and data, including inspection
findings and outcomes, are collectively reviewed, analyzed, evaluated and interpreted in
a systematic way to identify gaps, strengths, and trends. The output of this systematic
review and analysis represents the body of inspection metrics. The availability of
these inspection metrics enables the NRA to identify relevant guidelines that require
developing or updating, to set up and amend inspection plans and resources, and to
re- structure the format of inspection process.

Objective: The objective of this sub-indicator is to ensure that a system is in place for checking
and confirming the effectiveness of the regulatory inspection programme through
the systematic review, analysis, evaluation and interpretation of the inspection
reports.

Requirement: Monitoring progress and assessing outcomes and impact

Evidence to review: The assessor should ask for and review:

1. Documentation of available inspection data and analyses of outcomes (i.e., inspection
metrics). 07
2. Documentation showing frequency of data analysis and review.
08
References: 1. Effective drug regulation: A multicountry study. Geneva: World Health Organization;
2002, (7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf) 09
2. Quality management systems — Requirements. International Standard ISO 9001:2015.
Geneva. International Organization for Standardization, 2015, (4), (https://www.iso.org/
iso-9001-quality-management.html)
3. Quality systems requirements for national good manufacturing practice inspectorates.
In: WHO Expert Committee on Specifications for Pharmaceutical Preparations:
thirty-sixth report. Geneva: World Health Organization; 2002: Annex 8 (WHO Technical
Report Series, No. 902), (8), (http://apps.who.int/medicinedocs/documents/s22112en/
s22112en.pdf)
4. Guiding principles for small national drug regulatory authorities. In: WHO
Expert Committee on Specifications for Pharmaceutical Preparations: thirty-first
report. Geneva: World Health Organization; 1990: Annex 6 (WHO Technical Report
Series, No. 790), (5), (http://apps.who.int/medicinedocs/documents/s21964en/
s21964en.pdf)
5. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical
Preparations: thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO
Technical Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/documents/
s21964en/s21964en.pdf)

Page 233 of 340

WHO Global Benchmarking Tool (GBT)

Framework: Output

Rating Scale:  NOT IMPLEMENTED (NI): Inspection data are not systematically evaluated and
interpreted.
 ONGOING IMPLEMENTATION (OI): The NRA is in the process of establishing a
system for inspection data evaluation and interpretation.
 PARTIALLY IMPLEMENTED (PI): The NRA recently established a system
for evaluation and interpretation of Inspection data; however, consolidated
documentation over a time (at least two years) is not yet available. 01
 IMPLEMENTED (I): Inspection data are systematically evaluated and interpreted.
02
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 03

Sub Indicator: RI05.05: Performance indicators for regulatory inspection activities are established 04
and implemented
05
Maturity Level: 4
06
Scope: 1. Medicines

Regulatory Inspection (RI): Indicators and Fact Sheets

2. Vaccines

Description: The assessor should verify the existence and implementation of performance indicators
for different activities included under the regulatory inspection functions.
Specifically, the system should define key performance indicators (KPIs) along the entire
regulatory inspection activity chain and all indicators should be adequately justified.
For the purpose of clarity and consistency, established KPIs should be supported with
guidelines for monitoring and maintenance of the KPIs. The guidelines in turn should
be supported by SOPs and tools that define the procedures to be used for monitoring
and evaluating the performance indicators and that define procedures and timelines for
reviewing and revising the indicators.
Established KPIs might be qualitative, quantitative or combination of both. In general,
quantitative indicators are preferred to avoid bias or misinterpretation. However,
qualitative indicators are also accepted. Qualitative indicators may or may not include
scoring or scaling to render them semi-quantitative and thus more informative. The
assessor should ensure that indicators are measured on a regular basis to monitor
progress and advancement.
In addition, the assessor should verify measured indicators are analyzed to identify 07
trends or abnormalities. Justifications for any identified abnormalities should be
provided; when necessary, process optimizations should be introduced to avoid 08
recurrence.
09
Objective: The objective of this sub-indicator is to ensure that a system, mechanism, or procedure
exists to require the NRA to establish performance indicators along the entire
regulatory inspection chain. Additionally, the objective is to ensure that KPIs are actually
contributing to monitoring of regulatory performance, to measuring effectiveness of
regulatory inspection regulatory activities, and to making any necessary adjustments or
optimizations.

Requirement: KPIs for regulatory inspection activities

Evidence to review: The assessor should ask for and review:

1. Documents supporting the system, mechanism, or procedure compelling the NRA to
establish and implement performance indicators along the entire regulatory inspection
activity chain.
2. Evidence that the performance indicators have been established and implemented,
and that the members of staff involved in the regulatory inspection function are aware
of the indicators and the guidelines and SOPs used for monitoring and evaluating their
performance.
3. The current performance indicators for regulatory inspection activities

Page 234 of 340

WHO Global Benchmarking Tool (GBT)

Regulatory Inspection (RI): Indicators and Fact Sheets

5. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical
Preparations: thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO
Technical Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/documents/
s21964en/s21964en.pdf)
6. World Health Organization. European Observatory on Health Systems and Policies.
Performance measurement for health system improvement: experiences, challenges
and Prospects, (125), (https://www.who.int/management/district/performance/
PerformanceMeasurementHealthSystemImprovement2.pdf)
7. World Health Organization. European Observatory on Health Systems and Policies.
Health System Performance Comparison: an agenda for policy, information and
research, (126), (http://www.euro.who.int/__data/assets/pdf_file/0009/244836/Health-
System-Performance-Comparison.pdf)
8. Quality management -- Quality of an organization -- Guidance to achieve sustained
success, ISO 9004:2018, International Organization for Standardization (ISO), (115),
(https://www.iso.org/standard/70397.html)

Framework: Output 07
08
Rating Scale:  NOT IMPLEMENTED (NI): There are no KPIs for regulatory inspection activities.
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted KPIs for 09
regulatory inspection activities but they have not yet been reported.
 PARTIALLY IMPLEMENTED (PI): The NRA has developed KPIs for regulatory
inspection activities and has been applying them for less than two year or they
have not covered all critical steps.
 IMPLEMENTED (I): The NRA has established and implemented KPIs for regulatory
inspection activities. The indicators are reviewed regularly, and appropriate actions
are taken, and decisions made.

Page 235 of 340

WHO Global Benchmarking Tool (GBT)

 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator
will always apply for all benchmarked NRAs).

Indicator: RI06 Mechanism exists to promote transparency, accountability

and communication.

Objective: The objective of this indicator is to ensure that mechanisms are in place that promote
communication within and outside the NRA, that promote transparency and outreach to
the public, and that establish milestones that reflect the accountability of the NRA to its 01
mandate. Additionally, these contribute to mutual understanding and involvement of all
stakeholders relevant to regulatory inspection activities. Consequently, confidence in the 02
regulatory system is raised.
03
Category: 08. Transparency, accountability and communication
04
Sub Indicator: RI06.01: The list of inspectors is publicly available and the identity of the
designated team for each inspection is communicated to the relevant institutions 05
subject to inspections.
06
Maturity Level: 4

Regulatory Inspection (RI): Indicators and Fact Sheets

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that a proper mechanism is available, enacted and used,
together with internal procedures, to ensure that information related to the list and
identity of the inspectors is publicly available. The assessor should also verify that
the identity of the designated team for each inspection is provided to the institutions
that are being inspected. The communications regarding information of designated
inspectors may be done via public communication aids (e.g., website or national bulletin)
or, for a specific inspection, via a letter of designation, a notification of inspection, or a
similar approach.

Objective: The objective of this sub-indicator is to ensure confidence building and accountability of
the licensing structure via enhanced transparency, specifically, a mechanism to ensure
the list and identity of inspectors is publicly available.

Requirement: Transparency, accountability and communication 07

Evidence to review: The assessor should ask for and review: 08

1. Procedures and processes for publishing the list of inspectors.
2. Documents comparing the current NRA list of inspectors to the publicly available list 09
of inspectors.
3. Procedures and processes for communicating the identity of the designated team for
each inspection to the relevant institution subject to inspection.
4. Examples of inspection notifications, letters of designation or similar communications
using another mechanism.

Page 236 of 340

WHO Global Benchmarking Tool (GBT)

Geneva: World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No.
790), (5), (http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
5. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical
Preparations: thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO
Technical Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/documents/
s21964en/s21964en.pdf)
6. PIC/S Pre-Accession Procedure, related documents, PIC/S Audit Checklist, (138),
(https://www.picscheme.org/en/pre-accession) 01
Framework: Output 02
03
Rating Scale:  NOT IMPLEMENTED (NI): The NRA is not notifying the inspectee of the designated
inspection team and the list of the inspectors is not published 04
 ONGOING IMPLEMENTATION (OI): The NRA notifies the inspectee of the
designated inspection; however, the list of inspectors is not established and is 05
neither internally nor publicly available.
 PARTIALLY IMPLEMENTED (PI): The NRA notifies the inspectee of the designated 06
inspection team; however, the list of inspectors is not published.

Regulatory Inspection (RI): Indicators and Fact Sheets

 IMPLEMENTED (I): The NRA publishes an updated list of inspectors and a
procedure for notifying the inspectee of the designated inspection team is well-
established.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: RI06.02: The updated list or database of all inspected facilities along their
regulatory decisions, actions and enforcement activities, is regularly published
and publicly available.

Maturity Level: 4

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that a well-established system is in place to ensure
that information related to inspections performed is available to the general public. 07
This information includes a list of inspections performed, along with the following
information for each inspection: conclusions, regulatory decisions, regulatory actions 08
and enforcement actions (e.g., recalls, production suspensions, license suspensions
or revocations, sanctions, or prosecutions). In addition, the assessor should make sure 09
that urgent regulatory decisions (e.g., those related to quality, safety or efficacy of a
medical product) and enforcement actions affecting the public health are immediately
communicated to the public to ensure high level of public health protection.

Objective: The objective of this sub-indicator is to build confidence and accountability of the
licensing structure via enhanced transparency through making the information related
to regulatory inspections available for the general public.

Requirement: Transparency, accountability and communication

Evidence to review: The assessor should ask for and review:

1. Latest publicly available list or database of inspected facilities along with their
conclusions, regulatory decisions, regulatory actions and enforcements.
2. Earlier publicly available list of inspected facilities.

Page 237 of 340

WHO Global Benchmarking Tool (GBT)

9001:2015. Geneva. International Organization for Standardization, 2015, (4), (https://

www.iso.org/iso-9001-quality-management.html)
3. Quality systems requirements for national good manufacturing practice
inspectorates. In: WHO Expert Committee on Specifications for Pharmaceutical
Preparations: thirty-sixth report. Geneva: World Health Organization; 2002: Annex
8 (WHO Technical Report Series, No. 902), (8), (http://apps.who.int/medicinedocs/
documents/s22112en/s22112en.pdf)
4. Guiding principles for small national drug regulatory authorities. In: WHO Expert
Committee on Specifications for Pharmaceutical Preparations: thirty-first report. 01
Geneva: World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No.
790), (5), (http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf) 02
5. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical 03
Preparations: thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8
(WHO Technical Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/ 04
documents/s21964en/s21964en.pdf)
6. PIC/S Pre-Accession Procedure, related documents, PIC/S Audit Checklist, (138), 05
(https://www.picscheme.org/en/pre-accession)
06
Framework: Output

Regulatory Inspection (RI): Indicators and Fact Sheets

Rating Scale:  NOT IMPLEMENTED (NI): There is no publicly available list or database of
all inspections performed along with their conclusions, regulatory decisions,
regulatory actions and enforcements.
 ONGOING IMPLEMENTATION (OI): The NRA is preparing to establish a publicly
available list or database of all inspections performed along with their conclusions,
regulatory decisions, regulatory actions and enforcements, but there is no evidence
of results from such activities.
 PARTIALLY IMPLEMENTED (PI): There is evidence that the NRA has the elements
(i.e., procedures, documentation, management systems, and other relevant
information) and the capacity to perform the processes mentioned in the indicator,
however, it has only limited experience or a limited number of documented events.
 IMPLEMENTED (I): There is a publicly available list or database of all inspections
performed along with their conclusions, regulatory decisions, regulatory actions
and enforcements.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 07

Sub Indicator: RI06.03: Inspection metrics are regularly published and publicly available. 08
09
Maturity Level: 4

Scope: 1. Medicines
2. Vaccines

Description: The assessor should check that a well-established and regularly updated system or
mechanism is available to communicate to the public information related to inspection
metrics or performance.
The available quality metrics can be used as input in inspection models, but also can
be used to predict possible drug shortages, to determine inspection schedules for a
manufacturer, or to optimize the format of inspection processes.
Some examples of possible inspection metrics include: total number inspections,
relative proportion of each type of inspection (e.g., routine, cause- triggered, new
product, new license, innovative pharma, or generics), number in each category of
findings (i.e., critical,
major or other), number of inspection findings for each inspection type, average number
of findings reported for each type of inspection, average number of inspection findings
over time, and the number of selected frequently occurring observations. Inspection

Page 238 of 340

WHO Global Benchmarking Tool (GBT)

metrics are always dependent on inspectorate mandate and activities planned. The
above examples do not represent an exhaustive list of the possible metrics. Rather,
other inspection metrics may be adopted based on multiple factors which include the
objective, the relevance, and the ease of measurements.

Objective: The objective of this sub-indicator is to ensure confidence building and accountability of
the licensing structure via enhanced transparency provided by a system or mechanism
in place to inform the general public of inspection metrics.
01
Requirement: Transparency, accountability and communication
02
Evidence to review: The assessor should ask for and review:
1. Procedures and processes addressing inspections metrics 03
2. Procedures and processes for publishing the inspection metrics and performance
3. Published inspection metrics. 04

References: 1. Effective drug regulation: A multicountry study. Geneva: World Health Organization; 05
2002, (7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)
2. Quality management systems — Requirements. International Standard ISO 06
9001:2015. Geneva. International Organization for Standardization, 2015, (4), (https://

Regulatory Inspection (RI): Indicators and Fact Sheets

www.iso.org/iso-9001-quality-management.html)
3. Quality systems requirements for national good manufacturing practice
inspectorates. In: WHO Expert Committee on Specifications for Pharmaceutical
Preparations:
thirty-sixth report. Geneva: World Health Organization; 2002: Annex 8 (WHO Technical
Report Series, No. 902), (8), (http://apps.who.int/medicinedocs/documents/s22112en/
s22112en.pdf)
4. Guiding principles for small national drug regulatory authorities. In: WHO Expert
Committee on Specifications for Pharmaceutical Preparations: thirty-first report.
Geneva: World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No.
790), (5), (http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
5. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical
Preparations: thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8
(WHO Technical Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/
documents/s21964en/s21964en.pdf)

Framework: Output 07

Rating Scale:  NOT IMPLEMENTED (NI): Inspection metrics are not regularly published or publicly 08
available.
 ONGOING IMPLEMENTATION (OI): The NRA is preparing to establish publicly 09
available inspection metrics, but there is no evidence of results from such activities.
 PARTIALLY IMPLEMENTED (PI): There is evidence that the NRA has the elements
(i.e., procedures, documentation, management systems, and other relevant
information) and the capacity to perform the processes mentioned in the indicator,
however, it has only limited experience or a limited number of documented events.
 IMPLEMENTED (I): Inspection metrics are not regularly published and publicly
available.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: RI06.04: Information on inspections conducted is regularly published and publicly
available in accordance with national confidentiality requirements.

Maturity Level: 4

Scope: 1. Medicines
2. Vaccines

Page 239 of 340

WHO Global Benchmarking Tool (GBT)

Description: The assessor should check that the regulatory inspections performed, and reports for
those inspections, are available for the general public. The information that may be
shared with the public varies according to the national legislations. In some countries,
full inspection reports (redacted or non-redacted) may be published. In other countries,
only inspection summaries (or excerpts) may be publicly available. Both approaches are
accepted; however, the earlier one is encouraged. It should be noted that publication of
inspection reports or summaries would significantly contribute to regulatory reliance
and recognition in the area of regulatory inspection.
01
Objective: The objective of this sub-indicator is to build confidence and accountability of the
licensing structure via enhanced transparency through public availability of information 02
on inspections performed and reports from those inspections.
03
Requirement: Transparency, accountability and communication
04
Evidence to review: The assessor should ask for and review:
1. Procedures and processes for publishing the list of inspections performed 05
2. The list of inspections performed
3. Examples of actual published Inspection reports or summaries (or excerpts). 06

Regulatory Inspection (RI): Indicators and Fact Sheets

References: 1. Effective drug regulation: A multicountry study. Geneva: World Health Organization;
2002, (7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)
2. Quality management systems — Requirements. International Standard ISO
9001:2015. Geneva. International Organization for Standardization, 2015, (4), (https://
www.iso.org/iso-9001-quality-management.html)
3. Quality systems requirements for national good manufacturing practice
inspectorates. In: WHO Expert Committee on Specifications for Pharmaceutical
Preparations: thirty-sixth report. Geneva: World Health Organization; 2002: Annex
8 (WHO Technical Report Series, No. 902), (8), (http://apps.who.int/medicinedocs/
documents/s22112en/s22112en.pdf)
4. Guiding principles for small national drug regulatory authorities. In: WHO Expert
Committee on Specifications for Pharmaceutical Preparations: thirty-first report.
Geneva: World Health Organization; 1990: Annex 6 (WHO Technical Report Series, No.
790), (5), (http://apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
5. National drug regulatory legislation: guiding principles for small drug regulatory
authorities. In: WHO Expert Committee on Specifications for Pharmaceutical
Preparations: thirty-fifth report. Geneva: World Health Organization; 1999: Annex 8 (WHO
Technical Report Series, No. 885), (1), (http://apps.who.int/medicinedocs/documents/ 07
s21964en/s21964en.pdf)
08
Framework: Output
09
Rating Scale:  NOT IMPLEMENTED (NI): Inspection reports or summaries (or excerpts) are not
publicly available.
 ONGOING IMPLEMENTATION (OI): The NRA established a procedure or
mechanism by which relevant inspection reports or summaries (or excerpts) could
be publicly available; however, this is not yet implemented.
 PARTIALLY IMPLEMENTED (PI): Inspection reports or summaries are publicly
available, but only upon request (i.e. pull rather than push mechanism).
 IMPLEMENTED (I): Inspection reports or summaries (or excerpts) are regularly and
consistently published and publicly available.

Limitations and remarks:

Page 240 of 340

WHO Global Benchmarking Tool (GBT)

07 01

National Regulatory System (RS): Indicators and Fact Sheets

Laboratory Testing
(LT): Indicators and
Fact Sheets

WHO Global
Benchmarking 02
Tool (GBT) 03
for Evaluation of
04
National Regulatory
System of Medical 05
Products 06
07
08
09

Page 241 of 340

WHO Global Benchmarking Tool (GBT)

07. Laboratory Testing (LT):

Indicators and Fact Sheets

Function:
07 – LABORATORY TESTING (LT)
01
Description: The laboratory testing regulatory function is intended to ensure that the National 02
Regulatory Authority (NRA) is able to assess the quality of medical products by
performing quality tests on them in certain situations. For example, this testing can 03
be a requirement to corroborate manufacturer’s test results as a part of the evaluation
for marketing authorization or for a variation to a marketing authorization. Testing 04
can be a requirement for lot release for certain products depending upon national
regulations. Testing also may be needed for products for which there has been a 05
complaint or a report or for products that are under investigation due to an adverse
event. As part of the market surveillance function, laboratory testing is utilized for 06
checking and confirming the quality of medical products placed on the market and
for detecting substandard and falsified medical products. In order to do this product 07
testing, the NRA must have access to suitable laboratories where these tests can

Laboratory Testing (LT): Indicators and Fact Sheets

be performed.

If a country is able to provide all the resources needed, a laboratory under the
responsibility of the NRA or a governmental laboratory represents the best choice.
Commonly, this governmental laboratory is a national control laboratory (NCL). As an
alternative option, the regulatory system may have access to external laboratories,
either inside or outside the country, to perform the required tests on behalf on the NRA.
When external laboratories are used, regulatory decisions and actions remain at the
discretion of the NRA and the NRA retains accountability.

A well-functioning laboratory for medical products testing is an important resource

for the national regulatory system. The staff generally has expertise in different
life science disciplines and can help in other regulatory activities, for example, the
assessment and review of marketing authorization applications and the review of
clinical trial data.

When the regulatory laboratory testing activities are decentralized to one or more
internal or external laboratories, the NRA or NCL must ensure that a continuous 08
information exchange mechanism is established so the central authority can issue
guidance and the decentralized entity can report back with the information needed 09
for making decisions.

Indicator: LT01 Legal provisions, regulations and guidelines required to define the regulatory
framework of laboratory testing activities.

Objective: The objective of this indicator is to ensure that laboratory testing activities are
supported by a comprehensive set of legal provisions, regulations and guidelines
which provide the necessary mandate to implement all activities related to this
regulatory function.

Category: 01. Legal provisions, regulations and guidelines

Sub Indicator: LT01.01: There are legal provisions to establish a national quality control laboratory
(NCL) to perform quality control (QC) testing, and/or to authorize the National
Regulatory Authority (NRA) to sub-contract the required testing services.

Maturity Level: 1

Page 242 of 340

WHO Global Benchmarking Tool (GBT)

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that legal provisions exist and are enacted for establishing
a NCL that is responsible for controlling medical products and/or for authorizing the
national authority to establish an operational agreement with an independent external
laboratory. In the latter case, the independent laboratory may be inside or outside the-
country.
The assessor should verify that legal provisions exist and are enacted that authorize the 01
NCL to perform QC testing and to issue official results of these tests.
02
Objective: The objective of this sub-indicator is to ensure the existence of legal provisions,
regulations or other official government publications that establish the laboratory 03
testing function and that provide the mandate for the responsible entity to actually
implement these activities. 04

Requirement: Legal provisions for establishing the laboratory testing function through an established 05
NCL and/or authorizing the NRA to have an operational agreement with an independent
laboratory. 06
Legal provisions authorizing the NCL to issue official results of these tests.
07
Evidence to review: The assessor should ask for and review:

Laboratory Testing (LT): Indicators and Fact Sheets

1. Legal provisions establishing the laboratory testing function;
2. Regulations or other official government publication establishing the laboratory testing
function;
3. Legal provisions authorizing the NCL to perform testing and officially issue results
of the same;
4. Regulations or other official government publication establishing NRA’s responsibilities
when sub-contracting the required testing services.

Framework: Structure/Foundation/Input 08
Rating Scale:  NOT IMPLEMENTED (NI): There is no legal basis, regulation, or other official 09
government publication establishing an NCL to perform QC testing, authorizing the
NRA to sub-contract the required testing services, or authorizing the NCL to issue
official results
 ONGOING IMPLEMENTATION (OI): Although there is no legal basis, demonstrable
steps have been taken towards establishing one.
 PARTIALLY IMPLEMENTED (PI): The legal basis was recently established and
implementation is underway.
 IMPLEMENTED (I): There is a legal basis authorizing the NRA to establish a national
control laboratory that has the duties and responsibilities required to perform this
function or to sub-contract the required testing services.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: LT01.02: Legal provisions and regulations allow the NRA to recognize and use
laboratory testing-related decisions, reports or information from other NRAs
or regional and international bodies.

Maturity Level: 1

Page 243 of 340

WHO Global Benchmarking Tool (GBT)

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that legal provisions and regulations allow the NRA and the
NCL to recognize and /or rely on laboratory decisions, data and information from NCLs of
other countries or regional and international institutions.

Objective: The objective of this sub-indicator is to ensure that legal provisions and regulations provide
the necessary mandate for the NCL to implement regulatory recognition and/or reliance. 01
Requirement: Legal provisions and regulations allowing the reliance on and/or recognition of regulatory 02
decisions in the area of laboratory testing
03
Evidence to review: The assessor should ask for and review:
1. Legal provisions allow reliance on and/or recognition of other NCL regulatory decisions; 04
2. Agreements and memoranda of understanding with other NCLs to apply reliance on
and/or recognition of regulatory decisions; 3. Documentation providing the rationale and 05
justification for reliance on and/or recognition of other NCL regulatory decisions.
4. Examples of communication with other NCLs; 06
5. Examples of NRA consideration of decisions, information and data from other NCLs;
6. If legal provisions prevent the NCL from relying on and/or recognizing other NCLs 07
regulatory decisions, examples of any impact this condition may have in the laboratory

Laboratory Testing (LT): Indicators and Fact Sheets

testing function.

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence of legal provisions or regulations
allowing reliance on and/or recognition of other NCL regulatory decisions.
 ONGOING IMPLEMENTATION (OI): The NRA is preparing to draft such legal provisions
and regulations, but there is no evidence of results from such activities.
 PARTIALLY IMPLEMENTED (PI): The legal basis was established recently and is at the
implementation stage, so this practice is not consolidated yet.
 IMPLEMENTED (I): The NRA has such legal provisions and regulations and also 08
consistently maintains documentation of the results of related activities over time.
 NA: When the legal provisions prevent reliance on and/or recognition of other NCL 09
regulatory decisions.

Limitations and remarks: In some countries legal provisions may prevent the NCL from relying on and/or recognizing
other NCL regulatory decisions; in this case the sub-indicator may be considered as not
applicable.

Indicator: LT02 Arrangement for effective organization and good governance.

Objective: The objective of this indicator is to ensure the implementation of effective organization
and good governance practices at the entities responsible for laboratory testing activities
which, in turn, contribute to effective and efficient functioning of this regulatory function.

Category: 02. Organization and governance

Sub Indicator: LT02.01: There is a defined organizational structure with clear responsibilities
to conduct laboratory testing activities.

Maturity Level: 2

Page 244 of 340

WHO Global Benchmarking Tool (GBT)

Scope: 1. Medicines
2. Vaccines

Description: The assessor should identify the organization designated to establish, implement or
maintain the laboratory testing regulatory function, as well as the specific organizational
structure taking on the different relevant activities. Responsibilities, duties and roles of
these structures should be clearly defined and documented. If more than one structure
is involved, the assessor should check the ways and approaches by which coordination
among these structures is taking place. When some of the activities of the NCL can be 01
performed by laboratories of other national or international institutions, these institutions
must not have any conflict of interests. 02

Objective: The objective of this sub-indicator is to ensure effective organization and good governance 03
of laboratory testing of medical products and that these activities are taken over by defined
structures with clear roles and responsibilities. 04

Requirement: Defined structure and clear responsibilities 05

Evidence to review: The assessor should ask for and review: 06

1. Organization chart of the organizations responsible for the implementation of laboratory
testing activities along with identification of the particular structures implementing the 07
function.

Laboratory Testing (LT): Indicators and Fact Sheets

2. Documentation clarifying roles and responsibilities of the organizational structures
implementing laboratory testing activities. This may include administrative decrees, terms
of reference, and other relevant documents.
3. Documented contract with laboratories of other national or international institutions,
when applicable, establishing responsibilities of each party and stating there is no conflict
of interest.

References:

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There are no documents available establishing a defined
structure with clear responsibilities to conduct laboratory testing activities.
 ONGOING IMPLEMENTATION (OI): Although there are no approved documents, there
is evidence that they are being prepared.
 PARTIALLY IMPLEMENTED (PI): Such documents were approved recently (e.g. less
than one year ago). 08
 IMPLEMENTED (I): Such documents were approved some time ago and there has
already been a cycle of internal or external audits (with or without accreditation) or a 09
system review.

Limitations and remarks:  In some countries there may be a centralized NCL; in this case the assessor should
look at roles and responsibilities within the structure implementing the laboratory
testing function.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator
will always apply for all benchmarked NRAs).

Sub Indicator: LT02.02: Documented procedures are implemented to ensure the involvement
and contributions of the NCL to support regulatory oversight.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the existence of written documentation defining the
responsibilities and roles of the NCL for its involvement and contribution to other regulatory
functions. Examples of the responsibilities and roles include, but are not limited to, providing

Page 245 of 340

WHO Global Benchmarking Tool (GBT)

technical and scientific input before and after a product receive marketing authorizations
(MAs), contributing to clinical trials authorizations, participating in regulatory inspections,
performing quality testing on samples collected within the context of the market
surveillance programme, detecting substandard and falsified medical products, contributing
to analyses of vigilance data, and providing technical advice for national lot release.

Objective: The objective of this sub-indicator is to ensure that standard procedures are implemented
for effective involvement and contributions of the NCL in support of other regulatory
functions. 01
Requirement: Procedures relevant to involvement and contributions of the NCL to other regulatory 02
functions.
03
Evidence to review: The assessor should ask for and review:
1. Standard procedures and other documentations detailing roles and responsibilities of 04
the NCL with respect to its contribution to other regulatory functions and to regulatory
oversight as applicable. 05
2. Records demonstrating active involvement and contribution of the NCL to other
regulatory functions. 06

References: 1. WHO good practices for pharmaceutical quality control laboratories, (35), (http:// 07
digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en)

Laboratory Testing (LT): Indicators and Fact Sheets

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There are no documented procedures or results that
demonstrate the NCL involvement in and contributions to other regulatory
functions or to regulatory oversight.
 ONGOING IMPLEMENTATION (OI): Although there are no approved documents,
there is evidence that they are being prepared.
 PARTIALLY IMPLEMENTED (PI): Such documents were approved recently (e.g.
less than one year ago).
 IMPLEMENTED (I): Such documents were approved some time ago (at least one
year ago), and there has already been a cycle of internal or external audits (with or
without accreditation) or system review.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).
08
Indicator: LT03 Laboratory activities implemented as per well-established plans and policies
according a Quality Management System (QMS). 09
Objective: The objective of this indicator is to ensure that the laboratory establishes, implements,
and maintains a Quality Management System appropriate to the scope of its activities. It
is essential to have clear policies and plans in place for effective and reliable laboratory
operations. Laboratory policies should cover a range of activities, including calibration
and qualification of equipment, validation of testing methods, establishment of reference
standards, and performance of testing and retesting. In addition, policies should be in place
that cover the establishment, qualification, distribution, and use of reference materials
for laboratory testing, including their calibration against international reference materials
or standards, when available. The policies should cover the various types of reference
materials used in the laboratory, e.g., primary standards, working standards, and official
national or international reference preparations.

Category: 03. Policy and strategic planning

Sub Indicator: LT03.01: Documented and implemented policy for testing exists that is based
on the product’s risk.

Maturity Level: 3

Page 246 of 340

WHO Global Benchmarking Tool (GBT)

Scope: 1. Medicines
2. Vaccines

Description: The assessor should identify documented evidence that demonstrates there is a risk-
based assessment to support products to be tested and the testing to be done. Market
surveillance methodologies, such as field screening technologies, can serve as a basis for
prioritizing products for testing in the laboratory.

Objective: The objective of this sub- indicator is to ensure efficient use of resources to address 01
products of concern for public health such as substandard and falsified medical products.
02
Requirement: Prioritization of testing activities using a risk-based approach.
03
Evidence to review: The assessor should ask for and review:
1. Documents outlining the information used to support decisions. 04
2. Documentation of the criteria used to arrive at decisions, and evidence confirming these
are risk-based. 05

References: 1. WHO good practices for pharmaceutical quality control laboratories, (35), (http:// 06
digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en)
2. General requirements for the competence of testing and calibration laboratories, ISO 07
ISO/IEC 17025:2015 International Organization for Standardization, 2015, (33), (https://

Laboratory Testing (LT): Indicators and Fact Sheets

www.iso.org/standard/66912.html)
3. WHO good practices for pharmaceutical microbiology laboratories, (34), (http://
digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en)
4. WHO guidelines on quality risk management, (55), (http://digicollection.org/
whoqapharm/p/about and http://apps.who.int/medicinedocs/en)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There is no documented evidence that a risk-based testing
policy exists
 ONGOING IMPLEMENTATION (OI): Although there are no approved documents, there
is evidence that they are being prepared.
 PARTIALLY IMPLEMENTED (PI): Such documents were approved recently (e.g. less
than one year ago).
 IMPLEMENTED (I): Such documents were approved some time ago and there has
already been a cycle of internal or external audits (with or without accreditation) or
system review. 08
Limitations and remarks: Testing should be based on the products’ characteristics. The choice of tests and the 09
extent of testing should be based on risk management principles.
Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: LT03.02: Documented and implemented policy exists on the validation, verification
and transfer of analytical procedures.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify there is a policy on validation, verification and transfer of
analytical procedures, which establishes that all procedures should be validated. This
policy should include pharmacopoeial procedures, manufacturer’s procedures, NCL
procedures, and other procedures used to support the regulatory oversight function.
The assessor should verify that the NCL is using validated analytical methodologies
that ensure reliable results. If validated methods are transferred into the NCL from a
manufacturer or any other laboratory, the assessor should verify that the transferred

Page 247 of 340

WHO Global Benchmarking Tool (GBT)

procedure was re-validated, and that supporting documentation is available. The process
by which these methodologies were validated should be described, and documentation
should identify the validation parameters that were evaluated. The validation parameters
should be selected based on the category of analysis. Documentation should also identify
the required validation parameters when pharmacopoeial methodologies are used or when
methods are transferred.
To evaluate this, the assessor should check and review protocols and reports of test
method validations.
01
Objective: The objective of this sub- indicator is to ensure that policies and procedures that cover
test method validation, verification, and transfer are available and will ensure reliability of 02
laboratory test results.
03
Requirement: Test method validation and verification protocols and reports.
Standard procedures for transfer of validated test methods from the manufacturer or other 04
laboratories.
05
Evidence to review: The assessor should ask for and review:
1. Policies and procedures for validation and verification of test methods and transfer of 06
validated test methods;
2. List of transferred validated test methods; 07
3. Examples of documentation of validation or revalidation of transferred test methods;

Laboratory Testing (LT): Indicators and Fact Sheets

4. List of validated test methods;
5. Examples of validation protocols for selected test methods;
6. Examples of validation reports of selected test methods.

References: 1. WHO good manufacturing practices for pharmaceutical products: main principles, (51),
(http://digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en)
2. General requirements for the competence of testing and calibration laboratories, ISO
ISO/IEC 17025:2015 International Organization for Standardization, 2015, (33), (https://
www.iso.org/standard/66912.html)
3. WHO good practices for pharmaceutical quality control laboratories, (35), (http://
digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en)
4. Quality Management (QM) documents, European Directorate for the Quality of Medicines
and Healthcare (EDQM), Council of Europe (107) (https://www.edqm.eu/en/quality-
management-guidelines-86.html)
5. WHO good practices for pharmaceutical microbiology laboratories, (34), (http://
digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en)
6. Validation of analytical procedures, European Directorate for the Quality of Medicines 08
and Healthcare (EDQM), OMCL Network of the Council of Europe, PA/PH/. MCL (13) 82
2R, (106), (https://www.edqm.eu/medias/fichiers/validation_of_analytical_procedures_ 09
paphomcl_13_82_2r.pdf)
7. WHO International Reference Preparations Catalogue, (103), (http://www.who.int/
bloodproducts/catalogue/en/)
8. WHO manual for the establishment of national and other secondary standards for
vaccines. WHO/IVB/11.03, (136), (http://apps.who.int/iris/bitstream/10665/70669/1/
WHO_IVB_11.03_eng.pdf)
9. WHO manual for the preparation of secondary reference materials for in vitro diagnostic
assays designed for infectious disease nucleic acid or antigen detection: calibration to
WHO International Standards, (137), (http://www.who.int/entity/bloodproducts/norms/
SecStandManWHO_TRS_1004_web_ Annex_6.pdf?ua=1)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There is no documented evidence that a policy on the
validation, verification, and transfer of analytical procedures exist.
 ONGOING IMPLEMENTATION (OI): Although there are no approved documents, there
is evidence that they are being prepared.
 PARTIALLY IMPLEMENTED (PI): Such documents were approved recently (e.g. less
than one year ago).

Page 248 of 340

WHO Global Benchmarking Tool (GBT)

 IMPLEMENTED (I): Such documents were approved some time ago (at least one year
ago), and there has already been a cycle of internal or external audits (with or without
accreditation) or system review.

Limitations and remarks:  In this context, “verification” refers to the verification of compendial procedures,
namely, the demonstration of acceptable performance under the conditions in which
the procedure is used.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 01
Sub Indicator: LT03.03: A policy is in place to establish or qualify all reference standards used 02
in laboratory testing activities.
03
Maturity Level: 3
04
Scope: 1. Medicines
2. Vaccines 05

Description: The assessor should verify the existence of a national policy for establishment of all 06
reference standards used in laboratory activities. The NCL may set a policy not to develop
and establish reference standard materials, and instead, rely on reference standards 07
developed by other entities (e.g. World Health Organization (WHO) or other NCLs). However,

Laboratory Testing (LT): Indicators and Fact Sheets

in this case, the policy should explain how the reliability of those reference standards that
are used is ensured by the laboratory staff. If there is a policy to establish national reference
standards, the assessor should review the policy in detail including guidance that it provides
related to methodologies and criteria for establishing such national reference standards
and for ensuring traceability to international reference standards. In all cases, the policy for
establishment or qualification of national reference standards (if any) should be based on
relevant WHO guidelines (listed in the reference section) or other internationally accepted
standards and practices. In addition, the assessor should verify that a suitable system is in
place for using reference standards in the laboratory. In addition, the assessor should verify
that the policy includes plans and procedures for replacement or replenishment of reference
standards in case the standards have exhausted. Furthermore, if an NCL is supplying
national standards to users nationwide (i.e., as a major responsibility), there should be plans
and procedures for distribution and monitoring the use of the standards.
The reference standards should be identified, recorded and properly stored. An individual or
entity within the NCL should be assigned to monitor their use and to verify that they used
appropriately.
08
Objective: The objective of this sub- indicator is to ensure that specific policies are in place with
respect to establishment and maintenance of reference standard materials. Reliable 09
reference standard materials, in turn, contribute to effective laboratory testing activities.

Requirement: Policy for establishment or qualification of all reference standards used in laboratory
activities.

Evidence to review: The assessor should ask for and review:

1. Policy for establishment or qualification of all reference standards used in laboratory;
2. Implemented procedures for validation or bridging studies and for implementation and
use of control charts;
3. Examples of reference standards used in the laboratory, if any, along with the
documentation supporting qualification (including qualification criteria) and providing
evidence for traceability to international reference standards.

References: 1. WHO good practices for pharmaceutical quality control laboratories, (35), (http://
digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en)
2. Recommendations for the preparation, characterization and establishment, (37),
(https://www.who.int/biologicals/reference_preparations/en/)
3. WHO good manufacturing practices for pharmaceutical products: main principles, (51),
(http://digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en)

Page 249 of 340

WHO Global Benchmarking Tool (GBT)

4. General requirements for the competence of testing and calibration laboratories, ISO
ISO/IEC 17025:2015 International Organization for Standardization, 2015, (33), (https://
www.iso.org/standard/66912.html)
5. WHO International Reference Preparations Catalogue, (103), (http://www.who.int/
bloodproducts/catalogue/en/)
6. WHO manual for the establishment of national and other secondary standards for
vaccines. WHO/IVB/11.03, (136), (http://apps.who.int/iris/bitstream/10665/70669/1/
WHO_IVB_11.03_eng.pdf)
7. WHO manual for the preparation of secondary reference materials for in vitro diagnostic 01
assays designed for infectious disease nucleic acid or antigen detection: calibration to
WHO International Standards, (137), (http://www.who.int/entity/bloodproducts/norms/ 02
SecStandManWHO_TRS_1004_web_ Annex_6.pdf?ua=1)
03
Framework: Structure/Foundation/Input
04
Rating Scale:  NOT IMPLEMENTED (NI): There is no documented evidence of a policy in place to
establish or qualify all reference standards used in the laboratory. 05
 ONGOING IMPLEMENTATION (OI): Although there are no approved documents, there
is evidence that they are being prepared. 06
 PARTIALLY IMPLEMENTED (PI): Such documents were approved recently (e.g. less
than one year ago). 07
 IMPLEMENTED (I): Such documents were approved some time ago (at least one year

Laboratory Testing (LT): Indicators and Fact Sheets

ago), and there has already been a cycle of internal or external audits (with or without
accreditation) or system review.

Limitations and remarks:

Sub Indicator: LT03.04: Documented and implemented procedures exist for handling atypical
or out-of-specification (OOS) results, including a retest policy.

Maturity Level: 2

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the existence and implementation of a re-testing policy in case
of OOS test results. The policy should initially define OOS and set clear rules to handle
these results. The assessor should check the policy and make sure it considers the
properties procedures provide guidelines on handling appeals against the decisions of the 08
NCL. of different test methods including their variabilities and sensitivities (e.g. qualitative
or quantitative methods or in vitro or in vivo methods). 09
The assessor should verify the establishment and implementation of standard procedures
for handling events of non-compliance with pre-set specifications. Procedures should
provide comprehensive guidance on how to report such non-compliant events and
how to communicate and report these cases to supervisors or higher-level officials
in the laboratory. In addition, the procedures should provide guidance for notifying
the manufacturer and for meeting with its representatives, when necessary. Relevant
procedures should cover notification of other departments or entities within the regulatory
system. In all cases, assessor should check that a notice of non-compliance authorized by
pre-assigned official is issued by the NCL.
The assessor should verify that procedures for the evaluation of test results and decision-
making are available and enacted. The assessor should also confirm that relevant

Objective: The objective of this sub- indicator is to ensure that specific policies are in place that
contribute to effective laboratory testing activities. In particular, a re-testing policy is
required to ensure consistency in handling OOS test results.

Requirement: Standard procedures for dealing with and handling events of non-compliance.Re- testing
policy is established and implemented. Standard procedures for evaluation of test results
and decision-making.

Page 250 of 340

WHO Global Benchmarking Tool (GBT)

Evidence to review: The assessor should ask for and review:

1. Policy for handling OOS test results.
2. List of OOS test results.
3. Examples of documentation detailing the handling of selected OOS test results.
4. Standard procedures for dealing with and handling events of confirmed non-compliance
of test results with pre-set specifications.
5. List of non- compliant events within a particular time interval (e.g. last year).
6. Example records of non-compliant (i.e., OOS) events along with their investigations and
final decisions. . 01
7. Standard procedures for evaluation of test results and for decision-making regarding
compliance or non-compliance. 02
8. Example records for which test results were evaluated and for decisions made on
compliance or non-compliance (i.e., for specific test methods that are being reviewed by 03
the assessor).
9. Standard procedures for handling appeals against decisions made by the NCL. 04
10. List of appeals against NCL decisions within a particular time interval (e.g., last year).
11. Example records (i.e., for specific test methods that are being reviewed by the 05
assessor) from selected cases in which the NCL decision was appealed.
06
References:
07
Framework: Structure/Foundation/Input

Laboratory Testing (LT): Indicators and Fact Sheets

Proces

Rating Scale:  NOT IMPLEMENTED (NI): There is no documented evidence of a policy and
procedures in place to handle atypical or OOS results and no evidence for a
retesting policy.
 ONGOING IMPLEMENTATION (OI): Although there are no approved documents,
there is evidence that they are being prepared.
 PARTIALLY IMPLEMENTED (PI): Such documents were approved recently (e.g.
less than one year ago).
 IMPLEMENTED (I): Such documents were approved some time ago (at least one
year ago), and there has already been a cycle of internal or external audits (with or
without accreditation) or system review.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Indicator: LT04 Human resources to perform laboratory testing activities. 08

09
Objective: The objective of this indicator is to ensure to that all entities within an NRA are adequately
resourced with a trained, experienced and skilled workforce that is empowered to fully
perform the function of laboratory testing of medical products. This will ensure that
medical products testing activities are performed in accordance with international best
practices.
The objective of this indicator is to evaluate the human resource capacity of the entities
with respect to the number of personnel, the skills and experience of the personnel, and
the overall composition the workforce, with the goal of evaluating whether the workforce
possesses the specific expertise required to perform the laboratory testing function.

Category: Resources (HR, FR, infrastructure and equipment)

Sub Indicator: LT04.01: Sufficient competent staff (i.e., education, training, skills and experience)
are assigned to perform laboratory testing activities.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Page 251 of 340

WHO Global Benchmarking Tool (GBT)

Description: The assessor should verify that the human resources assigned to perform laboratory
testing activities should be sufficient with respect to numbers and competent with respect
to the requisite skills, education, experience and training. There should be technical
documents and standard operating procedures (SOPs) that provide guidance on the
required background for laboratory testing activities and that consider the requirements for
educational background, competencies, skills, experience, and training.
The assessor should verify that the NRA estimated the number of staff required to
effectively and efficiently perform laboratory testing function and that the NRA actually
recruited that number. In addition, the assessor should verify that these competency 01
requirements are well-established and maintained by the NRA. Metrics and statistics
on the different activities performed as well as performance indicators can be used for 02
estimating the adequacy of the number of the assigned staff. The assessor should also
verify that the competency of the assigned staff is built, maintained and improved through 03
recruitment as well as continuous on-the-job training.
04
Objective: The objective of this sub-indicator is to ensure the existing human resources for laboratory
testing is sufficient, in terms of numbers, experience, and specific competencies, to 05
perform all the activities along the entire laboratory testing chain.
06
Requirement: Sufficient number of competent human resources in charge of laboratory testing activities.
07
Evidence to review: The assessor should ask for and review:

Laboratory Testing (LT): Indicators and Fact Sheets

1. Evidence that the number of staff members involved in each of the documented
activities along the entire laboratory testing process flow is adequate.
2. Evidence that the systems and structures are in place to ensure appropriate placement
of staff with respect to competence and skills.
3. Evidence that the system and structures have been implemented. The documentation
should include the records to verify that the staff competence is appropriate for the job
requirements.
4. Evidence that the professional profiles of the human resources engaged in laboratory
testing activities are appropriate with respect to education, skills, and expertise, to perform
a particular function along the laboratory testing chain. Documentation should include a
list of the requisite skills and training for each position.
5. Recruitment plan.

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): The NRA does not have enough competent staff (i.e.,
education, training, skills and experience) to perform laboratory testing activities
 ONGOING IMPLEMENTATION (OI): The NRA has recently developed a plan to recruit
adequate competent staff; however, the plan has not been implemented.
 PARTIALLY IMPLEMENTED (PI): The NRA has initiated the implementation of
the human resources development plan; however, there is need to complete the
competency profile.
 IMPLEMENTED (I): The NRA has a sustained number of competent staff (i.e.,
education, training, skills and experience) assigned to perform laboratory testing
activities.

Page 252 of 340

WHO Global Benchmarking Tool (GBT)

 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: LT04.02: Duties, functions, and responsibilities of the staff in charge of laboratory
testing activities are established and updated in the respective job descriptions.

Maturity Level: 3

Scope: 1. Medicines 01
2. Vaccines
02
Description: The assessor should verify that procedures are in place to maintain a current and
updated structure for managing job descriptions for personnel participating in 03
laboratory testing activities. In addition, job descriptions should address current
staff duties, responsibilities and the requisite competencies. A job description with 04
this format and content should be established and implemented for all staff. The
management of job descriptions should be supported by a guidance document 05
that provides direction on when and how to update the information and where the
information should be kept for easy access. The guidance document should present 06
the appropriate duties and responsibilities that are assigned to each member of the
organization involved in laboratory testing activities. Thus, the professional profiles of 07
staff are reflected in their respective roles and responsibilities within the NRA. There

Laboratory Testing (LT): Indicators and Fact Sheets

should be procedures to guide responsible persons to document that duties, functions
and responsibilities are revised and kept up to date. In addition, procedures should
be available to guide the keeping and documenting of up to date work schedules and
enforcing the implementation of the documented guidelines and procedures.

Objective: The objective of this sub-indicator is to ensure that duties and responsibilities of
the staff are clear and well defined, that job descriptions are kept up to date with
current duties, functions and responsibilities, and that these activities are adequately
documented.

Requirement: Duties, roles and responsibilities of the staff relevant to laboratory testing activities.

Evidence to review: The assessor should ask for and review:

1. Procedures and guidelines that guide placement of staff members within the NRA;
2. The professional profiles of staff (i.e., job descriptions) and documentation that they
are related to their current roles and duties;
3. The professional profiles of the external experts and documentation which ensures that 08
experts’ profiles are complete and consistent with that prescribed in the legal provisions;
4. Procedures to guide the documentation of up to date duties and work schedules, and 09
to enforce the implementation of the documented guidelines and procedures;
5. Job descriptions for designated staff.

Framework: Structure/Foundation/Input

Page 253 of 340

WHO Global Benchmarking Tool (GBT)

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence of defined or established duties,
functions, responsibilities, respective job descriptions and necessary required
competencies.
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted or developed the role
and responsibilities document but it has not yet been implemented.
 PARTIALLY IMPLEMENTED (PI): The NRA has initiated implementation of this
requirement but it has not been defined or followed for all staff or the roles and
responsibilities documents, including staff job descriptions, are not up to date.
 IMPLEMENTED (I): The NRA has defined and established all required duties, 01
functions, and responsibilities, and respective job descriptions are up-to-date.
02
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 03

Sub Indicator: LT04.03: Training plan developed, implemented and updated at least once a year 04
for staff in charge of laboratory testing activities.
05
Maturity Level: 3
06
Scope: 1. Medicines
2. Vaccines 07

Laboratory Testing (LT): Indicators and Fact Sheets

Description: The assessor should verify that training plans are developed, implemented and updated
at least once every year to reflect the current situation by considering education and
experience of the staff. The training plan should be complemented with guidelines or
similar documents that guide the development and implementation of training plans. The
assessor should ensure that induction training for new staff as well as continued on- the-
job training for staff is planned and implemented. There should be procedures to approve
the training plan and the budget allocated for implementing and updating the training
plan. The plans should present clearly defined training goals and should include training
in certain topics and skills to address identified deficiencies. Learning objectives, training
methods and activities, evidence of learning, and evaluation and assessment of training
should be documented. This documentation should confirm that the learning objectives
were achieved and were designed to address weaknesses within the entities. Procedures
should be in place to ensure that a training plan is developed, implemented and updated
at least once every year. The assessor should verify that there is a system in place for
monitoring the implementation and effectiveness of the training plan and for documenting
the skills acquired in training activities for internal and external experts.
08
Objective: The objective of this sub-indicator is to ensure that a training plan for staff exists, and that
it is implemented and updated annually. Through the training plan, NRA can be sure that 09
competency of staff in charge of laboratory testing activities is maintained and enhanced.

Requirement: Implementation of training plan

Evidence to review: The assessor should ask for and review:

1. Guidelines for development, implementation and annual update (i.e., at least once per
year) of the training plan. Guidelines should also provide for a mechanism to measure
effectiveness of training.
2. Documentation for the system or structures used to approve the training plan and to
evaluate the adequacy of the budget allocated to the training activities.
3. The current or existing staff training plan (or matrix) for staff. The assessor should
assess this in in relation to the respective individual job descriptions.
4. SOP for developing and maintaining the training plan.
5. Evidence that the NRA has investigated and identified training needs.
6. List of trainings performed.
7. Example records for training activities.

References: 1. Effective drug regulation: A multicountry study. Geneva: World Health Organization;
2002, (7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)

Page 254 of 340

WHO Global Benchmarking Tool (GBT)

2. Quality systems requirements for national good manufacturing practice inspectorates.

In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: thirty-sixth
report. Geneva: World Health Organization; 2002: Annex 8 (WHO Technical Report Series,
No. 902), (8), (http://apps.who.int/medicinedocs/documents/s22112en/s22112en.pdf)

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): There is no systematic training program including training
plan (or matrix). 01
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted or developed the
training plan but there is no evidence of implementation. 02
 PARTIALLY IMPLEMENTED (PI): The NRA has developed and initiated the training
plan implementation. However, the NRA has not fulfilled all required planned training 03
or has been applying the plan for less than two years.
 IMPLEMENTED (I): The NRA has an updated training plan developed that is supported 04
by adequate records to demonstrate effective plan implementation, including induction
training for new staff and routine on-the-job training for recruited staff. 05

Limitations and remarks:  Training plans must be updated regularly; ideally on an annual basis, but not less 06
frequently that once every two years.
 Some regulatory functions may include many training activities that are not 07
incorporated in the institutional training programme. Such training normally is

Laboratory Testing (LT): Indicators and Fact Sheets

offered by invitation. In this case, the assessor should recognize reports from non-
routine laboratory testing relevant training not included in the NRA training plan.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator
will always apply for all benchmarked NRAs).

Sub Indicator: LT04.04: The NRA generates and maintains records of staff training activities
and training effectiveness verification.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that records of staff training that is performed or organized by
the NRA are generated, maintained, regularly updated. This activity should be supported by
guidelines that direct the NRA to generate and maintain records of staff training activities.
Procedures should be in place to document and propose staff training needs and to 08
allocate a budget for continuous staff capacity building and development. The assessor
should check that there is an evaluation or assessment mechanism to verify the quality 09
of learning and to confirm that learning objectives are achieved. Documentation should
include an inventory (i.e., soft and/or hard) system that records all impactful and non-
impactful trainings and identifies all staff members who participated. A system to measure
or estimate impact of trainings should be established.

Requirement: Training records

Evidence to review: The assessor should ask for and review:

Page 255 of 340

WHO Global Benchmarking Tool (GBT)

Laboratory Testing (LT): Indicators and Fact Sheets

 PARTIALLY IMPLEMENTED (PI): The NRA has recently initiated plans to generate,
document and keep records of staff training activities but they are not fully followed
for all training activities or they have been established for less than two years.
 IMPLEMENTED (I): The NRA generates and maintains records of staff training activities.

Limitations and remarks:  The assessor should note that some NRAs out-source training including staff
capacity development activities. In this case the assessor should request the identity
of the provider, as well as evaluations of the provider. The assessor may request the
professional profiles of tutors or resource persons used to offer training. The assessor
may also request records covering the archiving systems in place.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Indicator: LT05 Well maintained and equipped infrastructures for laboratory activities.

Objective: The objective of this indicator is to confirm that buildings and equipment fulfill quality and 08
technical requirements in order to enable adequate performance of the quality testing in
the laboratories. For some tests, specific requirements for rooms might be necessary to 09
achieve reliable results.

Category: 06. Resources (HR, FR, infrastructure and equipment)

Sub Indicator: LT05.01: Laboratory facilities are adequate to perform quality testing activities.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should review the appropriateness and the adequacy of facilities for
performing the laboratory testing and other NCL-related activities. The assessor should
specifically evaluate and check the adequacy of the laboratory premises, the work
environment and the work space.

Objective: The objective of this sub-indicator is to confirm the adequacy of laboratory facilities for
contributing to effectiveness and reliability of quality testing at the NCL.

Page 256 of 340

WHO Global Benchmarking Tool (GBT)

Requirement: Adequate facilities

Evidence to review: The assessor should ask for and review:

1. Evidence for the adequacy of the laboratory premises and facilities, including the work
environment and work space;
2. Evidence for the adequacy of the layout of the laboratory premises;
3. Evidence that the assigned biosafety level for the laboratory work is in accordance with
WHO recommendations (when applicable).
01
References: 1. General requirements for the competence of testing and calibration laboratories, ISO
ISO/IEC 17025:2015 International Organization for Standardization, 2015, (33), (https:// 02
www.iso.org/standard/66912.html)
2. WHO good manufacturing practices for pharmaceutical products: main principles, (51), 03
(http://digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en)
3. Quality management systems — Requirements. International Standard ISO 9001:2015 04
International Organization for Standardization, 2015, (4), (https://www.iso.org/iso-9001-
quality-management.html) 05
4. WHO good practices for pharmaceutical quality control laboratories, (35), (http://
digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en) 06
5. WHO good practices for pharmaceutical microbiology laboratories, (34), (http://
digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en) 07
6. Laboratory Biosafety Manual, World Health Organization, (105), (https://www.who.int/

Laboratory Testing (LT): Indicators and Fact Sheets

csr/resources/publications/biosafety/WHO_CDS_CSR_LYO_2004_11/en/)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): Laboratory facilities are not adequate for performing the
targeted testing activities.
 ONGOING IMPLEMENTATION (OI): The NCL is planning to establish, qualify or
upgrade the laboratory facilities in order to be adequate for performing the targeted
testing activities, however no evidence of such plans exists.
 PARTIALLY IMPLEMENTED (PI): The NCL recently (less than a year ago) established
or upgraded the laboratory facilities so it became adequate for performing the
targeted testing activities.
 IMPLEMENTED (I): Laboratory facilities are adequate and are used for performing the
targeted testing activities in a proficient manner.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 08
Sub Indicator: LT05.02: Equipment calibration, qualification and maintenance plans have been 09
defined and implemented and records have been maintained.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should identify the equipment and instruments used in the laboratory and
confirm the existence of the relevant operation manuals and SOPs.
The assessor should verify the calibration, qualification and maintenance status of
laboratory equipment as well as plans and procedures for regular updates. In addition, the
assessor should check the log books and records to verify that the equipment is being
correctly calibrated, qualified and maintained.

Objective: The objective of this sub- indicator is to ensure that calibrated, qualified and well-
maintained laboratory equipment is available and contributes to the effectiveness and
reliability of laboratory tests.

Requirement: Calibration, qualification and maintenance of laboratory equipment.

Page 257 of 340

WHO Global Benchmarking Tool (GBT)

Evidence to review: The assessor should ask for and review:

1. List of laboratory equipment;
2. Operational manuals and SOPs for selected equipment;
3. Log books that record use of selected equipment;
4. Log books that record calibration and maintenance of selected equipment;
5. Plans and procedures for annual equipment calibration;
6. Plans and procedures for annual equipment maintenance;
7. Sample calibration certificates;
8. Sample maintenance records. 01
References: 1. WHO good practices for pharmaceutical quality control laboratories, (35), (http:// 02
digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en)
2. General requirements for the competence of testing and calibration laboratories, ISO 03
ISO/IEC 17025:2015 International Organization for Standardization, 2015, (33), (https://
www.iso.org/standard/66912.html) 04
3. WHO good manufacturing practices for pharmaceutical products: main principles,
(51), (http://digicollection.org/whoqapharm/p/about and http://apps.who.int/ 05
medicinedocs/en)
06
Framework: Structure/Foundation/Input
07
Rating Scale:  NOT IMPLEMENTED (NI): There are no plans for equipment calibration,

Laboratory Testing (LT): Indicators and Fact Sheets

qualification and maintenance.
 ONGOING IMPLEMENTATION (OI): Although there are no approved documents,
there is evidence that they are being prepared.
 PARTIALLY IMPLEMENTED (PI): Such documents were approved recently (e.g.
less than one year ago).
 IMPLEMENTED (I): The NCL has up-to-date equipment calibration, qualification
and maintenance plans and records, which were approved some time ago (at
least one year ago), and there has already been a cycle of internal or external
audits (with or without accreditation) or system review.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Indicator: LT06 Procedures established and implemented to perform laboratory testing

activities according to Quality Management System.

Objective: The objective of this indicator is to ensure that a management system is in place to 08
ensure traceability of actions. Standard procedures are an essential element of overall
good regulatory practices; importantly, they contribute to consistency, effectiveness 09
and impartiality of the regulatory processes. Written documentation supporting the
laboratory quality system should exist, and the regulatory testing laboratory should have
procedures in place for the different laboratory testing activities performed. Procedures
should be available for receipt, storage and handling of test samples (including retention
samples); receipt, handling, and storage of test reagents and materials (including
reference standards); performance of different quality tests; notification and issuance of
test results; handling of out-of-specification results (including re-testing when necessary);
and handling events of non-compliance with pre-set specifications. An auditing system
(internal and external) should be established and implemented to assess the performance
of the management system of the laboratory testing activities with the objective of
promoting on-going improvement.

Category: 07. Regulatory process

Sub Indicator: LT06.01: There are procedures for receipt, handling, storage and retention
of samples.

Maturity Level: 3

Page 258 of 340

WHO Global Benchmarking Tool (GBT)

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that written and enforced standard procedures exist for
receipt, handling and storage of medical products samples received for testing. The
assessor should evaluate whether special handling procedures (e.g., requirements for
storage temperature (i.e., cool or frozen) and humidity or recommendations to protect
from light) are considered when required. The assessor should, as well, verify that
procedures are providing instructions for storage and handling of retention samples 01
which may be required for re-testing or verification purposes. In addition, the assessor
should verify the existence of an identification system to ensure traceability. 02

Objective: The objective of this sub- indicator is to ensure that implemented procedures for 03
ensuring traceability and consistency of laboratory testing activities are available.
04
Requirement: Standard procedures for handling and retention of samples under testing.
05
Evidence to review: The assessor should ask for and review:
1. Standard procedures for receipt, handling, storage and retention of samples received 06
for quality testing.
2. Example records of the reviewed procedures. 07
3. In the case of biological testing samples, verify that procedures for sampling from

Laboratory Testing (LT): Indicators and Fact Sheets

the manufacturer are available.
4. Documentation for the identification system used in paper or electronic records to
trace samples from receipt to its testing and storage.

References: 1. General requirements for the competence of testing and calibration laboratories, ISO
ISO/IEC 17025:2015 International Organization for Standardization, 2015, (33), (https://
www.iso.org/standard/66912.html)
2. WHO good practices for pharmaceutical quality control laboratories, (35), (http://
digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en)
3. WHO good manufacturing practices for pharmaceutical products: main principles, (51),
(http://digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en)

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): There are no documents for receipt, handling, storage or
retention of samples.
 ONGOING IMPLEMENTATION (OI): Although there are no approved documents, there 08
is evidence that they are being prepared.
 PARTIALLY IMPLEMENTED (PI): Such documents were approved recently (e.g. less 09
than one year ago).
 IMPLEMENTED (I): The NCL has all the essential documents, which were approved
some time ago (at least one year ago), and there has already been a cycle of internal
or external audits (with or without accreditation) or system review.

Limitations and remarks:  Samples may come from market surveillance or directly from manufacturers.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: LT06.02: There are documented procedures for performing tests in accordance
with MA documentation.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that documented procedures for testing are in accordance with
the MA. The assessor should review the procedures by which the NCL staff has access to

Page 259 of 340

WHO Global Benchmarking Tool (GBT)

the MA. Ideally, the NCL staff should be involved in the MA evaluation process (at least with
respect to information on pharmaceutical quality).

Objective: The objective of this sub- indicator is to ensure that standard procedures for performing tests
are in accordance with the MA in order to ensure consistency of laboratory testing activities.

Requirement: Standard procedures for testing of medical products as per their MA documentation.

Evidence to review: The assessor should ask for and review: 01

1. List of SOPs for testing of different medical products;
2. Examples of SOPs for quality testing of selected medical products; 02
3. Examples of records for testing done according to the SOPs reviewed in item #2 above;
4. Documented evidence that the NCL has access to the QC part of the MA (i.e., when the 03
NCL is part of the NRA);
5. When testing activities are sub-contracted, the contract should reflect that the NCL 04
should have access to the QC part of the MA (including updates), or a verified copy of the
QC part of the MA. 05

Laboratory Testing (LT): Indicators and Fact Sheets

Rating Scale:  NOT IMPLEMENTED (NI): There is no documented evidence that tests are performed
in accordance with MA documentation.
 ONGOING IMPLEMENTATION (OI): Although there are no approved documents, there
is evidence that they are being prepared.
 PARTIALLY IMPLEMENTED (PI): Such documents were approved recently (e.g. less
than one year ago).
 IMPLEMENTED (I): Procedures and records for performing tests in accordance
with MA documentation were implemented some time ago (at least one year ago),
and there has already been a cycle of internal or external audits (with or without
accreditation) or system review.

Limitations and remarks:  The methods approved in the MA application should ideally be used by the NCL. If not,
the assessor should review the NCL justification for deviation from the MA approved
testing methods along with the NCL procedure to handle this situation.
If the method approved in the MA application is not routinely followed by the NCL
because of some limitations in terms of technology or resources or for some other 08
reason, the method included in the MA application should be determinative in the
event of OOS test results. This is not only for technical and scientific reasons but also 09
for legislative reasons.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: LT06.03: A documented procedure is implemented for notification of test results
and for ensuring that test results are issued following a standardized format.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that the NCL has established and implemented standard
procedures for notifying interested parties (e.g., the manufacturer, the MA holder, and,
when necessary, the inspectorate) of the test results. When the NCL is not part of the NRA,
the test results should be forwarded to the NRA. Such procedures should provide guidance
on issuing the test results in a standardized format containing the information needed to
identify the sample, the testing laboratory, the applicant or other source of the sample, the
type of tests conducted, the procedures applied, and the relevant specifications.

Page 260 of 340

WHO Global Benchmarking Tool (GBT)

Objective: The objective of this sub- indicator is to ensure the implementation of standard
procedures for notification of test results and for issuing test results following a
standardized format. These procedures will ensure consistency of laboratory
testing activities.

Requirement: Standard procedures for following a standardized format for laboratory test results.

Evidence to review: The assessor should ask for and review:

1. Standard procedures for notification and issuance of test results; 01
2. List of issued test results within a specified time (e.g., last year);
3. Documentation of examples of decision-making process; 02
4. Examples of issued test results;
5. Examples of records showing implementation of the reviewed procedures; 03
6. Examples of test results forwarded to the NRA when the NCL is not part of the NRA.
04
References: 1. Model certificate of analysis, World Health Organization (WHO), (102), (http://
digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en) 05
2. WHO good practices for pharmaceutical quality control laboratories, (35), (http://
digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en) 06

Framework: Process 07

Laboratory Testing (LT): Indicators and Fact Sheets

Rating Scale:  NOT IMPLEMENTED (NI): There is no documented evidence for a procedure for
notification of test results or for ensuring that test results are issued following
standardized format.
 ONGOING IMPLEMENTATION (OI): Although there are no approved documents,
there is evidence that they are being prepared.
 PARTIALLY IMPLEMENTED (PI): Such documents were approved recently (e.g.
less than one year ago).
 IMPLEMENTED (I): Procedures and records are in place for notification of test
results and for ensuring that results are issued following standardized format.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: LT06.04: There are appropriate procedures for obtaining and handling of all materials
required for testing.

Maturity Level: 3 08
Scope: 1. Medicines 09
2. Vaccines

Description: The assessor should verify the existence and implementation of standard procedures for
procuring, receiving, storing and handling of all materials, including reference materials,
required for quality testing. The procedures should ensure a regular and adequate supply
of the required materials and standards. The assessor should verify that all reagents used
in the NCL are of assured quality and labelled accordingly (e.g., preparation date, expiry
date, specifications, storage conditions, and identity of tests in which they are used).

Objective: The objective of this sub- indicator is to ensure the implementation of standard procedures
for procuring and handling of all materials required for testing. These procedures will
ensure consistency of laboratory testing activities.

Requirement: Standard procedures for procuring and handling of materials required for laboratory
testing.

Evidence to review: The assessor should ask for and review:

1. Standard procedures for purchase, receipt, storage, handling, and use of all materials
required for testing;

Page 261 of 340

WHO Global Benchmarking Tool (GBT)

2. Examples of records for the reviewed procedures;

3. List of materials required for different quality tests;
4. List of suppliers of materials required for testing.

Rating Scale:  NOT IMPLEMENTED (NI): There is no documented evidence of procedures for 03
procuring and handling of all materials required for testing.
 ONGOING IMPLEMENTATION (OI): Although there are no approved documents, there 04
is evidence that they are being prepared.
 PARTIALLY IMPLEMENTED (PI): Such documents were approved recently (e.g. less 05
than one year ago).
 IMPLEMENTED (I): Documented procedures for procuring and handling of all 06
materials required for testing are implemented some time ago.
07
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will

Laboratory Testing (LT): Indicators and Fact Sheets

always apply for all benchmarked NRAs)

Sub Indicator: LT06.05: Staff has access to reference documents, including pharmacopoeias,
textbooks and operational manuals.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that the laboratory staff has access to all documentation
required for quality testing including, but not limited to, pharmacopoeias, textbooks,
operational manuals, scientific publications, and publications of the WHO Technical Report
Series. The assessor should also verify that the most updated version of each document is
available.

Objective: The objective of this sub- indicator is to ensure that staff has access to the documentation 08
required for testing. Access to these documents will contribute to effective operations of
the different laboratory activities. 09
Requirement: Staff access to the necessary documentation for performing different laboratory
activities.

Evidence to review: The assessor should ask for and review:

1. Documentation of the information resources available to laboratory staff;
2. Evidence that current and updated versions are available to laboratory staff;
3. List of the scientific publications for which the NCL has subscriptions.;
4. Procedures describing access to documents and location.

References: 1. WHO good practices for pharmaceutical quality control laboratories, (35), (http://
digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en)
2. Guidelines for EU Official Control Authority Batch Release, European Directorate for the
Quality of Medicines and Healthcare (EDQM), Council of Europe, (104), (https://www.edqm.
eu/en/human-ocabr-guidelines)
3. WHO Expert Committee on Biological Standardization (ECBS), Vaccine-specific
standardization, (135), (https://www.who.int/biologicals/vaccines/en/)

Framework: Process

Page 262 of 340

WHO Global Benchmarking Tool (GBT)

Rating Scale:  NOT IMPLEMENTED (NI): There are no documents establishing access to reference
documents, including pharmacopoeias, textbooks and operational manuals.
 ONGOING IMPLEMENTATION (OI): Although there are no approved documents,
there is evidence that they are being prepared.
 PARTIALLY IMPLEMENTED (PI): Such a document was approved recently (e.g.
less than one year ago).
 IMPLEMENTED (I): The NCL has demonstrated access to the essential documents,
and has had access for some time (at least one year), and there has already been
a cycle of internal or external audits (with or without accreditation) or system 01
review.
02
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 03

Indicator: LT07 Mechanism exists to promote transparency, accountability and 04

communication.
05
Objective: The objective of this indicator is to ensure that mechanisms are in place that promote
effective communication of laboratory testing activities and related information within the 06
NRA and NCL, that promote transparency and outreach to the public, and that establish
milestones that encourage accountability of the NRA and NCL to its mandate. Additionally, 07
these activities contribute to mutual understanding and involvement of all stakeholders

Laboratory Testing (LT): Indicators and Fact Sheets

relevant to laboratory testing activities. Consequently, confidence in the regulatory system
is raised.

Category: 08. Transparency, accountability and communication

Sub Indicator: LT07.01: Laboratory testing activities are appropriately communicated to the
public community.

Maturity Level: 4

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that different laboratory testing activities and findings are
appropriately communicated to the public community. This may be done through an
annual report of the NCL or through other regular publications. In addition, the assessor
should confirm that, when needed in the event of public health issues and concerns, 08
immediate and rapid communications to the public community are done to inform them
about the status of the issues of concern. Such communications will help avoid any 09
misunderstandings.

Objective: The objective of this sub- indicator is to ensure transparency to the pubic community.
This transparency will contribute to accountability of the NRA and NCL, promote
consistency in activities undertaken, and increase the public trust and confidence in the
regulatory system.

Requirement: Outreach and communication of laboratory testing activities to the public community.

Evidence to review: The assessor should ask for and review:

1. Documentation of communication and any outreach to the public community regarding
laboratory testing activities and findings.
2. Evidence for public communications from website

Framework: Output

Page 263 of 340

WHO Global Benchmarking Tool (GBT)

Rating Scale:  NOT IMPLEMENTED (NI): Laboratory testing activity reports are not publicly available.
 ONGOING IMPLEMENTATION (OI): The NRA established a procedure or mechanism
by which laboratory testing activities could be publicly available, however, this is not
yet implemented.
 PARTIALLY IMPLEMENTED (PI): There is evidence that the NRA has the elements (i.e.,
procedures, documentation, management systems, and other relevant information)
and the capacity to perform the processes mentioned in the indicator; however, it has
only limited experience or a limited number of documented events.
 IMPLEMENTED (I): Laboratory testing activity reports are regularly and consistently 01
published and publicly available.
02
Limitations and remarks: In some countries, these public communications may be done through the NRA.
Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will 03
always apply for all benchmarked NRAs).
04
Indicator: LT08 Mechanism in place to monitor regulatory performance and output.
05

Objective: The objective of this indicator is to ensure that mechanisms are in place to track regulatory 06
effectiveness of laboratory testing activities, to measure relevant advancements in the
programme, and to establish, implement and regularly verify performance indicators. The 07
NRA or NCL should monitor and analyze laboratory results. These analyses are essential

Laboratory Testing (LT): Indicators and Fact Sheets

to monitor the quality of medical products for each specific product and manufacturer, to
detect trends in non-compliance, and to detect shifts in results towards the specified upper
or lower confidence limits.

Category: 09. Monitoring progress and assessing outcomes & impact

Sub Indicator: LT08.01: There is an updated database of all medical products batches that
have undergone quality testing.

Maturity Level: 4

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that a database (electronic or paper based) of all medical
products batches tested is established and maintained. The database should include
comprehensive information about these medical products batches including name, batch 08
number, manufacturer, tests conducted, test results, and any other relevant information.
09
Objective: The objective of this sub- indicator is to ensure the establishment of a database (electronic
or paper based) of all medical products batches that have undergone quality testing by
the NCL. This database is essential for storing, consolidating and analyzing information
for these batches, including test results and test details (e.g., methods, analysts, reagents,
and equipment). Thus, tracing of relevant regulatory testing activities and decisions for
the sake of institutional memory is ensured. Furthermore, analyses of the data would
contribute to optimization of regulatory performance.

Requirement: An updated database of medical products batches tested for quality by the NCL.

Evidence to review: The assessor should ask for and review:

1. Database (electronic and/or paper based) of medical products batches tested for quality
by the NCL.

Page 264 of 340

WHO Global Benchmarking Tool (GBT)

3. WHO good manufacturing practices for pharmaceutical products: main principles, (51),
(http://digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en)

Framework: Output

Rating Scale:  NOT IMPLEMENTED (NI): There are no updated databases of all medical products
batches that have undergone quality testing.
 ONGOING IMPLEMENTATION (OI): Although there are no databases, there is evidence
that they are being prepared. 01
 PARTIALLY IMPLEMENTED (PI): Such databases were established recently (e.g. less
than one year ago). 02
 IMPLEMENTED (I): The NCL has updated databases, which were established some
time ago (at least one year ago), and there has already been a cycle of internal or 03
external audits (with or without accreditation) or system review.
04
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 05

Sub Indicator: LT08.02: Monitoring and trend analyses are carried out for laboratory testing results 06
data of reference materials and medical products.
07
Maturity Level: 3

Laboratory Testing (LT): Indicators and Fact Sheets

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that quality of all reference materials and working standards
that are used in routine testing are under control through use of control charts, and also
that they are regularly monitored through periodic testing against established references
(e.g. international, regional or national standards).
Test results of the laboratory should be monitored using trend analyses to identify
any trends (positive or negative) or deviations from their mean values. When required,
confidence intervals could be reconsidered. When necessary, new replacement reference
materials may be required when trend analyses reveal a problem with a current reference.
The assessor should check the knowledge and understanding of the laboratory staff of
basic statistical concepts and inference including, for example, the theory of probability
and methods for detection of data trends.

Objective: The objective of this sub- indicator is to ensure reliability and consistency of laboratory 08
testing activities. Performance of reference materials should be monitored, and their
test results should be evaluated using trend analyses. Laboratory test results of medical 09
products should also be monitored using trend analyses.

Requirement: Monitoring and trend analyses of laboratory result data of reference materials and
medical products.

Evidence to review: The assessor should ask for and review:

1. List of reference materials;
2. Documentation of periodic quality testing of selected reference materials;
3. Documentation of trend analyses of selected reference materials;
4. Documentation of trend analyses of medical products laboratory results.

References: General requirements for the competence of testing and calibration laboratories, ISO ISO/
IEC 17025:2015 International Organization for Standardization, 2015, (33), (https://www.
iso.org/standard/66912.html)1

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): There are no monitoring and trend analyses of laboratory
testing data of reference materials and medical products.

Page 265 of 340

WHO Global Benchmarking Tool (GBT)

 ONGOING IMPLEMENTATION (OI): Although there are no monitoring and trend

analyses of laboratory testing data of reference materials and medical products, the
NCL is taking some actions to develop and formalize these procedures.
 PARTIALLY IMPLEMENTED (PI): There are monitoring and trend analyses of
laboratory testing data of reference materials and medical products however these
are recently implemented (e.g., less than two years ago).
 IMPLEMENTED (I): There are monitoring and trend analyses of laboratory
testing data of reference materials and medical products, and these have been
consistently implemented over a relatively long period of time (e.g. more than 01
two years).
02
Limitations and remarks:
03
Sub Indicator: LT08.03: Regular participation in proficiency schemes, collaborative studies
and inter-laboratory comparisons. 04

Maturity Level: 4 05

Scope: 1. Medicines 06
2. Vaccines
07
Description: The assessor should verify that the technical competency of the NCL is periodically

Laboratory Testing (LT): Indicators and Fact Sheets

monitored to confirm consistency and reliability of test results. The monitoring may be
conducted in various ways, including: performance evaluation systems, collaborative studies,
and inter-laboratory comparisons. The assessor should verify that the NCL participates in
proficiency schemes or collaborative studies organized by WHO, the European Directorate for
the Quality of Medicines and HealthCare, the National Institute for Biological Standards and
Control, or other institutions. Participation allows the NCL to compare its own performance
to an international benchmark. The assessor should check dates of participation, scope
of study, the product tested, and coordinating institution. In addition, the assessor should
ascertain the number of collaborative studies in which laboratory participated, the specific
tests evaluated, and whether or not the NCL had appropriate performance.

Objective: The objective of this sub- indicator is to ensure that NCL participation in proficiency
schemes, collaborative studies and inter-laboratory comparisons contributes to regulatory
harmonization and networking. These, in turn, lead to improvements in the effectiveness
and efficiency of the regulatory function.

Requirement: Participation in proficiency schemes, collaborative studies and inter-laboratory comparisons. 08

Evidence to review: The assessor should ask for and review: 09
1. Records of participation in proficiency schemes, collaborative studies and inter-
laboratory comparisons.
2. Records of performance during participation in proficiency schemes, collaborative
studies and inter- laboratory comparisons.

Framework: Output

Rating Scale:  NOT IMPLEMENTED (NI): There are no documents establishing regular participation
in international proficiency schemes, collaborative studies and inter-laboratory
comparisons.

Page 266 of 340

WHO Global Benchmarking Tool (GBT)

 ONGOING IMPLEMENTATION (OI): At least some monitoring has been done with
results that warrant actions for improvement.
 PARTIALLY IMPLEMENTED (PI): At least some monitoring has been done with
acceptable results. Or, if there are decentralized activities, most of the laboratories
involved have demonstrated acceptable results in at least one evaluation.
 IMPLEMENTED (I): Laboratory has participated in several studies to monitor
performance. For these studies, acceptable results were obtained, demonstrating the
technical competency of the NCL (including all laboratories when the activities are
decentralized). 01
Limitations and remarks:  When results from participation in proficiency studies are not satisfactory the NCL 02
should show evidence of corrective actions and preventive actions.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will 03
always apply for all benchmarked NRAs).
04
Sub Indicator: LT08.04: Performance indicators for laboratory testing activities are established
and implemented. 05

Maturity Level: 4 06

Scope: 1. Medicines 07
2. Vaccines

Laboratory Testing (LT): Indicators and Fact Sheets

Description: The assessor should verify the existence and implementation of performance indicators
for different activities included under the laboratory testing functions.
Specifically, the system should define key performance indicators (KPIs) along the entire
laboratory testing activity chain and all indicators should be adequately justified. For the
purpose of clarity and consistency, established KPIs should be supported with guidelines
for monitoring and maintenance of the KPIs. The guidelines in turn should be supported
by SOPs and tools that define the procedures to be used for monitoring and evaluating the
performance indicators and that define procedures and timelines for reviewing and revising
the indicators.
Examples of performance indicators for activities included under the laboratory access
and testing function include, but are not limited to: number of product batches tested,
number of tests performed, number of non-compliant events, mean time for performing
different quality tests, and other relevant parameters.
Established KPIs might be qualitative, quantitative or combination of both. In general,
quantitative indicators are preferred to avoid bias or misinterpretation. However, qualitative
indicators are also accepted. Qualitative indicators may or may not include scoring 08
or scaling to render them semi-quantitative and thus more informative. The assessor
should ensure that indicators are measured on a regular basis to monitor progress and 09
advancement. ¨
In addition, the assessor should verify measured indicators are analyzed to identify
trends or abnormalities. Justifications for any identified abnormalities should be provided;
when necessary, process optimizations should be introduced to avoid recurrence.

Objective: The objective of this sub-indicator is to ensure that a system, mechanism, or

procedure exists to require the NRA to establish performance indicators along the entire
laboratory testing chain. Additionally, the objective is to ensure that KPIs are actually
contributing to monitoring of regulatory performance, to measuring effectiveness of
laboratory testing regulatory activities, and to making any necessary adjustments
or optimizations.

Requirement: KPIs for laboratory testing activities

Evidence to review: The assessor should ask for and review:

1. Documents supporting the system, mechanism, or procedure compelling the NRA to
establish and implement performance indicators along the entire laboratory testing activity
chain.
2. Evidence that the performance indicators have been established and implemented,

Page 267 of 340

WHO Global Benchmarking Tool (GBT)

and that the members of staff involved in the laboratory testing function are aware of
the indicators and the guidelines and SOPs used for monitoring and evaluating their
performance.
3. The current performance indicators for laboratory testing activities
4. Analyses of the measured indicators along with the investigations done to identify
trends or abnormalities.
5. Documentation for follow-up of any observed abnormalities, including justifications
for any identified abnormalities as well as any process optimizations introduced to avoid
recurrence. 01
References: 1. Quality management -- Quality of an organization -- Guidance to achieve sustained 02
success, ISO 9004:2018, International Organization for Standardization (ISO), (115),
(https://www.iso.org/standard/70397.html) 03

Framework: Output 04

Rating Scale:  NOT IMPLEMENTED (NI): There are no KPIs for laboratory testing activities. 05
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted KPIs for laboratory
testing activities but they have not yet been reported. 06
 PARTIALLY IMPLEMENTED (PI): The NRA has developed KPIs for laboratory testing
activities and has been applying them for less than two year or they have not covered 07
all critical steps.

Laboratory Testing (LT): Indicators and Fact Sheets

 IMPLEMENTED (I): The NRA has established and implemented KPIs for laboratory
testing activities. The indicators are reviewed regularly, and appropriate actions are
taken, and decisions made.

Limitations and remarks:  When they refer to outcomes, indicators may be ambiguous and difficult to interpret,
as outcomes are the result of many factors that are difficult to disentangle. When
they refer to processes, indicators are often too specific, as they may focus on a
particular intervention or condition or they may quickly become outdated as business
models develop.
Different methodologies are used to measure the NRAs performance on laboratory
testing activities. In this case, the assessor should verify that adequate supporting
documents are available. The assessor should consider that developed performance
indicators should be Specific, Measurable, Achievable, Realistic, and Time- bound (i.e.,
“SMART”).
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator
will always apply for all benchmarked NRAs).
08
Indicator: LT09 Measures for occupational health and safety.
09
Objective: The objective of this indicator is to ensure that safety issues to protect staff and the
environment are considered of critical importance. Laboratory work is linked to specific
risks because of the use of chemical, radiological and biological materials.

Category: 03. Policy and strategic planning

Sub Indicator: LT09.01: A laboratory hazardous substances list exists and documented procedures
for storage, handling and disposal of these substances are implemented.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that lists of hazardous substances used in the laboratories
are available. In addition, the assessor should determine if standard procedures for
storing, labelling, handling and disposal of hazardous substances are available and
implemented.

Page 268 of 340

WHO Global Benchmarking Tool (GBT)

Objective: The objective of this sub- indicator is to ensure that hazardous laboratory substances are
carefully handled and adversely affect neither the health of the laboratory staff nor the
environment.

Requirement: List of hazardous substances and standard procedures for handling them.

Evidence to review: The assessor should ask for and review:

1. List of hazardous substances used in the laboratory;
2. Standard procedures for the storage, handling and disposal for hazardous substances 01
used in the laboratory;
3. Standard procedures for waste management and final disposal according national 02
regulations;
4. Examples of records for each of the reviewed procedures. 03

References: 1. WHO good practices for pharmaceutical quality control laboratories, (35), (http:// 04
digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en)
2. WHO good manufacturing practices for pharmaceutical products: main principles, (51), 05
(http://digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en)
3. Environmental management systems -- Requirements with guidance for use, ISO 06
14001:2015, International Organization for Standardization (ISO), (108), (https://www.iso.
org/standard/60857.html) 07

Laboratory Testing (LT): Indicators and Fact Sheets

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): There is no list of hazardous substances used in the
laboratory and no documented procedure for storage, handling and disposal of these
substances.
 ONGOING IMPLEMENTATION (OI): Although there are no approved documents, there
is evidence that they are being prepared.
 PARTIALLY IMPLEMENTED (PI): Such documents were approved recently (e.g. less
than one year ago).
 IMPLEMENTED (I): The NCL has a list of laboratory hazardous substances used
in the laboratory and documented procedures for storage, handling and disposal
of these substances. These have been in place for at least one year, and there has
already been a cycle of internal or external audits (with or without accreditation) or
system review.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 08
Sub Indicator: LT09.02: A laboratory safety programme exists and a designated person 09
is responsible for its management.

Maturity Level: 4

Scope: 1. Medicines
2. Vaccines

Description: The assessor should confirm that a laboratory safety programme is established and that
a staff member (or a group of staff members) is responsible for managing all aspects
within that safety programme. The programme should provide comprehensive guidance
with general and specific instructions on occupational safety in the laboratory. Specific
attention
should be given to protection from microbiological materials such as live bacteria and
viruses. In addition, the assessor should confirm that the laboratory safety programme is
well communicated to all laboratory staff.

Objective: The objective of this sub- indicator is to ensure that a laboratory safety programme exists.
Hazardous laboratory activities should adversely affect neither the environment nor the
health of the laboratory staff.

Page 269 of 340

WHO Global Benchmarking Tool (GBT)

Requirement: Establishment of a laboratory safety programme and designation of qualified staff for its
management

Evidence to review: The assessor should ask for and review:

1. Policies and procedures for the laboratory safety programme;
2. Evidence that staff has been designated to manage the laboratory safety programme;
3. Examples of documented communications of the laboratory safety programme to all
laboratory staff;
4. Examples of laboratory staff training records covering the handling of hazardous 01
substances and spills, and the protection of staff from exposure to chemical
substances. 02

References: 03

Framework: Structure/Foundation/Input 04

Rating Scale:  NOT IMPLEMENTED (NI): There is no laboratory safety programme. 05

 ONGOING IMPLEMENTATION (OI): Although there are no approved documents, there
is evidence that they are being prepared. 06
 PARTIALLY IMPLEMENTED (PI): Such documents were approved recently (e.g. less
than one year ago). 07
 IMPLEMENTED (I): The NCL has a laboratory safety programme, which was

Laboratory Testing (LT): Indicators and Fact Sheets

approved and implemented some time ago (at least one year ago), and there has
already been a cycle of internal or external audits (with or without accreditation) or
system review.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: LT09.03: Staff immunization requirements are defined, implemented

and monitored.

Maturity Level: 4

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that immunization programs for laboratory staff are defined,
implemented and monitored. Staff compliance with the predefined immunization 08
requirements should be verified prior to permitting access to designated work areas or use
of designated equipment. 09
Objective: The objective of this sub- indicator is to ensure that requirements for staff immunization
are defined, implemented and monitored.

Requirement: Laboratory staff immunization programme is established, implemented and monitored.

Evidence to review: The assessor should ask for and review:

1. Immunization requirements, if any, for staff from different laboratory work areas
2. Examples of health records for selected staff;
3. Documentation of regular monitoring of staff immunization programme;
4. Examples of antibody titer measurement records for selected staff.

Framework: Structure/Foundation/Input

Page 270 of 340

WHO Global Benchmarking Tool (GBT)

Rating Scale:  NOT IMPLEMENTED (NI): There are no defined staff immunization requirements.
 ONGOING IMPLEMENTATION (OI): Although there are no defined requirements, there
is evidence that they are being prepared.
 PARTIALLY IMPLEMENTED (PI): Such requirements were approved recently (e.g. less
than one year ago).
 IMPLEMENTED (I): The NCL has defined staff immunization requirements, and there
is documented evidence for implementation and monitoring. The program has been
in place for some time ago (at least one year), and there has already been a cycle of
internal or external audits (with or without accreditation) or system review. 01
Limitations and remarks: 02

Indicator: LT10 Measures for good management of outsourced laboratory activities. 03

04
Objective: The objective of this indicator is to ensure that an NRA or NCL that is outsourcing
laboratory testing activities has the following processes in place: an established, formal 05
evaluation process for identifying qualified laboratories; a decision-making process
for issuing, renewing or rescinding contracts for laboratory services; established and 06
implemented procedures for managing outsourced activities; and procedures for handling
communication exchanges. 07

Laboratory Testing (LT): Indicators and Fact Sheets

Category: 07. Regulatory process

Sub Indicator: LT10.01: Documented procedures are implemented for managing outsourced
QC activities.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the existence and implementation of standard procedures for
managing all outsourced testing activities.
The assessor should check these procedures and confirm that they are providing
comprehensive guidance on implementation and management of quality agreements
between the NCL (i.e., contract giver) and the independent laboratory or organization (i.e.,
contract acceptor). In addition, the procedures should provide guidance on the selection of
contract acceptor, on the pathways of communication with clear roles and responsibilities 08
(e.g., sourcing of materials required for testing including reference materials) and on a
programme for regular audits (internal or external) of the contract acceptor by the NCL. 09
The assessor should verify that the information exchange is handled in a confidential
manner and that a system is in place to ensure there are no conflicts of interest.

Objective: The objective of this sub- indicator is to ensure that outsourcing and sub-contracting is
well controlled by the responsible NRA in order to ensure reliability of the outsourced test
results.

Requirement: Standard procedures for managing outsourced testing activities.

Evidence to review: The assessor should ask for and review:

1. Standard procedures for managing outsourced testing activities;
2. Documentation of quality audits (internal or external) and other evidence that
demonstrates that defined criteria have been met before accepting the sub-contracting
laboratory;
3. Quality agreements between the NCL and the independent laboratory or organization;
4. Evidence from quality audits (internal or external) that confirm the sub-contracting
laboratory has a QMS in place;
5. Documents that demonstrate that confidentiality agreements and declarations of
Interests have been signed;

Page 271 of 340

WHO Global Benchmarking Tool (GBT)

6. Examples of records of communication (including notifications) between

both parties.

References: 1. WHO good practices for pharmaceutical quality control laboratories, (35), (http://
digicollection.org/whoqapharm/p/about and http://apps.who.int/medicinedocs/en)
2. General requirements for the competence of testing and calibration laboratories, ISO
ISO/IEC 17025:2015 International Organization for Standardization, 2015, (33), (https://
www.iso.org/standard/66912.html)
01
Framework: Process
02
Rating Scale:  NOT IMPLEMENTED (NI): There are no documented procedures implemented for
managing outsourced QC activities. 03
 ONGOING IMPLEMENTATION (OI): Although there are no approved documents, there
is evidence that they are being prepared. 04
 PARTIALLY IMPLEMENTED (PI): Such documents were approved recently (e.g. less
than one year ago). 05
 IMPLEMENTED (I): The NCL has implemented documented procedures for managing
outsourced QC activities. These procedures were approved some time ago (at least 06
one year ago), and there has already been a cycle of internal or external audits (with or
without accreditation) or system review. 07

Laboratory Testing (LT): Indicators and Fact Sheets

Limitations and remarks:

08
09

Page 272 of 340

WHO Global Benchmarking Tool (GBT)

08
Clinical Trials Oversight
02
03
04
05
01

(CT): Indicators and 06

Fact Sheets 07

Laboratory Testing (LT): Indicators and Fact Sheets

WHO Global
Benchmarking
Tool (GBT)
for Evaluation of
National Regulatory
System of Medical
Products
08
09

Page 273 of 340

WHO Global Benchmarking Tool (GBT)

08. Clinical Trials Oversight (CT):

Indicators and Fact Sheets

Function:
08 — CLINICAL TRIALS OVERSIGHT (CT)
01
Description: National Regulatory Authorities (NRAs) should have the legal mandate to authorize 02
regulate and, if necessary, terminate clinical trials (CTs). The necessary requirements,
guidelines, procedures and forms should be developed to be in line with country 03
and region-specific guidelines as well as major international CT guidance including
guidelines from the Declaration of Helsinki, the Nuremberg code, International 04
Council on Harmonization, and World Health Organization Good Clinical Practices. CT
oversight is aimed at protecting the safety and rights of humans participating in CTs, 05
ensuring that trials are adequately designed to meet scientifically sound objectives,
and preventing any potential fraud and falsification of data. 06
NRAs are responsible at two stages for the critical evaluation of the documentation 07
supporting clinical studies: when CTs are being proposed for authorization and
when the results are submitted in an application for marketing authorization. CT 08
protocols should be reviewed and approved by Independent Ethics Committees

Clinical Trials Oversight (CT): Indicators and Fact Sheets

before the trial commences. A CT review committee should review the protocols
and should have the authority, when necessary, to require protocol revisions. The CT
review committee should be composed of members who have the appropriate
medical and scientific knowledge, experience and skills and who are free of conflicts
of interest.

In order to ensure the quality and safety of investigational products, the investigational
products should be manufactured in compliance with Good Manufacturing Practices
for investigational medical products, and the supporting preclinical studies should
be in compliance with Good Laboratory Practices. Additionally, the importation,
storage, use, and/or destruction of investigational products should follow national
requirements. Qualified and experienced inspectors should carry out on-site
inspections of the CT sites to verify compliance with Good Clinical Practices, ethical
principles and regulatory requirements, and to provide assurance of the quality and
reliability of the data obtained. The oversight activities should be conducted with due
concern for confidentiality.

The legal provisions should allow the NRA to recognize and/or rely on relevant CT
decisions, reports and information from other NRAs or from designated regional and 09
international bodies. In special circumstances (e.g., for public health interest), the
legal provisions should allow the NRA to elect not to follow the routine CT procedures.
Transparency in the entire oversight process is fundamental to ensuring the safety of
patients and to ensuring that no product with unacceptable benefit to risk balance will
be made available to the public.

Indicator: CT01 Legal provisions, regulations and guidelines required to define regulatory
framework of clinical trials oversight.

Objective: The objective of this indicator is to ensure that the NRA has the legal mandate to both
authorize and suspend the implementation of CTs. Among other benefits, the mandate
will ensure that regulations and guidelines are in place to protect the safety and rights of
the subjects participating in a trial and to ensure that trials are adequately designed to
meet scientifically sound objectives.

Category: Legal provisions, regulations and guidelines

Page 274 of 340

WHO Global Benchmarking Tool (GBT)

Sub Indicator: CT01.01: Legal provisions and regulations for clinical trials (CTs) oversight exist.

Maturity Level: 1

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that the legal provisions and regulations for CTs oversight exist 01
and are enacted and implemented. The legal provisions should clearly specify the entities
with the mandate for CT oversight and the extent of the mandate for each entity. When 02
more than one entity is involved, the provisions should specify the responsibilities shared
among the entities. 03
The assessor should verify that legal provisions and regulations that require National
Regulatory Authority (NRA) authorization prior to initiation of clinical studies exist and are 04
enacted and implemented. The assessor should check a representative sample of provided
documentation to verify that acceptable procedures were followed. Examples include: 05
1. Review the mandate for on-site inspections that ensure the location is acceptable and
that product quality is maintained during storage; 06
2. Review informed consent forms and investigator brochure to verify that information and
documentation provided are following Good Clinical Practices (GCPs). 07
The assessor should be guided by the existing law and regulation before applying the
scoring.
08

Clinical Trials Oversight (CT): Indicators and Fact Sheets

Objective: The objective of this sub-indicator is to ensure the existence of legal provisions and
regulations that require that CTs be authorized by the NRA prior to initiation. The NRA
should review the protocol and other relevant documentation to be sure safety of
participants is considered and that all required aspects are conducted according to GCPs.

Requirement: Legislation on CT oversight

Evidence to review: The assessor should ask for and review:

1. Legal provisions and regulations that grant the NRA the legal mandate for CT oversight.
2. The sections in the law that define the extent and scope of the CT oversight mandate
allocated to the NRA and other entities involved in CT-related activities.
3. Relevant sections of the law on CTs that stipulate that NRA authorization is required
prior to initiating and conducting a clinical study.
4. The guidelines that define the format and content of protocol, the procedure for
submission, and the timeframe for review of application.

References: 1. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization: forty-fifth 09
report. World Health Organization; 1995: Annex 1 (WHO Technical Report Series, No. 858),
(3), (http://www.who.int/bloodproducts/publications/WHO_TRS_858_A1.pdf)
2. Guidelines for good clinical practice (GCP) for trials on pharmaceutical products, (40),
(http://apps.who.int/medicinedocs/en)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There are no legal provisions or regulations for CTs
oversight.
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted and adopted legal
provisions for CTs oversight but they have not yet been followed.
 PARTIALLY IMPLEMENTED (PI): The NRA has legal provisions for conducting CTs
oversight and has been applying it for less than two years.
 IMPLEMENTED (I): The NRA has legal provisions for conducting CTs oversight and
has adequate documented evidence to support this.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Page 275 of 340

WHO Global Benchmarking Tool (GBT)

Sub Indicator: CT01.02: Legal provisions and regulations that stipulates that notification to the
NRAand authorization from the NRA is required for any changes or variations
(i.e., amendments) in the original protocol or in any relevant documents of the CT.

Maturity Level: 2

Scope: 1. Medicines
2. Vaccines
01
Description: The assessor should verify the existence, enactment, and implementation of legal
provisions and regulations related to changes or variations to the original (approved) 02
CT protocols. These provisions should state clearly that the NRA should be notified of
any changes or variations to the original protocol, and that authorization from the NRA 03
is required before the changes or variations (i.e., amendments) are implemented. The
assessor should note that some protocol amendments may not require authorization 04
from the NRA prior to implementation. Some changes may require a notification prior to
or even after implementation, while others require NRA approval before implementation. 05
The assessor should be guided by the existing laws and regulations before applying the
scoring. 06

Objective: The objective of this sub-indicator is to ensure that legal provisions and regulations require 07
that the NRA should be notified of changes or variations (i.e., amendments) to the original
protocol, and should authorize changes and variations to the original protocol before they
08
are implemented.

Clinical Trials Oversight (CT): Indicators and Fact Sheets

Requirement: Changes and variations to already approved (original) CT protocols

Evidence to review: The assessor should ask for and review:

1. Relevant sections of the legal provisions (laws, decrees, regulations or any legal binding
documents) on CTs with emphasis on the requirement for notification and authorization
prior to implementation of changes or variations to original protocol and related documents.
2. The guidelines that specify the format and content of submissions related to changes or
variations to original protocol, the procedure for submission, and the timeframe for review.

References: 1. WHO guidelines on nonclinical evaluation of vaccines, (39), (https://www.who.int/

biologicals/WHO_ECBS/en/ and https://www.who.int/biologicals/vaccines/en/)
2. Guidelines for good clinical practice (GCP) for trials on pharmaceutical products, (40),
(http://apps.who.int/medicinedocs/en)
3. Guidelines on clinical evaluation of vaccines: regulatory expectations, (38), (https://www.
who.int/biologicals/WHO_ECBS/en/ and https://www.who.int/biologicals/vaccines/en/)
4. WHO guidelines on the nonclinical evaluation of vaccines adjuvants and adjuvanted
vaccines, (119), (https://www.who.int/biologicals/WHO_ECBS/en/ and https://www.who. 09
int/biologicals/vaccines/en/)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There are no legal provisions or regulations requiring
notification to and authorization from the NRA when there are changes or variations
(i.e., amendments) in the original protocol of the CT.
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted or adopted this legal
provision but they have not yet been followed.
 PARTIALLY IMPLEMENTED (PI): The NRA has legal provision and has been applying it
for less than two years.
 IMPLEMENTED (I): The NRA has legal provisions and regulations requiring
notification to the NRA and authorization from NRA on changes or variations (i.e.,
amendments) in the original protocol of the CT; these amendments should be
supported with documented evidence.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Page 276 of 340

WHO Global Benchmarking Tool (GBT)

Sub Indicator: CT01.03: Legal provisions and regulations requiring research centers, researchers,
sponsors, clinical research organizations (CROs) and all relevant institutions in the
CT to comply with GCP

Maturity Level: 2

Scope: 1. Medicines
2. Vaccines
01
Description: The assessor should verify the existence, enactment, and implementation of legal provisions
and regulations that require stakeholders, including research centers, researchers, sponsors, 02
CROs and everyone involved in CTs to comply with the principles of GCP. GCP is an
international ethical and scientific quality standard for designing, conducting, recording and 03
reporting trials that involve the participation of human participants. Compliance with this
standard provides assurance that the rights, 04
safety and well-being of trial subjects are protected consistent with the principles established
in the Declaration of Helsinki, and that CT data are credible. The legal provisions should be 05
supported by detailed and published regulatory requirements for GCP. The published GCP
principles should be current and up to date with information on inspections and on suspension 06
or stoppage of trials. When GCP is referenced or adopted, documentation should be available
to demonstrate compliance. The assessor should 07
verify that Ethics Committees (ECs) and the NRA consider GCP principles when
conducting evaluations.
08

Clinical Trials Oversight (CT): Indicators and Fact Sheets

Objective: The objective of this sub-indicator is to ensure the existence of legal provisions and regulations
that mandates stakeholders, including research centers, researchers, sponsors, CROs and
everyone involved in CTs to comply with GCP. These provisions will ensure all stakeholders,
including research centers, researchers, sponsors, CROs and everyone involved CTs operate
within a set of globally-accepted standards for the conduct of biomedical research on human
participants.

Requirement: Stakeholder acting in compliance with GCP

Evidence to review: The assessor should ask for and review:

1. The legal provisions (laws, decrees, regulations or any legal binding document) that require
stakeholders involved in CT to comply with GCP principles.
2. The current GCP requirements for all CTs. The GCP standards should be published
and easily accessible.
3. Guidelines or similar documents (e.g., checklists and standard operating procedures (SOPs))
used to provide guidance in the application of the legal provisions and
regulations.
09
References: 1. Guidelines for good clinical practice (GCP) for trials on pharmaceutical products, (40),
(http://apps.who.int/medicinedocs/en)
2. Regulation and licensing of biological products in countries with newly developing
regulatory authorities. In: WHO Expert Committee on Biological Standardization: forty-fifth
report. World Health Organization; 1995: Annex 1 (WHO Technical Report Series, No. 858), (3),
(http://www.who.int/bloodproducts/publications/WHO_TRS_858_A1.pdf)
3. Guidelines on clinical evaluation of vaccines: regulatory expectations, (38), (https://www.
who.int/biologicals/WHO_ECBS/en/ and https://www.who.int/biologicals/vaccines/en/)
4. WHO guidelines on the nonclinical evaluation of vaccines adjuvants and adjuvanted
vaccines, (119), (https://www.who.int/biologicals/WHO_ECBS/en/ and https://www.who.int/
biologicals/vaccines/en/)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There are no legal provisions or regulations requiring research
centers, researchers, sponsors, CROs and everyone involved in the CTs to comply with
GCP
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted or adopted these legal

Page 277 of 340

WHO Global Benchmarking Tool (GBT)

provisions and regulations but they have not yet been followed.
 PARTIALLY IMPLEMENTED (PI): The NRA has developed this legal provision and has
been applying it for less than two years, or not all involved organizations have fully
followed this regulation.
 IMPLEMENTED (I): The NRA has legal provisions and regulations requiring research
centers, researchers, sponsors, CROs and everyone involved in CTs to comply with
GCP; compliance is supported with documented evidence.

Limitations and remarks: The assessor should consider that existing written documents are legally binding. 01
Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator
will always apply for all benchmarked NRAs). 02

Sub Indicator: CT01.04: Legal provisions, regulations and guidelines requiring that investigational 03
medical products (IMPs) comply with good manufacturing practices (GMP) for IMPs.
04
Maturity Level: 3
05
Scope: 1. Medicines
2. Vaccines 06

Description: The assessor should verify the existence, enactment, and implementation of legal 07
provisions, regulations and guidelines that require that IMPs comply with the GMPs for
IMPs. The legal provisions should clearly state that only IMPs produced in compliance with
08
current GMP standards for IMPs will be used in CTs, and that the applicant or sponsor of

Clinical Trials Oversight (CT): Indicators and Fact Sheets

the CT is responsible for supplying IMPs produced in accordance with GMP principles.
The legal provisions should be supported by regulations or guidelines that clearly present
the applicable GMP requirements and the type of supporting evidence to present (e.g.,
certificates and summary product release documents). The assessor should note that
some NRAs may not require a GMP certificate if the IMP is already licensed or registered
in the country of origin. However, according to good regulatory practices, a current GMP
certificate from the NRA of the country of origin should be required when the IMP has a
marketing authorization in the country of origin or has a marketing authorization, but the
original indication is modified for the purpose of the trial.

Objective: The objective of this sub-indicator is ensure the existence of legal provisions, regulations
and guidelines that mandate that IMPs used in CTs comply with GMP requirements.

Requirement: IMPs GMP requirements

Evidence to review: The assessor should ask for and review:

1. The legal provisions (laws, decrees, regulations or any legal binding document) and
guidelines requiring that IMPs used in CTs are produced in compliance with the principles 09
of GMPs for IMPs.
2. The legal provisions, regulations and guidelines that state that it is the responsibility of
the sponsors or applicants to supply IMPs produced in compliance with the principles of
GMP for IMPs.
3. Documentation specifying the format for the IMP’s GMP certificate that is required
prior to CT authorization. The GMP certificate should be issued by the national competent
authority of the country of origin of the IMP.

References: 1. WHO good manufacturing practices for pharmaceutical products: main principles,
(51), (http://digicollection.org/whoqapharm/p/about and http://apps.who.int/
medicinedocs/en)
2. Guidelines for good clinical practice (GCP) for trials on pharmaceutical products, (40),
(http://apps.who.int/medicinedocs/en)
3. Good manufacturing practice: supplementary guidelines for the manufacture of
investigational pharmaceutical products for clinical trials in humans, (120), (http://apps.
who.int/medicinedocs/en)

Framework: Structure/Foundation/Input

Page 278 of 340

WHO Global Benchmarking Tool (GBT)

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence of existence of legal provisions,
regulations or guidelines requiring that IMPs comply with GMPs for IMPs.
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted or adopted these
legal provisions, regulations and guidelines but they have not yet been implemented.
 PARTIALLY IMPLEMENTED (PI): The NRA has legal provisions, regulations and
guidelines and has been applying them for less than two years.
 IMPLEMENTED (I): The NRA has legal provisions, regulations and guidelines requiring
that IMPs comply with GMPs for IMPs.
01
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 02

Sub Indicator: CT01.05: There are legal provisions or regulations covering circumstances in which the 03
routine CT evaluation procedures may not be followed (e.g. for public-health interests).
04
Maturity Level: 2
05
Scope: 1. Medicines
2. Vaccines 06

Description: The assessor should verify the existence, enactment, and implementation of legal 07
provisions or regulations that allow the NRA to apply non-routine CT procedures such
as fast-track or expedited decision-making (e.g., receipt, screening, evaluation, review,
08
and authorization) of a CT application under certain circumstances (e.g., public health

Clinical Trials Oversight (CT): Indicators and Fact Sheets

emergencies). The legal provisions and regulations should provide clear directives on the
circumstances under which the non-routine CT authorization should apply. The provisions
should provide guidance on the relationship between the non-routine CT authorization
and the mandate of the Independent Ethics Committee (IEC). The legal provisions should
be supported with guidelines that describe the content (e.g., the critical requirements)
and format of CT applications submitted for approval by the non- routine authorization
procedure. This guidance should also describe the scope of the review and evaluation
process (i.e., screening, verification, and other relevant activities). The supporting
guidelines may provide clarity on whether a CT application rejected through the non-routine
procedure could be re-submitted through the routine procedure. In addition, there should
be a guidance document to provide direction to the NRA on how to justify application of
the non-routine procedure to the CT.
The assessor should verify that the proposed procedures meet targeted timeframes for
public health emergencies or any other unmet medical needs.

Objective: The objective of this sub-indicator is to ensure legal provisions and regulations are
established to allow the NRA or responsible regulatory authority to apply non-routine
CT procedures such as fast-track or expedited receipt, evaluation, and authorization 09
of a CT application under certain circumstances (e.g. public health emergencies). This
sub- indicator will establish whether there is an abridged route other than the normal CT
application route for CT authorization.

Requirement: Exemptions from routine CT application evaluation procedures

Evidence to review: The assessor should ask for and review:

1. Guidelines or similar document that provides the list of situations in which the routine
CT procedures may not be required.
2. Legal provisions (laws, decrees, regulations or any legal binding document) that cover
circumstances or instances under which the non-routine CT procedures such fast-track CT
authorization may be applied. Assessor should review evidence that the legal provisions
are published and implemented.
3. The supporting guidelines and regulations.
4. Guidelines describing the content and format of CT applications requesting application
of non-routine CT procedures such as fast-track.
5. Guidelines specifying the scope of the evaluation process (i.e., screening, verification, or
other relevant activities).

Page 279 of 340

WHO Global Benchmarking Tool (GBT)

References: 1. Guidelines for good clinical practice (GCP) for trials on pharmaceutical products, (40),
(http://apps.who.int/medicinedocs/en)
2. Emergency Use Assessment and Listing Procedure (EUAL) for Candidate Medicines for use
in the Context of a Public Health Emergency, (121), (http://apps.who.int/medicinedocs/en/)
3. Emergency Use Assessment and Listing Procedure (EUAL) for Candidate Vaccines for
use in the Context of a Public Health Emergency, (122), (http://apps.who.int/medicinedocs/
en/)

Framework: Structure/Foundation/Input 01
Rating Scale:  NOT IMPLEMENTED (NI): There are no legal provisions or regulations to cover 02
circumstances under which the NRA may elect to follow the non-routine CT
application process 03
 ONGOING IMPLEMENTATION (OI): The NRA has drafted or adopted these legal
provisions or regulations, but they have not yet been followed. 04
 PARTIALLY IMPLEMENTED (PI): The NRA has legal provision or regulation and has
been applying it for less than two years. 05
 IMPLEMENTED (I): The NRA has legal provisions or regulations to cover
circumstances in which the NRA may elect to follow the non-routine CT application 06
process (e.g., for public health interest).
07
Limitations and remarks: The assessor should consider any kind of legal document that address this requirement.
08
Sub Indicator: CT01.06: Legal provisions, regulations or guidelines exist for NRA to inspect,

Clinical Trials Oversight (CT): Indicators and Fact Sheets

suspend or stop CTs.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the existence, enactment, and implementation of legal
provisions or regulations that give the NRA enforcement powers to inspect, suspend or
stop CTs. The legal provisions should be supported with clear regulations or guidance on
when (i.e., routine, random or for specific reasons) and how to inspect, suspend or stop a
CT. Clear guidance on what to inspect (e.g., evidence supporting quality and reliability of
data and reported results or records for subjects, equipment, protocols, and environment)
should also be considered.

Objective: The objective of this sub-indicator is to ensure legal provisions or regulations are
established to give the NRA enforcement powers to inspect, suspend or stop CTs.
09
Requirement: Legal provisions, regulations or guidelines to inspect, suspend or stop CTs

Evidence to review: The assessor should ask for and review:

1. The legal provisions (laws, decrees, regulations or any legal binding document) that give
the NRA the enforcement power to inspect, suspend or stop CTs.
2. Documents providing guidance on procedures to follow for inspecting, suspending or
stopping CTs.
3. List of CTs inspected, suspended or stopped.

References: 1. Guidelines on clinical evaluation of vaccines: regulatory expectations, (38), (https://www.

who.int/biologicals/WHO_ECBS/en/ and https://www.who.int/biologicals/vaccines/en/)
2. WHO guidelines on nonclinical evaluation of vaccines, (39), (https://www.who.int/
biologicals/WHO_ECBS/en/ and https://www.who.int/biologicals/vaccines/en/)
3. Guidelines for good clinical practice (GCP) for trials on pharmaceutical products, (40),
(http://apps.who.int/medicinedocs/en)
4. WHO guidelines on the nonclinical evaluation of vaccines adjuvants and adjuvanted
vaccines, (119), (https://www.who.int/biologicals/WHO_ECBS/en/ and https://www.who.
int/biologicals/vaccines/en/)

Page 280 of 340

WHO Global Benchmarking Tool (GBT)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There are no legal provisions, regulations or guidelines
for NRA to inspect, suspend or stop CT
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted or developed the
regulations or guidelines, but these have not yet been followed.
 PARTIALLY IMPLEMENTED (PI): The NRA has legal provisions, regulations or
guidelines and has been applying them for less than two years or the guidelines
or regulations are not fully implemented for all CT applications. 01
 IMPLEMENTED (I): The NRA has legal provisions, regulations or guidelines to
inspect, suspend or stop CTs and these are implemented and followed for all 02
approved CT applications.
03
Limitations and remarks:  If there is no CT ongoing, the assessor may score this sub-indicator as
implemented by reviewing guidelines or other required documentation. 04
Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-
05


indicator will always apply for all benchmarked NRAs).

Sub Indicator: CT01.07: There are legal provisions or regulations that require the establishment 06
of an IEC.
07
Maturity Level: 2
08
Scope: 1. Medicines

Clinical Trials Oversight (CT): Indicators and Fact Sheets

2. Vaccines

Description: The assessor should verify the existence, issuance, and implementation of regulations
that require the establishment of an IEC. The regulations should be clear on the specific
authority that should host the IEC and that should assist it to successfully discharge
its duties and mandates. The legal provisions should provide guidance on what it
means to be “independent”, and also provide supporting regulations and guidelines that
ensure that the independence of the IEC is sustained. The objectives, functions and
composition of the IEC should be clearly defined and documented, and a general policy
on potential conflicts of interest for IEC members should be provided. The assessor
should note that there may be more than one legally established IEC in a country. The
assessor should therefore verify the legal mandate of the IEC in question and the scope
of its activities.

Objective: The objective of this sub-indicator is to ensure regulations are in place requiring the
establishment of an IEC to review the CTs. For each CT, the IEC should verify that safety,
integrity, and human rights of participating subjects are protected. The IEC should
consider the general ethics of the trials, and thereby promote public reassurance. 09
Requirement: Establishment of IEC

Evidence to review: The assessor should ask for and review:

1. Legal provisions (laws, decrees, regulations or any legal binding document)
mandating the establishment of an IEC.
2. Documentation defining the identity of the designated authority that has the mandate
to host the IEC and to assist the IEC in discharging its duties.
3. Documentation defining the selection criteria for the members of the IEC, the number
of members on the IEC, and the term of office for each member.
4. Documentation defining the mechanisms and structures to ensure the independence
of the IEC, and documentation providing the code of conduct for ICE members.
5. The policy on potential conflicts of interest for members of the IEC.

References: 1. Handbook for good clinical research practice (GCP): Guidance for Implementation,
(41), (http://apps.who.int/medicinedocs/en)
2. Guidelines for good clinical practice (GCP) for trials on pharmaceutical products, (40),
(http://apps.who.int/medicinedocs/en)

Page 281 of 340

WHO Global Benchmarking Tool (GBT)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There are no legal provisions or regulations requiring the
establishment of an IEC.
 ONGOING IMPLEMENTATION (OI): The legal provisions or regulations that require the
establishment of an IEC have recently defined or drafted, but these have not yet been
implemented.
 PARTIALLY IMPLEMENTED (PI): The legal provisions or regulations requiring the
establishment of an IEC have been developed and the IEC has been established for 01
less than two years.
 IMPLEMENTED (I): The NRA has legal provisions or regulations requiring the 02
establishment of an IEC and the IEC is established.
03
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 04

Sub Indicator: CT01.08: Legal provisions, regulations and guidelines that require authorization 05
for the import or destruction of IMPs.
06
Maturity Level: 2
07
Scope: 1. Medicines
2. Vaccines
08

Clinical Trials Oversight (CT): Indicators and Fact Sheets

Description: The assessor should verify the existence, enactment, and implementation of legal
provisions, regulations and guidelines requiring that the NRA authorize the import and
destruction of IMPs. The legal provisions should be supported with regulations and
guidelines that provide guidance on the permitted quantities to import. The assessor
should note that the quantities to import or destroy should be justified in relation to the
timelines for the CT and the use of the IMPs as specified in the CT protocol. The provisions
should provide guidance on the import or destruction process, and define the roles and
responsibilities of the various stakeholders, including the sponsor, principal investigator,
and other participants. The guidelines should include:
1. Guidelines on the acceptable or allowable quantities of IMPs to import;
2. Guidelines on the appropriate destruction methods (i.e., by the NRA or supervised by
the NRA);
3. Procedures for a feedback mechanism to inform the NRA on quantities left over after
the CT.
The guideline and application forms should be developed and easily available to the
sponsor and other stakeholders. The processes and procedures to seek authorization for
import or destruction should be published and easily available.
09
Objective: The objective of this sub-indicator is to ensure the existence of legal provisions, regulations
and guidelines that require the NRAs (or responsible regulatory authority) to authorize
import and destruction of IMPs.

Requirement: Legal provisions, regulations and guidelines to import and destroy IMPs

Evidence to review: The assessor should ask for and review:

1. Legal provisions (laws, decrees, regulations or any legal binding document) and
guidelines on the import and destruction of IMPs. The assessor should verify that IMPs
imported for the purposes of CT application submissions require authorization.
2. The guidance documents and application forms for requesting assistance from the
NRA to import or destroy IMPs, as well as guidance on the processes to be executed upon
receipt of an application.
3. Records of IMPs imported or destroyed since the last NRA inspection.
4. Guidelines or similar documents providing guidance on the justifiable quantities of IMPs
that should be imported relative to the timelines in the CT protocol.
5. Guidance defining the stage in the CT application process at which IMPs may be
imported.

Page 282 of 340

WHO Global Benchmarking Tool (GBT)

References: 1. Guidelines for good clinical practice (GCP) for trials on pharmaceutical products, (40),
(http://apps.who.int/medicinedocs/en)
2. Guidelines on clinical evaluation of vaccines: regulatory expectations, (38),
(https://www.who.int/biologicals/WHO_ECBS/en/ and https://www.who.int/biologicals/
vaccines/en/)
3. WHO guidelines on nonclinical evaluation of vaccines, (39), (https://www.who.int/
biologicals/WHO_ECBS/en/ and https://www.who.int/biologicals/vaccines/en/)
4. WHO guidelines on the nonclinical evaluation of vaccines adjuvants and adjuvanted
vaccines, (119), (https://www.who.int/biologicals/WHO_ECBS/en/ and https://www.who. 01
int/biologicals/vaccines/en/)
02
Framework: Structure/Foundation/Input
03
Rating Scale:  NOT IMPLEMENTED (NI): There are no legal provisions, regulations or guidelines that
require authorization for the import and destruction of IMPs. 04
 ONGOING IMPLEMENTATION (OI): The legal provisions, regulations or guidelines that
require authorization for the import, and destruction of IMPs exist, but these have not 05
yet been implemented.
 PARTIALLY IMPLEMENTED (PI): The legal provisions, regulations or guidelines that 06
require authorization for the import and destruction of IMPs have been established for
less than two years. 07
 IMPLEMENTED (I): The NRA has legal provisions, regulations or guidelines that
require authorization for the import and destruction of IMPs, and there is documented
08
evidence that it is applied to all IMPs

Clinical Trials Oversight (CT): Indicators and Fact Sheets

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: CT01.09: There are requirements for monitoring and reporting of adverse events
and reactions during conduct of CT.

Maturity Level: 2

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the existence and implementation of requirements for
monitoring and reporting of adverse events and reactions. Such requirements should
clearly provide guidance on monitoring of adverse reactions and on following up when
they are observed. The guidance should provide clarity on the roles, responsibilities, and
duties of all stakeholders, including the NRA. The guidance should also specify the exact
requirements for monitoring during the period of the CT, including, for example, collection 09
and evaluation of data and reporting of adverse reactions and events. The guidance should
define the type, nature, and form of adverse reactions or events to report. In the event that
reporting to more than one NRA, authority, or stakeholder (e.g., sponsor) is required, the
sequence for reporting should be indicated. The guideline should be supported by a form
for monitoring and reporting adverse reactions and events.

Objective: The objective of this sub-indicator is to ensure the existence of requirements that mandate
that investigators and sponsors monitor all subjects involved in CTs. The requirements
should include an obligation to report, to the NRA or responsible regulatory authority,
all adverse or serious adverse reactions and events within the approved time frame for
reporting.

Requirement: Monitoring and reporting adverse reactions and events

Evidence to review: The assessor should ask for and review:

1. The guidelines and regulations on monitoring and reporting of adverse events and
reactions, as well as the guidance on required follow up. The assessor should verify that
the guidelines have been implemented.

Page 283 of 340

WHO Global Benchmarking Tool (GBT)

2. Documentation that specifies the exact roles, responsibilities, and duties of

each stakeholder. These should be compared to World Health Organization (WHO)
recommendations or other international standards.
3. The NRA recommendations to the sponsors and investigators on the procedures for
monitoring and reporting of adverse events and reactions. The assessor should verify that
the recommendations are in line with WHO or other international standards.
4. Documentation establishing the committee responsible for reviewing reports of adverse
reactions and events, and documentation defining the composition of the committee and
specialties of committee members. 01
5. Guidelines defining the timelines allocated for reporting adverse reactions and events on
the part of the investigator or sponsor and timelines for generating and submitting a report 02
on the adverse reaction or event to the NRA.
6. Examples of records of completed forms sent to the NRA. The assessor should verify 03
that the format and content is in compliance with applicable guidelines.
04
References: 1. Handbook for good clinical research practice (GCP): Guidance for Implementation, (41),
(http://apps.who.int/medicinedocs/en) 05
2. Guidelines for good clinical practice (GCP) for trials on pharmaceutical products, (40),
(http://apps.who.int/medicinedocs/en) 06

Framework: Structure/Foundation/Input 07

Rating Scale:  NOT IMPLEMENTED (NI): There are no requirements or guidelines on monitoring and
08
reporting of adverse events and reactions.

Clinical Trials Oversight (CT): Indicators and Fact Sheets

 ONGOING IMPLEMENTATION (OI): The requirements on monitoring and reporting
of adverse events and reactions are drafted or defined, but these have not yet been
followed.
 PARTIALLY IMPLEMENTED (PI): The requirements and guidance on monitoring and
reporting of adverse events and reactions have been developed for less than two
years or there are no reports from any CTs.
 IMPLEMENTED (I): The requirements and guidance on monitoring and reporting
adverse events and reactions exist and all sponsors are following them.

Limitations and remarks:  The assessor may score this sub-indicator as implemented if there are no records
because there are no ongoing CT studies.
 The review of the reported adverse events and reactions can be conducted by
internal staff if the NRA has access to adequate number of competent reviewers
and experts.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator
will always apply for all benchmarked NRAs).

Sub Indicator: CT01.10: There are guidelines on the format and content of CT applications. 09

Maturity Level: 2

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that guidelines on the format and content of CT applications
exist and are issued and implemented. The guidelines should provide clear guidance on the
format and content (i.e., a list of required information and documents) of the CT application
dossier to be submitted to the NRA. The content may include but is not limited to;
1. CT registration with a registry that is approved and recognized by the NRA;
2. covering letter addressed to head of the NRA;
3. completed CT application forms that should be signed and dated by authorized persons;
4. CT Protocol;
5. investigator’s brochure;
6. investigational product dossier;
7. GMP certificate;

Page 284 of 340

WHO Global Benchmarking Tool (GBT)

8. evidence of approvals from EC and institutional review board;

9. evidence of insurance coverage (from a recognized insurance company);
10. statement or evidence of financial support and declaration of support;
11. data safety monitoring board membership and signed charter;
12. evidence of contractual agreement between sponsor and principal investigator;
13. sample of informed consent form;
14. assent forms (if applicable);
15. statistical analysis plan, if applicable;
16. professional profile (i.e., qualifications, experience, expertise, evidence of excellent 01
knowledge of local CT regulatory requirements, and list of previous CTs managed) for
potential principal investigator, study pharmacist, local monitor, and other relevant 02
participants.
The guidelines should also provide guidance on the sequence of events to be followed 03
before the actual submission (e.g., meetings, workshops, seminars, IEC meetings, and
other relevant activities). The guidelines may provide guidance on choosing a CT site and 04
selection of subjects.
05
Objective: The objective of this sub-indicator is to ensure the existence of a clear guidance document
that guides the applicant and sponsor on the information (i.e., content and format) to 06
be included in the application package. The guidance should include the procedure
for submitting the application to the NRA for evaluation. The content of the guidance 07
document should be sufficient to facilitate CT activities.
08
Requirement: Format and content of CT applications

Clinical Trials Oversight (CT): Indicators and Fact Sheets

Evidence to review: The assessor should ask for and review:
1. Evidence that the guidelines on the format and content of the CT application are
available, implemented and published on the NRA’s website.
2. Evidence that accessory documents such as CT application forms and other CT related
forms exist and are issued and used.
3. Documentation specifying the format and nature of CT application submission package
(i.e. electronic or face-to-face communications, hard or soft copies, and number of copies
to be submitted).
4. List of critical documents that should be included in the application package.

References: 1. Guidelines on clinical evaluation of vaccines: regulatory expectations, (38), (https://www.

who.int/biologicals/WHO_ECBS/en/ and https://www.who.int/biologicals/vaccines/en/)
2. Guidelines for good clinical practice (GCP) for trials on pharmaceutical products, (40),
(http://apps.who.int/medicinedocs/en)
3. Handbook for good clinical research practice (GCP): Guidance for Implementation, (41),
(http://apps.who.int/medicinedocs/en)
4. WHO guidelines on nonclinical evaluation of vaccines, (39), (https://www.who.int/ 09
biologicals/WHO_ECBS/en/ and https://www.who.int/biologicals/vaccines/en/)
5. WHO guidelines on the nonclinical evaluation of vaccines adjuvants and adjuvanted
vaccines, (119), (https://www.who.int/biologicals/WHO_ECBS/en/ and https://www.who.
int/biologicals/vaccines/en/)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There are no guidelines on the format and content of CT
applications.
 ONGOING IMPLEMENTATION (OI): The guidelines on the format and content of CT
applications have been drafted or adopted, but they have not yet been implemented.
 PARTIALLY IMPLEMENTED (PI): The format and content of CT applications has been
established and they have been applied for less than two years.
 IMPLEMENTED (I): The NRA has guidelines on the format and content of CT
applications and they are being followed for all received CT applications.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Page 285 of 340

WHO Global Benchmarking Tool (GBT)

Sub Indicator: CT01.11: Legal provisions or regulations allow the NRA to recognize and use
relevant CT decisions, reports or information from other NRAs or from regional
and international bodies.

Maturity Level: 1

Scope: 1. Medicines
2. Vaccines
01
Description: The assessor should verify the existence, enactment, and implementation of legal
provisions or regulations that allow the NRAs to recognize and use relevant CT 02
decisions, reports or information from other NRAs or from regional and international
bodies. The legal provision and regulations should provide guidance on the modalities, 03
processes and procedures to employ when recognizing and using relevant CT
decisions, reports or information from other NRAs or from regional and international 04
bodies. The legal provisions should also provide guidance on the conditions (i.e., when
and how) to recognize and use relevant CT decisions, reports or information from other 05
NRAs or from regional and international bodies. The legal provisions and regulations
should also provide clear directives on the scope or extent of recognition and use of the 06
CT decisions, reports or information from other NRAs or from regional and international
bodies. 07

Objective: The objective of this sub-indicator is to ensure the existence of legal provisions or
08
regulations that allow the NRA to recognize and use relevant CT decisions, reports or

Clinical Trials Oversight (CT): Indicators and Fact Sheets

information from other NRAs or from regional and international bodies.

Requirement: Recognition of and reliance on CT application decisions and information

Evidence to review: The assessor should ask for and review:

1. The legal provisions (laws, decrees, regulations or any legal binding document) that
permits the NRA to recognize and use relevant CT decisions, reports or information from
other NRAs or from regional and international bodies.
2. Guidelines that define the scope and extent of recognition and use of relevant CT
decisions, reports or information from other NRAs or from regional and international
bodies.
3. Examples of instances or situations in which the NRA permitted recognition and use
relevant CT decisions, reports or information from other NRAs or from regional and
international bodies.
4. List of NRAs or regional and international bodies whose relevant decisions, reports
or information may be used to influence a CT application decision. Documentation that
indicate whether those NRAs or regional and international bodies are aware that their
relevant CT decisions, reports or information on certain CTs may be used (i.e., aware of 09
the legal provision).

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There are no legal provisions or regulations that allow the
NRA to recognize and use relevant CT decisions, reports or information from other
NRAs or from regional and international bodies.
 ONGOING IMPLEMENTATION (OI): The legal provisions or regulations that allow the
NRA to recognize and use relevant CT decisions, reports or information from other
NRAs or from regional and international bodies are drafted or pending for adoption.
 PARTIALLY IMPLEMENTED (PI): Legal provisions or regulations allowing the NRA to
recognize and use relevant CT decisions, reports or information from other NRAs or
from regional and international bodies have been established and the NRA has been
applying them for less than two years.

Page 286 of 340

WHO Global Benchmarking Tool (GBT)

 IMPLEMENTED (I): There are legal provisions or regulations allowing the NRA
to recognize and use relevant CT decisions, reports or information from other
NRAs or from regional and international bodies; this is supported with adequate
documentation (e.g., criteria for selection or list of NRAs recognized or used)

Limitations and remarks: Reliance or recognition should be considered optional if the NRA has the expertise,
capacity, and resources to conduct reviews of all CT applications received.

Indicator: CT02 Arrangement for effective organization and good governance. 01

02
Objective: The objective of this indicator is to ensure that there is a legal basis for the organizational
structure and governance that allows for the smooth exchange of information within and 03
outside the entity responsible for CTs (e.g., NRA, ethical committee, or clinical research
organization). The arrangement should define the roles and responsibilities of those 04
persons within the entities which are in charge of the various component activities within
the CT. The arrangement should also clarify how the roles of these individuals relate to the 05
governance structure of the organization responsible for CTs, as well as how they relate
to outside organizations such as the other CT-related NRAs or other competent regulatory 06
authorities. Effective implementation of these arrangements will ensure that the NRA
responsible for CTs has complete control of all the information related to CTs, including 07
information about ongoing CTs, new directives, authorizations, suspensions and rejections,
and other relevant activities.
08
The objective of this indicator is to establish that structures are in place at the

Clinical Trials Oversight (CT): Indicators and Fact Sheets

organizational and governance levels to promote effective intra- and inter-NRA
relationships so that information traffic is efficiently managed.

Category: Organization and governance

Sub Indicator: CT02.01: There is a defined structure with clear responsibilities to conduct CT
oversight activities.

Maturity Level: 2

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that the existence and implementation of a defined structure
with clear responsibilities to conduct CT oversight activities. The organizational structure
should be supported by legal provisions, regulations and guidelines. These legal provisions
should clearly delineate the roles, responsibilities and duties of all stakeholders inside
and outside the NRA. The guidelines should define the scope and extent of the roles and 09
responsibilities of those within the NRA with respect to CT oversight activities. Additionally,
lines of reporting should be clearly established. The structure of the entity or authority,
with respect to relationships and ranks, should be established and implemented. Similarly,
the mechanisms for information exchange within and outside (e.g., the IEC) the entity or
authority should be established and implemented. Assessor should verify the existence and
implementation of guidelines for the establishment of advisory committees (e.g., technical
expert committees or external expert committees) with clearly-defined objectives, functions,
composition, and terms of reference (ToRs).

Objective: The objective of this sub-indicator is to ensure there are structures in place, with clearly defined
roles and responsibilities for each structural and governance level, for CT oversight activities.

Requirement: Structure and Responsibilities

Evidence to review: The assessor should ask for and review:

1. Guidelines that define the roles, responsibilities, and duties of the entity responsible for CT
oversight within the NRA, and guidelines that define their placement on the organizational
chart in relation to other entities involved in CTs.

Page 287 of 340

WHO Global Benchmarking Tool (GBT)

2. The regulations or guidelines that provide the mandate to this entity or authority to
conduct CT oversight activities within the NRA.
3. Documentation of a clearly defined policy on conflicts of interest.
4. The operational manual (or similar document) of the entity responsible for this activity.
The manual should contain all the authorized and approved guidelines (published or not),
application forms (published or not), and SOPs (or similar documents).
5. Published CT application processing flow diagrams and corresponding timelines.
6. Published schedule of any applicable fees and charges.
01
References:
02
Framework: Structure/Foundation/Input
03
Rating Scale:  NOT IMPLEMENTED (NI): There is no defined structure with clear responsibilities to
conduct CT oversight activities. 04
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted a defined structure
with clear responsibilities to conduct CT oversight activities. 05
 PARTIALLY IMPLEMENTED (PI): The NRA has established a defined structure with
clear responsibilities to conduct CT oversight activities; however there is the need to 06
improve this function by providing required support for effective implementation and
coordination. 07
 IMPLEMENTED (I): The NRA has established a defined structure with clear
responsibilities to conduct CT oversight activities that is fully supported with required
08
resources.

Clinical Trials Oversight (CT): Indicators and Fact Sheets

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: CT02.02: Documented procedures are implemented to ensure the involvement
and communication among all stakeholders relevant to CTs.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the existence and implementation of documented procedures
that promote involvement and communication among all stakeholders. The procedures
should provide clear directives on the regulatory divisions and guidance on how to
implement the operational procedures to ensure the involvement and communication
among all stakeholders within and outside the NRA (e.g., the entity responsible
for CT oversight, IEC, GCP inspectorate, sponsors, principal investigator, and other 09
relevant entities). Procedures should be clear on the scope and extent of the roles and
responsibilities of each stakeholder. In addition, the procedure should be supported by an
information and documentation transfer policy to ensure that directives, information, and
documentation reach the intended recipients and that feedback is received. SOPs or
similar documents and guidelines should be approved, authorized and implemented.

Objective: The objective of this sub-indicator is to ensure there are documented procedures,
structures, and mechanisms in place to ensure proper relationships within and among the
entities involved in CTs in order to ensure effective and efficient exchange of information
among stakeholders.

Requirement: CT stakeholders interactions

Evidence to review: The assessor should ask for and review:

1. Evidence that procedures are documented and implemented, and that the impact of the
implementation is periodically assessed.
2. SOPs or similar documents that guide and inform effective communication and
collaboration among stakeholders.

Page 288 of 340

WHO Global Benchmarking Tool (GBT)

3. The CT application guidelines that capture the duties, roles and responsibilities of the
various stakeholders involved in CT activities. The assessor should request evidence that
the document is known to the relevant organizations, institutions, and departments.
4. Documentation for the CT application processing flow that captures the roles, duties and
responsibilities of the various stakeholders and that defines the timelines allocated to the
various stages
5. Documentation for the feedback mechanisms in the information and documentation
transfer policy among the various stakeholders.
01
References:
02
Framework: Process
03
Rating Scale:  NOT IMPLEMENTED (NI): There are no procedures to ensure involvement and
communication with all stakeholders relevant to CTs. 04
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted procedures to
ensure involvement and communication among all stakeholders relevant to CTs. 05
 PARTIALLY IMPLEMENTED (PI): The NRA established the procedures to ensure
involvement and communication among all stakeholders relevant to CTs but these 06
have not been fully incorporated and are not consistent with relevant guidance.
 IMPLEMENTED (I): The NRA has implemented procedures to ensure involvement and 07
communication with all relevant stakeholders including required documentation and
records of communications and of feedback for these communications.
08

Clinical Trials Oversight (CT): Indicators and Fact Sheets

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Indicator: CT03 Human resources to perform clinical trials oversight activities.

Objective: The objective of this indicator is to ensure to that all entities within an NRA are adequately
resourced with a trained, experienced and skilled workforce that is empowered to fully
perform CT oversight activities. This will ensure that CT oversight activities are performed
in accordance with international best practices.
The objective of this indicator is to evaluate the human resource capacity of the entities
with respect to the number of personnel, the skills and experience of the personnel, and
the overall composition of the workforce, with the goal of evaluating whether the workforce
possesses the specific expertise required to perform CT oversight activities. The assessor
should consider that some NRAs may outsource CT activities.

Category: Resources (HR, FR, infrastructure and equipment)

09
Sub Indicator: CT03.01: Sufficient competent staff (i.e., education, training, skills and experience)
are assigned to perform CT oversight activities.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that the human resources assigned to perform CT oversight
activities should be sufficient with respect to numbers and competent with respect to the
requisite skills, education, experience and training. There should be technical documents
and SOPs that provide guidance on the required background for CT oversight activities
and that consider the requirements for educational background, competencies, skills,
experience, and training.
The assessor should verify that the NRA estimated the number of staff required to
effectively and efficiently perform CT oversight function and that the NRA actually recruited
that number. In addition, the assessor should verify that these competency requirements
are well-established and maintained by the NRA. Metrics and statistics on the different

Page 289 of 340

WHO Global Benchmarking Tool (GBT)

activities performed as well as performance indicators can be used for estimating the
adequacy of the number of the assigned staff. The assessor should also verify that the
competency of the assigned staff is built, maintained and improved through recruitment as
well as continuous on- the-job training.

Objective: The objective of this sub-indicator is to ensure the existing human resources for CT
oversight is sufficient, in terms of numbers, experience, and specific competencies, to
perform all the activities along the entire CT oversight chain.
01
Requirement: Sufficient number of competent human resources in charge of CT oversight activities
02
Evidence to review: The assessor should ask for and review:
1. Evidence that the number of staff members involved in each of the documented 03
activities along the entire CT oversight process flow is adequate.
2. Evidence that the systems and structures are in place to ensure appropriate placement 04
of staff with respect to competence and skills.
3. Evidence that the system and structures have been implemented. The documentation 05
should include the records to verify that the staff competence is appropriate for the job
requirements. 06
4. Evidence that the professional profiles of the human resources engaged in CT oversight
activities are appropriate with respect to education, skills, and expertise, to perform a 07
particular function along the CT oversight chain. Documentation should include a list of the
requisite skills and training for each position.
08
5. Recruitment plan.

Clinical Trials Oversight (CT): Indicators and Fact Sheets

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): The NRA does not have enough competent staff (i.e.,
education, training, skills and experience) to perform CT oversight activities
 ONGOING IMPLEMENTATION (OI): The NRA has recently developed a plan to recruit
adequate competent staff; however, the plan has not been implemented.
 PARTIALLY IMPLEMENTED (PI): The NRA has initiated the implementation of
the human resources development plan; however, there is need to complete the
competency profile.
 IMPLEMENTED (I): The NRA has a sustained number of competent staff
(i.e., education, training, skills and experience) assigned to perform CT oversight 09
activities.

Limitations and remarks:  Assessment of the adequacy and appropriateness of the number of staff members
is quite subjective and should be linked to some process or output indicators. When
estimating staff adequacy, the assessor should consider the workload, backlog, and
delays in delivery based on established timeframes.
 Assessor should recognize the existence of personnel from other departments who
are often engaged in CT oversight functions such as assessments of protocols and
GCP inspections. All of these, together with in-house staff, should be included when
number and competency of CT oversight staff are evaluated. Other government
officials from outside the NRA should also be included in this assessment.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: CT03.02: Duties, functions, and responsibilities of the staff in charge of CT oversight
activities are established and updated in the respective job descriptions.

Maturity Level: 3

Page 290 of 340

WHO Global Benchmarking Tool (GBT)

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that procedures are in place to maintain a current and
updated structure for managing job descriptions for personnel participating in CT
oversight activities. In addition, job descriptions should address current staff duties,
responsibilities and the requisite competencies. A job description with this format and
content should be established and implemented for all staff. The management of job
descriptions should be supported by a guidance document that provides direction on when 01
and how to update the information, and where the information should be kept for easy
access. The guidance document should present the appropriate duties and responsibilities 02
that are assigned to each member of the organization involved in CT oversight activities.
Thus, the professional profiles of staff are reflected in their respective roles and 03
responsibilities within the NRA. There should be procedures to guide responsible persons
to document that duties, functions and responsibilities are revised and kept up to date. In 04
addition, procedures should be available to guide the keeping and documenting of up to
date work schedules and enforcing the implementation of the documented guidelines 05
and procedures.
06
Objective: The objective of this sub-indicator is to ensure that duties and responsibilities of the
staff are clear and well defined, that job descriptions are kept up to date with current 07
duties, functions and responsibilities, and that these activities are adequately
documented
08

Clinical Trials Oversight (CT): Indicators and Fact Sheets

Requirement: Duties, roles and responsibilities of the staff relevant to CT oversight activities.

Evidence to review: The assessor should ask for and review:

1. Procedure and guidelines that guide placement of staff members within the NRA;
2. The professional profiles of staff (i.e., job descriptions) and documentation that they are
related to their current roles and duties;
3. The professional profiles of the external experts and documentation that the profiles
provide a composition that is complete and consistent with that prescribed in the legal
provisions;
4. Procedures to guide the documentation of up to date duties and work schedules and to
enforce the implementation of the documented guidelines and procedures;
5. Job descriptions for designated staff.

Framework: Structure/Foundation/Input

Page 291 of 340

WHO Global Benchmarking Tool (GBT)

 IMPLEMENTED (I): The NRA has defined and established all required duties,
functions, and responsibilities, and respective job descriptions are up-to-date.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: CT03.03: Training plan developed, implemented and updated at least once a year
for staff in charge of CT oversight activities.
01
Maturity Level: 3
02
Scope: 1. Medicines
2. Vaccines 03

Description: The assessor should verify that training plans are developed, implemented and updated 04
at least once every year to reflect the current situation by considering education and
experience of the staff. The training plan should be complemented with guidelines or 05
similar documents that guide the development and implementation of training plans. The
assessor should ensure that induction training for new staff as well as continued on- the- 06
job for staff is planned and implemented. There should be procedures to approve the
training plan and the budget allocated for implementing and updating the training plan. The 07
plans should present clearly defined training goals and should include training in certain
topics and skills to address identified deficiencies. Learning objectives, training methods
08
and activities, evidence of learning, and evaluation and assessment of training should

Clinical Trials Oversight (CT): Indicators and Fact Sheets

be documented. This documentation should confirm that the learning objectives were
achieved and were designed to address weaknesses within the entities. Procedures should
be in place to ensure that a training plan is developed, implemented and updated at least
once every year. The assessor should verify that there is a system in place for monitoring
the implementation and effectiveness of the training plan and for documenting the skills
acquired in training activities for internal and external experts.

Objective: The objective of this sub-indicator is to ensure that a training plan for staff exists, and that
it is implemented and updated annually. Through the training plan, NRA can be sure that
competency of staff in charge of CT oversight activities is maintained and enhanced.

Requirement: Implementation of training plan

Evidence to review: The assessor should ask for and review:

1. Guidelines for development, implementation and annual update (i.e., at least once per
year) of the training plan. Guidelines should also provide for a mechanism to measure
effectiveness of training.
2. Documentation for the system or structures used to approve the training plan and to 09
evaluate the adequacy of the budget allocated to the training activities.
3. The current or existing staff training plan (or matrix) for staff. The assessor should
assess this in in relation to the respective individual job descriptions.
4. SOP for developing and maintaining the training plan.
5. Evidence that the NRA has investigated and identified training needs.
6. List of trainings performed.
7. Example records for training activities.

Page 292 of 340

WHO Global Benchmarking Tool (GBT)

apps.who.int/medicinedocs/documents/s21964en/s21964en.pdf)
4. Effective drug regulation: A multicountry study. World Health Organization; 2002, (7),
(http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)
5. Quality systems requirements for national good manufacturing practice inspectorates.
In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: thirty-sixth
report. World Health Organization; 2002: Annex 8 (WHO Technical Report Series, No. 902),
(8), (http://apps.who.int/medicinedocs/documents/s22112en/s22112en.pdf)

Framework: Process 01
Rating Scale:  NOT IMPLEMENTED (NI): There is no systematic training program including training 02
plan (or matrix).
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted or developed the 03
training plan but there is no evidence of implementation.
 PARTIALLY IMPLEMENTED (PI): The NRA has developed and initiated the training plan 04
implementation. However, the NRA has not fulfilled all required planned training or has
been applying the plan for less than two years. 05
 IMPLEMENTED (I): The NRA has an updated training plan developed that is
supported by adequate records to demonstrate effective plan implementation, 06
including induction training for new staff and routine on-the-job training for recruited
staff. 07

Limitations and remarks:  Training plans must be updated regularly; ideally on an annual basis, but not less
08
frequently that once every two years.

Clinical Trials Oversight (CT): Indicators and Fact Sheets

 Some regulatory functions may include many training activities that are not
incorporated in the institutional training programme. Such training normally is offered
by invitation. In this case, the assessor should recognize reports from non-routine CT
oversight-relevant training not included in the NRA training plan.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator
will always apply for all benchmarked NRAs).

Sub Indicator: CT03.04: The NRA generates and maintains records of staff training activities
and training effectiveness verification.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that records of staff training that is performed or organized by
the NRA are generated, maintained, regularly updated. This activity should be supported by
guidelines that direct the NRA to generate and maintain records of staff training activities. 09
Procedures should be in place to document and propose staff training needs and to
allocate a budget for continuous staff capacity building and development. The assessor
should check that there is an evaluation or assessment mechanism to verify the quality
of learning, and to confirm that learning objectives are achieved. Documentation should
include an inventory (i.e., soft and/or hard) system that records all impactful and non-
impactful trainings and identifies all staff members who participated. A system to measure
or estimate impact of trainings should be established.

Requirement: Training records

Evidence to review: The assessor should ask for and review:

1. Guidelines or similar documents that guide the NRA to generate and maintain records of

Page 293 of 340

WHO Global Benchmarking Tool (GBT)

staff training activities;

Clinical Trials Oversight (CT): Indicators and Fact Sheets

records of staff training activities.
 ONGOING IMPLEMENTATION (OI): The NRA has recently initiated plans to generate,
document and keep records of staff training activities, however they are not yet
followed.
 PARTIALLY IMPLEMENTED (PI): The NRA has recently initiated plans to generate,
document and keep records of staff training activities but they are not fully followed
for all training activities or they have been established for less than two years.
 IMPLEMENTED (I): The NRA generates and maintains records of staff training
activities

Limitations and remarks:  The assessor should note that some NRAs out-source training including staff
capacity development activities. In this case the assessor should request the identity
of the provider, as well as evaluations of the provider. The assessor may request the
professional profiles of tutors or resource persons used to offer training. The assessor
may also request records covering the archiving systems in place.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).
09
Indicator: CT04 Procedures established and implemented to perform clinical trials oversight.

Objective: The objective of this indicator is to ensure that the NRA (or responsible regulatory
authority) has documented procedures establishing how CTs should be implemented.
The documented procedures should be efficiently designed to assist in the preparation
of the CT application and in the receipt and evaluation of the enclosed information.
The procedures should present an overview of the activities to be carried out, the steps
to be followed, the resources required, the processes to be followed in evaluation of
the submitted documents, and the interrelationships among the various documents.
The responsible authority should verify that all procedures follow and address all legal
principles. The established procedures will give guidance on how CT applications are
handled to ensure efficient CTs.

The objective of this indicator is to establish whether procedures have been established
and implemented to effectively perform CT oversight activities.

Category: Regulatory process

Page 294 of 340

WHO Global Benchmarking Tool (GBT)

Sub Indicator: CT04.01: NRA has access to an advisory committee for review of CT applications
and post-approval safety and compliance issues.

Maturity Level: 4

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that the involvement of an advisory committee, which is 01
external to the NRA and which participates in the review of CT applications, is documented
and supported by legal provisions and regulations. The assessor should also verify that 02
the legal provisions and regulations that guide the establishment and use of an advisory
committee in the review of CT applications are established and implemented. There should 03
be clear guidance on when and how to use the committee, as well as guidance on the
scope and extent of the use of committee expertise in the review of CT applications and in 04
the evaluation of post approval safety and compliance issues. The ToRs of the members
of the committee should be available and documented. There should be documented 05
objectives and functions for the committee and for each member of the committee.
Professional profile of the committee members should be available, and the composition 06
of the committee should be guided by a legal provisions or regulations. In addition,
supporting guidelines should clearly define, for each member of the committee, their 07
roles and responsibilities in the review of CT applications. The code of conduct for the
committee members, as well as a general policy on conflicts of interest, should also be
08
documented and available. The assessor should note that some NRAs (or responsible

Clinical Trials Oversight (CT): Indicators and Fact Sheets

regulatory authorities) may not involve the advisory committee in the review of the
submitted CT applications. In such cases, the NRAs may consult the advisory committee
for direction in response to adverse events or reactions including serious ones.

Objective: The objective of this sub-indicator is to ensure that the NRA has the option, when needed,
of using an advisory committee in the review of CT applications and in the evaluation of
post approval safety and compliance issues. This option is particularly important in the
event of new or emerging technology.

Requirement: Advisory committee for CT

Evidence to review: The assessor should ask for and review:

1. Legal provisions (laws, decrees, regulations or any legal binding document) for the
establishment and use of an advisory committee in the review of CT applications
2. The ToRs of the committee and members of the committee, including their signed
updated conflicts of interest forms.
3. Documentation of the composition of the advisory committee, and evidence that it is in
line with the directives prescribed in the legal provision and regulations 09
4. Guidelines that define the scope and extent of the advisory committee contributions in
the review of CT applications.
5. SOPs for the advisory committee activities.
6. Records of advisory committee reports

References:

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): The NRA does not have access to an advisory committee
involved in the review of CT applications or post-approval safety and compliance
issues.
 ONGOING IMPLEMENTATION (OI): The NRA has drafted a proposal to establish the
advisory committee that is involved in the review of CT activities.
 PARTIALLY IMPLEMENTED (PI): The advisory committee has been established for
less than two years or is not involved in all required CT activities.
 IMPLEMENTED (I): The expert advisory committee is established, officially endorsed
and involved in all required CT activities, as needed, to ensure quality of CT reviews.

Page 295 of 340

WHO Global Benchmarking Tool (GBT)

Limitations and remarks:  In the case the NRA has all required internal resources for CT oversight activities, this
sub-indicator can be scored as not applicable.
 The assessor should consider this sub- indicator as implemented, if the NRA can
provide adequate evidence of access to advisory committees when needed.

Sub Indicator: CT04.02: The existence of the ECs with clearly defined composition.

Maturity Level: 3 01
Scope: 1. Medicines 02
2. Vaccines
03
Description: The assessor should verify the existence, enactment, and implementation of legal
provisions and regulations that define the composition of the ECs. The legal provisions and 04
regulations should stipulate that the composition of the ECs should be multidisciplinary and
multi-sectorial and should have a balanced age and gender distribution. Additionally, the EC 05
should include members with relevant scientific expertise and laypersons who represent
the interests and the concerns of the community. The legal provisions should be supported 06
with guidelines or SOPs that provide guidance on selection of members, on selection of the
committee chair, and on the ToRs for members. The legal provisions and regulations should 07
also provide direction on how the activities of the ECs should be regulated.
08
Objective: The objective of this sub-indicator is to ensure there is a defined and documented

Clinical Trials Oversight (CT): Indicators and Fact Sheets

composition of the ECs for CT activities.

Requirement: Existence, performance and composition of ECs

Evidence to review: The assessor should ask for and review:

1. Legal provisions (laws, decrees, regulations or any legal binding document) that provide
guidance on the composition of the ECs
2. The ToRs for each member along with guidelines on the selection process. The assessor
should review the ToRs and SOPs that provide guidance.
3. Professional profiles for members of the ECs. Assessor should assess these for balance
in terms of relevant expertise, gender, age and representation.
4. Current list of the EC members.

References: 1. Guidelines on clinical evaluation of vaccines: regulatory expectations, (38), (https://www.

who.int/biologicals/WHO_ECBS/en/ and https://www.who.int/biologicals/vaccines/en/)
2. WHO guidelines on nonclinical evaluation of vaccines, (39), (https://www.who.int/
biologicals/WHO_ECBS/en/ and https://www.who.int/biologicals/vaccines/en/)
3. WHO guidelines on the nonclinical evaluation of vaccines adjuvants and adjuvanted 09
vaccines, (119), (https://www.who.int/biologicals/WHO_ECBS/en/ and https://www.who.
int/biologicals/vaccines/en/)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): ECs are not established for CT activities.
 ONGOING IMPLEMENTATION (OI): The composition of the ECs has been recently
drafted but the committees have not been implemented.
 PARTIALLY IMPLEMENTED (PI): The ECs are established but are not officially
approved or are not routinely involved in CT activities.
 IMPLEMENTED (I): The ECs are established, officially endorsed and routinely involved
in CT activities.

Limitations and remarks:  The assessor should note that in most cases, the NRA will not be the agency that
hosts the EC, and as a result may not be able to provide sufficient information to
address this sub-indicator. The assessor may have to consult the Ministry of Health or
the relevant Ministry or institution for more information. Failure to secure information
should justify the scoring of this sub-indicator as “not or partially implemented”.

Page 296 of 340

WHO Global Benchmarking Tool (GBT)

 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: CT04.03: Nonclinical data is considered within CT application review.

Maturity Level: 3

Scope: 1. Medicines 01
2. Vaccines
02
Description: The assessor should verify the existence and implementation of the regulations and
guidelines on the preclinical data requirements for CT applications. The guidelines should 03
clearly state that the CT application requirements include submission of preclinical data
which is generated in accordance with the principles of Good Laboratory Practices. The 04
guidelines should provide guidance on the type and scope of data to submit to support the
CT application. 05

Objective: The objective of this sub-indicator is to ensure the availability of data from preclinical 06
studies that will provide sufficient insight into potential safety issues which may influence
eventual clinical application of the IMP. The objective of the sub-indicator is to ensure 07
that preclinical data are part of the application package, and that the data submitted are
reviewed according to a documented procedure.
08

Clinical Trials Oversight (CT): Indicators and Fact Sheets

Requirement: Preclinical data as part of CT application

Evidence to review: The assessor should ask for and review:

1. The regulations and guidelines that states that nonclinical data may be required in a CT
application.
2. SOPs that provide guidance during the review of the nonclinical data submitted as part
of the CT application.
3. Sample records of CT application forms.

References: 1. Guidelines for good clinical practice (GCP) for trials on pharmaceutical products, (40),
(http://apps.who.int/medicinedocs/en)
2. WHO guidelines on nonclinical evaluation of vaccines, (39), (https://www.who.int/
biologicals/WHO_ECBS/en/ and https://www.who.int/biologicals/vaccines/en/)

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): The NRA does not require submission and review of
nonclinical data within CT application. 09
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted guideline or similar
documents to consider nonclinical data as part of CT application.
 PARTIALLY IMPLEMENTED (PI): The NRA considers nonclinical data as part of CT
application however there is no systematic review, records or capacity to review these
data.
 IMPLEMENTED (I): The NRA requires submission of nonclinical data within CT
applications, and these data are reviewed.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: CT04.04: There are defined roles for ECs at all levels (e.g., national, sub-national,
or institutional).

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Page 297 of 340

WHO Global Benchmarking Tool (GBT)

Description: The assessor should verify that the legal provisions, regulations and guidelines which
establish the EC, also provide guidance on the composition of the EC, and on the roles and
responsibilities at each level. These legal provisions should be up to date. The regulations
and guidelines should define the objectives, functions, mandates, missions, roles and
responsibilities of the ECs as a whole, as well as for each member of the EC. In addition,
the scope and extent of their mandate should be well-described. The interactions among
the different levels of ECs should be defined and documented.

Objective: The objective of this sub-indicator is to ensure that there are defined roles for the ECs at 01
all levels
02
Requirement: Defined roles for ECs
03
Evidence to review: The assessor should ask for and review:
1. The legal provisions (laws, decrees, regulations or any legal binding document) and 04
guidelines that provide defined roles for the EC at each level of CT activity including
guidance on information sharing and interactions with relevant stakeholders. 05
2. Guidelines detailing the objectives, functions, roles and responsibilities of the EC at
each level of CT activity. 06
3. Evidence that the legally mandated scope of work is acceptable according to WHO
guidelines 07
4. Records of interaction among different EC levels.
08
References: 1. WHO guidelines on nonclinical evaluation of vaccines, (39), (https://www.who.int/

Clinical Trials Oversight (CT): Indicators and Fact Sheets

biologicals/WHO_ECBS/en/ and https://www.who.int/biologicals/vaccines/en/)
2. Guidelines on clinical evaluation of vaccines: regulatory expectations, (38),
(https://www.who.int/biologicals/WHO_ECBS/en/ and https://www.who.int/biologicals/
vaccines/en/)
3. WHO guidelines on the nonclinical evaluation of vaccines adjuvants and adjuvanted
vaccines, (119), (https://www.who.int/biologicals/WHO_ECBS/en/ and https://www.
who.int/biologicals/vaccines/en/)

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): The role and responsibilities of all levels of ECs have
not been defined.
 ONGOING IMPLEMENTATION (OI): The role and responsibilities of ECs have been
drafted at all levels however they are not yet followed.
 PARTIALLY IMPLEMENTED (PI): The role and responsibilities of ECs have been
assigned at all levels however they have been used for less than two years or they
are not fully documented.
 IMPLEMENTED (I): The role and responsibilities of ECs have been established 09
and documented for CTs, and there is evidence of consistent implementation.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: CT04.05: Documented and implemented procedures exist to review CT applications.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the existence and implementation of guidelines or SOPs
that provide the guidance needed for review of CT applications. The guidelines or
SOPs should provide guidance on how to review the various components of the CT
application. The guidance should be supported by a checklist or CT application review
form that should be completed during the review process. Such a form or checklist

Page 298 of 340

WHO Global Benchmarking Tool (GBT)

should be used to generate a review report that captures the recommendations

and comments that are used to justify authorization, rejection or deferral of the CT
application.

Objective: The objective of this sub-indicator is to ensure a uniform evaluation of CT applications that
is devoid of bias. There should be regulations and guidelines instituting a defined set of
criteria for the NRA (or responsible regulatory authority) to follow and use in the discharge
of their duties. Employment of this approach will avoid bias.
01
Requirement: Document procedures to review CT applications
02
Evidence to review: The assessor should ask for and review:
1. The guidelines and SOPs that establish the criteria for reviewing CT applications. 03
2. Evidence that the guidelines and SOPs are implemented during the review process.
3. Examples of completed review reports. The assessor should review completed review 04
reports and compare for compliance with the applicable guidance.
05
References:
06
Framework: Process
07
Rating Scale:  NOT IMPLEMENTED (NI): There are no documented procedures for review of CT
applications.
08
 ONGOING IMPLEMENTATION (OI): The NRA has drafted or adopted the documented

Clinical Trials Oversight (CT): Indicators and Fact Sheets

procedures for review of CT applications.
 PARTIALLY IMPLEMENTED (PI): The NRA developed and officially approved the
documented procedures for review of CT applications less than two years ago
or there is no evidence to demonstrate the implementation of this procedure for all
applications.
 IMPLEMENTED (I): The NRA has developed and implemented the documented
procedures for review of CT applications and has adequate evidence to demonstrate
that they have been followed for all CT applications.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: CT04.06: There are procedures for EC responsibility for clearance and follow
up until completion of the CT.

Maturity Level: 3

09
Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the existence, documentation, implementation of procedures
that provide guidance on the ECs responsibility for clearance and follow-up until
completion of the CT. The documentation of procedures should be in the form of
guidelines or SOPs, and should provide detailed guidance on the roles and responsibilities
of the EC in the clearance and follow-up processes through the completion of the CT. The
guidelines should specify the scope and extent of the EC’s activity in the CT activities, the
corresponding timelines for each activity, the expected outcomes of their activities, and the
impact of each activity on the entire CT. Importantly, EC retains these responsibilities until
CT is completed.

Objective: The objective of this sub-indicator is to ensure the NRA (or responsible regulatory
authority) has established procedures on the responsibilities of the EC from CT clearance
until the completion of the CT.

Requirement: Documented procedures for the EC along the entire CT authorization chain

Page 299 of 340

WHO Global Benchmarking Tool (GBT)

Evidence to review: The assessor should ask for and review:

1. The regulations, guidelines and SOPs detailing the roles and responsibilities of the
EC in the clearance and follow-up process of CTs until completion of the trial. The
assessor should verify that the procedures have been appropriately documented and
implemented.
2. Documentation that defines the scope and extent of EC responsibility at each stage
of the CT.
3. The processes and procedures guiding the EC operations and activities at each stage
of the CT. 01
References: 02

Rating Scale:  NOT IMPLEMENTED (NI): There are no procedures that define EC responsibility for 03
clearance and follow up until completion of the CT.
 ONGOING IMPLEMENTATION (OI): There are drafted or adopted procedures to define 04
EC responsibility for clearance and follow up until completion of the CT.
 PARTIALLY IMPLEMENTED (PI): There are procedures to define EC responsibility 05
for clearance and follow up until completion of the CT, however they have not been
implemented. 06
 IMPLEMENTED (I): The defined procedures regarding EC responsibility for clearance
and follow up until completion of the CT are implemented and supported with 07
documented evidence.
08
Limitations and remarks:  The assessor should note that some ECs may have a limited mandate along the

Clinical Trials Oversight (CT): Indicators and Fact Sheets

entire CT chain in accordance to the existing regulations. In such cases, the assessor
should be guided by the existing laws or regulations. The scoring may be partial or not
applicable depending on the country regulation.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: CT04.07: The same policies are used for the evaluation of CT applications regardless
of the applicant (e.g., domestic, foreign, public sector, or private sector).

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the existence, enactment, and implementation of regulations
and guidelines that dictate that the criteria applied to evaluate CT applications should
be the same regardless of the applicant. A single set of criteria for the evaluation
process should be employed during evaluation of all CT applications. The criteria should 09
be supported with a checklist or evaluation form that should be completed during the
evaluation process. The assessor should note that in some instance, such as emergencies,
this defined set of criteria for evaluating CT applications may not apply. The assessor
should request for and review the documented and implemented procedures and
processes that are used to provide guidance for the review of CT applications in the event
of emergencies.

Objective: The objective of this sub-indicator is to ensure that the same criteria are applied to
evaluate CT applications from various sources. Use of consistent procedures will avoid
bias.

Requirement: Uniform criteria and policy for evaluating CT application regardless of applicant

Evidence to review: The assessor should ask for and review:

1. The regulations and guidelines that require that all CT applications are evaluated with
the same single set of criteria regardless of the applicant.
2. The guidelines and SOPs that specify the detailed criteria to be used for evaluating CT
applications.

Page 300 of 340

WHO Global Benchmarking Tool (GBT)

3. Evidence that the reviewers of CT applications are aware of the criteria and understand
how to apply the criteria in their assessment activities. Assessor should review evidence
that the criteria are documented, published and implemented.

References:

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): There are no defined policies and criteria, or different 01
policies are used for the evaluation of CT applications from different sources (e.g.
domestic, foreign, public sector, or private sector). 02
 ONGOING IMPLEMENTATION (OI): There is a drafted policy or documentation that
requires that the same criteria should be followed for the evaluation of CT applications 03
regardless of the applicant (e.g. domestic, foreign, public sector, or private sector).
 PARTIALLY IMPLEMENTED (PI): There are regulations or guidelines that require that 04
the same criteria are used for the evaluation of CT applications regardless of the
applicant (e.g. domestic, foreign, public sector, or private sector); however, these have 05
been available for less than two years or there is no evidence to demonstrate full
implementation for all received CT applications. 06
 IMPLEMENTED (I): The same criteria and policy are used for the evaluation of CT
applications regardless of the applicant (e.g. domestic, foreign, public sector, or 07
private sector) for all received and approved CT applications.
08
Limitations and remarks:  The assessor should note that in some countries only one source may exist.

Clinical Trials Oversight (CT): Indicators and Fact Sheets

The assessor can score this sub-indicator as implemented if all regulation and
requirements are the same.
The assessor should consider there are different requirements based on the nature of
each product.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Indicator: CT05 Mechanism exists to promote transparency, accountability and communication.

Objective: The objective of this indicator is to ensure mechanisms are in place to ensure that
information on CT applications, including authorized, suspended, rejected and completed
CTs, are published to promote transparency and information sharing among stakeholders
and potential stakeholders such as trial subjects.

Category: Transparency, accountability and communication

Sub Indicator: CT05.01: There is clarity about the funding of the EC and its members 09

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify that the source of funding for the EC activities is known,
documented, perceived as appropriate, and devoid of any conflicts of interest. The source
of funding for EC activities should be documented and supported by regulations that
provide clarity on the funding mechanisms and on the management and disbursement of
funds. All fees and charges should be legally approved and implemented. The regulation
should provide clarity on the source of funding and provide assurance about the absence
of any conflicts of interest.

Objective: The objective of this sub-indicator is to ensure that there is a dedicated source of funding
for the operations of the EC and that the source of funding is known, transparent and
devoid of any conflicts of interest.

Page 301 of 340

WHO Global Benchmarking Tool (GBT)

Requirement: Clarity on the source of funding

Evidence to review: The assessor should ask for and review:

1. The regulation detailing the source of funding that is made available to the EC to fund its
activities.
2. Evidence that the identity of the source is in compliance with the general policy on
conflicts of interest.
3. Evidence that the level of transparency regarding the source of funding and the
management of the funds is sufficient to eliminate doubts on conflicts of interest. 01
4. Evidence that the fees and charges are legally approved.
5. Records and reports of EC funding. 02

References: 1. Guidelines on clinical evaluation of vaccines: regulatory expectations, (38), 03

(https://www.who.int/biologicals/WHO_ECBS/en/ and https://www.who.int/biologicals/
vaccines/en/) 04
2. WHO guidelines on nonclinical evaluation of vaccines, (39), (https://www.who.int/
biologicals/WHO_ECBS/en/ and https://www.who.int/biologicals/vaccines/en/) 05
3. WHO guidelines on the nonclinical evaluation of vaccines adjuvants and adjuvanted
vaccines, (119), (https://www.who.int/biologicals/WHO_ECBS/en/ and https://www.who. 06
int/biologicals/vaccines/en/)
07
Framework: Structure/Foundation/Input
08
Rating Scale:  NOT IMPLEMENTED (NI): NRA does not have information or clarity about the funding

Clinical Trials Oversight (CT): Indicators and Fact Sheets

of the EC and its members.
 ONGOING IMPLEMENTATION (OI): There are drafted procedures and policies to
provide information about the funding process of the EC and its members.
 PARTIALLY IMPLEMENTED (PI): There are procedures to provide information about
the funding process of the EC and its members, however there is no evidence to
demonstrate availability of this information regarding all ECs.
 IMPLEMENTED (I): The NRA has clear information about the funding of the EC and its
members. Information is supported with documented evidence.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: CT05.02: The list of the CTs (approved and rejected applications), including
summarized evaluation reports by the NRA, are publicly available or recorded
in a domestic or international database.

Maturity Level: 4
09
Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the existence of regulations, guidelines or similar documents
that dictate that all CT applications, either approved or rejected, should be listed in a
domestic or international database. This database should also include summaries of the
evaluation for each CT application. The regulations should be supported with guidelines
that provide guidance on the information to be listed, as well as the content, format and
information that should be included in the summary evaluation reports that are made
available to the public. The guidelines should provide guidance on the database to the
used (i.e., local, international, hosting organization, and other relevant details) and on the
mechanism to be used for updating the database.

Objective: The objective of this sub-indicator is to ensure that approved and rejected CT applications,
as well as summarized CT evaluation reports, are listed and published in a local or
international database.

Requirement: Publication of approved and rejected CT applications as well as CT evaluation reports

Page 302 of 340

WHO Global Benchmarking Tool (GBT)

Evidence to review: The assessor should ask for and review:

1. Regulations and guidelines that require that all approved and rejected CT applications, as
well as summary evaluation reports, should be listed and available in an easily-accessible
local or international database.
2. Regulations and guidelines that require that the list of all approved and rejected CT
applications, as well as summary evaluation reports, should be updated periodically.
3. Documentation for the rate (i.e., when and how) at which the list is updated.
4. The list of approved and rejected CT applications, as well as summary evaluation reports,
on the local or international database. 01
5. Guidelines and SOPs that provide guidance on the content, format and information that
should be uploaded to the database. 02
6. Evidence that stakeholders are aware of the availability of the list, and that they have a
clear understanding of how and where to access the list and its contents. 03

References: 1. Guidelines on clinical evaluation of vaccines: regulatory expectations, (38), (https://www. 04

who.int/biologicals/WHO_ECBS/en/ and https://www.who.int/biologicals/vaccines/en/)
2. WHO guidelines on nonclinical evaluation of vaccines, (39), (https://www.who.int/ 05
biologicals/WHO_ECBS/en/ and https://www.who.int/biologicals/vaccines/en/)
3. WHO guidelines on the nonclinical evaluation of vaccines adjuvants and adjuvanted 06
vaccines, (119), (https://www.who.int/biologicals/WHO_ECBS/en/ and https://www.who.int/
biologicals/vaccines/en/) 07

Framework: Output
08

Clinical Trials Oversight (CT): Indicators and Fact Sheets

Rating Scale:  NOT IMPLEMENTED (NI): There is no list of CTs (approved and rejected applications),
including summarized NRA evaluation reports, that is publicly available or recorded in a
domestic or international database.
 ONGOING IMPLEMENTATION (OI): There is a draft procedure to publish the list of CTs
(approved and rejected applications), including summarized NRA evaluation reports,
however it is not yet published.
 PARTIALLY IMPLEMENTED (PI): There is a list of CTs (approved and rejected
applications), including summarized NRA evaluation reports, and list is publicly
available or recorded in a domestic or international database; however, it is not updated
or it was published less than two years ago.
 IMPLEMENTED (I): There is an updated list of CTs (approved and rejected applications),
including summarized NRA evaluation reports, that is publicly available or recorded in a
domestic or international database.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Indicator: CT06 Mechanism in place to monitor regulatory performance and output. 09

Objective: The objective of this indicator is to ensure that CT oversight mechanisms are in place to
verify that all CT activities are subjected to quality controls and other checks to reduce
errors, increase objectivity and ensure that the processes are consistent. These activities will
generate an assured output and enhance the reliability of results at the various stages of the
CT application processing flow.
Oversight activities should cover the various stages of the CT application processing
flow, including receipt of CT applications, acknowledgement of applications, processing,
evaluation, and assessment of the various parts of the applications, generation of
recommendations (i.e., approval, deferral, or rejection) following the evaluation and
assessment process, issuance of certificates, implementation of regulatory actions when
necessary (e.g. stoppage or termination of CTs) and publication of the decision made for
each application (i.e., approval or rejection) along with summary evaluation reports.
This would lead to consistency in the regulatory performance of the CT oversight function
as well as reliable outputs.

Category: Monitoring progress and assessing outcomes and impact

Page 303 of 340

WHO Global Benchmarking Tool (GBT)

Sub Indicator: CT06.01: There is an internal list or database of all approved and rejected CTs,
and the NRA maintains a record of each approved and rejected CT.

Maturity Level: 3

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the establishment of a register, list, or database into which all 01
approved or rejected CT applications are entered. The register, list or database should be
continually updated and maintained to reflect the current situation. The establishment 02
and maintenance of the register, list, or database should be supported by guidelines and
SOPs. These procedures should also provide guidance on the content, format and type of 03
information to enter and maintain in the database and guidance on procedures (i.e., when
and how) to update the register, list or database. In addition, guidance on who should be 04
given access to the stored information should be clearly documented.
05
Objective: The objective of this sub-indicator is to ensure that an internal list or database is kept for
all approved and rejected CT applications and that the NRA (or responsible regulatory 06
authority) maintains a record of all approved and rejected CTs.
07
Requirement: Internal list or database for all CT applications
08
Evidence to review: The assessor should ask for and review:

Clinical Trials Oversight (CT): Indicators and Fact Sheets

1. The guidelines requiring the establishment and maintenance of a register, list or
database of approved or rejected CT applications.
2. The register, list or database of all CT applications that have been approved or rejected,
including details of the type of information that was entered and kept.
3. List of persons with the authority to access stored information, as well as information on
the documentation that must be completed before access is granted.

References: 1. Guidelines on clinical evaluation of vaccines: regulatory expectations, (38),

(https://www.who.int/biologicals/WHO_ECBS/en/ and https://www.who.int/biologicals/
vaccines/en/)
2. WHO guidelines on nonclinical evaluation of vaccines, (39), (https://www.who.int/
biologicals/WHO_ECBS/en/ and https://www.who.int/biologicals/vaccines/en/)
3. WHO guidelines on the nonclinical evaluation of vaccines adjuvants and adjuvanted
vaccines, (119), (https://www.who.int/biologicals/WHO_ECBS/en/ and https://www.who.
int/biologicals/vaccines/en/)

Framework: Output
09
Rating Scale:  NOT IMPLEMENTED (NI): There is not an internal list or database of all approved and
rejected CTs.
 ONGOING IMPLEMENTATION (OI): There is a drafted procedure to create an internal
list or database; however the list or database does not yet exist.
 PARTIALLY IMPLEMENTED (PI): There is an internal list or database of all approved
and rejected CTs, and the NRA maintains a record of each approved and rejected CT;
however, the list or database is not up to date or all applications are not included.
 IMPLEMENTED (I): There is an updated internal list or database of all approved and
rejected CTs, and the NRA has maintained records of each approved and rejected CT
for more than two years.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: CT06.02: Performance indicators for CT oversight activities are established
and implemented.

Maturity Level: 4

Page 304 of 340

WHO Global Benchmarking Tool (GBT)

Scope: 1. Medicines
2. Vaccines

Description: The assessor should verify the existence and implementation of performance indicators
for different activities included under the CT oversight functions.
Specifically, the system should define key performance indicators (KPIs) along the entire
CT oversight activity chain and all indicators should be adequately justified. For the
purpose of clarity and consistency, established KPIs should be supported with guidelines
for monitoring and maintenance of the KPIs. The guidelines in turn should be supported 01
by SOPs and tools that define the procedures to be used for monitoring and evaluating the
performance indicators and that define procedures and timelines for reviewing and revising 02
the indicators.
Examples of performance indicators for CT oversight function include: CT applications 03
received, granted or rejected, timelines for CT application processing, and regulatory
actions taken with respect to CTs. 04
Established KPIs might be qualitative, quantitative or combination of both. In general,
quantitative indicators are preferred to avoid bias or misinterpretation. However, qualitative 05
indicators are also accepted. Qualitative indicators may or may not include scoring or
scaling to render them semi- quantitative and thus more informative. The assessor 06
should ensure that indicators are measured on a regular basis to monitor progress and
advancement. 07
In addition, the assessor should verify measured indicators are analyzed to identify trends
or abnormalities. Justifications for any identified abnormalities should be provided; when
08
necessary, process optimizations should be introduced to avoid recurrence.

Clinical Trials Oversight (CT): Indicators and Fact Sheets

Objective: The objective of this sub-indicator is to ensure that a system, mechanism, or procedure
exists to require the NRA to establish performance indicators along the entire CT
oversight chain. Additionally, the objective is to ensure that KPIs are actually contributing
to monitoring of regulatory performance, to measuring effectiveness of CT oversight
regulatory activities, and to making any necessary adjustments or optimizations.

Requirement: KPIs for CT oversight activities

Evidence to review: The assessor should ask for and review:

1. Documents supporting the system, mechanism, or procedure compelling the NRA to
establish and implement performance indicators along the entire CT oversight activity
chain.
2. Evidence that the performance indicators have been established and implemented, and
that the members of staff involved in the CT oversight function are aware of the indicators
and the guidelines and SOPs used for monitoring and evaluating their performance.
3. The current performance indicators for CT oversight activities
4. Analyses of the measured indicators along with the investigations done to identify 09
trends or abnormalities.
5. Documentation for follow-up of any observed abnormalities, including justifications
for any identified abnormalities as well as any process optimizations introduced to avoid
recurrence.

References: 1. Guidelines on clinical evaluation of vaccines: regulatory expectations, (38), (https://www.

who.int/biologicals/WHO_ECBS/en/ and https://www.who.int/biologicals/vaccines/en/)
2. WHO guidelines on nonclinical evaluation of vaccines, (39), (https://www.who.int/
biologicals/WHO_ECBS/en/ and https://www.who.int/biologicals/vaccines/en/)
3. WHO guidelines on the nonclinical evaluation of vaccines adjuvants and adjuvanted
vaccines, (119), (https://www.who.int/biologicals/WHO_ECBS/en/ and https://www.who.
int/biologicals/vaccines/en/)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There are no KPIs for CT oversight activities.
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted KPIs for CT
oversight activities but they have not yet been reported.

Page 305 of 340

WHO Global Benchmarking Tool (GBT)

 PARTIALLY IMPLEMENTED (PI): The NRA has developed KPIs for CT oversight
activities and has been applying them for less than two year or they have not covered
all critical steps.
 IMPLEMENTED (I): The NRA has established and implemented KPIs for CT oversight
activities. The indicators are reviewed regularly, and appropriate actions are taken and
decisions made.

Limitations and remarks: When they refer to outcomes, indicators may be ambiguous and difficult to interpret, as
outcomes are the result of many factors that are difficult to disentangle. When they refer to 01
processes, indicators are often too specific, as they may focus on a particular intervention
or condition or they may quickly become outdated as business models develop. 02
Different methodologies are used to measure the NRAs performance on CT oversight
activities. In this case, the assessor should verify that adequate supporting documents are 03
available. The assessor should consider that developed performance indicators should be
Specific, Measurable, Achievable, Realistic, and Time-bound (i.e., “SMART”). 04
Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 05

Sub Indicator: CT06.03: Progress reports from sponsors or CROs during and after CTs sent to 06
and shared among NRAs and ECs.
07
Maturity Level: 3
08
Scope: 1. Medicines

Clinical Trials Oversight (CT): Indicators and Fact Sheets

2. Vaccines

Description: The assessor should verify the establishment and implementation of mechanisms to
receive, store and disseminate information received from stakeholders during and after
CTs. The mechanism should be supported with guidelines that state that feedback reports
along the entire CT chain should be sent periodically by the sponsor (or CRO) to the
NRAs and ECs. Receipt of the reports should be documented by the NRAs and ECs. The
guidelines should provide guidance on the content and format for preparing the reports,
as well as guidance on the procedures for submitting the reports. In addition, SOPs
should provide guidance on how to review, summarize and store the reports received. The
database where reports are kept should be documented.

Objective: The objective of this sub-indicator is to ensure a progress reporting and documentation
system is in place to receive, store and disseminate reports from sponsors or CROs.

Requirement: Accessibility and availability of feedback from stakeholders

Evidence to review: The assessor should ask for and review: 09

1. The guidelines that stipulate that feedback reports along the entire CT chain should be
sent periodically by the sponsor (or CRO) to the NRAs and ECs and that the NRAs and ECs
should document receipt of the reports. The content of these reports may differ between
NRA and ECs.
2. Guidelines on the content and format for preparing the reports, as well as on the
procedure for submitting the reports.
3. Documented list of reports received from sponsors or CROs. Content and format should
be reviewed for compliance with guidelines
4. Examples of reports from sponsors and CROs.
5. Documentation for the database or data storage facility where the reports are stored.

References: 1. Guidelines on clinical evaluation of vaccines: regulatory expectations, (38), (https://www.

who.int/biologicals/WHO_ECBS/en/ and https://www.who.int/biologicals/vaccines/en/)
2. WHO guidelines on nonclinical evaluation of vaccines, (39), (https://www.who.int/
biologicals/WHO_ECBS/en/ and https://www.who.int/biologicals/vaccines/en/)
3. WHO guidelines on the nonclinical evaluation of vaccines adjuvants and adjuvanted
vaccines, (119), (https://www.who.int/biologicals/WHO_ECBS/en/ and https://www.who.
int/biologicals/vaccines/en/)

Page 306 of 340

WHO Global Benchmarking Tool (GBT)

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): There is no mechanism for feedback and progress
reporting between sponsors or CROs and NRAs and ECs during and after CTs.
 ONGOING IMPLEMENTATION (OI): There are draft guidelines to define feedback
mechanisms and to send progress reports from sponsors or CROs to NRAs and ECs
during and after CTs; however, it is not followed.
 PARTIALLY IMPLEMENTED (PI): There are mechanisms for feedback and progress
reporting from sponsors or CROs to NRAs and ECs during and after CTs, however all
reports are not available. 01
 IMPLEMENTED (I): There are mechanisms for feedback and progress reporting
from sponsors or CROs to NRAs and ECs during and after CTs. The mechanisms 02
have been established for more than two years and are supported with adequate
documentation, guidelines and records. 03

Maturity Level: 3 07

Scope: 1. Medicines
08
2. Vaccines

Clinical Trials Oversight (CT): Indicators and Fact Sheets

Description: The assessor should verify the establishment and implementation of documented
timelines and internal timeline tracking systems that guide the CT application processing.
These timelines should be supported by guidelines requiring that CT applications should
be processed and assessed according to published timelines. The timelines should be
monitored internally for compliance. The guidelines should be supported with SOPs that
provide guidance on how to establish the timelines and how to monitor the established
timelines. The guidance should be designed to be adaptable for routine and non-routine CT
applications (e.g., public health emergencies). In addition, the guidance should be designed
to address each stage of the CT application process flow. The timelines should be known
by all stakeholders for the purposes of transparency and trust.

Objective: The objective of this sub-indicator is to ensure that timelines exist for processing CT
applications and that internal systems or mechanisms are in place to monitor the CT
application processing for compliance with the timelines.

Requirement: Availability of timelines and a system to monitor processing of CT applications

09
Evidence to review: The assessor should ask for and review:
1. The guidelines that stipulate that CT applications should be processed and assessed
according to prescribed timelines.
2. Regulations establishing that timelines for CT application assessment should be
internally monitored for compliance with published timelines.
3. Documentation for the prescribed timelines for CT application assessment,
and guidelines for the internal tracking system used to monitor the timelines for
effectiveness;
4. Sample report from the internal tracking system for CT application assessment.
Assessor should review for content as well as for compliance with CT application
assessment guidelines.
5. The work schedules and work plans for staff responsible for monitoring timelines.

References: 1. Guidelines on clinical evaluation of vaccines: regulatory expectations, (38), (https://www.

3. WHO guidelines on the nonclinical evaluation of vaccines adjuvants and adjuvanted

Page 307 of 340

WHO Global Benchmarking Tool (GBT)

vaccines, (119), (https://www.who.int/biologicals/WHO_ECBS/en/ and https://www.who.

int/biologicals/vaccines/en/)

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): There are no defined timelines for the assessment of CT
applications and no internal tracking system to follow the targeted time frames.
 ONGOING IMPLEMENTATION (OI): There are drafted timelines and an internal
tracking system to follow the targeted time frames for the assessment of CT; 01
however, they are not followed.
 PARTIALLY IMPLEMENTED (PI): The timelines for the assessment of CT applications 02
are defined; however, the internal tracking system to follow the targeted time frames
has not been fully documented. 03
 IMPLEMENTED (I): The timelines for the assessment of CT applications and an
internal tracking system to follow the targeted time frames have been implemented 04
for more than two years.
05
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 06
07
08

Clinical Trials Oversight (CT): Indicators and Fact Sheets

Page 308 of 340

WHO Global Benchmarking Tool (GBT)

09 01

National Regulatory System (RS): Indicators and Fact Sheets

NRA Lot Release
(LR): Indicators
and Fact Sheets

WHO Global
Benchmarking 02
Tool (GBT) 03
for Evaluation of
04
National Regulatory
System of Medical 05
Products 06
07
08
09

Page 309 of 340

WHO Global Benchmarking Tool (GBT)

09. NRA Lot Release (LR):

Indicators and Fact Sheets

Function:
09 – NRA LOT RELEASE (LR)
01
Description: National Regulatory Authority (NRA) lot release (also called official authority batch release) 02
is a non-common regulatory function that does not apply to all medical products. Lot
release is a system specifically established for the regulatory release of specified biological 03
products. The goal of the regulatory function is to ensure the quality, safety and efficacy of
biological products through a regulatory release system. Lot release is done on a lot-by-lot 04
basis and takes into account the nature and inherent variability of these products.
NRAs should have the legal mandate to perform independent lot release. They should 05
develop and implement the necessary policies, guidelines, procedures and forms in line
with World Health Organization and major international guidelines. 06
In the case of vaccines, different approaches are currently used for conducting lot release.
The options include: review of the summary protocols only, review of the summary 07
protocols combined with independent testing (i.e., either full or selected testing), and
recognition and acceptance of lot release certificates from the responsible NRA or
08
National Control Laboratory (NCL). The NRA or NCL has the responsibility to decide on
an appropriate strategy for each vaccine. The decision should take into consideration
09
the nature of the vaccine, the post-marketing experience for each vaccine (including

NRA Lot Release (LR): Indicators and Fact Sheets

production history and safety profile), and the availability of other independent evidence of
product quality.

Indicator: LR01 Legal provisions, regulations and guidelines required to define regulatory
framework of independent lot release by the NRA.

Objective: The objective of this indicator is to ensure that NRA activities are backed by legislation and
provisions for independent lot release.
The lot release of vaccines by regulatory authorities is part of the regulation of vaccines
and involves the independent assessment of each lot of a licensed vaccine before it is
released onto the market. The impact of using substandard lots may not be known for a
very long time (i.e., years). Similarly, safety issues with a particular lot may not be known
immediately (i.e., within a few hours) after administration. Due to these delays, there
could be a drastic impact if a large number of healthy persons receive a vaccine before a
problem is recognized. For these reasons, a careful, independent review of manufacturing
and quality control data on every lot is necessary before a lot is marketed.

Category: 01. Legal provisions, regulations and guidelines

Sub Indicator: LR01.01: Legal provisions and regulations exist to conduct and enforce lot release
for all vaccines.

Maturity Level: 1

Scope: Vaccines

Description: The assessor should verify the existence of legal provisions, regulations or other
administrative provisions that give the National Regulatory (NRA) authority to implement
and enforce lot release for vaccines. The legal provisions should define the responsible
officer authorized to sign the regulatory lot release certificate.
The assessor should verify that the legal basis to perform lot release applies for all
vaccines marketed in the country and includes Expanded Programme on Immunization
and non- Expanded Programme on Immunization vaccines as well as imported and
domestically- produced vaccines. Current approaches for conducting lot release of

Page 310 of 340

WHO Global Benchmarking Tool (GBT)

vaccines include: review of the summary protocol only; review of the summary protocol
with independent testing (i.e., either full or selected testing); and recognition and
acceptance of lot release certificates from the responsible NRA or National Control
Laboratory (NCL). These approaches are not
mutually exclusive and different approaches may be used for different products in the
same country. It is the responsibility of the NRA or NCL to decide on the appropriate
strategy for each vaccine, taking into consideration the nature of the vaccine, the post-
marketing experience (including production history and safety profile) for the vaccine, and
the availability of other independent evidence of product quality. 01
The assessor should verify that the criteria have been defined when the NRA may elect
to follow non-routine procedures for lot release. Reasons for exemption from lot release 02
could include, for example, a shortage of a product on the market or the need to import a
non-authorized product for a defined time. The assessor should verify that legal provisions 03
and regulations permit the use of a fast -track mechanism with specific lot release
requirements. 04

Objective: The objective of this sub-indicator is to ensure that lot release of vaccines is part of the 05
regulatory framework and involves the independent assessment of each lot of a licensed
vaccine before it is released on to the market. 06
Each country should establish the national guidelines for lot release. The guidelines should
define all required procedures, from the submission of the lots for release to the issuance 07
of lot release certificates.
All vaccines lots should be released by an NRA or NCL; however, in defined exceptional
08
circumstances (e.g., a public health emergency), exemptions could be allowed.
09
Requirement: Lot release regulatory framework and regulations for lot release of all vaccines.

NRA Lot Release (LR): Indicators and Fact Sheets

Evidence to review: The assessor should ask for and review:
1. Legal provisions for lot release;
2. Documented procedures and records to ensure that staff participating in lot release
function contribute, as appropriate, to other regulatory functions;
3. Legal provisions and regulations that require lot release for all vaccines;
4. Legal provisions and regulations for exemptions from lot release exist;
5. Legal provisions and regulations defining fast-track mechanism for lot release;
6. List of products, if any, where the routine procedure for independent lot release may have
not been followed.

References: 1. Guidelines for independent lot release of vaccines by regulatory authorities, (42),
(https://www.who.int/biologicals/WHO_ECBS/en/ and http://apps.who.int/medicinedocs/
en)

Framework: “Structure/Foundation/Input”

Rating Scale:  NOT IMPLEMENTED (NI): There is no legal basis for lot release.
 ONGOING IMPLEMENTATION (OI): The NRA is preparing to establish the legal basis,
but there is no evidence of results from such activities.
 PARTIALLY IMPLEMENTED (PI): The legal basis was established recently and is at the
implementation stage, so this practice is not consolidated yet.
 IMPLEMENTED (I): The NRA has the legal provisions and also consistently maintains
documentation of related activities over time.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: LR01.02: Acceptance policy and criteria for lot release performed by another
NRA are documented.

Maturity Level: 2

Scope: Vaccines

Page 311 of 340

WHO Global Benchmarking Tool (GBT)

Description: If the country does not have the capacity to perform lot release on its own, the assessor
should determine whether the country recognizes certificates from the country of origin or
from other competent NRAs and NCLs.
Different processes are used for implementing this recognition. Examples of these
processes include establishment of a list of countries that are acceptable to the importing
country or creation of mutual recognition agreements. Establishment of mutual recognition
agreements is a legal approach. Many NRAs and NCLs use such agreements to: enhance
international regulatory cooperation in order to maintain high standards of product safety
and quality; reduce the regulatory burden for NRAs and NCLs and manufacturers; and 01
improve the free flow of goods and increase the accessibility of medical products globally.
Reciprocal mutual recognition of release certificates involves a number of legal aspects 02
that should be addressed. However, the key to successful mutual recognition is the building
of mutual confidence among the interested parties. This requires strong collaboration and 03
communication among the different NRAs and NCLs and a good level of transparency.
Situations may exist where a two-way recognition of certificates or test results is not 04
possible, owing to technical or other limitations. However, even in cases where reciprocity
is not attainable, an NRA or NCL may still wish to recognize a release certificate from 05
another NRA or NCL. This should be possible, provided the releasing NRA or NCL has clearly
established procedures that are transparent and relevant to the NRA or NCL wishing to 06
recognize the certificate or test results.
07
Objective: The objective of this sub-indicator is to ensure that, in cases when a lot has already been
released by another NRA or NCL, it may be possible for the NRA to accept that lot for
08
release on the basis of the existing release certificate.
09
Requirement: Recognition of other NRA’s decision

NRA Lot Release (LR): Indicators and Fact Sheets

Evidence to review: The assessor should ask for and review:
1. Documented provisions and criteria for recognition of decisions, reports or certificates
from other authorities;
2. Documentation for the recognition decision process including its rationale and reasoning;
3. List of NRAs or NCLs that are considered acceptable

References: 1. Guidelines for independent lot release of vaccines by regulatory authorities, (42), (https://
www.who.int/biologicals/WHO_ECBS/en/ and http://apps.who.int/medicinedocs/en)

Framework: “Structure/Foundation/Input”

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence of the policies mentioned in the
indicator.
 ONGOING IMPLEMENTATION (OI): The NRA is preparing to draft such policies or
procedures, but there is no evidence of results from such activities.
 PARTIALLY IMPLEMENTED (PI): The policy was established recently and is at the
implementation stage, so this practice is not consolidated yet.
 IMPLEMENTED (I): The NRA has such policies and procedures and also consistently
maintains documentation of related activities over time.

Limitations and remarks:  In some counties, the legal framework prevents the NRA or NCL from recognizing
decisions from other NRAs. In such cases, this sub-indicator should be scored as
non- applicable.
 The product’s manufacturer should be involved in the establishment of an agreement
for sharing product information, since there are issues of confidentiality that need to
be addressed.

Indicator: LR02 Arrangement for effective organization and good governance.

Objective: The objective of this indicator is to establish that structures are in place at the
organizational and governance levels to promote effective intra- and inter-NRA and NCL
relationships and efficient management of information traffic.

Page 312 of 340

WHO Global Benchmarking Tool (GBT)

The quality, safety and efficacy of a medical product, such as a vaccine, are the
responsibility of the manufacturer. The regulatory authority of the country is responsible
for establishing procedures to ensure that this responsibility is met. The same
requirements for regulatory oversight should apply to the production of all vaccines,
regardless of whether they are intended for domestic use or for export.

Category: 01. Legal provisions, regulations and guidelines

Sub Indicator: LR02.01: There is a defined organizational structure with clear responsibilities 01
to conduct independent lot release activities.
02
Maturity Level: 2
03
Scope: Vaccines
04
Description: The assessor should verify that roles and responsibilities for all regulatory entities involved
in the independent lot release are documented and implemented. It is critical that the roles 05
and responsibilities of both the NRA and the NCL are clearly defined, particularly when they
are separate entities. When all elements are available for final evaluation, a formal decision- 06
making process should be in place to decide whether the lot can be released.
07
Objective: The objective of this sub-indicator is to ensure there are appropriate structures in place,
with clearly defined roles and responsibilities for each entity involved in lot release
08
activities.
09
Requirement: Roles and responsibilities in regard to lot release function

NRA Lot Release (LR): Indicators and Fact Sheets

Evidence to review: The assessor should ask for and review:
1. Written document defining organizational structure, with clearly-defined responsibilities,
to conduct independent lot release activities;
2. Documented evidence designating the post holder responsible for signing lot release
certificates;
3. Organization chart including the department, unit, group or post holder responsible for
independent lot release.

References: 1. Guidelines for independent lot release of vaccines by regulatory authorities, (42), (https://
www.who.int/biologicals/WHO_ECBS/en/ and http://apps.who.int/medicinedocs/en)
2. Quality management systems — Requirements. International Standard ISO 9001:2015.
International Organization for Standardization, 2015, (4), (https://www.iso.org/iso-9001-
quality-management.html)

Framework: “Structure/Foundation/Input”

Rating Scale:  NOT IMPLEMENTED (NI): There is no defined structure with clear responsibilities to
conduct independent lot release activities.
 ONGOING IMPLEMENTATION (OI): The NRA is preparing to establish the legal basis,
but there is no evidence of results from such activities.
 PARTIALLY IMPLEMENTED (PI): The legal basis was established recently and is at the
implementation stage, so this practice is not consolidated yet.
 IMPLEMENTED (I): The NRA has defined a structure with clear responsibilities to
conduct independent lot release activities and consistently maintains documentation
of related activities over time.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: LR02.02: Documented procedures are implemented to ensure coordination and
communication among all regulatory entities involved in independent lot release.

Maturity Level: 3

Page 313 of 340

WHO Global Benchmarking Tool (GBT)

Scope: Vaccines

Description: The assessor should review the adequacy of the implemented coordination
mechanisms.
Lot release is one component of a regulatory framework which includes marketing
authorization (MA), good manufacturing practices (GMP) inspection, and post-
marketing surveillance. The relationship between NRA and NCL varies from country
to country, but in all cases, it is essential that the different entities of the regulatory
structure interact and exchange information effectively. 01
Good coordination and communication are needed, especially when different regulatory
entities are involved in this process. 02

Objective: The objective of this sub-indicator is to ensure that documented procedures, structures, 03
and mechanisms are implemented for proper relationships within and among entities
involved in lot release, quality control (QC) laboratory testing, MA and facilities inspection. 04
These coordination mechanisms will ensure effective and efficient exchange of
information for lot release activities. 05

Requirement: Communication among all regulatory entities relevant to independent lot release. 06

Evidence to review: The assessor should ask for and review: 07

1. Documentation that defines role and responsibilities;
2. Documentation for established communication channels (i.e., written documents that
08
describe methodology and procedures);
3. Records of relevant communications;
09
4. Records of regulatory actions taken based on lot release findings

NRA Lot Release (LR): Indicators and Fact Sheets

References: 1. Guidelines for independent lot release of vaccines by regulatory authorities,
(42), (https://www.who.int/biologicals/WHO_ECBS/en/ and http://apps.who.int/
medicinedocs/en)

Framework: “Structure/Foundation/Input”

Rating Scale:  NOT IMPLEMENTED (NI): There are no documented procedures to ensure
coordination and communication among all regulatory entities involved in
independent lot release.
 ONGOING IMPLEMENTATION (OI): The NRA is preparing to establish the
procedure, but there is no evidence of results from such activities.
 PARTIALLY IMPLEMENTED (PI): The procedure was established recently and is at
the implementation stage, so this practice is not consolidated yet.
 IMPLEMENTED (I): The NRA has documented procedures and also consistently
maintains documentation of related activities over time.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Indicator: LR03 Human resources to perform NRA lot release.

Objective: The objective of this indicator is to ensure that all entities within an NRA are adequately
resourced with a trained, experienced and skilled workforce, and are empowered to fully
perform the function of independent lot release. This will ensure that NRA lot release
activities are performed in accordance with international best practices.
The objective of this indicator is to evaluate the human resource capacity of the entities
with respect to the number of personnel, the skills and experience of the personnel,
and the overall composition the workforce, with the goal of evaluating whether the
workforce possesses the specific expertise required to perform the NRA lot release
function.

Category: Resources (HR, FR, infrastructure and equipment)

Page 314 of 340

WHO Global Benchmarking Tool (GBT)

Sub Indicator: LR03.01: Sufficient competent staff (i.e., education, training, skills and experience)
are assigned to perform NRA lot release activities.

Maturity Level: 3

Scope: Vaccines

Description: The assessor should verify that the human resources assigned to perform NRA lot release
activities should be sufficient with respect to numbers and competent with respect 01
to the requisite skills, education, experience and training. There should be technical
documents and standard operating procedures (SOPs) that provide guidance on the 02
required background for NRA lot release activities and that consider the requirements for
educational background, competencies, skills, experience, and training. 03
The assessor should verify that the NRA estimated the number of staff required to
effectively and efficiently perform NRA lot release function and that the NRA actually 04
recruited that number. In addition, the assessor should verify that these competency
requirements are well-established and maintained by the NRA. Metrics and statistics 05
on the different activities performed as well as performance indicators can be used for
estimating the adequacy of the number of the assigned staff. The assessor should also 06
verify that the competency of the assigned staff is built, maintained and improved through
recruitment as well as continuous on-the-job training. 07

Objective: The objective of this sub-indicator is to ensure the existing human resources for NRA
08
lot release is sufficient, in terms of numbers, experience, and specific competencies, to
perform all the activities along the entire NRA lot release chain.
09

NRA Lot Release (LR): Indicators and Fact Sheets

Requirement: Sufficient number of competent human resources in charge of NRA lot release activities.

Evidence to review: The assessor should ask for and review:

1. Evidence that the number of staff members involved in each of the documented
activities along the entire NRA lot release process flow is adequate.
2. Evidence that the systems and structures are in place to ensure appropriate placement
of staff with respect to competence and skills.
3. Evidence that the system and structures have been implemented. The documentation
should include the records to verify that the staff competence is appropriate for the job
requirements.
4. Evidence that the professional profiles of the human resources engaged in NRA lot
release activities are appropriate with respect to education, skills, and expertise, to perform
a particular function along the NRA lot release chain. Documentation should include a list
of the requisite skills and training for each position.
5. Recruitment plan.

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): The NRA does not have enough competent staff (i.e.,
education, training, skills and experience) to perform NRA lot release activities
 ONGOING IMPLEMENTATION (OI): The NRA has recently developed a plan to recruit
adequate competent staff; however, the plan has not been implemented.
 PARTIALLY IMPLEMENTED (PI): The NRA has initiated the implementation of
the human resources development plan; however, there is need to complete the
competency profile.
 IMPLEMENTED (I): The NRA has a sustained number of competent staff (i.e.,
education, training, skills and experience) assigned to perform NRA lot release
activities

Page 315 of 340

WHO Global Benchmarking Tool (GBT)

Limitations and remarks: Assessment of the adequacy and appropriateness of the number of staff members is quite
subjective and should be linked to some process or output indicators. When estimating
staff adequacy, the assessor should consider the workload, backlog, and delays in delivery
based on established timeframes.
Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: LR03.02: Duties, functions, and responsibilities of the staff in charge of NRA lot release
activities are established and updated in the respective job descriptions. 01
Maturity Level: 3 02

Scope: Vaccines 03

Description: The assessor should verify that procedures are in place to maintain a current and updated 04
structure for managing job descriptions for personnel participating in NRA lot release
activities. In addition, job descriptions should address current staff duties, responsibilities 05
and the requisite competencies. A job description with this format and content should be
established and implemented for all staff. The management of job descriptions should 06
be supported by a guidance document that provides direction on when and how to
update the information, and where the information should be kept for easy access. The 07
guidance document should present the appropriate duties and responsibilities that are
assigned to each member of the organization involved in NRA lot release activities. Thus,
08
the professional profiles of staff are reflected in their respective roles and responsibilities
within the NRA. There should be procedures to guide responsible persons to document that
09
duties, functions and responsibilities are revised and kept up to date. In addition, procedures

NRA Lot Release (LR): Indicators and Fact Sheets

should be available to guide the keeping and documenting of up to date work schedules and
enforcing the implementation of the documented guidelines and procedures.

Requirement: Duties, roles and responsibilities of the staff relevant to NRA lot release activities.

Evidence to review: The assessor should ask for and review:

1. Procedure and guidelines that guide placement of staff members within the NRA;
2. The professional profiles of staff (i.e., job descriptions) and documentation that they are
related to their current roles and duties;
3. The professional profiles of the external experts and documentation that the profiles
provide a composition that is complete and consistent with that prescribed in the legal
provisions;
4. Procedures to guide the documentation of up to date duties and work schedules, and to
enforce the implementation of the documented guidelines and procedures;
5. Job descriptions for designated staff.

Page 316 of 340

WHO Global Benchmarking Tool (GBT)

Framework: Structure/Foundation/Input

Rating Scale:  NOT IMPLEMENTED (NI): There is no evidence of defined or established duties,
functions, responsibilities, respective job descriptions and necessary required
competencies.
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted or developed the role
and responsibilities document but it has not yet been implemented.
 PARTIALLY IMPLEMENTED (PI): The NRA has initiated implementation of this
requirement but it has not been defined or followed for all staff or the roles and 01
responsibilities documents, including staff job descriptions, are not up to date.
 IMPLEMENTED (I): The NRA has defined and established all required duties, 02
functions, and responsibilities, and respective job descriptions are up-to-date.
03
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 04

Sub Indicator: LR03.03: Training plan developed, implemented and updated at least once a year 05
for staff in charge of NRA lot release activities.
06
Maturity Level: 3
07
Scope: Vaccines
08
Description: The assessor should verify that training plans are developed, implemented and updated
at least once every year to reflect the current situation by considering education and
09
experience of the staff. The training plan should be complemented with guidelines or

NRA Lot Release (LR): Indicators and Fact Sheets

similar documents that guide the development and implementation of training plans.
The assessor should ensure that induction training for new staff as well as continued
on- the-job training for staff is planned and implemented. There should be procedures
to approve the training plan and the budget allocated for implementing and updating
the training plan. The plans should present clearly defined training goals and should
include training in certain topics and skills to address identified deficiencies. Learning
objectives, training methods and activities, evidence of learning, and evaluation
and assessment of training should be documented. This documentation should
confirm that the learning objectives were achieved and were designed to address
weaknesses within the entities. Procedures should be in place to ensure that a training
plan is developed, implemented and updated at least once every year. The assessor
should verify that there is a system in place for monitoring the implementation and
effectiveness of the training plan and for documenting the skills acquired in training
activities for internal and external experts.

Objective: The objective of this sub-indicator is to ensure that a training plan for staff exists, and
that it is implemented and updated annually. Through the training plan, NRA can be
sure that competency of staff in charge of NRA lot release activities is maintained and
enhanced.

Requirement: Implementation of training plan

Evidence to review: The assessor should ask for and review:

1. Guidelines for development, implementation and annual update (i.e., at least once per
year) of the training plan. Guidelines should also provide for a mechanism to measure
effectiveness of training.
2. Documentation for the system or structures used to approve the training plan and to
evaluate the adequacy of the budget allocated to the training activities.
3. The current or existing staff training plan (or matrix) for staff. The assessor should
assess this in in relation to the respective individual job descriptions.
4. SOP for developing and maintaining the training plan.
5. Evidence that the NRA has investigated and identified training needs.
6. List of trainings performed.
7. Example records for training activities.

Page 317 of 340

WHO Global Benchmarking Tool (GBT)

References: 1. Effective drug regulation: A multicountry study. Geneva: World Health Organization;
2002, (7), (http://apps.who.int/medicinedocs/pdf/s2300e/s2300e.pdf)
2. Quality systems requirements for national good manufacturing practice inspectorates.
In: WHO Expert Committee on Specifications for Pharmaceutical Preparations: thirty-sixth
report. Geneva: World Health Organization; 2002: Annex 8 (WHO Technical Report Series,
No. 902), (8), (http://apps.who.int/medicinedocs/documents/s22112en/s22112en.pdf)

Framework: Process
01
Rating Scale:  NOT IMPLEMENTED (NI): There is no systematic training program including training
plan (or matrix). 02
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted or developed the
training plan but there is no evidence of implementation. 03
 PARTIALLY IMPLEMENTED (PI): The NRA has developed and initiated the training plan
implementation. However, the NRA has not fulfilled all required planned training or has 04
been applying the plan for less than two years.
 IMPLEMENTED (I): The NRA has an updated training plan developed that is 05
supported by adequate records to demonstrate effective plan implementation,
including induction training for new staff and routine on-the-job training for recruited 06
staff.
07
Limitations and remarks: Training plans must be updated regularly; ideally on an annual basis, but not less
08


frequently that once every two years.

Some regulatory functions may include many training activities that are not
09


incorporated in the institutional training programme. Such training normally is offered

by invitation. In this case, the assessor should recognize reports from non-routine

NRA Lot Release (LR): Indicators and Fact Sheets

NRA lot release-relevant training not included in the NRA training plan.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: LR03.04: The NRA generates and maintains records of staff training activities
and training effectiveness verification.

Maturity Level: 3

Scope: Vaccines

Requirement: Training records

Evidence to review: The assessor should ask for and review:

1. Guidelines or similar documents that guide the NRA to generate and maintain records of
staff training activities;
2. Evaluations of training effectiveness;

Page 318 of 340

WHO Global Benchmarking Tool (GBT)

3. The training inventory, and procedures for completing the inventory;

4. Examples of archived records of staff training, and procedures for the archiving
system

NRA Lot Release (LR): Indicators and Fact Sheets

Limitations and remarks:  The assessor should note that some NRAs out-source training including staff
capacity development activities. In this case the assessor should request the identity
of the provider, as well as evaluations of the provider. The assessor may request the
professional profiles of tutors or resource persons used to offer training. The assessor
may also request records covering the archiving systems in place.
 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Indicator: LR04 Procedures established and implemented to perform NRA lot release.

Objective: The objective of this indicator is to ensure documented and implemented procedures to
perform independent lot release.
The lot release of vaccines by regulatory authorities is part of the regulation of vaccines
and involves the independent assessment of each lot of a licensed vaccine before it is
released onto the market. This assessment is based, at a minimum, on the review of
manufacturer’s summary protocols. Lot release may be supplemented by other documents
such as the release certificates from the responsible NRA or NCL. In some circumstances,
this information is also supplemented by product testing that is independent of the
manufacturer’s quality control testing. The summary protocol content should follow
existing guidelines from the World Health Organization Technical Report Series or other
internationally-accepted guidelines. If accepted guidelines are not available, the country
should define the summary protocol template for the product.
Current approaches to conducting lot release of vaccines include: review of the summary
protocols only, review of the summary protocols combined with independent testing (i.e.,
either full or selected testing), and recognition and acceptance of lot release certificates
from the responsible NRA or NCL. These approaches are not mutually exclusive and
different approaches may be used for different products in the same country. It is the

Page 319 of 340

WHO Global Benchmarking Tool (GBT)

responsibility of the NRA or NCL to decide on an appropriate strategy for each vaccine,
taking into consideration the nature of the vaccine, the post-marketing experience for
the vaccine (including production history and safety profile), and the availability of other
independent evidence of product quality.

Category: 07. Regulatory process

Sub Indicator: LR04.01: Independent lot release is based, at a minimum, on summary lot
protocol review and the appropriate documentation exists. 01
Maturity Level: 2 02

Scope: Vaccines 03

Description: The assessor should verify that summary lot protocols are mandatorily required for the lot 04
release of vaccines.
The assessor should make sure the NRA or NCL issue lot release certificates based, at a 05
minimum, on the review of the summary lot protocols issued by the manufacturer. The
protocol review should be performed by comparing the critical data of each lot (including 06
testing data) to the licensed product specifications.
The assessor should verify that the SOP for summary protocol review describes the steps 07
and criteria required for a complete review of the summary protocol. The SOP should cover
all review steps up to and including the final conclusion based on review of the summary
08
protocol. When needed, these steps may include a request for corrections from the
manufacturer and a review of corrected pages. In some cases, the NRA or NCL may elect
09
to conduct an investigation before reaching a conclusion. The NRA or NCL should produce

NRA Lot Release (LR): Indicators and Fact Sheets

a formal written conclusion regarding the summary protocol review. A summary decision
form should be filled out to verify that the product complied with approved specifications.
This summary decision should be signed by the responsible staff.
The assessor should determine whether the competent authority’s approach to
independent lot release is appropriately described in the NRA or NCL process charts.
Procedures should cover the options used: release upon review of summary protocol
only or release upon review of summary protocol plus independent testing by the NCL.
The procedures should also define how and by whom the final decision is made. The
summary decision form should define the specific option used and include a formal written
conclusion. SOPs or documents are necessary to cover the essential elements.

Objective: The objective of this sub-indicator is to ensure that summary lot protocols are mandatorily
required, at a minimum, for vaccines lot release.
The manufacturers’ summary protocols summarize information taken from the production
and QC to ensure that the lot meets the specifications in the approved MA. In addition,
summary protocols submitted to the NRA or NCL should be approved by the person from
the manufacturer who is designated as responsible for quality assurance or QC. In general,
the format and content of the protocol is finalized and approved by the NRA or NCL during
the review of the license application. The format of the protocol should be amended in
response to changes in the approved production process. Amendments to the protocol
should be approved by the NRA or NCL.

Requirement: Documentation for national lot release

Evidence to review: The assessor should ask for and review:

1. SOP or guideline for performing lot release;
2. List of required documents for performing lot release;
3. Relevant written and enforced SOPs developed as part of the quality management
system for reviewing lot release protocols for each product;
4. Examples of lot release records;
5. Documented evidence of evaluation process.

References: 1. Guidelines for independent lot release of vaccines by regulatory authorities, (42), (https://
www.who.int/biologicals/WHO_ECBS/en/ and http://apps.who.int/medicinedocs/en)

Page 320 of 340

WHO Global Benchmarking Tool (GBT)

2. WHO Expert Committee on Biological Standardization (ECBS), Vaccine-specific

standardization, (135), (https://www.who.int/biologicals/vaccines/en/)
3. Guidelines for EU Official Control Authority Batch Release, European Directorate for the
Quality of Medicines and Healthcare (EDQM), Council of Europe, (104), (https://www.edqm.
eu/en/human-ocabr-guidelines)

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): No procedures for lot release based, at a minimum, on 01
summary protocol review are available.
 ONGOING IMPLEMENTATION (OI): The NRA is preparing the procedures but there is 02
no evidence of results from such activities (i.e., reports or certificates).
 PARTIALLY IMPLEMENTED (PI): The procedure was established recently and is at the 03
implementation stage, so this practice is not consolidated yet.
 IMPLEMENTED (I): The NRA has procedures for lot release based, at a minimum, on 04
summary protocol review and consistently maintains documentation of the results
over time. 05

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will 06
always apply for all benchmarked NRAs).
07
Sub Indicator: LR04.02: NRA or NCL staff involved in lot release have access to MA relevant
files and updates.
08

Maturity Level: 3
09

NRA Lot Release (LR): Indicators and Fact Sheets

Scope: Vaccines

Description: The assessor should verify that the NRA or NCL staff involved in lot release has access to
files and updates relevant to MA.
NRA or NCL lot release should be performed only for medical products that have a valid
MA in which specifications have been approved by the competent NRA or NCL of the
country using the vaccine. In addition, the development and adoption of more effective
test methods should be encouraged; however, any changes in testing should be approved
by the NRA or NCL. If a different test method is used by the NRA or NCL, and if there is a
discrepancy in test data between the manufacturer and the NRA or NCL, then the approved
test method defined in the MA should be used to resolve the issue.

Objective: The objective of this sub-indicator is to ensure that specifications as described in the MA
are used to judge the test results. Also, there should be a mechanism in place to allow
the testing staff of the NRA or NCL to be aware of the latest version of the approved MA
specifications.
In the decision-making process for MA, the responsible NRA or NCL staff should be
involved in assessing the test methods, validity criteria and product specifications.

Requirement: Access to MA data for lot release

Evidence to review: The assessor should ask for and review:

1. SOPs that define how MA data are considered for lot release;
2. Records of relevant updated information that was communicated to NRA or NCL.

References: 1. Guidelines for independent lot release of vaccines by regulatory authorities, (42), (https://
www.who.int/biologicals/WHO_ECBS/en/ and http://apps.who.int/medicinedocs/en)

Framework: “Structure/Foundation/Input”

Rating Scale:  NOT IMPLEMENTED (NI): No procedures to define how MA data are considered for
lot release are available.
 ONGOING IMPLEMENTATION (OI): The NRA is preparing a procedure, but there is no
evidence of results from such activities.

Page 321 of 340

WHO Global Benchmarking Tool (GBT)

 PARTIALLY IMPLEMENTED (PI): The procedure was established recently and is at the
implementation stage, so this practice is not consolidated yet.
 IMPLEMENTED (I): The NRA has procedures to define how MA data are considered
for lot release, and also consistently maintains documentation of the results of related
activities over time.

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).
01
Sub Indicator: LR04.03: Analysis of lot-to-lot consistency is conducted.
02

Maturity Level: 3 03

Scope: Vaccines 04

Description: The assessor should verify that statistical analyses are conducted once sufficient data 05
have been accumulated. Alert limits (i.e., warning limits) and action limits should be
defined on the basis of statistical assessments and trend analyses of test data. In general, 06
when data are distributed normally, ±2 and ±3 standard deviations from the mean are set
for the alert limits (i.e., warning limits) and action limits respectively. The variability and 07
precision of the test should be considered when defining the limits. Care should be taken
in interpreting such limits when they are based on small datasets. Trend analyses of key
08
parameters may be requested from manufacturers or from the responsible NRA or NCL.
More complex statistical approaches can be used for trend analyses when sufficient
09
data and expertise are available, particularly when data are not normally distributed. In

NRA Lot Release (LR): Indicators and Fact Sheets

addition, a set of data from a certain period (e.g., 6 months or 1 year) should be analyzed
statistically and compared to data from the previous period, in order to detect any
significant differences, shifts, or trends.
In order to conduct an appropriate trend analysis, it is important to have data from an
adequate number of release batches for each product.
When the NRA or NCL does not receive consecutive lots, or when it receives only a small
number of production lots, interpretation of trends may require additional information (e.g.,
yearly biological product reports).

Objective: The objective of this sub-indicator is to ensure that all critical quantitative data from QC
testing, especially from potency testing, from the manufacturer or other sources are used
for trend analysis as an essential part of lot release.

Requirement: Data monitoring

Evidence to review: The assessor should ask for and review:

1. Procedures and mechanisms employed to ensure lot-to-lot consistency.
2. Documentation that these procedures are performed by NRA or NCL on regular basis.

References:

Framework: Output

Rating Scale:  NOT IMPLEMENTED (NI): No procedures for analysis of lot-to-lot consistency are
available
 ONGOING IMPLEMENTATION (OI): The NRA is preparing a procedure for analysis of
lot-to-lot consistency but there is no evidence of results from such activities.
 PARTIALLY IMPLEMENTED (PI): The procedure was established recently and is at the
implementation stage, so this practice is not yet consolidated.
 IMPLEMENTED (I): The NRA has such procedures and also consistently maintains
documentation of related activities over time.

Limitations and remarks:  Information obtained from lot-to -lot consistency analyses could be used to establish
criteria for testing by the NCL.

Page 322 of 340

WHO Global Benchmarking Tool (GBT)

 Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Indicator: LR05 Mechanism for information-sharing exists to promote transparency

and accountability.

Objective: The objective of this indicator is to ensure that the mechanisms for independent lot
release, including requirements and timelines, are made public in a clear and transparent
way. In addition, these mechanisms could contribute to informing the public about any 01
risks, e.g., with respect to access, availability or shortages of some medical products
subject to national lot release. Public transparency of decision-making and resource- 02
management is one of the principles of truth and accountability.
03
Category: 08. Transparency, accountability and communication
04
Sub Indicator: LR05.01: Results of lot release process are publicly available.
05

Maturity Level: 4 06

Scope: Vaccines 07

Description: The assessor should verify the existence of a list of product batches that pass or fail NRA
08
lot release. In addition, the assessor should verify that the NRA or NCL has the required
authority and an available process to publish the list of product batches that passed or
09
failed NRA lot release. At a minimum, a publicly available website (or another form of

NRA Lot Release (LR): Indicators and Fact Sheets

communication) should be available to publish the list of product batches that passed
NRA lot release. The assessor should determine the frequency at which this information is
updated.

Objective: The objective of this sub-indicator is to ensure that processes and procedures for lot
release decision-making, as well as the list of product batches that pass or fail NRA lot
release, are documented and available to the public.
Transparency enhances public trust, permits the timely application of corrective measures,
prevents public use of unqualified products, and promotes the timely identification of
potential vulnerabilities due to possible acts of corruption.

Requirement: Results of lot release process available

Evidence to review: The assessor should ask for and review:

1. List of product batches that pass or fail NRA lot release;
2. Published list of product batches released;
3. List of product batches released compared with the list of product batches failed;
4. Guideline or SOP that defines the process to publish the list of product batches that
passed or failed NRA lot release.

References: 1. Good governance for medicines: Model framework. Updated version 2014. WHO, (110),
(https://www.who.int/medicines/areas/governance/ggm_modelframe_updated/en/)
2. Guidelines for independent lot release of vaccines by regulatory authorities, (42), (https://
www.who.int/biologicals/WHO_ECBS/en/ and http://apps.who.int/medicinedocs/en)

Framework: Output

Rating Scale:  NOT IMPLEMENTED (NI): No results of lot release process are available.
 ONGOING IMPLEMENTATION (OI): The NRA is preparing a procedure but there is no
evidence of results from such activities.
 PARTIALLY IMPLEMENTED (PI): The procedure or the list was established recently
and is at the implementation stage, so this practice is not yet consolidated.
 IMPLEMENTED (I): The NRA has the list with results of lot release process and
consistently maintains documentation of related activities over time.

Page 323 of 340

WHO Global Benchmarking Tool (GBT)

Limitations and remarks: In some countries, the legal framework does not allow publishing for the public results of
product batches that fail lot release. Nevertheless, in case of emergency or risk, the NRA or
NCL should inform public in timely manner.
There may be specific situations to take into consideration, e.g., during recalls or when
specifications are different in country of origin than in country of use.

Sub Indicator: LR05.02: Follow-up and communication with involved parties, including the
manufacturer, on issues of data quality.
01
Maturity Level: 3
02
Scope: Vaccines
03
Description: The assessor should verify the establishment and implementation of communication
procedures with involved parties, including the manufacturer, on issues of data quality. 04
Good communication with the manufacturer of the product is an important element in
developing an effective system. NCLs should discuss with the manufacturer the transfer 05
of assays, if required. This should begin as early as possible in the MA process, to allow
for transfer, qualification, and validation of the methodology prior to application of the 06
method for lot release testing of the first lot. It is also necessary to establish documented
and approved procedures and guidelines for performance of lot release testing, both for 07
internal use and for transparency with regard to partners, including other NCLs and the
manufacturer of the product.
08
A procedure to communicate QC and national lot release issues should be developed by
the NRA or NCL. These procedures may include formal notifications by memorandum
09
or letter, email communications, or minutes of telephone discussions. Manufacturers’

NRA Lot Release (LR): Indicators and Fact Sheets

responses should be reviewed and documented when making the decision on the lot. This
response can include submission by the manufacturer of a corrected page or revision of
the summary protocol. These corrections should then be properly traced by the NRA or
NCL as per good documentation practices. Depending upon the nature and severity of the
discrepancies or errors, the manufacturer may be asked to perform an investigation to
determine the root cause of the issues, and to initiate any corrective and preventive actions
required to avoid similar problems in the future.
A feedback mechanism from the NCL to the NRA, the GMP inspectorate and the MA staff
is highly advisable, in order to coordinate and optimize regulatory actions (e.g., encouraging
license variations or refinements in product specification based on trend analyses).

Objective: The objective of this sub-indicator is to ensure that any discrepancies, errors or out-
of- specification results found in the summary protocol submitted are documented and
verified before they are communicated to the manufacturer. The manufacturer should be
notified when an out-of-specification result is confirmed, and exchanges should ensue to
try to identify the cause of the discrepancy.

Requirement: Follow-up and communication on issues of data quality

Evidence to review: The assessor should ask for and review:

1. Evidence that the NRA or NCL has a suitable mechanism to follow up and communicate
on issues related to data quality with all involved parties such as the manufacturer,
importer, wholesaler or user of the products.
2. Written documents that describe the process;
3. Records of any actions.

References: 1. Guidelines for independent lot release of vaccines by regulatory authorities, (42), (https://
www.who.int/biologicals/WHO_ECBS/en/ and http://apps.who.int/medicinedocs/en)

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): No procedures or reports are available regarding follow-up
and communication with involved parties, including the manufacturer, on issues of
data quality.

Page 324 of 340

WHO Global Benchmarking Tool (GBT)

 ONGOING IMPLEMENTATION (OI): The NRA is preparing a procedure for this but there
is no evidence of results from such activities.
 PARTIALLY IMPLEMENTED (PI): The procedure was established recently and is at the
implementation stage, so this practice is not consolidated yet.
 IMPLEMENTED (I): The NRA has such procedures and also consistently maintains
records of the results of related activities over time.

Limitations and remarks: For imported lots, communication with the NRA of the producing or releasing country
may be required. For producing or releasing countries, communication with the country 01
inspectorate may be required. Such information exchange can help to evaluate the
corrective and preventive actions introduced by the manufacturer. When needed, 02
confidentiality issues should be taken into consideration during the communication
process. 03
In some countries, communications among government, manufacturer and NRA are used,
for planning purposes, for all batch release processes and not only for product analysis. 04
Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs). 05

Indicator: LR06 Mechanism in place to monitor regulatory performance and output. 06

07
Objective: The objective of this indicator is to ensure the implementation of a system or
mechanism for monitoring regulatory performance and output of the independent lot
08
release function.
09
Category: 09. Monitoring progress and assessing outcomes and impact

NRA Lot Release (LR): Indicators and Fact Sheets

Sub Indicator: LR06.01: Lot release records, reports and certificates available.

Maturity Level: 3

Scope: Vaccines

Description: The assessor should verify that there is requirement to issue a certificate of release for
all vaccines that have undergone lot release. This release certificate is issued by the
responsible NRA or NCL on the basis of, at a minimum, a review of the lot summary
protocol for the relevant lot. The assessor should verify that lot-to-lot consistency is
appropriately analyzed and documented by the NRA or NCL on regular basis.

Objective: The objective of this sub-indicator is to ensure that all documentation related to
independent lot release is available and readily accessible to internal staff whenever
needed. A general lot release process chart that outlines the lot approval process and
identifies the persons responsible for each activity should be available. Documentation
supporting compliance with approved specifications (i.e., summary protocol review and
test reports, if applicable) should also be available.

Requirement: Documentation system

Evidence to review: The assessor should ask for and review:

1. SOP describing the acceptance criteria for NCL test results;
2. Records of all the individual test results from the certificates of analysis (i.e., for all
lots that have undergone lot release);
3. Records of evaluation of lot release including reports and certificates;

References: 1. Guidelines for independent lot release of vaccines by regulatory authorities,

(42), (https://www.who.int/biologicals/WHO_ECBS/en/ and http://apps.who.int/
medicinedocs/en)

Framework: Output

Page 325 of 340

WHO Global Benchmarking Tool (GBT)

Rating Scale:  NOT IMPLEMENTED (NI): No lot release records, reports or certificates are available.
 ONGOING IMPLEMENTATION (OI): The NRA is preparing a procedure for lot
release but there is no evidence of results from such activities (i.e., reports or
certificates).
 PARTIALLY IMPLEMENTED (PI): The procedure was established recently and is at
the implementation stage, so this practice is not yet consolidated yet.
 IMPLEMENTED (I): The NRA has procedures, records, reports and certificates
for lot release and also consistently maintains documentation of related activities
over time. 01
Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will 02
always apply for all benchmarked NRAs).
03
Sub Indicator: LR06.02: Corrective actions taken in case of deviations due to laboratory error.
04

Maturity Level: 3 05

Scope: Vaccines 06

Description: The assessor should review that appropriate corrective actions are taken by the NCL 07
in case of deviations (i.e. significant discrepancy between NCL testing results and
manufacturer results due to laboratory or operator error and not due to product quality).
08
Depending upon the nature and severity of the discrepancies or errors, the NCL may be
asked to perform an investigation to determine the root cause of the issues, including
09
steps for corrections, corrective actions and/or preventive actions to avoid recurrence or

NRA Lot Release (LR): Indicators and Fact Sheets

Requirement: Corrective and preventive actions system

Evidence to review: The assessor should ask for and review:

1. Evidence for availability of procedures for corrections, corrective actions and preventive
actions;
2. Examples of corrections, corrective actions and preventive actions.
3. Evidence for implementation of corrections, corrective actions and preventive actions.

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): No procedures or reports regarding corrections, corrective
actions and continued improvements are available.
 ONGOING IMPLEMENTATION (OI): The NRA is preparing a procedure for this, but
there is no evidence of results from such activities.
 PARTIALLY IMPLEMENTED (PI): The procedure was established recently and is at the
implementation stage, so this practice is not consolidated yet.
 IMPLEMENTED (I): The NRA has such procedures and also consistently maintains
documentation of related activities over time.

Page 326 of 340

WHO Global Benchmarking Tool (GBT)

Limitations and remarks: A selected sample of yearly biological product reports should be reviewed as well. This is
a report that is submitted annually to the NRA or NCL by manufacturers and that contains
production information on both bulk and final lots. The report should include test methods
and results, reasons for any recalls and corrective actions taken, and any pertinent post-
marketing information.
Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: LR06.03: Regulatory action taken in case of product non-compliance. 01

02
Maturity Level: 3
03
Scope: Vaccines
04
Description: The assessor should verify that in case of non-compliances, the NRA or NCL should
confirm, through appropriate laboratory investigation, that the non-compliant results 05
reflect the quality of the lot tested and is not due either to an analytical error by the NCL or
to the influence of variables unrelated to the product. Once confirmed, the manufacturer 06
should be notified as soon as possible with prompt exchanges of information to try to
identify the cause of the discrepancy through manufacturer investigation. Test reports, 07
including the results and outcomes of all of testing performed, should be prepared. These
test reports should be used in the final evaluation and decision-making process for the
08
lot (or lots) under consideration. A feedback mechanism from the NCL to the NRA, the
GMP inspectorate, and the MA staff is highly advisable in order to coordinate and optimize
09
regulatory actions (e.g., batch recall, product withdrawal, MA revocation, encouraging

NRA Lot Release (LR): Indicators and Fact Sheets

license variations, or refinements in product specifications based on trend analyses). For
imported lots, communications between the NRAs or NCLs of the producing and releasing
country may be required. For producing and releasing countries, communications with the
country inspectorate may be required. Such information exchange can help to evaluate the
corrective and preventive actions introduced by the manufacturer.

Objective: The objective of this sub-indicator is to ensure that the NRA or NCL has an implemented
procedure dealing with the actions to be taken when test results do not comply with the
specifications. These procedures should also cover the appropriate regulatory actions to
be taken.

Requirement: Non-compliance action

Evidence to review: The assessor should ask for and review:

1. Written documents that describe regulatory actions (e.g., revoking of authorization,
recalling of product, or stopping of importation) to be taken in case of non-compliance.
2. Records of actions.

References: 1. Guidelines for independent lot release of vaccines by regulatory authorities, (42), (https://
www.who.int/biologicals/WHO_ECBS/en/ and http://apps.who.int/medicinedocs/en)
2. Quality management systems — Requirements. International Standard ISO
9001:2015. International Organization for Standardization, 2015, (4), (https://www.iso.
org/iso-9001-quality-management.html)

Framework: Process

Rating Scale:  NOT IMPLEMENTED (NI): No procedures or reports regarding regulatory actions
taken in cases of non- compliance are available.
 ONGOING IMPLEMENTATION (OI): The NRA is preparing a procedure for this,
but there is no evidence of results from such activities.
 PARTIALLY IMPLEMENTED (PI): The procedure was established recently and is
at the implementation stage, so this practice is not consolidated yet.
 IMPLEMENTED (I): The NRA has such procedures and also consistently maintains
records of the results of related activities over time.

Page 327 of 340

WHO Global Benchmarking Tool (GBT)

Limitations and remarks: Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Sub Indicator: LR06.04: Performance indicators for national lot release activities are established
and implemented.

Maturity Level: 4

Scope: Vaccines 01
Description: The assessor should verify the existence and implementation of performance indicators 02
for different activities included under the national lot release functions.
Specifically, the system should define key performance indicators (KPIs) along the entire 03
national lot release activity chain and all indicators should be adequately justified. For the
purpose of clarity and consistency, established KPIs should be supported with guidelines 04
for monitoring and maintenance of the KPIs. The guidelines in turn should be supported
by SOPs and tools that define the procedures to be used for monitoring and evaluating the 05
performance indicators and that define procedures and timelines for reviewing and revising
the indicators. 06
Examples of performance indicators for national lot release activities include, but are not
limited to: number of released lots per year, number of rejected lots per year, and average 07
number of days to reach a decision on the release of the received batches.
Established KPIs might be qualitative, quantitative or combination of both. In general,
08
quantitative indicators are preferred to avoid bias or misinterpretation. However, qualitative
indicators are also accepted. Qualitative indicators may or may not include scoring
09
or scaling to render them semi-quantitative and thus more informative. The assessor

NRA Lot Release (LR): Indicators and Fact Sheets

should ensure that indicators are measured on a regular basis to monitor progress and
advancement.
In addition, the assessor should verify measured indicators are analyzed to identify trends
or abnormalities. Justifications for any identified abnormalities should be provided; when
necessary, process optimizations should be introduced to avoid recurrence.

Objective: The objective of this sub-indicator is to ensure that a system, mechanism, or procedure
exists to require the NRA to establish performance indicators along the entire national lot
release chain. Additionally, the objective is to ensure that KPIs are actually contributing to
monitoring of regulatory performance, to measuring effectiveness of national lot release
regulatory activities, and to making any necessary adjustments or optimizations.

Requirement: KPIs for national lot release activities

Evidence to review: The assessor should ask for and review:

1. Documents supporting the system, mechanism, or procedure compelling the NRA
to establish and implement performance indicators along the entire national lot release
activity chain.
2. Evidence that the performance indicators have been established and implemented,
and that the members of staff involved in the national lot release function are aware
of the indicators and the guidelines and SOPs used for monitoring and evaluating their
performance.
3. The current performance indicators for national lot release activities
4. Analyses of the measured indicators along with the investigations done to identify
trends or abnormalities.
5. Documentation for follow-up of any observed abnormalities, including justifications
for any identified abnormalities as well as any process optimizations introduced to avoid
recurrence.

References: 1. Quality management systems — Requirements. International Standard ISO 9001:2015.

International Organization for Standardization, 2015, (4), (https://www.iso.org/iso-9001-
quality-management.html)
2. World Health Organization. European Observatory on Health Systems and Policies.
Performance measurement for health system improvement: experiences, challenges

Page 328 of 340

WHO Global Benchmarking Tool (GBT)

and Prospects, (125), (https://www.who.int/management/district/performance/

PerformanceMeasurementHealthSystemImprovement2.pdf)
3. World Health Organization. European Observatory on Health Systems and Policies.
Health System Performance Comparison: an agenda for policy, information and research,
(126), (http://www.euro.who.int/__data/assets/pdf_file/0009/244836/Health-System-
Performance-Comparison.pdf)
4. Quality management -- Quality of an organization -- Guidance to achieve sustained
success, ISO 9004:2018, International Organization for Standardization (ISO), (115),
(https://www.iso.org/standard/70397.html ) 01
Framework: Output 02

Rating Scale:  NOT IMPLEMENTED (NI): There are no KPIs for national lot release activities. 03
 ONGOING IMPLEMENTATION (OI): The NRA has recently drafted KPIs for national lot
release activities but they have not yet been reported. 04
 PARTIALLY IMPLEMENTED (PI): The NRA has developed KPIs for national lot release
activities and has been applying them for less than two year or they have not covered 05
all critical steps.
 IMPLEMENTED (I): The NRA has established and implemented KPIs for national lot 06
release activities. The indicators are reviewed regularly, and appropriate actions are
taken, and decisions made. 07

Limitations and remarks: When they refer to outcomes, indicators may be ambiguous and difficult to interpret, as
08
outcomes are the result of many factors that are difficult to disentangle. When they refer to
processes, indicators are often too specific, as they may focus on a particular intervention
09
or condition or they may quickly become outdated as business models develop.

NRA Lot Release (LR): Indicators and Fact Sheets

Different methodologies are used to measure the NRAs performance on national lot
release activities. In this case, the assessor should verify that adequate supporting
documents are available. The assessor should consider that developed performance
indicators should be Specific, Measurable, Achievable, Realistic, and Time- bound (i.e.,
“SMART”).
Scoring this sub-indicator as “not applicable NA” is excluded (i.e. this sub-indicator will
always apply for all benchmarked NRAs).

Page 329 of 340

WHO Global Benchmarking Tool (GBT)

Glossary and Definitions

The definitions given below apply to the terms as used in different fact sheets of the World Health Organization
(WHO) global benchmarking tool (GBT) for evaluation of national regulatory system of medical products. These
terms may have different meanings in other contexts.

Accountability
The result of the process which ensures that health actors take responsibility for their obligations and are made
answerable for their actions.

Advisory/Scientific committee
See expert advisory body

Benchmarking
A formal process of evaluation of a process or system, preferably quantitative, but sometimes necessarily
qualitative.

Benchmark
A measurement or point of reference at the beginning of an activity which is used for comparison with
subsequent measurements of the same variable.

Commitment
In accounting usage, commitments refer to a stage in the expenditure process at which contracts or other
forms of agreement are entered into, generally for future delivery of goods or services. A liability will not be
recognized until delivery of the item, but the government is contractually committed to meeting the obligation
once delivery is made. The term is also used in a more general, non-contractual sense to mean firm promises of
the government made in policy statements.

Competency
Competency combines knowledge, skills and attitude. Competencies describe how the work is to be carried
out while objectives indicate what must be accomplished. They also provide a sound basis for consistent
and objective performance standards by creating a shared language for what is needed and expected by the
organization.

Clinical Trials Oversight (CT)

NRAs should have the legal mandate to authorize regulate and, if necessary, terminate clinical trials (CTs). The
necessary requirements, guidelines, procedures and forms should be developed to be in line with country and
region-specific guidelines as well as major international CT guidance including guidelines from the Declaration
of Helsinki, the Nuremberg code, International Council on Harmonization, and World Health Organization
Good Clinical Practices. CT oversight is aimed at protecting the safety and rights of humans participating in
CTs, ensuring that trials are adequately designed to meet scientifically sound objectives, and preventing any
potential fraud and falsification of data.

NRAs are responsible at two stages for the critical evaluation of the documentation supporting clinical studies:
when CTs are being proposed for authorization and when the results are submitted in an application for
marketing authorization. CT protocols should be reviewed and approved by Independent Ethics Committees
before the trial commences. A CT review committee should review the protocols and should have the authority,
when necessary, to require protocol revisions. The CT review committee should be composed of members who
have the appropriate medical and scientific knowledge, experience and skills and who are free of conflicts of
interest.

In order to ensure the quality and safety of investigational products, the investigational products should be
manufactured in compliance with GMPs for investigational medical products, and the supporting preclinical

Page 330 of 340

WHO Global Benchmarking Tool (GBT)

studies should be in compliance with Good Laboratory Practices. Additionally, the importation, storage, use,
and/or destruction of investigational products should follow national requirements. Qualified and experienced
inspectors should carry out on-site inspections of the CT sites to verify compliance with Good Clinical Practices,
ethical principles and regulatory requirements, and to provide assurance of the quality and reliability of the data
obtained. The oversight activities should be conducted with due concern for confidentiality.

The legal provisions should allow the NRA to recognize and/or rely on relevant CT decisions, reports and
information from other NRAs or from designated regional and international bodies. In special circumstances
(e.g., for public health interest), the legal provisions should allow the NRA to elect not to follow the routine CT
procedures. Transparency in the entire oversight process is fundamental to ensuring the safety of patients and
to ensuring that no product with unacceptable benefit to risk balance will be made available to the public.

Decentralization
Political reform designed to promote local autonomy; decentralization entails changes in authority and
financial responsibility for health services. Hence, decentralization can have a large impact on health service
performance. There are several forms of decentralization affecting the health sector in different ways: (i)
decentralization, which transfers authority and responsibility from the central level of the Ministry of Health
(MOH) to its field offices; (ii) delegation, which transfers authority and responsibility from the central level
of the MOH to organizations not directly under its control; (iii) devolution, which transfers authority and
responsibility from the central level of the MOH to lower level autonomous units of government; and (iv)
privatization, which involves the transfer of ownership and government functions from public to private bodies.
These bodies may consist of voluntary organizations and for-profit and not-for-profit private organizations, with
varying degree of government regulation. (http://www.who.int/health- laws/topics/governance-decentralisation/
en/ )

Effectiveness
The extent to which a specific intervention, procedure, regimen or service, when deployed in the field in routine
circumstances, does what it is intended to do for a specified population.

Efficacy
The extent to which a specific intervention, procedure, regimen or service, produces the intended result under
ideal conditions.

Efficiency
The capacity to produce the maximum output for a given input.

Emergency
Emergency is a term describing a state. It is a managerial term, demanding decision and follow-up in terms of
extra-ordinary measures (Oxford Pocket Dictionary, 1992). A “state of emergency” demands to “be declared”
or imposed by someone in authority, who, at a certain moment, will also lift it. Thus, an emergency is usually
defined in time and space, requires threshold values to be recognized, and implies rules of engagement and an
exit strategy. Conceptually, it relates best to Response.

Entities
In this document, entities refer to different composition units within the NRA (e.g., divisions, departments,
or sections), or to different institutions, organizations, or bodies which are involved in the national regulatory
system.

Ethics committee (also called research ethics committee or independent ethics committee)
An independent body (i.e., an institutional, regional or national review board or committee), constituted of
medical professionals and non-medical members, whose responsibility is to verify that the safety, integrity and
human rights of the subjects participating in a particular trial are protected and to consider the general ethics
of the trial thereby providing public reassurance. Ethics committees should be constituted and operated so that
their tasks can be executed free from bias and from any influence of those who are conducting the trial.

Page 331 of 340

WHO Global Benchmarking Tool (GBT)

Event
A specific identifiable happening or occurrence, e.g., the taking of a medicine or the experience of an adverse
effect.

Expert advisory body

An advisory board or committee of experts, including academic experts and practicing health care
professionals.

Falsified medical products

Medical products that deliberately and fraudulently misrepresent their identity, composition or source.

Findings
See inspection observation

Good governance
See governance

Good Clinical Practice (GCP)

A standard for the design, conduct, performance, monitoring, auditing, recording, analysis and reporting of
clinical trials or studies. The standard provides assurance that the data and results that are reported are
credible and accurate.

Good review practices

Documented best practices for any aspect related to the process, format, content and management of a
medical product review.

Governance
Governance includes (i) the exercise of political, economic and administrative authority in the management of
a country’s affairs at all levels, comprising the complex mechanisms, processes, relationships and institutions
through which citizens and groups articulate their interests, exercise their rights and obligations and mediate
their differences. (ii) the traditions and institutions by which authority in a country is exercised for the common
good, including the processes by which those in authority are selected, monitored and replaced; the capacity of
the government to effectively manage its resources and implement sound policies; and the respect of citizens
and the state for the institutions that govern economic and social interactions among them; (iii) the process
of creating an organizational vision and mission (i.e., what it will be and what it will do) in addition to defining
the goals and objectives that should be met to achieve the vision and mission; of articulating the organization,
its owners and the policies that derive from these values; (iv) policies concerning the options that its members
should have in order to achieve the desired outcomes; and adopting the management necessary for achieving
those results and a performance evaluation of the managers and the organization as a whole. See stewardship.

Guidelines or guidance documents

Non-statutory advisory publications intended to assist those parties affected by legislation to interpret and
apply requirements.

Impact
Impact includes (i) the total, direct and indirect, effects of a programme, service or institution on a health status
and overall health and socioeconomic development; (ii) the positive or negative, long-term or medium-term
effects produced by a programme or intervention; and (iii) the degree of achievement of an ultimate health
objective.

Independent ethics committee (IEC)

See Ethics committee.

Input
A quantified amount of a resource put in a process.

Page 332 of 340

WHO Global Benchmarking Tool (GBT)

Inspection observation
A finding or statement of fact made during an inspection and substantiated by objective evidence. Such
findings may be positive or negative. Positive observations should take the form of a description of the
processes that the firm is carrying out particularly well and that may be considered as examples of particularly
good practice. Negative observations are findings of noncompliance with requirements.

Investigational product
Any pharmaceutical product (i.e., a new product or a reference product) or placebo being tested or used as a
reference in a clinical trial.

Key performance indicators (KPIs)

Factors that are under the control of the organization and are critical for its sustained success subject to
performance measures

Legislation
The first stage of the legislative process, in which laws are passed by the legislative body of government with
regard to a subject matter (e.g., control of pharmaceuticals). Laws define the roles, rights and obligations of all
parties involved in the subject matter in general terms (see regulations).

Laboratory Testing (LT)

The laboratory testing regulatory function is intended to ensure that the NRA is able to assess the quality of
medical products by performing quality tests on them in certain situations. For example, this testing can be a
requirement to corroborate manufacturer’s test results as a part of the evaluation for marketing authorization
or for a variation to a marketing authorization. Testing can be a requirement for lot release for certain products
depending upon national regulations. Testing also may be needed for products for which there has been a
complaint or a report or for products that are under investigation due to an adverse event. As part of the market
surveillance function, laboratory testing is utilized for checking and confirming the quality of medical products
placed on the market and for detecting substandard and falsified medical products. In order to do this product
testing, the NRA must have access to suitable laboratories where these tests can be performed.

If a country is able to provide all the resources needed, a laboratory under the responsibility of the NRA or a
governmental laboratory represents the best choice. Commonly, this governmental laboratory is a national
control laboratory (NCL). As an alternative option, the regulatory system may have access to external
laboratories, either inside or outside the country, to perform the required tests on behalf on the NRA. When
external laboratories are used, regulatory decisions and actions remain at the discretion of the NRA and the
NRA retains accountability.

A well-functioning laboratory for medical products testing is an important resource for the national regulatory
system. The staff generally has expertise in different life science disciplines and can help in other regulatory
activities, for example, the assessment and review of marketing authorization applications and the review of
clinical trial data.

When the regulatory laboratory testing activities are decentralized to one or more internal or external
laboratories, the NRA or NCL must ensure that a continuous information exchange mechanism is established
so the central authority can issue guidance and the decentralized entity can report back with the information
needed for making decisions.

Licensing Establishments (LI)

In order to protect public health, licensing activities are of outstanding importance and are considered
fundamental, together with inspections activities, for guaranteeing the quality, safety and efficacy of medical
products used within or exported out of the country. The NRA is responsible for coordinating licensing activities
and should be supported by published and readily available legal provisions, regulations and guidelines which
ensure that licensing of facilities throughout the supply chain is based on compliance with Good Practices
(GXPs) and that the NRA is empowered to issue, suspend or revoke licenses for premises and establishments.

Page 333 of 340

WHO Global Benchmarking Tool (GBT)

Premises, facilities, establishments and companies throughout the supply chain should possess a license
to operate issued by the NRA. These facilities include, but are not limited to, manufacturers, distributors,
wholesalers, importers, exporters and retailers. The process of issuing licenses should be based on
the implementation of and compliance with quality standards of GXP. An inspection for confirmation
of compliance with GXP is required in order to grant or re-grant a license or approval of a substantial
modification.

An updated list or database of all licensed facilities should be published and publicly available.

The GXPs considered most relevant for this function are good manufacturing practices and good distribution
practices, including good cold chain management practices. Good clinical practices and good vigilance
practices, which are generally excluded from this function, are addressed, however, in other functions.

A general limitation to this function occurs when there is no domestic manufacturing of medical products. In
this case, the function cannot apply to manufacturers, because none exist in the country. Nevertheless, the
function will always apply to distribution practices, including wholesaling. Another general limitation applies
to those countries that depend on the regulatory inspection function, without licensing, to ensure compliance
to GXPs at the premises, facilities, establishments and companies throughout the supply chain. In the latter
case, the whole function might not apply; however, the assessor of this establishment licensing function
should liaise with the assessor of the regulatory inspection function to verify that proper and appropriate
controls are in place despite the absence of any licensing activities in the country.

Market Surveillance and Control (MC)

Market surveillance and control function plays a crucial role in assuring medical products consumer safety
since its objective is to ensure compliance of the products placed on the market with pre-set criteria for
quality, safety and efficacy (i.e., verify compliance with marketing authorization and GXP guidelines). Market
surveillance and control function activities are primarily concerned with four themes: (1) control of import
activities, (2) prevention and detection of and response to substandard and falsified medical products, (3)
market surveillance program for monitoring the quality of medical products throughout the supply chain, and
(4) control of promotional, marketing and advertising activities. The aforementioned activities may or may not
be undertaken by a single entity (e.g., organization, division, or department).

A general limitation to one of the market surveillance and control function activities exists in countries where
advertisement is not allowed or is restricted to specific medical products (e.g. over-the-counter medical
products). However, in all cases, control of marketing and promotional materials (i.e., in its wider meaning that
includes promotion to healthcare professionals) will always apply.

Medical products
In this document, a term that includes medicines and vaccines.

NRA Lot Release (LR)

NRA lot release (also called official authority batch release) is a non-common regulatory function that does
not apply to all medical products. Lot release is a system specifically established for the regulatory release of
specified biological products. The goal of the regulatory function is to ensure the quality, safety and efficacy
of biological products through a regulatory release system. Lot release is done on a lot-by-lot basis and takes
into account the nature and inherent variability of these products.

NRAs should have the legal mandate to perform independent lot release. They should develop and implement
the necessary policies, guidelines, procedures and forms in line with WHO and major international guidelines.

In the case of vaccines, different approaches are currently used for conducting lot release. The options
include: review of the summary protocols only, review of the summary protocols combined with independent
testing (i.e., either full or selected testing), and recognition and acceptance of lot release certificates from the
responsible NRA or NCL. The NRA or NCL has the responsibility to decide on an appropriate strategy for each

Page 334 of 340

WHO Global Benchmarking Tool (GBT)

vaccine subject to NRA lot release. The decision should take into consideration the nature of the vaccine, the
post-marketing experience for each product (including production history and safety profile), and the availability
of other independent evidence of product quality.

National Regulatory System (RS)

The National Regulatory System (RS) provides the framework that supports the WHO recommended regulatory
functions. The NRA is the institution in charge of assuring the quality, safety, and efficacy of medical products
as well as ensuring the relevance and accuracy of product information. A sustainable, well-functioning
regulatory system will ensure an independent and competent oversight of medical products.

Non-conformity
Refers to a failure to comply with requirements. A requirement is a need, expectation or obligation. It can
be stated or implied by an organization, its customers or other interested parties. There are many types of
requirements. These include quality requirements, customer requirements, management requirements, product
requirements, process requirements and legal requirements. Whenever an organisation fails to meet one of
these requirements, a nonconformity occurs.

Observations
See inspection observation

Output
The quantity and quality of activities carried out by a programme.

Performance indicators
Measurable values used to quantify quality objectives to reflect the performance of an organization, process or
system, also known as “performance metrics” in some regions.

Performance monitoring
The continuous process of collecting and analysing data to compare how well a project, program, or policy is
being implemented against expected results.

Post-Marketing
The stage when a drug is approved and generally available on the market.

Promotion
All informational and persuasive activities by manufacturers and distributors, the intended effect of which is to
induce the prescription, supply, purchase and/or use of medicinal products. Ethical criteria for drug promotion,
WHO, 1988. For the purposes of this manual, promotion includes advertising.

Public health emergency of international concern (PHEIC)

Defined in the International Health Regulation (2005) as “an extraordinary event which is determined, as
provided in these Regulations:

to constitute a public health risk to other States through the international spread of disease; and

to potentially require a coordinated international response”.

This definition implies a situation that is serious, unusual or unexpected, carries implications for public health
beyond the affected State’s national border, and may require immediate international action.

Recognition
The routine acceptance by the NRA in one jurisdiction of the regulatory decision of another NRA or other trusted
institution. Recognition indicates that evidence of conformity with the regulatory requirements of country A is
sufficient to meet the regulatory requirements of country B. Recognition may be unilateral or multilateral, and
may be the subject of a mutual recognition agreement.

Page 335 of 340

WHO Global Benchmarking Tool (GBT)

Regulations
The second stage of the legislative process (the first stage being legislation, see above). Regulations are
specifically designed to provide the legal machinery to achieve the administrative and technical goals of
legislation.

Registration and Marketing Authorization (MA)

The issuance of marketing authorizations (MAs, also referred as product licensing or registration) is critical to
any NRA. MA refers to a procedure for approval of a medical product for marketing after it has undergone a
process of evaluation to determine the safety, efficacy and quality of the product and the appropriateness of
the product information. The objective of this regulatory function is to provide a system which ensures that
only medical products which have been duly authorized by the NRA are allowed to be manufactured, imported,
distributed, sold or supplied to end-users. The process of assessment for MA includes the review of data on
quality, safety and efficacy submitted by the applicant. The same standards should be applied to imported
and locally manufactured medical products. Nevertheless, the evaluation of the complex data used to support
market authorization of new or novel medical products may require specialized resources and experience not
available in the NRA; therefore, countries with limited resources may wish to give priority to well-established
products.

Mechanisms should exist to ensure that information on MA applications, including authorized, suspended,
rejected or completed applications, is published to promote transparency and information sharing among
stakeholders. Regular publication of MA decisions is helpful to procurement and distribution networks and to
other NRAs. Mechanisms and procedures should be in place to monitor and ensure that all activities within the
MA function are checked to reduce errors and to reasonably ensure that the processes are consistent and will
provide assurance of high-quality outputs. Such approaches would lead to consistency in the performance of
the MA regulatory function and to reliability of the regulatory outputs.

Regulatory framework
The collection of laws, regulations, guidelines, and other regulatory instruments through which a government
controls medical products manufacture, clinical evaluation, marketing, promotion and post-marketing safety
benchmarking.

Reliance
The act whereby the regulatory authority in one jurisdiction may take into account and give significant weight to
(i.e., totally or partially rely upon) evaluations performed by another regulatory authority or trusted institution in
reaching its own decision. The relying authority remains responsible and accountable for decisions taken, even
when it relies on the decisions and information of others.

Research ethics committee

See ethics committee.

Regulatory Inspection (RI)

Inspection of establishments across the medical product supply chain is an essential regulatory function.
The supply chain includes manufacturers, distributors, re-packagers, re-labelers, importers, agents, traders,
wholesalers and retailers of medical products. The purpose of regulatory inspections is to ensure that
operations at these establishments are carried out in accordance with approved standards, norms, and
guidelines and are in compliance with the national medical products legislation and regulations. These, in turn,

Page 336 of 340

WHO Global Benchmarking Tool (GBT)

should be consistent with WHO recommendations and other internationally recognized guidelines. The scope
of the function applies to different GXPs and is not limited to GMPs. Good Distribution Practices and GCPs
also come under the scope of this function. Good Vigilance Practices are not addressed in this function but are
addressed under the vigilance function.

NRAs should have the legal mandate to inspect and enforce GXPs throughout the supply chain, to make
decisions concerning the issuance, suspension or withdrawal of establishment licenses, and to issue
authorizations or certifications for the activities performed by these establishments. Additionally, the NRA
should develop policies, regulatory actions and procedures on the handling of medical products with suspected
quality defects and medical products identified as substandard and falsified. Commonly, inspectors perform
several types of inspections: pre-licensing or post-licensing, pre-approval or post-approval, announced or
unannounced, and domestic or overseas inspections.

Inspection activities should be conducted based on a risk management approach. Inspections reveal
weaknesses and deficiencies, as well as actual or potential errors in the production, quality control, storage or
distribution of medical products. Therefore, inspection activities are fundamental for guaranteeing the quality,
safety and efficacy of medical products used by the population. An appeal system that is independent of the
body that made the initial decision should be available.

The credibility of the inspection depends on the transparency and clarity of the process, on the absence of
conflicts of interest, and on the availability of regulations, directives, guidelines and procedures related to the
quality management and assurance system of the inspectorate. The system also is highly dependent on the
technical competence and integrity of the inspectors. The inspectorate must also ensure confidentiality of the
information obtained in the course of its inspection activities.

Networking with other international bodies and NRAs is an important method for acquiring, sharing, and
exchanging information relevant to the quality and safety of medical products; in turn, this information
contributes to informed science-based decisions. The inspectorate of the NRA should follow uniform
procedures incorporating quality system principles.

A general limitation for this function exists in countries where no domestic manufacturing capacities exist.
In this case, domestic regulatory inspections based on GMP are not applicable; however Good Distribution
Practices would always apply. Similarly, GCP inspections might not apply in countries where no clinical trials
are conducted. Even in the absence of a domestic pharmaceutical manufacturer, a GMP inspection function is
needed. GMP compliance is always a requirement for granting marketing authorization for a medical product.
In this case, GMP compliance can be assured through overseas inspections or desk assessments, which may
or may not include verification of documentation. Thus, even if there is no domestic manufacturer, there will be
a need for a function which can ensure the GMP compliance of foreign pharmaceutical manufacturers either
by physical inspection or desk review (e.g., taking into consideration the GMP certificates from a stringent
regulatory authority).

Regulatory system (national regulatory authority or NRA)

The system composed of entities responsible for the registration, marketing authorization and other regulatory
functions concerning medical products. The number of regulatory entities responsible for different regulatory
functions may vary from one country to another (i.e., NRA may or may not be a single entity). The terms
national medicines regulatory authority and drug regulatory authority are also used, however less encouraged,
for designating the NRA.

Safety signal
See signal

Signal
A hypothesis of a risk with a medicine, with various levels of evidence and arguments to support it. The
complexity of the signal detection process cannot easily be captured in a single, precise definition. In addition to

Page 337 of 340

WHO Global Benchmarking Tool (GBT)

detecting previously unknown risks with medicines, signal detection should aim to find and communicate any
important and relevant information that adds to previous safety knowledge about a medicine, including also risk
factors and at-risk groups, details of severity, time at risk, and duration of adverse effects.

Stakeholders
An individual, group or an organization that has an interest in the organization and delivery of health care.

Stewardship
The very essence of good government, the careful and responsible management of the well-being of the
population. Stewardship includes: health policy formulation (i.e., defining the vision and direction of health
system), regulation (i.e., setting fair rules of the game with a level playing field) and intelligence (i.e., assessing
performance and sharing information). See governance.

Substandard and falsified products (SF)

See Substandard medical products and falsified medical products

Substandard medical products (also called “out of specification”)

Authorized medical products that fail to meet either their quality standards or their specifications, or both.

Summary of product characteristics (SPC)

Product information as approved by the regulatory authority, and which may have different names globally (e.g.,
also known as the ‘product label’). The summary of product characteristics serves as the basis for production
of information for health personnel as well as for consumer information on labels and leaflets of medicinal
products and for control of advertising (see also Product information).

Transparency
Defining policies and procedures in writing and publishing the written documentation and giving reasons to the
public.

Unregistered or unlicensed medical products

Medical products that have not undergone evaluation and approval by the national or regional regulatory
authority for the market in which they are marketed, distributed or used, and for which marketing, distribution
and use is subject to permitted conditions under national or regional regulation and legislation.

Vigilance (VL)
Medical products vigilance, defined as the science and activities relating to the detection, assessment,
understanding and prevention of adverse effects or any other medical product-related problems, is extremely
important for guaranteeing that safe and effective medical products of high quality are used within the country.
Vigilance activities should be established in the countries based on a risk management approach.

A reporting system should be established to monitor the safety of medical products. One important activity
within that function is to monitor and assess side effects and other product-related safety issues (e.g., adverse
drug reactions for medicines and adverse events following immunization for vaccines). While common side
effects are likely to be detected during pre-approval clinical trials (phases I. II and III), rare events are more likely
to be observed after the marketing of medical products. Other unexpected events may also be due to errors and
thus could occur at any time during product development and marketing. Side effects may differ with respect
to severity, causes and public health consequences. Hence, it is advised that each country establish its own
vigilance system. A vigilance system, in general, monitors all kinds of patient harm potentially related to medical
products, be it due to inadequate product quality, inappropriate use (e.g., medication errors) or intrinsic adverse
effects.

Serious effects (e.g., adverse events following immunization) often lead to public concerns and could erode the
confidence in medical products and the overall regulatory and health systems. If not dealt with adequately, such
concerns could have significant negative implications on the public health. A post-marketing vigilance system
of medical products is therefore essential.

Page 338 of 340

WHO Global Benchmarking Tool (GBT)

Networking with other international bodies and regulators is a logical method for acquiring, sharing, and
exchanging the relevant information on medical products safety. This information, in turn, contributes
to informed science-based decisions. To facilitate networking and exchange of information with other
international bodies and regulators, harmonization across countries of the vigilance systems and safety
reporting requirements in accordance with internationally agreed standards, is expected.

Vigilance events
See event

Page 339 of 340

WHO Global Benchmarking Tool (GBT)

Page 340 of 340

01 GBT RS RevVI PDF
No ratings yet
01 GBT RS RevVI PDF
99 pages
Trs1025-Annex13 WHO Technical Report 2020
No ratings yet
Trs1025-Annex13 WHO Technical Report 2020
55 pages
Building Regulatory Capacity in Countries To Improve The Regulation of Health Products
No ratings yet
Building Regulatory Capacity in Countries To Improve The Regulation of Health Products
38 pages
Certification Scheme
No ratings yet
Certification Scheme
28 pages
Drug Regulation Harmonization Efforts
No ratings yet
Drug Regulation Harmonization Efforts
5 pages
WHO - OMS - 2012 - Assessment Criteria For National Blood Regulatory Systems
No ratings yet
WHO - OMS - 2012 - Assessment Criteria For National Blood Regulatory Systems
40 pages
Zimbabwe's Regulatory Review: Challenges & Opportunities
No ratings yet
Zimbabwe's Regulatory Review: Challenges & Opportunities
16 pages
WHO Pharmaceuticals Newsletter No 3 2023 1693146010
No ratings yet
WHO Pharmaceuticals Newsletter No 3 2023 1693146010
18 pages
WHO Expert Committee On Biological Standardization: WHO Technical Report Series
No ratings yet
WHO Expert Committee On Biological Standardization: WHO Technical Report Series
100 pages
Annex 3 - Who Guidelines On Evaluation of Biosimilars - 22 Apr 2022
No ratings yet
Annex 3 - Who Guidelines On Evaluation of Biosimilars - 22 Apr 2022
44 pages
Newsletter: WHO Pharmaceuticals
No ratings yet
Newsletter: WHO Pharmaceuticals
27 pages
WHO TRS 1054 Annex-1
No ratings yet
WHO TRS 1054 Annex-1
117 pages
WHO Expert Committee On Specifications For Pharmaceutical Preparations
No ratings yet
WHO Expert Committee On Specifications For Pharmaceutical Preparations
430 pages
Inf 37
No ratings yet
Inf 37
110 pages
TRS 1039 - 9789240046870 Eng
No ratings yet
TRS 1039 - 9789240046870 Eng
178 pages
Good Governance Medicine
No ratings yet
Good Governance Medicine
27 pages
Immpleminting Quality Mangement System
No ratings yet
Immpleminting Quality Mangement System
64 pages
Standardization WHO
No ratings yet
Standardization WHO
118 pages
Manual For Performance Evaluation of Regulatory Authorities Seeking Designation As WHO-listed Authorities
No ratings yet
Manual For Performance Evaluation of Regulatory Authorities Seeking Designation As WHO-listed Authorities
272 pages
Annex 3 - Who Guidelines On Evaluation of Biosimilars - SJ Ik 5 May 2022
No ratings yet
Annex 3 - Who Guidelines On Evaluation of Biosimilars - SJ Ik 5 May 2022
44 pages
Perte Auditive
No ratings yet
Perte Auditive
43 pages
Who MHP Hps Eml 2021.02 Eng
No ratings yet
Who MHP Hps Eml 2021.02 Eng
66 pages
WHO Pharmaceuticals Newsletter No.3 2022
No ratings yet
WHO Pharmaceuticals Newsletter No.3 2022
21 pages
Guidelines Developing National Drug Policies: World Organization
No ratings yet
Guidelines Developing National Drug Policies: World Organization
56 pages
Engineering Part of Pharmacy
No ratings yet
Engineering Part of Pharmacy
54 pages
Who Trs 993 Web Final
No ratings yet
Who Trs 993 Web Final
284 pages
WHO Quality Local Production Training Workshop
No ratings yet
WHO Quality Local Production Training Workshop
23 pages
Informe 45 de La Oms
No ratings yet
Informe 45 de La Oms
292 pages
03 GBT VL Rev-VI Ver 1 Nov 2018 Final Adjusted
No ratings yet
03 GBT VL Rev-VI Ver 1 Nov 2018 Final Adjusted
48 pages
Who Essential Drugs
No ratings yet
Who Essential Drugs
66 pages
Inf. 47 TRS 981
No ratings yet
Inf. 47 TRS 981
202 pages
Who Expert Committee On Biological Standardization: Fifty-Fifth Report
No ratings yet
Who Expert Committee On Biological Standardization: Fifty-Fifth Report
146 pages
Good Regulatory Practice
No ratings yet
Good Regulatory Practice
48 pages
The Selection and Use of Essential Medicines: WHO Technical Report Series
No ratings yet
The Selection and Use of Essential Medicines: WHO Technical Report Series
940 pages
Essential Public Health Functions, Health Systems, and Health Security
No ratings yet
Essential Public Health Functions, Health Systems, and Health Security
74 pages
Harrison Internal Medicine
No ratings yet
Harrison Internal Medicine
681 pages
Eng
No ratings yet
Eng
480 pages
Good Regulatory Practices
No ratings yet
Good Regulatory Practices
30 pages
WHO - TRS - 834 - Informe 37
No ratings yet
WHO - TRS - 834 - Informe 37
73 pages
WHO Benchmarks For Strengthening Health Emergency Capacities
No ratings yet
WHO Benchmarks For Strengthening Health Emergency Capacities
400 pages
WHO TRS 999 Corrigenda Web PDF
No ratings yet
WHO TRS 999 Corrigenda Web PDF
292 pages
WHO EMP MAR 2009.4 Eng PDF
No ratings yet
WHO EMP MAR 2009.4 Eng PDF
196 pages
Listado de Medicamentos Esenciales OMS 2025
No ratings yet
Listado de Medicamentos Esenciales OMS 2025
72 pages
Trs 1052
No ratings yet
Trs 1052
382 pages
WHO Tr -1024 2020年
No ratings yet
WHO Tr -1024 2020年
247 pages
Who Emp qsm2008.3
No ratings yet
Who Emp qsm2008.3
57 pages
World Health Organization - INGLES
No ratings yet
World Health Organization - INGLES
17 pages
Working Together For Health 2006
No ratings yet
Working Together For Health 2006
237 pages
Inf. 49 TRS 992
No ratings yet
Inf. 49 TRS 992
228 pages
WHO TRS 992 Web
No ratings yet
WHO TRS 992 Web
228 pages
WHO Study Group On Tobacco Product Regulation
No ratings yet
WHO Study Group On Tobacco Product Regulation
179 pages
Good Reliance Practices in The Regulation of Medical
No ratings yet
Good Reliance Practices in The Regulation of Medical
31 pages
TGS2 Estabilishing Stability of An in Vitro Diagnostic For WHO Prequalification
No ratings yet
TGS2 Estabilishing Stability of An in Vitro Diagnostic For WHO Prequalification
55 pages
Family Laws in Pakistan1
No ratings yet
Family Laws in Pakistan1
37 pages
Affidavit of Truth: No Contract Exists
No ratings yet
Affidavit of Truth: No Contract Exists
4 pages
MLC 2006 Regulation 2.5 - Seafarers Repatriation Requirements
No ratings yet
MLC 2006 Regulation 2.5 - Seafarers Repatriation Requirements
3 pages
Import Licensing Procedures
No ratings yet
Import Licensing Procedures
55 pages
Model UN Procedure Guide
No ratings yet
Model UN Procedure Guide
5 pages
Labor Law Case Digest Compilation
100% (1)
Labor Law Case Digest Compilation
65 pages
Form 9 Particulars of Directors and Officers
No ratings yet
Form 9 Particulars of Directors and Officers
4 pages
SIC Constitution and Powers
No ratings yet
SIC Constitution and Powers
3 pages
Philosophical Justice and Exchange
No ratings yet
Philosophical Justice and Exchange
2 pages
CASE OF CHURCH OF GREECE v. GREECE
No ratings yet
CASE OF CHURCH OF GREECE v. GREECE
23 pages
3 Years' LL.B. Revised Syllabus July 2024
No ratings yet
3 Years' LL.B. Revised Syllabus July 2024
117 pages
Barry Shaffer v. State Farm Mutual Automobile I, 3rd Cir. (2016)
No ratings yet
Barry Shaffer v. State Farm Mutual Automobile I, 3rd Cir. (2016)
9 pages
GDPR Compliance
100% (6)
GDPR Compliance
53 pages
Legal Motions and Pre-Trial Brief
No ratings yet
Legal Motions and Pre-Trial Brief
6 pages
Case Analysis On State of Haryana Vs State of Punjab: Submitted by
No ratings yet
Case Analysis On State of Haryana Vs State of Punjab: Submitted by
5 pages
Damnum Absque Injuria Doctrine
No ratings yet
Damnum Absque Injuria Doctrine
2 pages
Essential Elements of Sales Contracts
No ratings yet
Essential Elements of Sales Contracts
4 pages
Land Reforms in Pakistan
No ratings yet
Land Reforms in Pakistan
3 pages
Contract Import Shablon
No ratings yet
Contract Import Shablon
4 pages
Module 2 E-Text - Universal Origins of IP
No ratings yet
Module 2 E-Text - Universal Origins of IP
28 pages
Glossary of Terms
No ratings yet
Glossary of Terms
13 pages
Articles of Incorporation
No ratings yet
Articles of Incorporation
3 pages
Governance Handout-1 (Class-1) by Mr. Sahil Goyal
No ratings yet
Governance Handout-1 (Class-1) by Mr. Sahil Goyal
6 pages
Constitutional and Legal System of East Africa-Open University Manual
No ratings yet
Constitutional and Legal System of East Africa-Open University Manual
273 pages
Wright Vs CA
No ratings yet
Wright Vs CA
15 pages
The Beggar Maid Court Dilly PDF Download
100% (1)
The Beggar Maid Court Dilly PDF Download
37 pages
Notice of Public Records Correction - PDF Version 3
75% (4)
Notice of Public Records Correction - PDF Version 3
2 pages
Mutual Nda. Fermatagro - Individual
No ratings yet
Mutual Nda. Fermatagro - Individual
4 pages
The Queen of The Dark Things by C Robert Cargill Extract
0% (1)
The Queen of The Dark Things by C Robert Cargill Extract
13 pages
Regional Trial Court 4 Judicial Region Branch 42
No ratings yet
Regional Trial Court 4 Judicial Region Branch 42
3 pages