Kimberly C.

Molato August 2, 2010

BSN 16 Mr. Jose Isaak Calamlam

Sherlock Holmes is a movie essentially structured as an action- adventure film.

Moreover, it seems that this movie has a lot of science to inspect. Specifically, the evil
Lord Blackwood has a few more tricks up to his sleeve. Dipping into chemistry,
pentagrams and other earliest forms of electricity, the plot is all over the place and I’m
not entirely certain I understand how it all fit together in the end.

At the end of the movie, Sherlock debunks Blackwood’s “magic”. First, Sir
Thomas, Blackwood’s father was killed by a poisonous compound that's activated by
water, bath powder and copper from the bath tub. The compound wasn't traced
because the police drained the water from the tub. Second, the U.S. ambassador
caught on fire by a combination of a substance on his clothes and a spark from firing his
gun. Third, those who were on Blackwood's side wouldn't have suffered if the cyanide
gas had been released in Parliament because they drank the antidote on the night that
the ambassador died. As it happens, there is a natural and easily obtained antidote to
cyanide poisoning: Vitamin B12. Regarding on the radio control apparatus with which
Blackwood intended to activate his device, the setting was in between 1887 and 1892.
Physicist Heinrich Hertz experimented with the first transmission and detection of radio
waves in 1886 and 1887. Therefore, Blackwood might be able to obtain a device, either
through successful contact with Hertz or through another inventor unknown to history.

About Blackwood’s faked death, it had three parts: 1) Hanging not by his neck,
but by a hook hidden in the rope attached to a body harness. 2) Ingesting a substance
called rhododendron (another ingredient found in the dwarf's lab). 3) Blackwood's tomb
wasn't destroyed from the inside; it was shattered in pieces, glued together and
shattered again. Holmes figured this out by examining the ingredients of the dwarf's
experiments and the tomb. The first and third are matters of good preparation. He paid
off his executioner, and had his minions break the stone in advance. The second one
turned out to be one of the examples of just how the writers of this movie did their
research. A substance found in rhododendron plants, called grayanotoxin that produces
a syndrome called “Mad Honey Disease”, just as Holmes labels it. Two of the symptoms
are low blood pressure and bradycardia (slowed heartbeat). Thus it is entirely possible
that he could seem to have no pulse.

That does it for Blackwood. There are other two things I wanted to emphasize.
The first one is the case of the counterclockwise circling flies. Early in the movie,
Holmes gets drunk off surgical anesthetic and spends some quality time with a jar of
flies and his violin. He discovers that while a chromatic scale has no effect, a sixth chord
causes all of them to fly in circles. Now, is there any effect of music of any sort of
insects of any type? The other one is When Holmes and Watson enter the lab of the
chemist that helped Lord Blackwood with all of his illusions, Holmes reasons that the
room "smells of sodium phosphate, among other aromas". I wonder if Sodium
phosphate is an aromatic compound.

Well, anyway, Sherlock Holmes deals more with science. It is good to know that
this is a kind of movie wherein it will make the audience puzzle out the mysteries behind
every scene. The story is very amazing. At first I thought it was the “black magic” of
Lord Blackwood that makes the story enigmatical only to find out that science is the
reason behind it.
 WHO Drug Information Vol. 30, No. 2, 2016

Concept paper for discussion

A stepwise approach for pharmaceutical companies in
developing countries to attain WHO GMP standards

This concept paper aims to provide a risk-based, phased approach for

development of a country-specific, achievable roadmap towards WHO GMP
for the manufacture of finished pharmaceutical products in low- and middle-
income countries. The concept paper is based on the experience of the United
Nations Industrial Development Organization (UNIDO) in developing and
implementing national level GMP roadmaps. It describes the concept, the tools
that have been developed and the process utilised for implementation. It is
proposed for discussion on how policy makers and regulators can mitigate risk
to public health during transition of an industry to GMP compliance based on a
risk-based, phased roadmap.
Comments and suggestions on this paper are invited to facilitate further
discussion. They should be sent to druginfo@who.int.

Introduction States only “about 20% are known to have

Background
Adherence to Good Manufacturing Practices
(GMP) is essential for consistent quality
assurance of medicinal products and is
important for ensuring their consistent safety
and efficacy. However, due to the lack of
financial, technical and human resource
capacities, pharmaceutical manufacturers in
developing countries are often overwhelmed
by the vast array of GMP requirements and
therefore fail to operate in line with such
internationally acceptable standards. The
situation is fuelled by the fact that regulatory
authorities in many developing countries
cannot meet the demands associated with
internationally acceptable GMP standards.
Of the more than 190 WHO Member
well developed drug regulation” whereas
“30% either have no drug regulation in
place or а very limited capacity that hardly
functions” (1). As a result, pharmaceutical
companies located in developing countries
frequently feature operating environments
and procedures that fall below standards
that should ultimately be acceptable. Due
to the lack of unified quality requirements,
individual companies trying to improve their
manufacturing standards are struggling to
remain competitive in the low-priced market
while many manufacturers are discouraged
from making the necessary investments
that are required to upgrade. According to
WHO low- and middle-income countries
bear the greatest burden of substandard
products (2), whereby the rate of substandard

This document has been prepared by Kay Weyer, Ph.D., UNIDO Senior Pharmaceutical Manufacturing
Expert (Consultant). It is largely based on the document listed as Reference 10 on page 196.
This article is published in the interest of receiving feedback and initiating discussion. The views presented
in this paper remain the sole responsibility of the authors. Publication of this material in WHO Drug
Information does not necessarily mean endorsement of its content by WHO.

186
WHO Drug Information Vol. 30, No. 2, 2016 UNIDO concept paper for discussion
products can even exceed 60% in particular
countries for certain life-saving drugs such
as anti-malarials (3). The use of substandard
medicines can lead to harmful and even
lethal consequences including therapeutic
failure, drug resistance or toxicity.
Thus, there clearly exists an urgent need
for improvement of existing manufacturing
standards. However, there are significant
challenges to raising standards. A central
requirement to address the multitude of
issues is the establishment of a stepwise
technical pathway towards GMP compliance
based on an assessment of the current
situation within a country.
Purpose
This concept paper explains UNIDO’s
approach to developing a country-specific,
achievable roadmap towards internationally
acceptable GMP standards, such as those
issued by the World Health Organization
(WHO). The paper points out the need for a
risk-based, phased roadmap towards WHO
GMP compliance and explains required steps
and tools for its development.
The purpose of this document is to provide
the technical aspects of developing a
stepwise, phased, and risk-based approach
for pharmaceutical manufacturers to reach
full WHO GMP compliance. This roadmap
shall set the path for the industry in individual
countries to progress within a specified
period of time to compliance with the
internationally acceptable GMP standards
defined by WHO1.
The document also highlights the
various benefits that such a GMP roadmap
approach has for the pharmaceutical sector
in developing countries. Finally, this paper
provides an example of the successful
1
The GMP standard referred to in this document
is as outlined in (4). Updated WHO GMP
guidelines are published in the WHO Technical
Report Series (TRS). The most recent version is
TRS 986, Annex 2 (5).
application of the roadmap concept in
a developing country as evidenced by
the implementation of the Kenya GMP
Roadmap (6).
This document should be read in
conjunction with the respective WHO GMP
guidelines mentioned in Footnote 1.
Scope
This document focuses on WHO GMP
requirements for the manufacture of
medicines in their finished dosage forms.
Although the concept was initially
developed for the manufacture of non-sterile
dosage forms containing small molecular
entities, the strategic approach presented
in this paper can be applied to various GMP
environments as long it is ensured that
internationally recognized GMP guidelines
are utilized as reference standard and that
risks resulting from existing manufacturing
practices are adequately mitigated during
the progress of companies from existing
practices to full compliance with GMP.
GMP roadmap concept
Need for a risk-based, phased approach
towards WHO GMP
Observations from GMP inspections
performed by WHO at manufacturers of
medicinal products in developing countries
revealed a high number of deviations
from WHO GMP, including some critical
deficiencies posing a potential risk of
harm to patients (7). GMP compliance
assessments conducted recently by
UNIDO as part of the GMP roadmap
work resulted in the observation of similar
deficiencies, underlining the urgent need
for improvement of existing manufacturing
standards. For the majority of pharmaceutical
manufacturers in developing countries the
gap between WHO GMP requirements
and current manufacturing practices is
187
UNIDO concept paper for discussion WHO Drug Information Vol. 30, No. 2, 2016
substantial. Therefore, the transition from
current manufacturing practices to full
compliance with WHO GMP standards is a
time-consuming process which cannot be
achieved overnight. In order for the upgrading
approach to be realistic and achievable,
a stepwise, phased pathway with clearly
defined milestones and targets at the end of
each phase should be developed, guiding the
pharmaceutical sector from the status quo to
the targeted WHO GMP compliance.
While developing such an approach, it
is essential to identify those areas of WHO
GMP where companies are least compliant.
These areas pose the biggest threat to the
quality, safety and efficacy of the medicinal
products manufactured and therefore have
to be addressed with priority in order to avoid
exposing patients to preventable risks.
In summary, this highlights that to ensure
an achievable and hence realistic pathway
towards full WHO GMP compliance the
roadmap approach has to be
• risk-based, focusing first on those areas
of WHO GMP with which least compliance
exists and which are hence posing the
highest risk to the quality, safety and
efficacy of medicines manufactured; and
• structured in phases allowing a stepwise
transition to full WHO GMP compliance
with clearly defined targets at the end of
each phase.
Steps for a risk-based, phased approach
towards WHO GMP compliance in a
specific country
Step 1: Baseline assessment of existing
manufacturing practices
The baseline assessment is the starting
point for the development process leading
to a GMP roadmap. During the baseline
assessment field studies are performed on
a sample of pharmaceutical manufacturers
which have not yet achieved full compliance
188
with WHO GMP. Thereby it has to be
ensured that the sample of pharmaceutical
manufacturers selected for the baseline
assessment is representative of the various
levels of compliance with WHO GMP within a
country.
WHO GMP is a highly suitable GMP
reference standard as it is based on
unified principles and practices agreed by
the world’s leading regulatory agencies
and hence receives wide international
acceptance. Besides, many pharmaceutical
manufacturers in developing countries strive
to achieve compliance with WHO GMP as
part of the requirements for having their
products prequalified by WHO.
It is essential that this baseline assessment
is well prepared and conducted thoroughly,
as its results provide the basis for the specific
design of the GMP roadmap. Therefore,
unified tools have to be developed and
applied equally to all pharmaceutical
manufacturers participating in the baseline
assessment in order to ensure transparency
and consistency of obtained results. These
tools include 1) the definition of key elements
and focus areas during assessments;
2) preparation of an assessment schedule
to be applied to all companies; and 3) the
definition of rating of observations.
WHO GMP can be divided into 17 key
areas which are called “key quality elements”,
listed below.
1. Pharmaceutical Quality System
2. Utilities impacting Good Manufacturing
Practices (GMP)
3. Sanitation and hygiene
4. Qualification and validation
5. Complaints
6. Product recalls
7. Contract production, analysis and other
activities
8. Self-inspection, quality audits and
suppliers’ audits and approval
9. Personnel
WHO Drug Information Vol. 30, No. 2, 2016 UNIDO concept paper for discussion
10. Training
11. Personal hygiene
12. Premises
13. Equipment
14. Materials
15. Documentation
16. Good practices in production
17. Good practices in quality control
For each of these key quality elements the
assessment focus has to be defined. Based
on the defined key quality elements and
focus areas, an assessment schedule is
prepared which is uniformly applied to all
companies. In order to allow for a thorough
assessment while at the same time avoiding
too lengthy a time period for the field study, it
is recommended that the assessment of each
company takes two full days. Deficiencies
of individual companies observed during
the assessment are rated using a standard
rating scheme of “critical”, “major”, “other”, as
outlined for example in the compilation of EU
“Community Procedures on Inspections and
Exchange of Information” (8).
Step 2: Evaluation of assessment results
and identification of common main
technical challenges
In order to evaluate the level of compliance
with WHO GMP and to identify the main
technical challenges across the range of
pharmaceutical companies within individual
countries, two tools have been developed:
1. Identification of key quality elements
affected by highest and lowest compliance
with WHO GMP; and
2. Risk categorization of companies based on
their compliance with WHO GMP.
Tool 1: Identification of key quality
elements affected by highest and lowest
compliance with WHO GMP
Using the plain ratings of individual
observations made during each company
assessment would not be suitable to
identify common main challenges across
the pharmaceutical sector in a given
country. Rather, a tool is required to
compare individual companies in terms of
their compliance with WHO GMP and to
identify those key quality elements where
highest and lowest compliance rates are
observed. Therefore, a rating scheme has
been developed that enables aggregation
of individual observations related to a
specific key quality element so as to reflect
its composite compliance with WHO GMP
requirements. The rating scheme comprises
the following three levels:
• Compliance of a key quality element with
WHO GMP is rated “acceptable” if no or
only “other” (i.e. “minor”) deficiencies have
been observed in areas related to this
specific key quality element.
• Compliance of a key quality element with
WHO GMP is rated “requires improvement”
(short: “improve”) if only few “major”
deficiencies (n ≤ 5) were observed in areas
related to this specific key quality element.
• Compliance of a key quality element with
WHO GMP is rated “inadequate” if at
least one “critical” and/or a considerable
number (n > 5) of “major” deficiencies are
observed in the respective area or if the
entire key quality element is not available
at a company.
This rating key makes it possible to compare
company performances and to identify those
key quality elements for which highest and
lowest compliance has been observed. In
this way the main technical challenges for
compliance can be identified. The rating
key is a useful tool to evaluate particular
weaknesses in compliance of pharmaceutical
manufacturers within a country.
The described evaluation tool can also
be used for trending of GMP compliance
of companies and for monitoring their
development towards full WHO GMP
compliance throughout the implementation of
the roadmap.
189
UNIDO concept paper for discussion WHO Drug Information Vol. 30, No. 2, 2016
Tool 2: Risk categorization of companies
based on their compliance with WHO GMP
GMP compliance can be understood as the
result of compliant structural and compliant
organizational measures. In this paper the
term “site” applies to the physical entity of
mainly premises, utilities and equipment
used for pharmaceutical manufacturing. The
term “quality management system (QMS)” is
applied for all documentation systems and
procedures used by a company to ensure
GMP compliance. The interconnection
between site, QMS and GMP is illustrated in
Figure 1.
The risk classification uses a matrix to
categorize companies based on the two
risk-indicating factors for GMP compliance:
1) Compliance of site with WHO GMP
standards; and 2) Compliance of quality
management system with WHO GMP
standards.
A score of “1”, “2” or “3” is assigned to
both the site and the quality management
system to describe their respective degree
of compliance with WHO GMP. A score
of “3” represents high compliance-related
risk whereas a score of “1” indicates low
compliance-related risk.
A matrix is used for combining these two
scores in order to generate an estimate of the
overall compliance-related risk associated
with a pharmaceutical manufacturer
(Table 1). The resulting risk ratings are “A”,
Figure 1: Interconnection between Site, QMS and GMP
Good manufacturing practice
(GMP)
Site
Physical entity
mainly:
▪▪ Premises
▪▪ Utilities
▪▪ Equipment
190
“B” and “C”, with a rating of “C” representing
a high- risk company and a rating of “A”
indicating a low risk company.
In order to increase the transparency
and objectivity of the scores given for the
compliance of site and QMS with WHO GMP,
indicator criteria have been defined which are
presented in Annex 1.
This risk categorization is a suitable
tool for benchmarking GMP compliance
of companies and can also be used in
conjunction with “Tool 1” to monitor the
companies’ progress in the upgrading
process towards full WHO GMP compliance.
Additionally, the tools presented above
can be utilized by individual pharmaceutical
manufacturers in the context of a gap
analysis and in order to prioritize and
streamline corrective and preventive actions
(CAPA).
Step 3: Design of a GMP roadmap based
on evaluation results
Based on the evaluation outcomes a
risk-based, phased GMP roadmap can
be designed. Tool 1 identifies the key
quality elements for which the most
severe deficiencies versus WHO GMP
exist and hence identifies the main
technical challenges for the sector within
the country which need to be addressed
with highest priority. Tool 2 allows one to
determine whether the main reason for
Quality
management
system
(QMS):
▪▪ Systems
▪▪ Procedures
WHO Drug Information Vol. 30, No. 2, 2016 UNIDO concept paper for discussion

Table 1: Risk matrix for the categorization of companies based on their GMP compliance
Quality Management System (QMS)
3 – No QMS in 2 – Requirements 1 – A systematic
place are implemented approach in line
sporadically only; a with WHO GMP
systematic approach in place and
to GMP is not in place implemented
1 – Site is in general compliant with
WHO GMP
C B A
2 – Site shows significant
Site deficiencies versus WHO GMP, but C B B
production safety is not impaired
3 – Site unsuitable for
pharmaceutical manufacturing C C C
→ production safety impaired
A: Existing approach towards pharmaceutical manufacturing in general in line with WHO GMP
requirements → low-risk company
B: Existing approach towards pharmaceutical manufacturing not in line with WHO GMP but reduced risk
with regard to production safety → medium-risk company
C: Existing approach towards pharmaceutical manufacturing not in line with WHO GMP and high risk with
regard to production safety → high-risk company

low compliance with WHO GMP is caused
by site- or QMS-related aspects of GMP,
which helps to streamline the upgrading
approach. Furthermore, this tool allows one
to characterize the currently predominating
level of compliance-related risk associated
with pharmaceutical manufacturers within
a country, and provides guidance in
determining the number of phases needed
to achieve full compliance with WHO GMP.
If the predominantly existing compliance-
related risk of the pharmaceutical companies
in a country is rated as class “C” (i.e.
predominance of high risk companies with
inadequate manufacturing standards and
procedures impairing production safety)
at least 2 main phases will be needed to
gradually improve from the existing level
to full WHO GMP compliance: Phase I
from level “C” to “B” will primarily focus on
reducing the risk to production safety, Phase
II from level “B” to “A” will aim to achieve full
compliance with WHO GMP. In this context,
it is well acknowledged that depending on
the outcome of the evaluation it might be
advisable to further divide the main phases
into sub-phases. The content of those (sub-)
phases will be primarily defined by the
outcome of the compliance assessment of
the key quality elements, with the first phase
focusing particularly on those elements
that show the most severe deviations from
WHO GMP. Whether the first phase will put
emphasis on site- or QMS-related GMP
aspects will depend on the outcome of the
company risk assessment to the extent
that a distinct trend of compliance-related
risk distribution between the two aspects is
revealed.
As the individual phases of the GMP
roadmap are defined according to the
severity of deficiencies versus WHO GMP
and the compliance-related risk observed at
pharmaceutical manufacturers, the evaluation
results are instrumental in realizing a
stepwise, risk-based approach towards
achievement of full WHO GMP compliance.

191
UNIDO concept paper for discussion WHO Drug Information Vol. 30, No. 2, 2016
Benefits of a risk-based, phased roadmap
towards WHO GMP
A risk-based, phased approach towards
WHO GMP compliance has many benefits
for the pharmaceutical sector of developing
countries, including the following.
• The development of a risk-based, phased
roadmap results in an achievable and
scientifically sound pathway towards
internationally acceptable GMP standards,
which will eventually lead to a significant
reduction of substandard medicines.
• A step-wise transition of pharmaceutical
manufacturing practices towards a unified,
internationally acceptable quality standard
following clearly defined requirements,
activities and milestones ensures the
presence of a level playing field throughout
the phases of the roadmap.
• The risk-based, phased roadmap ensures
that all stakeholders have the same
understanding of GMP throughout each of
the transition phases,
–– demystifying requirements of WHO
GMP and hence leading to an increased
willingness to implement WHO GMP by
the industry, and
–– increasing transparency during licensing
procedures and regulatory GMP
inspections and hence strengthening
regulatory authorities.
A well-defined risk-based, phased roadmap
will enable:
• already existing companies to perform a
gap analysis between their current GMP
compliance and WHO GMP requirements
and to follow a stepwise approach towards
WHO GMP compliance;
• new start-up companies to assure that all
necessary elements and systems are taken
into consideration and are in place before
the actual launch of the company; and
• the regulatory authority to review licensing
criteria for new and existing facilities,
192
allowing existing companies to improve
gradually until they are in line with WHO
GMP requirements and ensuring that new
companies comply with WHO GMP before
their licensing.
Outlook on aligning the approach with
other stakeholders’ activities
The risk-based, phased roadmap towards
WHO GMP has to be anchored as a key
component in a holistic approach for
the development of the pharmaceutical
manufacturing industry. In addition to a
GMP roadmap many other components
essential for the industrial development of the
pharmaceutical sector in developing countries
have to be taken into consideration. Those
components, which include strengthening of
the regulatory functions, access to affordable
finance, development of incentive schemes,
development of necessary human resources
and securing distribution chains have to be
addressed to enable sustainable high quality
local production to be achieved through the
GMP roadmap approach.
While the roadmap approach presented
by UNIDO in this concept paper focuses
on improving the quality of pharmaceutical
manufacturing, other stakeholders, especially
the Essential Medicines Department of
WHO, are working on a risk assessment
regarding the suitability of products for
manufacture in companies according
to their respective levels of compliance
to WHO GMP. Both organizations have
acknowledged the complementarities of
the respective methodologies and have
indicated willingness to incorporate them
into a joint approach correlating the phases
of the roadmap and the risk classification of
pharmaceutical manufacturers with product
manufacturing requirements.
WHO Drug Information Vol. 30, No. 2, 2016 UNIDO concept paper for discussion

A practical example of successful of manufacturers in the country. The scope

development of a risk-based,
phased roadmap to WHO GMP
compliance
The approach outlined above has been
used to develop GMP roadmaps for Kenya
and Ghana delineating the pathway from
existing GMP compliance to full WHO GMP
compliance. Country-specific roadmaps were
devised taking into consideration the main
technical challenges faced by manufacturers
of medicinal products.
The development of the Kenya GMP
Roadmap (6) is described here as a practical
example for the successful development of a
risk-based, phased GMP roadmap using the
aforementioned approach.
The baseline assessment of existing
manufacturing practices was performed
at seven pharmaceutical companies that,
while all falling short of full WHO GMP
compliance, represented the different levels
of the baseline assessment was limited to
the manufacture of small molecule, non-
sterile medicinal products. The results shown
below are anonymized and the sequence of
the companies assessed is randomized, not
allowing participants to be traced.
Using evaluation tool 1 the compliance
of participating companies with key quality
elements of WHO GMP was assessed.
The results of this assessment, as shown
in Figure 2, indicate that the companies’
compliance with WHO GMP was not rated
acceptable for the majority of key quality
elements.
Figure 2 also shows that seven key quality
elements were associated with the lowest
possible compliance rating (i.e. inadequate)
in more than half of the participating
companies. The elements listed below should
be addressed with priority as they pose

Figure 2: Compliance of participating companies with key quality elements of GMP

Key quality element:
Utilities impacting GMP*
Premises*
Material handling*
Good practices in quality control*
Quality assurance**
Sanitation and hygiene
Qualification and validation
Personal hygiene
Complaints
Product recalls
Equipment
Documentation
Good practices in production
Self-inspection and quality audits**
Training
Contract production and analysis**
Personnel
0 1 2 3 4 5 6 7
Number of companies
■Acceptable   ■Improve   ■Inadequate   □Not applicable
* Key quality elements written in red indicate those for which the highest number of companies showed
least compliance.
** As the assessment had been performed before TRS 986, Annex 2 (5) became official, terms used for
the key quality elements were based on TRS 961, Annex 3 (9), e.g. the term “Quality assurance” was
used instead of “Pharmaceutical quality system”.

193
UNIDO concept paper for discussion WHO Drug Information Vol. 30, No. 2, 2016

the most severe risk to quality, safety and from “2” to “3”. This result underlines that the
efficacy of the manufactured products: selection of companies was suitable for the
• Quality assurance assessment, as the selection criteria were
• Utilities impacting Good Manufacturing to include companies representing different
Practices (GMP) levels of GMP compliance (risk scores
• Sanitation and hygiene between “1” and “3”) while having not yet
• Qualification and validation achieved full compliance with WHO GMP (no
• Premises company with an overall GMP rating of “A”).
• Material handling The risk assessment reveals that in general
• Good practices in quality control the scores for the site were higher than
those related to the QMS. The usually higher
Evaluation tool 2 was used to categorize risk associated with site was for almost all
participating companies regarding their companies the main cause for downgrading
compliance with WHO GMP based on two the overall GMP compliance rating. This
risk-indicating factors, namely: clearly indicates that particular guidance is
• Compliance of site with WHO GMP needed regarding site-related GMP aspects
standards and design requirements.
• Compliance of quality management The following conclusions can be
systems with WHO GMP standards drawn from the assessment performed at
The results are displayed in Table 2. pharmaceutical companies in Kenya and
needed to be reflected in the design of the
Table 2: Results of the risk categorization of roadmap to WHO GMP compliance:
companies based on their compliance with • Site-related GMP aspects need to be
WHO GMP prioritized for improvement.
Company name Risk Risk Overall • Immediate measures are also required to
score score GMP reduce product-related risks caused by
Site QMS rating inadequacies of the QMS, with a special
Company 1 2 1 B focus on those key quality elements with
Company 2 2 2 B the lowest observed compliance ratings.
Company 3 3 2 C Taking into account the evaluation results, a
Company 4 3 2 C risk-based, two-phased approach has been
Company 5 3 2 C designed for the Kenya GMP Roadmap as
Company 6 3 2 C shown in Figure 3.
Company 7 3 3 C
Phase I focuses on the mitigation
of risks impairing production safety by
The risk assessment shows that out of establishment of WHO GMP compliant sites
the seven companies assessed only and improvement of those QMS elements
two achieved an overall GMP rating of for which the majority of companies showed
“B” (medium risk company) whereas the the most severe deficiencies versus WHO
remaining companies received an overall GMP (“QMS 1”). Using the results of the
GMP rating of “C” (high risk company). The risk assessment, the majority of companies
risk scores for compliance of the QMS with initially rated as “C” should reach a “B” rating
WHO GMP requirements ranged from “1” at the end of Phase I as their sites (being a
to “3”, the risk scores for compliance of the main contributory factor for their low GMP
site with WHO GMP requirements ranged compliance rating) should then be in line with

194
WHO Drug Information Vol. 30, No. 2, 2016 UNIDO concept paper for discussion

WHO GMP requirements. Besides, those
key quality elements for which the majority of
companies showed least compliance will be
in line with WHO GMP requirements at the
end of Phase I, enabling companies to have
at least a sporadic implementation of QMS in
place.
During Phase II the main focus will be on
establishing a comprehensive, WHO GMP
compliant quality management system
(“QMS 2”) so that ultimately both structural
(“site”) and organizational (“QMS”) measures
for GMP compliance will be in line with
WHO GMP. As the definition of the individual
phases of the GMP roadmap is based on
both the severity of deficiencies versus
WHO GMP and the compliance-related
risk observed at Kenyan pharmaceutical
manufacturers, a stepwise, risk-based
approach has been realized for the Kenyan
roadmap towards achievement of full
WHO GMP compliance. This technical
roadmap provides for each of its phases
a detailed breakdown of required actions
and milestones for improvement of site-
related and QMS-related GMP aspects. The
structure of the Kenya GMP Roadmap is
provided in Annex 2.
The roadmap has been complemented with
an implementation plan embracing all facets
required for successful implementation of the
Kenyan roadmap towards compliance with
WHO GMP requirements including definition
of near- and mid-term requirements. The
roadmap and implementation plan were
agreed and endorsed by key stakeholders
including representatives from industry
and government (both policy-makers and
regulators) during meetings in 2013 and 2014
and are part of the implementation of the
Kenyan Pharmaceutical Sector Development
Strategy.
The targeted timeline for implementation
of the Kenya GMP Roadmap, as agreed
amongst all stakeholders, is five years,
whereby the first phase is targeted to take
no longer than three years; and the second
phase is targeted to be completed within two

Figure 3: Risk-based, phased approach of the Kenyan roadmap towards achievement of full
WHO GMP compliance

Overall GMP Phase focus and targeted outcomes Phase

compliance rating number
Site and Quality Management Systems in line with
A
WHO GMP requirements
▲
Improvement and implementation of those QMS
── Phase II
with identified lower risk (QMS 2)
▲
Site generally in line with WHO GMP requirements /
B
QMS 1 in line with WHO GMP requirements
▲
Improvement and implementation of those QMS
for which majority of companies showed least
compliance (QMS 1)
── Phase I
Construction / modification of sites as per WHO
GMP requirements
▲
Site and Quality Management Systems not in line
C
with WHO GMP requirements

195
UNIDO concept paper for discussion WHO Drug Information Vol. 30, No. 2, 2016
years. The time allocated to Phase I is longer
due to the need for modification of existing
sites or construction of new sites during this
phase.
In line with the implementation plan for
the roadmap all Kenyan manufacturers
of finished pharmaceutical products were
assessed by internationally experienced GMP
inspectors in conjunction with inspectors from
the Pharmacy and Poisons Board in 2015.
The outcome of the gap analysis project
confirmed observations, conclusions and
prioritizations made based on the baseline
assessment for the development of the GMP
Roadmap and hence verified the adequacy
of the approach and the aforementioned
tools used for the development of the GMP
roadmap.
Conclusion
This document summarizes a methodology
to develop a pathway for pharmaceutical
manufacturers in developing countries to
move towards WHO GMP standards. In order
to be manageable and scientifically sound,
the GMP roadmap should be risk-based and
structured into phases. The concept has been
successfully applied in Kenya and Ghana
where country-specific GMP roadmaps
have been developed in consultation
with key domestic stakeholders including
the pharmaceutical industry, regulators
and relevant governmental departments.
Assessments of all Kenyan manufacturers
of finished pharmaceutical products by
internationally experienced GMP inspectors
in conjunction with inspectors from the
Pharmacy and Poisons Board confirmed the
observations, conclusions and prioritizations
made based on the baseline assessment
for the development of the Kenya GMP
Roadmap and hence verified adequacy of the
approach and the tools used for development
of the GMP roadmap.
196
References
1 WHO. Untitled document. Available at: http://
www.who.int/medicines/services/counterfeit/
faqs/QACounterfeit-October2009.pdf.
2 WHO. Substandard, spurious, falsely
labelled, falsified and counterfeit (SSFFC)
medical products. Fact sheet. Updated
January 2016.
3 WHO. Survey of the quality of selected
antimalarial medicines circulating in six
countries of sub-Saharan Africa. Geneva:
World Health Organization, 2011.
4 WHO. Quality assurance of
pharmaceuticals: a compendium of
guidelines and related materials. Volume
2, Good manufacturing practices and
inspection. 2nd updated edition. Geneva:
World Health Organization, 2007.
5 WHO. WHO good manufacturing practices
for pharmaceutical products: main
principles. Annex 2. In: WHO Technical
Report Series, No. 986. Geneva: World
Health Organization, 2014.
6 United Nations Industrial Development
Organization (UNIDO), Republic of Kenya.
Kenya GMP Roadmap Concept. A Stepwise
Approach for the Pharmaceutical Industry to
Attain WHO GMP Standards. [On line] 2014.
7 Thrussell I. Examples of critical and major
observations from GMP inspections of
Manufacturing, QC and Contract Research
Organisations. Presented at the UNICEF
Pharmaceutical Supplier Meeting 2012 on
26 September 2012.
8 EMA. Compilation of Community
Procedures on Inspections and Exchange
of Information (EMA/572454/2014 Rev 17).
[Online] 3 October 2014. .
9 WHO good manufacturing practices for
pharmaceutical products: main principles.
Annex 3. In: WHO Technical Report
Series, No. 961. Geneva: World Health
Organization, 2011.
10 United Nations Industrial Development
Organization. White Paper on UNIDO’s
GMP Roadmap Concept. Design of a
Stepwise Approach for the Pharmaceutical
Industry in Developing Countries to Comply
with WHO GMP.
WHO Drug Information Vol. 30, No. 2, 2016 UNIDO concept paper for discussion

The Kenyan roadmap towards achievement of full WHO GMP compliance

Source: (6)

Annex 1: Indicator rating system

Indicator criteria have been defined in order to increase transparency when rating the compliance risks
associated with “Site” and “Quality Management System” (“QMS”) of the companies assessed.
A score of “3” represents a high compliance risk, whereas a score of “1” represents a low compliance risk.
Indicators for score criteria for site
Prerequisite Rating
1 2 3
Premises Premises are designed to be Premises show significant deficiencies Premises are unsuitable for
suitable for pharmaceutical from WHO GMP but do not impair pharmaceutical manufacturing
manufacturing production safety → Production safety impaired

Utility Utilities which have direct Utilities which have direct product Utilities which have direct
product contact (e.g. water, contact (e.g. water, air handling, product contact (e.g. water, air
air handling, compressed compressed dried air) are in place as handling, compressed dried
dried air) are in place as required but not fully compliant with air) are not available although
required, suitable and effective/ WHO GMP required, or available utilities
functioning are unsuitable
Equipment Equipment for all manufacturing Equipment for at least critical Equipment for critical
steps and quality controls manufacturing steps and quality manufacturing steps and
are suitable to perform the controls are in place and suitable to quality controls are not
operation and functioning perform the operation and functioning available or not functioning
When assigning the overall site rating, the rating (1, 2 or 3) which best reflects the various individual ratings that were
assigned to the site attributes should be chosen.
Indicators for score criteria for QMS
Prerequisite Rating
1 2 3
GMP A systematic holistic approach No systematic approach towards No GMP documentation is in
documenta­ towards GMP documentation is a documentation system is in place; procedures are totally
tion and in place; procedures performed place; sporadic implementation of inadequate
procedures are adequate and based on a GMP requirements; procedures
documented system performed are not always based on a
documented system
Calibration/ A systematic approach Checks for performance of critical No calibration, qualification,
qualifica­tion/ based on master documents, equipment, instruments and methods validation are performed
validation schedules, protocols and done but not to an extent required
reports is in place and/or not based on a systematic
approach
Preventive Comprehensive preventive Preventive maintenance for No preventive maintenance is
main­tenance maintenance procedures based critical systems is performed but performed.
on a systematic approach are no systematic approach including
in place. schedules, protocols, reports/logs is
in place
Sanitation Cleaning is adequate; a No signs of inadequate cleaning are Evidence of widespread
systematic approach to cleaning observed, but no systematic approach accumulation of residues/
consisting of validation, cleaning to cleaning including cleaning extraneous matter exists;
schedules, logs is in place validation, schedules, logs is in place evidence of gross infestation is
observed
Material Documented procedures for Testing of materials/products is No testing of materials/products
handling all types of material handling performed but not to the extent is performed. Procedures for
are in place in line with required by pharmacopoeia and receipt, sampling, storage,
pharmacopoeia/ international international guidelines. Procedures manufacturing and distribution
guidelines for receipt, sampling, storage, are inadequate; no GMP
manufacturing and distribution are documentation is in place
defined but documentation is not in
place for all operations
Continued

197
UNIDO concept paper for discussion WHO Drug Information Vol. 30, No. 2, 2016

Indicators for score criteria for QMS (continued)

Prerequisite Rating
1 2 3
Personnel/ Personnel has the right Personnel has the right qualification Personnel does not have the
training qualification, experience and and knowledge to perform duties right qualification, knowledge
knowledge to perform duties assigned, but no training program is and experience to perform the
assigned, training program is in place duties assigned
in place
When assigning the overall QMS rating, the rating (1, 2 or 3) which best reflects the various individual ratings that were
assigned to the QMS attributes should be chosen.

Annex 2: Exemplary structure of the Kenya GMP Roadmap

Start: Site and quality management systems not in compliance with WHO GMP requirements
Section 1.1: Phase I, Site
Phase/Ref. No. Key quality element Actions for implementation Milestones
1.1.1 Premises Define scope of premises by taking into account: Scope of the
▪▪ Environment in which the premises are built premises defined
▪▪ Targeted product classes (e.g. if toxic, ...
sensitizing, mutagenic, beta-lactams, sensitive to
light, temperature and/or humidity)
▪▪ Targeted production capacity
▪▪ ...
... ... ... ...
End of section: Phase I, Site
Site compliant with WHO GMP, but quality management systems (QMS) not in line with WHO GMP
Section 1.2: Phase I, QMS
Phase/Ref. No. Key quality element Actions for implementation Milestones
1.2.1 Quality assurance Development of an organizational structure Authorized
(organogram) within the company outlining organizational charts
hierarchy, functional levels and reporting lines. The in place
organizational structure has to ensure a separation ...
of quality assurance/control from production.
...
... ... ... ...
End of section: Phase I, QMS
Note: During the improvement of site and QMS at already existing pharmaceutical manufacturers
towards WHO GMP, activities outlined in Sections 1.1 and 1.2 should be conducted concurrently.

Site and QMS identified as main technical challenges in Kenya compliant with WHO GMP
Section 2: Phase II
Phase/Ref. No Key quality element Actions for implementation Milestones
2.1 Complaints Development and implementation of a Documented
documented system regarding handling, system for handling,
investigation, corrective and preventive actions of investigation,
complaints containing: corrective and
▪▪ Responsible person(s) and responsibilities preventive actions of
▪▪ Procedures to be followed for handling, complaints in place
investigation, corrective and preventive actions and implemented
of complaints including timelines ...
▪▪ ….
... ... ... ...
End of section: Phase II
Completion: Site and QMS in compliance with WHO GMP å

198
Dissolved Oxygen
Report

UP641402
 UP641402

Contents
1. INTRODUCTION 1

1.1 THE AIM. 1

1.2 DISSOLVED OXYGEN 1

2. METHODS AND MATERIALS 2

2.1 HEALTH AND SAFETY. 2

2.2 METHODS. 3

2.3 MATERIALS 3

2.4 EXPERIMENTAL DESIGN. 4

2.5 CALCULATIONS. 4

3. RESULTS 5

3.1 WINKLER TITRATION METHOD. 5

3.2 CALCULATIONS. 5

3.3 COMPARISONS. 5

3.4 TETRA TEST KIT. 5

3.5 BIOCHEMICAL OXYGEN DEMAND (BOD). 6

4. DISCUSSION 6

5. CONCLUSION 7

BIBLIOGRAPHY 7

Table of figures
Figure 1: DO tolerances for fish, Water research (2011). 2
Figure 2: Hanna probe 3
Figure 3: Oxyguard probe 3
Figure 4: Tetra test kit 3
Figure 5: Oxygen saturation nomogram 4

Table of tables
Table 1: Titration calculation 5
Table 2: DO results 5
Table 3: BOD results 6

i
 UP641402

1. Introduction

1.1 The Aim. The aim of this experiment was to investigate the level of dissolved
oxygen (DO) using a variety of techniques and equipment in a sample of water from an
unknown source and to establish the biochemical oxygen demand (BOD) in a number of
different water sources.

1.2 Dissolved oxygen. Landau (1992), Nakova et al. (2009), Bahadori and Vuthaluru
(2010) and Andrews et al. (2011) state that a sufficient supply of dissolved oxygen (DO) is
vital for life in aquaculture and is a good indicator of water quality. There is 20-30 times
less oxygen in water than in the same volume of air and this makes for a demanding
environment for aerobic creatures to survive in (Landau 1992; Andrews et al. 2011). Most
aquatic animals need more than 1mg/l concentration of oxygen for survival and 4-5mg/l to
avoid stress (Parker 2002). Girdler et al. (2010), mentions that the optimum DO level
should be above 8ppm and dangerous levels are below 6ppm in a course fishery however,
Landau (1992) contradicts that statement by stating that many culturists try to maintain a
5ppm.

1.3 Oxygen levels in water will decrease with increasing temperatures and at higher
altitudes (D’Autilia et al., 2004; Bahadori and Vuthaluru, 2010; Girdler et al., 2010 and
Andrews et al., 2011), likewise the opposite will increase the DO in a fishery. A 1%
increase in temperature will result in an 11% decrease in O 2, Girdler et al. (2010). There
are natural fluctuations between the seasons with the summer (warmer) months will on
average be lower in DO levels than the rest of the year, D’Autilia et al., 2004; Yoshikawa et
al., 2007; O’Boyle et al., 2009 and Girdler et al., 2010. Landau (1992), Parker (2002),
D’Autilia et al. (2004) and Reeder (2011) state that DO levels will also change through the
day, with DO reaching its maximum during the late afternoon and will be at its lowest
around sunrise and Girdler et al. (2010) states that DO should be recorded each day to aid
in identifying possible periods of hypoxia.

1.4 Aquatic plants and vegetation will use carbon dioxide (CO 2) which is a by-product
from respiration of fish and aquatic plants to produce oxygen, (Landau, 1992; Parker,
2002; Girdler et al., 2010 and Andrews et al., 2011), this is known as photosynthesis.
However if there is insufficient plants to support the aquatic system, this will result in low

1
 UP641402

DO levels, (Parker, 2002 and Andrews et al., 2011), as the fish and organisms will
consume more O2 than the plants can produce. The amount of oxygen required to support
fish, organisms and bacteria is called Biological Oxygen Demand (BOD), (Landau, 1992;
Misraa, 2006 and Girdler et al., 2010). Oxygen consumed during algal respiration in some
cases may account for 50% of the total BOD, Girdler et al. (2010). It is also worth
mentioning that stressed fish will also use more oxygen, (Landau, 1992; Parker, 2002 and
Girdler et al., 2010). Sensitivity of low concentrations of DO differs between species and
between the stages of life, (Alabaster and Lloyd, 1980). Fig 1 below shows the tolerances
of DO in all stages of growth for fish in general.

Figure 1: DO tolerances for fish, Water research (2011).

2. Methods and materials

2.1 Health and safety. Before commencing the experiment a detailed brief is given on
the health and safety implications when working in the laboratory. Numerous chemicals
some of which are hazardous were used in the laboratory therefore care must be taken
during the experiment meant that rubber gloves and goggles had to be worn at all times.
Glass beakers and pipettes were used during the experiment, which meant great care had
to be taken by all when handling and using the instruments. Dissections also take place
regularly in the labs so there could be a slim chance of fish slime and bodily fluids still
being present on the tables and equipment being use, therefore all those entering and
leaving the laboratory must wash their hands thoroughly.

2
 UP641402

2.2 Methods. To establish an accurate reading of DO in the samples the Winkler titration
was used, as it is the industry standard method of finding the level of DO in a sample,
Horstkottea et al. (2010). However this method is very much a lab based experiment and
therefore not ideal to every aquaculturists. Other methods used in this experiment were 2
commercially available DO probes and a Tetra test kit that can be purchased in any
aquatics shop.

2.3 Materials.

3 x 50ml conical flask

1 x 250 ml volumetric flask
Pipette and pipette filler
1 x Burette
Sample of water (bottle D) with a salinity of 3.5ppt

For the Winkler titration experiment the following reagents were used:

B2 Sodium thiosulphate NaS2O3 approximately 0.0125m

C Ortho phosphoric acid H3PO4 (sg 1.75)
D Alkaline iodide-azide solution
E Manganous sulphate MnSO45H2O 50%
F Starch indicator 1%

2.4 The following commercially available probes and test kits were used during the
experiment: Fig 2 the Hanna H9146 DO probe, fig 3 the Oxyguard-Handy Polaris DO
probe and fig 4 a Tetra DO test kit.

Figure 2: Hanna probe Figure 3: Oxyguard probe Figure 4: Tetra test kit

3
 UP641402

2.4 Experimental design. In summary the water sample from bottle D was poured
into 3 bottles and analysed using Winkler titration, DO probes and a Tetra test kit. Finally
the individual water sources were tested for BOD, to ensure photosynthesis didn’t occur
within the samples they were poured into dark coloured bottles and placed in dark room for
5 days. The results were also compared with an oxygen saturation nomogram, see figure
5 below. Annex A to this report shows the procedures and experimental design that was
used during the experiments.

Figure 5: Oxygen saturation nomogram

2.5 Calculations. To work out the DO levels in the water samples from titration the
following calculation was used:
X = O2 concentration in water (mg/l-1)

X= 8.0 x CB x VB
Va (Vf-4.0) / Vf

CB = Concentration of reagent B
VB = Volume of reagent B
Va = Volume of aliquot taken for titration
Vf = Bottle volume, with stopper in place (ml) this should be measured by weighing with an
accuracy and precision of 0.1%

4
 UP641402

3. Results

3.1 Winkler titration method. Below gives the amount of base (Sodium NaS2O3)
that was required to initiate the chemical change in the 3 samples.

Bottle 1 4 ml
Bottle 2 3.9 ml
Bottle 3 4 ml
Average: 3.96 ml

3.2 Calculations.

X= 8.0 x CB x VB
Va (Vf-4.0) / Vf

Table 1: Titration calculation

CB (Mm) VB (ml) Va (ml) Vf (ml) X (mg/l-1)
12.5 3.96 50 (250-4.0) = 246 8.04

X = 8.04 mg/l
O2 saturation using a nomogram was estimated to be 86% saturation.

3.3 Comparisons. The Winkler titration method was used as a bench mark for
establishing the known quantity of DO in the samples for the experiment. The results from
this were then cross examined with the probes and a DO nomograph; the results are in
table 2 below.

Table 2: DO results

Method DO (mg/l) Saturation (%) Temp 0C Nomogram (%)

Winkler Titration 8.04 19.3 86
Hanna DO Probe H9146 6.05 79 18 63
Oxyguard DO probe 8.3 87 19.3 87

3.4 Tetra test kit. The test kit was used 3 times on the different samples and each time
gave a non-conclusive result and therefore will not be included in the results.

5
 UP641402

3.5 Biochemical oxygen demand (BOD). Table 3 below shows the BOD of each
sample tested.

Table 3: BOD results

Water Samples Initial (mg/l) Final (mg/l) BOD (mg/l)

Tilapia pond 4.8 2.8 2.0
Tilapia pond and milk 5.69 .59 5.1
Pond 3.7 3.2 0.5
Tap 7.7 5.9 1.8

4. Discussion

The results show that the Oxyguard probe was accurate and could be compared directly to
the titration experiment results. According to Singh et al. (2004) and Horstkottea et al.
(2010), titration is one of the most accurate methods of establishing DO, but should really
only be done in a laboratory due to the nature of the kit required to undertake the
experiment in the field. Whereas a probe can be used at any time of day and on a boat,
making it a very useful and quick method of establishing the level of DO in a piece water.
For increased accuracy, samples can be removed from the system for titration in a
laboratory to confirm the results.

The difference in the 2 methods was 0.26mg/l in DO and 1% in saturation, showing that
the DO probe is a useful and accurate tool when calibrated (Girdler et al., 2010 and
Andrews et al., 2011). Once calibrated the Oxyguard probe has an accuracy of 1%,
(Oxyguard,2007), however the Hanna DO probe showed 6.05 mg/l and oxygen saturation
level of 79%, which is considerably different to the Oxyguard and titration methods. The
calibration process with the Hanna probe was long winded compared to the Oxyguards
user friendly device and this could have been the reason for the data. The DO results
from both probes were cross referenced with a DO nomogram to establish the O2
saturation in the sample, the Oxyguard showed 87% only 1% difference from the probe,
however the Hanna showed 62% saturation but the probe showed 79% a difference of
17%, it must be noted that the Hanna probe has an accuracy of +/- 1.5%, Hanna
Instruments (2010) and the calibration process may have been incorrect as previously
stated.

The results show that the water sample is suitable for Carp (Cyprinus carpio), as carp
need a minimum of 4mg/l, (Alabaster and Lloyd, 1980; Parker, 2002; Girdler et al., 2010

6
 UP641402

and Andrews et al., 2011), while DEFRA (2011) state that cyprinid waters should have a
minimum oxygen saturation of >50% and DO of 7 mg/l. Girdler et al. (2010) and Andrews
et al. (2011) state that trout (Salmo trutta) require a minimum of 8mg/l, also a trout has an
oxygen demand of 28.3mg/l due to its high energy requirement, (Girdler et al., 2010), while
more sedimentary fish such as the south American leaf-fish (Monocirrhus polyacanthus)
will survive on 1-2mg/l, (Andrews et al., 2011). If trout were to inhabit the water source
then feeding must cease and the water needs to be aerated immediately to prevent
anoxia, (Landau, 1992; Parker, 2002; Girdler et al., 2010 and Andrews et al., 2011).

The BOD samples show that oxygen had been consumed in each sample as expected,
however the tilapia (Oreochromis niloticus) and milk sample shows the biggest loss of DO.
Milk has a BOD of 17 x 105mg/-1, (Elliot et al., 2001) and will contain active bacteria that will
consume more O2 in the samples due to respiration. The results also show that the pond
water had a relativity low level of BOD at 0.5mg/l, however DEFRA (2007), state that
between the years 1980-2005, the average levels of BOD in its water ways in the south
west of the UK were 1.86mg/l and 3.45mg/l in the North West, it would suggest that that
there were lower levels of bacteria in the pond water sample. BOD guidelines for cyprinid
pond waters should have a level of <6 mg/l DEFRA (2007). However the tap water
sample should have had the lowest level of DO as there should be very little in the way of
bacteria consuming the DO DEFRA (2011).

5. Conclusion

While titration is an accurate method of establishing DO, it is really a laboratory tool and
will take up valuable time before a result is made, any time wasted could result in major
fish kills due to lack of dissolved oxygen in the water system. The use of DO probes are a
necessity for any aquaculturist in maintaining water quality, once calibrated they provide a
quick and reliable results that can be acted upon.

Bibliography
Alabaster, J.S and Lloyd, R (1980) Water Quality Criteria for Freshwater Fish. London-
Boston. Butterworth

7
 UP641402

Andrews, C, Exell, A. and Carrington, N. (2011), The Manual of Fish Heath. Surrey.
Interpet publishing.
Bahadori, A. and Vuthaluru, H,B. (2010). Simple Arrhenius-type function accurately
predicts dissolved oxygen saturation concentrations in aquatic systems. Process Safety
and Environmental Protection. 8: 335–340.
D’Autilia, R, Falcucci, M, Hull, V and Parrella, L. (2004). Short time dissolved oxygen
dynamics in shallow water ecosystems. Ecological Modelling. 179: 297–306.
DEFRA, (2007). Dissolved oxygen and Biochemical Oxygen demand statistics, Retrieved
on Mar 04, 2012. DEFRA website:
http://www.defra.gov.uk/environment/statistics/index.htm
DEFRA, (2011). Draft partial regulatory impact assessment of environmental quality
standards for implementation of the Water Framework Directive in the UK. Retrieved on
Mar 6, 2012 from the DEFRA website:
http://archive.defra.gov.uk/environment/quality/water/wfd/documents/pdf-ria-draft/ria-wfd-
annex1.pdf
Elliot, A. J, Wilkins, B.T and Mansfield, P. (2001). Disposal of contaminated milk on
coastal waters. Marine pollution bulletin. 42: 927-934
Girdler, A, Welby, I and Welcomme, R. (2010). Fisheries Management, A manual for still-
water course fisheries. Part ii Management of still-water fisheries, p200. Blackwell
publishing, Chichester, Sussex
Hanna Instruments. (2010). Hanna H9146 DO probe. Retrieved on Mar 4, 2012. Hanna
instruments website:
http://www.hannainst.co.uk/product_info.php?cPath=1203&products_id=3078&osCsid=je1
vn7f7tmt6nvc1bk2a99l6a5
Horstkottea, B, Alonsob, J.C, Miróa, M and Cerdàa, V. (2010). A multi syringe flow
injection Winkler-based spectrophotometric analyser for in-line monitoring of dissolved
oxygen in seawater, Talanta, 80: 1341–1346
Lake access. (2012). Oxygen Nomogram. Retrieved on Mar 4, 2012. Lake access
website: http://www.lakeaccess.org/russ/nomogram.gif
Oxyguard (2007). Oxyguard handy polaris DO probe. Retrieved on Mar 4. 2012.
Keetonaqua website:
http://www.keetonaqua.com/wp-content/uploads/2010/03/4_Point-Four-Systems-
OxyGuard-Handy-Polaris-Brochure-1001-1-2.jpg
Landau, M. (1992). Introduction to Aquaculture. New York: John Wiley & Sons Inc.

8
 UP641402

Misraa, A,K, Chandraa, P and Shuklab, J,B. (2006). Mathematical modelling and analysis
of the depletion of dissolved oxygen in water bodies. Nonlinear Analysis: Real World
Applications. 7: 980–996
Nakova, E, Linnebank, F.E, Bredeweg, B, Salles, P. and Uzunov, Y. (2009). The river
Mesta case study: A qualitative model of dissolved oxygen in aquatic ecosystems,
Ecological Informatics. 4: 339–357
O’Boyle, S, McDermott, G and Wilkes, R. (2009).. Dissolved oxygen levels in estuarine
and coastal waters around Ireland. Marine Pollution Bulletin. 58: 1657–1663
Parker, R. (2002). Aquaculture Science (2nd ed.), Health of aquatic animals. New York:
Delmar.
Reeder, B.C (2011). Assessing constructed wetland functional success using diel changes
in dissolved oxygen, pH, and temperature in submerged, emergent, and open-water
habitats in the Beaver Creek Wetlands Complex, Kentucky. Ecological Engineering. 37:
1772– 1778
Singh, R.K, Vartak, V.R, Balange, A.K and Ghughuskar, M.M (2004). Water quality
management during transportation of fry of Indian major carps, Catla catla (Hamilton),
Labeo rohita (Hamilton) and Cirrhinus mrigala (Hamilton). Aquaculture. 235: 297–302
Tetra. (2011). Tetra O2 test kits. Retrieved on Mar 4. 2012. 123 aquatics website:
http://www.123aquatics.co.uk/products/aquarium/treatments/water-
treatments/medication/test-kits/tetratest-oxygen-1283.aspx
Water research. (2011). Dissolved oxygen. Retrieved on Mar 4, 2012. Water research
website:
http://www.water-research.net/Watershed/dissolvedoxygen.htm
Yoshikawa, T, Murata, O, Furuya, K and Eguchi, M. (2007). Short-term covariation of
dissolved oxygen and phytoplankton photosynthesis in a coastal fish aquaculture site.
Estuarine, Coastal and Shelf Science. 74: 515-527

9
 Forensic Science International 297 (2019) 35–46

Contents lists available at ScienceDirect

Forensic Science International

journal homepage: www.elsevier.com/locate/forsciint

Review Article

Cognitive bias research in forensic science: A systematic review

Glinda S. Cooper* , Vanessa Meterko
Innocence Project, 40 Worth St, Suite 701, New York, NY, 10013, United States

A R T I C L E I N F O A B S T R A C T

Article history: The extent to which cognitive biases may influence decision-making in forensic science is an important
Received 2 August 2018 question with implications for training and practice. We conducted a systematic review of the literature
Received in revised form 20 December 2018 on cognitive biases in forensic science disciplines. The initial literature search including electronic
Accepted 8 January 2019
searching of three databases (two social science, one science) and manual review of reference lists in
Available online 22 January 2019
identified articles. An initial screening of title and abstract by two independent reviewers followed by full
text review resulted in the identification of 29 primary source (research) studies. A critical
Keywords:
methodological deficiency, serious enough to make the study too problematic to provide useful
Forensic science
Cognitive bias
evidence, was identified in two of the studies. Most (n = 22) conducted analyses limited to practitioners
Confirmation bias (n = 17), forensic science trainees (n = 2), or both forensic science practitioners and students (n = 3); other
Contextual information analyses were based on university student or general population participants. Latent fingerprint analysis
Training was examined in 11 studies, with 1–3 other studies found in 13 other disciplines or domains. This set of
studies provides a robust database, with evidence of the influence of confirmation bias on analysts
conclusions, specifically among the studies with practitioners or trainees presented with case-specific
information about the “suspect” or crime scenario (in 9 of 11 studies examining this question),
procedures regarding use of exemplar(s) (in 4 of 4 studies), or knowledge of a previous decision (in 4 of 4
studies). The available research supports the idea of susceptibility of forensic science practitioners to
various types of confirmation bias and of the potential value of procedures designed to reduce access to
unnecessary information and control the order of providing relevant information, use of multiple
comparison samples rather than a single suspect exemplar, and replication of results by analysts blinded
to previous results.
© 2019 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND
license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

Contents

1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2. Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3. Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.1. Description of studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.2. Synthesis of studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.3. Evaluation of study methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
4. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Declarations of interest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Funding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

1. Introduction

Cognitive bias is a broad term that includes a variety of

processes that may lead to inaccurate judgments or interpreta-
* Corresponding author.
E-mail addresses: gcooper@innocenceproject.org (G.S. Cooper), tions; cognitive biases can affect memory, reasoning, and decision-
vmeterko@innocenceproject.org (V. Meterko). making [1]. Although the term may be interpreted with a negative

https://doi.org/10.1016/j.forsciint.2019.01.016
0379-0738/© 2019 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
36 G.S. Cooper, V. Meterko / Forensic Science International 297 (2019) 35–46
connotation, it is important to understand that these processes are
a natural consequence of the need to attune to patterns and
develop heuristics to process a variety of complex stimuli [2].
Confirmation bias, described in 1998 [3] as “seeking or interpreting
evidence in ways that are partial to existing beliefs, expectations,
or a hypothesis in hand,” is one type of cognitive bias with the
potential to undermine the objective evaluation of forensic
evidence. This could arise, for example, by an unconscious focusing
of attention on similarities and away from dissimilarities because
of information about other evidence in a case, attributes of the
comparison procedures used, or from previous conclusions drawn
by another analyst.
Interest in the extent to which cognitive biases may influence
decision-making in forensic science has grown in the past decade;
this was one of the research needs identified in the 2009 National
Academy of Sciences report on forensic sciences [4]. The topic has
been the focus of considerable discussion, as exemplified by the
2013 review by Kassin et al. [5] and the accompanying responses
[2,6–14]. Restricting access to task-irrelevant information and
controlling the order and time in which information is provided
have been proposed as methods to reduce the potential for bias
[15]. A recent survey of 403 expert forensic science examiners,
however, found a general lack of acceptance of the need for
procedures designed to minimize cognitive biases, and a failure to
recognize susceptibility to biases [16].
We conducted a scoping review [17] of the literature on
cognitive biases in criminal investigations and prosecutions to map
the landscape of the existing research (e.g., topics, populations,
variables), by the point in the criminal justice process being
addressed. Forensic science was one of the areas with the most
research relating to cognitive biases, and thus we focused on this
set of research for a more in-depth analysis. In this second phase,
we conducted a systematic review of the scope, design, and results
of the forensic science-cognitive bias research using a set of criteria
defined a priori. The purpose of this review was to evaluate the
basis for conclusions regarding the potential influence of
contextual bias and other forms of bias on decision making in
different forensic science disciplines
2. Methods
We began our review using PsycINFO and Social Sciences Full
Text, two electronic databases of social science articles, chapters,
books, and dissertations. Specifically, we searched for work with
the words (criminal or justice or police or investigation* or
forensic* or jury or juries or judge* or conviction*) in any field (e.g.,
Table 1
Summary of 27 studies of confirmation bias in forensic sciences.
Discipline or domain N studies Participants
(Type, N)
Fingerprints 8 Practitioner
3 University students
Forensic Anthropology 3 practitioners, students
Bitemark 1 Dental students, other students
Bloodstain 1 Practitioners
Dog handling 1 Practitioners
DNA 1 Practitioners
Hair 1 Forensic science trainees
Handwriting 3 Forensic science trainees,
practitioners, general population
Shoeprint 1 Practitioners
Speech (auditory evidence) 1 University students
Toolmarks (bullets) 1 Practitioners
Crime scene investigation 1 Practitioners, students
Forensic pathology 1 Practitioners
Technology-human interactions 2 Practitioners, students
text, title) and (“cognitive bias” or “implicit bias” or “cognitive
dissonance” or “tunnel vision” or “confirmation bias” or “interpre-
tive bias” or “belief perseverance” or “asymmetrical skepticism”) in
any field, which produced 890 results. Based on title and abstract
review by two independent reviewers, we identified 92 seemingly
relevant abstracts, 20 of which pertained to forensic science. We
also reviewed the reference lists within the identified forensic
science publications to identify additional relevant references. As
additional checks on the completeness of our search strategy, we
repeated the search using the PubMed database, and our list of
references was reviewed by three external researchers working on
different questions relating to forensic science and confirmation
bias. The last review of the reference list was in July, 2018. At the
suggestion of a journal reviewer, we expanded our search to
include studies of the interaction of humans with technology and
databases, using a combination of search terms relating to bias,
technology, and database systems. This search strategy identified a
total of 41 primary source publications pertaining to 36 studies.
This review focuses on studies of the decisions or conclusions
made in an analysis. Specifically, we searched for studies, in any
language, addressing the following questions: (1) Does contextual
or case information influence the results of an analysis? And (2)
Does expectation bias (i.e., comparison to a single sample versus
comparison to a set of samples; knowledge of a previous
conclusion) influence the results of an analysis? These were issues
that were frequently discussed in the commentaries and review
articles found in the literature search. We limited our review to
disciplines considered within the Organization of Scientific Area
Committees for forensic science (OSAC) [see https://www.nist.gov/
topics/forensic-science/scientific-area-committees]. This criterion
was an easily reproducible, objective method to define the scope of
our review, and was broad enough to include a wide variety of
disciplines. This inclusion criterion resulted in the exclusion of 5
studies focusing on adversarial allegiance in forensic psychology
[18–21] or medical expertise in the context of medical malpractice
[22]. We included published studies and unpublished reports, but
excluded one citation for which a full text report was not available
[23]. Although all of the studies identified were in English,
language had not been designated as an exclusion criterion. To
avoid duplication of data, we did not include a meta-analysis based
on two previous studies [24], but we did include the two
underlying studies in our analysis. These exclusions resulted in
a final set of 34 primary source publications pertaining to 29
studies for review (Table 1).
A systematic review would typically include an evaluation of
the methods used in the identified studies to determine the “risk
N samples Variable(s)
5–70 1 >300 Case information, previous conclusion, comparison
27–107 20–96 procedures, emotion, complexity
38–99 1 Case information
178 15 Complexity, emotion
27; 30 12–16 Case information
18 8 Case information
17 1 Case information
12 4 Comparison procedures
12–28 6–30 Case information, previous conclusion, comparison
procedures
12 8 Case information, complexity
145 17 Case information, complexity
6 6 Case information
58, 36 1 Case information
192 31 Case information
23–30 160–210 Previous (technology-based) conclusion
 G.S. Cooper, V. Meterko / Forensic Science International 297 (2019) 35–46
of bias,” or the potential that methodological limitations would
affect the interpretation of the results. Evaluation methods have
been developed and used for randomized clinical trials and
observational studies in medicine [25,26], policy initiatives in
education, development, and other social sciences (see www.
campbellcollaboration.org), and environmental health [27,28].
We drew from this previous work four domains relating to the
internal validity of the study (allocation process or comparability
of groups, blinding of participants, blinding of other outcome
assessors, and analysis and reporting of results) and one domain
relating to external validity or generalizability (participant
eligibility and selection). The purpose of this evaluation was to
(1) provide descriptive information pertaining to the extent each
of these domains is addressed in this area of forensic science —
cognitive bias research, and (2) as is recommended in systematic
reviews, to identify attributes of the design (including imple-
mentation and analysis) that would be considered a critical
deficiency, making the study too problematic to provide useful
evidence. Exclusions for critical deficiencies can occur in the
identification phase, through the delineation of specific exclusion
criteria (e.g., excluding clinical trials that did not involve
randomized allocation) [25] or in the evaluation phase, through
the inclusion of a “critical risk of bias” category for the domains
under review [26]. We began with a description of what would be
considered an ideal or target study design, as discussed by Sterne
et al. [26] and Hernán et al. [29]. We next considered design
attributes within each domain that would be considered a critical
deficiency, and then developed examples for intermediate levels
(i.e., between “ideal” and “critical deficiency”) of decisions or
information that would strengthen the study (Appendix). In this
way we were able to assess the studies’ methods on a spectrum
from ideal to deficient, focusing on suggestions that could help
authors, and reviewers, strengthen the utility of research reports.
Critical limitations are noted in the summary of the studies. To
summarize the studies, we first stratified by type of research
question addressed and discipline, and then abstracted the results
for review. All studies, including those conducted with university
student or general population participants, are included in the
tabular summaries (Tables 2–5), but the synthesis focuses on
studies of forensic science practitioners or trainees. We also
provide a summary of the areas of improvement we noted in the
study evaluations.
3. Results
3.1. Description of studies
The 29 forensic science-confirmation bias studies identified
through this search included 11 studies of fingerprint (latent print)
analysis, with 1–3 studies in each of 13 other disciplines or
domains (Table 1). Most (n = 22) conducted analyses limited to
practitioners (n = 17), forensic science trainees (n = 2), or both
forensic science practitioners and students (n = 3); other analyses
were based on university student or general population partic-
ipants. Sample size was highly variable, ranging from 5 to 145
participants and from 1 to >300 samples to be analyzed per
participant. The studies examined the influence of various types of
case information or procedural details on decision-making; several
studies also examined the interaction between complexity and
case information. The two studies in the technology-domain
examined the influence of software generated information (i.e.,
position in AFIS candidate list or classification using facial
recognition software) on decision-making [30,31]. After the initial
two studies were published in the 1980’s [32,33], almost twenty
years passed before the additional studies were conducted, with
most published since 2010.
37
3.2. Synthesis of studies
Studies examining the influence of contextual (case) informa-
tion on decision-making are summarized in Table 2. Examples of
the type of information include information on whether there had
been a confession, results from other analyses, and other details
regarding the crime. Eleven disciplines are represented, most with
only one study, but there were three studies in forensic
anthropology [34–36]. Six experimental studies among forensic
science practitioners demonstrated the influence of either the
presence or absence of case information on analysts’ conclusions.
This included studies of analysis of DNA mixtures [37], fingerprints
[38], bloodstain patterns [39–41], dog handling [42], crime scene
investigation [43,44], and forensic pathology [45–47]. This effect
was also seen in three studies using participants with variety of
experience levels in forensic anthropology [34–36]. Two other
studies, in shoeprint analysis [48] and bullet analysis [49] did not
observe this type of influence. One study was considered to have
critical deficiencies, due to the lack of comparability between
groups [50]. The most relevant data in this study was the
comparison among the 51 crimes against people among the 845
cases from 2009 to 2010, divided into 13 cases defined as high level
of contextual information and high level of interaction with
investigators and 38 cases with low levels of contextual informa-
tion and low level of interaction with investigators. Residual
confounding by type of crime within this broad category is likely,
and there may be other differences between these groups that
were not assessed or addressed in the analysis.
Four studies examined procedures relating to the type of
exemplar or comparison sample(s) used in the analysis (Table 3).
Two studies examined use of more than one comparison sample in
microscopic hair analysis [33] and handwriting analysis [32]; the
hair analysis study compared use of a “line-up” of 6 samples to use
of a single exemplar, and the handwriting study compared a single
suspect with three possible suspect scenarios. In both of these
studies, the number of incorrect “identification” decisions was
higher when only a single comparison sample was used. Two other
studies examined assessment of minutia or suitability decisions in
fingerprint analysis made with and without a target exemplar
[51,52]. These two studies demonstrated an impact of a target
exemplar on decision-making, with a higher number of minutia
assessed with the presence of a target exemplar [51] and with the
presence of an exemplar, particularly a non-matching exemplar,
decreasing the correlation between the examiner’s decision and
the predetermined expert decision regarding suitability [52].
Six studies examined another procedural feature, specifically
the influence of knowledge of a previous decision, on an analyst’s
conclusions (Table 4); one of these was critically limited by the
failure of the blinding procedures, which was particularly difficult
given the setting of this study outside of a workplace [53]. Three
studies in fingerprints [52,54] and handwriting [55] were
conducted among experienced practitioners; one of the technolo-
gy-based studies was conducted with experienced latent print
examiners [30] and the other technology-based study was
conducted with university students [31]. Each of these studies
provide evidence that being told or shown a decision, ostensibly
based on an analyst’s work or on a computer algorithm, resulted in
conclusions that were more likely to be in agreement with the
previous decision, even if the initial decision was incorrect.
Four studies examined emotional context or seriousness of
crime, three in fingerprints [56–58] and one in bitemarks [59], but
only two were conducted in experienced practitioners [57,58]
(Table 5). Results for these two studies of suitability decisions
among fingerprint examiners were mixed, with one study showing
no difference in decision making [58] and the other showing a
higher number of “false positive” decisions (a submission decision
38 G.S. Cooper, V. Meterko / Forensic Science International 297 (2019) 35–46

Table 2
Studies of influence of case information on analyst decision-making, by discipline [64–66].
G.S. Cooper, V. Meterko / Forensic Science International 297 (2019) 35–46 39
40 G.S. Cooper, V. Meterko / Forensic Science International 297 (2019) 35–46

a
“not significant” designation of the authors replaced with actual p-value (http://www.socscistatistics.com/pvalues/fdistribution.aspx); p-values are 2-sided.

Table 3
Studies of Influence of Comparison Procedures on Analyst Decision-making, by Discipline.
 G.S. Cooper, V. Meterko / Forensic Science International 297 (2019) 35–46 41

Table 4
Studies of Influence of Knowledge of a Previous Conclusion on Analyst Decision-making, by Discipline or Domain.
42 G.S. Cooper, V. Meterko / Forensic Science International 297 (2019) 35–46

Table 5
Studies of Influence of Emotional Level or Seriousness of Case on Analyst Decision-making, by Discipline.

a
We report p-value for Fisher’s 2x4 test (http://vassarstats.net/fisher2x4.html) because expected cell values are < 5; For Chi-square test, df for 4 x 2 table = 3.

made for samples in which a priori expert decision was insufficient
quality) for serious (high emotional context) crimes [57].
3.3. Evaluation of study methods
As noted previously, we identified two studies with critical
deficiencies relating to participant blinding and to comparability of
groups. Among the other studies, design elements fell short of
what we considered “ideal,” either because important details were
not reported, or because specific standards were not met. Although
most studies used an experimental design with allocation of
participants into study groups (or randomization of the ordering of
samples for analysis), some of the studies did not specify that the
allocation process was randomized, and none of the studies
provided information about the randomization procedures, as is
recommended for randomized clinical trials [see, for example,
Chapter 8.9, Sequence Generation, in [60]]. In addition, randomi-
zation does not necessarily provide assurance of comparability,
particularly for small sample sizes, and many studies did not report
data on group-specific characteristics. For example, level of
experience may be an important variable to consider in these
analyses, and it is important to provide data on this variable for
each of the study groups, rather than only for the full sample, to
assure comparability across groups. The extent of blinding also
varied among the studies, with some studies designed in a way that
prevented participants from knowing that a particular sample was
a study sample, and other studies only blinding participants to the
specific hypothesis or to group allocation. Most studies reported
descriptive analyses in enough detail to allow determination of the
effect size (e.g., magnitude of differences between groups). More
difficulty was seen with some of the specific statistical tests used,
as methods may not have been described completely, or incorrect
 G.S. Cooper, V. Meterko / Forensic Science International 297 (2019) 35–46
procedures were used (e.g., use of Chi-square test with sparse
cells). Most studies did not discuss eligibility criteria or details of
the selection or recruitment process, including participation rates.
4. Discussion
The set of studies identified in this review provide evidence of
the influence of contextual and confirmation bias on analysts’
conclusions, specifically among the studies with practitioners or
trainees examining case-specific information about the “suspect”
or crime scenario (in 9 of 11 studies examining this question),
procedures regarding use of exemplar(s) (in 4 of 4 studies), or
knowledge of a previous decision (in 4 of 4 studies). The two
studies among fingerprint analysts of the effect of emotional
context or seriousness of a crime on suitability decision-making
provide differing results; this may be a less important avenue for
future research than the other questions examined.
The two studies that did not observe an influence of case-
specific information on decision making were in shoe print and
toolmark (bullet) analysis [48,49]. It is not possible to draw
conclusions regarding differences in susceptibility to case-
specific information among disciplines because the number of
studies within a particular discipline is so small (in most cases,
n = 1), and in many of these studies, the small sample size, in
terms of participants, samples, or both, provides limited
statistical power. If differences are established, it would be
important to assess level of subjectivity in the analytical
procedures as an explanatory factor.
The provision of case-specific information to forensic scientists
raises several issues. As demonstrated in this set of studies, case-
specific information can be influential, even when the information
is wrong. Some types of information may be needed within the
context of the analysis, however, so it is important to carefully
consider whether, and when, information is provided to the
analyst [15]. Another issue is that the use of ancillary information
blurs the line between conclusions based on the attributes of the
analysis of the specified evidence (e.g., the features of a latent
print), which is the purview of the forensic scientist, and
conclusions based on consideration of multiple lines of evidence,
which is the purview of the trier of fact. The ancillary information
may also give an illusion of a stronger basis for a conclusion than
would be warranted by the analytical tools available to the forensic
science practitioner.
The results of the studies examining aspects of comparison
samples (target exemplars) provide insights regarding potential
improvements to standard operating procedures. As has been
suggested by Miller [33] and Wells et al. [14], the use of multiple
comparison samples, analogous to using fillers in a photo array,
may lead to a reduction in errors of identification.
The set of studies examining knowledge of a previous decision
suggest additional procedural improvements. A verification step
that is used only for some types of decisions creates a situation in
which the second analyst has knowledge of the previous
conclusion. If knowledge, or the perception of knowledge, of
another analyst’s conclusions influences decision-making, it would
be important to develop procedures that allow for truly indepen-
dent replication of results, for example through replication of all
analyses, or of an appropriately weighted sample of analyses to
prevent the second analyst from predicting or guessing the first
analyst’s decision. The influence of the results of an algorithm-
generated matching process on human decision-making is another
area requiring additional research.
Several studies examined, or at least discussed, a measure of
complexity or sample quality as an effect modifier in their analysis
of cognitive bias [38,45,48,52,56,57,59,61]. This is an important
avenue to pursue, as the effects may be most pronounced in more
43
complex or difficult analyses. Examination of effect modification
requires attention to sample selection and allocation in the study
design, and a larger sample size.
Our evaluation of study methods revealed areas in which
improvements could be made to the design or the reporting of
methods and results. Guidelines for reporting randomized
controlled trials [62] and observational epidemiology studies
[63], developed within the medical and health arena, may provide
a useful foundation to think about ways to achieve, or come close
to, an optimal design, and about the information that needs to be
reported to allow readers to fully appreciate the strengths and
limitations of the research. For example, even if the study will
depend on practitioners who volunteer to participate, thinking
through the population of interest, the recruitment process used
for this population, and data that could be used to compare the
sample to the population (or to compare participants and non-
participants) may lead to decisions that result in stronger designs
and more informative publications. If a study population is drawn
from a single workplace, it would be useful to collect and report
information on the total number of people eligible in the
workplace (to be able to determine the participation rate), and
relevant demographic and work-related variables that allow
assessment of differences between eligible participants and
non-participants. For studies with practitioners, it is important
to incorporate study samples within the workplace flow or to
conduct the study under workplace conditions. Allocation
procedures, for example for grouping participants for between-
person comparison designs, should be based on a randomization
scheme, and the randomization process should be described.
Blinding to the specific samples used in a study may be difficult,
but possible, for within- and between-person designs; it can be
useful to include in the design an assessment of the extent to which
blinding was achieved.
Sample size is relevant to questions regarding precision, rather
than bias; within-person comparisons can be more efficient
compared with between-person designs because of they allow
greater control of sources of variability. Although we did not
designate an ideal sample size, or a size that would be a critical
deficiency, we do note that many studies were quite small (<15
participants in total for within person comparisons, or per group).
Larger studies would be better able to address interactions
between study variables and provide a stronger basis for
estimating effects even if methods to combine results across
studies are used.
In summary, the available research supports the idea of
susceptibility of forensic science practitioners to various types
of cognitive bias, and the potential value of procedures designed to
reduce access to unnecessary information and to promote use of
multiple comparison samples rather than a single suspect
exemplar and replication of results by analysts blinded to previous
results. Future research should provide additional data in under-
studied disciplines, assess level of subjectivity in the analytical
procedures in relation to presence or magnitude of bias, and assess
sample complexity or difficulty as an effect modifier. Attention to
guidelines for designing and reporting studies may result in
methodologically stronger, and more comprehensively described,
studies.
Declarations of interest
None.
Funding
This research did not receive any specific grant from funding
agencies in the public, commercial, or not-for-profit sectors.
44 G.S. Cooper, V. Meterko / Forensic Science International 297 (2019) 35–46

Acknowledgements (Continued)
Ideal design attributes Critically deficient (high Intermediate levels: what
The authors thank Dr. Sherry Nakhaeizadeh, Centre for the risk of bias) could be improved?
Forensic Sciences of the University College of London, and Dr.  For any subjective as-
Sharon Kelley, Institute of Law, Psychiatry, and Public Policy of sessment of partici-
University of Virginia School of Medicine for their thoughtful pants’ decisions,
reviews of earlier drafts of this work. We also thank Drs. describe training, use
two raters and assess
Nakhaeizadeh, Dr. Kelley, and Dr. Itiel Dror, University College of level of agreement
London, for their reviews of the list of references.

Analysis and reporting of results

Appendix A. Study evaluation
 Data presentation or  Data presentation or  Provide everything
analysis provide effect results did not provide needed to interpret
1. Internal validity size, direction and effect size, direction results (magnitude, di-
variability measure and variability mea- rection, and variability
 Statistical procedures sure for the results in effect measures)
Ideal design attributes Critically deficient (high Intermediate levels: what
appropriate for the  Inappropriate statisti-  Assess adequacy of
risk of bias) could be improved?
data cal tests, without ad- assumptions-require-
Allocation and comparability  Relevant variables, in- equate data ments for statistical
For randomized designs: For randomized designs: For randomized designs: fluence of outliers and presentation to allow tests
 Randomization pro-  Between person com-  Provide details of the robustness of results correct testing to be  Where applicable,
cess described in de- parison with random- randomization proce- addressed in the conducted considered relevant
tail; would not allow ized allocation to dure; assure that par- analysis  Insensitive outcome variables, influence of
anticipation of alloca- groups; demographic ticipant or study  Outcome measure measure (would not outliers and robustness
tion and experience data personnel could not appropriately sensi- be able to ascertain an of results
 Within-person com- indicate differences anticipate order or tive to the type of effect if an effect was  Make sure text
parison with random- between groups that group assignment effect being assessed present) matches tables-figures
ized allocation of are not addressed in  Assess adequacy of
samples (sample se- the analysis randomization proce-
lection or order); dures with respect to
 Between person com- comparability between
parison with random- groups and provide
ized allocation to group-specific data for 2. External validity - participant eligibility and selection – if target
groups; demographic relevant characteristics
population is experienced practitioners
and experience data  Assure selection pro-
indicate similarity of cess and experience of
groups a “control” group dif- In Order of Decreasing  Limited to practitioners with specified
fers only with respect Generalizability to Target minimum level of experience (e.g., 2
to the specific inter- Population years)
vention under investi-  Limited to practitioners, no minimum
gation level of experience
 Address relevant dif-  Limited to practitioners, no information
ferences in the analysis about level of experience
and discussion  Students or trainees in the relevant
forensic science discipline
For non-randomized For non-randomized For non-randomized  General population or university students
designs: designs: designs:
 Relevant differences  Differences in case or  Assess comparability of
in case or person person attributes not groups (potential con- Other important design  Eligibility criteria, sampling frame and
attributes adequately addressed in the founding) in the anal- attributes recruitment-selection process specified
addressed in the analysis ysis and discussion  To the extent possible, participation rate
analysis (consider at each step of recruitment process
processes leading to assessed and high participation rate (e.g.,
selection into the >95%) achieved for all steps with a
sample/groups) specified sampling frame
 Data pertaining to participant character-
istics allow analysis of representativeness
Blinding – participants of study sample, and for selected charac-
 Blinding to purpose-  Evidence of failure of  Incorporate into work- teristics, allow comparison between par-
hypothesis-study participant blinding place or course activi- ticipants and non-participants
question ties
 Conducted as part of  Include assessment of
regular workplace or adequacy of blinding
course activities (do procedures
not know samples are
study samples)
References
 Adequacy of blinding
procedures assessed
[1] A. Tversky, D. Kahneman, Judgment under uncertainty: heuristics and biases,
Science 185 (1974) 1124–1131.
[2] R.N. Haber, L. Haber, The culture of science: bias and forensic evidence, J. Appl.
Blinding – Others Interacting with Participants (including anyone making
Res. Mem. Cogn. 2 (2013) 65–67, doi:http://dx.doi.org/10.1016/j.jar-
subjective interpretations of participants’ results or decisions)
mac.2013.01.005.
 Blinded to partici-  Evidence of failure of  To the extent possible,
[3] R.S. Nickerson, Confirmation Bias: a ubiquitous phenomenon in many guises,
pant group blinding, affecting build in protections to Rev. Gen. Psychol. 2 (1998) 175–220.
participant behavior prevent interactions [4] National Research Council, Strengthening Forensic Science in the United
or assessors’ decision- with participants by States: a Path Forward, National Academy of Sciences, 2009.
making people knowledgeable [5] S.M. Kassin, I.E. Dror, J. Kukucka, The forensic confirmation bias: problems,
of hypothesis and perspectives, and proposed solutions, J. Appl. Res. Mem. Cogn. 2 (2013) 42–52,
group assignments doi:http://dx.doi.org/10.1016/j.jarmac.2013.01.001.
 G.S. Cooper, V. Meterko / Forensic Science International 297 (2019) 35–46
[6] L. Butt, The forensic confirmation bias: problems, perspectives, and proposed
solutions—commentary by a forensic examiner, J. Appl. Res. Mem. Cogn. 2
(2013) 59–60, doi:http://dx.doi.org/10.1016/j.jarmac.2013.01.012.
[7] D. Charlton, Standards to avoid bias in fingerprint examination? Are such
standards doomed to be based on fiscal expediency?, J. Appl. Res. Mem. Cogn. 2
(2013) 71–72, doi:http://dx.doi.org/10.1016/j.jarmac.2013.01.009.
[8] S.D. Charman, The forensic confirmation bias: a problem of evidence
integration, not just evidence evaluation, J. Appl. Res. Mem. Cogn. 2 (2013)
56–58, doi:http://dx.doi.org/10.1016/j.jarmac.2013.01.010.
[9] S.A. Cole, Implementing counter-measures against confirmation bias in
forensic science, J. Appl. Res. Mem. Cogn. 2 (2013) 61–62, doi:http://dx.doi.
org/10.1016/j.jarmac.2013.01.011.
[10] I.E. Dror, S.M. Kassin, J. Kukucka, New application of psychology to law:
improving forensic evidence and expert witness contributions, J. Appl. Res.
Mem. Cogn. 2 (2013) 78–81, doi:http://dx.doi.org/10.1016/j.jar-
mac.2013.02.003.
[11] E. Elaad, Psychological contamination in forensic decisions, J. Appl. Res. Mem.
Cogn. 2 (2013) 76–77, doi:http://dx.doi.org/10.1016/j.jarmac.2013.01.006.
[12] R. Heyer, C. Semmler, Forensic confirmation bias: the case of facial image
comparison, J. Appl. Res. Mem. Cogn. 2 (2013) 68–70, doi:http://dx.doi.org/
10.1016/j.jarmac.2013.01.008.
[13] M. Triplett, Errors in forensics: cause(s) and solutions, J. Appl. Res. Mem. Cogn.
2 (2013) 63–64, doi:http://dx.doi.org/10.1016/j.jarmac.2013.01.007.
[14] G.L. Wells, M.M. Wilford, L. Smalarz, Forensic science testing: the forensic
filler-control method for controlling contextual bias, estimating error rates,
and calibrating analysts’ reports, J. Appl. Res. Mem. Cogn. 2 (2013) 53–55, doi:
http://dx.doi.org/10.1016/j.jarmac.2013.01.004.
[15] I.E. Dror, W.C. Thompson, C.A. Meissner, I. Kornfield, D. Krane, M. Saks, M.
Risinger, Letter to the editor- context management toolbox: a linear sequential
unmasking (LSU) approach for minimizing cognitive Bias in forensic decision
making, J. Forensic Sci. 60 (2015) 1111–1112, doi:http://dx.doi.org/10.1111/
1556-4029.12805.
[16] J. Kukucka, S.M. Kassin, P.A. Zapf, I.E. Dror, Cognitive bias and blindness: a
global survey of forensic science examiners, J. Appl. Res. Mem. Cogn. 6 (2017)
452–459, doi:http://dx.doi.org/10.1016/j.jarmac.2017.09.001.
[17] H. Arksey, L. O’Malley, Scoping studies: towards a methodological framework,
Int. J. Soc. Res. Methodol. 8 (2005) 19–32.
[18] B.D. McAuliff, J.L. Arter, Adversarial allegiance: the devil is in the evidence
details, not just on the witness stand, Law Hum. Behav. 40 (2016) 524–535, doi:
http://dx.doi.org/10.1037/lhb0000198.
[19] D. Mossman, When forensic examiners disagree: bias, or just inaccuracy?
Psychol. Public Policy Law 19 (2013) 40–55, doi:http://dx.doi.org/10.1037/
a0029242.
[20] D.C. Murrie, M.T. Boccaccini, J.T. Johnson, C. Janke, Does interrater (dis)
agreement on Psychopathy Checklist scores in sexually violent predator trials
suggest partisan allegiance in forensic evaluations? Law Hum. Behav. 32
(2008) 352–362, doi:http://dx.doi.org/10.1007/s10979-007-9097-5.
[21] D.C. Murrie, M.T. Boccaccini, L.A. Guarnera, K.A. Rufino, Are forensic experts
biased by the side that retained them? Psychol. Sci. 24 (2013) 1889–1897, doi:
http://dx.doi.org/10.1177/0956797613481812.
[22] D.V. Yokum, Debiasing the Courtroom: using Behavioral Insights to Avoid and
Mitigate Cognitive Biases, ProQuest Information & Learning, 2016.
[23] C. Semmler, The Impact of Individual Differences and Eye Movements on Facial
Comparison Performanc, (2012) .
[24] I. Dror, R. Rosenthal, Meta-analytically quantifying the reliability and
biasability of forensic experts, J. Forensic Sci. 53 (2008) 900–903, doi:
http://dx.doi.org/10.1111/j.1556-4029.2008.00762.x.
[25] J.P.T. Higgins, D.G. Altman, P.C. Gotzsche, P. Juni, D. Moher, A.D. Oxman, J.
Savovic, K.F. Schulz, L. Weeks, J.A.C. Sterne, Cochrane Bias Methods Group,
Cochrane Statistical Methods Group, The Cochrane Collaboration’s tool for
assessing risk of bias in randomised trials, BMJ 343 (2011), doi:http://dx.doi.
org/10.1136/bmj.d5928 d5928–d5928.
[26] J.A. Sterne, M.A. Hernán, B.C. Reeves, J. Savovic, N.D. Berkman, M. Viswanathan,
D. Henry, D.G. Altman, M.T. Ansari, I. Boutron, J.R. Carpenter, A.-W. Chan, R.
Churchill, J.J. Deeks, A. Hróbjartsson, J. Kirkham, P. Jüni, Y.K. Loke, T.D. Pigott, C.
R. Ramsay, D. Regidor, H.R. Rothstein, L. Sandhu, P.L. Santaguida, H.J.
Schünemann, B. Shea, I. Shrier, P. Tugwell, L. Turner, J.C. Valentine, H.
Waddington, E. Waters, G.A. Wells, P.F. Whiting, J.P. Higgins, ROBINS-I: a tool
for assessing risk of bias in non-randomised studies of interventions, BMJ
(2016) i4919, doi:http://dx.doi.org/10.1136/bmj.i4919.
[27] G.S. Cooper, R.M. Lunn, M. Ågerstrand, B.S. Glenn, A.D. Kraft, A.M. Luke, J.M.
Ratcliffe, Study Sensitivity: evaluating the ability to detect effects in
systematic reviews of chemical exposures, Environ. Int. 92–93 (2016) 605–
610, doi:http://dx.doi.org/10.1016/j.envint.2016.03.017.
[28] A.A. Rooney, G.S. Cooper, G.D. Jahnke, J. Lam, R.L. Morgan, A.L. Boyles, J.M.
Ratcliffe, A.D. Kraft, H.J. Schünemann, P. Schwingl, T.D. Walker, K.A. Thayer, R.
M. Lunn, How credible are the study results? Evaluating and applying internal
validity tools to literature-based assessments of environmental health
hazards, Environ. Int. 92–93 (2016) 617–629, doi:http://dx.doi.org/10.1016/j.
envint.2016.01.005.
[29] M.A. Hernán, B.C. Sauer, S. Hernández-Díaz, R. Platt, I. Shrier, Specifying a
target trial prevents immortal time bias and other self-inflicted injuries in
observational analyses, J. Clin. Epidemiol. 79 (2016) 70–75, doi:http://dx.doi.
org/10.1016/j.jclinepi.2016.04.014.
[30] I.E. Dror, K. Wertheim, P. Fraser-Mackenzie, J. Walajtys, The impact of human-
technology cooperation and distributed cognition in forensic science: biasing
45
effects of AFIS contextual information on human experts, J. Forensic Sci. 57
(2012) 343–352, doi:http://dx.doi.org/10.1111/j.1556-4029.2011.02013.x.
[31] M.C. Fysh, M. Bindemann, Human-computer interaction in face matching,
Cogn. Sci. 42 (2018) 1714–1732, doi:http://dx.doi.org/10.1111/cogs.12633.
[32] L.S. Miller, Bias among forensic document examiners: a need for procedural
change, J. Police Sci. Adm. 12 (1984) 407–411.
[33] L.S. Miller, Procedural bias in forensic science examinations of human hair,
Law Hum. Behav. 11 (1987) 157–163, doi:http://dx.doi.org/10.1007/
BF01040448.
[34] S. Nakhaeizadeh, I.E. Dror, R.M. Morgan, Cognitive bias in forensic
anthropology: visual assessment of skeletal remains is susceptible to
confirmation bias, Sci. Justice J. Forensic Sci. Soc. 54 (2014) 208–214, doi:
http://dx.doi.org/10.1016/j.scijus.2013.11.003.
[35] S. Nakhaeizadeh, I. Hanson, N. Dozzi, The power of contextual effects in
forensic anthropology: a study of biasability in the visual interpretations of
trauma analysis on skeletal remains, J. Forensic Sci. 59 (2014) 1177–1183, doi:
http://dx.doi.org/10.1111/1556-4029.12473.
[36] S. Nakhaeizadeh, R.M. Morgan, C. Rando, I.E. Dror, Cascading Bias of initial
exposure to information at the crime scene to the subsequent evaluation of
skeletal remains, J. Forensic Sci. (2017), doi:http://dx.doi.org/10.1111/1556-
4029.13569.
[37] I.E. Dror, G. Hampikian, Subjectivity and bias in forensic DNA mixture
interpretation, Sci. Justice J. Forensic Sci. Soc. 51 (2011) 204–208, doi:http://dx.
doi.org/10.1016/j.scijus.2011.08.004.
[38] I.E. Dror, D. Charlton, Why experts make errors, J. Forensic Identif. 56 (2006)
600–616.
[39] T. Laber, P. Kish, M. Taylor, G. Owens, N. Osborne, J. Curran, Reliability
Assessment Of Current Methods In Bloodstain Pattern Analysis, NIJ, 2014.
[40] M.C. Taylor, T.L. Laber, P.E. Kish, G. Owens, N.K.P. Osborne, The reliability of
pattern classification in bloodstain pattern analysis - PART 2: bloodstain
patterns on fabric surfaces, J. Forensic Sci. 61 (2016) 1461–1466, doi:http://dx.
doi.org/10.1111/1556-4029.13191.
[41] M.C. Taylor, T.L. Laber, P.E. Kish, G. Owens, N.K.P. Osborne, The reliability of
pattern classification in bloodstain pattern analysis, part 1: bloodstain
patterns on rigid non-absorbent surfaces, J. Forensic Sci. 61 (2016) 922–927,
doi:http://dx.doi.org/10.1111/1556-4029.13091.
[42] L. Lit, J.B. Schweitzer, A.M. Oberbauer, Handler beliefs affect scent detection
dog outcomes, Anim. Cogn. 14 (2011) 387–394, doi:http://dx.doi.org/10.1007/
s10071-010-0373-2.
[43] C.A.J. van den Eeden, C.J. de Poot, P.J. van Koppen, Forensic expectations:
investigating a crime scene with prior information, Sci. Justice 56 (2016) 475–
481, doi:http://dx.doi.org/10.1016/j.scijus.2016.08.003.
[44] C.A.J. van den Eeden, C.J. de Poot, P.J. van Koppen, The forensic confirmation
bias: a comparison between experts and novices, J. Forensic Sci. (2018).
[45] W.R. Oliver, Effect of history and context on forensic pathologist interpretation
of photographs of patterned injury of the skin, J. Forensic Sci. 62 (2017) 1500–
1505, doi:http://dx.doi.org/10.1111/1556-4029.13449.
[46] W.R. Oliver, X. Fang, Forensic pathologist consensus in the interpretation of
photographs of patterned injuries of the skin, J. Forensic Sci. 61 (2016) 972–
978, doi:http://dx.doi.org/10.1111/1556-4029.13092.
[47] W.R. Oliver, Reasons for lack of consensus in forensic pathologist interpreta-
tion of photographs of patterns of injury of the skin, J. Forensic Sci. 62 (2017)
674–680, doi:http://dx.doi.org/10.1111/1556-4029.13322.
[48] J.H. Kerstholt, R. Paashuis, M. Sjerps, Shoe print examinations: effects of
expectation, complexity and experience, Forensic Sci. Int. 165 (2007) 30–34,
doi:http://dx.doi.org/10.1016/j.forsciint.2006.02.039.
[49] J. Kerstholt, A. Eikelboom, T. Dijkman, R. Stoel, R. Hermsen, B. van Leuven, Does
suggestive information cause a confirmation bias in bullet comparisons?
Forensic Sci. Int. 198 (2010) 138–142, doi:http://dx.doi.org/10.1016/j.for-
sciint.2010.02.007.
[50] G. Langenburg, F. Bochet, S. Ford, A report of statistics from latent print
casework, Forensic Sci. Policy Manag. Int. J. 5 (2014) 15–37, doi:http://dx.doi.
org/10.1080/19409044.2014.929759.
[51] I.E. Dror, C. Champod, G. Langenburg, D. Charlton, H. Hunt, R. Rosenthal,
Cognitive issues in fingerprint analysis: inter- and intra-expert consistency
and the effect of a “target” comparison, Forensic Sci. Int. 208 (2011) 10–17, doi:
http://dx.doi.org/10.1016/j.forsciint.2010.10.013.
[52] P.A.F. Fraser-Mackenzie, I.E. Dror, K. Wertheim, Cognitive and contextual
influences in determination of latent fingerprint suitability for identification
judgments, Sci. Justice J. Forensic Sci. Soc. 53 (2013) 144–153, doi:http://dx.
doi.org/10.1016/j.scijus.2012.12.002.
[53] G. Langenburg, C. Champod, P. Wertheim, Testing for potential contextual bias
effects during the verification stage of the ACE-V methodology when
conducting fingerprint comparisons, J. Forensic Sci. 54 (2009) 571–582, doi:
http://dx.doi.org/10.1111/j.1556-4029.2009.01025.x.
[54] I.E. Dror, D. Charlton, A.E. Péron, Contextual information renders experts
vulnerable to making erroneous identifications, Forensic Sci. Int. 156 (2006)
74–78, doi:http://dx.doi.org/10.1016/j.forsciint.2005.10.017.
[55] M. Merlino, Reliability Validity, Accuracy, and Bias in Forensic Signature
Identification, NIJ, 2015.
[56] I.E. Dror, A.E. Péron, S.-L. Hind, D. Charlton, When Emotions Get the Better of
Us: The Effect of Contextual Top-down Processing on Matching Fingerprints,
Appl. Cogn. Psychol. 19 (2005) 799–809, doi:http://dx.doi.org/10.1002/
acp.1130.
[57] H. Earwaker, R.M. Morgan, A.J.L. Harris, L.J. Hall, Fingermark submission
decision-making within a UK fingerprint laboratory: Do experts get the marks
46 G.S. Cooper, V. Meterko / Forensic Science International 297 (2019) 35–46
that they need? Sci. Justice J. Forensic Sci. Soc. 55 (2015) 239–247, doi:http://
dx.doi.org/10.1016/j.scijus.2015.01.007.
[58] L.J. Hall, E. Player, Will the introduction of an emotional context affect
fingerprint analysis and decision-making? Forensic Sci. Int. 181 (2008) 36–39,
doi:http://dx.doi.org/10.1016/j.forsciint.2008.08.008.
[59] N.K.P. Osborne, S. Woods, J. Kieser, R. Zajac, Does contextual information bias
bitemark comparisons? Sci. Justice J. Forensic Sci. Soc. 54 (2014) 267–273, doi:
http://dx.doi.org/10.1016/j.scijus.2013.12.005.
[60] J.P. Higgins, S. Green, Cochrane Handbook for Systematic Reviews of
Interventions, (2011) . http://handbook-5-1.cochrane.org/.
[61] N.D. Lange, R.P. Thomas, J. Dana, R.M. Dawes, Contextual biases in the
interpretation of auditory evidence, Law Hum. Behav. 35 (2011) 178–187, doi:
http://dx.doi.org/10.1007/s10979-010-9226-4.
[62] K.F. Schulz, D.G. Altman, D. Moher, CONSORT 2010 Statement: updated
guidelines for reporting parallel group randomised trials, BMJ 340 (2010), doi:
http://dx.doi.org/10.1136/bmj.c332.
[63] J.P. Vandenbroucke, E. von Elm, D.G. Altman, P.C. Gøtzsche, C.D. Mulrow, S.J.
Pocock, C. Poole, J.J. Schlesselman, M. Egger, STROBE Initiative, Strengthening
the reporting of observational studies in epidemiology (STROBE): explanation
and elaboration, Int. J. Surg. Lond. Engl. 12 (2014) 1500–1524, doi:http://dx.doi.
org/10.1016/j.ijsu.2014.07.014.
[64] L. Smalarz, S. Madon, Y. Yang, M. Guyll, S. Buck, The perfect match: do criminal
stereotypes bias forensic evidence analysis? Law Hum. Behav. 40 (2016) 420–
429, doi:http://dx.doi.org/10.1037/lhb0000190.
[65] S.V. Stevenage, A. Bennett, A biased opinion: demonstration of cognitive bias
on a fingerprint matching task through knowledge of DNA test results,
Forensic Sci. Int. 276 (2017) 93–106, doi:http://dx.doi.org/10.1016/j.for-
sciint.2017.04.009.
[66] J. Kukucka, S.M. Kassin, Do confessions taint perceptions of handwriting
evidence? An empirical test of the forensic confirmation bias, Law Hum. Behav.
38 (2014) 256–270, doi:http://dx.doi.org/10.1037/lhb0000066.
"The Devil in the White City": How to Get Away with Murder
Some people do not realize what is really happening in front of them, no matter how obvious it
seems to other people. In the case of H.H. Holmes, he is able to lie and charm his way into
making people trust him so that he can get away with murder. In Erik Larson’s The Devil in the
White City, the author presents his audience with the thoughts of both Holmes and his victims,
clouding the light of perfection Holmes creates with the dark reality of his true intentions in order
to let the readers see how his victims’ ignorance allows his evil ways to hide under the good
that they fail to look past.

Larson gives his readers a glimpse into Holmes’ mind in order to allow them to contrast what
he says and what he truly feels. In the first known murder presented in the novel, Larson tells
his audience that Holmes “knew he possessed great power over Julia… [He] possessed [her]
now as fully as if she were an antebellum slave, and he reveled in his possession” (146). The
om

reader can tell that Holmes believes he dominates Julia, that she has no control of what he will
do to her. This is frightening to the audience as they now see Holmes is quite crazy. Not only
c

does Holmes possess these women, he sees them as objects. Larson entitles one of his
r.

chapters “Acquiring Minnie” (198) to tell of how Holmes seduced Minnie so he could kill her. He
xe

uses the same word “acquired” to describe how Holmes “acquired high-grade furnishings”
(198) for his hotel. This directly compares Minnie to an inanimate piece of furniture, showing
fi

how Holmes sees these women as objects to be bought with charm and gifts and fake love.
Larson hopes his audience will think poorly of Holmes because of the way he thinks of these
es

women; he hopes they will be able to easily see past Holmes fake plays and discover his true
ad

psychopathic schemes. Larson also mentions a few very disturbing details about the murder of
Julia. He tells us how Holmes finds it “singularly arousing” (148) when Julia begins to fight
gr

back, and how “the sensation, as always, was pleasant and induced in him a warm languor, like
the feeling he got after sitting too long in front of a hot stove” (148-149). Larson allows his
readers to see how Holmes gets this soothing sexual release from suffocating this poor woman
who he faked out and lied to so he could murder her. He hopes to disgust the readers with this
description and cause them to view Holmes in a negative light. This way when they see what
these women are thinking, they realize the tricks that Holmes plays and how he clouds the truth
with his charm, taking advantage of the fact that these women are too ignorant, mesmerized by
Holmes, to see the dangerous truth.

Holmes’ victims are so charmed, they don’t realize how dangerous he really is, Larson lets his
audience into the mind of a few of Holmes’ victims, one of them being Georgiana Yoke. The
audience hears that “she had never met anyone like him. He was handsome, articulate, and
clearly well off” (Larson 307). They are able to see what Georgiana truly believes that Holmes is

Need help with the assignment?

Our professionals are ready to assist with any writing!

GET HELP
a wonderful man. Because Larson reveals this point of view to the audience, they are able to
contrast it with the view of Holmes and see how easily he tricked these women and got away
with it. Another girl, Anna, was suspicious of Holmes until she met him and “his warmth and
smile and obvious affection for Minnie caused [her] suspicions to quickly recede” (Larson 264).
“Holmes was such a charming man. And now that Anna knew him, she saw that he really was
quite handsome” (Larson 292). Something about him caused her, like many before her, to let
her guard down and not question his actions, no matter how skeptical they appear to the reader.
The audience, though, is able to realize that when Holmes invites her to his hotel, alone,
something bad is going to happen. It is especially apparent that Holmes plans on murdering
Ana when he asks her to step into his walk-in vault and “cheerfully, she [complies]” (294). She
trusts Holmes so much; she is so fascinated by his perfection, that the thought of danger never
crosses her mind. These women, so charmed by and trusting of Holmes, let their guard down
and walk to their death, but they don’t see it that way. The only reason the audience can see
the danger is because Larson reveals Holmes’ point of view to them. Since the audience is
aware of Holmes’ tricks, these women appear to be to blame for their death because they
should have seen it coming. It is not entirely their fault though, as Holmes charms them to gain
om
their trust, causing them to be unaware of his true intentions. It is only so obvious to the reader
because Larson allows his audience to see inside both Holmes and his victims’ minds, getting
c

both views of the situations and always being aware of what is really about to happen while
r.

others remain innocent.

xe

No matter what Holmes does, the people around him never suspect him of any sketchy
fi

business. He is so narcissistically confident, that when he hires Charles Chappell to make a

murdered body into a skeleton, then man doesn’t think anything of the corpse on the table
es

which “looked like that of a jack rabbit which had been skinned by splitting the skin down the
face and rolling it back off the entire body” (Larson 151). Larson informs his audience that the
ad

body did not bother Chappell, “for [he] knew that Holmes was a physician” (151). The man was
gr

easily fooled into thinking Holmes was simply dissecting the body for research. Holmes, as
convincing as he is, was able to let someone walk into his torture chamber, see a skinless dead
body, and still have no suspicion of Holmes. Larson makes sure to include the details about the
body so the audience sees just how obvious it was to us, knowing Holmes, that he murdered
this person, and Chappell did not notice anything like his audience did. Larson also includes
Chappell’s reasoning for not thinking anything of the dead body; Chappell knew Holmes was a
physician so it was perfectly normal to have a dismantled dead body lying on a table in the
basement of a hotel. The audience only realizes this is not right because Larson has given them
a glimpse into the mind of Holmes; these bystanders are completely oblivious to the murders
literally in front of them. Even to a victim, it is not apparent that she is about to be murdered. In
the case of Anna, after Holmes locks her in the walk-in vault, she continues to not believe that
he is a bad guy. She “guessed that [Holmes], unaware of her plight, had gone elsewhere in the
building” (Larson 295). She figured that would “explain why he still had not come despite her

Need help with the assignment?

Our professionals are ready to assist with any writing!

GET HELP
 pounding” (Larson 295). In this hypnotized frame of mind Holmes has put her in by use of his
charm and sly seduction, she is unable to fathom what is really going on. The audience knows
what is really about to happen because of the glimpse into Holmes’ mind. Had the audience not
had a general idea of what goes on in Holmes’ mind, they might not have figured out that she
was about to be killed. Since Larson gave them that opposite point of view, though, they are
able to tell that this woman is about to be murdered. The audience sees how little this woman
knows about Holmes, and realizes how, because she was so charmed by this man and didn’t
suspect him of anything, he was able to get away with everything without a problem. After
Anna’s panic finally starts to settle in, the readers get a glimpse of Holmes’ thoughts with Anna
trapped and dying in the fault. They see Holmes deciding whether or not to “open the door and
look in on Anna and give her a big smile – just to let her know that this was no accident – then
close the door again, slam it, and return to his chair” (Larson 295). This clearly sadistic thought
process is visible to the reader, as they are observing the situation from an outside point of view
presented to them by Larson, but Anna was completely unable to see this coming. Larson
hopes that by allowing his readers to see into both Holmes and his victims’ minds, they can see
the difference between the fake utopian reality Anna is living in and the real mad reality that
om
Holmes is killing her in.
c

Larson gives the readers a chance to see two sides of the same series of stories: Holmes’
r.

view, and his victims’ views. They are first able to see what goes on in the mind of Holmes.
Larson portrays him as a sadistic psychopath who gets off on torturing and murdering young
xe

women. On the other hand, Larson also reveals to his audience the thoughts of the women who
fi

become Holmes’ victims. They see him as a charming, handsome young man, and they trust
him almost immediately. They suspect nothing of actions that the reader might be skeptical of.
es

Because Larson portrays both points of view, the reader is able to see how Holmes might get
away with these murders. He allows the audience to see how, as Holmes charms and seduces
ad

these young women, he is really making sure these women trust him, so he will have no
gr

problem taking their lives; he takes advantage of these women’s innocence in order to get
away with murder.

Need help with the assignment?

Our professionals are ready to assist with any writing!

GET HELP

Powered by TCPDF (www.tcpdf.org)