EBM Self Instructional Manual 2010

EVIDENCE BASED
MEDICINE
Self-Instructional Manual
Noel L. Espallardo, MD, MSc

Department of Clinical Epidemiology
UP College of Medicine and Philippine General Hospital

Evidence-based Medicine Self-Instructional Manual
EVIDENCE BASED MEDICINE

INTRODUCTION

Medical practice is a dynamic endeavor. Challenging problems arise, new modalities of
treatment are promoted, disease management done under minimal or far from ideal
conditions. Sometimes decisions are made with incomplete information.

Suppose a patient who consulted for cough productive of yellowish phlegm and asked for a
prescription of an antibiotic. Will you prescribe it or not? In the old practice faced with this
question, a physician will just ask a colleague or an expert for the answer or rely on his/her
prior understanding of the disease. He/she may also prescribe a drug because of the
promotional lecture sponsored by the manufacturer.

In evidence‐based medicine (EBM) a new paradigm is introduced. Before the physician makes a
decision, he/she will first try to retrieve his latest article about the topic, appraise the article
then makes a decision. Later, he/she evaluates the effectiveness of his decision. This loop
ensures improvement in the quality of care.

DEFINITION OF EBM

Evidence‐based medicine is defined as the process of systematically finding, appraising, and
using research findings as the basis for clinical decisions (NLM, 2008). Evidence‐based medicine
follows four steps:

• formulate a clear clinical question from a patient's problem
• search the literature for relevant clinical articles
• evaluate (critically appraise) the evidence for its validity and usefulness
• implement useful findings in clinical practice

THE PARADIGM SHIFT

The new paradigm makes learning new things more self‐directed and less reliant on teachers.
Students can gain the skills to make independent assessments of evidence, and thus evaluate
the credibility of opinions being offered by experts. The purpose of this course is to introduce
the concept of evidence based medicine and the use of these concepts to improve the quality
of his/her own practice.

Page 1
WHY TEACH EBM

Medical education faces a problem in a present setting: too much information, too little time,
too many students in crowded rooms, and exams that discouraged real life‐long learning
(Rangachari, 2007). There is a need to make doctors asks questions about useful information
and try to seek the answer for themselves.

The term "evidence based medicine" was coined at McMaster Medical School in Canada in the
1980's to label an “active clinical learning” strategy, which people at the school had been
developing for over a decade (NLM, 2008). Randomized controlled trials have shown that
evidence‐based medicine learning is more effective than didactic learning among medical
interns in family medicine. They provide better care in terms of providing treatment to patients
with hypertension (Espallardo, 2006).

Currently most physicians report a moderate amount of exposure to EBM. Physicians in clinical
research careers were more favorable towards EBM than those in the clinical practice careers
(Luebbe, 2007). But among those who were already exposed to EBM there was mismatch
perceived competence and their actual performance. This suggests that better education in
EBM is needed (Caspi, 2006).

Starting the training in the undergraduate program (medical students) is also a logical
approach. Early experience helps medical students learn, helps them develop appropriate
attitudes towards their studies and future practice and orients medical curriculums towards
local needs. (Littlewood, 2005)

OBJECTIVES

After going through the readings and workshops of this manual, the participants should be able
to:

• define and describe the steps in applying evidence‐based medicine into his/her own
clinical practice
• appraise and use randomized controlled trials and other types of studies in solving
clinical problems in clinical practice
• make an efficient literature search and identify problems and solutions in the
application of evidence based medicine

METHODS

This is a series of self‐reading materials and group discussions. Allot a fix time for you to read
the reading assignment in the manual. Conduct the group discussion with at least 5 of your
Page 2
colleagues. Assign a facilitator and a co‐facilitator/scribe for each discussion. Observe the rules
enumerated in the succeeding section. Monitor your progress by reviewing the checklist
provided before each section.

The pace of learning depends on your time schedule. However, I suggest that you allot one day
a week for the reading time and group discussion. You can also try to apply critical appraisal by
yourself with other topics of interest.

RULES OF DISCUSSION

The ground rules that are encouraged to be observe during the group discussion are as follows:

• Honor the established time limits.
• Allow one person to talk at a time.
• Focus on the topic. Avoid sideline conversation.
• Listen to what others have to say. All ideas are valued.
• Encourage participation in the discussion for all participants.
• Critique on ideas and thoughts, not on the person.

ROLES AND RESPONSIBILITIES

FACILITATOR

The facilitator serves as the process facilitator. He/she is also responsible for the content and
final outcome of the discussion. His/her responsibilities are to:

• Provide the process to achieve the objectives and desired outcomes.
• Pose probing but non‐threatening questions to provoke thought, clarify discussion and
bring insight on some points.
• Provide balance. Facilitate rather than lead the discussion.
• Remain neutral on content and avoid evaluation and decisions on ideas.
• Encourage equal participation among group members.

CO‐FACILITATOR/SCRIBE

The co‐facilitator helps the facilitator to achieve his/her objectives. He/she may join the group
discussion, but must bear in mind of his/her other functions:

• Be a timekeeper to ensure progress.
• Contribute ideas to the topic being discussed.
Page 3
• Meet with the facilitator during the break to discuss the process and ideas on how to
proceed.
• Record essential information (content) and observation (group process) for post‐
discussion processing and evaluation.

PARTICIPANTS

The participants are encouraged to actively participate in the discussion. The amount of
learning you will get from this course is proportional to the degree of your participation.

REFERENCES

Caspi O, McKnight P, Kruse L, Cunningham V, Figueredo AJ, Sechrest L. Evidence‐based medicine: discrepancy
between perceived competence and actual performance among graduating medical students. Med Teach. 2006
Jun;28(4):318‐25.

Espallardo NL. Effectiveness of Critical Appraisal Workshop as a Method for Disseminating a Clinical Practice
Guideline on Hypertension. Fil Fam Phys 2006; 44 (2): 54‐60.

Littlewood S, Ypinazar V, Margolis SA, Scherpbier A, Spencer J, Dornan T. Early practical experience and the social
responsiveness of clinical education: systematic review. BMJ. 2005 Aug 13;331(7513):387‐91.

Luebbe AM, Radcliffe AM, Callands TA, Green D, Thorn BE. Evidence‐based practice in psychology: perceptions of
graduate students in scientist‐practitioner programs. J Clin Psychol. 2007 Jul;63(7):643‐55.

National Library of Medicine. Medical Subject Headings. www.ncbi.nlm.nih.gov/sites/entrez (May 27, 2008).

Rangachari PK. Back to the future? Active learning of medical physiology in the 1900s. Adv Physiol Educ. 2007
Dec;31(4):283‐7.

Users' Guides to Evidence‐based Medicine. JAMA. 1992;268(17):2420‐5.

Page 4
SEARCHING THE MEDLINE

OBJECTIVES

The purpose of the reading assignment is to introduce to the participants how to formulate a
question and search the MEDLINE for the answer.

At the end of the reading session, you should be able to search the MEDLINE with skill and
confidence that you were able to retrieve the right answer to the problem.

INSTRUCTIONS

Read the assignment for an article on MEDLINE search. Focus on the use of key terms and how
to use the operant words AND or OR. After reading the paper, proceed to the library or internet
and try searching in the MEDLINE.

Page 5
SEARCHING THE MEDLINE

CLINICAL SCENARIO

Supposing a patient with cough came in to your clinic and asks for an anti‐biotic drug because
he wants to be relieved of his cough right away. Will you prescribe it?

Patients usually come in to the clinic for problems. Unfortunately these problems are vague and
sometimes not clearly stated. To state the problem clearly, you must bear in mind that there
are only three important elements that the patient want to know:

• What their diseases are
• What treatment they should be given
• What is the expected outcome of the treatment

These three important elements in clinical research are basically:

• the patient (P)
• the intervention/exposure (I)
• the outcome (O)

Sometimes the researcher can add

• method (M)

Going back to our scenario, my clearly stated problem will be: “Among patients with cough (P)
will anti‐biotic (I) provide symptom relief faster?”

TRADITIONAL METHOD OF LITERATURE SEARCH

A recent survey of important knowledge sources that influence clinical practice was conducted
among faculty members, fellows and residents of a large teaching tertiary care hospital. The
results showed that the most important resources were English journals, text books and
experience (Yousefi‐Nooraie, 2007). Thus educational programs to develop skills of efficiently
searching the research literature need to be developed. Brief (two‐hour) instructional
intervention on EBM‐based techniques for searching Medline for evidence related to a clinical
problem provided to the students have been shown to be effective. With this training, students
had fewer search errors and correspondingly higher quality searches. (Gruppen, 2005)

Page 6
PUBMED, ENTREZ AND MEDLINE

PubMed was developed by the National Center for Biotechnology Information (NCBI) at the
National Library of Medicine (NLM), located at the U.S. National Institutes of Health
(NIH). PubMed provides access to bibliographic information that includes MEDLINE, as well as:

• out‐of‐scope citations (e.g., articles on plate tectonics or astrophysics) from certain
MEDLINE journals, primarily general science and chemistry journals
• citations that precede the date that a journal was selected for MEDLINE indexing.
• additional life science journals that submit full text to PubMed Central and receive a
qualitative review by NLM.

MEDLINE is the largest component of PubMed and is the freely accessible online database of
biomedical journal citations and abstracts created by the U.S. National Library of Medicine
(NLM). Approximately 5,200 journals published in the United States and more than 80 other
countries have been selected and are currently indexed for MEDLINE. Entrez is the text‐based
search and retrieval system used at NCBI for services including PubMed.

THE PUBMED HOMEPAGE

BASIC SEARCH STRATEGY

The first step in using PubMed is to first develop a search strategy, a plan that helps you look
for the information you need. This can be done by doing the following steps:

Page 7
• Identify the key concepts (should include the PIOM discussed earlier)
• Determine alternative terms for these concepts (can be facilitated with MESH term
search)
• Refine your search (use limits like publication dates, study subjects, study designs,
patient age, etc)

Our clinical question in the previous scenario was “Among patients with cough (P) will anti‐
biotic (I) provide symptom relief faster?” The key concepts in my search terms are:
• Cough
• Antibiotics
• Cure

When I type the key term “cough” in the search box the yield was 31,188 articles and it will be
impossible for me to browse these articles. When I typed “cough AND antibiotics" the yield was
2,410 and when I typed “cough AND antibiotics AND cure” the yield was 51 articles. Then I can
use the “limit” function to randomized controlled trials. Now I can browse through these
articles.

Page 8

REFERENCES

Gruppen LD, Rana GK, Arndt TS. A controlled comparison study of the efficacy of training medical students in
evidence‐based medicine literature searching skills. Acad Med. 2005 Oct;80(10):940‐4.

National Library of Medicine. PubMed OVerview. www.ncbi.nlm.nih.gov/sites/entrez (May 27, 2008).

Yousefi‐Nooraie R, Shakiba B, Mortaz‐Hedjri S, Soroush AR. Sources of knowledge in clinical practice in
postgraduate medical students and faculty members: a conceptual map. J Eval Clin Pract. 2007 Aug;13(4):564‐8.

Page 9
CRITICAL APPRAISAL OF AN ARTICLE

ABOUT DIFFERENTIAL DIAGNOSIS
OBJECTIVES

The purpose of the reading assignment is to introduce to the participants the concept of
medical decision making using an article about differential diagnosis.

At the end of the reading session, you should be able to answer the user guides questions for
the workshop.

INSTRUCTIONS

Read the assignment for an article about a differential diagnosis. Focus on the critical appraisal
questions, why they are asked and how to get the answers from the paper. After reading the
paper you can proceed to conduct the group workshop.

Page 10
CLINICAL DECISION ON
DIFFERENTIAL DIAGNOSIS
CLINICAL SCENARIO

Suppose a 35‐year old female patient came in to your clinic for fever and abdominal pain for a
week. There was neither diarrhea nor dysuria. You read in the papers that there was an
increased incidence of dengue fever in children. How will you optimize (request only for what is
essential) the diagnostic laboratory tests for this patient?

Naturally you can do that if you already have an initial diagnosis in mind. Unfortunately there
might be many of them. This session will help you trim down the differential diagnosis and
request only for the laboratory tests that are essential. Differential diagnosis is the method of
limiting the possible causes of the patient’s symptoms before making a final diagnosis.
Identifying the right differentials will make patient management more focused and efficient.

Differential diagnosis can be arrived at by using the anatomic approach i.e. considering the
possibilities based on organs that may be affected within the proximity of the symptom like
chest pain may have differential diagnosis like herpes zoster (skin), costochondritis (ribs),
pneumonia (lings) or angina (heart). If the symptom is systemic like fever, the differentials can
by be pathophysiology i.e. vascular, inflammatory/infectious, neoplastic/neurologic,
degenerative, intoxication/idiopathic, congenital, allergic/autoimmune, trauma and endocrine
(VINDICATE) (Friedland, 1998). With these approaches however the frequency or probability of
each differential will not be known..

If we consider all known causes equally possible (the ‘possibilistic’ approach), then the patient
will have unnecessary diagnostic tests performed on them. Instead, we must considering first
those that are more common (a ‘probabilistic’ approach), or more serious if left undiagnosed
and untreated (a ‘prognostic’ approach) or more responsive to treatment (Richardson, 1999).
And they are important because the probabilities of the individual differential will help us focus
our diagnostic strategies as shown in the table below.

The disease probabilities can be taken from population prevalence statistics or from original
research. Research studies focus more directly on the frequency of diseases that cause
symptoms (Kroenke, 1997) are preferred over population survey because they are more
associated with presenting symptoms.

Page 11
Probabilities of Differential Diagnosis and Recommended Diagnostic Strategies

Differential Diagnosis Diagnostic Tests Treatment

Working diagnosis – most Choose test(s) with high Start empiric treatment
specificity and LR+ much
possible cause that should be larger than one.
ruled in

Alternative diagnosis – other Choose test(s) with high Start supportive treatment
sensitivity and LR- much
possibilities that should be smaller than one.
ruled‐out

Remote diagnosis None None

SEARCH

You found an unpublished retrospective study in the archives of your department written by a
previous resident Santos AR, entitled “Differential diagnosis of typhoid fever in the emergency
room”.

CRITICAL APPRAISAL

RELEVANCE QUESTION

• Is the objective of the article on differential diagnosis similar to your clinical dilemma?

To answer this question, look at the objective of the study. It is important that your article is
relevant to the question you have raised in order for you to make maximum use of the results
of the study and be able to apply it to decision making that influences patient care. For
differential diagnosis it is important that the focus is to find the cause of symptoms, clinical and
laboratory presentation among patients similar to your patient or case scenario.

VALIDITY GUIDES

• Did the study patients represent the full spectrum of patients who present with this
clinical problem?
Page 12

Study designs that answer clinical questions like differential diagnosis can be a cross‐sectional
study or cohort study. An important element with these designs is how the subjects are
recruited so they can represent other patients who may also have the same symptoms i.e.
representativeness.

The definition of the clinical problem under study describes the population to which the study
will be applied. The problem usually is a symptom or an abnormal physical examination such as
headache or abdominal mass or a combination of symptoms and abnormal physical findings like
headache and facial asymmetry. This is usually defined in the inclusion and exclusion criteria of
the study.

With the symptom already defined, the other strategies that can assure representativeness are
any of the following:
• Random selection – not always possible in clinical setting
• Consecutive patient recruitment – most feasible
• Recruitment in defined setting – must always be done

The Article by Santos included patients consulting for fever in the emergency room. Although
the inclusion criteria were fever alone as the chief complaint, there was a subgroup analysis of
patients with fever and abdominal complaint. The total number of patients included in the
study was 235. This coincides with your case scenario.

• Were the criteria for each final diagnosis explicit and credible?

Determination of final diagnosis must be clearly described, may not necessarily be based on the
ultimate reference standard. However the criteria must be explicit enough to make sure that
different clinician will arrive at the similar diagnosis (inter‐rater reproducibility).

The final diagnosis in Santos’s paper was based on clinical syndromes and criteria. Blood
cultures, ultrasound and other tests were not done to establish the final diagnosis in only 48%
of the cases.

• Was the diagnostic work‐up comprehensive and consistently applied?

The set of diagnostic work‐up should be thorough to come up with an accurate diagnosis. Then
a minimum set of diagnostic work‐up that includes a thorough history and physical examination
and a few initial laboratory tests should have been applied consistently for all patients. This can
be answered when the study described a prospective approach in identifying patients in the
study. Retrospective approach is usually limited because, records cannot guarantee a standard
diagnostic approach for everybody.

Page 13
The diagnostic tests done for 85% of patients in the Santos study were CBC, urinalysis and stool
examination. Temperature was measured using a mercury type thermometer and records with
“febrile” as reported documentation of fever were excluded.

• For initially undiagnosed patients, was follow‐up to come up with a diagnosis
sufficiently long and complete?

Sometimes the diagnosis at the early stage of the disease is really difficult and the patient may
be classified as not having the disease or undetermined. To assure ourselves with the eventual
diagnosis of undetermined cases, we may have to observe them over time.

The Santos study observed the patients for 24 to 48 hours in the emergency room. In most
patients antibiotics were started and the patients were sent home without fever.

OVERALL, IS THE STUDY VALID?

Although the study was a retrospective study, you decided that you can use this article because
it is the only available study in your setting.

WHAT ARE THE RESULTS?

What were the diagnoses and their probabilities? How precise are the estimate of the
probabilities?

The probabilities of the differential diagnosis are reported as either incidence or prevalence
with their 95% confidence interval.

In the Santos study, the following top three diseases were the most common diagnosis given to
adult patients with fever: a) typhoid fever, 34%; b) urinary tract infection, 32%; and c) acute
gastroenteritis, 29%. No confidence intervals were reported.

CAN THE RESULTS HELP ME IN CARING FOR MY PATIENTS?

• Are the study patients similar to my own?

For a study on differential diagnosis be applied to your patient you have to be assured that the
characteristics of your patient is similar to the study’s inclusion criteria.

In the Santos study, they included patients consulting in the emergency room but were
eventually sent home. The cases seen in this study seemed to be the milder cases similar to
your patient.

• Do you think the disease probabilities in the study still apply today?

Page 14
Disease prevalence and incidence change across time. Old disease can be controlled because of
effective treatment. Thus a paper on differential diagnosis may still include smallpox for
patients with fever and skin lesions in the 1950’s, the probability is almost zero today. The
probability of Dengue fever may differ in different times of the year. A little knowledge on
epidemiology of disease across time may be necessary to have an accurate answer to this
question. However if the disease in question does not vary over time then this is not a problem.

The study of Santos was a three‐year retrospective study from January 1988 to December 1991.
Seasonal variation may have been accounted for but the study is already 9 years old.
Unfortunately you cannot find a more recent one.

RESOLUTION OF THE PROBLEM IN THE SCENARIO

After appraising the study of Santos you decided that your diagnostic tests will focus on ruling
in or ruling out typhoid fever, urinary tract infection and gastroenteritis.

REFERENCES

Friedland ed. Evidence‐based Medicine: A framework for clinical practice. Appleton and Lange, 1998.

Kroenke K. Symptoms and science: the frontiers of primary care research [Editorial]. J Gen Intern Med 1997; 12:
509 ‐ 510.

Richardson WS, Wilson MC, Guyatt GH, Cook DJ, Nishikawa J, and the Evidence Based Medicine Working Group.
JAMA, 1999 Apr 7; 281(13):1214‐9.
Page 15
WORKSHOP
CRITICAL APPRAISAL OF
AN ARTICLE ABOUT DIFFERENTIAL
DIAGNOSIS
(SESSION BRIEFING)

OBJECTIVES

The purpose of the workshop is to introduce to the participants the concept of medical decision
making about differential diagnosis. Another objective is to introduce concepts of critical
appraisal of an article regarding differential diagnosis focusing on the following:

• validity
• interpretation of the results
• applicability of the results

INSTRUCTIONS

Divide the participants into groups of six to ten persons. Assign a case scenario to each group
and formulate an answerable problem from the scenario. Establish initial group consensus on
how to proceed with the scenario.

Ask the group to read the article retrieved to answer the problem in the scenario. Focus on the
abstract, methods and results section. Again establish a group consensus on how to proceed
with the scenario. Note any change in decisions.

Critically appraise the article using the appraisal sheet provided. Answer validity questions,
analyze the results and determine the applicability of the results. Establish another group
consensus and note any change in decision.

Process the exercise. Focus on barriers and solution to the application of the exercise in usual
clinic practice.

Page 16
TIME ALLOTTED

The time allotted for this workshop is two hours. The recommended break‐up is:

• 15 minutes to analyze the scenario and develop consensus
• 15 minutes to read the article
• 45 minutes to appraise the validity
• 15 minutes to analyze results
• 20 minutes to establish applicability
• 10 minutes to summarize the process

DESIRED OUTCOME

The participants should make a clinical decision on the scenario based on the critical appraisal
of the evidence.

Page 17
Appraisal Sheet
CLINICAL DECISION ON DIFFERENTIAL DIAGNOSIS

CLINICAL SCENARIO OR
QUESTION

SEARCH

CRITICAL APPRAISAL

RELEVANCE Is the objective of the article on differential diagnosis similar to
your clinical dilemma?

PRIMARY VALIDITY Did the study patients represent the full spectrum of patients who
GUIDES present with this clinical problem?
Definition of the clinical problem or the patient whom the study
will be applied.

Were the criteria for each final diagnosis explicit and credible?
Determination of final diagnosis must be clearly described, may not
necessarily be the ultimate reference standard.

SECONDARY VALIDITY Was the diagnostic work‐up comprehensive and consistently
GUIDES applied?

Page 18
For initially undiagnosed patients, was follow‐up to come up with
a diagnosis sufficiently long and complete?

OVERALL, IS THE STUDY
VALID?

WHAT ARE THE What were the diagnoses and their probabilities? How precise are
RESULTS? the estimates of probabilities?

CAN THE RESULTS HELP Are the study patients similar to my own?
ME IN CARING FOR MY Inclusion criteria, exclusion criteria, clinical definition
PATIENTS?

Do you think the disease probabilities in the study still apply
today?
Is the study recent? Could the probabilities change since the study
publication?

RESOLUTION OF THE
PROBLEM IN THE
SCENARIO
Page 19
AN ARTICLE ABOUT A DIAGNOSTIC
TEST
(SESSION BRIEFING)

OBJECTIVES

medical decision making using an article about a diagnostic test. At the end of the reading
session, you should be able to answer the user guides questions for the workshop.

INSTRUCTIONS

Read the reading assignment for an article about a diagnostic test. Focus on the critical
appraisal questions, why they are asked and how to get the answers from the paper.

After reading the paper you can proceed to conduct the group workshop.

Page 20
CLINICAL DECISION ON A
DIAGNOSTIC TEST

CLINICAL SCENARIO

A 70 years old female patient came in to the clinic complaining of forgetfulness. She’s afraid
that she has dementia just like her sister. She does not want to be subjected to MRI or CT scan.
You referred her to the psychiatric resident and she suggested that you perform the Mini‐
mental State Examination (MMSE), but you doubt her decision.

The next weekend you went to the library and try to learn more about the MMSE.

SEARCH

After searching in the MEDLINE you found the article by Mulligan et al entitled “ A comparison
of alternative methods of screening for dementia in clinical settings” published in the Archive of
Neurology, June 1996. Luckily the full text was also available.

CRITICAL APPRAISAL

RELEVANCE

• Was the objective of the paper relevant to your clinical question?

Most of the time we read journal articles because the topic is interesting. Because of this
application to clinical practice is not ensured. We can only ensure that the results of the article
are applied to practice if the objectives of the article are relevant to the clinical problems we
see in clinical practice. Thus the objective of the study must determine the accuracy (outcome)
of the contemplated diagnostic test (intervention/exposure) among patients (population)
similar to your case scenario.

VALIDITY GUIDES

• Was there an independent comparison with a reference standard?

There are two elements in this guide question i.e. use of a reference standard and independent
comparison. A reference standard for a diagnostic test is the test that gives the information
Page 21
nearest to the “truth”. Thus the accuracy of the test should be compared against the standard.
If the diagnostic test approximated the standard, that means the test also approximates the
“truth”. An independent comparison means that the reader of the reference standard did not
know the result of the diagnostic test being evaluated (Jaeschke, 1994). Awareness of the initial
test result may lead to increase confirmation with reference standard leading to bias on the
accuracy of the diagnostic test being evaluated. Thus the first question you should answer is
whether there was a comparison with the reference standard and whether the reference
standard used was acceptable to your setting. The second is whether the reader of the
reference standard was blinded to result of the diagnostic test being evaluated.

In the study by Mulligan et al the reference standard used was the diagnosis of dementia based
on the DSM‐III‐R.

• Did the patient sample include an appropriate spectrum of patients to whom the test
will be used?

The accuracy of a diagnostic test among patients with low risk for the disease is different from
patients with high risk of the disease. The clinical utility of a test can be seen when used among
persons who are healthy, patients who are very sick and mostly those in‐between because
these are the patients who will be requiring the test. Patients consulting in family practice
usually belong to the healthy and in‐between groups while patients consulting in the hospitals
are those in the in‐between and more severe groups. The in‐between groups may give an
underestimate of the accuracy of the test (but it is the accuracy value to whom the test will be
used) while the healthier and more severe may give an overestimate of the accuracy. If all
groups are equally represented the average accuracy will be obtained.

The study of Mulligan et al included elderly patients consulting in a geriatric hospital and
memory clinic. The elderly age group is the population with the highest risk of dementia thus
the results from this study may be an overestimate.

• Was the reference standard done regardless of the result of the diagnostic test being
evaluated?

In some studies, the accuracy of a diagnostic test is examined retrospectively (chart review of
actual practice). In actual practice however, physicians request to perform the reference
standard based on the initial result of the diagnostic test. The reference standard is used to
verify the initial finding i.e. when positive. When this happens most of the data available will be
those positive for the diagnostic test and will likely be positive in the reference standard. This
will increase the accuracy of the test. This is called verification bias (Jaeschke, 1994). To avoid
this, the study must show that the reference standard was done regardless of the result of the
diagnostic test being evaluated.

It was mentioned in the Mulligan et al study that the comparison with the DSM‐III‐R was blind
and independent.
Page 22

• Were the methods for performing the test described in sufficient detail to permit
replication?

This is necessary so that the reader will be able to duplicate the test in his/her own setting and
get the same valid result. Description should include preparation for the patient such as diet,
drugs to avoid, precautions, ideal conditions for performing the diagnostic test and a step by
step description of how the diagnostic test is done and interpreted.

There are a lot of papers dealing with instructions on how to administer the MMSE and its
interpretation.


Yes. You accepted the validity since most of the questions were answered adequately.


• What were the likelihood ratios for the different possible test results?

Likelihood ratio indicates by how much a given test result increases the pre‐test probability of
the disease. A likelihood ratio of 1 means that the post‐test probability is similar to the pre‐test
probability. A likelihood ratio of greater than 1 increase the chance that the disease is present,
and the greater the likelihood ratio the greater is the increase in chance.

Some papers give the sensitivity and specificity values rather than the likelihood ratio. The
formula for computing the likelihood ratio from sensitivity and specificity is shown below:

Likelihood ratio of a positive test LR (+) = Sn/1‐Sp
Likelihood ratio of a negative test LR (‐) = 1‐Sn/Sp

A rough guide in evaluating LR values:
• LRs >10 or < 0.1 generate large, and often conclusive changes from pre‐ to post‐test
probability;
• LRs of 5‐10 and 0.1‐0.2 generate moderate shifts in pre‐ to post‐test probability;
• LRs of 2‐5 and 0.5‐0.2 generate small (but sometimes important) changes in probability;
and
• LRs of 1‐2 and 0.5‐1 alter probability to a small (and rarely important) degree.

In the Mulligan study the LR (+) MMSE is 2.46 and the LR (‐) is 0.14. The alternative clinical test
is the Antisaccadic Eye Movement Test (AEMT) but did not do very well compared with the
MMSE.

Page 23

• Will the reproducibility of the test result and its interpretation be satisfactory in my
setting?

Even if the test was described very well, the reproducibility of the interpretation is necessary
for the test to be adequately applied in your setting. The paper should report measures of
agreement between interpreters or raters.

If agreement is not reported, decide for yourself. Is the interpretation simple enough? Is the
basis of the interpretation clear and specific?

The MMSE questionnaire also contains instruction on how to administer the test and interpret
the result.

• Are the results applicable to my patient?

If your setting is somewhat similar to that in the study, and the inclusion criteria include
characteristics of your patient, then you can apply the results to your patient. Sometimes your
clinical judgment is required.

• Will the results change my management?

The usefulness of a diagnostic test result is whether it will help the clinician manage his/her
patient. If the diagnostic test will lead the doctor decide to give treatment or not, then the test
is helpful and may be requested. However if after applying the LR value to determine the post‐
test probability and this did not help in the decision, then you don’t have to perform the test.


After looking at the Mulligan et al study, you agreed with the psychiatry resident and even
asked her help to administer the MMSE to the patient yourself.

REFERENCES
Jaeschke R, Guyatt G, Sackett D, and the Evidence Based Medicine Working Group. How to Use an Article About a
Diagnostic Test: Validity Guides. JAMA, 1994; 271(5):389-391.
Jaeschke R, Guyatt G, Sackett D, and the Evidence Based Medicine Working Group. How to Use an Article About a
Diagnostic Test: Results and Applicability. JAMA, 1994; 271(9):703-707.

Page 24
WORKSHOP
ABOUT A DIAGNOSISTIC TEST
(SESSION BRIEFING)

OBJECTIVES

making about the usefulness of a diagnostic test. Another objective is to introduce concepts of
critical appraisal of an article regarding a diagnostic test focusing on the following:

• Validity
• Reference standard
• Sensitivity and specificity
• Likelihood ratios
• Applicability of the results

INSTRUCTIONS


abstract, methods and results section. Again, establish a group consensus on how to proceed


clinic practice.

Page 25
TIME ALLOTTED



DESIRED OUTCOME

of the evidence.

Page 26
Appraisal Sheet
CLINICAL DECISION ON A DIAGNOSTIC TEST

QUESTION

SEARCH

CRITICAL APPRAISAL

RELEVANCE Is the objective of the study relevant to your clinical question?

VALIDITY GUIDES Was there an independent and blind comparison with a reference
standard?
What was the reference standard. Were they assessed
independently?

Did the patient sample include an appropriate spectrum of
patients to whom the test will be used?

Was the reference standard done regardless of the result of the
diagnostic test being evaluated?

Page 27
Were the methods for performing the test described in sufficient
detail to permit replication?

VALID?

WHAT ARE THE What were the likelihood ratios for the different possible test
RESULTS? results?

CAN THE RESULTS HELP Will the reproducibility of the test result and its interpretation be
ME IN CARING FOR MY satisfactory in my setting?
PATIENTS?

Are the results applicable to my patient?

Will the results change my management?

RESOLUTION OF THE
PROBLEM IN THE
SCENARIO

Page 28

ABOUT THERAPY OR PREVENTION
(SESSION BRIEFING)

OBJECTIVES

medical decision making using an article about therapy or prevention.

the workshop.

INSTRUCTIONS

Read the reading assignment for an article about therapy or prevention. Focus on the critical
appraisal questions, why they are asked and how to get the answers from the paper.


Page 29
CLINICAL DECISION ON THERAPY OR

PREVENTION

CLINICAL SCENARIO

You are working at the out‐patient clinic in your institution when a 55 year‐old, male bank
executive came to your clinic for constricting chest pain located at mid‐sternum precipitated by
exertion and relieved by rest. His blood pressure was 130/80, heart rate was 85/minute and
the respiratory rate was 20/minute. An ECG was done revealing lateral wall ischemia. You sent
the patient home on oral nitrates and aspirin. However, further work‐ups revealed
hypercholesterolemia with a level of 6.5 mol/liter. You gave dietary advice but the patient
claimed that he has been on a high fiber diet with low fat intake for the past 6 months.

He is now inquiring if he should take a drug for his elevated cholesterol.

SEARCH

You decided to translate this clinical dilemma to an answerable question. You also decided that
the article should include a population of patients that has an elevated cholesterol who have
already undergone dietary therapy for at least 6 months to be consistent with your patient’s
profile. The drug intervention should be compared with placebo. The article must report
clinically important outcomes, such as reduction in cardiovascular deaths. Finally, you wanted
an article that employed randomization.

The article you finally retrieved included 4444 patients randomized to either simvastatin or
placebo. However, before applying the results to your patient, you decided to appraise the
article using the following guides.

RANDOMIZED CONTROLLED TRIALS

Randomized controlled trials are the standards design to prove effectiveness of drugs or other
forms of intervention. When done properly, it can provide the best evidence of effectiveness. In
this type of design, individuals are randomly assigned (randomization) to either of the two or
more groups, one with the intervention the other without the intervention being tested or
another intervention. Randomization tries to make the two groups similar for both known and
unknown factors that may affect the outcome other than the intervention being tested. Then
they are observed forward in time and their outcome compared. The outcome can be the cure
of a disease, relief of symptoms or improvement in quality of life (Espallardo, 2000).
Page 30

CRITICAL APPRAISAL

RELEVANCE

• Is the objective of the article comparing therapeutic interventions similar to your
clinical dilemma?

Before going any further, first ascertain if the objective of the study addresses the clinical
problem you face. Appraising an irrelevant article would not be helpful to your clinical dilemma
and will be a waste of time. Below are a few tips that will help you decide on relevance:

• Population of the study (P ) – should be similar to the characteristic of your patient.
• Intervention/comparative intervention/exposure (I) – should include the therapeutic
intervention you want to test.
• Outcome of the study (O) – one of the outcomes measured should be the goal you and
your patient wish to work for.

VALIDITY GUIDES

• Was the assignment of patients to treatment randomized?

In order to answer this question, the reader is advised to look into the article’s abstract or
methodology section. For the 4S study, randomization was done and was written both in the
abstract and methods section.

The strength of randomization is that if the sample size is sufficiently large, it assures that both
known and unknown determinants of outcome are evenly distributed between the treatment
and control groups. In the absence of randomization, these factors might be difficult to control
and might be the one strongly influencing outcome rather than the treatment itself (Guyatt,
1994).

At times though because of the rarity of the disease and small patient sample size,
randomization might not be feasible. In these cases, a clinician must rely on weaker studies but
should be aware of its potentials for errors.

Were all patients who entered the trial properly accounted for and attributed at its conclusion?

• Was follow‐up complete?

This is best checked by looking at the number of patients enrolled at the outset and comparing
this with the number of patients reported in the results table.

Page 31
Every patient who entered the trial should be accounted for at its conclusion. If substantial
numbers are “lost to follow‐up”, the validity of the conclusions are open to question. A drop‐
out rate of 20% or more is usually declared as substantial. If the number lost to follow‐up is
less than this the reader can decide if this affects the conclusion by assuming a “worst case
scenario”. This means that the numbers lost in the treatment group are assumed to have bad
outcomes and the numbers lost in the control group are assumed to have been cured and if the
conclusions differ, a substantial number was lost to follow‐up.

Another way of deciding whether follow‐up was complete is to check whether an intention to
treat analysis was done. If this is reported one can safely assume that follow‐up was complete.

• Were patients analyzed in the groups to which they were randomized?

It simply means that all those belonging to the control group or treatment group are analyzed
from beginning to end in this same grouping including those who were dropped or withdrawn
or changed treatment. No crossing over treatment modalities were done as this would likely
lead to biased results.

Excluding non‐compliant patients from the analysis leaves behind those who may be destined
to have a better outcome and destroys the unbiased comparison provided by randomization.
This principle of attributing all patients to the group to which they were randomized results in
an "intention‐to‐treat" analysis. This strategy preserves the value of randomization: prognostic
factors that we know about, and those we don't know about, will be, on average, equally
distributed in the two groups, and the effect we see will be just that due to the treatment
assigned (Guyatt, 1994).

• Were patients, their clinicians, and study personnel "blind" to treatment?

To answer this question the reader is again advised to look into the abstract or the
methodology section.

Blinding is the process by which the intervention being given is concealed from the patient, the
clinicians and the one who analyzes the data. Patients, clinicians and data analysts are likely to
have an opinion regarding the experimental treatment. These opinions, whether optimistic or
pessimistic, can systematically distort reporting of treatment outcomes. As to avoid these
“reporter and observer” bias, blinding is necessary.

• Were the groups similar at the start of the trial?

To answer this question, one should look for a report of the comparison of the baseline
characteristics of the experimental and control group. For most studies, this is labeled as table
1. In the 4S trial, baseline characteristics were similar.

Page 32
For reassurance about the study’s validity, readers would like to be informed that the
treatment and control groups were similar for all the factors that determine clinical outcomes
of interest save for the experimental therapy. The greater the similarity between known
prognostic factors for the control and experimental group, the more likely that the results can
be attributed to the intervention, rather than due to the differences in these factors.

• Aside from the experimental intervention, were the groups treated equally?

Interventions other than the treatment under study, when differentially applied to the
treatment and control groups, are called “co‐interventions”. This might distort the results since
they in themselves might cause changes in reported outcomes.


If you want to be strict about it, you should answer yes in all 5 questions. However, you as the
user of the journal can make the decision. A simple rule might be to answer yes to at least, one
primary guide and two secondary guides.

The 4S study yielded yes to all the appraisal questions, hence you decided that over‐all the
study was valid. You now proceed to analyze the results.


• How large was the treatment effect?

Most randomized controlled trials report outcomes either as treatment success or treatment
failures and adverse effects. Examples of outcomes include cure rates, side effects or death.
Patients either do or do not suffer these events and the article frequently reports the
proportion of patients who develop such events. In the 4S study, 11.5% died in the placebo
group and 8.2% died in the simvastatin group. By eyeballing these figures, simvastatin seems
better in reducing deaths. But how else could this figures be compared? The following simple
computations could help:

Risk in Control (Rc) = Death in control/N patients in the control

Risk in Treatment (Rt) = Death in treatment/N patients in treatment

Absolute Risk Reduction (ARR) = Rc – Rt = 0.115 ‐ 0.082 = 0.033

Relative Risk (RR) = Rt/Rc = 0.082/0.115 = 0.71

Relative Risk Reduction (RRR) = 1 – RR = 1‐ 0.71 = 0.29 (29%)

Page 33
Absolute risk reduction is the absolute difference between the proportion who died in the
placebo group compared to the simvastatin group. Relative risk is the risk of events in the
simvastatin group or new treatment relative to the placebo or control group. The most useful
measure to use in explaining the benefit of treatment to patients is the relative risk reduction.
In this case, simvastatin treatment reduces deaths 29% more than placebo.

• How precise was the estimate of treatment effect?

To decide regarding precision, one should look at the reported 95% confidence interval. The
closer these values, the more precise your estimates. If this is not reported check the p‐value,
anywhere from </= .10 is acceptable.

Ideally, the reported minimum and maximum values of this interval should all be positive or all
be negative for the absolute risk reduction, all above below one for the relative risk and relative
risk reduction to be considered precise.


• Can the results be applied to my patient care?

If your patient meets all of the inclusion criteria and none of the exclusion criteria, the
applicability of the study’s results to your patient is without question. It is rare however that
we get a patient who conforms to all the characteristics of the study subjects. In these cases,
we should decide if the reason is compelling enough not to apply the results of our study to our
particular patient.

• Were all clinically important outcomes considered?

Clinically important outcomes may range form decreasing mortality, morbidity,
improving quality of life. These are outcomes that are important to the patients and will lead
directly to reducing symptoms or decreasing death. Some studies might report improvement in
cholesterol levels, improvement in PFTs but these are what might be labeled as surrogate
endpoints. That is, the researchers have substituted these physiologic measures for important
outcomes we have just mentioned. For reduction in these laboratory parameters does not
always translate into decrease in morbidity and mortality.

A dramatic example of the danger of substitute endpoints was found in the evaluation of the
usefulness of clofibrate as anti‐cholesterol drug. It shown to decrease serum cholesterol but
was shown to increase all‐cause mortality. Similar findings were noted in an anti‐arrhythmia
trial hen the investigators had to stop the trials when they discovered that mortality was
substantially higher in patients receiving antiarrhythmic treatment than in those receiving
placebo.

Page 34
• Are the likely treatment benefits worth the potential harm and costs?

Computation for cost effectiveness and checking for side effects might be done to check if the
treatment benefits are worth the potential harm and costs.


The article was valid and giving simvastatin reduces all deaths by 29% compared to placebo.
However when we look into costs, giving this drug treatment is fairly expensive. However due
to the marked cardiac risks for this patient you decide to give simvastatin since your patient
could also afford drug treatment.

At this point, we hope that you are encouraged to adapt this new paradigm in your medical
decision making. The steps are relatively simple. First, define the problem clearly. Second, use
one of several search strategies to come up with relevant articles. Third, appraise the article.
Assess the results and the applicability of your article to your patient. Guided by this, you then
decide on the action to take.

This may sound like a tedious process but as they say practice makes perfect. And if each of us
will continue asking, searching, we will continue learning and hopefully improving our patient
care.

REFERENCES

Espallardo, NL. Research Protocol Development for Resident Physicians. Family Medicine Research Group, Inc.
Manila, 2000.

Guyatt GH, Sackett D, Cook DJ, for the Evidence Based Medicine Working Group. How to Use an Article About
Therapy or Prevention: Validity. JAMA, 1993;270(21):2598‐2601.

Guyatt GH, Sackett D, Cook DJ, for the Evidence Based Medicine Working Group. How to Use an Article About
Therapy or Prevention: Results and Apllicability. JAMA, 1994;271(1):59‐63.

Page 35
WORKSHOP
AN ARTICLE ABOUT THERAPY OR
PREVENTION
(SESSION BRIEFING)

OBJECTIVES

making about treatment. Another objective is to introduce concepts of critical appraisal of an
article regarding treatment focusing on the following:

• validity
• randomization
• intention‐to‐treat
• interpretation of the results
• applicability of the results
• clinically relevant endpoints
• cost‐effectiveness

INSTRUCTIONS




clinic practice.

Page 36
TIME ALLOTTED



DESIRED OUTCOME

of the evidence.

Page 37
Appraisal Sheet
CLINICAL DECISION ON THERAPY OR PREVENTION

QUESTION

SEARCH

CRITICAL APPRAISAL

RELEVANCE Is the objective of the article comparing therapeutic interventions
similar to your clinical dilemma?

VALIDITY GUIDES Was the assignment of patients to treatment randomized?
Randomization vs. random selection

Were all patients who entered the trial properly accounted for
and attributed at its conclusion?

Was follow‐up complete?
Dropouts, withdrawals

Were patients analyzed in the groups to which they were
randomized?
Intention‐to‐treat analysis

Page 38

Were patients, their clinicians, and study personnel "blind" to
treatment?

VALID?

WHAT ARE THE How large was the treatment effect?
RESULTS? Risk in control, risk in treatment, relative risk, relative risk
reduction, absolute risk reduction

How precise was the estimate of treatment effect?
95% confidence interval, p value

CAN THE RESULTS HELP Can the results be applied to my patient care?
ME IN CARING FOR MY Inclusion criteria, exclusion criteria
PATIENTS?

Were all clinically important outcomes considered?
Outcome, results

Page 39
Are the likely treatment benefits worth the potential harm and
costs?
Side effects, NNT, costs

RESOLUTION OF THE
PROBLEM IN THE
SCENARIO

Page 40

ABOUT HARM
(SESSION BRIEFING)

OBJECTIVES

medical decision making using an article about harm.

the workshop.

INSTRUCTIONS

Read the reading assignment for an article about harm. Focus on the critical appraisal
questions, why they are asked and how to get the answers from the paper.


Page 41
CLINICAL DECISION ON HARM OR

RISK CAUSATION

CLINICAL SCENARIO

You are the resident physician covering for the consultant in his clinic. You met one of his
patient, 45 year old male who is on anti‐cholesterol medication. You noted that his cholesterol
is now only 4.35 mmol/L but the patient was still prescribed with anti‐cholesterol drug from his
last week visit. You approach the consultant and mentioned it to him. He told you its okay; it’s
the high cholesterol that we should be concerned with anyway.

Still doubtful, you went to the library and look for the harmful effect of very low cholesterol.

SEARCH

You found the article by Zureik et al, entitled “Serum cholesterol concentration and death from
suicide in men: Paris prospective study 1” published in the British Medical Journal, September
1996.

COHORT AND CASE-CONTROL STUDIES

Harmful effects of drugs or other exposure can best be studied with cohort or case‐control
study designs. A cohort is any group of individuals who share the same characteristics. In a
cohort study, selection of subjects starts with identifying individuals who have the same
characteristics or presence or absence of a particular cause or exposure. They are then divided
into two groups, those with the characteristics or causes and those without the characteristics.
They are then observed forward in time and determine who among them develop the outcome
or effect (Espallardo, 2000).

Cohort studies can also be prospective or retrospective depending on the manner of patient
recruitment. If recruitment is being done forward in time it is prospective, but if the cohort
already existed in the past and data gathering is being done by reviewing existing clinical
records then it is retrospective (Espallardo, 2000).

In a case‐control study, inclusion of subjects starts with defining or selecting those who have
the outcome or effect. These are considered as cases. Then this group is compared with
subjects who don’t have the outcome or effect. These are considered as the controls. Both the
cases and the control should be taken from within the same population. Then the two groups
Page 42
are investigated as to the presence or absence of hypothesized causes or risk factors for the
outcome (Espallardo, 2000).

Case‐control study can be prospective or retrospective depending on the manner of patient
recruitment. If recruitment is being done as cases develop forward in time it is prospective, but
if the cases have already developed in the past and patient recruitment is being done by
reviewing existing clinical records then it is retrospective (Espallardo, 2000).

CRITICAL APPRAISAL

RELEVANCE

• Is the objective of the article on harm similar to your clinical dilemma?

Your formulated clinical question must be addressed by the objective of the study. For a
decision to stop the harmful exposure the article must be designed to determine the harmful
effect (outcome) of the a drug, chemical or environmental substances (intervention/exposure)
to the patient (population). This must be clearly stated by the objective of the study.

VALIDITY GUIDES

• Were there clearly identified comparison groups?

Just like studies of effectiveness of treatment, studies of harm require that a control group for
comparison should be done. Unfortunately, randomized controlled trials to prove harm is not
ethical, so studies of weaker design like cohort or case‐control studies may be relied upon. Both
designs has a control group for comparison, in the former controls are chosen by absence of
exposure and in the latter by absence of the outcome or disease.

In a cohort study, a group of patients are observed. They are divided into those with the
exposure and those without the exposure. Then the outcome is observed forward in time. In a
case‐control study, patients with the outcome are gathered. Then a group of patient without
the outcome matched to the preceding group for certain characteristics other than the
exposure is also gathered. The presence of the exposure in both groups is then ascertained
(Levine, 1994).

In a cross‐sectional study, patients are grouped and analyzed with respect to their outcome and
exposure. This study design can also be a basis for establishing harmful effect, but the temporal
relationship of the exposure occurring before the outcome cannot be established.

The study by Zureik observed 6,728 men who had measurements of serum cholesterol. They
grouped and categorized cholesterol levels into low (<4.78 mmol/L), normal (4.78 to 6.21
Page 43
mmol/L) and high (>6.21 mmol/L). The changes were also categorized. These were the
comparison groups.

• Were the exposures and outcomes measured in the same way in the groups
compared?

Measurement of outcomes must be similar in both groups. In cohort study, the investigators
must show that diligent observation for the outcome was done in the groups with the exposure
(high risk) as well as the groups without the exposure (low risk). When the patient with the
exposure were observed more diligently, there will be a higher detection rate of the outcome
leading to increase incidence of the disease in the exposed group. This is called surveillance bias
(Levine, 1994). This must be avoided. This bias may also occur in case‐control studies, when the
detection of exposure is more diligent in the group with the disease or outcome.

Suicide data were taken from the national databases and death certificates in all groups in the
Zureik study.

• Was follow‐up sufficiently long and complete?

The length of follow‐up must be sufficiently long enough to detect the outcome. If follow‐up is
short, the chance of underestimating the effect of the exposure is high. When the relation
between asbestos and lung cancer was being investigated the relative risk was only 1.4 in the
early years of observation compared to the subsequent relative risk of 18.2 when the years of
observation was extended to 15 years and beyond.

The Zureik study observed their patients for 4 years after enrolment. They had 95% follow‐up.

• Is the temporal relationship between the exposure and outcome correct and dose
response gradient present?

For an exposure to cause an effect, two important criteria may be considered. First the
exposure must be present before the outcome and second there must be a dose response
gradient, i.e. the higher the dose the higher is the probability of the outcome. Cross‐sectional
studies usually cannot establish temporal relationship but it can establish a dose response
gradient and a comparison between groups.

The Zureik study measured serum cholesterol before the event of suicide so the temporal
relationship was correct.


Since all the answers to the validity guides, the study can be considered valid.

Page 44

• What is the magnitude of the association between exposure and outcome? Was the
estimate of the risk precise?

The relative risk is the incidence of the adverse effect in the group with the exposure divided by
the incidence of adverse effect in the group without the exposure. If the relative risk is more
than 1, then the exposure is causing harm and if less than 1 the exposure reduces harm.
Relative risk is usually computed when the design is a cohort study.

In a case control study, the odds ratio is computed. The odds ratio approximates the relative
risk, especially when the disease is rare.

The values of 95% confidence interval should be greater than 1 to say that the exposure really
causes harm. If one value of the 95% confidence interval is less than 1 and the other is more
than 1, then the effect of the exposure is uncertain.

The Zureik study showed that those with low cholesterol had increased risk of suicide with a
relative risk of 3.16 with a 95% CI of 1.38 to 7.22. The analysis was also adjusted for age,
smoking and mean corpuscular volume.



Just like in an article about a beneficial intervention, for the harmful effect to be extrapolated
to your patient you have to be assured that the characteristics of your patient is similar to the
study’s inclusion criteria.

The Zureik study recruited men between 43 to 52 years old with similar demographic
characteristics with our patient.

• Should I attempt to stop the exposure?

In answering this question you should consider the following:
• How large and precise is the risk of harm?
• What are the consequences if I withdraw the exposure?
• Do I have any alternative for the exposure?

Decision is simple when the answers to these questions are clear. For example cigarette
smoking has been associated with increase incidence of lung cancer and cardiac deaths, but
Page 45
withdrawing smoking may lead to “decrease quality of life” for smokers. But recently an
alternative like nicotine patch has been shown to decrease withdrawal discomfort and
eventually improve smoking cessation. So the decision to withdraw smoking for every patient
consulting in the clinic is warranted.


Based on the appraisal you decided to go back to your consultant and inform him about the
study that you found. He thanked you for the information and promised to withdraw the anti‐
cholesterol drug when the patient comes back.

REFERENCES

Manila, 2000.

Levine M, Walter S, Lee H, Haines T, Holbrook A, Moyer V, for the Evidence Based Medicine Working Group. How
to Use an Article about Harm. JAMA, 1994;271(20):1615‐1619.

Page 46
WORKSHOP
ABOUT HARM OR RISK CAUSATION
(SESSION BRIEFING)

OBJECTIVES

making about harmful effect of a substance or drug. Another objective is to introduce concepts
of critical appraisal of an article regarding a harmful effect of a substance or drug focusing on
the following:

• Validity
• Cohort study, case‐control study, case series and case report
• Interpretation of the results

INSTRUCTIONS




clinic practice.

Page 47
TIME ALLOTTED



DESIRED OUTCOME

of the evidence.

Page 48
Appraisal Sheet
CLINICAL DECISION ON HARM

QUESTION

SEARCH

CRITICAL APPRAISAL

RELEVANCE Is the objective of the article on harm similar to your clinical
dilemma?

PRIMARY VALIDITY Were there clearly identified comparison groups?
GUIDES

Were the exposures and outcomes measured in the same way in
the groups compared?

Was follow‐up sufficiently long and complete?
Is the temporal relationship between the exposure and outcome
correct and dose response gradient present?

Page 49

VALID?

WHAT ARE THE What is the magnitude of the association between exposure and
RESULTS? outcome? Was the estimate of the risk precise?

ME IN CARING FOR MY
PATIENTS?

Should I attempt to stop the exposure?

RESOLUTION OF THE
PROBLEM IN THE
SCENARIO

Page 50

ABOUT PROGNOSIS
(SESSION BRIEFING)

OBJECTIVES

medical decision making using an article about prognosis.

the workshop.

INSTRUCTIONS

Read the reading assignment for an article about prognosis. Focus on the critical appraisal
questions, why they are asked and how to get the answers from the paper.


Page 51
CLINICAL DECISION ON PROGNOSIS

CLINICAL SCENARIO

Your brother consulted you because of what happened to his wife lately. She just had a
miscarriage (after 6 months of pregnancy) last week and she had not taken her meals lately.
They are already in their five years of marriage and they don’t have a child yet. Your brother is
asking you if they should take a vacation despite his being very busy at work and he is trying to
save for the house mortgage. What will you advice him?

If you advice him to take a vacation, they might loss their house or even his job. If you advice to
continue working and let time heal his wife’s grief, the chance of a psychological problem
worsening is great.

Since you have attended a workshop on evidence based family practice, you formulated the
question “What is the chance that my brother’s wife will go further into grief after the
miscarriage considering that they still don’t have any child yet?” and went to the library.

SEARCH

You typed the combination of the terms, pregnancy loss and prognosis and grief and you were
able to get across the article of Janssen et al. study entitled “A prospective study of risk factors
predicting grief intensity following pregnancy loss” publish in the Archive of General Psychiatry
last January 1997.

Now you proceed to see if the information in this study can be used to answer your brother’s
question.

WHAT IS PROGNOSIS

Prognosis refers to the development of possible “outcome” of disease i.e. death in patient with
cancer. Prognostic factors are characteristics of a particular patient can be used to predict that
patient's eventual outcome i.e. patients advanced TNM cancer stage may have more death
than those with less advance TNM cancer stage. Prognostic factors need not necessarily cause
the outcomes but just predict their development. Thus prognosis is a prediction of the probable
outcome of a disease based on a individual's condition and the usual course of the disease as
seen in similar situations. Risk factors on the other hand are patient characteristics associated
with the development of the disease rather than the outcome of the disease.

Page 52
The study designs for prognostic and risk factors are cohort study and case‐control study. Cross‐
sectional studies do not give valid conclusions about prognostic or risk factors because
temporal relationship between factors and outcome is not established.

A cohort study follows one or more groups (cohorts) of individuals who have not yet suffered
an adverse event and monitor the number of outcome events over time. An ideal cohort study
consists of well defined sample of subjects representative of the population of interest, and
uses objective outcome criteria.

Investigators can also collect "cases" of individuals who have already suffered the outcome
event (death due to cancer) and compare them to "controls" who have not (cancer patients
who are alive). In these "case‐control" studies the investigators count the number of individuals
in each group with a particular prognostic factor (advance or less advance TNM cancer stage).

To be valid studies on prognosis, these observational studies must be conducted and reported
with information that address this appraisal guide (Von Elm, 2007)

CRITICAL APPRAISAL

RELEVANCE

• Is the objective of the article on prognosis similar to your clinical dilemma?

Your formulated clinical question must be addressed by the objective of the study. The PIO can
still be applied in this type of article. The objective of the study must clearly state that it is
determining the prognosis (outcome) of some patients with the prognostic factor
(intervention/exposure) among patients with the disease being studied (population).

VALIDITY GUIDES

• Was there a representative sample of patients without the outcome at the start of
observation?

The authors must specify how they defined or diagnosed the patients included in the study. The
authors should also specify at what stage of the disease they started observing their patients. If
these were not done bias can distort the result of the study. If the study included patients who
are more severe, the prognosis will naturally be poor and if they include patients who are mild,
the prognosis will be good. However if you mix these patients in one study without subgroup
analysis, the results will be mixed and biased.

Page 53
If this is not explicit in the study, you can look at the inclusion criteria or examine the setting
where the study was done. The inclusion criteria may give the basis for the diagnosis, and the
setting may give the stage of the disease i.e. outpatient setting may have included patients in
the earlier stage and hospital setting may be patients in the late stage.

In the Janssen et al study, 221 women were recruited through a magazine add. They had a
stable marriage and reported a recent pregnancy loss. So these women may have been
recruited at a relatively similar stage.

• Was follow‐up sufficiently long and complete?

Just like in the paper about harm, the length of follow‐up must be sufficiently long enough to
detect the outcome. If follow‐up is short, the chance of arriving at a good prognosis is high
because few will develop the outcome resulting to false hopes for the patient. If it is too long,
the prognosis will be poor because everybody will eventually die in the long term. Measuring
prognosis over a given period is usually acceptable i.e. 5 year survival for chronic diseases, 6‐24
months survival for cancer, 30 days survival after ICU admission etc.

The number of lost to follow‐up will also lead to bias results especially when the outcome is
unknown. If patients were lost to follow‐up because they felt bad about the outcome,
prognosis will be better if they are excluded in the analysis. If they were lost to follow‐up
because they felt better and the investigators assumed the worse scenario, the prognosis will
look bad.

In the Janssen et al study, follow‐up was 94%, a relatively high rate.

• Were the criteria for determining the prognostic factor and outcome explicit and
credible?

The criteria for determining the outcome in study about prognosis are usually straightforward
I.e. mortality. Mortality or survival can be taken from death certificates and other medical
records; morbidity can be taken from hospitalization records, etc. In some cases outcomes are
recurrence of disease or disease progression in which case this must be clearly defined.
Definitions can be taken from the NLM MESH definitions or ICD 10 classification of the WHO.

The Janssen et al study, measured the following prognostic factors; a) Symptom Checklist‐90, b)
Dutch Personality Inventory, and c) information about quality of partnership, education,
religion, social support etc. using existing records and surveys. The outcomes were measured
using the Perinatal Grief Scale immediately after pregnancy loss and at 6, 12 and 18 months.

• Was there adjustment for other prognostic factors?

Age and sex are factors that can affect prognosis but something we cannot do about. Thus
many prognostic studies look at the effect of other modifiable prognostic factors by adjusting
Page 54
for age and sex. Doing subgroup analysis does this. In subgroup analysis, the results of the study
are presented for each subgroup i.e. age and sex. Thus the prognosis of different TNM stage for
cancer can be presented in different age group or in different sex.

Another method is through multivariate analysis or regression model approach. In this
approach the basic variables included in the model are the prognostic factor and the outcome.
To adjust for age and sex, these variables are included into the model. This is usually described
in the analysis section of the methodology.

In the Janssen et al study, multivariate analysis was done.


Since all the validity questions were fulfilled, the study can be considered to be valid.


• How large is the likelihood of outcome to occur in those with the prognostic factor in a
specified period of time? Was it statistically significant?

The relative risk is the incidence of the outcome in the group with the prognostic factor divided
by the incidence of the outcome in the group without the prognostic factor. If the outcome
being measured is death and the relative risk is more than 1, then the factor results into poor
prognosis and if less than 1 the factor causes good prognosis. Relative risk is usually computed
when the design is a cohort study. In a case control study, the odds ratio is computed. The odds
ratio approximates the relative risk, especially when the disease is rare.

If the relative risk or odds risk is 1.11, it means the chance of developing the outcome is just
slightly higher if the patient has the prognostic factor. If the relative risk or odds risk is 1.99 it
means the chance of developing the outcome is almost two times (2x) and if the relative risk or
odds risk is 9.89 the chance is almost ten times (10x). The question of “how large is the chance”
involves preferential judgment from the patient and the physician.

To be statistically significant, the upper and lower values of 95% confidence interval should be
greater than 1 to say that the factor gives a bad prognosis when the outcome is death. If one
value of the 95% confidence interval is less than 1 and the other is more than 1, then the effect
of the prognostic factor is uncertain. In some cases, studies report the p value for statistical
significance i.e p <0.05 as significant.

The Janssen et al study reported that grief intensity was higher for a) women who had been
pregnant longer, b) pre‐loss neurotic personalities, c) pre‐loss psychiatric symptoms, and d) did
not have any living children. All these factors were significant at p <0.05.

Page 55


Just like in an article about harm, for the prognostic factor to be extrapolated to your patient
you have to be assured that the characteristics of your patient is similar to the study’s inclusion
criteria. The setting may also be important. Patients being observed in setting where the
facilities are advanced and complete may have better prognosis than among patients who are
being observed in resource poor setting even though they have the same prognostic factor.

The subjects in the Janssen et al study were women who reported recent pregnancy loss with
stable marital relationship. The subjects were similar to the sister‐in‐law’s case.

• Can I use the results to decide on the intervention or reassure my patient?

Prognostic data should be used in decisions about therapy. Knowing the probability of the
outcome based on the prognostic factors present in the patient should influence the decision to
give or withhold treatment. For example surgical excision for cancer with the hope of improving
survival should be withheld in favor of palliation treatment if the prognosis of the patient is
very poor.

Prognosis data may also be helpful in reassuring anxious patients about their outcome. For
example some patients with dyspepsia may become too worried about the chronic epigastric
symptom and can be reassured and counseled about the low prognosis of dyspepsia leading to
bleeding ulcer or cancer.


Based on the Janssen et al study, you would rather advice your brother to take a vacation,
because his wife’s grief may even intensify based on the results of Janssen et al study.

REFERENCES

Andreas Laupacis, George Wells, W. Scott Richardson, Peter Tugwell for the Evidence‐Based Medicine Working
Group. How to Use an Article about Prognosis. JAMA. 1994;272(3):234‐237.

National Library of Medicine. Medical Subject Headings. www.ncbi.nlm.nih.gov/sites/entrez (May 27, 2008).

Von Elm E, Altman D, Egger M, Pocock S, Gøtzsche P, Vandenbroucke J. STROBE Initiative. Strengthening the
reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational
studies. BMJ 2007; 335: 806‐808.

Page 56
WORKSHOP
ABOUT PROGNOSIS
(SESSION BRIEFING)

OBJECTIVES

making using an article about prognosis. Another objective is to introduce concepts of critical
appraisal of an article regarding prognosis:

• Validity
• Representative sample

INSTRUCTIONS




clinic practice.

TIME ALLOTTED


Page 57

DESIRED OUTCOME

of the evidence.

Page 58
Appraisal Sheet
CLINICAL DECISION ON PROGNOSIS

QUESTION

SEARCH

CRITICAL APPRAISAL

dilemma?

PRIMARY VALIDITY Was there a representative sample of patients without the
GUIDES outcome at the start of observation?

Was follow‐up sufficiently long and complete?

Were the criteria for determining the prognostic factor and
outcome explicit and credible?

Was there adjustment for other prognostic factors?

Page 59
VALID?

WHAT ARE THE How large is the chance of the outcome to occur in a specified
RESULTS? period of time? How precise were they?

ME IN CARING FOR MY
PATIENTS?

Can I use the results to decide on the intervention or reassure my
patient?

RESOLUTION OF THE
PROBLEM IN THE
SCENARIO

Page 60

ABOUT HEALTH ECONOMIC ANALYIS
(SESSION BRIEFING)

OBJECTIVES

medical decision making using an article about a health economic analysis.

the workshop.

INSTRUCTIONS

Read the reading assignment for an article about a health economic analysis. Focus on the
critical appraisal questions, why they are asked and how to get the answers from the paper.


Page 61
CLINICAL DECISION USING AN

ARTICLE ON HEALTH ECONOMIC
ANALYSIS

CLINICAL SCENARIO

Your father is the mayor of one of the town in the Visayas. His budget for health care is minimal
but he wants to start a program for pregnant women. Although most of the pregnancies in the
area were low risk, most patients still go to the nearest obstetrician who is in the city 50
kilometers away. He is planning to set‐up a maternity hospital with a trained obstetrician in
your hometown. He is now asking you for a recommendation.

What is your recommendation?

SEARCH

Having attended an EBM workshop you thought that the best way to convince your father is to
show cost‐effectiveness of handling low risk deliveries with different types of trained
physicians. You were able to get an article by Ratcliffe and Tucker entitled “The costs of
alternative types of routine antenatal care for low‐risk women: obstetrician‐led shared care vs
care by general practitioners and midwives” published in the Journal of Health Services and
Policy, 1996.

Appraising the article and conveying the information to your father seemed to be a good
strategy, so you proceeded in appraising the article.

HEALTH ECONOMIC ANALYSIS

Health economic analysis is a formal, quantitative methods used to compare alternative
strategies with respect to their cost and their expected outcomes (Eisenberg, 1989). Its purpose
is to inform decisions on resource allocation. It can potentially inform decisions in institutions
like hospitals and in regional or national health policy (Russell, 1996). In this design the cost of a
particular intervention is estimated. Estimation include direct and indirect costs. There are
three types of economic analysis depending on the type of outcome. If the outcome being
considered is effectiveness of treatment, it is called cost‐effectiveness analysis. If the outcome
is savings in terms of monetary units it is called cost‐benefit analysis. If the outcomes are equal
and the cost is the only one being compared it is called cost minimization.
Page 62
CRITICAL APPRAISAL

RELEVANCE

• Is the objective of the article on economic analysis similar to your clinical dilemma?

Your scenario must be addressed by the objective of the study. The perspective or “point of
view” in economic analysis usually refers to the one who will pay for the intervention. Often,
the point of view of the economic analysis is stated in the objective and this is important to
determine its relevance to your case scenario. If you are using an economic analysis for policy
decision, then the point of view must be the societal point of view. If your scenario is to assist a
patient to make a decision on an intervention in a “pay‐for‐service” setting, then the
perspective must be from the patient perspective. Health insurance perspective is also a payer
perspective.

VALIDITY GUIDES

• Did the analysis provide a full economic comparison of health care strategies?

Physicians usually choose between two alternatives. The range of alternative strategies
examined must include at least the currently accepted standard and the new alternative
(O’Brien, 1997). Another alternative is the “do nothing” alternative but may not be realistic in
some cases because of ethical issues.

When we compare the cost of giving each of the two alternatives, this is cost analysis. When we
use this to make a decision, we only give the alternative with the lowest cost that may not be
necessarily effective. When we use comparison of effectiveness such as a randomized
controlled trial in making a decision, we give an effective alternative that the patient may not
be able to afford. Thus it makes sense to consider cost and effectiveness when making clinical
decisions. A full economic analysis compares not only the cost of the two alternatives but also
integrate information about efficacy of the alternatives. Thus cost and outcomes should both
be analyzed for each alternative strategies being compared. This can be achieved if the study
design is a cost‐benefit or a cost effectiveness approach.

The Ratcliffe and Tucker study was a full economic valuation of the cost of tests, investigations,
personnel, cost incurred by the patients of pregnancies delivered by obstetricians vs family
physicians. The study included patients enrolled in a randomized controlled trial to answer the
effectiveness outcome.

• Were the costs and outcomes properly measured and valued?

Cost pertains to resources used and this must be differentiated from charges or prices of
commodities. The point of view of costing refers to the one who will pay for the cost and this
Page 63
may differ. For example cost to government hospital or funder may be different from the point
of view of the patient. What may be cost saving for the funder may actually be an increased
cost for the patient. The ideal point of view is from the society’s view, but this is difficult to
measure. Thus proper measurement of cost may differ from the health care system. In a system
where the payment is a fee‐for‐service set‐up, an economic analysis on the point of view of the
paying patient may be a good basis for making decision. In a system where health care is being
paid for by the government, an economic analysis from the point of view of society may be a
good basis. Thus in measuring cost, the point of view of the analysis must be established
(O’Brien, 1997).

Outcomes in health care must be an outcome that is of value to the patient and society. It
should be something that can be appreciated by the patient. For example in making a decision
about the treatment for hypertension, outcomes like decrease in incidence of mortality or
stroke, decrease in hospitalization or myocardial infarction instead of just the lowering of blood
pressure should be the outcome to be considered. In addition these outcomes must be
measured in the best possible designs i.e. randomized controlled trials for treatment,
controlled comparison for complex intervention etc. Systematic reviews or meta‐analysis of
these interventions are better methods for establishing outcomes.

Lastly, the cost and outcome must be expressed as a ratio i.e. cost per outcome (cost per life‐
year gained, or cost per death avoided etc.). This expression of result will give the most relevant
information for decision making.

In the Ratcliffe and Tucker study costs were extracted from clinical data that came from a
randomized controlled trial. They included cost per patient, staffing cost, non‐health services
cost and mean societal cost. Cost data was available in about 94% of subjects included in the
trial.

• Was appropriate allowance made for uncertainties in the analysis?

Economic analysis usually depends on analysis of secondary data and since not all data are
available for each alternative, some assumptions need to be made. There are uncertainties to
these assumptions. A good economic analysis is one that recognizes these uncertainties and
look at how it affects their findings. This method is called sensitivity analysis (O’Brien, 1997).

In sensitivity analysis the estimates for key variables in costs are changed in order to assess
their impact they on the results. The changes can be based on variation in price index,
opportunity costs, geographical price differences etc. In terms of outcomes, the variation can
be from confidence intervals or from lowest and highest effect noted from the studies that
were reviewed.

Page 64

Since the study was a full economic comparison and wide perspective of cost was considered
you decided that the study was valid.


• What were the costs and outcomes of each strategy?

Economic analysis papers should have tables that report the costs of resources used in the
alternative intervention such as drugs, personnel services, facilities, supplies etc. It should also
contain tables about the outcome of each alternative. Lastly, this is expressed as a cost‐
effectiveness ratio or cost‐benefit.

The table below is a good guide to help decide the alternative to choose. “C” will be the best
choice since it is more effective and less cost. “B” is not a good choice because it is less effective
but more expensive. “A” is more effective but more expensive as well.

Effectiveness
High Low
Cost High A B
Low C D

Sometimes, an alternative may be more effective but also more expensive. To make a decision
in this scenario, an incremental analysis should be done. Incremental cost is the amount we pay
for the added effectiveness of the alternative. Usually the availability of resources and personal
judgment may be needed to decide whether the incremental cost is worth it.

Ratcliffe and Tucker showed that the total societal mean cost for GP or midwife care was lower
by P 2,178 and was statistically significant.

• How much does allowance for uncertainty change the result?

Looking at how uncertainties affect the results is called sensitivity analysis. A direct approach
for doing this is by computing for the cost effectiveness ratio using the lower and upper limit of
the 95% confidence interval of the effectiveness outcome. If the cost‐effectiveness values are
reversed with sensitivity analysis then the results are considered to be soft and its reliability is
less.


• Could my patients expect similar outcomes?
Page 65

The inclusion criteria of the cited clinical trial or other studies reviewed to determine the
outcome and the setting from which the trial was done can be duplicated in your setting will
play an important factor. If the patients in the study are similar to your patient and setting for
which the intervention was given can be duplicated, then you can expect the same outcome
(O’Brien, 1997).

• Could my patients expect similar costs?

The health care system may be different from the setting where the economic analysis was
done. This difference may lead to difference in costing once applied for decision making in your
setting. Cost data may be different for two reasons: 1) clinical practice vary in resource
consumption associated with the treatment and 2) prices for resources differ from those used
in the study (O’Brien, 1997). This can only be answered if the analysis presented the detailed
cost so the reader can decide whether the costing in the study can also be applied in his/her
own setting.

Countries may differ with respect to the value they place on health benefits. If $50,000 per life‐
year is an acceptable cost‐effectiveness threshold for the US it may not be affordable in the
Philippines. Countries vary in their willingness to pay for health care (O’Brien, 1997).


Since most pregnancies in your hometown were low risk, you showed to your father that the
town will save more money if they hire family physicians or midwives rather than an
obstetrician. You called up your father one month later and the town council opted to hire
more midwives.

REFERENCES

Manila, 2000.

Eisenberg JM. Clinical economics. A guide to the economic analysis of clinical practices. JAMA, 1989; 262:2879‐86.

O'Brien B, Heyland D, Richardson WS, Levine M, Drummond M, for the Evidence‐Based Medicine Working Group.
How to use an Article on Economic Analysis of Clinical Practice: Validity Guides. JAMA, 1997; 277(19):1552‐1557.

O'Brien B, Heyland D, Richardson WS, Levine M, Drummond M, for the Evidence‐Based Medicine Working Group.
How to use an Article on Economic Analysis of Clinical Practice: Results and Applicability. JAMA, 1997;
277(22):1802‐1806.

Russell LB, Gold MR, Siegel JE, Daniels N, Weinstein MC. The role of the cost‐effectiveness analysis in health and
medicine. JAMA, 1996; 276(1)‐1172‐7.

Page 66
WORKSHOP
ON HEALTH ECONOMIC ANALYSIS
(SESSION BRIEFING)

OBJECTIVES

making about an economic analysis. Another objective is to introduce concepts of critical
appraisal of an article regarding an economic analysis focusing on the following:

• Validity
• Costs in health care
• Incremental costs

INSTRUCTIONS




clinic practice.

TIME ALLOTTED

Page 67


DESIRED OUTCOME

of the evidence.

Page 68
Appraisal Sheet
CLINICAL DECISION ON ECONOMIC ANALYSIS

QUESTION

SEARCH

CRITICAL APPRAISAL

RELEVANCE Is the objective of the article on economic analysis similar to your
clinical dilemma?

VALIDITY GUIDES Did the analysis provide a full economic comparison of health care
strategies?

Were the costs and outcomes properly measured and valued?

Was appropriate allowance made for uncertainties in the analysis?

Are estimates of costs and outcomes related to the baseline risk in
the treatment?

Page 69
VALID?

WHAT ARE THE What were the incremental costs and outcomes of each strategy?
RESULTS?

Do incremental costs and outcomes differ between subgroups?

How much does allowance for uncertainty change the result?

CAN THE RESULTS HELP Could my patients expect similar outcomes?
ME IN CARING FOR MY
PATIENTS?

Could my patients expect similar costs?

RESOLUTION OF THE
PROBLEM IN THE
SCENARIO

Page 70

ABOUT SYSTEMATIC REVIEWS OR
META-ANALYSIS
(SESSION BRIEFING)

OBJECTIVES

medical decision making using an article about systematic reviews or meta‐analysis.

the workshop.

INSTRUCTIONS

Read the assignment for an article about systematic reviews or meta‐analysis. Focus on the


Page 71
CLINICAL DECISION ON
SYSTEMATIC REVIEW OR
META-ANALYSIS

CLINICAL SCENARIO

Your grandmother had a history of fall a week ago after taking a bath. She was brought to the
hospital for treatment of minor bruises in her knees. The x‐rays were normal. Your mother
asked you if she needs a walker or cane to prevent falls and subsequent injury.

SEARCH

You ask a colleague from Rehabilitation Medicine and she gave you an article from the NHS
Center for Reviews and Dissemination she got from the internet entitled “Preventing falls and
subsequent injury in older people”.

REVIEWS, SYSTEMATIC REVIEWS AND META-ANALYSIS

A review is secondary study design that integrates findings of two or more studies that discuss
similar topic usually defined by PIO. There may be some bias when the reviewer subjectively
decides which studies to include or exclude in the review. A systematic review is similar to
review but has a way to systematically search the literature searching and has systematic rules
in combining the studies to be reviewed. The results of systematic reviews are more often
objective than a review. Meta‐analysis is like a systematic review but applies some statistical
analysis to the results.

A meta‐analysis is a procedure that integrates and combine the results of two or more primary
studies that are similar in the population enrolled the intervention used and the outcome
measured. The pooled result is then subjected to a statistical analysis. A well conducted meta‐
analysis allows a more objective appraisal of the existing evidence about a problem than a
traditional review or systematic review. Meta‐analysis may also be biased owing to the
inclusion or exclusion of some irrelevant or relevant studies respectively (Espallardo, 2000).
Page 72

Sample of meta‐analysis result

CRITICAL APPRAISAL

RELEVANCE

• Is the objective of the article on meta‐analysis similar to your clinical dilemma?

Your formulated clinical question must be addressed by the objective of the study. The
objective of an appropriately done meta‐analysis is often a focused clinical objective with PIO
and the method being clearly defined. This is often used for the systematic literature search
and basis for inclusion or exclusion.

VALIDITY GUIDES

• Did the review address a focused clinical problem?

Systematic reviews of the medical literature try to summarize publications related to a similar
topic. Because several articles are combined together, sometimes the purpose of the review is
not clear or very broad. It therefore becomes difficult to determine what the review is trying to
achieve.

In order to know what the objective of the review is, the clinical problem must be focused.
There must be a clear description of the patient and its relation with an exposure or an
outcome.
Page 73

The NHS study specifically stated that they tried to identify strategies that prevent falls and
subsequent injury in older people. The focused objective seemed to apply to your problem.

• Were the criteria for searching and selecting articles for inclusion and exclusion
explicit and credible?

In conducting systematic reviews, a systematic search and appraisal of the literature should be
done in order to:
• ensure that no relevant articles were missed
• studies were included because they are good studies and not because they agree with
the authors opinion
• studies were excluded because they are bad studies and not because they disagree with
the author’s opinion.

Thus paper should describe the method of searching for the medical literature. Statements like
“an electronic search of published articles in the MEDLINE using the terms . . . from 1966 to
2000 was done” must be found somewhere in the methodology section. This assures the
readers that the findings of the study were based on a wide range of literature source and
represent the most current and complete information about the clinical problem.

The paper should also describe how they include or exclude retrieved articles. The paper must
contain statements like “all retrieved abstracts were reviewed by three independent reviewers
and articles that were randomized controlled trial on . . . using the intervention . . .” somewhere
in the method section. This assures the readers that the articles used in the study were
objectively chosen and not because they agree with the authors opinion.

The NHS study identified trials published in computerized databases like Social Science Citation
Index, PSYCHLIT, EMBASE, RCN database, AMED and UNCOVER. The citations also identified
reviews and peer contribution from reviewers and other experts in the field.

• Was the validity of included studies appraised and the appraisal reproducible?

Even if all included studies are randomized controlled trials, there may be some small
differences among different trials that might affect the results of the study. Thus a standard
appraisal of each article must be done. Peer review may not be a reliable method of appraisal.
Differences in peer perception and interest may lead to differences in the result of the
appraisal.

While there are no agreed standards to evaluate validity, the review must have at least
developed a checklist of criteria that focused on the methodology of the study being appraised.

The NHS study included only randomized controlled trials that evaluated strategies to prevent
falls.
Page 74


Overall the study is valid.


• What are the overall results of the systematic review?

When looking at the results of a systematic review or meta‐analysis, you should look for
clinically relevant presentation like lower mortality rates in one group compared to the other,
or the difference in quality of life scores between the two groups. Sometimes subgroup analysis
i.e. patients with high risk and patients with low risk, to see the different effect of an exposure
may also be helpful.

In a meta‐analysis, the confidence interval of the overall results can also be computed and this
can provide information about the precision of the results.

Thirty‐six randomized controlled trials were included in the meta‐analysis. The results showed
that 10‐24 weeks of exercise including balance training showed an effective risk reduction by as
much as 37% with an adjusted fall incidence ratio of 0.90 and with a 95% CI of 0.81 to 0.99.



In a systematic review, the patient’s characteristics are varied because they came from
different studies. Application to patients therefore becomes wider. When variation in patient
inclusion may influence the effect, subgroup analysis between different patient characteristics
may also help decide what kind of patient will benefit from the intervention or will be affected
by the exposure.

This information can be seen in the methodology section where the researchers describe the
type of patients in the literature search and inclusion criteria of the studies.

The NHS meta‐analysis included only studies done on elderly.

• Are the results of the review relevant to my patient?

When the clinical question of the review is focused, the answer to this question becomes
evident. You only need to focus on the outcome measured and decide whether this is relevant
to your patient.

Page 75
When the outcome differs between studies, they are combined and this is reported in
systematic reviews or meta‐analysis as effect size. This is a difficult situation because the
outcomes are combined and you cannot easily decide whether the outcome is relevant or not.
In this case you can look at the results of individual studies and choose studies that give
relevant outcomes and use them to make a decision.


Based on the review, balance training rather than walker devices will help prevent further fall
and subsequent injury.

REFERENCES

Manila, 2000.

Oxman A, Cook D, Guyatt G, for the Evidence Based Medicine Working Group. How to Use an Overview. JAMA,
1994;272(17):1367‐71.

Page 76

ABOUT A SYSTEMATIC REVIEW OR
META-ANALYSIS
(SESSION BRIEFING)

OBJECTIVES

making using an article about a systematic review or meta‐analysis. Another objective is to
introduce concepts of critical appraisal of an article regarding differential diagnosis focusing on
the following:

• Validity
• Review, systematic review or overview, meta‐analysis

INSTRUCTIONS

Divide the participants into groups of six to ten persons per group. Assign a case scenario to
each group and formulate an answerable problem from the scenario. Establish initial group
consensus on how to proceed with the scenario.



clinic practice.

TIME ALLOTTED

Page 77


DESIRED OUTCOME

of the evidence.

Page 78
Appraisal Sheet
CLINICAL DECISION ON SYSTEMATIC REVIEW OR META‐ANALYSIS

QUESTION

SEARCH

CRITICAL APPRAISAL

dilemma?

PRIMARY VALIDITY Did the review address a focused clinical problem?
GUIDES

Were the criteria for searching and selecting articles for inclusion
and exclusion explicit and credible?

Was the validity of included studies appraised and the appraisal
reproducible?

Page 79

VALID?

WHAT ARE THE What are the overall results of the systematic review?
RESULTS?

ME IN CARING FOR MY
PATIENTS?

Are the results of the review relevant to my patient?

RESOLUTION OF THE
PROBLEM IN THE
SCENARIO

Page 80
READING ASSIGNMENT
CRITICAL APPRAISAL OF AN A
CLINICAL PRACTICE GUIDELINE
(SESSION BRIEFING)

CHECKLIST

Have you read the previous topic on systematic reviews?
Yes No

Have you undergone the workshop on systematic reviews with your group?
Yes No

Did you enjoy the workshop?
Yes No

Please state the reasons below and share it to the group before starting the next workshop.

OBJECTIVES

medical decision making using an article about a clinical practice guideline.

the workshop.

INSTRUCTIONS

Read the reading assignment for an article about a clinical practice guideline. Focus on the

Page 81
HOW TO USE A CLINICAL PRACTICE

GUIDELINE

CLINICAL PRACTICE GUIDELINES

"Clinical Practice Guidelines are systematically developed statements to assist practitioner
decisions about appropriate health care for specific clinical circumstances."(Field, 1990).
Guidelines were developed to:

• Make evidence‐based management explicit.
• Make clinical decision making more objective and scientific.
• Assess professional performance.
• Educate the patients and practitioners about current "best practice."

If guidelines are to improve practice, they need to be developed by the people who are actually
going to have to apply them. Guidelines, like so much else in healthcare today are no longer as
immutable as the Laws of the Medes, you will have to periodically check that they are working,
that they are getting to the people who are going to use them and that they are up to date.

In general, good topics for guidelines are those which:

• contribute a high workload
• poor treatment risks disastrous outcomes
• change is practical with the resources you have available
• there is variation in current management (i.e. there is uncertainty about the best
management strategy), but some hope of reaching a consensus
• the changes you wish to implement will be acceptable to patients
• there is good evidence to back up the protocols

It might be useful to consider the criteria used to select illnesses for screening programs (how
common, how serious, how preventable, how acceptable?

Clinical practice guidelines, which have been defined as "systematically developed statements
to assist practitioner and patient decisions about appropriate health care for specific clinical
circumstances," represent an attempt to distill a large body of medical knowledge into a
convenient, readily useable format. Like overviews, they gather, appraise and combine
evidence. Guidelines, however, go beyond most overviews in attempting to address all the
issues relevant to a clinical decision and all the values that might sway a clinical
recommendation. Like decision analyses, guidelines refine clinical questions and balance trade‐
Page 82
offs. Guidelines differ from decision analyses in relying more on qualitative reasoning and in
emphasizing a particular clinical context.

CASE SCENARIO

At the end of a busy day in your clinic, you are glad to find that your last patient is a 45/male
previously diagnosed to have hypertension. He claims that his highest blood pressure for the
past month was 160/90 and his usual blood pressure was 130/90 and he has been taking a
beta‐blocker for the past year. He was relatively symptom free save for occasional headache
and nape pains during BP spikes. Thinking that this was just another run of the mill
hypertensive patient you were ready to refill his prescription and give your usual advice
regarding diet and exercise. As you were about to do just that, your patient began asking you
the relative benefits of the alternative drugs available in the market. He was also asking if he
needed to have an ECG done together with blood chemistries, urinalysis and a 2D‐Echo since
his friend who consulted another physician was advised to do that. Since he was not
overweight he was asking if a regular exercise program would add any additional benefit in
controlling his symptoms. You gave him the usual advice you knew based on your knowledge
of pathophysiology and pharmacokinetics. However, as you were finally closing your clinic you
decided that you were not satisfied with the answers you gave him and decided to do a search
for the best available evidence.

SEARCH FOR CPG

You decide that in order to find relevant answers to a variety of clinical questions, a clinical
practice guideline would be the best article to retrieve. You initially searched
ww.guidelines.gov and found numerous guidelines for hypertension. However, you did not
have sufficient time to download and print the full text version of these guidelines. You then
remembered a copy of the Philippine compendium sent to you by mail 2 years ago and taught
of the Philippine Clinical Practice Guidelines on the Detection and Management of
Hypertension.

You then go home, sit at your desk and decide to review this article in order to be more
prepared for your next patient encounter.

CRITICAL APPRAISAL

RELEVANCE

• Is the objective of the article on clinical practice guideline similar to your clinical
dilemma?
Page 83

Your formulated clinical question must be addressed by the objective of the clinical practice
guideline. Usually guideline objectives are broad i.e. answers questions about the best
diagnostic test, the recommended treatment, the expected outcome or prognosis etc.

VALIDITY GUIDES

• Were all important options and outcomes considered?

Guidelines aid us in our decision making skills and we make better judgment calls if we know all
the alternative options open to us and the relative harm and benefits of each choice. Guideline
developers then should present most of the reasonable options seen in practice and their
corresponding outcomes.

In the case of the Philippine Clinical practice guideline on the detection and management of
hypertension, several treatment options were presented ranging from beta‐blockers, diuretics,
ACE inhibitors, calcium channel blockers with recommendations of the best alternative for
hypertensive patients with co‐morbid conditions. The guideline however did not include the
newer generation anti‐hypertensives.

As important as presenting all the options, the corresponding outcomes such as morbidity and
mortality data, prevention of complications and other measures that improve health related
quality of life should be reported. Inasmuch as all these will be helpful and clinically relevant to
individual patients.

In this hypertension guideline, mortality and morbidity data together with prevention of
hypertensive complications were the outcomes given emphasis. However costs and side effects
were not well mentioned.

• Was an explicit and sensible process used to identify, select, and combine evidence?

Guideline developers must allow the reader to know how the evidence has been tracked,
reviewed, appraised and combined in order to allow them to ascertain the validity of the
gathered evidence. Developers should specify a focused question, search the literature for
available evidence, critically appraise this evidence and summarize the results in an easy to
understand material.

The Philippine Clinical Practice Guidelines for Hypertension was not very clear regarding how
they searched the literature and what database they used. Mention of tracking, retrieving and
appraising was done in Phase 1 of the introduction section, but the complete way on how this
was done was not mentioned. However, summary on how the articles were reviewed and
graded was provided.

• Is the guideline likely to account for important recent developments?
Page 84

You should look for two important dates: the publication date of the most recent evidence
considered and the date on which the final recommendations were made. Some authorities
also identify important studies in progress and new information that could change the
guideline. Ideally, these considerations may be used to qualify guidelines as "temporary" or
"provisional," to specify dates for expiration or review, or to identify key research priorities. For
most guidelines, however, you must scan the bibliography to get an impression of how current
a particular guideline may be.

Once you are confident that the clinical practice guideline addresses your clinical question and
is based on a rigorous up‐to‐date assessment of the relevant evidence, you can review the
recommendations to determine how useful they will be in your practice.

The reader is advised to check the bibliography section of the guideline and check the dates of
the most recent articles included. Ideally, the evidence should be within the last 2 years before
the guideline was published. Since medical knowledge rapidly transforms, this will ensure that
our recommendations will not be outdated. Hence, there is a need to revise guidelines
periodically.

This guideline on hypertension was released in 1995 and the latest evidence upon looking at
the bibliography was in that same year. Being at present in the year 2000 and with the rapid
developments in antihypertensive medications, the guideline should be due for review and
revision.

• Has the guideline been subjected to peer review and testing?

People may interpret evidence differently and their values as to what important options and
outcomes are may differ. As such, a guideline that has been subjected to scrutiny by external
reviewers and tested in an actual clinical practice setting and found acceptable might be easier
to use.

OVERALL, IS THE GUIDELINE VALID?

Once you are confident that the guideline meets at least 2 out of the 3 requirements above,
you can review the recommendations and its applicability to our individual patients. At
present, the Philippine Clinical Practice Guidelines on Detection and Management of
hypertension although a little bit outdated at the present time will do.

WHAT ARE THE RECOMMENDATIONS

• Are practical, clinically important, recommendations made?

To be useful guidelines should give practical, unambiguous advice addressing a particular
clinical situation. Recommendations should be simple and specific at the same time
Page 85
comprehensive enough to allow the reader a chance to assess the benefits and costs of
following the particular recommendation.

In this hypertension guideline, recommendations are divided into every aspect of any
encounter with a hypertensive patient. The following 6 questions are addressed by the
guideline:

• How should blood pressure be measured?
• How should hypertension be diagnosed?
• How should hypertension be worked up?
• What advice should hypertensive patients receive regarding lifestyle modification?
• How should hypertension be treated?
• How can hypertension be prevented among normotensives?

This then allows the clinician to answer aspects regarding diagnosis, laboratory work‐ups,
treatment options, non‐pharmacologic advice and preventive measures.

• How strong are the recommendations?

The "strength," "grade," "confidence," or "force" of a recommendation should be informed by
multiple considerations:

• the quality of the investigations which provide the evidence for the recommendations
• the magnitude and consistency of positive outcomes relative to negative outcomes
(adverse effects, burdens to the patient and the health care system, costs)
• the relative value placed upon different outcomes.

Thus, grading of the recommendations are based on the methodological soundness of the
available evidence, the number of positive outcomes in relation to negative ones and the
consistency of findings across different evidences available. It is not enough to look into the
fact that randomized controlled trials were used as evidence but also if findings across different
trials were consistent. Inconsistent findings are at times the reason why different guideline
developers have different recommendations regarding certain clinical issues.

It is also important to note that different guideline developers use different standards for
grading their recommendations and that this should explicitly be placed in the guideline for
ease of understanding.

The Philippine Clinical Practice Guidelines for Hypertension used a system adopted by the
Canadian Hypertension Society. Therapy wise, a lot of Grade A recommendation meaning that
evidence is based from well‐conducted trials was made.

Page 86
WILL THE RECOMMENDATIONS HELP YOU IN CARING FOR YOUR PATIENTS?

• Is the primary objective of the guideline consistent with your objectives?

The purpose of the guideline developers for coming up with recommendations may vary from
your own. Guidelines may be disseminated to assist physicians in decision making (clinical
algorithms), to evaluate their practice and the standard of care they give to their patient
(quality assurance) or to set limits for physician choices (reimbursements, recertification). In
any case, in order to find recommendations most suited to your needs, the purpose of the
guideline should be in line with your intended objective.

This hypertension guideline was made to ensure the availability of a local guideline for the
detection and management of hypertension. Since your questions dealt with management
issues, this guideline is appropriate for your purpose.

• Are the recommendations applicable to your patients?

You must determine if the kind of patients you have are similar to those patients targeted by
the guideline. If your patients have a different prevalence or risk of disease, if the diagnostic
and therapeutic options recommended are not available in your area, the guideline might not
apply.

The advantage of reviewing and applying a Philippine practice guideline is that it takes into
account the characteristics of our setting and hopefully allows it to be more responsive.


Having deemed that the aforementioned guideline is valid, you would still opt to give this
patient with uncomplicated hypertension a beta‐blocker. In terms of diagnostics, you would
request for an FBS, Serum Creatinine, Serum potassium and urinalysis. You would request for
these tests since they would have an effect on the antihypertensive you would choose.
Furthermore they would provide the following additional benefits: a)detection and early
treatment of diabetes, b)detection of asymptomatic renal disease, c)detection of possible
secondary hypertension. You would explain to your patient that since he has no symptoms of
any cardiac disease, performing an ECG and Echo is not routinely recommended.

In terms of non‐pharmacologic advice, you would encourage him to go with regular aerobic
exercise such as walking, jogging or cycling 30 minutes per day 3‐4x/week since regular physical
activity reduces blood pressure and results in a decrease in all cause mortality.

A lot of decision making, considerations of options and outcomes came into play for a relatively
simple case of hypertension. And as medical knowledge improves, more options will be made
available. As such the need for relevant, well‐constructed and tested guidelines to improve our
clinical decision making will always be there. Again, we as clinicians should be able to adapt
Page 87
guidelines that are valid and whose recommendations will be most appropriate and feasible in
our respective settings.

REFERENCES
Hayward R, Wilson M, Tunis S, Bass E, Guyatt G for the Evidence Based Medicine Working Group. How to Use a
Clinical Practice Guideline: Validity. JAMA, 1995;274(7):570-4
Hayward R, Wilson M, Tunis S, Bass E, Guyatt G for the Evidence Based Medicine Working Group. How to Use a
Clinical Practice Guideline: Recommendations and Applicability. JAMA, 1995;274(20):1630-2.

Page 88
WORKSHOP
CRITICAL APPRAISAL OF CLINICAL
PRACTICE GUIDELINE
(SESSION BRIEFING)

OBJECTIVES

making about multiple problems related to disease management. Another objective is to
introduce concepts of critical appraisal of a clinical practice guideline:

• validity
• recommended options and outcomes
• up‐to‐date recommendations
• relevance and certainty of the recommendations
• applicability of the recommendations

INSTRUCTIONS

Divide the participants into groups of six to ten persons per group. Assign a case scenario to
each group and formulate an answerable problem from the scenario. Establish initial group
consensus on how to proceed with the scenario.


analyze the recommendations and determine the applicability of the recommendations.
Establish another group consensus and note any change in decision.

clinic practice.

TIME ALLOTTED

The time allotted for this workshop is one hour. The recommended break‐up is:
Page 89

• 10 minutes to read the clinical practice guideline
• 10 minutes to analyze the recommendations

DESIRED OUTCOME

of the clinical practice guideline.

Page 90
Appraisal Sheet
HOW TO USE A CLINICAL PRACTICE GUIDELINE

CASE SCENARIO OR
QUESTION

SEARCH

CRITICAL APPRAISAL

RELEVANCE Is the objective of the article on clinical practice guideline similar
to your clinical dilemma?

VALIDITY GUIDES Were all important options and outcomes considered?
Alternatives, expected results

Was an explicit and sensible process used to identify, select, and
combine evidence?

Is the guideline likely to account for important recent
developments?
Last update

Has the guideline been subjected to peer review and testing?

Page 91
OVERALL, IS THE
GUIDELINE VALID?

WHAT ARE THE Are practical, clinically important, recommendations made?
RECOMMENDATIONS

How strong are the recommendations?
Grading

WILL THE Is the primary objective of the guideline consistent with your
RECOMMENDATIONS objectives?
HELP YOU IN CARING
FOR YOUR PATIENTS?

Are the recommendations applicable to your patients and setting?

RESOLUTION OF THE
PROBLEM IN THE
SCENARIO

Page 92

EBM Self Instructional Manual 2010

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

EBM Self Instructional Manual 2010

Uploaded by

Copyright:

Available Formats

EVIDENCE BASED

Noel L. Espallardo, MD, MSc

EVIDENCE BASED MEDICINE

WHY TEACH EBM

SEARCHING THE MEDLINE

SEARCHING THE MEDLINE

PUBMED, ENTREZ AND MEDLINE

CRITICAL APPRAISAL OF AN ARTICLE

CRITICAL APPRAISAL OF AN ARTICLE

CLINICAL DECISION ON THERAPY OR

CRITICAL APPRAISAL OF AN ARTICLE

CLINICAL DECISION ON HARM OR

CRITICAL APPRAISAL OF AN ARTICLE

CLINICAL DECISION ON PROGNOSIS

CRITICAL APPRAISAL OF AN ARTICLE

CLINICAL DECISION USING AN

CRITICAL APPRAISAL OF AN ARTICLE

CRITICAL APPRAISAL OF AN ARTICLE

HOW TO USE A CLINICAL PRACTICE

You might also like