Critical Appraisal

By
Dr. Hassan Bin Usman
Assistant Professor

Learning objectives
• Define the term critical appraisal and discuss its
relevance to medical practice, policy and research.

• To provide a brief overview of the critical appraisal
process.

• Discuss the purpose of critical appraisal.

• Identify research questions and study types from
published papers and abstracts

Why read critically?
• Most doctors/people “read” journal articles by scanning the
abstracts
• Abstracts rarely tell the whole story, do not contain enough
details or nuances, and are frequently biased
• Abstracts are the appetizers that should get your interest up,
but it cannot be the main course (Methods, Results, and
Conclusions)
• The rest of the article contains the important nuances
• Beware of reading only abstracts, especially if you plan to
change your practice based on the study!

Why read critically? • Need to be able to evaluate the quality of published studies • Understand the ideal features of a study • Identify study errors and biases • Appropriateness of data analysis methods • statistics .

. This frees time to concentrate on a more systematic evaluation of those studies that cross the quality threshold and then to extract their salient points.Appraisal Increases the effectiveness of reading by enabling you to exclude research studies that are too poorly designed to inform practice.

. Why is this necessary? So much information is now easily available. that it is not possible to read or study it all. or forced upon us. Criteria for selection need to be established. Selection is required.

. • There are many ways of finding literature. • Selectivity is the key to successful critical appraisal. • Remember to ask a librarian – they are the experts.Identify Relevant Literature.

.identifying the limitations of.understanding. What is critical appraisal? Critical appraisal is a systematic approach to: . .deciding upon the usefulness of results of scientific papers. and .interpreting. . .reading.

etc. completeness of reporting. methods and procedures..Definition Can be defined as • “the assessment of methodological quality” • “Application of rules of evidence to a study to assess the validity of the data. The rules of evidence vary with circumstances” (Last JE (Ed. Critical Appraisal. 2001) A Dictionary of Epidemiology (4th Edn). compliance with ethical standards. New York: Oxford University Press) . conclusions.

.1). p. and relevance before using it to inform a decision” (Hill and Spittlehouse. and its value and relevance in a particular context. “Critical appraisal is the process of systematically examining research evidence to assess its validity. What is Critical Appraisal? Definition contd. 2001. results. Critical appraisal is the process of carefully and systematically analyzing the research paper to judge its trustworthiness. .

. • Time is limited – you should aim to quickly stop reading the dross. • Simple checklists enable the useful information to be identified.Critically Appraise What You Read • Separating the wheat from the chaff. • Others contain useful information mixed with rubbish.

How do I appraise? • Mostly common sense. . • You don’t have to be a statistical expert! • Checklists help you focus on the most important aspects of the article. • Different checklists for different types of research. • Will help you decide if research is valid and relevant.

Brief General guidelines/Questions to Ask One should keep these basic questions in mind before reading a topic • Is it of interest? • Why was it done? • How was it done? • What has been found? • What are the implications? • What else is of interest? .

source. • Or they didn’t find anything but wanted to publish! . • Should end with a clear statement of the purpose of the study. • The absence of such a statement can imply that the authors had no clear idea of what they were trying to find out. • Why was it done? • Introduction. abstract.General guidelines/Questions to Ask • Is it of interest? • Title.

• Brief but should include enough detail to enable one to judge quality. • Basic demographics must be there. .General guidelines/Questions to Ask • How was it done? • Methods. • An important guide to the quality of the paper. • Must include who was studied and how they were recruited.

• The data should be there – not just statistics. bland statements of results. . • All research has some flaws – this is not nit picking. the impact of the flaws need to be assessed. • Are the aims in the introduction addressed in the results? • Look for illogical sequences.General guidelines/Questions to Ask • What has it found? • Results. • Flaws and inconsistencies.

• All authors will tend to think their work is more important than the rest of us! • What is new here? • What does it mean for health care? • Is it relevant to my patients? .General guidelines/Questions to Ask • What are the implications? • Discussion. • The whole use of research is how far the results can be generalised.

• Useful references? • Important or novel ideas? • Even if the results are discounted it doesn’t mean there is nothing of value.General guidelines/Questions to Ask • What else is of interest? • Introduction / discussion. .

Classification of Epidemiological Studies .

almost all addressing one of five main research questions: 1. what are the effects of an intervention?) 4. how common is this disease or condition in a specific population?) 2.e. can the risk of disability and death be predicted?) . What is the aetiology of a certain condition or disease? (i. condition or risk factor? (i. How frequent is a disease.e. Does a test accurately classify groups with a condition or disease? (i. symptom or test in predicting a condition or disease?) 5.Main topics/research questions in Public health Although there are numerous public health research projects conducted every year.e.e.e. What is the prognosis of a person with a certain condition or disease? (i. Does an intervention work? (i. How accurate is a sign. are there factors associated with a particular condition or disease?) 3.

2008) . you must find an appropriately designed study.Clinical Questions and Study Design “There are 5 fundamental types of clinical questions: • 1) therapy • 2) harm • 3) differential diagnosis • 4) diagnosis • 5) prognosis” • It is important “to correctly identify the category of study. because.” (Guyatt. to answer your question.

Web-based Courses: EBM and the Medical Librarian. Web-based Learning.mlanet. diferent research designs require evidence from Diferent clinical questions Study Category Suggested Best Method of Investigation Therapy RCT>cohort>case control>case series Diagnosis prospective. Hp. 2001. Available: http://www. Education. blind comparison to a gold standard Cost Economic Analysis Medical Library Association. 2005. MLANET.org/education/web/web_courses.html 10 Apr. . blind comparison to a gold standard Etiology/Harm RCT>cohort>case control>case series Prognosis cohort>case control>case series Prevention RCT>cohort>case control>case series Clinical Exam prospective. Nov.

.

Types of Studies .

Anatomy of a Scientific Article • Abstract • Introduction • Materials and Methods • Results • Discussion • Conclusion .

Critically Appraise What You Read
• Critical appraisal is most easily carried out using structured check
lists.

• Simple checklists enable the useful information
to be identified.

• Different checklists are available like:
• CONSORT checklist for trials
• STROBE checklist for case control/cohort and cross sectional
studies
• AGREE checklist for clinical guidelines

Checklists
• Checklists for particular types of literature are a
quick and easy way of learning critical appraisal.

• They all have 3 stages: (Australian checklist)
• Basic questions.
• Essential appraisal.
• Detailed appraisal

Formal Critical Appraisal - CASP
• Useful source of checklists that over many study
designs is Critical Appraisal Skills Program
(CASP)

• CASP developed by Public Health Resource Unit
to bring about improvement in health and health
care

casp-uk. It is not a definitive guide and extensive further reading is recommended. Critical Appraisal Skills Program (CASP) making sense of evidence • 10 questions to help you make sense of qualitative research • This assessment tool has been developed for those unfamiliar with qualitative research and its theoretical perspectives.net/#!casp-tools-checklists/c18f8 ) . This tool presents a number of questions that deal very broadly with some of the principles or assumptions that characterize qualitative research. • Questionnaire attached… link • CASP Checklists are available on: (http://www.

Prof David Australia. Public Health 2004 . Critical Appraisal Check list based on widely used checklist by Prof Jane Hall.

Critical Appraisal Check list based on widely used checklist by Prof Jane Hall. Prof David Australia. Public Health 2004 .

Structure of scientific publications: • Title • Summary/Abstract • Key words • Main text (IMRAD) • Introduction • Methods • Results and • Discussion • Conclusions • References .

Introduction • Reading a scientific article • Subject selection • Literature search • Title of the scientific article: • Helps the reader to decide whether this matches with the subject • First impression of the article’s content .

intervention versus ‘natural experiment’ • i. a study assessing association between exposure to a known harmful substance and a new disease – need to use ‘natural experiment’ (or observational) design .e.What is the research question? • Identifying the Research Question (RQ) is the first and most important part of critical appraisal • Directly related to choice of study type • Needs to be appropriate to answer the RQ • Ideal study type may not always be feasible or ethical • E.g.

e.Stating the Research Question • Not able to judge suitability of study type if RQ not clearly stated in literature • Ideal RQ contains the following information • Population under study • Intervention (or exposure / study factor) • Comparison group (unexposed or controls ) • Outcome (or disease) • i. PICO .

who have taken hormone replacement therapy (HRT) for a year or more. have a higher risk of ovarian cancer than women aged 60 years and over who have never taken HRT?” • Population under study • Intervention (or study factor or exposure) • Comparison group • Outcome (or disease) .Example of PICO • Do women aged 60 years and over.

Example of PICO • Do women aged 60 years and over. have a higher risk of ovarian cancer than women aged 60 years and over who have never taken HRT?” • Population under study • Intervention (or study factor or exposure) • Comparison group • Outcome (or disease) . who have taken hormone replacement therapy (HRT) for a year or more.

Critical Appraisal.Abstract Abstract: • Same basic structure as the article • Essential points of the article in shortened form • Not a substitute for the article • Important to know if the abstract has been able to summarize the aims. methods. results and conclusions .

Introduction Introduction: • To familiarize the reader with the subject matter • Current state of knowledge presented with reference to the recent literature • The necessity of the study should be clearly laid out • Findings of the studies cited should be given in detail with numerical results • Avoid phrases such as ‘inconsistent findings.” or "somewhat better" .Critical Appraisal.

• A ‘good’ publication backs up its central statements with references to the literature • Proceeds from general to specific • Should explain clearly what question the study is intended to answer • Choice of appropriate study design • Should include rationale and aims/objectives of the study ..Introduction contd.Critical Appraisal.

Critical Appraisal.Methods Methods: • Section resembles a ‘cookbook’ with procedures as ‘recipes’ that can be followed to repeat the study • Can be divided into sub-sections Should describe: • All stages of planning • Composition of the study sample • Execution of the study • Statistical methods .

. • Most important element of methods is Study Design • Choice of study design should be explained and depicted in clear terms • Inclusion and exclusion criteria • Sample size calculation • Response rate (80% is good. 30% is no or only slight power) • Rate of loss to follow-up • Non-participation rate • Information on missing values • Ethical review • Written informed consent .Critical Appraisal.Methods contd.

Methodology contd. Questions to think about:   • Was a study protocol written before the study commenced? • Was the investigation preceded by a pilot study? • Are location and study period specified? • Is it explained how measurements were conducted? ..Critical Appraisal.

Methodology contd. Metric)? • Was there a careful power calculation before the study started? . and time point. e. described in sufficient detail? • Were the measurements made under standardized and thus comparable conditions in all patients? • What kind of scale the variables are being measured (e. measuring devices.. laboratory data.g. • Are the instruments and techniques.g.Critical Appraisal. nominal. scale of measured values. tumor stage. eye color. bodyweight. ordinal.

.

Was a control group used? 3. Levels of internal validity 1. Was a pretest used? 5. Were there enough subjects in the study? 2..Methodology contd. Was the study started prior to the intervention or event? 6. Were the subjects randomly assigned? 4. Critical Appraisal. Was the outcome measured in an objective and reliable way? 6 x yes = very high (A) 5 x yes = high (A) 4-3 x yes = limited (B) 2 x yes = low (C) 1-0 x yes = very low (D) .

• Includes comprehensive description of the study population . stating statistical parameters such as case numbers. without interpretation • Should first be formulated descriptively. and confidence intervals.Critical Appraisal. i.Results Results: • Findings should be presented clearly and objectively. measures of variation.e. mean values.

and may include calculation of appropriate statistical models • Statistical significance in the form of p values. or estimates the effect of a risk factor.. • Second sub-section describes relationship between characteristics. comprehensive description of the data and details on confidence intervals and effect sizes are strongly recommended • Tables and figures with self-explanatory data .Results contd.Critical Appraisal.

Two methods of assessing the role of chance • P-values (Hypothesis Testing) • use statistical test to examine the ‘null’ hypothesis • associated with “p values” .if p<0.05 then result is statistically significant • Confidence Intervals (Estimation) • estimates the range of values that is likely to include the true value .

P values • P stands for probability . • P values and confidence intervals should be consistent • Confidence intervals provide a range of values .05 means likelihood of results being due to chance is less than 1 in 20 = “statistically significant”.how likely is the result to have occurred by chance? • P value of less than 0.

Confidence intervals (CI) • The range of values within which the “true” value in the population is found • 95% CI – 95% confident the population lies within those limits • Wide CI = less precise estimates of effect .

Critical Appraisal.Discussion Discussion: • Comparison of the findings with the status quo (available literature) • Questions to keep in mind: • How has the study added to the body of knowledge on the given topic? • What conclusions can be drawn from the results? • Will the findings of the study lead the author to reconsider or change his/her own professional behavior? .

Critical Appraisal. clinical routine. patient care..Discussion contd. • Do the findings suggest further investigations? • Does the study raise new. why might that be? • Do the results appear plausible from the biological or medical viewpoint? . unanswered questions? • What are the implications of the results for science. and medical practice? • Are the findings in accord with those of the majority of earlier studies? If not.

.Study Limitations Critical analysis of the study's limitations: • Check what are the sources of bias (random or systematic) • When comparing groups should know whether there is any intergroup difference in the composition of participants lost to follow-up • Missing values should be discussed • Results that do not attain statistical significance must also be published.Critical Appraisal.

Conclusion Conclusion: • The interpretations should follow logically from the results • Avoid conclusions that are supported neither by one's own data nor by the findings of others • Study can attain objectivity only if the possibility of erroneous or chance results is admitted • Inclusion of non significant results contributes to the credibility of the study .Critical Appraisal.

Conclusion contd.Critical Appraisal.. • "Not significant" should not be confused with "no association“ • Significant results should be considered from the viewpoint of biological and medical plausibility .

Critical Appraisal.References Reference: • Should be presented in the journal's standard style • Reference list must include all sources cited in the text. tables and figures of the article • Should be up to date and help the reader to explore the topic further • Do these reflect fairly and appropriately the current state of knowledge? .

financial or otherwise. must be revealed in full .Critical Appraisal.Acknowledgements and conflict of interest Acknowledgements and conflict of interest • Must provide information on any sponsors of the study • Any potential conflicts of interest.

Critical appraisal questionnaires Examples .

Examples of Check lists .

questions Did the study address a clearly focused issue? 2. What do the findings mean? 11.Standard appraisal 1. Is the design appropriate to the stated aims? 4. Did untoward events occur during the study? 7. Is the sample size justified? 3. Were the basic data adequately described? 8. Are important effects overlooked? 12.Examples of Check lists. What implications does the study have for your practice? . Are the measurements likely to be valid and reliable? 5. Do the numbers add up? 9. Was the statistical significance assessed? 10. Are the statistical methods described? 6.

could this have introduced bias? 3. could this have introduced bias? 7. Were outcomes assessed blind? If not.Examples of Check lists. Were subjects randomly allocated to the experimental and control group? If not. Did the study address a clearly focused issue? 2. Were groups comparable at the start of the study? 5. Is the size of effect practically relevant? 8. Are the conclusions applicable? . Are objective and validated measurement methods used and were they similar in the different groups? (misclassification bias) 6.Appraisal of a controlled study 1. Are objective inclusion / exclusion criteria used? 4.

Was the cohort/ panel representative of a defined population? 4.Appraisal of a cohort / panel study 1. Is the size of effect practically relevant? 9. Are objective and validated measurement methods used and were they similar in the different groups? (misclassification bias) 6.Examples of Check lists. Was a control group used? Should one have been used? 5. Did the study address a clearly focused issue? 2. Are the conclusions applicable? . Was the follow up of cases/subjects long enough? 7. Was the cohort / panel recruited in an acceptable way? (selection bias) 3. Could there be confounding? 8.

Was the selection of cases and controls based on external. Was there data-dredging? 7. Did the study address a clearly focused issue? 2. Were the cases and controls defined precisely? 3. objective and validated criteria? (selection bias) 4. Are objective and validated measurement methods used and were they similar in cases and controls? (misclassification bias) 5.Examples of Check lists. Did the study incorporate blinding where feasible? (halo-effect) 6.Appraisal of a case-control study 1. Could there be confounding? 8. Is the size of effect practically relevant? 9. Are the conclusions applicable? .

Could the way the sample was obtained introduce (selection)bias? 4.Was the statistical significance assessed? 7.Assessment of a survey 1.Examples of Check lists.Did the study address a clearly focused issue? 2.Is the sample representative and reliable? 5.Are the measurements (questionnaires) likely to be valid and reliable? 6.Are important effects overlooked? .Was the sample size justified? 3.

Group Activity • Read abstracts 1 to 5. analytic or observational? c) Was the study design used by the researcher the best one to answer the research question? Why? d) Identify the study factor and the outcome factor. e) What did the researchers conclude from the study? f) In your own words. For each abstract: a) What was the research question? b) Was the study descriptive. briefly summarise the main results of the study .

Dent R. P=0. DESIGN: Randomised controlled trial.6% and 6. SETTING: Five general practices in Hertfordshire. Secondary outcomes were reported changes in daily consumption of cigarettes and identification of new diagnoses of chronic obstructive lung disease. Griffin M. People with worse spirometric lung age were no more likely to have quit than those with normal lung age in either group. INTERVENTION: All participants were offered spirometric assessment of lung function. Those in the control group received a raw figure for forced expiratory volume at one second (FEV1).2% to 12. 336(7644):598‐600.2%. BMJ. 2008. a total of 16% (89/561) of participants. Effect on smoking quit rate of telling patients their lung age: the Step2quit randomised controlled trial. A new diagnosis of obstructive lung disease was made in 17% in the intervention group and 14% in the control group. Both groups were advised to quit and offered referral to local NHS smoking cessation services. number needed to treat 14). were 13. Cost per successful quitter was estimated at 280 pounds sterling (366 euros. PARTICIPANTS: 561 current smokers aged over 35.1%. respectively. Participants in intervention group received their results in terms of "lung age" (the age of the average healthy individual who would perform similar to them on spirometry).4% (difference 7. TRIAL REGISTRATION: National Research Register N0096173751 . but the mechanism by which this intervention achieves its effect is unclear. Greenhalgh T. 95% confidence interval 2. MAIN OUTCOME MEASURES: The primary outcome measure was verified cessation of smoking by salivary cotinine testing 12 months after recruitment.005. CONCLUSION: Telling smokers their lung age significantly improves the likelihood of them quitting smoking. England. • Abstract • OBJECTIVE: To evaluate the impact of telling patients their estimated spirometric lung age as an incentive to quit smoking. Independently verified quit rates at 12 months in the intervention and control groups. RESULTS: Follow‐up was 89%. Abstract 1 • Parkes G. $556).

Molitor JN.56] across the average within‐community interquartile range of 6. • Abstract • BACKGROUND: The question of whether air pollution contributes to asthma onset remains unresolved.85). with a hazard ratio (HR) of 1. OBJECTIVES: In this study. hazard ratio can be considered to be similar to relative risk.35‐7. Gauderman WJ. .9 ppb increased the HR to 3.2 ppb in annual residential NO2.25 (95% CI. Using the total interquartile range for all measurements of 28. Kunzli N. We used multilevel Cox models to test the associations between asthma and air pollution. • Note: for this activity. Gilliland F. The risks observed suggest that air pollution exposure contributes to new‐onset asthma. RESULTS: In models controlling for confounders. Thomas DC. CONCLUSIONS: In this cohort. Peters J. incident asthma was positively associated with traffic pollution. Molitor JT. METHODS: We selected a sample of 217 children from participants in the Southern California Children's Health Study. we assessed the association between asthma onset in children and traffic‐related air pollution. Abstract 2 • Jerrett M. Environmental Health Perspectives 116(10):1433‐8. Berhane K.29 [95% confidence interval (CI). Individual covariates and new asthma incidence (30 cases) were reported annually through questionnaires during 8 years of follow‐up. Traffic‐related air pollution and asthma onset in children: a prospective cohort study with individual exposure measurement. 1. McConnell R. Avol E. 2008. markers of traffic‐related air pollution were associated with the onset of asthma. a prospective cohort designed to investigate associations between air pollution and respiratory health in children 10‐18 years of age. Children had nitrogen dioxide monitors placed outside their home for 2 weeks in the summer and 2 weeks in the fall‐winter season as a marker of traffic‐ related air pollution. 1.07‐1. Lurmann F. Shankardass K.

Franceschi S. Serraino D. age at menopause and use of hormone replacement therapy or oral contraceptives. for ≥ 25 years vs. International Journal of Cancer 123(9):2213‐6. <25 years). case‐control study on RCC risk. Talamini R. with histologically confirmed. Dal Maso L. central and southern Italy. Montella M. a major role of female hormone‐related factors. Negri E.0. • Abstract • A role of hormone‐related factors in renal cell cancer (RCC) etiology has been hypothesized. and other hormone‐related factors and risk of renal cell cancer. however. Cases were 273 women. menstrual and other gender‐specific variables on RCC risk among women.3‐4. menstrual. The present study aimed at evaluating the effect of reproductive. but the epidemiological evidence is inconsistent. A negative association of borderline‐statistical significance emerged for age at menarche. Odds ratios (OR) and 95% confidence intervals (CI) were computed using multiple logistic regression models. La Vecchia C. This study is part of a larger hospital‐based. menopausal status. Canzonieri V. frequency‐matched to cases by age and center.3. Ramazzotti V. conducted in northern. . Polesel J. Controls were 546 women hospitalized for acute.7. Our findings give support to a role of hysterectomy in increasing RCC risk without corroborating. RCC risk was inversely related to age at first birth (OR = 0. 2008. Reproductive. nonneoplastic conditions. whereas. incident RCC. 95% CI 0. Abstract 3 • Zucchetto A. below age 80. 95% CI 1. Hysterectomy was found to double RCC risk (OR = 2.2). no associations were found between RCC risk and parity.5‐1.

5% vs.0%. history of kidney transplantation.1% vs. we studied cross‐sectionally the hepatitis surface antigen (HbsAg) status in blood samples of 167 active chronic hemodialysis patients at our center with enzyme linked immunosorbant assay (ELISA). Survey of hepatitis B status in hemodialysis patients in a training hospital in Urmia.5%. sex. Urmia. 2008. 2. marital status. 0. in males than females (10. duration of the therapy. and in those on three times dialysis than twice per week (7. respectively). The mean frequency of HbsAg+ was 6. respectively). 19(3):466‐9. Abstract 4 • Khameneh ZR.3% vs. Iran. • Abstract • To evaluate the prevalence of HBV infection in chronic hemodialysis patients at our dialysis center of Urmia's Taleqni Hospital. 5. Iran. history of blood transfusion. Sepehrvand N. Saudi Journal of Kidney Diseases & Transplantation. . and prevalence of HbsAg+.3%. which was higher in patients less than 50 years old than in those above 50 years (9. respectively). job status.58%. We did not find a significant relationship between the factors of: age. being resident in city or village.

mental disorders. chlamydia. medication utilization. low birthweight. Canadian Journal of Public Health Revue Canadienne de Sante Publique 97(6): 435‐9.683). gonorrhea. diabetes. chronic obstructive pulmonary disease. Neudorf C. infant mortality and all‐cause mortality. • Abstract • BACKGROUND: Canadian cities are becoming more segregated by income. There were two comparison groups: all other Saskatoon residents (N=184. average‐ and high‐income neighbourhoods in order to quantify the level of health disparity at the scale of an urban city. teen birth. Opondo J. injuries and poisonings. Abstract 5 • Lemstra M. 2006. As such. The rate ratios increased in size when comparing low‐income neighbourhoods to high‐income neighbourhoods. FINDINGS: Statistically significant differences in health care utilization by neighbourhood income status were observed for suicide attempts. investigation is required into the magnitude of health disparity between low‐. public health information and vital statistics for an entire city by neighbourhood income status.228). No clear trend was observed for stroke or cancer. Postal code information was used to identify six existing contiguous residential neighbourhoods in the city of Saskatoon that were defined as low‐income cut‐off neighbourhoods (N=18. coronary heart disease. Health disparity by neighbourhood income. . METHODS: A cross‐sectional ecological study design was used to review all hospital discharges. physician visits.284) and the five most affluent neighbourhoods in Saskatoon (N=16. hepatitis C.

Questions ?? .

Thank You .