Critical Appraisal of a Cohort Study plus tips for EBP assignment writing

Yoon et al. Association between human papillomavirus vaccination and serious adverse events in
South Korean adolescent girls: nationwide cohort study. BMJ 2021; 372 doi:
https://doi.org/10.1136/bmj.m4931

1. Did the study address a clearly focused issue?

The objective was “to evaluate the association between human papillomavirus (HPV) vaccination
and serious adverse events in adolescent girls in South Korea” and aimed to provide evidence for an
Asian population due to under-representation in previous studies. The population was girls in South
Korea aged 11-14 who were vaccinated against HPV, Japanese encephalitis, and tetanus, diphtheria,
and acellular pertussis in 2017. The outcome was a list of 33 serious adverse events based on the
medical literature of reported adverse outcomes.

2. Was the cohort recruited in an acceptable way?

The researchers used a nationwide linked database created from combining the Korea Immunization
Registry Information System (KIRIS) and the National Health Information Database (NHID) which Commented [PW1]: Note- it is fine to use abbreviations in
covers the entire population of South Korea. All the included participants had received at least one your assignment but you must introduce these by writing
them out in full the first time of use.
of the listed vaccinations- this was to avoid any bias due to fundamental differences between
vaccinated and unvaccinated populations and the rationale for this was explained. As the
researchers used anonymous nationally routinely collected data there was no need to recruit
participants into the study and enabled the maximum amount of data to be analysed.

3. Was the exposure accurately measured to minimise bias?

The exposure to HPV vaccination was measured using objective data from the Korea Immunization
Registry Information System. This is good example of using a national database with very high
completeness of data capture to provide verifiable objective study data. The data was inputted
before the study was started by practitioners independent of the research team so is unlikely to be
affected by bias, although it is possible bias could occur if there are fundamental differences in those
whose data is missing from the registry.

4. Was the outcome accurately measured to minimise bias?

Outcomes were identified from the NHID on medical diagnosis and included date of diagnosis,
prescription, and treatment given. Diagnosis and drug prescription information was coded using ICD-
10 (international classification of diseases, 10th revision) which is the internationally agreed
standard for disease classification. The disease outcomes were recorded by medical practitioners
independently of the researchers, diagnoses were not made as part of the study and therefore were
not biased by the objectives of the study.

5a. Have the authors identified all important confounding factors?

Other potential confounders include previous adverse effects from other childhood vaccinations as
this may linked, although this has not been reported in the medical literature. Commented [PW2]: Demonstrate that you have tried to
identify evidence to support your ideas even if not available.
5b. Have they taken account of the confounding factors in the design and/or analysis?

Secondary analysis using self-controlled risk interval (SCRI) to minimise possible bias present in the
cohort analysis was performed, sensitivity analysis was conducted using short and 2 year follow-up
times and sub-group analysis was performed on the two different types of HPV vaccine. The
statistical analysis was adjusted for age, region of residence, type of health insurance, income level,
and anaemia which takes account of all the listed confounders. The secondary analysis by SCRI
enabled the researchers to assess the relative risk of conditions even when there were zero cases in
the unvaccinated group.

SCRI only uses data from vaccinated people with an outcome of interest. It compares, within each
case, the risk of that outcome during a defined risk interval after vaccination with the risk a during a
control interval. SCRI is widely used in vaccine safety monitoring to detect potential elevated risks of
adverse events following vaccinations1. The self-controlled design has the advantage of controlling
fixed confounders, although because the HPV vaccine is given at around the time of menarche any
confounding causes by puberty would not be controlled for2 Age at vaccination has been identified
as a confounder in studies of chronic fatigue following HP vaccination3.
6a. Was the follow up of subjects complete enough?
The NHID covers 97% of the South Korean population4. Those not included receive healthcare from a
different medical aid program or are temporary or illegal residents, many of whom would not
receive vaccination in South Korea and so would not be included in the inclusion criteria for this
study.
Commented [PW3]: This is explained in the article in
figure 1- by explaining this is your own words you show the
examiner that you understand how this method is different
from the standard cohort analysis.
Commented [PW4]: Show the examiner that you have
done some wider reading and understand the strengths and
weaknesses of any usual methods or techniques used in the
study.
Commented [PW5]: For high scores show the examiner
that you have done wide reading to verify this independently
and not just relied on what the authors have told us in the
paper.

The list of 33 possible serious adverse events were based on HPV vaccination reported in the
medical literature. The list of adverse events are all serious medical conditions and it is highly
unlikely that they would not be reported and recorded. Other chronic adverse effects have been
reported in the medical literature following HPV vaccination, such as primary ovarian insufficiency,
postural orthostatic tachycardia syndrome (POTS), complex regional pain syndrome (CRPS) and
alopecia5,6,7. As Cervantes and Doan point out CRPS and POTS are difficult to diagnose so data on
these conditions may be missing or incomplete in the NHID and reaching a diagnosis may take longer
than follow up of this study8. Any association between HPV vaccine and outcomes other than the 33
conditions listed cannot be ruled out by this study- the omission of these conditions that have been
reported in the medical literature is a major weakness of this study. Commented [PW6]: Show the examiner that you have
done some wider reading on the subject and identified other
relevant studies that highlight either strengths or
weaknesses in the study.
6b. Was the follow up of subjects long enough?

Sensitivity analysis provided two years of follow up which should be adequate to capture any
adverse events, as the plausibility of new disease onset being caused by the vaccine decreased as
time since vaccination increases. A case series from Japan reported a small number of girls
presenting with adverse effects up to four years following their first HPV vaccine dose but the vast
majority of cases presented within the first year9. Commented [PW7]: Show the examiner that you are not
just making assumptions but that you are supporting your
7. What are the results of this study? ideas with evidence.

No association with HPV vaccine was found for 32 of the 33 outcomes. There was a small increased
risk of migraine associated with receiving HPV vaccination in the cohort analysis (relative risk, RR
1.11) but this association was not demonstrated by the SCRI analysis (RR 0.67). The sensitivity and
sub-groups analyses gave consistent results indicating that the results are robust.

8. How precise are the results?

The precision of the rate ratios indicated by greater narrowness in the 95% confidence interval (CI)
varies by outcome. This can be partly explained by the commonness of the outcomes, with narrower
CIs for common diseases, such as migraine (1.02 to 1.22) and wide CIs for rare diseases, such as
intracerebral haemorrhage (0.47 to 5.30).
Commented [PW8]: This is a slightly unusual cohort study
in that we are not really expecting a narrow CI around a
point estimate of effect size but you can still show the
examiner that you understand that narrow CIs indicate
precision and wider CIs indicate uncertainty.
9. Do you believe the results?

The fact that the sensitivity analyses and sub-group analyses all gave consistent results suggests that
the results are robust. The results are biologically plausible as there is no obvious causal pathways
between the listed outcomes and HPV vaccine. Commented [PW9]: This comment demonstrates that I
am thinking about the criteria for causality.
10. Can the results be applied to the local population?

Previous studies have already provided evidence of HPV vaccine safety in the UK population10,11. This
study adds to that evidence. 7% of the UK population, although majority of this group are Pakistani,
Indian and Bangladeshi and may not be represented by this study12. The same HPV vaccines are used
in the UK as in South Korea and the vaccine is administered at around the same age13. This study did
not include boys, who since 2019 have received HPV vaccine in the UK14. Commented [PW10]: Demonstrate some wider reading
comparing epidemiology of the study population with your
11. Do the results of this study fit with other available evidence? own country or region. Highlight similarities or differences
between the ability to implement the intervention. Higher
The conclusion that HPV vaccination is not associated with any adverse events is consistent with a marks for supporting your answer with contemporary and
recent Cochrane review of 23 RCTs with a follow-up of up to seven years15. A WHO review also relevant evidence.
concluded it was extremely safe16. Commented [PW11]: Presentation point- it is usual
academic writing convention to spell out number less than
12. What are the implications of this study for practice? 10.
Commented [PW12]: Demonstrate the you have looked
This study indicates no cause for concern regarding the safety of the HPV vaccine and provides
for best evidence.
reassurance that the vaccine is safe to administer to girls without risk of serious adverse effects. This
Commented [PW13]: Show wider reading by citing
study can be used to provide reassurance to parents and girls that receiving the HPV vaccine will not references beyond those cited in the paper
cause them serious harm. A repeat of the study in boys would be necessary to extend reassurance to
that group.

How are the expectations for the weekly practice critical appraisals different to the expectations
for a critical appraisal assignment?

For the practice weekly critical appraisal I expect you to spend 1 or 2 hours so this would not include
wider reading.

For the final assignment you are allocated 30 hours of study time so there is an expectation for you
to demonstrate wider reading related to the study methodology and the topic it is examining.

What I had to do to prepare this critical appraisal over and above simply reading the article:

 Read up about the SCRI study design in order to be able to fully understand the
methodology used and be able to spot potential biases
 Check through all the supplementary files
 Do a specific search for other studies about HPV vaccine adverse effects and compare my
findings with the evidence base reported in this paper- don’t just accept that the wider
evidence reported by the authors truly represents the totality of the evidence base.
 Research the key differences between my own population and South Korea to understand
the external validity of the study to my own setting.
 Compare any differences in HPV vaccination programme between South Korea and my own
setting that could alter outcomes, e.g. age/ setting of vaccination and inclusion/ exclusion of
boys
  Look for other research papers conducted on my own population for comparison

References

1. Li R, Stewart B, Weintraub E. Evaluating efficiency and statistical power of self-controlled case

series and self-controlled risk interval designs in vaccine safety. J Biopharm Stat,
2016;26(4):686-93. doi: 10.1080/10543406.2015.1052819.
2. Tokars JI et al. The Risk of Guillain–Barré Syndrome Associated with Influenza A (H1N1) 2009
Monovalent Vaccine and 2009–2010 Seasonal Influenza Vaccines: Results from Self‐
Controlled Analyses. Pharmacoepidemiol Drug Saf, 2012;21:546-552.
https://doi.org/10.1002/pds.3220
3. Palmieri B et al. Severe somatoform and dysautonomic syndromes after HPV vaccination:
case series and review of literature. Immunol Res, 2017; 65(1): 106–116. doi:
10.1007/s12026-016-8820-z
4. Kim HK et al. Data Configuration and Publication Trends for the Korean National Health
Insurance and Health Insurance Review & Assessment Database. Diabetes Metab J, 2020;
44(5): 671–678. doi: 10.4093/dmj.2020.0207
5. Gong L et al. Human papillomavirus vaccine-associated premature ovarian insufficiency and
related adverse events: data mining of Vaccine Adverse Event Reporting System. Sci Rep,
2020;10(1):10762. doi: 10.1038/s41598-020-67668-1.
6. Phillips A et al. Adverse events following HPV vaccination: 11 years of surveillance in
Australia. Vaccine, 2020;38(38):6038-6046. doi: 10.1016/j.vaccine.2020.06.039.
7. Neha R, Subeesh V, Beulah E, Gouri N, Maheswari E. Postlicensure surveillance of human
papillomavirus vaccine using the Vaccine Adverse Event Reporting System, 2006-2017.
Perspect Clin Res, 2020;11(1):24-30. doi: 10.4103/picr.PICR_140_18.
8. Cervantes JL and Doan AH. Discrepancies in the evaluation of the safety of the human
papillomavirus vaccine. Mem Inst Oswaldo Cruz, 2018; 113(8): e180063. doi: 10.1590/0074-
02760180063
9. Ozawa K et al. Suspected Adverse Effects After Human Papillomavirus Vaccination: A
Temporal Relationship Between Vaccine Administration and the Appearance of Symptoms in
Japan. Drug Saf, 2017; 40(12): 1219–1229. doi: 10.1007/s40264-017-0574-6

10. Andrews N, Stowe J, Miller E. No increased risk of Guillain-Barré syndrome after human
papilloma virus vaccine: A self-controlled case-series study in England. Vaccine.
2017;35(13):1729–1732.
11. Willame C, Rosillon D, Zima J, Angelo MG, Stuurman AL, Vroling H, Boggon R, Bunge EM,
Pladevall-Vila M, Baril L. Risk of new onset autoimmune disease in 9- to 25-year-old women
exposed to human papillomavirus-16/18 AS04-adjuvanted vaccine in the United Kingdom.
Hum Vaccin Immunother. 2016 Nov;12(11):2862-2871. doi:
10.1080/21645515.2016.1199308.
12. Office for National Statistics. 2011 Census [internet]
https://www.ons.gov.uk/census/2011census last updated 8 March 2017
13. Public Health England. HPV vaccination programme. [internet], 2019
https://www.gov.uk/government/collections/hpv-vaccination-programme
14. Public Health England. HPV universal immunisation programme: implementation letter
[internet], June 2019. https://www.gov.uk/government/publications/hpv-universal-
immunisation-programme-implementation-letter

15. Arbyn M, Xu L, Simoens C, Martin‐Hirsch PPL. Prophylactic vaccination against human

papillomaviruses to prevent cervical cancer and its precursors. Cochrane Database of
 Systematic Reviews 2018, Issue 5. Art. No.: CD009069. DOI:
10.1002/14651858.CD009069.pub3.
16. WHO Global Advisory Committee on Vaccine Safety. Safety update of HPV vaccines [internet],
2017 https://www.who.int/vaccine_safety/committee/topics/hpv/June_2017/en/