Systema(c  review  and  meta-­‐analysis   of  Diagnos(c  Test  Studies    

Kuan-­‐Fu  Chen   CGMH  EM   SR/MA  workshop     January  8,  2013  

Objec(ves  
•  •  •  •  •  •  •  •  Differences  of  diagnos(c  SR/MA?     Steps  of  diagnos(c  SR/MA   Formula(ng  a  focused  ques(on     Reviewing  guidelines:  QUADAS     Literature  searching   Evalua(on  of  studies     Data  extrac(on   Data  synthesis  
KFC  CGMH  EM  Dx  SR/MR  talk   2  

1/8/13  

Difference  of  Dx  SR/MA  
•  •  •  •  ID  studies,  assessing  bias     Method  to  combine  results   Paired  of  summary  sta(s(cs  to  pool   Design  of  diagnos(c  studies    

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

3  

Diagnos(c  studies  
•  Defini(on    
–  Dia  +  gnosis  =  apart/separated  +  knowledge   –  Dia:  Greek  ‘through’   –  Gnosis:  Greek  ‘knowledge’   –  To  reduce  uncertainty  

•  Purposes    
–  Screening,  triage,  add-­‐on  or  replacement  tests,   predict  outcomes  or  monitor  dz  process   –  Purpose  must  be  considered  during  evalua(on  
1/8/13   KFC  CGMH  EM  Dx  SR/MR  talk   4  

Phases  of  Dx  test  evalua(on    
•  Prelim  evalua(on  (case-­‐control)  
–  I:  whether  results  different  for  pts  ±  dz   –  II:  whether  dz  more  likely  to  have  +  results    

•  Accuracy  &  probability  of  condi(on      (cross-­‐sec(onal/cohort  study)  
–  III:  how  well  dis(nguishes  btw  pts  ±  dz  with    suspicion   –  IV:  how  informa(ve  as  add-­‐on  

•  Health  outcomes  (Randomized  or  before-­‐ader  study)  
–  V:  whether  leads  to  beeer  outcomes   –  VI:  acceptable  costs  
1/8/13   KFC  CGMH  EM  Dx  SR/MR  talk   5  

Phases  of  biomarker  evalua(on    

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

6  

Objec(ves  
•  •  •  •  •  •  •  •  Differences  of  diagnos(c  SR/MA?     Steps  of  diagnos(c  SR/MA   Formula(ng  a  focused  ques(on     Reviewing  guidelines:  QUADAS     Literature  searching   Evalua(on  of  studies     Data  extrac(on   Data  synthesis  
KFC  CGMH  EM  Dx  SR/MR  talk   7  

1/8/13  

Steps  of  (diagnos(c)  SR/MA  
•  •  •  •  •  •  •  •  Formula(ng  a  focused  ques(on   Reviewing  guidelines   ID  databases/sources  of  studies   Run  searches  and  save  cita(ons   First  screen  by  two  reviewers     Second  screen  by  two  reviewers     Data  extrac(on  and  quality  assessment     Data  analyses    
KFC  CGMH  EM  Dx  SR/MR  talk   8  

1/8/13  

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

9  

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

10  

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

11  

Objec(ves  
•  •  •  •  •  •  •  •  Differences  of  diagnos(c  SR/MA?     Steps  of  diagnos(c  SR/MA   Formula(ng  a  focused  ques(on     Reviewing  guidelines:  QUADAS     Literature  searching   Evalua(on  of  studies     Data  extrac(on   Data  synthesis  
KFC  CGMH  EM  Dx  SR/MR  talk   12  

1/8/13  

PICO(S)  for  of  diagnos(c  studies  
•  Popula(on    
–  What  are  the  characteris(cs  of  the  pa(ents?     –  What  is  the  condi(on  that  may  be  present?   –  Which  diagnos(c  test  am  I  considering?   –  What  is  the  diagnos(c  gold  standard?   –  How  likely  is  the  test  to  predict/rule  out  this  condi(on?   –  What  study  design  would  provide  the  best  level  of  evidence?   –  What  clinical  sekng  would  this  study  apply  to?  
KFC  CGMH  EM  Dx  SR/MR  talk   13  

•  Interven(on  (diagnos(c  test)   •  Comparison   •  Outcome  

•  Study  design/sekng  

1/8/13  

Objec(ves  
•  •  •  •  •  •  •  •  Differences  of  diagnos(c  SR/MA?     Steps  of  diagnos(c  SR/MA   Formula(ng  a  focused  ques(on     Reviewing  guidelines:  QUADAS     Literature  searching   Evalua(on  of  studies     Data  extrac(on   Data  synthesis  
KFC  CGMH  EM  Dx  SR/MR  talk   14  

1/8/13  

Guidelines  for  (SR/MA)  studies  
•  QUA  D  A  S  
–  QUAlity  of  Diagnos(c  Accuracy  research  Studies  in  SR  

•  STA  R  D  
–  STAndard  for  Repor(ng  of  Diagnos(c  accuracy  

•  RE  MARK  
–  REpor(ng  recommenda(ons  for  tumor  MARKer   prognos(c  studies  

•  P  R  I  S  M  
–  Preferred  Repor(ng  Items  for  SR/MA  
1/8/13   KFC  CGMH  EM  Dx  SR/MR  talk   15  

QUADAS  (2)  
•  Four  phases  to  establish  tool  for  each  review   •  Four  key  domains  to  review   •  Three  sec(ons  in  each  domain  for  risk  of  bias  

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

16  

QUADAS  –  phases    
•  •  •  •  Review  ques(ons   Review  specific  tailoring   Flow  diagram   Judgments  on  bias  and  applicability  

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

17  

QUADAS  –  phases    
•  Review  ques(ons  
–  Four-­‐part  ques(on:    
•  (P)  Pa(ents,     •  (I)  Index  test(s),     •  (C)  Reference  standard,  and     •  (O)  Target  condi(on  

–  Diagnos(c  pathway:    
•  Sekng,  inten(on,  pa(ent  presenta(on,  and/or  prior   tes(ng  
1/8/13   KFC  CGMH  EM  Dx  SR/MR  talk   18  

QUADAS  –  phases    
•  Review  specific  tailoring  

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

19  

QUADAS  –  phases    
•  Flow  diagram  

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

20  

QUADAS  –  phases    
•  Judgments  on  bias  and  applicability  
–  Three  sec(ons  in  each  domain  for  risk  of  bias  
•  Informa(on  to  support     •  Signaling  ques(ons   •  Judgment  of  risk  of  bias  

–  Applicability  
•  Similar  but  not  including  signaling  ques(ons   •  Refer  to  first  phase  (review  ques(on)  

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

21  

QUADAS  –  domains    
•  •  •  •  Pa(ent  selec(on     Index  test     Reference  standard     Flow  and  (ming    

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

22  

QUADAS  –  domains    
•  Pa(ent  selec(on  
–  Signaling  ques(ons  
1.  Was  a  consecu(ve  or  random  sample  of  pa(ents  enrolled?     2.  Was  a  case-­‐control  design  avoided?   3.  Did  the  study  avoid  inappropriate  exclusions?   •  Spectrum  bias  (one  of  selec(on  biases)  (lab)?   •  Excluding  difficult  to  diagnose;  liele  diagnosis  uncertainty:  
–   overop(mis(c  

–  Risk  of  bias  

•  Exclusion  of  “red  flags”:  underes(ma(on     •  Selec(on  towards  more  severe  manifesta(ons  
–  Increase  prevalence   –  Bias  in  any  direc(on    

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

23  

QUADAS  –  domains    
population

Study&

population

Study&

•  Pa(ent  selec(on  
–  Applicability  concerns  

Target' Population

Target' Population

•  Target  vs.  Study  popula(on:  generalizability       •  Any  discrepancy  re:  severity,  demographics,  presence   of  differen(al  diagnosis,  comorbidity,  sekng  and   previous  tes(ng  protocol  
Target' Population

population

Study&

–  Supplemental:  prospec(ve  vs.  Retrospec(ve?    
•  Retrospec(ve:    
–  No  verifica(on  of  Dz  status   –  Same  reference  standard  for  all?   –  Same  assessors  for  reference  standard  for  all?    
1/8/13   KFC  CGMH  EM  Dx  SR/MR  talk   24  

QUADAS  –  domains    
•  Index  test    
–  Signaling    ques(ons  
1.  Were  the  index  test  results  interpreted  without   knowledge  of  the  results  of  the  reference  standard?   2.  If  a  threshold  was  used,  was  it  pre-­‐specified?  

–  Risk  of  bias  
•  Ques(on  1:  blinding,  subjec(vity   •  Ques(on  2:  overop(mis(c      

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

25  

QUADAS  –  domains    
•  Index  test    
–  Applicability  concerns  
•  Same  index  tests?    

–  Supplemental:  (QUADAS  1)  
•  Reproducibility:  for  all  tests,  different  phases  

 

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

26  

QUADAS  –  domains    
•  Reference  standard    
–  Signaling    ques(ons  
1.  Is  the  reference  standard  likely  to  correctly  classify   the  target  condi(on?   2.  Were  the  reference  standard  results  interpreted   without  knowledge  of  the  results  of  the  index  test?  

–  Risk  of  bias  
•  Ques(on  1:  direc(on?  Misclassifica(on  bias   (textbook?)  
–  Bias  confounding  and  role  of  chance  slide  

•  Ques(on  2:  similar  to  index  test,  Incorpora(on  bias  
1/8/13   KFC  CGMH  EM  Dx  SR/MR  talk   27  

QUADAS  –  domains    
•  Reference  standard    
–  Applicability  concerns:    
•  target  condi(on  defined  by  the  reference  standard   does  not  match  the  ques(on?   •  Example:  U/C  for  UTI  with  different  cutoffs  

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

28  

QUADAS  –  domains    
•  Flow  and  (ming    
–  Signaling    ques(ons  
1.  Was  there  an  appropriate  interval  between  index  test  and   reference  standard?   2.  Did  all  pa(ents  receive  the  same  reference  standard?   3.  Were  all  pa(ents  included  in  the  analysis?  

–  Risk  of  bias  
•  Ques(on  1:  misclassifica(on  bias  (if  delay  or  treatment   started).  Direc(on?  Acute  vs.  Chronic     •  Ques(on  2:  Verifica(on  bias:  overes(mate   •  Ques(on  3:  lost  to  follow-­‐up:  Direc(on?  Could  be  either   direc(on  (selec(on  bias)  
1/8/13   KFC  CGMH  EM  Dx  SR/MR  talk   29  

Objec(ves  
•  •  •  •  •  •  •  •  Differences  of  diagnos(c  SR/MA?     Steps  of  diagnos(c  SR/MA   Formula(ng  a  focused  ques(on     Reviewing  guidelines:  QUADAS     Literature  searching   Evalua(on  of  studies     Data  extrac(on   Data  synthesis  
KFC  CGMH  EM  Dx  SR/MR  talk   30  

1/8/13  

Literature  searching    
•  Electronic  database  
–  Less  produc(ve  and  more  difficult  

•  Different  strategies  
–  Different  indexing  terms/text  words   –  MeSH:  diagnosis   –  Textwords:  specificity,  sensi(vity  

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

31  

Objec(ves  
•  •  •  •  •  •  •  •  Differences  of  diagnos(c  SR/MA?     Steps  of  diagnos(c  SR/MA   Formula(ng  a  focused  ques(on     Reviewing  guidelines:  QUADAS     Literature  searching   Evalua(on  of  studies     Data  extrac(on   Data  synthesis  
KFC  CGMH  EM  Dx  SR/MR  talk   32  

1/8/13  

Evalua(on  of  studies      
•  Selec(on  of  sample.    
–  Ideal:  consecu(ve/randomly  selected  

•  •  •  • 

Reference  test   Blinding   Quality  repor(ng   Evidence  of  bias:    
–  table  of  rela(ve  DOR  in  different  study  design  

•  Incorpora(ng  QA  in  SR:  Checklist    
1/8/13   KFC  CGMH  EM  Dx  SR/MR  talk   33  

Objec(ves  
•  •  •  •  •  •  •  •  Differences  of  diagnos(c  SR/MA?     Steps  of  diagnos(c  SR/MA   Formula(ng  a  focused  ques(on     Reviewing  guidelines:  QUADAS     Literature  searching   Evalua(on  of  studies     Data  extrac(on   Data  synthesis  
KFC  CGMH  EM  Dx  SR/MR  talk   34  

1/8/13  

Data  extrac(on  
•  Accuracy  expression    
–  Sensi(vity/specificity:  Based  on  cut-­‐off   –  ROC  curve     –  PPV  →  influence  of  prevalence   –  May  consider  DOR  instead  of  Sensi(vity/specificity,  LR    
•  DOR  (Diagnos(c  Odds  Ra(o)  also  =  LR(+)/LR(-­‐)   •  However,  discarded  informa(on  
–  DOR  25:  LR(+)  5,  LR(-­‐)  0.2,  DOR  100:  LR(+)  10,  LR(-­‐)  0.1  

•  Choice  of  parameters:    

•  Predic(ve  value:  high  heterogeneity→prevalence  varia(on   •  Examples  of  abstract/ar(cle  review  form  
–  Post-­‐test  probability  can  be  used  instead    

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

35  

Objec(ves  
•  •  •  •  •  •  •  •  Differences  of  diagnos(c  SR/MA?     Steps  of  diagnos(c  SR/MA   Formula(ng  a  focused  ques(on     Reviewing  guidelines:  QUADAS     Literature  searching   Evalua(on  of  studies     Data  extrac(on   Data  synthesis  
KFC  CGMH  EM  Dx  SR/MR  talk   40  

1/8/13  

Data  synthesis  
Test of heterogeneity
heterogenous

Bayesian adaptation SROC asym

homogenous

Test of cut-off effect no

yes

LM method test of asymmetry, SROC or pool ROC curves

sym

Meta-regression DOR on characteristics, investigate source of heterogeneity

Pool sensitivity & specificity

SROC:  summary  ROC   LM  method:  Lieenburg  –  Moses  regression  method   DOR:  diagnos(c  odds  ra(o    
KFC  CGMH  EM  Dx  SR/MR  talk   41  

1/8/13  

Systema(c  review  and  meta-­‐analysis  of  Diagnos(c  Test  Studies    

WORKSHOP  TIME!  

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

42  

Objec(ves  
•  Understand  and  workout  poten(al  bias  during   diagnos(c  SR/MA  
–  Spectrum  bias     –  Misclassifica(on  bias   –  Incorpora(on  bias   –  Lost  of  follow  up   –  Selec(ng  threshold   –  verifica(on  bias    

•  Use  QUADAS-­‐2  form  to  evaluate  one  study  
1/8/13   KFC  CGMH  EM  Dx  SR/MR  talk   43  

Selec(on  of  pa(ent  (spectrum  bias)  
•  Excluding  difficult  to  diagnose  
–  Overop(mis(c  or  underes(ma(ng,  why?  

•  Excluding  ‘red  flag’  
–  Overop(mis(c  or  underes(ma(ng,  why?  

•  Evaluate  and  discuss  the  reasons  in  examples  
1/8/13   KFC  CGMH  EM  Dx  SR/MR  talk   44  

Example  1:  Centor’s  CPR  

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

45  

Example  2:  BNP  

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

46  

Example  2:  BNP  

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

47  

Misclassifica(on  bias  
•  Use  those  tables  to  discuss  the  reasons  
–  PCR  vs.  B/C  for  sepsis  diagnosis   –  PCT  vs.  B/C  for  sepsis  diagnosis  

•  Use  blood  culture  as  example  
–  Discuss  interval  and  reference  test  issue  

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

48  

Misclassifica(on  bias  
Cases:  B/C  +   PCR  +   PCR  -­‐   False     100   50   150   Sensi(vity=66%   Control:  B/C  -­‐   40   60   100   Specificity=60%   Total   150   100   250  

DifferenAal  misclassificaAon  bias   Say  Sensi(vity  of  B/C  =  80%,  specificity  =  90%,  and  related  to  PCR  results       Cases:  real  sepsis   PCR  +   PCR  -­‐   True  
1/8/13  

Control:  real  ctrl   40   60   100   Specificity=60%  

Total   150   100   250  

110   40   150   Sensi(vity=73%  

KFC  CGMH  EM  Dx  SR/MR  talk  

49  

Misclassifica(on  bias  
Cases:  B/C  +   PCT  +   PCT  -­‐   False     100   50   150   Sensi(vity=66%   Control:  B/C  -­‐   40   60   100   Specificity=60%   Total   150   100   250  

Non-­‐DifferenAal  misclassificaAon  bias   Say  Sensi(vity  of  B/C  =  80%,  specificity  =  75%,  and  not  related  to  PCT  results       Cases:  real  sepsis   PCT  +   PCT  -­‐   True  
1/8/13  

Control:  real  ctrl   25   37   62   Specificity=60%  

Total   150   100   250  

125   63   188   Sensi(vity=66%  

KFC  CGMH  EM  Dx  SR/MR  talk  

50  

Incorpora(on  bias  
•  Example:  SIRS  vs.  WBC  for  sepsis  diagnosis  
–  Overop(mis(c  or  underes(ma(ng,  why?  

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

51  

Lost  of  follow  up  
•  Example:  transferred  to  other  hospital  
–  Overop(mis(c  or  underes(ma(ng,  why?  

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

52  

Selec(ng  threshold  
•  Example:    
–  Post-­‐hoc  determina(on  of  BUN/Cre  >  15  as  risk   factor  in  acute  stroke   –  Overop(mis(c  or  underes(ma(ng,  why?  

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

53  

Verifica(on  bias    
•  Example:  PSA  
–  Verifica(on  and  incorpora(on  biases  in  studies   assessing  screening  tests:  prostate-­‐specific   an(gen  as  an  example  (Gupta  &  Roehrborn)   –  Overop(mis(c  or  underes(ma(ng,  why?  

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

54  

Verifica(on  bias    
Disease  +   PSA  +   PSA  -­‐   Total     231   27   258   Disease  -­‐   32   54   86   Total   263   8   344   %  Verified     61%   37%  

Sensi(vity=90%   Specificity=63%  

Corrected     PSA  +   PSA  -­‐   Total    

Disease  +  

Disease  -­‐  

Total  

Sensi(vity=__%   Specificity=__%  

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

55  

Verifica(on  bias    
Disease  +   PSA  +   PSA  -­‐   Total     231   27   258   Disease  -­‐   32   54   86   Total   263   8   344   %  Verified     61%   37%  

Sensi(vity=90%   Specificity=63%  

Disease  +   PSA  +   PSA  -­‐   Total     377   74   451  

Disease  -­‐   52   147   199  

Total   429   221   650  

Sensi(vity=84%   Specificity=74%  

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

56  

Verifica(on  bias    

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

57  

Verifica(on  bias    

1/8/13  

KFC  CGMH  EM  Dx  SR/MR  talk  

58  

Master your semester with Scribd & The New York Times

Special offer for students: Only $4.99/month.

Master your semester with Scribd & The New York Times

Cancel anytime.