Diagnostic research

Lecture Contents
I.
II. Diagnostics in practice
- Explained with a case

Scientific diagnostic research – Design
– – Data-analysis Reporting

III. IV.

Exercises Summary

Diagnostics in practice
Diagnostics always start with a patient with a complaint/symptom Case: neck stiffness • Child, 2 years-old, comes to ER with parents • Child turns out to have a very stiff neck What is the physician’s aim?

Diagnostics in practice Aim of the physician • Quickly and efficiently determine the correct diagnosis Why diagnose? • Basis medical handling • Determines treatment choice • Gives information about prognosis What are possible diagnoses for neck stiffness? .

g.Diagnostics in practice Differential diagnosis (DD) • Bacterial meningitis • Viral meningitis • Pneumonia • ENT infection • Other (e. myalgia) What is the most important diagnosis? Which one does the physician not want to miss? .

Diagnostics in practice Most important diagnosis • Bacterial meningitis (BM) • If missed: often fatal .

Diagnostics in practice Suppose: 20% of all children on the ER with neck stiffness has BM – 20% with disease in that population = prevalence – Prior-probability What is your decision for the child in this case? .

Diagnostics in practice Decision for child in case • Prior-probability too low to treat • Prior-probability too high to send home Decision: reduce uncertainty  diagnostics What is the best test? .

Diagnostics in practice Best test Lumbal punction (liquor culture) .

Diagnostics in practice Gold standard • True disease status. ‘truth’ – Never 24 karat • Reference standard/test • Decisive test with doubt Perform reference test for everybody (=every child on ER with neck stiffness)? .

Diagnostics in practice Reference test for everybody? • Unethical  too invasive/risky • Inefficient  too expensive • Do not perform unnecessarily How should we then determine the probability of disease presence and what would be ideal? .

etc. imaging. physical exam. – Ideal: diagnosis without reference test • Diagnostic process in practice: – Stepwise process: less  more invasive – Not one diagnosis based on 1 test – Each item: separate test .Diagnostics in practice How then? • Simpler diagnostics: – Usually anamnesis. simple lab tests.

Diagnostics in practice Suppose: after anamnesis & PE 10% probability of BM • Probability of disease given test results = posteriorprobability • The bigger the difference between prior and posterior probability. the better the diagnostic value of the tests Our decision for child in case: probability is too high to send home --> next step? .

etc. blood tests (leucocytes. sedimentation. CRP.Diagnostics in practice Next step – Additional research.g.) . e.

Diagnostics in practice Suppose: 1% posterior-probability after anamnesis. PE+ simple lab tests posterior probability low enough to send home • Ideal diagnostic process: simple tests reduce posterior probability to 0 or 100% (without reference) • Most often physician continues testing until sufficiently sure (approximation of 0 or 100%) • Choose when sufficiently sure: depends on prognosis of disease if untreated + risks/costs treatment .

Diagnostics in practice Summarizing • What does diagnosing involve in practice? – Estimation of probability of disease presence based on test results of the patient When is the probability of disease best estimated? Why is this usually not done? .

Diagnostics in practice Why not all possible tests? – – – Invasive (for patient and budget) Unnecessary: different test results give same info However: In practice often more tested than necessary! What diagnostics truly necessary diagnostic research scientific .

Study design Scientific diagnostic research – What tests truly contribute to probability estimation? – Has to serve practice follow practice .

Study design • • • • • • • Research question Domain Study population Determinant(s): test(s) to study Endpoint: presence/absence disease (outcome) Study design: design Data analysis. interpretation + reporting .

safe. and cheap tests estimate the probability of the presence/absence of disease.Research question • With as few as possible simple. • Determinant-outcome relation: – probability of disease as a function of test results – outcome = probability of disease = % = prevalence – test results = determinants .

blood CRP. etc) . blood leucocytes.Research question Case • What tests contribute to probability estimation of presence or absence of BM in children with neck stiffness at the ER? • Or: Determinants of presence/absence disease (BM)? • %BM = ƒ(age. fever. gender.

Research population
Case: • All children with neck stiffness in 2002 at ER Utrecht

Domain
• For whom domain, generalisation
= type of patient with certain symptom/complaint + setting

• Research population = 1 sample from domain Case:
All children (e.g. in Western world) suspected of disease (BM) based on neck stiffness (characteristic) in secondary care (setting)

Determinants
= Tests to study
• Diagnostic determinants • All possible important tests (in domain) Case Items anamnesis, PE and lab (blood and urine) tests

Endpoint ‘True’ presence/absence disease = Diagnostic outcome = Results reference test NB: reference = not infallible but always best available test in practice at that moment Case • Positive liquor culture .

PICO EBM • • • • Population/ problem Intervention Comparison/ control Outcome • • • • Domain Determinant Reference test Outcome .

Measure determinants/endpoint • Determinants – – Without knowledge (blinded) of the outcome Same method in study and practice never measure more precisely than in practice (overestimation information yield) • Endpoint – – Assessment blind for determinants With the best possible test known in practice .

Study design • Observational and descriptive – Observational = no manipulation of determinants – Descriptive = not causal – if the determinant only predicts – no hypothesis functional mechanism determinantoutcome • >1 determinant .

Study design • Cross-sectional = Simultaneously measure determinants and outcome .

per patient – Value determinants (test results) – Diagnostic outcome (reference test) .Data-analysis After data collection.

Data-analysis • Data analysis: 3 steps 1) Estimation a priori probability (without test results) 2) Compare each test result separately with reference = univariate 3) Compare combination of test results with reference = multivariate (via model) .Determine added value test result to already collected (previous) test results .Following order in practice .

Data-analysis Case Data scientific research available: 200 patients with neck stiffness at ER Liquor culture positive (BM+) n=40 Liquor culture negative (BM-) n=160 Step 1: A priori probability (prevalence) of BM? = % BM+ = 40/ 200 patients = 20% .

Data-analysis reading 2 by 2 table • • Step 2: Analysis per determinant (univariate) Use 2 by 2 table Disease Presence Absence False positive B D True negative Test Positive True positive A Negative C False negative .

Data-analysis reading 2 by 2 table Horizontally • Positive predictive value (PV+) = probability Disease + if test + PV+ = A / A + B • Negative predictive value (PV-) = probability disease .if test PV.if disease SP = D / B + D What numbers do you think are most useful in practice (PV+ and PV.= D / C + D Gold standard Disease + Disease – Test + TP FN A B C D FP TN Vertically • Sensitivity (SE) = probability test + if disease + SE = A / A + C • Specificity (SP) = probability test .or SE and SP)? Test – .

Data-analysis Perfect diagnostic test False Positive = 0 False Negative = 0 e.tot. Fever > 380C as predictor for BM Yes (+) Fever > 380C No (-) BM+ BM. 20 20 90 110 70 90 40 160 200 .g.

= 70/160 = 44% SP = D / B + D Fever – TP FN 20 A B C D FP TN 70 What numbers do you think are most useful in practice (PV+ and PV.= 70/90 = 78% PV.if BM.if fever.= D / C + D Gold standard BM+ BM– 20 Fever + 90 Vertically • probability fever+ if BM+ = 20/40 = 50% SE = A / A + C • probability fever.or SE and SP)? .Data-analysis reading 2 by 2 table Horizontally • probability BM+ if fever+ = 20/110 = 18% PV+ = A / A + B • probability BM .

Exercise 1 Mercury thermometer or timpanic membrane infrared meter .

Exercise 1 Ad question 1 Research question: Can fever be determined with the TIM? Determinant: test under study = timpanic membrane infrared meter Outcome: fever determined with rectal mercury thermometer (RMT) Domain: Children in secondary/tertiary care (ER hospital) .

Exercise 1 Ad question 2 GS RMT Fever+ Fever– Se = probability TIM+ if RMT+ = 77/96 = 80 % SP = probability TIM.if RMT.= 108/117 = 92% 77 TIM >38° 9 TIM 38° TP FN 19 A B C D FP TN 108 .

Exercise 1 Ad question 3 PV+ = probability RMT+ if TIM+ = 77/86 = 90 % PV.if TIM.= probability RMT.= 108/127 = 85% GS RMT Fever+ Fever– 77 TIM >38° 9 TIM 38° TP FN 19 A B C D FP TN 108 .

92  D=156) 34 14 48 TIM+ –PV+ becomes lower (34/48 = 70%) TIM9 156 164 –PV – becomes higher (156/164 = 95%) 43 170 213 . 20% (X/213=0.8  A=34) (D/170=0.2  X=43) – For similar Se and SP: GS RMT Fever+ Fever– (A/43=0.Exercise 1 Ad question 4 –The prior probability of fever in the general practice is lower.g. e.

Exercise 1 Ad question 5 – In the general practice an unjustly referred or treated child is less of a problem than an unjust reassurance of the parents – Especially the negative predictive value must therefore be sufficiently high .

Data-analysis: combination of determinants • In practice not one single diagnosis based on 1 test – Tests together distinguish ill/non-ill – Method: statistical model • Moreover: diagnostic process is hierarchical – (simple --> invasive/expensive) --> always start with anamnesis model --> see case .

pain) • Statistical model can be seen as 1 (composed) test • Quantify diagnostic value model with area under ROC curve (Receiver Operating Characteristic =Area Under Curve (AUC)) . age. fever.Data-analysis Case: model with all anamnestic tests gender + age + fever + pain %BM = ƒ(gender.

25 .75 Sensitivity .Specificity .00 .00 0.Data-analysis ROC Curve 1.25 0.50 .00 .00 1 .75 1.50 .

0 AUC = 0.0  perfect (Se=100% & 1-Sp=0%) .5 1.7  reasonable AUC > 0.9  excellent AUC = 1.71 Informal interpretation AUC = % correctly diagnosed The larger the ROC area  the better the model AUC range: 0.Data-analysis Case: AUC anamnesis model = 0.5  bad (Se = 1.8  good AUC > 0.Sp  diagonal [coin]) AUC > 0.

Data-analysis Quantify added value additional tests to previous tests • Extend previous model (follow order practice) • Quantify change in AUC Case Model 1 anamnesis model + physical exam (5 extra tests) --> AUC = 0.72  interpretation? Model 2 anamnesis model + 3 blood tests ---> AUC = 0.90  interpretation? .

00 0.Specificity .50 .25 Coin flip Patient hisotry 0.50 .Data-analysis ROC Curve 1.00 Pat hist + test 1 .00 .00 .25 .75 Sensitivity .75 1.

Data-analysis • The AUC does not directly say anything about individual patients and is therefore not directly applicable .

design Results • Predictive values (new) test and/or ROC curve • ROC curve combination of tests • Added value new test --> ROC curve . setting. determinants.Reporting Research question Study set-up • Research population. outcome.

Exercise 2 Ad question 1 .For all patients anamnestic data were collected .Cross-sectional study in patients suspected of a stomach or duodenum ulcer .For all patients a gastroscopy was done .Independent diagnostic value of anamnestic factors (determinant) for the diagnosis of ulcer (outcome: gastroscopy) were calculated .

Exercise 2 Ad question 2 Adults with stomach complaints referred to a gastroenterology policlinic in a peripheral hospital .

Exercise 2 Ad question 3 Score is 5. risk is 57% .

Of course this is not true and the score loses precision .Everybody above the cut-off point has the same risk (and the same below the cut-off point) .Preferably predictive values for score-categories and predictive values for more cut-off points .Exercise 2 Ad question 4 .

Exercise 2 Ad question 5 PV+ = 20/31 = 65% PV.= 64/69 = 93% Test + Peptic ulcus + – 20 11 A B C D TP FN Test - FP TN 64 5 .

Predictive values more favourable and therefore preferred .Exercise 2 Ad question 6 .But it is not about the isolated predictive value but about the added diagnostic value given the results of the anamnestic score .

. ... breath test) • Then compare the Receiver Operating Characteristic (ROC)-curve of the models .) • P(ulcus) = ƒ (age. gender.. anamnesis.. anamnesis.Exercise 2 Ad question 7 • Perform the anamnestic score and the breath test for a population from the domain. . gender. Subsequently perform the reference test (endoscopy) for everybody • Compare the next determinant-outcome relations: • P(ulcus) = ƒ (age.

Exercise 2 Ad question 8 . then the breath test also measures age and therefore the added value is less than when the breath test would be completely independent of the score .Suppose that the breath test is more often positive with age.Breath test partially contains the same information as the score .

than it should be the same in the study . but if the assessor is informed of the data in the score in practice.Preferably not.Exercise 2 Ad question 9 .

Diagnostics Summary (1) Diagnostics in practice – Uncertainty reduction – Determines prognosis & determines policy Diagnostic research Design – Observational – Descriptive .

Diagnostics Summary (2) – Cross-sectional • Simultaneous measurement determinant and outcome (reference standard) – Always study >1 determinant Design – Assess determinants as in practice – Assess disease status & determinant status with double blinding .

Diagnostics Summary (3) Analysis – Univariate (per determinant) – Multivariate: combination of test results in relation to outcome • Endpoint = ƒ(combination of determinants) • Determine added value. first analyse least invasive tests (as in practice) Reporting – Mainly added value of test .