.

.
" .
Interobserver variability between Radiology residents
and specialists in the interpretation of abdominal CT for
the diagnosis of acute abdomen at Hadassah University
Hospital, Ein Kerem.

: .
' .2003 :
:

" .
," ,.
.____________________ :

' .
" ," ,.
.___________________ :
07.2004 :

6 ................................................................................................ :
9 ........................................................................................................ :
19 ............................................................................................................:
28 ........................................................................................................ :
30 ........................................................................................... :
'32 ..................................................................................................... :
36 .............................................................................................. :



, " ].[1
1998 "Institute of Medicine" -" .
" ,

] .[2
,
].[2
.
,
.
" CT
" - ,
".
] ,[3
, .
.
) (
, .
, , . CT -
,
] CT .[4-6 , "
].[3
CT , , ) (
, Siewert , 91 , CT
90% 76% ) gold standard - /
/ CT ( ]" .[6 CT intraobserver variability --
3

 ,interobserver variability
] .[7 Leslie " Lewisham 50,
" , , CT
) , , ,( , " , .60% -

] Wise .[8 ,
, "
.Kappa 100 CT
. " ) (.
Kappa - 0.36 Albano .[9] 0.45 - CT
" . 7
103 CT 9:00 8:00 - ,
. 96 - 103 ) 93%,
.(97.2% - 87.8% :95%
].[10
, , ,
, CT,
.

 :
.1 ,Kappa
) ( .
Kappa ,0.8 - Kappa
0.4 .0.8
.2 CT -"
.gold standard
.3 ) (
.
.4 CT
.
"
" -
.

 :
.

: , .
2002 .2003 CT
, " ,
" .

:
, .
' , ' .

:
) ( .80% ) 20% 70%,(90% -
)( .95%
" Freedman ,
60 .[11] CT

.1 -
.
.2
, - ,
, ) (.
.3 :
: .
.

CT- " )
( , ) : (.
. CT -"
) '( , " .

 " ) '( , .
" '.
: -
)/ , ( , .
- " .1 :
.2 ,
.3 , . - " "1 " "2

) . :" (.
Pre test .
CT .
, CT
.

" .
-1 :
. -2
, -3 .
" '
.
,
.
. 60 CT
350 - CT .
CT - ,
, . .1 :
15cc ) ,(Telebrix
.2 . 90cc , Telebrix 1.8 ml/sec" .3 .

 ,
6.5" 5.5" . ,
, )
: (.
.

:
.1
.

" :
Kappa -
, CT
. Kappa -
, .

.
) (overall agreement
,
] . [7

.2

"
. .gold standard

.3

.:
.

,CT
).(appa>0.7

,CT -
) .(75% -

,
.

:
2002 2003 60 CT .
" )( ," ) '(
" ) '(.
"
, "
) .(1
-1 :

1.5

10.0

12

22

16

26

48

3.5

12

60

18

30.0

90

4.5

92

100.0

"

60

100.0

 CT 41/60 .
) (68% ) .(2
-2 " CT
:

41

11

" )( 16:00 8:00 .

(55%) 33/60 , , 16:00) .24:00- .(3
-3 :

* 8:00-16:00

16:00-24:00

33

24:00-8:00

15

:3
* ,16:00- 8:00 , CT .

10

 13
) " "1( . ' 11 - , 9
" . ' 14- , 9
" 9 . " ' , " '.
45/60 - , 34/45 -
) " "2( 11/45 -
) " "3( . ' 49/60-
, 36 " 37/49 .
, 30 " 12/49 -
, 6 " . ' 46/60 -
, 30 " 27/46 ,
, 22 "
, 19/46 - , 8 " .
) (60 - ,
) " '
( , ,
, ) , " '
( . ,
.
.
) '/'( " Kappa
.0.5-0.6 , Kappa ,
) 0.7 .(4
-4 ) Kappa (:

Kappa

95% Confidence Interval

'

0.6

0.4-0.8

'

0.5

0.3-0.7

' ' 0.7

0.5-0.8

11

 , , gold
standard .
) .(5
-5 :gold standard

)(95% CI

)(95% CI

)(95% CI

)(95% CI

90

91

75

97

'

)(60-100

)(78-98

)(46-93

)(86-100

64

91

69

91

'

)(41-89

)(80-97

)(41-89

)(80-97

'

78

93

'

)(52-94

)(84-98

92

97

96

97

)(62-100

)(87-100

)(62-100

)(87-100

12

 , "
,Kappa ) Kappa 0.4 , (0.6
) .(6
-6 :
Kappa

)(95%CI

)(95%CI

)(95%CI

0.6

100

100

)(0.3-0.9

)(47-100

)(89-100

'
-

0.5

70

100

)(0.3-0.8

)(26-94

)(88-100

'
3

0.6

80

80

)(0.4-0.8

)(41-99

)(63-94

'
3

0.4

71

82

)(0.2-0.7

)(33-95

)(62-94

'

) ( 36-
) (60% ) ' '( 47 -
) .(78% )) (1.7%
.(7
38 - 60 , .
,38 - 7 - ,8
, 1 - 8
.

13

 , ' , ,
, '
, , ,SMV - ,
. ) 13 (38
) "" 2( ,"
. ,13 - CT -
) ( .
) " "3( - ,
, " .

14

 -7 ) :" 38
60(.

'

: .

15

22

37

13

13

*1

.
:

.
"

36

22

60

15

12

20

.
*

11

18

***1

.
60

"

100

37

10

27

:7
* ) (.
* 2/11 - 9/11 -
.
*** , ' '
.
* 3/12 -
, 9/12 -
, .CT -

15


,
, ) .(8
, CI .
-8 , :
, .
" 95%
.
%

)
(
95% CI

50

94

30

67

63

100

50

50

)(4 ,2

)(16 ,15

)(2 ,3

)(2 ,3

)(7 ,8

)(5 ,5

)(2 ,2

)(2 ,1

'

9-91

73-100

13-100

13-100

52-99

55-100

22-100

2.5-98

50

80

100

80

80

30

)(4 ,2

)(16 ,15

)(2 ,2

)(2 ,2

)(7 ,5

)(5 ,5

)(2 ,0

)(2 ,3

'

9-91

73-100

22-100

22-100

33-95

55-100

0-78

13-100

87

67

63

80

)(2 ,2

)(15 ,15

)(3 ,2

)(3 ,2

)(8 ,5

)(5 ,5

)(2 ,0

)(1 ,3

'

22-100

)(82-100

13-100

13-100

28-89

55-100

2-87

' 100

:8
*

16

 :
: / ,
/ , / , , ,
, , , ,
, - , / .
15- , ' 28- '
16-.
:
CT :1 25 . ","1
,
' ,
" ."3 ) ,
, (, PV/PS -
33 X 35" ,
. ,
.

CT -" .
.
:

17

 :2 " ,"2
.

. '
1" , . '
, ,
. ,12 . CT -
.
-
.CT "
) (.
:

18

:
:
CT
.
, " Kappa )
0.5 Kappa ,0.6 ' '( .
)) (0.6 = Kappa .(4
CT ) Kappa 0.36 (0.45
].[12
, gold
.standard 64% ) 90%
' '( 91% CT )
.(5
) ( , 92% 97%
) .(6 ,
, 38 - 60 .
" .

,
) ,(6 , ,
) (.
" 36) 60% -( )
.(7

19

- , 60 )
- ( .
) ,Kappa (
.

:Selection bias .
,
.
)
( , ,
" ) . 60 CT
350 - (.

:Information bias ) ,
US ( , ' '
.
.
" )(
. , "
.CT -

:Misclassification )\(
." ,
) ,
(.

:Hawthorne effect " ,

.

20

 .
.

' , , '
, " ,
, " ' .

:
,
, "
. James '96-'93

" ,
] .[13 :
.
:

,
.
.

, .

":

,
, .

,
.

,
.
"

21

 , .
" ,
, 12 " .
, .
) :
( , .
.
,

].[13
" " .

" . "
"" , ,"inter observer variation" ,
)(
" ] .[14 CT
.1 :
.2 , .3 , .
Berlin ,
].[14
.1 :

] .[14 CT -
. CT -
3 - : ,
.

22

] .[4
: ) ,pitch ,(collimation
/ / )
( ] .[15 ,"
- , CT - " ,
) ( ,
. ,
,
, , , ,
, , , , ].[4
.1 : .2 , " .3 ,"
] .[4 ,
/ . .1 : ""
.2 , ,
.3 , " ].[4
)"
CT ( .
, ' )
( ] .[16-19
, // ,
] .[4 CT : ,
,

].[3

.2 :

) (.

23

) (
,
.

.CT -
Siewer , 91 93% CT
90%
] .[6 Taourel 97.5%
CT ][6

.3 :

,
Bechtold ,
CT ] .[14 694 ) 329 365 - ,
.(54 "
."error rate" -
: ) ,(inter reader variability
, , ,
. 53 " 56,
""overall error rate 19 ,7.6% 7-
. error rate -
) ,(16.1% - 3.6% ,0.00062 =P
.
] .[20" ,Renfrew ,
].[14

, .
.

24

CT .
, "
. ,
.

, " . ,

.
, .
" ,"1
" ,"2
, .
87 ,
.

" ].[14
" , ,
" ] .[3
,
.

:
) (1 .
, . "
. .
) (2 .
,
.

25

) (3 .
'
'.
) (4 " .
" ] ,[14
] .[14 Yoon
],[21
500 - CT ,
. " "
. 30% - , 8% -
, , 2% - .

] .[21 CT
"
.

) (.
)" (5" - .
.
) (6 .Computer-Aided Detection -
,
,
] .[22 ,
. ,Computer-Aided Detection , ,
.
,

26

 . ,
].[22
) (7 ,"Telecommunication" -
" , .
, ].[23
) (8 " "
. ].[24
, " ,

.


. ,
.
" , ,
.
.
: ,
].[25

27

:
: ,
,
.
" , CT
, "
".
.1 :
CT .2 . .3 .
" ,
.4 .gold standard
.
: , ,
) 2002 .(2003 60 ,CT
)( " , .
,
.
CT- " .1 : ) (
CT - .2 ,
CT -" ) '( ,
.3 , " ) , '(
" '.
" ,
" (1) : (2) ,
(3) , .
.

28

 : ,
CT ) 0.5 Kappa .(0.6
). (Kappa = 0.6
, :gold standard 64% 90%
.91%
, : 92%
.97%
.
.
.
: CT
. , "
, "
" .

29

:
Background: Errors in the interpretation of imaging examinations lead to errors in
diagnosis and treatment, and are most significant in patients presenting with acute
medical conditions.
In the current study, comparison was made between the interpretations given by
residents and specialists for abdominal CT conducted for the workup of acute
abdomen at the department of Radiology of Hadassah Ein Kerem hospital.
The study was part of the hospital's quality assurance program.
Purpose: 1- To evaluate interobserver variability between radiology residents and
specialists in the interpretation of abdominal computed tomography (CT) for the
diagnosis of acute abdomen at Hadassah University Hospital, Ein Kerem. 2- To
determine the percentage of agreement between the readers. 3- To determine the
sensitivity and specificity of the residents' interpretations compared to the specialists'
interpretation, serving as the "gold standard". 4- To compare interobserver variability
between residents and specialists as a function of the number of years in residency.
Materials and Methods: CT reports of 60 patients with clinically suspected acute
abdomen were collected prospectively. The CT's were interpreted by three
radiologists: (1) A resident who interpreted the CT during a night shift, (2) a specialist
(specialist A) who's interpretation was published as the final report of the
examination, (3) a second specialist (specialist B) who, interpreted the CT
retrospectively, (without knowledge of the resident and specialist As' interpretations).
The resident and the specialist B filled out a questionnaire to determine whether there
was: (1) a pathology which required an immediate surgical intervention, (2) a
pathology with no need for an immediate surgical intervention, (3) no evidence of
pathology. A third questionnaire was filled out according to specialist A's report.

30

Interobserver variability between the readers was determined by calculating Kappa

values. The sensitivity and specificity of the residents' interpretations were calculated
while the specialists' interpretations were considered as the gold standard.
Results: interobserver agreement between the residents and the specialists was
moderate for the diagnosis of acute abdomen in CT, with Kappa values of 0.5-0.6.
Kappa values for agreement between specialists A and B were similar. The sensitivity
and specificity of the residents' interpretation were 64%-90% and 91% respectively
when compared to the specialists' interpretation. The sensitivity and specificity of the
residents' interpretation were higher for cases with agreement between specialists.
Correlation between the radiologists' interpretations and the clinical decisions, as
found in the medical records, showed that most disagreements were minor and would
have not affected clinical management.
Conclusions: Although interobserver agreement was found to be moderate,
interobserver variability had only insignificant impact on clinical decisions.
Nevertheless, errors in interpretation can be minimized by adopting methods and
strategies such as training of observers, structured reporting, external quality control
of a sample of the examinations and in the future possibly computer-aided detection,
thus improving the quality of care.

31

 ':
\.

C.T .
\ :
\.
.___________ :
._____________ :
.________ :
:
:
.______________ :
..______________ :
:' \ ' .._____ :
.____________ :
.____________________ :
:

)/ x - (

.1 \ .
.2 .
.
:
.

:
.
.
.
.
.
.
.
.____________ :

32

:
:

.
.
.

.
).(Transition zone
.(Closed loop) .
.
: \ \ \ .
.

:Crohn's Disease
.
.
.
.____________ :
.

:
:Torsion of ovary
.
.____________ :
: PID
.
.
.__________ :

:
:
.
.
:
.Intimal flap
.
.

33

:
:
.________ :
. __________ :
.
.
:
.______________ :
.
.
.
.
.____________ :
:
.___________ :
.
.
'(Appendigitis) :
.___________ :
.

:
: \ \ .
: \ \ .
.
.____________ -

:
\ .
.
.
.

34

:
.
.
.
.
.___________________ :

._________________________ :

? \ . .____________ :
___________________________________________:
____________________________________________________________________
____________________________________________________________________
____________________________________________________________________

) : (
. .______________ :
._______ : .________ :
.
.
.

35

:
.1
.2
.3
.4
.5
.6
.7
.8
.9
.10
.11
.12
.13
.14
.15
.16
.17
.18
.19
.20

Leavitt, M., Medscape's response to the Institute of Medicine Report:

Crossing the quality chasm: a new health system for the 21st century.
MedGenMed, 2001. 3(2): p. 2.
Homsted, L., Institute of Medicine report: to err is human: building a safer
health care system. Fla Nurse, 2000. 48(1): p. 6.
Urban, B.A. and E.K. Fishman, Tailored helical CT evaluation of acute
abdomen. Radiographics, 2000. 20(3): p. 725-49.
Gore, R.M., et al., Helical CT in the evaluation of the acute abdomen. AJR
Am J Roentgenol, 2000. 174(4): p. 901-13.
Siewert, B. and V. Raptopoulos, CT of the acute abdomen: findings and
impact on diagnosis and treatment. AJR Am J Roentgenol, 1994. 163(6): p.
1317-24.
Siewert, B., et al., Impact of CT on diagnosis and management of acute
abdomen in patients initially treated without surgery. AJR Am J Roentgenol,
1997. 168(1): p. 173-8.
Gruis, L., Epideiology - Chap 4: Assesing the Validity and Reliability of
Diagnosis and Screening tests. PP:63-80.
Leslie, A., A.J. Jones, and P.R. Goddard, The influence of clinical information
on the reporting of CT by radiologists. Br J Radiol, 2000. 73(874): p. 1052-5.
Wise, S.W., et al., Comparative assessment of CT and sonographic techniques
for appendiceal imaging. AJR Am J Roentgenol :(4)176 .2001 ,p. 933-41.
Albano, M.C., et al., Resident interpretation of emergency CT scans in the
evaluation of acute appendicitis. Acad Radiol, 2001. 8(9): p. 915-8.
Freedman, L.S., M.K. Parmar, and S.G. Baker, The design of observer
agreement studies with binary assessments. Stat Med, 1993. 12(2): p. 165-79.
Torbati, S.S. and D.A. Guss, Impact of helical computed tomography on the
outcomes of emergency department patients with suspected appendicitis. Acad
Emerg Med, 2003. 10(8): p. 823-9.
Espinosa, J.A. and T.W. Nolan, Reducing errors made by emergency
physicians in interpreting radiographs: longitudinal study. Bmj, 2000.
320(7237): p. 737-40.
Robinson, P.J., Radiology's Achilles' heel: error and variation in the
interpretation of the Rontgen image. Br J Radiol, 1997. 70(839): p. 1085-98.
Mindelzun, R.E. and R.B. Jeffrey, The acute abdomen: current CT imaging
techniques. Semin Ultrasound CT MR, 1999. 20(2): p. 63-7.
Rao, P.M., et al., Helical CT with only colonic contrast material for
diagnosing diverticulitis: prospective evaluation of 150 patients. AJR Am J
Roentgenol, 1998. 170(6): p. 1445-9.
Rao, P.M. and R.A. Novelline, Case 6: primary epiploic appendagitis.
Radiology, 1999. 210(1): p. 145-8.
Rao, P.M., CT of diverticulitis and alternative conditions. Semin Ultrasound
CT MR, 1999. 20(2): p. 86-93.
Rao, P.M., et al., Helical CT combined with contrast material administered
only through the colon for imaging of suspected appendicitis. AJR Am J
Roentgenol, 19 :(5)169 .97p. 1275-80.
Bechtold, R.E., et al., Interpretation of abdominal CT: analysis of errors and
their causes. J Comput Assist Tomogr, 1997. 21(5): p. 681-5.

36

.21
.22
.23
.24
.25

Yoon, L.S., et al., Evaluation of an emergency radiology quality assurance

program at a level I trauma center: abdominal and pelvic CT studies.
Radiology, 2002. 224(1): p. 42-6.
Astley, S.M. and F.J. Gilbert, Computer-aided detection in mammography.
Clin Radiol, 2004. 59(5): p. 390-9.
Vari, S.G., et al., Telepathology and imaging spectroscopy as a new modality
in histopathology. Stud Health Technol Inform, 1999. 68: p. 211-6.
Selman, A.E., et al., Quality assurance of second opinion pathology in
gynecologic oncology. Obstet Gynecol, 1999. 94(2): p. 302-6.
Dolmans, D.H ,.H.A. Wolfhagen, and A.J. Scherpbier, From quality
assurance to total quality management: how can quality assurance result in
continuous improvement in health professions education? Educ Health
(Abingdon), 2003. 16(2): p. 210-7.

37