Professional Documents
Culture Documents
Satisfaction of Search in Periapical Radiograph Interpretation
Satisfaction of Search in Periapical Radiograph Interpretation
Johnny D. Huynh, D.D.S, S. Craig Rhodes, D.M.D., M.S.D, John F. Hatton, D.M.D,
John A. Khademi, D.D.S., M.S.
PII: S0099-2399(20)30852-9
DOI: https://doi.org/10.1016/j.joen.2020.11.001
Reference: JOEN 4721
Please cite this article as: Huynh JD, Rhodes SC, Hatton JF, Khademi JA, SATISFACTION OF
SEARCH IN PERIAPICAL RADIOGRAPH INTERPRETATION, Journal of Endodontics (2020), doi:
https://doi.org/10.1016/j.joen.2020.11.001.
This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition
of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of
record. This version will undergo additional copyediting, typesetting and review before it is published
in its final form, but we are providing this version to give early visibility of the article. Please note that,
during the production process, errors may be discovered which could affect the content, and all legal
disclaimers that apply to the journal pertain.
of
ro
-p
re
lP
na
From the Department of Endodontics, Center for Advanced Dental Education, Saint Louis
University, Saint Louis, MO
ur
Address requests for reprints to Dr. S. Craig Rhodes, Department of Endodontics, Center for
Advanced Dental Education, Saint Louis University, Saint Louis, MO, 63104-1122. E-mail
Jo
address: craig.rhodes@health.slu.edu
of
ro
SATISFACTION OF SEARCH IN PERIAPICAL RADIOGRAPH INTERPRETATION
-p
re
lP
na
ur
Jo
Abstract
Introduction
and patients with multiple trauma 5. No research to date has been published evaluating
the possible existence of the satisfaction of search error phenomenon made during
of
Purpose
ro
The purpose of the present pilot study was to determine if there exists a
-p
satisfaction of search error effect when dental clinicians interpret periapical radiographs.
re
The null hypothesis is that the detection accuracy will be the same or will improve for
lP
the detection of native lesions in the presence of an added abnormality. The alternative
na
hypothesis is that there will be a decrease in detection accuracy for native lesions in the
Six images were selected to be part of the present experiment. One of the six
images served as the positive control and another image served as the negative control.
Four images, each including a single subtle carious lesion, were selected to represent
the experimental images. The single subtle carious lesion present within the four
or recurrent carious lesion was artificially inserted into the image as the added
pathology. Thus, the second set of images consisted of the same four images
1
containing the native pathology including an added pathology that was inserted into the
image using Adobe Photoshop CS6 (Adobe, Inc., San Jose, CA).
from endodontics and periodontics, as well as alumni and faculty from the Saint Louis
minimum period of three months duration, in order to prevent memory bias. Before
of
starting each interpretation session, the participants were given verbal instructions.
ro
Subjects were instructed to provide a location (by tooth number), identify and rate the
-p
presence of all suspected pathology using a Likert scale of 1-5: (1: Definitely normal, 2:
re
Probably normal, 3: Possibly abnormal, 4: Probably abnormal, 5: Definitely abnormal).
lP
In the second session, the radiographs that were initially presented containing only the
na
native lesion were presented again with the added abnormality, and vice versa. The
observer’s reports and confidence ratings were recorded and analyzed. Ratings of 3-5
ur
Results
A true satisfaction of search (SOS) error occurs when the presence of the native
lesion is reported correctly without an added abnormality, but is not reported (missed) in
the presence of an added abnormality. In our study, a true SOS error occurred in 13 of
the 64 interpretation sets (20.31%). There was a total of 64 expected native lesions
present within the 4 native images viewed by 16 observers. In the 4 four added images,
there was a total of 64 expected added findings. In the images with containing only
native lesions, the observers reported 30 of the 64 expected native lesions. In the
2
images containing an artificially added abnormality, the observers reported 58 of the 64
Conclusions
SOS error occurred. This study is clinically relevant because it can help clinicians in
of
reducing false-negative errors made during radiographic interpretation, thus preventing
ro
misdiagnosis.
-p
re
lP
na
ur
Jo
3
Introduction
Dental periapical radiographs typically include soft tissues, teeth and bone
Interpreting such 2-D radiographic images accurately can be a difficult task that may be
modalities used in dentistry, periapical radiographs remain the most commonly utilized
images for endodontic evaluation purposes. Depending on the film or sensor size, these
of
individual images will typically contain the teeth within a single sextant, along with the
ro
surrounding anatomy. When both clinical testing and radiographic findings are
-p
ambiguous, or perhaps even when in conflict, endodontists often will make treatment
re
decisions based mainly on what is viewed within the radiographic image. This practice
lP
can often lead to errors and can have negative consequences, when such clinicians
na
several subjective factors that could play a part in negatively influencing and
Jo
Errors in perception are problematic and are often manifested in the form of a
radiology has been attributed to what has been called the "satisfaction of search” (SOS)
phenomenon, which occurs when lesions lie undetected because the search process
4
Tuddenham and Calvert first demonstrated that observers missed pathological
Although their study was limited by using only four radiologist observers, it served to
highlight that reading radiographs was not as simple and orderly as had been previously
believed. Christensen et al found that the likelihood for a positive report actually being
false increases with search time. They demonstrated that as more obvious lesions are
detected and reported, the unreported additional lesions and noise will increasingly
of
overlap.8
ro
Several studies in radiology and medicine have evaluated the SOS error effect:
-p
in chest radiography 1, 2, abdominal radiography 3, osteoradiology 4, and for patients with
re
multiple trauma 5. No research exists to date evaluating the SOS effect during dental
lP
as compared with those images having multiple abnormalities 9, 10, 11. Although the
ur
radiograph or a computed tomography scan, the importance of perception and need for
accurate interpretation remain the same. Such satisfaction of search error studies have
has not been demonstrated whether this phenomenon can also occur during the
phenomenon and aims to assess if it also impacts the interpretation of dental periapical
5
radiographs. The null hypothesis is that the detection accuracy will be the same or will
improve for the detection of native lesions in the presence of an added. The alternative
hypothesis is that there will be a decrease in detection accuracy for native lesions in the
This experimental study was designed to assess the SOS error phenomenon in
of
dental radiography, specifically as encountered in periapical imaging diagnostics. SOS
ro
errors would be demonstrable in periapical radiograph interpretations if the false-
-p
negative error rates for native lesions were increased when an added abnormality was
re
inserted into the periapical image. In other words, if the observers identify the native
pathology (the independent variable, e.g. caries) in the absence of the added pathology
lP
or recurrent carious lesion) but miss the native pathology in the presence of the added
ur
the Saint Louis University Center for Advanced Dental Education. Institutional Review
Board (IRB) approval was obtained (IRB protocol #30686). A Delphi panel unanimously
teeth having direct or indirect restorations, teeth with coronal caries, teeth with
6
Six such images were selected for the experiment. Each periapical image
contains at least 3 teeth (examples: the first molar, the second molar, and at least the
second premolar). One of the six images served as the positive control and contained
an obvious, large carious lesion and an apical radiolucency from a single tooth. Another
image containing clinically verified normal anatomical structures served as the negative
control. Four images, each including a single subtle carious lesion, were selected as
the experimental images. The single subtle carious lesion within the four radiographs
of
served as the native pathology and an abnormality such as a periapical radiolucency,
ro
resorption, non-ideal root canal obturation material, or a recurrent carious lesion was
-p
the added pathology. Thus, the second set of images consisted of the same four
re
images containing a native pathology, in addition to an added pathology created using
lP
Adobe Photoshop CS6 (Adobe Inc., San Jose, CA). The experimental manipulation was
na
the various pathologies (both the added and native) and the observer’s responses were
analyzed.
ur
and periodontics postgraduate residents, alumni, and faculty at the Saint Louis
subject in two separate and distinct experimental sessions. Each session was
observers recalling the lesion locations. The observers were told that the purpose of the
study was to test various image processing filter’s effect on diagnostic performance to
7
the periapical images within a Portable Document Format (PDF) (Adobe Inc., San Jose,
CA) presentation, and their responses were recorded. Before starting the interpretation
session, the participants were given verbal instructions. No time limit was placed on any
examiner or image presentation duration. They participants were informed that there
was a total of 6 cases and their task was to note all suspected pathologies contained
within each image, as they would for an actual patient examination. Examiners were
asked to “Please report any radiographic findings that are consistent with or suggestive
of
of dental pathology, disease processes or anomalies for which you would recommend
ro
treatment, active surveillance or otherwise make note”. Subjects were instructed to
-p
provide a location (by tooth number), and to identify and rate all suspected pathology on
re
a Likert scale of 1-5 (1: Definitely normal, 2: Probably normal, 3: Possibly abnormal, 4:
lP
initially presented without the added abnormality were presented with the added
abnormality, and vice versa. The observer’s reports and confidence ratings were
ur
recorded and analyzed. Ratings of 3-5 were considered as being positive for the
Jo
presence of pathology.
Results
A total of 291 findings (2-5 normality rating) were reported. For images with
solely native lesions, a total of 144 findings were reported. When an added abnormality
was added to the same images, a total of 147 findings were reported. When teeth rated
a 1 or 2 were combined as probably normal, there were a total of 252 findings (3-5
normality rating) reported, 115 findings reported from images with only native lesions
and 137 findings were reported when an added abnormality was included.
8
There was a total of 64 expected native lesions in the 4 native images viewed by
16 observers. For the 4 added images, there was a total of 64 expected added findings.
In the images with only native lesions, the observers reported 30 of the 64 expected
native lesions. In the images with an added abnormality, the observers reported 58 of
Observers recalled fewer native lesions in the presence of an added abnormality. A true
SOS error occurs when a native lesion was reported without an added abnormality but
of
was missed in the presence of an added abnormality 6. In our study, a true SOS error
ro
occurred in 13 of the 64 interpretation sets (20.31%).
-p
Case #1 did not demonstrate a difference in reports between the two images.
re
The corresponding dot plot including false-positives (FPs) reveal similar amounts of
lP
reports between interpretative sets of images (Figure 1). Each dot represents a finding
na
reported by the observer and the dot size is relative to the confidence rating, with higher
ratings as larger dots. Forty-one findings were reported in both the native and added
ur
metal crown was present on tooth #14 than when it was replaced with a deep
restoration. There were 13 (2-5 confidence rating) reports when there was a crown
present, as compared to 4 reports when there was a deep restoration. Nine observers
reported a periapical radiolucency for tooth #14 when a crown was present, as
9
In Case #3, observers called less FPs when an obvious periapical radiolucency
was added for tooth #20 (Figure 3). Seven FPs (3-5 confidence rating) were reported for
tooth #18 and 11 FPs for tooth #19 were reported in the native image. In the added
image, 4 FPs on for tooth #18 and 3 FPs for tooth #19 were reported. Four observers
reported the native lesion for tooth #18 in the presence of an added abnormality for
tooth #20 whereas 12 observers reported the native lesion for tooth #18 without the
added abnormality. Case #3 contained the highest number of true SOS errors, 8 out of
of
13, by observers E1, E6, E8, E10, P1, P2, P3 and P4.
ro
Case #4 also had less abnormalities reported when a large resorption was added
-p
(Figure 4). Thirty-five abnormalities were reported in the native image, versus 22 that
re
were reported when an obvious resorption was added for tooth #25. True SOS errors
lP
Discussion
ur
these images to help us to appreciate whether there may pathology present, and to
evaluate the outcome of our treatment. This appreciation, as gained through the visual
accompanied by the potential for making errors. Garland first reported on errors in
medical image interpretation 12. There are some radiographic techniques that are
particularly prone to such errors, with traditional chest x-rays having a “miss rate” (false
10
Dental research has not yet been conducted to define the percentage rates for errors
studied them extensively and have become keenly aware of specific errors in perception
Errors in perception and cognition have been rigorously studied, to clarify potential
reasons for them during radiographic interpretation tasks. Tuddenham and Calvert
described that once an observer detects an abnormality, the “search for meaning” has
of
been satisfied, thus resulting in the premature termination of that search effort 7. Kundel
ro
et al categorized false-negative errors into three distinct categories: scanning errors,
-p
pattern-recognition errors, and decision-making errors as a part of what is known as the
re
Kundel-Nodine error classification system15. Search errors are ones in which the lesion
lP
is never fixated upon, nor reported. Recognition errors are ones in which lesions are
na
fixated upon, but the duration lies below the threshold of dwell time sufficient to be
considered abnormal. Decision errors occur when the lesion is fixated upon for a
ur
sufficiently long time, however the clinician actively dismisses or does not recognize it
Jo
terminated search efforts because the likelihood for false-positive errors increases with
additional search time 8. Berbaum et al hypothesized that SOS errors may be due to a
perceptual set bias, that is a readiness to interpret images in a particular way because
missed in the presence of another abnormality but is detected in the absence of the
same abnormality. Berbaum et al developed a laboratory model to study the SOS effect
11
in medical radiography and have extensively documented the SOS phenomenon in
chest radiograph interpretation studies 6. The SOS phenomenon has been studied
modalities may contribute to the cause of SOS errors. Eye-tracking studies have helped
to demonstrate that the process underlying such SOS effects while clinicians review
of
images from plain-film contrast abdominal radiography is different from other types of
ro
radiologic examinations. Visual neglect of the non-contrasted regions as compared to
-p
contrasted regions was common, and was attributed to faulty scanning, using the
re
Kundel-Nodine error classification system 15. However, Berbaum et al’s investigation of
lP
SOS error made by clinician’s treating multi-trauma patients suggested otherwise 11. For
na
these patients, the observer clinicians almost always fixated on the native bone
fractures when an SOS error was being made, thus demonstrating that the observer
ur
had indeed scanned the region of interest. The authors concluded that the number of
Jo
SOS false-negative errors was directly proportional to the severity of the detected
fracture. These findings highlighted the complex nature of cognitive science and the
For the present study, we followed Berbaum et al’s chest radiography model to
investigate SOS errors in periapical radiography. In their study, chest radiographs were
used containing a native abnormality, and the images were modified to include an
added pulmonary nodule as a “distractor” lesion 6. The same process was followed with
periapical radiographs.
12
The results from the present study suggest that SOS false-negative errors also
occur in 20.31% of the radiographic interpretations. It should be noted that the images
shown in both sessions were identical, except for the presence of a lesion added to one
of the paired images. Therefore, it can be accurately inferred that any observed
differences between the evaluations are most likely to be due to the presence of the
added abnormality. The present findings agree with Berbaum et al, and demonstrate a
of
decrease in the detection rate of native abnormalities, as reported in the presence of an
ro
added abnormality, indicating a true SOS effect 6.
-p
Within the present study, the authors added a short endodontic obturation length
re
in case #1 as an added abnormality, to draw the attention of observers to a subtle
lP
periapical radiolucency on the same tooth. This decision added confusion because the
na
native image then had two native lesions, which could lead some observers to call both
the native carious lesion and the subtle periapical radiolucency. Future studies should
ur
be designed to have the subtle periapical radiolucency made less obvious. Case #2
Jo
included tooth #14 with a full coverage metal crown and a subtle periapical radiolucency
present on the mesial root. Observers reported more abnormalities as being present for
tooth #14 when a metal crown was present, as opposed to when it was replaced with a
deep restoration. In case #3, observers called less FPs when an obvious periapical
radiolucency was added for tooth #20. Case #4 also had less abnormalities reported
when an obvious lesion was added. The reduction in false-positives in case #3 and #4
could be considered as being like Berbaum et al’s plain-film abdominal contrast study 17.
Berbaum et al concluded that the reduction in FPs in such contrast studies is likely due
13
to faulty visual scanning. The large periapical radiolucency in case #3, and the obvious
resorption in case #4 may be considered as being like the radiographic contrast material
present in the Berbaum et al abdominal studies, wherein the observers focused their
attention on the more obvious, higher contrast abnormality and in doing so neglected
The present study has several potential limitations. The authors were careful to
select radiographs that contained a single, subtle native lesion, without other
of
abnormalities present, to avoid having other potential distractors. It was realized that
ro
lesions cannot be adequately removed from radiographs without the creation of
-p
distortion in the background. The most predictable method to simulate a lesion required
re
finding another lesion on a separate image and digitally cutting and pasting it into the
lP
Another limitation of this study is that observers were making a diagnosis based
from their interpretation of a single radiograph. Brynolf showed that increasing the
ur
from experiments, but attempts can be made to mitigate their effects. One method is to
use a large sample size and a between-subjects experimental design, rather than a
within-subjects design.
14
The goal of this pilot study was to investigate if SOS false-negative errors exist in
periapical radiograph interpretation tasks. Although the SOS error was assumed to be
medicine and radiology without employing a more thorough statistical analysis and ROC
curve development.
of
Conclusion
ro
The current investigation has demonstrated the existence of the satisfaction of
-p
search effect during periapical radiographic interpretations. In 20.31% of the image
re
interpretations, the native lesion was not reported in the presence of an added
lP
abnormality but was reported in the absence of the added abnormality. This study is
clinically relevant because it highlights the importance for clinicians having cognitive and
na
of search effect in periapical radiographs can help clinicians reduce their false-negative
Jo
error rates, thus preventing misdiagnosis. Future investigations should also focus on the
15
References
1. Berbaum KS, Franken EA, Dorfman DD et al. Role of faulty visual search in the
satisfaction of search effect in chest radiography. Acad Radiol. 1998;5:9–19.
2. Samuel S, Kundel HL, Nodine CF et al. Mechanism of satisfaction of search: eye
position recordings in the reading of chest radiographs. Radiology.
1995;194:895–902.
3. Franken EA, Berbaum KS, Lu CH et al. Satisfaction of search in the detection of
plain-film abnormalities in abdominal contrast studies. Invest Radiol.
1994;29:403–409.
of
4. Ashman CJ, Yu JS, Wolfman D. Satisfaction of search in osteoradiology. AJR Am
J Roentgenol. 2000;175:541–544.
ro
5. Berbaum KS, El-Khoury GY, Franken EA et al. Missed fractures resulting from
satisfaction of search effect. Emerg Radiol. 1994;1:242–249.
-p
6. Berbaum KS, Franken EA, Dorfman DD et al. Satisfaction of search in diagnostic
re
radiology. Invest Radiol. 1990;25:133–140.
7. Tuddenham WJ, Calvert WP. Visual search patterns in roentgen diagnosis.
lP
Radiology. 1961;76:255–256.
8. Christensen EE, Murry RC, Holland K et al. The effect of search time on
na
10. Berbaum KS, Franken EA, Dorfman DD et al. Time course of satisfaction of
search. Invest Radiol. 1991;26:640–648.
Jo
11. Berbaum KS, El-Khoury GY, Ohashi K et al. Satisfaction of search in multitrauma
patients: severity of detected fractures. Acad Radiol. 2007;14:711–722.
12. Garland LH. Studies on the accuracy of diagnostic procedures. Am J Roentgenol
Radium Ther Nucl Med. 1959;82:25–38.
13. Forrest JV, Friedman PJ. Radiologic errors in patients with lung cancer. West J
Med. 1981;134:485–490.
14. Yankaskas BC, Schell MJ, Bird RE et al. Reassessment of breast cancers
missed during routine screening mammography: a community-based study. AJR
Am J Roentgenol. 2001;177:535–541.
15. Kundel HL, Nodine CF, Carmody D. Visual scanning, pattern recognition and
decision-making in pulmonary nodule detection. Invest Radiol. 1978;13:175–181.
16.Christensen EE, Murry RC, Holland K et al. The effect of search time on
perception. Radiology. 1981;138:361–365.
16
17.Berbaum KS, Franken EA, Dorfman DD et al. Cause of satisfaction of search
effects in contrast studies of the abdomen. Acad Radiol. 1996;3:815–826.
18.Brynolf I. Roentgenologic periapical diagnosis. II. One, two or more
roentgenograms? Sven Tandlak Tidskr. 1970;63:345–350.
19.Orne MT. Demand Characteristics and the Concept of Quasi-Controls1. In:
Artifacts in behavioral research. Oxford University Press; 2009:110–137.
of
ro
-p
re
lP
na
ur
Jo
17
Legend
of
ro
-p
re
lP
na
ur
Jo
18
of
ro
-p
Fig. 1 – Case #1 with Dot Plots
re
lP
na
ur
Jo
19
of
ro
-p
re
Fig. 3 – Case #3 with Dot Plots
lP
na
ur
Jo
20