Journal Pre-proof

SATISFACTION OF SEARCH IN PERIAPICAL RADIOGRAPH INTERPRETATION

Johnny D. Huynh, D.D.S, S. Craig Rhodes, D.M.D., M.S.D, John F. Hatton, D.M.D,
John A. Khademi, D.D.S., M.S.

PII: S0099-2399(20)30852-9
DOI: https://doi.org/10.1016/j.joen.2020.11.001
Reference: JOEN 4721

To appear in: Journal of Endodontics

Received Date: 28 August 2020

Accepted Date: 1 November 2020

Please cite this article as: Huynh JD, Rhodes SC, Hatton JF, Khademi JA, SATISFACTION OF
SEARCH IN PERIAPICAL RADIOGRAPH INTERPRETATION, Journal of Endodontics (2020), doi:
https://doi.org/10.1016/j.joen.2020.11.001.

This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition
of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of
record. This version will undergo additional copyediting, typesetting and review before it is published
in its final form, but we are providing this version to give early visibility of the article. Please note that,
during the production process, errors may be discovered which could affect the content, and all legal
disclaimers that apply to the journal pertain.

Copyright © 2020 Published by Elsevier Inc. on behalf of American Association of Endodontists.

 SATISFACTION OF SEARCH IN PERIAPICAL RADIOGRAPH INTERPRETATION

Johnny D. Huynh, D.D.S.

S. Craig Rhodes, D.M.D., M.S.D.
John F. Hatton, D.M.D.
John A. Khademi, D.D.S., M.S.

of
ro
-p
re
lP
na

From the Department of Endodontics, Center for Advanced Dental Education, Saint Louis
University, Saint Louis, MO
ur

Address requests for reprints to Dr. S. Craig Rhodes, Department of Endodontics, Center for
Advanced Dental Education, Saint Louis University, Saint Louis, MO, 63104-1122. E-mail
Jo

address: craig.rhodes@health.slu.edu
 of
ro
SATISFACTION OF SEARCH IN PERIAPICAL RADIOGRAPH INTERPRETATION

-p
re
lP
na
ur
Jo
 Abstract

Introduction

Several studies in radiology and medicine have evaluated the satisfaction of

search error effect in chest radiography 1, 2, abdominal radiography 3, osteoradiology 4,

and patients with multiple trauma 5. No research to date has been published evaluating

the possible existence of the satisfaction of search error phenomenon made during

dental periapical radiograph interpretations.

of
Purpose

ro
The purpose of the present pilot study was to determine if there exists a
-p
satisfaction of search error effect when dental clinicians interpret periapical radiographs.
re
The null hypothesis is that the detection accuracy will be the same or will improve for
lP

the detection of native lesions in the presence of an added abnormality. The alternative
na

hypothesis is that there will be a decrease in detection accuracy for native lesions in the

presence of an added abnormality.

ur

Materials and Methods

Jo

Six images were selected to be part of the present experiment. One of the six

images served as the positive control and another image served as the negative control.

Four images, each including a single subtle carious lesion, were selected to represent

the experimental images. The single subtle carious lesion present within the four

experimental radiographs served as the native pathology, and an abnormality such as a

periapical radiolucency, resorption, inadequate non-ideal root canal obturation material,

or recurrent carious lesion was artificially inserted into the image as the added

pathology. Thus, the second set of images consisted of the same four images

1
containing the native pathology including an added pathology that was inserted into the

image using Adobe Photoshop CS6 (Adobe, Inc., San Jose, CA).

Purposive sampling was obtained from 16 examiners that included residents

from endodontics and periodontics, as well as alumni and faculty from the Saint Louis

University Center for Advanced Dental Education. Each observer participated as a

subject during two, time-separated sessions. Each session was separated by a

minimum period of three months duration, in order to prevent memory bias. Before

of
starting each interpretation session, the participants were given verbal instructions.

ro
Subjects were instructed to provide a location (by tooth number), identify and rate the
-p
presence of all suspected pathology using a Likert scale of 1-5: (1: Definitely normal, 2:
re
Probably normal, 3: Possibly abnormal, 4: Probably abnormal, 5: Definitely abnormal).
lP

In the second session, the radiographs that were initially presented containing only the
na

native lesion were presented again with the added abnormality, and vice versa. The

observer’s reports and confidence ratings were recorded and analyzed. Ratings of 3-5
ur

were considered as being positive for the presence of pathology.

Jo

Results

A true satisfaction of search (SOS) error occurs when the presence of the native

lesion is reported correctly without an added abnormality, but is not reported (missed) in

the presence of an added abnormality. In our study, a true SOS error occurred in 13 of

the 64 interpretation sets (20.31%). There was a total of 64 expected native lesions

present within the 4 native images viewed by 16 observers. In the 4 four added images,

there was a total of 64 expected added findings. In the images with containing only

native lesions, the observers reported 30 of the 64 expected native lesions. In the

2
images containing an artificially added abnormality, the observers reported 58 of the 64

expected added abnormalities and 25 of the 64 expected native lesions. Observers

reported fewer native lesions in the presence of an added abnormality.

Conclusions

The current investigation demonstrated the existence of the satisfaction of search

effect during periapical radiographic interpretations. In 20.31% of interpretations, a true

SOS error occurred. This study is clinically relevant because it can help clinicians in

of
reducing false-negative errors made during radiographic interpretation, thus preventing

ro
misdiagnosis.
-p
re
lP
na
ur
Jo

3
Introduction

Dental periapical radiographs typically include soft tissues, teeth and bone

structures all superimposed onto one another in a two-dimensional (2-D) image.

Interpreting such 2-D radiographic images accurately can be a difficult task that may be

negatively impacted by errors. Among the various types of radiographic imaging

modalities used in dentistry, periapical radiographs remain the most commonly utilized

images for endodontic evaluation purposes. Depending on the film or sensor size, these

of
individual images will typically contain the teeth within a single sextant, along with the

ro
surrounding anatomy. When both clinical testing and radiographic findings are
-p
ambiguous, or perhaps even when in conflict, endodontists often will make treatment
re
decisions based mainly on what is viewed within the radiographic image. This practice
lP

can often lead to errors and can have negative consequences, when such clinicians
na

assume there to be an infallible objective science underlying the radiographic

perception and interpretation processes employed by humans. In reality, there are

ur

several subjective factors that could play a part in negatively influencing and
Jo

skewing cognition, leading to errors culminating in a misdiagnosis. Clinicians must first

understand the fundamental processes inherent to image perception to be able to

accurately interpret radiographic images, and to prevent such diagnostic errors.

Errors in perception are problematic and are often manifested in the form of a

missed diagnosis. A subset of under-reading errors (false-negative responses) in

radiology has been attributed to what has been called the "satisfaction of search” (SOS)

phenomenon, which occurs when lesions lie undetected because the search process

has been prematurely terminated after the detection of an initial lesion 6.

4
 Tuddenham and Calvert first demonstrated that observers missed pathological

shadows even when they had a methodological approach to scanning an image.7

Although their study was limited by using only four radiologist observers, it served to

highlight that reading radiographs was not as simple and orderly as had been previously

believed. Christensen et al found that the likelihood for a positive report actually being

false increases with search time. They demonstrated that as more obvious lesions are

detected and reported, the unreported additional lesions and noise will increasingly

of
overlap.8

ro
Several studies in radiology and medicine have evaluated the SOS error effect:
-p
in chest radiography 1, 2, abdominal radiography 3, osteoradiology 4, and for patients with
re
multiple trauma 5. No research exists to date evaluating the SOS effect during dental
lP

periapical radiograph interpretations. Medical research studies have shown a clear

na

difference in the accuracy of interpretation of radiographs having a single abnormality,

as compared with those images having multiple abnormalities 9, 10, 11. Although the
ur

relative size of a periapical radiograph may be small, as compared to a chest

Jo

radiograph or a computed tomography scan, the importance of perception and need for

accurate interpretation remain the same. Such satisfaction of search error studies have

also been used as an explanation for false-negative findings, or the under-interpretation

(a source of diminished accuracy) of various medical radiographs. However, to date, it

has not been demonstrated whether this phenomenon can also occur during the

interpretation process of dental periapical radiographs.

The present investigation further explores the satisfaction of search error

phenomenon and aims to assess if it also impacts the interpretation of dental periapical

5
radiographs. The null hypothesis is that the detection accuracy will be the same or will

improve for the detection of native lesions in the presence of an added. The alternative

hypothesis is that there will be a decrease in detection accuracy for native lesions in the

presence of an added abnormality.

Materials and Methods

This experimental study was designed to assess the SOS error phenomenon in

of
dental radiography, specifically as encountered in periapical imaging diagnostics. SOS

ro
errors would be demonstrable in periapical radiograph interpretations if the false-

-p
negative error rates for native lesions were increased when an added abnormality was
re
inserted into the periapical image. In other words, if the observers identify the native

pathology (the independent variable, e.g. caries) in the absence of the added pathology
lP

(Independent Variable, e.g. periapical radiolucency, resorption, inadequate obturation,

na

or recurrent carious lesion) but miss the native pathology in the presence of the added
ur

pathology, we can attribute the false-negative error to the satisfaction of search

Jo

phenomenon. The score/confidence rating would represent the dependent variable.

Digital periapical images were obtained, in conjunction with endodontic

diagnostic or therapeutic procedures, from patients treated in the endodontics clinic at

the Saint Louis University Center for Advanced Dental Education. Institutional Review

Board (IRB) approval was obtained (IRB protocol #30686). A Delphi panel unanimously

selected images to include a variety of radiographic findings, including normal teeth,

teeth having direct or indirect restorations, teeth with coronal caries, teeth with

periapical radiolucent areas, and those having prior endodontic treatment.

6
 Six such images were selected for the experiment. Each periapical image

contains at least 3 teeth (examples: the first molar, the second molar, and at least the

second premolar). One of the six images served as the positive control and contained

an obvious, large carious lesion and an apical radiolucency from a single tooth. Another

image containing clinically verified normal anatomical structures served as the negative

control. Four images, each including a single subtle carious lesion, were selected as

the experimental images. The single subtle carious lesion within the four radiographs

of
served as the native pathology and an abnormality such as a periapical radiolucency,

ro
resorption, non-ideal root canal obturation material, or a recurrent carious lesion was
-p
the added pathology. Thus, the second set of images consisted of the same four
re
images containing a native pathology, in addition to an added pathology created using
lP

Adobe Photoshop CS6 (Adobe Inc., San Jose, CA). The experimental manipulation was
na

the various pathologies (both the added and native) and the observer’s responses were

analyzed.
ur

Purposive sampling was obtained from 16 examiners that included endodontics

Jo

and periodontics postgraduate residents, alumni, and faculty at the Saint Louis

University Center for Advanced Dental Education. Each observer participated as a

subject in two separate and distinct experimental sessions. Each session was

separated by a minimum of three months’ duration, to prevent memory bias from

observers recalling the lesion locations. The observers were told that the purpose of the

study was to test various image processing filter’s effect on diagnostic performance to

mitigate demand characteristics. The randomized images were displayed on an LCD

monitor (1920x1080 resolution) (Hewlett-Packard, Palo Alto, CA). Participants viewed

7
the periapical images within a Portable Document Format (PDF) (Adobe Inc., San Jose,

CA) presentation, and their responses were recorded. Before starting the interpretation

session, the participants were given verbal instructions. No time limit was placed on any

examiner or image presentation duration. They participants were informed that there

was a total of 6 cases and their task was to note all suspected pathologies contained

within each image, as they would for an actual patient examination. Examiners were

asked to “Please report any radiographic findings that are consistent with or suggestive

of
of dental pathology, disease processes or anomalies for which you would recommend

ro
treatment, active surveillance or otherwise make note”. Subjects were instructed to
-p
provide a location (by tooth number), and to identify and rate all suspected pathology on
re
a Likert scale of 1-5 (1: Definitely normal, 2: Probably normal, 3: Possibly abnormal, 4:
lP

Probably abnormal, 5: Definitely abnormal). In the second session, the radiographs

na

initially presented without the added abnormality were presented with the added

abnormality, and vice versa. The observer’s reports and confidence ratings were
ur

recorded and analyzed. Ratings of 3-5 were considered as being positive for the
Jo

presence of pathology.

Results

A total of 291 findings (2-5 normality rating) were reported. For images with

solely native lesions, a total of 144 findings were reported. When an added abnormality

was added to the same images, a total of 147 findings were reported. When teeth rated

a 1 or 2 were combined as probably normal, there were a total of 252 findings (3-5

normality rating) reported, 115 findings reported from images with only native lesions

and 137 findings were reported when an added abnormality was included.

8
 There was a total of 64 expected native lesions in the 4 native images viewed by

16 observers. For the 4 added images, there was a total of 64 expected added findings.

In the images with only native lesions, the observers reported 30 of the 64 expected

native lesions. In the images with an added abnormality, the observers reported 58 of

the 64 expected added abnormalities and 25 of the 64 expected native lesions.

Observers recalled fewer native lesions in the presence of an added abnormality. A true

SOS error occurs when a native lesion was reported without an added abnormality but

of
was missed in the presence of an added abnormality 6. In our study, a true SOS error

ro
occurred in 13 of the 64 interpretation sets (20.31%).
-p
Case #1 did not demonstrate a difference in reports between the two images.
re
The corresponding dot plot including false-positives (FPs) reveal similar amounts of
lP

reports between interpretative sets of images (Figure 1). Each dot represents a finding
na

reported by the observer and the dot size is relative to the confidence rating, with higher

ratings as larger dots. Forty-one findings were reported in both the native and added
ur

image. True SOS errors were made by observer E4 and P3.

Jo

Case #2 is depicted in Figure 2. Observers reported more abnormalities when a

metal crown was present on tooth #14 than when it was replaced with a deep

restoration. There were 13 (2-5 confidence rating) reports when there was a crown

present, as compared to 4 reports when there was a deep restoration. Nine observers

reported a periapical radiolucency for tooth #14 when a crown was present, as

compared to 5 reports of a periapical radiolucency being present with a deep

restoration. One true SOS error was made by observer E8.

9
 In Case #3, observers called less FPs when an obvious periapical radiolucency

was added for tooth #20 (Figure 3). Seven FPs (3-5 confidence rating) were reported for

tooth #18 and 11 FPs for tooth #19 were reported in the native image. In the added

image, 4 FPs on for tooth #18 and 3 FPs for tooth #19 were reported. Four observers

reported the native lesion for tooth #18 in the presence of an added abnormality for

tooth #20 whereas 12 observers reported the native lesion for tooth #18 without the

added abnormality. Case #3 contained the highest number of true SOS errors, 8 out of

of
13, by observers E1, E6, E8, E10, P1, P2, P3 and P4.

ro
Case #4 also had less abnormalities reported when a large resorption was added
-p
(Figure 4). Thirty-five abnormalities were reported in the native image, versus 22 that
re
were reported when an obvious resorption was added for tooth #25. True SOS errors
lP

were made by observer P2 and P5.

na

Discussion
ur

As dentists, perceiving and interpreting radiographs represent

Jo

fundamentally important aspects of our clinical examination process. We depend on

these images to help us to appreciate whether there may pathology present, and to

evaluate the outcome of our treatment. This appreciation, as gained through the visual

interpretation process, has been demonstrated to be very complex and is often

accompanied by the potential for making errors. Garland first reported on errors in

medical image interpretation 12. There are some radiographic techniques that are

particularly prone to such errors, with traditional chest x-rays having a “miss rate” (false

negatives) of 20-50% and traditional mammography a miss rate of up to 30% 13,14.

10
Dental research has not yet been conducted to define the percentage rates for errors

made in the interpretation of radiographic images. However, medical radiologists have

studied them extensively and have become keenly aware of specific errors in perception

and cognition that are prone to being made by clinicians.

Errors in perception and cognition have been rigorously studied, to clarify potential

reasons for them during radiographic interpretation tasks. Tuddenham and Calvert

described that once an observer detects an abnormality, the “search for meaning” has

of
been satisfied, thus resulting in the premature termination of that search effort 7. Kundel

ro
et al categorized false-negative errors into three distinct categories: scanning errors,
-p
pattern-recognition errors, and decision-making errors as a part of what is known as the
re
Kundel-Nodine error classification system15. Search errors are ones in which the lesion
lP

is never fixated upon, nor reported. Recognition errors are ones in which lesions are
na

fixated upon, but the duration lies below the threshold of dwell time sufficient to be

considered abnormal. Decision errors occur when the lesion is fixated upon for a
ur

sufficiently long time, however the clinician actively dismisses or does not recognize it
Jo

as representing a true lesion. Christensen et al theorized that observers strategically

terminated search efforts because the likelihood for false-positive errors increases with

additional search time 8. Berbaum et al hypothesized that SOS errors may be due to a

perceptual set bias, that is a readiness to interpret images in a particular way because

certain features are indicative of a specific diagnosis 6.

Satisfaction of search error is defined as occurring when one abnormality is

missed in the presence of another abnormality but is detected in the absence of the

same abnormality. Berbaum et al developed a laboratory model to study the SOS effect

11
in medical radiography and have extensively documented the SOS phenomenon in

chest radiograph interpretation studies 6. The SOS phenomenon has been studied

across a spectrum of imaging modalities and clinical situations, including chest

radiography 1, 2, abdominal radiography 3, osteoradiology 4, and in patients having

experienced multiple traumas 5. The inherent differences between these imaging

modalities may contribute to the cause of SOS errors. Eye-tracking studies have helped

to demonstrate that the process underlying such SOS effects while clinicians review

of
images from plain-film contrast abdominal radiography is different from other types of

ro
radiologic examinations. Visual neglect of the non-contrasted regions as compared to
-p
contrasted regions was common, and was attributed to faulty scanning, using the
re
Kundel-Nodine error classification system 15. However, Berbaum et al’s investigation of
lP

SOS error made by clinician’s treating multi-trauma patients suggested otherwise 11. For
na

these patients, the observer clinicians almost always fixated on the native bone

fractures when an SOS error was being made, thus demonstrating that the observer
ur

had indeed scanned the region of interest. The authors concluded that the number of
Jo

SOS false-negative errors was directly proportional to the severity of the detected

fracture. These findings highlighted the complex nature of cognitive science and the

occurrence of satisfaction of search errors.

For the present study, we followed Berbaum et al’s chest radiography model to

investigate SOS errors in periapical radiography. In their study, chest radiographs were

used containing a native abnormality, and the images were modified to include an

added pulmonary nodule as a “distractor” lesion 6. The same process was followed with

periapical radiographs.

12
 The results from the present study suggest that SOS false-negative errors also

exist in periapical image interpretations. SOS false-negative errors were observed to

occur in 20.31% of the radiographic interpretations. It should be noted that the images

shown in both sessions were identical, except for the presence of a lesion added to one

of the paired images. Therefore, it can be accurately inferred that any observed

differences between the evaluations are most likely to be due to the presence of the

added abnormality. The present findings agree with Berbaum et al, and demonstrate a

of
decrease in the detection rate of native abnormalities, as reported in the presence of an

ro
added abnormality, indicating a true SOS effect 6.
-p
Within the present study, the authors added a short endodontic obturation length
re
in case #1 as an added abnormality, to draw the attention of observers to a subtle
lP

periapical radiolucency on the same tooth. This decision added confusion because the
na

native image then had two native lesions, which could lead some observers to call both

the native carious lesion and the subtle periapical radiolucency. Future studies should
ur

be designed to have the subtle periapical radiolucency made less obvious. Case #2
Jo

included tooth #14 with a full coverage metal crown and a subtle periapical radiolucency

present on the mesial root. Observers reported more abnormalities as being present for

tooth #14 when a metal crown was present, as opposed to when it was replaced with a

deep restoration. In case #3, observers called less FPs when an obvious periapical

radiolucency was added for tooth #20. Case #4 also had less abnormalities reported

when an obvious lesion was added. The reduction in false-positives in case #3 and #4

could be considered as being like Berbaum et al’s plain-film abdominal contrast study 17.

Berbaum et al concluded that the reduction in FPs in such contrast studies is likely due

13
to faulty visual scanning. The large periapical radiolucency in case #3, and the obvious

resorption in case #4 may be considered as being like the radiographic contrast material

present in the Berbaum et al abdominal studies, wherein the observers focused their

attention on the more obvious, higher contrast abnormality and in doing so neglected

the surrounding regions having the subtler findings.

The present study has several potential limitations. The authors were careful to

select radiographs that contained a single, subtle native lesion, without other

of
abnormalities present, to avoid having other potential distractors. It was realized that

ro
lesions cannot be adequately removed from radiographs without the creation of
-p
distortion in the background. The most predictable method to simulate a lesion required
re
finding another lesion on a separate image and digitally cutting and pasting it into the
lP

image containing the native lesion.

na

Another limitation of this study is that observers were making a diagnosis based

from their interpretation of a single radiograph. Brynolf showed that increasing the
ur

number of radiographs can significantly improve a clinician’s diagnostic accuracy 18. In

Jo

addition, we tried to mitigate demand characteristics by falsifying the purpose of the

study in our prompt. Demand characteristics refer to a change in a participant’s

behavior to fit their interpretation of the experiment’s purpose. Research on demand

characteristics was pioneered by Orne 19. Demand characteristics cannot be eliminated

from experiments, but attempts can be made to mitigate their effects. One method is to

use a large sample size and a between-subjects experimental design, rather than a

within-subjects design.

14
 The goal of this pilot study was to investigate if SOS false-negative errors exist in

periapical radiograph interpretation tasks. Although the SOS error was assumed to be

present in this investigation, it is impossible to correlate the findings to previous work in

medicine and radiology without employing a more thorough statistical analysis and ROC

curve development.

of
Conclusion

ro
The current investigation has demonstrated the existence of the satisfaction of
-p
search effect during periapical radiographic interpretations. In 20.31% of the image
re
interpretations, the native lesion was not reported in the presence of an added
lP

abnormality but was reported in the absence of the added abnormality. This study is

clinically relevant because it highlights the importance for clinicians having cognitive and
na

perceptual awareness during radiographic interpretation. Research on the satisfaction

ur

of search effect in periapical radiographs can help clinicians reduce their false-negative
Jo

error rates, thus preventing misdiagnosis. Future investigations should also focus on the

causes of SOS errors pertaining to periapical radiographs and possible interventions to

reduce these errors.

15
 References

1. Berbaum KS, Franken EA, Dorfman DD et al. Role of faulty visual search in the
satisfaction of search effect in chest radiography. Acad Radiol. 1998;5:9–19.
2. Samuel S, Kundel HL, Nodine CF et al. Mechanism of satisfaction of search: eye
position recordings in the reading of chest radiographs. Radiology.
1995;194:895–902.
3. Franken EA, Berbaum KS, Lu CH et al. Satisfaction of search in the detection of
plain-film abnormalities in abdominal contrast studies. Invest Radiol.
1994;29:403–409.

of
4. Ashman CJ, Yu JS, Wolfman D. Satisfaction of search in osteoradiology. AJR Am
J Roentgenol. 2000;175:541–544.

ro
5. Berbaum KS, El-Khoury GY, Franken EA et al. Missed fractures resulting from
satisfaction of search effect. Emerg Radiol. 1994;1:242–249.
-p
6. Berbaum KS, Franken EA, Dorfman DD et al. Satisfaction of search in diagnostic
re
radiology. Invest Radiol. 1990;25:133–140.
7. Tuddenham WJ, Calvert WP. Visual search patterns in roentgen diagnosis.
lP

Radiology. 1961;76:255–256.
8. Christensen EE, Murry RC, Holland K et al. The effect of search time on
na

perception. Radiology. 1981;138:361–365.

9. Berbaum KS, Franken EA, Dorfman DD et al. Satisfaction of search in diagnostic
radiology. Invest Radiol. 1990;25:133–140.
ur

10. Berbaum KS, Franken EA, Dorfman DD et al. Time course of satisfaction of
search. Invest Radiol. 1991;26:640–648.
Jo

11. Berbaum KS, El-Khoury GY, Ohashi K et al. Satisfaction of search in multitrauma
patients: severity of detected fractures. Acad Radiol. 2007;14:711–722.
12. Garland LH. Studies on the accuracy of diagnostic procedures. Am J Roentgenol
Radium Ther Nucl Med. 1959;82:25–38.
13. Forrest JV, Friedman PJ. Radiologic errors in patients with lung cancer. West J
Med. 1981;134:485–490.
14. Yankaskas BC, Schell MJ, Bird RE et al. Reassessment of breast cancers
missed during routine screening mammography: a community-based study. AJR
Am J Roentgenol. 2001;177:535–541.
15. Kundel HL, Nodine CF, Carmody D. Visual scanning, pattern recognition and
decision-making in pulmonary nodule detection. Invest Radiol. 1978;13:175–181.
16.Christensen EE, Murry RC, Holland K et al. The effect of search time on
perception. Radiology. 1981;138:361–365.

16
17.Berbaum KS, Franken EA, Dorfman DD et al. Cause of satisfaction of search
effects in contrast studies of the abdomen. Acad Radiol. 1996;3:815–826.
18.Brynolf I. Roentgenologic periapical diagnosis. II. One, two or more
roentgenograms? Sven Tandlak Tidskr. 1970;63:345–350.
19.Orne MT. Demand Characteristics and the Concept of Quasi-Controls1. In:
Artifacts in behavioral research. Oxford University Press; 2009:110–137.

of
ro
-p
re
lP
na
ur
Jo

17
Legend

Fig 1 - Case #1 with Dot Plots

Fig 2 - Case #2 with Dot Plots

Fig 3 - Case #3 with Dot Plots

Fig 4 - Case #4 with Dot Plots

of
ro
-p
re
lP
na
ur
Jo

18
 of
ro
-p
Fig. 1 – Case #1 with Dot Plots
re
lP
na
ur
Jo

Fig. 2 – Case #2 with Dot Plots

19
 of
ro
-p
re
Fig. 3 – Case #3 with Dot Plots
lP
na
ur
Jo

Fig. 4 – Case #4 with Dot Plots

20