ORIGINAL ARTICLES
Quality of dental care: development of standards
Howard Bailit, DMD, PhD
Meni Koslowsky, PhD
Joseph G rasso, MS, DDS, Farmington, Conn
Stanley Holzman, DDS
Robert Levine, MS, DDS, West Hartford, Conn
Paula Valluzzo, BS
Paula Atwood, BA, Farmington, Conn
Standards of dental care were developed by a
group of nine private general dental practitioners
and dental school faculty members. Five assessors
were trained to use the standards that were then
tested for reliability, variability, validity, and prac­
ticality on a sample of 47 patients. The results
showed high reliability for most items and moder­
ate variability and validity. The time necessary to
train assessors and evaluate individual patients
was reasonable. The potential use of this quality
assessment system is discussed in terms of contin­
uing education, peer review committees, and den­
tal program evaluation.
R ecen t passage o f legislation mandating the de­
velopm ent o f professional standards review or­
ganizations and the increasing involvem ent o f
insurance com panies in monitoring the quality o f
care make it apparent that providers o f dental
care eventually w ill be involved in som e form o f
organized review system . A major challenge fac­
ing organized dentistry is to ensure that the sy s­
tem is acceptable to both the dentists and the
public.
M any factors might affect the profession’s
support o f a particular evaluation system . First,
the dental profession must have a significant role
in developing the system . Second, the m ethod
o f evaluation m ust be objective. Third, the cost
842 ■ JADA, Vol. 89, October 1974
o f evaluation m ust be reasonable in terms o f time
and m oney. Fourth, the basic dentist-patient
relationship must be kept intact.
With regard to the first factor, the A m erican
D ental A ssociation and state associations have
influenced, to som e degree, the evaluation pro­
cedures o f third parties. In general, how ever,
m ost insured programs function independently
o f organized dentistry. With the expected in­
crease in prepaid dental care, the dental profes­
sion might consider the formation o f founda­
tions that contract with third parties to monitor
the cost and quality o f dental care in a given state
or area. A precedent for this type o f organiza­
tion has already been established in m edicine.1
A t the national level, dentists m ust have som e
role in determining P SR O policy; this organiza­
tion will probably be responsible for evaluating
the quality o f dental services purchased with fed­
eral and state funds.
In terms o f objectivity, one o f the bases for an
evaluation system m ust be reliable criteria o f
care; that is, similar judgm ents m ust be made by
tw o or more assessors w ho independently evalu­
ate the sam e patient. F or reliability to be
achieved, written criteria m ust be tested under
controlled conditions. Som e progress has been
made in this area, notably the work o f Abram-
ow itz,2 Friedm an,3 Schonfeld and co-w orkers,4
S oricelli,5 and C o n s,6 although much remains to
be done.
The cost m ust also be reasonable. M ethods o f
evaluation m ust not rely solely on the use o f den­
tists, the m ost expensive form o f dental man­
power. Through the u se o f com puters and non­
professional raters, the review system can min­
im ize the tim e practicing dentists must spend
away from direct patient care.
Finally, although the evaluation system may
have som e effect on dentist-patient relationships,
the mutual respect and trust that exist betw een
m ost dentists and patients must be maintained.
 T his brief review is a general background to
the present paper on an experim ental quality
review system for general dentistry, developed
jointly by a group o f com m unity practitioners
and dental school faculty, the latter serving
mainly as technical advisers. T his paper d e­
scribes the process o f developm ent and the test­
ing o f the criteria. E xcept for exam ples, the ac­
tual criteria are not presented.
■ Fram ew ork fo r evaluating quality: F or a
com m on fram e o f reference, it is necessary to
discuss briefly the concept o f quality and the
various dim ensions o f quality that can be eval­
uated. A com m only accepted definition o f quality
is the one o f L ee and J on es7: “ G ood medical
care is the kind o f medicine practiced and taught
by the recognized leaders o f the m edical profes­
sion at a given tim e or period o f social, cultural,
and professional developm ent in a com m unity or
population group.”
Several aspects o f this definition need em ­
phasis. First, it is clear that the definition o f qual­
ity depends on the professional leaders in a given
com m unity, and although there will be general
agreem ent on som e standards in all com m unities,
individual differences will exist. T h ese regional
differences should be incorporated into any na­
tional standards.
A second point is that standards are not con­
stant over tim e. Even within a single com m unity,
standards o f care will change as n ew biologic or
technical discoveries are made in the diagnosis
and treatm ent o f disease. T his is also true for
changes in social and cultural values that often
define what society and the profession consider
abnormal conditions requiring treatment.
W ith this definition in mind, there are several
general conceptual m odels for the m easurem ent
o f the quality o f care, including th ose o f F ried­
m an,3 R oem er,8 and S chon feld.9 A ll have many
points in com m on, but the m odel o f D onabedian10
has received w ide acceptance and will be used in
this study.
Donabedian proposed that the quality o f care
can be evaluated from essentially three different
but related aspects: the structure, the p rocess,
and the outcom e o f care. T he outcom e o f receiv­
ing dental service is usually m easured by the
D M F , periodontal, and other morbidity indexes.
T he outcom e o f care is often considered the m ost
important m easure o f quality, but sin ce many
factors other than dental care affect the dental
health o f the patient, outcom e m easures have
som e lim itations. F or exam ple, if the patient is
not conscientious about oral hygiene, the best
dentistry may deteriorate.
T h e second dim ension o f quality is the process
o f care. Regardless o f the outcom e o f care, did
the dentist plan the proper treatment for the pa­
tient and w as the treatment adequately carried
out? Criteria must be available for com parison to
make judgm ents on the process o f care.
T h e structural dim ension o f care focu ses on
the setting in which care takes place. A sse ss­
ment can be made o f such things as physical fa­
cilities and equipm ent, qualifications o f pro­
viders, and administration o f the p ractice.11 It
is assum ed that if the structure is adequate, then
the process o f care is likely to be acceptable.
There is a relationship among the three dimen­
sions o f care; though one does not necessarily
cause the other, they are related “ as links in a
ch ain .” 10 Within a proper structure, dentists can
deliver adequate care that should result in healthy
oral tissues. T he present project is concerned
with all three aspects o f quality and especially
the interrelationships am ong them; how ever,
this paper is directed to the developm ent and
testing o f standards for the “ process o f care.”
Methods
■ Development o f criteria and standards: Since
the evaluation o f the process o f care depends on
criteria developed by leaders in the dental com ­
m unity, the first step was to select a standards
com m ittee. On the advice o f the Hartford (Conn)
D ental S o ciety ’s execu tive com m ittee, a nine-
mem ber com m ittee was formed: one dentist was
a m em ber o f the C onnecticut State Board o f
D ental Exam iners, tw o w ere on the executive
com m ittee, four w ere active members o f the so­
ciety, and tw o w ere U niversity o f C onnecticut
dental school faculty in the department o f gen­
eral dentistry. T hey all considered them selves to
be general dental practitioners and had at least
seven years’ experience in practice. In addition,
the com m unity m embers o f the standards com ­
mittee w ere generally w ell known am ong their
peers, and it was assum ed that criteria and stan­
dards developed by this group would be accept­
able to local dentists.
A t the first m eeting o f the com m ittee a deci­
sion was made to try to reach unanimity when­
Bailit—others: QUALITY OF DENTAL CARE ■ 843
 ever possible on criteria, but in areas o f contro­
versy a majority vote w ould be binding. A lso , it
was realized that the com m ittee w as rather large
and a subcom m ittee w ould be needed to do m ost
o f the staff work. T he subcom m ittee consisted
o f one dental school faculty m em ber and tw o
com m unity practitioners.
c r i t e r i a o f c a r e . A n exten sive bibliography
w as available on evaluation procedures and cri­
teria used in other investigations. From a general
perspective, the work o f P eterson and co-w ork­
ers12 on general m edical practitioners w as par­
ticularly helpful, as was a manual o f dental cri­
teria and standards developed by Friedm an.3
For the m ost part, the actual content o f the cri­
teria w as based on the practical experience o f
the clinicians on the com m ittee; reference was
also made to dental textbooks. A nother useful
source, dealing with som e o f the conceptual
problem s that had to be resolved before criteria
and standards w ere developed, w as a book by
D on ab ed ian .13 Four o f these conceptual prob­
lem s d eserve com m ent.
First w as the question o f whether the criteria
should be norm ative or empirical. Should the cri­
teria reflect what dentists should do in practice
(normative) or what they actually do (empirical)?
O bviously, these are not necessarily the sam e.
T he com m ittee decided that criteria and stan­
dards should not be unrealistic. “ Ideal dentist­
ry” or the “ best possible dentistry” are intan­
gibles that are probably not definable and that,
in any event, suggest a level o f care that is seldom
a ch iev ed .11 T he m ore important issu e is w hether
the care is adequate. T hus, the criteria are nor­
m ative in that they are oriented to what the av­
erage dentist should do to provide adequate care.
T he second major issu e w as w hether to d evel­
op criteria for all aspects o f general dentistry or
for m ore limited parts o f it. Since general dental
practitioners deal with many different typ es o f
problem s, it w ould be im possible to develop cri­
teria for every conceivable situation.
O ne approach to this conceptual problem that
has recently received attention is the “ tracer
m ethod .” 14 H ere, criteria are developed for cer­
tain specific disease entities and providers are
evaluated with respect to these conditions. It is
assum ed that the level o f care provided for the
tracer d iseases approxim ates the level o f care
for all diseases.
In dentistry, the tracer method could mean
evaluation o f only the quality o f amalgam restor­
ations, if it is assum ed that the quality o f endo­
844 ■ JADA, Vol. 89, October 1974
dontic or crown and bridge treatm ent w ould be
comparable. Since there is little evid en ce about
the relationships in quality among different treat­
ment areas, limiting the criteria to four or five
conditions did not seem appropriate. Rather,
the com m ittee decided to prepare criteria and
standards for the com m on conditions that gen­
eral dental practitioners treat. T hen, after eval­
uation o f the standards clinically, it w ould be
possible to determine the feasibility o f a m ore
limited approach.
A third conceptual problem w as the division
o f the process o f care into its com ponents. There
are m any w ays in which this could be d one, but a
logical method is by the sequence o f events in
the treatm ent o f the average patient. T hus, the
first com ponent would be the quality o f the his­
tory and examination; second, the diagnosis;
third, the treatment plan; and fourth, the treat­
ment.
T he final issue w as the m ethods available to
collect information about patient care. T o som e
extent, the m ethods used w ould define the areas
o f patient care that could be evaluated. Three
approaches to data collection w ere considered:
observation o f the dentist, record audit, and pa­
tient exam ination. O f th ese three m ethods, ob­
servation o f the dentist w hile he w as treating a
patient was discarded as being too exp en sive and
probably unacceptable to m ost practitioners.
T he evaluation o f dental records including radio­
graphs seem ed a m ore reasonable approach,
since all dentists keep records and these records
can be evaluated relatively inexpensively.
T h e other major source o f data w as the clin­
ical exam ination o f patients. F or the evaluation
o f the technical quality o f m ost services related
to the teeth per se, patient exam ination should
provide an acceptable method because teeth do
not repair and a permanent record o f the treat­
ment rendered is left. This is not true for ser­
vices involving the soft tissu es, which can repair
without any permanent changes in size, shape,
or color.
c o m p o n e n t s o f c a r e . With th ese general
guidelines in mind, the subcom m ittee started to
develop criteria and standards for each o f the
four com ponents.
— H istory and examination: For sim plicity, it
was decided to limit the criteria initially to the
exam ination o f a new patient seeking com pre­
h en sive care. F or these patients, the history and
exam ination should be m ost com plete.
F iv e major elem ents w ere identified as n eces­
sary in an adequate history and examination:
the chief complaint and, if associated with a spe­
cific problem, a description of the present illness;
a personal history; the past medical history; the
past dental history; and the dental examination
(including use of diagnostic aides). For adequate
care to be provided, some description of these
five sections should be found in the dental rec­
ords. The information does not have to appear
in the categories or sequence suggested. The im­
portant issue is whether the appropriate data
exist somewhere in the record.
For each of the five areas, specific criteria
were developed to assess the adequacy of the
information (illustration). It became apparent
that adequacy involved two aspects—the level
o f detail and its accuracy. Although it is useful
to determine if the information recorded was ac­
curate, in this report only the level of detail in
the record is evaluated.
— Diagnosis: Evaluation of the diagnosis pre­
sented several practical problems because, in
many instances, dentists do not specifically list
diagnoses. In a few situations such as soft tissue
lesions, specific diagnoses might be in the record,
whereas for most other conditions they are not.
A lso, if the treatment plan is done well, it is rea­
sonable to assume that the diagnoses were ade­
quate. Therefore, since it is almost impossible to
obtain direct evidence on diagnoses and since
the quality o f the diagnoses is covered, to some
extent, in the treatment plan assessment, a de­
cision was made not to have a specific section on
the diagnostic phase of care.
—Treatment plan: It is necessary to separate
the evaluation o f the plan of treatment from the
actual treatment. The former includes the judg­
ments o f the dentist on what should be done for
the patient, whereas the latter reflects the tech­
nical quality of the treatment provided. This is
an important issue, since the dentist should not
be rated for the same performance in two differ­
ent parts of the evaluation.
Information on the treatment plan is elicited
from the treatment progress notes and from the
examination of the patient. The presence of a
specific treatment plan in the record would be of
great value but, except for the more complex
and difficult situations, most dentists probably
plan their treatment but do not record it.
The criteria for the treatment plan assessment
are divided into ten categories primarily on the
basis of the information obtained in the history
and examination. The categories are personal,
medical, extraoral tissues, preventive services,
restorative services, intraoral soft tissues, perio­
dontal services, occlusion, and sequence of
treatments (illustration).
—Treatment: The assessment of the technical
quality o f treatment can be done with relative
objectivity where there is a permanent change
in hard tissues such as in restorative dentistry.
H owever, the treatment o f tissues such as the
periodontium that can repair may be impossible
to evaluate. In a period of several months after
treatment, the tissues can return to their previous
state for reasons that may be independent o f the
dentist’s therapy; the actual therapy cannot be
directly evaluated, only the outcome o f care. As
a result o f these technical limitations, criteria for
assessing periodontal care were not developed.
In addition to periodontal services, endodontics
and oral surgery presented somewhat similar
problems so that their assessment was limited to
just a few criteria.
Most o f the information to assess the technical
quality of treatment was based on the examina­
tion of patients. For a few areas such as endo­
dontics and oral surgery, radiographs and treat­
ment notes were the main source of data.
A s previously implied, the treatment criteria
are divided in terms of the primary categories of
services commonly provided in general dental
practice. These include restorations, endodon­
tics, crowns and fixed bridges, partial dentures,
complete dentures, and oral surgery (illustra­
tion).
With the subcommittee doing much of the de­
velopmental work, the meetings of the standards
committee proved to be productive. Each set of
criteria was discussed in depth and, although
there was often controversy over particular is­
sues, it was seldom necessary to have a formal
vote. M ost decisions were reached by consen­
sus, and in six months, a first draft of the criteria
was completed.
■ Scoring system : Both quantitative and cate­
gorical (qualitative) scoring system s were de­
signed.
q u a n t i t a t i v e s c o r e . Before the criteria
could be used in the evaluation of patients, it was
necessary to develop a reliable scoring system
that was easily understood by the assessors. At
first, each criterion was scored on a three-point
scale: 1, unsatisfactory; 2, adequate; and 3, su­
perior. A value of 9 was assigned when no deci­
sion could be made; this was usually because of
Bailit— others: QUALITY OF DENTAL CARE ■ 845
Examples of criteria used to assess history and examination, treatment plan, and treatment.
History and examination
M e d ica l h is to ry : Foran adequate
medical history, these data should
be in the record:
1. A general description of the
patient’s general health, including
past serious illnesses.
2. In addition to a general state­
ment of the patient’s health, specific
references should be made to these
conditions: sensitivity to drugs and
other allergies, rheumatic fever,
bleeding problems, liverand kidney
disease, heart disease, diabetes,
and pregnancy status (if fem aieo f
child-bearing age).
3. The names of the patient’s
personal physicians (if any).
4. The date of the patient’s last
visit to a physician for a physical
examination.
5. The medications being
taken by the patient (if any) and the
reasons for taking them.
Treatment plan
S equence o f tre a tm e n t: Inm ost
instances there is an orderly se­
quence of priorities in planning a
patient’s care. Although some pa­
tients will not require certain treat­
ments such as referral to a physi­
cian, treatm ent plans should follow
the general sequence listed here.
As a broad guideline, the patient’s
chief complaint should be dealt
with on th efi rst visit. Of course, if
the complaint concerns the need for
dentures or other treatments that
cannot be provided until extensive
dental or medical care is completed,
the chief com plaint cannot be re­
sponded to at the beginning of treat­
ment.
This sequence of treatments is
recommended:
846 ■ JADA, Vol. 89, October 1974
1. Alleviation of acute condi­
tions (pain, bleeding, or acute infec­
tion) or the chief complaint.
2. Consultation with or referral
to a physician for systemic evalua­
tion.
3. Placement of the patient on
antibiotics or other premedications.
4. Institution of primary pre­
vention prog rams such as oral hy­
giene instruction, prophylaxes,
plaque control, or fluoride treat­
ments.
5. Control of deep caries that
may cause pulpal exposure. (The
sequence of items 6 through 10 is
interchangeable.)
6. Extractions and soft tissue
surgery.
7. Treatm ent of teeth endodon-
tically.
8. Treatm ent of the periodon­
tium.
9. Treatm ent of orthodontic
problems.
10. Adjustment of occlusion.
11. Operative treatm ent of
teeth (restorations).
12. Replacement of teeth pros-
thetically.
13. Recall examinations and
maintenance treatment.
Treatment
C o m p le te d e n tu re s : The factors
considered in judging complete
dentures are retention, stability,
vertical dimension, extension of
flanges, occlusion, placement of
teeth over ridges, and appearance.
1. Retention
Retention in the maxillary den­
ture should be sufficient to allow
the patient to perform the normal
mouth functions of talking, eating,
and opening w ithoutdislodgingthe
denture.
Retention in the maxillary den­
ture should be strong enough to
resist light hand pressure in a
downward direction.
2. Stability
For both the maxillary and man­
dibular dentures, there should be
only slight movement in a plane hor­
izontal with the ridge, when light
twisting pressure is placed on the
denture by the hand.
3. Vertical dimension
The teeth should not come in
contact when the patient talks.
There should not be excessive
free-way space with overclosure
when the teeth are in contact.
4. Extension of the flanges
The flanges of the denture
should b e to th e d e p th o fth e m u c o -
buccal folds without displacement
of tissue.
On the lingual aspect of the
mandibular denture, the flanges
should make contact with the floor
of the mouth at rest and should
not dislodge the denture when the
tongue is extended to moisten the
surface of the lower lip.
5. Occlusion
There should be bilateral con­
tact of all molar teeth.
There should be no movement
of denture bases when the teeth are
in light occlusion.
With repeated closure, the teeth
should meet without sliding.
6. Placement of posterior teeth
The buccal cusps of the molars
should be placed over the alveolar
ridge in the m andibular denture.
7. Appearance
The shade of the teeth should
blend with the patient’s remaining
natu ral teeth (if any) in the opposite
arch.
The labial position of the maxil­
lary anterior teeth should provide
adequate support to the lips.
lack of information. Preliminary attempts to use
this system showed that assessors had difficulty
making reliable judgments between the second
and third classes, especially in evaluating the
treatment plan and treatment. Therefore, for
these latter two components o f care, the scale
was reduced by removing the third classifica­
tion (superior).
In the rating of the treatment plan and treat­
ment, a problem arose if the dentist had multiple
opportunities to meet specific criteria. For ex­
ample, treatment-planning criteria were devel­
oped on the materials used to restore teeth. Since
the dentist usually restored more than one tooth
and often met the standards for some teeth but
not others, the question arose as to whether the
treatment plan in this area was satisfactory.
Arbitrarily, a general rule was established. If
90% or more of the decisions were correct, then
a satisfactory score was assigned. By the same
token, if less than 90% o f the decisions were cor­
rect for that item, an unsatisfactory score was
assigned. In practical terms this meant that if a
dentist used the correct materials in nine of ten
teeth restored for a given patient, a satisfactory
score was assigned for that criterion. If only five
restorations were evaluated and one was not ade­
quate, an unsatisfactory score was assigned.
One other issue on scoring deserves mention.
Many components o f care involve more than one
criterion. A s an example, assessors were asked
to evaluate the use of restorative materials. For
this item, four of five specific criteria related to
the material’s biological effects on pulpal tissue,
strength, esthetics, and so on are available. The
issue then is scoring the use of restorative ma­
terials if three of the four standards are met but
one is consistently unmet. It can be argued that
in this situation use of materials should receive
an adequate rating. Y et, from a clinical perspec­
tive, if one important aspect of material use is
consistently not achieved, it is likely that this
aspect of the treatment plan is clinically inade­
quate. A s an example, for anterior teeth a dentist
might use a synthetic filling material that has ex­
cellent strength, matches the color of the natural
tooth, but causes a permanent, adverse pulpal re­
action even with a cavity liner or base. The use of
materials should be given a low score even
though most o f the criteria are met. Thus, an
“ adequate” score would be assigned to an item if
all criteria within that item were met 90% or more
of the time.
c a t e g o r i c a l s c o r e . For the validation o f the
quantitative evaluation system just described, a
five-point categorical (or qualitative) scale was
devised to provide an overall, rather than an item
by item, measure o f quality. The assessors gave
their general impression of the quality of care
provided after examining the patient and briefly
reviewing the record (treatment notes and radio­
graphs). Then, the quality of care was rated by
one of five general categories.
— Category 1: The dentist has met few of the
standards commonly accepted for adequate care.
The treatment is clearly o f the low est quality,
technically and judgmentally.
— Category 2: The dentist has some area
where his work is adequate, but overall the level
of care has not reached an acceptable level.
— Category 3: The dentist has achieved ade­
quate care in all phases of the treatment plan and
treatment.
— Category 4: The dentist has done superior
work in most of the major phases o f care. There
are, however, one or two areas where the work
is adequate but not superior.
— Category 5: The dentist has provided out­
standing care in all phases o f treatment.
The assessor who performed the overall eval­
uation was not involved in the detailed (quanti­
tative) evaluation o f that particular patient.
In addition to its use as a validity check, a ma­
jor advantage of the categorical approach is the
speed with which an assessment can be com­
pleted; it requires about five minutes. Conse­
quently, it is crucial to compare both systems
and to judge whether One can be used as a substi­
tute for the other.
■ Training o f assessor: Two types o f assessors
were trained. For the history and examination,
two research assistants without any background
in dentistry or other health disciplines were
taught to rate the dental records and to abstract
them for use by the dental assessors.
The rationale for using nondental personnel
for this task was that decisions were being made,
not on the appropriateness of treatment, but
rather on the amount of detail in the record, and
nonprofessional record assessors could do this
job at less cost than could trained dentists.
Three members o f the standards committee—
two from the community and one from the dental
school—were trained as assessors to evaluate
the treatment plan and treatment. Although the
assessors were already familiar with the evalua­
tion system , they were asked to take a test to
examine their didactic knowledge o f the criteria.
After the assessors had successfully completed
Bailit—others: QUALITY OF DENTAL CARE ■ 847
the test, patients were examined clinically, and
preliminary data were collected on the criteria
and scoring system. It soon became apparent
that som e o f the criteria were not practical, could
not be rated reliably, or for some other reason
were not usable. The standards committee met
several times to modify the criteria.
The usual procedure during the training period
was for two or three patients to be scheduled for
an afternoon; each patient was rated indepen­
dently by the three assessors. Then, the asses­
sors met to discuss any differences among them.
This quickly led to the assessors’ developing
comparable interpretations o f the standards and
a better understanding of the entire process of
evaluation.
A fter two months o f training and after modifi­
cations were made in the criteria, a manual was
prepared outlining in detail the method of eval­
uation. A t this time, two additional assessors
were recruited, one a member of the standards
committee, the other a community practitioner.
■ Examination o f patients: With both sets of
assessors trained, the criteria were evaluated for
reliability. Approximately 50 patients were sche­
duled for examination. These participants were
active dental patients from three sources: the
private practices o f community dentists, the den­
tal school clinic, or the Veterans Administra­
tion hospital (these patients received their care
in the private practices o f local dentists). The pa­
tients selected were above the age o f 18, had com ­
pleted a course of treatment (were in a main­
tenance phase of care), and had been treated for
more than one condition. In regard to the latter,
most patients had, as a minimum, received care
requiring restorative and periodontal therapy.
A description of the evaluation process fol­
lows. First, before the clinical evaluation, the
research assistants rated the history and exam­
ination based on the dentist’s records. They then
abstracted the record by writing a brief descrip­
tion o f the treatment given at each visit in se­
quence. This allowed the dental assessor to eval­
uate the plan o f treatment without wasting time
trying to organize the material in the original rec­
ord; it also prevented the dental assessor from
knowing the name of the dentist who treated the
patient.
Immediately before the clinical assessment,
the patient was asked to complete a short ques­
tionnaire noting the history of dental treatments,
any outstanding medical problems, and payment
848 ■ JADA, Vol. 89, October 1974
sources for care. A lso, attitudinal questions
were used to assess the patient’s willingness to
practice good oral hygiene and to pay for need­
ed services.
N ext, the patient was then seen by the dental
assessor who evaluated the treatment plan and
treatment. H is sources o f information were the
abstracted treatment notes, the patient question­
naire, and his own clinical examination of the
patient. After the first assessor had finished, the
patient was seen independently by a second as­
sessor who proceeded to do the same type of
evaluation. Finally, a third clinical assessor eval­
uated the patient using the qualitative scoring
system.
■ D ata analysis: The process variables were
examined in terms of their reliability, variability,
validity, and practicality. The ideal evaluation
system would be highly consistent and stable
(reliable), would highly discriminate between
patients (variable), would be meaningfully inter­
pretable (valid), and easily learned and adminis­
tered (practical).
— Reliability: Two measures were used to de­
termine interrater reliability. The first was sim­
ply a descriptive indicator that compared the per­
centage o f agreement between any two judges
across all items. The value of this measure can
range from 0 (perfect disagreement) to 100% (per­
fect agreement). A chance finding would be 25%
in history and examination and 33% in treatment
plan and treatment. H ere, a chance result refers
to the number of ways two judges can agree, di­
vided by the sum o f possible agreements and dis­
agreements.
Another indicator of reliability was the com­
parison o f mean scores (on items 1,2, and 3 only)
between judges for the same record or patient.
The t statistic was used to test the null hypothe­
sis that means for the two judges were equal. A
significant value indicated lack of agreement be­
tween judges on a particular patient or record.
—Variability: Before any practical meaning
can be assigned to a criterion, variance must be
present. An item that everyone agrees on and
has been assigned the same value (for example,
2) does not distinguish between dentists. A s the
evaluation of such an item does take some finite
amount o f time, it reduces the efficiency of the
instrument.
The method used to measure variability was
the standard deviation. It was calculated for
each item across all patients; items rated 9, no

judgment, were excluded. The range o f vari­
ability scores with this data was 0 to a value
slightly greater than 1. A value o f 0 meant that
only one score had been assigned to the item
(that is, either 1, 2, or 3) whereas a value near 1
signified that an equal number o f Is, 2s, and 3s
had been assigned.
—Validity: The validity of an instrument an­
swers the question, “ Is the instrument measur­
ing what it is supposed to measure?” Three mea­
sures of validity are of concern here: content or
consensual, predictive, and concurrent. The first
refers to the appropriateness of the items in the
instrument. This criterion was met by having all
items reviewed and approved by the standards
committee. The second measure investigates the
predictive ability of the items in regard to other
variables such as outcome indexes.
The only estimate o f validity statistically de­
termined in this paper, concurrent validity, de­
scribes the relationship between the quantita­
tive and qualitative scores. This was accom­
plished by calculating the product moment cor­
relation between the sum of the treatment plan
and treatment scores (a quantitative measure)
and the qualitative score assigned to each pa­
tient.
— Practicality: Practicality refers to several
different measures including the length of time
required to learn the criteria and to evaluate a
particular patient clinically. These considera­
tions are crucial for wide dissemination of the
system.
Results
■ D escription o f sample: The sample consisted
o f 32 men and 15 women, ranging in age from 20
to 65 with a mean age of 30.6. Since about half
the patients were recruited from the Veterans
Administration dental clinic, the distribution o f
age and sex is clearly skewed to men aged 20 to
25 years. Thus, the patient population cannot
be considered representative o f the total popula­
tion o f patients in the region.
Each patient additionally was classified ac­
cording to areas of treatment planning and treat­
ment that could be assessed. For example, how
many patients had restorative or endodontic
treatment that could be evaluated? From Table
1 it is evident that most patients had received
restorative, periodontal, and some type o f pros-
Table 1 ■ Percentage of patients assessed by category
of evaluation: treatment planning and treatment.
C a te g o ry o f e v a lu a tio n % p a tie n ts
T re a tm e n t plan
M e d ic a l 48.9
E x tra o ra l 10.6
S o ft tis s u e 2.1
P re v e n tiv e
R e s to ra tiv e
84.0
89.4
P e rio d o n ta l 63.8
O c c lu s a l 10.6
P ro s th e tic s 66.0
P e ria p ic a l 21.3
T re a tm e n t
R e s to ra tiv e 89.4
C row n and b rid g e 53.2
R e m o v a b le p a rtia l d e n tu re s 21.3
C o m p le te d e n tu re s 12.8
E n d o d o n tic s 30.0
O ral s u rg e ry 12.8
thetic services. H owever, few could be evalu­
ated for intra- or extra-oral soft tissue lesions,
occlusal problems, and oral surgery treatment.
This assessment could not be made for two main
reasons: either the patient never needed treat­
ment (for example, periapical lesion), or if he did,
there was not enough information on the extent
o f the problem or how the dentist handled it (for
example, soft tissue lesion). Often, it was impos­
sible to distinguish between these two reasons,
but the end result was the same: no judgment
could be made.
■ Reliability: First, so that the reliability o f the
record audit of the history and examination could
be measured, the audits o f the research assist­
ants were compared with those of a dentist.
When it was apparent that there was little differ­
ence between the non-dentist and dentist audi­
tors, ten records were chosen at random and
evaluated twice for detail by the two research
assistants. The average agreement between as­
sessors for all items was 95%. Similarly, the per­
centage agreement between assessors for each
item ranged from 80% to 100%. All the demo­
graphic items such as sex, age, and occupation
were scored identically by both judges. Those
relating to charting o f caries, periodontal exam­
ination, and oral hygiene evaluation were the
most troublesome and received a lower reliability
of 80%.
These results were supported by the findings
of no significant differences in mean scores be­
tween judges for any o f the records. It is apparent
that for the assessment of history and examina­
tion a high degree o f reliability was achieved.
The analysis of reliability for treatment plan
showed similar results. The percentage agree­
ment between assessors across all items ranged
from 79% to 96% (Table 2). Although some pairs
Bailit—others QUALITY OF DENTAL CARE ■ 849
Table 2 ■ Reliability between assessors (percentage
agreement) for all items: treatment plan and treatment.
No.
P a ir no. A s s e s s o rs p a tie n ts
1 A /B
2 A/C
3 A/D
4 A/E
5 B/C
6 B/D
7 B/E
8 C/D
9 C/E
D/E
13
10
Table 3 ■ Proportion of items classified by percentage
of agreement between assessors.*
C om ponent of
% a g re e m e n t b etw e en a s s e s s o rs
-------------------------------------------------------------------------------
e v a lu a tio n <80%
H is to ry & e x a m in a tio n
31 ite m s
T re a tm e n t plan
29 item s 0.21
T re a tm e n t
50 ite m s 0 .0 6
* Each c e ll re p re s e n ts th e p ro p o rtio n o f in d iv id u a l ite m s w ith in a
s p e c ifie d ra n g e o f a g re e m e n t.
o f dentists saw very few patients in common,
others such as D and E, who saw a total of 13
patients together, had a 92% agreement score.
For the reliability between assessors for each
item, only a fifth of the items had less than 90%
agreement; the lowest o f these was 59% for a spe­
cific criterion related to the control of periodon­
tal inflammation. As is evidenced from Table 3,
more than a third of the items were agreed on at
least 90% o f the time.
Finally, o f a total of 4 7 t tests, each determined
by a comparison of the two assessments for each
patient, only four were significant (2.4 are ex­
pected by chance). For the treatment plan, then,
the reliability was high regardless o f the measure
used.
For the evaluation o f treatment, the percen­
tage agreement between assessors for all items
ranged from 90% to 96% (Table 2). Similarly,
more than half of the items had agreement per­
centages higher than 90 and only three items
were agreed on less than 80% o f the time (Table
3). This occurred for criteria related to embra­
sure space in amalgams and the marginal ridge
heights in crowns or fixed bridges. The agree­
ment level for these items was 79%.
In the calculation of the 47 t tests, 11 showed
significance beyond the 5% level. This is about
nine more than would have been expected by
chance. A close examination of the data showed
850 ■ JADA, Voi. 89, October 1974
% a g re e m e n t
T re a tm e n t plan T re a tm e n t
(N = 2 9 item s) (N = 5 0 ite m s )
4 86
4 85
9 85
6 86
2 83
2 83
1 96
4 81
2
79
92
8 0 % -9 0 % 9 0 % -9 9 %
0.10
0.41
0.28
that judges C and D , who did the least number of
evaluations, had the highest proportion o f signifi­
cant differences compared with those o f their
partners. The lack of practice probably contrib­
94 uted to their unreliability. Even with this result,
96
90
close to 80% of the t tests were not significant;
94 this indicates that, in general, treatment assess­
94
92 ment for a particular patient was quite reliable.
94
92
95 ■ Variability: Since each patient was seen twice,
92
scores by each assessor were assigned to one of
two groups. This procedure yielded two stan­
dard deviations for each item across all patients.
These values were averaged to obtain a better
indication of the “ true” population standard
deviation.
100% Although it is quite difficult to clearly define an
acceptable value for the standard deviation, for
0.25 0.65
practical purposes 0.25 was set as a lower limit.
0.03
0.35 Items with standard deviations below this cutoff,
0.48 0.18 representing about 25% o f the maximum score,
would serve as poor discriminators and predic­
tors. A s can be seen from Table 4, about a third
of the items on history and examination and
treatment had standard deviations below 0.25
and more than half the items in the treatment
plan component failed to reach this arbitrary
cutoff.
A close examination o f the items in each com­
ponent showed where the low standard devia­
tions were most prominent. For example, in the
history and examination nearly all the demo­
graphic criteria had little or no variance. Simi­
larly, the medical items in the treatment plan
were very low on the standard deviation. Finally,
in treatment, the items relating to extractions
showed little, if any, variability.
■ Validity: A s a measure o f concurrent validity
the summed treatment plan and treatment scores
were compared to the qualitative score for each
patient. A product moment correlation o f 0.30
was found between the measures. This was sig­
nificant at the 0.05 level. Although there is some
Table 4 ■ Percentage of items classified by standard
deviation.*
S ta n d a rd d e v ia tio n
C om ponent of No. item s
e v a lu a tio n in co m p o n e n t =£0.25 > 0 .2 5 < 0 .7 5 ^ 0 .7 5
H is to ry and
e x a m in a tio n 31 32% 26% 42%
T re a tm e n t plan 29 58 31 11
T re a tm e n t 50 34 28
38
*Each c e ll re p re s e n ts th e p e rc e n ta g e o f ite m s w ith in a s p e c ifie d
s ta n d a rd d e v ia tio n range.
indication that the two scores were tapping the
same underlying variable, 91% of the variance
is unaccounted for; the process standards con­
tain unique variance unexplained by a simple
evaluation measure.
■ Practicality: The training time for assessors,
both professional and nonprofessional, averaged
about 20 hours. About 18 more hours were spent
by each clinical assessor in practicing the system
with patients. The time for examination of pa­
tients varied indirectly with the number o f pa­
tients seen. A t the beginning o f the evaluation
sessions, about 40 minutes were required to do
the treatment plan and treatment, but by the end
o f the sessions, very few patients required more
than 20 minutes of the examiner’s time. The qual­
itative measure required no more than five min­
utes at any time.
These times are all based on the particular as­
sessors, recorders, and patients used in the pres­
ent test run. However, these results are probably
not unique and may be generalized to most set­
tings.
Discussion
Perhaps the most important results of this study
are the insights gained on the process of devel­
oping criteria and standards and not on the con­
tent of the criteria per se. Private dental practi­
tioners, without special expertise or training,
were able to formulate criteria of care within a
six-month period. Some o f these same practi­
tioners then used these criteria in the assess­
ment o f patients. While technical advice was
available from dental faculty with a background
in tests and measurements, this type of advice
is probably obtainable in most communities; this
suggests that many dental groups could develop
standards.
A s noted in the introduction, there are many
advantages to the development o f criteria at a
local as well as a national level. First o f all, there
is a greater probability that the criteria will be
acceptable to community dentists. A second ad­
vantage is the educational value o f having a group
of practitioners who know and respect each other
discuss the elements of adequate care. Third, the
development of standards may be the beginning
of a community peer review system that goes be­
yond problems o f patient or third party com­
plaints, and becomes a form o f continuing edu­
cation. Objective evaluation o f the quality of
care provided in a practice by colleagues en­
ables the individual dentist to receive feedback
on his clinical strengths and weaknesses. This
assessment should be carried out in an atmos­
phere o f helpful concern rather than of sanctions
and punishment—the usual connotations asso­
ciated with peer review. Even the dentist doing
the assessment will find peer review a learning
experience. In fact, it is recommended that all
dentists should, at some time, serve as asses­
sors.
The preliminary results o f testing the criteria
for reliability, validity, and practicality were en­
couraging. A ssessor reliability was relatively
high for most items. For the few categories in
which there were problems, it is our impression
that the criteria were not explicit and the asses­
sors were unsure of themselves. These criteria
have been modified by the standards committee,
and the reliability for these items should improve.
In terms of the variability scores, the standard
deviations were not as high as expected; this in­
dicates that many items were always assigned
the same value by the assessors. T o some extent,
this may have led to artificially high reliability
values for some items.
Several reasons could account for the low var­
iability of some items. The standards may have
been set too low or high; however, a more im­
portant reason is probably the mix of patients
evaluated. With a larger number o f patients cov­
ering a wider range o f clinical problems, greater
variance would, in all likelihood, result.
Practically, the total time needed for the den­
tist to evaluate one patient was less than 25 min­
utes. O f equal significance, the time necessary to
train a practicing dentist to be an assessor was
also reasonable. These results suggest that this
system , or a modified version of it, could be in­
stituted on a larger scale. It could serve as a form
o f continuing education, as a framework for pa­
tient assessment by peer review committees,
and as one level of quality assessment for den­
tal care programs. With reference to the last
item, there must be different levels o f evalua­
tion when large numbers of patients are involved.
Certainly, it is unrealistic to suppose that a sig­
nificant percentage o f patients in a program
could be examined as part of a quality monitor­
ing system. The cost and inconvenience to both
providers and patients would be too great. In­
stead, the initial or screening review will prob­
ably depend on information obtained from insur­
ance claim forms or patient records. Then, on
Bailit—others: QUALITY OF DENTAL CARE ■ 851
 the basis o f these results, selected dentists would
be subject to a more detailed review. It is at this
point that the evaluation system proposed here
would be o f value.
Mention was made previously of problems
associated with patient sampling. This whole
question needs considerable study before any
organized evaluation system can be effected.
Currently, there is little knowledge of the num­
ber and types o f patients who must be examined
to obtain a stable estimate of quality in the aver­
age dental practice. Certainly, assessment of
two or three patients from an active list of 1,500
to 2,000 probably yields little information on the
quality o f the entire practice. This point should
be emphasized since some recent publications
on the quality of dental care have suggested that
a low but significant percentage of dentists are
delivering inadequate care; unfortunately these
results are based on samples o f only a few pa­
tients from each practice.15,16 Although this con­
clusion might be correct, it is premature to make
such statements without further evaluation.
A related issue that has both theoretical and
practical significance is the relationship of qual­
ity among different treatment areas. For exam­
ple, if it is assumed that the average general den­
tist spends most o f his time providing restora­
tive, periodontal, and prosthetic services, is the
quality of care for each o f these treatment areas
approximately the same within a particular prac­
tice? From a theoretical point of view , this prob­
lem is o f interest because it leads to speculations
on the reasons for any quality differences found.
Are they a result of personal preferences of the
dentist? D oes he enjoy one area but not the oth­
er? Are they a factor of his education and train­
ing or, perhaps, the amount of time spent in pro­
viding specific services? Obviously, these are
not mutually exclusive categories and probably
all reasons are relevant.
Practically, if the quality of care seen in one or
two types of services is indicative o f quality in
the entire range of services provided by a den­
tist, then the evaluation process can be greatly
restricted and limited to the evaluation o f a few
patients of a specific type. H owever, if this is
not true, then it is important that enough patients
o f different types be seen to obtain a valid es­
timate of quality. Although these problems seem
somewhat esoteric at this time, they may soon
assume major importance.
The attempt to validate the criteria with use
of an independent evaluation o f the patient and
852 ■ JADA, Vol. 89, October 1974
rating o f the quality o f care on a scale of from one
to five resulted in a low correlation (r=0.30) be­
tween the qualitative and quantitative scores.
In the five to ten minutes allowed for the qualita­
tive assessm ent, it is our impression that the ex­
aminers tended to focus on one or two items that
did not meet acceptable standards of care and to
weigh these items heavily in the final score they
assigned. This suggests that it is necessary to
have an orderly schedule to follow in the exam­
ination o f patients.
O f course, the value o f the qualitative versus
the quantitative assessment systems cannot be
decided finally until the predictive validity of
each method is evaluated—the relationship be­
tween the process scores (qualitative and quan­
titative) and scores based on indexes of patient
health. Although unlikely, it is possible that the
qualitative score correlates more closely with
the number of D M F teeth or periodontal index
than does the quantitative score. This issue is
currently under investigation.
Summary
This paper presents a description o f the process
of development and testing of clinical criteria
and standards for general dentistry. A standards
committee made up of nine dental practitioners
established criteria for three components of care:
the history and examination, treatment planning,
and treatment. The standards establish what the
average general dental practitioner should do to
provide adequate care. Criteria could not be de­
veloped for all services performed by dentists,
so only the more common ones were considered.
After a limited pretest of the criteria, five
dentists were trained to use them in the evalua­
tion o f patients. Two independent assessments
of 47 dental patients were made in an effort to
determine the reliability and variability o f speci­
fic items and the practicality of the entire sys­
tem. In addition, the validity o f the criteria
was evaluated by a comparison of the scores as­
signed on the quantitative review system to those
obtained independently on the basis of a qualita­
tive rating scheme.
The results indicated a high degree of reliabil­
ity for most items. H owever, the variability was
often quite low; this may be explained partially
by the limited number and diversity of patients
evaluated.
 The time needed to train assessors and to ex­
amine patients was within practical limits; thus,
this system is feasible on a larger scale. With
further refinements in the review procedures,
the time and, therefore, the expense of evalua­
tion can probably be reduced even more.
The correlation coefficient between the quan­
titative and qualitative scoring systems was low
but statistically significant. This indicates that
the simpler qualitative method cannot be sub­
stituted for the quantitative approach.
The major point to be emphasized from this
study is that criteria and standards of dental care
can be developed and tested by local dental
groups. These standards may serve as the basis
for any government-sponsored evaluation sys­
tem.
A uthor’s note: G . Ryge and M. Snyder pub­
lished an investigation (“ Evaluating the clinical
quality o f restorations” ) on the development and
testing o f process standards for assessing the
technical quality o f restorations (JA D A 87:369
Aug 1973).
This w ork was supported in part by contract NIH-72-4207 from
the Division o f Dental Health, National Institutes o f Health, US
P ublic Health Service.
The au thors acknow ledge the assistance of Dr. Earle Yeamans
and Robert Villanova; the members o f the standards com m ittee:
Dr. Nathan Dubin, Dr. Stanley Holzman, Dr. Robert Levine, Dr.
Calvin Mass, Dr. S edrick Rawlins, Dr. Robert Villanova, and Dr.
John Zazzaro; the Hartford Dental Society and the State Board
o f Dental Examiners; Dr. Michael Zazzaro, secretary o f the board;
and Ms. Joan Jannace and Ms. Sharon Zarcaro.
Dr. B ailit is professor and head of the departm ent o f behavioral
sciences and com m unity health, School o f Dental Medicine,
University o f C onnecticut Health Center, Farm ington, Conn 06032.
Dr. Koslowsky is assistant professor, departm ent of behavioral
sciences and com m unity health, and Dr. Grasso is assistant pro­
fessor, departm ent o f general dentistry, School o f Dental Med­
icine, University o f C onnecticut Health Center. Dr. H olzman and
Dr. Levine are private practitione rs and members o f the Hartford
Dental Society. Ms. Valluzzo and Ms. A tw ood are research as­
sistants in the departm ent of behavioral sciences and com m un­
ity health, U niversity o f C onnecticut Health Center.
1. Egdahl, R.H. Foundations fo r medical care. New Engl J Med
288:491 March 8, 1973.
2. Abram ow itz, J. Planning fo r the Indian Health Service. J
P ublic Health Dent 31:70 S pring 1971.
3. Friedman, J.W. A guide fo r th e evaluation of dental care. Los
Angeles, School o f P ublic Health, University o f C alifornia, 1972.
4. S chonfeld, H.K., and others. Professional dental standards
fo r the con tent of dental exam inations. JADA 77:870 O ct 1968.
5. S oricelli, D.A. M ethods o f adm inistrative control fo r the pro­
m otion o f quality in dental programs. Am J P ublic Health 58:1723
Sept 1968.
6. Cons, N.C. Method fo r posttreatm ent evaluation of the qual­
ity of dental care. J P ublic Health Dent 3 1 :104 S pring 1971.
7. Lee, R.I., and Jones, L.W. The fundam entals o f good med­
ical care. P ublication of the C om m ittee on the Costs of Medical
Care, no. 22. Chicago, C hicago University Press, 1933.
8. Roemer, M.l. Evaluation o f health service program s and lev­
els of measurement. HSMHA Health Rep 86:839 Sept 1971.
9. Schonfeld, H.K., and others. The content o f good dental
care: m ethodology in a form ulatio n fo r clinical standards and
audits, and prelim inary find in gs. Am J P ublic Health 57:1137
July 1967.
10. Donabedian, A. Evaluating the quality o f medical care.
M ilbank Mem Fund Q 44:166 July (Suppl) 1966.
11. Friedman, J.W. Study and appraisal guide fo r dental care
programs. Berkeley, School o f P ublic Health, D ivision o f P ublic
Health and Medical A dm inistration, University of C alifornia, May
1963.
12. Peterson, O.L., and others. An analytical study o f North
C arolina general practice: 1953-1954. J Med E duc31:1 Dec 1956.
13. Donabedian, A. A guide to medical care adm inistration.
V olum e II: Medical care appraisal—qu ality and utilization. New
York, The American P ublic Health Association, Inc., 1969.
14. Kessner, D.M.; Kalk, C.E.; and Singer, J. Assessing health
qu ality—the case fo r tracers. New Engl J Med 288:189 Jan 25,
1973.
15. B eilin, L.E.; and Kavaler, F. P olicing p u blicly funded health
care fo r poor quality, overutilization, and fraud— the New Y ork
C ity M edicaid experience. Am J P ublic Health 60:811 May 1970.
16. Denenberg, H.S. A shopper's guide to dentistry. Harrisburg,
Pennsylvania Insurance Department, 1973.
Bailit—others: QUALITY OF DENTAL CARE ■ 853