TITLE: The Development of Two Performance Appraisal Systems for Firefighters

AUTHOR NAME: Robert L. Holmgren

PUBLISHED DATE: 20 april 1981
OBJECTIVES: A behaviorally anchored rating scale (BARS) and a mixed
standard rating scale (MSS) were developed to evaluate the performance of
firefighters in a large midwestern city. Both formats were developed from the same
pool of items
RESEARCH METHODOLOGY: Firefighters, lieutenants and captains from the
fire department in a large midwestern city took part in this study. All personnel had
been in their current job title for at least six months, some of them for as long as
twenty years or more. The fire department is organized into groups of up to six
individuals assigned to a specific truck or apparatus.
ANALYSIS AND INTERPRETATION: Three additional factors must be
considered: restriction of range, leniency and halo. These three factors all have an
influence on the alpha reliability. They may each serve to artificially inflate the
reliability. Even though the BARS has a higher reliability, this effect may be
diminished or even changed when these three other error factors are considered.
CONCLUSION: Finally, it would be inadvisable to sum together criterion scores
that measure different concepts. One would be hard pressed to state exactly what
this composite criterion was measuring. Rather, it was deemed wiser to assess all
the information in the five criteria separately.
INFERENCE: Restriction of range also has an influence on the leniency and halo
error analysis. Leniency error analysis can be performed with respect to the
theoretical definition of leniency or on the basis of the actual distribution of scores.