MENC: The National Association for Music Education

A Facet-Factorial Approach to Rating High School Choral Music Performance

Author(s): John M. Cooksey
Source: Journal of Research in Music Education, Vol. 25, No. 2 (Summer, 1977), pp. 100-114
Published by: Sage Publications, Inc. on behalf of MENC: The National Association for Music
Education
Stable URL: http://www.jstor.org/stable/3345190 .
Accessed: 22/06/2014 05:34

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .
http://www.jstor.org/page/info/about/policies/terms.jsp

.
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of
content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms
of scholarship. For more information about JSTOR, please contact support@jstor.org.

Sage Publications, Inc. and MENC: The National Association for Music Education are collaborating with
JSTOR to digitize, preserve and extend access to Journal of Research in Music Education.

http://www.jstor.org

This content downloaded from 194.29.185.227 on Sun, 22 Jun 2014 05:34:41 AM

All use subject to JSTOR Terms and Conditions
100/JRME

A FACET-FACTORIAL
APPROACH TO RATING
HIGH SCHOOL CHORAL
MUSIC PERFORMANCE

John M. Cooksey

The purpose of this study was to construct and test a rating scale for the
evaluation of high school choral music performance, using a facet-factoria I
approach. A scale was developed by collecting descriptions of high school
choral performances, transforming them into items, and pairing them with
a Likert-typescale. Fiftyjudges used the scale to rate one hundred high school
choral performances. The ratings were factor analyzed and inter-judge
reliability estimates were obtained. Two criterion-related studies, using a
global performance rating and the NIMACadjudication scale as criteria,
were completed. Seven factors of choral performance were produced by the
analysis: diction, precision, dynamics, tone control, tempo, balance/blend,
and interpretation/musical effect. Thirty-six criteria (dimensions-items)
were selected to form subscales to measure these seven factors. The final
scale achieved high inter-judge reliability and high criterion-related validity.

Key Words: choral organizations, performance ability, performance tests,

secondary schools, testing, tests.

Some recent researchers have placed more emphasis upon the

neurological-physiological aspects of performance behavior (Madsen
and Madsen, for example)-those parts of the performance act
that are more easily identified and measured.' Others (Leonhard
and House, for example)2 have continued to insist that musical
performance be considered from more than just a physiological
point of view, that any attempts at evaluation and measurement in-
1 Clifford K. Madsen and Charles H. Madsen
Jr., Experimental Research in
Music, Contemporary Perspectives in Music Education Series, edited by Charles
Leonhard (Englewood Cliffs, New Jersey: Prentice-Hall, Inc., 1970), p. 40.
2 Charles Leonhard and Robert W. House, Foundations and
Principles of Music
Education, second edition (New York: McGraw-Hill Book Co., 1972), p. 284.

This content downloaded from 194.29.185.227 on Sun, 22 Jun 2014 05:34:41 AM

All use subject to JSTOR Terms and Conditions
 Cooksey/101
clude "expressiveness" and "meaning" as vital parts of the performance
act itself.
In the choral field, few attempts have been made to develop group
measures of performance achievement. The obstacles are numerous:
(1) there are no precise, objective measuring instruments for choral per-
formance achievement; (2) teachers and adjudicators must depend on
their subjective opinions in judging the performance achievement of
choral groups; (3) there is a lack of agreement concerning which criteria
(expressive versus objective, for example) should be used in rating choral
performance; and (4) there is difficulty in defining, arranging, and identi-
fying the most basic factors involved in group performance behavior.
These same difficulties suggest, however, that more systematic attempts
should be made in devising instruments for measuring choral perform-
ance achievement.
The present study attempts to examine and solve some of the prob-
lems related to choral music performance evaluation. A systematic
approach to the development of a measuring instrument was taken
through scale construction, using factorial analysis to identify the main
components of choral performance. From these components precise cri-
teria were set up from which more accurate scales could be devised. The
primary purpose of the study, therefore, was to develop and validate a
rating scale to measure high school choral performance achievement.

The Problem
The problem was to construct a rating scale for the evaluation of high
school choral music performance using a facet-factorial approach. Analy-
sis of the problem led to the development of the following subproblems:

(1) What evaluative criteria (dimensions-items) should be used in the

description of high school choral performance?
(2) What factors seem to be central to rating high school performance
as identified by facet-factorial analysis?
(3) What items for the subscales formed for each factor should be
included in a choral performance rating scale (CPRS)?
(4) What is the inter-judge reliability of the CPRS?
(5) What is the criterion-related validity of the CPRS?

TheScale
The first step in the development of the CPRS was to determine what
evaluative criteria should be used in the description of high school
choral performance. Evaluative statements were collected from three
major sources: (1) 618 adjudication sheets containing judges' comments
about actual high school choral performances at district and state vocal
contests; (2) fifty-two critiques written by choral teachers on recorded per-
formances of high school choruses; and (3) twelve essays written by choral

This content downloaded from 194.29.185.227 on Sun, 22 Jun 2014 05:34:41 AM

All use subject to JSTOR Terms and Conditions
102/JRME
experts on aural aspects of high school choral performances. A content
analysisproducedmore than 500 statementsabout the evaluativeaspects
of high school choral performance.The statementswere pooled and
placed under the following categories of the National Interscholastic
Music Activities Commission (NIMAC) choral evaluation structure:
balance, diction, intonation, technique, tone, interpretation, and musical
effect.8During succeedingrevisionsof the list, statementswhich dupli-
cated others or seemed vague or irrelevant were eliminated. After several
revisions, 147 statements were chosen for the initial item pool.
The next step was to determine the factors that seemed central to rat-
ing high school choral performance as identified by facet-factorial analy-
sis. Facet-factorial scales were used by Butt and Fiske (1968)4 to measure
dominance in personality:

Strategiesfor the measurementof dominancewere classifiedas facet vs. trait,

and factorialvs. rational,yieldingthe four approachescomparedin the study:
rationalfacet,factorialfacet, rational trait, factorialtrait. ... The distinction
between trait and facet involves the degree to which the personalityvariable
is conceptually delineated and subdivided before scales are developed to
measure it. In the trait strategy,the construct is identified by a label or
sentence and is measuredat a global level. In contrast, the facet strategy
assumesthat a trait has several facets, each with several forms or elements.
The proposedobjectiveis a homogeneousscale for each specifiedpart of the
construct (e.g., each element of each facet).5

Considered within the context of the present study, choral perform-

ance behavior was thereforeviewed as a construct (complex behavior)
which consisted of facets (performancecomponents or factors such as
interpretation, tempo, tone quality, and so on) which could be further
divided into elements (descriptive statements or items concerning the
performancebehaviorswhich fit into the "facet"categories)for which
separate scales could be developed. Accordingly, before facet-factorial
analysis could be applied, the 147 statements chosen for the initial item
pool were randomly ordered, transformed into items (phrased both posi-
tively and negatively), and paired with a five-point Likert response
scale.6 The 147-item pool that resulted was employed by fifty judges in
rating 100 recorded high school choral performances. The responses
of these judges ranged from "strongly agree" to "strongly disagree" that
3 The NIMAC Manual, "The Organization and Management of Interscholastic Music
Activities" (Washington, D.C.: National Interscholastic Music Activities Commission of
the Music Educators National Conference, 1963), p. 15.
4 Dorcus S. Butt and Donald W. Fiske, "Comparison of Strategies in Developing
Scales for Dominance," Psychological Bulletin, Vol. 70, No. 6 (1968), pp. 505-519.
5 Butt and Fiske, p. 505.
6 R. Likert, "A
Technique for the Measurement of Attitudes," Archives of Psychology,
No. 140 (1932), 55 pages.

This content downloaded from 194.29.185.227 on Sun, 22 Jun 2014 05:34:41 AM

All use subject to JSTOR Terms and Conditions
 Cooksey/103
the statement was descriptive of the performance. Several of these items
appear in Table 1.
Positive items in the above scales were assigned five points for
"strongly agree," four points for "agree," and so on. Negative items were
scored in the reverse manner. The data were factor analyzed using the
Dixon BMD X727 and the SOUPAC8 principal factors factor analysis pro-
grams. The Kaiser criterion (only factors having latent roots greater than 1
are considered as common factors)9 and the Cattell scree test (Eigen values
greater than 1 are plotted on a graph against their ordinal numbers;
cut-off points for determining the number of factors to extract are dis-
covered by examining the shape of the resultant curve on the graph)1o
were employed for factor extraction. After a twenty-one factor structure
was produced in the first application of the BMD factor analysis pro-
gram to the item pool data, the cumulative proportion of the total
variance for the twenty-one factors was examined, and, using a liberal
estimate, fourteen factors were extracted. To further examine the results
and derive a factor structure for choral performance, four to fourteen
factors were rotated, using the Varimax rotation method contained
within the BMD X72 factor analysis program."1From six to ten factors
were rotated in the SOUPAC binomial program.12 The factor structures
resulting from these rotations were compared with each other and inter-
preted in terms of (1) factor structure simplicity, in other words, signifi-
cant loadings of items appeared only under the factors they were chosen
to measure; (2) the relative constancy of factors across successive rota-
tions (as identified by the various configurations and clusters of items
having significant loadings on the factors they were selected to define);
and (3) a comparison with the categories included in the a priori NIMAC
structure of choral performance. The selection and identification of fac-
tors that formed the basis structure of the CPRS thus rested upon both
statistical and musical considerations.
After an examination and comparison of the various factor matrices
produced in the BMD factor analysis program by the various rotations,
an eight-factor structure, resulting from a nine-factor Varimax rotation
in which one factor was not identified, was tentatively selected to form
the basis of the CPRS. This particular factor structure accounted for
.633 of the total common factor variance. The eight factors were labeled

7 W. J. Dixon, editor, Biomedical Computer Programs: X-Series Supplement, No. 3

(Los Angeles: University of California Press, 1970), pp. 90-103.

8 SOUPAC Program Descriptions, Book 2, CSO Volume 9 (USER), (Urbana, Illinois:
Computing Services Office, October 1, 1973), Section VII, Bin. 1.
9 Dennis Child, The Essentials of Factor Analysis (London: Holt, Rinehart and
Winston, 1970), p. 43.
10Child, p. 44.
11 Dixon, pp. 90-103.
12SOUPAC Program Descriptions, Section VII, Bin. 1.

This content downloaded from 194.29.185.227 on Sun, 22 Jun 2014 05:34:41 AM

All use subject to JSTOR Terms and Conditions
104/JRME
as diction, unity, balance,precision,tempo, dynamics,tone control, and
interpretation/musicaleffect.
In general, the eight-factorstructureof the CPRS was confirmedby
the SOUPACfactoranalysisprogram.The nine-factormatrix produced
by the Binormaminrotation contained two factors dealing with tone,
however.Since tone quality is a controversialarea in choral music upon
which experts do not always agree, it was not surprising to see this
particularfactorsplit in the SOUPACfactoranalysisprogram.

Table 1
Sample Items from the PerformanceDescriptionItem Pool
Judges' Responses Sample Items
SD D NN A SA 130. Top voice flat at times. (-)
SD D NN A SA 131. Excellent feeling of ensemble. (+)
SD D NN A SA 132. Phrases lack consistent contour and arch. (-)
SD D NN A SA 133. Excellent articulation of pitches. (+)

A comparisonof the CPRS eight-factorstructurewith the a priori

NIMAC category structure of choral performanceshows that several
differencesemergedbetween the two structuresas a result of the factor
analysisinterpretation.The NIMAC categoriesof tone and intonation
combined to form one factor, tone control, in the CPRS structure.
Another pair of NIMAC categories,interpretationand musical effect,
also combined to form a single factor in the CPRS structure.Such fac-
tors as tempo,dynamics,and precision,which had been subsumedunder
broaderheadingsof the NIMAC form,emergedas separatefactorsin the
CPRS structure.Only the factorsof balanceand diction were identified
as separate factors in both structures.Finally, the CPRS contained a
factor of choral performancenot included in the NIMAC form; it was
identifiedby the descriptoras "unity."
For the third and final step in developingand constructingthe CPRS,
thirty-nineitems from the original 147-itempool were selected to form
subscalesfor the eight factorsincluded in the CPRS. Four or five items
were chosen to measureeach factor in the scale. Items chosen had rela-
tively high loadings on the factors they were selected to define, were
relativelyfactor simple, and sampledthe broadestpossiblecontent area
within the eight factorsof the scale.Table 2 containsthe Varimaxfactor
loadings for items selected for the CPRS. The items listed in Table 2
were then randomlyorderedand paired with a Likert scale to form the
initial CPRS.

This content downloaded from 194.29.185.227 on Sun, 22 Jun 2014 05:34:41 AM

All use subject to JSTOR Terms and Conditions
 Cooksey/105

Reliabilityand Validity
To test the stability of the eight-factor CPRS structure and to gain
initial data on its inter-judge reliability and criterion-related validity,
four groups of judges were asked to rate three sets of ten different re-
corded high school choral performances using the CPRS instrument.
Table 3 presents the CPRS evaluation form used by the adjudicators.
Each of the first three groups of judges rated a different set of per-
formances, and the fourth group (students) rated the third set of per-
formances. The data were factor analyzed using the Dixon X72 and the
SOUPAC principal factor factor analysis programs. The student data
from group 4 were not included in this phase of the study; the same gen-
eral criteria employed in selecting and extracting factors for the initial
CPRS again were employed, with one major exception. The CPRS eight-
factor structure was used as the primary frame of reference for the factor
structure interpretation instead of the NIMAC categories.
The factor analysis of the results of the CPRS evaluations confirmed
seven of the eight factors included in the initial CPRS structure: diction,
precision, dynamics, tone control, tempo, balance, and interpretation/
musical effect. Thirty-six out of the original thirty-nine items comprising
the initial CPRS were selected to form subscales for the seven factors.
Since the unity factor, included in the factor structure of the initial
CPRS, was not confirmed as a separate factor in the second application
of the factor analysis technique, the five items defining that factor either
had to be dropped or included under some of the other seven factor
categories of the revised CPRS. Since two of the five items for unity had
high loadings on factors already confirmed, they were added to the sub-
scales of those particular factors (balance and interpretation/musical
effect). The name change for the balance factor (from balance to bal-
ance/blend) was made because the new item added to the subscale
measuring that factor dealt primarily with the blend aspect of choral
performance. The remaining three items of the unity factor did not
achieve any significant loadings across the seven-factor structure of
choral performance and hence were dropped. Table 4 contains the
Varimax factor loadings for items selected for the revised CPRS.
The results of the administration of the CPRS also were used to pro-
vide an inter-judge reliability estimate for the rating scales produced in
the revised CPRS. Reliability coefficients were obtained for subscale as
well as total scores for the four groups of judges rating the three sets of
choral performances. The Hoyt analysis of variance procedure was em-
ployed.13 The revised CPRS achieved high inter-judge reliability across
the four groups of adjudicators. Reliability coefficients for the total re-

13Joy P. Guilford, Psychometric Methods, second edition (New York: McGraw-Hill

Book Co., Inc., 1954), p. 383.

This content downloaded from 194.29.185.227 on Sun, 22 Jun 2014 05:34:41 AM

All use subject to JSTOR Terms and Conditions
106/JRME
Table2
VarimaxFactorLoadingsfor ItemsSelectedforthe CPRS*

04)
oo
0tc
--
Q Q) o o.-
&-
13.0 O
._ O
o cE o
E
cl
i,.-
0-
-CE a. i. • 0)r• p
Items1I IV V VI VII VI
Items I I III IV V VI VII VIII
A. Diction
1. Articulation was clear
and precise .375 .378 .550
2. Words clearly under-
standable .721
3. The diction of this
group is excellent .316 .638
4. Diction is muddy .755
5. Initial consonants need
more emphasis .618

B. Unity
1. Excellent unity of style .450 .605
2. Overall effect is choppy
and over-sectionalized .382 .606
3. Too much emphasis on
unimportant words and
syllables .632
4. Excellent feeling of
ensemble .436 .572
5. Good overall blend of
all parts .386 .537

C. Balance
1. Top voices cover up
lower voices .819
2. Men's voices (Sop. II/
Alto for Women's Ch.)
balance the choir very
well .768
3. Excellent balance
between all parts .742
4. Inner parts balance the
outer voices very well .647
5. Lowest part balances
upper parts very well .669

D. Precision
1. Sloppy rhythms .334 .610
2. All part entrances are
very precise .301 .692
3. Attacks and releases of
many notes are imprecise .321 .562

This content downloaded from 194.29.185.227 on Sun, 22 Jun 2014 05:34:41 AM

All use subject to JSTOR Terms and Conditions
 Cooksey/107
4. Some poor entrances by
different parts .724
5. Attacks are consistently
weak .514 .427
E. Tempo
1. Tempo unsteady in some
sections .596
2. Excellent control of
tempo .425 .592
3. Tendency to rush the
tempo .636
4. Unsteady rhythmic
sections .482 .537
F. Dynamics
1. Needs wider dynamic
contrasts .630
2. Lovely changes in
dynamics .373 .685
3. Delicate, expressive
shading in dynamics .467 .574
4. Excellent use of "pp" .514
5. Dynamics handled well
in relation to phrase
development .398 .494
G. Tone Control
1. The tone quality is too
harsh in forte passages .678
2. Sopranos (Ten. I for
Men's Ch.) sound forced
in upper pitch and dy-
namic ranges .741
3. The tone quality is often
forced in this choir .655
4. Excellent control of in-
tonation at forte levels .396 .644
5. Intonation in all parts
excellent throughout .438 .545
H. Interpretation/Musical Effect
1. Performance exhibits the
proper stylistic interpre-
tation .674
2. This choir projects the
mood of the selection
very well .843
3. Emotional concept of
word meanings very
well expressed .740
4. A musical and artistic
effect... fluid and vital .663 .363
5. Excellent forward
rhythmic drive .714
* All item
loadings above .30 included across factors.

This content downloaded from 194.29.185.227 on Sun, 22 Jun 2014 05:34:41 AM

All use subject to JSTOR Terms and Conditions
108/JRME
Table 3
The ChoralPerformanceRatingScale
The purpose of the following questions is to have you as accurately as possible
describe the performance which you have just heard. Respond to each statement
on the basis of how much you agree or disagree that the statement is descriptive
of the performance. Use the following five-point scale:
SD - Strongly disagree that the statement is descriptive
D - Disagree that the statement is descriptive
NN- Neither disagree nor agree that the statement is descriptive
A - Agree that the statement is descriptive
SA - Strongly agree that the statement is descriptive
+ sign indicates positive statement; - sign indicates negative statement.

Please choose only one response to each question. Please attempt to answer every
question. Circle responses.
SD D NN A SA 1. Excellent forward rhythmic drive +
SD D NN A SA 2. Attacks and releases of many notes are imprecise -
SD D NN A SA 3. Excellent unity of style +
SD D NN A SA 4. Dynamics handled well in relation to phrase develop-
ment +
SD D NN A SA 5. Lowest part balances upper parts very well +
SD D NN A SA 6. Excellent use of "pp" +
SD D NN A SA 7. Tendency to rush the tempo -
SD D NN A SA 8. Excellent feeling of ensemble +
SD D NN A SA 9. The tone quality is too harsh in forte passages -
SD D NN A SA 10. Lovely changes in dynamics +
SD D NN A SA 11. Sloppy rhythms -
SD D NN A SA 12. All part entrances are very precise +

vised CPRS scores for the first three groups were above .98. The student
group of judges achieved a reliability estimate of .97. Inter-judge reli-
ability coefficients for each of the subscales of the revised CPRS were
generally above .95 for the first three groups of judges, and above .92
for the student group. Table 5 contains the inter-judge reliability esti-
mates for the revised CPRS subscale scores.
To estimate inter-judge reliability for adjudication panels of different
sizes, a generalized Spearman-Brown prophecy formula was applied to
the reliability estimates of the total and subscale revised CPRS scores.14
For the first three groups of judges, reliability estimates for the total
revised CPRS scores were above .84 for as few as two judges; the reli-
ability estimate for the student group was .76 for as few as two judges.
The subscale reliability coefficients for the first three groups were above
.72 for as few as three judges, and for the student group were above .70
for three judges.
To examine the criterion-related validity of the revised CPRS, two
14Harold Gulliksen, Theory of Mental Tests
(New York: John Wiley and Sons, 1950).

This content downloaded from 194.29.185.227 on Sun, 22 Jun 2014 05:34:41 AM

All use subject to JSTOR Terms and Conditions
 Cooksey/109
SD D NN A SA 13. Unsteady rhythmic sections -
SD D NN A SA 14. Needs wider dynamic contrasts -
SD D NN A SA 15. The tone quality is often "forced" in this choir -
SD D NN A SA 16. Words clearly understandable +
SD D NN A SA 17. Top voices cover up lower voices -
SD D NN A SA 18. A musical and artistic effect...fluid and vital +
SD D NN A SA 19. Good overall blend of all parts +
SD D NN A SA 20. Initial consonants need more emphasis -
SD D NN A SA 21. This choir projects the mood of the selection very well +
SD D NN A SA 22. Excellent control of tempo +
SD D NN A SA 23. Excellent control of intonation at forte levels +
SD D NN A SA 24. Overall effect is choppy and over-sectionalized -
SD D NN A SA 25. Emotional concept of word meanings very well ex-
pressed +
SD D NN A SA 26. Excellent balance between all parts +
SD D NN A SA 27. Attacks are consistently weak -
SD D NN A SA 28. Performance exhibits the proper stylistic interpretation +
SD D NN A SA 29. Articulation was clear and precise +
SD D NN A SA 30. Men's voices (Sop. 2/Alto for Women's Ch.) balance
the choir very well +
SD D NN A SA 31. Delicate, expressive shading in dynamics +
SD D NN A SA 32. Diction is muddy -
SD D NN A SA 33. Too much emphasis on unimportant words and syl-
lables -
SD D NN A SA 34. Tempo unsteady in some sections -
SD D NN A SA 35. Intonation in all parts excellent throughout +
SD D NN A SA 36. Some poor entrances by different parts -
SD D NN A SA 37. Soprano (Ten. 1 for Men's Ch.) sound forced in upper
pitch and dynamic ranges -
SD D NN A SA 38. Inner parts balance the outer voices very well +
SD D NN A SA 39. The diction of this group is excellent +

studies were conducted. In the first study a paired-comparison procedure

was used. The ten performances in each of the three sets used for the
CPRS evaluations and the inter-judge reliability analysis were given a
global rating by three groups of judges who had not participated pre-
viously in the study. Inter-judge reliability estimates were obtained using
the Hoyt analysis of variance procedure. The resultant reliability coeffi-
cients were above .98 for each of the three groups of judges. To provide
a measure of the degree of agreement among the judges concerning the
ranking of the ten performances in each set, Kendall's coefficient of
concordance, W, was used.15 Concordance coefficients above .80 were
produced within the three sets of performances.
To compare the results of the revised CPRS evaluations of each of the
three performance sets to the global performance ratings, the zero-order
correlation coefficients between the revised CPRS total scores, subscale
scores, and both the rank order and the normalized scale scores for the
global ratings were obtained. Correlation coefficients between the total
15Fred N. Kerlinger, Foundations of Behavioral Research, second edition (New York:
Holt, Rinehart and Winston, Inc., 1973), p. 283.

This content downloaded from 194.29.185.227 on Sun, 22 Jun 2014 05:34:41 AM

All use subject to JSTOR Terms and Conditions
110/JRME
Table4
VarimaxFactorLoadingsfor Items Selected for the Revised CPRS

000 o 0-
o C- Q-
? E-
o cno E .-
a oC
U I
C
>%
C
0
E
0 V
o a in c

Items 1 11 III IV V VI VII

A. Diction
1. Articulation was clear
and precise .678
2. Words clearly under-
standable .776
3. The diction of this group
is excellent .780
4. Diction is muddy .789
5. Initial consonants need
more emphasis .589

B. Precision
1. Sloppy rhythms .334 .376 .306 .389
2. All part entrances are
very precise .338 .601
3. Attacks and releases of
many notes are imprecise .343 .498
4. Some poor entrances by
different parts .327 .585
5. Attacks are consistently
weak .371 .488 .304

C. Dynamics
1. Needs wider dynamic
contrasts .744
2. Lovely changes in dy-
namics .737
3. Delicate, expressive shad-
ing in dynamics .665 .321
4. Excellent use of "pp" .716
5. Dynamics handled well
in relation to phrase de-
velopment .578 .382

D. Tone Control
1. The tone quality is too
harsh in forte passages .657 .316
2. Soprano (Ten. I for
Men's Ch.) sound forced
in upper pitch and dy-
namic ranges .577 .351

This content downloaded from 194.29.185.227 on Sun, 22 Jun 2014 05:34:41 AM

All use subject to JSTOR Terms and Conditions
 Cooksey/111
3. The tone quality is often
forced in this choir .688
4. Excellent control of into-
nation at forte levels .520 .378 .343
5. Intonation in all parts
excellent throughout .466 .385 .345

E. Tempo
1. Tempo unsteady in some
sections .702
2. Excellent control of
tempo .679 .414
3. Tendency to rush the
tempo .593
4. Unsteady rhythmic sec-
tions .344 .577

F. Balance/Blend
1. Top voices cover up lower
voices .647
2. Men's voices (Sop. II/
Alto for Women's Ch.)
balance the choir very
well .743
3. Excellent balance between
all parts .321 .660
4. Inner parts balance the
outer voices very well .654
5. Lowest part balances up-
per parts very well .735
6. Good overall blend of all
parts .377 .525 .407

G. Interpretation/Musical Effect
1. Performance exhibits the
proper stylistic interpre-
tation .346 .685
2. This choir projects the
mood of the selection
very well .319 .741
3. Emotional concept of
word meanings very well
expressed .304 .348 .623
4. A musical and artistic
effect...fluid and vital .323 .353 .322 .315 .552
5. Excellent forward rhyth-
mic drive .629
6. Excellent unity of style .547

This content downloaded from 194.29.185.227 on Sun, 22 Jun 2014 05:34:41 AM

All use subject to JSTOR Terms and Conditions
112/JRME
Table 5
Inter-JudgeReliabilityEstimates for the Revised CPRSSubscale
Scores
Group4
Group1 Group2 Group3 (students)
Subscale Categories (N = 20) (N = 20) (N = 20 (N = 20)
Diction .951 .972 .949 .879
Precision .971 .976 .959 .935
Dynamics .961 .967 .974 .952
Tone Control .984 .980 .975 .959
Tempo .953 .957 .935 .923
Balance/Blend .976 .964 .971 .933
Interpretation/MusicalEffect .980 .971 .976 .950

scores for the revised CPRS evaluations in each set and both ranks and Z
scores for the global performance criterion were generally above .80 for
the first three groups of judges, and above .70 for the student group of
judges. Correlation coefficients between the subscale scores and the ranks
and Z scores ranged from .623 to .923 for the first three groups of judges,
and from .442 to .901 for the student judges.
To examine the contributions of the subscale scores of the revised
CPRS in predicting the global criterion, a step-wise multiple regression
analysis was performed on the data generated by the revised CPRS. The
subscale scores for each of the four groups of judges who used the CPRS
in evaluating the three sets of choral performances served as the inde-
pendent variables in predicting the global performance rating criterion.
The corrected R2's for the revised CPRS subscale scores of the four
groups of judges for performance sets I-III and the criterion of ranks
ranged from .630 to .993, and from .787 to .955 for the criterion of
normalized scores. Since the results of the multiple regression analysis
showed R2's which were highly inflated, multiple R coefficients were
computed from a pooled within-groups correlation matrix. The values
of the two criteria were .874 (for ranks) and .876 (for Z scores). The sub-
scale scores of the revised CPRS for the four groups of judges therefore
produced substantial evidence of success in predicting the global per-
formance rating criterion.
Finally, to test the stability of the weights for the subscale scores for
each of the four groups of judges generated by the step-wise multiple
regression across the three sets of performances, the weights for each
group of subscale scores were applied to the subscale scores of the other
three sets. The data indicated that the weights for sets I, III, and IV were

This content downloaded from 194.29.185.227 on Sun, 22 Jun 2014 05:34:41 AM

All use subject to JSTOR Terms and Conditions
 Cooksey/113
comparable, but the weights for set II produced poor results when
applied to the other three sets.
In the final study concerning the criterion-related validity of the re-
vised CPRS, a new group of twenty judges (group 5) used the NIMAC
adjudication scale in evaluating set III of the high school choral per-
formances, previously judged by groups 3 and 4 using the CPRS. The
same procedures used for comparing the revised CPRS scores and the
global performance ratings were applied. The results of this test were:
(1) The inter-judge reliability estimates for the NIMAC total and
subscale scores of group 5 were above .94.
(2) The zero-order correlation coefficients between the revised CPRS
total scores, subscale scores (generated by the ratings of the judges in
groups 3 and 4 on the set III performances), and the NIMAC criterion
were generally above .85.
(3) A step-wise multiple regression analysis for the revised CPRS sub-
scale scores (generated by the ratings of the judges in groups 3 and 4 on
the set II performances) and the NIMAC scale criterion produced sub-
stantial evidence (R2 = .970 for group 3 subscale scores: group 5 NIMAC
total scores; R2 = .900 for group 4 subscale scores: group 5 NIMAC
total scores) that the revised CPRS served as a good predictor for the
criterion.

Summaryand Conclusions
(1) The facet-factorial approach produced a seven-factor structure of
choral performance. These factors were diction, precision, dynamics, tone
control, tempo, balance/blend, and interpretation/musical effect.
(2) Thirty-six items-dimensions were selected to form the subscales to
measure the seven factors.
(3) The revised CPRS achieved high inter-judge reliability.
(4) The revised CPRS achieved high criterion-related validity when a
global performance rating was used as the criterion for comparison.
(5) The revised CPRS achieved high criterion-related validity when
the NIMAC scale was used as the criterion for comparison.
These results suggest that some of the difficulties involved in measur-
ing choral performance achievement can be overcome. Some of the basic
components of choral performance were successfully identified by factor
analysis. These components were defined by the items selected to measure
them. Since the selection of these items rested upon musical, statistical,
and empirical considerations, a significant degree of objectivity was
maintained in both the selection and definition of the essential criteria
needed to develop a structure of choral performance. Some of the items
selected for the revised CPRS included "expressive" aspects of choral
performance as well as the more easily measurable technical items.

This content downloaded from 194.29.185.227 on Sun, 22 Jun 2014 05:34:41 AM

All use subject to JSTOR Terms and Conditions
114/JRME

Objectivity in choral performance adjudication was increased with the

use of the revised CPRS because adjudicators using the instrument
shared a common set of criteria. In each group of judges, including the
student evaluators, reliability coefficients were very high for both sub-
scale and total revised CPRS scores. The scale thus demonstrated utility
for both students and choral directors in public schools.
The revised CPRS also proved effective in measuring "musicality" in
choral performance. When a global performance rating was used as the
criterion for comparison, validity coefficients of the revised CPRS were
quite high. When the subscale scores of the revised CPRS were used as
the independent variables in predicting the global performance rating
criterion, substantial R2's were achieved.

CaliforniaStateUniversity
Fullerton,California

This content downloaded from 194.29.185.227 on Sun, 22 Jun 2014 05:34:41 AM

All use subject to JSTOR Terms and Conditions