Kendall's Coefficient of Concordance for Sociometric Rankings with Self Excluded

Author(s): Gordon H. Lewis and Richard G. Johnson

Source: Sociometry , Dec., 1971, Vol. 34, No. 4 (Dec., 1971), pp. 496-503
Published by: American Sociological Association

Stable URL: https://www.jstor.org/stable/2786195

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide
range of content in a trusted digital archive. We use information technology and tools to increase productivity and
facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at
https://about.jstor.org/terms

American Sociological Association is collaborating with JSTOR to digitize, preserve and extend
access to Sociometry

This content downloaded from

59.100.1.159 on Mon, 18 Jan 2021 05:30:35 UTC
All use subject to https://about.jstor.org/terms
Sociometry
1971, Vol. 34, No. 4, 496-503

Kendall's Coefficient of Concordance for

Sociometric Rankings with Self Excluded*
GORDON H. LEWIS AND RICHARD G. JOHNSON
Carnegie-Mellon University

Evaluating the extent of agreement among a set of judges each of whom has
ranked all of the members of a group except for himself is a common problem
in sociometric studies. Although Kendall's extension of the Coefficient of Con-
cordance, W, to Youden arrays is available as a measure of agreement in suck
a case, critical values of W have been given only by approximation and only
for large n. The current paper reports critical values of S for small n (4-10).
In addition, the paper demonstrates how Kendall's Coefficient of Concordance
can be used to test more specific hypotheses about agreement on the criteria
of choice within a group. This suggested extension of W applies not only to
the special case of square data matrices with zero major diagonals but to data
matrices in general.

Kendall (1955) presented a Coefficient of Concordance, W, to evaluate the

extent of agreement among a set of judges each of whom ranks in entirety a
set of objects; this statistic is well known and has been widely used in the
sociometric literature. Kendall also presented an extension of W to handle
certain cases of incomplete rankings, rankings of elements presented in an
incomplete Latin square, or Youden array; this extension has wide ap-
plicability but appears to be unused.
A special case of Youden arrays occurs when each member of a group is
asked to rank each of the other members of the group, excluding himself.
This type of question has been used by many investigators (e.g., Bock, 1955;
Bock and Husain, 1952; Goldhammer, n.d.; Morton, 1959; Newcomb, 1961;
Weisman, 1968). In addition, data appropriate for analysis by the presently-
discussed technique have been reported by many others (e.g., Bales, 1953;
Bonacich, 1968; Rehagen, 1948; Slater, 1955; Theodorson, 1957). Although
Kendall's extension of W is appropriate in these cases, to the best of our
knowledge, only once has it ever been used (Jones, 1959). The lack of usage
may have resulted from a lack of familiarity with the extension of W to this
case, or from the fact that in this case the problem of testing significance of
W was considered only for "large n." Unlike the case of complete rankings,

* Revision of a paper presented at the annual meeting of the Pacific Sociological As-
sociation, Honolulu, Hawaii, April 1971.
496

This content downloaded from

59.100.1.159 on Mon, 18 Jan 2021 05:30:35 UTC
All use subject to https://about.jstor.org/terms
 KENDALL'S COEFFICIENT FOR SOCIOMETRIC RANKINGS 497

Kendall did not report critical values of S (the statistic on which W

for small n, and in a great many sociometric studies the number of judges
is small, quite frequently fewer than 10. The present paper presents critical
values of S for small (n = 4... 10) ranking matrices with zero major
diagonals. The results are based on Monte Carlo techniques.

Kendall's Coefficient of Concordance

Let R be an n x m matrix in which rij is the rank of the j-th of m objects as

judged by the i-th of n judges. Then

W = 12S/m2n (n2 - 1)

where S is the sum of the squared deviations of the column sums from the
mean column sum,

S E(rj - r /M) 2
In the case where the objects to be ranked are the judges themselves, n = m
and

W= 12S/n3 (n2- 1)

As applied to ranking matrices, Youden arrays are characterized by the

fact that each pair of objects to be ranked appears the same number of times.
In this case

W= 12S/X2n (n2 - 1)

where X is the number of times that a given comparison occurs. When each
member of a group ranks all of the members except himself, each pair of
members is ranked n - 2 times. This is obvious from the fact that the (ij)
pair is omitted by person i and person j but by no one else. In this case,
A = n - 2 and

W= 12S/n(n - 2)2 (n2- 1) (1)

The data matrix, R, corresponding to (1) is characterized by zeros on the

major diagonal. The following results for the sampling distribution of S
pertain to that case.

Critical Values of S

The sampling distribution of S was estimated for group sizes 4 through 10

by Monte Carlo techniques. The simulation assumes that each of n judges
ranks the n - 1 others in a random fashion.1 Results from the simulation

1 A FORTRAN program was written that did the following: a random generator filled
the n(n-1) off-diagonal cells of a given data matrix, in each row the off-diagonal entries

This content downloaded from

59.100.1.159 on Mon, 18 Jan 2021 05:30:35 UTC
All use subject to https://about.jstor.org/terms
 498 SOCIOMETRY

(based on 50,000 data matrices per group size), together with maximum
values of S are shown in Table 1. Thus, under the null hypothesis that the
judges have ranked the objects at random, with seven judges Pr(S t 396)
9 .001.

TABLE 1

Values of S Associated with A Given Probability and A Given Number of Judges

Pr(5SX)C 4 5 6 7 8 9 10

.990 2 4 10 22 44 88 146
.980 4 4 12 26 54 104 178
.900 4 10 22 50 96 176 292
.500 8 24 56 114 208 358 566
.250 14 36 82 164 288 488 758
.100 16 48 108 214 372 624 956
.050 .. 56 126 246 428 712 1094
.025 20 62 142 276 480 796 1214
.010 .. 68 158 312 542 900 1363
.005 .. 74 170 338 586 966 1462
.001 .. 80 194 396 686 1106 1710
Smax 20 90 280 700 1512 2940 5280

To test the significance of W, K

large n:

X2_A (n2 _1 ) W/ (k + 1 )

with n - 1 degrees of freedom. The parameter k is the number of objects

presented at one time. In the present instance k = n - 1. Simple substitu-
tion yields

><=-12S/n2(n -2) (2)

The differences between the approximation formula (2) and the estimated
sampling distribution based on Monte Carlo results are given in Table 2,
for n = 10. The largest difference in Table 2 occurs for S = 292 where the
probability given by the simulation is .8986 and by the approximation
formula is .8836.

Comments

The purpose of the present paper is both to call attention to the extension
of the Coefficient of Concordance to data matrices with missing entries, and

were replaced by their rank, the column sums were calculated, S was computed, and
finally the frequency distribution for S was compiled. The program was run on the
Univac 1108 at Carnegie-Mellon University.

This content downloaded from

59.100.1.159 on Mon, 18 Jan 2021 05:30:35 UTC
All use subject to https://about.jstor.org/terms
 K1ENDALL'S COEFFICIENT FOR SOCIOMETRIC RANKINGS 499

TABLE 2

Differences between Results from Simulation and Kendall's Approximation, N

Pr(S X) Based On

X (1) Simulation (2) Kendall's Approx. ()-(2)

146 .9900 .9879 .0021

178 .9795 .9762 .0033
292 .8986 .8836 .0150
566 .4991 .4853 .0138
758 .2493 .2521 -.0028
956 .1000 .1110 -.0110
1094 .0498 .0589 -.0091
1214 .0248 .0329 -.0081
1362 .0099 .0157 -.0058
1462 .0050 .0091 -.0041
1710 .0010 .0023 -.0013

to present the critical values for that statistic in the case where the major
diagonal is zero. Since it might appear that the extended form of the statistic
is severely limited in application, some comments are perhaps in order.
First, although the data collected may yield a complete data matrix, it is
not always the case that one wants to apply Kendall's Coefficient of Con-
cordance in its familiar form, that is, in the form appropriate to complete
data matrices. In many situations it may be more appropriate to eliminate the
judge's evaluation of himself in attempting to assess the extent to which
judges agree in their evaluations of the other members of a group. In most
situations, the judge's information about himself probably differs significantly
in both quantity and quality from the information he holds about others.
Jones (1959) has discussed this case in some detail.
Second, typical uses of Kendall's Coefficient of Concordance take the items
being judged as fixed and test the null hypothesis that the similarities in the
rankings of the objects could have been produced by chance. The alternative
hypothesis is that the judges are applying a common or similar standard of
judgment. There is an important case, however, where researchers would fail
to reject the null hypothesis even though the alternative hypothesis is true,
unless the researcher is careful in the test he performs. This can easily occur
if the criterion of evaluation is a function not simply of the items being
judged but is a function of the characteristics of the items and the judges
combined. Perhaps the point can be made most quickly by a short example.
Homans (1961) has suggested that if a group is characterized by a
prestige hierarchy (which we shall assume for present purposes is a strong
ordering) then, if the judges are asked to rank the other members of the
group according to their "preference" for them, the i-th judge will rank the

This content downloaded from

59.100.1.159 on Mon, 18 Jan 2021 05:30:35 UTC
All use subject to https://about.jstor.org/terms
 500 SOCIOMETRY

other members of the group in the following order: i - 1, i + 1, i - 2,

i + 2, . . . This hypothesis implies that in a five-person group one would
observe the matrix in Table 3. Although an observed pattern of preferences
such as shown in Table 3 would be in perfect agreement with the hypothesis,
application of Kendall's Coefficient of Concordance to Table 3 would not lead
to rejection of the null hypothesis of random orderings (W = .38; p > .25).
However, the matrix can be rearranged so that the item represented by the
first column is the item expected to be ranked first by the i-th judge. If
the cells of Table 3 are rearranged so that they proceed across a row accord-

TABLE 3

Expected Rankings by Judges Where the Judges Are Ordered by Prestige in the Group-
Homans' Hypothesis

Person Ranked
1 2 3 4 5
1l-l [ 1 2 3 4
2 12 3 4

Judge 3 3 1 2 4

4 4 3 1 2

5 4 3 2 1

ing to the hypothesized preferences of the i-th judge, then the resulting table,
if Homans' hypothesis is correct, will result in a table in which the data of
the i-th row will contain the vector (1, 2, . . . , n - 1), with 0 as the i-th
element. Application of Kendall's Coefficient of Concordance to the trans-
formed table is appropriate. One would reject Homans' hypothesis either if
the results of the statistical test were consistently non-significant or if the
results were significant but not in the "direction" of the hypothesis. Direction
in this case refers to a monotonic increase of the column sums.
Table 4 shows the application of W to a hypothetical matrix and to the
same matrix after it has been transformed to test Homans' hypothesis.2
Whereas the untransformed data do not show substantial departure from
results which might be obtained in the absence of a common criterion of
judgment, the transformed matrix does. Secondly, and equally important, the
column sums increase in the manner suggested by the hypothesis.

2 To obtain the transformed matrix, rearrange the observed data matrix as follows: for
j <i, place in the (i,j) cell the observed rank of the person expected to be ranked j-th
by the i-th judge; for j = i, enter zero in the (i,j) cell; and for j > i, place in the (i,j)
cell the observed rank of the person expected to be ranked j - 1.

This content downloaded from

59.100.1.159 on Mon, 18 Jan 2021 05:30:35 UTC
All use subject to https://about.jstor.org/terms
 TABLE 4

Application of Kendall's W to a Transformed Matrix *

Original Matrix Transf

1 2 3 4 5

I 1?-0 2 1 4 3 1 0
2 1 0 3 2 4 2 1 0

3 3 1 0 2 4 3 1 2
4 3 1 0 2 4 1 2

4 3 1 2 0 5 2 1
12 9 6 10 13 5 7 1

S=30 S
W = .33 W =
p> .25 p=.0

* Transfor

This content downloaded from

59.100.1.159 on Mon, 18 Jan 2021 05:30:35 UTC
All use subject to https://about.jstor.org/terms
 502 SOCIOMETRY

The generality of the suggested usage of Kendall's Coefficient of Con-

cordance to test hypotheses which specify different standards of evaluation
for specific judges is not limited to cases where each judge has a separate
preference ordering or choice rule. It extends to any partitioning of the set of
judges such that to each element, to each judge, there is a specified preference
ordering over the set of objects. The case of all judges having the same
preference ordering and the case of each individual judge having a different
preference ordering are merely special cases.
The previous comments on a more extensive use of Kendall's Coefficient
of Concordance also apply mutatis mutandis for the case where the judges
rank all of the objects rather than a subset of them. Instead of being used to
test merely whether a set of judges agrees on the ranking of a set of objects,
Kendall's Coefficient of Concordance can be used to test whether judges are
employing specified criteria or choice rules. This may or may not require
transformation of the data matrix, depending on the hypothesis tested.

REFERENCES

Bales, Robert F.
1953 "The equilibrium problem in small groups." Pp. 111-161 in T. Parsons,
R. F. Bales and E. A. Shils, Working Papers in the Theory of Action. New
York: Free Press.
Bock, R. Darrell
1955 Unpublished study of a ninth grade science class, Hyde Park High School,
Chicago.
Bock, R. Darrell and S. Z. Husain
1952 "Factors of the tele: A preliminary report." Sociometry 15:206-219.
Bonacich, Phillip
1968 Specialization and Differentiation in Small Laboratory Groups. Ph.D. Dis-
sertation: Harvard University.
Burke, Peter J.
1967 "The development of task and social-emotional role differentiation."
Sociometry 30 (December) :379-392.
Goldhammer, D.
n.d. Unpublished study of discussion groups in the Great Lakes Naval Training
Station.
Homans, George C.
1961 Social Behavior: Its Elementary Forms. New York: Harcourt, Brace and
World.
Jones, James A.
1959 "An index of consensus on rankings in small groups." American Sociological
Review 24:533-537.
Kendall, Maurice G.
1955 Rank Correlation Methods. New York: Hafner.
Morton, Anton S.
1959 Similarity as a Determinant of Friendship: A Multi-Dimensional Study.
Princeton University and Educational Testing Service.

This content downloaded from

59.100.1.159 on Mon, 18 Jan 2021 05:30:35 UTC
All use subject to https://about.jstor.org/terms
 KENDALL'S COEFFICIENT FOR SOCIOMETRIC RANKINGS 503

Newcomb, Theodore M.
1961 The Acquaintance Process. New York: Holt, Rinehart and Winston.
Rehagen, Kenneth J.
1948 A Comparison of Pupil-Teacher Planning and Teacher-Directed Procedures
in Eighth Grade Social Studies Classes. Ph.D. Dissertation, University of
California, Department of Education.
Slater, Philip E.
1955 "Role differentiation in small groups." American Sociological Review 20:
300-310.
Theodorson, G. A.
1957 "The relationship between leadership and popularity roles in small groups."
American Sociological Review 22:58-67.
Weisman, Thomas
1968 "Liking and participating in small task-oriented problem-solving groups."
A.B. honors thesis, Yale University.

This content downloaded from

59.100.1.159 on Mon, 18 Jan 2021 05:30:35 UTC
All use subject to https://about.jstor.org/terms