EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

1978.38

GENERALIZABILITY IN FACTORABLE DOMAINS:

"DOMAIN VALIDITY AND GENERALIZABILITY"
1

RODERICK P. McDONALD
The Ontario Institute for Studies in Education

It is shown that if a behavior domain can be described by the

common factor model with a finite number of factors, the squared
correlation between the sum of a selection of items and the domain
total score is actually greater than coefficient alpha. Equality is
attained only if the selected items are parallel. Generalizability can
be correctly assessed as a function of the item uniquenesses and the
test variance.

ARGUMENTS have been presented by Tryon (1957),Cronbach, Raja-

ratnam and Gleser (1963), and, most recently, by Kaiser and Michael
(1975), for regarding coefficient alpha, the generalized version of the
Kuder-Richardson formula twenty for the reliability of a total test
score, as a measure of generalizability from a selection of items to the
behavior domain from which they are deemed to be selected. In these
treatments, the behavior domain is undefined, and the selection of
items is not assumed to be random, yet, counterintuitively, a definite
result is obtained, namely alpha, for the square of the correlation
between the sum of the arbitrarily selected items and the sum of the
remaining items in the undefined domain.
The assumptions that enable this result to be reached are difficult to
evaluate, essentially because nothing in the theory serves to define the
domain or the manner of selection from it.
In view of the work of Guttman (1955) and Kaiser and Caffrey
1
This paperwas written while the author was a Visiting Professor in the Department
of Psychology, University College London.
2
The proof of this inequality is a variant on one given by McDonald (1970), which in
turn follows, in part, a proof of a slightly different inequality given by Novick and Lewis
(1967).

75

Downloaded from epm.sagepub.com at UNIV OF PITTSBURGH on June 2, 2015

76 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

(1965), at least, it should seem reasonable to suppose that some

behavior domains (though not necessarily all) can be described by the
common factor model, with a finite number of common factors. The

object of this note is to derive a generalizability coefficient for &dquo;facto-

rable&dquo; behavior domains, and compare it with alpha, to check the
consistency of theory in this area.
Suppose we have a selection of p items, not in general dichotomous,
from a behavior domain. Let jCi, ’’’, xp be the item scores on these,
with a sum score

Let ~i, ’’’, y, be item scores on the remaining q items in the domain,
with sum score

The squared correlation between x and y is given by

where

(Note that the last two sums include the diagonal variance terms.)
Let us now suppose that the behavior domain is explained by just
one factor, i.e., we have a Spearman case, or factorially homogeneous
case. (Shortly we consider multiple factors.) Let the factor loadings
and uniquenesses of ~i, - - -, xp be /i, - - -, fp, and U12, ..., Up2, and let
those of ~, &dquo; -, y, be gi, - &dquo;, gQ, and W12, ..., wQ2. Then

Downloaded from epm.sagepub.com at UNIV OF PITTSBURGH on June 2, 2015

 RODERICK P. McDONALD 77

Hence (3) becomes

or, what is the same thing,

Since

exists, (7) yields the limit

Alternatively, (8) may be written as

Note that no assumption has been made concerning the relation

between covariances or factor loadings in the selection, and those in
the domain. The sole assumption is that an infinite domain is explain-
able by one factor.
More generally, let us now suppose that r common factors explain
the covariances in the domain. We extend the notation in an obvious
way, writing fll, gsl, for the loadings of Xj, Ys, on the lth factor. In this
more general case we require an additional assumption that in effect
makes the selection representative, in respect of each factor. We sup-
pose that the average factor loading is the same in the selection and the

Downloaded from epm.sagepub.com at UNIV OF PITTSBURGH on June 2, 2015

78 EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT

domain, for each factor in turn, i.e.

Substitution of the more general counterparts of (5) in (3) yields, with

(10), the result

may be rewritten to be the same as (9).

Alternatively, (11)
We denote the limit (11) by W. Coefficient alpha is well known to be
defined by

where the ranges of j and k are 1, ’’’, p. We now show that

and equality is attained if and only if there is only one factor, and all
factor loadings are the same.2
Substituting factor loadings and uniquenesses in the covariances in
(12), together with (11), yields

Using the mathematical identity, for any numbers ~i, ’’’, bp,

we obtain from (14),

.

The expression on the right in (16) is a sum of squares of real numbers,

hence we see that ~ ~ a. Equality is attained if and only ifs flel for =

Downloaded from epm.sagepub.com at UNIV OF PITTSBURGH on June 2, 2015

 RODERICK P. McDONALD 79

every j, k, and l. But this can only be true if r = 1, and /n =

f2l = ... =

fp¡, or, an equivalent, the items are parallel in the sense that all
covariances of items are the same.
We see, then, that if a behavior domain is explainable by a finite
number of common factors, coefficient alpha is in general less than the
coefficient of generalizability, given by the squared correlation be-
tween selection score and domain score. It follows also that the as-
sumptions made by Tryon (1957) and by Kaiser and Michael (1975)
cannot be true in such a behavior domain, except under the much
stronger condition of parallelism of items, which they were trying to
free the theory from. But then, such assumptions are unnecessary, if by
factor analysis we can both assess the general character of the behavior
domain, and estimate generalizability by sample counterparts of (9) or
(11).
A more extended account of these and related matters will be given
elsewhere.

REFERENCES

Cronbach, L. J., Rajaratnam, N., and Gleser, G. C. Theory of general-

izability : A liberalization of
reliability theory. British Journal of
Statistical Psychology, 1963, 16, 137-163.
Guttman, L. The determinacy of factor score matrices with implica-
tions for five other basic problems of common-factor theory. Brit-
ish Journal of Statistical Psychology, 1955, 8, 65-82.
Kaiser, H. J. and Caffrey, J. Alpha factor analysis. Psychometrika
,
1965, 30, 1-14.
Kaiser, H. J. and Michael, W. B. Domain validity and generalizability.
EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 1975, 35, 31-
35.
McDonald, R. P. The theoretical foundations of principal factor
canonical factor analysis, and alpha factor analysis. Brit
analysis,
ish Journal of Mathematical and Statistical Psychology, 1970, 23,
1-21.
Novick, M. R. and Lewis, C. Coefficient alpha and the reliability of
, 1967, 32, 1-13.
composite measurements. Psychometrika
Tryon, R. C. Reliability and behavior domain validity: Reformulation
and historical critique. Psychological Bulletin, 1957, 54, 229-249.

Downloaded from epm.sagepub.com at UNIV OF PITTSBURGH on June 2, 2015