54 views

Uploaded by smmlsu

Theta Reliability and Factor Scaling

- Wahyu_Widhiarso_1 (1)
- Dissertation 007
- Developing a Checklist
- Done2
- Mohammad Mansour Fawwaz Al-Khasawneh
- mb0040
- yjmt-23-188
- Measure Criativity Haberland Dacin
- About DISC.pdf
- AJBMR_17_25i1n9a1
- 1-s2.0-S0966636208002646-main.pdf
- udsm-muce-ep300-basic-concepts-emr.docx
- Mathematical Symbols List
- Sternberg
- BE4 Research
- reliabil
- Lampiran 5
- Reliability and validity are central to judgments about the quality of research in the natural sciences and quantitative research in the social sciences.docx
- Chapter 5 Measurement
- 7448-26945-1-PB

You are on page 1of 35

Source: Sociological Methodology, Vol. 5 (1973 - 1974), pp. 17-50

Published by: Wiley

Stable URL: http://www.jstor.org/stable/270831 .

Accessed: 10/02/2015 01:34

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .

http://www.jstor.org/page/info/about/policies/terms.jsp

.

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of

content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms

of scholarship. For more information about JSTOR, please contact support@jstor.org.

Wiley is collaborating with JSTOR to digitize, preserve and extend access to Sociological Methodology.

http://www.jstor.org

All use subject to JSTOR Terms and Conditions

THETA RELIABILITY AND

FACTOR SCALING

David3. Armor

THE RAND CORPORATION

presshis gratitude

toAlbertE. Beatonforpointingouttherelationship be-

tweenthetaand maximumalpha. David Heise, Donald Olivier,and Fred

Mostellerreadearlierdraftsand providedhelpfulcriticisms

and suggestions

forthefinal manuscript.

considerableamount of interestamong sociologistsin the last several

years.' Althoughsociologistshave always lamentedthe problemof reli-

ability in sociologicalmeasurement,until recentlylittle attentionhas

beenpaid to techniquesofreliabilityassessmentand improvementwithin

the sociologicalmethodologyliteratureitself.While much of the recent

1 Throughoutthis chapterthe term

reliabilityrefersto internal-consis-

tancy reliability-as opposed to stabilityreliability-unless otherwisestated

(Heise, 1969).

17

All use subject to JSTOR Terms and Conditions

18 DAVID J. ARMOR

conveyingbasic principles(Upshaw, 1968; Blalock, 1969), in connecting

reliabilityto path analysis(Siegeland Hodge, 1968; Heise, 1969), and in

providingsimplified computationalprocedures(Bohrnstedt,1969).

In spite of this activity,thereare still many technicalproblems

facingthe practitionerof reliabilityassessmentand improvementfor

compositemeasures.Most proceduresforreliabilityassessmentdepend

on the applicationof Cronbach's alpha (Cronbach, 1951), and its im-

provementdependson variousstepsofitemanalysis(whichI laterdefine

as covariancescaling;Upshaw, 1968). Unfortunately thesemethodshave

severaldrawbacksforthe sociologistand perhapsforothersocial scien-

tistsas well: themathematicalassumptionsforalpha reliabilityare often

not met; the usual steps of itemanalysis-throwingout "bad" itemsto

enhancealpha reliability-maynot in factproduceoptimumalpha reli-

ability;and item analysisdoes not include clear and systematicproce-

duresfordetectingand takingintoaccountmultidimensionality-that is,

the presenceof mutuallyindependentsubelustersof items withinthe

total composite.

This chapteraims to solve some of theseproblemsby describing

an approachto reliabilitybased on principal-component factoranalysis.

This approachrestson a different and, forsociologists,a morerealistic

definitionof compositereliability.Moreoverit leads to a little-known

measureof reliabilityI call theta (to distinguishit fromalpha reliabil-

ity) that assesses optimal reliability,and it providesfor a method of

factorscaling that can take account of multidimensionality in a set of

items,therebyenhancingreliabilityand validity.

Beforepresentingthesemethodsit is necessaryto reviewcurrent

alpha reliabilitytechniquesand the associated method of covariance

scaling.In thisway it willbe possibleto illustratemoreclearlythe spe-

cificadvantagesof theta reliabilityand factorscaling.

ALPHA RELIABILITY

an appropriatecoefficienthas been a troublesomestep formanyinvesti-

gators.Althoughthe usual definitionof reliabilityis the simple prod-

uct-momentcorrelationbetween two parallel variables that measure

"identical" things,thereis no singlecoefficientthat has been adopted

universallyforcompositereliability.Guilford(1954) liststencoefficients,

and thisis not an exhaustivelist. Some ofthestandardones he givesare

the Spearman-Brownprophecyformula;the split-halfand odd-even

All use subject to JSTOR Terms and Conditions

THETA RELIABILITY AND FACTOR SCALING 19

Kuder-Richardsonformula20; and Cronbach's alpha. The reason for

thismultiplicity has to do withthe different waysofestimatingtheerror

componentsof a set of items.

Due in partto theconceptualand-computationalsimplicityofthe

method,it was probablythemostcommonlyused coefficient

split-half in

sociologicalresearchbeforethe middle1950s. Since that time,however,

Cronbach'salpha has become the popularmeasureof reliability(Cron-

bach, 1951): it is a generalformulathat subsumesmost of the split-half

and Kuder-Richardson ;2 it has also provedto be a lowerbound

coefficients

to the truereliability(Novick and Lewis, 1967). This lattercharacteris-

tic means that alpha is a conservativeestimateof the reliabilityof a

composite.Basically the alpha coefficient treatseach itemin a composite

as a parallelvariable. Since moderatedeparturesfromthe model can be

expected,the reliabilityforeach itemis estimatedby the averageinter-

itemcorrelation.If we let f be theaveragecorrelationbetweenitemsand

ifwe assumethattheitemsare in standardform(or thattheyhave equal

variances),then Cronbach'salpha is simply

aO=-pr/[l + r(p - 1)] (1)

wordsalpha is identicalto the Spearman-Brownprophecyformulawith

the average correlationf used to estimatethe reliabilityof each item.

Model

The usual model forthe alpha reliabilityformulagivenin Equa-

tion (1) comesfrompsychologicaltest theory(Lord and Novick, 1968).

This modelassumesthat the observedscoreforany itemin a composite

can be partitionedaccordingto "true" and "error" components.For

item i and forany subjectj this model is

wherexij = the observedscoreforitem i, subjectj; Tj = the truescore

forsubjectj; eij = the errorscoreforitemi, subjectj. That is, all items

measurethe subject on an underlyingpropertyTj to an equal extent.It

is normallyfurtherassumedthat the errorcomponentsare uncorrelated

withtruecomponents(forthesame itemas wellas acrossdifferent items)

and witheach other;in otherwords,all itemsare parallelforms.It would

2 Cronbach(1951) provedthat alpha is the mean ofall possiblesplit-half

reliabilities.

All use subject to JSTOR Terms and Conditions

20 DAVID J. ARMOR

correlationswouldbe equal, as would be all populationitemvariances.

Given moderatedeparturesfromthese assumptionsand giventhat the

inter-itemcorrelationsand item variances may not be all equal in a

sample,Equation (1) providesan estimatedreliabilityby standardizing

itemsand by averagingoverobservedcorrelationsto obtainan estimate

of the truepopulationinter-item correlation.

However adequate this model may be forpsychologicaltests,for

manycompositemeasuresin sociologytheseassumptionsare not realis-

tic. Since the contentof each item in a compositeusually differsby a

substantialdegree,particularlyin attitudinalor behavioralcomposites

(such as politicalefficacy,anomie,social class), we mightexpectthat the

true populationcorrelationsare not all equal. For example,in a social

class indexcomposedofmeasuresofincome,education,and occupational

status,one usuallyfindsthat thecorrelations are not all equal; the corre-

lations betweenincomeand the other two variables are usually lower

than the correlationbetweeneducation and occupation-even for the

fullU.S. Census (Siegel and Hodge, 1968). A similarcase mightbe made

forany numberof attitudescales wheninter-item correlationsare con-

sistentlyunequal acrossmany different samples.

A more parsimoniousand more realisticbasis for alpha can be

givenby utilizingthe familiarformulafordecomposingthe variance of

a sumofitems.Lettingxijstandfora scoreon itemi forsubject., letting

p be the numberof items, and lettingscale scoresXj = Xlj + X2j +

* + xj, we have

=x2 = 20IZ i + 21 cov(xi,xj) (3)

i<j

whereax2 = varianceof the sum; cK2 = varianceof itemi; cov(x, xj)=

covariancebetweenitemi and itemj. Or, morecompactly,

S = I+C (4)

That is, the varianceofa scale or compositesum S is equal to the sumof

the individualitemvariancesI plus the sum of all possiblecovariances

betweendifferent items C.

For a set of itemsin a compositethat are measuringthe same or

similarproperties,one would expect the covariancesto be large com-

pared to the itemvariances;therefore we mightmeasurethe composite

reliability(or scalability)by consideringthe ratio C/S: this is the pro-

portionof scale variance due to item covariation.Unfortunately this

ratiodoes not have an upperlimitof 1, sinceI is neverzeroexceptin the

trivialcase when all item variancesare zero. The maximumvalue for

All use subject to JSTOR Terms and Conditions

THETA RELIABILITY AND FACTOR SCALING 21

with everyotheritem, whichmeans that all oa, are equal and cov(xi,

X,) = o2. This suggestsan indexof compositereliabilitywiththe desir-

able rangeof 0 to 1 as follows:3

itemcovariationadjusted to providean upperlimitof 1.

Althoughthis derivationprovidesthe same coefficient, the ad-

vantages of the rationale (aside fromits simplicity)are that no postu-

lates have to be made about an underlying modelof trueand errorscore

componentsand that thereis no necessityto make the parallelvariable

assumptions(such as equalityoftruescoresand equal precisionofmeas-

urement).It is this latterset of assumptionsthat may be particularly

unrealisticin many sociologicalapplications.On the otherhand, if the

parallel formassumptionsare reasonableforall items in a composite,

thenalpha as definedin Equations (1) or (5) is an appropriatemeasure

of reliabilityin the traditionalsense. In eithercase alpha offersa single

criterionforestimatingthe internalconsistencyand coherencyof a set

of itemsmakingup a scale. The greaterthe covariation,given a fixed

numberof items,the greaterthe value foralpha. A vahle of zero means

that the average covariationis zero; a value of 1 means that all items

have intercorrelations equal to 1.

Computational Forms

The formulasforalpha given in Equations (1) and (5) are not

particularlyusefulforcomputation.It mightbe helpfulto summarize

the major computationalformsand pointout some practicaldifferences

betweenthem.It mightcome as a surpriseto some analyststhat not all

formsproduce the same answers,and, given various conditions,that

some formsare moreappropriatethan others.

In the followingformulasp is the numberof items; rij is the

product-moment correlationbetweenitemsx. and xj; a-2 is the variance

of item i; X is the sum of the itemscoreswithan associatedscale vari-

ance ax; and X is the arithmeticmean of the itemscoreswithan associ-

2

ated variance ax. Depending on how the compositescale scores are

if not, that appropriatereversalsare made to eliminatenegativecovariances.

Withreal data it is always possibleto make reversalsso thatC remainspositive

(and so that alpha is always greaterthan or equal to zero).

All use subject to JSTOR Terms and Conditions

22 DAVID J. ARMOR

formedby summingstandardizeditemscores,

~(6) p + 2'5~~~"

rij

i<j )

If the scale is formedby summingraw itemscores,

a = ( (7)

_ -

U2)

If the scale is formedby averagingraw itemscores,

a = ( P T ) (8)

tationalconvenience;at least threeconsiderations apply. First,if one is

makingthetraditionalparallel-form assumptions,thenall itemvariances

shouldbe equal; if theyare not (as is usual withmost real data), then

differences wouldordinarily be seen as nonsubstantiveand shouldnot be

allowedto affectthe reliabilityestimate.In this case standardizationof

items removesvariance differences and Equation (6) can be used. Of

courseEquation (7) can also be used ifthe scale variancehappensto be

known.But Equation (6) is moreusefulbecause the scale variancedoes

not have to be known;it dependsonlyon theinter-item correlations and

the numberof items.

Equations (7) or (8) should be used wheneverraw item scores

are used to producethe scale scores.Since theyemploythe actual item

variances(whichare normallynot all equal), theseformulasdo not pro-

duce the same estimateas Equation (6). In effectthe alpha coefficient is

weightedaccordingto itemvariances.Whetherthe reliabilityaccording

to Equations (7) and (8) is higheror lowerthan that givenby Equation

(6) dependson complexrelationships betweentheitemvariancesand the

inter-item Basicallyscales formedfromrawitemscoreshave

correlations.

higherreliabilityif the itemswithhighervarianceshave highercorrela-

tions; if the oppositeholds,then Equation (6) producesa morereliable

scale. This relationshipshould become cleareraftertheta reliabilityis

derivedin the nextsection.

A thirdconsiderationarises when some items have missingob-

servationsdue to nonresponses.If scales are formedby a straightsum-

mation methodwith no special treatmentof nonresponses(that is, if

nonresponsesare in effecttreatedas zeros), subjectswithmany nonre-

sponseswill have spuriouslylow scale scores;the scale variancea2 will

be spuriouslylarge; and the resultingalpha will be spuriouslyhigh.In

All use subject to JSTOR Terms and Conditions

THETA RELIABILITY AND FACTOR SCALING 23

on responsespresent)and to apply Equation (8). Equation (6) is also

appropriatein thiscase, assumingthat the rijare computedwithoutthe

nonresponsesforeach pair of items.

Covariance Scaling

Given that real data may departfromthe parallel-itemassump-

tions,techniquesare necessaryto decide on the adequacy of individual

items. It is possiblethat one or moreitemsmay not be measuringthe

propertyin commonto the rest of the items; if so, theirelimination

mightenhancereliabilityas well as the conceptualclarityof the com-

posite. Under the parallel-itemassumptions,the greaterthe numberof

itemsthe higherthe reliability(this relationshipis given by the Spear-

man-Brownprophecyformula).But in the absence oftheseassumptions

such is not necessarilythe case. Specificallyitemsthat do not correlate

sufficientlywithotheritemsin a composite(or correlatein a negative

directionafterappropriatereversals)may actuallyreducethe reliability

of a compositeand shouldtherefore be excluded.4

The techniqueof maximizingthe alpha reliabilityof a composite

undervarious constraintswill be called covariancescaling.Otherterms

used to describethis processincludeitemanalysis (Upshaw, 1968), test

construction (Cronbach,1960), summatedratings(Edwards, 1957), and

Likertscaling after the psychologistwho inventedthe agree-disagree

itemscoringscheme.Sociologistsalso oftenuse the termindexconstruc-

tionto denote a similarprocedure.The termcovariancescalingcan be

applied to all thesemethodssince theyare all based on the same basic

and self-evident assumption:if a set of itemsis measuringthe same or

similarpropertiesand the propertycomprisesa singlecontinuumor di-

mension,the itemsshouldall covaryto some extent.For a fixednumber

of items,the greaterand moreconsistentthe inter-item correlationsthe

morereliablethe composite.

There is no single,uniformprocedurethat all researchersfollow

when constructing a covariancescale. Nonethelessseveralsteps can be

identifiedin the literature;an investigatorwould normallyuse at least

one of them:

basedontheremaining itemsafteritemh hasbeenexcludedwillbe greater

than

ap if

_p

2rih < pap/[p - a, (p - 1)] [p - 1 - ap (p - 2)]

i3 h

whererihis thecorrelation

betweenitemi and itemh.

All use subject to JSTOR Terms and Conditions

24 DAVID J. ARMOR

meanings

2. Calculation and inspectionof item-to-scalecorrelationsto

pinpointitems that are not contributingto the scale as a

whole5

3. Calculationof a T-test foreach itembetweenthe highest25

percentand lowest 25 percentof the compositescale scores

(usuallya substituteforstep 2)6

4. Examination of all inter-itemcorrelationsfor patterns of

loweror negativecorrelations

5. Recalculationof alpha reliabilityaftereliminatingitems ac-

cordingto information obtainedin steps 1 to 4

It is unlikelythat any but the most carefulresearchersgo throughall

thesesteps.Step 4 is probablythe one mostlikelyto be leftout unlessa

computeris beingused fortheanalysis.In factthisstepis notmentioned

in several standarddiscussionsof the methodof summedratings(Ed-

wards,1957; Upshaw, 1968). As willbecomeclear,however,step 4 con-

tains all the informationneeded to decide on scalability,and it may be

the mostimportantstep of all.

The pointin describingthesefamiliartechniquesis not so much

to instructthe reader in covariancescaling; ratherI want to be clear

about the meaningof my claim that the traditionalcovariancestepsre-

viewed here do not necessarilyresultin optimalscales fromthe stand-

point of compositereliability.Althoughthe applicationof covariance

scalingmay enhancereliability,the success of the applicationmay de-

pend moreon subjectivejudgmentsthan on analyticcriteria.Although

subjectivejudgmentscan never be eliminatedfromscaling techniques

(norshouldtheybe), the methodsof thetareliabilityand factorscaling

offera substantiallymoreanalyticapproachthat can help to reducethe

idiosyncraticvariationsof individualinvestigators.

Limitations of Alpha Reliability and Covariance Scaling

It has been pointedout that the methodof alpha reliabilityde-

pendson the assumptionthat all itemsin a compositeare parallelitems,

whichfurtherimpliesthat all itemsmeasure a singleunderlyingscale

propertyequally. Thereforethereare two major conditionsunderwhich

a set ofrealdata clearlyviolatestheassumptions:theitemsmay measure

a singlepropertybut do so unequally; the itemsmay measuretwo or

(or item-to-total)

itemcontributionbut frequentlytheyare not.

6 The proceduresforthisstep are describedin Edwards (1957).

All use subject to JSTOR Terms and Conditions

THETA RELIABILITY AND FACTOR SCALING 25

moreindependentpropertieseitherequally or unequally.Althoughthe

techniquesof covariancescalingmay helpsome aspectsof the firstprob-

lem theymay not do so optimally.As faras the second problemis con-

cerned,covariancescalingoffers littleor no solutionat all. The littlehelp

it may offerdependsverymuchon thenumberofitemsand thepatience

of the investigator.

These two problemscan be illustratedby simple hypothetical

examples.Considera four-item compositewhoseintercorrelations are as

shownin Table 1. The firstthreeitemshave intercorrelations that are

quite highand consistent,rangingfrom0.4 to 0.6. But item4 has much

lowercorrelationswith the otherthreeitems (all being 0.1). Assuming

that the correlationsare based on a sufficiently large sample,it might

turnout thatthe0.1 correlationsare significantly different fromzeroand

an investigatormay thereforedecide to keep item 4 in the composite.

But ifitem4 is leftin the compositewithoutsomekindofweighting, the

compositereliabilitywillbe lowerthanifthe itemsare weightedaccord-

ing to theirdifferential The four-item

contributions. (unweighted)alpha

reliabilityis 0.63; the compositereliabilitywithweightingis 0.68 (apply-

ing the theta techniquedefinedin the nextsection).

The second problemis generallyharderto detectwithreal data,

but the simplehypotheticalexamplein Table 2 offersa cleardemonstra-

tion. Basically the existenceof two or moreindependentpropertiesin a

compositeis usually revealed wheneverthereare two or more sets of

itemswithrelativelyhighwithin-setcorrelationsand relativelylow be-

TABLE 1

Inter-Item Correlations for Hypothetical Four-Item Composite

Item 1 2 3 4

1 1.0

2 0.6 1.0

3 0.5 0.4 1.0

4 0.1 0.1 0.1 1.0

TABLE 2

Inter-Item Correlations Revealing Two Dimensions

Item 1 2 3 4 5 6

1 1.0

2 0.5 1.0

3 0.5 0.5 1.0

4 0.0 0.0 0.0 1.0

5 0.0 0.0 0.0 0.5 1.0

6 0.0 0.0 0.0 0.5 0.5 1.0

All use subject to JSTOR Terms and Conditions

26 DAVID J. ARMOR

increasesas the between-setcorrelationsapproachzero.

Table 2 illustratesa six-itemcompositewhose intercorrelations

revealtwo independentdimensions,one represented by items1 to 3 and

anotherby items4 to 6. Each subset has within-setcorrelationsof 0.5;

all the between-setcorrelationsare zero. If all six items are combined

into a single scale, the alpha reliabilitywill be 0.60 (r = 0.2); but if

items1 to 3 and items4 to 6 are combinedinto two separatethree-item

scales, theywould each have the substantiallyhigheralpha reliabilities

of 0.75. In factthe six-itemscale reliabilitywouldnot surpassthe relia-

bilitiesof the two three-item scales untiltheaverageofall thebetween-

set correlationssurpasses0.2 (holdingthe within-setcorrelationscon-

stantat 0.5). Quite aside fromthe questionofreliability, ofcourse,is the

factthatdecomposingthea priorisix-itemcompositeintotwoindepend-

ent subscales providesnew informationthat may have an important

bearingon conceptualand theoreticalissues.

Althoughmost investigatorscould discoverthe independentdi-

mensionsin simpleexampleswhosecorrelationalpatternsare as clear as

those in Table 2, the situationwith real data is generallymuch more

complex.As thenumberofitemsincreases(say, overten),as thenumber

of dimensionsincreases,and as items contributedifferentially to each

dimension,nonebut themostpatientand diligentanalystwouldbe able

to produceoptimumscalingwiththe usual alpha reliabilityand covari-

ance scalingtechniques.

THETA RELIABILITY

manyresearchershave turnedto factoranalysisforassistancein identi-

fyingthe numberof propertiesor dimensionsin a set of data and in re-

latingspecificitemsto each dimension.With a fewnotable exceptions,

however,littleattempthas been made to relatefactoranalysisformally

to reliabilityand scalingmethods,and almostnoneofthisworkis widely

knownand applied in sociology(Lord, 1958; Bentler,1968; Heise and

Bohrnstedt,1971). Even factoranalysisis not as well understoodand

applied as it could be. When factoranalysisis applied to scalingtasks,

the applicationdependsmoreon the intuitionand experienceof the in-

vestigatorthan on analytictechniques;it remainsmoreof an art than a

technology.

Althoughthe interpretiveaspects of factoranalysis can never be

eliminated,thereare a numberof proceduresthat can aid the establish-

mentofoptimallyreliablescales. The keyto theseproceduresis a formal

All use subject to JSTOR Terms and Conditions

THETA RELIABILITY AND FACTOR SCALING 27

ponentfactoranalysis.7Otherfactor-analytic methodshave been devel-

oped, but principal-component analysisoffersthe most straightforward

and preciseconnectionbetweenreliabilityand scaling.The resultsof a

componentanalysisenable one to computean optimalreliabilitycoeffi-

cientI call thetaand to deriveoptimalscales througha seriesof steps I

call factorscaling.

The basic hypothesisof componentanalysisis that,givena set of

p iteins,the scoreof a subjecton each itemcan be decomposedinto any

numberof componentsor factors(up to the total numberof items).

Generallyonlya small numberof the largestfactorsin termsof the pro-

portionof the total itemvariationtheyrepresentare givensubstantive

meaning;the remainderare groupedtogetherand consideredas an error

componentor factor.Each nonerrorfactor,say m ofthem(withm < p),

is a hypotheticalvariablegivensubstantivemeaningby individualitems

accordingto factorloadings-a set of weightsdeterminedby the contri-

butionofeach itemto a givenfactor.This meansthateach itemcan con-

tributedifferentially to a givenfactor(or not at all if the factorloading

is zero). Moreover each factorrepresentsa statisticallyindependent

sourceofvariationamongtheset ofitems,and mostcomponent-analysis

solutionsextractthese factorsin an ordercorresponding to the size of

theircontribution; thatis, thefirstfactoraccountsforthemostvariance,

the second factorforthe second largestamountof variance,and so on.

From the standpointof scaling,principal-component analysis offersa

means of detectingthe most importantindependentdimensionsamong

a set of items (if thereis morethan one) and, foreach of these dimen-

sions,providesfordifferential contributions by individualitems.

A principal-component analysiscan be used to constructa set of

factorscores,one set foreach factor.A factorscoreis simplythe scoreof

a subjecton a givenfactor:it is in effecta compositescale scorebased on

a weightedsum of the individualitemsusingthe factorloadingsas the

weights.The task of the thetacoefficient is to providea compositerelia-

bilityestimateforfactorscoresbased on principal-component analysis.

In componentanalysisthe amountof varianceaccountedforby

a factoris called a rootand is denotedby the symbolXk (forthe kthfac-

tor).8 Althoughthese quantitiesare providedby most computersolu-

tions forcomponentanalysis,they actually have a simple relationship

7 By principal-component analysis is meanta methodof factoranalysis

wherebyunities(is) are leftin the diagonalof a correlationmatrixwhenfactor-

ing. See Harman (1967).

8 The fulltechnicaltermis latent

root.Othercommontermsare eigenvalue

and characteristicroot.

All use subject to JSTOR Terms and Conditions

28 DAVID J. ARMOR

torloadingsforthe kthfactor.These are the quantitiesthat providethe

basis fortheta reliability.

Beforegivingthe formulaand the derivationforthetareliability,

it must be pointedout that thereare two generalcases in component

analysisthat need to be distinguished.First,in a single-factor solution

the firstfactorand its root Xi sufficefora completespecification of the

scale and its reliability.Second, if a multiple-factor

solutionis adopted

meaningfulinterpretation generallyrequiresa rotationof the m factors

that are retained.Rotationsare dealt within moredetail later; suffice it

to say herethatafterrotationthevarianceofa factor-whichstillequals

the sum of the squared rotatedloadings-no longerequals the mathe-

matical rootsderivedin the originalprincipal-component solution.Ac-

cordinglythesymbolXkis used to denotethevarianceofthe kthrotated

factor.

Given a set of p itemsand a single-factor solutionwithroot Xi,

the reliabilityof the compositescoresbased on thisfactoris givenby

0 = [p/(p - )1- (1/XI)] (9)

this formulais not new, it is littleknownin sociology(Bentler,1968).

It is mathematicallyequivalentto alpha fora compositescale formed

by weightingitemsaccordingto theirprincipal-component factorload-

ings; this has been shownby Lord (1958) to be the maximumpossible

alpha. By takingintoaccountdifferential itemcontributions to the cen-

in

tral property commonto a set of items,one obtains the maximum

compositereliabilityaccordingto the definition in

given Equation (5).

The situationfora multiple-factor solutionwithrotatedfactors

is somewhatmorecomplicated.LettingOhk stand forthe squared corre-

lation betweenthe originalunrotatedscores forfactorh and the new

factor k, and given a rotated m-factorsolution with original roots

Xl, X2, , Xm,the reliabilityof the kth set of rotatedfactorscoresis

givenby

solution).The quantity khk is actually the elementin the hth row and

kth columnof the transformation matrixthat maps the originalfactor

loadingsinto the rotatedloadings.

Althoughtheta-staris mathematicallysimilarto theta,it will be

clearlaterthatit does not have quite the same utility.Since the number

All use subject to JSTOR Terms and Conditions

THETA RELIABILITY AND FACTOR SCALING 29

theta-staroffersmaximumreliabilityforeach subset of itemsin a com-

posite.AMoreover it turnsout that in spiteof the weighting, ifmorethan

one factoris presentoptimal reliabilityis seldom approached if all p

itemsare used to derivethe weightedscale scores.Nonethelesstheta-star

is the properformulaforreliabilitywhenone is usingthe completeset of

rotatedfactorscoresfroma principal-component analysis.

I now give more precisemeaningto the precedingideas, and I

providea derivationfortheta. Given an observedstandardizedscorexij

forsubjectj on item i, the basic model forcomponentanalysisis9

xij= ailflj + ai2f2j + * + aimfmj + eij ()

where aik is a factor loading of item i on factor k; fkj is a factor score for

subjectj on factork; eij is an errorcomponentdue to randomand other

specificinfluencesofitemi to the observedscore.If m = p or ifas many

factorsare extractedas thereare items,then eij = 0 forall i, j and the

originalscoresare decomposedperfectlywithouterror.This case is gen-

erallyuninteresting, however,sinceone ends up withas manyfactorsas

there are items. In most cases an investigatorinterpretsonly a small

numberoffactorsm; in thiseventtheremainingp-mfactorsare assumed

to representan errorcomponent.The problemof choosingthe cutofffor

m is taken up in a later section.

It is not necessaryto go into the mathematicsof exactly how

these quantitiesare obtained; mostresearchersrelyon widelyavailable

computerprogramsforderivingthem.10For the presentit suffices to say

that a standardprincipal-component analysisresultsin a set of factor

loadingsaik that measurethe magnitudeof the contributionof itemi to

factork on a scale rangingfrom-1.0 to + 1.0; a set ofrootsXk such that

Xk = 5 aik; and a set of factorscoresfkj that providethe score of sub-

9 The assumptionofstandardizedscoresdoes iiotdestroy

thegenerality

of thetechnique.Thereis usuallyno a priorireasonwhyitemsthatmeasure

similarpropertiesshouldbe allowedto have differenitial

weightsby virtueof

different

arbitrarily variances;indeedthe wholepointof factoranalysisis to

base weightson contributionsto underlyingfactorswithoutregardto initial

variances.

10It can be notedin passingthat thefactorloadingsare simlply theset of

latent vectors (eigenvectors,characteristicvectors) which satisfythe charac-

teristicequation (R - XI) v = 0, whereR is the correlationmatrixwithunities

in the diagonal,X is a constant,I is the identitymatrix,and v is a latentvector.

The factorloadingsin Equation(11) are normalizedso that akak = Xk, whereak

is the vectorof factorloadingsforfactork; the lateintvectorsv are normallyas-

sumed to be normalizedto 1 (v'v = 1).

All use subject to JSTOR Terms and Conditions

30 DAVID J. ARMOR

TABLE 3

Factor Loadings and Roots

Item Factor 1 Factor 2 ... Factor in

1 all al2 *-- a,m

2 a2l a22 * a2m

a k =1 X2 X,n

onlyforconstructing thefactorscoresbut also forinterpreting thefactor

in termsof the contentof the originalitem. The higherthe absolute

value ofa loading,thegreaterthe contribution ofthatitemto thefactor

in question.11

A root Xk can be interpretedas the amount of variance in the

originalset of itemsaccountedforby factork. Since the total variance

in a set ofp standardizeditemsis just p, and sincethe sum of all p roots

mustalso equal p, theratioXk/PiS theproportion oftotalvariancedue to

factork. These proportions are used to givean idea about theimportance

of a given factor;moreoverthe sum of these proportionsform factors

tells us how much variationhas been explainedby the total set of m

factors.

Given the factorloadingsand roots,it is possible to derivethe

factorscoresfromthe followingsimplerelationship:

standardizedscoresusing the ratio aik/Xk as the weightforitem i and

factork.

In orderto derive theta, considera single-factor solution with

root Xi,loadings ail, and scoresfij. We shall use the basic of

definition

compositereliabilityprovidedin Equation (5) as thebasis of our deriva-

tion;thatis, C(p)/S(p - 1). Accordingly we needto knowthequantities

as thecorrelation

loadingcanbe interpreted an item

and thefactorscores.A negativeloadingmeansthattheitemhas a negative

withthefactor

correlation theitemscore,thelowerthefactor

scores(thehigher

scoreorviceversa).

All use subject to JSTOR Terms and Conditions

THETA RELIABILITY AND FACTOR SCALING 31

relationship

forS based on a weightedsum of item scores:

S = 2ZW ,Ti 2 + Zwiwh Cov(Xi, Xh) = I + C (13)

standardized,we have C. 2 = 1 and cov(xi, xh) = rih. Therefore,since

Wi = ail/Al,

S = f 22 (ail/X\)2+ (ail/Xl)(ahl/\l)rih

i h

2

And since EZai21 =

i,--h J

i,h ailahlrih = SO

E ailah1rih =

iHh

Z ailahlrih - Zi a2

i,h (16)

= X12-1

Thus

S = /X1+ (X2-Xl)/X2

1 1 1

~~~~~~~(17)

= 1/X1+ (1 -1/X1)

This means that forthe factorscoresbased on the firstprincipalcompo-

nent,S = 1, I = 1/X1,and C = 1 - 1/X1.For the compositereliability

fromEquation (5) we have

whichis the formulafortheta givenin Equation (9).

That this formulafor theta provides the maximumcomposite

reliability(given our definition)followsfroma fundamentalproperty

of principalcomponents:that the quantityX1is the maximumvalue for

the quadratic form i,hailahlrih = X. That is, no set of weightsail

otherthan those providedby the firstprincipal-component factorload-

ings produce a value largerthan the firstroot.12Thus theta gives the

maximumvalue forthe ratio C(p)/S(p - 1).

12 In principal-component theorythisrelationshipis generallydefinedas

v1Rv1= Xi, wherethe firstprincipalcomponentis that vector v, which pro-

duces thelargestrootXi (see footnote10). Withtheusual normalization forfactor

loadingsthis is equivalentto a'Ral = X2.

All use subject to JSTOR Terms and Conditions

32 DAVID J. ARMOR

line. Startingwiththe usual relationshipS = I + C, forthe kthrotated

factorscoresf we have

2 p 22 p

S= =

S fk - Z zik -T + z Wikwjkrij Ik + Ck (19)

Equation (5) is the quantityCk = Z p WikWjkrij. In the case ofrotated

factorscores,the weightWik is morecomplexthan forunrotatedfactor

scores:

m

Wik = 'E (aihOhk)/\h (20)

h=l

tated factork scores,or the (h, k) elementin the orthonormaltrans-

formationmatrix.Using this relationshipforCk we have

P P m m

Ck = E WikWjkrij =

iHj=l

E

iHj=l

[~h=lI (aihhk)\h [E (ajhOhk)/h

h=1

rij

P m m'(1

=~~E~ ,2rij

E(aihOhkajhOhk)/Xhr

+

+ , (aihchkaj0gk)/Xh\rij (21)

ii--dj-1 h=1 hp,hg=l

m 2p m P

= --I a'haJhrij + E (chk0gk)/(XhX\g) E2 aihaigrij

h=1 Ah iHj=1 h,g=1 idj1-

Z ipj aihajhrij = Xh2 hand Z ipj aihaigrij = 0. Therefore

Ck E2y(Xh- Ah) =Z phk(I- 1 )= hk- _E hklh/A

hsl Ah h=l A h=1 h=1

matrix,

we knowthat E am 1 = 1. We have, finally,

h=1

tionshipa < 0. As we have said, alpha tendsto get largeras the amount

of covariationin a set of itemsincreases.But alpha assumes that each

itemis weightedequally and that each inter-item correlationis close to

All use subject to JSTOR Terms and Conditions

THETA RELIABILITY AND FACTOR SCALING 33

alpha are approximatelyequal. When this is not the case, each item

contributesa different amount to the covariance.A factorloading ex-

pressessomethinganalogous to the average amountof covariationdue

to an item: those items with highercorrelationswith otheritems have

higherfactorloadings.Thereforea compositescale that weightsitems

accordingto thefactorloadingsexpressesa moreaccurateratioofcovari-

ance to scale variance.Theta and alpha are equal onlyin thelimitingcase

whenall inter-item correlationsrijare equal.

The relationshipbetweena and O*is moredifficult to state since

a is appropriateonly fora single-factor model. Given an m-factorsolu-

tion, one can constructm unweightedscales using the highestloading

itemson each factor.Lettinga' be the alpha reliabilityforthe subscale

correspondingto factor k, empiricalresultsshown later demonstrate

that usuallyO*< a'. Moreover,ifeach subsetofitemsis refactoredas a

single-dimension scale and a new theta reliabilityis computed,say Ok,

then generallya' and O' are approximatelyequal. Given a multiple-

factorsolution,therefore, optimallyreliablescales are derivedby apply-

ing the usual unweightedcovariancetechniquesto each subset of items

ratherthan by usingrotatedfactorscores.More is said on thisissue in

the nextsection.

principal-component analysis to enhancethe reliabilityof a composite.

First, as we have already seen, if one uses factorscores forcomposite

scale scoresinsteadofthe traditionalunweightedsum ofitemscores,the

methodof thetareliabilityshowsus that we are guaranteeda reliability

equal to or greaterthan that given by alpha reliability.Second, in the

case ofa single-factorsolutionwe muststillraisethe questionofwhether

weightingalone takes care ofitemsthat do not contributeto a factor(or

contributeto onlya smallextent).That is, willreliabilitybe enhanced

beyond that providedby factorscoring-by factorscales that exclude

low-loadingan item altogether?By factorscale I mean a composite

formedto measurea factorby usingonlythe highest-loading itemson a

factor.Finally,both reliabilityand conceptualclaritymay be improved

by adoptinga multiple-factor solution(whetheror not it was hypothe-

sized). If eitherof these last two conditionsis present,we must again

considertherelativebenefitsofrotatedfactorscoresversusfactorscales.

In thissectionI take up somepracticalconsiderations thatbear on these

issues.

All use subject to JSTOR Terms and Conditions

34 DAVID J. ARMOR

Single-Factor Case

Althoughwe know that theta is algebraicallylargerthan alpha,

it mightbe usefulto presentsomesimplehypotheticalexamplesthat re-

veal how seriousthe differences mightbe and showhow we can improve

scale reliability.We shall examinean applicationoffactorscalingto real

data in a latersection.

Table 4 shows fourcases, the firstthreeof whichinvolve four

items.I have helditems1 to 3 constantso that thedifference in reliabili-

ties can be seen as we add differentfourthitems.

Case 1 reveals the largestdiscrepancybetweentheta and alpha

(0.05). The reasonis that the fourthitemhas low correlationswiththe

otheritemsand consequentlya low factorloading (0.24). A scale score

that weightsitemsaccordingto the loadingstherefore has a higherreli-

abilitythan a simplesummedscore.

In case 2 the fourthitem still has somewhathighercorrelations

and a higherloading (0.44), so that the difference betweentheta and

alpha is diminishedto 0.02. In case 3 thereis practicallyno difference

betweentheta and alpha even thoughthe correlationsand the loadings

do varyconsiderably.Clearlythetaand alpha differ in a substantialway

only whensome itemshave consistently lowercorrelationswithall the

TABLE 4

Comparing Theta and Alpha

Factor

Case Item Correlations Loading (ail)

Case I 1 2 3 4

Xi = 2.033 1 1.0 0.86

o = 0.68 2 0.6 1.0 0.82

a = 0.63 3 0.5 0.4 1.0 0.76

4 0.1 0.1 0.1 1.0 0.24

Case 2 1 2 3 4

Xi = 2.117 1 1.0 0.87

a = 0.70 2 0.6 1.0 0.80

a = 0.68 3 0.5 0.4 1.0 0.72

4 0.3 0.2 0.1 1.0 0.44

Case 3 1 2 3 4

X = 2.310 1 1.0 0.81

o = 0.756 2 0.6 1.0 0.80

a = 0.754 3 0.5 0.4 1.0 0.76

4 0.3 0.4 0.4 1.0 0.66

Case 4 1 2 .3

X= 2.004 1 1.0 0.87

0 =0.75 2 0.6 1.0 0.82

a = 0.75 3 0.5 0.4 1.0 0.76

All use subject to JSTOR Terms and Conditions

THETA RELIABILITY AND FACTOR SCALING 35

largerthan alpha.

Case 4 shows the statisticsfora scale formedby the firstthree

itemsalone. It is clearthat the three-item scale in case 4 is morereliable

than the weightedfour-item scale in cases 1 and 2 (0 = 0.75 compared

to 0.68 and 0.70), and it is almost as reliableas the four-item scale in

case 3. In otherwords,even thoughfactorscoresgive less weightto the

weak itemsand producea morereliablescale than a simple unweighted

summation,an even higherreliabilitymay be obtainedby throwingout

weak itemsaltogether.

This fourthcase motivates the procedure that I call factor

scaling.In some cases the highestreliabilityis obtainedby formingthe

scale using only items that load higheston a given factor.A scale so

formedis called a factorscale to distinguishit fromfactorscores. Al-

thoughthereis no preciseruleto definethe highestloadings,experience

has shownthat itemswithloadingsbelow0.3 shouldbe excluded.Items

in the 0.3 to 0.4 range are generallyborderline;that is, sometimesthey

increasereliabilityand sometimestheydecreaseit, but usuallytheydo

not affectit in any appreciableway.

One practicalconsequenceofconstructing scalesby usingonlythe

highest-loading itemsis that theta and alpha do not oftendiffer by any

importantdegree.This means that one can use the simple,unweighted

sum oftheitemscoresforthe compositescale withoutsacrificing reliabil-

ity. This procedureis onlyvalid, of course,whenthe itemsused do not

have widelydiffering factorloadings.Whetherone uses the unweighted

sums or refactorsto get factorscoresforthe subset of itemsto be used

in the scale is mainlya matterof convenience.

I must issue one importantcaution at this point. The most reli-

able scale forone sample of subjects may not correspondto the most

reliable scale for anothersample. When I suggestthat throwingout

itemsmay producea morereliablescale, this holds only in the context

of a singlesample. If one is buildinga scale of some type (ratherthan

analyzingdata froma particularsample), manysamplesand manyfac-

tor analyses must be taken beforea finalscale can be derived.In this

case a "bad" item is one that shows consistentlypoor factorloadings

acrossdifferent samples.Nonethelesswhenone is analyzingdata froma

particularsample and scales are beingused, thereis no reasonnot to ex-

clude weak itemsforthat analysiseven thoughtheymay be maintained

whencollectingdata fromothersamples.

Multiple-Factor Case

In the event that a set of itemsmeasuresmorethan one dimen-

sion,the methodof factorscalingrequiressome extension.Althoughthe

All use subject to JSTOR Terms and Conditions

36 DAVID J. ARMOR

case, the mathematicaleleganceis somewhatdiminishedifthereis more

than one factor.The reasonshave to do withthe nonuniquenessof the

rotationalmethodsthat are used to interpreta multiple-factor analysis.

Given two or more clustersof intercorrelating items, the principal-

componentsolutionis notsuitableforfactorscalingbecause it maximizes

thevarianceofthefirstfactor,thenthesecondfactorafterthefirstfactor

varianceis removed,and so on. Therefore, giventwo or moreindepend-

entclustersofitems,unlesstheclustersare perfectly uncorrelated(which

is seldom the case) the firstfactorwill pick up more variance than it

should.

Figure 1 gives a geometricinterpretation of this problem.With-

out rotationthe firstfactorconsistsof strongand positiveloadingsby

all items whereasthe second factoris bipolar with both positive and

negativeloadings.The rotatedsolutionassociates each factorwithone

subsetof itemsor the other,facilitating both interpretation and scaling.

I want to stressthat rotationis not advocatedonlyas a convenience.As

we shall see in the real-dataexamplein a later section,rotationcan be

crucial foridentificationof thoseclustersofitemsthat producethe most

reliablescales.

Given a set ofm principal-component factorloadingsail, ai2, ,

,

fmj can be foundby applyingan arbitrary orthogonaltransformation to

theoriginalloadingsand scores.Withone exceptionthe rotatedloadings

and scoreshave all the propertiesof the unrotatedloadingsand scores:

theyare statisticallyindependent;takentogethertheyexplainthe same

amountofthetotalvariation;and theyreproducetheoriginalcorrelation

matrixto thesame extent.The exceptionis thatthesumofsquaresofthe

loadingsX*, ,* (whereXk =

*, i a*k) do notequal theoriginal roots.

That is, we can say thatE km=l X = 57,=m X*but we also have that

Xk # X*.This means that althoughthe amount of variance accounted

forby in factorsis invariantunderarbitraryorthogonalrotations,the

portionof varianceattributableto each factoris not. Obviously,then,

dependingon the ro-

the scales derivedas well as theirreliabilitiesdiffer

tationalsolution.The onlyway to achievesome standardizationat this

stage is to specifythe rotationmethodwhenscales and reliabilitiesare

presented.

Althoughmany different rotationalschemeshave been devised

(Harmon,1967), the varimaxmethodseemsto be the mostwidelyused

orthogonalsolutionat the presenttime,and it is the one I shall assume

throughout.The varimaxmethodrotatesall m factorsuntil the factor

All use subject to JSTOR Terms and Conditions

THETA RELIABILITY AND FACTOR SCALING 37

I 1*I 11*

0

1.0

0~~~~~~

ClusterA ail ClusterB a*

CILISLserAC /(luster B

-1.0 0 1.0M

a

a i2

-1.0

Uinrotate(I Rotated

ofunrotatedand rotatedfactors.

1. Geometricrepresentation

Figutre

factorhas itemswitheitherveryhighor verylow loadings(in absolute

value) withinthe constraintsimposedby the total varianceof the load-

ings.Since thissolutiontendsto give the mostsatisfactoryalignmentof

factorswithclustersof items,giventhe requirementof factorindepend-

ence, it seems the most suitablesolutionforfactorscaling.

The mostdifficult issueby farformostanalystshas to do withthe

numberof factorsm to extractand rotate. Clearly this numberdrasti-

cally affectsthe natureand reliabilityof the scales produced.Unfortu-

nately,however,Ino singleanalyticmethodcan solve this problempre-

cisely.About all I can offerhereare threerulesthatmay aid thedecision.

First,no factorshouldbe used ifits rootis near to or less than 1.

Since factoranalysis works on standardizeditems, the variance of a

singleitem is 1. Thereforea factorwhose root is near to 1 explainsno

morevariancethan a singleitem.

Second, afterthe mthroot, if the m + Ith and successiveroots

are fairlyequal exceptfora gradual tapering-off, thenm factorsshould

probablybe used (or at least rotatedforinterpretation).A large drop

fromone root to anotherfollowedby slightlydecreasingroots usually

rnarksthe end of meaningfulfactorsand the beginningof erroror spe-

cificvariancecomponents.Thereare statisticaltestsavailable fortesting

equality of the remainingroots afterm have been extracted,but they

depend heavilyon sample size and may be of limiteduse in large socio-

logicalstudies (Lawley and i-\iaxwell,1963).

Third, in many sociologicalstudies with inter-itemcorrelations

rangingfrom0.3 to 0.5, an m that accountsfor40 to 60 percentof the

total variance is a good solution.In this contextit can be pointedout

that, for ten items,single-factor solutionswhich account for no more

All use subject to JSTOR Terms and Conditions

38 DAVID J. ARMOR

the order ofO.8.13

Aside fromthese rules of thumb,most carefulanalysts rotate

suecessivenumbersof factorsuntiltheyfinda solutionthat is substan-

tivelymeaniingful and interpretable.Obviouslythis last criterionis as

importantas the others.

Given a satisfactorysolutionof in factorsas in the single-factor

case, thereare twomajor waysin whichto constructscales and compute

reliabilities.One methodis based on the rotatedfactorscores;the other

uses factorscales based on the highestloadingitems.

For principal-componientfactoranalysis,factorscoresforthe ro-

tated factorloadingscannotbe computedwitha simpleformulasuch as

Equation (12). Neverthelessfactorscoresare usuallyprovidedby most

computerprogramsforprincipal-component analysis.The procedurein-

volves rotatingthe originalfactorscores using the same orthogonal

transformation matrixthat is derivedforrotatingthe factorloadings.

Given rotated factorscoresfkj, a reliabilitycoefficient can be

computedforeach factork by applyingthe generalformulafortheta-

star given in Equation (10). It must be stressedthat these reliabilities

are not unique,however,sincetheydependon a givennumberoffactors

and on a givenrotationalmethod.I also stressthat thesereliabilitiesare

generallysignificantly lowerthan thoseobtainedby factorscaling.

As illustratedin the single-factorcase, even betterreliabilities

can be obtainedby formingscales usingonlythe itemsthat load highly

on a givenfactor.This methodis even moreimportantin the multiple-

factorcase sincethe existenceof two or morerotatedfactorsguarantees

that each factor*willhave many itemswithloadingsclose to zero. The

set of itemsthat loads highlyon each factor(say, above 0.40 or so with

no higherloadingson otherfactors)is takento forma scale. Since this

approachguaranteesrelativelyequal factorloadings(that is, equal item

w-eights), simpleunw-eighted summedscales can be constructed;alpha

reliabilitycan then be computed as a close approximationto theta.

Alternativelyeach subset of itemscan be refactoredfora single-factor

solution;both factorscoresand theta can thenbe obtained.Items that

load highon two or more factorsshould be excludedif one desiresto

maintainscale inidependence.

Althoughthe method of factorscaling generallyproduces the

most reliablescales, thereis a disadvantagein that the scales will not

of varianceex-

betweenthetaand the proportioin

13 The relationship)

)lained by thefirstfactor(say, Pi) is givenby Pi = 1/[p - 0 (p - 1)]. This can

be usedinconjunctionwithdiffering

numbers ofitemsto seewhatproportion of

varianceshouldbe explained

bythefirst toattaina giventhetareliability.

factor

All use subject to JSTOR Terms and Conditions

THETA RELIABILITY AND FACTOR SCALING 39

finesa factoris somewhatrelatedto anothercluster,however,thiscorre-

lation is probablymeaningful.If the correlationbecomes too large, of

course,it may mean that a smallernumberof factorswill be a better

solutionfromthe perspectiveof scalabilitythat we have developed.

Rather than givinghypotheticalexamplesto illustratethe mul-

tiple-factorcase, I shall turnimmediatelyto an exampleusingreal data.

The issues we have been discussingshouldbe made clearby way of this

completeillustration.

been primarilytheoreticalratherthan empirical.In orderto show how

to apply theseideas and howto see differencesbetweenthetwomethods,

thissectionpresentsa fullexampleusingreal data.

The data are drawnfroma surveyof collegeprofessors' viewsand

behaviorregardingthe VietnamWar (Armorand others,1967). A set of

13 attitudeitemswas designedto measurepoliticalideologyalonga gen-

eral conservative-liberal dimension.The intentionwas to constructan

attitudescale to measure politicalliberalismso that politicalideology

could be related to feelingsabout the Vietnam War. Althoughit was

knownthat politicalideologyis not a unidimensionalconstructin na-

tional cross-sections,

the special natureof the sample led to an expecta-

tion that therewould be a singleliberal-conservative dimensionin this

case. Since the originalintentionwas to createa singlescale, it is a par-

ticularlyappropriateexampleforcomparingunidimensionalcovariance

scalingwithmultidimensional factorscaling.

Covariance Scaling

Table 5 showstheitemwordingsalongwitha numberofstatistics

used in theconstruction ofa covariancescale. All itemswerescoredorigi-

nally on a 1 to 5 agree-disagreecontinuumwith5 indicating"strongly

agree"; itemsmarkedwitha minussignhave been reverse-scored so that

all itemsare scoredin the same direction(that is, agreeingmeans more

liberal). In additiononlyrespondentswho had no missingobservations

on any of the 13 itemswereincludedin the analysis.

If an analystconsidereditemface contentalone, withoutconsid-

erationof the quantitativeitem-analysisinformation shownin Table 5,

and developeda 13-itemscale by a straightsummationofitemscore,the

alpha reliabilitywouldbe 0.72 usingEquation (7). The alpha reliability

based on r (standardizeditems)is slightlyhigherin thiscase (0.73).

All use subject to JSTOR Terms and Conditions

TABLE 5

Covariance Scaling forPolitical Ideology Scale

(a) Item-Scale Correlations

Itemb Uncorrectedc Correctedd

1 War is not justifiableunder any circumstances 0.608 0.483

given a major part in deciding company policy 0.557 0.428

systemto any otheryet devised 0.073 -0.062 a = 0.

vide health servicesthan our presentsystem 0.503 0.384

5 It is up to the governmentto make sure that

everyonehas a secure job and a good standard of

living 0.434 0.289

(-)6 It is never wise to introduce changes rapidly, in

governmentor in the economic system 0.367 0.213

(-)7 In taking part in any formof world organization,

this countryshould make certain that none of its

independenceand power is lost 0.397 0.255

essary 0.535 0.411

9 An occupation by a foreignpower is betterthan a

nuclear war 0.604 0.481

All use subject to JSTOR Terms and Conditions

(-)10 In general, complete economic security is bad;

most men wouldn't work if they didn't need

moneyforeating and living 0.483 0.339

11 War could always be avoided 0.440 0.285

(-) 12 Nationalization of the basic industries would

lead to an intolerable degree of governmental

control 0.566 0.435

13 Nationalization of the basic industrieswould lead

to a more equitable distributionof wealth 0.623 0.514

1 2 3 4 5 6 7 8 9 10

< 1 1.000

2 0.223 1.000

3 0.014 -0.048 1.000

4 0.103 0.334 -0.028 1.000

5 0.018 0.315 -0.261 0.324 1.000

6 -0.027 0.197 0.088 0.115 0.118 1.000

7 0.261 0.207 -0.121 0.005 0.010 0.193 1.000

8 0.771 0.197 -0.011 0.059 -0.028 -0.149 0.226 1.000

9 0.439 0.115 -0.039 0.325 0.119 0.187 0.260 0.322 1.000

10 0.036 0.305 0.005 0.156 0.388 0.244 0.106 0.024 0.065 1.00

11 0.622 0.062 -0.043 0.112 0.023 -0.159 0.045 0.579 0.354 -0.020

12 0.588 0.228 0.033 0.329 0.392 0.266 0.177 0.049 0.197 0.38

13 0.200 0.314 0.023 0.356 0.227 0.202 0.077 0.249 0.370 0.26

a N = 104 forall statistics.

bAll items scored on a 1 to 5 agree-disagreescale; reverseditems indicated by a minus sign

c Item contributionincluded.

d Significancelevels: r > 0.19 significantat 0.05

level; r > 0.25 significantat 0.01 level.

All use subject to JSTOR Terms and Conditions

42 DAVID J. ARMOR

tiallylower(and nonsignificant) correlationforitem3. Even ifthisitem

is eliminated,however,the alpha reliabilityof the summedscale based

on the remaining12 itemswouldbe unchanged.The remainingitemsall

have significantand fairlyconsistentitem-to-scalecorrelations;it is un-

likely,based on thesestatistics,that an investigatorwould change the

scale beyonddroppingitem3.

The inter-item correlationsgive moredetail about the itemrela-

tionships and show the problemwith item 3: it has a large (and sig-

nificant)negativecorrelationwithitem 4 and generallynegligiblecor-

relationswiththe remainingvariables. This clearlylowersthe average

correlation,althoughin this case not enoughto affectalpha reliability

significantly.In spite of the item-to-scalecorrelations,it is apparent

fromthe correlationmatrixthat a numberof the itemshave low corre-

lationsand that someeven have slightlynegativecorrelations.A careful

inspectionof the pairwisecorrelationsmightreveal other items that

shouldbe eliminatedor combinedseparatelyto formsubscales. Rather

thanundertaking thistediousjob, however,we turnto factorscalingfor

a morepreciseand compactmethodofexaminingthepatternofrelations

amongthe items.

Factor Scaling

The relevantstatisticsforfactorscalingare presentedin Table 6.

I shall discussonlythefirsttwo factors,whichaccountforabout 43 per-

cent of the total variance.Because the thirdroot was close to 1 (X3 =

1.236), it was decidedto considerat mostthe firsttwo factors.

solution,we shouldfocuson the prin-

If we adopt a single-factor

cipal-component loadingsforthe firstunrotatedfactor.The firstrootis

3.29, accountingforabout 25 percentof the total variance. As we can

see fromthe firstcolumnof Table 6, thereis considerablevariationin

the item contributionsto the firstfactor. The loadings range from

-0.08 foritem3 to 0.65 foritem 1. These differences resultin a higher

reliabilityfora 13-itemscale based on the factorI scores,whichtake

item weightinginto account. Theta is 0.75 forthe factorI scores,com-

pared to an alpha of0.72 forthe 13-itemsummedscale. Since in thisex-

ample only one item is trulyweak, the difference between alpha and

thetais not great.Obviouslythe difference betweenalpha and thetade-

pendson how many"bad" itemsoccurin a set ofitemsbeingused fora

scale.

Ratherthan exploringothercombinationsof first-factor itemsto

discoverthe best single scale, we must considerrejecting the single-

factorhypothesis.The secondrootis fairlylarge (X2 = 2.32, 18 percent

All use subject to JSTOR Terms and Conditions

THETA RELIABILITY AND FACTOR SCALING 43

TABLE 6

Factor Scaling for Political Ideology Scalea

Unrotated Principal-

Component Loadings Rotated Loadings

Itemb I II I* II*

1 0.650 -0.615 0.032 -0.895

2 0.553 0.247 0.567 -0.212

3 -0.082 -0.030 -0.080 0.036

4 0.523 0.283 0.571 -0.165

5 0.438 0.477 0.647 0.033

6 0.262 0.450 0.502 0.137

7 0.377 -0.058 0.228 -0.306

8 0.593 -0.636 -0.023 -0.869

9 0.637 -0.184 0.325 -0.578

10 0.425 0.472 0.634 0.038

11 0.483 -0.637 -0.103 -0.793

12 0.535 0.504 0.735 -0.016

13 0.648 0.217 0.613 -0.300

Roots Xi = 3.289 X2 = 2.322 Xi = 2.813 X2 = 2.798

Percent of

total variance 25.3% 17.9% 21.7% 21.5%

Reliabilities

Factor Scores

O = 0.75 forfactor scores based on unrotated factor I

01 = 0.70 forfactor scores based on rotated factor I*

02 = 0.70 forfactor scores based on rotated factor II*

Factor Scales

01 -0.74 for socialism scale based on items 2, 4-6, 10, 12, and 13 [ai = 0.73

using Equation (6)]

02 = 0.85 forpacifismscale based on items 1, 8, and 11 [a2 = 0.85 using Equa-

tion (6)]

a N = 104 for all statistics.

I See Table 5 for item wording.

factorII than on factorI (namely,items5, 6, 8, 10, and 11). Therefore

we mustexaminethe two-factor solution.For the two-factorhypothesis

therotatedloadingsmustbe inspectedin orderto obtaina moreaccurate

interpretation.These are shownin columnsI * and II * in Table 6.

The rotated loadings clearly reveal two relativelyorthogonal

clustersof items (see Figure 2 for a geometricrepresentation).The

clusterof items (2, 4-6, 10, 12, 13) that largelydeterminesfactorI * has

contentconcernedmainlywitheconomicissues (the sole exceptionbeing

item6); the itemsthatdeterminefactorII * (1, 8, 11) deal withtheissue

All use subject to JSTOR Terms and Conditions

44 DAVID J. ARMOR

Iioe ( I* (rotatedl)

I (unrlotaletdl) (12) *(?> II

0.7

0.4-

0 ~~~~0.3

Ci) b0~~~.2-

-0.3

-0).4

-0.5

rpe

-0.7-

not made in ada -,cof seein -0.8

the fatranlss

U.S. power) do not load substantiallyon eitherfactor.Item 9 ("better

red thandead") fallsbetweenthe two clusters.Accordingly we shall say

thatfactorI * representsa socialismdimensionand that factor * repre- I,

sents a pacifismdimension.This distinctioncertainlymakes sense con-

ceptually,althoughin the originalstudy this two-factor predictionwas

not made in advance of seeingthe factoranalysis.

Aside fromthe conceptualadvantagesof the two-factor solution,

there are some importantquantitativeadvantages as well1.First, the

two-factorsolutionaccounts for substantiallymore variance than the

single-factorsolution,witheach factoraccountingfora littlemorethan

21 percentof the total variance. Second, scale reliabilitiescan be in-

creasedsubstantially.The 0* reliabilitiesforthe scales based on the full

13-itemfactorscoresare somewhatlowerthan the unrotatedfactorn,

with tlieta-star0.70 forboth. If we are guided by the highestrotated

loadingsin Table 6 and clustersshownin Figure2, however,and if we

refactoritems2, 4-6, 10, 12, and 13 forthe socialismscale and items1,

8, and 11 forthe pacifismscale, we thenhave 0' reliabilitiesof0.74 and

0.85. The reliabilityof the socialismscale is slightlyhigherthan the

original13-itemalpha reliability;the pacifismscale reliabilityis con-

All use subject to JSTOR Terms and Conditions

THETA RELIABILITY AND FACTOR SCALING 45

are 0.73 and 0.85, so that weightedscoringis unnecessaryand either

alpha or theta can be used to estimatereliability.

In effectthe factoranalysis enables us to find two relatively

homogeneousclustersof itemsthat can be used to constructtwo inde-

pendent subscales which closely approximatethe basic assumptions

behindthe alpha coefficient.

A slightlymorereliablescale (0 = 0.75) can be obtainedby add-

ingitems7 and 9 to thefactorI *scale, but thisdoes not seemjustifiable

in view of the largeloadingthat item9 has on factorII *. Moreoverthis

raises the correlationbetweenthe two subscales from0.09 (indicating

practicallyindependentscales) to 0.20. Addingitem 9 to the factor11*

scale substantiallyreduces the latter's reliability.Basically Figure 2

shows that items7 and 9 fall betweenthe two major clustersand that

theirinclusionin one scale or the other eitherdecreases reliabilityor

does not sufficiently increasereliabilityto justifythe resultantloss of

scale independence.

This examplemakes it clear that factorscalingcan substantially

improve scale reliability,particularlywhen multiple dimensionsare

present.Reliabilitycan be improvedin a single-factor case by identifica-

tionofitemsthatdo not contributeto the centraldimension.But when-

ever multipledimensionsare present,factorscalingis even moreimpor-

tant. In the multiple-factorcase the reliabilityof factorscoresis often

considerablylowerthanthereliabilityoffactorscales (otherthingsbeing

equal). In the presentexamplethe rotatedfactorscorereliabilitieswere

0.70 each; they were improvedto 0.74 and to 0.85 when factorscales

wereconstructed.These factorscales had one further advantage:it made

littledifference whetheralpha (unweighted)or theta (weighted)coeffi-

cientswereused to estimatereliability.

Validity

I have stated that the ultimatetest of the importanceof a scale

is not simplyits internalconsistency.The acid test of a sociological

measureis its validity; that is, whetherit is related to other,external

sociologicalvariableswithintheframework ofa model.If scale reliability

is importantand if we believe that we have enhanced the conceptual

clarityand reliabilityof our measureof politicalideology,we shouldbe

able to show improvedrelationshipswithothervariables.

I mentionedthat the purposeofbuildinga politicalideologyscale

was to relateit to supportforor oppositionto the VietnamWar. Using

a scale to measureattitudeson Vietnam,we can testtherelativevalidity

of the covarianceand factorapproachesby comparingthe predictive

All use subject to JSTOR Terms and Conditions

46 DAVID J. ARMOR

pacifismscales derivedby the methodof factorscaling.

The VietnamWar scale is scoredso that a highscore indicates

strongsupportforthe war and a low score denotesstrongopposition.

The regressionresultscan be summarizedas follows:

R2 withVietnam

War Scale

1. Covariancescaling: 13-itemsummedscale 0.26

2. Factor scaling:pacifismscale alone 0.25

pacifismplus socialismscale 0.30

In otherwordsthemodelthatuses thetwo morereliablescales raisesthe

explainedvariancefrom26 to 30 percentforour externalvariable.This

is directevidencethat the use of factorscalingto developmorereliable

scales can have a substantialimpacton the testofa model,even whena

covariancescale has an acceptablelevel of reliability.

the relationshipbetweentheta and omega,a new reliabilitycoefficient

proposedby Heise and Bohrnstedt(1971). Their workrepresentsan im-

portantcontributionto understandingthe relationshipbetweeinfactor

analysisand reliability.They take a unique approachby usingthe tech-

nique of path analysistogetherwithtraditionalfactoranalysisto derive

measuresof reliabilityand validity.At the momentwe are most inter-

ested in theiromega,whichlike thetais based on factorainalysis.

Assumingstandardizeditems for convenience,the formulafor

omega reliabilityis

Q ( Erti?+Pr)

+L ) ( (25)

omega and theta stems fromthe relianceof omega on the conceptof

communalityhi. In the classic theoryof factoranalysis, communality

is definedas that componentof an item's variance shared in common

withthe remainingset of itemsbeingfactor-analyzed. It is to be distin-

guishedfromunique variance-that part of an item's variationdue to

the item alone (includingerrorcomponents).Unfortunately, since the

communalitydepends oIn the numberof factorsextractedthereis no

analyticmethodforcomputinga unique,invariantcommunality foreach

item.That is, since the communalityforitemi is definedas

2 2 2 2

h. =ail+ ai2 + +

*. faim (26)

solutionand not a

(where the aik are derived froma common-factor

All use subject to JSTOR Terms and Conditions

THETA RELIABILITY AND FACTOR SCALING 47

ingsof an itemon the m factorsextracted,we do not have an invariant

h2 unlessthenumberoffactorsm is also invariant.But as I have pointed

out,thenumberofmeaningful factorscannotbe determinedanalyticallv;

it dependson the typeof factoranalysisused and, mostof all, on inter-

pretivejudgmentsmade by the analyst.Thus forthe same set of items

omega differs fromone analysisto anotherdependingon themethodand

investigator.In generalthe greaterthe numberof factorsextracted,the

greaterthe omega coefficient.

Since experiencedsocial scientistshave acquired a hightolerance

forambiguity,the variant propertiesof omega may not be viewed as

seriousones. Thereis, however,a secondand moreseriousdifference be-

tweenomega and theta.We have seen that thereis no restriction on the

hypothesizednumberof factorsforomega. But if an analyst decides

thereare reallytwo or moreindependentfactorsin a set of items,each

factorshouldresultin a separatescale. That is the wholepointof apply-

ing factoranalysisin the firstplace. From the standpointof reliability

theoryand covariancescaling,thereis no morejustification in combining

uncorrelatedclustersof items into a singlescale than thereis in com-

bininga set of uncorrelatedindividualitems into a single scale. Thus

omega does not assess the reliabilityof separate scales in the event of

multipledimensions.

The comparisonof omega and theta can be done meaningfully

only if we considerthe single-factorcase. In additionit is importantto

2

stress that the hi for omega depends on a common-factorsolution, such

as the maximum-likelihood

method (Lawley and Mlaxwell,1963). The

a2 obtained from a principal-component solution should not be used in

2

Equation (26) to estimatethe hi. If thisis done the omega formulare-

sultsin a reliabilityestimatethat is spuriouslyhigh.'4

Assumingthat these conditionsare met, what can we say about

the differencebetweenomega and theta?First,practicalexperiencewith

common-factor and principal-component solutions reveals few differ-

ences betweenthe relativemagnitudesof the factorloadings.That is,

althoughthe loadingsdifferin absolute value both methodsgenerally

rank the itemsin the same orderwith respectto theircontributionto

the firstfactor.Thus if a scale is formedby takingthe items withthe

highestloadings,both methodsshould resultin similarscales. Second,

experiencealso indicatesthat the values of omega and theta are gener-

ally comparable,withvariationsusuallyunder0.01. The main pointto

14 If all intercorrelations

equalr, thena, 0, and Q shouldbe equal.But

using principal-componentloadings gives Q = a + (1 - r)/pX,, making

Q> or 0 unlessr = 1.

All use subject to JSTOR Terms and Conditions

48 DAVID J. ARMOR

componentanalysisand thatomegashouldbe used withcommon-factor

hypothesis.

solutionsassuminga single-factor

SUMMARY

liningthe steps that investigatorsshould take to develop a composite

measure of some sociological construct.Although the basic idea of

covariancescalingis sound,sinceit is moreappropriateformanysocio-

logical conceptsthan otherscaling methods,it has several weaknesses

in the traditionalways it is applied to sociologicalvariables.Such steps

as item-to-scale correlations and alpha reliabilitydo not take properac-

countof varyingitemcontributions to a construct,and theycannotun-

cover the existenceof multiple,independentconstructsthat mightbe

presentin a set of items.Theta reliabilityand factorscaling,based on

traditionalprincipal-component factoranalysis,is offeredas a method

ofimprovingscale reliabilityand conceptualclarity.It presentsa means

of discoveringmultidimensionality and allows items to relate differen-

tially to these dimensions.These methodshave the advantage of con-

nectingmultidimensional scaling closelyto the traditionalconceptsof

measurementreliabilityand scalability.

Perhapsthebest way to summarizethetechniqueoffactorscaling

is to listthestepsan investigator can followwhenapplyingthetechnique

forimprovingscale reliability.

First,select a set of variables that representsa domain within

whichone or morescales or indicesis predicted.

Second,omitany subjects(or otherunitsofanalysis)withmissing

observationsforthepurposeoffactorscalingand reliabilitycomputation

(factoranalysiswith mean substitutionfor missingdata gives a good

approximation).

Third, apply a principal-component factoranalysis, extracting

factorsuntileithertherootsapproach1 or therootsbeginto taperoffby

approximately equal decrements.Onlyrarelywillit be necessaryto take

morethan fouror fivefactors;even feweris the norm.

Fourth,if a single-factor hypothesisis supportedby the factors,

the unrotatedfirst-factor loadingscan be used forinterpreting the scale.

If regularfactorscoresare used to create the scale, the theta formula of

Equation (9) shouldbe used forreliability.

Fifth,if the factorrevealsitemswithsmall loadings (say, under

0.3 or 0.4) higherreliabilitiescan be obtainedby eliminatingthe weak

itemsaltogether.In thiscase a simplesummedscale can be created(as-

suming approximatelyequal item variances; if not, items should be

All use subject to JSTOR Terms and Conditions

THETA RELIABILITY AND FACTOR SCALING 49

mate. Alternatively thesubsetcan be refactored in orderto obtainfactor

scoresand the corresponding theta coefficients.

Sixth, if a multiple-factorsolution is suggestedon eithercon-

ceptual or empiricalgrounds,rotationsof two or morefactorsshouldbe

done (usingthe varimaxor some similarmethod)untilan interpretable

solutionis obtained.Althoughthe rotatedfactorscorescan be used for

scales withreliabilitiescomputedaccordingto Equation (10), reliabilities

may be substantiallyhigherif only the highest-loadeditems on each

factorare used forscales. In this case, as with single-factor solutions,

scales can be constructedby simplesummationand reliabilityestimates

can be made usingalpha; alternativelyeach subsetcan be refactoredfor

factorscoresand theta.

I close with several cautions. First, reliabilityas we have dis-

cussed it pertainsonly to the internalconsistencyof a scale. High relia-

bilityis no guarantee,in and of itself,of a usefuland meaningfulcon-

struct.Second, thereare several otherrationalesformultidimensional

scalingand the covariancemodel may not be the best one forall socio-

logicalapplications.In particular,althoughnonmetricmultidimensional

scalingis not tied closelyto reliabilitytheorytheremay be applications

wherereliability(as defined)is not of primaryconcernand wherethe

nonmetricmethodsare more appropriatelytailoredto the problemat

hand.

Finally,even ifan analystfindsthe assumptionsof factorscaling

suitedto hisproblemhe cannotbe certainthatfactorscalingwillimprove

his scalingproblem.On the one hand the experiencedsociologistwho is

especiallysensitiveto the substantivemeasurementissuesin his fieldof

interestmay be able to constructhighlyreliablescales withlittleor no

quantitativeassistance (althoughhe may want to have a check on his

intuition!).On the other hand factorscaling cannot produce reliable

constructsout of a hodge-podgeof items assembledwithoutregardto

conceptualdomain and content.The view presentedhereis that factor

scaling is a quantitativeadjunct to scientificintuitionand judgment

thatmakesit possibleforall analysts-not just theparticularlysensitive

ones-to producecomparableresults.

REFERENCES

1967 "Professors'attitudes toward the Vietnam war." Public Opinion

Quarterly(Summer):159-175.

BUTLER, P. M.

1968 "Alpha-maximizedfactor analysis and its relation to alpha and

cannonicalfactoranalysis." Psychometrika

33 (March) :335-346.

All use subject to JSTOR Terms and Conditions

50 DAVID J. ARMOR

tomeasurement

andthecausalapproach

1969 "Multipleindicators error."

Journal

American 75 (September):264-272.

ofSociology

BOHRNSTEDT, G. W.

1969 "A quick methodfordetermining and validityof

the reliability

scales."American

multiple-item Review

Sociological :542-

34 (August)

548.

CRONBACH, L. J.

alphaandtheinternal

1951 "Coefficient oftests."Psychometrika

structure

16 (September):297-334.

Testing.(2nded.) New York:Harper&

1960 EssentialsofPsychological

Row.

EDWARDS, A. L.

of Attitude

1957 Techniques New York: Appleton

Scale Construction.

Crofts.

Century

GUILFORD, J. P.

NewYork:McGraw-Hill.

Methods.

1954 Psychometric

HARMAN, H. H.

1967 ModernFactorAnalysis.(2nded.) Chicago:University

of Chicago

Press.

HEISE, D. R.

and stabilityin test-retest

1969 "Separatingreliability correlations."

American Review34 (February):93-101.

Sociological

HEISE, D. R. AND BOHRNSTEDT, G. W.

1971 "Validity,invalidity, Pp. 104-129inE. F. Borgatta

andreliability."

San Francisco:

Methodology.

(Ed.), Sociological Jossey-Bass.

LAWLEY, D. N. AND MAXWELL, A. E.

1963 FactorAnalysisas a Statistical London:Butterworth.

Method.

LORD, F. M.

1958 "SomerelationsbetweenGuttman'sprincipal of scale

components

analysisand otherpsychometric (Decem-

theory."Psychometrika

ber):291-296.

LORD, F. M. AND NOVICK, M. R.

1968 Statistical ofMentalTestScores.Reading,Mass.:Addison-

Theories

Wesley.

NOVICK, M. R. AND LEWIS, C.

alphaand thereliability

1967 "Coefficient measurements."

ofcomposite

32 (March):1-13.

Psychometrika

SIEGEL, P. M. AND HODGE, R. W.

1968 "A causalapproachto thestudyofmeasurement error."In H. M.

in Social Re-

Blalock,Jr.and A. B. Blalock(Eds.), Methodology

search.NewYork:McGraw-Hill.

UPSHAW, H. S.

1968 "Attitude measurement."In H. M. Blalock,Jr.and A. B. Blalock

NewYork:McGraw-Hill.

in SocialResearch.

(Eds.), Methodology

All use subject to JSTOR Terms and Conditions

- Wahyu_Widhiarso_1 (1)Uploaded byJose Luis Ventura Leon
- Dissertation 007Uploaded byJaybird9er
- Developing a ChecklistUploaded byHà Vy
- Done2Uploaded byArshad Hussain
- Mohammad Mansour Fawwaz Al-KhasawnehUploaded byNije Mohd
- mb0040Uploaded byjobin
- yjmt-23-188Uploaded byebrahimpan
- Measure Criativity Haberland DacinUploaded byiasilvac
- About DISC.pdfUploaded byjoeaston
- AJBMR_17_25i1n9a1Uploaded byvvunited
- 1-s2.0-S0966636208002646-main.pdfUploaded byWojciech Kurzydło
- udsm-muce-ep300-basic-concepts-emr.docxUploaded bykitmad
- Mathematical Symbols ListUploaded byHammad Rehman
- SternbergUploaded byNoorleha Mohd Yusoff
- BE4 ResearchUploaded byNuuradiin Xasan Dhaqane
- reliabilUploaded byAmit A
- Lampiran 5Uploaded byyunita
- Reliability and validity are central to judgments about the quality of research in the natural sciences and quantitative research in the social sciences.docxUploaded byConstance Lima-gonzalez
- Chapter 5 MeasurementUploaded byTangMei
- 7448-26945-1-PBUploaded bySachin Manjalekar
- paper2Uploaded byTauqeer Syed
- CITIZENS’ ACCEPTANCE OF E-GOVERNMENT SERVICESUploaded byAnonymous Gl4IRRjzN
- SPSSUploaded byNur Ayu
- Advanced TestUploaded byHarsh Mehta
- Factors that Influence the Service Experience.pdfUploaded bynewaznahian
- Sampling .pptxUploaded byGeraldJoelBillih
- Propensity Score MatchingUploaded byHe H
- hasil univariatUploaded byuswatun
- MT1.S15.SolvedUploaded byPPP
- Tutorial 01Uploaded byAnonymous pcanQ3

- Coaching Model: Lisa Sennhauser-Kelly The LIVE Coaching Model – incremental transformation to achieve your goalsUploaded byCoach Campus
- Hawthorne Studies.pptUploaded bychristian
- Building a Coaching Culture Report 2014Uploaded byAcuy Sang Pemimpi
- 2nd Sem Concept PapaerjayUploaded bymaylmutia
- For and against essay.pdfUploaded byAna Clara Valdez
- Counseling MattersUploaded byclee6336
- 7.Dimensiones Culturales en La Empresa_publicacion_cUploaded byAlejandro Martin
- Fact and OpinionUploaded byRestu Dwi S
- obesity lesson planUploaded byapi-371977679
- Spector -2008- Grounds of Moral Agency Locke-s Account of Personal IdentityUploaded byeffetpapillon
- Communication SkillsUploaded byAhmad
- fumage daliUploaded byTeofil Stiop
- We Have a Future- An Interview With Sharon Hayes JULIA BRYAN-WILSONUploaded byerrazu4890
- Spring 2016_ENG301_3Uploaded bySadaf Latif
- Elementary Reading Comprehension Test 01Uploaded byAmairani Guarneros
- Kelloggs PptUploaded bydauvbafna
- Cognitive Approaches to SLA - Manual Za Risrč u Ovom PodručjuUploaded bytiquismiquis
- single lesson plan week 5Uploaded byapi-316651663
- Comparing Methods for Measuring EmotionsUploaded bynesuma
- Embodied Research and WritingUploaded byonlyying
- A Doll House- Character Analysis EssayUploaded byReginald D. White
- L&DLuentokalvot.pptUploaded bySai Kumar
- Samuel a. Stouffer El Soldado AmericanoUploaded bylubar56
- Critique of Disobedience as a Physiological and Moral ProblemUploaded byShaheryar Khan
- PSYCH 328 SyllabusUploaded bymiwa921
- Ellen Jaye Benson: Philosophy of Visual Art Teaching StatementUploaded byEllen Jaye Benson
- ORIGINAL - Language Anxiety Among Malaysian Gifted StudentsUploaded byhasrulkamarul
- mogt- nov 09Uploaded byapi-281359843
- Maha Bodhi Journal 1972-04Uploaded bySameer
- New Microsoft Office Word Document (3).docxUploaded byShivam Saxena