A PRIMER OF a ree STATISTICS Tie ae Richard J, HarrisA Primer of Multivariate Statistics, Third Edition Richard J. Harris University of New Mexico [EA LAWRENCE ERLBAUM ASSOCIATES, PUBLISHERS 2001 Mahwah, New Jersey LondonThe final camera copy for this work was prepared by the author, and there- fore the publisher takes no responsibility for consistency or correctness of typographical style. However, this arrangement helps to make publication of this kind of scholarship possible. Copyright © 2001 by Lawrence Erlbaum Associates, Inc. All rights reserved. No part of this book may be repro- duced in any form, by photostat, microform, retrieval system, or any other means, without prior written permission of the publisher. Lawrence Erlbaum Associates, Inc., Publishers 10 Industrial Avenue Mahwah, NJ 07430 Cover design by Kathryn Houghtaling Lacey Library of Congress Cataloging-in-Publication Data Harris, Richard J. A primer of multivariate statistics / Richard J. Harris.—3rd ed. p. cm. Includes bibliographical references and index. ISBN 0-8058-3210-6 (alk. paper) 1. Multivariate analysis. I. Title. QA278 .H35 2001 519.5'°35—dc21 2001033357 CIP Books published by Lawrence Erlbaum Associates are printed on acid-free paper, and their bindings are chosen for strength and durability. Printed in the United States of America 1009 8 7 65 43 2 =41Dedicated to classmates and colleagues at Lamar (Houston) High School, Class of ’58; at Caltech ’58-’61; in the Stanford Psychology doctoral program ’63—'68; and at the University of New Mexico 68-01. Thanks for setting the bar so high by your many achievements and contributions.A Primer of Multivariate Statistics Third Edition Richard J. Harris University of New Mexico Preface to the Third Edition Well, as I looked over the syllabus and the powerpoint presentations for my multivariate course I realized that the material I was offering UNM students had progressed beyond what’s available in the first two editions of the Primer of Multivariate Statistics. (You did ask “Why did you bother revising the Primer?”, didn’t you?) That of course begs the question of why I didn’t just adopt one of the current generation of multivariate textbooks, consign the Primer of Multivariate Statistics to the Museum of Out-of-Print Texts, and leave the writing of statistics texts to UNM colleagues and relatives. (See M. B. Harris, 1998, and Maxwell & Delaney, 2000.) First, because somebody out there continues to use the Primer, which at the turn of the millenium continues to garner about fifty SCZ and SSCI citations annually. Sheer gratitude for what this has done for my citation counts in my annual biographical reports suggests that I ought to make the Primer a more up-to-date tool for these kind colleagues. Second, because I feel that the current generation of multivariate textbooks have moved too far in the direction of pandering to math avoidance, and I vainly (probably in both senses) hope that renewed availability of the Multivariate Primer will provide a model of balance between how-to and why in multivariate statistics. Third, because of the many new examples I have developed since the second edition. I have found that striking, “fun” data sets are also more effective. As you work your way through this third edition, look for the Presumptuous Data Set, the N = 6 Blood Doping dataset, the Faculty Salary data showing an overall gender bias opposite in direction to that which holds in every individual college, and the tragic tale of the Beefy Breasted Bowery Birds and their would-be rescuer, among other dramatic datasets both real and hypothetical. Fourth, because a good deal of new material (ammunition?) has become available in the effort to convince multivariate researchers and authors that they really should pay attention to the emergent variables (linear combinations of the measures that were “fed into” your MRA or Manova or Canona) that actually produced that impressive measure of overall relationship — and that these emergent variables should be interpreted on the basis of the linear combination of the original variables that generates or estimates scores on a given emergent variable, not on structure coefficients (zero-order correlations). Several of the new datasets mentioned in the previous paragraph reinforce this point. I am especially pleased to have found a colleague (James Grice) with the intellectual vuvill Preface commitment and the skill actually to carry out the Monte Carlo analyses necessary to show how much more closely regression-based estimates of (orthogonal) factors mimic the properties of those factors than do loadings-based estimates. (Cf. the discussion in Chapter 7 of Dr. Grice’s dissertation and of Harris & Grice, 1998.) Fifth, because structural equation modeling has become so ubiquitous in the literature of so many areas, so user friendly, so readily available, and so unconstrained by cpu time that it simply must be represented in any attempt at a comprehensive treatment of multivariate statistics. The focus of this third edition is still on the “classic” multivariate techniques that derive emergent variables as deterministic linear combinations of the original measures, but I believe that it provides enough of a taste of latent-variable approaches to give the reader a feel for why she should consider diving into more detailed treatments of confirmatory factor analysis, SEM, etc. Some of the above reasons for my feeling compelled to produce a third edition can also be seen as reasons that you and all the colleagues you can contact should consider reading the result and prescribing it for your students: e New coverage of structural equations modeling (Chapter 8), its manifest-variable special case, path analysis (Chapter 2), and its measurement-model-only special case, confirmatory factor analysis (Chapter 7). e New, more interesting and/or compelling demonstrations of the properties of the various techniques. Additionally, the ease with which multivariate analyses can now be launched (and speedily completed) from anyone’s desktop led me to integrate computer applications into each chapter, rather than segregating them in a separate appendix. Unsurprisingly, I feel that the strengths of the first two editions have been retained in the Multivariate Primer you’re about to read: One of the reviewers of the second edition declared that he could never assign it to his students because I took too strong a stand against one-tailed tests, thereby threatening to curb his right to inculcate unscientific decision processes in his students. (Well, he may have put it a bit differently than that.) As a glance through Harris (1997a, b) will demonstrate, I feel at least as strongly about that issue and others as I did fifteen years ago, and although I try to present both sides of the issues, I make no apology for making my positions clear and proselytizing for them. Nor do I apologize for leaving a lot of references from the “good old” (19)70’s, 60’s, and 50’s and even earlier (look for the 1925 citation) in this edition. Many of the techniques developed “back then” and the analyses of their properties and the pros and cons of their use are still valid, and a post-Y2K date is no guarantee of ... well, of anything. Naturally there have been accomplices in putting together this third edition. First and foremost I wish to thank Publisher Larry Erlbaum, both on my behalf and on behalf of the community of multivariate researchers. Larry is a fixture at meetings of the Society for Multivariate Experimental Psychology, and the home Lawrence Erlbaum Associates has provided for Multivariate Behavioral Research and for the newly launched series of multivariate books edited by SMEP member Lisa Harlow has contributed greatly to the financial stability of the society and, more importantly, to making multivariate techniques broadly available. Personally I am grateful for Larry’s quick agreement toPreface 1X consider my proposal for LEA to publish a third edition of the Multivariate Primer and for his extraordinary forbearance in not abandoning the project despite my Jong delay in getting a formal prospectus to him and my even longer delay getting a complete draft to Editors Debra Riegert and Jason Planer (Editor Lane Akers having by then moved on to other projects). Elizabeth L. Dugger also made a major contribution to the book (and to helping me maintain an illusion of literacy) through her extremely detailed copyediting, which went beyond stylistic and grammatical considerations to pointing out typos (and displays of sheer illogicality) that could only have been detected by someone who was following the substance of the text in both its verbal and its mathematical expression. I would also like to thank three reviewers of the prospectus for the third edition, Kevin Bird (University of New South Wales), Thomas D. Wickens (UCLA) and Albert F. Smith (Cleveland State University) for their support and for their many constructive criticisms and helpful suggestions. Dr. Bird was a major factor in getting the second edition of the Primer \aunched (“feeding” chapters to his multivariate seminar as fast as I could get them typed and copied), and he was on hand (via the wonder of intercontinental email) to be sure the third edition didn't stray too far from the core messages and organization of the first two editions. In addition to reviewing the prospectus, Dr. Wickens has also helped maintain my motivation to prepare a third edition by being one of the intrepid few instructors who have continued to use the second edition in their courses despite its having gone out of print. He has also made my twice-a-decade teaching of a seminar on mathematical modeling in psychology easier (thus freeing up more of my time to work on the Primer revision) by making available to me and my students copies of his out-of-print text on Markov modeling (Models for Behavior, 1982). They also serve who only have to read the darned thing. Thanks to the UNM graduate students whose attentive reading, listening, and questioning (beloved are those who speak up in class) have helped me see where better examples, clearer explanations, etc. were needed. I’m especially grateful to the most recent class (Kevin Bennett, Heather Borg- Chupp, Alita Cousins, Winston Crandall, Christopher Radi, David Trumpower,and Paula Wilbourne), who had to forego a complete hardcopy, instead pulling the most recent revision of each chapter off the web as we progressed through the course. Paula was especially brave in permitting her dissertation data to be subjected to multivariate scrutiny in class. I’ve thanked Mary, Jennifer, and Christopher for their patience with my work on the first two editions, but a family member (Alexander) who arrived after publication of the second edition surely deserves to get his name mentioned in a Primer preface. Thanks, Alex, for being willing to rescue a member of the hopeless generation (with respect to real computers, anyway) on so many occasions, and for putting up with many months of “once the book’s shipped off” excuses. So, why are you lingering over the witty prose of a preface when you could be getting on with learning or reacquainting yourself with multivariate statistics? Dick Harris November 2000Contents 1 The Forest before the Trees 1.0 Why Statistics? 1 1.01 Statistics as a Form of Social Control 1 1.02 Objections to Null Hypothesis Significance Testing 2 1.03 Should Significance Tests be Banned? 3 1.04 Math Modeling's the Ultimate Answer 5 1.05 Some Recent Developments in Univariate Statistics 6 1.0.5.1 The MIDS and FEDs criteria as alternatives to power calculation 7 Table 1.1 Fraction of Population Effect Size That Must Be Statistically Significant in Order to Achieve a Given Level of Power for Your Significance Test 9 1.0.5.2 Prior-Information Confidence Intervals (PICIs) 9 1.1 Why Multivariate Statistics? 10 1.1.1 Bonferroni Adjustment: An Alternative to Multivariate Statistics. 13 1.1.2 Why Isn’t Bonferroni Adjustment Enough? 14 1.2 A Heuristic Survey of Statistical Techniques 14 Table 1.2 Statistical Techniques 16 1.2.1 Student’s ttest 17 1.2.2 One-Way Analysis of Variance 18 1.2.3 Hotelling’s 7 21 Example 1.1 Anglo versus Chicano Early Memories 23 1.2.4 One-Way Multivariate Analysis of Variance 24 Example 1.2 Inferring Social Motives from Behavior 25 1.2.5 Higher Order Analysis of Variance 26 1.2.6 Higher Order Manova 27 Example 1.3 Fat, Four-eyed, and Female 28 1.2.7 Pearson r and Bivariate Regression 28 1.2.8 Multiple Correlation and Regression 31 Example 1.4 Chicano Role Models, GPA, and MRA — 33 1.2.9 Path Analysis 34 1.2.10 Canonical Correlation 35 Figure 1.1] Multivariate Analyses of Between-Set Relationships 36 Example 1.5 Television Viewing and Fear of Victimization 37 1.2.11 Analysis of Covariance 38 1.2.12 Principal Component Analysis 40 1.2.13 Factor Analysis 42 Example 1.6 Measuring Perceived Deindividuation 44 1.2.14 Structural Equation Modeling 44 XIXi Contents 1.3 Learning to Use Multivariate Statistics 45 1.3.1 A Taxonomy of Linear Combinatons 45 1.3.1.1 Averages of subsets of the measures 45 1.3.1.2 Profiles 47 1.3.1.3 Contrasts 47 1.3.2 Why the Rest of the Book? 51 Quiz 1 See How Much You Know after Reading Just One Chapter! 55 Sample Answers to Quiz1 56 2 Multiple Regression: Predicting One Variable from Many Data Set1 58 2.1 The Model 59 2.2 Choosing Weights 62 2.2.1 Least Squares Criterion 62 Table 2.1 Multiple Regression Analyses of DataSet 1 66 Table 2.2 Data Set 1b: A Presumptuous Data Set 68 2.2.2 Maximum Correlation Criterion 69 2.2.3 The Utility of Matrix Algebra 70 2.2.4 Independence of Irrelevant Parameters 72 2.3 Relating the Sample Equation to the Population Equation 74 Table 2.3 Summary of Significance Tests for Multiple Regression 77 2.3.1 R, versus S, versus x’x as the Basis for MRA 81 Table 2.4 Alternative MRA Formulae 84 2.3.2 Specific Comparisons 84 2.3.3 Illustrating Significance Tests 86 Example 2.1 Locus of Control, the CPQ, and Hyperactivity 86 Computer break 2-1: CPQ vs. LOC, CPT-C, CPT-E 89 2.3.4 Stepwise Multiple Regression Analysis 95 Example 2.1 Revisited 96 2.4 Computer Programs for Multiple Regression 96 2.4.1 Computer Logic and Organization 97 2.4.2 Sage Advice on Use of Computer Programs 98 2.4.3 Computerized Multiple Regression Analysis 100 2.4.3.1 MATLAB _ 100 2.4.3.2 SPSS REGRESSION, Syntax Window 101 2.4.3.3 SPSS REGRESSION , Point-and-Click 102 2.4.3.4 SAS PROC REG and PROC RSQUARE __ 103 2.5 Some General Properties of Covariance Matrices 105 2.6 Measuring the Importance of the Contribution of a Single Variable 107 Table 2.5 Measures of Importance in MRA 110Contents Xiii 2.7 Anova viaMRA _ 103 Table 2.6 Relationship Between MRA and Anova Effects Model 105 Example 2.2 In-Group/Out-Group Stereotypes 106 Table 2.7 Coding of MRA Level-Membership Variables for Study of Stereotypes 113 Example 2.3 Negative Shares and Equity Judgments 106 Table 2.8 Alternative Codings of MRA Predictor Variables, Equity Study 113 Example 2.4 Gender Bias in Faculty Salaries? 116 Table 2.9 Mean Faculty Salary at Hypo. U. as f(College, Gender) 117 Table 2.10 Data for MRA-Based Anova of Gender-Bias Data Set 118 2.8 Alternatives to the Least-Squares Criterion 121 2.9 Path Analysis 122 2.9.1 Path analytic Terminology 123 2.9.2 Preconditions for Path Analysis 124 2.9.3 Estimating and Testing Path coefficients 126 2.9.4 Decomposition of Correlations into Components 128 2.9.5 Overall Test of Goodness of fit 129 2.9.6Examples 130 Example 2.5 Mother's Effects on Child'sIQ 130 Example 2.6 Gender Bias Revisited: More Light on "Suppression" 134 2.9.7 Some Path-Analysis References 136 Demonstration Problem 136 Answers 139 Some Real Data and a Quiz Thereon 143 Table 2.11 Data Set 2: Ratings of Conservatism of Statement 144 Answers 146 Table 2.12 Buildup of R’ for Different Orders of Addition of Predictors 146 Figure 2.1 Venn diagram of correlations among Y and four predictors 147 Path Analysis Problem 149 Answers to Path Analysis Problem 150 3 Hotelling’s 7°: Tests on One or Two Mean Vectors 3.1 Single-Sample tand 7 155 Table 3.1 Data Set 3: Divisions of Potential Prize, Experiment 3, Harris & Joyce (1980). 157 Example 3.1 162 3.2 Linearly Related Outcome Variables 165 Example 3.2 166 Table 3.2 Data Set 4: Results of Deutsch Replication 1 167 3.3 Two-Samplet and7 170 3.4 Profile Analysis 173 Figure 3.1 Response vectors for groups differing in level and slope 174XIV Contents 3.5 Discriminant Analysis 182 3.6 Relationship between T and MRA 184 3.7 Assumptions Underlying T 186 3.7.1 The Assumption of Equal Covariance Matrices 186 3.7.2 Known Covariance Matrix 187 3.7.3 The Assumption of Multivariate Normality 188 3.8 Analyzing Repeated-Measures Designs viaT 188 Table 3.3 Repeated-Measures Anova of Data Set3 189 Example 3.2 Blood Doping 192 Table 3.4 10K Running Time as Affected by an Infusion of One’s Own Blood 192 3.9 Single-Symbol Expressions for Simple Cases 196 3.10 Computerized T 198 3.10.1 Single-Sample and Two-Sample T 198 3.10.2 Within-Subjects Anova 199 Demonstration Problems 200 Answers 202 4 Multivariate Analysis of Variance: Differences Among Several Groups on Several Measures 4.1 One-Way (Univariate) Analysis of Variance 210 4.1.1 The Overall Test 210 Table 4.1 Summary Table of Anova on Dependent Variable 213 4.1.2 Specific Comparisons 213 Table 4.2 Summary Table for Effects of Instructions on Frequency of DD Outcomes 215 4,2 One-Way Multivariate Analysis of Variance 218 Table 4.3 Critical Values for Contrasts Performed on Linear Combinations of Variables 222 4.3 Multiple Profile Analysis 224 Example 4.1 Damselfish Territories = 227 Table 4.4 Mean Percentage Coverage of Damselfish Territories 227 4.4 Multiple Discriminant Analysis 229 4.5 Greatest Characteristic Roots versus Multiple-Root Tests in Manova 231 4.5.1 "Protected" Univariate Tests 233 4.5.2 Simultaneous Test Procedures and Union Intersection 234 4.5.3 Invalidity of Partitioned-U Tests of Individual Roots 234 4.5.4 Simplified Coefficients as a Solution to the Robustness Problem 237 4.5.5 Finite-Intersection Tests 238Contents XV 4.6 Simple Cases of Manova 240 4.7 Higher Order Anova: Interactions 243 4.8 Higher OrderManova 245 Example 4.2 Eyeball to Eyeball ina Prisoner's Dilemma 248 Table 4.5 Proportion of Mutually Cooperative Choices as f(Contact, Communic'n) 248 Table 4.6 Mean Proportion of Total Responses Accounted for by Each Outcome 249 Table 4.7 Summary Table for Anova on Discriminant Function from One-Way Manova 251 4.9 Within-Subject Univariate Anova Versus Manova 252 Example 4.3 Stress, Endorphins, and Pain 256 4.10 Computerized Manova = 257 4.10.1 Generic Setup for SPSS MANOVA 257 4.10.2 Supplementary Computations 259 4.10.3 Pointing and Clicking toa Manovaon SPSS PC —_ 259 4.10.4 Generic Setup for SAS PROC GLM ~—_. 260 Demonstration Problems 262 Answers 264 5 Canonical Correlation: Relationships Between Two Sets of Variables 5.1 Formulae for Computing Canonical Rs 268 5.1.1 Heuristic Justification of Canonical Formulae 270 5.1.2. Simple Cases of Canonical Correlations 272 5.1.3. Example of a Canonical Analysis 274 Table 5.1 Correlations of Background Variables with Marijuana Questions 275 Table 5.2 Canonical Analysis of Background Variables versus Marijuana Questions 276 5.2 Relationships to Other Statistical Techniques 277 5.3 Likelihood-Ratio Tests of Relationships between Sets of Variables 279 5.4 Generalization and Specialization of Canonical Analysis 280 5.4.1 Testing the Independence of m Sets of Variables 281 Example 5.2 Consistency of Behavior across Different Experimental Games 282 Table 5.3 Correlation Matrix for Game OutcomeVariables, Flint (1970) 283 5.4.2 Repeated-Battery Canona 284 5.4.3 Rotation of Canonical Variates 288 Example 5.3 A Canonical Cautionary 290 Figure 5.1 Naturally Occurring versus Canona-based Pairings of Beefy-Breasted Bowery Birds (BBBBs) 291 5.4.4 The Redundancy Coefficient 293 5.4.5 What’s Missing from Canonical Analysis? 295XVi Contents 5.5 Computerized Canonical Correlation 297 5.5.1 Matrix-Manipulation Systems 297 5.5.1.1 MATLAB 297 5.5.1.2 SAS PROC MATRIX and SPSS Matrix/End Matrix 299 5.5.2 SAS PROC CANCORR 301 5.5.3. Canona viaSPSS MANOVA — 304 5.5.4 SPSS Canona From Correlation Matrix : Be Careful 305 Demonstration Problems and Some Real Data Employing Canonical Correlation 307 Answers 309 6 Principal Component Analysis: Relationships Within a Single Set of Variables 6.1 Definition of Principal Components 319 6.1.1 Terminology and Notation in PCA and FA 320 6.1.2 Scalar Formulae for Simple Cases of PCA 322 6.1.3 Computerized PCA 325 6.1.4 Additional Unique Properties (AUPs) of PCs 326 6.2 Interpretation of Principal Components 327 Example 6.1 Known generating variables 332 6.3 Uses of Principal Components 333 6.3.1 Uncorrelated Contributions 333 6.3.2 Computational Convenience 334 6.3.3 Principal Component Analysis as a Means of Handling Linear Dependence 335 6.3.4 Examples of PCA 338 Example 6.2 Components of the WISC-R 338 Example 6.3 Attitudes toward cheating 343 Table 6.1 PCA on Questions 12-23 of Cheating Questionnaire 344 Example 6.4 Fat, four-eyed, and female again 344 Table 6.2, Manova Test of Obesity Main Effect 345 Table 6.3 PCA-Based Manova of Obesity Main Effect 347 6.3.5 Quantifying Goodness of Interpretation of Components 348 6.4 Significance Tests for Principal Components 351 6.4.1 Sampling Properties of Covariance-Based PCs 353 6.4.2 Sampling Properties of Correlation-Based PCs 354 6.5 Rotation of Principal Components 356 Example 6.1 revisited 356 6.5.1 Basic Formulae for Rotation 358 Figure 6.1 Factor structures, example 6.1 358 Figure 6.2 Rotation, general case 358 6.5.2 Objective Criteria for Rotation 360Contents XVii Table 6.4 Quadrant within which 4¢ Must Fall as Function of Signs of Numerator and Denominator of Expression (6.9) 364 6.5.3 Examples of Rotated PCs 365 Table 6.5 Intermediate Calculations for Quartimax and Varimax Rotation 365 Table 6.6 Varimax Rotation of PC,- PC,, Cheating Questionnaire 367 Table 6.7 Varimax Rotation of All Twelve PCs, Cheating Questionnaire 368 Table 6.8 Large Loadings for Cheating Questionnaire 369 6.5.4 Individual Scores on Rotated PCs 369 Example 6.5 A factor fable 373 Figure 6.3 Architectural dimensions of houses 373 Figure 6.4 Schematic representation of 27 houses 374 Table 6.9 Scores on Observed and Derived Variables for27 Houses 375 Figure 6.5 Same 27 houses sorted on basis of loadings-based interpretation of Factor 1 377 6.5.5 Uncorrelated-Components Versus Orthogonal-Profiles Rotation 379 Demonstration Problems 381 Answers 383 Figure 6.6 Rotation of factor structure for problem 1 386 ? Factor Analysis: The Search for Structure 7.1 The Model 394 7.2 Communalities 397 7.2.1 Theoretical Solution 398 7.2.2 Empirical Approximations 400 7.2.3 Iterative Procedure 401 7.2.4 Is the Squared Multiple Correlation the True Communality? 401 7.3 Factor Analysis Procedures Requiring Communality Estimates 404 7.3.1 Principal Factor Analysis 404 7.3.2 Triangular (Choleski) Decomposition 405 7.3.3 Centroid Analysis 406 7.4 Methods Requiring Estimate of Number of Factors 406 7.5 Other Approaches to Factor Analysis 409 7.6 Factor Loadings versus Factor Scores 410 7.6.1 Factor Score Indeterminacy 411 7.6.2 Relative Validities of Loadings-Derived versus Scoring-Coefficient-Derived Factor Interpretations 412 Table 7.1 Mean Validity, Univocality, and Orthogonality of Regression and Loading Estimates for Three Levels of Complexity 413 7.6.3 Regression-Based Interpretation of Factors is Still a Hard Sell 414 7.7 Relative Merits of Principal Component Analysis versus Factor Analysis 416 7.7.1 Similarity of Factor Scoring Coefficients 416 Table 7.2 Comparison of Factor Structures for PCA versus Two PFAs of Same Data 417XVili Contents Table 7.3 Comparison of Kaiser-Normalized Factor Structures 418 Table 7.4 Comparison of Factor-Score Coefficients 418 7.7.2 Bias in Estimates of Factor Loadings 420 7.8 Computerized Exploratory Factor Analysis 421 Example 7.1 WISC-R Revisited 423 7.9 Confirmatory Factor Analysis 433 7.9.1 SAS PROC CALIS Example 7.1 Revisited: Model Comparisons Galore 434 8 The Forest Revisited 8.1 Scales of Measurement and Multivariate Statistics 444 Table 8.1 Representative Critical Values for Measures of Association 446 8.2 Effects of Violations of Distributional Assumptions in Multivariate Analysis 450 8.3 Nonlinear Relationships in Multivariate Statistics 453 8.4 The Multivariate General Linear Hypothesis 456 Example 8.1 Unbalanced Manova via the multivariate general linear model 460 8.5 Structural Equation Modeling 464 8.5.1 General Approach and Examples 464 Example 8.2 Path Analysis of Scarr (1985) viaSEM 464 Example 8.3 All Three Colleges in the Faculty Salary Example 468 Example 8.4 Increment to Canonical R’ via CALIS LinEgs? 470 8.5.2 SEM Is Not a General Model for Multivariate Statistics 473 Example 8.5 Higher-Order Confirmatory Factor Analysis via SEM: WISC-R One More Time 473 8.5.3 Other User-Friendly SEM Programs 478 8.6 Where to Go from Here 479 8.7 Summing Up 480 Digression 1 Finding Maxima and Minima of Polynomials D1 .1 Derivatives and Slopes 482 D1.2 Optimization Subject to Constraints 485 Digression 2 Matrix Algebra D2.1 Basic Notation 487 D2.2 Linear Combinations of Matrices 489 D2.3 Multiplication of Matrices 489Contents X1X D2.4 Permissible Manipulations 493 D2.5 Inverses 493 D2.6 Determinants 496 D2.7 Some Handy Formulae for Inverses and Determinants in Simple Cases 500 D2.8 Rank 501 D2.9 Matrix Calculus 502 D2.10 Partitioned Matrices 503 D2.11 Characteristic Roots and Vectors 506 D2.12 Solution of Homogeneous Systems of Equations 512 Digression 3 Solution of Cubic Equations 514 Appendix A Statistical Tables A.1 -A.4 (Why omitted from this edition) 517 A.5 Greatest Characteristic Root Distribution 518 Appendix B Computer Programs Available from the Author B.1 evinter: p values and Critical Values for Univariate Statistics 532 B.2 gcrinter: Critical Values for the Greatest Characteristic Root (g.c.r.) Distribution 532 Appendix C Derivations Derivation 1.1 Per-Experiment and Experimentwise Error Rates for Bonferroni-Adjusted Tests 533 Derivation 2.1 Scalar Formulae for MRA with One, Two, and Three Predictors 536 Derivation 2.2 Coefficients That Minimize Error Also Maximize Correlation 539 Derivation 2.3 Maximizing r via Matrix Algebra 541 Derivation 2.4 Independence of Irrelevant Parameters 542 Derivation 2.5 Variances of bs and of Linear Combinations Thereof 542 Derivation 2.6 Drop in R= b, (1-R'ja,,) 543 Derivation 2.7 MRA on Group-Membership Variables Yields Same F As Anova 544 Derivation 2.8 Unweighted Means and Least-Squares Anova Are Identical in the 2 Design 545XX Contents Derivation 3.1 7° and Associated Discriminant Function 546 Single-Sample T 546 Two-Sample 7 548 Derivation 3.2 Relationship between T and MRA 549 Two-Sample ¢t Versus Pearson r With Group-Membership Variables 549 Single-Sample ¢ Test versus "Raw-Score” r,, 550 T Versus MRA 551 Derivation 4.1 Maximizing F(a) in Manova 552 Derivation 5.1 Canonical Correlation and Canonical Variates 554 Derivation 5.2 Canonical Correlation as “Mutual Regression Analysis” 556 Derivation 5.3 Relationship between Canonical Analysis and Manova 557 Derivation 6.1 Principal Components 560 Derivation 6.2 PC Coefficients Define Both Components in Terms of Xs and Xs in Terms of PCs 562 Derivation 6.3 What Does Rotation of Loadings Do to Coefficients? 564 Derivation 7.1 Near Equivalence of PCA and Equal-Communalities PFA 566 References 567 Index 584A Primer of Multivariate Statistics, Third Edition1 The Forest before the Trees 1.0 WHY STATISTICS? This text and its author subscribe to the importance of sensitivity to data and of the wedding of humanitarian impulse to scientific rigor. Therefore, it seems appropriate to discuss my conception of the role of statistics in the overall research process. This section assumes familiarity with the general principles of research methodology. It also assumes some acquaintance with the use of statistics, especially significance tests, in research. If this latter is a poor assumption, the reader is urged to delay reading this section until after reading Section 1.2. 1.0.1 Statistics as a Form of Social Control Statistics is a form of social control over the professional behavior of researchers. The ultimate justification for any statistical procedure lies in the kinds of research behavior it encourages or discourages. In their descriptive applications, statistical procedures provide a set of tools for efficiently summarizing the researcher’s empirical findings in a form that is more readily assimilated by the intended audience than would be a simple listing of the raw data. The availability and apparent utility of these procedures generate pressure on researchers to employ them in reporting their results, rather than relying on a more discursive approach, On the other hand, most statistics summarize only certain aspects of the data; consequently, automatic (e.g., computerized) computation of standard (cookbook?) statistics without the intermediate step of “living with” the data in all of its concrete detail may lead to overlooking important features of these data. A number of authors (see especially Anscombe, 1973, and Tukey, 1977) offered suggestions for preliminary screening of the data so as to ensure that the summary statistics finally selected are truly relevant to the data at hand. The inferential applications of statistics provide protection against the universal tendency to confuse aspects of the data that are unique to the particular sample of subjects, stimuli, and conditions involved in a study with the general properties of the population from which these subjects, stimuli, and conditions were sampled. For instance, it often proves difficult to convince a subject who has just been through a binary prediction experiment involving, say, predicting which of two lights will be turned on in each of several trials that the experimenter used a random-number table in selecting the sequence of events. Among researchers, this tendency expresses itself as a proneness to generate complex post hoc explanations of their results that must be constantly revised because they are based in part on aspects of the data that are highly unstable from one2 1 The Forest Before the Trees replication of the study to the next. Social control is obtained over this tendency, and the“garbage rate” for published studies is reduced, by requiring that experimenters first demonstrate that their results cannot be plausibly explained by the null hypothesis of no true relationship in the population between their independent and dependent variables. Only after this has been established are experimenters permitted to foist on their colleagues more complex explanations. The scientific community generally accepts this control over their behavior because 1. Bitter experience with reliance on investigators’ informal assessment of the generalizability of their results has shown that some formal system of “screening” data is needed. 2. The particular procedure just (crudely) described, which we may label the null hypothesis significance testing (NHST) procedure, has the backing of a highly developed mathematical model. If certain plausible assumptions are met, this model provides rather good quantitative estimates of the relative frequency with which we will falsely reject (Type I error) or mistakenly fail to reject (Type II error) the null hypothesis. Assuming again that the assumptions have been met, this model also provides clear rules concerning how to adjust both our criteria for rejection and the conditions of our experiment (such as number of subjects) so as to set these two “error rates” at prespecified levels. 3. The null hypothesis significance testing procedure is usually not a particularly irksome one, thanks to the ready availability of formulae, tables, and computer programs to aid in carrying out the testing procedure for a broad class of research situations. 1.0.2 Objections to Null Hypothesis Significance Testing However, acceptance is not uniform. Bayesian statisticians, for instance, point out that the mathematical model underlying the null hypothesis significance testing procedure fits the behavior and beliefs of researchers quite poorly. No one, for example, seriously entertains the null hypothesis, because almost any treatment or background variable will have some systematic (although possibly miniscule) effect. Similarly, no scientist accepts or rejects a conceptual hypothesis on the basis of a single study. Instead, the scientist withholds final judgment until a given phenomenon has been replicated on a variety of studies. Bayesian approaches to statistics thus picture the researcher as beginning each study with some degree of confidence in a particular hypothesis and then revising this confidence in (the subjective probability of) the hypothesis up or down, depending on the outcome of the study. This is almost certainly a more realistic description of research behavior than that provided by the null hypothesis testing model. However, the superiority of the Bayesian approach as a descriptive theory of research behavior does not necessarily make it a better prescriptive (normative) theory than the null hypothesis testing model. Bayesian approaches are not nearly as well developed as are null hypothesis testing procedures, and they demand more from the user in terms of mathematical sophistication. They also demand more in terms of ability to specify the nature of the researcher’s subjective beliefs concerning the hypotheses about which the study is designed to provide evidence. Further, this dependence of the result of Bayesian1.0 Why Statistics 3 analyses on the investigator’s subjective beliefs means that Bayesian “conclusions” may vary among different investigators examining precisely the same data. Consequently, the mathematical and computational effort expended by the researcher in performing a Bayesian analysis may be relatively useless to those of his or her readers who hold different prior subjective beliefs about the phenomenon. (The mays in the preceding sentence derive from the fact that many Bayesian procedures are robust across a wide range of prior beliefs.) For these reasons, Bayesian approaches are not employed in the Primer. Press (1972) has incorporated Bayesian approaches wherever possible. An increasingly “popular” objection to null hypothesis testing centers around the contention that these procedures have become too readily available, thereby seducing researchers and journal editors into allowing the tail (the inferential aspect of statistics) to wag the dog (the research process considered as a whole). Many statisticians have appealed for one or more of the following reforms in the null hypothesis testing procedure: 1. Heavier emphasis should be placed on the descriptive aspects of statistics, including, as a minimum, the careful examination of the individual data points before, after, during, or possibly instead of “cookbook” statistical procedures to them. 2. The research question should dictate the appropriate statistical analysis, rather than letting the ready availability of a statistical technique generate a search for research paradigms that fit the assumptions of that technique. 3. Statistical procedures that are less dependent on distributional and sampling assumptions, such as randomization tests (which compute the probability that a completely random reassignment of observations to groups would produce as large an apparent discrepancy from the null hypothesis as would sorting scores on the basis of the treatment or classification actually received by the subject) or jackknifing tests (which are based on the stability of the results under random deletion of portions of the data), should be developed. These procedures have only recently become viable as high-speed computers have become readily available. 4. Our training of behavioral scientists (and our own practice) should place more emphasis on the hypothesis-generating phase of research, including the use of post hoc examination of the data gathered while testing one hypothesis as a stimulus to theory revision or origination. Kendall (1968), Mosteller and Tukey (1968), Anscombe (1973), and McGuire (1973) can serve to introduce the reader to this “protest literature.” 1.0.3 Should Significance Tests be Banned? Concern about abuses of null hypothesis significance testing reached a peak in the late 1990s with a proposal to the American Psychological Association (APA) that null hypothesis significance tests (NHSTs) be banned from APA journals. A committee was in fact appointed to address this issue, but its deliberations and subsequent report were quickly broadened to a set of general recommendations for data analysis, framed as specific suggestions for revisions of the data-analysis sections of the APA publication manual—anot including a ban on NHSTs (Wilkinson and APA Task Force on Statistical4 1 The Forest Before the Trees Inference, 1999). Most of the objections to NHST that emerged in this debate were actually objections to researchers’ misuse and misinterpretation of the results of NHSTs—most notably, treating a nonsignificant result as establishing that the population effect size is exactly zero and treating rejection of Ho as establishing the substantive importance of the effect. These are matters of education, not of flawed logic. Both of these mistakes are much less likely (or at least are made obvious to the researcher’s readers, if not to the researcher) if the significance test is accompanied by a confidence interval (CI) around the observed estimate of the population effect—and indeed a number of authors have pointed out that the absence or presence of the null-hypothesized value in the confidence interval matches perfectly (at least when a two-tailed significance test at level o is paired with a traditional, symmetric (l-a)-level CI) the statistical significance or nonsignificance of the NHST. This has led to the suggestion that NHSTs simply be replaced by CIs. My recommendation is that CIs be used to supplement, rather than to replace, NHSTs, because 1. The p-value provides two pieces of information not provided by the corresponding CI, namely an upper bound on the probability of declaring statistical significance in the wrong direction (which is at most half of our p value; Harris, 1997a, 1997b) and an indication of the likelihood of a successful exact replication (Greenwald, Gonzalez, Harris, & Guthrie, 1996). 2. Multiple-df overall tests, such as the traditional F for the between-groups effect in one-way analysis of variance (Anova), are a much more efficient way of determining whether there are any statistically significant patterns of differences or among the means or (in multiple regression) statistically reliable combinations of predictors than is examining the confidence interval around each of the infinite number of possible contrasts among the means or linear combinations of predictor variables. The aspect of NHST that Les Leventhal and I (Harris, 1997a, 1997b; Leventhal, 1999a, 1999b; Leventhal & Huynh, 1996) feel should be banned (or, more accurately, modified) is the way its underlying logic is stated in almost all textbooks—namely as a decision between just two alternatives: the population parameter 0 is exactly zero or is nonzero in the case of two-tailed tests and 8 > 0 or @ <0 in the case of one-tailed tests. As Kaiser (1960b) pointed out more than forty years ago, under this logic the only way to come to a decision about the sign (direction) of the population effect is to employ a one- tailed test, and under neither procedure is it possible to conclude that your initial hypothesis about this direction was wrong. That is, researchers who take this decision- making logic seriously are “thus faced with the unpalatable choice between being unable to come to any conclusion about the sign of the effect ... and violating the most basic tenet of scientific method” — namely, that scientific hypotheses must be falsifiable (Harris, 1997a, p. 8). This book thus adopts what Harris (1994, 1997a, 1997b) referred to as three-valued hypothesis-testing logic and what Leventhal and Huynh (1996) labeled the directional two-tailed test. Specifically, every single-df significance test will have one of three possible outcomes: a conclusion that 6 > 0 if and only if (iff) 6 (the sample estimate of