You are on page 1of 4

Education 353C Problems in Measurement: Generalizability Theory

Richard J. Shavelson & Edward H. Haertel Tuesdays 9:00AM – 11:50AM Cubberley 207 Generalizability Theory (G theory) provides a powerful class of psychometric models for behavioral and social measurement data. G theory was developed beginning in the 1960s by Lee J. Cronbach here at Stanford University with his collaborators Goldine Gleser, Harinder Nanda, and Nageswari Rajaratnam (Cronbach, Rajaratnam, & Gleser, 1963; Cronbach, Gleser, Nanda, & Rajaratnam, 1972). It encompasses classical test theory (CTT) as a special case, but goes far beyond CTT in clarifying conceptual confusions and providing more powerful statistical tools. Most importantly, G theory systematizes distinctions among the many reliability coefficients of CTT, and enables the calculation of reliability coefficients and/or standard errors of measurement tailored to specific measurement applications and/or interpretations. G theory also provides a powerful extension of the Spearman-Brown "prophecy" formula, enabling the design of efficient data collection plans for specific measurement applications. In this course, we will provide a systematic introduction to the concepts and models of G theory, as well as special-purpose software useful in carrying out the requisite calculations. The goals of the course are to provide a working knowledge of common Gtheory models and applications, to enable students to read critically journal articles employing G theory, and to introduce some unsolved problems and current research topics related to G-theory. The class will typically be organized as follows. The instructors might begin with a minilecture on the topic of the day with discussion. Then, a problem with data will be introduced and students will work in teams to solve the problem within the G theory framework. Toward the end of the quarter, the problems will be replaced by student-led presentation of self-chosen paper that uses G theory. Occasional homework will consist of students working on a brief problem with data. There will be a final project in which either a student will apply G theory to a somewhat novel set of data and report her/his findings to the class or will read up on a topic not covered in class (e.g., newly developed standard errors and confidence intervals for variance component estimates) and teach the class about the topic. There will be no final examination. The primary text for the course is Generalizability Theory: A Primer by Richard J. Shavelson and Noreen M. Webb. In addition, Generalizability Theory by Robert L. Brennan is recommended. Additional readings will be provided during the quarter. Note that the Primer is just that—an introductory text on G theory. Brennan’s is the currently authoritative text. So, when you complete a chapter in the Primer and you want more

Webb. Handbook of Statistics: Psychometrics (Vol. Thousand Oaks. Westport. (1997). Y. C. Tourneur.J. M. Multivariate generalizability theory. 81-124). V. (2004). British Journal of Statistical Psychology. Shavelson. Shavelson. Brennan covers roughly the same materials in his Chapters 1-4 as the Primer does—but in a somewhat different structure. In L. N.. H. R. (2000a). (1999). 61-71. J. Stanford University. Wiley. Amsterdam: Elsevier B. N. 64(3). (1997). In C. (2006). R. J. L.. 922-932. Shavelson. Editor’s preface to Lee J. (1963). Generalizability Theory: A Primer. Y. Fyans (Ed. J. L. R. Note on sources of sampling variability in science performance assessments.. E. Haertel. & Gleser. C. In R. Shavelson. & Wiley. New York: Springer-Verlag. J. Rajaratnam.. CA: Sage.. E. Journal of Educational Measurement. (1981). American Psychologist. (Mis)conceptions about generalizability theory. Brennan (Ed. J. H.. M. 26. M. N. H. Sinharay (Eds. L. A perspective on the history of generalizability theory.. Generalizability theory. & Haertel. & Allal. 14-20. Rao & S.. (1983). San Francisco: Jossey-Bass. (2007). 16(4). & Haertel. (1981). Extension of generalizability theory and its applications in educational measurement. Educational and Psychological Measurement. L. R. 19(1). R. W. 67-82).. L. 57. L. we have copied the first two chapters of the Primer and will email them to you. 5-10. N. G. L. Theory of generalizability: A liberalization of reliability theory. N. Gleser. & Rajaratnam. R. N.G Theory Seminar Shavelson & Haertel statistical background. My current thoughts on coefficient alpha and successor procedures. L. Cronbach. 36. Nanda. R. 373-399. For convenience. (1976).). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. J. W. R. pp. Shavelson... L. turn to Brennan. R.. N. Brennan. Educational Measurement: Issues and Practice. New York: Wiley.. & Rowley. J. Generalizability theory: 1973-1980. Some References Brennan. & Allal. You can purchase both textbooks online so they are not in the bookstore. Reliability coefficients and generalizability theory.. 133-166. Generalizability Theory.). E. J. R. 389-391. Cronbach. R. A. J. J. 119-135 Cardinet.H. Brennan. (1991). Journal of Educational Measurement. Generalizability analysis for performance assessments of student achievement or school effectiveness. pp. (1972). M. Educational and Psychological Measurement. Linn. Bootstrap strategies for variance component estimation: Theoretical and empirical results. Cronbach. & Webb. & Maddahian. G.. 183-204. Webb.” Educational and Psychological Measurement. 18. E.M. Tourneur. 34. L. (1989). Unpublished doctoral dissertation. R. 44(6). (2000). J. & Webb. Educational Measurement: Issues and Practice. J. Webb.13. Ruiz-Primo.. 391-418 Cronbach. British Journal of Shavelson. Shavelson. Journal of Educational Measurement. E. Brennan. Spring 2007 2 . G. Cardinet. L.. 65-110). 16. The symmetry of generalizability theory: Application to educational measurement. 137-163. Cronbach’s “My Current Thoughts on Coefficient Alpha and Successor Procedures. (2001). Mathematical and Statistical Psychology. CT: American Council on Education/Praeger. M. R..).. E. Educational measurement (4th ed.. L..J. 64(3).. Generalizability theory: New directions for testing and measurement (pp. (2004). Reliability.

Tentative Course Schedule1 Date 4-3 Topic Introduction to G Theory & Statistical Demo Software (GENVA & EduG) SW Tables 1.1 and 1.3 AERA Week—No class meeting Readingsa SW Chapter 1 CGNR Preface & Pp 1-14 Homework #1 A. . 3 with GENOVA & EduG Homework #2—Brief proposal for final project (bring handouts to class 4/24) Read Cardinet’s Why EduG Homework #3—Read & prepare to lead discussion: Group 1: Cardinet et al 1981 Group 2: CLBH Group 3: SRW Homework #4 Prepare final project description. F EduG Users Guide SW Chapters 2 & 3 4/17 Discuss Homework #1 Statistical Model for G Theory G Studies with crossed facets One-facet G Study simulation 4/24 Discuss final project ideas Discuss Why EduG In Class Exercise #1 G studies with nested facets Group presentations of papers Random & Fixed Facets Decision Studies Exercise#2_Cardinet #1 Collect project descriptions SW Chapter 4 5-1 SW Chapter 5 5-8 SW Chapter 6 1 This schedule is tentative and may change depending on coverage in class or other unexpected events. hand in at class on 5-8 Homework #5 Cardinet Exercise #2 & Cardinet Complex Exercise Assignmentb 4-10 See 4-3 Readings SW Chapter 2 Brennan Appx. Handout P x I G study B. Run Exercise 3 in SW Ch.

& Rajaratnam (1972). SRW refers to Shavelson. & Haertel (1997). Gleser. SW refers to Shavelson & Webb (1991). CLBH refers to Cronbach. Linn. WS refers to Webb & Shavelson (1983). Ruiz-Primo. and Wiley (1999). Nanda. Brennan.G Theory Seminar Shavelson & Haertel 5-15 Discuss Homework #5 G & D Studies for same and different Designs Multivariate G Theory mGenova Exercise Presentations of student projects SW Chapter 7 & 8 Work on projects 5/22 WS Brennan 267-285 (Optional) 5-29 6-5 Presentations of student projects ___________________________________________ a Readings to be done before class on date given CGNR refers to Cronbach. b Assignments due the week following their assignment Spring 2007 4 .