Development and Validation of Indicators of Teacher Proficiency in Diagnostic Classroom Assessment: A Mixed Methods Study

Madhabi Chatterji, Ph.D.

mb1434@columbia.edu

FOCUS OF THE STUDY

CONSTRUCTAPPROACH -

AIM-

Teachers Proficiency in Formative, Diagnostic Classroom Assessment Apply a Mixed-Methods approach to derive operational indicators to define the construct domain A) Literature review and content analysis B) Case study--analysis of an expert teachers classroom applications C) Attitude measurement traditions Examine utility of methodology and domain framework for designing future instruments

 Despite rhetoric on formative classroom assessment , limited empirical research and literature on learner diagnosis approaches that draw on the latest KNOW-HOW from the COGNITIVE SCIENCES AND LEARNING THEORY (Pellegrino, Chudowsky & Glaser, 2001). To support teacher changes in classroom assessment practices, many recommendations for tools and wellconceived models of ideal professional practice as guides for research, monitoring, coaching in TEACHER PROFESSIONAL DEVELOPMENT CONTEXTS (Shepard, 2006). Not much knowledge about how teachers WOULD ADD new theory from the cognitive sciences into their existing repertoires of classroom assessment practice.

CALLS FOR MODELS AND TOOLS TO GUIDE TEACHER PRACTICES

SIGNIFICANCE: NEED FOR MODELS AND TOOLS

STUDIES SHOW THAT THE TYPICAL TEACHER: Still views assessment primarily as a means for assigning student grades (Shepard, 2006) Lacks knowledge necessary for pupil assessment, individualization, and re-teachingespecially lacks skills in ongoing, diagnostic analysis of learner needs and mediation techniques (Erickson, 2007; Macintyre
Latta, Buck & Beckenhauer, 2007).

Belongs in schools that are beset with problems and shortcomings when it comes to supporting formative assessment (Black, Harrison, Lee, Marshall, & Wiliam,
2003, p. 10)

SIGNIFICANCE: NEED FOR MODELS AND TOOLS

In the U.S., federal policies and institutional pressures for accountability-oriented, large scale standardized testing, coverage of curriculum, and a summative assessment culture in schools serve as threats to formativity and learning diagnosis in classrooms
(Chatterji, Koh, Choi & Iyengar, 2009; Erickson, 2007, p. 190).

The reported research addresses the instrumentation void in research and applied settings to support formative, diagnostic assessment by teachers.

RESEARCH CONTEXT
Stems from a 2-year Research and Development Project (supported by the National Science Foundation) Purpose of Larger Study : Develop and Field-Test new approach to diagnostic classroom assessment drawing on the cognitive sciences, called Proximal Assessment for Learner Diagnosis (PALD) Expert teacher case: Selected from 14 PALD participant teachers in 4 New York schools who were all trained in PALD (Chatterji et al., 2009).

Question: Stage One

Based on the cognitive sciences literature, what are some observable (verbal and non-verbal) behaviors, attitudes, or practices that an expert teacher would likely display when using diagnostic assessment strategies effectively with students?

METHOD-STAGE ONE
Stage One: Domain Specification based on a Literature Review following Traditions in Attitude Measurement and Domain Sampling Theory
1) Identify behaviors or observable indicators of the construct of interest via content analysis of existing literature or documentary sources 2) Classify indicators using a theoretically-supported taxonomy 3) Select or write items matched to the domain indicators using established item-writing guidelines 4) Sample items from domain to assemble an instrument
(AERA, APA, & NCME, 1999; Chatterji, 2003; Crocker & Algina, 2006; Nunnally & Bernstien, 1994).

PALD Proficiency construct domain was envisaged as a behavioral-attitudinal construct (Bandura, 1997; Rosenberg & Hovland, 1960). (1) Beliefs/Opinions: represent what teachers perceive they know or hold to be true about some attitudinal object, herediagnostic assessment. Belief could also represent ones self-efficacy in a given proficiency domain (2) Behaviors: represent what teachers might actually do or practice that reflects their underlying attitude about diagnostic assessment (3) Feelings: represent what individuals feel about an object [Dropped; not considered relevant to proficiency construct].

Taxonomy for Organizing and Classifying Indicators

Selected Themes from Research -Proximal, Diagnostic Assessment

Cognitive Modifiability and Continuous Human Development Continuous Expertise Development in a Domain involves Error Diagnosis, Clear Goals and Formative Instruction in a Supportive Environment Changes in Skill and Expertise Levels Occur with Self-Monitoring and Deliberate Practice by Learners Themselves
IMPLICATIONS: Domains and tasks must be developmentally ordered when designing educational assessments. Dynamic classroom assessment model: Test/assessIdentify Errors---Re-teach---Mediate Give Practice--Test Again (Repeat) (Anderson, 2005; Black & Wiliam, 1998; Dick, Carey, & Carey, 2005 ; Feuerstein, Rand, & Hoffman, 1979; Goleman, 1994; Kuhn, Katz, & Dean, 2004; Pressley, 1995; Nichols, 1994; Tatsuoka, 1983; Siegler, 2000; Vygotsky, 1978; Wellman & Gelman; 1992 Webb, Franke, De, Chan, Freund, Shein, & Melkonian, 2009) )

The Proximal Assessment for Learner Diagnosis (PALD) Model: How Teachers are Expected to Plan and Work
I. Goal-Setting Specify: Long-term instructional goals
R E V I S E

Specify: Culminating performance tasks and domain of embedded concepts and skills

II. PALD Planning Develop lesson plans linked to domain Design instructional strategies Develop ordered diagnostic assessment tasks

IV. End of Unit Assessment Make summative decisions on student progress

R E V I S E

III. PALD Implementation Instruct (New concepts)

Embed PALD Cycles to Detect Learner Gaps Assess Analyze Errors/-Diagnoses Mediate Give Practice

Make formative decisions

Table 1. The PALD Proficiency Domain for Teachers: Theoretically-derived Indicators of Teacher Belief

1.0 Believes in principles underlying diagnostic classroom assessment and self-capacity to execute the same.
General indicator(s) 1.1 Belief in cognitive modifiability and continuous human development in a domain Specific indicator(s) As a part of teaching in a domain, believes that: 1.1.1 Learning and development is a continuous process for all learners 1.1.2 Assessment and learner development are inter-related 1.1.3 Learner success and growth are facilitated with diagnostic assessment and mediation 1.1.4 Learning/development occurs in both cognitive and non-cognitive domains 1.1.5 Affective states of the learner (motivation, value, comfort) influence learning and development To facilitate learner development in a domain, believes in the utility of: 1.2.1 Embedding self-designed assessments for diagnosis of learner needs 1.2.2 Probing and performing error analysis to identify learning gaps 1.2.3 Interacting, modeling and targeting immediate mediations/interventions to support learner growth 1.2.4 Connecting students prior learning with new learning goals 1.2.5 Scaffolding while teaching or testing 1.2.6 Encouraging student self-monitoring and reflection 1.2.7 Situating learning experiences in real life 1.2.8 Giving students planned practice 1.2.9 Nurturing positive student affect 1.2.10 Creating a risk-free assessment environment 1.2.11 Taking graduated steps (from simple to complex) towards difficult goals To facilitate learner development, believes in the importance of: 1.3.1 Clarifying learning outcomes and instructional goals when designing instruction and assessment 1.3.2 Aligning learning outcomes/goals with instruction and assessment 1.3.3 Coaching, re-teaching, and giving feedback to students after assessment 1.3.4 Using formative assessment results to alter and improve teaching strategies 1.3.5 Gathering evidence of new learning via formative assessment before performing summative assessment Expresses belief in own ability to: 1.4.1 Make changes towards learning-centered and diagnostic assessment practices 1.4.2. Balance external assessment and accountability demands with formative and diagnostic learner assessment 1.4.3 Balance the external or organizational assessment culture with best practices in classroom assessment 1.4.4 Manage ongoing, diagnostic assessment tasks (designing tests/items, scoring, analyzing errors, making formative decisions) as part of regular work routine

1.2 Belief that proximal, timely assessment and mediation to bridge learning gaps

1.3 Belief in student-centered, outcome-driven pedagogy

1.4 Belief in self-capacity to conduct diagnostic and formative assessment in typical school environments.

RESULTS Stage One: A Sample of Items for a Survey tapping Teacher Beliefs in Diagnostic Classroom Assessment
General and specific indicator(s) 1.1 Belief in cognitive modifiability and continuous human development in a domain 1.1.1 Learning and development in an area is continuous 1.1.5 Affective states of the learner (motivation, value, comfort) influence learning and development Items for a self-report survey 1. Students can succeed and grow in any subject area (1.1.1). 2. How students feel about what they learn influences how much they learn (1.1.5).

1.2 Belief that proximal, timely assessment and mediation helps bridge learning gaps Believes in the importance of: 1.2.1 Embedding self-designed diagnostic assessments during teaching is important for facilitating learning 1.2.2 Performing error analysis and probing 1.2.3 Interacting, modeling and targeting immediate mediation/interventions Note. Matching indicator number in parenthesis.

3. Teachers become more aware of student learning needs when they embed their own assessments during teaching (1.2.1). 4. If teachers model after they find student mistakes, learning gaps are removed (1.2.2). 5. Student mistakes are summative evidence that a student has reached his/her capacity to master something (1.2.2, negative orientation) ITEM RESPONSE SCALE: STRONGLY AGREE, AGREE, UNCERTAIN, DISAGREE, STRONGLY DISAGREE

Questions-STAGE TWO
Stage Two: Validation On a day-to-day basis and in real time, what does an expert teacher say or do in her classroom that is consistent with the theoretical literature on PALD? How did the teacher assimilate diagnostic assessment approaches in her daily teaching repertoire? Is there a theory of PALD practice grounded in the data?

METHOD-STAGE TWO
Stage Two: Domain Validation using Qualitative Techniques (Mabry, 2009; Strauss & Corbin, 1998; Creswell, 2003). Case Study Methodology: One female, grade 5-6 PALD expert teacher, who taught math; 9 years of experience; class of 17 students Grounded Theory Analysis: Data on her classroom assessment activities and artifacts; multiple data sources; 2 year observation period

METHODS-STAGE TWO
Expert Teacher Definition Case selected based on : (1) Attendance in all PALD workshops and coaching sessions and increased frequency of practices in Year 2 (2) Class exposure to PALD-trained teachers for 2 years and showed math achievement gains (3) Continuing participation in PALD project.

METHODS-STAGE TWO
Data Sources (1) Teachers lesson plans, assessments, scoring rubrics, analytic recorts and artifacts (2) Running records 30-minute classroom observations of the teachers assessment activities (3) Journal records kept by the teacher of assessment applications

TEACHERS DATA SOURCES: Ordered, Diagnostic Assessment Tasks in Long Division Unit
Task examples Task specifications and embedded competencies Item Difficulty Level: % students answering correctly in Grade 5 (n=16) .75 (Easy Item)

Part 1 Look at the problem below to answer Questions 1-6.

5 175
1. The problem is telling me to: a. Multiply 5 times 175 b. Find out how many groups of 5 are in 175 c. Divide 5 into 175 equal parts d. Add 5 and 175 e. Do something else (explain): Part 2 I began to do the following problem but know I have made some mistakes. Please find and mark my mistakes. Then, show me how I could fix them by re-doing the problem. 610 9) 552 -54 12 9 3 -0 3R

Question designed to probe application of long division concepts. Item situated in an item series. Learners/examinees are expected to restate what the problem is asking by using the structured-response item format. Scaffolding is provided via multiple answer options that aim to guide learners thinking.

Probe for checking concept understanding and procedural skills in long division. This is a situated task in that learners apply long division vocabulary, identify errors, check solutions, and explain answer, when given a problem that is incorrectly solved. Task uses an unscaffolded, open-ended item format.

.42 (Difficult Item)

Coding running observation records using grounded theory analysis: How indicators were derived from a teacher-class interaction during a unit on probability
Theme/ Indicator Probing Description Sentence by sentence codes applied to text segments Teacher uses a bag of letters (Scrabble pieces) (T uses materials to teach to teach probability concepts and vocabulary probability-related related to dependent and independent vocabulary) observations. T: Well, dependent means it relies on something else. And I put the M back. Right? And I reach in the bag and pull another letter. Do those pulls affect each other? S (2): No T: You think its gonna change the probability on one? Ss (1): Yes, yes. (T clarifies word dependent) (T probes with 3 Qs while demonstrating) (S gives content-related response) (T probing) Running record

Uses probing questions to check for concept understanding, mediates with whole class, and demonstrates as a part of instructional exchange specific to a math problem/concept.

Probing

Re-teaching, demonstrating concept

Giving feedback-large group mediation.

(S gives content-related response) Ss (2): No, no. (S gives content-related response) T (repeats): I reach into the bag and pull a (T clarifies and demos letter out, look at it, put it back in, and reach again) in the bag and pull another one out. Ss: (Unanimously) No (S content-related response) T: Theyre independent. One does not affect (T Feedback during direct the other. Thats independent. But the other concept instruction) example that Sean picked up on, is that if I (T repeats concept, pulled out the M and I put it over on the side, vocabulary, and demo) and I reach in the bag again and pulled out another letter. Do you understand how the (T probes with 1 Q.) second time, my probability is not out of 11 anymore?

TEACHERS DATA SOURCES: Assessment Task with Scoring Rubric

Task: Coconut cookies come in packets of 12. I would like to give 1 cookie to each 5th grader at (your school) to celebrate the last day of school (in June). There are 219 5th graders at (school name) Elementary. How many packets of cookies should I buy so that every 5 th grader gets at least 1 cookie? Will there be any cookies left over? Show your plan to solve the problem. Then set up the numbers and solve it. Explain all the parts of your answer, and show how you would check if your answer is correct. Show all your work. Rubric: Checking What I Know in Long Division

I can: -Say in my own words what the long division problem is asking me to do -Read a story problem and set up the long division and/or other operations needed to solve it -Identify the dividend and divisor in an long division problem -After the long division problem is set up, know how to start the algorithm (find digit with the highest place value in the dividend) -Recall and use multiplication facts correctly when doing long multiplication and long division -Recall place value concepts and give place value labels for digits in whole and decimal numbers, when doing long multiplication and long division -Follow the steps of the long division algorithm correctly* -Repeat the long division algorithm until I get a remainder that cannot be further divided (grade 5) OR -Continue the long division algorithm so I can express remainder as a decimal of the quotient (grade 6) -Check the long division answer with backwards operations -Explain what the answer means in my own words -Think-aloud and write the steps of long division -Think aloud and explain the answer and parts of the answer in a story problem -Keep my scratch work neat (so that I dont get lost) -Look at my mistakes to help me grow (without fear) -Not give up before finishing a problem -Teach long division to a friend -Make connections between fractions, decimals and division/ long division operations
*Students used a mnemonic in classDivide (Dad), Multiply (Mom), Subtract (Sister), Bring Down (Brother), Again (Aunt), Remainder (Repeat? End): DMSBAR

Table 3. PALD Proficiency Domain based on Case Study: Teacher Practice Indicators derived from Coaching Session
2.0 Demonstrates diagnostic classroom assessment practices while teaching General and specific indicator(s) 2.1 Clarifies targeted goals, outcomes, and embedded competencies on a continuum to prepare for student need diagnosis 2.1.1 Identifies embedded learning outcomes, concepts and skills from a culminating target (problem or task) 2.1.2 Designs/uses ordered items or tasks that test mastery of embedded concepts and skills in a unit 2.1.3 Gauges difficulty of goals vis--vis student background levels 2.1.4 Designs/uses pre-tests, homework, and class work exercises for formative assessment of student learning needs 2.1.5 Evaluates difficulty of assessment tasks vis--vis student levels and learning targets 2.2 Plans and delivers instruction to class linked to formative assessment results 2.2.1 Deliberately plans instruction using assessment results from pre-tests, mid-unit homework, or classwork exercises before using assessment results for grading 2.2.2 Makes time to examine student work formatively 2.2.3 Identifies student errors/mistakes vis--vis embedded competencies and learning targets 2.2.4 Identifies mediation techniques specific to observed learning gaps in students 2.2.5 Tries more than one mediation strategy with students before moving on 2.2.6 Repeats formative assessment and teaching cycles, as needed (re-teaches and re-assesses) 2.2.7 Makes time to summarize and review student responses to tests/assessments (analyzes data quantitatively or qualitatively) 2.2.8 Encourages student engagement or dialogue about concepts and skills relevant to domain/unit mastery
Note. Matching indicator number in parenthesis.

Matching items for a behavior-based observation tool or self report survey 1. I use more difficult problems in my early unit assessments to identify student needs (2.1.2). 2. My goals for a unit are broken down to match levels of different students (2.1.4). 3.. I give homework exercises to check where students are in learning new goals for a unit (2.1.5).

4. I use the pre-unit assessments results to plan my lessons. (2.2.1) 5. I test formatively to see if students make errors in the concepts I am teaching (2.2.2). 6. Depending on errors my students make on exercises I give them, I change what I teach (2.2.4). 7. I conduct formative assessment more than once in a unit (2.2.6). RESPONSE SCALE: RARELY OR NEVER, SOMETIMES (ABOUT ONCE A WEEK), OFTEN, VERY OFTEN (MORE THAN TWICE A WEEK)

Table 4. PALD Proficiency Domain: Added Teacher Practice Indicators derived from Classroom Observations

2.0 Demonstrates diagnostic classroom assessment practices while teaching Matching items for a behavior-based observation General and specific indicator(s) tool or self report survey 2.2 Plans and delivers instruction to class linked to 8. I use everyday examples during formative formative assessment results assessment and instruction (2.2.9). 2.2.9 Uses situated tasks meaningful to learners during 9. I ask students to think about their mistakes assessment and instruction. (2.2.11). 2.2.10 Probes student thinking about mistakes related 10. I re-teach concepts to the whole class, when to concepts in unit needed (2.2.13). 2.2.11 Encourages examination of errors by students 2.2.12 Mediates individually while teaching 2.2.13 Mediates with whole class or small groups while teaching 2.2.14 Creates a supportive, learning-oriented assessment culture Negatively-oriented indicator Negatively-oriented items 2.2.15 Uses classroom assessment to force behavioral 11. If they are not attentive when I teach, I warn compliance from students or for behavior management students of external exams.

Table 5. PALD Proficiency Domain: Added Teacher Practice Indicators derived from Teacher Journal Records

2.0 Demonstrates diagnostic classroom assessment practices while teaching Matching items for a behavior-based observation General and specific indicator(s) tool or self report survey 2.2 Plans and delivers instruction to class linked to 12. I use different types of assessment tasks formative assessment results written, oral, projects--during instruction 2.2.16 Uses formative assessment tasks in a variety (2.2.16). of formats 13. I keep records of student growth on 2.2.17 Repeats error analyses of student work, as particular skills (2.2.18). needed 14. I recognize students publicly in class if they 2.2.18 Tracks student growth via formative show growth after a formative assessment. assessment records, quantitatively or qualitatively (2.2.20). 2.2.19 Communicates performance expectations to students via rubrics or other means 2.2.20 Rewards/recognizes students for participating in formative assessment processes 2.2.21 Provides practice exercises as part of formative assessment cycle Negatively-oriented indicator 2.2.22 Communicates negative beliefs about students learning capacities
Note. Matching indicator number in parenthesis.

Negatively-oriented items 15. To be realistic, I let weaker students know they may not master the entire unit.

CONCLUSIONS
In general, the Belief and Practice sub-domains of PALD Proficiency were cross-validated using mixed-methods A veteran teacher could make positive shifts towards diagnostic assessment-related attitudes and behaviors, but needed at least two years of support Limitations in current domain: single case data --based on one teacher and classroom with limited generalizability Future Research: Utility of this methodological approach should continue to be tested
SOURCE Chatterji, M. (2012). Development and validation of indicators of teacher proficiency in diagnostic classroom assessment. The International Journal of Educational and Psychological Assessment, 9 (2), 4-25. Special Issue on Teacher Assessments.