RUNNING HEAD: On the Development of Reflective Lessons FROM FORMAL EMBEDDED ASSESSMENTS TO REFLECTIVE LESSONS: THE DEVELOPMENT OF FORMATIVE

ASSESSMENT SUITES

Carlos Ayala Sonoma State University Richard J. Shavelson Stanford University

Paul Brandon University of Hawaii Yue Yin University of Hawaii

Erin M. Furtak Max Planck Institute for Human Development

Maria Araceli Ruiz-Primo University of Colorado at Denver and Health Sciences Center

Donald Young University of Hawaii

Miki Tomita

On the Development of Reflective Lessons Stanford University and University of Hawaii

2

On the Development of Reflective Lessons Abstract The idea that formative assessments embedded in a curriculum could help guide teachers toward better instructional practices that lead to student learning has taken center stage in science assessment research (Atkins & Coffey, 2003; Black & Wiliam, 1998). In order to embed formative assessments in a curriculum, curriculum developers and assessment specialists need to collaborate to create these assessment tasks. This paper describes the development of the formal embedded formative assessment suites and implementation plans that were designed for the Romance Study using a Science Achievement Framework. It describes the fundamental shift away from “summative assessment scripts” to reflective lessons scripts. Samples of the assessment tasks and implementation plans are described along with the rationale for why these tasks were selected and where these tasks were placed in the curriculum. Finally we conclude about how to successfully embed formative assessments in new or existing curriculum and how to help teacher use these assessments. For example, we point out the critical importance of collaboration, of professional development aimed at enabling teachers to reconceptualize the role of assessments in their teaching, of linking formative assessments to overall goals, and of providing a learning trajectory as reference for teachers to locate students’ ideas developmentally and provide feedback accordingly.

3

On the Development of Reflective Lessons Introduction While some empirical evidence suggests that the use of curriculum-embedded (formative) assessment leads to increased learning (Bell & Cowie, 2001; Black & Wiliam, 1998; 2004; Shepherd, 2000), how these formative assessments are designed and used by curriculum developers and then eventually implemented by teachers is poorly understood. Moreover, assessment specialists and curriculum developers rarely collaborate on assessment development, let alone on embedding formative assessments in existing curricula, on preparing end-unit summative assessments, or in thinking about how teachers might use these assessments. The “Romance” project, described by Shavelson et al. (this issue), attempted to fill this knowledge gap. Here we describe how we went about building, refining and embedding formative assessments into an inquiry science curriculum. Most noteworthy is the use of an interdisciplinary Assessment Development Team that laid out a blueprint for formative assessment definition and development. We report the fruits of this collaboration, and how they “matured” over time. We then describe how we trained teachers to use formative assessments in their inquiry science teaching. Finally, we draw conclusions about these activities with the goal of informing collaborative assessment and curriculum developers with knowledge of how to successfully embed formal formative assessments in a new or existing curriculum and of the challenges of helping teachers use this emerging technology. Background

4

this issue). They are formative assessments because they are developed to give a snap shot to students and teachers about what students know and are able to do at a particular time such that this information could be used to make teaching decisions—what does the teacher or student do next (see Shavelson et al. National Staff Development Council. See Shavelson et al. 1983. Department of Education’s Expert Panel on Mathematics and Science Education. because previous studies have supported FAST’s efficacy (Pauls. and because various national organizations have found it to be an exemplary program (U. this issue) The content for this project focused on the conception of “why things sink and float” that was built up from an understanding of mass. density and relative 5 .. S. 1977. & Lapitkova. We chose a curriculum for the project. 2001. Formal because we crafted assessment tasks which would be available for teachers to use at critical times in curriculum implementation. They are embedded assessments because they are inserted into a curriculum to be used at a particular time as opposed to the end of a unit. Young. for details). this issue.1993). Young. Tamir & Yamamoto. for more information. 1999. 1999. Foundational Approaches in Science Teaching (FAST). because the Stanford Educational Assessment Laboratory (SEAL) had collaborated previously with Curriculum Research & Development Group at the University of Hawaii (CRDG).. volume. this contrasts with on-the-fly and planned-for formative assessment that capitalize on informal ongoing clinical observations or create teachable moments for enhancing students’ understanding (see Shavelson et al.. the curriculum developers.On the Development of Reflective Lessons We define the activities that we developed in this project as embedded formal formative assessments.

On the Development of Reflective Lessons density (see Table 1). we conceived and developed embedded and end-of-unit assessments. From the framework. 1966). More specifically we conceived of science achievement as comprised of (at 6 . We begin by describing the framework and then turn attention to the Assessment Development Team. collaborative infrastructure for the project. That is. King & Brownell. which turned out to be a very important. students’ understandings of why things sink and float are developed via explanations of sinking and floating phenomena sequentially beginning with the concepts of mass and volume and moving to the concepts of density and relative density. and evaluation of embedded and end-of-unit assessments. in significant part. Science Achievement Framework The project’s assessment development was guided by conceptual framework for science achievement (see Shavelson et al. development. This curriculum develops students’ science understandings incrementally in a manner that parallels how science knowledge was developed in the Western world (cf. this issue).. --------------------------Insert Table 1 Here --------------------------Assessment Development A science achievement framework guided the Assessment Development Team in the conceptualization. we conceived of science achievement as. and as such. the acquisition of and reasoning with knowledge.

in the form of actions or steps that can be carried out to achieve a certain goal leading to task completion. and reasoning with this knowledge. knowing how to design and reason about a study that manipulates one relevant variable and controls others.” For example. descriptions.” For example.” reflect their procedural knowledge of separating mixtures. Procedural knowledge includes if-then production rules or a sequence of steps. such as.” A scientific fact would be. a statement like.” Procedural knowledge and reasoning is “knowing how. I will use a screen to separate them. To know why is to have a scientifically 7 . the water goes through the screen and the gravel is left on it. In a broad sense. For instance. for example. knowing why Delaware has a change of seasons or knowing why we see different phases of the moon. this issue). statements. mostly in the form of terms. schematic and strategic (see Figure 1 in Shavelson et al. “If I have a mixture like gravel and water. Schematic knowledge and reasoning is “knowing why. Declarative knowledge and reasoning is “knowing that”—for example. “combining two or more materials together forms a mixture” is the scientific definition of “mixture.. or how to measure the density of an object or how to graph the relation between the angle of an incline plane and the force needed to move an object up it. knowing that force is a push or pull and light is a form of energy and reasoning with this knowledge. “the density of water is 1 gram per milliliter at 4 degrees centigrade and at one atmosphere of pressure. students’ statements.” or “After I pour the mixture on the screen. Declarative knowledge includes scientific definitions and facts.On the Development of Reflective Lessons least) four different but overlapping types of knowledge and reasoning: declarative. or data. procedural.

And strategic knowledge and reasoning is “knowing when. to troubleshoot systems. 1996a. 8 . (c) evaporation can be used to separate a material from a liquid because the liquid is changed into the gas state. where and how” to use and reason with certain types of knowledge in a new situations. to explain what happened.. or to control and monitor cognitive processing. and to predict the effect that changes in some concepts will have on other concepts (De Kleer & Brown. People use strategic knowledge to recognize the situations where some procedures can be carried out. short-answer and concept maps provide valid evidence (Ruiz-Primo & Shavelson. to set task goals. 1983). Our research has linked certain types of assessment to this science achievement framework. To measure procedural knowledge. Baxter. Briefly put. schemes. 1 Schematic knowledge includes principles. performance assessments are appropriate (e. to measure the structure of declarative knowledge. 1983. For instance. or more generally. multiplechoice. Shavelson & Ruiz-Primo. Strategic knowledge includes domain-specific conditional knowledge and strategies such as planning and problem-solving as well as monitoring progress toward a goal. to examine the features of tasks in order to decide what schematic knowledge can be applied. 1996b. and mental models. Schematic knowledge can be used to interpret problems. & 1 A theory or model often called a “mental model” (Gentner & Stevens.g. the schematic knowledge students may develop when they learn about mixtures and solutions includes: (a) The method used to separate mixtures (screening or filtering) is determined by the size of the mixture’s particles. 1983). Shavelson. (b) evaporation is needed to separate a dissolved material like salt from a liquid. 1999).On the Development of Reflective Lessons justifiable “theory” or “model” or “conception” that explains the physical world. Gentner & Stevens. Ruiz-Primo & Shavelson.

this issue. Assessment Development Team The Assessment Development Team guided the conceptualization. To measure schematic knowledge. and performance assessments are appropriate. except for schematic knowledge. this issue). especially with novel assessment tasks. formative and summative assessments. The team discussed what FAST teachers typically used as assessments and student 9 . First. the science achievement framework for assessment development and corresponding assessment methods. the Team went through four major tasks to establish rapport. and begin assessment development.On the Development of Reflective Lessons Pine. To this end. the team reviewed the curriculum for the study. Strategic knowledge is difficult to measure directly but is essential. Second. (The collaboration of the Assessment Development Team towards the implementation of this study is more fully covered in Shavelson et al. the project goals and each members’ responsibilities.) Once formed. development and evaluation of the embedded and end-of-unit assessments. FAST Physical Science (PS) investigations 1 to 14. The Team consisted of Stanford assessment specialists and researchers and CRDG curriculum developers. 1992). and Brandon et al. The CRDG participants confirmed that the knowledge framework fit well with FAST. the curriculum developers provided abbreviated hands-on demonstrations of PS 1-14 while other Team members participated as students. develop a common language. identify assessment targets... multiple-choice and short-answer items. which was not addressed in the unit that was to be the focus of the study (see Brandon. and researchers. the team reviewed the project background. et al. FAST trainers and teachers.

the multiple-choice. will be able to explain why things sink and float using relationships between mass. We refer to the collection of assessment items and prompts given at a particular point in the curriculum implementation as an assessment suite (e. carry out.. report. there is a body of knowledge and 10 . graphing and lab performance assessment suite at FAST investigation 4). In addition. and identified the points (natural joints) in the instructional sequence in which the formative assessments were to be embedded. The team came up with three criteria to identify the natural joints: (1) a subgoal of the end-of-unit goal is achieved. report and defend scientific investigations of buoyancy and develop. a SEAL member presented storyboards showing the types of knowledge that were addressed in each of the PS 1-14 investigations. Thirdly.g. The group’s discussions about the lessons helped the team members understand the science achievement framework and helped clarify the purpose of the investigations. the Assessment Development Team identified the most important end-of-unit concepts taught in the 14 investigations to be used in a posttest assessment suite. carry out. and the match of the proposed assessments with the curriculum.On the Development of Reflective Lessons corresponding responses. develop. target difficulty levels. that is. volume and density. Many assessment items discussed here ended up in the study’s final assessment suites. What and where to embedded assessment? SEAL researchers developed a comprehensive assessment blue print for the PS 1-14 investigations using the science achievement framework described above. the knowledge types. Using the comprehensive blueprint. The ADT then discussed possible assessments. the Assessment Development Team developed a goal statement for the entire unit: Students individually and in groups.

SEAL & CRDG. it would impossible for the embedded assessments or the end of unit assessments to cover all of the topics and so decisions had to be made as to how to select the most important ones. there are many important concepts and procedures that students learn. a SEAL assessment developer collected and then presented a large list of terms to the curriculum developers and asked them to identify the most important terms that could be used to in a concept map assessment. For example. (2) teachers need to know about student understanding before they proceed with further instruction. a natural joint in the sequence of physical science investigations occurred between investigations 4 and 5 where students move away from examining why things sink and float using mass holding volume constant and begin using volume holding mass constant (Table 1). and (3) feedback to students is critical to help them improve their understanding and skills of the material already taught (Shavelson. However. The curriculum 11 . In this curriculum the natural joints were clearly identifiable because students move from learning about sinking and floating phenomena using mass and then move on to another set of investigations that are focused on the next concept. 6). in the identification of the nine concepts to be used in the concept map. volume.On the Development of Reflective Lessons skills sufficiently comprehensive to be assessed. p. For example. Since these important topics are clearly identifiable to the curriculum developers all of these topics became equally important to them as assessment targets. Deciding what to assess in the embedded and in the end-of-the-unit assessments was not as straight forward as finding the natural joints. Four embedded-assessment natural joints were identified in the 14 investigation sequence (Figure 1). In curriculum that typically takes eight to ten weeks to implement. 2005.

Additionally. “which terms are the most important for explaining why things sink and float?” This list narrowing still generated a large number of concepts that was then whittled down in pilot testing using empirical information. In response to this concern. In the case of the terms to be used in the concept map. The tensions between time spent on instruction vs.On the Development of Reflective Lessons developers selected all the terms because all the terms were important. and depth vs. the Assessment Development Team narrowed down the concept list by focusing on the overall goal of the unit and then asked. coverage of the assessments remained a concern throughout the project. One team member felt that student progress might be slowed by the embedded assessments. Once the important concepts had been selected and the joints identified. 12 . piloting of assessments and validation with the rest of the Team. The final selection of the assessments attended to the main goal of the unit. a SEAL team member noted that the assessments should help teachers decrease their teaching time. a FAST teacher and trainer on the team expressed concern about the length of time the assessments would add to the unit. the assessment specialists of the Assessment Development Team began an iterative process of designing and refining assessments based on the science achievement framework. assessment. because they will know the students’ deficiencies and as such the teacher would more efficiently deliver information to the students. The first complete iteration of the assessment tasks identified four locations in the curriculum (excluding the pre-post tests) where formative assessments were to be placed (Figure 1). We reduced the list again because of the amount of time it would take to complete the assessments.

. Once all the embedded assessments were reviewed and discussed. For example. The Team reviewed the assessments by discussing and/or carrying out the assessments. We exclude a detailed discussion here of the pre and post-tests because we are focused on the embedded assessments (see Yin et al. using metric rulers. the first version of the assessments contained performance tasks for students. The information that was collected although important for investigations was indirectly related to the unit’s goal: explaining “Why 13 . measuring volume. The passport was a student record book where information is stored about whether or not a student passed a performance goal (e.. A series of “passport” assessments was created to assess these performances. The Assessment Development Team dropped these passports for several reasons. The Assessment Development Team reviewed the first complete version of the assessments and began a process of refining them to match the purpose of the project and intent of the curriculum. Second the team had to make hard decisions about what not to include in the embedded assessments. the assessments went through multiple iterations. These tasks included using a balance. First the team decided that the project should only focus on the 12 physical science lessons in FAST. Can Mary mass the rock? If so then give her a stamp in her Passport book).On the Development of Reflective Lessons ----------------------Insert Figure 1 ----------------------Assessment refinement. As assessment development and refinement continued. etc.g. the Team decided to make several important modifications to the project. this volume).

and then having the teachers use the embedded assessment with students who were involved in the FAST investigations in summer school. The pilot study’s findings suggested that the embedded assessments should be 1) short in duration and tightly focused on the key outcomes of the unit—explaining “why things sink and 14 . to reduce the amount of time at each major embedded assessment joint. The pilot study focused on the teachers’ implementation of the embedded assessments. Three teachers were trained in the use of the embedded assessments and asked to use these with their classes during the project’s second year.” Moreover. carrying out the embedded assessments with the teachers as students. The Assessment Development Team carried out a pilot study with the embedded assessments. and to ameliorate an expected pre-post testing effect with the concept maps. We trained the pilot study teachers by reviewing the curriculum. From Embedded Assessments to Reflective Lessons Pilot Study. student performance on these assessments.and posttests. the amount of teacher time and effort to carry out these assessments outweighed the value of the information collected.On the Development of Reflective Lessons things sink and float. This focus on explaining “Why things sink and float” was significant because it foreshadowed larger changes to come. and professional development. The Assessment Development Team also dropped the concept map from the summative tests and moved the concept maps after investigation 6 and investigation 11. the information about student performance could be verified in the conduct of hands-on investigations. The purpose for this was to reduce the amount of time in the pre. Finally.

in later studies we found the assessment externalization in non-FAST teachers. 2) allow for immediate feedback to teacher and students. and 4) set the stage for the next set of investigations The study’s findings also suggested that teachers treated the embedded assessments just like any other test they might give. These pilot findings led to significant changes in the embedded assessments. It became clear that how these assessments were to be used in the classroom was very important and that teachers preconceived notions about assessments influence their implementation. This may have lead to the assessment externalization. A summative assessment teaching script can be 15 . the teachers would review the material covered in the unit before the embedded assessments even though the purpose of the formative assessments was to do just that. The teachers who participated in this pilot study had already been trained in FAST and knew that the assessments were added to the curriculum after the fact. Furthermore. 3) provide opportunities for students to test their “why-things-sink-and-float” explanations with evidence from the assessment event and to take a stand on what they believe. pilot study teachers often would provide feedback to students weeks after the assessment thus missing the teachable moments provided for by the embedded assessments—none of which was intended by the project staff. The teachers treated the embedded assessments as external to the unit of instruction and did not use the assessments to inform their teaching. Overall. However.On the Development of Reflective Lessons float” based on relative density. the teachers believed these embedded assessments to be summative assessments and would revert from formative assessment pedagogy to a summative assessment “teaching script” (Shavelson. For example. 1986).

Finally. this issue). The Reflective Lessons also evolved from assessment activities to learning activities intended to provide instructional information to both the student and the teacher by: 1) building on what students already know. If left untouched. or reviewing lecture notes with the students prior to actually giving the test. pilot-study evidence suggested the need to reduce the number of Reflective Lessons.On the Development of Reflective Lessons conceptualized as a formalized teaching pattern consisting of a set of expectations of what events are necessary and about the temporal order of such events. A summative assessment teaching script might include studying for the tests. the Reflective Lessons would have taken a teacher about 15 lesson periods to complete (see Furtak et al. and 5) reflecting on material covered. 2) attending to student conceptions and misconceptions.. we changed the name from embedded assessments to Reflective Lessons. as such. 3) making student conceptions and misconceptions public and observable. detailed information about how to use the “teachable moments” should be provided with the Reflective Lessons. but they did not necessarily use these conceptions to further student learning. 4) priming students for future learning. Other pilot study evidence collected suggested that teachers needed increased structure in order to use the Reflective Lessons and. This shift from assessment activities to learning activities represents a fundamental change for teachers in the way to look at the formative assessment practices. taking practice tests. In order to avoid the usual summative assessment teaching scripts. For example. teachers were able to elicit student conceptions about why things sink and float. The Assessment Development Team removed some assessments because they believed that regular 16 .

On the Development of Reflective Lessons classroom practices allowed for formative assessment of students declarative and procedural knowledge and because they believed that too much time was taken away from the regular science curriculum. the Team set the goals of adding no more than two lesson periods at the natural joints and one period each for the concept maps. These Reflective Lessons and scripts were used in the final Romance experiment. Further. the Team set a goal of simplifying the structure of the assessments by identifying two reflective lesson scripts. The tension between time spent on assessments and time spent on instruction as if the assessment were not instructional continued even though the Team began to move towards believing that these assessment were instructional. 17 . Consequently.

help students track their sinking and floating conceptions and help students reflect about these concepts (e.g. In order to achieve this purpose. Duschl.On the Development of Reflective Lessons Reflective Lessons The Reflective Lessons were composed of a carefully designed sequence of investigations (prompts) that enabled teachers to step back at key points during an instruction unit (natural conceptual joints) to check student understanding. The SEAL team developed drafts of the assessments and.. the Reflective Lessons were designed to elicit and make public student sinking-and-floating conceptions. Elicit and make students’ conceptions public. these assessments were carefully designed to match the content and structure of the existing FAST investigations.. “Why things sink and float?”). 2003). using talk-aloud methods.g.. and to reflect on the next steps that they must take to move forward (Figure 2). Reflective Lesson Goals In order to promote student understanding of why things sink and float via relative density concepts. The Reflective Lessons were intended to make students’ thinking about different physical phenomena explicit (e. push on students’ why things sink and float conceptions. Asking students.g. having students predict 18 . refined them in tryout-refine-tryout cycles conducted with students from a local school near Stanford and the University of Hawaii CRDG Laboratory School. Reflective Lessons included a set of activities/investigations that bring forth students’ conceptions of density and why things sink and float (e. Furthermore. encourage communication and argumentation based on evidence from the investigations or assessment tasks.

students could be provided with the opportunity to see competing student conceptions. small and/or large group discussions. they included different suggested teaching strategies. For example. We designed the Reflective Lessons to provide opportunities for students to discuss and debate what they know and understand. results of an investigation). and how these explanations could be generalized to other similar situations. that is. the universality of the scientific principles. and hear and evaluate the supporting evidence provided for the different competing views. the Reflective Lessons provided teachers with scripts to help students support their why-things-sink-and-float explanations based on scientifically sound evidence and help test the universality of student claims. sharing activities. Moreover. how this evidence is used to support their predictions. Reflective Lessons were concerned with what data counts as evidence for students (Uncle Joe’s tales of the sea vs. collecting student 19 . That is. Furthermore. and questions and prompts that help teachers reveal students’ conceptions and thinking and make these conceptions public. By having students take a stand and make their why-things-sink-and-float conceptions public. That is. Encourage communication and argumentation of ideas using evidence.On the Development of Reflective Lessons whether a small-sized high density plastic block will sink or float and explain why). can the notion that more bbs means more sinking be applied in all sinking and floating events? The use of evidence for explanations in the Reflective Lessons was an extension of the pedagogical practices already found in FAST. such as student group work. the Reflective Lessons provided for examples and models for setting up the lessons. decisions and explanations.

The Reflective Lessons were also designed to push on students’ ideas about why things sink and float. the Reflective Lessons asked students to predict whether a large-sized low-density plastic block (polypropylene) or a small-sized lightweight high-density plastic block (PVC) would sink or float. Students focused on the size and weight of the two blocks (decisions based on everyday life) rather than on the density of the plastic (decisions based on scientific evidence). and providing questions to push students evidence-based why things sink and float explanations: “How do you know that?” “What evidence do you have to support your explanations?” “In what cases does your explanation apply and what cases does it not apply?” Such discussion allows students to think about how evidence should be used to generate and justify explanations as well as think about the universality of their explanations. That is. For example.On the Development of Reflective Lessons conceptions. The Reflective Lessons were also designed in part to track student why-things-sink-and-float conceptions. establishing discussion events. but rather the problems provide instances where everyday knowledge about sinking and floating events cannot be easily applied. each Reflective Lesson included the Why Things Sink and Float prompt and the same concept 20 . Push on students’ why-things-sink-and-float conceptions. students are presented with problems that seem to be buoyancy anomalies. in the Reflective Lesson suites there were assessment items that were used each time and that allow for teacher and students to track student understanding. With the goal of tracking students’ conceptual development. Help students track why-things-sink-and-float conceptions. In the Reflective Lessons.

teachers and students find the problems they are having as they move towards the unit goals. the Reflective Lessons provided teachers with strategies for progress that can help students achieve these goals.On the Development of Reflective Lessons terms were used in both Reflective-Lesson concept map administrations. push on student buoyancy conceptions. class discussion. and helping teachers and students reflect about 21 . Reflective Lessons provided specific prompts that have proved useful for eliciting students’ conceptions. when to do it with students and how to do it with students. looking for universality of student claims. student progress could be observed. reflect on student’s buoyancy conceptions) in their instructional materials about group work. Types of Reflective Lessons In order to accomplish these goals. and help guide the development of students’ understanding by identifying where students were going wrong. The curriculum developers stated the importance of these types of learning activities (i.e. The Reflective Lesson goals were derived directly from the FAST curriculum. What the reflective lessons did. The Reflective Lessons were intended to help teachers pinpoint students’ conceptual development at key instructional points. By comparing earlier with later assessments.. encourage argumentation about evidence. is to make the whole reflective discussion explicit in terms of what to do with students. Connect to the curriculum. Reflect with students about their conceptions. encouraging communication and argumentation based on evidence. Furthermore. Through making student conceptions public and then through discussions and debate based on evidence.

On the Development of Reflective Lessons their learning and instruction. The prompts and questions focus students on the use of evidence to support their conceptions of why things sink and float. Graph Prompt. These lessons were embedded at three joints in the unit. In order to simplify the Reflective Lessons. Each Type I Reflective Lesson has four prompts for students to: (1) interpret and evaluate a graph. Students are familiar with the data representation used because the scatter plots are like those created by the students in their FAST investigations and reveal the important relationship between two variables that were just studied. 7 and 10 (Figure 2). to encourage students to bring evidence to support their conceptions. Type I Reflective Lesson The Type I Reflective Lessons were designed to expose students’ conceptions of why things sink and float. two types were identified. This prompt requires students to interpret. and to raise questions about the universality of their conceptions when applied to new situations. These prompts vary according to where the Reflective Lessons are embedded within the unit. after investigations 4. The Assessment Development 22 . and (4) predict-observe an event related to sinking and floating. resulting data and explanations they have collected in different FAST investigations as evidence to support their responses and conclusions (Figure 3). This prompt asks students to use knowledge of investigations. (2) predict-observe-explain an event related to sinking and floating. (3) answer a short question. evaluate and complete graphs that focus on the variables involved in sinking and floating specific to the FAST investigations that they have just completed.

This prompt get directly at schematic knowledge. These activities focus on the variables that the students have just completed in their FAST investigations. Careful administration is important for this prompt type. 1992). Students use what they have learned in the unit to help them make predictions and to explain and justify their ideas. Graphs like these are used throughout the 12 physical science investigations in the unit. For example an important administration consideration is when and how to collect student predictions and evidence based explanations without affecting student reconciliations—to make ideas public and anonymous.On the Development of Reflective Lessons Team chose the graphing prompts because of the importance of interpreting graphs in the curriculum in helping students use data to draw conclusions and support explanations. Students are expected to use evidence in their explanations and reconciliations. In order to make students’ thinking explicit. observations. It asks students to predict the outcome of a sinking-and-floating event and to justify their prediction. 23 . The activities that were used in this assessment came from the literature on student’s understanding about density. then observe the event. --------------------------Insert Figure 3 --------------------------Predict-Observe-Explain Prompt. and then reconcile their predictions and observations (White & Gunstone. and from the repertoire of FAST and non-Fast science teachers associated with the project. they must write their predictions. explanations and reconciliations. developmental psychology literature of dimensionality (relating two variables together) in student learning.

the pre and post tests). The same prompt is used as is same in all three Type I Reflective Lesson after investigations at 4. ----------------------Insert Figure 5 ----------------------Predict-Observe Prompt.e. This prompt is a direct result of the ADT work in clarifying the goal of the instructional unit. this prompt would eventually serve as a vehicle to track student understanding across FAST 1 implementations thus providing a link between the embedded formal formative assessments and the summative assessments (i. These FAST challenge questions were restructured using a modified POE model and student worksheets were developed for these POs. this prompt asks students to predict and observe an event (Figure 6). Furthermore. Students are not asked to reconcile predictions and observations. These prompts are based on the FAST 1 challenge questions that are provided at the end of each natural joint investigation. POs act as a launching point for the 24 . 7 and 10. A slight variation on the Predict-Observe-Explain prompt. This prompt was a single question that asks students to explain why things sink and float with supporting examples and evidence (Figure 5)..On the Development of Reflective Lessons ------------------Insert Figure 4 ------------------Short-Answer Prompt.

The teacher’s manual for the Reflective Lessons suggests that the discussion during the first leg focuses on the 25 . this issue). they in themselves represent only part of what needs to happen to promote the goals of the Reflective Lessons. -----------------Insert Figure 6 ------------------Type I Reflective Lesson Implementation..On the Development of Reflective Lessons next instructional activity of the unit in which the explanation will emerge. The Team allowed for three class sessions for Reflective Lesson Type I implementation. POs can be thought of as a springboard that motivates students to start thinking about the next investigation might be about. although the Team expected implementation might take only two or two-and-a-half sessions (see Furtak et al. The implementation is as follows (Figure 7): -----------------Insert Figure 7 ------------------The important components of the reflective lessons are the discussions that occur after the students have completed a few of the Reflective Lesson prompts. the teacher makes student conceptions and reasoning public and where the relevance and importance of supporting one’s explanations with evidence and pushing for universality are brought to the surface. The discussions are key because in the discussions. Each session was planned to take about 45 minutes or so to complete. While the actual Reflective Lessons are important.

The concept maps were based on the terms that make up the content of 26 . Only one kind of prompt is used in this type of reflective lesson. or divided as shown in Figure 6 above so that part one is carried out at the end of class session two. Reflective Lessons and discussions.Observe-Explain prompt as evidence. The two parts may be carried out on the same day. the discussion focuses on students’ conceptions and the universality of the ideas and evidence. the teacher is expected to push them beyond these toward universality and encourage students to use information learned in class. In the subsequent short answer question Why do things sink and float?. while part two is carried out at the beginning of class session 3. Many teachers carried out the Predict-Observe prompt on one day and left the explanation and recap for the next day as an introduction to the next investigation. Type II Reflective Lesson This type of reflective lesson focuses on checking students’ progress in their conceptual understanding of the key concepts across the twelve investigations. is performed in two parts. If students rely primarily on the graph or Predict. First students make a prediction based on recent class investigations. although both might be used as evidence.Observe-Explain prompt. The last Reflective Lesson prompt. a Concept Map. and then they observe a demonstration that may challenge their ideas.On the Development of Reflective Lessons information from the graph and the Predict-Observe-Explain prompt as well as what has happened in class. Concept maps provide evidence on how students see the relationships between ideas/concepts. Students are asked to extend what they know beyond the context of the Graph or Predict.Observe prompt. the Predict.

the whole class concept map is not desirable. For example. but it needs to be done carefully and teachers need to make sure that students know how to construct concept maps in accordance with as set of rules. The second time students construct the maps they need only to be reminded of the rules and do not need the entire training. -----------------Insert Figure 8 ------------------The first box in the implementation model reflects that students need to be trained to draw concept maps. the most important aspect of student interaction occurs when the students construct a group concept map. students are not expected to know how to 27 .On the Development of Reflective Lessons the entire series of twelve investigations. This training takes about 30 minutes. looking at the terms most often used in the group maps. we found in our pilot study that only a handful of students participated in constructing a whole class concept map and consequently. The sharing of main ideas benefits from looking for those terms that students have not been able to place into the concept maps. Based on the pilot study observations. Concept maps make evident students’ understanding and the evolving sophistication of their thinking as their investigations progress. These Reflective Lessons are implemented in one session (Figure 8). and having students describe how their group concept maps were constructed. The concept map Reflective Lessons were inserted after Investigations 6 and 11. While a class concept map may be desirable.

the students in the first three investigations work with alternative conceptions of sinking and floating events (size. number of bbs. That is. while in Investigation 11 students 28 . In Investigation 10 (and the in part in 9). assessment blueprints were drawn and the Reflective Lessons refined. FAST Buoyancy Learning Trajectory While the curriculum was reviewed for natural joints. especially displaced volume. plays an important role in explaining sinking and floating events. greater or lesser amounts of something) and then in investigation 4 students focus in on mass as an important variable for sinking-and-floating events. In Investigation 7-9 students now combine both volume and mass together to explain sinking-and-floating events. students hold mass and volume variables independent of each other. In Investigations 5 and 6 students learn that volume. a learning trajectory for student understanding of why things sink and float using a relative density explanation was developed to guide teachers. the 12 FAST investigations build a student’s understand of why things sink and float through a series of models. -----------------Insert Figure 9 ------------------That is.and posttests. students learn about the density of an object. In Investigations 4 through 6. This model is evident in the curriculum and was made explicit through the development of the Reflective Lessons and the study’s pre.On the Development of Reflective Lessons relate one of the concept map terms (density) with the other concept terms in their whythings-sink-and-float concept map until investigation 9.

Many student responses can readily be placed in the model and a teacher armed with this knowledge. students learn that sinking and floating events can be explained by comparing the density of an object and the density of medium in which it is placed—relative density. might fashion a discussion question that gets at what is important for a student to know to move forward. In this assessment development. The learning trajectory is useful when a teacher is trying to understand where a student is in her understanding of why things sink and float.On the Development of Reflective Lessons learn that liquids have density too. collaboration of the assessment specialists along with curriculum developers is vital. Finally in Investigation 12. several considerations may be drawn from this project. Furthermore. For example. Conclusions In order to create efficacious embedded formal formative assessments or Reflective Lessons in new or existing curriculum. if during the Reflective Lesson after investigation 4 a student speaks about how more sand makes the straw sink more a teacher might ask this student “What do you think the bbs and sand have in common that affects why things sink and float?” Knowing where a student’s response is in the FAST learning trajectory may help a teacher think about what a student needs to know in order to increase their understanding. the line between assessment specialists and curriculum developers was very thin and often blurred because these Reflective Lessons appeared seamless to students and teachers in terms of instructional 29 . First. providing teachers with this learning trajectory allowed teachers to identify where students might be in their conceptions about sinking and floating thus simplifying teachers’ categorization of student conceptions into five levels.

This may raise questions about the need for some lessons in established curricula. Assessment specialists or curriculum developers cannot expect teachers to effectively use reflective lessons without training in their use of the strength of teachers’ summative assessment scripts which may defeat the purpose of the reflective lessons. Third. the reflective lessons must be linked to the overall goal of the curriculum and not only to the material that the students have just covered. If the assessments do not look like the other lessons it is hard to make the information connect to what the teachers and students are already and will be doing. professional development should be provided to teachers to reconceptualize the value of assessment.. focusing on concepts or procedures that might not be vital to the overall goal of the instructional unit (i. Fourth. This helps ameliorate the summative assessment teaching script and student testing scripts as well as helps teachers and students make use of the information that they gather from these reflective lessons. This trajectory albeit useful in the reflective lessons proved helpful beyond those lessons because teachers were expected to use the trajectory as a vehicle to track student understanding throughout unit implementation. focusing on students’ abilities to mass an object in a why-things-sink-and-float unit). If the reflective lessons are only linked to the material just covered. in the development of these reflective lessons. 30 . Second.On the Development of Reflective Lessons approaches and in content. a useful tool was the development of the why things sink and float learning trajectory. instructional time may be misused.e. especially formative assessment.

using the information gained (see Ayala. When we consider formative assessments. Finally. 2005). Having some embedded formal formative assessments or reflective lessons in a curriculum is useful because it reminds teachers to reflect back on what has been learned and hopefully guide future lessons toward unit goals. and most importantly. developing the assessments. implementing the assessments. we lack knowledge of how teachers use reflective-lesson information (but see Furtak et al. this project raises questions about the quantity and frequency of formative assessments in a curriculum. These formative assessments should be coherently focused on the overall goal of the unit rather than each minuscule instructional or assessment target.. 31 . The more efficacious use of these assessments may reside in the application of formative assessment principles to all instructional activities (including reflective lessons). this issue). assessment specialists and curriculum developers must consider all five important assessment pedagogies: knowing the content. collecting information from the assessments.On the Development of Reflective Lessons Fifth.

Black. P. 31. J. Cognitive skills and their acquistion. 32 . Classroom assessment is not (necessarily) formative assessment (and vice-versa). D. de Jong. M. Hillsdale.J. VA: NSTA Press. NJ: Erlbaum. Atkin & J.R. 41-59). Hillsdale. C. & Ferguson-Hesser. The characteristics of formative assessment in science education. Mental models. Assessment in education. 7-73. In J. T. Bell. NJ: Erlbaum. R. deKleer. & Cowie. Everyday assessment in the science classroom (pp.).. (1998). Gentner. & Wiliam. Towards a coherence between classroom assessment and accountability. Black. M. Coffey (Eds. J (Eds. Ayala. Paper presented at the annual meeting of the American Educational Research Association. & Wiliam. Montreal Canada. (2004).). 5(1). D.. B. (1996). Assessment of inquiry. (2001)..On the Development of Reflective Lessons References Atkin.. B. 85. A.). M. J. Types and qualities of knowledge. Everyday Assessment in the science classroom. & Stevens. (2003). G. 105-113. (1981). J. M. Duschl. In M.S. Science Education. Anderson (ed. & Brown. Educational Psychologist. D. Arlington. VA: NSTA Press. April). A. (1983). Arlington. Assessment Pedagogies. Mental models of physical mechanisms and their acquisition. P. Chicago: University of Chicago Press. In J.) (2003). & Coffey. 536–553. Wilson (Ed. Assessment and classroom learning. (2005.

R. Laboratory for learning. & Shavelson. (1999). University of Hawaii Manoa: Curriculum Research and Development Group.).. and Young. Paper presented at the American Educational Research Association Annual Conference. V. San Diego. M. In S. A. National Research Council. M. Washington.C. (1992). Young. 27–29. The Local Environment: FAST 1 Foundational Approaches in Science Teaching. Pauls.On the Development of Reflective Lessons King.. R. Informal assessment of students' understanding of scientific inquiry. London: 33 . B. Contexts of learning mathematics and science: Lessons learned from TIMSS. Brownell. & Furtak. (1966). and J.: National Academy Press. The curriculum and the disciplines of knowledge. M. & Lapitkova. Classroom assessment and the national science education standards. Examining the links between science achievement and assessment. (2001). Pottenger. Towards a science achievement framework: The case of TIMSS 1999. A Paper presented at the American Educational Research Association Annual Meeting.A.A. The science teacher. WA. D. Wiley: New York. (2001). CA. Ruiz-Primo.. F. Plomp (Eds. Seattle. M. D. Howie & T... M. A. (2006). & Shavelson. J.J. Li.J. Li. D. 66. R. (2004). a theory of curriculum practice.. Routledge. Ruiz-Primo. E.

M. Seville. Servicio de Publicaciones. 4(4). Embedding assessments in the FAST curriculum: The romance between curriculum and assessment. Unterrichts wissenschaft.. (1999). (1986). R.On the Development of Reflective Lessons Ruiz-Primo.J. Toma de decision interactiva: Algunas reflexiones sobre los procesos cognoscitivos de los profesores. & Pine. Journal of Research in Science Teaching.. Angulo (Ed. 33(10). Baxter. R. Shavelson. (Translation: Interactive decision making: Some thoughts on teacher cognition. (1996b). Problems and issues in the use of concept maps in science assessment. Spain. M. 33(6).).M. R.) In L. (2005). 10451063. Performance assessment in science. Final Report. Pensamientos de los profesores y toma de decisiones Universidad de Sevilla.. J.. On the assessment of science achievement. J. (English version). R. 569-600. 27(2). Stanford Educational Assessment Laboratory (SEAL) and Curriculum Research & Development Group (CRDG). Ruiz-Primo. Shavelson. Journal of research in science teaching. 34 . & Shavelson. & Ruiz-Primo. 347-362. (1996a). J. Applied measurement In education.A. (1991).V. Shavelson. A.J. R.. G. & Shavelson. R. Rhetoric and Reality in Science Performance Assessments: An Update. 102-127. A. Shavelson.

(2000). 35 . 5. Santa Barbara. University of California. Tamir.3(1). CA: Center for Research on Education. & Yamamoto. 23–28. Falmer Press: London. Studies in Educational Evaluation.On the Development of Reflective Lessons Shepherd. and R. R.. D. Diversity & Excellence.A. England. Pacific-Asian Education. K. B. 35–49. Gunstone. Local science program makes good: The evaluation of FAST. and Education. 1. 7–17 White. Probing understanding. (1993). B. (1982). Young. The role of classroom assessment in teaching and learning. D. Science achievement and thinking skills. L. Santa Barbara. The effects of junior high ‘FAST’ program on student achievement and preferences in high school biology. Technology. 1977. Human Sciences. 1992. P. Young.

Figure 2. 2003) Figure 8. Reflective Lesson graph after investigation 7. Figure 4. Figure 3.On the Development of Reflective Lessons Figure Captions Figure 1: Assessment suite timeline. Type I Reflective Lesson Suite implementation by session (adapted from SEAL’s Teacher guide to the reflective lessons. Final reflective lesson suites and placement. Figure 7. FAST Buoyancy Learning Trajectory. Type I Reflective Lesson Predict-Observe after investigation 7. Type I Reflective Lesson Predict-Observe-Explain prompt after investigation 7 Figure 5. Figure 9. 36 . Figure 6. Implementation of the Type II Reflective Lesson in one session. Type I Reflective-lesson short-answer after Investigation 7.

On the Development of Reflective Lessons Figure 1: 37 .

On the Development of Reflective Lessons Figure 2. 38 .

On the Development of Reflective Lessons Figure 3 39 .

40 .On the Development of Reflective Lessons Figure 4.

41 . Figure 6.On the Development of Reflective Lessons Figure 5.

. Concept Map Training Individual Map Group Map Sharing Main Ideas from Group Maps 42 Figure 8.On the Development of Reflective Lessons First Session   Graph  Interpretation  Piece of Evidence #1 Predict­Observe­Explain  DISCUSSION (POE)  Piece of Evidence # 2 Second Session  Short­Answer  Why Things Sink and Float  DISCUSSION Predict  (P) Third Session  Observe  (O)  Next  Investigation  Figure 7.

On the Development of Reflective Lessons Figure 9. 43 .

student tasks. density graph. displace volume of floating and sinking objects Discover how a Cartesian Diver works Find the density of Cartesian divers of different masses and volumes Find the density of floating and sinking objects. Determining the density of liquids Experiment with different objects in liquids of different densities 44 . 2004) Lesson 1 Investigation Liquids and Vials Sinking Straws Student Tasks Observing vials of different liquids sinking and floating in different liquids (a buoyancy anomaly) Adding BBs to a straw and measuring the depth of sinking Graphing number of BBs versus depth of sinking Learning Goals Make scientific observations and test predictions Predict the number of BBs needed to sink a straw to particular depth Represent BB data in line graphs. the carton size and the depth of sinking Calculate the displaced volume of different cartons Graph mass vs. First twelve FAST 1 investigations. more BBs more sinking Conclude that more mass more sinking Discover the relationship between the amount of ballast.On the Development of Reflective Lessons Table 1. and learning goals (Adapted from Ruiz-Primo & Furtak. Discover that different liquids have different densities Understand relative density 2 3 4 5 Graphing Sinking Straws Data Mass and the Finding the relationship between total mass Sinking Straws and depth of sinking of straws Sinking Cartons Measuring the depth of sinking of different sizes and of equal mass Volume and Sinking Cartons Floating and Sinking Objects Introduction to Cartesian Divers Density and the Cartesian Diver Density of Objects Density of Liquids Buoyancy of Liquids Finding the submerged and total volume of cartons Finding the mass and displaced volume of different objects Experimenting with Cartesian Divers 6 7 8 9 10 11 12 Investigating the relationship between the diver’s mass and volume at different sinking and floating positions Finding the relationship between mass and total volume and sinking and floating.

Sign up to vote on this title
UsefulNot useful