Research perspectives and formative assessment

ASME Conference: Researching Medical Education, November 2009: RIBA, London Dylan Wiliam

www.ioe.ac.uk

Overview
The nature of educational research What should educational research try to do? How should it try to do it? Formative assessment Definitions Implementations Researching formative assessment

Pasteurs quadrant

Educational research
An elusive science (Lagemann, 2000) A search for disciplinary foundations Making social science matter (Flyvbjerg, 2001) Contrast between analytic rationality and value-rationality Physical science succeeds when it focuses on analytic rationality Social science fails when it focuses on analytic rationality, but succeeds when it focuses on value-rationality

Research methods 101: causality

Does X cause Y? In the presence of X, Y happened (factual) Problem: post hoc ergo propter hoc Desired inference: If X had not happened, Y would not have happened (counterfactual) Problem: X did happen So we need to create a parallel world where X did not happen Same group different time (baseline measurement) Need to assume stability over time Different group same time (control group) Need to assume groups are equivalent Randomized contolled trial

Plausible rival hypotheses

Example: Smoking cigarettes causes lung cancer Randomized controlled trial not possible Have to rely on other methods Logic of inference-making Establish the warrant for chosen inferences Establish that plausible rival interpretations are less warranted

Knowledge
Not justified-true-belief Discriminability (Goldman, 1976) Elimination of plausible rival hypotheses Building knowledge involves: marshalling evidence to support the desired inference eliminating plausible rival interpretations Plausible determined by reference to a theory, a community of practice, or a dominant discourse

Inquiry systems (Churchman, 1971)

System Leibnizian Lockean Kantian Hegelian Singerian Evidence Rationality Observation Representation Dialectic Values, ethics and practical consequences

Inquiry systems
The Lockean inquirer displays the fundamental data that all experts agree are accurate and relevant, and then builds a consistent story out of these. The Kantian inquirer displays the same story from different points of view, emphasising thereby that what is put into the story by the internal mode of representation is not given from the outside. But the Hegelian inquirer, using the same data, tells two stories, one supporting the most prominent policy on one side, the other supporting the most promising story on the other side (Churchman, 1971 p. 177).

Singerian inquiry systems

The is taken to be is a self-imposed imperative of the community. Taken in the context of the whole Singerian theory of inquiry and progress, the imperative has the status of an ethical judgment. That is, the community judges that to accept its instruction is to bring about a suitable tactic or strategy [...]. The acceptance may lead to social actions outside of inquiry, or to new kinds of inquiry, or whatever. Part of the communitys judgement is concerned with the appropriateness of these actions from an ethical point of view. Hence the linguistic puzzle which bothered some empiricistshow the inquiring system can pass linguistically from is statements to ought statements is no puzzle at all in the Singerian inquirer: the inquiring system speaks exclusively in the ought, the is being only a convenient faon de parler when one wants to block out the uncertainty in the discourse. (Churchman, 1971: 202).

Educational research
can be characterised as a never-ending process of assembling evidence that: particular inferences are warranted on the basis of the available evidence; such inferences are more warranted than plausible rival inferences; the consequences of such inferences are ethically defensible. The basis for warrants, the other plausible interpretations, and the ethical bases for defending the consequences, are themselves constantly open to scrutiny and question.

Effective learning environments

A prevalent, mistaken, view Teachers create learning The teachers job is to do the learning for the learner A not so prevalent, not quite so mistaken, but equally dangerous view Only learners can create learning The teachers job is to facilitate learning A difficult to negotiate, middle path Teaching as the engineering of effective learning environments Key features: Create student engagement (pedagogies of engagement) Well-regulated (pedagogies of contingency) Develop habits of mind (pedagogies of formation)

Formative assessment: a definition

An assessment functions formatively to the extent that evidence about student achievement elicited by the assessment is interpreted and used to make decisions about the next steps in instruction that are likely to be better, or better founded, than the decisions that would have been taken in the absence of that evidence. Formative assessment therefore involves the creation of, and capitalization upon, moments of contingency (short, medium and long cycle) in instruction with a view to regulating learning (proactive, interactive, and retroactive). (Wiliam, 2009)

The formative assessment hi-jack

Long-cycle Span: across units, terms Length: four weeks to one year Impact: Student monitoring; curriculum alignment Medium-cycle Span: within and between teaching units Length: one to four weeks Impact: Improved, student-involved, assessment; teacher cognition about learning Short-cycle Span: within and between lessons Length: day-by-day: 24 to 48 hours minute-by-minute: 5 seconds to 2 hours Impact: classroom practice; student engagement

Unpacking assessment for learning

Key processes Establishing where the learners are in their learning Establishing where they are going Working out how to get there Participants Teachers Peers Learners

Five key strategies

Clarifying, understanding, and sharing learning intentions curriculum philosophy Engineering effective classroom discussions, tasks and activities that elicit evidence of learning classroom discourse, interactive whole-class teaching Providing feedback that moves learners forward feedback Activating students as learning resources for one another collaborative learning, reciprocal teaching, peer-assessment Activating students as owners of their own learning metacognition, motivation, interest, attribution, self-assessment
(Wiliam & Thompson, 2007)

and one big idea

Use evidence about learning to adapt instruction to better meet learner needs

A model for professional change

Content Evidence Ideas Process Choice Flexibility Small steps Accountability Support

KMO Formative Assessment Project

24 teachers, each developing their practice in individual ways Different outcome variables No possibility of standardized controls Polyexperiment with local design Synthesis by standardized effect size

1.5 1.4 1.3 1.2 1.1 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 -0.1 -0.2 -0.3 -0.4

0 5

6 0 4 0 5 3 4 5 6 6 8 8 8 3 4 5 6 6 7 9 9

Jack-knife estimate of mean effect size: 0.32; 95% C.I. [0.16, 0.48)

Effect size by comparison type

I S P L D N Parallel set taught by same teacher in same year Similar set taught by same teacher in previous year Parallel set taught by different teacher in same year Similar set taught by different teacher in previous year Non-parallel set taught by different teacher in same year National norms

Summary
Educational research is a never-completed process of assembling evidence that: particular inferences are warranted on the basis of the available evidence; such inferences are more warranted than plausible rival inferences; the consequences of such inferences are ethically defensible. The basis for each of these is constantly open to scrutiny and question