EXPERIMENTAL RESEARCH Martin Kozloff An experimental research strategy usually involves comparison of two or more situations (classes, schools,

districts) that (through random assignment or through matching on important factors, such as race and social class) are the same in every way possible, but that differ in the few factors (such as curriculum materials, teaching methods, leadership style) whose effects or outcomes (e.g., achievement) you are testing. One group is the e perimental gro!p, that receives the intervention. The other is the control gro!p (that receives the usual treatment ) or is an alternati"e treatment gro!p, that receives a different intervention that is being tested. !ertain #esign elements ma"e the data from experiments more "ali# (accurate), and therefore more believable and useable. These design elements are# a. Samples are representati"e of the population for which the findings are relevant. b. Ran#omization or matc$ing are used to produce e$uivalent groups. c. %ariables to measure are clearl% #efine## conceptual definitions (general) and operational definitions (examples& things to measure). d. A wi#e range of "aria&les is measured so that a comprehensive picture is obtained& e.g., achievement, student engagement, the $uality of a curriculum, the $uality of instruction, and teacher satisfaction are all measured.

e. The same variables are measured in several different ways (triang!lation)to see if they support the same finding& e.g., achievement is measured by a standardi'ed test, by fre$uent curriculum(based measures (tests that are in the curriculum materials), and teachers) perceptions of students) progress. f. *easures (e.g., how to count behavior) and instruments (for tests) are "ali#ate#. g. *easurement (e.g., testing) is chec"ed for tester or observer relia&ilit%. h. +oth pre'tests an# post'tests are given. ,ven better, data are also collected periodically #!ring an inter"ention to see if there is any trend& e.g., gradual vs. rapid change. i. The experiment is replicate# (done again) with similar samples (to see if the findings are consistent and strong(((not a flu"e), and with #ifferent samples (to see if the findings apply elsewhere). This is a "ind of experiment in which data are ta"en (for example, on students) math s"ill) before and at the end of an intervention, a teaching method is used, or a change is made in class. -f nothing else changed between the pre( test and post(test (except the delivery of instruction), then it is li"ely that any increase in students) "nowledge (shown by comparing the pre(test scores and post(test score) is the result of instruction. A pre-test, post-test design with one group is not as powerful as a pretest, post-test design that uses an experimental group and control group . -f you have only one group, other factors !O./0 have operated between the pre(test and the post(test that affected post(test scores. 1or example, some children got tutoring, and that made their scores higher. -f the researcher concludes that the class scores were higher at the post(test +,!A.2, of the new math curriculum, this claim would be in"ali#(

The experimental design that has e$uivalent experimental and control groups means that any OT3,4 changes in the groups between the pre(test and the post(test (e.g., tutoring) could have happened to both groups. Therefore, the O5, main difference is 2T-// the difference in curriculum.

Researc$ Strategies) E perimental Researc$ 6.T -5 0,1 14O* TA+/, ,xperiments involve collecting data on groups of people (from small groups to samples of communities or larger units) that have experienced differential exposure to some variable or variables (the 7condition7). ,xperiments are of several "inds. A( La&orator% E periments, in which participants are studied in a place established by the experimenter& e.g., a laboratory study in which children in a play room are reinforced by an adult every time they put their toys away or clean up after themselves, in order to test the effects of reinforcement on cleaning(up behavior. *( +iel# E periments, in which participants are observed in natural surroundings, such as existing classrooms, in stores, on the street, and at the beach. 1or example, at a national par", the experimenter might reinforce children whenever they put trash in trash cans. Obviously, the generaliza&ilit% of the findings from the field experiment is li"ely to be greater than that from a laboratory experiment. 3owever, the laboratory allows more control over possible sources of contamination. (There always are trade(offs). C( Contri"e# E periments, in which the experimenter (either in a laboratory or in a field setting) manip!lates participants8 exposure to the experimental variables whose effects are being studied. ,( Nat!ralistic E periments, in which the experimenter has little or no control over the participants8 exposure, or perhaps even over possible extraneous variables. ,xamples of naturalistic experiments include#

9) studies of changes in social support, cooperation, and interpersonal sentiments before and after a natural disaster& :) differences in students8 en;oyment of the learning process, and accomplishments, under different instructional situations (self(paced vs. teacher(paced& peer(tutoring vs. independent wor")& and <) staff morale and rate of ;ob turnover, and scores on student achievement tests, before and during A+! legislation. 5umerous experimental designs (e.g., the pre(test, post(test, experimental and control group design vs. the time series design) and experimental procedures (e.g., blind vs. double(blind& random allocation vs. matching) are employed to wea"en rival hypotheses and to provide evidence of a causal connection between the independent (experimental) variable(s) and the dependent variable(s). The designs and procedures differ in their cost, feasibility, ethical acceptability, and inference(power. ,xperiments can use all of the recording methods described above. I( LE-EN, X = exposure of a group or an individual (e.g., in a single(sub;ect design) to experimental (i,e., independent) variables (i.e., events, conditions or treatments). The experimenter may or may not have control over this exposure. . = observation or measurement. >ere the "ali#it% and relia&ilit% of instruments and measurement established prior to use? >as observer reliability chec"ed periodically during the experiment? Left'to'rig$t #imension = time order. X8s and .8x in a vertical line mean that observation and exposure to or change in experimental variables occur simultaneously. R = random assignment to comparison groups. >as matc$ing used to ma"e groups more e$uivalent on certain variables? Or was ran#omization used to give all possibly contaminating (e traneo!s) variables an e$ual chance to be

in all groups? >as the e$uivalence of comparison groups assessed pior to or after a pre(test?

II( ,ESI-NS) PRE'EXPERIMENTAL/ EXPERIMENTAL/ AN, EXPERIMENTAL A( Pre'e perimental ,esigns

01ASI'

6re(experimental designs have either no comparison groups or comparison groups whose e$uivalence is indeterminate. 2( .ne's$ot Case St!#%. -n this design there is almost complete absence of control over (or at least a determination of) extraneous variables that might account for the findings. 2ince there is no pre(test and no comparison group, inferences that @ had an effect are based upon a comparison between the one case studied and other cases observed and remembered. That is, a causal inference (that @ has an effect on the dependent variables observed) is based on general expectations of what the data wo!l# have been had @ not occurred. @ O

3( .ne -ro!p Pre'test Post'test. One group is exposed to the presence of @ or a change in @ and is measured before and after this has occurred. The design at least enables you to compare the dependent variable(s) before and after @. 3owever, the absence of a comparison group (e.g., in design A) or the absence of a series of changes in the independent variable(s) (e.g., introducing and then removing an intervention several times Bin design CD), means that one cannot rule out plausible rival hypotheses (explanations) for differences between pre( and post(test (e.g., maturation).

O9

O:

4( Static'gro!p Comparison. One group is exposed to @ and is compared with another group which is not exposed to @. 3owever, the absence of a pre( test means that one cannot determine whether the groups were e$uivalent with respect to the dependent variable(s) or other extraneous variables. @ O9 O:

*( Tr!e E perimental ,esigns These designs provide formal means (pre(tests andEor comparison groups created by random allocation) for handling many of the extraneous variables that wea"en internal and external validity. 5( Pre'test/ Post'test/ E periment'gro!p/ Control'gro!p ,esign( This is the classic design. -t allows comparisons within and between groups that are as similar as randomi'ation can ma"e them. 5ote that the design is such that *ills8 *ethod of 0ifference can be used to infer a causal relationship between @ and changes in dependent variables from pre( to post(test, and differences between , and ! groups at the post(test. 4 4 O9 O< @ O: OA

5ote, however, that the pre(tests wea"en external validity. That is, groups outside of the experimental situation may not normally be pre(tested. 6( Solomon +o!r'gro!p ,esign( This design formally considers sources of external invalidity, especially the reactive effects of testing. 4 O9 @ O:

4 4 4

O<

@ @

OA OF OG

!omparison of O: and OA suggests the effects of @ (but with the pre(test). !omparison of OF and OG suggests the effects of @ without the pre(test, and enables you to assess the effects of the pre(test. !omparison of O: and OF also suggests effects of the pre(test. !omparison of O9, OG, O< and OA suggests the effects of the passage of time. 7( Post'test'.nl% Control -ro!p ,esign( This is a useful design when pre(tests are not feasible. 5ote that randomi'ation is an all(purpose assurance of a lac" of initial bias between the groups. !ompare with 0esigns < and F. -s this design preferable to 0esign A because it handles the rival hypothesis of 7testing7? 4 4 @ O9 O:

C( 0!asi'e perimental ,esigns Huasi(experimental designs lac" control over exposure to @& i.e., when to expose, to whom, and ability to randomi'e group assignment. 3owever, they do have true experiment(li"e features regarding measurement& e.g., whom and when. 8( Time'series ,esign( -n this design, the apparent effects of @ are suggested by the discontinuity of measurements recorded in the time series. The design is a moral alternative (and probably ;ust as strong as) control(group designs in which sub;ects are denied a potentially(beneficial treatment. O9 O: O< OA @ OF OG OI OC

!ompare this design with 0esign :. 73istory7 is the ma;or wea"ness of this design, which might be handled using control by constancy or a different design (e.g., multiple time series). 9( E:!i"alent Time'samples ,esign( This is a form of time(series design with repeated introduction and removal of @ with the same group or individual. -t is sometimes called an 7intra(sub;ect7 or 7intra(group reversal (or replication)7 design. 5ote the use of concomitant variation to establish the effect of @. @O9 O: @O< OA @OF OG @OI

;( None:!i"alent Control -ro!p ,esign( The groups involved might constitute natural collectivities (e.g., classrooms), that are as similar as availability permits, but not so similar that one can dispense with a pre(test as a way to determine e$uivalendce. ,xposure to @ is assumed to be under the experimenter8s control. O9 O< @ O: OA

2<( Co!nter&alance# ,esign( This design enables you to assess the effects of numerous conditions (e.g., treatments((9, :, <, A) by exposing each group to the treatments, but in a different order. Jroup A @9O @:O @<O @AO Jroup + @:O @<O @AO @9O Jroup ! @<O @AO @9O @:O Jroup 0 @AO @9O @:O @<O A ma;or problem with this design is multiple treatment interference. That is, Jroup A might show the most change during treatment A, but this may

be because Jroup A was previously affected by treatments 9(<. This possibility is partly handled by comparing Jroup A (at condition A) with the other groups, where condition A happens earlier. ,ven so, it is a good idea to use random allocation. Otherwise, it could be argued that Jroup 0, for example, was so low during treatment A because Jroup 0 was somehow different from Jroup A. 22( M!ltiple Time'series ,esign( -n this design, each group is given several pre(tests and several post(tests, but the groups differ in their exposure to @. As with the counterbalanced design, the effects of several conditions can be tested. (,) (!) O O O O O O O O or (!) (,9) (,:) (,<) O O O O O O O O O O O O O O O O O @9O @:O @<O O @9O @:O @<O O @9O @:O @<O O @9O @:O @<O O @9O @:O @<O @ O O O O O O O O

,9(,< are different experimental conditions& e.g., different treatments.

The following table indicates how well each experimental design handles threats to validity. A 7K7 means that the design handles the threat. A 7(7 means that a design does not ade$uately handle the threat. A 7?7 means that the design8s handling a threat is indeterminate. And a blan" space means that the threat is irrelevant to the design.

How =ell Eac$ ,esign Han#les T$reats to Internal an# E ternal >ali#it% Threats to -nternal %alidity ,xternal %alidity 3 , x p t a o i t r y One(shot case study One group pre(post 2tatic group campar 6re(post exper M contr 2olomon a t n g u m s t t i l a m c l s n n r i r i c r a r a u e e l r t r @ r *T i u r s l e t u t t t r e s e t e Ms e f p p ao c t eM e t s s ag l a p r t l u mf m m vm a s t l t a e n t e e i x o ne a i i o o i e e e ne n * T 24 2+ ,* -2 !T 0 ! !4 0 4T -2 L

A group 6ost only contr group Time series ,$uival time samples 5one$uiv control group !ounter( bal *ulti time series