This presentation was prepared by Del Siegle.

Some of the material is from an earlier presentation by Scott Brown.

Experimental

DESIGNS

c. 2000 Del Siegle

RESEARCH

Experimental Research Designs have Two Purposes:

to provide answers to research questions ...to control variance (differences)
c. 2000 Del Siegle

The main function of the experimental research design is to control variance.

Principle: maximize systematic variance, control extraneous systematic variance, and minimize error variance.
In other words control variance.
c. 2000 Del Siegle

Therefore the researcher attempts to:

maximize the variance of the variable(s) of the research hypothesis (i.e., maximize the difference in the dependent variable [outcome] caused by maximizing the differences in the independent variable [treatment]). control the variance of extraneous or "unwanted" variables that may have an effect on the experimental outcomes, but which he/she is not interested (limit factors other than the treatment (IV) that could be causing differences in the outcome (DV) . minimize the error or random variance (i.e., avoid unreliable measurement instruments which have high errors of measurement ).

c. 2000 Del Siegle

Maximization of Experimental Variance

experimental variance
the variance due to the manipulated (i.e., treatment) or attribute (i.e., gender) variables (IV) research precept: design, plan and conduct research so that experimental conditions are as different as possible on the independent variable.

c. 2000 Del Siegle

Control of Extraneous Variables (EV)

eliminate the variable (i.e., if you are worried about gender, only include one gender in the study). randomization (i.e., if you randomly assign subjects to groups, the extraneous variable should be equally distributed among the groups) build it into the design make it a moderator variable (i.e., if you are worried about gender, build it into the analysis [2way ANOVA]well learn about this later) match subjects (i.e., match the characteristics of subjects and put one of each matched pair in each group) statistically equate groups (i.e., use ANCOVA [Analysis of Covariance] to analyze the data with the extraneous variable used as a covariatewell learn about this later)

c. 2000 Del Siegle

Minimizing Error Variance has Two Principle Aspects:

reduction of errors of measurement through controlled conditions (i.e., standardize testing procedures) increase in the reliability of measures (i.e., revise test instruments or find more reliable ones)
c. 2000 Del Siegle

c. 2000 Del Siegle

Experimental Designs Should be Developed to Ensure Internal and External Validity of the Study

Internal Validity:
Are the results of the study (DV) caused by the factors included in the study (IV) or are they caused by other factors (EV) which were not part of the study?

There are 16 common threats to internal validity.

c. 2000 Del Siegle

Threats to Internal Validity

Subject Characteristics
(Selection Bias/Differential Selection) -- The groups may have been different from the start. If you were testing instructional strategies to improve reading and one group enjoyed reading more than the other group, they may improve more in their reading because they enjoy it, rather than the instructional strategy you used.

c. 2000 Del Siegle

Threats to Internal Validity

Loss of Subjects
(Mortality) -- All of the high or low scoring subject may have dropped out or were missing from one of the groups. If we collected posttest data on a day when the honor society was on field trip at the treatment school, the mean for the treatment group would probably be much lower than it really should have been.
c. 2000 Del Siegle

Threats to Internal Validity

Location
Perhaps one group was at a disadvantage because of their location. The city may have been demolishing a building next to one of the schools in our study and there are constant distractions which interfere with our treatment.
c. 2000 Del Siegle

Threats to Internal Validity

The testing instruments may not be scores similarly. Perhaps the person grading the posttest is fatigued and pays less attention to the last set of papers reviewed. It may be that those papers are from one of our groups and will received different scores than the earlier group's papers

c. 2000 Del Siegle

Instrumentation Instrument Decay

Threats to Internal Validity

The subjects of one group may react differently to the data collector than the other group. A male interviewing males and females about their attitudes toward a type of math instruction may not receive the same responses from females as a female interviewing females would.

Data Collector Characteristics

c. 2000 Del Siegle

Threats to Internal Validity

The person collecting data my favors one group, or some characteristic some subject possess, over another. A principal who favors strict classroom management may rate students' attention under different teaching conditions with a bias toward one of the teaching conditions.

c. 2000 Del Siegle

Data Collector Bias

Threats to Internal Validity

Testing
The act of taking a pretest or posttest may influence the results of the experiment. Suppose we were conducting a unit to increase student sensitivity to prejudice. As a pretest we have the control and treatment groups watch Shindler's List and write a reaction essay. The pretest may have actually increased both groups' sensitivity and we find that our treatment groups didn't score any higher on a posttest given later than the control group did. If we hadn't given the pretest, we might have seen differences in the groups at the end of the study.

c. 2000 Del Siegle

Threats to Internal Validity

History
Something may happen at one site during our study that influences the results. Perhaps a classmate dies in a car accident at the control site for a study teaching children bike safety. The control group may actually demonstrate more concern about bike safety than the treatment group.

c. 2000 Del Siegle

Threats to Internal Validity

There may be natural changes in the subjects that can account for the changes found in a study. A critical thinking unit may appear more effective if it taught during a time when children are developing abstract reasoning.
c. 2000 Del Siegle

Maturation

Threats to Internal Validity

Hawthorne Effect
The subjects may respond differently just because they are being studied. The name comes from a classic study in which researchers were studying the effect of lighting on worker productivity. As the intensity of the factory lights increased, so did the worker productivity. One researcher suggested that they reverse the treatment and lower the lights. The productivity of the workers continued to increase. It appears that being observed by the researchers was increasing productivity, not the intensity of the lights.

c. 2000 Del Siegle

Threats to Internal Validity

One group may view that it is in competition with the other group and may work harder than they would under normal circumstances. This generally is applied to the control group "taking on" the treatment group. The terms refers to the classic story of John Henry laying railroad track.

c. 2000 Del Siegle

John Henry Effect

Threats to Internal Validity

The control group may become discouraged because it is not receiving the special attention that is given to the treatment group. They may perform lower than usual because of this.

c. 2000 Del Siegle

Resentful Demoralization of the Control Group

Threats to Internal Validity

Regression
(Statistical Regression) -- A class that scores particularly low can be expected to score slightly higher just by chance. Likewise, a class that scores particularly high, will have a tendency to score slightly lower by chance. The change in these scores may have nothing to do with the treatment.

c. 2000 Del Siegle

Threats to Internal Validity

The treatment may not be implemented as intended. A study where teachers are asked to use student modeling techniques may not show positive results, not because modeling techniques don't work, but because the teacher didn't implement them or didn't implement them as they were designed.

c. 2000 Del Siegle

Implementation

Threats to Internal Validity

Someone may feel sorry for the control group because they are not receiving much attention and give them special treatment. For example, a researcher could be studying the effect of laptop computers on students' attitudes toward math. The teacher feels sorry for the class that doesn't have computers and sponsors a popcorn party during math class. The control group begins to develop a more positive attitude about mathematics.
c. 2000 Del Siegle

Compensatory Equalization of Treatment

Threats to Internal Validity

Experimental Treatment Diffusion

Sometimes the control group actually implements the treatment. If two different techniques are being tested in two different third grades in the same building, the teachers may share what they are doing. Unconsciously, the control may use of the techniques she or he learned from the treatment teacher.

c. 2000 Del Siegle

c. 2000 Del Siegle

Once the researchers are confident that the outcome (dependent variable) of the experiment they are designing is the result of their treatment (independent variable) [internal validity], they determine for which people or situations the results of their study apply [external validity].

External Validity:
Are the results of the study generalizable to other populations and settings?

External validity comes in two forms: population and ecological.

c. 2000 Del Siegle

Threats to External Validity (Population) Population Validity is the extent to which the results of a study can be generalized from the specific sample that was studied to a larger group of subjects. It involves... 1. ...the extent to which one can generalize from the study sample to a defined population-If the sample is drawn from an accessible population, rather than the target population, generalizing the research results from the accessible population to the target population is risky.

2. ...the extent to which personological variables interact with treatment effects-If the study is an experiment, it may be possible that different results might be found with students at different grades (a personological variable).

c. 2000 Del Siegle

Threats to External Validity (Ecological)

Ecological Validity

is the extent to which the results of an experiment can be generalized from the set of environmental conditions created by the researcher to other environmental conditions (settings and conditions).

c. 2000 Del Siegle

There are 10 common threats to external validity.

Threats to External Validity (Ecological)

Explicit description of the experimental treatment

c. 2000 Del Siegle

(not sufficiently described for others to replicate) If the researcher fails to adequately describe how he or she conducted a study, it is difficult to determine whether the results are applicable to other settings.

Threats to External Validity (Ecological)

Multiple-treatment interference
(catalyst effect) If a researcher were to apply several treatments, it is difficult to determine how well each of the treatments would work individually. It might be that only the combination of the treatments is effective.
c. 2000 Del Siegle

Threats to External Validity (Ecological)

Hawthorne effect
(attention causes differences) Subjects perform differently because they know they are being studied. "...External validity of the experiment is jeopardized because the findings might not generalize to a situation in which researchers or others who were involved in the research are not present" (Gall, Borg, & Gall, 1996, p. 475)

c. 2000 Del Siegle

Threats to External Validity (Ecological)

Novelty and disruption effect

(anything different makes a difference) A treatment may work because it is novel and the subjects respond to the uniqueness, rather than the actual treatment. The opposite may also occur, the treatment may not work because it is unique, but given time for the subjects to adjust to it, it might have worked.
c. 2000 Del Siegle

Threats to External Validity (Ecological)

Experimenter effect
(it only works with this experimenter) The treatment might have worked because of the person implementing it. Given a different person, the treatment might not work at all.

c. 2000 Del Siegle

Threats to External Validity (Ecological)

Pretest sensitization
(pretest sets the stage) A treatment might only work if a pretest is given. Because they have taken a pretest, the subjects may be more sensitive to the treatment. Had they not taken a pretest, the treatment would not have worked.
c. 2000 Del Siegle

Threats to External Validity (Ecological)

Posttest sensitization
(posttest helps treatment "fall into place") The posttest can become a learning experience. "For example, the posttest might cause certain ideas presented during the treatment to 'fall into place' " (p. 477). If the subjects had not taken a posttest, the treatment would not have worked.
c. 2000 Del Siegle

Threats to External Validity (Ecological)

Interaction of history and treatment effect

(...to everything there is a time...) Not only should researchers be cautious about generalizing to other population, caution should be taken to generalize to a different time period. As time passes, the conditions under which treatments work change.

c. 2000 Del Siegle

Threats to External Validity (Ecological)

Measurement of the dependent variable

(maybe only works with M/C tests) A treatment may only be evident with certain types of measurements. A teaching method may produce superior results when its effectiveness is tested with an essay test, but show no differences when the effectiveness is measured with a multiple choice test.

c. 2000 Del Siegle

Threats to External Validity (Ecological)

Interaction of time of measurement and treatment effect

c. 2000 Del Siegle

(it takes a while for the treatment to kick in) It may be that the treatment effect does not occur until several weeks after the end of the treatment. In this situation, a posttest at the end of the treatment would show no impact, but a posttest a month later might show an impact.

First, and foremost, an experiment must have internal validity. If the researchers cannot be certain that the results of the experiment are dependent on the treatment, it does not matter to which people or situations they wish to generalize (apply) their findings. The importance of external validity is reliant on having internal validity in much the same way that the validity of a measurement instrument is reliant on the instrument being reliable.
c. 2000 Del Siegle

However, the more tightly experimenters design their study, the more they limit the populations and settings to whom they can generalize their findings.

c. 2000 Del Siegle

Suppose a researcher wants to study the effect of a reading program on reading achievement. She might implement the reading program with a group of students at the beginning of the school year and measure their achievement at the end of the year.

XO
c. 2000 Del Siegle

This simple design is known as a one-shot case study design.

Unfortunately, the students end of year reading scores could be influenced by other instruction in school, the students maturation, or the treatment. We also do not know whether the students reading skills actually changed from the start to end of the school year. We could improve on this design by giving a pretest at the start of the study.

OXO

This is known as a one-group pretest-posttest design.

c. 2000 Del Siegle

Unfortunately, the students end of year reading scores still could be influenced by other instruction in school, the students maturation, or the treatment. Our researcher may wish to have a comparison group.

OXO O O
This is a static-group pretest-posttest design.
c. 2000 Del Siegle

If our researcher believes that the pretest has an impact on the results of the study, she might not include it.

OXO O O
This is a static-group comparison design.
c. 2000 Del Siegle

Because our researcher did not pretest, she might wish to randomly assign subjects to treatment and control group.
Random assignment of subject to groups should spread the variety of extraneous characteristics that subjects possess equally across both groups.

ROXO RO O
This is a randomized posttest-only, control group design.
c. 2000 Del Siegle

Of course, our researcher could also include a pretest with her random assignment.

ROXO RO O
This is a randomized pretest-posttest control group design.
c. 2000 Del Siegle

Occasionally researchers combine the randomized pretest-posttest control group design with the randomized posttest-only, control group design.

ROXO RO O R XO R O
This is a randomized Solomon four-group design.
c. 2000 Del Siegle

With the randomized Solomon four-group design, all groups are randomly assigned and given the posttest. Two of the groups are given pretests.

ROXO RO O R XO R O

One of the pretest groups is assigned to treatment and one of the non-pretest groups is assigned to treatment.

c. 2000 Del Siegle

Each of the designs described in this section has advantages and disadvantages that influence the studies internal and external validity.
This presentation was prepared by Del Siegle. Some of the material is from an earlier presentation by Scott Brown.
c. 2000 Del Siegle