Types of validity

Internal Validity: How sure are we that the cause leads to the expected results? In other words, is it
appropriate for us to infer that the relationship between variables is causal
External Validity: How sure are we that we can generalize the finding of causation to other populations,
settings, or variables?
Construct Validity: How sure are we that the variables we are using actually measure the concept (i.e., the
construct) that we are seeking to measure?
Statistical Conclusion Validity: Do the statistical tests that we perform accurately measure the relationships
between the variables under study?
7 Threats to Internal Validity (Campbell & Stanley)
History: events that occur between the first and second measurement that are unrelated to the experiment but
that could affect the results.
Maturation: Changes in the participants that occur as a function of the passage of time and not specific to the
experiment.
Testing: The effects of taking a first test on the scores of a second test.Instrumentation: Changes in the
measurement instrument or changes or the observers make changes in the obtained measurements.
Statistical regression (toward to mean): Groups having extreme scores on the pretest (or selected on the
basis of extreme scores) will tend to have scores closer to the mean on the posttest.
Selection: Biases resulting in differentials selection of respondents for the comparison groups.
Experimental mortality: Differential loss of respondents from the comparison groups
.Selection-maturation interaction, other interaction effects:
8 Threats to External Validity
Reactive or interaction effects of testing: The pretest itself might be a learning experience such that by
taking the pretest students gain information that will affect posttest results
Interaction of selection and the experimental variable: Different groups may respond differently to the
experimental variable.
Reactive effects of experimental arrangements: Subjects respond differently because they know they are in
an experiment (i.e., the Hawthorne effect)
Multiple treatment interference: Multiple treatments applied to the same respondents; the effects of prior
treatments cannot be erased.
10 Before we get started… Some definitions:
Construct: An idea or concept that you are attempting to measureLatent Construct: A construct that cannot be
measured directly (e.g., group cohesion)
Independent Variable: Variables that are presumed to be the cause of an effect being studied; independent
variables are manipulated to examine their impact on results
Dependent Variables: Variables that are observed to understand the result of causation.
Hypothesis: A statement of a possible explanation for causation. An hypothesis is tested by drawing
conclusions from an experimental examination of the variables that are expected to be related

11 Types of Experimental Designs

Pre-experimental designs: One group designs and designs that compare pre-existing groups
Quasi-experimental designs: Experiments that have treatments, outcome measures, and experimental
conditions but that do not use random selection and assignment to treatment conditions.
True experimental designs: Experiments that have treatments, outcome measures, and experimental
conditions and use random selection and assignment to treatment conditions. This is the strongest set of
designs in terms of internal and external validity.
12 Pre-Experimental Designs
Design 1: One-Shot Case Study: A single group is studied once after some intervention/treatment that is
presumed to cause change.
For example, a training program is implemented and participants are given a posttest at the conclusion
of the training.X O
13 Pre-Experimental Designs
Design 2: One-Group Pretest-Posttest Design: One group, not randomly selected nor randomly assigned, is
given a pretest, followed by a treatment/intervention, and finally a posttest. There is no comparison group.
Generally done with intact groups.
 For example, a classroom teacher gives her students a pretest then implements an instructional
strategy followed by a posttest.O1 X O2
14 Pre-Experimental Designs
Design 3: The Static-Group Comparison: One group which has experienced a treatment/intervention (X) is
compared to another group that has not had the intervention. The groups are not randomly selected nor
randomly assigned and are generally pre-existing groups. There is no pre-observation/pretest
.For example, comparison of GRE scores for students who attended a rural high school versus those
who attended an urban high school.X1 OX2 O
15 True Experimental Designs
Design 4: Pretest-Posttest Control Group Design: One group is administered a treatment while the other is
not; all groups are observed before and after the treatment is administered.
For example, 50 freshman students are randomly selected to participate in a tutoring study. Half are
randomly assigned to a tutor for their first semester and half are not. All students are given a pretest at the
beginning of the term and a posttest at the end of the term.R O1 X O2R O1 O2
16 True Experimental Designs
Design 5: Solomon Four-Group Design: This design involves four experimental groups. Two of the groups
parallel the structure of Design 4 while the remaining two groups include no pre-test (so that the effects of the
pretest can be evaluated).
For example, 100 freshman students are randomly selected to participate in a tutoring study. 25 are
randomly assigned to a tutor for their first semester and given a pretest. 25 are randomly assigned to a group
where no tutor is assigned and they are given a pretest. Another 25 are randomly assigned to a tutor but not
given a pretest. The remaining 25 are randomly assigned to a group where no tutor is assigned and they are
not given a pretest. Whew!R O1 X O2R O1 O2R X O2R O2
17 True Experimental Designs
Design 6: Posttest Only Control Group Design: One group is administered a treatment while the other is
not; all groups are observed after the treatment is administered BUT not before the treatment.
For example, students are randomly assigned to two groups of 50 each. The experimental (treatment)
group receives a new teaching method during a special class session. The second group (the control) receives
a traditional teaching method during a special class session. No pretest is used for each group. Issues such as
existing grades, SAT scores, and other factors are examined as covariates. R X O2R O2
18 Quasi-Experimental Designs
Design 7: The Time-Series Experiment: This design involves periodic measurements of some group or
individuals and the introduction of a change into the conditions during the series.
For example, studying a group of workers over time and taking several measures of productivity during
this period. At some point a new work process is introduced and measures of productivity are taken over
several weeks following the intervention.O1 O2 O3 X O4 O5 O6
19 Quasi-Experimental Designs
Design 8: Equivalent Time-Samples Designs: This design involves periodic introduction of treatments
followed by measurements with the treatments varied consistently over time.
For example, to study the effect on student discussions of having an observer appear in a classroom.
At time period one, an observer is present and a measure of discussion level is made. At time two, no observer
is present and a measure of discussion level is made. At time three an observer is present, a measure is
taken. At time four an observer is not present, a measure is taken. Etc.X1 O X2 O X1 O X2 O
20 Quasi-Experimental Designs
Design 9: The Equivalent Materials Design: This design involves giving equivalent samples of materials to
subjects, imparting interventions, and then making observations.
For example, subjects are asked to complete a survey instrument about their opinions related to current
events. The students are then split into two groups and given two different sets of (falsified) survey results
indicating how other students answered the survey. Both groups are then asked to complete the survey again
to observe how they respond.Experimental Materials A(O) X0 OExperimental Materials B(O) X0 O
21 Quasi-Experimental Designs
Design 10: Nonequivalent Control Group: This design involves an experimental and control group with both
given pretests and posttest; however, these groups are not randomly selected because they constitute
naturally assembled groups (e.g. classrooms). The assignment of X (the treatment) to one group or the other is
randomly selected by the researcher.
 For example, four sections of a course are chosen to participate in a study of teaching methods. Half
are randomly assigned a new teaching method and half are not. All are given pretests at the beginning of the
term and all are given posttests at the end of the semester.O X OO O
22 Quasi-Experimental Designs
Design 11: Counterbalanced Designs: In this design all subjects receive all treatments but in a different
order. Each treatment occurs once at each time period and once for each treatment group. A Latin-square
design is a type of counterbalanced design in which four treatments are applied to four naturally assembled
pools of subjects.
For example, consider a study of the effect of different training methods on learning. Subjects are
placed into four groups (A,B,C, D) for different training methods, X1-X4.Group A X1O X2O X3O X4OGroup B
X2O X4O X1O X3OGroup C X3O X1O X4O X2OGroup D X4O X3O X2O X1O
23 Quasi-Experimental Designs
Design 12: The Separate Sample Pretest-Posttest Design: Often used with large populations (i.e., in public
opinion studies) where the researcher cannot randomize or segregate subgroups for different treatments. Two
equivalent groups are identified, one sample is measured prior to the treatment and a different (but equivalent)
sample is measured after the treatment. This design is also called the "simulated before and after" design.
For example, 100 community members are randomly surveyed concerning their opinions about local
government policies. A PR campaign is then conducted for six weeks. A follow-up survey is then conducted
with 100 different residents who are randomly selected.R O XR X O
24 Quasi-Experimental Designs
Design 13: The Separate Sample Pretest-Posttest Control Group Design: This design is similar to Design
12; however, a control group is added to the design.
For example, consider the PR campaign described in Design In this case, the same design is used, but,
in addition, the measurements are made in a similar nearby city where no PR campaign is run.R O XR X OR
OR O
25 Quasi-Experimental Designs
Design 15: Recurrent Institutional Cycle Design (A "Patched-Up" Design): This is an approach used in
field research. A researcher begins with an inadequate design and then adds features to control for one or
more sources of invalidity. The result is an "inelegant accumulation of precautionary checks." The researcher is
aware of rival interpretations (sources of internal invalidity) and incrementally identifies other data that would
rule out rivals. The design exploits contextual features to refine the research as it progresses.
For example, this design would combine a longitudinal and cross sectional structure. One group will be
exposed to X and measured at the same time as a second group that is just about to be exposed to X. A
comparison of the two groups would be able to be made because it is equivalent to a static group comparison.
The second group would be remeasured (posttest), which would make the design comparable to the one group
pretest-posttest design.Group A X O1Group B O1 X O2