Experimental, QuasiExperimental, and Ex Post Facto (Causal-Comparative) Research

Characteristics of Experimental Research

There is a control or comparison group Subjects are randomly assigned to groups The treatment is randomly assigned to groups.

Characteristics of QuasiExperimental Research

There is a control or comparison group Intact groups are used The treatment is randomly assigned to groups.

Characteristics of Ex Post Facto Research

There is a control or comparison group Intact groups are used The treatment is not manipulated, it has already occurred.

Diagramming Research
To illustrate research designs, a number of symbols are used
X1 = Treatment X2 = Control Group O = Observation (pretest or posttest) R = Random Assignment

A Sample Research Design

Single-Group Pretest-TreatmentPosttest Design
R O X1 O
This means subjects are randomly assigned to a group, which is then given a pretest, then there is a treatment, then there is a posttest.

R O X1 O
This is not really an experimental design because there is no control group
It is often referred to as a preexperimental design

Novice researchers often use this research design There are some major problems with this design did the treatment really make the difference or was something else happening.

R O X1 O
What are the threats to the Internal Validity of this type of research (Did the treatment really cause a difference?)

R O X1 O Internal Validity Threats

History
Another event occurs during the time of the experiment that might cause the difference
An experiment to heighten racial awareness was conducted by a researcher during February. This is Black History month; so the results might be affected by events that occur during Black History month and not the treatment.

R O X1 O Internal Validity Threats

Maturation
People naturally change and evolve over time. This may cause the difference.
A college develops a new housing plan to promote more open-mindness and acceptance of others. The students are tested when they enter college and when they graduate. The results show they are now more open-minded and tolerant of others. Did the housing plan work or do students just mature and grow as a result of the college experience.

R O X1 O Internal Validity Threats

Mortality
Some people drop out during an experiment. This may affect the outcome.
I am teaching a new experimental seminar on study skills. About half of the class stopped coming to the seminar before the semester was over. The students who remained improved their study skills. So my course was effective!
Probably not. The half that stopped coming might not have gained anything; that is why they stopped attending.

R O X1 O Internal Validity Threats

Testing
Whenever you give a pretest, the students may remember the test questions, and get them correct on the posttest.
I gave a test to my study skills group on Monday, presented some unique concepts on Tuesday, then gave them the posttest on Wednesday. The grades were significantly higher on the posttest.
It is possible the grades were higher because the students still remembered the questions from the pretest.

R O X1 O Internal Validity Threats

Instrumentation
To overcome the testing threat to internal validity, a researcher develops a different form of the test instrument, but it is not really equivalent.
I gave a test to my study skills group on Monday, presented some unique concepts on Tuesday, then gave them an alternative form of the pretest on Wednesday. The grades were significantly higher on the posttest.
It is possible the grades were higher because the second test was easier than the first.

O X1 O
Internal Validity Threats
Regression
When subjects are selected because of extreme scores on some type of instrument, there is tendency for their scores to move more toward the average on subsequent tests.
An experimenter selected students for a reading program based on their low test scores. At the end of the treatment, the test scores had improved.
Extreme scores naturally move toward the mean on subsequent tests.

How to Handle Internal Validity Threats

Have a control group and use randomization.This design is the TwoGroup Pretest-Treatment-Posttest Design. The Control Group would experience the same

R O X1 O R O X2 O

history and maturation. Mortality should be the same because of random assignment. Random assignment eliminates the selection threat. However testing and instrumentation could still be a threat.

Other Research Designs

Two-Group Treatment-Posttest-Only Design

R X1 O R X2 O

There is no pretest so this eliminates the testing and instrumentation threat to internal validly but you dont know about their knowledge or attitude coming into the study.

Other Research Designs

Solomon 4-Group Design

R O X1 O R X1 O R O O R O

Note: A blank indicates the control group, same as X2

Quasi-Experimental Designs
Posttest Only Nonequivalent Group Design The absence of R indicates

X1 O X2 O

there is no random assignment. Sometimes you will see a dotted line between the two groups. This indicates the two groups may not be equivalent.

Quasi-Experimental Designs
Pretest-Posttest Nonequivalent Group Design

O X1 O O X2 O

Time Series Designs O O O X1 O O O

In the next course, AEE 579 Research Design, many more research designs are examined.

External Validity
Can the research be generalized to other settings?
Population Validity Personological Variables Ecological Validity

Population Validity
Is the sample population similar to the population the researchers wishes to generalize to

Personological Variables
Different people have different personalities, learning styles, etc., so the results may not be generalizable to people who are substantially different on these personological variables.

Ecological Validity
The setting or situation in which the experiment occurred may be different than other settings.

Social Interaction Validity Threats

Diffusion or Imitation of Treatment
This occurs when a comparison group learns about the program either directly or indirectly from program group participants.
This group may try to imitate or emulate what the treatment group is getting.

Social Interaction Validity Threats

Compensatory Rivalry
The comparison group knows what the program group is getting and develops a competitive attitude with them.

Social Interaction Validity Threats

Resentful Demoralization
This is almost the opposite of compensatory rivalry. Here, students in the comparison group know what the program group is getting. But here, instead of developing a rivalry, they get discouraged or angry and they give up.

Social Interaction Validity Threats

Compensatory Equalization of Treatment
The researcher is under pressure to enrich the experiences of the control group. This pressure may come from parents, school administrators, etc.

Ex Post Facto (CausalComparative) Research

Explores possible causes and effects The independent variable is not manipulated, it has already been applied Focuses first on the effect, then attempts to determine what caused the observed effect.

Statistical Analysis
If we are comparing the scores of two groups a t-test is normally used. The value of t means nothing by itself (unlike the value of R). We have to determine if t is statistically significant

Tea for two

Statistical Analysis
If we are comparing the scores of three (or more) groups Analysis of Variance (ANVOA) is used. This test gives us a f value which means nothing by itself. We have to determine if it is statistically significant.

Statistical Analysis
If we want to statistically equate two or more groups (because one group had a high pretest score) we use Analysis of Covariance (ANCOVA). This test gives us a f value which means nothing by itself. We have to determine if it is statistically significant.