Some Threats to Internal Validity- Adapted from Campbell & Stanley (1963)

Drop-out. More of one type of person may drop out of one of the groups. For example,
those less committed, less achievement-oriented, less intelligent.

History (aka Confounding Variables). Events that happen to participants during the
research which affect results but are not linked to the ‘treatment’ In an extended study
comparing relaxation to no relaxation on headache occurrence, those in the no relaxation
condition sought out other means of reducing their headache occurrence (e.g. took more
pills).

Maturation: the growth or improvement of people being studied over time, regardless of
any treatment they are receiving.

Multiple tests of significance. The more significance tests you conduct on the one set of
data, the more likely you are to claim that you made a significant finding when you
should not have. You will be capitalising on chance fluctuations

Order effects. If we measure something over a series of trials, we might find that a
change occurs because our participants are becoming bored, tired, disinterested, fatigued,
less motivated than they were at the beginning of the series. "Counterbalancing" is a way
of overcoming this problem in repeated measures designs.

Reliability of measures and procedures. Unreliable operationalisation (definition and

understanding) of constructs (variables), or inconsistency in giving instructions to
participants, or training to assessors can invalidate the study

Regression to the Mean: The tendency for performance scores to creep towards the
mean over repeated measures. For example, a tendency for low achieving students to
improve test scores (to nudge upwards towards the mean) and for high achieving students
to drop (slip downwards towards the mean.) For a good explanation and a sample
animated illustration, go to the Rice Virtual Statistics ab page on this at
http://www.ruf.rice.edu/%7Elane/stat_sim/reg_to_mean/index.html

Selection bias. Occurs when more of one type of person gets into one group for a study.
For example, the people who return your questionnaire may be different, in some
important way, to the people who did not return your questionnaire. The students who
volunteer for your project might be different to the ones who do not volunteer (for
example, more altruistic, more achievement oriented, more intelligent). Do these
variables have an effect on the thing you are trying to measure? We usually do not know.

Using a design of low power. In particular, a small sample size may have insufficient
power to detect a real effect even if it is there. As a result, the researcher claims the
manipulation had no effect when in fact it does; he just could not pick it up. As well,
different statistical tests have varying sensitivity to detect differences.
 Reliability and Validity

“ For a research study to be accurate, its findings must be reliable and valid. Reliability
means that the findings would be consistently the same if the study were done over again.
It sounds easy, but think of a typical exam in college; if you scored a 74 on that exam,
don't you think you would score differently if you took if over again? Validity refers to
the truthfulness or findings; if you really measured what you think you measured, or
more precisely, what others think you measured. Again, think of a typical multiple choice
exam in college; does it really measure proficiency over the subject matter, or is it really
measuring IQ, age, test-taking skill, or study habits?

A study can be reliable but not valid, but it cannot be valid without first being reliable.
You cannot assume validity no matter how reliable your measurements are. There are
many different threats to validity as well as reliability, but an important early
consideration is to ensure you have internal validity. This means that you are using the
most appropriate research design for what you're studying (experimental, quasi-
experimental, survey, qualitative, or historical), and it also means that you have screened
out spurious variables as well as thought out the possible contamination of other variables
creeping into your study. Anything you do to standardize or clarify your measurement
instrument to reduce user error will add to your reliability.

It's also important early on to consider the time frame that is appropriate for what
you're studying. Some social and psychological phenomena (most notably those
involving behavior or action) lend themselves to a snapshot in time. If so, your research
need only be carried out for a short period of time, perhaps a few weeks or a couple of
months. In such a case, your time frame is referred to as cross-sectional. Sometimes,
cross-sectional research is criticized as being unable to determine cause and effect, and a
longer time frame is called for, one that is called longitudinal, which may add years onto
carrying out your research. There are many different types of longitudinal research, such
as those that involve tracking a cohort of subjects (such as schoolchildren across grade
levels), or those that involve time-series (such as tracking a third world nation's economic
development over four years or so). The general rule is to use longitudinal research the
greater the number of variables you've got operating in your study and the more confident
you want to be about cause and effect”