Research PPT

Statistical Analyses And Threats

to Validity.
Background
 Recall from last lab that there are two major types of
designs.
 If you are interested in the association between continuous
variables, you should use a correlational design.
 If you are interested in comparing two (or more) groups, it
is likely you will use an experimental design.
 The design you use will determine:
 statistical analysis you will perform
 threats to validity you will most likely need to address
 conclusions you can make from your research.
Correlational/Observational
Design
 In this design, you are not comparing groups.
Rather, you are examining the association between
two continuous variables.
 You can use either correlation or regression
analysis to analyze your data.
Correlation
 Used when interested in assessing the strength
of association between two variables

 Used with continuous variables

 Example research hypothesis: are higher levels

of self-esteem associated with lower levels of
depression (what direction correlation would be
expected?)
Regression
Simple Linear Regression is a statistical method
concerned with the amount of variance that one
variable accounts for in another variable (that is,
the amount of shared variance between variables).
You also want to use regression when you are
predicting one variable from the other.
Used when you have continuous variables (e.g.,
motivation measured on a Likert scale, body
weight).
Regression

Shared variance

Variable X Variable Y

http://www.math.csusb.edu/faculty/stanton/m262/regress/regress.html
Correlation vs. Regression
 Both correlation and regression are concerned with the
amount of variance that variables share

 The major difference is that with correlation we are

only interested in an association between variables, but
with regression we assign one variable to be the
independent variable (x; the predictor) and the other to
be the dependent variable (y; the criterion)

 Note: While we typically are concerned with

predictive relationships when using regression, the
certainty with which we can say that two variables are
causally related depends upon the design (true
experiments are best!...why?) and quality of the study
Comparing Groups
 If you have an experimental design (manipulation
of IV), you should use the following analyses to
compare groups (non-continuous variables).
 In some cases, you will want to compare groups
even when you don’t manipulate anything.
 Comparing male versus female on performance.
 Examining differences in opinion of various political
or religious groups.
Independent samples (between-
groups) t-test
 Used to determine dependent variable mean differences
between two different groups

 Used with categorical or dichotomous variables (e.g.,

gender, democrat or republican, yes/no responses)

 Only one independent variable with only two

levels/groups

 Example research hypothesis: do women have higher

levels of depression on average than do men?
Analysis of Variance (ANOVA)
 Like t-test, used to determine dependent variable
mean differences between groups/levels, but can
use when there are more than two groups (which
is the same as more than two levels of your IV)

 Remember, though, you’re still only using one IV,

just there are more than two groups/levels (e.g.,
you might have the IV of political party, with the
groups Democrat, Republican, and Independent)
Analysis of Variance (ANOVA)
 Can use for your final project if you have just one IV
with more than two groups (i.e., one-way ANOVA), but
can be used for more than one IV (i.e. two-way ANOVA,
where say you have political party [3 groups] that may
differ by gender [2 groups] for a 2x3 two-way ANOVA)

 Independent variable(s) have two or more levels; if you

only have one IV with two levels, though, use a t-test

 Example research question: Do psychology graduate

students have higher IQs than medical students, physics
graduate students, and English graduate students?
Repeated measures (within-groups) t-
test or ANOVA
 Used to test mean differences in the DV for the group
measured at two or more times (e.g., the same group of
clients is exposed to every type of counseling [the IV]
and symptoms [the DV] are measured)

 Remember, the t-test involves an independent variable

with two levels; ANOVA involves independent
variable(s) with more than two levels.

 Population parameters and sample statistics the same as

between-group t-test and ANOVA above.
Repeated measures (within-groups) t-
test or ANOVA
 Example research hypothesis for repeated-measures t-
test: Do people like the taste of Pepsi more than Coke
(same people would taste both types of pop)?

 Example research hypothesis for repeated-measures

ANOVA: Does exposure to death metal music lead to
better cognitive performance than exposure to jazz,
classical, or country music (same people would listen to
each type of music [IV] and perform a cognitive task
[DV])?
THREATS TO VALIDITY
Threats to Internal & External Validity

 Keep in mind that in science it is always important

to balance multiple concerns.
 Ethics: Harm versus Benefit
 Sampling: Representativeness versus Practicality
 When choosing a research design, it is important to
address whether:
 You can address or eliminate alternative explanations
for your results (Internal Validity).
 You can generalize your results (External Validity).
Research Designs & Validity
 In general, experiments with random assignment
are high on internal validity.
 Why? (What does random assignment do?)
 Correlational studies are often higher than
experiments on external validity but lower on
internal validity.
 Ideally, you would want studies to both be high
internal and external validities.
 No single method does this.
 Multiple methods important.
An example
 You are interested in studying why individuals help
each other. You believe that sadness leads to
helping because it makes individuals feel better.
 Let’s say you conduct your study on September 12,
2001. How do you think this would influence your
results?
 To the extent that other explanations (outside of
sadness) can account for the associations that you
find, your internal validity is threatened.
Internal Validity Threats
 Given that the IV is related to the DV, is it plausibly causal from
one operational variable to another?
 History – is there something about the experience & history of one group
that could lead to differences?
 Maturation – would natural development influence the associations?
 Testing – does actual participation in the study change behavior or
responses?
 Instrumentation – what happens if the instruments you use change during
the course of a study or if different groups of participants interpret your
questions differently?
 Regression towards the mean – if you start with extreme groups, they are
likely to be less extreme at another time point.
 Selection – are participants in your study somehow different than those who
are not?
 Mortality – those who drop out of your study could be qualitatively
different than those who remain.
Internal Validity Threats
 Given that the IV is related to the DV, is it plausibly
causal from one operational variable to another?
 Diffusion or imitation – what would happen if participants in
the different conditions communicated?
 Interaction with Selection – what if any of the above happen
differentially for one group?
 Direction of causality – can the association happen in the
opposite direction than what you predicted?
 Compensatory equalization of treatments – what if
experimenters know what conditions participants are in?
 Compensatory rivalry – what would happen if participants in
the different conditions started competing?
 Resentful demoralization – is it possible that participants in one
condition give up?
External Validity Threats
 Given that there is probably a causal relationship
from construct A to construct B, how generalizable
is this relationship across persons, settings, and
times?
 Interaction of selection and treatment (sample to
population)
 Interaction of setting and treatment
 Interaction of history and treatment (time)
For your paper
 You will need to discuss the internal and external
threats to validity in the discussion of your paper.
 Keep in mind that no study is perfect.
 However, when you bring up threats to validity,
you MUST explain:
 How it is a threat to validity
 How your results would change if the threat were
addressed
 What you could do to address the threat
 In other words, DO NOT simply state the threat.
Homework 9
 For each of the three research hypotheses described below, list what type of
statistical test you would use to test it (independent (between-groups) t-test,
between-groups ANOVA, repeated measures t-test, repeated measures
ANOVA, correlation, regression), and whether external or internal validity
is most threatened and why (6 points; 2 for each hypothesis).

 Research Hypotheses:
 Women who experience a great deal of stress during pregnancy are

more likely to give birth to low birth weight or premature babies than
pregnant women who do not experience a great deal of stress.

 The degree to which someone believes in God relates to his/her

psychological well-being. Further, the more someone believes in God
the higher his/her well-being will be.
Homework 9
 Research Hypotheses:
 A new manualized (standardized) treatment for depression will

decrease depressive symptoms in a group of people seeking help at a

mental health clinic. They will be assessed when they come into the
clinic, immediately after the treatment is completed, and 6 months
after treatment is complete.

 Lastly, describe your hypothesis, what design you will use for your
project (from previous homework), what statistical analysis you plan to
conduct, and identify 2 threats to validity (any combination of external
or internal) and how they could influence the interpretation of your
results. (4 points)