Can Results Generalize to Other Populations?

● If we wish to generalize a set of results to all of humanity, the best—but completely impractical
—method for doing so would be to randomly select a sample from the earth’s entire population.
In contrast, most psychological research relies on convenience samples, people selected simply
because they are available and willing
Beyond University Students
● undergraduate students tend to be young and to possess the characteristics of late adolescence: a
developing sense of self-identity, social and political attitudes that are in a state of flux, a high
need for peer approval, and unstable peer relationships. They are also intelligent, have good
cognitive skills, and know how to win approval from authority
● Researchers at the University of British Columbia have described this group as WEIRD: Western,
Educated, Industrialized . As a result of being WEIRD, the conclusions drawn from these
participants seem unlikely to generalize to all of humanity.
Beyond Volunteers
● At many universities, introductory psychology students are often required to volunteer for
experiments for course credit. Recruitment for these populations might involve asking parents
visiting a science center to participate in a study on parenting, or asking users of a particular
Internet forum to complete a survey online.Volunteers tend to be more highly educated, more in
need of approval, and more social
● Another factor that can influence who signs up for a study is whether some financial
compensation is offered. Studies that recruit participants by emphasizing financial rewards tend
to attract less-altruistic participants than studies emphasizing the potential learning
opportunity.Similarly, studies offering course credit for participating tend to attract less motivated
participants than studies that do not offer course credit , although offering either money or course
credit boosts volunteer rates more than offering nothing.
Beyond the Gender of Participants
● Sometimes, researchers use either mostly females or males, or only one gender, simply because
this is convenient or the procedures seem better suited to one gender. Similarly, individuals who
identify as a gender minority might be excluded from analysis
● In psychological research with humans, gender could influence results and subsequently the
conclusions drawn from these results. Denmark and colleagues have identified several ways that
gender bias may arise throughout the research process. If a sample is solely or predominantly one
gender, it is best to avoid concluding that a widely generalizable truth has been found.
● It is also possible, when a sample includes a good balance of genders, to include gender as a
variable in the analyses to investigate if the results generalize across the genders studied.
Beyond Culture
● Today, however, many samples of university students are ethnically diverse because the
population of university students across North America has become increasingly diverse. In
addition, more and more psychological research is being done in countries around the world. As a
result, the overall external validity of research has improved. It is also now much easier to
compare ethnic groups, to examine cross-cultural differences and similarities
● So far, much of cultural research has centered on identifying cross-cultural similarities and
differences in responses to the same environments, along with personality and other
characteristics
 ● Once again, students with a collectivist cultural background reported more parental involvement
in their choice of romantic partner than students with an individualist cultural background
(Cohen’s d = 1.42). This type of research informs us about the generality of effects across cultural
groups.
Can Results Generalize beyond the Specific Study Situation?
Beyond the Experimenter
● The person who actually conducts the experiment can trigger another generalization problem. In
some studies, only one experimenter is used to reduce variability in how the experimenter
influences participants. Because little attention is typically paid to the personal characteristics of
experimenters , it is possible that the results of a study using only one experimenter cannot be
generalized to other types of experimenters. Some of the important characteristics of
experimenters include personality, gender, and amount of practice in the role of an experimenter
A warm, friendly experimenter may produce different results than a cold, unfriendly
experimenter. Participants are also more productive and cooperative when experimenters are
dressed in accordance with stereotyped gender roles
● One solution to the problem of generalizing to other experimenters is to use two or more
experimenters, with differing characteristics .Another option is to deliver instructions using a
computer, which minimizes the amount of interaction between experimenters and participants,
thereby reducing the potential for influence
Beyond a Pretest
● Researchers must often decide whether to give a pretest . Intuitively, pretesting seems to be a
good idea. The researcher can examine whether groups are equivalent on the pretest, and
sometimes it is imps. In longitudinal studies that have the risk of participants withdrawing from
the study, a pretest allows uortant to examine changes in people’s scores from pretest to posttest,
rather than simply comparing posttest scores to look for any effects of selective attrition. A
pretest lets us determine whether the people who withdrew from the study were different from
those who completed it.
● Pretesting, however, may limit the ability to generalize any results to populations that do not
receive a pretest. In the real world, people are rarely given a pretest. For example, people do not
regularly take stock of their attitudes before listening to a political speech or viewing an
advertisement
● In the Solomon four group design, the same experiment is conducted with and without the
pretest. The researcher can then examine whether there is an interaction between the independent
variable and the pretest variable. If the pretest has no effect, posttest scores on the dependent
variable are the same regardless of whether or not the pretest was given.
Beyond the Laboratory
● Research conducted in a laboratory setting has the advantage of allowing the experimenter to
study variables under highly controlled conditions. In experiments, the goal of high internal
validity may sometimes conflict with the goal of external validity
● Field experiments are one way that researchers try to examine phenomena under more realistic
circumstances, and thereby increase the external validity of their experiments (Chapter 4). In a
field experiment, the researcher manipulates the independent variable in some natural setting, like
a factory, a school
 ● Conducting research in both laboratory and field settings provides the greatest opportunity for
advancing our understanding.Both lab- and field-based studies are vital contributors to our
understanding of eyewitness testimony. Although lab studies and field studies do tend to find
similar effects, there is substantial variability in the degree to which this is true, based on the field
of psychology and topic of study.
● Replication is an important way to overcome some of the questions of generalization that stem
from the results of a single study. There are two types of replications to consider: direct
replications and conceptual replications
● A direct replication is an attempt to replicate the procedures of a study as closely as possible to
see whether the same results are obtained. In best practice, individual researchers attempt to
directly replicate their own work when possible, especially when the results from the initial study
are unexpected or are based on small samples . Direct replications are crucial for determining
whether an original finding can generalize to other samples drawn from the same population —in
other words, to offer evidence that the initial result was not simply a Type I error
● A single failure to replicate does not always mean that the original phenomenon does not truly
exist, much as a single study demonstrating an effect should not convince us on its own that a
phenomenon is real. Failures to replicate share the same difficulties of interpretation as
statistically non-significant results. A failure to replicate could mean that the original results are
invalid, but it could also mean that the replication attempt was flawed.The so-called “Mozart
effect” offers an example of the importance of replications
● Recent commitments made by journal editors in psychology are changing that norm by accepting
particular types of replication studies after adequate peer review. Researchers are also developing
a set of best practices that are helping to ensure that direct replications are high quality and
objective . Some recommendations for convincing replication attempts include ensuring high
statistical power; following the original procedures as closely as possible, including using the
original materials if obtainable
● Not only does a multi-labs approach add external validity to the results, it also promotes
objectivity.The RRR and other published direct replications represent a major commitment to
high-quality psychological science.
Replicate Conceptually
● In a conceptual replication, the independent variable is manipulated in a different way and/or the
dependent variable is measured in a different way from the original study. A relationship that
appears with one set of operationalizations should generalize to different ways of manipulating
and measuring the same variables. Sometimes a conceptual replication may involve an alternative
stimulus
● When conceptual replications produce similar results as the original study, a case can be made
that the relationship between the variables generalizes beyond the original
operationalizations.One problem with conceptual replications is the potential to promote Type I
errors. If a conceptual replication fails to find an effect, it is possible to toss that study aside as
methodologically problematic, and continue trying different operationalizations until the original
effect is conceptually replicated
● consider conducting a direct replication first, to ensure that the original results generalize beyond
the original sample. Then, carefully substitute alternative operationalizations in a conceptual
replication to develop a theory about how the underlying theoretical variables relate.
Recognize the Limits of Convenience Samples and Seek Diverse Samples
● Psychologists rely heavily on undergraduate students for their research participants. As we noted
earlier, this unique population does not represent the general population of the country from
which the sample was drawn, nor does it represent humanity more generally. This represents a
challenge to the external validity of behavioral science., we emphasized the importance of
randomly sampling from a population when you wish to generalize to that population
● However, it is not always possible to collect data from diverse sources. That said, researchers
should strive to do so whenever possible.The Internet is one relatively inexpensive way for
researchers to reach samples beyond undergraduate students. Although online samples raise their
own issues of generalization, they typically reach a broader population than undergraduate
samples. These samples tend to have greater diversity with respect to socio-economic status,
ethnicity, age, and work experience
● However, research with animals now relies on a great diversity of species to explore different
research questions.Factorial designs allow us to study interactions, with an interaction occurring
when an effect exists under one condition but not another, or when the nature of the effect is
different in one condition than in another. This can be leveraged to study generalizability across
groups directly.
● Researchers can address generalization issues that stem from the use of different populations by
including individual differences as a variable in a factorial design. By including variables such as
age or cultural background in the design of a study, it can be determined whether there are
interaction effects like the ones illustrated
5 Rely on Multiple Studies to Draw Conclusions: Literature Reviews and Meta Analysis
● In a literature review, a researcher reads many studies that address a particular topic and then
writes a paper that summarizes, organizes, and evaluates the literature, sometimes proposing
advances to theory. The literature review offers a summary of existing results, indicating what
findings are strongly or weakly supported in the literature
● A currently more prevalent approach to the review paper is to use statistical techniques to
combine the results from many past studies, known as a metaanalysis .In a meta-analysis,
information from each past study is gathered and considered statistically, including effect-sizes,
sample sizes, and other features of the studies that might influence the effect-size for the outcome
● A meta-analysis compares the strength of a given finding across many different studies that tested
the same or similar variables.The example of a meta-analysis on the so-called Mozart Effect.One
advantage of meta-analyses over literature reviews is that a single conclusion is evident, in the
form of an estimation of the effect-size observed across many past studies. With a literature
review, it is very difficult to integrate so many past results and arrive at a clear conclusion. In
contrast, a meta-analysis using a quantitative approach, analyzing the data with statistics, to arrive
at an estimate that integrates all this past information
● Meta-analyses are used to evaluate the relationship between variables, to test hypotheses, and to
refine theories. People tend to procrastinate because they are disorganized and impulsive, but less
so because they are anxious or perfectionistic. Moreover, people tend to procrastinate on tasks
they don’t enjoy and when the reward for completing a task is not immediately delivered.
● Both narrative literature reviews and meta-analyses provide valuable information, and the two are
often complementary. A meta-analysis allows statistical, quantitative conclusions, whereas the
narrative review uses a more qualitative approach .
● The goal of anyone doing a meta-analysis or review is to consider all of the relevant research,
whether it is published or not. This goal is complicated by publication bias.Therefore, researchers
conducting these types of reviews often email other researchers, or make posts to forums and
listservs, requesting relevant unpublished data.
● Simply knowing about meta-analysis can improve the way we interpret information for literature
reviews.Whenever possible, seek large and diverse samples for participation in research, to
enable more accurate and generalizable results