t

os
rP
REPRINT H04GIR
PUBLISHED ON HBR.ORG
JULY 27, 2018

ARTICLE yo
ECONOMICS & SOCIETY
op
How to Be a Smart
Consumer of Social
tC

Science Research
No

by Eva Vivalt
Do

This document is authorized for educator review use only by Carlos Eduardo Lourenco, FGV - EAESP until Aug 2021. Copying or posting is an infringement of copyright.
Permissions@hbsp.harvard.edu or 617.783.7860
 t
os
ECONOMICS & SOCIETY

How to Be a Smart

rP
Consumer of Social
Science Research
by Eva Vivalt
JULY 27, 2018

yo
op
tC
No

PCHYBURRS/GETTY IMAGES

Academic studies in the social sciences often find very different results. Even in disciplines like
Do

medicine, where one might imagine there to be a direct, physical relationship between the
intervention being tested and its consequences, results can vary — but many think the situation is
worse in the social sciences. This is because the relationship between an intervention and its effects
may depend on multiple factors, and differences in context or implementation can have a large
impact on the studies’ results.

COPYRIGHT © 2018 HARVARD BUSINESS SCHOOL PUBLISHING CORPORATION. ALL RIGHTS RESERVED. 2

This document is authorized for educator review use only by Carlos Eduardo Lourenco, FGV - EAESP until Aug 2021. Copying or posting is an infringement of copyright.
Permissions@hbsp.harvard.edu or 617.783.7860
 There are other reasons that studies might report different effects. For one, chance errors could affect
a study’s results. Researchers may also consciously or subconsciously bias their results. All these

t
sources of variability have led to fears of a “replication crisis” in psychology and other social sciences

os
relevant to business. Given this variability, how should we consume evidence?

The immediate answer is to not rely too much on any one study. Whenever possible, look for meta-
analyses or systematic reviews that synthesize results from many studies, as they can provide more-
credible evidence and sometimes suggest reasons why results differ.

rP
When considering how much weight to give a study and its results, pay attention to its sample size.
Studies are particularly likely to fail to replicate if they were based on a small sample. The most
positive and negative results are often those with the smallest samples or widest confidence
intervals. Smaller studies are more likely to fail to replicate in part due to chance, but effects may also
be smaller as sample size increases, for several reasons. If the study was testing an intervention,

yo
there may be capacity constraints that prevent high-quality implementation at scale. For example, if
you were testing out a training program, you might not need to hire any full-time personnel to run it
— but if you were to expand the program, you would need to hire new staff, and they may not run it
quite as well.
op
Smaller studies also often target the exact sample that would yield the biggest effects. There’s a logic
to this: If you have a costly intervention that you can allocate to only a few people, you might
perform triage and allocate it to those who could benefit from it the most. But that means the effect
would likely be smaller if you implemented the intervention in a larger group. More generally, it can
be helpful to think about what things might be different if the intervention were scaled up. For
tC

example, small interventions are unlikely to affect the broader market, but if scaled up, competitors
or regulators might change their behavior in response.

Similarly, consider peculiarities of the sample, context, and implementation. How did the
researchers come to study the people, firms, or products they did? Would you expect this sample to
have performed better or worse than the sample you are interested in? The setting could have
No

affected the results too. Was there anything special about the setting that could have made the
results larger?

If the study was evaluating an intervention, how that intervention was implemented is very
important. For example, suppose you hear that reminder messages can improve attendance at
appointments. If you were considering implementing a reminder system, you would probably want
to know the frequency of the messages the researchers sent and their content in order to gauge
Do

whether you might have different results.

You may also have more confidence in the results of a study if there is some clear, causal mechanism
that explains the findings and is constant across settings. Some results in behavioral economics, for
instance, suggest that certain rules of human behavior are hardwired. Unfortunately, these

COPYRIGHT © 2018 HARVARD BUSINESS SCHOOL PUBLISHING CORPORATION. ALL RIGHTS RESERVED. 3

This document is authorized for educator review use only by Carlos Eduardo Lourenco, FGV - EAESP until Aug 2021. Copying or posting is an infringement of copyright.
Permissions@hbsp.harvard.edu or 617.783.7860
 mechanisms can be very hard to uncover, and many experiments in behavioral economics that
initially seemed to reflect a hardwired rule have failed to replicate, such as a finding that happiness

t
increases patience. Nonetheless, if there is a convincing reason that we might expect to see the

os
results that a study has found, or if there is a strong theoretical reason that we might expect a
particular result to generalize, that should lead us to trust the results more.

Finally, if it sounds too good to be true, it probably is. This might sound like a cliché, but it’s based on
a principle from Bayesian statistics: Stranger claims should require stronger evidence in order to

rP
change one’s beliefs, or “priors.” If we take our priors seriously — and there is reason to believe that,
on average, humans are pretty good at making many kinds of predictions — then results that seem
improbable actually are less likely to be true. In other words, holding constant the significance of a
result or the power of a study, the probability of a “false positive” or “false negative” report varies
with how likely we thought it to be true before hearing the new evidence.

yo
This article emphasizes the importance of drawing on many studies, rather than relying too much on
any one study. What if there haven’t been many studies? If that’s the case, you may wish to consider
other sources of evidence, such as advice or predictions from others. Just as with social science
research, you may get conflicting advice, but aggregated forecasts can be quite accurate. However,
make sure your sources are not relying on the same information; research has found that people are
op
subject to “correlation neglect,” so that when multiple experts or news outlets base their reports on
the same study, people incorrectly treat those sources as independent and end up over-weighting
that study’s results.

Overall, trust a mix of your experience and the evidence, but be careful not to be overconfident in
tC

your assessments. Most people could benefit by weighing the evidence more, even when results
vary.

Eva Vivalt is a Research Fellow and Lecturer at the Australian National University.
No
Do

COPYRIGHT © 2018 HARVARD BUSINESS SCHOOL PUBLISHING CORPORATION. ALL RIGHTS RESERVED. 4

This document is authorized for educator review use only by Carlos Eduardo Lourenco, FGV - EAESP until Aug 2021. Copying or posting is an infringement of copyright.
Permissions@hbsp.harvard.edu or 617.783.7860