Meta-analysis offers research on research for MR | Articles | Quirks.

com 9/19/17, 9'50 AM

Meta-analysis offers research on

research for MR Meta-analysis is a
relatively new research-on-research
tool that marketing researchers can
use to examine a collection of results
across multiple primary studies. This
article covers the basics of meta-
analysis and addresses some criticism
toward it.
Editor's note: Kevin Gray is president of Cannon Gray LLC, a marketing
science and analytics consultancy. This article appeared in the May 12,
2014, edition of Quirk's e-newsletter.

Meta-analysis is a set of procedures for statistically synthesizing the results

of several primary studies. It is research on research and is used when we
want examine the body of evidence about a topic, rather than relying on the
results of a single study. Even in rigorous disciplines such as medicine,
studies often do not agree in their central conclusions and meta-analysis
provides a systematic way to examine a collection of results, typically effect
sizes, across multiple primary studies. It also sheds light on factors
potentially underlying dissimilarities in findings and helps us make sense of
the patterns of effects. It began to take root in the '90s and is applied in
medicine, ecology, psychology, education, business and other fields as well as
for the evaluation of government programs. It is still quite new to marketing

https://www.quirks.com/articles/meta-analysis-offers-research-on-research-for-mr Page 1 of 9
Meta-analysis offers research on research for MR | Articles | Quirks.com 9/19/17, 9'50 AM

research and arguably underutilized.

Estimate the true size

Meta-analysis is increasingly employed in place of the traditional narrative

literature review, which has been criticized by its detractors for being
subjective, unsystematic and as "arithmetic with words." Meta-analysis is
appropriate, provided the primary studies are not too disparate. Since real
effects cannot be reliably established through one study alone, meta-analysis
helps us better estimate the true size of effects and is better suited when we
wish to generalize to a broad population.

Effect sizes are typically measured by means, proportions, risk ratios, odds
ratios or correlations. However, meta-analysis is not simply a matter of
adding up effect sizes in each study and dividing by the number of studies we
are examining, nor is it just counting the number of significant differences
favoring one hypothesis over another and choosing a winner. Instead, it uses
statistical techniques to combine the results of multiple studies. Ideally, data
from individual subjects (individual participant data) in each study would be
used but more typically this is not obtainable and aggregate data must be
analyzed instead.

Weighted by precision

In meta-analysis, the individual studies are weighted by precision, which is

derived from the standard errors of the individual studies. The term "inverse
variance" is often used to describe this weighting mechanism, which is
primarily a function of sample size. Larger studies (i.e., those with larger
sample sizes) will be more influential in meta-analysis. It is not necessary that
all primary studies in a meta-analysis measure effect in the same way and
formulas exist to convert among effect sizes based on dissimilar metrics.
There are other methods besides the inverse variance scheme. For details,
https://www.quirks.com/articles/meta-analysis-offers-research-on-research-for-mr Page 2 of 9
Meta-analysis offers research on research for MR | Articles | Quirks.com 9/19/17, 9'50 AM

readers can refer to standard references such as Introduction to Meta-

Analysis (Borenstein et al.) and Modern Epidemiology (Rothman et al.).

Forest plots

Results of meta-analyses are normally summarized with forest plots, a handy

graphical device that efficiently conveys the highlights of the analysis. (Forest
stands for "forest of lines" and we are interested in both the trees and the
forest.)

Figure 1 shows a simple example, taken from Wikipedia, based on five

studies. In practice, sometimes dozens or even hundreds of primary studies
are used. In the plot, the size of a square is proportional to the corresponding
study's weight in the meta-analysis and the horizontal line represents the
confidence interval of the study's odds ratio (OR), an effect measure that is
standard output from logistic regression and equals 1.0 when there is no
effect. The diamond at the bottom of the plot reports an overall odds ratio of
2.2, with the real effect size estimated to lie between 1.9 and 2.4.

https://www.quirks.com/articles/meta-analysis-offers-research-on-research-for-mr Page 3 of 9
Meta-analysis offers research on research for MR | Articles | Quirks.com 9/19/17, 9'50 AM

Variations of meta-analysis

Fixed-effect versus random-effects

Statisticians will be familiar with fixed-effects and random-effects models. In

meta-analysis, fixed-effect (singular form) has a different meaning and
Borenstein et al. suggest "common-effect" might be a more suitable term. The
fixed-effect model assumes the true effect is the same in all studies and that
differences we observe among studies are the result of sampling error, while
random-effects models allow that the true effect may vary from study to
study.

When fixed-effect models are used, studies with larger sample sizes will

https://www.quirks.com/articles/meta-analysis-offers-research-on-research-for-mr Page 4 of 9
Meta-analysis offers research on research for MR | Articles | Quirks.com 9/19/17, 9'50 AM

receive relatively more weight and smaller studies relatively less weight,
though this by itself should not determine whether a fixed-effect or random-
effects model is chosen. Confidence intervals under random-effects models
will be wider because both within and between (across studies) sources of
variation are taken into account. Random-effects meta-analysis tends to be
preferred because the primary studies as a rule are heterogeneous in
participant (i.e., respondent) characteristics, in the implementation of
treatments themselves or in other important operational aspects. It is
generally, though not always, more realistic to assume that the true effect
could vary from one study to the next. A new method, the quality-effects
model, has been advocated as an alternative to random-effects models.

Meta-regression

Regression is widely used in primary research and also has an important

place in meta-analysis. Rather than subjects in a primary study, however,
observations in the meta-regression data file are the results of the primary
studies and both the dependent and independent variable(s) are at the study
level, not the subject (i.e., respondent) level. As with standard regression,
independent variables may be continuous, categorical or a combination of the
two. Fixed- and random-effects models are both possible, as are variations of
regression (e.g., polynomial). Analogous to regression in primary studies, the
number of studies should be sufficiently large for the analysis to be
meaningful and a minimum of 10 studies per independent variable is one rule
of thumb, though this is not set in stone.

Psychometric meta-analysis

Any study has methodological faults that affect its results, sometimes
substantially. Psychometric meta-analysis, also known as validity
generalization or Hunter-Schmidt meta-analysis, attempts to adjust effect

https://www.quirks.com/articles/meta-analysis-offers-research-on-research-for-mr Page 5 of 9
Meta-analysis offers research on research for MR | Articles | Quirks.com 9/19/17, 9'50 AM

estimates for methodological limitations of the studies and perform a meta-

analysis of the corrected results. The quality-effects model shares some of the
aims of these methods.

Psychometric meta-analysis uses procedures based on psychometrics theory

to make corrections for attenuation of effect size due to measurement error
and other flaws. Effects will generally be larger and their variation smaller
after corrections have been made and the distribution of true effects is
emphasized, rather than the overall mean effect. Another distinction is that
raw correlations are favored as the metric of effect and sample size is
preferred for weighting.

Take into account

Researchers must take into account many factors when selecting primary
studies. Obvious ones include the topic, the sample composition, how the
sample was drawn and from what population and when and where the
research was conducted. Less obvious, perhaps, are details of the study
design (e.g., whether it was experimental or observational). (See my article,
"Forget exact science: Drawing conclusions from observational research," in
the July 22, 2013, edition of Quirk's e-newsletter, for an overview of
experimental versus observational research.)

Other considerations include which potential confounders were accounted

for, how they were measured, the analytic methods employed and, of course,
whether sufficient information regarding the study has been published or can
be obtained. To a degree, heterogeneity is desirable since we will have more
confidence that the findings of the meta-analysis are generalizable to real-
world circumstances, though this is not to say anything goes.

Publication bias is another concern. This is a complex topic but essentially

refers to the fact that many studies go unpublished, not because of poor
https://www.quirks.com/articles/meta-analysis-offers-research-on-research-for-mr Page 6 of 9
Meta-analysis offers research on research for MR | Articles | Quirks.com 9/19/17, 9'50 AM

quality, but because no statistically-significant effects were detected. Effects

that achieve statistical significance will, in general, be larger than those that
have not. Publication bias reflects editorial decisions but researchers must
first decide on their own whether to submit results for publication.
Statistically-significant findings are also more likely to be published quickly,
to be published more than once, to be cited frequently and to be published in
English and thus easier to locate. Meta-analysis cannot not magically make
these potential biases disappear but statistical methods have been developed
(e.g., funnel plots) to help assess whether they are a serious problem in the
set of studies we have tentatively selected. In short, judgment plays a large
part in study selection but it must be remembered that science is not merely
plugging numbers into formulas in mechanical fashion. Whatever approach
we take and regardless of our topic, it's garbage in, garbage out.

Somewhat contentious

Despite having advantages over alternatives, meta-analysis remains

somewhat contentious. There are several reasons for this. Because the
combined sample sizes can be very large, very small effects can be depicted as
significant, even when effect sizes in all the individual studies are not
statistically different from zero. Larger studies, moreover, are often
operationally more challenging and can actually be more error-prone than
ones that are smaller-scale and more carefully supervised. But larger studies
will have more weight in meta-analysis because of their bigger sample size.
Also, as noted, inclusion criteria for primary studies will have a major impact
on the results and another critique is that meta-analysis can make it easier for
the investigator to "cook" the results by cherry-picking "ingredients" that
support a favored hypothesis. In other words, it can be used for advocacy
instead of science.

Proponents of meta-analysis often respond to criticisms such as these by

https://www.quirks.com/articles/meta-analysis-offers-research-on-research-for-mr Page 7 of 9
Meta-analysis offers research on research for MR | Articles | Quirks.com 9/19/17, 9'50 AM

admitting that many of them do indeed have merit but that they apply even
more to traditional narrative literature reviews or simply counting significant
differences (vote-counting). In meta-analysis, it is claimed that these dangers
are more transparent. My own view is that, while properly-conducted meta-
analyses have important benefits, we do need to be vigilant against potential
malpractice, perhaps more so than in the case of narrative literature reviews,
which, in my experience, tend to be circumspect in their conclusions.

Who cares?

To this point, meta-analysis has been little-used in marketing research and

some might wonder "Who cares? I'm not a doctor or a psychologist, so why
should I incorporate these methods into mine?" My first response would be
that, as researchers, we need to be attentive to what researchers in other
fields are doing. After all, nearly all our methods and research protocols have
been borrowed from other disciplines!

At a more day-to-day level, marketing researchers often undertake desk

research as part of a project, especially at the design phase, and when doing
so may occasionally encounter meta-analyses. It is therefore helpful to have
some background on what it is and its strengths and limitations.
Furthermore, we can conduct it ourselves when designing new research.
Lastly, we can use meta-analysis for R&D on new methodologies, for
examining questionnaire design issues, in ROI analysis, in pricing research
and for testing product claims, to name just a few.

In a tiny nutshell

The foregoing is meta-analysis in a tiny nutshell. For those who'd like to study
it in depth, there are numerous online sources and papers in addition to
textbooks. Experimental and Quasi-Experimental Designs for Generalized
Causal Inference (Shadish et al.) is a widely-cited textbook that offers a
https://www.quirks.com/articles/meta-analysis-offers-research-on-research-for-mr Page 8 of 9
Meta-analysis offers research on research for MR | Articles | Quirks.com 9/19/17, 9'50 AM

comprehensive perspective on research, including meta-analysis, as well as

detailed guidelines on research design and how it affects interpretation. The
Cochrane Collaboration provides a very large repository of research on health
care and health policy and is a good source for examples of meta-analysis.
The Campbell Collaboration is a similar organization for social, behavioral
and educational research.

https://www.quirks.com/articles/meta-analysis-offers-research-on-research-for-mr Page 9 of 9