Session 5A: Evaluation in IR 2
Meta-Analysis for Retrieval Experiments Involving
Multiple Test Collections
Ian Soboroff
National Institute of Standards and Technology
Gaithersburg, Maryland
ian.soboroff@nist.gov
ABSTRACT
Traditional practice recommends that information retrieval experi-
ments be run over multiple test collections, to support, if not prove,
that gains in performance are likely to generalize to other col-
lections or tasks. However, because of the pooling assumptions,
evaluation scores are not directly comparable across different test
collections. We present a widely-used statistical tool, meta-analysis,
as a framework for reporting results from IR experiments using
multiple test collections. We demonstrate the meta-analytical ap-
proach through two standard experiments on stemming and pseudo-
relevance feedback, and compare the results to those obtained from
score standardization. Meta-analysis incorporates several recent
recommendations in the literature, including score standardization,
reporting effect sizes rather than score differences, and avoiding a
reliance on null-hypothesis statistical testing, in a unified approach.
It therefore represents an important methodological improvement
over using these techniques in isolation.
CCS CONCEPTS
· Information systems → Evaluation of retrieval results;
KEYWORDS
meta-analysis, score standardization, effect sizes
ACM Reference Format:
Ian Soboroff. 2018. Meta-Analysis for Retrieval Experiments Involving Multi-
ple Test Collections. In The 27th ACM International Conference on Information
and Knowledge Management (CIKM ’18), October 22–26, 2018, Torino, Italy.
ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3269206.3271719
1 INTRODUCTION
Laboratory experiments in information retrieval commonly use
test collections. A test collection comprises a set of documents, a
set of search needs, and a set of relevance judgments that indicate
the łcorrectž documents that should be retrieved for each search
need. Test collections are intended to model an abstract search task,
such as the well-known ad hoc search task from the Text Retrieval
Conference (TREC) test collections [24].
This paper is authored by an employee(s) of the United States Government and is in
the public domain. Non-exclusive copying or redistribution is allowed, provided that
the article citation is given and the authors and agency are clearly identified as its
source.
CIKM ’18, October 22–26, 2018, Torino, Italy
2018. ACM ISBN 978-1-4503-6014-2/18/10.
https://doi.org/10.1145/3269206.3271719
713
CIKM’18, October 22-26, 2018, Torino, Italy
Because test collections have a fixed set of documents and search
needs, they necessarily represent some kind of sample of the uni-
verse of document collections and search needs. Hence, experimen-
talists in information retrieval have long recommended that tests of
retrieval effectiveness be repeated and reported over multiple test
collections, lest a result be an artifact of one particular collection’s
quirks.
Voorhees reinforces this recommendation with respect to how
modern test collections violate the assumptions of the original
Cranfield-II experiments [23]. Because modern test collections are
known to have incomplete relevance judgments, evaluation met-
ric values are only meaningful for relative comparisons between
systems as measured on the same test collection.1 This imparts a de-
gree of noise into any retrieval experiment, and as we reduce noise
from variance across topics by using many topics in a single test
collection, we can control for the noise of the collection by using
many test collections. However, now we have a problem: because
of the noise and sampling error inherent in a single test collection,
we should use more than one test collection. But because of incom-
plete judgments and (again) the sampling error inherent in test
collections, we can’t compare system scores between collections
directly.
This paper proposes using a statistical method, meta-analysis, to
perform comparisons across multiple test collections in an infor-
mation retrieval experiment. We compare meta-analysis with score
standardization [25] through several demonstration experiments
that highlight the advantage of meta-analysis over score standard-
ization. The goal of this paper is not to claim that meta-analysis
is new, or to present new empirical evidence for well-known IR
phenomena, but to make a methodological contribution that solves
a real problem in IR experiment.
2 RELATED WORK
Information retrieval experiments through the 1970s and 1980s
were necessarily performed with the data at hand. There were a
few test collections held by multiple research groups, including
Cranfield (CRAN), CISI, and CACM, and in the mid-to-late 1980s
experiments conducted across more than one test collection be-
gan to appear. Two particularly interesting examples appeared in
SIGIR 1987: one study of phrase indexing [8], and another on stem-
ming [10]. Everyone at the time expected retrieval performance
to vary across different types of document collections, and these
papers clearly demonstrated it experimentally. With the emergence
of TREC and other annual evaluation forums in the 1990s, along
1 One might wonder, given subsequent observations of variances in relevance assess-
ment, even by the same user, whether any set of relevance judgments can be considered
to be complete.
Session 5A: Evaluation in IR 2
with cheaper computer and storage technology, it became feasible
to expect researchers to investigate results in multiple test collec-
tions. Nevertheless, there was (and remains) no accepted method
for rigorously drawing a general conclusion from results gleaned
from multiple test collections.
Webber et al. [25] proposed score standardization as a solution
to this problem: each test collection should come with a standard-
ization factor that scales scores into a comparable interval. The
standardization approach replaces raw scores with z-scores, scores
normalized to zero-mean, unit-variance based on the sample of
scores in a designated set of system runs such as those used to pool
the collection. This scales the differences in score between experi-
mental conditions as some number of standard deviations from the
mean, which is comparable across test collections with identically
standardized scores. Sakai proposed adding a linear transformation
to the z-score which improves sensitivity to the topic set while
maintaining discriminative power [18]. Webber et al. give an excel-
lent summary of information retrieval literature on the subject of
normalization of evaluation scores to which we refer the interested
reader.
Score standardization is a reasonable approach. We see three
issues which motivate us to propose meta-analysis as an improve-
ment. First, z-scores are in units of standard deviations, which could
be hard to interpret. Second, the z-score parameters are fixed at
what might be an early point in the improvement of IR technology
for that test collection, and so may need recalibration over time.
Third, all scores must be standardized using the identical approach,
for example either that of Webber et al. [25] or Sakai [18], with
identical parameters, for comparability to be maintained.
One can argue that standardization assumes that metric scores
are normally distributed, or alternatively that the mean and stan-
dard deviation are meaningful characterizations of the distribution
of scores, which is plainly not the case when scores are bounded
to the range [0,1]. In practice z-score normalization is a common
practice in many statistical methods, and is even inherent in many
common techniques such as the t-test which are known to be robust
to violations of distributional assumptions.
What is missing in the score standardization approach is a proce-
dure for inference about the scaled scores. Ideally, score standardiza-
tion would generate a summary score over multiple test collections,
and predict whether outcomes in similar test collections will behave
similarly.
3 META-ANALYSIS
Meta-analysis is a statistical technique designed to integrate in-
formation from multiple separate experiments in order to draw
a single overall conclusion [3]. This is a long-standing problem
in many fields, including medicine, public health, and agriculture,
where researchers are rarely fortunate to experiment with the same
data [13]. Consider the question of whether iocaine powder2 is
effective for treating persistent headaches. Different studies would
conduct slightly different experiments around this question, differ-
ing in dosage, how the powder is delivered to the subject, whether
the subject is a human (the target population) or an animal (a model
of the population), the number of subjects surviving the study, etc.
2A fictional substance featured in the film łThe Princess Bridež (1987)
714
CIKM’18, October 22-26, 2018, Torino, Italy
All of these differences could affect the outcome of the study. Some
of the studies could be statistically underpowered or neglect to
describe an essential part of the procedure clearly. Meta-analysis
was developed to allow a researcher to select an appropriate, com-
parable subset of studies and integrate them to derive an overall
łtreatment effectž. We refer the reader to Borenstein et al. [3] for
an excellent and readable gateway of the basics of meta-analysis
beyond those presented here.
The problem of comparing IR experiments across multiple test
collections is similar to the problem of integrating separate scien-
tific studies. The documents in the test collection are samples of an
undefined wider population: ClueWeb12 is a sample of the entire
web as it existed between February and May 2012, during which
time the web certainly changed; at the same time, it is a sample of
the written output of the human species. TREC CDs 4 and 5, used
in many ad hoc TREC tasks over the years, are a set of news articles
from the Wall Street Journal and the Los Angeles Times alongside
government publications from the US Federal Register and For-
eign Broadcast Information Service. Each of these is a sample of
its source, and is a sample of the larger population of newspaper
articles. Similarly, the topics in a test collection are an opportu-
nity sample: in TREC they are developed by assessors or mined
from query logs. There is often a noisy selection process, since
topics with large numbers of relevant documents present problems
for evaluating ad hoc search. These sampling and selection pro-
cesses are difficult if not impossible to characterize completely, so
we certainly can’t say how similar the samples might be to each
other. Numerous papers on experimental methods in IR have ex-
amined subsets of test collection topic sets (for example [9]) and
observed that some topics have a larger effect on the experimental
outcome, or that sometimes different topics are equally predictive
and thus having multiple such topics in a test collection increases
bias towards topics of that type, whatever that type might be.
Even without being able to understand fully the relationship
of the components of test collections to the populations they are
drawn from, we want to be able to integrate results across multiple
test collections, and take away from that some sort of statistical
prediction about future datasets. Meta-analysis scales the results of
the so-called łprimaryž experiments so that they are directly com-
parable, and summarizes the experiments with a mean treatment
effect in a confidence interval. If the interval crosses the point of
no difference, then this is equivalent to result that is not statisti-
cally significant. Otherwise the interval contains the true treatment
effect in similar studies 95% of the time.
We propose here to use meta-analysis as a framework for pre-
senting results of a series of experiments conducted by a single team
using multiple test collections. To accomplish this we need to give
a brief tutorial on meta-analysis, with specific recommendations
on choices that are sensible for IR experiments.
3.1 Effect size
The first step in meta-analysis is to quantify the effect size we
observe in each experiment. In an experiment we keep a control
or a baseline, and apply a treatment, for example stemming query
terms. We observe some difference, and the true value of that change
is due to the treatment but also possibly other spurious causes. The
Session 5A: Evaluation in IR 2
goal of meta-analysis is to set up a comparison within which we
can estimate the true treatment effect with explicit assumptions
about other effects.
There is a wide array of types of effect sizes in the biomedical
and social sciences, for results reported as tables of outcomes, or
relative risks, or categorical measures. For measures generically,
the chief questions are whether the measure is continuous or not,
whether it is ratio-valued with a meaningful zero point, and how
that value is distributed. In common IR experiments we are lucky
in that most of our system effectiveness measures are continuous
ratios with a natural zero. Even for measures like average precision
which are not normally distributed, they are well characterized by
their mean and standard deviation.
In a meta-analysis, we are not concerned with the score of a
single retrieval run, rather, we want to quantify the effect of the
łtreatmentž. A treatment in information retrieval is whether we ap-
ply stemming or not, or the difference between a ranking function
and a baseline, or the effect we see from adding relevance feed-
back. We will consider each experiment as computing the effect
size between the treatment and the control, between the experi-
mental condition and the baseline. This already offers a significant
methodological advantage: the baseline is always explicit. It is also
possible to look at covariates, as in multiple regression or analysis
of variance, to compare multiple treatment variables, but in this
paper we restrict ourselves to the common case: I, the experimenter,
propose a novel retrieval algorithm and wish to demonstrate its
improvement over a reasonable baseline.
For each test collection, we have a measurement of the baseline
and the treatment for each topic. The simplest effect size is the
difference in the raw mean scores y = T̄ − B̄. We also need the
variance of the effect size, which can be pooled (the sample variance
of the per-topic differences) or not. Since we don’t expect the per-
topic scores between the baseline and the treatment to be perfectly
correlated, we use the unpooled variance:
s 2 = sT2 + s B2 − 2 × sT × s B
where s 2 is the sample variance over the topics in either the treat-
ment or the baseline.
An alternative effect size we might choose for IR experiments is
the łresponse ratiož, or the logarithm of the ratio of the mean of
the treatment to the mean of the baseline:
 
T̄
y = ln
B̄
The variance of the log response ratio is:
 
1 1
s 2 = s pooled
2
+
nT T̄ 2 n B B̄ 2
The response ratio is equivalent to reporting a percent improvement
in the score, rather than a raw difference.
It is also possible to use the standardized mean difference as
the effect size. In meta-analysis this is typically done when the
measures in each study are different or are on different scales. The
meta-analysis process will weight the observed effects by dividing
by the variance, so we effectively standardize anyway. Since in
a typical IR experiment we have full control of the experimental
setup, we will use a single metric across all test collections, and
standardization is not necessary.
715
CIKM’18, October 22-26, 2018, Torino, Italy
3.2 Model
There are two primary meta-analysis models: the fixed-effect model
and the random-effects model. The fixed-effect model reports the
summary estimate of the true effect as the weighted average of the
study effects, with the weight as w i = 1/si2 one over the within-
experiment variance. By using a fixed-effect model, we assume that
the true effect size is constant across all the collections. A random-
effects model allows us to say the true effect size varies within an
interval.
Of course, now that we assume that the effect size varies across
the collections, that amount of variance is something we need to
be concerned about. In meta-analysis this is called łheterogeneity.ž
Within each test collection, the effect varies, and that amount of
variance varies across collections. Heterogeneity is the excess vari-
ation we see from collection to collection beyond what we would
expect to see if the true effect were the same in all collections.
In a random-effects model we add an estimate of the between-
studies variance to the study variance when we compute the weight.
There are a number of methods for estimating the between-studies
variance T 2 , of which the basic one is the DerSimonian and Laird
(łDLž) estimate [7]:
Q − df
T2 =
C
Í 2
k k w y
Õ i=1 i i
Q= w i yi2 − Ík
I =1 i=1 w i
Õ Í 2
w
C= wi − Í i
wi
df = k − 1
k = number of studies
Then the new study weight w i′ = 1/(si2 + T 2 ).
There are of course more complicated estimates, but the DL
estimate is quick to compute and is accurate. For the meta-analyses
in this paper, we use the R package metafor, which supports many
models from the literature.
3.3 Summary and inference
The meta-analysis process scales the results of the primary studies
according to their variance, and computes a summary effect as
a weighted average of the primary studies. A common form for
reporting these results is a forest plot, as in Figure 2 for the stem-
ming experiment below. Each primary study effect is shown with a
normal model confidence interval. The summary effect is shown as
a diamond extending for its confidence interval.
A n% confidence interval indicates that the true value in similar
experiments will lie within that interval n% of the time. In contrast
to a p-value from a hypothesis test, which reports the probability
of correctly rejecting a null hypothesis despite sampling error, a
retrospective statement, confidence intervals are predictive about
future outcomes in similar experiments. The distinction is subtle,
and in fact confidence intervals can be interpreted as illustrating
hypothesis test outcomes, but this flexibility is welcome here, be-
cause it allows us to distinguish the magnitude of the effect with
the probability that we will see it again.
Session 5A: Evaluation in IR 2
Document set Genre Topic set
301 - 350
Disks 4 and 5 news, gov’t 351 - 400
401 - 450
701 - 750
Gov2 web 751 - 800
801 - 850
851 - 900
Blogs06 blog 901 - 950
1001 - 1050
LDC New York Times news 1 - 60
Table 1: Test collections used in the stemming experiment.
4 STEMMING
Here we present an demonstrator experiment to determine if stem-
ming is effective at improving search. Stemming is the process of
conflating words to a łstemž term, for example combining łcom-
putež, łcomputerž, łcomputersž and łcomputingž into a single term
łcomputž, analogous to a morphological root except that the al-
gorithms are usually based on heuristic rules or corpus statistics
rather than linguistics. Stemming can be as simple as dropping the
letter łsž from word endings. Porter [14] and Krovetz [12] proposed
more complicated algorithms. Stemming can be done at index time,
storing stemmed terms only, or at search time by expanding the
query to include all terms with the common stem. Stemming ideally
should improve recall at the expense of precision, and was widely
used in information retrieval for many years, but nowadays has a
mixed reputation and many systems do not stem in their default
configuration. Harman [10] identified excessive term łexpansionž
by stemming algorithms through a failure analysis. Singh and Gupta
[20] present a lengthy survey of stemming, not only for information
retrieval but also in text classification and morphological analysis.
Silvello et al. [19] examined modern statistical stemming methods
and compared them across a variety of test collections, with a focus
on reproducing published results in CLEF; stemming can have a
stronger effect in languages that are more morphologically rich
than English.
The purpose of this experiment is to illustrate the meta-analysis
method, and contrast it to score standardization, in an experimental
context familiar to all IR researchers. We use ten test collections
from TREC: The ad hoc test collections from TREC-6 through TREC-
8, with Disks 4 and 5 (leaving aside the Congressional Record sub-
collection) as the document collection;3 the terabyte track ad hoc
collections from TREC 2004 through 2006, with the Gov2 crawl of
websites in the .gov domain;4 the blog track opinion finding ad hoc
collections from TREC 2006 through 2008, with the Blogs06 blog
crawl for data;5 and the dynamic domain collection from TREC 2017,
built on the LDC New York Times collection.6 For the blog track
topics, we treated all relevant posts of any opinion categorization
as relevant. For the dynamic domain collection, we only used the
3 https://www.nist.gov/srd/nist-special-database-22 and -23
4 http://ir.dcs.gla.ac.uk/test_collections/access_to_data.html
5 ibid.
6 https://catalog.ldc.upenn.edu/ldc2008t19
CIKM’18, October 22-26, 2018, Torino, Italy
Year titles of the topics and collapsed all subtopics such that a document
TREC-6 relevant to any subtopic was relevant to the topic. For all other
TREC-7 topics we use the title or short query field as the query.
TREC-8 One question that arises is how to best deal with the fact that the
TREC 2004 document collection is a factor linking nine of the test collections.
TREC 2005 Meta-analysis calls these factors łmoderatorsž and we can include
TREC 2006 them in our model, such that the between-studies variance takes
TREC 2006 this into account; the resulting model is called a łmixed-effects
TREC 2007 modelž. However, this makes it impossible to compute a pooled,
TREC 2008 summary effect. We take a different approach, and show summary
TREC 2017 effects for the subset of experiments using each test collection. As
an alternative method we collapsed the three ad hoc collections, the
three terabyte collections, and the three blog collections each into
a single collection using all topics created for the same task and
document collection. Combining the collections gives 150 topics
each, giving higher statistical power [17] and stability [5]. We show
each approach below.
We use the ATIRE system for all of our experiments.7 Except as
specified in the experiments below, we used the default settings, in-
cluding using BM25 as the ranking model. Retrieval was limited to
the top 1000 (-k1000) and evaluation measures were computed with
trec_eval.8 The specific stemmer employed is the Krovetz stem-
mer as provided by ATIRE. The document collections were indexed
following instructions on the ATIRE website and examples from the
website of the SIGIR 2015 RIGOR workshop on reproducibility [1].
Scripts for reproducing the results here, tested under MacOS and
Linux, can be found at https://github.com/isoboroff/meta-analysis/.
As we mentioned above, we use the metafor package in R. The
effect size is calculated by the escalc function, and takes the num-
ber of topics in each test collection, and the means and standard
deviations in the two experimental conditions, no stemming or
stemmed with the Krovetz stemmer. The ‘MD’ parameter indicates
that the effect we are calculating is the raw mean difference. The
‘DL’ parameter for the model indicates the DerSimonian and Laird
estimate for between-studies variance:
library(tidyverse)
library(metafor)
dat <- read.table('all', col.names=c('treat', 'coll', 'meas'
, 'topic', 'value'))
maps <- subset(dat, meas=='map')
stems <- maps
%>% group_by(coll, treat)
%>% spread(treat, value)
%>% summarise(n = n(), m_nostem = mean(no_stem), sd_
nostem = sd(no_stem), m_kro=mean(krovetz), sd_kro=
sd(krovetz))
stems.md <- escalc('MD', m2i=m_nostem, m1i=m_kro, sd2i=sd_
nostem, sd1i=sd_kro, n2i=n, n1i=n, data=stems)
model.md <- rma(stems.md, method='DL')
forest(model.md, slab=stems.md$coll)
Figure 1 shows the complete experiment using a forest plot. The
leftmost plot uses the raw difference in mean average precision as
the effect. Each row shows a test collection. The numbers in the
7 http://www.atire.org/, Mercurial revision 9a105a28370b
8 https://github.com/usnistgov/trec_eval
716
Session 5A: Evaluation in IR 2 CIKM’18, October 22-26, 2018, Torino, Italy

blogs06 32.05% −0.01 [−0.06, 0.03] blogs06 37.81% 0.96 [0.84, 1.09]

gov2 33.27% 0.02 [−0.02, 0.06] nyt 16.42% 1.07 [0.87, 1.32]

nyt 6.62% 0.03 [−0.06, 0.12] gov2 30.08% 1.08 [0.93, 1.25]

cd45 28.06% 0.04 [−0.01, 0.08] cd45 15.69% 1.19 [0.96, 1.47]

Summary effect, RE model 100.00% 0.01 [−0.01, 0.04] Summary effect, RE model 100.00% 1.05 [0.96, 1.14]

−0.1 0 0.1 0.8 1 1.2 1.4 1.6

Mean Difference Ratio of Means

Figure 1: Forest plots for the stemming experiment. The left figure shows the mean difference in AP as the effect, while the
right shows the log ratio of means.

right column are the effect, shown as the raw difference and as a
95% confidence interval. The percentages give the weights of the
respective test collection sub-experiments; recall that the weight is
one over the total variance for that test collection (including our
estimate of between-collection variance). The lines and polygons
blogs06.2006 −0.01 [−0.08, 0.06]
give a graphical illustration of the effect in each test collection
blogs06.2007 −0.01 [−0.09, 0.06]
and its confidence interval. The vertical line at zero means zero blogs06.2008 −0.01 [−0.09, 0.06]
difference from using stemming ś no effect. Finally, the bottom line cd45.TREC6 0.06 [−0.02, 0.14]
shows the summary effect with interval. cd45.TREC7 0.01 [−0.05, 0.08]
The results here show that we cannot conclude that stemming cd45.TREC8 0.04 [−0.05, 0.12]
gov2.2004 0.01 [−0.06, 0.08]
improves effectiveness. The summary effect line indicates that we gov2.2005 0.05 [−0.02, 0.11]
would expect stemming to change the mean average precision be- gov2.2006 0.00 [−0.07, 0.08]
tween -0.01 and 0.04 95% of the time. Because that interval includes nyt 0.03 [−0.06, 0.12]
zero, we are essentially saying we can’t detect this effect size (0.01
RE model, all document sets 0.01 [−0.01, 0.04]
MAP) at this confidence level. From the hypothesis testing perspec-
tive, this is equivalent to saying that the results are not statistically RE model, blogs06 only −0.01 [−0.06, 0.03]
significant ś we would not reject the null hypothesis of no effect RE model, cd45 only 0.03 [−0.01, 0.08]
due to stemming. We will return to the question of significance RE model, gov2 only 0.02 [−0.02, 0.06]
below in the section on power analysis.
The right-hand plot in Figure 1 shows the same experiment, but
−0.1 0 0.05 0.15
using the log ratio of means as the effect (‘ROM’ instead of ‘MD’
Mean Difference
in the escalc function). While the effect and model work with log
ratios, in the plot the results are transformed to the plain ratio of
average precision: now, the 1 point on the x-axis indicates no differ-
ence, ratios less than 1 are where stemming hurts performance, and Figure 2: Stemming experiment broken out by individual
ratios greater than 1 are where it helps. We can see that the results test collection. Comparative models within each test collec-
are the same, we’ve just expressed the effect of using stemming tion are summarized below.
with different units. We feel that the raw difference in the score is
easier to understand, and so going forward we will only report raw
differences. 4.1 Comparison with score standardization
Figure 2 shows the analysis when we don’t combine the test How does the meta-analysis method compare to score standard-
collections with common document sets. The overall summary ization? Here we analyze the data from the stemming experiment,
of the experiment is the same, but the plot now lets us see some following [25]. To standardize the scores in each test collection,
interesting differences among the test collections. We add a plot we need to have standardization factors: the mean µ and standard
of the summary effect for the studies within each test collection; deviation σ of the performance of a set of systems on each topic
these intervals are the same as those on the left side of Figure 1. in the collection. Webber et al. [25] advises taking the set of runs

717
Session 5A: Evaluation in IR 2 CIKM’18, October 22-26, 2018, Torino, Italy

that contributed to the pools for that collection, and investigates None Krovetz δ p [95% CI δ ]
the impact of improved performance over time, showing for the blogs06.2006 0.062 0.060 -0.001 0.05 [-0.004, 0.000]
TREC Robust track data, the standardization factors computed from blogs06.2007 0.039 0.038 -0.000 0.17 [-0.002, 0.000]
the TREC 6-8 pools were suitable. For those collections and for blogs06.2008 0.037 0.036 -0.001 0.08 [-0.003, 0.000]
the Gov2 collections, we re-use their standardization factors as cd45.TREC6 0.186 0.199 0.013 0.02 [0.002, 0.024]
published online.9 cd45.TREC7 0.120 0.122 0.001 0.57 [-0.004, 0.008]
There are no published standardization factors for the Blogs06 cd45.TREC8 0.220 0.231 0.011 0.00 [0.004, 0.019]
collections. We compute them from all the runs that were submitted gov2.2004 0.153 0.156 0.003 0.36 [-0.004, 0.009]
to the TREC Blog track in each year. In 2008, Blog track participants gov2.2005 0.058 0.062 0.003 0.00 [0.002, 0.006]
ran their systems over the 2006 and 2007 topics as well, and those gov2.2006 0.068 0.067 -0.001 0.56 [-0.005, 0.003]
systems were probably improved over those that were originally
pooled for those topics in 2006 and 2007, so these factors might Table 2: Stemming experiment results, reported using score
reflect some level of improvement and bug fixing that factors from standardization. łNonež indicates the run without stem-
pooled runs would not. In any event, it is not a problem to use a ming used.
slightly different method to compute the standardization factors in
different collections, as long as all experiments using a particular
collection are normalized using the same factors.
The TREC 2017 Dynamic Domain track data was not pooled; on future collections. While we might decide on the basis of stan-
rather, an active learning system based on relevance models was dardized scores that stemming might or might not help a little,
used to attempt to find and judge as many relevant documents as the meta-analysis computes a confidence interval on the overall
possible during topic development. Moreover, the measure for that effect size. Whereas the confidence intervals in the score standard-
track, the Cube Test, is quite different than average precision, and ization approach are predictive with respect to those individual
only a few teams participated in the track. As a result, we were collections, the meta-analysis summary interval is predictive for
wary of computing standardization factors for that dataset and so all collections similar to those analyzed. Thus, while the figures
we drop it from the analysis here. are similar, meta-analysis provides a more useful result than score
To standardize an AP score for a topic, the score is normalized standardization.
to zero-mean, unit-variance: Analysis of variance is an alternative approach that can be taken
with standardized scores. For ANOVA we would model the collec-
s AP − µ
s std = tion and the presence/absence of stemming as factors. This would
σ
allow us to compare the effect of the collection to the effect of stem-
As stated above, this score is in units of standard deviations from ming, but would still not provide inference about future collections.
the mean used to compute the standardization factor, so a z-score
of 0 equals the mean average precision µ, and a z-score of -2 would 5 PSEUDO-RELEVANCE FEEDBACK
indicate performance two standard deviations below that mean. For a second experiment, we looked at the improvement given
Since the z-score can range from [−∞, ∞], we follow [25] and con- by pseudo-relevance feedback over a baseline. Pseudo-relevance
vert it into a probability using the cumulative density function of feedback (PRF) is a method of query expansion. The user’s initial
the standard normal distribution. After this scaling a z-score of 0 query is run and a small number of top-n documents are retrieved.
converts to a probability of 0.5 (performing greater than half the Pretending that these documents are actually relevant, they are fed
systems used to compute the standardization factors), and a z-score to a query-expansion model. The results of the expanded query
of 1.96 gives a probability of 0.975. are shown to the user. There are a number of variations and each
Table 2 shows the results using standardized scores. The col- can be applied to different query expansion models. Historically
umn δ shows the difference in mean standardized AP obtained by (see, for example, Singhal [21]), PRF is felt to provide 5-10% relative
applying stemming. The p and 95% confidence interval columns improvement in MAP, but as in stemming, results are mixed in dif-
show the outcome of a paired, two-tailed t-test for the difference ferent collections. The Reliable Information Access (RIA) workshop
in mean standardized AP. Stemming provides a statistically signifi- in 2003 performed an in-depth study of PRF effectiveness using
cant advantage for three of the nine collections; the most hopeful multiple systems and per-topic failure analysis, finding that often
end of the confidence intervals indicate a best-case advantage of PRF can improve average precision but it can frequently be fooled
around 2%. Those three collections are the three collections with the into abysmal performance by quirks in query processing or the
highest mean effect size in the meta-analysis, so in that sense, the expansion model [4].
two analyses agree. One should keep in mind that the standardized The baseline here is the same as the unstemmed baseline in the
scores and deltas are not plain average precision figures, but rather stemming experiment. The query expansion model is Rocchio’s
represent the fraction of standardizing systems that this system algorithm [16], as implemented in the ATIRE system. Rocchio’s
improves upon. algorithm is a vector-space model approach: relevant document
Unfortunately, the score-standardization approach cannot pro- vectors are added to the query vector, and nonrelevant ones sub-
vide any kind of summary effect or prediction about performance tracted, and in the limit this should produce an optimal query. In
practice we don’t have complete feedback and don’t use all terms
9 https://people.eng.unimelb.edu.au/ammoffat/ir_eval/
in the documents for expansion, so we operate quite a step back

718
Session 5A: Evaluation in IR 2 CIKM’18, October 22-26, 2018, Torino, Italy

None PRF δ p [95% CI δ ]

blogs06.2006 0.062 0.062 0.000 0.77 -0.001 0.001
blogs06.2007 0.039 0.038 -0.001 0.44 -0.001 0.002
blogs06.2008 0.037 0.037 -0.000 0.95 -0.001 0.001
cd45.TREC6 0.186 0.188 0.002 0.07 -0.004 0.000
blogs06.2006 0.00 [−0.07, 0.07]
cd45.TREC7 0.120 0.122 0.002 0.00 -0.003 -0.001
blogs06.2007 0.00 [−0.08, 0.08]
blogs06.2008 0.01 [−0.07, 0.08]
cd45.TREC8 0.220 0.224 0.004 0.00 -0.006 -0.001
cd45.TREC6 0.01 [−0.07, 0.08] gov2.2004 0.153 0.157 0.004 0.01 -0.008 -0.001
cd45.TREC7 0.02 [−0.05, 0.08] gov2.2005 0.058 0.061 0.003 0.00 -0.004 -0.001
cd45.TREC8 0.01 [−0.07, 0.09] gov2.2006 0.068 0.071 0.003 0.02 -0.006 -0.000
gov2.2004 0.02 [−0.05, 0.09]
gov2.2005 0.03 [−0.04, 0.10]
gov2.2006 0.02 [−0.05, 0.10] Table 3: Pseudo-relevance feedback results presented using
nyt 0.02 [−0.07, 0.12] score standardization

RE model, all studies 0.01 [−0.01, 0.04]

RE model, blogs06 only 0.00 [−0.04, 0.05]

RE model, cd45 only 0.01 [−0.03, 0.05]
RE model, gov2 only 0.02 [−0.02, 0.07]
−0.1 0 0.05 0.1 0.15
Mean Difference
Figure 3: Pseudo-relevance feedback using Rocchio’s algo-
rithm.
from that limit. Here, we start with the title or short query fields of
the topic (as in the stemming experiment), and expand the query
using the top 50 terms from the top 20 retrieved documents. We
tuned effectiveness using the TREC 6-8 topics, using a variety of
ranking models and parameter settings, before settling on these set-
tings as representative of the upper range of performance. ATIRE’s
implementation of the Rocchio model selects terms based on the
KL-divergence of the term in the collection. We then re-rank the
entire collection using the expanded query.
Figure 3 shows the experimental results in a forest plot. There is
a small improvement in MAP for almost all collections, except in
two of the Blogs06 collections. However, that is just the average,
and the variance across topics indicates that some topics average
precision falls below the baseline in every collection, consistent with
the result found in the RIA experiments [4]. Performance on the
dynamic domain (łNYTž) collection has the largest variance. The
summary effect interval again crosses zero, indicating no significant
change by using PRF should be expected, on average, from test
collections like these.
One might find these results surprising, given that historically
PRF was seen as a critical component for ad hoc search. One ex-
planation could be that early work on PRF may not have had
other parameters of the models well-tuned. For example, the stan-
dard ATIRE BM25 parameters are (k 1 : 0.9, b : 0.4) in contrast to
(k 1 : 1.2, b : 0.75) reported by the Okapi team in TREC-8 [15].
We also present the parallel analysis using score standardization,
following the same procedures as in the stemming experiment and
using the same nine collections that we could standardize. Table 3
shows the results. The TREC-7, TREC-8, and all the Gov2 collections
show a statistically significant improvement, and again these are the
collections with the largest positive effect size in the meta-analysis.
The confidence intervals for these collections are negative at both
ends, despite a positive delta in the experiment, which would seem
to reflect numerical error due to the tiny difference in performance.
Because the differences in the systems are small in the standard-
ized probability interval, the analysis from standardization indicates
little or no effect from pseudo-relevance feedback in these collec-
tions. Both analyses support the hypothesis that pseudo-relevance
feedback is slightly beneficial on average, but not all topics improve.
The meta-analysis further supports an inferential conclusion, that
we would see little or no effect in future collections. There are of
course many parameters to the PRF setup which could be tweaked
here: the number of feedback documents, the number of feedback
terms, the feedback model, the starting query, and more. We might
see different results by exploring those parameters. Meta-analysis
with multiple covariates is called meta-regression, and we plan
to explore the use of meta-regression in IR experiments in future
work.
6 POWER ANALYSIS
Both of these experiments failed to achieve statistical significance.
Why is that? This is a question of statistical power. Statistical power
is the probability of rejecting a null hypothesis when the alternative
hypothesis is true. Higher values of power indicate a smaller prob-
ability of Type II error, incorrectly accepting the null hypothesis.
Power in meta-analysis is a function of the number of test collection
studies we use, the variance we see in each collection, the variance
between collections, and the effect size we expect to see. Just as
individual IR experiments can be underpowered by having too few
topics to detect a small effect, we can fall into the same trap in
meta-analysis [22].
Prospective power analysis is complicated by the need to es-
timate these parameters, but since we’ve already done the meta-
analysis, we can compute the power retrospectively, and use those
results to help guide us on what we might have done differently.
Note that we strongly recommend against retrospective power anal-
ysis as a tool for understanding the p-value or confidence interval
of the experiment at hand. Rather, we can explore the components
that make up statistical power, starting from the data we have, to
imagine the experiment we need to design next.

719
Session 5A: Evaluation in IR 2
0.8
0.6
Power
0.4
0.2
20 40 60 80
Number of test collections
Figure 4: Statistical power as we increase the number of test collections, or alternatively the effect size with only 10 test
collections, given the effect variances we see in the stemming experiment.
The power for the random-effects model that the mean effect is
different than zero is
p = 1 − Φ(c α − λ)
where Φ is the standard normal cumulative distribution function
and c α is the critical value. For α = 0.05, c α = 1.96 for the two-
tailed test we are doing here. λ is the extent to which the null
hypothesis is false, which we compute from the observed effects
and variances:
p all the individual test collections.
λ = Y / v ∗ /k
v ∗ = v̄ + var yi
Y is the mean effect, and k is the number of test collections we’ve
used. v∗ is the variance component and because we are using a
random-effects model, it is the sum of the average variance within
each test collection and the variance in the effect size across test
collections.
The observed power of the stemming experiment with four test
collections is 16.5%, and 18.9% with ten test collections. If we work
with the effects and variances we see in the ten-collection experi-
ment, we can ask about the design of a follow-on experiment with
more studies or a different target effect size.
Figure 4 illustrates how we might design the experiments differ-
ently, expecting that the individual test collections we would use
would have similar properties. The left-hand plot shows how power
would increase given more test collections, and the right-hand plot
shows how power would increase if we tried to detect a larger
effect size. We can see that we would have to use 60 or more test
collections to get 80% power for a 0.016 mean effect size, but if we
could observe effect sizes of 0.05 or greater, we could achieve 80%
power with these ten test collections only.
CIKM’18, October 22-26, 2018, Torino, Italy
1.0
0.8
Power (10 test collections)
0.6
0.4
0.2
100 0.05 0.10 0.15 0.20 0.25
Effect size
7 DISCUSSION
Meta-analysis is essentially performing the same function as score
standardization: scores are transformed by dividing them by the
variance in the test collection. However, there are no fixed standard-
ization factors, rather the standardization is done for the data at
hand, specifically in order to show the effect size of the experimen-
tal condition on the same scale in each test collection. Additionally,
the variance in each collection and between the collections is used
to compute a weighted average effect summarizing the results of
Because meta-analysis and score standardization are equivalent,
as far as computing the effect size within each experiment, what is
there to be gained here? Meta-analysis provides a statistical frame-
work for asking about the overall effect given many test collections.
The mean effect is understandable either from the perspective of
null-hypothesis testing, or interpreting confidence intervals.
We were forced to drop the TREC 2017 Dynamic Domain collec-
tion from the score standardization analysis because of concerns
regarding the existing runs we would use to compute the standard-
ization factors: the collection was built with active learning instead
of pooling, and only a few systems participated in the track, so the
set of runs to standardize from is limited. This is not a problem
for meta-analysis, because as long as the collection is sufficiently
complete to measure effectiveness on the baseline and experimental
condition runs, the effect size is still meaningful. If the collection
were missing judgments for many highly-ranked retrieved docu-
ments in either the baseline or experimental condition, the effect
size would be in doubt and we would not advise using the collection
in any analysis.
In meta-analysis generally there are a great number of choices to
make regarding effect sizes, models, etc. [3] In this paper we concen-
trate on the raw mean difference as the effect size, do not assume
720
Session 5A: Evaluation in IR 2
the variance of the effect size is constant among test collections,
employ a random-effects model to allow our estimate of the true
effect size to lie within an interval, look at a single experimental
variable, and employ traditional estimators where we need them.
We recommend that IR experiments generally follow this approach
except that multiple variables could be of interest. Industry studies
which might consider a binary response like search success would
compute the effect size differently.
Traditional meta-analysis is done as part of a systematic review
of past literature. This is complicated in information retrieval, where
many studies make use of the same test collections, but may have
other differences in systems used or parameters set. A future step
for meta-analysis in IR would be to show how it can be used to
understand variations in performance from multiple systems that
claim to implement the łsamež algorithm.
Another question which confronts IR is that of additivity: should
we expect to see such-and-such effect size when applied to a dif-
ferent baseline? Research using poor baselines has historically led
to exaggerated claims of improved effectiveness. Armstrong et al.
[2] surveyed more than 100 articles in SIGIR 1998-2008 and CIKM
2003-2008 to compare choices in baselines, and found that often
authors employ weak baselines, often not even competitive with
official TREC results. (Unfortunately, their survey did not standard-
ize scores, although some of their other experiments do.) This work
was extended by Kharazmi et al. [11] to explore more recent IR
methods like result diversification. Their analyses of additivity are
complicated by the lack of strong comparisons across multiple test
collections. Meta-analysis can’t solve the problem of choosing a
weak baseline, but by trying to derive a general effect size of an IR
feature over multiple test collections, meta-analysis could bring us
a step closer to understanding additivity.
Cormack and Lynam [6] also suggested meta-analysis as an alter-
native to averaging per-topic scores within a single test collection,
but we have not investigated that here. We could consider the mea-
sure of a single topic to be drawn from the set of judged documents
retrieved between the baseline and the treatment run. Within-topic
variance could come from observations of differing relevance judg-
ments or from models of assessor disagreement. Topics would not
need to be scored by a common metric, but instead each topic could
use the measure best suited to that information need. Meta-analysis
would scale the per-topic scores and then output a weighted average
as the summary treatment effect.
8 CONCLUSION
Information retrieval experiments are best done using multiple test
collections, but the community has lacked a framework for making
comparisons across multiple test collections in a statistically sound
way. Score standardization as proposed by Webber et al. [25] was
an important step, but didn’t say anything about how inferences
should be subsequently drawn from the data. If we accept score
standardization as an approach, then meta-analysis completes that
step. Meta-analysis provides a method to obtain a summary effect
and to bound that in a confidence interval.
We have shown this with two demonstrator experiments, one
on stemming and one on pseudo-relevance feedback. The meta-
analysis approach agrees with the analysis by score standardization,
721
CIKM’18, October 22-26, 2018, Torino, Italy
but goes further in providing a summary overall effect that we might
expect from these algorithms on similar sets of test collections.
We have also demonstrated using meta-analysis that experi-
menters do very much need to use multiple test collections, and
that results from two or three very similar collections should be
suspect. Existing methodologies did not support drawing a conclu-
sion from the suite of test collection experiments, but meta-analysis
does.
A feature of the meta-analysis method is that baselines are an
explicit component of the analysis; the effect size is always with
respect to a specified baseline. The summary effect is only with
respect to that baseline, and there is no implication that a result
generalizes beyond the application analyzed. If a researcher wants
to claim that an improvement is orthogonal to other system com-
ponents and would yield comparable effects when added to any
system, the onus is on the researcher to demonstrate that. The
meta-analysis makes it clear whether additivity is the subject of
the experiment.
This paper has only presented the simplest meta-analysis models.
Meta-analysis as a field of statistics has been around since at least
the 1980s and very active starting in the 2000s, as computational
methods have made systematic reviews easier to manage. There are
new methods being invented all the time. We feel that the control
we as experimenters bring to the IR experiment mean that older
models and estimation methods are robust, but exploring the space
of meta-analytic methods is part of our planned future work.
Like any subfield of statistics, meta-analysis is not without its
detractors. When meta-analysis is used to summarize existing re-
search studies, it can be difficult to anticipate issues of bias in results,
and there is much work in the meta-analysis literature trying to
solve that problem. We feel that in information retrieval exper-
iments, using meta-analysis to integrate multiple test collection
experiments for which we have total control, and can completely
observe effects, variances, and design decisions, meta-analysis is ro-
bust and provides distinct advantages over alternative approaches.
9 ACKNOWLEDGEMENTS
We deeply thank Sauparna Palchowdhury for the initial work on
this study, and Andrew Trotman, Matt Crane, and the RIGOR work-
shop for helpful advice on using the ATIRE system. We also thank
the reviewers of several venues for their comments on earlier ver-
sions of this paper.
REFERENCES
[1] Jaime Arguello, Fernando Diaz, Jimmy Lin, and Andrew Trotman. 2015. SIGIR
2015 Workshop on Reproducibility, Inexplicability, and Generalizability of Re-
sults (RIGOR). In Proceedings of the 38th International ACM SIGIR Conference on
Research and Development in Information Retrieval (SIGIR ’15). ACM, New York,
NY, USA, 1147ś1148. https://doi.org/10.1145/2766462.2767858
[2] Timothy G. Armstrong, Alistair Moffat, William Webber, and Justin Zobel. 2009.
Improvements That Don’t Add Up: Ad-hoc Retrieval Results Since 1998. In Pro-
ceedings of the 18th ACM Conference on Information and Knowledge Management
(CIKM ’09). ACM, New York, NY, USA, 601ś610. https://doi.org/10.1145/1645953.
1646031
[3] Michael Borenstein, Larry V. Hedges, Julian P. T. Higgins, and Hannah R. Roth-
stein. 2009. Introduction to Meta-Analysis. John Wiley & Sons, Ltd.
[4] Chris Buckley. 2009. Why Current IR Engines Fail. Inf. Retr. 12, 6 (Dec. 2009),
652ś665. https://doi.org/10.1007/s10791-009-9103-2
[5] Chris Buckley and Ellen M. Voorhees. 2000. Evaluating Evaluation Measure
Stability. In Proceedings of the 23rd Annual International ACM SIGIR Conference on
Session 5A: Evaluation in IR 2
Research and Development in Information Retrieval (SIGIR ’00). ACM, New York,
NY, USA, 33ś40. https://doi.org/10.1145/345508.345543
[6] Gordon V. Cormack and Thomas R. Lynam. 2006. Statistical Precision of Infor-
mation Retrieval Evaluation. In Proceedings of the 29th Annual International ACM
SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’06).
ACM, New York, NY, USA, 533ś540. https://doi.org/10.1145/1148170.1148262
[7] Rebecca DerSimonian and Nan Laird. 2015. Meta-analysis in clinical trials re-
visited. Contemporary Clinical Trials 45, 0 0 (2015), 139ś145. https://doi.org/10.
1016/j.cct.2015.09.002
[8] Joel L. Fagan. 1987. Automatic Phrase Indexing for Document Retrievals: An
Examination of Syntactic and Non-Syntactic Methods. In Proceedings of the 10th
Annual Conference on Research and Development in Information Retrieval (SIGIR
1987). New Orleans, LA, 91ś101.
[9] John Guiver, Stefano Mizzaro, and Stephen Robertson. 2009. A Few Good Topics:
Experiments in Topic Set Reduction for Retrieval Evaluation. ACM Trans. Inf. Syst.
27, 4, Article 21 (Nov. 2009), 26 pages. https://doi.org/10.1145/1629096.1629099
[10] Donna Harman. 1987. A Failure Analysis on the Limitations of Suffixing in an
Online Environment. In Proceedings of the 10th Annual Conference on Research and
Development in Information Retrieval (SIGIR 1987). New Orleans, LA, 102ś108.
[11] Sadegh Kharazmi, Falk Scholer, David Vallet, and Mark Sanderson. 2016. Exam-
ining Additivity and Weak Baselines. ACM Trans. Inf. Syst. 34, 4, Article 23 (June
2016), 18 pages. https://doi.org/10.1145/2882782
[12] Robert Krovetz. 1993. Viewing morphology as an inference process.. In Pro-
ceedings of the 16th Annual International ACM SIGIR Conference on Research and
Development in Information Retrieval. ACM, New York, NY, 191ś202.
[13] Keith O’Rourke. 2007. An historical perspective on meta-analysis: dealing quan-
titatively with varying study results. Journal of the Royal Society of Medicine 100,
12 (Dec 2007), 579ś582. https://dx.doi.org/10.1258/jrsm.100.12.579
[14] Martin F. Porter. 1980. An algorithm for suffix stripping. Program 14, 3 (1980),
130ś137.
[15] Stephen E. Robertson and Stephen Walker. 1999. Okapi/Keenbow at TREC-8.
In Proceedings of the Eighth Text Retrieval Conference (TREC-8). 151ś162. https:
//trec.nist.gov/pubs/trec8/papers/okapi.pdf
[16] Joseph J. Rocchio. 1965. Relevance Feedback in Information Retrieval. In
Information Storage and Retrieval (ISR-9), Gerard Salton (Ed.). NSF. http:
//sigir.org/files/museum/pub-08/XXIII-1.pdf
722
CIKM’18, October 22-26, 2018, Torino, Italy
[17] Tetsuya Sakai. 2006. Evaluating Evaluation Metrics Based on the Bootstrap. In
Proceedings of the 29th Annual International ACM SIGIR Conference on Research
and Development in Information Retrieval (SIGIR ’06). ACM, New York, NY, USA,
525ś532. https://doi.org/10.1145/1148170.1148261
[18] Tetsuya Sakai. 2016. A Simple and Effective Approach to Score Standardisation. In
Proceedings of the 2016 ACM International Conference on the Theory of Information
Retrieval (ICTIR ’16). ACM, New York, NY, USA, 95ś104. https://doi.org/10.1145/
2970398.2970399
[19] Gianmaria Silvello, Riccardo Bucco, Giulio Busato, Giacomo Fornari, Andrea
Langeli, Alberto Purpura, Giacomo Rocco, Alessandro Tezza, and Maristella
Agosti. 2018. Statistical Stemmers: A Reproducibility Study. In Advances in
Information Retrieval - 40th European Conference on IR Research, ECIR 2018, Greno-
ble, France, March 26-29, 2018, Proceedings. 385ś397. https://doi.org/10.1007/
978-3-319-76941-7_29
[20] Jasmeet Singh and Vishal Gupta. 2016. Text Stemming: Approaches, Applications,
and Challenges. ACM Comput. Surv. 49, 3, Article 45 (Sept. 2016), 46 pages.
https://doi.org/10.1145/2975608
[21] Amit Singhal. 1997. AT&T at TREC-6. In Proceedings of the Sixth Text Retrieval
Conference (TREC-6). 215ś226. http://trec.nist.gov/pubs/trec6/papers/att.ps.gz
[22] Jeffrey C. Valentine, Therese D. Pigott, and Hannah R. Rothstein. 2010. How
Many Studies Do You Need? A Primer on Statistical Power for Meta-Analaysis.
Journal of Educational and Behavioural Sciences 35, 2 (2010), 215ś247. https:
//doi.org/10.3102/1076998609346961
[23] Ellen M. Voorhees. 2002. The Philosophy of Information Retrieval Evaluation. In
Evaluation of Cross-Language Information Retrieval Systems, Carol Peters, Martin
Braschler, Julio Gonzalo, and Michael Kluck (Eds.). Springer Berlin Heidelberg,
Berlin, Heidelberg, 355ś370.
[24] Ellen M. Voorhees and Donna K. Harman (Eds.). 2005. TREC: Experiment and
Evaluation in Information Retrieval. MIT Press.
[25] William Webber, Alistair Moffat, and Justin Zobel. 2008. Score Standardization
for Inter-collection Comparison of Retrieval Systems. In Proceedings of the 31st
Annual International ACM SIGIR Conference on Research and Development in
Information Retrieval (SIGIR ’08). ACM, New York, NY, USA, 51ś58. https:
//doi.org/10.1145/1390334.1390346