Session 8: Users and Evaluation II
Generalising Kendall’s Tau
for Noisy and Incomplete Preference Judgements
Riku Togashi
Mercari Inc., Waseda University
Tokyo, Japan
riktor@mercari.com
ABSTRACT
We propose a new ranking evaluation measure for situations where
multiple preference judgements are given for each item pair but
they may be noisy (i.e., some judgements are unreliable) and/or
incomplete (i.e., some judgements are missing). While it is generally
easier for assessors to conduct preference judgements than absolute
judgements, it is often not practical to obtain preference judgements
for all combinations of documents. However, this problem can
be overcome if we can effectively utilise noisy and incomplete
preference judgements such as those that can be obtained from
crowdsourcing. Our measure, η, is based on a simple probabilistic
user model of the labellers which assumes that each document is
associated with a graded relevance score for a given query. We
also consider situations where multiple preference probabilities,
rather than preference labels, are given for each document pair.
For example, in the absence of manual preference judgements, one
might want to employ an ensemble of machine learning techniques
to obtain such estimated probabilities. For this scenario, we propose
another ranking evaluation measure called ηp . Through simulated
experiments, we demonstrate that our proposed measures η and
ηp can evaluate rankings more reliably than τ -b, a popular rank
correlation measure.
CCS CONCEPTS
• Information systems → Retrieval effectiveness; Relevance
assessment;
KEYWORDS
crowdsourcing, evaluation measures, graded relevance, preference
judgements
ACM Reference format:
Riku Togashi and Tetsuya Sakai. 2019. Generalising Kendall’s Tau for Noisy
and Incomplete Preference Judgements. In Proceedings of The 2019 ACM
SIGIR International Conference on the Theory of Information Retrieval, Santa
Clara, CA, USA, October 2–5, 2019 (ICTIR ’19), 4 pages.
https://doi.org/10.1145/3341981.3344246
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or a
fee. Request permissions from permissions@acm.org.
ICTIR ’19, October 2–5, 2019, Santa Clara, CA, USA
© 2019 Association for Computing Machinery.
ACM ISBN 978-1-4503-6881-0/19/10. . . $15.00
https://doi.org/10.1145/3341981.3344246
193
ICTIR '19, October 2-5, 2019, Santa Clara, CA, USA
Tetsuya Sakai
Waseda University
Tokyo, Japan
tetsuyasakai@acm.org
1 INTRODUCTION
Collecting relevance assessments is a crucial factor in information
retrieval evaluation. However, as it is widely known, manual rel-
evance assessments are costly and do not scale well; the burden
on the labellers is especially high if graded relevance [6] is utilised.
There are many more innovations that may help researchers over-
come this problem: the first is the use of crowdsourcing [4], and
the second is the use of preference judgements [2].
Crowdsourced relevance assessments are cheap and fast, and
therefore it is not difficult to obtain labels from multiple crowd
workers for the same target item to judge. However, the downside
is that crowdsourced assessments are generally noisy, and even
after applying a few tricks for filtering out low-quality labels, the
labels from multiple labellers may disagree with one another. One
simple approach to handle the disagreements would be to employ
majority voting, but this means losing information about the origi-
nal distribution of the labels. Approaches to preserving the original
distribution for the evaluation (See [5][6]) would be useful.
On the other hand, collecting preference judgements, rather than
absolute relevance judgements is advantageous especially from the
reliability point of view [2]. Generally speaking, it is easier for a
labeller to decide which of the two documents presented side-by-
side is more relevant than to decide whether a single document is
relevant or not relevant. However, due to combinatorial explosion,
obtaining preference judgements for all possible pairs of documents
is not practical. Hence, if we are to leverage preference judgements
to evaluate a ranked list of documents, the measure should be robust
to incompleteness. That is, it should be able to evaluate rankings
even if some preference judgements are missing.
Based on the above considerations, the present study proposes
a new ranking evaluation measure akin to Kendall’s τ , which can
handle noise and incompleteness when multiple preference judge-
ments are available per document pair. Our proposed measure, η,
is based on a simple probabilistic user model of the labellers, and
assumes that each document is associated with a graded relevance
score for a given query. We also consider situations where multiple
preference probabilities, rather than preference labels, are given
for each document pair. For example, in the absence of manual
preference judgements, one might want to employ an ensemble of
machine learning techniques to obtain such estimated probabilities.
For this scenario, we propose another ranking evaluation measure
called ηp . Through simulated experiments, we demonstrate that our
proposed measures η and ηp can evaluate rankings more reliably
than τ -b, a popular rank correlation measure.
Session 8: Users and Evaluation II ICTIR '19, October 2-5, 2019, Santa Clara, CA, USA

2 RELATED WORK By letting δi j = 1i j − 1ji , δi j can be written as follows.

The original Kendall’s τ is computed simply by subtracting the num- (
1 i is ranked higher than j
ber of discordant pairs from the number of concordant pairs and δi j = (8)
then applying normalisation so that its range becomes [−1, 1]. In −1 i is ranked lower than j
the present study, we use a popular variant, τ -b, which can handle We obtain the following measure η̂ which can be regarded as a
ties in the ranking, as the baseline measure. There are several vari- generalisation of Kendall’s τ with probabilistic preference labels.
ants of Kendall’s τ . Yilmaz et al. [8] developed τap measure based N −1 ÕN
on Kendall’s τ , which has a probabilistic interpretation and a top- 2 Õ
η̂ = (2pi j − 1)δi j (9)
weighted property. Webber et al. [7] proposed RBO (Rank-Biased N (N − 1) i=1 j=i+1
Overlap), a top-weighted measure that can handle nonconjoint
and incomplete rankings. However, these measures cannot handle When pi j for each documents pair (i, j) is fixed, η̂ has the max-
distributions of multiple judgements for a documents pair. Mad- imum value which can be obtained when the ranking satisfies
dalena et al. [5] proposed AANDCG (Agreement Aware NDCG) δi j = siдn(2pi j − 1) for all combinations (i, j). However, when the
measure, which considers the agreement between labellers, espe- preferences given by comparisons are not transitive, there is no
cially in low-quality data collection environments such as those ranking that actually satisfies this.
based on crowdsourcing. Although AANDCG takes into considera- Here, by substituting δi j to ∆i j = siдn(2pi j − 1) in Eq.(9), we
tion the distribution of graded relevance given by multiple labellers, obtain the ideal value of η:
N N
−1 Õ
to the best of our knowledge, there is no measure that leverages 2 Õ
the distribution of preference judgements. η∗ = (2pi j − 1)∆i j (10)
N (N − 1) i=1 j=i+1
3 PROPOSED MEASURES Our proposed measure η is obtained by normalising η̂ by η ∗ .
3.1 η measure η̂
η= ∗ (11)
For a given query, we hire n labellers and let them independently η
make a preference judgement on each document pair (i, j): 3.2 Interpretation of η
η is based on a simple user model of labellers to handle graded
(
(k ) 1 labeller k prefers i over j
vi j = (1) relevance. Let µ i∗ denote the true relevance grade for document
0 labeller k prefers j over i i (i = 1, . . . , N ). In our labeller model, a labeller compares two
for k = 1, . . . , n. As a result, the probability pi j of “labellers prefer documents (i, j) in the process of creating a preference judgement
i over j” can be computed as the average of the observed votes as and chooses document i that they prefers over document j with the
follows. following probability.
n µ∗
pi j =
1 Õ (k ) p(i ≻ j) = ∗ i ∗ (12)
n
vi j (2) µi + µ j
k =1
Thus, the winning rate p(i ≻ j) is generated by the Bradley-Terry
We regard this probability as a preference label that means “i should model [1] under the assumption that preferences are transitive, as
be ranked higher than j in a ranking”. The number of concordant in the case with graded relevance. In this model, the winning rate
pairs nc and the number of disconcordant pairs nd in the ranking varies depending on the magnitude of the difference of relevance
can be expressed using these probability labels. between a document pair. This generalises the preference label
N
Õ −1 N
Õ which traditional Kendall’s τ and τ -b rely on. In a situation where
nc = (pi j 1i j + (1 − pi j )1ji ) (3) multiple preference judgements per item are available, τ may be
i=1 j=i+1 used by applying a majority vote to produce the final verdict, and
N
Õ −1 N
Õ then use this as a single preference label. However, this approach
nd = (pi j 1ji + (1 − pi j )1i j ) (4) ignores the original distribution of votes, even if they are nearly
i=1 j=i+1 tied. In such cases, majority voting will have strong adverse effects
( on the reliability of τ , as we shall demonstrate later.
1 i is ranked higher than j
1i j = (5) 3.3 ηp measure
0 i is ranked lower than j
In the absence of manual preference judgements for a query, semi-
where N is the number of documents to be ranked. automatic evaluation by pseudo-labeling is a promising approach
Then, the difference of nc and nd is given by: to low-cost evaluation in large datasets. In this situation, multiple
N
Õ −1 N
Õ preference probabilities, rather than preference labels, are given
= for each document pair as predicted values of machine learning


nc − nd (2pi j − 1)1i j − (2pi j − 1)1ji (6)
i=1 j=i+1 techniques. In general, although the evaluation utilising predicted
N
Õ −1 N
Õ preference labels are less accurate than human-in-the-loop evalua-
= (2pi j − 1)(1i j − 1ji ) (7) tion, it is possible to make a safe decision by computing uncertainty
i=1 j=i+1 with techniques such as Bayesian inference or ensemble of multiple

194
Session 8: Users and Evaluation II
models. For this scenario, we also propose a measure for reliably
evaluating a given ranking.
(t )
Suppose that the prediction probability of the t-th model pi j (t =
1, . . . , n) is given for each documents pair (i, j). In Eq.(9), we now
treat the expectation E[pi j ] as the final probabilistic label for each
document pair, instead of pi j . Hence, the aforementioned η̂ measure
can be rewritten as follows.
N −1 ÕN
2 Õ (2E[pi j ] − 1)δi j
η̂p = (13)
N (N − 1) i=1 j=i+1 1 + V [pi j ]
(t )
where V [pi j ] is the variance of the predicted probabilities pi j ,
which can be used as the uncertainty of prediction. Note that
2E[pi j ] − 1, the gain for (i, j), also handles the uncertainty of pref-
erence, because the closer E[pi j ] is to 0.5 (tie), the closer this term
is to 0. Thus, the gain for a documents pair (i, j) is discounted by
the risk V [pi j ] while avoiding division by zero.
η̂p also has the maximum value when a ranking satisfies δi j =
siдn(2E[pi j ] − 1). Hence, we can define ηp , by normalising η̂p by
ηp as follows.
η̂p
ηp = ∗ (14)
ηp
4 EVALUATING THE PROPOSED MEASURES
This section evaluates the reliability of our proposed measures
through simulation.
4.1 Artificial Data
In each trial of our simulation, we consider д = 4 relevance grades
and N = 50 documents, and for each document we assign a true
relevance grade µ k∗ (k = 1, . . . , N ) as follows.
r ∼ U ni f orm(0, д) , µ k∗ = ⌊r ⌋ (15)
Since τ -b for any document ranking can be computed based on
the number of concordant pairs and the number of discordant pairs
relative to the gold pairwise preferences, a true τ -b score for any
ranking can be obtained based on the gold preferences derived from
the above true relevance grades.
(t )
On the other hand, we obtain artificial human votes {vi j |t =
1, . . . , n} as: ! 50 ∗ 49/2 = 1, 225 preferences. The y axis is the correlation with
(t ) µ i∗
vi j ∼ Bern ∗ (16)
µ i + µ ∗j
Given this noisy data, a τ -b score for any ranking can be computed
by first taking a majority vote of the above votes to obtain a prefer-
ence label for each document pair. Compared to the true τ -b, this
τ -b reflects the noise in the labels. In contrast, from the above votes,
we can compute an η score for any ranking directly.
In each trial, we generated 500 different document rankings by
degrading an ideal ranked list (as defined according to the true rele-
vance grades) to different degrees: we randomly applied S pairwise
document swaps (S = 1, . . . , 20) to the original ideal list, and 25
different rankings were obtained for each S. We then computed
the above three measures for each of the 20 ∗ 25 = 500 rankings,
and then the Pearson correlation between the true τ -b and the τ -b
(based on noisy labels), and that between the true τ -b and η. Hence
the Pearson correlations reflect the robustness of τ -b and that of η
when noise is introduced to labels.
195
ICTIR '19, October 2-5, 2019, Santa Clara, CA, USA
The above trial is repeated 100 times to obtain 100 Pearson cor-
relation values for τ -b and for η.
4.2 Number of Labellers
First, we examine the effect of the number of labellers n on the
robustness of η, as well as that of τ -b. Figure 1 compares the Pearson
correlation between η and the true τ -b and that between τ -b and the
true τ -b, while varying n. It can be observed that τ -b is unreliable
when the number of votes n is a small even number, because ties
can occur often and therefore majority voting introduces a lot of
noise in such cases. While we should avoid hiring an even number
of assessors if majority votes will be taken, this cannot always be
ensured due to (say) exclusions of data from low quality labellers.
Hence, this is a problem for τ -b. In contrast, it can be observed that
η consistently outperforms τ -b.
Figure 1: Effect of the Number of labellers on the Pearson
Correlation with true τ -b. The error bars represent 95% CIs.
4.3 Abused Preferences
In crowdsourcing, there might be abusers in labellers. This may be
a challenge for η as it does not ignore minority votes. Hence, we
investigate how η is affected by such abused votes. More specifically,
we define abused preferences to be those that oppose the true
preferences, and examine the effect of the amount of abuser votes
on the reliability of η.
Figure 2 shows the effect of abused preferences. The x axis is the
number of abused preferences we intentionally injected among the
the true τ -b as before. It can be observed that, while η is at least as
accurate as τ -b when the number of abused preferences is relatively
small, it degrades more rapidly than τ -b as this number increases, as
was expected. However, it can be observed that η performs at least
as well as τb when the number of abused preferences is less than
700 (out of 1,225). Since abusers should not dominate in a practical
crowdsourcing experiment, the above property of η probably does
not pose any practical threat.
4.4 Inferring Preferences
This section discusses the robustness of η when the number of
available preference judgements is limited. To do this, we set up
a simulation environment similar to the one described earlier, but
gradually reduce the number of observed preferences from 1,225,
while holding the number of labellers to a constant (n = 3). Given
such a situation, it is possible to estimate the relevance grades of
unobserved documents from observed preferences; we apply this
relevance estimation approach to both η and τ -b. Given the set of
Session 8: Users and Evaluation II ICTIR '19, October 2-5, 2019, Santa Clara, CA, USA

4.5 Using Predicted Preference Probabilities

We finally compare ηp with η when predicted preference proba-
bilities from multiple models are available: the basic simulation
conditions are similar to before, but here we assume that each
model introduces uncertainty to the preference probabilities, as
follows. (t ) (t ) (t )
νi j ∼ U ni f orm(0, 1.0) , ϵi j ∼ N (νi j , 0.1) (20)
µ∗
pi j = ∗ i ∗ + ϵi j
(t ) (t )
(21)
µi + µ j
Then, for each document ranking, ηp is computed as defined in
Figure 2: Effect of the number of abused preferences on the Section 3.3. As for η, since it requires an exact preference probability,
Pearson correlation with true τ -b, with 95% CIs. we let pi j = E[pi j ]. As before, we measure the Pearson correlation
with the true τ -b for 100 trials, each with 500 document rankings.
sum of observed votes W = {w i j |i = 1, . . . , N , j = 1, . . . , N , j ,
Figure 4 compares ηp with η while varying the number of models
i} where w i j = nt=1 vi j , we maximise the following likelihood
Í (t )
n; the y axis is the correlation for ηp minus that for η. Hence, the
function that is also based on the Bradley-Terry model: figure shows that ηp consistently outperforms η even though the
N −1 Ö N   wi j   w ji differences are very small. The results demonstrate that ηp can
Ö µi µj
P(W |µ 1 , . . . , µ N ) = (17) handle the uncertainty introduced by each model.
i=1 j=i+1
µi + µ j µi + µ j

To this end, we estimate each graded relevance score µ i using the

MM algorithm [3] through iteration as follows.
ÍN
(k +1) j=1 w i j
µi = ÍN n
(18)
j=1 µ (k ) +µ (k )
i j

The τ -b for a given ranking can then be computed based on prefer-

ences derived from the estimated graded relevance µˆi ; we denote
this as τ -b-MM. As for η, each probability label can be obtained
from µˆi as: µˆi
pˆi j = (19)
µˆi + µˆj
Then η can be computed for any given ranking based on the above;
we refer to this as η-MM. As before, the straight η, η-MM, and
τ -b-MM for 500 document rankings can be compared with the true
τ -b in terms of Pearson correlation, for 100 trials, each with 500
document rankings.
Figure 3 shows the effect of the number of judged document
pairs on the correlation with the the true τ -b for η, η-MM, and
τ -b-MM. It can be observed that all correlation values are quite
high even when the number of judged pairs is small, and more
importantly, that η-MM outperforms η, which in turn outperforms
τ -b-MM. That is, even without relevance estimation, η can provide
rankings that are more reliable than τ -b (with relevance estimation).
Figure 3: Effect of the number of judged documents pairs on
the correlation with true τ -b, with 95% CIs.
Figure 4: Effect of the number of models on the Pearson cor-
relation with true τ -b, with 95% CIs.
4.6 Conclusion
We proposed a ranking evaluation measure η for situations where
multiple preference judgements are given for each item pair, but
they may be noisy and/or incomplete. We also proposed ηp for sit-
uations where multiple preference probabilities are given. Through
simulation, we have demonstrated the reliability of our measures
in comparison to the popular τ -b.
The code for computing our proposed measures can be obtained
from a github repository (github.com/mercari/eta).
REFERENCES
[1] Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block
designs: I. The method of paired comparisons. Biometrika 39, 3/4 (1952), 324–345.
[2] Ben Carterette, Paul N Bennett, David Maxwell Chickering, and Susan T Dumais.
2008. Here or there. In European Conference on Information Retrieval. Springer,
16–27.
[3] David R Hunter et al. 2004. MM algorithms for generalized Bradley-Terry models.
The annals of statistics 32, 1 (2004), 384–406.
[4] Matthew Lease and Emine Yilmaz. 2013. Crowdsourcing for information retrieval:
introduction to the special issue. Information retrieval 16, 2 (2013), 91–100.
[5] Eddy Maddalena, Kevin Roitero, Gianluca Demartini, and Stefano Mizzaro. 2017.
Considering assessor agreement in ir evaluation. In Proceedings of the ACM SIGIR
International Conference on Theory of Information Retrieval. ACM, 75–82.
[6] Tetsuya Sakai. 2019. Graded Relevance Assessments and Graded Relevance Mea-
sures of NTCIR: A Survey of the First Twenty Years. CoRR abs/1903.11272 (2019).
arXiv:1903.11272 http://arxiv.org/abs/1903.11272
[7] William Webber, Alistair Moffat, and Justin Zobel. 2010. A similarity measure for
indefinite rankings. ACM TOIS 28, 4 (2010), 20.
[8] Emine Yilmaz, Javed A Aslam, and Stephen Robertson. 2008. A new rank cor-
relation coefficient for information retrieval. In Proceedings of the 31st annual
international ACM SIGIR conference on Research and development in information
retrieval. ACM, 587–594.

196