See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/327282582

Online Evaluation for Effective Web Service Development: Extended Abstract of

the Tutorial at TheWebConf'2018

Thesis · April 2018

DOI: 10.13140/RG.2.2.12362.62400/1

CITATIONS READS

4 469

6 authors, including:

Roman Budylin Alexey V. Drutsa

Yandex Lomonosov Moscow State University
14 PUBLICATIONS   26 CITATIONS    36 PUBLICATIONS   212 CITATIONS   

SEE PROFILE SEE PROFILE

Gleb Gusev Eugene Kharitonov

Yandex FB
45 PUBLICATIONS   422 CITATIONS    34 PUBLICATIONS   236 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

algebraic geometry View project

online evaluation View project

All content following this page was uploaded by Alexey V. Drutsa on 08 September 2018.

The user has requested enhancement of the downloaded file.

 Online Evaluation for Effective Web Service Development:
Extended Abstract of the Tutorial at TheWebConf’2018
Roman Budylin
Yandex; Moscow, Russia
budylin@yandex-team.ru
Eugene Kharitonov
Criteo Research; Paris, France
eugene.kharitonov@gmail.com
ABSTRACT
Development of the majority of the leading web services and soft-
ware products today is generally guided by data-driven decisions
based on evaluation that ensures a steady stream of updates, both in
terms of quality and quantity. Large Internet companies use online
evaluation on a day-to-day basis and at a large scale. The number of
smaller companies using A/B testing in their development cycle is
also growing. Web development across the board strongly depends
on quality of experimentation platforms. In this tutorial, we will
overview state-of-the-art methods underlying everyday evaluation
pipelines at some of the leading Internet companies.
We invite software engineers, designers, analysts, service or prod-
uct managers — beginners, advanced specialists, and researchers
— to join us at The Web Conference 2018, which will take place
in Lyon from 23 to 27 of April, to learn how to make web service
development data-driven and do it effectively.
KEYWORDS
online evaluation; A/B testing; online controlled experiment; inter-
leaving; industrial practice; online metrics
INTRODUCTION
Nowadays, the development of most leading web services and soft-
ware products, in general, is guided by data-driven decisions that
are based on online evaluation which qualifies and quantifies the
steady stream of web service updates. Online evaluation is widely
used in modern Internet companies (like search engines [15, 22, 30],
social networks [5, 64], media providers [5], and online retailers)
in permanent manner and on a large scale. Yandex run more than
100 online evaluation experiments per day; Bing reported on more
than 200 run A/B tests per day [42]; and Google conducted more
than 1000 experiments [30]. The number of smaller companies that
use A/B testing in the development cycle of their products grows
as well. The development of such services strongly depends on
the quality of the experimentation platforms. In this tutorial, we
Permission to make digital or hard copies of part or all of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
TheWebConf 2018, April 23–27, 2018, Lyon, France
© 2018 Copyright held by the owner/author(s).
ACM ISBN 0. . . $0.00
https://doi.org/0
Alexey Drutsa Gleb Gusev
Yandex; Moscow, Russia Yandex; Moscow, Russia
adrutsa@yandex.ru gleb57@yandex-team.ru
Pavel Serdyukov Igor Yashkov
Yandex; Moscow, Russia Yandex; Moscow, Russia
pavser@yandex-team.ru excel@yandex-team.ru
overview the state-of-the-art methods underlying the everyday
evaluation pipelines.
At the beginning of this tutorial, we will make an introduction to
online evaluation and will give basic knowledge from mathematical
statistics (50 min, Section 1). Then, we will share rich industrial
experiences on constructing of an experimentation pipeline and
evaluation metrics: emphasizing best practices and common pitfalls
(55 min, Section 2). This will be followed by approaches to develop-
ment of online metrics (55 min, Section 3) and by foundations of
interleaving (55 min, Section 4). A large part of our tutorial will be
devoted to modern and state-of-the-art techniques (including the
ones based on machine learning) that allow to conduct online ex-
perimentation efficiently (80 min, Section 5). Finally, we will point
out open research questions and current challenges that should be
interesting for research scientists.
1 STATISTICAL FOUNDATION
We introduce the main probabilistic terms, which form a theoretical
foundation of A/B testing. We introduce the observed values as
random variables sampled from an unknown distribution. Evalu-
ation metrics are statistics based on observations (mean, median,
quantiles, etc.). Overview of statistical hypothesis testing is pro-
vided with definitions of p-value, type I error, and type II error. We
discuss several statistical tests (Student’s t-test, Mann Whitney U,
and Bootstrap [26]), compare their properties and applicability [25].
2 EXPERIMENTATION PIPELINE AND
WORKFLOW IN THE LIGHT OF
INDUSTRIAL PRACTICE
We share rich industrial experiences on constructing of an experi-
mentation pipeline in large Internet companies. First, we introduce
the notion of an A/B test (also known as an online controlled exper-
iment): it compares two variants of a service at a time, usually its
current version (control) and a new one (treatment), by exposing
them to two groups of users [44, 45, 49]. The aim of controlled
experiments is to detect the causal effect of service updates on its
performance relying on an Overall Evaluation Criterion (OEC) [45],
a user behavior metric (e.g., clicks-per-user, sessions-per-user, etc.)
that is assumed to correlate with the quality of the service. Sec-
ond, we discuss how can experiments be used for evaluation of
changes in various components of web services (e.g., the user inter-
face [21, 23, 40, 48], ranking algorithms [21, 23, 48, 61], sponsored
search [10], and mobile apps [64]).
TheWebConf 2018, April 23–27, 2018, Roman
Lyon, France
Budylin, Alexey Drutsa, Gleb Gusev, Eugene Kharitonov, Pavel Serdyukov, and Igor Yashkov
Third, we consider several real cases of experiments, where
pitfalls [12, 17, 41, 43] are demonstrated and lessons are learned. In
particular, we discuss: conflicting experiments, network effects [28],
duration and seasonality [21, 60], logging, and slices. Fourth, we
provide our management methodology to conduct experiments
efficiently and to avoid the pitfalls. This methodology is based on
pre-launch checklists and a team of Experts on Experiments (EE).
Finally, we discuss how large-scale experimental infrastructure [42,
62, 65] can be used to collect experiments for metric evaluation [19].
3 DEVELOPMENT OF ONLINE METRICS
We deeply discuss how to build evaluation metrics, what is the main
ingredient in online experimentation pipeline. First, main compo-
nents of a metric are presented: key metric, evaluation statitistic,
statistical significance test, Overall Evaluation Criterion (OEC) [45],
and Overall Acceptance Criterion (OAC) [25]. We show that de-
velopment of a new good metric is a challenging goal, since an
appropriate OAC should possess two crucial qualities: direction-
ality (the sign of the detected treatment effect should align with
positive/negative impact of the treatment on user experience) and
sensitivity (the ability to detect the statistically significant differ-
ence when the treatment effect exists) [17, 41, 48, 51]. The former
property allows to make correct conclusions on the system quality
changes [17, 41, 48], while improvement of the latter one allows
to detect metric changes in more experiments and to utilize less
users [37, 38, 51].
Second, we provide evaluation criteria beyond averages: (a) how
to evaluate periodicity [20, 23, 24] and trends [20, 21, 23] of user be-
havior over days, e.g., for detection of delayed treatment effects; and
(b) how to evaluate frequent/rare behavior and diversity in behavior
between users that cannot be detected by mean values [25]. Third,
product-based aspects in metric building are presented. Namely,
we discuss vulnerability of metrics such as a click on a button to
switch a search engine [2] (i.e., how can a metric be gamed or
manipulated); ways to measure different aspects of a service (i.e.,
speed [43, 46], absence [8], abandonment [43]); difference between
metrics of user loyalty and ones of user activity [20–22, 25, 47, 57];
dwell time to improve click-based metrics [39]; how to evaluate the
number of user tasks (which can have a complex hierarchy [6]) by
means of sessions [61]; and issues in session division [34] as well.
Fourth, math-based approaches to improve metrics are consid-
ered. In particular, we discuss the powerful method of lineariza-
tion [7] that reduces any ratio metric to the average of a user-level
metric preserving directionality and allowing usage of a wide range
of sensitivity improvement techniques developed for user-level met-
rics. We also describe different methods of noise reduction (such as
capping [43], slicing [14, 61], taking into account user activity, etc.)
and of utilization of a user generated content approach.
Finally, some system requirements for metric building are dis-
cussed. We explain how to get a set of experiments with verdicts
(known positiveness or negativeness), how to construct a pipeline to
easily implement and test metrics, and how to measure metrics [19].
4 INTERLEAVING FOR ONLINE RANKING
EVALUATION
The online behavior might vary dramatically across the users and
queries. This can lead to significant noise in the online measure-
ments. The interleaving evaluation is an online evaluation tech-
nique that aims to eliminate these inter-user and inter-query vari-
abilities in the ranker comparisons. This is achieved by combining
results from the compared rankers in a single result page (inter-
leave). After that, the user’s click behavior is analyzed to infer the
preference direction.
The first interleaving method, Balanced Interleaving, was pro-
posed by Joachims et al. [31, 32]. Later, multiple other methods and
extensions were proposed, including Team Draft [55], Probabilistic
Interleave [29], Optimized Interleaving [53], Multileaving [59], and
Generalized Team Draft [36].
In this part of the tutorial, we start with discussing the motiva-
tion behind the interleaving methods. We introduce and illustrate
the core interleaving algorithms: Balanced Interleave [31], Team
Draft [55], and Probabilistic Interleave [29]. Further, we overview
some of the available experimental studies [4, 9] that compare in-
terleaving methods with offline and online evaluation approaches.
Finally, we introduce the extension of the interleaving methods
towards comparing multiple rankers, Multileaving [59].
Overall, the interleaving evaluation is a fruitful area of research
with numerous interesting ideas. Unfortunately, in the available
time frame, we are not able to cover all of them. Hence, we en-
courage an interested participant to look into other relevant tutori-
als [27, 52, 54, 56].
5 MACHINE LEARNING DRIVEN A/B
TESTING
A large part of our tutorial is devoted to modern and state-of-the-art
techniques (including the ones based on machine learning) that
improve the efficiency of online experiments. We start this section
with the comparison of randomized experiments and observational
studies. We explain that the difference between averages of the
key metric may be misleading when measured in an observational
study. We introduce the Neyman–Rubin model and rigorously for-
mulate implicit assumptions we make each time when evaluating
the results of randomized experiments.
Then we overview several studies devoted to the variance reduc-
tion of evaluation metrics. Regression adjustment techniques such
as stratification, linear models [18, 63], and gradient boosted deci-
sion trees [51] reduce the variance related to the observed features
(covariates) of the users.
We also consider experiments with user experience, where the
effect of a service change is heterogeneous (is different for users
of different types). We overview the main approaches to estima-
tion of the heterogeneous treatment effect depending on the user
features [3, 50].
We explain the Optimal Distribution Decomposition (ODD) ap-
proach that is based on the analysis of the control and treatment
distributions of the key metric as a whole, and, for this reason,
is sensitive to more ways the two distributions may actually dif-
fer [48].
Online Evaluation for Effective Web Service Development
Method of virtually increasing of the experiment duration through
the prediction of the future [22] is discussed. We also provide an-
other way to improve sensitivity that is based on learning of metric
combinations [35]. This approach showed outstanding sensitivity
improvements in the large scale empirical evaluation [35].
Finally, we discuss ways to improve the performance of experi-
mentation pipeline as a whole. Optimal scheduling of online evalu-
ation experiments is presented [37] and approaches for early stop-
ping of them are highlighted (where inflation of Type I error [33]
and ways to correctly make sequential testing [16, 38] are dis-
cussed).
We also highlight important topics not covered by the tutorial:
Bayesian approaches [13, 16] and non-parametric mSRPT [1] in
sequential testing; network A/B testing [28, 58]; two-stage A/B
testing [15] (also mentioned as tournaments in Sec. 2); and Imperfect
Treatment Assignment [11].
TUTORIAL MATERIALS
The tutorial materials (slides) are available at https://research.yandex.
com/tutorials/online-evaluation/www-2018.
REFERENCES
[1] Vineet Abhishek and Shie Mannor. 2017. A nonparametric sequential test for
online randomized experiments. In Proceedings of the 26th International Conference
on World Wide Web Companion. International World Wide Web Conferences
Steering Committee, 610–616.
[2] Olga Arkhipova, Lidia Grauer, Igor Kuralenok, and Pavel Serdyukov. 2015. Search
Engine Evaluation based on Search Engine Switching Prediction. In SIGIR’2015.
ACM, 723–726.
[3] Susan Athey and Guido Imbens. 2015. Machine Learning Methods for Estimating
Heterogeneous Causal Effects. arXiv preprint arXiv:1504.01132 (2015).
[4] Juliette Aurisset, Michael Ramm, and Joshua Parks. 2017. Innovating Faster on
Personalization Algorithms at Netflix Using Interleaving. https://medium.com/
netflix-techblog/interleaving-in-online-experiments-at-netflix-a04ee392ec55.
(2017).
[5] Eytan Bakshy and Dean Eckles. 2013. Uncertainty in online experiments with
dependent data: An evaluation of bootstrap methods. In KDD’2013. 1303–1311.
[6] Paolo Boldi, Francesco Bonchi, Carlos Castillo, and Sebastiano Vigna. 2009. From
Dango to Japanese cakes: Query reformulation models and patterns. In Proceed-
ings of the 2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence
and Intelligent Agent Technology-Volume 01. IEEE Computer Society, 183–190.
[7] Roman Budylin, Alexey Drutsa, Ilya Katsev, and Valeriya Tsoy. 2018. Consistent
Transformation of Ratio Metrics for Efficient Online Controlled Experiments. In
Proceedings of the Eleventh ACM International Conference on Web Search and Data
Mining. ACM, 55–63.
[8] Sunandan Chakraborty, Filip Radlinski, Milad Shokouhi, and Paul Baecke. 2014.
On correlation of absence time and search effectiveness. In SIGIR’2014. 1163–1166.
[9] Olivier Chapelle, Thorsten Joachims, Filip Radlinski, and Yisong Yue. 2012. Large-
scale validation and analysis of interleaved search evaluation. ACM Transactions
on Information Systems (TOIS) 30, 1 (2012), 6.
[10] Shuchi Chawla, Jason Hartline, and Denis Nekipelov. 2016. A/B testing of auctions.
In EC’2016.
[11] Dominic Coey and Michael Bailey. 2016. People and cookies: Imperfect treatment
assignment in online experiments. In Proceedings of the 25th International Con-
ference on World Wide Web. International World Wide Web Conferences Steering
Committee, 1103–1111.
[12] Thomas Crook, Brian Frasca, Ron Kohavi, and Roger Longbotham. 2009. Seven
pitfalls to avoid when running controlled experiments on the web. In KDD’2009.
1105–1114.
[13] Alex Deng. 2015. Objective Bayesian Two Sample Hypothesis Testing for Online
Controlled Experiments. In WWW’2015 Companion. 923–928.
[14] Alex Deng and Victor Hu. 2015. Diluted Treatment Effect Estimation for Trigger
Analysis in Online Controlled Experiments. In WSDM’2015. 349–358.
[15] Alex Deng, Tianxi Li, and Yu Guo. 2014. Statistical inference in two-stage online
controlled experiments with treatment selection and validation. In WWW’2014.
609–618.
[16] Alex Deng, Jiannan Lu, and Shouyuan Chen. 2016. Continuous Monitoring of
A/B Tests without Pain: Optional Stopping in Bayesian Testing. In DSAA’2016.
TheWebConf 2018, April 23–27, 2018, Lyon, France
[17] Alex Deng and Xiaolin Shi. 2016. Data-Driven Metric Development for Online
Controlled Experiments: Seven Lessons Learned. In KDD’2016.
[18] Alex Deng, Ya Xu, Ron Kohavi, and Toby Walker. 2013. Improving the sensitivity
of online controlled experiments by utilizing pre-experiment data. In WSDM’2013.
123–132.
[19] Pavel Dmitriev and Xian Wu. 2016. Measuring Metrics. In CIKM’2016. 429–437.
[20] Alexey Drutsa. 2015. Sign-Aware Periodicity Metrics of User Engagement for
Online Search Quality Evaluation. In SIGIR’2015. 779–782.
[21] Alexey Drutsa, Gleb Gusev, and Pavel Serdyukov. 2015. Engagement Periodicity in
Search Engine Usage: Analysis and Its Application to Search Quality Evaluation.
In WSDM’2015. 27–36.
[22] Alexey Drutsa, Gleb Gusev, and Pavel Serdyukov. 2015. Future User Engagement
Prediction and its Application to Improve the Sensitivity of Online Experiments.
In WWW’2015. 256–266.
[23] Alexey Drutsa, Gleb Gusev, and Pavel Serdyukov. 2017. Periodicity in User
Engagement with a Search Engine and its Application to Online Controlled
Experiments. ACM Transactions on the Web (TWEB) 11 (2017).
[24] Alexey Drutsa, Gleb Gusev, and Pavel Serdyukov. 2017. Using the Delay in a Treat-
ment Effect to Improve Sensitivity and Preserve Directionality of Engagement
Metrics in A/B Experiments. In WWW’2017.
[25] Alexey Drutsa, Anna Ufliand, and Gleb Gusev. 2015. Practical Aspects of Sensi-
tivity in Online Experimentation with User Engagement Metrics. In CIKM’2015.
763–772.
[26] Bradley Efron and Robert J Tibshirani. 1994. An introduction to the bootstrap.
CRC press.
[27] Artem Grotov and Maarten de Rijke. 2016. Online learning to rank for information
retrieval: Tutorial. In SIGIR.
[28] Huan Gui, Ya Xu, Anmol Bhasin, and Jiawei Han. 2015. Network a/b testing: From
sampling to estimation. In Proceedings of the 24th International Conference on
World Wide Web. International World Wide Web Conferences Steering Committee,
399–409.
[29] Katja Hofmann, Shimon Whiteson, and Maarten De Rijke. 2011. A probabilistic
method for inferring preferences from clicks. In Proceedings of the 20th ACM
international conference on Information and knowledge management. ACM, 249–
258.
[30] Henning Hohnhold, Deirdre O’Brien, and Diane Tang. 2015. Focusing on the
Long-term: It’s Good for Users and Business. In KDD’2015. 1849–1858.
[31] Thorsten Joachims. 2002. Unbiased evaluation of retrieval quality using click-
through data. (2002).
[32] Thorsten Joachims et al. 2003. Evaluating Retrieval Performance Using Click-
through Data. (2003).
[33] Ramesh Johari, Pete Koomen, Leonid Pekelis, and David Walsh. 2017. Peeking
at A/B Tests: Why it matters, and what to do about it. In Proceedings of the 23rd
ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.
ACM, 1517–1525.
[34] Rosie Jones and Kristina Lisa Klinkner. 2008. Beyond the session timeout: auto-
matic hierarchical segmentation of search topics in query logs. In Proceedings
of the 17th ACM conference on Information and knowledge management. ACM,
699–708.
[35] Eugene Kharitonov, Alexey Drutsa, and Pavel Serdyukov. 2017. Learning Sensitive
Combinations of A/B Test Metrics. In WSDM’2017.
[36] Eugene Kharitonov, Craig Macdonald, Pavel Serdyukov, and Iadh Ounis. 2015.
Generalized Team Draft Interleaving. In CIKM’2015.
[37] Eugene Kharitonov, Craig Macdonald, Pavel Serdyukov, and Iadh Ounis. 2015.
Optimised Scheduling of Online Experiments. In SIGIR’2015. 453–462.
[38] Eugene Kharitonov, Aleksandr Vorobev, Craig Macdonald, Pavel Serdyukov, and
Iadh Ounis. 2015. Sequential Testing for Early Stopping of Online Experiments.
In SIGIR’2015. 473–482.
[39] Youngho Kim, Ahmed Hassan, Ryen W White, and Imed Zitouni. 2014. Model-
ing dwell time to predict click-level satisfaction. In Proceedings of the 7th ACM
international conference on Web search and data mining. ACM, 193–202.
[40] Ronny Kohavi, Thomas Crook, Roger Longbotham, Brian Frasca, Randy Henne,
Juan Lavista Ferres, and Tamir Melamed. 2009. Online experimentation at Mi-
crosoft. Data Mining Case Studies (2009), 11.
[41] Ron Kohavi, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker, and Ya
Xu. 2012. Trustworthy online controlled experiments: Five puzzling outcomes
explained. In KDD’2012. 786–794.
[42] Ron Kohavi, Alex Deng, Brian Frasca, Toby Walker, Ya Xu, and Nils Pohlmann.
2013. Online controlled experiments at large scale. In KDD’2013. 1168–1176.
[43] R. Kohavi, A. Deng, R. Longbotham, and Y. Xu. 2014. Seven Rules of Thumb for
Web Site Experimenters. In KDD’2014.
[44] Ron Kohavi, Randal M Henne, and Dan Sommerfield. 2007. Practical guide to
controlled experiments on the web: listen to your customers not to the hippo. In
KDD’2007. 959–967.
[45] Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M Henne. 2009.
Controlled experiments on the web: survey and practical guide. Data Min. Knowl.
Discov. 18, 1 (2009), 140–181.
 TheWebConf 2018, April 23–27, 2018, Roman
Lyon, France
Budylin, Alexey Drutsa, Gleb Gusev, Eugene Kharitonov, Pavel Serdyukov, and Igor Yashkov

[46] Ron Kohavi, David Messner, Seth Eliot, Juan Lavista Ferres, Randy Henne, Vig-
nesh Kannappan, and Justin Wang. 2010. Tracking Users’ Clicks and Submits:
Tradeoffs between User Experience and Data Loss. (2010).
[47] Mounia Lalmas, Heather O’Brien, and Elad Yom-Tov. 2014. Measuring user
engagement. Synthesis Lectures on Information Concepts, Retrieval, and Services 6,
4 (2014), 1–132.
[48] Kirill Nikolaev, Alexey Drutsa, Ekaterina Gladkikh, Alexander Ulianov, Gleb
Gusev, and Pavel Serdyukov. 2015. Extreme States Distribution Decomposition
Method for Search Engine Online Evaluation. In KDD’2015. 845–854.
[49] Eric T Peterson. 2004. Web analytics demystified: a marketer’s guide to under-
standing how your web site affects your business. Ingram.
[50] Scott Powers, Junyang Qian, Kenneth Jung, Alejandro Schuler, Nigam H Shah,
Trevor Hastie, and Robert Tibshirani. 2017. Some methods for heterogeneous
treatment effect estimation in high-dimensions. arXiv preprint arXiv:1707.00102
(2017).
[51] Alexey Poyarkov, Alexey Drutsa, Andrey Khalyavin, Gleb Gusev, and Pavel
Serdyukov. 2016. Boosted Decision Tree Regression Adjustment for Variance
Reduction in Online Controlled Experiments. In KDD’2016. 235–244.
[52] Filip Radlinski. 2013. Sensitive Online Search Evaluation. http://irsg.bcs.org/
SearchSolutions/2013/presentations/radlinski.pdf. (2013).
[53] Filip Radlinski and Nick Craswell. 2013. Optimized interleaving for online re-
trieval evaluation. In WSDM.
[54] Filip Radlinski and Katja Hofmann. 2013. Practical online retrieval evaluation. In
ECIR.
[55] Filip Radlinski, Madhu Kurup, and Thorsten Joachims. 2008. How does click-
through data reflect retrieval quality?. In CIKM’2008. 43–52.
[56] Filip Radlinski and Yisong Yue. 2011. Practical Online Retrieval Evaluation. In
SIGIR.
[57] Kerry Rodden, Hilary Hutchinson, and Xin Fu. 2010. Measuring the user experi-
ence on a large scale: user-centered metrics for web applications. In CHI’2010.
2395–2398.
[58] Martin Saveski, Jean Pouget-Abadie, Guillaume Saint-Jacques, Weitao Duan,
Souvik Ghosh, Ya Xu, and Edoardo M Airoldi. 2017. Detecting network effects:
Randomizing over randomized experiments. In Proceedings of the 23rd ACM
SIGKDD international conference on knowledge discovery and data mining. ACM,
1027–1035.
[59] Anne Schuth, Floor Sietsma, Shimon Whiteson, Damien Lefortier, and Maarten
de Rijke. 2014. Multileaved comparisons for fast online evaluation. In CIKM.
[60] Milad Shokouhi. 2011. Detecting seasonal queries by time-series analysis. In
Proceedings of the 34th international ACM SIGIR conference on Research and devel-
opment in Information Retrieval. ACM, 1171–1172.
[61] Yang Song, Xiaolin Shi, and Xin Fu. 2013. Evaluating and predicting user engage-
ment change with degraded search relevance. In WWW’2013. 1213–1224.
[62] Diane Tang, Ashish Agarwal, Deirdre O’Brien, and Mike Meyer. 2010. Overlap-
ping experiment infrastructure: More, better, faster experimentation. In KDD’2010.
17–26.
[63] Huizhi Xie and Juliette Aurisset. 2016. Improving the Sensitivity of Online
Controlled Experiments: Case Studies at Netflix. In KDD’2016.
[64] Ya Xu and Nanyu Chen. 2016. Evaluating Mobile Apps with A/B and Quasi A/B
Tests. In KDD’2016.
[65] Ya Xu, Nanyu Chen, Addrian Fernandez, Omar Sinno, and Anmol Bhasin. 2015.
From infrastructure to culture: A/B testing challenges in large scale social net-
works. In KDD’2015.

View publication stats