EC’20 Session 2a: X: Evidence from Y

The Engagement-Diversity Connection: Evidence From a

Field Experiment on Spotify

DAVID HOLTZ, MIT Sloan School of Management, USA

BEN CARTERETTE, PRAVEEN CHANDAR, ZAHRA NAZARI, and HENRIETTE CRAMER,
Spotify, USA
SINAN ARAL, MIT Sloan School of Management, USA
We present results from a large-scale, randomized field experiment on Spotify testing the effect of personalized
recommendations on consumption diversity. In the experiment, both control and treatment users were given
podcast recommendations, with the sole aim of increasing podcast consumption. However, the recommenda-
tions provided to treatment users were personalized based on their music listening history, whereas control
users were recommended the most popular podcasts among users in their demographic group. Consistent
with previous studies, we find that the treatment increased the average number of podcast streams per user.
However, we also find the treatment decreased the average individual-level diversity of podcast streams and
increased the aggregate diversity of podcast streams, indicating that personalized recommendations have
the potential to create consumption patterns that are homogenous within users and diverse across users.
Our results provide evidence of an “engagement-diversity trade-off" when optimizing solely for increased
consumption: while personalized recommendations increase user engagement, they also affect the diversity
of content that users consume. This shift in consumption diversity can affect user retention and lifetime
value, and also impact the optimal strategy for content producers. Additional analyses suggest that exposure
to personalized recommendations can also affect the content that users consume organically. We believe
these findings highlight the need for both academics and practitioners to continue investing in approaches to
personalization that explicitly take into account the diversity of content recommended to users.
CCS Concepts: • Information systems; • Applied computing → Electronic commerce; • Computing
methodologies → Machine learning;
Additional Key Words and Phrases: recommender systems, diversity, field experiments
ACM Reference Format:
David Holtz, Ben Carterette, Praveen Chandar, Zahra Nazari, Henriette Cramer, and Sinan Aral. 2020. The
Engagement-Diversity Connection: Evidence From a Field Experiment on Spotify. In Proceedings of the 21st
ACM Conference on Economics and Computation (EC ’20), July 13–17, 2020, Virtual Event, Hungary. ACM, New
York, NY, USA, 2 pages. https://doi.org/10.1145/3391403.3399532

EXTENDED ABSTRACT
Despite recommender systems’ increasing ubiquity, the ways in which they impact the types
of choices we make are still not well understood. While some scholars have speculated that
recommender systems will lead to “filter bubbles", others hypothesize that recommender systems
will homogenize user consumption, leading to the “rich getting richer." In this paper, we analyze a
large scale field experiment conducted on Spotify, one of the world’s leading streaming platforms.
During the experiment, both treatment and control users were recommended podcasts with the sole
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee
provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and
the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses,
contact the owner/author(s).
EC ’20, July 13–17, 2020, Virtual Event, Hungary
© 2020 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-7975-5/20/07.
https://doi.org/10.1145/3391403.3399532

75
 EC’20 Session 2a: X: Evidence from Y

aim of increasing podcast consumption; while control users were recommended podcasts popular
amongst those in their demographic group, treatment users were provided fully personalized
recommendations based on their existing music listening history. We measure the impact of
more personalized content recommendations on user engagement, as well as individual-level and
aggregate podcast category diversity.
We find that the recommender system in the experiment increased the average number of podcasts
streamed per user by 28.90% relative to the less-personalized, popularity-based recommendation
strategy. We also test for the impact of the algorithm on user-level podcast category diversity, as
measured through the Shannon entropy, and aggregate podcast category diversity, as measured
through a quantity that we call “intragroup diversity" [2]. We find that the more personalized
algorithm decreased individual-level diversity by 11.51%, but increased intragroup diversity by 5.96%.
These results indicate that recommender systems and personalization algorithms have the capacity
to push individual users into homogeneous consumption patterns that are increasingly dissimilar
from those of their peers. While the effects of the treatment are largest for streams originating
from the section of Spotify’s app where personalized recommendations are delivered, we observe
evidence that the treatment also affected streams originating from other parts of the app. This
suggests that exposure to personalized recommendations can also affect the diversity of content
that users engage with organically.
In aggregate, our findings highlight the potential for recommender systems to create an “engagement-
diversity trade-off” for firms when optimizing solely for increased consumption; while algorithmic
recommendations can increase user engagement, they can also homogenize individual users’
consumption and Balkanize user content consumption. This shift in consumption diversity can
negatively impact user churn rates and lifetime values [1], and can impact the optimal strategy
for content creators. It is possible that our findings also extend to cases where diversity is mea-
sured with respect to the ideological slant or extremity of content. If so, this tradeoff suggests that
recommender systems that increase engagement/consumption can also create costs for firms, due
to the high level of public scrutiny given to personalized recommendations, and impact public
discourse on platforms through the creation of “filter bubbles." In light of our results, we believe it is
worthwhile for both academics and practitioners to continue developing personalization techniques
that explicitly take into account the diversity of content recommended to users.
Our work contributes to an emerging research literature that uses randomized field experiments
to measure the impact of recommender systems on the diversity of content that users consume
[3, 4]. Although at first glance, our findings are not entirely consistent with previous studies, we
believe this contrast is due to differences in how diversity is quantified, as well as differences in the
recommendation algorithms being used and the research settings in which the experiments are
conducted. These conflicting results highlight the importance of measuring the impact of many
different recommendation algorithms in a variety of settings, and the need to develop a number
of different methodological approaches for measuring the impact of recommendation systems on
consumption diversity. The full paper can be accessed online at https://bit.ly/2ZrGSBe.

REFERENCES
[1] Ashton Anderson, Lucas Maystre, Ian Anderson, Rishabh Mehrotra, and Mounia Lalmas. 2020. Algorithmic effects on
the diversity of consumption on spotify. In Proceedings of The Web Conference 2020. 2155–2165.
[2] Sinan Aral and Paramveer Dhillon. 2016. Unpacking novelty: The anatomy of vision advantages. Available at SSRN
2388254 (2016).
[3] Jörg Claussen, Christian Peukert, and Ananya Sen. 2019. The Editor vs. the Algorithm: Targeting, Data and Externalities
in Online News. Data and Externalities in Online News (June 5, 2019) (2019).
[4] Dokyun Lee and Kartik Hosanagar. 2019. How Do Recommender Systems Affect Sales Diversity? A Cross-Category
Investigation via Randomized Field Experiment. Information Systems Research 30, 1 (2019), 239–259.

76