See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/268688632

Large-scale Recommendations in a Dynamic Marketplace

Conference Paper · October 2013

CITATIONS READS
8 682

3 authors:

Jayasimha Reddy Katukuri Rajyashree Mukherjee

17 PUBLICATIONS 96 CITATIONS
eBay
9 PUBLICATIONS 52 CITATIONS
SEE PROFILE
SEE PROFILE

Tolga Könik
Apple Inc.
30 PUBLICATIONS 328 CITATIONS

SEE PROFILE

All content following this page was uploaded by Rajyashree Mukherjee on 25 November 2014.

The user has requested enhancement of the downloaded file.

 Large-scale Recommendations in a Dynamic Marketplace
Jayasimha Katukuri Rajyashree Mukherjee
eBay Inc. eBay Inc.
jkatukuri@ebay.com rmukherjee@ebay.com
ABSTRACT
We present a recommendation system architecture for dynamic
marketplaces. Our system addresses several challenges
recommendation engines face in an open market. It can handle
open-ended and rapidly changing user generated listings and the
absence of a catalog behind them. It can control the trade-off
between relevance and predicted quality. Quality is affected by
many factors like price, item condition and seller trustworthiness
and it is a particularly important challenge in an open market
setting. The core of our solution involves learned cluster
definitions that map dynamic user listings into static identifiers.
These cluster identifiers allow us to pre-compute long-term
models and keep the runtime system scalable. Our solution is
currently deployed on ebay.com and has shown statistically
significant business gain in site-wide metrics.
Keywords
Clustering, recommender systems, hadoop, map-reduce,
similarity-based recommendations.
1. Introduction
Recommender systems are gaining wide popularity in e-
commerce as they are becoming major drivers of incremental
business value and user satisfaction. Building a recommendation
engine for a large open marketplace like eBay has several
challenges. Majority of the listings are unstructured and do not
have product catalogues behind them. The listings are short-lived
as the items are often bought within 1-2 weeks of availability.
Hence pre-computing recommendations using traditional
techniques like item-to-item collaborative filtering [1] is not
feasible. On the other hand, a solution based on completely online
computation is not scalable. Another challenge in an open market
setting is that the recommendation systems also need to address
factors like seller trustworthiness and item quality.
We propose a recommender system architecture that is
scalable, recommends short-lived items and provides control over
multiple competing objectives such as relevance and quality. Our
solution learns cluster functions that maps dynamically changing
listings to static cluster identifiers. This allows us to separate
processing into computationally intensive offline modeling and
very efficient runtime performance. Offline processing builds
static models using clusters as the main language. Although
computationally expensive, it is a highly parallelizable procedure
since it is based on local clustering of items as partitioned by user
queries. The runtime system efficiently combines the cluster
models with dynamic features on the site. As a result, our system
can cover hundreds of millions of active items in the inventory
Copyright © LSRS 2013 for the individual papers by the papers’
authors. Copying permitted only for private and academic purposes. This
volume is published and copyrighted by its editors.
Tolga Konik
eBay Inc.
tkonik@ebay.com
while serving millions active users. Our contributions in this
paper are a highly scalable algorithm for clustering the inventory
using user queries as seeds, a recommender system architecture
for transient items in a marketplace and a method that can
negotiate between relevance and item quality.
Our system is currently deployed at ebay.com on a large scale
and has shown statistically significant positive impact on site-
wide performance. We present A/B test results showing increase
in user engagement. Our proposed method is general and it is
suitable for other e-commerce marketplaces where relevance and
quality play an important role.
2. Related Work
Recommender systems can be broadly categorized into ‘content-
based’ and ‘collaborative-filtering-based’ systems. Content-based
algorithms use the item features to compute similarity with respect
to other items and recommendations are based on this similarity.
Collaborative filtering methods compute item-item matrix using
the user behavioral data such as co-purchase [1] or co-views [2].
Most of the existing recommender systems address
recommendations for long-living items or products. Amazon’s
system [1] recommends products that are stable and do not expire
in a short period time. YouTube video recommender system [2]
deals with a fast expanding collection, but most of the items stay
on the site for long periods of time. Netflix’s movie database is
slowly growing collection and this allows their recommendation
system to be based on pre-computing item-item relationships
using collaborative-filtering methods [4]. The Google news
personalization [5] is one of the few systems that addressed the
issue of recommendations for short-lived items. In the dynamic e-
commerce settings of eBay, our recommender systems need to
address the short-lived items as well as ‘seller and item quality’
issues. Our proposed system provides control over the trade-off
between relevance and quality.
3. Two Motivating Scenarios
In this paper, we discuss how our system addresses two typical
recommendation scenarios in ecommerce: pre-purchase and post-
purchase recommendations.
In the pre-purchase scenario our system recommends items that
are good alternatives for the item the user is viewing. eBay as a
marketplace is well known for auction items. When an item is
listed as an auction item, several interested buyers will bid on the
item, but the item will be bought by only one buyer. In this
scenario, buyers who placed a bid on the item but could not buy
will often want to buy a comparable item.
Figure 1 shows an example of this particular scenario. The
algorithm recognizes that the input item belongs to the learned
concept “handmade amish quilt” and makes recommendations all
within that class of objects, even though the details of the
recommendations can vary in some other dimensions. One naïve
solution to this problem is to use the seed item title as query and
recommend best matching items. We compared our method
against this baseline and showed superior results.
Figure 1: Pre-purchase Similar Item Recommendations
Figure 2: Post-purchase Related Item Recommendations
In the post-purchase scenario, we recommend items
complementary/related to an item, which the user has bought
recently. The input item in Figure 2 is a ‘samsung galaxy s3
phone’, and the recommended items are accessories for the phone.
The most common methods of post purchase recommender
systems is ‘collaborative filtering’. However, computing item-
related items using collaborative filtering method is not useful for
short-lived items. Next, we describe our architecture including
how we address the issue of short-lived items.
4. Architecture
Our recommendation architecture consists of a number of
components that can be partitioned in three major groups: The
data store contains data about the active and temporarily
changing state of the web site as well as models learned using that
information, the performance system generates recommendations
at the site given a session state and information in the data store,
and the offline model generation component creates models by
conducting computationally intensive offline analyses. Next we
describe these components in more detail.
4.1 Data store
The data store is the glue between the computationally intensive
offline model generation and the real-time performance
components. Since both systems use the information in the data
store, it often provides two versions of similar services. For
example, the offline modeling component has access to the
complete history of changes in the item inventory but does not
provide efficient keyword search on item properties, while the
performance system can search an indexed version of the current
inventory but does not have access to changes over time. Next we
describe the data sources that our system uses as input and output.
4.1.1 Input Information Sources
The data store contains two kinds of information sources,
continuously accumulated raw data about the individual actions
along with the resulting state on the website, and models which
contain generalized knowledge that can apply to new situations.
Like in many e-commerce sites, our data store can be categorized
into inventory data, which contains a set of items and their static
attributes; clickstream data, which stores actions and dynamic
state of the site, and transaction data, which stores a history of
purchases. Note that, while transactions could be recreated from
clickstream data, it is typically stored separately for efficient
access. The data store also contains a conceptual knowledge base
including category tree, which is a hierarchical ontology that
organizes items contained in the inventory, language specific
knowledge sources like stop words, spell correction rules, etc.,
and term dictionary that lists important terms/phrases in a given
category. Some of these knowledge sources are learned from raw
data, but we will not describe this process here due to space
limitations.
4.1.2 Output Cluster Model
Our architecture generates two main output models. The first
model contains cluster definitions, which groups a set of
conceptually similar items. One of the most important
architectural commitments of our approach is to represent clusters
as an explicit bag of phrases. One major advantage of this
representation is that they can be used as search queries. For
example, given a search engine indexing an inventory of items, we
can retrieve an ordered set of items that best match a cluster
expression. This representation also allows us to calculate term
similarity and item coverage overlap between clusters.
The second knowledge structure our system generates is the
related cluster model, which is used for post-purchase
recommendations of complimentary items. This model represents
a sparse graph between clusters. A strong link from one cluster to
another indicates that the likelihood of purchase of an item from
the second cluster increases after a purchase of an item from the
first cluster. Next, we describe how the cluster models are used in
the runtime system and how they are learned during offline
processing.
 Offline Model Generation The Data Store Real-time Performance System

user queries Inventory SIR query ?similarOf(item)

Query-Recall Cluster cluster
items assignment formation
Generation
clickstream
query
{query items}
concepts,categories Transactions Item items SIR
Cluster Search items
new clusters Clusters Ranking
generation
recommended user
items
bought item-item
Cluster Item items RIR
assignment Conceptual Search items Ranking
Knowledgebase
bought cluster-cluster Set of queries
related cluster-cluster Related clusters
Cluster-to- cluster
cluster Cluster-cluster RIR Query
Model relations Cluster Formation ?relatedOf(item)
cluster
assignment

Fig 3: Cluster Based Architecture

allows our system to balance between relevance and quality: while

Real-time Performance System
The performance system consists of two components, similar item
recommender (SIR) and related item recommender (RIR). Both of
these components take a seed item as input, and return a set of
items that are similar or related to that seed item.
Since the performance system is interacting with the user in real
time, it is crucial that any relevant complicated decision process is
compiled to the offline model. However, pre-caching all
processing is also not feasible since under the changing inventory
assumption we are making, the seed item can be an item the
performance system encounters the first time. As a result, we
require that any retrieval from a data source occurs through an
efficient indexing service and the computation done after retrieval
is kept limited.
Both SIR and RIR start the recommendation process by calling a
cluster assignment service that returns the best matching clusters
given an item. To achieve that, the service compiles normalized
versions of cluster expressions in a Lucene1 index and runs the
input item title and features (e.g. category, attribute-value pair)
through a similar normalization to return the best matching
clusters. Next, SIR creates a search query by taking the union of
best matching cluster tokens, while RIR uses related cluster
service to retrieve n related clusters and constructs n separate
search queries for them. Next, both algorithms use these queries
to call the item search service, which indexes active items. SIR
shows few best items returned from the search service as its
recommendations. On the other hand, RIR returns one item per
the query it has constructed to ensure that each recommendation is
related to the seed item in a different way.
In the item search call, the expressions constructed from clusters
are used as hard-constraints and within the scope of those
constraints, the retrieved items are sorted using a ranking
function that emphasize the recommended items’ quality based
on metrics like predicted conversion probability, seller quality,
price proximity to seed item, format affinity to seed item, etc. This
1
http://lucene.apache.org/
the cluster constraints keep the recommendations relevant, the
ranking process increases the quality of the recommendations.
4.2 Offline Model Generation
4.2.1 Clusters Generation
An online marketplace like eBay has a huge inventory of transient
items ranging in the hundreds of millions. Moreover, the
inventory covers a very broad spectrum of variety, ranging from
regular electronic items to unique collectible items. Given the
scale and variedness of the inventory, global clustering is not
feasible. We had a couple of options to partition the input data:
we could use the manually generated ontology of items created for
business reasons, namely the category hierarchy; or the historical
user queries from behavioral data on the site. We chose the latter
because the user queries are more contextual and provide a better
interpretation of similarity from the users’ perspective.
Our clustering pipeline runs in a Hadoop map-reduce distributed
environment. It starts with the recall sets of user queries and
performs local clustering using K-Means algorithm to further split
them into meaningful clusters of similar items. Later we merge
clusters with high overlap and remove duplicates. The features
used in the clustering algorithm include normalized tokens from
the items’ titles, attributes used to describe them and part of the
category hierarchy they belong to. With this clustering mechanism
we are able to build a stable representation for the transient items
with a 100X reduction in size.
4.2.2 Cluster-Related Clusters Generation
Our proposed method computes cluster-related cluster pairs using
the transactional data. From this data set, we first extract an item-
to-item co-purchase matrix. Next, we generate a directed cluster-
to-cluster graph applying cluster assignment on the item-to-item
co-purchase matrix. An edge from a cluster ‘i’ to cluster ‘j’
indicates that a group of users have purchased items in cluster ‘j’
after buying items in cluster ‘i’. The problem of finding related
clusters is modeled as that of ‘ranking the outgoing edges’ for a
given node. The challenges that we face in this problem include
scale of the data and noisy data. We address the scale using
Hadoop Map-Reduce implementation. The problem of noisy data
 is addressed using cluster-cluster content similarity. For each
node in the graph, we rank the outgoing edges. The ranking
function uses collaborative filtering features such as number of
users who co-purchased items from the respective clusters,
similarity between the two clusters.
5. Experimental Results
In this section we present experimental results for the two
recommendation engines described in the paper.
We compared the similar items recommendation system described
in the previous section with the legacy system that was in
production at eBay at the time. The legacy system took a naïve
approach of using the seed item title as a search query to
recommend the best matching items in the retrieval set. We
hypothesized that relevance would increase if we were to use
clusters learned from user queries, since they would better capture
user intention. The new system also performs more efficiently
since the bulk of the computation is done offline, but we are not
reporting on that in this paper.
We conducted an A/B test on the Closed View Item Page (CVIP)
in eBay to compare our algorithm with the legacy algorithm.
Users typically come to the CVIP upon losing an auction. The
goal of this page is to keep the users engaged with the site and
provide options to buy a different yet similar item in place of the
item they could not win. The test results showed that with 90%
confidence, our algorithm achieved statistically significant
improvement in user engagement and site-wide business metrics.
We are not allowed to report specific statistics on site-wide impact
and incremental purchase driven by the algorithm, but we report
the relative improvement in user engagement (Table 1).
We have also conducted an A/B test comparing our relation-item
recommendation system against the legacy system. The legacy
related items recommender system developed at eBay by Chen &
Canny [6] also first mapped the items to a stable representation
and then used transactional data to compute relationships between
the groups. However, there are important differences between our
proposed method and the original method. Unlike the earlier
method, ours uses clusters generated from user queries and we
hypothesized that they should better capture user intentions.
Additionally, our system is more scalable. It was built using
transactional data from a period of 1 year as opposed to 3 months
used by Chen & Canny [6]. Our test results show a statistically
significant improvement in site-wide business metrics (90%
confidence interval). We also show results on improvement in
user-engagement. (Table 1).
Table1. Relative improvement in user engagement over the legacy
similar and related items recommendation algorithms.
System Click-through-rate (CTR)
Similar Items Recommender 38.18%
Related Items Recommender 10.5%
Both A/B tests validate our proposition that user engagement and
conversion can be improved by striking a balance between item
similarity and quality and by utilizing user queries to better model
similarity from the perspective of the users. We saw significantly
higher user engagement in terms of click through rate and the both
View publication stats
new algorithms made a statistically significant positive impact on
eBay’s overall revenue.
6. Conclusion
In this paper, we presented recommendation system architecture
for dynamic marketplaces where item listings are user-generated
and short-lived. The proposed system provides control over
multiple competing objectives in recommendations such as
relevance, item quality and seller trustworthiness.
The core of our solution is the cluster definitions our system
learns that map item features into a bag of phrases. The clusters
are learned over item features in the inventory but clustering is
local in the coverage set of user queries. This ensures that the
problem is scalable and the clusters are biased by the concepts the
users are creating during their search term construction. Our
offline model generation system leverages information from
multiple large data sources: inventory, user click-stream data and
transactional data and large scale deployment was feasible due to
highly parallel algorithm and usage of large-scale Hadoop cluster.
The online performance system can deal with large number of
requests by indexing offline learned models in memory and
keeping computation on dynamic features limited.
Our system is deployed at ebay.com in large-scale and we showed
statistically significant improvement in user engagement metrics
as well as site-wide business metrics. Our future plans for
improvement includes using more structured representations in
clusters in form of attribute-value pairs, and better natural
language parsing to better retrieve important group of words.
Acknowledgements
We thank Santanu Kolay, Riyaaz Shaik, Kranthi Chalasani and
Venkat Sundaranatha for contribution to this research with ideas
and implementation.
7. REFERENCES
1. Linden, G., B. Smith, and J. York, Amazon.com
Recommendations: Item-to-Item Collaborative
Filtering. IEEE Internet Computing, 2003. 7(1): p. 76-
80.
2. Davidson, J., et al., The YouTube video
recommendation system, in Proceedings of the fourth
ACM conference on Recommender systems. 2010,
ACM: Barcelona, Spain. p. 293-296.
3. Adomavicius, G. and A. Tuzhilin, Toward the Next
Generation of Recommender Systems: A Survey of the
State-of-the-Art and Possible Extensions. IEEE Trans.
on Knowl. and Data Eng., 2005. 17(6): p. 734-749.
4. Koren, Y., Factorization meets the neighborhood: a
multifaceted collaborative filtering model, in
Proceedings of the 14th ACM SIGKDD international
conference on Knowledge discovery and data mining.
2008, ACM: Las Vegas, Nevada, USA. p. 426-434.
5. Das, A.S., et al., Google news personalization: scalable
online collaborative filtering, in Proceedings of the 16th
international conference on World Wide Web. 2007,
ACM: Banff, Alberta, Canada. p. 271-280.
6. Chen, Y. and J.F. Canny, Recommending ephemeral
items at web scale, in Proceedings of the 34th
international ACM SIGIR conference on Research and
development in Information Retrieval. 2011, ACM:
Beijing, China. p. 1013-1022.