Professional Documents
Culture Documents
Who I am?
! ! ! ! Olaf Zschiedrich eBay Classifieds Group Head of Technology @ eBay Kleinanzeigen Area of expertise/interest:
High traffic web-applications Agile development Java/JEE Search technologies
Agenda
! ! ! ! ! ! ! ! ! ! About eBay Classifieds Group/ebay Kleinanzeigen Metrics & Traffic Numbers Why Solr? Solr Features in Action Data Indexing Solr in Production Best Practices Problems Outlook Questions
4
Why Solr
! ! ! ! ! ! ! ! ! Open Source Good documentation / big community Java-based (the language we know/use) Widely used (especially lucene) Based on lucene (de-facto standard for full text search in java) Feature-rich (including enterprise features) Extensible (e.g. easy implementation of own tokenizers) Easy to integrate (HTTP, SolrJ client) Easy to setup (java web application)
Most promising option we looked at. Due to very aggressive timelines no time consuming research was possible!
9
10
Data Indexing
MySQL Slave
JDBC Delta Import Handler
Solr Master
! ! ! !
Use of Delta Import Handler Delta import runs every 10 minutes Full import only done in case schema change requires full index rebuild Index optimized once a day
Solr Slave
Solr Slave
Solr Slave
11
Solr In Production
! 2 datacenters ! 1 Master + 6 Slaves per datacenter
Slaves show very low resource consumption. Could go down to 4 slaves per datacenter while still having 50% overcapacity
! ! ! ! !
Master only used for indexing Load balancer in front of slaves Varnish in front of slaves (for dedicated use cases) Working closely with SITE-OPS Team DEV-OPS are part of development process
12
13
Best Practises
! Use solr cores right from the beginning
Allows you to run mutiple indexes on one box in dev and distribute indexes to mutiple boxes in production
! ! ! !
Use filter queries Use caching (FieldCache, QueryCache, Web Proxy Cache e.g. Varnish or Squid) Tune JVM properly Build search-layer hiding the usage of solr
SearchCommand cmd = new SearchCommand(); cmd.setKeywords(BMW 323); ... SearchResult result = searchService.searchActiveAds(cmd); List<Ad> ads = result.getAds();
Problems
! Distance search including sorting
Not supported in previous Solr versions LocalSolr
not working with Solr 1.4 final, GC issues, performance issues
Solution:
Got rid of sort by distance. Implemented own distance search based on bounding boxes and simple range queries.
Solved in 3.1
15
16
Questions ?
17
Contact
! Olaf Zschiedrich
ozschiedrich@ebay-kleinanzeigen.de ozschiedrich@ebay.com www.ebay-kleinanzeigen.de
18