Record every Referral for Flickr Realtime

While reading from the DB By Dathan Vance Pattishall

Who am I?
• • • • • Been working with mySQL since 1999 Scaled many companies with mySQL Such as Friendfinder / Friendster / Flickr Now doing the same for RockYou Words I like: Federation, Partitioning, Shards, Raid-10, Scale, Speed (Throughput)

Flickr Stats Backend Look
Flickr Stats is a feature that reports on every referrer an object receives, and stores that information for as long as you’re a pro member

Requirements
• Scale better then the number of page views. • Store the data forever • Associate time with the data • Allow for change • Keep it cheap • Oh and downtime is not an option

Spread the Data around
• Federate – not the scope of this talk (RAID-10 for the DataLayer) • All referrers are owned by the page owner • Spread data out by that • But Federate in a different direction • Add a new column to the global account lookup

What does the data look like?
Path_query Domain Owner When Object-ID Object-Type Counts and stuff Varchar(255) Varchar(50) Bigint Date Bigint Tinyint Various ints PK

May be some keys

That didn’t work
• Strings as primary keys was not good. • Every other index was a new page referencing the larger then ideal primary key size. • Inserts slowed when the table grew larger then memory size • Not enough I/O to handle double the load

Start over
• • • • • Converted the URL into a 64-bit ID CONV(SUBSTR(MD5(Url),0,16),16,10) == 64-bit number For added protection its unique for the owner • Reduced Primary key size to 8 bytes + owner, object, object-type

Split data up by Week
• Since a time requirement and a chance to drop data, lets drop it fast and move on • 53 Tables that have 7 columns to denote day • 1 Year Table to cook data down to a year view • 1 Totals table to reference the single copy of the string

INNODB & Strings
• Indexing a string takes a lot of space • Indexing a large string takes even more space • Each index has its own 16KB page. • Fragmentation across pages was hurting the app – chewing up I/O • That’s a lot of disk space chewed up per day

INNODB & High Concurrency of String Writes
• Requirement: 300 ms for total db access FOR ALL Apps • Writes when the datafile(s) are greater then the buffer_size slow down at high concurrency • 10 ms to 20 seconds sometimes for the full transaction • Noticed replication keeping up

Solution buffer the writes
• Put a java daemon that buffers up to 4000 messages (transactions) and apply it serially with one thread • It does not go down • It does not use much memory or cpu • It was written by a Zen Java Master • Even during peak messages do not exceed 200 outstanding transactions

Reduce the use of big strings
• Store 1 copy of the referrer and reference the big int • Since you store the data for a pro user forevar keep condensed formats • Keep only the data that is displayed

Cook data down
• Created a state machine • 99% of urls reduced by 30% • Probably could of used zlib to do it better thinking about it.

Keep even smaller amounts of data
• Use myISAM for non-pro users and keep only X weeks of data. MyISAM keeps the data 1/6 the size over InnoDB for my case • Migrate the data when they activate the product. • But don’t do it more then once – because that would suck.

Distributed Locks
• Use GET_LOCK & RELEASE_LOCK on the same server for which a user operates on. • Sick

What you have is
• A system that can record every referrer from every page on flickr • Keeps track of time for each referrer and associated with a page owner • Done in a transaction across 3 tables • Highly redundant • Backed up regularly (Buffered writes on MYISAM tables too baby!) • Crazy through-put (federated 1 application on dedicated hardware) • On 6 servers x2 for redundancy == 12 

Questions? Yes?