On Real-Time Twitter Analysis
Mikio L. Braun mikiobraun twimpact UG (haftungsbeschrnkt) with Matthias Jugel thinkberg Apache Hadoop Get Together, Berlin April 28, 2012 [Link] [Link]
Apache Hadoop Get Together, April 18, 2012, Berlin
2012 TWIMPACT
Big Data and Data Science and Social Media
There's a lot you can do with social media data
Trend analysis (trending topics) Sentiment analysis Impact analysis (Klout, Kred, etc.) More general studies (diameter of network, distribution patterns, etc.) Event treams (Twitter stream) Graph data (user relationships, retweet networks) Text data (sentiment analysis, word clouds) URLs
Types of data
Apache Hadoop Get Together, April 18, 2012, Berlin
2012 TWIMPACT
Social Media Streaming Data
Examples
Twitter firehose/sprinkler Click-through data [Link] URL resolution requests up to a few thousand events per second events are small up to a few kilobytes
Some numbers:
Apache Hadoop Get Together, April 18, 2012, Berlin
2012 TWIMPACT
What's in a Tweet?
Tweet Hashtag Link User Mention Keywords Retweeting User Retweeted User Retweeted Tweet
Timestamp
Apache Hadoop Get Together, April 18, 2012, Berlin
2012 TWIMPACT
TWIMPACT - Retweet trends
Trending by retweet activity Robust matching of tweets even if shortened, edited (slightly) Compute trends for links, hashtags, URLs Aggregate TWIMPACT score for users
Apache Hadoop Get Together, April 18, 2012, Berlin
2012 TWIMPACT
How to scale stream processing?
Apache Hadoop Get Together, April 18, 2012, Berlin
2012 TWIMPACT
History of approaches
Started in June 2009 Free Twitter stream (capped at 50 tweets/s)
Language Storage backend
Version 1 Version 2 Stream mining + in memory
Version 3
Apache Hadoop Get Together, April 18, 2012, Berlin
2012 TWIMPACT
Putting it all in a data base
Insert millions of rows into data base Get reports by
SELECT*,COUNT(*)FROMevents WHEREcreated_at> ANDcreated_at< GROUPBYid ORDERBYCOUNT(*)DESC LIMIT100;
Hardly real-time. Also, data bases will become slower and slower...
Apache Hadoop Get Together, April 18, 2012, Berlin
2012 TWIMPACT
NoSQL: Cassandra
Structure: Families Tables Rows Key Value pairs Easy clustering (peer-to-peer configuration) Flexible consistency, read-repair, hinted handoff, etc. No locking, (in 0.6.x:) no support for indices, counters complete rewrite Operations profile (about 50:50 read/write)
Apache Hadoop Get Together, April 18, 2012, Berlin
2012 TWIMPACT
Cassandra: Multithreading
Multithreading helps (but without locking support?)
64 32 Tweets per second 4 2 16 8
Core i7, 4 cores (2 + 2 HT)
2012 TWIMPACT
Seconds
Apache Hadoop Get Together, April 18, 2012, Berlin
Cassandra: Configuration
Flush
Memtables, indexes, etc.
Size of Memtable: 128M, JVM Heap: 3G, #CF: 12
Apache Hadoop Get Together, April 18, 2012, Berlin
Compaction
2012 TWIMPACT
Cassandra: Configuration
Tweets per second
Compaction
Big GC
Apache Hadoop Get Together, April 18, 2012, Berlin
2012 TWIMPACT
NoSQL/Cassandra - Summary
Works quite well, faster than PostgreSQL (from 200 to 600 tps) Lack of locking/indices require a lot of manual management Configuration messy 4 node cluster vs. single node:
Single node consistently 1.5 3 times faster!
Ultimately, becomes slower and slower Doesn't handle deletions gracefully
2012 TWIMPACT
Apache Hadoop Get Together, April 18, 2012, Berlin
Stream processing frameworks
Stream processing = scalable actor based concurrency For example:
Twitter's (backtype's) Storm [Link] Yahoo's S4 [Link] Esper [Link] Streambase [Link]
Apache Hadoop Get Together, April 18, 2012, Berlin
2012 TWIMPACT
Stream processing- some thoughts
Maximum throughput hard to estimate Not everything can be parallelized Scalable storage system still necessary How to deal with failure/congestion? Persistent messaging middleware not what you might want.
Apache Hadoop Get Together, April 18, 2012, Berlin
2012 TWIMPACT
The DataSift infrastructure
[Link]
Parse
Augment Content
Custom Filters
Delivery
Throughput: 120,000 tweets per second
936 CPU cores Analyzes 250 million tweets per day Peak throughput: 120,000 t/s monitoring & accounting
C++, PHP, Java/Scala, Ruby MySQL on SSDs, HBase (30 nodes, 400TB), memcached, Redis for some queues 0MQ, Kafka (LinkedIn)
but: 120,000 / 936 = 128.2 tweets per second per core
Apache Hadoop Get Together, April 18, 2012, Berlin
2012 TWIMPACT
Principles of Stream Processing
Keep resource needs constant Control maximum processing rates Disks too slow, keep data in RAM
Apache Hadoop Get Together, April 18, 2012, Berlin
2012 TWIMPACT
Stream mining
fixed number of slots asd erq qwe 13r13t erqew erq qer fgsa gwth 5z3 wet 42 37 25 20 17 13 10 7 4 qer 5 21
Focus on relevant data, discard the rest Provably approximates true counts Keep data in memory
Space Saving algorithm (Metwally, Agrawal, Abbadi, Efficient Computation of Frequent and Top-k Elements in Data Streams, International Conference on Database Theory, 2005.)
Apache Hadoop Get Together, April 18, 2012, Berlin
2012 TWIMPACT
TWIMPACT
Real-time Twitter Retweet Analysis
Stream mining to keep hot set of few hundred thousand most active retweets in memory Secondary indices, bipartite graphs, object stores Write snapshots to disk for later analysis Up to several thousand tweets per second in single threaded operation.
Apache Hadoop Get Together, April 18, 2012, Berlin
2012 TWIMPACT
2011 in Retweets
Apache Hadoop Get Together, April 18, 2012, Berlin
2012 TWIMPACT
2011 in Retweets
Apache Hadoop Get Together, April 18, 2012, Berlin
2012 TWIMPACT
Our Analysis Pipeline
JSON parsing Thread 1 Tweets Retweet Matching & Retweet Trends Thread k synchronized worker threads single threaded
Analyzing dependent trends (links/hashtags/etc.) Day 1 Day 2
Snapshots
Trends
Day n map reduce like
Apache Hadoop Get Together, April 18, 2012, Berlin
2012 TWIMPACT
Most retweeted users
Apache Hadoop Get Together, April 18, 2012, Berlin
2012 TWIMPACT
Most retweeted tweets
Apache Hadoop Get Together, April 18, 2012, Berlin
2012 TWIMPACT
Social network buzz
Apache Hadoop Get Together, April 18, 2012, Berlin
2012 TWIMPACT
Summary
Many interesting challenges in social media. Many different data types, including streams. MapReduce doesn't really fit stream processing You can't just scale into real-time Principles of Stream Processing
Bounded hot set of data in memory Mine stream, discard irrelevant data
Real world applications often include a mixture of multithreading, stream processing, map reduce and single thread stages.
2012 TWIMPACT
Apache Hadoop Get Together, April 18, 2012, Berlin
