Redis at Superfeedr

In production @superfeedr // @julien51

Superfeedr What?
• Realtime feed parsing in the cloud • Polling feeds • Redis for • Scheduling • Caching

The Ring
• Requirements • A datastructure that can be
“rotated”

• Dealing with priorities • Storing some “basic” stuff

Redis
• List rpoplupsuh(“ring”, “ring”) • Key/Value in Json • Async • Pipelining to avoid network latencies • Version • redis_version:1.2.5 • arch_bits:32 • multiplexing_api:epoll

Algorithm
loop do ids = $redis.rpoplpush("ring", "ring", 50) items = $redis.get(ids) ids.each do |item| if item.time_to_fetch < now() fetch(item) item.time_to_fetch = later() else # skip end end end

Cleaning the Ring
• No direct access to items in List • ‘maintenance’ script that takes

items in the back, check if they’re in k/v space and puts them back (or not) in the ring.

Numbers
• 450 processes with the same loop. • ~=2M items in ring in 15seconds
(limited by clients perf)

• 1.5GB slice @slicehost. RSS ~= 600MB • Total commands/sec : 102869 • 4core : load average: 1.00, 1.00, 1.00

Tricks
• Don’t snapshot/AOF on same
machine. Use slave for this

• Compile in 32 bits : less RAM • Pipeline! (cut down overhead %) • Client side sharding (ouch!) • Runit for process monitoring

Sign up to vote on this title
UsefulNot useful