Redis: Persistence Power

Nick Quaranto / @qrush / nick@quaran.to
Tuesday, August 10, 2010

What is Redis?
“advanced key-value store” REmote DIctionary Server data structures server

Tuesday, August 10, 2010

YOUR APP

Tuesday, August 10, 2010

YOUR APP

REDIS

Tuesday, August 10, 2010

the basics
persist data as you think of it in memory, sync to disk in background ridiculously fast master-slave replication keys = strings, value = data structures

Tuesday, August 10, 2010

http://try.redis-db.com

Tuesday, August 10, 2010

use it: redis-cli
% ./redis-cli SET user:1:name qrush OK % ./redis-cli GET user:1:name "qrush"

Tuesday, August 10, 2010

use it: redis-rb
% gem install redis % irb -rubygems -rredis >> $redis = Redis.new => #<Redis client v2.0.3 connected... >> $redis.set "user:1:name", "qrush" => "OK" >> $redis.get "user:1:name" => "qrush"
Tuesday, August 10, 2010

FEATURE SWITCHES
STRINGS

based on http://github.com/blog/677 http://github.com/bvandenbos/redis_feature_control
Tuesday, August 10, 2010

Rediswitch.features << :super_secret Rediswitch.features << :payment_gateway Rediswitch.features << :twitter if Rediswitch.enabled?(:twitter) # post to twitter else # failwhale ahoy! end

Tuesday, August 10, 2010

begin # take some money rescue PaymentGateway::TotallyDown => ohno Rediswitch.disable(:payment_gateway) # notify the troops end

Tuesday, August 10, 2010

class Rediswitch def self.enabled?(feature) $redis.exists(feature) end def self.enable(feature) $redis.incr(feature) end def self.disable(feature) $redis.del(feature) end end
Tuesday, August 10, 2010

feature switch lessons
the real win: no-deploy configuration fast enough to be transparent next step: separate users into buckets with sets http://github.com/jamesgolick/rollout

Tuesday, August 10, 2010

RATE LIMITER
STRINGS

soon to be in place at http://hoptoadapp.com
Tuesday, August 10, 2010

class Choker def restrict? track count_for > 60 end end

Tuesday, August 10, 2010

class Choker def count_for $memcache.get(key, true).to_i end end

Tuesday, August 10, 2010

class Choker def track if !$memcache.get(key, true) $memcache.add(key, "0", 1.minute.from_now, true) end $memcache.incr(key) end end

Tuesday, August 10, 2010

class Choker def track if !$redis.exists(key) $redis.setex(key, 60, 0) end $redis.incr(key) end end

Tuesday, August 10, 2010

class Choker def count_for $redis.get(key).to_i end end

Tuesday, August 10, 2010

rate limiter lessons
expire semantics are changing in redis 2.2 benchmark the crap out of it could use a sorted set instead of strings

Tuesday, August 10, 2010

API USAGE LOGGING
STRINGS SORTED SETS

based off http://www.production-hacks.com/2010/07/10/redis-api-access-logger/
Tuesday, August 10, 2010

# one way to do it class ActionHit < ActiveRecord::Base # t.string :controller # t.string :action # t.integer :counter end class UserHit # t.string # t.integer # t.integer end < ActiveRecord::Base :controller_action :user_id :counter

Tuesday, August 10, 2010

# for all controllers { "statuses#update" => 1410, "users#create" => 931, "home#index" => 2936 } # users { "101" "102" "103" }
Tuesday, August 10, 2010

hitting an action => 42, => 13, => 34

class StatusesController < ApplicationController def update $redis.incr "statuses#update" $redis.incr "statuses#update:#{user.id}" end end

Tuesday, August 10, 2010

class StatusesController < ApplicationController def update key = "statuses#update" $redis.zincrby "actions", 1, key $redis.zincrby "users:#{key}", 1, user.id end end

Tuesday, August 10, 2010

# hits for a specific user >> $redis.zscore "users:statuses#update", 1001 => 42 # list all the controller actions, sorted >> $redis.zrevrange "actions", 0, -1, :with_scores => true => ["home#index", "2936", "statuses#update", "1410", "users#create", "931"]

Tuesday, August 10, 2010

api usage logging lessons
sorted set = high score list bad at historical usage, trends good for a simple heartbeat or pulse

Tuesday, August 10, 2010

JOB QUEUE
LISTS

based on http://github.com/defunkt/resque
Tuesday, August 10, 2010

class Staple @queue = :default def self.perform(post_id, tempfile) # complex image resizing, cropping end end

Tuesday, August 10, 2010

class Post < ActiveRecord::Base after_save :process_with_stapler def process_with_stapler Resque.enqueue(Staple, self.id, @tempfile) end end

Tuesday, August 10, 2010

module Resque extend self def push(queue, item) redis.rpush "q:#{queue}", encode(item) end def pop(queue) decode redis.lpop("q:#{queue}") end end

Tuesday, August 10, 2010

class Resque::Worker def work loop do if job = Resque.pop(queue) job.perform else sleep 5 end end end end

Tuesday, August 10, 2010

module Resque extend self def bpop(queue) decode redis.blpop("q:#{queue}") end end

Tuesday, August 10, 2010

class Resque::Worker def work loop do job = Resque.bpop(queue) job.perform end end end

Tuesday, August 10, 2010

job queue lessons
guaranteed atomic actions, no row locking blocking commands simplify daemons many more queue commands in redis itself!

Tuesday, August 10, 2010

GLOBAL ERRORS
SETS MULTI/EXEC

a new feature at http://hoptoadapp.com
Tuesday, August 10, 2010

# text :globals, :default => '', :null => false class Project < ActiveRecord::Base def has_global?(name) @globals ||= globals.gsub(/,/,' ').split @globals.include?(name) end end

Tuesday, August 10, 2010

# MORE TABLES!!!! class Global < ActiveRecord::Base belongs_to :project end class Project < ActiveRecord::Base has_many :globals end

Tuesday, August 10, 2010

# Project#global_errors ["MySQL::Error", "MemCache::Error", "Net::HTTPFatalError"]

Tuesday, August 10, 2010

class Project < ActiveRecord::Base def global_key "project-#{id}-globals" end def has_global?(name) $redis.sismember(global_key, name) end end

Tuesday, August 10, 2010

class Project < ActiveRecord::Base after_save :save_globals def save_globals $redis.del global_key @globals.each do |g| $redis.sadd global_key, g end end end

Tuesday, August 10, 2010

SISMEMBER

SISMEMBER

[Mysql::Error, MemCache::Error, Net::HTTPFatalError]
Tuesday, August 10, 2010

DEL

[]
Tuesday, August 10, 2010

DEL

SADD

[Mysql::Error]

Tuesday, August 10, 2010

DEL

SADD SADD

[Mysql::Error, OpenURI::HTTPError]

Tuesday, August 10, 2010

DEL SISMEMBER SADD SADD

[Mysql::Error, OpenURI::HTTPError]

[]

Tuesday, August 10, 2010

MULTI DEL SADD SADD EXEC SISMEMBER

[Mysql::Error, OpenURI::HTTPError]
Tuesday, August 10, 2010

class Project < ActiveRecord::Base after_save :save_globals def save_globals $redis.multi do $redis.del global_key @globals.each do |g| $redis.sadd global_key, g end end end end

Tuesday, August 10, 2010

global error lessons
avoid joins for simple data consider race conditions use append-only file (AOF)

Tuesday, August 10, 2010

MULTIPLAYER NOTEPAD
PUB/SUB

based on http://github.com/laktek/realie
Tuesday, August 10, 2010

# usage: ruby pub.rb room username data = {"user" => ARGV[1]} loop do msg = STDIN.gets $redis.publish ARGV[0], data.merge('msg' => msg.strip).to_json end

Tuesday, August 10, 2010

# sub.rb $redis = Redis.new(:timeout => 0) $redis.subscribe('rubyonrails', 'rubymidwest') do |on| on.message do |room, msg| data = JSON.parse(msg) puts "##{room} - [#{data['user']}]: #{data['msg']}" end end

Tuesday, August 10, 2010

% ruby pub.rb rubymidwest qrush i give up, i hate markdown

% ruby sub.rb #rubymidwest - [qrush]: i give up, i hate markdown #rubyonrails - [railsn00b]: undefined method posts_path? wtf? #rubymidwest - [turbage]: seriously.

Tuesday, August 10, 2010

multiplayer notepad lessons
combine with other data structures can subscribe to channels via patterns concurrency in ruby is hard use eventmachine! (or node.js)

Tuesday, August 10, 2010

more to learn
know your data! (via @antirez) command reference on the wiki active IRC, mailing list

Tuesday, August 10, 2010

AKASENTAI.com
redis in the cloud
Tuesday, August 10, 2010

Thanks!
http://redis.io http://rediscookbook.com http://scr.bi/redispower
Tuesday, August 10, 2010

@qrush

BONUS ROUND!
I prepared way too many examples. Jackpot!
Tuesday, August 10, 2010

URL SHORTENER
STRINGS

based on http://github.com/mattmatt/relink
Tuesday, August 10, 2010

require 'sinatra' require 'redis_url' post '/' do RedisUrl.new(params[:url]).save end

Tuesday, August 10, 2010

class RedisUrl attr_accessor :url, :id def initialize(url) @url = url @id = seed # unique string algorithm end def save $redis.set("relink.url|#{@id}", @url) $redis.set("relink.url.rev|#{@url}", @id) end end

Tuesday, August 10, 2010

get %r{/(.+)} do |url| u = RedisUrl.find(url) if u u.clicked redirect u.url else status 404 end end

Tuesday, August 10, 2010

class RedisUrl def self.find(id) u = $redis.get("relink.url|#{id}") if u redis_url = RedisUrl.new(u) redis_url.id = id redis_url end end def clicked $redis.incr("relink.url.clicks|#{@id}") end end
Tuesday, August 10, 2010

url shortener lessons
common pattern: namespacing incr/decr assumes value is an integer wrap behavior into ActiveRecord-like objects next step: store URLs in a list

Tuesday, August 10, 2010

LIVE DEBUGGING
LISTS

based on http://github.com/quirkey/redisk
Tuesday, August 10, 2010

def after_save begin # make request to external service rescue Exception => ex logger.error "this shouldn't ever happen!" logger.error ex logger.error ex.backtrace end end

Tuesday, August 10, 2010

# config/initializers/logger.rb require 'redisk' path = "#{Rails.env}.log" config.logger = Redisk::Logger.new(path)

Tuesday, August 10, 2010

class Redisk::IO def write(string) redis.rpush "#{name}:_list", string end def self.readlines(name) redis.lrange("#{name}:_list", 0, -1) end end

Tuesday, August 10, 2010

live debugging lessons
enables real-time data about your system dump serialized/marshalled data fast run the redis instance on a different box dive deeper: hummingbird

Tuesday, August 10, 2010

COUNTING DOWNLOADS
STRINGS SORTED SETS HASHES

based on http://github.com/rubygems/gemcutter
Tuesday, August 10, 2010

# bad idea, dude class Download < ActiveRecord::Base belongs_to :rubygem end class Rubygem < ActiveRecord::Base has_many :downloads end

Tuesday, August 10, 2010

class Download def self.incr(rubygem) $redis.incr("all") $redis.incr(rubygem) $redis.zincrby("today", 1, rubygem) end end

Tuesday, August 10, 2010

class Download def self.rollover(version) $redis.rename "today", "yesterday" dls = Hash[*$redis.zrange("yesterday", 0, -1, :with_scores => true)]

dls.each do |key, score| $redis.hincrby key, Date.today, score Rubygem.find_by_name(key).increment!(:downloads, score) end end end

Tuesday, August 10, 2010

get "/api/v1/downloads/rails.json" do $redis.hgetall("rails").to_json end # returns... { "2010-07-09" => 1908, "2010-07-10" => 1032, "2010-07-11" => 1091, }

Tuesday, August 10, 2010

counting downloads lessons
hybrid approach does work! redis is really not for search test your migration away from SQL

Tuesday, August 10, 2010

Sign up to vote on this title
UsefulNot useful