to: Resque from: Backgroundrb

@kbrock
2010-10-12

Summary
!

Background queues let us defer logic outside the browser request and response. Background.rb was crashing for us often. Moved to resque and it hasn't crashed since. Background.rb is easier to run out of the box. Adding just a little code makes Resque just as easy without sacrificing all the added flexibility.

!

!

!

Why we upgraded?
!

bdrb pages Boss 4 times my first weekend
!

memory leaks caused crashes monit can't restart workers in backgroundrb

!

!

move to active project (ala heroku, github, redis)

web ! ! ! code code .What do each bring to the table bdrb adhoc (out of request) delay (run/remind) schedule (cron) mail (invisible/out of req) status reporting backgroundrb does most of what we need out of the box resque has plugins to make up the difference resque ! resque-schedule resque-schedule resque_mailer resque-meta.

managed by 1 process (is a runner/worker) .Bdrb Components scheduler workers rails enqueue main queue queue manager work mailer we started Monitored data bdrb yml simple w/ 1 queue (add started_at for delayed jobs) scheduler is a special worker .

Resque Components delayed queue scheduler schedule 2 rails enqueue 1 workers rake 4 resque web main main main queue queue queue work workers 6 mailer we started Monitored data 3 5 many moving parts simplified in all workers are the same scheduler simply adds entries in the queue (instead of MetaWorker/running jobs) web ui is a nice touch .

Ad-hoc Enqueuing bdrb args enqueue AR objects mail(invisible) AR objects . checked hash ! ! ! .1. but not enqueuing AR objects is best practice resque ruby.creeped up in the action_mailer deliver calls Looks like bdrb wins here.

enq_purge_job_logs( :job_key => new_job_key.perform_later(*args) MiddleMan.enq_purge_job_logs( :job_key => new_job_key.purge_expired! persistent_job.worker(:job_worker).worker(:job_worker).new_job_key() "purge_job_logs_#{ActiveSupport::SecureRandom. :arg => args) end def self.Ad-hoc/Delayed (bdrb) class JobWorker < BackgrounDRb::MetaWorker set_worker_name :job_worker def purge_job_logs() JobLog.shift MiddleMan.perform_at(*args) time=args.finish! end def self. :arg => *args.hex(8)}" end end don't need to do a command pattern (our code didn't) scheduled_at = beauty of SQL parent class enqueue knows queue name (code not loaded) .:scheduled_at => time) end def self.

later}) -> abstracted out to parent? .process() JobLog.perform_at(*args) time=args.perform_later(*args) Resque. *args) end def self. self.shift Resque.enqueue_at(time. *args) end end Enqueue needs worker class to know the name of the queue (even if called directly into Resque) interface only (perform_{at.purge_expired! end def self.Ad-hoc/Delayed (resque) class PurgeJobLogs @queue = :job_worker def self.enqueue(self.

yml and ruby. We ran into case where this caused a problem web ui for easy adhoc kicking off of resque commands. (very useful in staging) resque command !+ ! !x2 ! .2. Scheduled Enqueuing bdrb sched any method scheduler adhoc jobs Need to define schedule in 2 places.

1 :port: 11006 :environment: development :schedules: :scheduled_worker: :purge_job_logs: :trigger_args: 0 */5 * * * * Evidence of framework .0.0. need meta worker (so it can be run) .scheduled_worker defined here.Scheduled (bdrb) :backgroundrb: :ip: 127.

purge_expired! } end scheduler = MetaWorker.Scheduled (bdrb) class ScheduledWorker < BackgrounDRb::MetaWorker extend BdrbUtils::CronExtensions set_worker_name :scheduled_worker threaded_cron_job(:purge_job_logs) { JobLog. Defined 2 times .so it calls your code. so can call "any static method" .

Scheduled (resque) --clear_logs: cron: "*/10 * * * *" class: PurgeJobLogs queue: job_worker description: Remove old logs queue_name (so scheduler does not need to load worker into memory to enqueue) cron is standard format (remove 'seconds') .commands scheduler in separate process. (can run when workers are stopped / changed) .minimal env scheduler injects into queue (vs runs jobs) .so can adhoc inject via web no ruby code for this .

command. web us+ ! ! ! 1 <1 .∞ ! .3.1+ queues / worker pause/restart workers resque us. Processes/Worker management bdrb knows queues pids mem leak resistant workers/queue pause workers Discover previous queues (not all) via 'resque list' / web bdrb: creates 1 worker/queue (creates pid file + 1 pid for backgroundrb) .monit can't restart we manage processes: 1+ workers/queue .

*changed at runtime inverted priority list . * schedule focused.worker list (resque) primary: queues: background. * response focused.prevents starvation .background can have multiple workers running the same queues can have multiple queues in 1 worker worker pool can be * generalized.mail secondary: queues: mail.

log &" end end . config| sh ". Running Workers namespace :resque do desc 'start all background resque daemons' task :start_daemons do mrake_start "resque_scheduler resque:scheduler" workers_config.4.workers_config YAML.yml')) end def self./script/monit_rake start #{task} RAILS_ENV=#{ENV['RAILS_ENV']} >> log/daemons./script/monit_rake stop resque_#{worker} -s QUIT" end end def self.load(File.open(ENV['WORKER_YML'] || 'config/resque_workers.mrake_start(task) sh "nohup .each do |worker. config| mrake_start "resque_#{worker} resque:work QUEUE=#{config['queues']}" end end desc 'stop all background resque daemons' task :stop_daemons do sh ".each do |worker./script/monit_rake stop resque_scheduler" workers_config.

Deploying (cap) namespace :resque do desc "Stop the resque daemon" task :stop. true" end desc "Start the resque daemon" task :start. :roles => :resque do run "cd #{current_path} && RAILS_ENV=#{rails_env} WORKER_YML=#{resque_workers_yml} rake resque:start_daemons" end end . :roles => :resque do run "cd #{current_path} && RAILS_ENV=#{rails_env} WORKER_YML=#{resque_workers_yml} rake resque:stop_daemons.

config| %> check process resque_<%=worker%> with pidfile <%= @rails_root %>/tmp/pids/resque_<%=worker%>.com start program = "/bin/sh -c 'cd <%= @rails_root %>. RAILS_ENV=production ./script/monit_rake stop resque_scheduler'" <% YAML.5./script/monit_rake stop resque_<%=worker%>'" <% end %> use template to generate monit file .erb) check process resque_scheduler with pidfile <%= @rails_root %>/tmp/pids/resque_scheduler.each_pair do |worker.pid group resque alert errors@domain./script/monit_rake start resque_<%=worker%> resque:work QUEUE=<%=config['queues']%>'" stop program = "/bin/sh -c 'cd <%= @rails_root %>.load(File. Monitoring Workers (monit.com start program = "/bin/sh -c 'cd <%= @rails_root %>.root+'/config/production/resque/resque_workers. RAILS_ENV=production .open(Rails. RAILS_ENV=production ./script/monit_rake start resque_scheduler resque:scheduler'" stop program = "/bin/sh -c 'cd <%= @rails_root %>. RAILS_ENV=production .pid group resque alert errors@domain.yml')).

/log/${name}.log # . shift pid_file=.pid log_file=... .Monitoring Rake Processes #!/bin/sh # wrapper to daemonize rake tasks: see also http://mmonit.g.com/wiki/Monit/FAQ#pidfile usage() { echo "usage: ${0} [start|stop] name target [arguments]" echo "\tname is used to create or read the log and pid file names" echo "\tfor start: target and arguments are passed to rake" echo "\tfor stop: target and arguments are passed to kill (e.: -n 3)" exit 1 } [ $# -lt 2 ] && usage cmd=$1 name=$2 shift ./tmp/pids/${name}.

esac . then echo "ensure process ${name} (pid: ${pid_file}) is not running" exit 1 fi fi echo $$ > ${pid_file} exec 2>&1 rake $* 1>> ${log_file} . then echo -e "\nERROR: missing target\n" usage fi pid=`cat ${pid_file} 2> /dev/null` if [ -n "${pid}" ] ... *) usage .. then ps ${pid} if [ $? -eq 0 ] .Monitoring Processes case $cmd in start) if [ ${#} -eq 0 ] . stop) pid=`cat ${pid_file} 2> /dev/null` [ -n "${pid}" ] && kill $* ${pid} rm -f ${pid_file} .

Monitoring Web .

rb` end end .6.sync=true $stderr.sync=true puts `env RAILS_ENV=#{RAILS_ENV} resque-web #{RAILS_ROOT}/config/initializers/resque. Running Web namespace :resque do task :setup => :environment desc 'kick off resque-web' task :web => :environment do $stdout.

redis = redis_config[rails_env] require 'resque_scheduler' require 'resque/plugins/meta' require 'resque_mailer' Resque. :cucumber] ..yml') Resque.dirname(__FILE__).load_file(rails_root+'/config/resque_schedule..join(File.') redis_config = YAML.initializer #this runs in sinatra and rails ./.so don't use Rails.'.yml') Resque::Mailer.schedule = YAML.load_file(rails_root + '/config/redis.excluded_environments = [:test.env rails_env = ENV['RAILS_ENV'] || 'development' rails_root=ENV['RAILS_ROOT'] || File.

Monitoring Work bdrb ad-hoc queries did it run? did it fail? rerun have id que health ! sample controller SQL custom hoptoad resque redis query resque-meta ! ! resque-meta ! Did the job run? resque assumes all worked .only tells you failures.5. not good enough for us .

Pausing Workers signal quit term / int usr1 usr2 cont what happens wait for child & exit immediately kill child & exit immediately kill child don't start any new jobs start to process new jobs when to use gracefully shutdown shutdown now stale child .

Testing Worker bdrb testing queue testing command all workers same interface only mid-easy resque resque_unit ! ! ! .

:cucumber] .excluded_environments = [:test.Mail Resque::Mailer.

g.Extending with Hooks resque hooks around_enqueue after_enqueue before_perform around_perform after_perform all plugins want to extend enqueue .not compatible need to be able to alter arguments (e.: add id for meta plugins) " ! ! !/" ! .

Conclusion ! Boss got no pages in first month of implementation ! no memory leaks. great uptime (don't need monit...) ! Fast ! generalized workers increases throughput (nightly vs 1 hour) ! minimal custom code still some intimidation Eating flavor of the month ! ! .

rpm_contrib ! ! ! ! ! .References ! coders: @kbrock and @wpeterson great company: PatientsLikeMe (encouraged sharing this) resque_mailer resque-scheduler resque-meta monit. hoptoad.

Sign up to vote on this title
UsefulNot useful