Professional Documents
Culture Documents
morning, hi, my name is rany keddo, i run a little startup in frankfurt called play/type.
PLAY/TYPE
git clone \
git://github.com/purzelrakete/cows-not-kittens.git
OR...
you might want to start by grabbing the demo project. i’m serving it.
PLAY/TYPE
WTF?
this talk is about running code asynchronously in your rails application. this means removing
long running or side effect code from your request cycle.
for some reason cows started to creep into the slides while i was working on them. hoping to
start a trend *away* from cats in tech presentations... you’ll see this reflected in the example
project. make sure it’s working by rake db:migrate && starting irb in the project, then you
type
PLAY/TYPE
>> CowSubsystem.moo
PLAY/TYPE
this is an example app to demonstrate why you need background work, and how you can do
this. also, you can milk cows with this application.
PLAY/TYPE
Milk it
Real Examples
1 AnalyticsHit.create \
2 :potential_user_id => potential_user_id,
3 :event => "converted"
4
nor does this: stuff that does not have to have an immediate effect on the page you’re
rendering.
PLAY/TYPE
1 CommentMailer.deliver_created(comment)
2
1 Blackbook.get \
2 :username => "moo@cheapmail.com",
3 :password => "milky"
and especially this sort of long running process - scraping contacts from webmailer.
PLAY/TYPE
no active* way of handling this... something consistent that works for almost everybody.
instead: too many options. that’s why people come to this sort of talk.
You are a snowflake again.
Trust nobody!
Workling
Workling
• Easy plugging of new Job Runners
Workling
• Easy plugging of new Job Runners
• Nice Rails integration
Workling
• Easy plugging of new Job Runners
• Nice Rails integration
• Plays nicely with tests
Workling
• Easy plugging of new Job Runners
• Nice Rails integration
• Plays nicely with tests
• Lightweight and hackable
1 script/plugin install \
2 git://github.com/purzelrakete/workling.git
3
4 script/plugin install \
5 git://github.com/tra/spawn.git
1 #
2 # handle asynchronous mooing.
3 #
4 class CowWorker < Workling::Base
5
6 # let the moo-ings begin!
7 def moo(options = {})
8 cow = Cow.find(options[:id])
9 cow.moo
10 end
11 end
Milk it!
PLAY/TYPE
What’s Spawn?
1 script/plugin install \
2 git://github.com/tra/spawn.git
explain what’s going on here... we’ve used spawn as a runner for workling. what’s spawn?
PLAY/TYPE
1 spawn do
2 logger.info("I feel sleepy...")
3 sleep 11
4 logger.info("Time to wake up!")
5 end
by itself you can run it like this. it will fork the process....
PLAY/TYPE
like this, basically, but with all rails fixes and tweaks in place. above: drops to unix, the OS
copies the process & creates a child process. try this in your console and use top to look at
the processes.
PLAY/TYPE
“
Twitter’s Evan Weaver and nesting friend.
before i started on workling, i asked evan weaver of chow fame (twitter now) what he
thought. this is his what he said about spawn.
PLAY/TYPE
BackgroundJob
1 script/plugin install \
2 git://github.com/purzelrakete/workling.git
3
4 ./script/plugin install \
5 http://codeforpeople.rubyforge.org/svn/rails/plugins/bj
6
7 ./script/bj setup
lets start over, workling + bj. don’t need to do anything else, since bj is automatically
detected.
PLAY/TYPE
1 Workling::Remote.dispatcher =
2 Workling::Remote::Runners::BackgroundjobRunner.new
3
however, the workling runner can also be set manually like this, inside of environment.rb or
under config/initializers. this is being done automatically for you.
PLAY/TYPE
Milk it!
PLAY/TYPE
1 ./script/plugin install \
2 http://codeforpeople.rubyforge.org/svn/rails/plugins/bj
3
4 ./script/bj setup
1 create_table :bj_config do |t|
2 t.column "command" , :text
3 t.column "state" , :text
4 t.column "priority" , :integer
5 t.column "tag" , :text
6 t.column "is_restartable" , :integer
7 t.column "submitter" , :text
8 t.column "runner" , :text
9 t.column "pid" , :integer
10 t.column "submitted_at" , :datetime
11 t.column "started_at" , :datetime
12 t.column "finished_at" , :datetime
13 t.column "env" , :text
14 t.column "stdin" , :text
15 t.column "stdout" , :text
16 t.column "stderr" , :text
17 t.column "exit_status" , :integer
18 end
setup is running this migration.
PLAY/TYPE
1 if(job.finished) ...
If you want something back... these are some useful columns in the db. they are available on
the job object, too.
PLAY/TYPE
howz it work? this is why the moo came later than with spawn.
PLAY/TYPE
howz it work? this is why the moo came later than with spawn.
PLAY/TYPE
howz it work? this is why the moo came later than with spawn.
PLAY/TYPE
howz it work? this is why the moo came later than with spawn.
PLAY/TYPE
howz it work? this is why the moo came later than with spawn.
PLAY/TYPE
howz it work? this is why the moo came later than with spawn.
PLAY/TYPE
Added Bj Runner to
Workling like this...
Starling
PLAY/TYPE
add github to your sources if you havent already done so. explain fiveruns client.
PLAY/TYPE
1 mkdir /var/spool/starling
2 sudo starling -d
3 script/workling_starling_client start
1 Workling::Remote.dispatcher =
2 Workling::Remote::Runners::StarlingRunner.new
PLAY/TYPE
Milk it already...
PLAY/TYPE
Starling
lightweight queue that speaks memcached. developed at twitter by blaine cook 2 make
twitter arch more msg-oriented.
PLAY/TYPE
Memcache Client
PLAY/TYPE
Memcache Client
Memcache Client
“
... The main things lacking in
Starling are non-destructive
reads (transactions), and
speed.
twitter moving away from starling. putting msgs back onto queue not possible after kill/
crash.
PLAY/TYPE
“
... The main things lacking in
Starling are non-destructive
reads (transactions), and
speed.
twitter moving away from starling. putting msgs back onto queue not possible after kill/
crash.
PLAY/TYPE
“
... The main things lacking in
Starling are non-destructive
reads (transactions), and
speed.
twitter moving away from starling. putting msgs back onto queue not possible after kill/
crash.
apparently stable, millions of messages / day with workling + starling.
we are using starling at play/type and for us, it’s fine. but if replay for huge traffic /
destructive reads are an issue, starling isn’t for you.
PLAY/TYPE
TODOs
workling is up on github. fork it! here’s what needs to be done, come join the project.
PLAY/TYPE
MemcachelikeRunner
PLAY/TYPE
MemcachelikeRunner
take the StarlingRunner and refactor it to be generic for all Queue Systems that imitate the
memcache api. once this is done, we’ll be able to plug in the following...
sparrow + workling running out there, no code unfortunately.
PLAY/TYPE
MemcachelikeRunner
take the StarlingRunner and refactor it to be generic for all Queue Systems that imitate the
memcache api. once this is done, we’ll be able to plug in the following...
sparrow + workling running out there, no code unfortunately.
PLAY/TYPE
MemcachelikeRunner
take the StarlingRunner and refactor it to be generic for all Queue Systems that imitate the
memcache api. once this is done, we’ll be able to plug in the following...
sparrow + workling running out there, no code unfortunately.
PLAY/TYPE
BeanstalkdRunner
BeanstalkdRunner
BeanstalkdRunner
AMPQRunner
PLAY/TYPE
BackgroudndRB
FUD?
“
I wish, people will check their facts before
making any claims, I am kinda getting tired of
fighting this FUD within community. There are
few outstanding issues, but BackgrounDRb
supports many features that other similar
alternatives doesn’t offer. And I am working on
it.
- Hemant
BackgroundRB
PLAY/TYPE
BackgroundRB
• As of version1.0.3 - complete rewrite with
Packet, no DRB code in there anymore.
PLAY/TYPE
- Hemant
my personal impression: still heavy. waiting for somebody to try integrating it into workling,
no personal need.
PLAY/TYPE
have a real world examle. old school, circa Feb. 2008 social network imports over gmail
scraping... need this out of the request, but the response has to be shown, too.
PLAY/TYPE
explain how this works - scraping gmail. return store: again, using memcache api.
PLAY/TYPE
1 def poll
2 @results = Workling::Return::Store.get \
3 params[:workling_uid]
4
5 # TODO: handle no results, results
6 # and results with errors
7 end
PLAY/TYPE
Rany Keddo
rany@playtype.net
Questions?
Lunch!