You are on page 1of 44

one system to rule

them all
the challenge
• write a backend that scales well
• handle various frontends (web, iPhone, ...)
• runs “somewhere in the cloud”
• easy to maintain and extend
the solution
RabbitMQ
+CouchDB
=Awesome
Part I
RabbitMQ

• AMQP based Message Queue


• lots of additional transports (XMPP, HTTP,
STOMP, ...)
• erlang based
frontend frontend frontend

daemon RabbitMQ

daemon daemon daemon


• every functionality gets a backed daemon
• every daemon has a queue associated
• communication via a JSON message that
gets passed around
the JSON message
{
"response": {
"expiry": 710,
"ts": "1234425918",
"name": "Lenz Gschwendtner",
"last4": 0,
"cc_type": "master"
},
"data": {
"options": {
"billing_id": "E16003D4-B312-BEA23D9AB789"
},
"command": "billing_info"
},
"meta": {
"lang": "en",
"reply_to": "B312-E16003D4-BEA23D9AB789",
"platform": "iwmn"
}
}
works great for ...

• RPC style communication


• if the flow through the system is static
• fast transactions
gets fiddly when ...

• there are asynchronous callbacks from


outside
• message flow can change based on various
factors
• tasks can take days to complete
Part II
workflows
... again ...
what we have ...

• various frontends
• backend daemons
• JSON messages
what we need

• a way to describe message flow


• a easy flexible way to write and update it
CouchDB
... to the rescue
workflow

• really only a path definition through


backends
• some sort of condition management
• a backend definition
{
"_id": "f7a4408898e5...a05b4181045",
"_rev": "3-633060011",
"name": "billing_info",
"type": "workflow",
"user": "iwmn",
"path": [
"logger",
"billing",
"billing_iwmn_cc"
]
}
... and the backend
definition
{
"_id": "33adf2e3efc3...f423929f057ac",
"_rev": "3-3507820969",
"conn": "rpc",
"type": "worker",
"name": "billing_iwmn_cc",
"location": "billing.cc",
"description": "the CC interface",
"command": "billing_verify_cc"
}
now we need a
workflow daemon
frontend frontend frontend

daemon RabbitMQ workflow

daemon daemon daemon


where are we now?

• we can call backends as before


• we can call workflows via the workflow
queue
• we can manage workflows in CouchDB
Part III
asynchronous callbacks
problem

• we start a workflow
• we call some external API
• the external API somehow notifies us
(Mail, Callback URL, ...) about progress
• we need to continue in the workflow
we need a persistent
message store
{
"_id": "22cd37afd06c82129adb665d3150f570",
"_rev": "3-2669449653",
"status": "done",
"last_update": 1252296819,
"name": "E16003D4-F8DB-11DD-B312-BEA23D9AB789",
"message": {
"response": {
"expiry": 710,
"name": "Lenz Gschwendtner",
"last4": 0,
"cc_type": "master"
},
"data": {
"options": {
"billing_id": "E16003D4-F8DB-11DD-B312-BEA23D9AB789"
},
"command": "billing_info"
},
"meta": {
"lang": "de",
"name": "E16003D4-F8DB-11DD-B312-BEA23D9AB789",
"orig_reply_to": "D234ADE8-9B64-11DE-A92C-E5EF11C6FBDB",
"reply_to": "workflow",
"user": "iwmn",
"log": [
"logger",
"billing",
"billing.iwmn.info"
],
"platform": "iwmn"
}
},
"created": 1252296815,
"type": "bill"
}
one more problem
how do we correlate messages
... with some CouchDB
awesome ...
• the backend daemon for the callback polls a
view in CouchDB

• we can search for any value in the request


we sent

• all we need is a view that searches for the


very value this external API uses as
correlation ID
external frontend frontend frontend
callback

daemon RabbitMQ workflow

daemon daemon daemon


me
Lenz Gschwendtner
CTO iWantMyName
@norbu09
credits

• RabbitMQ
• CouchDB
• Erlang and Perl
more credits

• http://www.flickr.com/photos/djpd
• http://www.flickr.com/photos/lilyapp
• http://www.flickr.com/photos/b-tal
• http://www.flickr.com/photos/mikebaird
• http://www.flickr.com/photos/eyetwist