Plone, RabbitMQ

and messaging
that just works
Asko Soukka & Jukka Ojaniemi

Speakers' background
University of Jyvskyl
the largest Plone user in Finland
83 Plone sites
2.5 TB data served through Plone
700 content editors

Demo (serializing writes)

Browser A

Browser B

RabbitMQ

M1

M1

Queue

Queue

M1

Plone

M1

Why message queues?

Why message queues?

We needed a reliable, effective and
scalable solution for:
running asynchronous tasks
being a common integration point
between services (decoupling).

Why message queues?

None of the existing ZODB-based mechanism
really scales. When you are really in need for a
real and working queuing mechanism one
would use some external queuing solution (e.g.
something AMQP compatible like RabbitMQ).
Andreas Jung

Advanced Message Queue

Protocol (AMQP)
Think of it as an highly advanced
post office which can deliver
message to multiple recipients
based on address.

AMQP jargon 1/2

Message consists of label and payload.
Producer is a program which sends
messages to exchange.
Consumer is a program which mostly waits
to receive messages.

AMQP jargon 2/2

Exchange receives the messages from
producers and pushes them to queues.
Bindings are how the messages get routed
from the exchange to a queue.
Queue is a buffer that stores messages.

Basic messaging
Work queues
M1
Message broker
M2
Producer

M1

Consumer

M2
Exchange
(direct)

M1

M2
Queue

M1
M2
Consumer

Basic messaging
Publish/Subscribe queues
M1
Message broker
Queue
Producer

M1

Exchange
(fanout)

M1

Consumer

M1
M1
Queue

M1

Consumer

Basic messaging
RPC implementation
Message broker
Exchange

M1

RPC queueM1

R1
Client/producer M1

M1
Server

R1 Reply_to
queue

R1

Exchange

R1

RabbitMQ
Implements AMQP (Advanced Message
Queue Protocol) open standard
Written in Erlang
made for messaging
easy to cluster
fast (enough)

Industry tested and reliable

Sounds complicated?

Example with
collective.zamqp
Publish and subscribe announcements
M1
Message broker
Queue
Site A

M1

Exchange
(fanout)

M1

Site B

M1
M1
Queue

M1

Site C

Produce messages with

collective.zamqp
from zope.component import getUtility
from collective.zamqp.interfaces import IProducer

producer = getUtility(
IProducer, name="announcer")
producer.register() # for transaction
producer.publish("Hello World!")

Handle messages with

collective.zamqp
from five import grok
from collective.zamqp.interfaces import IMessageArrivedEvent

@grok.subscribe(IAnnouncement,
IMessageArrivedEvent)
def handleMessage(message, event):
logger.info(message.body)
message.ack()

Overview of
collective.zamqp
Message handlers
(as event subscribers)
Producers

Consumers
IMessageArrivedEvent
stamp
messages

produce messages

Consuming Servers

AMQP Broker Connections

Message broker

consume
messages

Connections in
collective.zamqp
buildout.cfg:
[instance]
zope-conf-additional =
%import collective.zamqp
<amqp-broker-connection>
connection_id example
...
</amqp-broker-connection>

Producers in
collective.zamqp
class Announcer(Producer):
grok.name("announcements")
connection_id = "example"
exchange = "announcements"
exchange_type = "fanout"
serializer = "text/plain"
durable = False

Consuming servers in
collective.zamqp
buildout.cfg:
zope-conf-additional =
%import collective.zamqp
....
<amqp-consuming-server>
connection_id example
site_id Plone
</amqp-consuming-server>

Consumers in
collective.zamqp
class Announcements(Consumer):
grok.name("announcements")
connection_id = "example"
exchange = "announcements"
exchange_type = "fanout"
queue = "" # anonymous temp. queue
durable = False
marker = IAnnouncement

Message handlers in
collective.zamqp
@grok.subscribe(IAnnouncement,
IMessageArrivedEvent)
def handleMessage(message, event):
logger.info(message.body)
message.ack()

Overview of
collective.zamqp
Message handlers
(as event subscribers)
Producers

Consumers
IMessageArrivedEvent
stamp
messages

produce messages

Consuming Servers

AMQP Broker Connections

Message broker

consume
messages

It's so simple!

It really is!

Real world example

Moniviestin video publishing service
Plone 4.1 site
Distributed encoder servers (48 CPUs
combined)
First version used XML-RPC based
communication - lots of error handling code.
New version uses AMQP-messaging.
Very fast and scalable. Recovers from
network outages.

Encoder
Mediainfo
memcached

FS

RabbitMQ

CherryPy
upload
service

Plone

Thank you
c.zamqp and examples are available at:
https://github.com/datakurre/collective.zamqp
https://github.com/datakurre/collective.zamqpdemos
https://github.com/datakurre/chatbehavior
https://github.com/datakurre/pubsubannouncements
other relevant links:
https://www.rabbitmq.com/getstarted.html
https://www.amqp.org/about/examples
http://www.manning.com/videla/ (RabbitMQ in Action)