Scaling RabbitMQ To 11

Scaling RabbitMQ to 11
Gavin M. Roy @crad - Surge 2011
INTERACTIVE POLLS

http://pollev.com/gmr SMS: 22333
myYearbook.com

#1 Comscore Teen Destination, Top Trafficked Sites in US Use of RabbitMQ dates back to 2009 Multiple clusters for different purposes Browser <-> Backend Message Bus Messaging, Event Processing
DISASTER PORN
ABOUT AMPQ

Platform, Vendor Neutral Messaging format 0.9.1 finalized in November 2008 JP Morgan Chase, Red Hat, Rabbit Technologies, iMatrix, IONA Technologies, Cisco Systems, Envoy Technologies, Twist Process Innovations
Members now include Bank of America, Barclays Banks, Microsoft Corporation, Novell, VMWare 1.0 Final: October 2011
5
ABOUT RABBITMQ

Born in 2006, Released in 2007, Acquired in 2010 Written in Erlang/OTP Talks AMQP 0.8 and 0.9.1 Features include: Clustering, Active/Active HA Queues Management UI and API Plugin Architecture
6
ABOUT YOU & RABBITMQ

RABBITMQ & AMQP CONCEPTS

PUBLISHERS
CONSUMERS
QUEUES

Durability: Queue survives reboot Auto-Delete: Delete the queue when consumer goes away Exclusive: Only one consumer may consume x-expires: Auto-Delete after given duration x-message-ttl: Auto-discard a message after ttl
11
MESSAGES

Have client specified properties: Content type, Encoding, Timestamp, App-Id, User-Id, Headers Delivery Mode: 1: Non-Persistent 2: Persistent
12
MESSAGE ROUTING
Its not just about Queues.
ROUTING KEY
In addition to the exchange, determines what is done with a message.
CC & BCC
More routing options per message, but violates AMQP.
HEADERS EXCHANGE
More on this later.
EXCHANGES
FANOUT

1:N Message Delivery Pattern No Routing Keys Involved Queues bound to a fanout exchange get all messages sent to the exchange
18
DIRECT

1:1 Message Delivery Pattern Routing Keys are direct, no wildcarding All queues bound to a routing key get the message
19
TOPIC EXCHANGES

Pattern Matching in Routing Keys Publish one message type to Surge.Sessions, another to Surge.Mentions Period delimiter, * stays within the period scope, # does not Example patterns: *.Mentions, Surge.*, # Slower than Fanout and Direct
20
HEADERS EXCHANGES

Matches on Basic.Properties headers table values to the arguments table specified when binding a queue Use x-match when declaring exchange to specify any or all headers must match Slow implementation in Erlang
21
PLUGIN EXCHANGES

Consistent Hash Exchange: Round-robin distribution to queues External: RPC Exchange Plugin Script: Similar in intent to External Riak Last-Value Cache Recent History Exchange: Keeps last 20 messages routed Global Fanout: Sends to all queues, regardless of routing key
22
ONE MORE THING...

Exchange-to-Exchange Bindings
CLUSTERS
STATS, DISK AND RAM NODES

WHERE DOES THE QUEUE LIVE?

CROSS-NODE PUBLISHING & CONSUMING

MAX CLUSTER SIZE

~32 notes is a realistic maximum node count for a cluster
SCALING ISSUES?
WHAT TO WATCH
RABBITMQ
Y U NO TAKE MESSAGE?
TCP BACKPRESSURE MUCH?

MEMORY MANAGEMENT
[{RABBIT, [{VM_MEMORY_HIGH_WATERMARK, 0.4}] }].
40%
MESSAGE PERSISTENCE
50-70% performance penalty compared to non-persisted messages. *
http://www.rabbitmq.com/faq.html#performance-persistent
CONSUMING
Ack, Nack and Reject vs Basic.Consume(no_ack=True)
Server Basic.Deliver
Client Basic.[Ack/Nack/Reject]
Basic.[Ack/Nack/Reject]Ok
BASIC.QOS

Prefetch Size and Count How many messages the client wants the RabbitMQ to pre-deliver to the client
Doesnt work in no_ack
TOO MANY SHORT LIVED CONNECTIONS

APACHE + MOD_PHP
CONNECTION PERSISTENCE
100 SERVERS 125 BACKENDS 1 REQUEST/S 12,500 CONN/S

AMQP CONNECTION SEQUENCE

Client Protocol Header Frame Connection.Start Connection.StartOk Connection.Tune Connection.TuneOk Connection.Open Connection.OpenOk Channel.Open Channel.OpenOk Server
685 BYTES PER 8.16 MB/S IN STARTUP @ 12,500 CONN/S

VORPAL BUNNY
VORPAL BUNNY

Light-weight PHP client for RabbitMQ Used the experimental JSON-RPC Channel plugin Handles the overhead of the HTTP, JSON-RPC to AMQP analog Fire and forget message publishing Can not be transparently load balanced
48
TOO MANY CONNECTIONS

VELOCITY TOO HIGH FOR ONE NODE?

MEMORY UTILIZATION DISPROPORTIONATE TO QUEUE DEPTHS

CLUSTER NODE FAILURES

UP TO 60 SECONDS TO DETECT
MESSAGES DROPPED ON THE FLOOR*

* 2.6.0 ADDED ACTIVE/ACTIVE QUEUES IN CLUSTERS

ACTIVE/ PASSIVE FAILOVER?

storage for Mnesia databases.
Recommendation is to use Pacemaker and DRBD or shared
RECOVERY?
CLUSTERING ALTERNATIVES
FEDERATION

Exchange Plugin Provides ability to route all messages to a x-federation exchange to another broker
SHOVEL PLUGIN

From Queue on one broker to an exchange on another broker. Effective for cross-cluster data broadcasting
TRENDING & MONITORING

RabbitMQ Message Velocity Rates Gotcha Management API Plugin exposes RESTful API including monitoring data points 2.6.1 allows monitoring tag for users to allow data access without full admin access Queue.Declare(passive=True) gets queue depths as well Plug in to Graphite, Nagios,or your favorite tool
61
PHOTO CREDITS
skateboard kids 7 2007 Volker Neumann http://flickr.com/prawnpie
methodcn contest 2009 Erin OHara http://flickr.com/erinohara73/
USPS Truck Accident Valley City Fire Department http://www.valleycityfire.com
62
QUESTIONS?

Scaling RabbitMQ To 11

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Scaling RabbitMQ To 11

Uploaded by

Copyright:

Available Formats

Scaling RabbitMQ to 11

Gavin M. Roy @crad - Surge 2011

Gavin M. Roy @crad - Surge 2011

Gavin M. Roy @crad - Surge 2011

Gavin M. Roy @crad - Surge 2011

Gavin M. Roy @crad - Surge 2011

ABOUT YOU & RABBITMQ

RABBITMQ & AMQP CONCEPTS

Gavin M. Roy @crad - Surge 2011

Gavin M. Roy @crad - Surge 2011

More routing options per message, but violates AMQP.

Gavin M. Roy @crad - Surge 2011

Gavin M. Roy @crad - Surge 2011

Gavin M. Roy @crad - Surge 2011

Gavin M. Roy @crad - Surge 2011

Gavin M. Roy @crad - Surge 2011

ONE MORE THING...

STATS, DISK AND RAM NODES

WHERE DOES THE QUEUE LIVE?

CROSS-NODE PUBLISHING & CONSUMING

MAX CLUSTER SIZE

Gavin M. Roy @crad - Surge 2011

Gavin M. Roy @crad - Surge 2011

Gavin M. Roy @crad - Surge 2011

Gavin M. Roy @crad - Surge 2011

TCP BACKPRESSURE MUCH?

[{RABBIT, [{VM_MEMORY_HIGH_WATERMARK, 0.4}] }].

Gavin M. Roy @crad - Surge 2011

Gavin M. Roy @crad - Surge 2011

Gavin M. Roy @crad - Surge 2011

Gavin M. Roy @crad - Surge 2011

Doesnt work in no_ack

Gavin M. Roy @crad - Surge 2011

TOO MANY SHORT LIVED CONNECTIONS

100 SERVERS 125 BACKENDS 1 REQUEST/S 12,500 CONN/S

AMQP CONNECTION SEQUENCE

Gavin M. Roy @crad - Surge 2011

685 BYTES PER 8.16 MB/S IN STARTUP @ 12,500 CONN/S

Gavin M. Roy @crad - Surge 2011

Gavin M. Roy @crad - Surge 2011

TOO MANY CONNECTIONS

VELOCITY TOO HIGH FOR ONE NODE?

MEMORY UTILIZATION DISPROPORTIONATE TO QUEUE DEPTHS

CLUSTER NODE FAILURES

MESSAGES DROPPED ON THE FLOOR*

* 2.6.0 ADDED ACTIVE/ACTIVE QUEUES IN CLUSTERS

ACTIVE/ PASSIVE FAILOVER?

Recommendation is to use Pacemaker and DRBD or shared

Gavin M. Roy @crad - Surge 2011

Gavin M. Roy @crad - Surge 2011

Gavin M. Roy @crad - Surge 2011

TRENDING & MONITORING

Gavin M. Roy @crad - Surge 2011

skateboard kids 7 2007 Volker Neumann http://flickr.com/prawnpie

methodcn contest 2009 Erin OHara http://flickr.com/erinohara73/

USPS Truck Accident Valley City Fire Department http://www.valleycityfire.com

Gavin M. Roy @crad - Surge 2011

You might also like