You are on page 1of 7

CLUSTERING HIVEMQ

Building highly available, horizontally


scalable MQTT Broker Clusters

5/2016
About this document
MQTT is based on a publish/subscribe architecture that decouples MQTT clients
and uses a central MQTT broker for distributing messages in a very performant and
efficient manner. Traditionally, MQTT brokers are deployed as a single instance. This
makes a broker deployment hard to scale and the single deployment is a Single
Point of Failure, which means the whole MQTT communication is offline when the
single broker fails.

This document discusses how you can eliminate these problems by creating a MQTT
broker cluster that is scalable in a horizontal fashion (which means you can add any
number of MQTT brokers at runtime) and eliminates the single point of failure by
clustering multiple broker nodes to one logical MQTT broker. You will learn how you
achieve true high availability and linear scalability with HiveMQ.

Table of Contents
1. Introduction .. 2
2. The Only MQTT Broker that scales horizontally 2
3. The Distributed Cluster explained .. 2
4. Availability - Paramount for the IoT . 3
5. Best in Class Performance and Scalability . 4
6. Zero Downtime Upgrades . 4
7. Discovery Mechanisms ...... 4
8. Support for all Major OS and Cloud Providers ........ 5
9. A Plugin System designed for MQTT Broker Clusters 5
10. Conclusion .... 6

dc-square GmbH 2016. All rights reserved. hivemq.com | 1


is why HiveMQ is built to be resilient
Introduction
and the cluster is able to self-heal in
case any disastrous events happen.
HiveMQ is a reliable and resilient
Network Splits are handled gracefully
high-performance MQTT Broker, en-
without losing the availability charac-
gineered in Germany for the profes-
teristics of certain cluster nodes; the
sional use in companies and enter-
cluster as a whole is always fully opera-
prises worldwide. Its a dedicated
tional.
MQTT Broker with a unique feature
set, optimized for millions of concur-
HiveMQ is able to scale to a vast
rent MQTT connections. The broker is
amount of cluster nodes and is truly
highly event-driven and multi-thread-
horizontal scalable by adding (or even
ed to ensure lowest latency and high-
removing) cluster nodes at runtime
est throughput even for the most am-
without any reconfiguration of existing
bitious MQTT projects.
cluster nodes. Its possible to achieve
extreme numbers of concurrent MQTT
One of the most outstanding and
clients by spawning additional
unique features of HiveMQ is the abili-
HiveMQ instances as soon as you
ty to form resilient MQTT broker clus-
need them - and easily remove them if
ters that eliminate a Single Point of
they are not needed anymore. All
Failure for the MQTT communication
prevalent load balancing solutions,
by providing high availability and true
both hardware and software, are fully
horizontal scalability due to the dis-
supported by HiveMQ.
tributed architecture of the HiveMQ
cluster.

The distributed Cluster


The only MQTT Broker explained
that scales horizontally
HiveMQ is designed with a true dis-
tributed and masterless cluster archi-
HiveMQ is the only MQTT Broker that
tecture, which means there is no sin-
scales in a true horizontal fashion
gle point of failure and the cluster can
without trading availability or MQTT
grow and shrink at runtime without
reliability guarantees. We all know
losing data or trading availability.
from basic computer science theory
MQTT clients can (re-)connect to any
that all systems eventually fail and this

dc-square GmbH 2016. All rights reserved. hivemq.com | 2


HiveMQ cluster node and can resume ker, even if a vast number of nodes is
their MQTT session. The HiveMQ clus- deployed. This is very important if a
ter implements a sophisticated and load balancer is in front of the nodes,
very efficient message routing that en- since the load balancer can distribute
sures the intra-cluster communication the client to any broker node.
is at a minimum while all MQTT guar-
antees are maintained at lowest laten-
cy.
Availability - Paramount
HiveMQs cluster is designed for both, for the IoT
high availability and scaling out use
cases. For ambitious MQTT deploy- Availability is key for critical in-
ments with many millions of clients, frastructure components - this is espe-
any number of broker nodes can be cially true for MQTT broker clusters.
deployed at runtime to scale out while Even if parts of the whole cluster fail,
maintaining stability and availability at the system as a whole must always be
any point of time. Full data replication available to avoid service interrup-
between cluster nodes is possible but tions.
is not needed - data can be dis-
tributed with configurable replica HiveMQ is a MQTT Broker that is re-
counts. silient against network splits and
strives to be always available, even in
The HiveMQ cluster is self-healing, the presence of network partitions.
which means that even if network This makes HiveMQ an eventual con-
splits occur or any kind of connectivity sistent system that leans towards con-
problem between nodes arise, the sistency under normal operation and
cluster as a whole is always available doesnt trade availability at any time.
and heals itself in error scenarios. No An arbitrary number of broker nodes
human interaction is required in such can fail or can be unavailable and the
cases. system as a whole is still online and
fully functional.
From a MQTT clients standpoint, the
cluster is completely transparent, This is one of the unique characteris-
which means the MQTT client can in- tics of HiveMQ that make it the most
teract with any node and the HiveMQ resilient and most scalable distributed
deployment always looks as one bro- MQTT Broker. A single MQTT broker

dc-square GmbH 2016. All rights reserved. hivemq.com | 3


node can potentially serve multiple
Zero Downtime Upgrades
hundreds of thousands and up to mil-
lions of open TCP connections, it HiveMQ supports Rolling Upgrades,
would not be viable to trade availabili-
which means that zero-downtime up-
ty characteristics and disconnect or
grades are possible when using a
halt these connected MQTT clients
HiveMQ cluster. All nodes can be up-
just because of a networking problem
graded one-by-one.
between brokers. HiveMQ never dis-
connects or halts MQTT clients just
When upgrading broker nodes, the
because the node theyre connected
cluster is still fully functional without
to is part of some kind of minority par-
any performance or reliability degra-
tition.
dations. In case of unforeseen prob-
lems when upgrading you can stop
the upgrade process and go back to
Best in Class Performance the older version any time, no data is
lost. DevOps will love the fact that the
and Scalability
update can be done in a completely
automated fashion.
A single HiveMQ node is designed for
multiple hundreds of thousands and
even millions of concurrently connect-
ed MQTT clients and its easy to scale Discovery Mechanisms
HiveMQ vertically by adding more re-
sources (CPU & RAM). Multiple Cluster Discovery Mecha-
nisms are supported by HiveMQ out-
The masterless and distributed archi- of-the-box. This means that you can
tecture of the HiveMQ cluster allows adapt the cluster elasticity to your
to scale horizontally, which means you project needs.
can add an arbitrary number of nodes
(or even remove nodes) at runtime to Beside static cluster configuration its
serve more MQTT clients when need- possible use auto-discovery mecha-
ed. Deployments that experience traf- nisms like UDP Multicast, TCP Broad-
fic or usage spikes can scale up quick- cast or even more sophisticated clus-
ly if needed. ter discovery mechanisms like AWS S3
discovery or Consul Discovery with off-
the-shelf plugins. You need to inte-

dc-square GmbH 2016. All rights reserved. hivemq.com | 4


grate with your own service registry programming styles are supported as
application? No problem, the discov- well as synchronous programming
ery mechanisms are pluggable via the paradigms for optimal plugin perfor-
powerful HiveMQ plugin API that is mance when needed.
available to developers.

Conclusion
Support for all Major OS
Due to HiveMQs sophisticated cluster
and Cloud Providers
design that is tailored specifically for
MQTT, HiveMQ is able to form highly-
When it comes to HiveMQ deploy-
available, elastically and horizontally
ments, public cloud providers like
scalable MQTT Broker clusters that
AWS, Microsoft Azure and Google
avoid a Single Point of Failure. The
Cloud Platform are first-class citizens
cluster is linearly scalable and nodes
as well as hybrid or private clouds.
can be added and removed at run-
HiveMQ runs on any operating system
time without any service interruption.
like Linux (e.g. RHEL, CentOS, Ubun-
tu, Debian) or Microsoft Windows
HiveMQ is designed to handle net-
Server.
work splits and the system is always
Its easy to scale out your cluster de- available, even if some parts of the in-
frastructure become unavailable. Ad-
ployments with Docker and tools like
vanced self-healing mechanisms pro-
Kubernetes or Mesos (with Marathon).
tect from fatal disasters.

These unique characteristics of the


A Plugin System designed cluster make HiveMQ a superior
for MQTT Broker Clusters choice for ambitious MQTT projects.

The HiveMQ Plugin system allows to


extend the broker with any custom Not sure if the HiveMQ cluster is suit-
business logic. All Plugin Services
able for your use case? Talk to one of
available for your own plugins are de-
our experts: contact@hivemq.com
signed for interaction with clustered
HiveMQ instances and single in-
stances. Very efficient asynchronous

dc-square GmbH 2016. All rights reserved. hivemq.com | 5


Contact us!
contact@hivemq.com
http://www.hivemq.com
dc-square GmbH 2016. All rights reserved. hivemq.com |