4/9/2020 2.

Event Driven Microservice Fundamentals - Building Event-Driven Microservices

 Building Event-Driven Microservices

PREV NEXT
⏮ ⏭
1. Why Event-Driven Microservices 3. Communication and Data Contracts
  🔎

Chapter 2. Event Driven Microservice

Fundamentals

Introduction
An event-driven microservice is a small application built to fulfil a specific
bounded context. Consumer microservices consume and process events from one
or more input event streams, whereas producer microservices produce events to
event streams for other services to consume. It is common for an event-driven
microservice to be both a consumer of one set of input event streams and a
producer to another set of output event streams. These services may be stateless
(Chapter 6), stateful (Chapter 7) and may also contain synchronous request-
response APIs (Chapter 12). These services all share the common functionality of
sourcing and/or sinking their event data from and to the event broker.
Communication between event-driven microservices is completely
asynchronous.

Event streams are served by an event-broker, which we will look at in more detail
in the second half of this chapter. Running microservices at any meaningful scale
often necessitates the usage of deployment pipelines and container management
systems, also discussed near the end of this chapter.

Building Topologies
The term topology will come up frequently in this book, as it also will in other
materials on event-driven microservices. This term can often be used to mean the
processing logic of an individual microservice; It may also be used to refer to the
graph-like relationship between individual microservices, event-streams, and
request-response APIs. For the sake of disambiguation, let’s use the following
terminology:

Microservice Topology The event-driven topology internal to a single

microservice. This defines the data-driven operations to be performed on
incoming events, including any transformation, storage, and emission of new
events.

The example below shows a single microservice topology ingesting from two
input event streams.

Figure 2-1. A simple microservice topology

In the above example, the microservice topology ingests events from Event
Stream A and stores them in a data store. Meanwhile, Event Stream B is
ingested, certain events are filtered out, transformed, and then joined against the
stored state. The results are output to a new event stream. The ingestion,
processing, and output of the microservice are part of its topology.

TIP

A microservice topology details the inner workings of a

single microservice. It does not detail the relationship
between services.

Business Topology Refers to the set of microservices, event-streams, and APIs

that fulfil complex business functions. A business topology is an arbitrary
grouping of services, and may reflect the services owned by a single team,
department, or that fulfils a superset of complex business functionality. The
business communication structures detailed in chapter one provide the form of
the business topology. Microservices implement the business bounded contexts
and event streams provide the data communication mechanism for sharing cross-
context domain data.

TIP

A business topology details the relationships between

services. It does not detail the inner workings of any specific
service.

The example below shows a business topology with three independent

microservices and event streams. Note that the business topology does not detail
the inner workings of a microservice.

Figure 2-2. A simple business topology

Microservice 1 consumes and transforms data from event stream A, and produces
the results to event stream B. Microservice 2 and Microservice 3 both consume
from event stream B. Microservice 2 acts strictly as a consumer and provides a
You have 2 days le in your
REST API in trial,
which dataReddychan. Subscribe
can be accessed synchronously. Meanwhile, today. See pricing options.

https://learning.oreilly.com/library/view/building-event-driven-microservices/9781492057888/ch02.html 1/6
4/9/2020 2. Event Driven Microservice Fundamentals - Building Event-Driven Microservices

microservice 3 performs its own transformations according to its bounded

context requirements and outputs to event stream C. New microservices and
event streams can be added to the business topology as needed, coupled
asynchronously through event streams.

The Contents of an Event

An event can be anything that has happened within the scope of the business
communication structure. Receiving an invoice, booking a meeting room,
requesting a cup of coffee (yes, you can hook a coffee machine up to an event
stream), hiring a new employee, and the successful completion of arbitrary code
are all examples of events that happen within a business. It is important to
recognize that events can be anything that is important to the business. Once
these events start being captured, event-driven systems can be created to harness
and use them across the organization.

An event is a recording of what happened, much like how logs of an application

are recordings of what happened in a given application. The main difference with
events is that events are also the single-source of truth, as we discussed in
chapter 1. As the single-source of truth, the event must contain all the
information required to accurately describe what happened. To get an idea as to
how we store this data, lets take a look at the basic structure of an event.

The Structure of an event

Events are typically represented using a key-value format. The value is used to
store the complete details of the event, while the key is used for identification
purposes, routing, and aggregation operations on events with the same key. The
key is not a required field.

Key Value

Unique ID Details payload

Let’s take a look at the three main event types that we’ll be working with in this
book, and that you are likely to encounter in your own domain.

Unkeyed Event

These are used when describing an event as a singular statement of fact. An

example of this could be an event that indicates a customer interacted with a
product, such as a user opening a book entity on a digital book platform.

Key Value

- ISBN: 372719, Timestamp: 1538913600

Keyed Event

This event type contains a key but does not represent any entity data. This is
often used for partitioning the stream of events to guarantee data locality within a
single partition of an event stream (more on this later in the chapter). The
distinction is that the keyed event describes an interaction with an entity, but not
the entity itself. An example of this could be a stream of events, keyed on ISBN,
indicating which user has interacted with the book.

Key Value

ISBN: 372719 UserId: A537FE781BC

Entity Event

An entity is a unique thing, and is keyed on the unique id of that thing. The entity
event describes the properties and state of this thing at a given point in time. An
entity is most commonly an object in the business context. For a book publisher,
an example of this could be a book entity, keyed on ISBN, as shown in the table
below. The value field contains all the necessary information related to the
unique entity.

Key Value

ISBN: 372719 Author: Adam Bellemare

Entity events are particularly important in event-driven architectures. They

provide a continual history of the state of an entity, and can be used to materialize
state at any point in time, as we will see in the next section.

Building State from Entity Events

A stateful table can be materialized by applying entity events, in order, from an
entity event stream. Each entity event is upserted into the key-value table, such
that the most recently read event for a given key is represented. Conversely, a
table can be converted into a stream of entity events by publishing each update to
the event stream. This is known as the Table-Stream duality, and it is
fundamental to the creation of state in an event-driven microservice. This can be
seen in the Figure 2, where AA and CC both have the newest values in their
materialized table.

Figure 2-3. Materializing an event stream into a

table
In the same way, a table can have all updates recorded, and in doing so produce a
stream of data representing the state of the table over time. In the following
example, BB is upserted twice, while DD is upserted just once. The output
stream in Figure 3 shows three upsert events representing these operations.

Figure 2-4. Generating an event stream from the

changes applied to a table
A relational database table, for instance, is created and populated by issuing a
series of data insertion commands. These commands can be tracked as events,
and in fact, they are actually recorded as such by the binary log which underpins
the relational database. Each new row added is a new event in the binary log, as
is each row that is modified or deleted. By playing back the entire contents of the
binary log you are able to exactly reconstruct the actual relational table and all
You have 2 days le in
the your trial,
data contents within Reddychan.
it. Subscribe today. See pricing options.

https://learning.oreilly.com/library/view/building-event-driven-microservices/9781492057888/ch02.html 2/6
4/9/2020 2. Event Driven Microservice Fundamentals - Building Event-Driven Microservices

This table-stream duality is used for communicating state between event-driven

microservices. Any consumer client can read an event stream of keyed events
and reproduce the same state of the upstream producer. This simple yet powerful
pattern allows microservices to share state through events alone, without any
direct coupling between producer and consumer services.

Maintaining state for the processing of business logic is an extremely common

pattern in an event-driven architecture. It is a near-certainty that your entire
business model will not be able to fit in purely stateless streaming domain, as the
need for recalling past business decisions is important for making decisions
today. As a specific example, if your business is retail, you will need to know
your stock level to identify when you need to re-order, and to avoid selling
customers items you do not have. You would also want to be able to keep track of
your accounts payable and accounts receivable. Perhaps you also wish to have a
weekly promotion sent to all the customers who have provided you their emails.
All of these systems require that you have the ability to materialize streams of
events into current-state representations.

Event Data Definitions and Schemas

Event data serves as both the means of data storage and communication between
services. As such, it is important that both the producers and consumers of events
have a common understanding of the meaning of the data. The consumer will
always be required to know the meaning of the data it is consuming, including
which fields to look for their underlying business meaning. This requires a
common language for communication between producers and consumers, and is
analogous to an API definition between synchronous request-response services.

Schematization selections such as Avro and Protobuf provide two features that
are leveraged heavily in event-driven microservices. First, they provide an
evolution framework, where certain sets of changes can be safely made to the
schemas without requiring downstream consumers to make a code change.
Secondly, they also provide the means to generate typed classes (where
applicable) to convert the schematized data into plain old objects of the language
of your choice. This makes the creation of business logic far simpler and more
transparent in the development of microservices. Chapter 3: Event Theory and
Design covers these topics in greater detail.

Microservice Single Writer Principle

Each event stream has one and only one producing microservice. This
microservice is the owner of each piece of data produced to that stream. This
allows for the authoritative source of truth to always be known for any given
event, permitting the tracing of data lineage back through the system. Access-
control mechanisms, as discussed in Chapter TODO on Tooling, should be used
to enforce ownership and write boundaries.

Powering Microservices with the Event Broker

At the heart of every production-ready event-driven microservice platform is the
event broker. This is a system which receives events, stores them in a queue or
ordered event stream, and provides them for consumption by other processes.
Events are typically published to different streams based on their underlying
logical meaning, similar to how a database will have many tables, each logically
separated to contain a specific type of data.

Event broker systems suitable for large-scale enterprises all generally follow the
same model. Multiple, distributed event brokers work together in a cluster to
provide a platform for the production and consumption of event streams. This
model provides several essential features that are required for running an event
driven ecosystem at scale

Scalability: Additional event broker instances can be added to increase

production, consumption and data storage capacity of the cluster.

Durability: Event data is replicated between nodes. This permits a cluster of

brokers to both preserve and continue serving data when a broker fails.

High Availability: A cluster of event broker nodes provides clients with the
means of connecting to other nodes in the case of a broker failure. This permits
the clients to maintain full uptime.

High Performance: Multiple broker nodes share the production and

consumption load. In addition, each broker node must be highly performant to be
able to handle hundreds of thousands of writes or reads per second.

Though there are different ways in which event data can be stored, replicated and
accessed behind the scenes of an event broker, they all generally provide the
same mechanisms of storage and access to their clients.

Event Storage and Serving

The minimal requirements of the underlying storage of the data by the broker.

Partitioning: Event streams can be partitioned into individual sub-streams, the

quantity of which can vary depending on the needs of the producer and
consumer. This partitioning mechanism allows for multiple instances of a
consumer to process each sub-stream in parallel, allowing for far greater
throughput. Note that queues do not require partitioning, though it may be useful
to do so anyways for performance purposes.

Strict Ordering: Data in an event stream partition is strictly ordered, and it is

served to clients in the exact same order that it was originally published.

Immutability: All event data is completely immutable once published. There is

no mechanism which can modify event data once it is published. Alterations to
previous data can be made only by publishing a new event with the updated data.

Indexing: Events are assigned an index when written to the event stream. This is
used by the consumers to manage the consumption of data, as they can specify
which offset to begin read from. The difference between the consumers current
index and the tail index gives a measurement of consumer lag. This metric can be
used to scale up the number of consumers when high, and scale them down when
it is low. Additionally, it can also be used to awaken Functions-as-a-Service
logic.

Infinite Retention: Events streams need the option to retain events for an infinite
period of time. This property is foundational for the maintenance of state in an
event stream.

Replayability: Event streams must be replayable, such that any consumer can
read whatever data it requires. This provides the basis for the single source of
truth and is foundational for communicating state between microservices.

Additional Factors to Consider

Support Tooling Support tools are essential for effectively developing event-
driven microservices. Many of these tools are bound to the implementation of the
event-broker itself. Some of these include: * Browsing of event and schema data
* Quotas, access-control, and topic management * Monitoring, throughput, and
lag measurements

Chapter TODO: Supportive Tooling covers this subject in greater detail.

Hosted Services

Will you purchase a hosted solution or host it internally?

Do hosted solutions exist?

You have 2 days le in your trial, Reddychan. Subscribe today. See pricing options.

https://learning.oreilly.com/library/view/building-event-driven-microservices/9781492057888/ch02.html 3/6
4/9/2020 2. Event Driven Microservice Fundamentals - Building Event-Driven Microservices

Does the hosting agent provide monitoring, scaling, disaster recovery,

replication, and multi-zone deployments?

Does it couple you to a single specific service provider?

Are there professional support services available?

Client Libraries and Processing Frameworks

Do client libraries and frameworks exist in the required languages?

Will you be able to build the libraries if they do not exist?

Are you using commonly-used frameworks or trying to roll your own?

Community Support * Is there online community support? * Is the technology

mature and production-ready? * Is the technology commonly-used across many
organizations? * Is the technology attractive to prospective employees? * Will
employees be excited to build with these technologies?

Event Brokers vs. Message Brokers

There can sometimes be confusion around what constitutes a message broker and
what constitutes an event broker. Event brokers may be usable in place of a
message broker, but a message broker cannot fulfill all the functions of an event
broker.

Message brokers have a long history and have been used in large-scale message-
oriented middleware architectures by numerous organizations. Message brokers
provide the means for systems to communicate across a network through
publish/subscribe message queues. Producers write messages to a queue, while a
consumer consumers these messages and processes them accordingly. Messages
are then acknowledged as consumed and deleted either immediately or shortly
thereafter. Message brokers are designed to handle a different type of problem
than event-brokers, and have been proven to be very successful at providing the
middleware necessary in message-oriented middleware architectures.

“Event brokers are much more about the immutable, append-only log of facts and
the model of consumers being being able to pick up and reprocess from
anywhere in the log at any time. It’s incredibly powerful and is why Kafka has
been so hugely successful (though obviously there are others). It’s about
preserving the state of event ordering.”

Event brokers, on the other hand, are designed around providing an ordered log
of facts. Event brokers meet two very specific needs that are not provided by the
message broker. For one, the message broker only provides queues of messages,
where the consumption of the message is handled on a per-queue basis.
Applications which share consumption from a queue will each receive only a
subset of the records. This makes it impossible to correctly communicate state
via events, since each consumer is not able to obtain a full copy of all events.
Unlike the message broker, the event broker maintains a single ledger of records
and manages individual access via indices, so that each independent consumer
can access all required events.

Additionally, a message broker deletes events after acknowledgement, whereas

an event broker will retain events for as long as is needed by the organization.
The deletion of the event after consumption makes a message broker insufficient
for providing the indefinitely-stored, globally-accessible, replayable, single-
source of truth for all applications.

Keep in mind that the concept of queues, as used in message brokers, still have a
role in event-driven microservices. Queues provide useful access patterns that
may be awkward to implement with strictly partitioned event streams. The
patterns introduced by message broker systems are certainly valid patterns for
EDM architectures, but they are not sufficient for the full scope of duties required
by a full EDM architecture. For the reasons discussed above, the remainder of the
book will not focus on any message broker architectures or application design,
but will focus instead on the usage of event brokers in event-driven microservice
architectures.

Consuming from the Immutable Log

Though not a definitive standard, most of the commonly available event brokers
use an append-only immutable log. Events are appended at the end of the log and
are given an auto-incrementing index ID. Consumers of the data use a reference
to the index ID to access data. Events can then be consumed as either an event-
stream or as a queue, depending on the needs of the business and the available
functionality of the event broker.

CO N S U M I N G A S A N E V E N T ST R E A M

Each consumer is responsible for updating its own pointers to previously-read

indices within the event stream. This index is known as the offset, as it is the
measurement of the current event from the beginning of the event stream. Offsets
permit multiple consumers to consume and track their progress independently of
one another, as shown in Figure 1.

Figure 2-5. Consumer groups and with their per-

partition offsets
This same offset mechanism allows for horizontal scaling of message
consumption. The consumer group concept allows for multiple consumers to be
viewed as the same logical entity. The events can be shared across all consumers
belonging to the same consumer group. The number of consumer processes in the
group is limited to the number of partitions in the event stream.

CO N S U M I N G A S A Q U E U E

In queue-based consumption, each event is consumed by one and only one

microservice instance. Upon being consumed, that event is marked as
“consumed” by the event broker and is no longer provided to any other instance.
Partition counts do not matter when consuming as a queue, as any number of
consumer instances can be used for consumption. Queues are not supported by
all event brokers. While Apache Pulsar supports queues at time of writing,
Apache Kafka does not.

Figure 2-6. Consuming from an immutable-log as a

queue
Example 2-1.

Event order is not maintained if multiple processing instances are consuming

from a queue, as one instance may process messages faster than another.

You have 2 days le in your trial, Reddychan. Subscribe today. See pricing options.

https://learning.oreilly.com/library/view/building-event-driven-microservices/9781492057888/ch02.html 4/6
4/9/2020 2. Event Driven Microservice Fundamentals - Building Event-Driven Microservices

Providing a Single Source of Truth

The durable and immutable log provides the storage mechanism for the single
source of truth, with the event broker becoming the only location in which
services consume and produce data. In doing so, each consumer is guaranteed to
be given an identical copy of the data as all the other consumers.

Adopting the event broker as the single source of truth often requires a culture
shift in the organization. Whereas previously a team may simply write direct
SQL queries to access data in a monolith’s database, now the monolith’s data
must additionally be published to the event broker. The developers managing the
monolith must ensure that the data produced is fully accurate, for any
disagreement between the event streams and the monolith’s database must always
be considered a failure of the producing team. Consumers of the data no longer
couple directly on the monolith, but instead consume directly from the event
streams.

The adoption of event-driven microservices results in the creation of services that

only use the event broker to store and access data. While local copies of the
events may certainly be used by the business logic of the microservice, the single
source of truth for all data remains the event broker.

Managing Microservices at Scale

Managing microservices can become increasingly difficult as the number of
services grows. Each microservice requires specific compute resources, data
stores, configurations, environment variables, and a whole host of other
microservice-specific properties. Each microservice must also be manageable
and deployable by the team that owns it, with simple control mechanisms
exposed to facilitate this. To achieve this, we generally look to containerization,
virtualization and their associated management systems. Both of these
approaches allows for individual teams to customize the requirements of their
microservices and provide the means of a single unit of deployability.

Putting Microservices into Containers

Containers, as recently popularized by Docker, provide the means to isolate

applications from one another. Containers leverage the existing host operating
system via a shared-kernel model. This provides basic isolation between
containers, while the container itself allows for isolated environment variables,
libraries and other dependencies. Containers provide most of the benefits of a
virtual machine at a fraction of the cost, with fast startup times and low resource
overhead.

The shared operating-system approach of containers does have some tradeoffs.

Containerized applications must be able to run on the host OS. If an application
requires a specialized operating systems, then an independent host will need to be
set up. Security is one of the more major concerns, since containers do share
access to the host machine’s operating system. If there is a vulnerability in the
kernel it can put all the containers on that host at risk. With friendly workloads it
is unlikely to be a problem, but current shared tenancy models in cloud
computing are beginning to make this a bigger concern, which brings us to our
next section.

Putting Microservices into Virtual Machines

While most microservice implementations tend to use containers, virtual

machines (VMs) may also be used as a much more isolated alternative.
Traditional VMs provide full isolation with a self-contained operating system and
virtualized hardware specified for each instance. Although this provides higher
security than containers, it has historically been much more expensive. Each VM
has higher overhead costs compared to containers, with slower startup times and
larger system footprints.

Initiatives are being taken by numerous competitors to make VMs cheaper and
more efficient. These include Google’s gVisor, Amazon’s Firecracker, and Kata
Containers, to mention just a few currently notable candidates. As these
technologies improve VMs will become a much more competitive alternative to
using containers for your microservice needs. It is worth keeping an eye on this
domain should your needs be driven by security-first requirements.

Managing Containers and Virtual Machines

The management of containers and VMs are available through a variety of

purpose-built software known as Container Management Systems (CMS). These
provide the means of controlling container deployment, resource allocation, and
integration with the underlying compute resources. Popular and commonly used
CMSs include Kubernetes, Docker Engine, Mesos Marathon, Amazon ECS and
Nomad.

Microservices must be able to scale up and down depending on changing

workloads, service level agreements, and performance requirements. Vertical
scaling must be supported, in which the compute resources such as CPU,
memory or disk are increased or decreased on each microservice instance.
Horizontal scaling must also be supported, with new instances added or removed.

Each microservice should be deployed as a single unit. For many microservices,

a single executable is all that is needed to perform its business requirements, and
it can be deployed within a single container. Others microservices may be more
complex, with multiple containers and external data stores requiring
coordination. This is where something like Kubernetes’ pod concept comes into
play, allowing for multiple containers to be deployed and reverted as a single
action. Kubernetes also allows for single-run operations, like database migrations
can be run during the execution of the single deployable.

Virtual Machine management is supported by a number of implementations, but

is currently more limited than that of containers. Kubernetes and Docker Engine
supports Google’s gVisor and Kata Containers, while Amazon’s platform
supports AWS Firecracker. The lines between containers and VMs will continue
to blur as development continues. Ensure that the CMS you select will handle the
containers and VMs that you require of it.

Finally, there are rich sets of resources available for Kubernetes

(http://shop.oreilly.com/product/0636920043874.do), Docker
(http://shop.oreilly.com/product/0636920153566.do), Mesos
(http://shop.oreilly.com/product/0636920039952.do), Amazon ECS, and Nomad. The
information provided goes far beyond what I can present here in this limited
scope. I encourage you to look into these materials for more information, as well
as look at some of the previous resources in this section for more information.

Summary
In this chapter we took a look at the basic requirements behind event-driven
microservices. The event broker forms the main mechanism of data
communiation, providing realtime event streams at scale for other services to
consume. Containerization and container management systems provide the
mechanisms necessary for running microservices at scale. Lastly, we take a look
at some of the important principles underlying events, event-driven logic and a
first look at how to manage state in an distributed event-driven world.

Settings / Support / Sign Out

You have 2 days le in your trial, Reddychan. Subscribe today. See pricing options.

https://learning.oreilly.com/library/view/building-event-driven-microservices/9781492057888/ch02.html 5/6
4/9/2020 2. Event Driven Microservice Fundamentals - Building Event-Driven Microservices

© 2020 O'Reilly Media, Inc. Terms of Service / Privacy Policy

PREV NEXT
⏮ ⏭
1. Why Event-Driven Microservices 3. Communication and Data Contracts

You have 2 days le in your trial, Reddychan. Subscribe today. See pricing options.

https://learning.oreilly.com/library/view/building-event-driven-microservices/9781492057888/ch02.html 6/6