Building Event Driven Services

with Apache Kafka and Kafka

Streams
Ben Stopford
@benstopford

1
There is a book!

http://bit.ly/designing-event-driven-systems

2
Event Driven Architectures
Event Sourcing (DDD)

Stream
Processing

3
Today’s ecosystems get pretty big

• 2.2 trillion messages per day (6

Petabytes)
• Up to 400 Microservices pre
cluster.
• 20-200 Brokers per cluster

4 4
Today’s ecosystems get pretty big

• 1 billion messages per day

• 20,000 messages per second
• 100 teams

5 5
Event Driven Architectures
Event Sourcing (DDD)

Events
Stream
Processing

6
Notification

Data
replication

7
Notification

Data
replication

8
Streaming Platforms
Kafka Streams /
KSQL
KAFKA

Serving
Layer
(Cassandra etc.)

High Throughput Data is embedded in Clustered

Messaging each engine Java App

9
 Streaming Pipeline
apps_opened
(a) Apps Opened,
Per App, Per day

opened_per_day

(c) Unstable
Applications unstable_apps
crashed_per_day

app_crashes (b) Crashes,

Per App, Per day
10
Streaming Platforms
Kafka Streams /
KSQL
KAFKA

Serving
Layer
(Cassandra etc.)

High Throughput Data is embedded in Clustered

Messaging each engine Java App

11
Streaming Platforms
Kafka Streams /
KSQL
KAFKA

Serving
Layer
(Cassandra etc.)

High Throughput Data is embedded in Clustered

Messaging each engine Java App

12
 An event log is a simple idea

Old New

Messages are added at the end of the log

13
Readers have a position all of their own

George
is here Scan

Old New

Fred Sally
is here Scan is here Scan

14
You can rewind and replay, just like Tivo!

Old New

Sally
is here Scan

15
The hard part: Tying it all together!

16
 Many ”logs” over many machines
Kafka

Producing Consuming
Services Services

17
 Resistant to Failure
Kafka

Producing Consuming
Services Services

18
Streaming Platforms
Kafka Streams /
KSQL
KAFKA

Serving
Layer
(Cassandra etc.)

High Throughput Data is embedded in Clustered

Messaging each engine Java App

19
 Streaming Example

apps_opened opened_per_day

changelog

20
 apps_opened opened_per_day

changelog

CREATE TABLE opened_per_day AS

SELECT app_id, count(*)
FROM apps_opened
WINDOW TUMBLING (SIZE 1 DAY)
GROUP BY app_id;

21
 apps_opened opened_per_day

changelog

CREATE TABLE opened_per_day AS

SELECT app_id, count(*)
FROM apps_opened
WINDOW TUMBLING (SIZE 1 DAY)
GROUP BY app_id;

22
 apps_opened opened_per_day

changelog

CREATE TABLE opened_per_day AS

SELECT app_id, count(*)
FROM apps_opened
WINDOW TUMBLING (SIZE 1 DAY)
GROUP BY app_id;

23
 apps_opened opened_per_day

changelog

CREATE TABLE opened_per_day AS

SELECT app_id, count(*)
FROM apps_opened
WINDOW TUMBLING (SIZE 1 DAY)
GROUP BY app_id;

24
 apps_opened opened_per_day

changelog

CREATE TABLE opened_per_day AS

SELECT app_id, count(*)
FROM apps_opened
WINDOW TUMBLING (SIZE 1 DAY)
GROUP BY app_id;

25
 apps_opened opened_per_day

changelog

CREATE TABLE opened_per_day AS

SELECT app_id, count(*)
FROM apps_opened
WINDOW TUMBLING (SIZE 1 DAY)
GROUP BY app_id;

26
Streaming is manipulating events in flight,
at scale.

27
 Event Driven
Architectures

Stream
Processing

28
Increasingly we build ecosystems

App Ecosystems

29
SOA / Microservices / EDA

Customer
Service

Shipping
Service

30
The Problem is DATA

31
Most services share the same core facts.

Most services live

in here

Catalog

32
 Events have two hats

Notification Data
replication
33
Buying an iPad (with REST/RPC)

Webserver
Submit
Order

Orders Shipping Customer

Service Service Service

shipOrder() getCustomer()

34
 Events for Notification Only

Webserver
Submit
Notification Order

Orders Shipping Customer

Service Service Service
REST

Order getCustomer()
Created
Message Broker (Kafka) KAFKA 35
 Pluggability

Webserver
Submit
Notification Order

Orders Shipping Customer

Service Service Service
Repricing
REST

Order getCustomer()
Created
KAFKA 36
Events for Data Locality

Webserver
Submit
Order Data is
replicated
Orders Shipping Customer
Service Service Service

Order Customer
Created Updated
KAFKA 37
 Events have two hats

Notification Data
replication
38
Stateless / Stateful Stream Processing
Relates to these hats

39
 Stateless Stream Processing
Webserver
Kafka Steams
Submit / KSQL
Notification Order KStreams
Shipping Service

Orders Customer
Service Service

REST/RPC
Order getCustomer()
Created

Message Broker (Kafka) KAFKA

40
 Stateful Stream Processing
Webserver

Data Submit
Order KStreams
KTable
Shipping Service
replication
Orders Customer
Service Service

Order Customer
Created Updated

Message Broker (Kafka) KAFKA

41
KSQL ~ KStreams

42
 Any code you desire

Orders

Table Kafka Streams API

Customers

43
 It’s just java

builder.stream(“Orders”)
.join(“Customers”,…)
.transform((key, value) ->
{//Any code your heart desires!}
)
.to(“Shipments”);

44
 Streaming can be stateful or stateless

1. Joining & Operating on

Streams On Notification

2. Joining & Operating on

Materialized Tables Data Replication
45
 Streaming Platform
Producer Consumer

Connectors The Log Connectors

Streaming Engine
46
Event Driven Example

47
1. Use events to decouple and to
collaborate

48
 Event Collaboration
Browser
Orders
Notification Service

Webserver

Order
Order Order
Requested
Order
Received Validated Completed

KAFKA
49
2. Use the Single Writer Principal

50
State changes to a topic owned by one service

Browser
Orders
Service

Webserver

Order
Order Order
Requested
Order
Received Validated Completed

KAFKA 51
Local consistency points in the absence
of Global Consistency
REST
Service
T4 T1 T2 Orders
Service

T3

Email
Service
T5
52
3. Convert legacy databases to Events
(with CDC)

53
Make Legacy Datasets Available via the Log
Browser
Orders
Service

Webserver

Order
Order Order
Requested
Order
Received Validated Completed

Products
Connect
KAFKA
54
4. Use Kafka as an Event Store

55
 Shared Source of Truth
Browser
Orders
Service

Webserver

Order
Order Order
Requested
Order
Received Validated Completed Reporting
Products

Connect
KAFKA
56
Product Catalogue stored in 3 places
Browser
Orders
Service

Webserver

Reporting view
may be “thinner”
Order
Order Order
Requested
Order
Received Validated Completed Reporting
Products

Connect
KAFKA
57
5. Derive “Materialized Views” instead of
caching

58
 Materialize Stock ‘View’ Inside Service
Browser
Orders Data
Service Stock Replication

Webserver

Order
Order
Requested
Order Order
Received Validated Completed
Products Stock
Connect KAFKA
59
 Is stateful a good idea?

- Standby Replicas
- Disk Checkpoints
- Compacted topics
- Or just use a database!

60
Database Inside Out Pattern
KAFKA: Retained Event Streams
Payments Products
Orders Customers

Queries run locally

(in database or
streaming engine)
61
6. Write to State Stores, just like a
local ‘database’, backed up in Kafka

62
 State stores behave like local databases
Browser State Store
Orders Reserved Stocks
Service Stock

Webserver

Order
Order
Requested
Order Order Reserved Stocks
Received Validated Completed
Products Stock
Connect KAFKA
63
7. Use Transactions to tie All
Interactions Together

64
 Transactions
TRANSACTION
Browser
Orders Reserved Stocks
Service Stock

Webserver

Order
Order
Requested
Order Order Reserved Stocks
Received Validated Completed
Products Stock
Connect KAFKA
65
 8. Evolve and
Grow through
Streaming
Functions

66
 Tiered Contexts
Operations
Finance

KAFKA

Kafka

Front Office
67
Span regions or clouds

68
Handle Disconnectedness

69
So…
70
Optimize for complexity vs optimize for scale

Event Driven
Architectures

Stream
Processing
71
Events provide the key to evolutionary
architectures

Notification Data
replication
72
 Spectrum of use cases

Finer Grained, Courser Grained,

Collaborative, Non-collaborative,
Connected Disconnected

Notification Data Replication

73
Events to transcend individual services

Notification

Data
replication
74
 Start Simple and Evolve

1. Broadcast events
2. Retain them in the log
3. Evolve the event-stream with
streaming functions
4. Recasting the event stream into
views when you need to query.
75
 Find out more
Book: http://bit.ly/designing-event-driven-systems
Software: https://confluent.io/download/

Cloud: https://www.confluent.io/confluent-cloud/

Kubernetes Operator: https://www.confluent.io/confluent-operator/

Example code: http://bit.ly/kafka-microservice-examples

Twitter: @benstopford
76