Building Event Driven Services

with Apache Kafka, Kafka

Streams & KSQL
Ben Stopford
@benstopford

1
There is a book!

http://bit.ly/designing-event-driven-systems

2
Event Driven Architectures
Business Events
Event Sourcing
DDD

Stream
Processing

3
Today’s ecosystems get pretty big

• 2.2 trillion messages per day (6

Petabytes)
• Up to 400 Microservices pre
cluster.
• 20-200 Brokers per cluster

4 4
Today’s ecosystems get pretty big

• 1 billion messages per day

• 20,000 messages per second
• 100 teams

5 5
Event Driven Architectures
Business Events
Event Sourcing
DDD

Stream
Processing

6
Streaming Platforms
Kafka Streams /
KSQL
KAFKA

Serving
Layer
(Cassandra etc.)

High Throughput Data is embedded in Clustered

Messaging each engine Java App

7
 Streaming Pipeline
apps_opened
(a) Apps Opened,
Per App, Per day

opened_per_day

(c) Unstable
Applications unstable_apps
crashed_per_day

app_crashes (b) Crashes,

Per App, Per day
8
Streaming Platforms
Kafka Streams /
KSQL
KAFKA

Serving
Layer
(Cassandra etc.)

High Throughput Data is embedded in Clustered

Messaging each engine Java App

9
 An event log is a simple idea

Old New

Messages are added at the end of the log

10
Readers have a position all of their own

George
is here Scan

Old New

Fred Sally
is here Scan is here Scan

11
You can rewind and replay, just like Tivo!

Old New

Sally
is here Scan

12
The hard part: Tying it all together!

13
 Many ”logs” over many machines
Kafka

Producing Consuming
Services Services

14
 Resistant to Failure
Kafka

Producing Consuming
Services Services

15
Streaming Platforms
Kafka Streams /
KSQL
KAFKA

Serving
Layer
(Cassandra etc.)

High Throughput Data is embedded in Clustered

Messaging each engine Java App

16
 Streaming Example

apps_opened opened_per_day

17
 apps_opened opened_per_day

CREATE TABLE opened_per_day AS

SELECT app_id, count(*)
FROM apps_opened
WINDOW TUMBLING (SIZE 1 DAY)
GROUP BY app_id;

18
 apps_opened opened_per_day

CREATE TABLE opened_per_day AS

SELECT app_id, count(*)
FROM apps_opened
WINDOW TUMBLING (SIZE 1 DAY)
GROUP BY app_id;

19
 apps_opened opened_per_day

CREATE TABLE opened_per_day AS

SELECT app_id, count(*)
FROM apps_opened
WINDOW TUMBLING (SIZE 1 DAY)
GROUP BY app_id;

20
 apps_opened opened_per_day

CREATE TABLE opened_per_day AS

SELECT app_id, count(*)
FROM apps_opened
WINDOW TUMBLING (SIZE 1 DAY)
GROUP BY app_id;

21
 apps_opened opened_per_day

CREATE TABLE opened_per_day AS

SELECT app_id, count(*)
FROM apps_opened
WINDOW TUMBLING (SIZE 1 DAY)
GROUP BY app_id;

22
 apps_opened opened_per_day

CREATE TABLE opened_per_day AS

SELECT app_id, count(*)
FROM apps_opened
WINDOW TUMBLING (SIZE 1 DAY)
GROUP BY app_id;

23
Streaming is manipulating events in flight,
at scale.

24
Event Driven Architectures
Business Events
Event Sourcing
DDD

Stream
Processing

25
Increasingly we build ecosystems

App Ecosystems

26
SOA / Microservices / EDA

Customer
Service

Shipping
Service

27
The Problem is DATA

28
Most services share the same core facts.

Most services live

in here

Catalog

29
 Events have two hats

Notification Data
replication
30
Buying an iPad (with REST/RPC)

Webserver
Submit
Order

Orders Shipping Customer

Service Service Service

shipOrder() getCustomer()

31
 Events for Notification Only

Webserver
Submit
Notification Order

Orders Shipping Customer

Service Service Service
REST

Order getCustomer()
Created
Message Broker (Kafka) KAFKA 32
 Pluggability

Webserver
Submit
Notification Order

Orders Shipping Customer

Service Service Service
Repricing
REST

Order getCustomer()
Created
KAFKA 33
Events for Data Locality

Webserver
Submit
Order Data is
replicated
Orders Shipping Customer
Service Service Service

Order Customer
Created Updated
KAFKA 34
 Events have two hats

Notification Data
replication
35
Stateless / Stateful Stream Processing
Relates to these hats

36
 Stateless Stream Processing
Webserver
Kafka Steams
Submit / KSQL
Notification Order KStreams
Shipping Service

Orders Customer
Service Service

REST/RPC
Order getCustomer()
Created

Message Broker (Kafka) KAFKA

37
 Stateful Stream Processing
Webserver

Data Submit
Order KStreams
KTable
Shipping Service
replication
Orders Customer
Service Service

Order Customer
Created Updated

Message Broker (Kafka) KAFKA

38
 Streams & Tables
KStreams
Shipping Service
Join

Customers
Orders (Buffer All)
Stream
(Buffer)

Message Broker (Kafka) KAFKA

39
KSQL ~ KStreams

40
 Streaming is about

1. Joining & Operating on

Streams On Notification

2. Joining & Operating on

Materialized Tables Data Replication
41
 Kafka: a Streaming Platform
Producer Consumer

Connectors The Log Connectors

Streaming Engine
42
8 Steps to Streaming Services

43
1. Use events to decouple and collaborate

44
 Event Collaboration
Browser
Orders
Notification CQRS/ Service
Websocket
Webserver

Order
Order Order
Requested
Order
Received Validated Completed

KAFKA
45
2. Use Connect (& CDC) to evolve away
from legacy

46
Make Legacy Datasets Available via the Log
Browser
Orders
CQRS Service

Webserver

Order
Order Order
Requested
Order
Received Validated Completed

Products
Connect
KAFKA
47
3. Use the Single Writer Principal

48
State changes to a topic owned by one service

Browser
Orders
Service

Webserver

Order
Order Order
Requested
Order
Received Validated Completed

Products

Connect
KAFKA 49
Local consistency points in the absence
of Global Consistency
REST
Service
T4 T1 T2 Orders
Service

T3

Email
Service
T5
50
4. Use Kafka as a Shared Source of Truth
(Messaging that Remembers)

51
 Shared Source of Truth
Browser
Orders
Service

Webserver

Order
Order Order
Requested
Order
Received Validated Completed Reporting
Products

Connect
KAFKA
52
Product Catalogue stored in 3 places
Browser
Orders
Service

Webserver

Reporting view
may be “thinner”
Order
Order Order
Requested
Order
Received Validated Completed Reporting
Products

Connect
KAFKA
53
5. Move Data to Code

54
55
 Materialize Stock ‘View’ Inside Service
Browser
Orders Data
Service Stock Replication

Webserver

Order
Order
Requested
Order Order
Received Validated Completed
Products Stock
Connect KAFKA
56
Kafka has several features for reducing
the need to move data on startup

- Standby Replicas
- Disk Checkpoints
- Compacted topics

57
6. Write to State Stores, just like a
local ‘database’, backed up in Kafka

58
 State stores behave like local databases
Browser State Store
Orders Reserved Stocks
Service Stock

Webserver

Order
Order
Requested
Order Order Reserved Stocks
Received Validated Completed
Products Stock
Connect KAFKA
59
7. Use Transactions to tie All
Interactions Together

60
 Transactions
TRANSACTION
Browser
Orders Reserved Stocks
Service Stock

Webserver

Order
Order
Requested
Order Order Reserved Stocks
Received Validated Completed
Products Stock
Connect KAFKA
61
8. Evolve and Grow

62
 Tiered Contexts
Operations
Finance

KAFKA

Kafka

Front Office
63
Span regions or clouds

64
Handle Disconnectedness

65
So…
66
Optimize for complexity vs optimize for scale

Event Driven
Architectures

Stream
Processing
67
Events provide the key to evolutionary
architectures

Notification Data
replication
68
 Spectrum of use cases

Finer Grained, Courser Grained,

Collaborative, Non-collaborative,
Connected Disconnected

Notification Data Replication

69
 Streaming is the
toolset for dealing with
events at scale

70
 Event Driven Services

• Broadcast events
• Retain them in the log
• Evolve the event-stream with
streaming functions
• Recasting the event stream into
views when you need to query.
71
 Find out more
Book: http://bit.ly/designing-event-driven-systems

Software: https://confluent.io/download/

Code: http://bit.ly/kafka-microservice-examples

Twitter: @benstopford

72