Issue #1

RISE OF
THE KAFKA HEROES
Located in the heart of Mountain View,
the trailblazing shoe retail titan, NewLimits,
is no stranger to success. However, confronted
by growing demands from customers for
real-time experiences and internal stakeholders
for up-to-date data, management is forced to
acknowledge the need for transformation. Yet,
the 'how' remains a mystery—hopefully one
that our intrepid heroes, Ada and Jax, can
solve with the aid of Apache Kafka®.

Renowned for its use by over 80% of Fortune

100 companies, Apache Kafka is hailed as
the defacto standard in the realm of data
streaming. As an open-source distributed
technology, it’s leveraged by countless
businesses to power their streaming data
pipelines and real-time applications.

That's where Confluent steps in,

transforming Kafka with a robust array of
enterprise-grade features that fosters
developer productivity, ensures efficient
operation at scale, and addresses all
architectural prerequisites for production.
Businesses deploying Confluent experience a
60% reduction in Kafka costs and accelerate
their streaming use cases into production by
over half a year, all the while significantly
diminishing operational risks and burdens.

Ada and Jax must fast-track the

implementation and scaling of solutions.
But wouldn’t it be simpler if a guide were to
illuminate the path ahead?
CORPORATE
HEADQUARTERS
OF NEwLIMITS.
Ada?
I said my treat.
Free caffeine.
I can’t Your favorite.
believe the This is the part
nightly batch where you agree
job failed and insist we
again last also get
week. a pastry.

You up for a
coffee break? Ada?
I can’t keep you OKAY?
looking at this.
My treat.

Don’t they seE

this homegrown
ETL tool is jusT
a disaster? Adding
point-to-point
INtegrations
doesn’t help
either.

NO,
JAX!
I’M
NOT
OKA
Y
!!!!!!

Sorry, Jax.
I’m just freaking Some of the Ada?
Um...Ada?
Um...
out a little bit. sales figures are Jax? I need
I’m still sifting screwed up and the to see you
through all the warehouses are all both right
data that got lost.
lost. looking at different now. Kind of
numbers for the
urgent.
inventory and--

Be
rIght
I FEEL there,
LIKE I’M JUST Erick.
WAITING FOR
ERICk TO COME
IN HERE AND
FIRE US.

1
 I’m It’s
They keep
telling not our
giving us
you; EricK fault this
more to do
is firing company
but refuse
us both. only updates
to give us
and processes
the tools we
their data
need to make
once a day.
any real
changes or
improvements.

It’s just
demoralizing.
Maybe ‘‘do more
with less’’ could
be NewLimits’
motto from
now on.
Ada,
you
don’t
know Ha!
that. Well...it’ll
only be
demoralizing
for a
couple
more
minutes.

If
you help
me find
a new
dev gig
someplace,
I’ll help you
find a new I know,
architECT right!
JOB. Everyone
is working
with stale
numbers!

Ready Look on
to the bright side.
get If we get fired,
fired? we won’t have to
sit through another
one of Erick’s boring
meetings where he
lectures us about
the “big picture” and
“bottom line”.

2
just a few moments later...

We’re on a tight Now I

schedule today, so wish we
I’ll cut to the bottom were
line here about this being
new project. fired.
Let me give you all the
big picture of what I’m
thinking about. It’s gonna
be a game changer.

The short
version is that
the big bosses I’m talking about
think it’s finally real-time inventory
time to bring in management. Maybe we
real-time neeD... uh... what’s it
capabilities called...
called ... macro event-
to this
company! Jax, who
is that? In the
corner? She a new
consultant?
Event-driven A developer?
architecture! No clue.
I’ve never
seen her
before.
Right! That!
Maybe we don’t need
it? Who knows? It’s
gonna be a challenge
to figure out.

The last update

And that’s why I’m
M,
putting Ada and Jax in to the inventory Go, TEA
charge of the entire Go, system took a Go!
TEAM, full year!
project! It’s August Go
TEA ,
now, so let’s say you GO!
have everything figured M
out by Black Friday for GO ,
the holiday rush? !

Wait... Black
we’re... Friday is
doing... only
WHAT four
??? months
away!

So... Erick
put you two
in charge
of the
real-time
inventory
projecT.

3
 Right. Any
changes you’d
recommend,
Pat?

Changes?!
How about this
for an idea...

We stick to batch
processing and just
run it a few more times
a day! Our homegrown
ETL tool works
works,, and I
don’t want you
screwing up our
systems.

You
understand
me?

There
I’m HAS
watching to be
you! Huh? a
Nightly batch better
job failed again?! way.
Maybe we WhaT the...
should just take Not again!!
Pat’s advice. He is the
senior developer Now, I’m going
after all. to have to run
the job again
from scratch!
Even if it’s
a nightmare to There is
maintain, it’s probably Maybe
you’re right, Jax. a better way.
better than building Follow me!
new event-driven I tried that at
solutions from my last job and it
scratch. ended up being a bunch
of complicated
Our point-to-point
system integrations. It was
works a disaster.
fine!
Just
F-

4
 I’m from
So... someplace
we didn’t much...
catch your away.
further away.
name.
And if you want
to join the
Data Streaming
Revolution, then
follow me.

It’s
Iris.

J-
J-
J-
JAX!!!
So,
Iris... you
Iris...
come here
from the
Mountain View
Headquaters?

You said
Data Streaming
Revolution?
Definitely Is that
NOT Mountain an app?
View.

Do...
do we
follow Well...

JAX
her? I guess there
Into a are much worse

!!!!
mysterious things in life
portal of than portals
death? of death.
What are you,
crazy?!

What
the-- Erick wants
?!? everyone in the
conference
room in five! He
said it’s about
the “big picture”.

5
MOMENTS LATER
We’re
in a place
known as
Umm... Batchland..
Batchland
where Wherever
are we it is, I
exactly? know it isn’t
And Mountain
WHAT View. Batchland?
are we Right... I mean... it
wearing?! kinda looks cool
and all... but... when
you said to follow
you... you sort of
hinted that you could
help us with our
project.

You know? The real-time Shhhh!

inventory project? We’re
Optimize efficiency waiting!
and seamless
customer experiences?
...
..All
All of that stuff
Erick loves.
Whoa!
What is
that?

That
means we
We don’ t have to
have time to start with
wait! Erick wants a proof-of
Remind her concept and
about the us to figure
everything out migrate the
Black Friday rest of our
deadline. before black
Friday hits. systems-

This is But I could have

a data dump. sworn I heard you
It’s the both say it’s not
daily data worth the effort
transfer here in to change from
Batchland. All batch processing.
these islands... all
the systems
in here... it’s all
run by batch
processing.

Well, Jax,
inefficient
processing
It looks kind is ugly
of... ugly. And and slow.
slow? And
inefficient?

6
 We’re still
searching for
a solution So it’s
that makes like...when
like...when
things easier... our batch
not harder. jobs fail?

Exactly!
When a batch
job fails...
the system
can be
inundated with
unreliable And
It’s data. even IF
complex, everything
isn’t it? Just runs
like all the successfully...
architecture you’re still
here in looking at
Batchland. stale data
right from
So... the start.
this other
All the way you
data here has a mentioned.
specific path that Is it--
it needs to follow
that depends on
the apps being used
and the sources
and sinks...

...and when a new

system enters the
picture or a new
connection is made,
things can get stuck
or crash or the flow
gets interrupted.

It’s Do you
right over want a
there. The closer
revolution. look?
Real-time
data
streaming.

Why
dn’t
coul
i c k have
Er
s t f ired
ju
u !?
s ?

7
 What’s
up in
It’s... the
it’s... sky?
beautiful!
The data
is always
in motion
here.

You two
skip your
Are we coffee this
still in morning?
Batchland? It
began by
harnessing
power from
Apache Kafka,
but now
It’s it’s being
data. It’s run by
flowing all Confluent.
around us.
WAIT!
Is this an
elaborate
sales
pitch?

Yes
That’s and no...
all Kafka itself
data?! is open source,
Flowing but there’s
around more than
us?! just Kafka
here...

Yes.
Does this And I’m
look like regretting
Batchland?! it. What
This place is makes this
called place
StreamSCAPE. run?

See that in the

center of the sky?
That’s Apache Kafka and
all the data is flowing
seamlessly through it.

It’s a
scalable,
open-source
distributed system
used to build And now
It
streaming data you just need
eliminates the
pipelines me to forward
need for
and real-time this info to ten
point-to-point
applications. of my closest
connections.
friends, right?

8
 I don’t need
you to forward Of
anything, hot shot. course, I’ll
The data is out I do. try this a
there already. I went to my different
neighbors’ way.
Quinceanera
last weekend.
It was
quite the
event.

You, But
Jax! not
You’re every
an event
event.. system
And Ada. needs
Us to know
coming here. all about
That’s us at
an event! this very
moment.

The same Point-to-point

way that integrations
not every must be
department updated every
in your office time a new data
needs access consumer
to the same appears.
information And each new
at the update
same time. can be --

It’s
data streaming!
instead of You woke up, Jax!
stagnant data we Each new
update can be And without your
look at data as a coFfee! With
real-time stream of slow or a point
of failure! And an event-driven
events.. You know
events architecture,
what an event is, it reduces
reliability! producers just
right? emit events and
That happens
constantly with consumers
our current make use of the
system. data as needed.

And when something Great! But unless

in this system actually Erick wants to learn
wants to learn about more about Jax, that
an event like an doesn’t really help
oddball named Jax US. We’re on a tight
coming to visit... deadline to complete
then it can. the POC to improve our
company’s inventory
system and-

9
 I’ll put that on
my to-do list with the
thousand other things
Erick needs. You need
And that’s what I’m trying me to post any tiktok
to show you, Ada! All this I showed videos too? I’ve got
light on your right is part of you the time to kill...
a system that’s already foundation
for I guess it’s
harnessing the power of worth a shot.
real-time data streaming. everything you
see here in
Streamscape.
Can you
feel it?
Try tapping
into it.

And on the
left is part of
your company’s
system! Nothing
streaming and
nothing flowing
in or out.

We don’t have the

manpower to run
something like this!
Do you have any
powers to help
manage this Kafka?
Kafka?

You feel that, right? Yes... but

The power of
real-time data? this looks
like a headache
to manage
and maintain.

Yep!
It’s called...

CONFLUENT!

10
 Okay... but what does Confluent Not to mention
actually do?
do? Erick still wants scalability so you
us to- challenges and don’t end up
consumer lag...
lag... with angry
customers
because of
performance
issues or
downtime.
We
don’t
want you
angry
either,
Iris.

Just a bit.
And this Did you
Erick, Kafka... does not hear me
Erick, Erick! that involve when I said
Feeling the complex broker Confluent?
crunch of settings, NOT KAFKA.
that looming capacity Confluent
Black Friday planning, offloads the
deadline, versions... infrastructure
are you? management.

With streaming, Show Uh...

You can leverage me Ada?
you can
the same data to build EVERYTHING! IRIS?!!!
seamlessly
new experiences.
integrate your
real-time store
inventory systems Wait! CAN WE
with your complete the
e-commerce POC for all this
services. right now?

I told
you not to
change
anything!

I TOLD
you our
SYSTEM
WAS FINE!

Of course!
What I’ve shown you
so far is just a taste
of the capabilities for
real-time data streaming.

I TO
LD
YOU
I’D
WATC BE
HING
YOU
!!!!!!

11
 He’s been
infected by I thought I might
Batchland! He’s find you here.
Is trying to lock up Trying to “fix”
that....
that.. all the data things. It’s just
is that... and control how and like when you
PAT?!? when it moves. convinced HQ to
switch from
rollerballs to
gel pens.

What
happened to
him?! He tried
programMING
something in “C”,
didn’t he? He knows
better than that.

I LIKE J-J-
ROLLERBALLS! Jax...
kaff...
There’s nothing h-h-
wrong with our help
system. I won’t let us!
you change it!!

Tell me You do
what to do!! have power,
I don’t have Jax! Focus!
any power!
Remember
what I showed
Ada! You’re not
locked into
running batch
processes
multiple times
a day!

I just
can’t
feel it.

I’m I
trying. can’t...
But I I just...
can’t-- I-
Hey,
Pat?

12
 g
lo s
i
th !
!!!

I
thought it I think
would’ve Jax might
taken him need your
help. Oh,
more time
I’ve got
to process.
this!

KAFK
A!!!
Hey, Pat! It’s not quite time
for that daily batch transfer
you love so much. But I’ve got
something else for you.

...how
about
just
Instead these
of five
endless digits?!!
lines
of code...

13
 It doesn’t have
to be this way, Pat!
We want the same thing
in the end... an efficient
system! We don’t have to
pull data from
a database once or
twice a day.
We don’t
even need to
change the data
producer systems,
we can use Change
Data Capture.

I think I can
get through
him, Jax. You
to him,
and Iris decouple
the rest of our
system to free I’ll
up all the data deal
and start building with
out that POC. Pat.

Want to join us
in the data streaming
revolution?
revolution ?
What do
you say, With CDC
CDC,,
Pat? our system can
automatically
detect changes to
We can data in the source
head back database and emit
to the office
office.. them as events
events..
Jax told me
this morning
the coffee
would be
on him.

NEVER!!!!!!

Bat
c
Gan h
Atta ! g
ck
!!!!

14
 It looks like
they’re trying
to build out the
POC to justify Batch
switching to an Script!
event-driven
architecture!

This ought to
put an end to
their little Bates
experiment. Number!
HAHAHA!

Batch
Up!

Batch
Job!

Now... Fill
the system with
bad data!

Batch
file!

15
 Batch
Job!

Oh, no! The data

they’re dumping
isn’t structured I’m not
properly and Maybe we let Pat
sure we
it’s breaking have his way and
can win
everything! just stick to batch
this fight!
processing?

There’s just too

much to manage at I know! All the
once! It’s only the infrastructure in here
three of us here! It and these added security
would take months! threats! It’s impossible!

It’s not
impossible!

Remember
I told you both that
everything here began And with
with Apache Kafka, Confluent it won’t
but now it runs on take months to get
Confluent. You just this system in
haven’t seen the full order... it will
capabilities yet. take minutes!

Minutes
?!?

16
 Confluent
has reinvented
Kafka for modern data
teams everywhere
to join the
data streaming
revolution..
revolution

By providing
developer tools
such as pre-built
connectors and
stream processing,
you can quickly access,
join, and enrich all
your data to build high
quality, reusable data
streams. And I can
focus on Pat and
his Batch
Gang!

So, we’ll be If we survive

able to do more this battle,
with less?!? Erick will be all
over that!

You’ll
be able
to do
more.
Period!

The Kora
engine powering First, we
Confluent will need to describe
help scale your the schemas
operations up and add schema
and down thirty enforcement to
times faster! block bad data.

Then look
at audit logs to
identify the very
moment they dumped
bad data.

And then
we use RBAC to
provide access
only to the
T!!! three of us.
CONFLUEN

17
 I JUST
DON’T LIKE
DON’
CHANGE!!!!!

No! I
won’t let this
happen! Batch
processing
has
everything!! With batch Rollerball?
processing
I can... it’s
easy to... Batch
it’s perfect File?
for-

NO
OOO O
O!!!

Batch
Fiiiiii-

18
 Real-time data! What We
Event-streaming! about can
Scalable for you, Pat? save
when our you,
So... company grows! Pat!
This is
the data Damn sure
streaming we’re ready!
revolution
you
mentioned
...

You...
I know deep you already
in your heart have...
have...
you must believe
in events.
I take
it you’re
ready to
make the
switch
now?
I see it now...
The future is
in events. You
were right. You
were right.

So, you’re
ready to help us
revolutionize the
way we process
data?

Yeah...
but...

...can i
still get
that coffee
first?

19
 A short
while later...
Ummmm...
we just
thought
Don’t
that...
look now,
but Erick Who?
is headed I don’t
this know any
way. Iris.

Amazing
work,
team!
All these
other teams
are coming to
me and want
to use it too.
We need to We’ll
figure out a take care
Which of it, boss.
one of chargeback
model now
now.. But it wasn’t
you added just us. We
Confluent Maybe you and
Jax can figure couldn’t
to our have done it
system?!? that out too?
without Iris
Iris..

...
I’ll
be right
back.

Go,
team,
go!

Hey...
Go,
team,
go,
huh?

20
Later that night...

The Data Streaming

Revolution Continues...

Y
READ HE
I N T
TO JO EAMING Get started with
ST R
DATA LUTION? Confluent Cloud for free.
RE O
V In just a few minutes, you’ll have the
power of Confluent at your
fingertips and unlock a world of
endless possibilities.

Special thanks to:

Sophia Jiang, Thao Nguyen, Carlson Bull,
Michael Santee, Alina Timoftica, Nick Asphodel,
Julie Newcomer, Diana Vornicov,
Igor Nechytaylo, Todorov Maximm,
Jack Vanlightly, Cathy Nunez, Chase Thomas,
Jade Monroe, Hailey Wheeler, and Kevin Chao

©2023 Confluent, Inc. Apache, Apache Kafka, Kafka, and

associated open source project names are trademarks of 21
the Apache Software Foundation.
 by
uced
Prod N T
LUE
CONF ed by
t
Crea ENT
I N M
E R TA tten by
ENT Wri FAND
LY! L
HEL BUL y
BUL I S N b
O
LEWARLS trated OUS
h C us L
wit Ill OPO
T
AN
N ST
KO
E
O RG
GE