Apache

Kafka

jaydeep patil
 Overview of the Event
Streaming
In programming, an event typically refers to an action
or occurrence that happens during the execution of a
program. It can be an interaction from a user, such as
clicking a button or pressing a key, or it can be a
system event, such as a timer or a network request
completing.

Event streaming platform records different types of

data like transaction, historical, and real-time data.

jaydeep patil
Also, this platform is used to process events and allow
the different consumers to process results
immediately and timely manner.

An event-driven platform allows us to monitor our

business and real-time data from different types of
devices like IoT and many more, after analyzing it
provides a good customer experience based on
different types of events and needs.

jaydeep patil
 Introduction of Apache
Kafka
Apache Kafka is an open-source distributed
streaming platform.

Kafka is written in Java and Scala.

The primary purpose of to designed Kafka by Apache

Foundation is to handle real-time data feeds and
provide high throughput and low latency platforms.

Kafka is an event streaming platform that has many

capabilities to publish(write) and subscribe to (read)
streams of events from a different system.

Also, to store and process events durably as long as

we want, by default Kafka store event 7 days of the
time period but we can increase that as per need and
requirement.

jaydeep patil
Kafka has distributed system which has servers and
clients that can communicate via TCP protocol.

It can be deployed on different virtual machines and

containers in on-premise as well as cloud
environments as per requirements.

In the Kafka world, a producer sends messages to the

Kafka broker. The messages will get stored inside the
topics and the consumer subscribes to that topic to
consume messages sent by the producer.

Zookeeper is used to manage metadata of Kafka-

related things. it tracks which brokers are part of the
Kafka cluster and partitions of different topics. Also,
manage the status of Kafka nodes and maintain a list
of Kafka topics and a list of messages.

jaydeep patil
 Main concepts and
foundation of Kafka
1. Event

An event or record is the message that we read and write

to the Kafka server; we do this in the form of events, and it
contains key, value, timestamp, and other metadata
headers.

Key: “Jaydeep”
Value: “Booked BMW”
Event Timestamp: “Dec. 11, 2022, at 12:00 p.m.”

2. Producer

The producer is a client application that sends messages

to the Kafka Node/Broker.

3. Consumer

The consumer is an application that receives data from

Kafka.

jaydeep patil
4. Kafka Cluster

The Kafka cluster is the set of computers that share the

workload with each other and serve some purpose.

5. Broker

The broker is a Kafka server that acts as an agent

between both the producer and consumer, who
communicate via the broker.

6. Topic

The events are stored inside the Topic it’s similar to our
folder in which we store multiple files.

jaydeep patil
Each topic has one or more producers and consumer
which write and reads data from the topic.

Events in topic we can read as often as needed persists

events and it’s not like another messaging system that
removes messages after consuming that

7. Partitions

Topics are partitions, meaning the topic is spread over

multiple partitions that we created inside the topic.
When the producer sends some event to the topic, it will
store it inside the particular partitions, and then the
consumer can read the event from the corresponding
topic partition in sequence.

8. Offset

Kafka assigns one unique ID to the message stored inside

the topic partition when the message arrives from the
producer.

jaydeep patil
9. Consumer Groups

In the Kafka world, the consumer group acts as a single

logical unit.

10. Replica

In Kafka, to make data fault-tolerant and highly available,

we can replicate topics in different regions and brokers.
So, in case something wrong happens with data in one
topic we can easily get that from another where replicate
the same.

jaydeep patil
 Different Kafka APIs
Admin API
This API manages different topics, brokers, and Kafka
objects

Producer API
This API is used to write/publish events to different Kafka
topics

Consumer API
This API is used to receive the different messages
corresponding to the topics which are subscribed by the
consumer

Kafka Stream API

This API is used to perform different types of operations
like windowing, joins, aggregation, and many others.
Basically, it uses to transform objects.

Kafka Connect API

This API works as a connector to Kafka, which helps
different systems connect with Kafka easily. It has
different types of ready-to-use connectors related to
Kafka.

jaydeep patil
 Use Cases of Apache Kafka
Following are the use cases of Kafka

Messaging Systems

User Activity Tracking

Log Aggregation

Stream Processing

Real-time data analytics

Event Sourcing

Microservices Communication

jaydeep patil
 Installation of Kafka on
Windows 10
Step 1

Download and install the Java SDK of version 8 or more.

(Note: I have Java 11 that’s why I put the same path in all
commands which I used over here)

https://www.oracle.com/java/technologies/downloads/#j
ava8

Step 2

Open and install exe

Step 3

Set the environment variable for Java using the

command prompt as admin.

setx -m JAVA_HOME “C:\Program Files\Java\jdk-

11.0.16.1”

setx -m PATH “%JAVA_HOME%\bin;%PATH%”

jaydeep patil
Step 4

After that, download and install Kafka

https://kafka.apache.org/downloads

Step 5

Extract the downloaded Kafka file at rename it as Kafka.

Step 6

Open D:\Kafka\config\ and create a zookeeper-data and

Kafka-logs folder inside that.

Step 7

Next, open the D:\Kafka\config\zookeeper.properties file

and add the folder path inside that.

D:\Kafka\config\zookeeper.properties

dataDir=D:/Kafka/zookeeper-data

jaydeep patil
Step 8

After that, open D:\Kafka\config\server.properties file

and change the log path over there

D:\Kafka\config\server.properties
log.dirs=D:/Kafka/kafka-logs

Step 9

Saves and closes both files.

Step 10

Run zookeeper

D:\Kafka> .\bin\windows\zookeeper-server-start.bat
.\config\zookeeper.properties

jaydeep patil
Step 11

Start Kafka

D:\Kafka>.\bin\windows\kafka-server-start.bat
.\config\server.properties

Step 12

Create Kafka Topic

D:\Kafka>\bin\windows>kafka-topics.bat — create —
bootstrap-server localhost:9092 — replication-factor 1 —
partitions 1 — topic testdata

jaydeep patil
 Step 13

Create a producer and send some messages after you've

started both a producer and a consumer.

D:\Kafka\bin\windows>kafka-console-producer.bat —
broker-list localhost:9092 — topic testdata

Step 14

Next, Create a Consumer and you can see a message

when the producer sent.

D:\Kafka\bin\windows>kafka-console-consumer.bat —
bootstrap-server localhost:9092 — topic testdata

jaydeep patil
Thank You
www.linkedin.com/jaydeep patil