Apache Kafka-Flink Syllabus

Day 1
Introduction to Apache Kafka

• What is Kafka?
• Need for Kafka
• Real time Analytics with Kafka
• Core Concepts of Kafka
• Kafka Architecture
• Kafka Usecases

Kafka Cluster

• Understanding Kafka Cluster

• Installing and Configuring Kafka Cluster
• Kafka Producer
• Kafka Consumer
• Producer and Consumer in Action
• Reading Data from Kafka
• Message Delivery Semantics
• Replication
• Log Compaction

Lab:
Kafka Installation and Configuration
Single Node and MultiNode Setup
Configuring Zookeeper

Kafka Core Components and operations

• Type of Messaging System

• Kafka Data Model
• Topics
• Partitions
• Partition Distribution
• Producer
• Consumer
• Kafka Reliability
• Replication in Kafka
• Persistence in Kafka
• High availability and reliability using Replication
• ISR - In Sync Replication
• Partition reassignment - Increasing Replication or partitions
• Creating a topic
• Modifying a Topic
• Creating a Message
• Reading a Message
• Java Interface to Kafka
• Producer Side API
• Consumer Side API

Day 2
Discussion:
1. Establish the contrast when using kafka consumer vs third party consumer such as flink.
2. Emphasize the role of Offset in Kafka.

Lab:
• Implement Kafka Producer, Consumer using real time streaming data.
• Working with Kafka API to implement Kafka producer , consumer , consumer group and Kafka Admin APIs to work with kafka
topics and fetch zookeeper metadata
Flink's Stream Processing Architecture
What is stream processing?
Stream processing vs Batch processing
Flink Architecture walkthrough
Stream processing with Apache Flink

Working with Flink

Installing Flink
Setting up your Flink project with Maven
Data Representation and Programming Model
Example 1: Writing a Flink program discussion

Lab:
Flink Instalation and Configuration
Setting up your Flink project with Maven
Writing a Flink program

Flink Transofrmations and Operations

The Filter operation;
The Map operation;
The Flat Map operation;
Stateless and Stateful Transformations;
Creating a stream of Tuples;
The Reduce Operation;

Lab:
Hands on problem solving using Flink API and transformations listed above

Day 3
Flink Transofrmations and Operations Continue..

Windows Transformation ;
Count Windows;
State and Check pointing;
Checkpointing strategies;
Operations on Multiple Streams ;
Unions;
Joining Streams;
coGroup;
Iterate
Discussion:
3. Flink sink operations : map vs flatmap
4. Flink sink as Oracle/RDBMS and best practices.

Lab:
Hands on problem solving using Flink API and transformations listed above
Configuration and Integration of Flink with Kafka

Project:
Real time streaming analytics using Kafka and Flink