Apache Kafka Distributed Systems
11 chapters
1. 1 Why KRaft Exists
[Verse 1]
In the old days Kafka had a friend
ZooKeeper helped from start to end
Broker registration, metadata store
Topic partitions, ACLs and more
[Verse 2]
Controller election, who's in charge today
ZooKeeper held the truth, showed the way
But running two systems side by side
Brought complexity we couldn't hide
[Chorus]
Why does KRaft exist, why the change
Two systems felt so strange
Metadata split in different places
Scaling up showed all the spaces
KIP five hundred came to save
One system, that's all we crave
[Verse 3]
Operations teams would pull their hair
Managing both systems everywhere
When metadata didn't match up right
Debugging took us through the night
[Verse 4]
Two hundred thousand partitions max
Hit the ceiling, felt the cracks
ZooKeeper's limits held us back
Scalability's what we lacked
[Chorus]
Why does KRaft exist, why the change
Two systems felt so strange
Metadata split in different places
Scaling up showed all the spaces
KIP five hundred came to save
One system, that's all we crave
[Bridge]
Simplify operations, that's the goal
Remove dependencies, take control
Self-managing metadata, all in one
Unified system, scaling's begun
[Chorus]
Why does KRaft exist, why the change
Two systems felt so strange
Metadata split in different places
Scaling up showed all the spaces
KIP five hundred came to save
One system, that's all we crave
[Outro]
No more ZooKeeper in the way
KRaft's here to save the day
2. 3 The Raft Consensus Protocol Primer
[Verse 1]
In the world of distributed systems today
We need consensus to show us the way
Three key players in this protocol dance
Leader election gives structure a chance
One node rises to coordinate the flow
While followers listen to what they should know
Safety first with our guarantees strong
Raft consensus keeps data along
[Chorus]
Leader, log, safety - three pillars standing tall
Pull-based replication when Kafka makes the call
Epoch-based fencing keeps the old leaders out
Raft consensus protocol, that's what it's about
Leader, log, safety - remember these three
Distributed consensus sets your data free
[Verse 2]
Kafka takes Raft but makes it its own
Pull-based replication, not push to the phone
Followers request the updates they need
Instead of the leader controlling the feed
Epochs act like barriers, fencing the past
Old leaders can't write when their time didn't last
This divergence from textbook makes Kafka shine
Optimized for throughput by design
[Chorus]
Leader, log, safety - three pillars standing tall
Pull-based replication when Kafka makes the call
Epoch-based fencing keeps the old leaders out
Raft consensus protocol, that's what it's about
Leader, log, safety - remember these three
Distributed consensus sets your data free
[Bridge]
ZooKeeper had ZAB, atomic broadcast way
But KRaft mode changes how we play
Three nodes in Docker, compose them with care
Metadata shell shows the quorum health there
No more external coordination needed
Self-managing clusters, complexity defeated
[Verse 3]
Set up your lab with three nodes to test
Docker compose makes deployment the best
Run kafka-metadata to verify the state
Quorum health check, don't hesitate
From ZAB to Raft, the evolution's clear
Simpler architecture, future is here
[Chorus]
Leader, log, safety - three pillars standing tall
Pull-based replication when Kafka makes the call
Epoch-based fencing keeps the old leaders out
Raft consensus protocol, that's what it's about
Leader, log, safety - remember these three
Distributed consensus sets your data free
[Outro]
KRaft mode rising, ZooKeeper's done
Raft consensus protocol, the battle is won
Three simple concepts to keep in your mind
Leader, log, safety - leave the old ways behind
3. 1 Node Roles
[Verse 1]
In the world of KRaft there's a choice to make
Three different roles for your cluster's sake
Controller nodes keep the metadata straight
While broker nodes handle client requests all day
[Chorus]
Controller, broker, or combined together
Choose your roles like choosing the weather
Process dot roles equals your decision
Controller, broker, perfect division
Size your cluster, pick your topology
KRaft node roles, that's the key
[Verse 2]
Set process roles to controller alone
Metadata quorum is their only zone
They manage the state but don't serve requests
Keeping cluster information at its best
[Chorus]
Controller, broker, or combined together
Choose your roles like choosing the weather
Process dot roles equals your decision
Controller, broker, perfect division
Size your cluster, pick your topology
KRaft node roles, that's the key
[Verse 3]
Broker nodes serve your produce and consume
Client connections fill up the room
Set process roles to broker they'll be
Handling data flows efficiently
[Bridge]
Combined nodes do it all in one
Broker comma controller, two roles as one
Single JVM runs them both
Perfect for dev or when cluster growth
Is small and simple, resources tight
Combined topology feels just right
[Chorus]
Controller, broker, or combined together
Choose your roles like choosing the weather
Process dot roles equals your decision
Controller, broker, perfect division
Size your cluster, pick your topology
KRaft node roles, that's the key
[Outro]
Deployment size will guide your way
Large clusters separate roles today
Small setups can combine with ease
KRaft node roles bring you peace
4. 1 The KRaft Protocol (Raft Implementation)
[Verse 1]
In the distributed world of Kafka's reign
KRaft protocol breaks the old chain
No more ZooKeeper holding us down
Leader election's the new game in town
Vote RPC starts the conversation
BeginQuorumEpoch seals our foundation
[Chorus]
Pull don't push, that's the KRaft way
Fetch-based replication every day
High-watermark rising, committed and true
Observer nodes watching what leaders do
Vote, fetch, commit, repeat the flow
KRaft protocol is how we grow
[Verse 2]
Traditional Raft would push logs out
But KRaft says followers pull, no doubt
Fetch requests come from every node
Pulling entries down the replication road
Less network pressure, more control
Pull model plays a starring role
[Chorus]
Pull don't push, that's the KRaft way
Fetch-based replication every day
High-watermark rising, committed and true
Observer nodes watching what leaders do
Vote, fetch, commit, repeat the flow
KRaft protocol is how we grow
[Bridge]
High-watermark tells us what's safe to read
Committed offsets meet every need
When majority confirms what we wrote
That's when we advance and take our note
Observer nodes learn but never vote
Non-voter followers stay remote
[Verse 3]
BeginQuorumEpoch starts a new age
Leader and followers on the same page
Tracking committed offsets with care
Making sure data's consistent everywhere
Vote RPC chooses who will lead
Fetch RPC gets the logs we need
[Chorus]
Pull don't push, that's the KRaft way
Fetch-based replication every day
High-watermark rising, committed and true
Observer nodes watching what leaders do
Vote, fetch, commit, repeat the flow
KRaft protocol is how we grow
[Outro]
From Vote to Fetch to high-watermark
KRaft lights up Kafka with its spark
No ZooKeeper needed anymore
KRaft protocol opens up the door
5. 2 What is KRaft?
[Verse 1]
ZooKeeper's been the guardian for years
Managing our cluster state and fears
But now there's something new to learn
KRaft has arrived, it's Kafka's turn
No more external dependencies
Built right into the core with ease
[Chorus]
KRaft is Kafka Raft, eating our own dog food
Metadata as events, just like we always should
Controller nodes in quorum, no ZooKeeper in sight
Event-based consensus, keeping everything tight
KRaft, KRaft, built into the heart
KRaft, KRaft, this is where we start
[Verse 2]
A quorum of controllers takes the stage
Replacing ZooKeeper's ensemble page
Three or five nodes voting as one
Raft protocol makes sure it gets done
Leader election happens clean and fast
Finally a solution that will last
[Chorus]
KRaft is Kafka Raft, eating our own dog food
Metadata as events, just like we always should
Controller nodes in quorum, no ZooKeeper in sight
Event-based consensus, keeping everything tight
KRaft, KRaft, built into the heart
KRaft, KRaft, this is where we start
[Bridge]
Every partition assignment flows as an event
Topic configurations, all the messages sent
Consumer group changes logged in time
Using Kafka's own power, everything's in rhyme
The event log holds the cluster's brain
Metadata streaming through every lane
[Chorus]
KRaft is Kafka Raft, eating our own dog food
Metadata as events, just like we always should
Controller nodes in quorum, no ZooKeeper in sight
Event-based consensus, keeping everything tight
KRaft, KRaft, built into the heart
KRaft, KRaft, this is where we start
[Outro]
From KIP five hundred came this gift
Architecture's getting quite a lift
KRaft consensus, strong and true
Kafka's future starts with you
6. 4 Broker Lifecycle in KRaft
[Verse 1]
When a broker wants to join the cluster scene
It sends registration to the controller machine
The controller checks the request and assigns with care
A broker epoch number that the broker will wear
[Chorus]
Register, heartbeat, shutdown or fail
Fenced or unfenced, that tells the tale
In KRaft lifecycle, four phases we see
Broker management in harmony
[Verse 2]
Heartbeat requests keep the connection alive
Broker to controller, proving it's online
The response comes back with updates to share
Metadata changes floating through the air
[Chorus]
Register, heartbeat, shutdown or fail
Fenced or unfenced, that tells the tale
In KRaft lifecycle, four phases we see
Broker management in harmony
[Bridge]
When shutdown comes, there are two different ways
Controlled shutdown gives time for graceful delays
But uncontrolled failure just cuts the line
No warning given, no cleanup time
[Verse 3]
Fenced brokers are blocked from serving requests
Their epoch is stale, they've failed the test
Unfenced brokers are healthy and ready to go
Active participants in the data flow
[Chorus]
Register, heartbeat, shutdown or fail
Fenced or unfenced, that tells the tale
In KRaft lifecycle, four phases we see
Broker management in harmony
[Bridge - Lab Section]
Kill the controller, watch what happens next
Leader election, timing gets complex
Metadata replay and broker confusion
JMX metrics show the resolution
[Final Chorus]
Register, heartbeat, shutdown or fail
Fenced or unfenced, that tells the tale
Epoch numbers and heartbeat calls
KRaft lifecycle manages it all
[Outro]
Four phases dancing in perfect time
Broker lifecycle by design
7. 2 The Metadata Log (`__cluster_metadata` topic)
[Verse 1]
Deep inside the cluster's heart there beats a special stream
A single partition log that holds the metadata dream
It's replicated safe and sound across the active nodes
Recording every change that through the system flows
[Chorus]
Metadata log, metadata log
Single partition, never clog
Topics born and topics die
Brokers join and configs fly
Metadata log, metadata log
Building cache from every blog
Every change gets written down
In this log the truth is found
[Verse 2]
When a topic gets created or deleted from the scene
The metadata log captures it in records crystal clean
Partition reassignments and broker registration
ACL changes, feature flags across the federation
[Chorus]
Metadata log, metadata log
Single partition, never clog
Topics born and topics die
Brokers join and configs fly
Metadata log, metadata log
Building cache from every blog
Every change gets written down
In this log the truth is found
[Bridge]
But logs can grow forever, that's a problem we must face
So compaction comes to rescue, saving precious storage space
Snapshots freeze the current state, a checkpoint in time
While brokers read these records to keep their cache in line
[Verse 3]
Every broker in the cluster consumes this precious feed
Reading all the metadata changes that they need
Building up their local cache from every single line
So they know the cluster state at any point in time
[Chorus]
Metadata log, metadata log
Single partition, never clog
Topics born and topics die
Brokers join and configs fly
Metadata log, metadata log
Building cache from every blog
Every change gets written down
In this log the truth is found
[Outro]
One log rules them all today
In the KRaft distributed way
Metadata flows like a river wide
With cluster metadata as our guide
8. 3 The Quorum Controller
[Verse 1]
In the cluster there's a group of three
Controllers waiting patiently
One will rise to lead the way
While the others stand and obey
Raft consensus makes the choice
Every node gets to have a voice
[Chorus]
Leader, followers, quorum control
Terms and votes, that's how we roll
Controller quorum voters config
Keeps the cluster running quick
Epochs fence the stale away
Active controller rules the day
[Verse 2]
When election time comes around
Candidates make their voting sound
Higher terms will always win
New leadership can begin
Majority makes the final call
One true leader above them all
[Chorus]
Leader, followers, quorum control
Terms and votes, that's how we roll
Controller quorum voters config
Keeps the cluster running quick
Epochs fence the stale away
Active controller rules the day
[Bridge]
Configuration holds the key
Who can vote, who's in the quorum spree
Three or five, it's odd you see
Split brain problems can't break free
When the leader writes its mark
Epoch numbers light the dark
[Verse 3]
Fencing keeps the old ones out
Stale controllers filled with doubt
Higher epochs take command
Lower numbers must disband
No more dual writes can occur
Active leader's voice is sure
[Chorus]
Leader, followers, quorum control
Terms and votes, that's how we roll
Controller quorum voters config
Keeps the cluster running quick
Epochs fence the stale away
Active controller rules the day
[Outro]
KRaft consensus, strong and true
Quorum controller sees us through
Leader election, clean and bright
Kafka's future burning bright
9. 3 Why Strimzi?
[Verse 1]
Kubernetes runs containers with ease and grace
But stateful workloads challenge this cloud-native space
Persistent data, ordered deployments too
Apache Kafka needs more than pods can do
Storage that survives when containers restart
Network identity that won't fall apart
[Chorus]
Why Strimzi, why choose this way?
Operator magic guides us day by day
CNCF trusted, community strong
Lifecycle management where we belong
Strimzi makes Kafka sing on Kubernetes wings
That's why we choose these operator things
[Verse 2]
Confluent Operator brings commercial might
Enterprise features and support so tight
But licensing costs can make budgets weep
Vendor lock-in runs expensive and deep
Bare metal Kafka gives you full control
But ops complexity takes its toll
[Chorus]
Why Strimzi, why choose this way?
Operator magic guides us day by day
CNCF trusted, community strong
Lifecycle management where we belong
Strimzi makes Kafka sing on Kubernetes wings
That's why we choose these operator things
[Bridge]
Custom resources declare your intent
Rolling updates with zero lament
Scaling brokers with a simple command
Health checks and monitoring, all so grand
Open source freedom, no vendor chains
Cloud Native Computing Foundation reigns
[Verse 3]
Operator pattern drives the lifecycle dance
From installation to upgrade, nothing left to chance
Reconciliation loops keep state in line
Configuration drift becomes benign
Community plugins extend what you can do
Ecosystem thriving, innovations new
[Final Chorus]
Why Strimzi, now we know the way
Operator magic guides us day by day
CNCF trusted, community strong
Lifecycle management where we belong
Strimzi makes Kafka sing on Kubernetes wings
That's why we love these operator things
[Outro]
Stateful workloads need not fear
Strimzi's operator makes the path clear
On Kubernetes clouds, our Kafka soars
Open source power that never ignores
10. 3 Metadata Records & Versioning
[Verse 1]
In the metadata world of KRaft design
Eight record types keep everything in line
Topic and Partition records hold the state
Partition Change records when we need to update
Register Broker when a new node comes online
Broker Registration Change to redefine
[Chorus]
Records and versions, keeping data flowing
API negotiation, compatibility showing
Forward and backward, rolling upgrades clean
Metadata versioning, the smoothest you've seen
T-P-P, R-B-B, A-C-E, C-F complete
Eight types of records make the system sweet
[Verse 2]
Access Control Entry manages who can play
Config Records store the settings for the day
Feature Level Records track what's supported now
API Version handshake shows the system how
When clients connect they negotiate the way
Older versions work, newer features display
[Chorus]
Records and versions, keeping data flowing
API negotiation, compatibility showing
Forward and backward, rolling upgrades clean
Metadata versioning, the smoothest you've seen
T-P-P, R-B-B, A-C-E, C-F complete
Eight types of records make the system sweet
[Bridge]
Rolling upgrades happen node by node
Different versions but they share one home
Forward compatibility means newer reads old
Backward compatibility, the story's retold
Graceful degradation when features don't align
Version negotiation keeps everything fine
[Chorus]
Records and versions, keeping data flowing
API negotiation, compatibility showing
Forward and backward, rolling upgrades clean
Metadata versioning, the smoothest you've seen
T-P-P, R-B-B, A-C-E, C-F complete
Eight types of records make the system sweet
[Outro]
From Topic Records to Feature Levels high
Metadata versioning keeps the cluster synchronized
KRaft evolution with compatibility's grace
Eight record types in their perfect place
11. 2 Custom Resource Definitions (CRDs)
[Verse 1]
In Kubernetes land where pods deploy
We need resources that we can employ
Custom definitions lead the way
Eight CRDs for Kafka's play
First comes Kafka, the cluster king
Primary resource that makes bells ring
Brokers, storage, configuration too
The foundation resource we build upon true
[Chorus]
CRDs declare the Kafka way
Topic, User, Connect today
Bridge and Mirror, Rebalance flow
NodePool makes the cluster grow
Eight resources, memory's key
Strimzi's custom destiny
[Verse 2]
KafkaTopic manages streams with care
Declarative topics everywhere
Partitions, replicas, retention time
No more command line paradigm
KafkaUser handles access control
Authentication plays its role
ACLs and permissions defined in code
Security on the declarative road
[Chorus]
CRDs declare the Kafka way
Topic, User, Connect today
Bridge and Mirror, Rebalance flow
NodePool makes the cluster grow
Eight resources, memory's key
Strimzi's custom destiny
[Verse 3]
KafkaConnect runs the connector show
With KafkaConnector in the flow
Source and sink connectors dance
Lifecycle management at a glance
KafkaMirrorMaker2 spans the divide
Cross-cluster replication as your guide
Topics mirror from here to there
Disaster recovery without care
[Bridge]
KafkaBridge opens HTTP doors
REST API that data pours
KafkaRebalance with Cruise Control
Optimizing every cluster's soul
[Verse 4]
KafkaNodePool is the newest friend
Since zero-thirty-six to depend
Manage worker nodes with precision
Scaling up is your decision
Eight CRDs in harmony
Custom resources, memory
Strimzi makes Kafka sing
On Kubernetes, everything
[Outro]
From Kafka cluster to the bridge so wide
These custom resources are your guide
Declare your intent, let Strimzi flow
Eight CRDs are all you need to know
Back to Home