Loading audiobook...

Apache Kafka & Strimzi

39 chapters

1. 1 Core Architecture

[Verse 1]
In the streaming world where data flows
Kafka's architecture everyone should know
Brokers hold the topics in their care
Partitions split the load for all to share
Segments store the messages in files
Offsets track position all the while

[Chorus]
Brokers Topics Partitions all in line
Segments Offsets keeping perfect time
Producers Consumers in the streaming dance
Kafka core gives your data flow a chance

[Verse 2]
Producer sends with acks of zero one or all
Zero fire forget might lose the call
One waits for leader confirmation
All replicas for durability's foundation
Idempotency keeps duplicates away
Exactly-once semantics saves the day

[Chorus]
Brokers Topics Partitions all in line
Segments Offsets keeping perfect time
Producers Consumers in the streaming dance
Kafka core gives your data flow a chance

[Verse 3]
Consumer groups divide the workload fair
Round robin range or sticky assignment there
When members join or leave the group
Rebalancing kicks in to regroup
Partition ownership shifts around
Until stability is found

[Bridge]
Log compaction keeps the latest key
Time retention clears what's old and free
Schema Registry guards the format tight
Avro Protobuf JSON in sight
Evolution modes control the change
Backward forward full compatibility range

[Chorus]
Brokers Topics Partitions all in line
Segments Offsets keeping perfect time
Producers Consumers in the streaming dance
Kafka core gives your data flow a chance

[Outro]
From broker down to schema at the gate
Kafka's architecture seals your streaming fate
Core components working as designed
Defense infrastructure peace of mind

2. 3 Kafka on Kubernetes

[Verse 1]
In the cluster where our data flows
Strimzi operator knows the way it goes
Custom resources define our streaming state
Kafka clusters that we orchestrate

KafkaTopic holds our message streams
KafkaUser grants access to our dreams
KafkaConnect bridges data left and right
Architecture planned to handle any fight

[Chorus]
Kafka on Kubernetes running strong
Storage, placement, sizing all along
Anti-affinity spreads the load around
JVM tuning keeps performance sound
Strimzi CRDs make deployment clean
Best streaming platform you've ever seen

[Verse 2]
When commercial support is what you need
Confluent for Kubernetes plants the seed
CFK delivers enterprise-grade care
Professional backing when systems dare

Local SSDs give the fastest speed
Networked storage for the backup need
Performance implications guide our choice
IOPS and latency give storage voice

[Chorus]
Kafka on Kubernetes running strong
Storage, placement, sizing all along
Anti-affinity spreads the load around
JVM tuning keeps performance sound
Strimzi CRDs make deployment clean
Best streaming platform you've ever seen

[Bridge]
Pod anti-affinity keeps brokers apart
Topology constraints are the spreading art
Different zones and nodes for resilience sake
Fault tolerance that will never break

Resource sizing for the perfect fit
CPU memory disk IOPS commit
KRaft controllers need their share too
Proper planning gets your cluster through

[Verse 3]
JVM tuning in containers tight
Heap sizing has to be just right
G1GC or ZGC for collection choice
Container-aware flags give JVM voice

Memory limits that the pod can see
CPU allocation sets performance free
Garbage collection pauses kept small
Container orchestration handling all

[Final Chorus]
Kafka on Kubernetes running strong
Storage, placement, sizing all along
Anti-affinity spreads the load around
JVM tuning keeps performance sound
Strimzi operators make it clean
Best defense infrastructure ever seen

[Outro]
From Strimzi CRDs to storage choice
Every component has its proper voice
Kubernetes and Kafka working as one
Defense infrastructure battle won

3. 4 Kafka Security

[Verse 1]
Data streams are flowing through your Kafka pipes tonight
But without security walls, you're fighting the wrong fight
Certificates in hand, we'll build our mTLS shield
SASL SCRAM passwords, OAuth tokens revealed
Authentication gates that guard your message store
Four pillars standing strong, defending at the core

[Chorus]
Auth then Author, Encrypt and Log
mTLS SASL, cutting through the fog
ACLs and RBAC, permissions that we trust
TLS in transit, at rest it's a must
Kafka security, four walls standing tall
Authentication, authorization, encryption, audit all

[Verse 2]
Once you prove who you are, now what can you do?
Access Control Lists mapping users to their view
Role-based controls from Confluent's enterprise way
Open Policy Agent integration holds sway
Granular permissions on topics, groups, and more
Authorization layer, second of our four

[Chorus]
Auth then Author, Encrypt and Log
mTLS SASL, cutting through the fog
ACLs and RBAC, permissions that we trust
TLS in transit, at rest it's a must
Kafka security, four walls standing tall
Authentication, authorization, encryption, audit all

[Verse 3]
Messages flying between brokers need protection strong
TLS configuration keeps them safe along
Controller connections, client communication too
End-to-end encryption, nothing bleeding through
At rest on disk, LUKS and dm-crypt shield
Or cloud KMS, same security yield

[Bridge]
Authorizer logs capturing every single call
Request logging shows who accessed it all
SIEM integration, correlating the flow
Audit trails revealing what you need to know
Four pillars together, defense infrastructure strong
Kafka security symphony, sing along

[Chorus]
Auth then Author, Encrypt and Log
mTLS SASL, cutting through the fog
ACLs and RBAC, permissions that we trust
TLS in transit, at rest it's a must
Kafka security, four walls standing tall
Authentication, authorization, encryption, audit all

[Outro]
From certificate handshake to the final log line
Four security layers, by design
Your streaming platform, bulletproof and sound
Kafka fortress built, security all around

4. 5 Cross-Cluster Replication and Active-Active

[Verse 1]
MirrorMaker Two connects our clusters wide
Three connectors working side by side
Source connector pulls the data through
Checkpoint tracks what offsets we've pursued
Heartbeat tells us replication's alive
Keeping distributed systems synchronized

[Chorus]
Cross-cluster, active-active flow
Mirror, checkpoint, heartbeat - that's how we grow
Namespace prefixes keep the topics clear
RPO and RTO - objectives we hold dear
Replicate, translate, never lose the beat
Defense infrastructure, mission complete

[Verse 2]
Topic naming needs a namespace plan
Prefix tells us where the data began
Source-dot-topic-name becomes the style
Checkpoint topics sync across each mile
Offset translation when consumers fail
Consumer groups can pick up the trail

[Chorus]
Cross-cluster, active-active flow
Mirror, checkpoint, heartbeat - that's how we grow
Namespace prefixes keep the topics clear
RPO and RTO - objectives we hold dear
Replicate, translate, never lose the beat
Defense infrastructure, mission complete

[Bridge]
Circular replication needs control
Provenance headers play the vital role
Loop prevention keeps the data clean
While replication lag stays within our screen
Confluent Cluster Linking offers more
But licensing costs you must explore

[Verse 3]
Active-active writes need resolution plans
CRDTs merge the data where conflict spans
Event sourcing keeps the history straight
Last-write-wins with vector clocks can calculate
Which update came first across the time
Keeping consistency in distributed rhyme

[Chorus]
Cross-cluster, active-active flow
Mirror, checkpoint, heartbeat - that's how we grow
Namespace prefixes keep the topics clear
RPO and RTO - objectives we hold dear
Replicate, translate, never lose the beat
Defense infrastructure, mission complete

[Outro]
From async replication to sync so tight
Choose your strategy to get RPO right
Monitor the lag and set your alerts
Cross-cluster mastery - that's how it works

5. 2 KRaft — The ZooKeeper Replacement

[Verse 1]
ZooKeeper's been the guardian for so many years
Managing metadata, calming all our fears
But now there's a new way to handle the load
KRaft consensus takes us down a different road
No more external systems to maintain and scale
Kafka's got its own story to tell

[Chorus]
Leader election, log replication
Committed entries through the nation
KRaft controller quorum standing strong
No ZooKeeper needed, we've moved along
Metadata topics flowing free
This is how it's meant to be

[Verse 2]
Raft protocol with leaders at the helm
Three phases guide us through this realm
Follower, candidate, then leader takes control
Heartbeats keep the system on a roll
Log entries replicated across the quorum
Majority consensus, that's the forum

[Chorus]
Leader election, log replication
Committed entries through the nation
KRaft controller quorum standing strong
No ZooKeeper needed, we've moved along
Metadata topics flowing free
This is how it's meant to be

[Bridge]
Migration path from old to new
Rolling upgrade sees you through
First the controllers make the switch
Then the brokers join without a glitch
Dual-write mode helps you transition
Legacy support during intermission

[Verse 3]
Kafka four point oh brings the change
ZooKeeper mode is out of range
Faster failover, seconds not minutes
Partition scaling with no limits
Metadata log stores the state
Performance improvements truly great

[Chorus]
Leader election, log replication
Committed entries through the nation
KRaft controller quorum standing strong
No ZooKeeper needed, we've moved along
Metadata topics flowing free
This is how it's meant to be

[Outro]
From ensemble old to quorum new
KRaft's the future coming through
Controllers handle all the load
Welcome to the KRaft mode

6. 2 Apache Kafka Fundamentals

[Verse 1]
In the stream of data flowing fast and free
Kafka stands as our distributed key
Brokers form a cluster, topics hold the data
Partitions split the load, replicas make it greater
Messages flow in order, offset tracks the way
Producers write the future, consumers read today

[Chorus]
B-T-P-R, brokers topics partitions replicas
P-C-G, producers consumers groups in harmony
Kafka keeps the data streaming, never losing what we need
Distributed architecture, built for scale and speed

[Verse 2]
ZooKeeper used to coordinate the dance
Managing the metadata, giving brokers their chance
But KRaft mode is rising, Kafka's own consensus
No more external keeper, internal and tremendous
Controller election happens within the cluster now
Self-managing architecture, Kafka shows us how

[Chorus]
B-T-P-R, brokers topics partitions replicas
P-C-G, producers consumers groups in harmony
Kafka keeps the data streaming, never losing what we need
Distributed architecture, built for scale and speed

[Bridge]
Log compaction keeps the latest state
Retention policies decide each message fate
Segments store the data, rolling when they're full
Connect moves the data, Streams make it beautiful
Schema Registry validates, ensuring data's clean
Trading throughput latency durability in between

[Verse 3]
Consumer groups balance the partition load
Each partition owned by one, that's the sacred code
Rebalancing happens when consumers join or leave
Offset commits track progress, recovery they achieve
Acknowledgments control the durability we gain
At-least-once or exactly-once, managing the pain

[Chorus]
B-T-P-R, brokers topics partitions replicas
P-C-G, producers consumers groups in harmony
Kafka keeps the data streaming, never losing what we need
Distributed architecture, built for scale and speed

[Outro]
From produce to consume, the data flows complete
Kafka fundamentals make our streaming sweet
Built for scale and speed

7. 4 Client-Side Implications

[Verse 1]
When clients need to know what's going on
They still talk to brokers, not the control zone
No direct controller contact, that's the rule
Metadata requests through brokers, that's the tool
The architecture keeps the old client way
But underneath, KRaft's here to stay

[Chorus]
Four implications you should know
How the client-side will flow
Metadata through brokers still
Faster updates, that's the thrill
Cluster ID and describe calls
KRaft improvements help us all

[Verse 2]
Propagation latency gets a boost today
ZooKeeper's slow path fades away
Controller to broker, then to client fast
No more waiting for consensus to last
The metadata flows like water downstream
KRaft makes it smooth, fulfilling the dream

[Chorus]
Four implications you should know
How the client-side will flow
Metadata through brokers still
Faster updates, that's the thrill
Cluster ID and describe calls
KRaft improvements help us all

[Bridge]
Take your kafka-metadata shell
Snapshot flag will serve you well
Dump the log and trace the flow
Topic creation, watch it grow
Decode the records, type by type
See how KRaft handles the hype

[Verse 3]
DescribeCluster API reveals the truth
Cluster ID shows the living proof
Lab three will teach you how to see
The metadata log's complexity
From creation to the final state
KRaft's efficiency you'll appreciate

[Chorus]
Four implications you should know
How the client-side will flow
Metadata through brokers still
Faster updates, that's the thrill
Cluster ID and describe calls
KRaft improvements help us all

[Outro]
Client perspective stays the same
But KRaft's playing a faster game
Through the brokers, information flows
How much faster, now you know

8. 1 Migration Strategy Overview

[Verse 1]
When it's time to leave ZooKeeper behind
Don't panic, there's a path that's well-designed
No big bang cutover, no midnight crash
Just a bridge that helps you make the dash
From the old coordination way
To KRaft's controller led array

[Chorus]
Bridge mode, bridge mode, that's the way to go
ZK to KRaft, take it nice and slow
Three point three for early access start
Three point six for production heart
Four point oh waves ZooKeeper goodbye
Bridge mode gets you there, here's why

[Verse 2]
First phase running in ZooKeeper mode
Same old metadata, same old code
Controllers talking to ZK trees
Managing your cluster with familiar ease
But underneath the preparation's done
For the migration that's about to come

[Chorus]
Bridge mode, bridge mode, that's the way to go
ZK to KRaft, take it nice and slow
Three point three for early access start
Three point six for production heart
Four point oh waves ZooKeeper goodbye
Bridge mode gets you there, here's why

[Bridge]
ZK mode to ZK plus KRaft bridge
Walking safely cross that metadata ridge
Then KRaft only, standing on its own
Controllers got a metadata home
Version by version, step by step
Migration promises that we can keep

[Verse 3]
Final phase is KRaft only mode
No more ZooKeeper in your cluster code
Controllers managing their own state
Quorum consensus seals your data fate
Bridge has carried you across the stream
To the KRaft controller team

[Chorus]
Bridge mode, bridge mode, that's the way to go
ZK to KRaft, take it nice and slow
Three point three for early access start
Three point six for production heart
Four point oh waves ZooKeeper goodbye
Bridge mode gets you there, here's why

[Outro]
No big bang, just a bridge today
Migration's smooth the KRaft way

9. 1 Strimzi Operators

[Verse 1]
In the world of Kubernetes where containers run free
Strimzi brings Apache Kafka with operators three
The Cluster Operator takes the leading role
Managing Kafka, ZooKeeper, keeping systems whole
Connect and MirrorMaker, Bridge components too
Reconciliation loops make everything run smooth

[Chorus]
Cluster manages all, Entity splits in two
Topic Operator waits, User Operator too
C-E-T-U, operators four
Reconcile and lead elect, that's what they're for
Strimzi operators working through the night
Keeping Kafka running, everything's alright

[Verse 2]
Entity Operator holds a special place
Contains two operators in one single space
Topic Operator watches KafkaTopic CRDs
Creates and updates topics with such expertise
While User Operator handles KafkaUser calls
Managing ACLs and permissions for all

[Chorus]
Cluster manages all, Entity splits in two
Topic Operator waits, User Operator too
C-E-T-U, operators four
Reconcile and lead elect, that's what they're for
Strimzi operators working through the night
Keeping Kafka running, everything's alright

[Bridge]
Leader election keeps the order right
Only one operator leads the fight
Reconciliation loops keep checking state
Desired meets actual, never too late
When custom resources change their form
Operators respond and transform

[Verse 3]
From YAML declarations to running pods
These operators work like digital gods
Cluster Operator stands at the top
Entity holds two that never stop
Topic and User working side by side
In Kubernetes where Kafka can reside

[Chorus]
Cluster manages all, Entity splits in two
Topic Operator waits, User Operator too
C-E-T-U, operators four
Reconcile and lead elect, that's what they're for
Strimzi operators working through the night
Keeping Kafka running, everything's alright

[Outro]
Four operators strong in Strimzi's way
Managing Kafka every single day

10. 2 Installing Strimzi

[Verse 1]
Three paths to install, choose your way
Helm charts make it clean and quick today
Pull from repositories, customize with ease
Values files configure what you need

[Chorus]
A-B-C, install Strimzi
Helm or YAML or OLM
Watch the namespaces carefully
Single scope or many realms
Check the operator's running free
Strimzi on Kubernetes

[Verse 2]
GitHub releases hold the YAML gold
Download manifests, stories to be told
Apply the files with kubectl command
Direct installation, take control in hand

[Chorus]
A-B-C, install Strimzi
Helm or YAML or OLM
Watch the namespaces carefully
Single scope or many realms
Check the operator's running free
Strimzi on Kubernetes

[Verse 3]
OperatorHub brings the marketplace way
OpenShift catalog, install today
OLM manages the lifecycle flow
Subscription model, watch your clusters grow

[Bridge]
Verify the cluster operator's alive
Check the pods and see them thrive
Logs will tell you if it's right
Green and running through the night

Namespace watching, make your choice
Single target or multiple voice
Environment variable sets the scope
WATCH_NAMESPACE gives you hope

[Chorus]
A-B-C, install Strimzi
Helm or YAML or OLM
Watch the namespaces carefully
Single scope or many realms
Check the operator's running free
Strimzi on Kubernetes

[Outro]
Installation's just the start
Kafka clusters, work of art
Strimzi's ready, now deploy
Apache Kafka you'll enjoy

11. 2 Step-by-Step Migration Process

[Verse 1]
Time to leave the old ZooKeeper days behind
Upgrade first to KRaft-compatible design
Check your version, make it right
Before we start this migration flight
Controllers need their special place
Format storage, set the base

[Chorus]
Six steps dancing, migration song
Upgrade, deploy, enable strong
Dual-write magic, restart the crew
Finalize bridge when we're through
KRaft controllers take the lead
ZooKeeper's time to let them succeed

[Verse 2]
Deploy controller-role nodes with care
Storage formatting everywhere
Run kafka-storage script today
Format metadata the proper way
New controllers standing by
Ready for the metadata high

[Chorus]
Six steps dancing, migration song
Upgrade, deploy, enable strong
Dual-write magic, restart the crew
Finalize bridge when we're through
KRaft controllers take the lead
ZooKeeper's time to let them succeed

[Verse 3]
Enable migration flag is true
ZooKeeper metadata migration enable too
Activate controllers now
Dual-writing starts somehow
Metadata flows to both sides
While the old and new collides

[Bridge]
Rolling restart every broker node
Switch process roles, change the code
From ZooKeeper base to broker-only mode
One by one down the cluster road
Watch them boot with their new role
KRaft connection takes control

[Verse 4]
Finally time to cut the cord
Disable bridge, migration's reward
Decommission ZooKeeper crew
Their faithful service now is through
KRaft stands alone at last
ZooKeeper becomes the past

[Chorus]
Six steps dancing, migration song
Upgrade, deploy, enable strong
Dual-write magic, restart the crew
Finalize bridge when we're through
KRaft controllers take the lead
ZooKeeper's time to let them succeed

[Outro]
From keeper old to KRaft so new
Migration complete, we made it through
Metadata flows in modern way
KRaft controllers here to stay

12. 3 How Strimzi Maps Kafka to Kubernetes Primitives

[Verse 1]
When Kafka meets Kubernetes, there's a bridge to build
Strimzi maps the concepts so your clusters can be filled
Brokers need persistence and a place to call their home
StatefulSets or StrimziPodSets give them room to roam

[Chorus]
Map it out, map it out, primitives align
Brokers to StatefulSets, everything's by design
Configs to ConfigMaps, secrets hold the keys
Storage to PVCs, networking with ease
Map it out, map it out, Strimzi shows the way
Kafka primitives dancing in the Kubernetes ballet

[Verse 2]
Configuration settings need a place to live and breathe
ConfigMaps hold the broker props that make your cluster weave
Every setting tuned and stored in Kubernetes style
Bootstrap servers, log retention, optimized by the mile

[Chorus]
Map it out, map it out, primitives align
Brokers to StatefulSets, everything's by design
Configs to ConfigMaps, secrets hold the keys
Storage to PVCs, networking with ease
Map it out, map it out, Strimzi shows the way
Kafka primitives dancing in the Kubernetes ballet

[Verse 3]
TLS certificates encrypted in Secret vaults secure
Client authentication, broker trust that will endure
Persistent storage calling for PVCs and their class
StorageClass definitions make your data built to last

[Bridge]
Services expose the brokers
Ingress routes the calls
LoadBalancers and NodePorts
Break down networking walls
Every primitive has purpose
In this orchestrated dance
Strimzi makes the mapping
Nothing left to chance

[Chorus]
Map it out, map it out, primitives align
Brokers to StatefulSets, everything's by design
Configs to ConfigMaps, secrets hold the keys
Storage to PVCs, networking with ease
Map it out, map it out, Strimzi shows the way
Kafka primitives dancing in the Kubernetes ballet

[Outro]
From Apache Kafka concepts
To Kubernetes native forms
Strimzi builds the bridges
Through the cloud computing storms

13. 1 Sizing & Topology Planning

[Verse 1]
When you're planning out your cluster design
Controllers are the heart and the spine
Three's the magic number for most cases
Five for enterprise's larger spaces
They coordinate the metadata flow
Making sure your brokers always know

[Chorus]
Three or five controllers, that's the way
CPU and memory, don't delay
Combined or dedicated, make your choice
Network partitions need a careful voice
Quorum placement across the racks
KRaft topology, cover all the cracks

[Verse 2]
Hardware specs matter for your controllers
CPU cores spinning like propellers
Memory gigabytes, four or eight
Disk IOPS can't afford to wait
Fast SSDs keep the logs in sync
Performance matters more than you think

[Chorus]
Three or five controllers, that's the way
CPU and memory, don't delay
Combined or dedicated, make your choice
Network partitions need a careful voice
Quorum placement across the racks
KRaft topology, cover all the cracks

[Bridge]
Combined nodes save you money and space
But dedicated gives performance grace
When the network splits and chaos starts
Quorum keeps the cluster from falling apart
Spread across availability zones
Never put all controllers in one home

[Verse 3]
Rack awareness is your safety net
Cross-zone placement, don't forget
If one goes down, two remain strong
Majority rules to keep moving along
Plan your topology from the start
Network failures won't break the heart

[Chorus]
Three or five controllers, that's the way
CPU and memory, don't delay
Combined or dedicated, make your choice
Network partitions need a careful voice
Quorum placement across the racks
KRaft topology, cover all the cracks

[Outro]
Size it right and plan ahead
KRaft controllers keep your cluster fed
Topology matters, don't wing it
Proper planning helps you win it

14. 2 Configuration Reference

[Verse 1]
Setting up your KRaft configuration
Seven parameters need your dedication
Process roles define what your node will be
Controller, broker, or both running free
Node ID makes each server unique
Integer values, no duplicates we seek

[Chorus]
Config reference, seven keys to know
Process roles, node ID, let the metadata flow
Controller quorum voters, listener names
Metadata log directory, snapshot games
Max record bytes and idle time
KRaft configuration, working in rhyme

[Verse 2]
Controller quorum voters in a list
One at host one, two at host two, can't be missed
Port nine zero nine three is the way
Three controllers keep the cluster in play
Controller listener names must be clear
CONTROLLER is the name that we hear

[Chorus]
Config reference, seven keys to know
Process roles, node ID, let the metadata flow
Controller quorum voters, listener names
Metadata log directory, snapshot games
Max record bytes and idle time
KRaft configuration, working in rhyme

[Bridge]
Metadata log directory needs a fast disk
Twenty megabytes triggers snapshots, don't miss
Five hundred milliseconds for idle time max
These defaults keep your cluster on track

[Verse 3]
Broker comma controller runs them both
Process roles honor your deployment oath
Dedicated storage for metadata logs
Performance matters, avoid the system clogs
Configure these seven, your cluster will sing
KRaft architecture, that's the new thing

[Chorus]
Config reference, seven keys to know
Process roles, node ID, let the metadata flow
Controller quorum voters, listener names
Metadata log directory, snapshot games
Max record bytes and idle time
KRaft configuration, working in rhyme

[Outro]
Seven parameters, commit them to mind
KRaft configuration, perfectly designed

15. 3 Rollback & Risk Management

[Verse 1]
Started with ZooKeeper, now we're bridging to the new
KRaft migration running, but what if something's wrong with you?
Monitor those metrics while the cluster's in between
CPU and memory usage, keep your dashboard clean

[Chorus]
Roll it back, roll it back, before the point of no return
Bridge mode saves you, bridge mode saves you, from the lessons that you'd learn
Watch the logs, check the health, migration's not complete
Until you see that magic line, rollback's your retreat

[Verse 2]
Common failures happening when the brokers lose their way
Network partitions forming, or the storage starts to fray
Quorum controller struggling, metadata out of sync
Stop the process quickly before you're on the brink

[Chorus]
Roll it back, roll it back, before the point of no return
Bridge mode saves you, bridge mode saves you, from the lessons that you'd learn
Watch the logs, check the health, migration's not complete
Until you see that magic line, rollback's your retreat

[Bridge]
Lab four practice time now, test cluster in your hands
Happy path migration, then rollback where you stand
Dual write mode active, both systems running strong
But once you flip that final switch, you can't undo what's wrong

[Verse 3]
Key metrics to monitor, throughput and latency
Under replicated partitions, offline broker spree
Controller event rate climbing, warning signs appear
Better safe than sorry when the rollback point is near

[Chorus]
Roll it back, roll it back, before the point of no return
Bridge mode saves you, bridge mode saves you, from the lessons that you'd learn
Watch the logs, check the health, migration's not complete
Until you see that magic line, rollback's your retreat

[Outro]
From ZooKeeper to KRaft mode, but safety comes first
Practice makes it perfect, for better or for worst
Remember rollback windows, they won't last forever
Bridge mode is your lifeline, migration done clever

16. 2 Metadata Snapshots

[Verse 1]
Log compaction cleans the old away
But that's not enough for metadata's way
When brokers start they need to know the state
Without reading millions from day one's date
The controller needs a faster plan
To bootstrap clusters across the span

[Chorus]
Snapshots save the day, freeze the current state
Generate and load, never be too late
Trigger when it's time, format keeps it clean
Storage saves the past, fastest start you've seen
Snapshots save the day, metadata's best friend
Start up quick and smooth, on this you can depend

[Verse 2]
Generation triggers when the log grows long
High water mark says the time is strong
Controller writes the current metadata down
Binary format, compact and sound
Stored alongside the partition log
A frozen moment through the fog

[Chorus]
Snapshots save the day, freeze the current state
Generate and load, never be too late
Trigger when it's time, format keeps it clean
Storage saves the past, fastest start you've seen
Snapshots save the day, metadata's best friend
Start up quick and smooth, on this you can depend

[Bridge]
When the broker starts up from the ground
It loads the snapshot, state is found
No need to replay every single write
Just the latest changes, everything's right
SnapshotAlterRequest comes into play
Managing the lifecycle every day

[Verse 3]
Controller restart means loading time
Snapshot first, then replay the timeline
Only process logs after the snap
Fill in the missing pieces, close the gap
The lifecycle flows from create to load
Efficient startup on this well-worn road

[Chorus]
Snapshots save the day, freeze the current state
Generate and load, never be too late
Trigger when it's time, format keeps it clean
Storage saves the past, fastest start you've seen
Snapshots save the day, metadata's best friend
Start up quick and smooth, on this you can depend

[Outro]
From trigger point to storage space
Snapshots give clusters a faster pace
KRaft metadata, now you understand
Snapshots make startup quick and grand

17. 3 Monitoring & Observability

[Verse 1]
In the world of KRaft we need to see
What's happening inside our cluster machinery
JMX metrics tell the story true
ActiveControllerCount starts our view
One controller leads while others wait
Zero means trouble at the gate

[Chorus]
Monitor and observe, keep your eyes on the prize
Raft metrics flowing, commit latency flies
Current leader standing, log end offset grows
MetadataLoader spinning, that's how KRaft flows
Watch the patterns, catch the signs
Keep your cluster running fine

[Verse 2]
Server raft metrics hold the key
Commit latency shows delivery speed
Current leader tells you who's in charge
Log end offset keeps the margin large
When brokers get fenced you'll see the logs
Leader elections clear the fog

[Chorus]
Monitor and observe, keep your eyes on the prize
Raft metrics flowing, commit latency flies
Current leader standing, log end offset grows
MetadataLoader spinning, that's how KRaft flows
Watch the patterns, catch the signs
Keep your cluster running fine

[Bridge]
Snapshot loading can sometimes fail
Last applied offset tells the tale
Grafana dashboard paint the scene
Red and green lights keep it clean
Health checks running through the night
Making sure your data's right

[Verse 3]
Build your dashboard piece by piece
Controller count brings you peace
Metadata loader metrics shine
Last applied offset stays in line
Pattern matching in your logs
Catches failures in the fog

[Chorus]
Monitor and observe, keep your eyes on the prize
Raft metrics flowing, commit latency flies
Current leader standing, log end offset grows
MetadataLoader spinning, that's how KRaft flows
Watch the patterns, catch the signs
Keep your cluster running fine

[Outro]
KRaft observability is the way
Monitor your cluster every day
JMX and logs will be your guide
Keep your data flowing with pride

18. 4 Lab: Hello Strimzi

[Verse 1]
Fire up your kind cluster, let's begin today
Three brokers we'll deploy in the Kubernetes way
Strimzi operator's ready, YAML files in hand
Persistent storage volumes across our local land

[Chorus]
Three brokers, three partitions, replication factor three
Hello Strimzi on K8s, streaming wild and free
Produce and then consume it, messages flowing through
Inspect those pods and secrets, PVCs waiting too

[Verse 2]
Kafka cluster custom resource, define it with care
Each broker needs persistence, storage everywhere
Apply the configuration, watch the pods arise
Three leaders, three followers, before your very eyes

[Chorus]
Three brokers, three partitions, replication factor three
Hello Strimzi on K8s, streaming wild and free
Produce and then consume it, messages flowing through
Inspect those pods and secrets, PVCs waiting too

[Verse 3]
Create your KafkaTopic now, partitions times three
Replication factor matching, for reliability
Console producer ready, exec into the pod
Send messages streaming through our Kafka mod

[Bridge]
kubectl get pods shows us all the running state
Persistent volume claims, they don't hesitate
Services route the traffic, secrets hold the keys
Strimzi makes it simple, Kafka with such ease

[Chorus]
Three brokers, three partitions, replication factor three
Hello Strimzi on K8s, streaming wild and free
Produce and then consume it, messages flowing through
Inspect those pods and secrets, PVCs waiting too

[Verse 4]
Console consumer listening, messages appear
From producer to consumer, the pipeline is clear
Check those Kubernetes resources, everything's in place
Strimzi lab completed with distributed grace

[Outro]
Hello Strimzi running strong
Apache Kafka singing along
On Kubernetes we deploy
Streaming data, pure joy

19. 4 Security in KRaft Mode

[Verse 1]
Controllers talking safe and sound
TLS encryption all around
SASL authentication strong
No more ZooKeeper holding on
Metadata logs now hold the keys
ACLs flowing with such ease

[Chorus]
Security in KRaft mode
TLS and SASL code
Metadata logs will store
ACLs and so much more
Audit trails keep track of change
KRaft security rearranged

[Verse 2]
Inter-broker trust rebuilt
New foundations without guilt
Controllers share their secrets tight
Encrypted channels day and night
Principal names and auth rules
KRaft brings us better tools

[Chorus]
Security in KRaft mode
TLS and SASL code
Metadata logs will store
ACLs and so much more
Audit trails keep track of change
KRaft security rearranged

[Bridge]
Gone are ZooKeeper ACL days
Metadata log shows us the way
Every change is logged and tracked
Security features tightly packed
Controller quorum keeps it clean
Best security we've ever seen

[Verse 3]
Audit logging tells the tale
Every metadata change in detail
Who did what and when they did
Nothing secret, nothing hid
Compliance teams can sleep at night
KRaft security done right

[Chorus]
Security in KRaft mode
TLS and SASL code
Metadata logs will store
ACLs and so much more
Audit trails keep track of change
KRaft security rearranged

[Outro]
From ZooKeeper to KRaft we go
Security improvements flow
Controllers, brokers, all aligned
Peace of mind for every mind

20. 3 Deploying Your First Kafka Cluster

[Verse 1]
First we write the Kafka custom resource
Minimal config, that's our starting course
Metadata name, specify the namespace
Three replicas for our cluster's base
YAML definition, clean and bright
Bootstrap servers coming to life tonight

[Chorus]
Deploy your first cluster, watch it come alive
Kafka custom resource, three replicas to thrive
Ephemeral or persistent, choose your storage way
ZooKeeper or KRaft mode, different paths today
kubectl get kafka, watch the reconciliation
Strimzi operator working through orchestration

[Verse 2]
Storage matters, make your choice with care
Ephemeral means data won't stay there
Persistent volumes keep your messages safe
Production workloads need that storage space
Size and class, configure what you need
Storage type will make your cluster succeed

[Chorus]
Deploy your first cluster, watch it come alive
Kafka custom resource, three replicas to thrive
Ephemeral or persistent, choose your storage way
ZooKeeper or KRaft mode, different paths today
kubectl get kafka, watch the reconciliation
Strimzi operator working through orchestration

[Verse 3]
ZooKeeper mode is the classic way
Metadata management, tried and true today
KRaft mode is newer, self-managing design
No ZooKeeper needed, everything's in line
Choose your architecture, both will work fine
Configuration flag sets the paradigm

[Bridge]
Watch the pods spin up, one by one they start
kubectl get kafka shows the beating heart
Ready status true means you're good to go
Operator logs will tell you what you need to know
Reconciliation magic, Strimzi's doing work
Custom resource controller, no need to shirk

[Verse 4]
Verification time, let's test what we built
Producer consumer, messages without guilt
Create a simple topic, send some data through
Consume the messages, prove the cluster's true
Bootstrap service endpoint, that's your connection door
Now your Kafka cluster's ready to explore

[Chorus]
Deploy your first cluster, watch it come alive
Kafka custom resource, three replicas to thrive
Ephemeral or persistent, choose your storage way
ZooKeeper or KRaft mode, different paths today
kubectl get kafka, watch the reconciliation
Strimzi operator working through orchestration

[Outro]
From YAML to running, that's the Strimzi way
Your first Kafka cluster's running today
Custom resources make it clean and neat
Kubernetes native, the integration's sweet

21. 2 ZooKeeper Configuration

[Verse 1]
In the world of Strimzi where Kafka needs to run
ZooKeeper holds the state for everyone
Three replicas minimum, that's the golden rule
For production workloads, keep your cluster cool
Memory allocation starts at one gig base
CPU cores depending on your cluster's pace
Network bandwidth matters when nodes communicate
Size it right or watch your performance deteriorate

[Chorus]
Configure your Zoo, allocate with care
Memory CPU storage everywhere
Persistent volumes keep your data safe
Tuning parameters set the perfect pace
Three two one, replicas must be odd
Ticktime heartbeat, sync limit's your rod
Configure your Zoo, make it bulletproof
Strimzi's foundation underneath your roof

[Verse 2]
Persistent storage is where your logs will live
Dynamic provisioning has so much to give
Storage class selection for your volume claims
SSD performance when you're playing big games
Data directory mounted at var lib zookeeper
Transaction logs need their own keeper
Separate volumes for optimal I O flow
Watch your disk space as your clusters grow

[Chorus]
Configure your Zoo, allocate with care
Memory CPU storage everywhere
Persistent volumes keep your data safe
Tuning parameters set the perfect pace
Three two one, replicas must be odd
Ticktime heartbeat, sync limit's your rod
Configure your Zoo, make it bulletproof
Strimzi's foundation underneath your roof

[Bridge]
Tick time sets the heartbeat rhythm
Two thousand milliseconds keep your system
Sync limit controls the follower pace
Init limit for startup's grace
Max client connections set the door
Auto purge keeps your disk from being sore
Four letter words for monitoring health
These parameters are your cluster's wealth

[Verse 3]
JVM heap size needs careful consideration
Too little memory brings poor performance frustration
Garbage collection tuning keeps your latency low
Parallel collector helps your throughput grow
Network threads handling all requests
I O threads serving clients at their best
Session timeout balancing client needs
Recovery time when leadership succeeds

[Chorus]
Configure your Zoo, allocate with care
Memory CPU storage everywhere
Persistent volumes keep your data safe
Tuning parameters set the perfect pace
Three two one, replicas must be odd
Ticktime heartbeat, sync limit's your rod
Configure your Zoo, make it bulletproof
Strimzi's foundation underneath your roof

[Outro]
From sizing to storage to parameters fine
Your ZooKeeper cluster will work every time
In Kubernetes pods with Strimzi's might
Configure it well and sleep through the night

22. 1 KRaft Performance Characteristics

[Verse 1]
ZooKeeper held us back for years so long
Metadata crawling when we needed strong
KRaft arrived to change the game we play
Milliseconds faster, leading the way
No external systems slowing us down
Built-in controller wearing the crown

[Chorus]
KRaft performance flying high
Latency low, throughput to the sky
Millions of partitions we can scale
Controller speed will never fail
Propagation fast, operations smooth
KRaft performance, feel the groove

[Verse 2]
Remember when a thousand partitions felt like max
ZooKeeper sessions creating cracks
Now we're talking millions without breaking sweat
Partition scalability, the best we've met
Linear growth pattern, predictable and clean
Most efficient cluster that you've ever seen

[Chorus]
KRaft performance flying high
Latency low, throughput to the sky
Millions of partitions we can scale
Controller speed will never fail
Propagation fast, operations smooth
KRaft performance, feel the groove

[Bridge]
Metadata operations per second climbing
Ten thousand requests, perfect timing
Thousand broker clusters running fine
Tuning parameters, everything's aligned
Batch processing keeps the pipeline flowing
Controller efficiency always growing

[Verse 3]
Large-scale tuning needs a careful hand
Metadata cache size, understand
Controller threads scaled to match the load
Network buffers optimized for the road
Replication factor balanced just right
Performance metrics shining bright

[Chorus]
KRaft performance flying high
Latency low, throughput to the sky
Millions of partitions we can scale
Controller speed will never fail
Propagation fast, operations smooth
KRaft performance, feel the groove

[Outro]
From ZooKeeper days to KRaft today
Performance improvements here to stay
Benchmark numbers tell the tale
Modern streaming systems never fail

23. 3 KRaft Mode (ZooKeeper-less Kafka)

[Verse 1]
ZooKeeper's been our trusted friend for years
Managing metadata, handling all our fears
But Strimzi's moving forward, there's a new way to go
KRaft mode is coming, though the pace is slow

[Chorus]
KRaft mode, KRaft mode, controllers lead the way
No more ZooKeeper dependency to stay
Node pools define the roles we need
Controller-only or combined indeed
KRaft mode, KRaft mode, the future's here today

[Verse 2]
Strimzi's roadmap shows us where we're bound
KRaft support is building, getting more sound
From experimental status to production grade
The transition path is carefully being made

[Chorus]
KRaft mode, KRaft mode, controllers lead the way
No more ZooKeeper dependency to stay
Node pools define the roles we need
Controller-only or combined indeed
KRaft mode, KRaft mode, the future's here today

[Verse 3]
Migration from ZooKeeper takes some planning time
Data transformation, keeping systems in line
KafkaNodePool CRD helps us organize
Controller nodes and brokers, each with their own size

[Bridge]
Controller-only nodes just manage state
Combined nodes handle both, they don't hesitate
Role-based management gives us the control
Each node pool serves its designated goal

[Chorus]
KRaft mode, KRaft mode, controllers lead the way
No more ZooKeeper dependency to stay
Node pools define the roles we need
Controller-only or combined indeed
KRaft mode, KRaft mode, the future's here today

[Outro]
Strimzi's KRaft journey, step by step we climb
Leaving ZooKeeper behind, it's just a matter of time
Controllers rule the cluster in this brand new way
KRaft mode in Kubernetes, it's here to stay

24. 4 Encrypting Data at Rest

[Verse 1]
When your Kafka data sleeps at night
On the disks where secrets hide
Encryption guards what's stored inside
From prying eyes that shouldn't pry
etcd holds your cluster state
Every secret, every key
Without encryption at the gate
Your data's not as safe as it could be

[Chorus]
Lock it down, encrypt at rest
K-eight-s level, that's the best
etcd secrets, CSI drives
Keep your Kafka data alive
Cloud provider, disk encryption
AWS, Azure, GCP protection
Lock it down, encrypt at rest
Data sleeping safely blessed

[Verse 2]
Kubernetes secrets engine runs
etcd encryption at the core
AES-CBC or AES-GCM for fun
Your cluster secrets now secure
CSI drivers take the wheel
Storage classes define the way
Container Storage Interface deal
Encrypts volumes where Kafka stays

[Chorus]
Lock it down, encrypt at rest
K-eight-s level, that's the best
etcd secrets, CSI drives
Keep your Kafka data alive
Cloud provider, disk encryption
AWS, Azure, GCP protection
Lock it down, encrypt at rest
Data sleeping safely blessed

[Bridge]
AWS EBS volumes encrypted tight
Azure Disk with keys so bright
GCP Persistent Disks aligned
Multi-layered, peace of mind
Key rotation, access control
Cloud provider plays their role
While Kubernetes does its part
Encryption layers, work of art

[Verse 3]
Configure encryption config files
Set your providers in a row
KMS integration all the while
Cloud-native security flow
Storage classes marked encrypted
CSI parameters set right
Your Kafka topics are protected
Even when they sleep at night

[Chorus]
Lock it down, encrypt at rest
K-eight-s level, that's the best
etcd secrets, CSI drives
Keep your Kafka data alive
Cloud provider, disk encryption
AWS, Azure, GCP protection
Lock it down, encrypt at rest
Data sleeping safely blessed

[Outro]
From etcd to the storage layer
Every bit protected well
Multi-cloud encryption prayer
Keeps your secrets safe to tell
Lock it down, encrypt at rest
Strimzi Kafka at its best

25. 1 TLS Encryption

[Verse 1]
In the world of Kafka streams and queues
Strimzi brings security we can use
Built-in Certificate Authority stands guard
Making TLS encryption less hard
Auto-generates what your cluster needs
Certificates planted like digital seeds

[Chorus]
CA creates, auto-renews, trust the flow
Bring your own or let Strimzi grow
Validity periods, configure with care
Truststore holds the keys you share
TLS encryption, lock it down tight
Kafka security shining bright

[Verse 2]
Renewal cycles happen behind the scene
Thirty days before expiry, fresh and clean
Default validity lasts three-sixty-five
But you can customize to keep certs alive
Set your own periods in the YAML config
Certificate lifecycle running like clockwork

[Chorus]
CA creates, auto-renews, trust the flow
Bring your own or let Strimzi grow
Validity periods, configure with care
Truststore holds the keys you share
TLS encryption, lock it down tight
Kafka security shining bright

[Bridge]
When you bring your own CA to the table
Custom certificates, strong and stable
Import your secrets to the cluster space
Replace the defaults with your own embrace
Client truststore needs the public key
Authentication flowing seamlessly

[Verse 3]
Configure clients with the truststore path
SSL context follows the security math
Certificate chain validation on every call
Mutual TLS protecting it all
From producer to broker, consumer to cluster
Encrypted connections growing stronger

[Chorus]
CA creates, auto-renews, trust the flow
Bring your own or let Strimzi grow
Validity periods, configure with care
Truststore holds the keys you share
TLS encryption, lock it down tight
Kafka security shining bright

[Outro]
Trust the process, trust the chain
Strimzi TLS breaking through the pain
Certificates rotating, security never sleeps
Your Kafka cluster safely keeps

26. 5 Lab: Securing a Strimzi Cluster

[Verse 1]
In our Kafka cluster running free
Security's the missing key
TLS encryption we must enable
On every listener round the table
Bootstrap servers need protection
Client connections need inspection
Set the security protocol right
SASL SSL shining bright

[Chorus]
Lock it down, encrypt the sound
SCRAM SHA five twelve all around
ACLs control the flow
Zero downtime as we go
TLS on every port
User auth of every sort
Strimzi security made strong
This is how we get along

[Verse 2]
KafkaUser resource we create
SCRAM SHA authentication fate
Password stored in secret space
Topic level access we embrace
Producer rights on my-topic name
Consumer groups join the game
Allow operations read and write
Authorization done just right

[Chorus]
Lock it down, encrypt the sound
SCRAM SHA five twelve all around
ACLs control the flow
Zero downtime as we go
TLS on every port
User auth of every sort
Strimzi security made strong
This is how we get along

[Verse 3]
Client application needs to know
Truststore path and how to flow
SASL mechanism set in stone
Username password not alone
Properties file configured clean
Bootstrap servers through the screen
Producer sends with credentials true
Authentication's what we do

[Bridge]
Certificate rotation time has come
Cluster operator gets it done
Rolling update keeps us live
Zero downtime we achieve
Old certs fade and new ones rise
No disruption to surprise
Kafka keeps on running strong
Security updated all along

[Chorus]
Lock it down, encrypt the sound
SCRAM SHA five twelve all around
ACLs control the flow
Zero downtime as we go
TLS on every port
User auth of every sort
Strimzi security made strong
This is how we get along

[Outro]
Secured cluster stands so tall
TLS protecting one and all
Users authenticated right
Kafka streaming through the night

27. 3 Network Policies

[Verse 1]
In the cluster where the data flows
Security must guide where traffic goes
Network policies stand as guards
Protecting Kafka from outside cards
Three key rules we need to know
To keep our message streams secure below

[Chorus]
Lock the brokers, shield the Zoo
Allow the ones you trust to get through
Network policies three by three
Kafka safety, that's the key
Lock the brokers, shield the Zoo
Only trusted pods break through

[Verse 2]
First restriction, broker traffic tight
Only designated pods get sight
Port nine zero nine two stays closed
Unless your label's been exposed
Ingress rules with selectors clean
Keep unwanted clients from the scene

[Chorus]
Lock the brokers, shield the Zoo
Allow the ones you trust to get through
Network policies three by three
Kafka safety, that's the key
Lock the brokers, shield the Zoo
Only trusted pods break through

[Verse 3]
ZooKeeper needs its private space
Coordination in a sacred place
Port two one eight one locked down tight
Internal cluster traffic only right
No external access to the state
Where Kafka's metadata finds its fate

[Bridge]
Namespace labels, pod selectors too
Define exactly who gets through
Match expressions tell the tale
Of which connections will not fail

[Verse 4]
Third policy allows with care
Specific namespaces to share
Label matching makes it clear
Which applications can draw near
Production separate from test
Access control at its best

[Chorus]
Lock the brokers, shield the Zoo
Allow the ones you trust to get through
Network policies three by three
Kafka safety, that's the key
Lock the brokers, shield the Zoo
Only trusted pods break through

[Outro]
Three network policies stand guard
Kafka security won't be marred
Strimzi makes the config clean
Safest streaming you've ever seen

28. 2 Authentication

[Verse 1]
When clients knock on Kafka's door
Three ways to prove who they are for sure
TLS certificates shake hands both ways
Mutual trust in encrypted displays
SCRAM with SHA takes username and pass
Hashed five-twelve times to make security last
OAuth tokens from providers you know
Keycloak, Azure, Okta's the flow

[Chorus]
Three ways in, three ways to authenticate
TLS mutual, SCRAM, OAuth delegate
Per listener config, you choose the gate
KafkaUser CRD seals their fate
Lock it down, lock it tight
Authentication done right

[Verse 2]
TLS client auth needs certificates paired
Server checks client, both sides are prepared
Private keys and public certs align
X-five-oh-nine format by design
SCRAM-SHA-512 salts and iterates
Password never travels, security validates
Store the credentials in Kubernetes secrets
User management that never retreats

[Chorus]
Three ways in, three ways to authenticate
TLS mutual, SCRAM, OAuth delegate
Per listener config, you choose the gate
KafkaUser CRD seals their fate
Lock it down, lock it tight
Authentication done right

[Bridge]
Listener by listener you decide the method
Plain or TLS or SASL connected
KafkaUser custom resource defines
Who gets access across the lines
Strimzi operator handles the rest
Certificate authority puts trust to test

[Verse 3]
OAuth brings external identity providers
Keycloak tokens, Azure suppliers
JWT validation at the broker door
Centralized auth you can't ignore
Each listener in your Kafka spec
Choose your method, earn respect
Authentication mechanisms three
Secure your streaming technology

[Chorus]
Three ways in, three ways to authenticate
TLS mutual, SCRAM, OAuth delegate
Per listener config, you choose the gate
KafkaUser CRD seals their fate
Lock it down, lock it tight
Authentication done right

[Outro]
Strimzi makes it simple to configure and deploy
Authentication layers that hackers can't destroy
Three methods strong, your choice to make
Kafka security for your data's sake

29. 3 Authorization

[Verse 1]
In the Kafka realm where messages flow
We need to control who can go where they go
KafkaUser spec authorization comes first
Simple ACLs to quench your access thirst
Define the rules in YAML clean and bright
Who can read and write, who gets the right

[Chorus]
Three ways to authorize, keep them in mind
Simple ACLs, Keycloak refined
OPA custom when you need more control
Literal prefix wildcard makes you whole
Authorization layers protecting your stream
Strimzi security living the dream

[Verse 2]
Keycloak steps up when simple won't do
Authorization Services coming through
Fine-grained policies with context aware
Role-based decisions handled with care
Resource servers and permissions defined
Policy evaluation peace of mind

[Chorus]
Three ways to authorize, keep them in mind
Simple ACLs, Keycloak refined
OPA custom when you need more control
Literal prefix wildcard makes you whole
Authorization layers protecting your stream
Strimzi security living the dream

[Bridge]
Patterns make the magic happen fast
Literal matches exact and they last
Prefix patterns start the same way
Wildcard asterisk saves the day
Match your topics, match your groups
Authorization follows all your loops

[Verse 3]
Open Policy Agent when you need to code
Custom authorizer down a different road
Rego language policies you can write
Decision engine working day and night
Plugin architecture flexible and strong
Custom logic where you belong

[Chorus]
Three ways to authorize, keep them in mind
Simple ACLs, Keycloak refined
OPA custom when you need more control
Literal prefix wildcard makes you whole
Authorization layers protecting your stream
Strimzi security living the dream

[Outro]
From simple specs to complex rules
Strimzi gives you all the tools
Protect your Kafka, control the flow
Authorization everywhere you go

30. 1 MirrorMaker 2 Architecture

[Verse 1]
Two clusters standing side by side
Source and target, worlds divide
Data flows across the space
MirrorMaker keeps the pace
Three connectors work as one
Replication's never done

[Chorus]
Mirror Source, Mirror Checkpoint, Mirror Heartbeat too
Three connectors working hard to see your data through
Source to target, topics flow with prefixes so clear
Consumer offsets translated here, MirrorMaker's architecture dear

[Verse 2]
MirrorSourceConnector leads the way
Copying topics day by day
Remote prefixes mark the trail
Source cluster name will never fail
Topics renamed with dot notation
Cross-cluster conversation

[Chorus]
Mirror Source, Mirror Checkpoint, Mirror Heartbeat too
Three connectors working hard to see your data through
Source to target, topics flow with prefixes so clear
Consumer offsets translated here, MirrorMaker's architecture dear

[Verse 3]
MirrorCheckpointConnector saves the state
Consumer offsets it translates
Groups and partitions mapped with care
Progress tracking everywhere
Failover smooth when systems fall
Checkpoint connector handles all

[Bridge]
Heartbeat connector sends the pulse
Monitoring without results in faults
Cluster connectivity it checks
Network health it never neglects
Three together form the team
MirrorMaker's perfect scheme

[Verse 4]
Topic naming follows rules so strict
Remote cluster names predict
Source dot topic dot partition clear
Target knows what data's here
Offset mapping keeps in sync
Missing nothing, not one link

[Chorus]
Mirror Source, Mirror Checkpoint, Mirror Heartbeat too
Three connectors working hard to see your data through
Source to target, topics flow with prefixes so clear
Consumer offsets translated here, MirrorMaker's architecture dear

[Outro]
When disaster strikes your source
Mirror target stays the course
Architecture built to last
Failover smooth and fast

31. 1 Data Governance

[Verse 1]
In the streaming world of Kafka flows
Every topic needs a label that shows
What kind of data passes through the gate
Classification marks decide its fate
PII gets tagged as sensitive grade
Public info flows without parade
Schema registry holds the metadata crown
While governance rules keep chaos down

[Chorus]
Tag it, track it, time it right
Data governance shining bright
Lineage flowing through the streams
GDPR and HIPAA dreams
Classify, retain, and trace
Every message finds its place
Governance rules the Kafka way
Protecting data every day

[Verse 2]
Lineage tracking tells the story clear
Where your data came from, where it steers
Kafka metadata maps the journey long
From producer source to consumer throng
Connect transforms leave their fingerprint
Schema evolution shows each hint
Audit trails through every partition
Data provenance with precision

[Chorus]
Tag it, track it, time it right
Data governance shining bright
Lineage flowing through the streams
GDPR and HIPAA dreams
Classify, retain, and trace
Every message finds its place
Governance rules the Kafka way
Protecting data every day

[Bridge]
Seven years for HIPAA hold
Thirty days when GDPR's told
Right to be forgotten calls
Delete requests through Kafka walls
Retention configs set the time
Log compaction keeps in line
Tombstone markers clear the way
Regulatory compliance stays

[Verse 3]
Topic configs hold the retention key
Time-based cleanup automatically
Size-based limits guard the storage space
Compaction keeps the latest case
Headers carry classification tags
Security policies with feature flags
Kubernetes secrets lock it tight
Governance working day and night

[Final Chorus]
Tag it, track it, time it right
Data governance shining bright
Lineage flowing through the streams
GDPR and HIPAA dreams
Classify, retain, and trace
Every message finds its place
Governance rules the Kafka way
Strimzi keeps your data safe today

[Outro]
From classification to the final delete
Data governance makes compliance complete
In Kubernetes clusters running strong
Kafka governance all along

32. 2 Audit Logging

[Verse 1]
When your Kafka cluster's running tight
Every action needs to see the light
Authorizer logs will track each call
Who accessed what, we log it all
Topic permissions, ACL checks too
Consumer groups and what they do
Security policies enforced with care
Audit trails show who was there

[Chorus]
A-C-E trace every face
Authorizer, CRD, Evidence base
Log it, lock it, never drop it
Compliance flows when audit shows
A-C-E trace every face
Immutable in every space
Track it, stack it, compliance backed it
Strimzi logs know where data goes

[Verse 2]
Kubernetes watches every change
CRD mutations in its range
When KafkaUser gets modified
Or KafkaTopic's been applied
API server logs the call
Who made changes, logs them all
Cluster operators leave their mark
In audit streams both light and dark

[Chorus]
A-C-E trace every face
Authorizer, CRD, Evidence base
Log it, lock it, never drop it
Compliance flows when audit shows
A-C-E trace every face
Immutable in every space
Track it, stack it, compliance backed it
Strimzi logs know where data goes

[Bridge]
Immutable pipelines never lie
Evidence stored up in the sky
Write-once patterns seal the deal
Compliance officers can feel
Confident that every trace
Has a permanent resting place
Tamper-proof and crystal clear
Audit history persists here

[Verse 3]
OpenShift or native K8s
Audit policies never miss
Webhook backends catch each event
Store them where they won't be bent
Syslog, files, or streaming flows
Pick the pipeline that best knows
Your compliance requirements true
Immutable logs will see you through

[Chorus]
A-C-E trace every face
Authorizer, CRD, Evidence base
Log it, lock it, never drop it
Compliance flows when audit shows
A-C-E trace every face
Immutable in every space
Track it, stack it, compliance backed it
Strimzi logs know where data goes

[Outro]
From Kafka auth to K8s state
Audit logging seals your fate
Compliance ready, evidence steady
Strimzi keeps your logs ready

33. 3 Encryption & Key Management

[Verse 1]
Data flows through Kafka streams but stops at every door
Plain text messages expose what we're fighting for
Application level encryption wraps each payload tight
Before it hits the broker, everything's out of sight

[Chorus]
End to end, wrap and send, keys in vaults secure
Certificates rotate, compliance we ensure
Vault or KMS, manage all the rest
Encryption patterns, put them to the test

[Verse 2]
Producer encrypts the message with a symmetric key
Consumer holds the other half to set the data free
While transport layer secures the network connection
Application layer gives us deeper protection

[Chorus]
End to end, wrap and send, keys in vaults secure
Certificates rotate, compliance we ensure
Vault or KMS, manage all the rest
Encryption patterns, put them to the test

[Bridge]
HashiCorp Vault with secrets engine running
Dynamic secrets, automatic key spinning
AWS KMS with envelope encryption
Cross region backup, zero interruption

[Verse 3]
Certificate lifecycle starts with generation
Sign and distribute across the federation
Monitor expiry dates, automate renewal
Compliance reports show our security accrual

[Chorus]
End to end, wrap and send, keys in vaults secure
Certificates rotate, compliance we ensure
Vault or KMS, manage all the rest
Encryption patterns, put them to the test

[Outro]
From producer to consumer, data stays encrypted
Key management systems keep our secrets scripted
Strimzi handles transport, we handle application
Together we achieve complete data protection

34. 4 Access Control Documentation

[Verse 1]
In the world of Kafka streams and queues
KafkaUser maps to roles we choose
Finance team gets read access clean
While operations owns the whole machine
Document every permission granted
Compliance officers won't be stranded

[Chorus]
Map, Approve, Change, Review
Access control in all we do
RBAC evidence crystal clear
SOC2 and ISO standards here
Map, Approve, Change, Review
Documentation sees us through

[Verse 2]
Sarah from accounting needs her data
Consumer permissions in the metadata
DevOps team requires admin rights
Producer access for deployment nights
Every role defined with purpose
Corporate structure at the surface

[Chorus]
Map, Approve, Change, Review
Access control in all we do
RBAC evidence crystal clear
SOC2 and ISO standards here
Map, Approve, Change, Review
Documentation sees us through

[Bridge]
Change requests flowing through the gate
Approval chains we validate
Topic modifications logged with care
User access changes declared
Audit trails that tell the story
Compliance frameworks in their glory

[Verse 3]
Workflows capture every single change
Permission updates we arrange
Pull requests for topic creation
User role modification
Evidence gathered day by day
Auditors will have their way

[Chorus]
Map, Approve, Change, Review
Access control in all we do
RBAC evidence crystal clear
SOC2 and ISO standards here
Map, Approve, Change, Review
Documentation sees us through

[Outro]
From KafkaUser to corporate role
Documentation makes us whole
Change management keeps us right
Access control day and night

35. 3 The Raft Consensus Protocol Primer

[Verse 1]
In the land of distributed nodes, chaos reigns supreme
Until one brave server steps up to lead the team
Raft protocol brings order from the scattered mess
Leader election starts when heartbeats go silent, I confess

[Chorus]
Leader logs and followers sync
Append entries in the chain, don't break the link
Safety first with term numbers climbing high
Raft consensus keeps our data unified

[Verse 2]
Leader sends append entries down the wire
Followers acknowledge or the leader must inquire
Majority votes commit each entry to the log
No split-brain scenarios in this algorithmic fog

[Chorus]
Leader logs and followers sync
Append entries in the chain, don't break the link
Safety first with term numbers climbing high
Raft consensus keeps our data unified

[Verse 3]
Kafka takes the textbook and rewrites the rules
Pull-based replication, not push-based tools
Epochs fence the old leaders when they resurrect
Controller quorum manages what ZooKeeper once protected

[Bridge]
ZAB versus Raft, two cousins in the night
ZooKeeper's broadcast protocol kept Kafka's metadata tight
Now KRaft mode eliminates that external dependency
Three nodes in Docker, check your quorum's harmony

[Verse 4]
Kafka metadata shell reveals the cluster state
Controller logs replicated, no more external fate
Bootstrap servers point to controllers now
Kraft mode simplifies what ZooKeeper used to allow

[Chorus]
Leader logs and followers sync
Append entries in the chain, don't break the link
Safety first with term numbers climbing high
Raft consensus keeps our data unified

[Outro]
Lab time: Docker Compose with three nodes standing guard
Verify quorum health, consensus isn't hard
KRaft protocol evolved from Raft's foundation
Distributed consensus for the streaming generation

36. 4 Client-Side Implications

[Verse 1]
When clients need the cluster map
They knock on broker doors, not controller caps
Metadata flows through familiar gates
While KRaft rebuilds what coordination creates
No direct line to the leader's throne
Brokers bridge the gap, they're not alone

[Chorus]
Metadata still routes through brokers
Four implications, learn the tokens
Faster propagation, less delay
Controller hidden, brokers relay
Cluster ID and Describe API
KRaft revolution, here's the why

[Verse 2]
ZooKeeper's sluggish whispers fade away
KRaft's metadata races, cuts the delay
Propagation speeds like lightning strikes
Controller to broker, then client likes
The latency drops, performance climbs
Milliseconds matter in modern times

[Chorus]
Metadata still routes through brokers
Four implications, learn the tokens
Faster propagation, less delay
Controller hidden, brokers relay
Cluster ID and Describe API
KRaft revolution, here's the why

[Bridge]
Cluster ID marks your domain
DescribeCluster breaks the chain
kafka-metadata shows the snapshot view
kafka-dump-log reveals what's new
Decode the records, trace creation
Topic birth through log foundation

[Verse 3]
Lab time calls with tools in hand
Snapshot inspection helps you understand
Dump the log and read the tale
Every record leaves a trail
From controller writes to broker reads
KRaft plants efficiency seeds

[Chorus]
Metadata still routes through brokers
Four implications, learn the tokens
Faster propagation, less delay
Controller hidden, brokers relay
Cluster ID and Describe API
KRaft revolution, here's the why

[Outro]
Four truths carved in KRaft stone
Brokers serve what controller owns
Speed increased, design refined
Architecture redefined

37. 2 What is KRaft?

[Verse 1]
ZooKeeper's reign is ending now
A quorum built from Kafka's own
Controllers cluster, take a bow
Event streams carved in metadata stone
No external keeper anymore
The protocol lives within the core

[Chorus]
KRaft means Kafka Raft inside
Eating dogfood with controller pride
Metadata flows like events should be
Consensus native, finally free
KRaft, KRaft, the future's here
ZooKeeper fades, the path is clear

[Verse 2]
Three controllers form the quorum base
Event logs hold the cluster state
Each decision leaves a metadata trace
No more external keeper's weight
The architecture speaks in streams
Events and consensus share the same dreams

[Chorus]
KRaft means Kafka Raft inside
Eating dogfood with controller pride
Metadata flows like events should be
Consensus native, finally free
KRaft, KRaft, the future's here
ZooKeeper fades, the path is clear

[Bridge]
From ensemble to controller nodes
The paradigm completely shifts
Event-based consensus explodes
Through streams where metadata drifts
KIP five hundred leads the charge
Making Kafka's footprint less large

[Verse 3]
Controllers vote on every change
Event sourcing tells the tale
No coordination that's strange
Built-in Raft will never fail
Simplified deployment waits
While event logs coordinate

[Chorus]
KRaft means Kafka Raft inside
Eating dogfood with controller pride
Metadata flows like events should be
Consensus native, finally free
KRaft, KRaft, the future's here
ZooKeeper fades, the path is clear

[Outro]
Event-driven all the way
Metadata streams and controllers play
KRaft consensus, here to stay

38. 1 Migration Strategy Overview

[Verse 1]
From ZooKeeper's reign to KRaft's domain
There's no sudden switch, no overnight chain
Three point three unlocked the early door
But three point six brought production shore

[Chorus]
Bridge mode flowing, ZK to KRaft
Migration pathway, carefully crafted
ZK mode to ZK plus KRaft bridge
Then KRaft-only, cross that ridge
No big-bang cutover, smooth transition
Bridge mode strategy, perfect mission

[Verse 2]
First phase running ZooKeeper alone
Second phase they coexist, dual zone
ZK plus KRaft working side by side
Until KRaft-only becomes your guide

[Chorus]
Bridge mode flowing, ZK to KRaft
Migration pathway, carefully crafted
ZK mode to ZK plus KRaft bridge
Then KRaft-only, cross that ridge
No big-bang cutover, smooth transition
Bridge mode strategy, perfect mission

[Bridge]
Version four point zero seals the fate
ZooKeeper vanished, KRaft's final state
Three phases dancing through the years
Bridge mode conquers all your fears

[Verse 3]
Early access brave souls took the leap
Production ready when GA runs deep
From metadata chaos to consensus clean
Smoothest migration you've ever seen

[Chorus]
Bridge mode flowing, ZK to KRaft
Migration pathway, carefully crafted
ZK mode to ZK plus KRaft bridge
Then KRaft-only, cross that ridge
No big-bang cutover, smooth transition
Bridge mode strategy, perfect mission

[Outro]
ZK to bridge to KRaft complete
Migration mastery, can't be beat

39. 2 Strimzi MirrorMaker 2 Configuration

[Verse 1]
KafkaMirrorMaker2 lives as custom resource definition
Bridging clusters across the void with data transmission
YAML specs declare the source and destination endpoints
While replication flows like rivers through these waypoints

[Chorus]
Mirror, mirror on the wall
Which direction serves them all?
Uni flows one way clean
Bi-directional active-active scene
Filters catch what should pass through
Include exclude patterns true
Sync intervals keep the beat
Replication policies complete

[Verse 2]
One direction streams data like a waterfall cascade
Source to target, never back, decisions cleanly made
But active-active dances both ways round the floor
Each cluster writes and reads while syncing more and more

[Chorus]
Mirror, mirror on the wall
Which direction serves them all?
Uni flows one way clean
Bi-directional active-active scene
Filters catch what should pass through
Include exclude patterns true
Sync intervals keep the beat
Replication policies complete

[Verse 3]
Topic filters act like sieves with regex magic spells
Include patterns welcome topics, exclude rings the bells
Group filters work the same way for consumer coordination
Controlling which assemblies cross the federation

[Bridge]
Sync intervals set the rhythm
Heartbeats pulsing through the system
Replication policies decide
How the naming rules collide
Rename topics as they travel
Watch the threading patterns unravel

[Chorus]
Mirror, mirror on the wall
Which direction serves them all?
Uni flows one way clean
Bi-directional active-active scene
Filters catch what should pass through
Include exclude patterns true
Sync intervals keep the beat
Replication policies complete

[Outro]
Configuration carved in stone
Custom resources you now own
Mirror maker understands
Data flowing cross the lands

Back to Home

Apache Kafka & Strimzi | 93