Loading audiobook...

Apache Kafka with Kubernetes

11 chapters

1. 1 Listener Types

[Verse 1]
In your Kafka cluster running on K8s land
Six listener types help clients understand
How to connect and reach your streaming data
Each one's designed for different use case patterns

Internal listeners keep it all inside
ClusterIP services where your pods reside
No external access, just cluster-to-cluster talk
Security tight, on the internal network walk

[Chorus]
Six ways to listen, six ways to connect
Internal, Route, LoadBalancer direct
NodePort opens, Ingress controls the flow
ClusterIP custom, that's how your clients go

Listen up, listen now, choose the right type
For your Kafka streams flowing through the pipe

[Verse 2]
Route listeners are OpenShift's special way
Creating Routes that external clients can relay
Hostname routing through the platform's edge
Automatic SSL at the router's pledge

LoadBalancer type calls your cloud provider
Creates an LB, traffic's smooth rider
AWS, GCP, Azure will provision
External IP for your client's mission

[Chorus]
Six ways to listen, six ways to connect
Internal, Route, LoadBalancer direct
NodePort opens, Ingress controls the flow
ClusterIP custom, that's how your clients go

Listen up, listen now, choose the right type
For your Kafka streams flowing through the pipe

[Bridge]
NodePort opens a port on every node
Thirty thousand range, access mode is known
Direct connection to your cluster's machines
Simple but crude for production scenes

[Verse 3]
Ingress listeners use your favorite controller
NGINX or Traefik, they're the traffic patroller
Path-based routing, host-based too
SSL termination, certificates renewed

ClusterIP custom with your own DNS name
Service discovery playing the networking game
Internal resolution but with your domain
Custom addressing in your cluster's domain

[Chorus]
Six ways to listen, six ways to connect
Internal, Route, LoadBalancer direct
NodePort opens, Ingress controls the flow
ClusterIP custom, that's how your clients go

Listen up, listen now, choose the right type
For your Kafka streams flowing through the pipe

[Outro]
From internal pods to external clients all
Strimzi listeners answer every call
Choose your type based on your architecture need
Let your Kafka cluster successfully feed

2. Official Documentation

[Verse 1]
ZooKeeper's been our metadata friend
But dependencies we need to end
KIP-500 shows a better way
Self-managed quorum saves the day
Controllers form their own consensus
Raft protocol keeps things defenseless

[Chorus]
KRaft mode, no Zoo needed
Quorum-based, self-completed
Metadata logs in perfect sync
Controllers vote without a blink
KRaft mode, streamlined and clean
Best architecture we've ever seen

[Verse 2]
Check the docs on Apache's site
KRaft section makes it crystal bright
Configuration starts with process roles
Controller, broker, or combined goals
Set your quorum voters carefully
Three or five nodes work perfectly

[Chorus]
KRaft mode, no Zoo needed
Quorum-based, self-completed
Metadata logs in perfect sync
Controllers vote without a blink
KRaft mode, streamlined and clean
Best architecture we've ever seen

[Bridge]
KIP-631 defines the controller
Quorum-based, it's so much smaller
Leader election through Raft consensus
Metadata partitions, zero expenses
Read the documentation thoroughly
Every detail matters, you will see

[Verse 3]
Bootstrap servers point to controllers
Cluster metadata, they're the providers
Log directories need proper planning
Controller logs need understanding
Official docs have all the answers
Migration guides for all advancers

[Chorus]
KRaft mode, no Zoo needed
Quorum-based, self-completed
Metadata logs in perfect sync
Controllers vote without a blink
KRaft mode, streamlined and clean
Best architecture we've ever seen

[Outro]
When you're ready to deploy KRaft
Read the docs, both fore and aft
Apache Kafka's future's bright
KRaft mode makes everything right

3. 2 Configuring External Access

[Verse 1]
When your Kafka cluster lives inside the cloud
External clients need a way to reach out loud
Advertised listeners tell the world your name
Where to find each broker in the network game
Bootstrap discovery gets the party started
But address resolution keeps connections charted

[Chorus]
Configure external access, make it shine
Advertised listeners, bootstrap design
TLS passthrough or termination choice
DNS and hostnames give your brokers voice
Per-broker services or shared bootstrap way
External access working every day

[Verse 2]
TLS passthrough lets the broker handle keys
Client talks direct with encrypted expertise
But termination stops the SSL at the gate
Load balancer decrypts, then seals the fate
Choose your strategy based on your security needs
Both approaches plant successful seeds

[Chorus]
Configure external access, make it shine
Advertised listeners, bootstrap design
TLS passthrough or termination choice
DNS and hostnames give your brokers voice
Per-broker services or shared bootstrap way
External access working every day

[Bridge]
DNS resolution maps the names to addresses
Hostname configuration handles all the stresses
Each broker needs its advertised identity
External clients find them with such clarity
Network topology shapes the paths they take
Good configuration prevents connection break

[Verse 3]
Per-broker services give each one its own door
Dedicated endpoints, scalability and more
Shared bootstrap service starts with single entry
Discovery protocol fills the inventory
LoadBalancer type or NodePort exposition
Choose the right method for your client's mission

[Chorus]
Configure external access, make it shine
Advertised listeners, bootstrap design
TLS passthrough or termination choice
DNS and hostnames give your brokers voice
Per-broker services or shared bootstrap way
External access working every day

[Outro]
From inside Kubernetes to the world outside
Strimzi makes the bridge, your traffic guide
External access configured right and true
Kafka clusters ready for the world to use

4. 4 Lab: External Access

[Verse 1]
Inside your cluster Kafka waits to speak
But external clients cannot reach its peak
LoadBalancer listeners are the key you need
To bridge the gap and make connections feed

Configure your cluster YAML with care
Set external type LoadBalancer there
Each broker gets its own public face
Ready to handle the networking race

[Chorus]
Load it up, balance out, TLS secure the way
External access through the LoadBalancer gateway
Connect and produce, consume with ease
Kafka flowing through Kubernetes

[Verse 2]
Your local client needs the proper setup
Certificates and truststore to connect up
Bootstrap servers point to external ports
TLS encryption for secure transports

Copy the cluster certificate file
Configure your client properties in style
Security protocol TLS is the name
SSL truststore location starts the game

[Chorus]
Load it up, balance out, TLS secure the way
External access through the LoadBalancer gateway
Connect and produce, consume with ease
Kafka flowing through Kubernetes

[Bridge]
When connections fail and you're stuck in doubt
Check your security groups and routes
DNS resolution might be the clue
Port forwarding issues could be blocking you

Network policies might restrict the flow
Service endpoints you need to know
Troubleshoot step by step with care
Debug the path from here to there

[Chorus]
Load it up, balance out, TLS secure the way
External access through the LoadBalancer gateway
Connect and produce, consume with ease
Kafka flowing through Kubernetes

[Outro]
External access configured right
Clients connecting day and night
Strimzi makes it smooth and clean
Best Kafka setup you've ever seen

5. 1 Kafka Broker Configuration

[Verse 1]
In your Strimzi cluster, there's a way to tune
Spec dot kafka config, make your brokers croon
Set the log retention, compression type as well
Cleanup policies running, stories they will tell
JVM heap memory, garbage collection too
Resource limits matter, CPU cores for you

[Chorus]
Configure, orchestrate, Kafka brokers sing
Spec config, JVM tricks, storage everything
Listeners internal, external TLS
SASL authentication, security's the best
Configure, orchestrate, make your data flow
Kafka broker settings, watch your cluster grow

[Verse 2]
Log directories scattered across your persistent disk
Storage class selection, performance is the risk
Request your memory, set your CPU bound
Limits keep you stable when the load comes around
Heap size calculations, garbage collector choice
Give your Java runtime a optimized voice

[Chorus]
Configure, orchestrate, Kafka brokers sing
Spec config, JVM tricks, storage everything
Listeners internal, external TLS
SASL authentication, security's the best
Configure, orchestrate, make your data flow
Kafka broker settings, watch your cluster grow

[Bridge]
Plain text listeners for internal cluster chat
TLS encrypted channels where the secrets are at
SASL mechanisms, username password auth
OAuth bearer tokens, choose your security path
Bootstrap servers ready, clients can connect
Internal service mesh, external load direct

[Verse 3]
Min in sync replicas, replication factor high
Default topic partitions, messages that fly
Log segment bytes limit, index interval time
Flush messages batched, performance so fine
Socket buffer sizes, network receive queue
Background threads working, processing all you do

[Chorus]
Configure, orchestrate, Kafka brokers sing
Spec config, JVM tricks, storage everything
Listeners internal, external TLS
SASL authentication, security's the best
Configure, orchestrate, make your data flow
Kafka broker settings, watch your cluster grow

[Outro]
From spec to production, your brokers stand tall
Configured and ready, they're handling it all
Kafka on Kubernetes, Strimzi leads the way
Broker configuration, mastered today

6. 4 The Road Ahead

[Verse 1]
ZooKeeper's gone, we've moved ahead
KRaft controller leads instead
Dynamic quorum's here to stay
Add and remove without delay
No restart needed anymore
Eight five three opens the door

[Chorus]
The road ahead is crystal clear
Dynamic quorum, leader here
Scale to millions, watch it grow
KRaft four point oh steals the show
Metadata flowing fast and free
Post-ZooKeeper destiny

[Verse 2]
Nine six six brings something new
Eligible leaders, just a few
Not every replica can lead
Only the chosen ones we need
Controller picks the worthy ones
Until the leadership race runs

[Chorus]
The road ahead is crystal clear
Dynamic quorum, leader here
Scale to millions, watch it grow
KRaft four point oh steals the show
Metadata flowing fast and free
Post-ZooKeeper destiny

[Bridge]
Tiered storage coming soon
Metadata spread around the moon
Geographic replication
Cross the world's coordination
Future holds such grand designs
Kafka's crossing all the lines

[Verse 3]
Lab six puts us to the test
Five hundred thousand at our best
Partitions flowing like a stream
Measure failover, live the dream
Controller switching, brokers sync
Convergence faster than you think

[Chorus]
The road ahead is crystal clear
Dynamic quorum, leader here
Scale to millions, watch it grow
KRaft four point oh steals the show
Metadata flowing fast and free
Post-ZooKeeper destiny

[Outro]
Benchmark shows the future's bright
KRaft cluster shining in the light
No more restarts, no more pain
Dynamic membership's our gain

7. 4 Storage Strategies

[Verse 1]
When you're setting up your Kafka cluster right
Storage strategies keep your data flowing bright
JBOD means just a bunch of disks you see
Multiple drives working independently
No RAID complexity to slow things down
Each disk operates safe and sound
Configure your broker with volumes galore
Spread the load across each storage core

[Chorus]
Four strategies to make your storage sing
JBOD, classes, expansion, benchmarking
Remember the flow - distribute and scale
Choose the right class so you never fail
Expand when you need it, test what you've got
Storage done right hits the sweet spot

[Verse 2]
Cloud providers offer storage classes wide
GP three for general workloads you can't hide
IO two for when you need those IOPS fast
Premium SSD makes performance last
Match your workload to the storage type
Don't overpay when standard's just right
Throughput versus latency decide
Let your requirements be your guide

[Chorus]
Four strategies to make your storage sing
JBOD, classes, expansion, benchmarking
Remember the flow - distribute and scale
Choose the right class so you never fail
Expand when you need it, test what you've got
Storage done right hits the sweet spot

[Bridge]
Volume expansion when you're running low
Kubernetes makes your storage grow
Online resizing keeps the service live
No downtime when you need more drive
But benchmark first before you choose
Test your throughput, latency you can't lose
FIO and iostat show the way
Performance metrics guide your day

[Verse 3]
Strimzi makes configuration clean
Storage specs in YAML seen
Persistent volume claims define your need
Size and class for guaranteed speed
Monitor your disk utilization tight
Alerting keeps your storage right
When bottlenecks start to appear
These four strategies keep performance clear

[Chorus]
Four strategies to make your storage sing
JBOD, classes, expansion, benchmarking
Remember the flow - distribute and scale
Choose the right class so you never fail
Expand when you need it, test what you've got
Storage done right hits the sweet spot

[Outro]
From JBOD config to benchmark tests
These storage strategies are the best
Keep your Kafka cluster running strong
With storage planned you can't go wrong

8. 2 Managing Connectors Declaratively

[Verse 1]
In Kubernetes land where Kafka flows
There's a custom resource that everyone knows
KafkaConnector CRD makes the magic real
Declarative management with mass appeal
No more REST calls or manual commands
Just YAML specs that the cluster understands

[Chorus]
Connectors flow, declare don't call
Source brings in, sink takes it all
Tasks and scaling, config so clean
Dead letter queues for errors unseen
CRD way, the modern day
Managing streams the Strimzi way

[Verse 2]
Source connectors pull from external stores
Database changes through your Kafka doors
Sink connectors push data out the gate
To warehouses where analytics await
Each connector spawns tasks to do the work
Parallel processing, no need to lurk

[Chorus]
Connectors flow, declare don't call
Source brings in, sink takes it all
Tasks and scaling, config so clean
Dead letter queues for errors unseen
CRD way, the modern day
Managing streams the Strimzi way

[Bridge]
When errors strike don't let them break
Error tolerance for goodness sake
Dead letter topics catch what's wrong
Keep your pipeline running strong
Scale your tasks both up and down
Best connector management around

[Verse 3]
Configuration lives in spec so neat
Class and config make it complete
Pause and resume with a simple change
No downtime required to rearrange
Status shows you what's happening now
Health and state, the operator's vow

[Chorus]
Connectors flow, declare don't call
Source brings in, sink takes it all
Tasks and scaling, config so clean
Dead letter queues for errors unseen
CRD way, the modern day
Managing streams the Strimzi way

[Outro]
KafkaConnector CRD
Declarative connectivity
Source and sink, tasks that scale
Error handling never fails
Strimzi makes connectors shine
Kubernetes native by design

9. 3 Common Connector Patterns

[Verse 1]
When your database changes flow like a stream
Debezium captures every single theme
PostgreSQL, MySQL, MongoDB too
Change data capture makes your pipeline true
Binlog and WAL files tell the tale
Of every insert, update without fail

[Chorus]
Three patterns strong, three patterns bright
Moving your data left and right
CDC flows, sinks store away
Connectors working night and day
Debezium streams, S3 saves
JDBC talks, Elasticsearch waves

[Verse 2]
Data lakes need feeding from your Kafka flow
S3 Sink and Azure Blob help data grow
Parquet files and JSON stored secure
Object storage makes your pipeline pure
Partition by time, organize by key
Analytics ready for your team to see

[Chorus]
Three patterns strong, three patterns bright
Moving your data left and right
CDC flows, sinks store away
Connectors working night and day
Debezium streams, S3 saves
JDBC talks, Elasticsearch waves

[Verse 3]
JDBC Source pulls from tables old
JDBC Sink writes stories to be told
Relational bridges built with SQL
Insert and update, connections never dull
From Oracle to Postgres, data moves
Database integration always proves

[Bridge]
MirrorMaker Two replicates across
Cross-cluster streaming, never at a loss
Elasticsearch indexing makes search fast
Document storage built to last
Three patterns dancing in the cloud
Strimzi makes Kafka sing out loud

[Chorus]
Three patterns strong, three patterns bright
Moving your data left and right
CDC flows, sinks store away
Connectors working night and day
Debezium streams, S3 saves
JDBC talks, Elasticsearch waves

[Outro]
From source to sink, the data flows
Through Kubernetes, your knowledge grows
Connector patterns, now you know
Let your streaming pipeline glow

10. Talks & Articles

[Verse 1]
There once was a time when Kafka relied
On ZooKeeper standing by its side
But scaling up meant complexity grew
Metadata scattered, performance fell through
Colin McCabe saw a better way
KIP-500 would save the day

[Chorus]
KRaft mode, no more ZooKeeper pain
Self-managing cluster, breaking the chain
Raft protocol keeps consensus strong
Leader election where it belongs
KRaft mode, streamlined and clean
The simplest Kafka you've ever seen

[Verse 2]
Jason showed us how Raft works inside
One leader chosen, others subside
Heartbeats flowing to stay alive
If leader fails, new one will thrive
Log replication keeps data safe
Majority wins, no time to waste

[Chorus]
KRaft mode, no more ZooKeeper pain
Self-managing cluster, breaking the chain
Raft protocol keeps consensus strong
Leader election where it belongs
KRaft mode, streamlined and clean
The simplest Kafka you've ever seen

[Bridge]
Confluent made it simple to see
Fewer moving parts means stability
Bootstrapping easier than before
No external dependencies to store
Metadata controllers lead the way
Built into Kafka, here to stay

[Verse 3]
Three key players in this new design
Controllers, brokers, working in line
Event-driven architecture flows
Self-contained system that truly knows
How to manage its own state
No ZooKeeper dependency weight

[Chorus]
KRaft mode, no more ZooKeeper pain
Self-managing cluster, breaking the chain
Raft protocol keeps consensus strong
Leader election where it belongs
KRaft mode, streamlined and clean
The simplest Kafka you've ever seen

[Outro]
From KIP-500 to production use
KRaft has set your clusters loose
No more ZooKeeper in the way
Welcome to Kafka's brighter day

11. 2 Strimzi MirrorMaker 2 Configuration

[Verse 1]
When you need to mirror Kafka streams across the wire
KafkaMirrorMaker2 CRD is what you require
Define your source and target clusters in the spec
Connect them with replication that you can inspect

[Chorus]
Mirror, mirror, data flows
From source to target, watch it go
Filters and patterns, sync in time
Replication policies by design
Mirror, mirror, keep it straight
Uni or bi-directional fate

[Verse 2]
Uni-directional sends one way, source to destination
Active-active flows both ways for full replication
Choose your pattern based on needs, disaster recovery
Or active-active for the load, shared delivery

[Chorus]
Mirror, mirror, data flows
From source to target, watch it go
Filters and patterns, sync in time
Replication policies by design
Mirror, mirror, keep it straight
Uni or bi-directional fate

[Verse 3]
Topic filters let you choose what gets replicated
Include patterns bring them in, exclude keeps them waited
Regular expressions rule the matching game you play
Consumer groups can filter too in the very same way

[Bridge]
Sync intervals control the speed
How often data flows you need
Seconds, minutes, find your pace
Replication policy in place

[Verse 4]
Default policy renames topics with cluster prefix
Source dot cluster dot topic name, that's how it sticks
Override with custom rules if defaults don't fit right
Heartbeats and checkpoints keep your mirroring tight

[Chorus]
Mirror, mirror, data flows
From source to target, watch it go
Filters and patterns, sync in time
Replication policies by design
Mirror, mirror, keep it straight
Uni or bi-directional fate

[Outro]
Configure your clusters, set your rules
MirrorMaker2 gives you the tools
Strimzi makes it Kubernetes native
Cross-cluster replication, so creative

Back to Home

Apache Kafka with Kubernetes | 93