1 Why KRaft Exists sitar blues, prog ambient noise wall, jungle afrobeat · 4:02 Discover why Apache Kafka is transitioning away from ZooKeeper dependency through KRaft mode, exploring the limitations of the traditional architecture and the benefits of Kafka's new self-managing consensus protocol.
2 What is KRaft? dark dance, accordion afro trap, sitar afro-funk, electro-bossa nova · 3:32 Learn how KRaft revolutionizes Apache Kafka by eliminating the need for ZooKeeper as an external dependency, instead building cluster management directly into Kafka's core architecture. Discover why this fundamental shift represents a major evolution in how Kafka handles distributed coordination and state management.
3 The Raft Consensus Protocol Primer griot shoegaze, hypnagogic garage, appalachian folk cloud rap · 3:07 Learn the fundamental building blocks of the Raft consensus protocol, including leader election mechanics and the critical safety guarantees that enable distributed systems to maintain consistency across multiple nodes.
1 Node Roles hyper-acid house, lo-fi afro-cuban jazz, bengali afroswing · 3:56 Learn the three fundamental node roles in Apache Kafka's KRaft architecture and how controllers, brokers, and combined nodes each serve distinct purposes in cluster management. Discover how to strategically assign these roles to optimize your Kafka deployment for metadata handling and client request processing.
2 The Metadata Log (`__cluster_metadata` topic) acoustic texas blues afrobeat, garage tango, dakar house, hypnagogic ambient trance · 4:21 Explore the __cluster_metadata topic, a single-partition log that serves as the central nervous system of KRaft clusters, recording and replicating all configuration changes across active controller nodes. Learn how this critical component maintains cluster state consistency and ensures reliable metadata management without the complexity of multi-partition coordination.
3 The Quorum Controller tokyo afro-funk, dark acid house · 4:01 Learn how Apache Kafka's KRaft protocol uses a three-node quorum controller system where one leader emerges through Raft consensus algorithm while the other controllers remain in standby mode. Discover the democratic process that ensures every node participates in electing the cluster's decision-making authority.
4 Broker Lifecycle in KRaft dark dance, accordion afro trap, sitar afro-funk, electro-bossa nova · 3:20 Learn how Apache Kafka brokers navigate their complete lifecycle in KRaft mode, from initial cluster registration and epoch assignment through heartbeat maintenance to graceful shutdown or failure scenarios. Discover the critical states of fenced and unfenced brokers and how the controller manages these transitions to ensure cluster stability.
1 The KRaft Protocol (Raft Implementation) acoustic texas blues afrobeat, garage tango, dakar house, hypnagogic ambient trance · 4:02 Learn how Apache Kafka's KRaft protocol revolutionizes distributed systems by eliminating ZooKeeper dependency and implementing a new leader election mechanism through Vote RPC and BeginQuorumEpoch processes. Discover the fundamental shift from traditional coordination services to a streamlined consensus protocol that forms the backbone of modern Kafka deployments.
2 Metadata Snapshots acoustic texas blues afrobeat, garage tango, dakar house, hypnagogic ambient trance · 3:36 Learn how Apache Kafka's metadata snapshots solve the critical bootstrap problem by allowing new brokers to quickly catch up on cluster state without processing massive historical logs. Discover the technical mechanics behind this essential optimization that enables faster cluster startup and recovery in distributed streaming systems.
3 Metadata Records & Versioning tokyo afro-funk, dark acid house · 3:34 Explore the eight essential metadata record types that form the backbone of Apache Kafka's KRaft architecture, learning how topics, partitions, and broker registrations maintain cluster state and handle real-time updates.
4 Client-Side Implications hyper-acid house, lo-fi afro-cuban jazz, bengali afroswing · 3:34 Learn how KRaft's new architecture maintains backward compatibility by keeping client-broker communication patterns intact while the underlying controller mechanism transforms completely. Discover why clients continue using familiar metadata request workflows through brokers rather than directly contacting the controller plane.
1 Migration Strategy Overview hyper-acid house, lo-fi afro-cuban jazz, bengali afroswing · 3:16 Learn the fundamentals of migrating from ZooKeeper to KRaft architecture through a systematic bridge approach that avoids risky big-bang deployments. Discover how to transition smoothly from traditional coordination methods to KRaft's modern controller-led system without downtime or crashes.
2 Step-by-Step Migration Process sitar blues, prog ambient noise wall, jungle afrobeat · 4:14 Learn the essential pre-migration steps for transitioning from ZooKeeper to KRaft architecture, covering version compatibility checks, controller configuration, and storage formatting requirements. This technical guide breaks down the critical preparation phase that ensures a smooth upgrade path for your Apache Kafka infrastructure.
3 Rollback & Risk Management acoustic texas blues afrobeat, garage tango, dakar house, hypnagogic ambient trance · 4:17 Learn essential rollback strategies and risk mitigation techniques for safely managing Kafka KRaft migrations, including monitoring key metrics and leveraging bridge mode to prevent irreversible cluster failures.
1 Sizing & Topology Planning sitar blues, prog ambient noise wall, jungle afrobeat Learn the fundamental principles of Apache Kafka cluster architecture, focusing on controller node sizing and the critical role of metadata coordination in distributed systems. Discover why three controllers work for most deployments while enterprise environments require five for optimal fault tolerance and performance.
2 Configuration Reference tokyo afro-funk, dark acid house Learn the seven essential KRaft configuration parameters that define node roles, unique identifiers, and server behavior in Apache Kafka's controller-less architecture. Master the fundamentals of setting up process roles, node IDs, and other critical settings needed to deploy KRaft clusters effectively.
3 Monitoring & Observability tokyo afro-funk, dark acid house · 3:51 Learn essential monitoring techniques for KRaft clusters through key JMX metrics, starting with ActiveControllerCount to ensure proper controller leadership and detect critical system failures. Master the fundamental observability practices needed to maintain healthy Kafka deployments in the KRaft architecture.
4 Security in KRaft Mode acoustic texas blues afrobeat, garage tango, dakar house, hypnagogic ambient trance · 4:18 Learn how KRaft mode revolutionizes Apache Kafka security by implementing TLS encryption, SASL authentication, and streamlined ACL management without ZooKeeper dependencies. Discover the enhanced security architecture where Controllers manage metadata logs directly, creating a more robust and simplified authentication system.
5 Disaster Recovery tokyo afro-funk, dark acid house Learn essential techniques for recovering from catastrophic failures in KRaft clusters, including how to restore metadata logs and rebuild your Kafka infrastructure when controllers go down and quorum is lost. Master the critical bootstrap procedures and backup strategies that will save your distributed system from complete collapse.
1 KRaft Performance Characteristics griot shoegaze, hypnagogic garage, appalachian folk cloud rap · 3:37 Learn how KRaft revolutionizes Kafka performance by eliminating ZooKeeper dependencies and introducing built-in controllers that deliver faster metadata operations and reduced system complexity.
2 KRaft and Kafka Streams / Connect tokyo afro-funk, dark acid house · 3:39 Learn how KRaft mode revolutionizes Apache Kafka's architecture by eliminating ZooKeeper dependencies while maintaining seamless Kafka Streams and Connect functionality. Discover how this fundamental shift changes cluster coordination without disrupting your existing streaming applications and data persistence patterns.
3 Multi-Tenancy & Quotas acoustic texas blues afrobeat, garage tango, dakar house, hypnagogic ambient trance · 3:54 Learn how KRaft controllers manage multiple tenants through clean isolation boundaries and discover the essential role of metadata quotas and rate limiting in maintaining optimal performance across shared environments.
4 The Road Ahead tokyo afro-funk, dark acid house Explore how Apache Kafka's revolutionary KRaft architecture eliminates ZooKeeper dependency and enables seamless cluster management through dynamic quorum control. Learn the technical foundations of version 8.5.3's breakthrough capabilities for adding and removing nodes without system restarts.
Official Documentation acoustic texas blues afrobeat, garage tango, dakar house, hypnagogic ambient trance Explore Apache Kafka's revolutionary KIP-500 architecture that eliminates ZooKeeper dependencies by implementing self-managed quorum controllers using the Raft consensus protocol. Learn how this fundamental shift simplifies Kafka's infrastructure while improving reliability and reducing operational complexity.
Talks & Articles acoustic texas blues afrobeat, garage tango, dakar house, hypnagogic ambient trance · 3:39 Learn how Apache Kafka evolved from its ZooKeeper dependency to the revolutionary KIP-500 architecture through Colin McCabe's vision for simplified metadata management. Discover how this transformation solved critical scaling and performance challenges that plagued earlier Kafka deployments.
Related KIPs hyper-acid house, lo-fi afro-cuban jazz, bengali afroswing Learn how KIP-595 introduces the Raft consensus protocol to Apache Kafka, replacing Zookeeper with a more streamlined approach where controller quorum leaders make metadata decisions through direct consensus. Discover the foundational building blocks that enable Kafka's evolution toward a more efficient and simplified architecture.