Kafka 2.8.0 is out with one major improvement:
Implemented Raft distributed consistency mechanism which means you can run independently of ZooKeeper.
ZooKeeper plays an important role in Kafka, storing Kafka metadata.
ZooKeeper stores metadata for partitions and brokers, as well as elections for Kafka Controllers.
ZooKeeper is an external system for Kafka. To deploy a Kafka cluster, you need to deploy, manage, and monitor ZooKeeper.
ZooKeeper has its own configuration methods and management tools, which are completely different from Kafka. Therefore, running two distributed systems at the same time increases the complexity and makes problems more likely. Sometimes the workload is doubled. For example, to enable some security features, Kafka and ZooKeeper require configuration.
In addition to complexity, external storage can also reduce system efficiency.
For example, each time a Kafka cluster starts, the Controller must load cluster status information from ZooKeeper.
For example, after a new Controller is elected, it will be troublesome because metadata needs to be loaded, and the amount of metadata may already be very large, which causes efficiency problems.
As a result, the complexity and efficiency of ZooKeeper has become a pain point for Kafka, and the Kafka team has been working hard to wean itself off ZooKeeper. Kafka 2.8.0 is finally available.
After Raft mode is used, metadata and configuration information is stored in the @metadata Topic and automatically replicated across the cluster. This makes Kafka much simpler and lighter.
However, zookeeper-less Kafka is still an early version and not perfect, so do not apply it in a production environment right now.