Kafka involves multiple voting mechanisms
When you think of elections in Kafka, the first thing that comes to mind is that there are elections between broker nodes that rely on Zookeeper for elections, elections between partitions, and elections in many other places. But these are all done internally in Kafka, and they all require a leader to control the whole process. The leader is responsible for reading and writing requests, processing message synchronization, monitoring partition changes, monitoring topic changes, saving some partition schemes, recording consumption displacement and other information. I’ve summarized the following elections.
- broker Leader
- partition Leader
- GroupCoordinator Leader
- Group Leader
Controller election
There are one or more brokers in a Kafka cluster. Only one of these brokers is elected as a Controller. The Kafka Controller is responsible for managing the state of all partitions and replicas in the cluster. Update metadata information of ISR collection. The broker election process is the creation of a/Controller temporary node in ZooKeeper, as shown below
Each broker still adds a listener to the /controller node to monitor changes in the node’s data. When the /controller node changes, a new election is triggered.
Election of the zone Leader
The election of the partition leader copy is the responsibility of the controller. When a zone is created (creating a theme or adding a zone involves creating a zone), or a zone goes online (for example, the original leader copy in a zone goes offline, and a new leader needs to be elected online to provide services externally), the leader election action needs to be performed. The basic strategy is to find the first surviving copy in the order of the copies in the AR collection, and that copy is in the ISR collection. The AR collection of a partition is specified at allocation time, and the order of the copies within the collection remains the same as long as no redistribution occurs, while the order of the copies in the ISR collection of a partition may change.
In the string following Partitions, the Leader replica is in square brackets.
The election of GroupCoordinator
A GroupCoordinator is responsible for allocating zones and rebalancing consumers. In the initial stage, if a consumer does not save information about a GroupCoordinator node corresponding to a consumer group, a GroupCoordinator needs to send a request to the node with the lowest load in the cluster to find it. Kafka calculates the partition number in __consumer_offsets using the hash value of the groupId of the consumer group. After finding the partition, Kafka finds the broker node where the leader copy of the partition resides, which is the corresponding GroupCoordinator. The final partition allocation scheme of the consumer and the consumption shift information submitted by the consumer within the group are sent to the broker node where the leader copy of the sub-partition resides.
Election of the consumer group Leader
A GroupCoordinator needs to elect a leader for a consumer group. The election algorithm is simple. If there is no leader ina consumer group, the first consumer to join the consumer group is the leader of the consumer group. The leader is selected as the first key-value pair in the consumer node data stored in the HashMap structure.