1. Zookeeper Overview

Zookeeper is a distributed coordination service with open source code. It is a high performance distributed data consistency solution. It encapsulates the complex and error-prone distributed consistency services. Users can invoke interfaces provided by Zookeeper to solve practical problems in distributed applications.

2, the characteristics of

  1. Zookeeper: A leader, a cluster of followers.
  2. If more than half of the nodes in the cluster are alive, the cluster can serve normally.
  3. Global data consistency: Each Server stores the same data copy. The data is consistent regardless of which Server the Client connects to.
  4. Update requests are ordered, and update requests from the same Client are executed in the order in which they are sent.
  5. Data update atomicity either succeeds or fails.
  6. Real-time: The Client can read the latest data within a certain period of time.

3. Data structure

The Structure of the Zookeeper data model is similar to that of the Unix file system and can be viewed as a tree, with each node called a ZNode. Each ZNode can store 1MB of data by default, and each ZNode can be uniquely identified by its path.

Like a file system, we can add and remove zNodes freely, and add and remove child ZNodes within a ZNode, except that zNodes can store data.

There are four types of ZNodes:

  • Persistent-persistent Directory nodes exist even after the client is disconnected from ZooKeeper
  • PERSISTENT_SEQUENTIAL- PERSISTENT_SEQUENTIAL- PERSISTENT_SEQUENTIAL- PERSISTENT_SEQUENTIAL- PERSISTENT_SEQUENTIAL- PERSISTENT_SEQUENTIAL- PERSISTENT_SEQUENTIAL- PERSISTENT_SEQUENTIAL- PERSISTENT_SEQUENTIAL- PERSISTENT_SEQUENTIAL
  • EPHEMERAL- Temporary Directory node was removed after the client was disconnected from ZooKeeper
  • After the directory node client is disconnected from ZooKeeper, the node is deleted. Zookeeper only numbers the node name sequentially

** Set the sequence identifier when creating zNode. A value will be appended after the name of zNode. The sequence number is a monotonically increasing counter maintained by the parent node

** Note: ** In distributed systems, ordinals can be used to globally sort all events so that clients can infer the order of events from ordinals

4. Application Scenarios

4.1 Unified Naming Service

In a distributed environment, applications and services need to be named uniformly for easy identification.

For example, IP is not easy to remember, but domain name is easy to remember.

4.2 Unified Configuration Management

  1. Configuration file synchronization is common in distributed environments. (1) The configuration information of all nodes in a cluster must be consistent, for example, a Kafka cluster. (2) After a configuration file is modified, you need to quickly synchronize the configuration file to all nodes
  2. ZooKeeper implements configuration management. (1) ZooKeeper writes configuration information to a Znode on ZooKeeper. (2) Each client server listens to the Znode. (3) ZooKeeper notifies each client server when data in the Znode is modified

Assuming that our program is distributed on multiple machines, if we want to change the configuration file of the program, we need to change it on each machine, which is very troublesome. Now, we put all these configurations into ZooKeeper and save them in a directory node of ZooKeeper, and then all related applications listen on this directory node. When configuration information is changed, each application receives notification from ZooKeeper and obtains the new configuration information from ZooKeeper to apply to the system.

4.3 Unified Cluster Management

  1. In distributed environment, it is necessary to grasp the status of each node in real time. (1) Some adjustments can be made according to the real-time status of nodes
  2. ZooKeeper can monitor node status changes in real time. (1) You can write node information to a ZNode on ZooKeeper. (2) You can monitor the ZNode to obtain its real-time status changes

4.4 Soft load Balancing

Zookeeper records the number of accesses to each server, allowing the server with the least number of accesses to process the latest client requests.

5. Election mechanism

  1. Half mechanism: More than half of the machines in the cluster are alive and the cluster is available. Therefore, Zookeeper can be installed on an odd number of servers. For example, 3 out of 5 servers are alive and the cluster is available, while only 2 servers are alive and the cluster is unavailable.
  2. Zookeeper does not specify Master and Slave in its configuration file. However, when Zookeeper works, one node serves as the Leader and the other nodes serve as followers. The Leader is temporarily elected through an internal election mechanism.

Suppose there is a Zookeeper cluster consisting of five servers, whose ids range from 1 to 5, and they are all newly started, that is, there is no historical data. In terms of data quantity, they are all the same. Suppose these servers are started sequentially:

  1. Server 1 starts and initiates an election. Server 1 votes for itself. At this point, server 1 has one vote, less than half (3 votes), the election cannot be completed, and the state of server 1 remains LOOKING.
  2. Server 2 is started, and another election is called. Server 1 and server 2 vote for each other and exchange vote information: server 1 finds that server 2’s ID is larger than the one it currently votes for (server 1) and changes the vote to server 2. At this time, server 1 has 0 votes, and server 2 has 2 votes. If there is no more than half of the results, the election cannot be completed. Server 1 and server 2 remain LOOKING
  3. Server 3 starts and initiates an election. Both servers 1 and 2 change the vote to server 3. The result of this vote: 0 votes for server 1, 0 votes for server 2, 3 votes for server 3. At this point, the number of votes of server 3 exceeds half, and server 3 is elected as the Leader. Change the state of server 1 and server 2 to FOLLOWING and the state of server 3 to LEADING.
  4. Server 4 starts and initiates an election. Servers 1, 2, and 3 are no longer LOOKING and will not change the ballot information. Result: 3 votes for server 3, 1 vote for server 4. At this point, server 4 follows the majority, changes the vote information to server 3, and changes the status to FOLLOWING;
  5. Server 5 starts, changing the state to FOLLOWING as with 4.

6. Principle of listener

  1. First, there is a main() thread
  2. When the Zookeeper client is created in the main thread, two threads are created, one for network connection communication (Connet) and one for listener.
  3. Registered listening events are sent to Zookeeper via the Connect thread.
  4. Add the registered listener events to the List of Registered listeners in Zookeeper.
  5. When Zookeeper detects a change in data or path, it sends the message to the listener thread.
  6. The listener thread calls the process() method internally.

Common listening:

  1. Get path [watch]
  2. Ls path [watch]

7. Data writing process

  1. The Client writes data to Server1 of ZooKeeper and sends a write request.
  2. If Server1 is not the Leader, then Server1 forwards the received requests to the Leader, because one of the ZooKeeper servers is the Leader. The Leader broadcasts the write request to various servers, such as Server1 and Server2, who queue the write request and send the Leader a success message.
  3. If the Leader receives success messages from more than half of the servers, the write operation can be performed. The Leader sends the submission information to each Server. After receiving the information, each Server implements the write request in the queue. Then, the write succeeds.
  4. Server1 further notifies the Client that the data write was successful, at which point the entire write operation is considered successful.