Zookeeper Basic Concepts 2.1 Cluster Roles 2.2 Cluster Node Division 2.3 Session 2.4 Data Node 2.5 Status Information 2.6 Transaction Operations 2.7 Watcher 3. Typical ZooKeeper application Scenarios 3.1 Data Publishing and Subscription (Configuration Center) 3.2 Naming Service 3.3 Distributed Coordination Service/Notification 3.4Master Election 3.5 Distributed Lock four. conclusion

Zookeeper function Overview

ZooKeeper is an open source distributed coordination service created by Yahoo and an open source implementation of Google Chubby. Distributed applications can implement functions such as data publishing/subscription, load balancing, naming service, distributed cocoordination/notification, cluster management, Master election, configuration maintenance, name service, distributed synchronization, distributed lock, and distributed queue based on ZooKeeper.Copy the code

2. Basic Concepts of ZooKeeper

This section introduces some of the core concepts of ZooKeeper so it’s important to understand them in advance.

  1. The cluster character

A ZooKeeper cluster has only one Leader at a time, and the others are followers or observers.

The ZooKeeper configuration is simple. The ZooKeeper configuration file (zoo.cfg) is the same for each node except the myID file. The value of myID must be the {value} part of server.{value} in zoo.cfg.

Example of zoo. CFG configuration file

Run the zookeeper-server status command on the terminal of the ZooKeeper host to check whether the ZooKeeper role of the current node is Leader or Follower.Copy the code

By default, ZooKeeper has only Leader and Follower roles, but no Observer role. To use the Observer mode, add peerType= Observer to the configuration files of any node that you want to become an Observer and in the configuration files of all servers, The line of configuration appended to the server configured in Observer mode :observer

2. Assign read and write services to nodes

1. All the machines in the ZooKeeper cluster select a Leader through a Leader election process. The Leader server provides read and write services for clients. 2. Followers and Observers provide read services but do not provide write services. The only difference is that the Observer machine does not participate in the Leader election process, nor does it participate in the "half-write success" policy of write operations, which allows the Observer to improve read performance of the cluster without affecting write performance.Copy the code

3 . Session

Session is a client Session, so before we talk about client sessions, let's talk about client connections. In ZooKeeper, a client connection refers to a TCP long connection between the client and the ZooKeeper server. The default ZooKeeper external service port is 2181. When the ZooKeeper client is started, a TCP connection is established with the server. After the first connection is established, the client session life cycle starts. It can also send requests to and receive responses from the ZooKeeper server, as well as receive Watch event notifications from the server through this connection. The Session SessionTimeout value is used to set the timeout period of a client Session. If the client is disconnected due to server pressure, network failure, or client disconnection, the session created before can be reconnected to any server in the cluster within the time specified by SessionTimeout.Copy the code

4. Data node

The structure of ZooKeeper is actually a tree structure, in which the leader is the root node and the other nodes are the follow nodes. Each node retains its own content. Zookeeper nodes are classified into two types: persistent nodes and temporary nodes - Persistent nodes: Once a persistent node is created in the tree structure, it will remain in ZooKeeper until the tree node is removed. - Temporary node: The life cycle of a temporary node is bound to the client session. Once the client session fails, all temporary nodes created by the client are removed.Copy the code

5. Status information

In addition to storing data content, each node also stores some state information of the node itself. In ZooKeeper, the version attribute is used to implement "write check" (atomicity of distributed data) in the optimistic locking mechanism.Copy the code

6. Transaction operation

In ZooKeeper, an operation that changes the state of the ZooKeeper server is called a transaction. The operations include creating and deleting data nodes, updating data content, and creating and invalidating client sessions. For each transaction request, ZooKeeper assigns a globally unique transaction ID, represented by an ZXID, usually a 64-bit number. Each ZXID corresponds to an update operation. From these ZXids, you can indirectly identify the global order in which ZooKeeper processes these transaction operation requests.Copy the code

7.Watcher(Event Listener)

ZooKeeper is an important feature in ZooKeeper. ZooKeeper allows users to register some Watcher on a specified node, and when certain events are triggered, the ZooKeeper server notifies interested clients of the events. This mechanism is an important feature of ZooKeeper to implement distributed coordination service.Copy the code

3. Typical ZooKeeper application scenarios

ZooKeeper is a highly available distributed data management and coordination framework. Based on the implementation of ZAB algorithm, the framework can guarantee the consistency of data in distributed environment. It is also based on this feature that Makes ZooKeeper a powerful tool to solve distributed consistency problems.Copy the code

1. Data Publishing and subscription (Configuration Center)

As the name implies, publishers publish data to ZooKeeper nodes for subscribers to subscribe data. In this way, data can be dynamically obtained and centralized management and dynamic update of configuration information can be realized. For: data volume is usually small. Data content changes dynamically at run time. All machines in the cluster share the same configuration. This global configuration information can then be published to ZooKeeper, allowing clients (clustered machines) to subscribe to the message. Publish/subscribe systems generally have two design patterns, namely Push and Pull patterns. - In push mode, the server proactively sends data updates to all subscribed clients. - In pull mode, the client proactively sends requests to obtain the latest data. Usually, the clients use scheduled polling and pull mode. Once the data of the node is changed, the server will send the Watcher event notification to the corresponding client. After receiving the message notification, the client needs to take the initiative to obtain the latest data from the serverCopy the code

2. Naming service

Naming services are also a common scenario in distributed systems. In distributed system, by using naming service, client application can obtain the address, provider and other information of resource or service according to the specified name. Named entities can usually be machines in a cluster, services provided, remote objects, and so on -- we can call them all names. A common one is the list of service addresses in some distributed service frameworks, such as RPC. By creating sequential nodes in ZooKeepr, it is easy to create a globally unique path that can be used as a name. ZooKeeper's naming service generates globally unique ids.Copy the code

3. Distributed coordination services/notifications

ZooKeeper has a special Watcher registration and asynchronous notification mechanism, which can well realize the notification and coordination between different machines and even different systems in a distributed environment, so as to realize the real-time processing of data changes. If the machine node changes, all subscribed clients will receive the corresponding Watcher notification and act accordingly. The distributed coordination/notification function of ZooKeeper is a universal communication method between distributed systems.Copy the code

4. Master election

The Master election is the most typical application scenario of ZooKeeper. For example, Active NameNode election in HDFS, Active ResourceManager election in YARN, and Active HMaster election in HBase.

In general, we can choose the primary key feature in the common relational database to implement: Each machine that wants to be Master inserts a record with the same primary key ID into the database, and the database checks for primary key conflicts for us. That is, only one machine inserts successfully -- then we consider the client machine that successfully inserts into the database to be Master. Relying on the primary key feature of a relational database does a good job of ensuring that a single Master is elected in the cluster. But what if the currently elected Master dies? Who's going to tell me that Master died? Obviously, the relational database cannot notify us of this event. But ZooKeeper can do it! Using the strong consistency of ZooKeepr, the global uniqueness can be guaranteed for the creation of nodes in the case of distributed high concurrency, that is, ZooKeeper will ensure that the client cannot create an existing data unit node. That is, if there are multiple client requests to create the same temporary node at the same time, only one client request will succeed. With this feature, it is easy to do Master elections in a distributed environment. The machine on which the client successfully created the node becomes the Master. Meanwhile, any client that fails to create this node will register a Watcher of child node changes on this node to monitor whether the current Master machine is alive or not. If the current Master machine is found to have hung, the other clients will re-elect the Master. This implements the dynamic election of the Master.Copy the code

5. Distributed lock

Distributed lock is a way to control the synchronous access to shared resources between distributed systems. Distributed lock is classified into exclusive lock and shared lock

  • Exclusive locking How does ZooKeeper implement exclusive locking?

    • Defining locks A machine node on ZooKeeper can represent a lock
    • A node on ZooKeeper is regarded as a lock, and the lock is obtained by creating temporary nodes. ZooKeeper ensures that only one client can be successfully created, and the client is considered to have acquired the lock. At the same time, all clients that have not obtained the lock need to register a Watcher on the /exclusive_lock node to listen for changes on the lock node in real time.

    • Since the lock is a temporary node, there are two ways to release the lock

      • If the client machine that currently holds the lock goes down or restarts, the temporary node is deleted and the lock is released
      • After the service logic is executed, the client automatically deletes the created temporary node to release the lock.

In any case, ZooKeeper notifies all clients that have registered node changes on the /exclusive_lock node that Watcher listens for. Upon receiving the notification, these clients re-initiate distributed lock acquisition, i.e., repeat the lock acquisition process.

  • A Shared lock

    Shared locks are easy to implement within the same process, but difficult to implement across processes or between different servers. Zookeeper makes this easy by creating an EPHEMERAL_SEQUENTIAL directory node that requires the Server to acquire the lock. Then we call getChildren to see if the smallest directory node in the current directory node list is the one we created. If it is, then it has the lock. If not, it calls the exists(String Path, Boolean watch) method and monitors the directory node list on Zookeeper until it creates the smallest directory node in the list to obtain the lock. Releasing the lock is simple. Just delete the directory node that it created itself earlier.

conclusion

This article introduces the basic knowledge of Zookeeper and introduces several typical application scenarios. These are the basic functions of Zookeeper. The most important is that Zoopkeeper provides a good distributed cluster management mechanism, which is the hierarchical directory tree-based data structure and effective management of nodes in the tree, so that a variety of distributed data management models can be designed. Not just the common application scenarios mentioned aboveCopy the code