What is a ZooKeeper

ZooKeeper is a top-level project of Apache that provides efficient and highly available distributed coordination services for distributed applications, providing distributed infrastructure services such as data publishing/subscription, load balancing, naming services, distributed coordination/notification, and distributed locks. ZooKeeper is widely used in large distributed systems such as Hadoop, HBase, Kafka, and Dubbo due to its convenient usage, excellent performance, and good stability. Compiled a Java interview treasure book complete PDF

Zookeeper supports three operating modes: single-machine deployment, pseudo-cluster deployment, and cluster deployment.

  • Stand-alone mode: This mode is generally suitable for development and test environments, on the one hand we do not have so many machine resources, and on the other hand, the normal development and debugging does not need excellent stability.

  • Cluster mode: A ZooKeeper cluster usually consists of a group of machines. Generally, more than three machines can form an available ZooKeeper cluster. Each machine that makes up the ZooKeeper cluster maintains the current server state in memory, and each machine communicates with each other.

  • Pseudo cluster mode: This is a special cluster mode where all the servers in the cluster are deployed on one machine. ZooKeeper allows you to start multiple Instances of the ZooKeeper service on a single machine by starting different ports on the same machine. In this case, you can start multiple instances of the ZooKeeper service on the same machine by starting different ports.

ZooKeeper

Roles in Zookeeper:

  • Leader: responsible for initiating and deciding votes and updating system status.

  • Followers: receive client requests and return results to the client. They vote in the master election process.

  • Observer: Can accept client connections and forward write requests to the leader, but the Observer does not participate in the process of voting, only to expand the system and improve read speed.

Data model for Zookeeper

  • Hierarchical directory structure, named according to the general file system specification, similar to Linux.

  • Each node is called a Znode in Zookeeper and has a unique path identifier.

  • A Znode can contain data and child nodes, but EPHEMERAL nodes cannot have child nodes.

  • Data in a Znode can have multiple versions. For example, data in a path can have multiple versions. In this case, data in this path needs to be queried with the version.

  • Client applications can set up monitors on nodes.

  • The node does not support partial read/write, but full read/write at one time.

Node features of ZooKeeper

ZooKeeper nodes are lifecycle, depending on the type of node. In ZooKeeper, nodes can be classified as PERSISTENT, EPHEMERAL, SEQUENTIAL, and out-of-order (by default, out-of-order) according to their duration.

Once a persistent node is created, it remains in Zookeeper until it is actively removed (it does not disappear because the client session that created the node fails).

Application scenarios of Zookeeper

ZooKeeper is a highly available distributed data management and system coordination framework. Based on the implementation of Paxos algorithm, the framework ensures the strong consistency of data in the distributed environment, and it is based on this feature that Makes ZooKeeper solve many distributed problems.

It is worth noting that ZooKeeper is not inherently designed for these application scenarios. Instead, it is a typical usage method developed by many developers based on the characteristics of its framework and a series of API interfaces (or primitives) provided by ZooKeeper.

Data Publishing and subscription (Configuration Center)

Publish and subscribe model, also known as the so-called configuration center, as the name implies, publishers publish data to the ZooKeeper node for subscribers to dynamically obtain data, so as to achieve centralized management and dynamic update of configuration information. For example, global configuration information, service address list of service-oriented service framework is very suitable for use.

Some configuration information used in applications is centrally managed on ZooKeeper. This scenario is usually like this: the application actively obtains a configuration at startup and registers a Watcher on the node. In this way, every configuration update will be notified to the subscribing client in real time, and the latest configuration information will never be obtained.

In the distributed search service, the meta information of the index and the node state of the server cluster machine are stored in some specified nodes of ZooKeeper for each client to subscribe to.

Distributed log collection system

The core of the system is to collect logs distributed across different machines. Collectors usually allocate collection task units based on applications. Therefore, a node P with the application name as path needs to be created on ZooKeeper, and all machine IP addresses of this application are registered on node P as child nodes. This allows the collector to be notified in real time to adjust task assignments as the machine changes.

Some information in the system needs to be acquired dynamically, and there will be a manual to modify this information. This is usually done by exposing an interface, such as a JMX interface, to get some runtime information. After ZooKeeper is introduced, you do not need to implement a solution by yourself. You only need to store the information on the specified ZooKeeper node.

Note: In the application scenario mentioned above, there is a default premise — a scenario where the amount of data is small, but the data is likely to be updated quickly.

Load balancing

Load balancing refers to soft load balancing. In a distributed environment, to ensure high availability, multiple applications or service providers are deployed to achieve peer services. The consumer needs to choose one of these peer servers to execute the relevant business logic, and the typical one is the producer of message middleware, consumer load balancing.

Naming Service

Naming services are also a common scenario in distributed systems. In distributed system, by using naming service, client application can obtain the address, provider and other information of resource or service according to the specified name. Named entities can be machines in a cluster, addresses of services provided, remote objects, and so on — all of which we can collectively call names. A common one is a list of service addresses in some distributed service frameworks. By calling the node creation API provided by ZooKeeper, it is easy to create a globally unique path that can be used as a name.

In Dubbo, the open source distributed service framework of Alibaba Group, ZooKeeper is used as its naming service to maintain the global service address list. In the implementation of Dubbo:

  • When the service provider starts, it writes its URL address to the specified node /dubbo/${serviceName}/providers directory on ZooKeeper to publish the service.

  • /dubbo/ serviceName/providers /{serviceName}/providers /dubbo/{serviceName}/providers /dubbo/{serviceName}/providers /dubbo/{serviceName}/providers /dubbo/{serviceName}/providers /dubbo/{serviceName}/providers /dubbo/{serviceName}/providers / And to dubbo/serviceName/will provider URL address in the directory, and to dubbo / {serviceName} / consumers directory into your URL address.

Note: All addresses registered with ZooKeeper are temporary nodes, ensuring that service providers and consumers are automatically aware of resource changes.

In addition, Dubbo also monitors service granularity. By subscribing to the /dubbo/${serviceName} directory for all providers and consumers.

Distributed notification/coordination

ZooKeeper has a unique Watcher registration and asynchronous notification mechanism, which can well realize notification and coordination between different systems in a distributed environment, and realize real-time processing of data changes. Generally, different systems register the same Znode on ZooKeeper and monitor the changes of Znode (including the content of Znode itself and its child nodes). When one system updates Znode, the other system can receive notification and take corresponding actions.

Another heartbeat detection mechanism: The detection system is not directly associated with the detected system, but is associated with a node on ZooKeeper, greatly reducing system coupling.

Another system scheduling mode: a system consists of a console and a push system. The responsibility of the console is to control the push system to carry out corresponding push work. Some of the actions that administrators make on the console actually change the state of some of ZooKeeper’s nodes, and ZooKeeper notifies clients that have signed up for Watcher, the push system. So, make corresponding push task.

Another mode of reporting work: something like a task distribution system. After the subtask starts, register a temporary node with ZooKeeper and report its progress periodically (write the progress back to the temporary node). This allows the task manager to know the progress of the task in real time.

A distributed lock

Distributed locking mainly benefits from ZooKeeper to ensure strong data consistency. Lock services can be divided into two categories: one is to maintain exclusivity, and the other is to control timing.

The so-called hold exclusive is that all the clients that attempt to acquire the lock, only one can succeed in obtaining the lock. A common approach is to treat a Znode on ZooKeeper as a lock, and create a Znode. All clients create the /distribute_lock node, and the client that is successfully created owns the lock.

The control timing, that is, the client that all views acquire the lock, is eventually scheduled to be executed, but there is a global timing. This is similar, except that the /distribute_lock already exists, under which the client creates a temporary ordered node (this can be specified by the node’s property control: createmode.ephemeral_sequential). The parent node of ZooKeeper (/distribute_lock) maintains a sequence to ensure that the child nodes are created sequentially, thus forming the global sequence of each client.

  1. The sub-node names of the same node cannot be the same. If a Znode is created on a node, the Znode is locked successfully. A listener is registered to listen on the Znode, and whenever the Znode is deleted, other clients are notified to lock it.

  2. Create temporary sequential nodes: a node is created under a node, and a node is created for each request. Since it is sequential, the one with the smallest sequence number obtains the lock, and when the lock is released, the next sequence number is notified of the lock.

Distributed queue

In terms of queues, there are simply two kinds: one is the conventional first-in, first-out queue, and the other is to wait for the queue members to gather up before executing in order. For the first queue, the basic principle is the same as for the control sequence scenario in the distributed lock service described above, which will not be described here.

The second queue is actually an enhancement of the FIFO queue. It is usually possible to pre-create a /queue/num node under the /queue Znode and assign n (or simply assign n to /queue) to indicate the queue size. Each time a queue member joins, it determines whether the queue size has been reached and whether it is ready to execute.

A typical scenario for this usage is A distributed environment where A large Task, Task A, needs to be completed (or conditioned) by A number of sub-tasks. At this point, as soon as one of the sub-tasks is complete, go to /taskList and create your own temporary sequential node. When /taskList finds that it has a specified number of children, it can proceed to the next step in order.

Build the cluster using dokcer-compose

We have introduced so many application scenarios about ZooKeeper, so next we will first learn how to build a ZooKeeper cluster and then practice the above application scenarios.

The directory structure of the file is as follows:

├ ─ ─ docker - compose. YmlCopy the code

Write the docker-comemess. yml file

Docker-comemess. yml file contains the following contents:

Version: '3.4' services: zoo1: image: Zookeeper restart: always hostname: zoo1 ports: -2181:2181 Environment: ZOO_SERVERS ZOO_MY_ID: 1: server. 1 = 0.0.0.0:2888-3888; 2181 server.2=zoo2:2888:3888; 2181 server.3=zoo3:2888:3888; 2181 zoo2: image: zookeeper restart: always hostname: zoo2 ports: - 2182:2181 environment: ZOO_MY_ID: 2 ZOO_SERVERS: server.1=zoo1:2888:3888; 2181 server. 2 = 0.0.0.0:2888-3888. 2181 server.3=zoo3:2888:3888; 2181 zoo3: image: zookeeper restart: always hostname: zoo3 ports: - 2183:2181 environment: ZOO_MY_ID: 3 ZOO_SERVERS: server.1=zoo1:2888:3888; 2181 server.2=zoo2:2888:3888; 2181 server. 3 = 0.0.0.0:2888-3888; 2181Copy the code

In this configuration file, Docker runs three Zookeeper images, binding local ports 2181, 2182, and 2183 respectively to port 2181 of the corresponding container through the ports field.

ZOO_MY_ID and ZOO_SERVERS are two environment variables required to set up a Zookeeper cluster. ZOO_MY_ID Specifies the service ID. The value is an integer ranging from 1 to 255 and must be unique in the cluster. ZOO_SERVERS is a list of hosts in the cluster.

Docker-compose up: docker-compose up: docker-compose up: docker-compose up: docker-compose.

Connect the ZooKeeper

After starting up the cluster, we can connect to ZooKeeper for node related operations.

  1. First, you need to download ZooKeeper.

  2. Unzip it.

  3. Go to the conf/ directory and change zoo_sample. CFG to zoo.cfg.

Configuration File Description

Milliseconds of each tick # tickTime: Number of CS communication heartbeats # Interval between Zookeeper servers or between clients and servers to maintain heartbeats, i.e., one heartbeat is sent per tickTime. TickTime is measured in milliseconds. TickTime =2000 # The number of ticks that The initial # synchronization phase can take # initLimit: LF Initial communication duration # Maximum number of heartbeats (ticktimes) that can be tolerated during the initial connection between the follower server (F) and the leader server (L) in the cluster. InitLimit =5 # The number of ticks that can pass between # sending a request and getting an acknowledgement # syncLimit: LF Synchronous communication duration # Maximum number of heartbeats (ticktimes) that can be tolerated between the follower server and the leader server in the cluster. syncLimit=2 # the directory where the snapshot is stored. # do not use /tmp for storage, / TMP here is just # example sakes. # dataDir: directory where Zookeeper saves data. By default, Zookeeper saves log files that write data in this directory. DataDir =/data/soft/zookeeper-3.4.12/data # dataLogDir: Directory for storing log files. DataLogDir =/data/soft/zookeeper-3.4.12/logs # the port at which the clients will connect # clientPort: Client connection port # Port used by the client to connect to the Zookeeper server. Zookeeper listens on this port to receive access requests from the client. clientPort=2181 # the maximum number of client connections. # increase this if you need to handle more clients #maxClientCnxns=60 # # Be sure to read the maintenance section of the # administrator guide before turning on autopurge.  # # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance # # The number of snapshots to retain in  dataDir #autopurge.snapRetainCount=3 # Purge task interval in hours # Set to "0" to disable auto purge feature PurgeInterval =1 # Server name and address: cluster information (server number, server address, LF communication port, election port) # N=YYY:A:B # Where N indicates the server number, YYY indicates the IP address of the server, and A indicates the LF communication port, which indicates the port through which the server exchanges information with the leader in the cluster. B is the election port, indicating the port used by the servers to communicate with each other when the new leader is elected. (When the leader fails, other servers communicate with each other to select a new leader.) In general, port A is the same for each server in the cluster, as is port B for each server. However, in A pseudo cluster, the IP addresses of ports A and B are the same.Copy the code

You can use the default configuration without modifying zoo. CFG. Run the./ zkcli. sh -server 127.0.0.1:2181 command in the bin/ directory after decompression to connect to the server.

Welcome to ZooKeeper! 2020-06-01 15:03:52,512 [myID :] -info [main-sendThread (localhost:2181):ClientCnxn$SendThread@1025] -opening socket Connection to server localhost / 127.0.0.1:2181 Will not attempt to authenticate using SASL (unknown error) JLine support Is enabled 2020-06-01 15:03:552,576 [myID :] -info [main-sendThread (localhost:2181):ClientCnxn$SendThread@879] -socket The connection established to localhost / 127.0.0.1:2181, Initiating session 2020-06-01 15:03:552,599 [myID :] -info [main-sendThread (localhost:2181):ClientCnxn$SendThread@1299] - The Session establishment complete on server localhost / 127.0.0.1:2181, sessionid = 0 x100001140080000, negotiated timeout = 30000 WATCHER:: WatchedEvent state:SyncConnected type:None path:null [zk: 127.0.0.1:2181 (CONNECTED) 0]Copy the code

You can then view the node using the command:

  • Run the ls command to view the contents in ZooKeeper. Command: ls /

    [zk: 127.0.0.1:2181(CONNECTED) 10] ls /

  • Creates a new ZNode node zk and its associated string. Command: create /zk myData

    [zk: 127.0.0.1:2181(CONNECTED) 11] create /zk myData

Created /zk [zk: 127.0.0.1:2181(CONNECTED) 12] ls / [zk, zookeeper] [zk: 127.0.0.1:2181(CONNECTED) 13]

  • Obtain the ZK of the Znode. Run the get /zk command

    [zk: 127.0.0.1:2181(CONNECTED) 13] get /zk myData cZxid = 0x400000008 ctime = Mon Jun 01 15:07:50 CST 2020 mZxid = 0x400000008 mtime = Mon Jun 01 15:07:50 CST 2020 pZxid = 0x400000008 cversion = 0 dataVersion = 0 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 6 numChildren = 0

  • Example Delete node ZK. Command: delete /zk compiled a complete PDF of the Java Interview handbook

    [zk: 127.0.0.1:2181(CONNECTED) 14] delete /zk [zk: 127.0.0.1:2181(CONNECTED) 15] ls / [zookeeper]