1. Basic Concepts of ZooKeeper

1. What is ZooKeeper?

Zookeeper official website: zookeeper.apache.org/

Zookeeper’s website documents address: zookeeper.apache.org/doc/trunk/i…

ZooKeeper, a subproject of Hadoop, is a reliable coordination system for large distributed systems. It provides the following functions: configuration maintenance, name service, distributed synchronization, group service, etc. Its goal is to encapsulate complex and error-prone key services, and provide users with easy-to-use interfaces and systems with high performance and stable functions.

One of the most common usage scenarios of Zookeeper is to act as a registry of service producers and service consumers. Service producers register their services with the Zookeeper center, and service consumers first search for the service in Zookeeper when invoking the service. After obtaining the detailed information of service producers, To invoke the content and data of the service producer, a simple example is shown below:

2. ZooKeeper design objectives:

ZooKeeper allows distributed processes to coordinate with each other through a shared hierarchical namespace, similar to a standard file system. Namespaces are made up of data registers in ZooKeeper – called ZNodes, which are similar to files and directories. Unlike typical file systems designed for storage, ZooKeeper data is kept in memory, which means ZooKeeper can achieve high throughput and low latency.

The Zookeeper namespace is shown as follows:

It is easy to find a specific service through the data model of tree graph structure.

3. Main Features of ZooKeeper:

1) Final consistency: Displaying the same view for the client is the most important performance of ZooKeeper. 2) Reliability: If a message is accepted by one server, it will be accepted by all servers. 3) Real-time: ZooKeeper cannot guarantee that the two clients get the newly updated data at the same time. If the latest data is needed, sync() interface should be called before reading the data. 4) Waiting has nothing to do with (waitFree: Slow or invalid clients do not interfere with requests from fast clients. 5) Atomicity: update can only succeed or fail, there is no other state in between. 6) Sequential: For all servers, the same message is published in the same order.Copy the code

2. Basic principles of ZooKeeper

1. ZooKeeper system architecture

First, take a look at the architecture diagram of ZooKeeper.

In the ZooKeeper architecture diagram, we need to know and master the following:

(1) ZooKeeper is divided into *** Server *** and *** Client ***. The Client can connect to any Server of the entire ZooKeeper service (unless the leaderServes parameter is explicitly set). The leader is not allowed to accept client connections.

(2) The client uses and maintains a TCP connection through which it sends requests, receives responses, obtains observed events, and sends heartbeats. If this TCP connection breaks, the client will automatically attempt to connect to another ZooKeeper server. When a client connects to the ZooKeeper service for the first time, the ZooKeeper server that accepts the connection establishes a session for the client. When the client connects to another server, the session is re-established by the new server.

(3) Each Server in the figure above represents a machine that installs Zookeeper service, that is, the whole cluster that provides Zookeeper service (or consists of pseudo-clusters);

(4) Servers that constitute the ZooKeeper service must know each other. They maintain an in-memory state image, as well as transaction logs and snapshots in persistent storage, and ZooKeeper services are available as long as most servers are available;

(5) When ZooKeeper is started, a leader will be elected from the instance. The leader is responsible for data update and other operations. A successful update operation is indicated only when most servers successfully modify data in memory. Each Server stores a copy of data in memory.

(6) Zookeeper can be replicated in a cluster. Data consistency is maintained between clusters using Zab protocol (Zookeeper Atomic Broadcast).

(7) Zab protocol consists of two stages: leader election stage and Atomic Brodcast stage.

  • A) THE cluster elects a leader and the other machines are called followers. All write operations are sent to the leader and all updates are told to the followers via brodcast.
  • B) When the leader crashes or loses most followers, a new leader needs to be elected to restore all servers to a correct state.
  • C) When the leader is elected and most servers have completed the synchronization with the leader status, the process of Leadder election will end and the Atomic Brodcast process will be entered.
  • D) Atomic Brodcast synchronizes the information between the leader and followers to ensure that the leader and followers have the same system state.

2. The Zookeeper role

After the Zookeeper server cluster environment is started, multiple Zookeeper servers elect a Leader before running. Before the leader is elected, all servers are required to participate in the vote on an equal basis without distinguishing their roles (except observers).

After the main selection process is complete, the following roles exist:

Think about:

1、为什么需要server?

Ensure high availability and consistency for ZooKeeper. ② In order to support more clients, more servers need to be added. (3) The increase of followers will lead to the increase of voting delay and affect the performance.Copy the code

2. What role does ObServer play in Zookeeper?

① The ObServer does not participate in the voting process and only synchronizes the status of the leader. Observers accept client connections and forward write requests to the leader node. ③ Add more ObServer nodes to improve scalability without affecting throughput.Copy the code

3. Why is the number of Servers in Zookeeper usually odd?

We know that the Leader election algorithm in Zookeeper uses Paxos protocol. The core idea of Paxos is that task data is successfully written when most servers are successfully written. (1) If there are three servers, a maximum of one Server can fail. ② If there are four servers, a maximum of one Server is allowed to fail. The reliability of three or four ZooKeeper servers is the same. Therefore, select an odd number of ZooKeeper servers. In this case, select three servers.Copy the code

3. Data writing process of ZooKeeper

The following shows the flowchart for writing data to ZooKeeper.

The data writing process of ZooKeeper is as follows:

  • A). For example, the Client sends a write request to Server1 of ZooKeeper.

  • B) If Server1 is not the Leader, Server1 will forward the received request to the Leader, because one of the ZooKeeper servers is the Leader. The Leader broadcasts the write request to each Server, such as Server1 and Server2, and notifies the Leader when each Server writes successfully.

  • C) When the Leader receives that most Server data has been written successfully, the data has been written successfully. If there are three nodes, two of them are considered successful. After the write is successful, the Leader tells Server1 that the data write is successful.

  • D) Server1 will further inform Client that the data write is successful, and then the whole write operation is considered successful.

4. ZooKeeper component

ZooKeeper Component Displays the advanced components of the ZooKeeper service. In addition to the request handler, each server that makes up the ZooKeeper service makes its own copy of each component.

The Replicated Database is an in-memory Database that contains the entire data tree. Update operations are logged to disk for recoverability, and writes are serialized to disk before being placed into the in-memory database.

Each ZooKeeper server serves clients. The client connects to a server to submit an iRequest. Read requests from the local replica service of each server database. Requests that change the state of the service (write requests) are handled by the protocol.

As part of the protocol protocol, all write requests from clients are forwarded to a single server, called the leader. The remaining ZooKeeper servers (called Followers) receive message proposals from the leader and agree to the message delivery. The message layer is responsible for replacing the leader in case of failure and synchronizing followers with the leader.

Summary of ZooKeeper application scenarios

1. Unified naming service

The naming structure of the unified Naming service is as follows:

2. Configuration management

The configuration management structure is as follows:

1. In distributed environment, configuration file management and synchronization is a common problem. A) In a cluster, the configuration information of all nodes is consistent, such as a Hadoop cluster. B) The modified configuration file can be quickly synchronized to each node. 2. Configuration management can be implemented by ZooKeeper. A) You can write the configuration information to a Znode on ZooKeeper. B) Each node listens to this Znode. C) Once data in Znode is modified, ZooKeeper will notify each node.

3. Cluster management

The cluster management structure diagram is as follows:

1. In distributed environment, it is necessary to master the status of each node in real time. A) Some adjustments can be made according to the real-time status of nodes. 2. It can be implemented by ZooKeeper. A) The node information can be written to a Znode on ZooKeeper. B) Monitor the Znode to obtain its real-time status changes. 3. Typical application a) Monitoring and voting Master status in HBase.

4. Distributed notification and coordination

1. In distributed environments, there is often a service that needs to know the state of the child services it manages. A) NameNode needs to know the status of each Datanode. B) JobTracker needs to know the status of each TaskTracker. 2. The heartbeat detection mechanism can be implemented using ZooKeeper.

3. Information push can be implemented by ZooKeeper, which is equivalent to a publish/subscribe system.

5. Distributed locks

Different services on different nodes, which may require sequential access to some resources, require a distributed lock.

Distributed locks have the following features:

1. ZooKeeper is strongly consistent. For example, each node runs a ZooKeeper client, and they create the same Znode at the same time, but only one client is successfully created. 2, to achieve lock exclusivity. The client that successfully created Znode gets the lock. The other clients have to wait. After the current client runs out of the lock, the Znode will be deleted, and other clients will try to create zNodes to obtain the distributed lock. 3. Control the timing of the lock. Each client creates a temporary Znode under a Znode. This type must be createmode.ephemeral_sequential so that the Znode has global access timing.Copy the code

6. Distributed queues

There are two types of distributed queues:

1. A queue is available only when all the members of the queue are together. Otherwise, the queue is waiting for all the members to arrive. A) A job consists of multiple tasks. A job is executed only after all tasks are completed. B) Create a /job directory for the job. In this directory, create a temporary Znode for each completed task. Once the number of temporary nodes reaches the total number of tasks, the job is completed. 2. Queue in and out of queue in FIFO mode, such as the realization of producer and consumer model.

3. Install and deploy ZooKeeper

Zookeeper can be installed in the following modes:

  • Standalone mode: single server;
  • Cluster mode: Multiple machines and multiple servers form a cluster.
  • Pseudo-cluster mode: Multiple servers are deployed in a single machine to form a pseudo-cluster.

Environment: Cent OS 7.0

1. Single-machine mode

(1) according to the need to create the directory, such as my directory is: / home/xuliugen/Desktop/zookeeper – install

(2) into the directory, using wget download zookeeper, download address: mirrors.tuna.tsinghua.edu.cn/apache/zook…

Other version download address: mirrors.tuna.tsinghua.edu.cn/apache/zook…

Complete as follows:

(3) Run the tar -xvf zookeeper-3.4.6.tar.gz command to decompress the file.

(4) Create the Zookeeper configuration file.

In the conf file in the Zookeeper installation directory, the default value is:

CFG zoo. CFG command to copy a zoo. CFG file, because Zookeeper uses the zoo. CFG configuration file by default during Zookeeper restart.

(5) Modify the configuration file as required:

Generally, the default configuration file can be used to demonstrate startup. The configuration file is as follows:

# Heartbeat detection time of the Zookeeper service, in ms
tickTime=2000

# initialization time to vote for a new leader
initLimit=10

# Leader and follower heartbeat detection Max tolerance time, response exceeds syncLimit*tickTime,
The follower is removed from the server list if the leader thinks the follower is dead
syncLimit=5

# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/tmp/zookeeper

# the port at which the clients will connect
clientPort=2181

# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60

# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3

# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

Copy the code

Tip:

Zookeeper’s official documentation gives a small lesson about performance tuning, which is that there are several other configuration parameters that can greatly improve performance:

To achieve low latency when updating, it is important to have a dedicated transaction log directory. By default, transaction logs are placed in the same directory as data snapshots and myID files. The dataLogDir parameter indicates the different directories used for transaction logging.Copy the code

This means that it is best to separate the dependency directory from the log directory to improve the efficiency of reading and updating data.

(6) Start Zookeeper

In the bin directory of the Zookeeper installation directory:

To start the service, run the./ zkserver. sh start command.

Run the./ zkcli. sh command to go to the CLI.

To this single machine mode installation is over!

Cluster mode 3. Pseudo cluster mode

The configuration of cluster mode and pseudo cluster mode, there is a lot of content on the Internet, which will not be demonstrated here, please go to check:

www.open-open.com/lib/view/op…

Appendix:

Zoo. CFG Configuration parameters:

Reference article:

1, the website of large distributed architecture – design and practice Chen Kangxian – the “2,” the Zookeeper – 3.3.5 source code analysis Shao-wei liu “3, m.blog.csdn.net/article/det… 4, mt.sohu.com/20160527/n4… 5, www.open-open.com/lib/view/op…

Java Backend technology (ID: JavaITWork), learn Java with 200,000 people!

Java Backend Technology Focus on Java related technologies: SSM, Spring family bucket, micro services, MySQL, MyCat, cluster, distributed, middleware, Linux, network, multi-threading, occasionally talk about the operation Jenkins, Nexus, Docker, ELK, occasionally share some technical dry goods, committed to Java full stack development!

Java Backend Technology (ID: JavaITWork)1024, you can get it for free! Includes SSM, Spring family bucket, microservices, MySQL, MyCat, cluster, distributed, middleware, Linux, network, multi-threading, Jenkins, Nexus, Docker, ELK and so on free learning video, continue to update!