What is ZooKeeper?

Official note:

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications. Each time they are implemented there is a lot of work that goes into fixing the bugs and race conditions that are inevitable. Because of the difficulty of implementing these kinds of services, applications initially usually skimp on them, which make them brittle in the presence of change and difficult to manage. Even when done correctly, different implementations of these services lead to management complexity when the applications are deployed.
Copy the code

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these types of services are used by distributed applications in one form or another. Each time they are implemented, a lot of work goes into fixing the inevitable errors and race conditions. Because these services are difficult to implement, applications often ignore them initially, making them vulnerable and difficult to manage when changes occur. Even when done correctly, different implementations of these services can cause administrative complexity when deploying the application.

A third-party coordinator, used by hadoop, Hive, Pig, and other distributed applications, provides configuration management, name services, distributed locks, and cluster management.

Other explanations:

Zookeeper understands this from a design pattern perspective: Zookeeper is a distributed service management framework designed based on the observer mode. It is responsible for storing and managing the data that everyone cares about, and then accepting the registration of observers. Once the status of the data changes, Zookeeper will be responsible for notifying those observers who have registered with Zookeeper to respond accordingly.

For example: Stations in each region send their remaining ticket information to the 12306 general database. Users can then check certain tickets they are interested in through the 12306 official APP. When tickets at a station are sold out, 12306 will send the corresponding information to concerned users.

Zookeeper architecture:

  • Leader: the unique scheduler and handler of core transaction requests (write operations) of the ZooKeeper cluster, ensuring the sequence of cluster transaction processing. Dispatcher of services within a cluster. For create, setData, DELETE and other write requests, they need to be uniformly forwarded to the leader for processing. The leader needs to decide the number and execute the operation. This process is called a transaction.
  • Follower: processes non-transaction (read) requests from the client. Forwards transaction requests to the Leader. Participates in voting for the cluster Leader election. In addition, you can add a Follower role for a ZooKeeper cluster with a large number of visits.
  • The Observer: The role of observer observes the latest state changes of ZooKeeper cluster and synchronizes these states. For non-transaction requests, it can independently process them; for transaction requests, it will forward them to the Leader server for processing and only provides services without participating in any form of voting. It is usually used to improve the non-transaction processing capability of the cluster and increase the number of concurrent requests without affecting the transaction processing capability of the cluster

Zookeeper:

  1. Zookeeper: A cluster consisting of a Leader and multiple followers.
  2. The Zookeeper cluster can run normally as long as more than half of the nodes in the cluster are alive.
  3. Global data consistency: Each Server stores the same data copy. The data is consistent regardless of which Server the Client connects to.
  4. Sequentiality: includes global order and partial order. Global order means that if message A is published before message B on one server, message A is published before message B on all servers. Partial order means that if message B is published by the same sender after message A, a must be placed before MESSAGE B.
  5. Atomicity: A data update either succeeds or fails.
  6. Real-time: ZooKeeper ensures that the client will receive updates or server failures within an interval.

What is a distributed application?

Distributed applications can run on multiple systems in a network at a given time (simultaneously), coordinating them to accomplish specific tasks in a fast and efficient manner. It has two parts, the Server application and the Client application. The server application is essentially distributed and has a common interface so that clients can connect to any server in the cluster and get the same results. Client applications are tools for interacting with distributed applications.

What are the downsides of distributed applications?

  • Race condition – Two or more machines trying to perform a particular task, actually only need to be done by a single machine at any given time. For example, shared resources can only be modified by a single machine at any given time.
  • Deadlock – Two or more operations that are waiting for each other to release resources or complete certain conditions are unable to complete.
  • Inconsistent – Distributed applications have many clients. How do you ensure that the data used by the clients is consistent?

2. Application scenarios

The following describes the application scenarios of ZooKeeper to solve problems encountered in distributed applications.

Unified Naming Service

In a distributed environment, applications and programs are often named in a unified way for easy identification

IP is not easy to remember, for example, the domain name is relatively easy to remember, for a website all users access to the same domain name, the domain name may correspond to many different servers, when user access domain can access the node of the IP address of the server without having to memory their concrete

Unified Configuration Management

  1. Configuration file synchronization is common in distributed environments.

    • The configuration information of all nodes in a cluster must be consistent, for example, Kafka cluster.
    • After modifying a configuration file, you need to quickly synchronize the modification to each node.

  2. Configuration file synchronization can be handled by ZooKeeper

    • Write configuration information to a ZNode.
    • Each client server listens on the ZNode.
    • When the content of the configuration file changes (zNode data changes), ZooKeeper notifies each client server.

Unified Cluster Management

  1. In distributed environment, it is necessary to know the status of each node in real time.

    • Some adjustments can be made based on the real-time status of nodes.

  2. ZooKeeper monitors node status changes in real time.

    • You can write node information to a ZNode on ZooKeeper. .
    • Listen on the ZNode for real-time status changes.

Soft load balancing: ZooKeeper records the number of requests received by each server. When new requests arrive, the server with the least number of requests is preferentially processed.

Dynamic server online and Offline: ZooKeeper monitors server online and offline in real time.

Election algorithm: Elects a leader of a node for coordination purposes.

Locking and synchronization services: Locking data while modifying it. Prevent data anomalies.

There are problems, there are needs, there are progress. Faced with the problems encountered by these distributed applications, the ZooKeeper framework provides a complete mechanism to overcome all the challenges.

  • Race conditions and deadlocks are handled using a fail-safe synchronization approach.

  • ZooKeeper uses atomic parsing to resolve data inconsistencies.

Three, basic knowledge

Hierarchical namespace

The following figure describes the tree structure of the ZooKeeper file system for memory representation. The ZooKeeper node is called a ZNode. Each Znode is identified by a name, separated by a sequence of paths (/), and each Znode can store 1 MB of data by default.

  • In the figure, there is first a ZNode separated by a “/”. Under the root directory, there are two logical namespaces, Config and workers.
  • The config namespace is used for centralized configuration management and the Workers namespace is used for naming.
  • Each ZNode can store up to 1MB of data in the Config namespace. This is similar to the UNIX file system, but in ZooKeeper the parent ZNode can also store data. The main purpose of this structure is to store synchronous data and describe the metadata of ZNode.

Each Znode consists of three parts:

  • Stat: Indicates status information, including the version and permission of the Znode.

    (1) czxID – the transaction zxID of the creation node. Each time you change the ZooKeeper status, you will receive a timestamp in the form of an ZXID, which is the ZooKeeper transaction ID. The transaction ID is the total order of all changes in ZooKeeper. Each modification has a unique zxID, and if ZXID1 is less than zxid2, then zxid1 occurs before zxid2.

    (2) The number of milliseconds ctime-zNode was created (since 1970)

    (3) mzxid – znode last updated transaction zxID

    (4) mtime-znode last modified milliseconds (since 1970)

    (5) pZxid-znode indicates the zxID of the last updated child node

    (6) cversion-znode child node change number, znode child node change times

    (7) DatAversion – Znode data change number

    (8) aclversion-znode ACL change number

    (9) ephemeralOwner- If it is a temporary node, this is the session ID of the ZNode owner. 0 if not a temporary node.

    (10) datalength-znode dataLength

    (11) Numchildren-znode Number of child nodes

  • Data: data associated with the Znode

  • Children: indicates the node under the Znode

The type of the Znode

Znodes are divided into persistent, sequential, and ephemeral nodes.

  • Persistent node – The persistent node exists even after the client that created this particular ZNode disconnects. By default, all ZNodes are persistent unless otherwise noted.

  • Temporary nodes – Temporary nodes are valid when the client is active. Temporary nodes are automatically deleted when the client disconnects from the ZooKeeper collection. Therefore, only temporary nodes are not allowed to have children. If the temporary node is removed, the next appropriate node fills its location. Temporary nodes play an important role in leader election.

  • Sequential nodes – Sequential nodes can be persistent or temporary. When a new ZNode is created as a sequential node, ZooKeeper sets the path of the ZNode by appending the 10-bit serial number to the original name. For example, if a ZNode with path/myApp is created as a sequential node, ZooKeeper changes the path to/MyApp0000000001 and sets the next sequence number to 0000000002. If two sequential nodes are created at the same time, ZooKeeper will not use the same number for each ZNode. Sequential nodes play an important role in locking and synchronization.

Sessions

Sessions are important for ZooKeeper operations. Requests in the session are executed in FIFO order. Once the client connects to the server, a session is established and a session ID is assigned to the client.

The client sends heartbeat at a specific interval to keep the session valid. If the ZooKeeper collection does not receive a heartbeat from the client for more than the period specified when the server is started (session timeout), it determines that the client has crashed.

Session timeouts are usually measured in milliseconds. When a session ends for any reason, temporary nodes created during that session are also deleted.

Watches

Monitoring is a simple mechanism for clients to be notified of changes in the ZooKeeper collection. The client can set the Watches while reading a particular ZNode. Watches will send notifications of any ZNode (client registry) changes to registered clients.

A Znode change is a modification of zNode-related data or a change in a Znode child. Activate the watches only once. If the client wants to notify again, it must do so through another read operation. When the connection session expires, the client will be disconnected from the server and the related watches will be deleted.

conclusion

Zookeeper is a coordination service that solves the problems faced by distributed applications. It can be used to solve the problems related to locks, resources, and updates, and the coordination process between all nodes is very simple.