Sorted out some Java architecture, interview materials (micro services, clusters, distributed, middleware, etc.), partners in need can pay attention to the public number [programmer internal point], no routine to get

  • Reeling off nine distributed ID generation methods, the interviewer gets a little confused

  • How about the score sheet? You can play him off like this

  • Rattle off six @Transactional annotation failure scenarios

  • Face recognition function based on Java

  • 9 kinds of distributed ID generation of the United States (Leaf) combat

nagging

This article mainly shares some basic concepts of ZooKeeper. Before getting into the topic formally, I would like to talk with you about my interview experience when I just started my career, which can be said to be honest and cute.

Interviewer: Have you ever tried ZooKeeper?

Me: Yes, the registration and discovery service for Dubbo

Interviewer: Do you know what Zookeeper is?

Me: Yeah, the registry

Interviewer: So how do you use ZooKeeper in your projects?

I: Just add a ZooKeeper service address to springBoot’s application.properties configuration file…

Why is zookeeper asked? Because it is written in my resume that I am proficient in using ZooKeeper, but the interviewer understands that “proficient” in using ZooKeeper is not as simple as knowing how to configure and not reporting errors in project startup. Therefore, it is necessary to have a comprehensive understanding of ZooKeeper.

First introduction to ZooKeeper?

As an open source subproject of Hadoop project, Zookeeper is a classic distributed data consistency solution. It is committed to providing distributed coordination service with high performance, high availability, and strict sequential access control for distributed applications.

1. Zookeeper data model

Zookeeper maintains a file system-like data structure, and each subdirectory (/ wechat, / wechat/official account) is called a Znode, or node. Like a file system, zNodes can be easily added to or deleted from a ZNode, and children can be added to or deleted from a ZNode. The difference is that in a file system, zNodes can store data (strictly, data must be stored, which is null by default).

Zookeeper is a directory node structure. When obtaining or creating a node, you must start with a slash (/). Otherwise, an error message is displayed when obtaining a node.

[zk: localhost:2181(CONNECTED) 13] get test
Command failed: java.lang.IllegalArgumentException: Path must start with / character
Copy the code

The root node name must be /XXX, and the root directory /XXX/CCC and /XXX/AAA must be included when creating child nodes.

For example, if you want to get the programmer’s internal point node below, you must concatenate the complete path get/wechat/public number/programmer’s internal point

Get/wechat/public number/programmer internal mattersCopy the code

Znode is used to store byte or KB data, up to 1MB (please note: The data amount of a node not only contains its own stored data, but also the names of all its child nodes are converted into Byte numbers. Therefore, the number of child nodes of ZNode is not infinite.) Although you can manually change the size of the node storage, it is generally not recommended to do so.

2. Znode attributes

A ZNode can store not only data, but also some other special properties. Next, let’s create a /test node to examine the meaning of its properties.

[zk: localhost:2181(CONNECTED) 6] get /test
456
cZxid = 0x59ac //
ctime = Mon Mar 30 15:20:08 CST 2020
mZxid = 0x59ad
mtime = Mon Mar 30 15:22:25 CST 2020
pZxid = 0x59ac
cversion = 0
dataVersion = 2
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 3
numChildren = 0  
Copy the code
Node properties annotations
cZxid The transaction Id when the data node was created
mZxid The latest transaction Id when the data node was modified
pZxid Transaction Id of the parent node of the current node
ctime Time when the data node was created
mtime Time when the data object was last modified
dataVersion Current node version (increasing by +1 with each change)
cversion Version number of a child node (the number of times that a child node is modified increases by +1 each time it is modified)
aclVersion Acl version of the current node (the acl permission of the node is changed, and the value increases by +1 each time the ACL permission is changed)
ephemeralOwner Temporary node identifier, if the current node is a temporary node, then store the sessionId (sessionId) of the creator, if not, then value =0
dataLength The length of data stored on the current node
numChildren Number of sub-nodes of the current node

We can see that a ZNode has many attributes, but the main attributes are ZXID, Version, and ACL.

Zxid:

A zNode status change will cause the node to receive a timestamp in the format of zxID. This timestamp is globally ordered. A new timestamp will be generated when a ZNode is created or updated. If the value of zxid1 is less than the value of zxid2, then zxid2 has changed after ZXid1. Each Znode has three ZXID attributes, cZxid (creation time of the node), mZxid (modification time of the node, independent of child nodes), and pZxid (last creation or modification time of the node or child nodes, independent of grandson nodes).

The zxID property is mainly applied to the ZooKeeper cluster, which will be described in detail later.

Version:

There are three version numbers in the znode attribute: datversion (dataversion number), cversion (child node version number), and aclversion (ACL permission version number of the node).

Znode data can have multiple versions. If there are multiple versions of data stored in a node, you need to query the node data with the version number. Every time we modify the data of zNode, the datSpanning version number of this node will increase. When the client requests the ZNode, both the node data and the version number are returned. In addition, when datAversion is -1, the version can be ignored for operation. When setting permissions for a node, the version number of aclVersion increases. ACL permission control is described below.

To verify this, we modify the data on the /test node to see what happens to datSpanning. We see that the DatSpanning property has changed to 3 and the version number has increased.

[zk: localhost:2181(CONNECTED) 10] set /test 8888
cZxid = 0x59ac
ctime = Mon Mar 30 15:20:08 CST 2020
mZxid = 0x59b6
mtime = Mon Mar 30 16:58:08 CST 2020
pZxid = 0x59ac
cversion = 0
dataVersion = 3
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 4
numChildren = 0


Copy the code
3. Type of zNode

Zookeeper provides four types of ZNodes. You need to specify the types when creating nodes using the Client.

zookeeper.create("/ public number/programmer internal matters"."".getBytes(), Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL_SEQUENTIAL);
Copy the code

  • Persistent-persistent Directory node: After a client creates a node and disconnects from ZooKeeper, the node is persisted. After the client re-connects to ZooKeeper, the node still exists.

  • PERSISTENT_SEQUENTIAL node: After the client creates a node and disconnects from ZooKeeper, the node will be persisted. Zookeeper numbers the node name sequentially, for example: /lock/0000000001, /lock/0000000002, /lock/0000000003.

  • EPHEMERAL- Temporary directory node: This node is deleted after the client is disconnected from ZooKeeper

  • EPHEMERAL_SEQUENTIAL- Temporary sequential node: After the client is disconnected from ZooKeeper and the node is deleted, the node name is sequentially numbered, for example: /lock/0000000001, /lock/0000000002, /lock/0000000003.

2. ACL permission control on nodes

The ACL: Access Control List (permission Control of nodes) is used to solve the Access permission problem of ZNode nodes. It should be noted that ZooKeeper controls permissions based on the ZNode level, which means that permissions between nodes are not inherited, that is, child nodes do not inherit permissions of their parent nodes.

The ACL permission format in ZooKeeper consists of

:< ID >:< ACL >.

Schema: Indicates the authorization mode

  • world: Indicates that anyone can access it
  • auth: Only authenticated users can access it
  • digest: Use username: password to generate an MD5 hash value as the authentication ID
  • host/ip: Uses the IP address of the client host for authentication

Id: scope of permission used to identify identity, depending on the schema selected.

Acl: Specifies the permissions granted to a node. The permissions of a node include CREATE, DELETE, write, read, and admin. All of these permissions are cdWRA.

1.world: Indicates that anyone can access it

Let’s use the getAcl command to see the default permissions of a Znode that has no permissions set.

[zk: localhost:2181(CONNECTED) 12] getAcl /test
'world,'anyone
: cdrwa
Copy the code

The default schema is World, the scope is Anyone, and the node permission is CDWRA, which means that anyone can access the node.

What if we want to set the world permission on a node whose schema is not world?

setAcl /test world:anyone:crdwa
Copy the code
2,auth: Only authenticated users can access it

Schema uses auth authorization to indicate that only authenticated users can access the schema. Therefore, you need to add authenticated users first. After adding authenticated users, you need to set ACL permissions for authenticated users.

Addauth Digest test: PasswordCopy the code

Note that the password for user authentication is in plain text.

[zk: localhost:2181(CONNECTED) 2] addauth digest user:user // User name: password
[zk: localhost:2181(CONNECTED) 5] setAcl /test auth:user:crdwa
[zk: localhost:2181(CONNECTED) 6] getAcl /test
'digest,'user:ben+k/3JomjGj4mfd4fYsfM6p0A=
: cdrwa
Copy the code

SetAcl /test auth:user:crdwa is equivalent to setAcl /test auth::crdwa.

3,digest: User name: password authentication mode

User name: The password authentication mode applies to a specific user. In this mode, you do not need to add an authentication user.

If the ZOOKeeper client is used to set the ACL in the code, the password is in plain text. However, if the ZK. CLI client is used, the password needs to be sha1 and Base64.

SetAcl <path> digest:<user>:<password(cipher-text)>:<acl> setAcl /test Digest :user:jalRr+ KNV /6L2uXdenC93dEDNuE=:crdwaCopy the code

So how to encrypt the password? There are several ways:

Encryption is performed using shell commands

echo -n <user>:<password> | openssl dgst -binary -sha1 | openssl base64
Copy the code

They are used to own a library org. Apache. Zookeeper. Server auth. DigestAuthenticationProvider generated

java -cp /zookeeper3.413./zookeeper3.413..jar:/ zookeeper 3.4.13 /lib/slf4j-api1.725..jar \
  org.apache.zookeeper.server.auth.DigestAuthenticationProvider \
  root:root
root:root->root:qiTlqPLK7XM2ht3HMn02qRpkKIE=
Copy the code
4,host/ip: Uses the IP address of the client host for authentication

This approach is easier to understand by granting authorization to a specific IP address, or an IP address segment.

[zk: localhost:2181(CONNECTED) 3] setAcl /test0000000014 ip:127.0. 01.:crdwa
cZxid = 0x59ac
ctime = Mon Mar 30 15:20:08 CST 2020
mZxid = 0x59b6
mtime = Mon Mar 30 16:58:08 CST 2020
pZxid = 0x59ac
cversion = 0
dataVersion = 3
aclVersion = 3 // This version is growing all the time
ephemeralOwner = 0x0
dataLength = 4
numChildren = 0
Copy the code

The soul of Zookeeper watcher

We mentioned at the beginning that ZooKeeper can register and discover services for Dubbo as a registry, but have you ever wondered why ZooKeeper can do this? This brings us to the soul of ZooKeeper, Watcher.

1. What is watcher?

Watcher is a very core function of zooKeeper. The client watcher can monitor the data changes of the node and its child nodes. Once the status changes, the zooKeeper server will notify all clients that have set Watcher on this node. As a result, each client will quickly sense that the state of the node it is listening to changes, and make the corresponding logical processing.

With a brief introduction to Watcher, let’s take a look at how ZooKeeper implements service registration and discovery. Zookeeper service registration and discovery mainly applies zooKeeper’s ZNode data model and Watcher mechanism. The general process is as follows:

  • Service registration: When a service Provider is started, it registers service information with the ZooKeeper server. That is, it creates a node, for example, user registration service com.xxx.user.register, and stores service data (such as the IP address and port of the service Provider) on the node.

  • Service discovery: When a service Consumer starts up, it obtains the registered service information from the ZooKeeper server according to its configured dependent service information and sets watch monitoring. After obtaining the registered service information, it caches the information of the service provider locally and invokes the service.

  • Service notification: Once the service provider breaks down for some reason and stops providing services, the client is disconnected from the ZooKeeper server, and the corresponding service node of the service provider on the ZooKeeper server will be deleted (for example: User registration service com.xxx.user.register), and then the ZooKeeper service end will asynchronously register the service com.xxx.user.register to all consumer users, and the service consumers monitored by Watch will send the notification of node deletion. The consumer pulls the latest service list based on the notification received and updates the locally cached service list.

This is how ZooKeeper can register and discover services.

2. Watcher type

Two types of Watch can be set on zNode. One is DataWatches, which triggers the watch event based on the data change of ZNode and triggers the conditions getData(), exists(), setData() and create().

The other is Child Watches. The Watch event triggered by the change of the Child node based on ZNode triggers the conditions getChildren() and create().

When the delete() method is called to delete zNode, both Data Watches and Child Watches will be triggered. If the deleted node has a parent node, the parent node will trigger a Child Watches.

3. Watcher features

Watch is a one-time listener on a node. The client sets the monitoring watch on the specified node. Once the client is notified once the node data changes, the client’s monitoring event on the node will become invalid.

If we want to continue listening on this node, we need to set the watch event of the node to True again in the client’s listening callback. Otherwise, the client can receive only one change notification from the node.

What functions can ZooKeeper achieve

Service registration and discovery function is only the tip of the iceberg of ZooKeeper, it can also achieve a series of functions such as distributed lock, queue, configuration center, and so on, next we only analyze the principle, the specific implementation of the Internet we check the information or relatively complete.

1. Distributed locks

Zookeeper is based on the Watcher mechanism and the ordered nodes of ZNode, which is naturally a distributed lock. First create a /test/lock parent as a lock, as PERSISTENT as possible. For each client that tries to acquire this lock, create temporary sequential children under the /test/lock parent.

Because of the increment of ordinal number, we stipulate that the node with the smallest ordinal number obtains the lock. For example, if the client obtains the lock, create node /test/lock/ seQ-00000001 under /test/lock. It is the smallest node, so it obtains the lock first. Other nodes wait for notification to acquire the lock again. /test/lock/ seQ-00000001 Delete the node after executing its own logic to release the lock.

Then the node/test/lock/seq-00000002Who do you want to get lock notification from?

/test/lock/seq-00000001 /test/lock/seq-00000002 /test/lock/ seQ-00000001 /test/lock/ seQ-00000001 Let it decide again if it is the smallest node, if it is, it will get the lock, not continue to wait for notification.

/test/lock/seq-00000003 node listening /test/lock/seq-00000002 node, always let the last node listen on the previous node, do not let all nodes listen on the smallest node, avoid setting unnecessary listening, so as not to cause a large number of invalid notifications. Form a “herd mentality”.

Compared with Redis distributed lock, zooKeeper distributed lock is not recommended because of poor performance in creating and deleting a large number of nodes.

2. Distributed queues

Zookeeper is also very simple to implement distributed queue. The ordered nodes of ZNode are naturally “first-in, first-out”. The node created later is always the largest, and the node with the smallest sequence number is always the only one to be taken out of the queue.

3. Configuration management

Many open source projects use Zookeeper to maintain configurations, such as message queue Kafka, which uses Zookeeper to maintain broker information. Configuration information about management services in Dubbo. The principle is also based on the Watcher mechanism. For example, a /config node is created to store some configurations. The client listens to this node, changes the configuration information of the /config node, and informs each client of data changes to pull the configuration information again.

4. Naming service

Zookeeper naming service: Is what we call service registration and discovery, mainly to secure resources or services according to specified name address, the information such as service providers, advantage of the characteristics of the znode node and watcher mechanism, as a dynamic registration and configuration to the information of the service center, unified management service name and its corresponding server list information, We can sense the status of back-end servers (online, offline, down) in near real time.

conclusion

This article is intended to introduce the basic knowledge of ZooKeeper, such as the concept of zooKeeper cluster selection, which is frequently asked in the interview, and I will not write it in this period, because the content of cluster is also quite large, I am afraid that the length of the article is too long for everyone to read it (actually a little lazy, ha ha ha!). If you are interested, you can pay attention to a wave of ZooKeeper clusters. See you next time.

So much for today, if this article is a little help to you, I hope you can get a thumbs up 👍 oh

Your approval is my motivation to write!

Small benefits:

Hundreds of technical e-books, SHH ~, free for your friends. Concern public number reply [666] to get by yourself