The cluster theory

A system to build a cluster mainly need to solve two problems: data synchronization and cluster fault tolerance.

Naive solution

A simple and crude solution is to deploy multiple identical Redis services and use load balancing to spread the load and monitor service status. The advantage of this scheme is that it is easy to tolerate faults. As long as one machine is alive, the whole cluster is still available. However, the problem is that ensuring the data consistency of these Redis services will lead to a large number of data synchronization operations, which will affect performance and stability.

Redis cluster solution

The Redis cluster solution is based on the idea of divide and conquer. Data in Redis is stored in the form of key-value, and data with different keys are independent of each other. Therefore, keys can be divided into multiple partitions according to certain rules, and data in different partitions can be stored on different nodes. This scheme is similar to the structure of a hash table in a data structure. In the implementation of Redis clustering, the hash algorithm (formula is CRC16(Key) mod 16383) is used to map keys to integers in the range of 0 to 16383. In this way, each integer stores several key-values. The abstract storage corresponding to an integer is called a slot. Each Redis Cluster node — or master node — is responsible for a range of slots. The Cluster of all nodes covers a range of slots from 0 to 16,383.

It is said that any computer problem can be solved by adding an intermediate layer. The concept of a slot is the same. It lies between data and nodes, simplifying expansion and contraction operations. The mapping between data and slots is performed by a fixed algorithm and does not require maintenance. Nodes only need to maintain the mapping between themselves and slots.

Slave

The above solution only solves the problem of performance expansion, but does not improve the fault tolerance of the cluster. The usual way to improve fault tolerance is to use some kind of backup/redundancy approach. Nodes that are responsible for a certain number of slots are called master nodes. To increase cluster stability, each master node can be configured with several backup nodes — called slave nodes. The Slave node is used as a cold backup to store the data of the master node and replace the master node when the master node is down. In some cases with heavy data access pressure, the slave node can also provide data reading function, but the real-time performance of the slave node is slightly worse. Writing data can only be done through the master node.

Request redirection

When the Redis node receives a command for a key whose slot is not its responsibility, it returns a MOVED redirection error, notifying the client to access the data on the correct node.

If redirection errors occur frequently, access performance will be affected. Because the algorithm for mapping a key to a slot is fixed, clients can maintain the mapping between slots and nodes internally. When accessing data, clients can calculate slots based on keys and find the correct node, reducing redirection errors. Redis clients in most current development languages implement this strategy. IO /clients to view redis clients in major languages.

Node communication

Although different nodes store data independently of each other, these nodes still need to communicate with each other to synchronize node state information. Redis cluster uses P2P Gossip protocol. The nodes constantly communicate and exchange information, and eventually all the nodes will reach the same status. Common Gossip messages include the following:

  • Ping messages: Each node continuously sends ping messages to other nodes to check whether the node is online and exchange node status information.
  • Pong message: Response message when a ping or meet message is received.
  • Meet message: new node joins message.
  • Fail message: indicates that a node is offline.
  • Forget messages: Forgets node messages and takes a node offline. This command must be executed on all nodes within 60 seconds, otherwise the node re-participates in the message exchange after 60 seconds. In practice, it is not recommended to use the forget command to offline a node.

Nodes offline

When a node has a problem, it takes a certain amount of propagation time for the majority of master nodes to consider the node to be truly unavailable before it can be marked offline. The Redis cluster node offline includes two steps: subjective offline (PFAIL) and objective offline (FAIL).

  • Subjective offline: If node A fails to communicate with node B within the cluster-Node-timeout period, node A considers node B unavailable, marks it as subjective offline, and transmits the status message to other nodes.
  • Objective offline: After a node is marked as subjective offline by most master nodes in the cluster, the objective offline process is triggered to mark the node as offline.

Fault recovery

When a master node that holds a slot goes offline objectively, the cluster selects a master node from the slave nodes to replace it. The Redis cluster uses an election-vote algorithm to select slave nodes. A slave node must be voted by a majority of master nodes, including the faulty master node, before it can be promoted to master node. Assume that the cluster size is 3 active and 3 secondary, at least two active nodes must survive to perform fault recovery. If two active nodes are deployed on the same server, the cluster cannot recover after the server breaks down.

By default, if a master node is unavailable in a Redis cluster, that is, if some slots have no responsible nodes, the whole cluster is unavailable. That is, when a master node fails, the entire cluster is unavailable until the fault is recovered. This is intolerable for some businesses. You can set cluster-require-full-coverage to no in the configuration, so the master node failure will only affect the access to the relevant slot data, and does not affect the access to other nodes.

Set up the cluster

Start a new node

Modify the Redis configuration file to start cluster mode:

Enable cluster mode
cluster-enabled yes
# Node timeout in milliseconds
cluster-node-timeout 15000
# Cluster node information file
cluster-config-file "nodes-6379.conf"Copy the code

Then start the new node.

Send meet messages to cluster nodes

Using the client command cluster < IP > , the node sends a meet message to add the new node with the specified IP address and port to the cluster.

Distribution of tank

What we have after the previous step is an “empty” cluster that is not yet responsible for any slots. To make the cluster available, we need to allocate all 16,384 slots to the master node count.

On the client run cluster add addslots {… } to allocate slots in the range to to nodes connected to the current client. After all slots are assigned to the master node, run the cluster Nodes command to view the slots of each node and node ids.

Next, the slave nodes need to be allocated. Use a client to connect to the slave node assigned to it, and run the Cluster replicate

command to allocate the node as a backup of the master node specified by

.

Use commands to create clusters directly

Cluster operation commands are added to Redis – CLI client in Redis 5.

Create a cluster with 3 master and 3 slave using commands as shown below:

Redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001\127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 \  --cluster-replicas 1Copy the code

If you are using an older version of Redis, you can use the redis-trib.rb script to create clusters:

./redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001\127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005Copy the code

The cluster scale

capacity

Expanding capacity is similar to creating a cluster except that the last step is to migrate slots from existing nodes to new nodes.

  1. Start a new node: The same as creating a cluster.
  2. Add a new node to the clusterredis-cli --cluster add-nodeThe command adds the new node to the cluster (implemented internally using the meet message).
  3. Migrating slots and data: After a new node is added, some slots and data need to be migrated from the old node to the new node. Using the commandredis-cli --cluster reshardMigrate slots.

shrinkage

To safely remove nodes, the Redis cluster can only take offline nodes that do not have responsible slots. Therefore, if you want to bring down a master node with a slot, you need to migrate its slot to another node first.

  1. Migration channel. Using the commandredis-cli --cluster reshardMigrate all slots of the node to be deleted to other nodes.
  2. Forget the node. Using the commandredis-cli --cluster del-nodeDelete a node (internally using the FORGET message).

  3. Cluster Configuration Tool

If your redis-CLI version is younger than 5, you can use the redis-trib.rb script to complete the above command. Click here for redis-cli and redis-trib.rb commands to operate clusters.

If you want to learn Java engineering, high performance and distribution for free, simple. Micro services, Spring, MyBatis, Netty source analysis of friends can add my Java advanced group: 478030634, group ali Daniel live explain technology, and Java large Internet technology video free to share to you.

persistence

Redis has RDB and AOF persistence strategies. This article explains RDB and AOF persistence in detail.

A pit for RDB persistence

RDB persistence trap:

  • Even if it’s set upsave ""Attempts to close RDB, however RDB persistence may still trigger.
  • The master node triggers RDB persistence to generate RDB files for full replication (for example, when a new slave node is added). The RDB file is then sent to the slave node. Finally, both the slave node and the corresponding master node have RDB files.
  • If AOF is not enabled when shutdown is executed, RDB persistence is also triggered.
  • Regardless of how save is set, as long as the RDB file exists, redis starts to load that file.

Consequences:

  • If RDB persistence is turned off (and AOF persistence as well), the RDB files saved from the last full copy from the node or shutdown are loaded when Redis restarts. And this RDB file is probably a piece of data that’s long out of date.
  • In Cluster mode, Redis restarts and recovers data from the RDB file. If the configuration of Nodes in cluster-config-file cannot be read, Redis marks itself as an independent master and occupies the slot corresponding to the Key of the data recovered from the RDB. As a result, the node cannot be added to other clusters.