Series of columns: Redis series of columns

Slice the cluster

Data expansion mode

If the total amount of data Redis wants to cache is not very large, such as 5GB of data, it is generally acceptable to use the master-slave model + sentinel cluster to ensure high availability. However, if the total amount of data Redis needs to cache is large, or may increase in the future, such as 20GB or 50GB data, then a master database cannot meet the requirements. In this case, there are generally two modes for expansion: vertical expansion and horizontal expansion.

Vertical scaling

Vertical scaling is to upgrade the configuration of a single Redis instance to increase server memory capacity, disk capacity, and higher CPU configuration. For example, the previous 4C 8GB 50GB is upgraded to 8C 32GB 100GB. If the amount of data increases, the configuration will be continued.

Vertical scaling has the advantage of being simple to implement, but it also has potential problems:

  • If RDB data persistence is used, as the amount of data increases, so does the amount of memory required. On the other hand, during RDB persistence, the main thread will fork child processes, and the fork operation time is positively correlated with the amount of Redis data. The larger the amount of data is, the longer the main thread will be blocked by fork operation, which will sometimes lead to the slow response of Redis. Of course, this problem does not exist if RDB persistence is not required.

  • Another problem is that vertical scaling is limited by hardware and cost. It’s easy to scale up from 32GB to 64GB of memory, but there are hardware capacity and cost constraints to scale up to 1TB.

Horizontal scaling

Horizontal expansion is to increase the number of Redis instances and spread the data among each instance. For example, to cache 15GB data, it is ok to use three 8GB servers, and one instance only caches 5GB data.

This is the slice cluster of Redis, also known as the slice cluster, which is to start multiple Redis instances to form a cluster, and then divide the received data into each instance according to certain rules. This solution simply increases the number of instances of Redis without worrying about the hardware and cost constraints of a single instance.

Although it is troublesome to set up a slice cluster, it can store a large amount of data. Since the amount of data saved by a single instance is small, RDB persistence has less impact on the main thread blocking. And as users or businesses grow, it is often inevitable to keep large amounts of data, so a sliced cluster is a good solution.

Cluster architecture

Redis deployment generally has different solutions according to different scenarios. Here is a simple comparison:

  • 1. Single-machine mode

If the amount of data is small, mainly for high concurrency and high performance scenarios, such as the cache is generally a few GIGABytes, deploying a stand-alone Redis instance is sufficient.

  • 2. Master/slave replication + sentinel

This mode mainly ensures high concurrent read and high availability scenarios. The master-slave mode means one master library + multiple slave libraries. Deploying several slave libraries is related to read throughput, and then deploying a sentinel cluster to ensure high availability of the master and slave clusters. This mode can be used in scenarios where the amount of data is not very large.

  • 3. Cluster mode

In the master-slave architecture mode of Redis, all data is written into one master library, which has some problems such as upper limit of single machine capacity and high concurrent write performance. Redis cluster has multiple master libraries, data is scattered to each master library according to certain rules, the number of master libraries depends on the size of the data volume, can dynamically add and delete master library nodes according to the data volume.

The Redis cluster needs at least three primary nodes to run normally. It is recommended to configure at least one secondary node for each primary node for master/standby switchover. When the primary database is unavailable, the secondary database can be installed on top. In the production environment, you are advised to deploy six servers with three primary libraries and three secondary libraries to ensure high availability of the cluster.

Different from the master-slave architecture, the secondary nodes in the Redis cluster are mainly for high availability. Each master library has one or two secondary nodes, which are used for hot backup and master/standby switchover of data to achieve high availability. Therefore, the primary and secondary nodes in the cluster mode do not need sentinels to ensure high availability.

In addition, if you want to improve the performance of high concurrent reads, you can generally increase the read throughput by adding the primary library node. Of course, you can also modify the configuration to configure the secondary library to be readable and perform read/write separation mode. However, this may be a bit complicated, and the Redis client also supports read/write separation.

Cluster deployment

First stop the previously deployed Redis master/slave and Sentinel clusters. The deployment mode of the Redis cluster is not the same as before. Then, based on the cluster architecture diagram above, we deployed six instances with three servers, namely three master and three slave, and tried to keep the master and slave instances on the same machine.

Six instances are deployed on three servers with ports 7001 to 7006 in sequence. The deployment architecture is as follows:

The server Redis instance
172.17.0.2 7001, 7002,
172.17.0.3 7003, 7004,
172.17.0.4 7005, 7006,

1. Create a data directory

Create a directory on the first server:

# mkdir -p /var/redis/log
# mkdir -p /var/redis/7001
# mkdir -p /var/redis/7002
Copy the code

2. Configuration files

Copy two copies of the Redis configuration file to /etc/redis:

# cp /usr/local/ SRC/redis - 6.2.5 / redis. Conf/etc/redis / 7001. Conf
# cp /usr/local/ SRC/redis - 6.2.5 / redis. Conf/etc/redis / 7002. Conf
Copy the code

Modify the following configuration in the configuration file to ensure that 7001 is the same as the port number in the file:

configuration value instructions
bind 172.17.0.2 Binding a Local IP address
port 7001 port
cluster-enabled yes Enabling the Cluster Mode
cluster-config-file /etc/redis/node-7002.conf Cluster configuration file
cluster-node-timeout 15000 Node lifetime timeout period
daemonize yes daemon
pidfile /var/run/redis_7001.pid The PID file
dir /var/redis/7001 The data directory
logfile /var/redis/log/7001.log The log file
appendonly yes Enable AOF persistence

Do not configure replicaof <masterip> <masterport> in the file.

3. Start the script

Copy two copies of the Redis startup script to /etc/init.d/ :

# cp /usr/local/ SRC/redis - 6.2.5 / utils/redis_init_script/etc/init. D/redis_7001
# cp /usr/local/ SRC/redis - 6.2.5 / utils/redis_init_script/etc/init. D/redis_7002
Copy the code

Then change the port number in the script:

And add the following two lines at the beginning of the script to boot:

# chkconfig: 2345 90 10 
# description: Redis is a persistent key-value database
Copy the code

Set boot:

# chkconfig redis_7001 on
# chkconfig redis_7002 on
Copy the code

4. Start the Redis instance

Install the configuration files and startup scripts on all three servers, and start each instance separately.

# cd /etc/init.d

# ./redis_7001 start
# ./redis_7002 start
Copy the code

View the Redis process:

If the startup fails, you can view the log:

5. Create a cluster

You can use the redis-cli –cluster tool to manage the cluster. The command format is as follows:

redis-cli --cluster SUBCOMMAND [ARGUMENTS] [OPTIONS]
Copy the code

–cluster-replicas

indicates that each master has several slaves:

redis-cli --cluster create host1:port1 ... hostN:portN --cluster-replicas <arg>
Copy the code

Let’s create a cluster of six instances. It will automatically assign which instances are master and which are slave, and try to keep the master and slave on different servers. The output shows the cluster creation process.

[root@centos-01 init.d]# redis-cli --cluster create 172.17.0.2:7001 172.17.0.2:7002 172.17.0.3:7003 172.17.0.3:7004 172.17.0.4 172.17.0.4:7005:7006 - cluster - replicas of 1>>> Performing hash slots allocation on 6 nodes...Master[0] -> Slots 0 - 5460 Master[1] -> Slots 5461 - 10922 Master[2] -> Slots 10923 - 16383 Adding replica 172.17.0.3:7004 to 172.17.0.2:7001 Adding Replica 172.17.0.4:7006 to 172.17.0.3:7003 Adding Replica 172.17.0.2:7002 to 172.17.0.4:7005 M: 9 b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 172.17.0.2:7001 slots: [0-5460] (5461 slots) master S: 9 b0751a4b6ea227658ed26576d9bda1f3b01fc04 172.17.0.2:7002 replicates d9aef1c511d77efee79a45a63eff18fb4f9 26776 M: 4 e2f6532200991c93bbf23f7d4996fbd2aff02ac 172.17.0.3:7003 slots: [5461-10922] (5462 slots) master S: 9 a627cad72e4192802237c00e4e570b37241feda 172.17.0.3:7004 replicates 9 b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 M: 26776 d9aef1c511d77efee79a45a63eff18fb4f9 172.17.0.4:7005 slots: [10923-16383] (5461 slots) master S: 4 e2f6532200991c93bbf23f7d4996fbd2aff02ac fc25b50d6afe39211a47609287afd250c9e3183b 172.17.0.4:7006 replicates Can I set the above configuration? (type 'yes' to accept): yes>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 172.17.0.2:7001)M: 9 b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 172.17.0.2:7001 slots: [0-5460] (5461 slots) master 1 additional up (s) M: 26776 d9aef1c511d77efee79a45a63eff18fb4f9 172.17.0.4:7005 slots: [10923-16383] (5461 slots) master 1 additional up (s) S: 9 a627cad72e4192802237c00e4e570b37241feda 172.17.0.3:7004 slots: (0 slots) slave replicates 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 S: 9 b0751a4b6ea227658ed26576d9bda1f3b01fc04 172.17.0.2:7002 slots: (0 slots) slave replicates 26776d9aef1c511d77efee79a45a63eff18fb4f9 S: Fc25b50d6afe39211a47609287afd250c9e3183b 172.17.0.4:7006 slots: (0 slots) slave replicates 4e2f6532200991c93bbf23f7d4996fbd2aff02ac M: 4 e2f6532200991c93bbf23f7d4996fbd2aff02ac 172.17.0.3:7003 slots: [5461-10922] (5462 slots) master 1 additional up (s) [OK] All nodes agree about slots configuration.>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Copy the code

You can run the redis-cli –cluster check command on other instances to check the cluster status:

[root@centos-02 init.d]# redis-cli --cluster check 172.17.0.3:7003 172.17.0.3:7003 (4e2f6532...) - > 0 keys 5462 slots | | 1 slaves. 172.17.0.4:7005 (26776 d9a...). - > 0 keys 5461 slots | | 1 slaves. 172.17.0.2:7001 (9 b4f3c2c...). 5461 slots - > 0 keys | | 1 slaves. [OK] 0 keys in 3 masters. 0.00 keys per slot on business.>>> Performing Cluster Check (using node 172.17.0.3:7003)M: 4 e2f6532200991c93bbf23f7d4996fbd2aff02ac 172.17.0.3:7003 slots: [5461-10922] (5462 slots) master 1 additional up (s) M: 26776 d9aef1c511d77efee79a45a63eff18fb4f9 172.17.0.4:7005 slots: [10923-16383] (5461 slots) master 1 additional up (s) S: fc25b50d6afe39211a47609287afd250c9e3183b 172.17.0.4:7006 slots: (0 slots) slave replicates 4e2f6532200991c93bbf23f7d4996fbd2aff02ac S: 9 a627cad72e4192802237c00e4e570b37241feda 172.17.0.3:7004 slots: (0 slots) slave replicates 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 S: 9 b0751a4b6ea227658ed26576d9bda1f3b01fc04 172.17.0.2:7002 slots: (0 slots) slave replicates 26776d9aef1c511d77efee79a45a63eff18fb4f9 M: 9 b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 172.17.0.2:7001 slots: [0-5460] (5461 slots) master 1 additional up (s) [OK]  All nodes agree about slots configuration.>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Copy the code

At this point, a Redis cluster with three servers, three masters and three slaves has been set up.

The cluster theory

Data distribution algorithm

Slicing Cluster is a universal mechanism for storing a large amount of data. This mechanism can be implemented in different ways. The official Redis Cluster scheme is Redis Cluster, which specifies the corresponding rules of data and instances.

Hash slot

The Redis Cluster uses Hash slots to map data to instances. The Cluster is divided into 16,384 slots, and each key-value pair is mapped to a Hash Slot based on its key.

When creating a cluster, the redis-cli –cluster create command automatically distributes 16384 slots evenly on the cluster instance. If the cluster has N masters, the number of slots on each Master is 16384 / N.

According to the previous cluster creation logs, you can know which slots are assigned to each Master and the Master -> Slave relationship.

Master[0] -> Slots 0 - 5460 Master[1] -> Slots 5461 - 10922 Master[2] -> Slots 10923 - 16383 Adding replica 172.17.0.3:7004 to 172.17.0.2:7001 Adding Replica 172.17.0.4:7006 to 172.17.0.3:7003 Adding Replica 172.17.0.2:7002 to 172.17.0.4:7005 M: 9 b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 172.17.0.2:7001 slots: [0-5460] (5461 slots) master S: 9 b0751a4b6ea227658ed26576d9bda1f3b01fc04 172.17.0.2:7002 replicates d9aef1c511d77efee79a45a63eff18fb4f9 26776 M: 4 e2f6532200991c93bbf23f7d4996fbd2aff02ac 172.17.0.3:7003 slots: [5461-10922] (5462 slots) master S: 9 a627cad72e4192802237c00e4e570b37241feda 172.17.0.3:7004 replicates 9 b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 M: 26776 d9aef1c511d77efee79a45a63eff18fb4f9 172.17.0.4:7005 slots: [10923-16383] (5461 slots) master S: 4 e2f6532200991c93bbf23f7d4996fbd2aff02ac fc25b50d6afe39211a47609287afd250c9e3183b 172.17.0.4:7006 replicatesCopy the code

Of course, if a machine has high memory, CPU, etc., we can allocate more slots to the Redis instance of that machine. The command for reassigning slots is redis-cli –cluster reshard, for example, 172.17.0.2:7001 (.. 2f5CA7BF646) moved 500 Slots to 172.17.0.2:7003 (.. Fbd2aff02ac).

[root @ centos - 01 /] # redis - cli - cluster reshard 172.17.0.2:7001 - cluster - the from 9 b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 --cluster-to 4e2f6532200991c93bbf23f7d4996fbd2aff02ac --cluster-slots 500Copy the code

After the reassignment, you can run the redis-cli –cluster check command to check slot configurations.

Slot location algorithm

By default, the Redis Cluster uses the CRC16 algorithm to compute a 16-bit hash integer for the key value, and then modulo 16384 with this integer value to obtain a module ranging from 0 to 16383. Each module represents a corresponding hash slot. You get the slot.

The Redis Cluster also allows the user to force a key to hang on a particular slot. CRC16 computes only the hash value of the tag by embedding the {tag} tag in the key string. For example, user_{system} and role_{system}, the two keys compute the hash value only for system, and the two keys end up in the same slot.

Advantages of hash slots

Redis Cluster does not directly maintain the mapping relationship between keys and instances, but maintains the mapping relationship between hash slots and instances. This requires the client to calculate the hash value of keys using CRC16 first. Why not directly maintain the mapping relationship between keys and instances?

  • First of all, the number of keys stored in the whole cluster cannot be estimated, and the mapping relationship between keys and instances is directly stored. This mapping table may be very large, and it will occupy very large memory space whether stored on the server or the client.

  • Redis Cluster adopts the decentralized mode. Each instance will exchange its slot configuration with other instances and store the complete mapping relationship of the whole Cluster. If the relationship between keys and instances is stored, information exchange between nodes will become huge and consume too much network resources.

  • When a cluster is being expanded, scaled down, or balanced, data will be migrated between nodes. During the migration, the mapping relationship of each key must be modified, resulting in high maintenance costs.

Through add a layer of hash slots in the middle, can put the data and node decoupling, key by hash calculation, only need to care about which hash map to the slot, and then through the hash slot and node mapping table to find the node, equivalent to consume little CPU resources, not only let the data distribution is more even, also can make the mapping table is very small, It is easy for clients and servers to save and lightweight for exchanging information between nodes. During data migration, hash slots are used as the basic unit, which simplifies node capacity expansion and reduction and facilitates cluster maintenance and management.

Slot configuration information

When a client (Jedis, lettuce, etc.) in a Redis cluster connects to a cluster instance, it gets a slot configuration for the cluster, and then the client caches the hash slot information locally. When a client requests a key-value pair, the hash slot corresponding to the key is computed and the request can then be sent to the instance corresponding to the hash slot.

So how does each instance know which slot the other instance is responsible for? This is because when instances in the cluster are connected to each other, they send their hash slot information to each other. When instances are connected to each other, each instance has a mapping of all hash slots. This allows clients to get the entire hash slot mapping of the cluster when they connect to an instance.

The slot information of a cluster is stored on each node, and it does not require additional distributed storage to store node slot information. When creating the cluster, we specified the cluster configuration file through cluster-config-file. Each node of the cluster will persist the hash slot information of the cluster to this configuration file, so you must ensure that the configuration file is writable and do not manually modify the configuration file. With this file, instance downtime and restart can restore the slot configuration.

For example, the following two configuration files:

Slot redirection

As mentioned earlier, the Redis cluster is decentralized in that each instance of the cluster has a complete slot mapping table for the entire cluster, which provides the client with error correction capability because the client may access slots that are not on the target instance.

When a client of the Redis cluster connects to the cluster, it also gets a copy of the slot configuration information of the cluster and caches it locally. However, if slot information in a cluster changes, such as capacity expansion or reduction, nodes can send messages to each other to obtain the latest hash slot allocation information, but clients cannot proactively detect the changes.

Redis Cluster provides a redirection mechanism. When a client sends a command to an instance and the instance finds that the key slot is not its own, it will return a special move instruction to the client, which contains the key slot and the instance. The client then reaccesses the new instance.

For example, the hash slot 5798 for the requested key-value pair is 172.17.0.3:7003. The client then updates the slot information cached locally and resends instructions to the new instance.

Data migration

In a Cluster, we might expand, shrink, or load balance, and the Redis Cluster redistributes the hash slots and migrates the data.

Data migration between instances is based on the hash slot. When a slot is being migrated, the slot is in an intermediate transition state. It is possible that some of the data in this slot has been migrated to the destination node and some of the data remains on the original node.

If the corresponding data is still in the original node when the client accesses the original node, the original node processes the data normally. If the corresponding data is not in the original node, there are two possibilities: the data is in the new node, or it does not exist at all. The old node does not know which case it is, so it returns an ASK redirect to the client, which tells the client that the slot is being migrated and the address of the new instance of the slot. After receiving the redirection command, the client first executes an ASKING command without any parameters on the target node, and then re-executes the original operation command on the target node.

GET hello:key
(error) ASK 13320 172.17.0.3:7003
Copy the code

The reason for sending the ASKING command first is that the slot will not be managed by the new node until the migration is complete. If you send a command for the slot to the new node, it will return a MOVED redirection to the client telling it to execute on the original node, which will cause a circular redirection. The ASKING instruction tells the new node to enforce the next instruction and treat the next instruction as its own slot.

Unlike the MOVED command, the ASK command does not update the hash slot mapping information cached by the client. So, if the client requests data from the slot again, it will still send the request to the original node.

In addition, the data migration of Redis Cluster is synchronous. Migrating a key will block the original node and the target node at the same time, resulting in certain performance problems in the migration process.

The main/backup

The Slave nodes in the Redis cluster are mainly used for hot backup of data and master/Slave switchover to realize the high availability of the cluster. After the master database breaks down, the cluster will elect a node from the Slave nodes of the master database to switch to the new master database.

The principle of active/standby switchover election is very similar to that of sentry mode, for reference: Redis series (3) — Sentry High availability. The principle of active/standby switchover in cluster mode is briefly described below.

The Master nodes in the cluster continuously send PING messages to exchange information and check whether the communication is normal.

If a node does not PING another node within cluster-Node-timeout, the other node is considered to be down, and the status of the node is subjective offline (SDown). If more than half of the nodes think that the other node is down, it is objective offline (ODown).

After judging that the node is offline objectively, a Slave node of the node is selected again. In this case, the election is conducted according to the following rules:

  • If the disconnection time between the Slave node and the Master node exceedscluster-node-timeout * cluster-replica-validity-factor, will be removed
  • The data copied from the Slave node to the Master node is sorted according to the offset. The larger the offset is, the data is selected first
  • If offset is the same, all Master nodes vote on the Slave nodes, and the one with more than half of the votes (N/2+1) is elected.
cluster-node-timeout 15000
cluster-replica-validity-factor 10
Copy the code

After a new node is elected, the active/standby switchover is performed. The whole process is very similar to Sentry, and you can see that Redis Cluster is more powerful, integrating master-slave replication and Sentry.

Cluster experiment

Master-slave switch

The following experiments verify whether the secondary database will automatically switch to the primary database after the primary database is down.

Based on the previous deployment, you can know the relationship between cluster nodes as follows. Take 7003 as an example:

172.17.0.2:7001 (Master) -> 172.17.0.3:7004(Slave) 172.17.0.4:7005 (Master) -> 172.17.0.2:7002(Slave) 172.17.0.3:7003 (Master) -> 172.17.0.4:7006(Slave)Copy the code

Verify the operation on 7003 (Master) first:

Not operable on the corresponding 7006 (Slave) :

Now kill 7003:

7006 (Slave) has been changed to Master:

[root@centos-01 ~]# redis-cli --cluster check 172.17.0.2:7001 Could not connect to redis at 172.17.0.3:7003: Connection refused 172.17.0.2:7001 (9 b4f3c2c...). - > 0 keys 5461 slots | | 1 slaves. 172.17.0.4:7005 (26776 d9a...). - > 0 keys 5461 slots | | 1 slaves. 172.17.0.4:7006 (fc25b50d...). - > 1 keys 5462 slots | | 0 slaves. [OK] 1 keys in 3 masters. 0.00 keys per slot on business.>>> Performing Cluster Check (using node 172.17.0.2:7001)M: 9 b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 172.17.0.2:7001 slots: [0-5460] (5461 slots) master 1 additional up (s) M: 26776 d9aef1c511d77efee79a45a63eff18fb4f9 172.17.0.4:7005 slots: [10923-16383] (5461 slots) master 1 additional up (s) S: 9 a627cad72e4192802237c00e4e570b37241feda 172.17.0.3:7004 slots: (0 slots) slave replicates 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 S: 9 b0751a4b6ea227658ed26576d9bda1f3b01fc04 172.17.0.2:7002 slots: (0 slots) slave replicates 26776d9aef1c511d77efee79a45a63eff18fb4f9 M: Fc25b50d6afe39211a47609287afd250c9e3183b 172.17.0.4:7006 slots: [5461-10922] (5462 slots) master [OK] All nodes something about slots configuration.>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Copy the code

After connecting to 7006, we find that we can operate:

Then restart the previous 7003:

Check the cluster status again, you can see 7003 (… 2AFF02AC) automatically changed to 7006 (… 50C9E3183b) slave library.

[root@centos-01 ~]# redis-cli --cluster check 172.17.0.2:7001 172.17.0.2:7001 (9b4f3C2C...) - > 0 keys 5461 slots | | 1 slaves. 172.17.0.4:7005 (26776 d9a...). - > 0 keys 5461 slots | | 1 slaves. 172.17.0.4:7006 (fc25b50d...). - > 1 keys 5462 slots | | 1 slaves. [OK] 1 keys in 3 masters. 0.00 keys per slot on business.>>> Performing Cluster Check (using node 172.17.0.2:7001)M: 9 b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 172.17.0.2:7001 slots: [0-5460] (5461 slots) master 1 additional up (s) M: 26776 d9aef1c511d77efee79a45a63eff18fb4f9 172.17.0.4:7005 slots: [10923-16383] (5461 slots) master 1 additional up (s) S: 9 a627cad72e4192802237c00e4e570b37241feda 172.17.0.3:7004 slots: (0 slots) slave replicates 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 S: 9 b0751a4b6ea227658ed26576d9bda1f3b01fc04 172.17.0.2:7002 slots: (0 slots) slave replicates 26776d9aef1c511d77efee79a45a63eff18fb4f9 M: Fc25b50d6afe39211a47609287afd250c9e3183b 172.17.0.4:7006 slots: [5461-10922] (5462 slots) master 1 additional up (s) S: 4 e2f6532200991c93bbf23f7d4996fbd2aff02ac 172.17.0.3:7003 slots: (0 slots) slave replicates fc25b50d6afe39211a47609287afd250c9e3183b [OK] All nodes agree about slots configuration.>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Copy the code

The above test shows that when the master library is down, the slave library will automatically switch to the top of the master library. After the original master library restarts, it will automatically become the slave library of the new master library.

The cluster expansion

If you want to increase the read/write throughput of the cluster or support more data, you need to expand the cluster and add nodes.

Add a 172.17.0.5 server. Add a primary node and a secondary node with ports 7007 and 7008 respectively.

First follow the previous steps of cluster setup to install the basic environment:

# mkdir -p /var/redis/log # mkdir -p /var/redis/7007 # mkdir -p /var/redis/7008 # mkdir -p /var/redis/7008 # mkdir -p /var/redis/7008 Conf /etc/redis/7007.conf # cp /usr/local/src/redis-6.2.5/redis.conf # cp /usr/local/src/redis-6.2.5/redis.conf # cp /usr/local/src/redis-6.2.5/redis.conf # cp /usr/local/src/redis-6.2.5/redis.conf /etc/redis/7008.conf # cp /usr/local/src/redis-6.2.5/utils/redis_init_script /etc/init.d/redis_7007 # cp / usr/local/SRC/redis - 6.2.5 / utils/redis_init_script/etc/init. D/redis_7008Copy the code

Start both nodes:

The command for adding a node is in the following format:

redis-cli --cluster add-node new_host:new_port existing_host:existing_port
    --cluster-slave --cluster-master-id <arg>
Copy the code

Add 7007 as Master of cluster:

[root@centos-01 ~]# redis-cli --cluster add-node 172.17.0.5:7007 172.17.0.2:7001
>>> Adding node 172.17.0.5:7007 to cluster 172.17.0.2:7001. [OK] All 16384 slots covered.>>> Send CLUSTER MEET to node 172.17.0.5:7007 to make it join the cluster.
[OK] New node added correctly.
Copy the code

Check the state of the cluster, found no slave 7007 slaves (0), also did not assign slot (0 slots), 7007 instance ID for bca31339ca8d8177e9d80831d72b64bc52fdbb87.

[root@centos-01 ~]# redis-cli --cluster check 172.17.0.2:7001 172.17.0.2:7001 (9b4f3C2C...) - > 0 keys 5461 slots | | 1 slaves. 172.17.0.4:7005 (26776 d9a...). - > 0 keys 5461 slots | | 1 slaves. 172.17.0.5:7007 (bca31339...). - > 0 keys | 0 slots | 0 slaves. 172.17.0.4:7006 (fc25b50d...). - > 1 keys 5462 slots | | 1 slaves. [OK] 1 keys in 4 masters. 0.00 keys per slot on business.>>> Performing Cluster Check (using node 172.17.0.2:7001)M: 9 b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 172.17.0.2:7001 slots: [0-5460] (5461 slots) master 1 additional up (s) M: 26776 d9aef1c511d77efee79a45a63eff18fb4f9 172.17.0.4:7005 slots: [10923-16383] (5461 slots) master 1 additional up (s) M: bca31339ca8d8177e9d80831d72b64bc52fdbb87 172.17.0.5:7007 slots: (0 slots) master S: 9 a627cad72e4192802237c00e4e570b37241feda 172.17.0.3:7004 slots: (0 slots) slave replicates 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 S: 9 b0751a4b6ea227658ed26576d9bda1f3b01fc04 172.17.0.2:7002 slots: (0 slots) slave replicates 26776d9aef1c511d77efee79a45a63eff18fb4f9 M: Fc25b50d6afe39211a47609287afd250c9e3183b 172.17.0.4:7006 slots: [5461-10922] (5462 slots) master 1 additional up (s) S: 4 e2f6532200991c93bbf23f7d4996fbd2aff02ac 172.17.0.3:7003 slots: (0 slots) slave replicates fc25b50d6afe39211a47609287afd250c9e3183b [OK] All nodes agree about slots configuration.>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Copy the code

Next add 7008 as 7007 Slave:

[root@centos-01 ~]# redis-cli --cluster add-node 172.17.0.5:7008 172.17.0.2:7001 --cluster-slave --cluster-master-id Bca31339ca8d8177e9d80831d72b64bc52fdbb87 > > > Adding node 172.17.0.5:7008 to cluster 172.17.0.2:7001 > > > Performing Cluster Check (using node 172.17.0.2:7001)...... [OK] All 16384 slots covered. >>> Send CLUSTER MEET to node 172.17.0.5:7008 to make it join the CLUSTER. Waiting for the Cluster to join >>> Configure node as replica of 172.17.0.5:7007. [OK] New node added correctly.Copy the code

7008 has become the secondary repository for 7007.

[root@centos-01 ~]# redis-cli --cluster check 172.17.0.2:7001 172.17.0.2:7001 (9b4f3C2C...) - > 0 keys 5461 slots | | 1 slaves. 172.17.0.4:7005 (26776 d9a...). - > 0 keys 5461 slots | | 1 slaves. 172.17.0.5:7007 (bca31339...). - > 0 keys | 0 slots | 1 slaves. 172.17.0.4:7006 (fc25b50d...). - > 1 keys 5462 slots | | 1 slaves. [OK] 1 keys in 4 masters. 0.00 keys per slot on business.>>> Performing Cluster Check (using node 172.17.0.2:7001)M: 9 b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 172.17.0.2:7001 slots: [0-5460] (5461 slots) master 1 additional up (s) M: 26776 d9aef1c511d77efee79a45a63eff18fb4f9 172.17.0.4:7005 slots: [10923-16383] (5461 slots) master 1 additional up (s) M: bca31339ca8d8177e9d80831d72b64bc52fdbb87 172.17.0.5:7007 slots: 1 additional (0 slots) master up (s) s: 5 bfc64bbce07a872d6d6048c62e80f7a8e56990a 172.17.0.5:7008 slots: (0 slots) slave replicates bca31339ca8d8177e9d80831d72b64bc52fdbb87 S: 9 a627cad72e4192802237c00e4e570b37241feda 172.17.0.3:7004 slots: (0 slots) slave replicates 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 S: 9 b0751a4b6ea227658ed26576d9bda1f3b01fc04 172.17.0.2:7002 slots: (0 slots) slave replicates 26776d9aef1c511d77efee79a45a63eff18fb4f9 M: Fc25b50d6afe39211a47609287afd250c9e3183b 172.17.0.4:7006 slots: [5461-10922] (5462 slots) master 1 additional up (s) S: 4 e2f6532200991c93bbf23f7d4996fbd2aff02ac 172.17.0.3:7003 slots: (0 slots) slave replicates fc25b50d6afe39211a47609287afd250c9e3183b [OK] All nodes agree about slots configuration.>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Copy the code

The new node doesn’t have a hash slot yet. Use reshard or rebalance to assign slots to the new node. For example from 7001 (… 2f5CA7BF646) moved 2000 hash slots to 7007 (… C52fdbb87) :

[root @ centos - 01 ~] # redis - cli - cluster reshard 172.17.0.2:7001 - cluster - the from 9 b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 --cluster-to bca31339ca8d8177e9d80831d72b64bc52fdbb87 --cluster-slots 2000Copy the code

Select * from 7007; select * from 7007;

[root@centos-01 ~]# redis-cli --cluster check 172.17.0.2:7001 172.17.0.2:7001 (9b4f3C2C...) - > 0 keys 3461 slots | | 1 slaves. 172.17.0.4:7005 (26776 d9a...). - > 0 keys 5461 slots | | 1 slaves. 172.17.0.5:7007 (bca31339...). - > 0 keys 2000 slots | | 1 slaves. 172.17.0.4:7006 (fc25b50d...). -> 1 keys | 5462 slots | 1 slaves. [OK] 1 keys in 4 masters. ... [OK] All 16384 slots covered.Copy the code

Cluster shrinkage capacity

The following demo delete 172.17.0.5:7007 (bca31339ca8d8177e9d80831d72b64bc52fdbb87) node.

To delete a node from the cluster, you must move all hash slots in the cluster to other nodes. Otherwise, an error message is displayed:

Reshard 7007(… C52fdbb87) node’s hash slot moved to 7001(… 2 f5ca7bf646) :

[root @ centos - 01 ~] # redis - cli - cluster reshard 172.17.0.2:7001 - cluster - from bca31339ca8d8177e9d80831d72b64bc52fdbb87 --cluster-to 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 --cluster-slots 2000Copy the code

Delete 7007 from node 7007:

[root @ centos - 01 ~] # redis - cli - cluster del -node 172.17.0.2:7002 bca31339ca8d8177e9d80831d72b64bc52fdbb87>> > o node bca31339ca8d8177e9d80831d72b64bc52fdbb87 from cluster 172.17.0.2:7002
>>> Sending CLUSTER FORGET messages to the cluster...
>>> Sending CLUSTER RESET SOFT to the deleted node.
Copy the code

7008 becomes a Slave of 7001. The Slave node 7008 is automatically attached to other Master nodes.

[root@centos-01 ~]# redis-cli --cluster check 172.17.0.2:7001 172.17.0.2:7001 (9b4f3C2C...) - > 0 keys 5461 slots | | 2 slaves. 172.17.0.4:7005 (26776 d9a...). - > 0 keys 5461 slots | | 1 slaves. 172.17.0.4:7006 (fc25b50d...). - > 1 keys 5462 slots | | 1 slaves. [OK] 1 keys in 3 masters. 0.00 keys per slot on business.>>> Performing Cluster Check (using node 172.17.0.2:7001)M: 9 b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 172.17.0.2:7001 slots: [0-5460] (5461 slots) master 2 additional up (s) M: 26776 d9aef1c511d77efee79a45a63eff18fb4f9 172.17.0.4:7005 slots: [10923-16383] (5461 slots) master 1 additional up (s) S: 5 bfc64bbce07a872d6d6048c62e80f7a8e56990a 172.17.0.5:7008 slots: (0 slots) slave replicates 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 S: 9 a627cad72e4192802237c00e4e570b37241feda 172.17.0.3:7004 slots: (0 slots) slave replicates 9b4f3c2ca93b576776e3d3ea5b0c12f5ca7bf646 S: 9 b0751a4b6ea227658ed26576d9bda1f3b01fc04 172.17.0.2:7002 slots: (0 slots) slave replicates 26776d9aef1c511d77efee79a45a63eff18fb4f9 M: Fc25b50d6afe39211a47609287afd250c9e3183b 172.17.0.4:7006 slots: [5461-10922] (5462 slots) master 1 additional up (s) S: 4 e2f6532200991c93bbf23f7d4996fbd2aff02ac 172.17.0.3:7003 slots: (0 slots) slave replicates fc25b50d6afe39211a47609287afd250c9e3183b [OK] All nodes agree about slots configuration.>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Copy the code

Of course we can also delete 7008 node:

[root @ centos - 01 ~] # redis - cli - cluster del -node 172.17.0.2:7002 5 bfc64bbce07a872d6d6048c62e80f7a8e56990a>> > o node 5 bfc64bbce07a872d6d6048c62e80f7a8e56990a from cluster 172.17.0.2:7002
>>> Sending CLUSTER FORGET messages to the cluster...
>>> Sending CLUSTER RESET SOFT to the deleted node.
Copy the code