preface

Although sentinel mode makes read/write separation more highly available, it is far from enough for high concurrency and big data service scenarios due to the memory and CPU bottlenecks of a single server. In this case, experienced partners will not hesitate to think of the cluster, he several nodes, load balancing plus failover, is not beautiful. Yeah, that’s the idea. Here we go.

The body of the

Cluster, believe that the word small partner should listen to the ears of the cocoon; Arrange several servers so that requests/commands are evenly distributed to each server to avoid excessive load on a single server; For the Redis cluster, in order to achieve automatic failover, it is necessary to add one or more slave nodes on each master node. When the master node fails, the slave nodes automatically replace the master node to achieve high availability.

In general, Redis clusters serve the following purposes:

  • The implementation of multi-master nodes can cope with high concurrency scenarios, and the concurrency increases, so the nodes can be expanded to meet the needs at any time.
  • Multi-master implementations can store more data because data is evenly distributed across nodes;
  • The implementation of multiple primary nodes with multiple secondary nodes makes high availability more stable. That is, when a primary node fails, the corresponding secondary node will be upgraded to the primary node to provide normal functions.

Same old rules, while the actual operation side summary, next set up a 3 master 3 slave cluster, this is the simplest. Redis cluster needs at least 3 master nodes, plus in order to achieve high availability, each master node must follow at least one slave node, otherwise a master node hung, can not find complete data, the whole cluster can not be used; As to why the complete data is not available, we will talk about it below.

The cluster environment to be set up is as follows:

Brief description:

  • 6370 is the master node, and 6381 is the slave node of 6370.
  • 6380 is the master node, and 6391 is the slave node of 6380.
  • 6390 is the master node, and 6371 is the slave node.
  • In a clustered environment the master nodes communicate with each other (there are no sentinels) and each node is a data node;

Here the cluster solution uses redis- CLI to automatically specify the primary/secondary relationship (the primary/secondary relationship of partners may be different from mine), or you can manually specify; It’s all the same idea;

The following demonstration on the same machine differentiates nodes by port; In real development, you typically use a different server.

Case presentation

  1. Prepare configuration files for six nodes and enable cluster configurations.

    Copy the original default configuration file and make changes, mainly to the following:

    port 6370 Specifies the Redis node port
pidfile /var/run/redis_6370.pid # specify the corresponding process file
dbfilename dump6370.rdb RDB persistence files for each node
cluster-enabled yes Start the cluster, this is important
cluster-config-file nodes-6370.conf It is important to specify the cluster configuration file for each node
Copy the code

    Above configuration file content in other nodes (6370637, 1638, 0638, 1639, 0639 (1) need to be modified, will only 6370 of them to the corresponding node of port, the purpose is to different nodes use different ports and can be used to distinguish different file; For example, you need to modify the configuration file of node 6371 as follows:

    port 6371 Specifies the Redis node port
pidfile /var/run/redis_6371.pid # specify the corresponding process file
dbfilename dump6371.rdb RDB persistence files for each node
cluster-enabled yes Start the cluster, this is important
cluster-config-file nodes-6371.conf It is important to specify the cluster configuration file for each node
Copy the code

    Cluster-enabled and cluster-config-file are the key points of cluster configuration.

  2. Start six nodes. At first, each node is independent of each other.

    Redis-server: redis-server: redis-server: redis-server: redis-server: redis-server: Redis-server

    ./redis-server ZoeCluster/redis6370.conf Start node 6370
./redis-server ZoeCluster/redis6371.conf Start node 6371
./redis-server ZoeCluster/redis6380.conf Start node 6380
./redis-server ZoeCluster/redis6381.conf Start node 6381
./redis-server ZoeCluster/redis6390.conf Start node 6390
./redis-server ZoeCluster/redis6391.conf Start node 6391
Copy the code

    The startup effect is as follows:

    If you do not believe this, you can use the redis-CLI to connect any node to view the cluster information, as follows:

    As shown in the figure above, cluster-size is 0, and the key slots of the cluster have not been allocated. Then the next step must be to make the cluster relationship of each node;

  3. Establish a node cluster relationship

    Since the Redis version I use is 5.0, I can directly use Redis – CLI to build the cluster. Before this version, IT is recommended to use Redis -trib.rb for related operations. This is a Ruby script, which needs to install the relevant environment.

    Use the following command:

    /redis-cli --cluster create --cluster-replicas 1 127.0.0.1:6370 127.0.0.1:6380 127.0.0.1:6390 127.0.0.1:6371 127.0.0.1:127.0.0.1 6381:6391Copy the code

    Brief Introduction to parameters:

    — cluster: specifies whether to create a cluster environment.

    -A: indicates the password. If there is a password, you can pass the parameter through -a. No password is set.

    –cluster-replicas: Set this parameter to 1, which is used to configure the number of secondary nodes on the primary node. 1 represents a primary and secondary nodes, 2 represents a primary and secondary nodes, and so on.

    Node IP: node port, usually the first is the master node, followed by the slave node.

    If you agree to the cluster solution, then proceed as follows:

    Similarly, you can connect to any node and view the cluster as follows:

    Note: The master-slave replication process between master and slave nodes is not described here. It is the same as the master-slave replication process mentioned earlier.

  4. Demonstrate access operations;

    Write data to any node using redis-CLI. If cluster connection is not specified, write data will fail as follows:

    In the cluster environment, the storage location of data is calculated according to the Key (consistent hash algorithm is involved here), so that the data can be evenly distributed to each node, so the cluster mode needs to be specified during the redis-CLI connection, as follows:

    As shown in the figure above, once the cluster mode is specified, normal access is available;

  5. Fault demonstration

    Since the cluster environment, certainly not a good test;

    Emulation falls from the node

    Try to find a secondary node and stop it. If you stop 6371 nodes, you know that the primary node is 6390;

    The master node displays the slave node disconnection and see its master node response:

    Other cluster nodes simply mark slave nodes as failing, or offline, as follows:

    Access to data is not affected, here is not screenshots, small partners try it;

    When the faulty secondary node 6371 is connected again, the primary node restores the relationship between the primary node and secondary node and performs the primary/secondary replication. Other cluster nodes will clear the original marked offline status and change it to online.

    Simulate the failure of the primary node

    So if I just stop 6390 manually, what happens?

    The slave node checks the connection every second, and if the connection times out (15 seconds by default), it elects the slave node to serve as the primary node of the cluster, as shown below:

    Data access is ultimately unaffected;

    For other cluster nodes, mark the failing nodes as failing and allow the new master node to provide services, as shown below:

    Access to data is also unaffected;

    If the primary node 6390 is recovered, it can only become the secondary node 6371, and perform related primary/secondary replication operations. Other nodes in the cluster only clear their original Fail status, indicating that they can be connected normally.

Redis cluster is so simple, as long as the idea is right, is manual work; Small partners can write scripts for automatic execution oh;

Next, storage of cluster data;

Data storage simple analysis

In the Redis cluster environment, the location of the data is specified based on the Hash calculation of the Key. In order to ensure uniform data distribution when nodes change, the Redis cluster introduces slot as the basic unit of migration. Slot decouples the relationship between data and actual nodes, so that the change of actual node number has little influence on the system.

In the cluster, a total of 16,384 slots are allocated evenly to primary nodes. Each slot has a storage space (storage space can be interpreted as a container, which can store a lot of data). The slots in the cluster environment are allocated as follows:

The procedure for storing data is as follows:

Connect the 6380 primary node and perform the following operations:

The specific process is as follows:

Brief description:

  1. The client initiates a command.
  2. The server performs CRC16 calculation on the Key and calculates the location of the Key to be stored with the total slot.
  3. The Key simulated here is Zoe, and the calculated slot is 14588, which is not on node 6380. The cluster node will redirect it to the node in the corresponding slot.
  4. Then find slot 14588 on 6371 for data storage; (Note that 6371 is already the primary node because of a failover simulation);

How do cluster nodes know the slot ranges and other information of other nodes?

The communication port is redis port +10000. For example, the cluster communication port of node 6371 is 16371 (if there are multiple machines, don’t forget to release this port by firewall), which can be seen through Cluster Nodes as follows:

The file name is specified by the configuration item cluster-config-file. When the cluster node is started, the system checks whether the file exists. If it does not, the system automatically creates the file. Find a configuration file of any node and have a look at it:

As shown in the preceding figure, the configuration file of each node records the master/slave relationship, slot allocation, and status of each node of other nodes. In this case, the cluster relationship will survive a restart.

Cluster scaling (node addition and deletion) demonstration

In actual application scenarios, clusters can be scaled based on service requirements, that is, nodes can be added or deleted. If the service concurrency is large and nodes are added for expansion, nodes may need to be deleted when they need to be adjusted.

Add node

Node expansion is carried out here, adding a 6360 master node, and 6361 as the slave node of 6360; Refer to the configuration file changes above when setting up the cluster and then start them all as follows:

6360 nodes

port 6360 Specifies the Redis node port
pidfile /var/run/redis_6360.pid # specify the corresponding process file
dbfilename dump6360.rdb RDB persistence files for each node
cluster-enabled yes Start the cluster, this is important
cluster-config-file nodes-6360.conf It is important to specify the cluster configuration file for each node
Copy the code

6361 nodes

port 6361 Specifies the Redis node port
pidfile /var/run/redis_6361.pid # specify the corresponding process file
dbfilename dump6361.rdb RDB persistence files for each node
cluster-enabled yes Start the cluster, this is important
cluster-config-file nodes-6361.conf It is important to specify the cluster configuration file for each node
Copy the code

Start both nodes and add 6360 to the cluster master node by executing the following command:

/redis-cli --cluster add-node 127.0.0.1:6360 127.0.0.1:6370Copy the code

Note: 127.0.0.1:6360 is the newly added node, and 127.0.0.1:6370 is any node in the existing cluster.

You can run the following command to check the cluster status:

. / redis - cli - cluster check 127.0.0.1:6370The following address is an arbitrary cluster node
Copy the code

You can see that 6360 has been added to the cluster environment, but it has not been allocated from nodes and slots yet, so we will first add 6361 as the slave node of 6360 as follows:

./redis-cli --cluster add-node --cluster-slave --cluster-master-id eaa814dc56beb0d5edb6a4fbb14f1384e78d4764 127.0.0.1:6361 127.0.0.1:6370 
Copy the code

Parameter Description:

  • –cluster-slave: joins slave nodes.
  • –cluster-master-id: specifies the id of the cluster 6360. You can view node ids from Cluster Nodes.
  • 127.0.0.1:6361: specifies the secondary node to be added.
  • 127.0.0.1:6370: Any primary node of an existing cluster;

Now there is still a gap in slot allocation. To evenly allocate 16384 slots to all nodes, run the following command:

/redis-cli --cluster rebalance --cluster-threshold 1 --cluster-use-empty-masters 127.0.0.1:6370Copy the code

Run the./redis-cli –cluster check 127.0.0.1:6370 command to check the allocation result, as shown in the following figure.

If you do not want to evenly distribute the configuration, run the following command to configure the configuration step by step.

. / redis - cli - cluster reshard 127.0.0.1:6360The newly added primary node is followed by #
You can also configure the desired data directly by executing the following commands/redis-cli --cluster reshard --cluster-from all --cluster-to SPECIFIES the ID of the slots to be allocated --cluster-slots 1000 --cluster-yes 127.0.0.1:6370# 1000 indicates the number of slots allocated
Copy the code

Here is not a screenshot of the demo, leave it to your friends to operate it;

Remove nodes

  1. Nodes are fragmented first to prevent data loss, that is, slots on a specified node are allocated to other nodes. /redis-cli –cluster reshard Node IP address :port to be deleted

  2. To remove a node, you are advised to delete the secondary node first and then the primary node.

    /redis-cli --cluster del-node NODE IP address :port node ID

Cluster Configuration Items

  • Cluster-enabled: Set it to yes to enable the cluster mode.
  • Cluster-config-file (name of the configuration file corresponding to each cluster node) : the file is automatically generated and does not need to be manually created.
  • Cluster-node-timeout (specifies the maximum timeout period for which a cluster node is unavailable. If this timeout period is exceeded, the node is considered unavailable.) The default value is 15000 milliseconds.
  • Cluster-migration-barrier (minimum number of slave hosts available to a host) : The default barrier is 1, indicating that at least one slave host must be available after a slave host is migrated. Otherwise, node migration will not take place.
  • Cluster-require-full-coverage: enable cluster-require-full-coverage: enable cluster-require-full-coverage: enable cluster-require-full-coverage: enable cluster-require-full-coverage: enable cluster-require-full-coverage: enable cluster-require-full-coverage: enable cluster-require-full-coverage: yes If the parameter is set to NO, services can be provided even if slots are not fully allocated. You need to ensure slot allocation.

conclusion

To the construction of this cluster is over, originally thinking of writing very simple, did not expect to dry 4000 words; For clusters, there are some restrictions. For example, the Keys command can only be used for the current node, which needs to be processed for multiple nodes. Each node in the cluster only supports DB0 database, other databases do not support, etc. So pay attention to the use of oh, later take some time to separate a note, the length is a little long, I will not continue to talk; Next, I’ll talk about cache penetration, cache breakdown, and cache avalanche.

A handsome guy who was made ugly by the program, pay attention to “Code variety circle “, learn with me ~~~