As an efficient caching framework, Redis is widely used. In terms of data storage, it stores data in memory during runtime to achieve efficient data read and write. According to different customized persistence rules, it will irregularly persist data to the hard disk.
In addition, compared with other NoSql databases, Redis provides very rich data structures, such as dict, SDS, LinkedList, Ziplist, set, Quicklist, geometry. Based on these storage structures, Redis provides users with a rich choice of operations, such as sorting certain types of data through Zskiplist, which is a time-consuming operation in a database.
1. Compared with other cache frameworks, redis singleton installation and redis singleton installation are very convenient. Just download redis.
Make install Make is a command in GCC. Make sure GCC is installed on your machine before installing. All commands in redis are in the SRC subdirectory of the redis installation directory, among which the most important ones are redis-server, redis-sentinel, and redis-cli.
After compiling, run /redis-server in the SRC directory to start Redis (close the window after starting), open a new window, and run /redis-cli to connect to the started Redis service. Run the following command to see that the compilation and installation are successful:
127.0.0.1:6379> set hello world
OK
127.0.0.1:6379> get hello
"world"
Copy the codeNote that if you start Redis in the above way, the IP address is 127.0.0.1, port is 6379, and the rest of the configuration is the default configuration. You can view the related configuration in the redis installation directory in the redis. If you want to start the system according to the specified configuration file, you can attach the configuration file name after redis-server, for example:
/ SRC /redis-server redis.conf In addition, if the redis cli is used to connect to the redis client without any parameters, the default IP address and port are 127.0.0.1:6379. To connect to a client with a specified IP address and port, use the following methods:
/ SRC /redis-cli -h 127.0.0.1 -p 6379 In this command, -h indicates the CONNECTED IP address, and -p indicates the connected port. After redis is configured, we can execute commands in Redis to manipulate the data.
Redis singletons provide a data caching method and rich DATA operation apis. However, there are two main problems in storing data in a single REDIS: data backup and performance degradation caused by large data volume. The master-slave mode of Redis here provides a better solution to these two problems.
Master-slave mode refers to using one Instance of Redis as the host and the remaining instances as the backup machines. The data on the master machine and slave machine are identical. The master machine supports data writing and reading, while the slave machine only supports data synchronization and reading with the host. That is to say, the client can write data to the master, and the master can automatically synchronize the data writing operation to the slave machine.
The master/slave mode solves the problem of data backup well. Since the master/slave service data is almost the same, the commands for writing data can be sent to the host for execution, while the commands for reading data can be sent to different slave machines for execution, thus achieving the purpose of read/write separation. As shown below, host Redis-A has four slave machines redis-b, redis-c, redis-d and redis-e respectively:
We have already introduced the configuration mode of REDis singleton in point 1, and we have also introduced that the master-slave mode is actually composed of multiple Redis instances. Therefore, the configuration of redis master-slave mode can be understood as multiple different Redis instances inform their master-slave relationship through certain configuration.
As described above, each Redis instance occupies a local port number. In master-slave mode, there are two main configuration points: the port number of the current instance and whether the current instance is a host or a slave, and the IP address and port of the host in slave mode. Conf file in the redis directory, save the default configuration and try not to modify it. Here we make three copies of the redis. Conf file named 6379.conf, 6380.conf and 6381.conf.
Bind 127.0.0.1 port 6379 logfile "6379.log" dbfilename "dump-6379.rdb"Copy the codeThe following is the configuration of the slave machine with ports 6380 and 6381:
Bind 127.0.0.1 port 6380 logfile "630.log" dbfilename "dump-6380. RDB "slaveof 127.0.0.1 6379 bind 127.0.0.1 port 6381 Logfile "6381.log" dbfilename "dump-6381. RDB "slaveof 127.0.0.1 6379Copy the codeYou can see that instances with ports 6380 and 6381 are configured as slave machines for instance 6379. After the configuration is complete, run the following command to start the three instances:
./src/redis-server 6379.conf
./src/redis-server 6380.conf
./src/redis-server 6381.conf
Copy the codeStart the three command line tools to connect to the redis instance by executing the following commands:
./src/redis-cli -p 6379
./src/redis-cli -p 6380
./src/redis-cli -p 6381
Copy the codeExecute a get command in each of the three command line tools to get the data with the key name MSG, as shown below:
127.0.0.1:6379> get msg
(nil)
127.0.0.1:6380> get msg
(nil)
127.0.0.1:6381> get msg
(nil)
Copy the codeAs you can see, there is no data with key MSG in any of the three redis instances. Now let’s set a data with key MSG on host 6379 as follows:
127.0.0.1:6379> set msg "hello"
OK
Copy the codeGet MSG on instance 6380 and instance 6381 as follows:
127.0.0.1:6380> get msg
"hello"
127.0.0.1:6381> get msg
"hello"
Copy the codeIt can be seen that although we only set MSG data on instance 6379, corresponding data also exists on instance 6380 and instance 6381, indicating that we have successfully configured the master-slave mode of REDis. In addition, if you do not specify the relationship between master and slave nodes in the configuration file, you can also use the slaveof command after starting the relevant Redis instance to specify that the current node is called a slave node of a node, for example:
127.0.0.1:6380> slaveof 127.0.0.1 6379 3. Sentinel configuration in RedisCopy the codeThe Redis master-slave pattern solves potential performance problems with data backup and singletons, but it also introduces new problems. Due to the configuration of master-slave mode for three redis instance, and each instance using different IP (if on different machines) and port number, according to the stated earlier, master-slave mode can be read and write operations assigned to different instances so as to achieve the purpose of improve the system throughput, but also because the use of inconvenience caused by this way Because each client is assigned the IP and port number when connecting to the Redis instance, if the connected Redis instance is offline due to failure, and the master/slave mode does not provide a certain means to notify the client of another accessible client address, so you need to manually change the client configuration to reconnect. In addition, in master-slave mode, if the master node goes offline due to a failure, the slave node synchronizes because there is no master node, and manual failover is required.
To address these two issues, the Sentinel architecture was officially provided by Redis after version 2.8. Here are a few concepts to illustrate with sentinel:
Each Sentinel node is actually a Redis instance. Different from the primary and secondary nodes, the Sentinel node is used to monitor the Redis data nodes, while the sentinel node set represents the collection of multiple Sentinel monitoring nodes monitoring a group of primary and secondary Redis instances. For example, there are master nodes and slave nodes slave-1 and slave-2. To monitor these three nodes, N sentinel nodes are configured. Sentinel-1, sentinel-2… Sentinel – N. The following figure is an example of sentinel monitoring of primary and secondary nodes:
As you can see from the figure, sentinel is just an additional set of Redis instances for monitoring outside of a set of master and slave nodes. After the master and slave nodes and sentinel node set are configured, the Sentinel nodes send messages to each other to check whether the other Sentinel nodes are working properly, and the Sentinel nodes also send messages to the master and slave nodes to check whether the monitored master and slave nodes are working properly.
As mentioned above, the main function of Sentinel architecture is to solve the failover work of the master node in master-slave mode. Here, if the primary node is offline due to failure, when a Sentinel node sends detection messages to the primary node, if no reply is received within a specified time, the Sentinel will subjectively judge that the primary node is offline and send messages to other Sentinel nodes. It is asked if it “thinks” that the primary node is offline. Other Sentinels receive the message and send a detection message to the primary node.
If it thinks the primary node is offline, it will reply to the sentinel node asking that it also thinks the primary node is offline. When the sentinel node first receives more than the specified number (the number configured in the configuration file and half of the current sentinel node set, The sentinel node replies that the current primary node is offline, then it will perform failover on the primary node. The basic idea of failover is to select a slave node from the nodes and send slaveof no one to it (assuming that the selected slave node is 127.0.0.1:6380). Make it an independent node (that is, the new master node), and then Sentinel sends the slaveof 127.0.0.1 6380 command to the remaining slave nodes to make them become slave again to the new master node.
After reallocation, the Sentinel node set will continue to monitor the primary node that has gone offline (let’s say 127.0.0.1:6379). If it comes back online, Sentinel will send it the Slaveof command, making it a slave node for the new host point, and the failover is complete.
As mentioned above, each Sentinel node is essentially a Redis instance, but unlike the Redis data node, its main function is to monitor the Redis data node. There is a default sentinel configuration file in the redis installation directory, sentinel.conf. Conf, sentinel-26380.conf and sentinel-26381.conf. Edit the three configuration files as follows:
Port 26379 daemonize yes logfile "26379.log" dir /opt/soft/redis/ Data Sentinel monitor myMaster 127.0.0.1 6379 2 sentinel down-after-milliseconds mymaster 30000 sentinel parallel-syncs mymaster 1 sentinel failover-timeout mymaster 180000 sentinel myid mm55d2d712b1f3f312b637f9b546f00cdcedc787Copy the codeFor sentinel ports 26380 and 26381, the configuration is similar. You only need to change the corresponding port number to the corresponding port number. Two points to note here:
The myID parameter of each sentinel should also be modified, because this attribute is used to uniquely distinguish sentinel nodes from other sentinel nodes.
Sentinel monitor myMaster 127.0.0.1 6379 2 Sentinel monitor myMaster 127.0.0.1 6379 2 Sentinel monitor myMaster 127.0.0.1 6379 2
After the configuration is complete, we first start the three primary and secondary nodes and then use three configuration files respectively to enable sentinel with the following command:
./src/redis-sentinel sentinel-26379.conf
./src/redis-sentinel sentinel-26380.conf
./src/redis-sentinel sentinel-26381.conf
Copy the codeSince the sentinel node is also an instance of Redis, we can use redis-cli to connect to the sentinel node:
/ SRC /redis-cli -p 26379 After being connected to the sentinel node, you can run the following command to check the sentinel status:
127.0.0.1:26379> Info Sentinel result:
# Sentinel sentinel_masters:1 sentinel_tilt:0 sentinel_running_scripts:0 sentinel_scripts_queue_length:0 Sentinel_simulate_failure_flags: 0 master0: name = mymaster, status = ok, address = 127.0.0.1:6379, slaves = 2, sentinels = 3Copy the codeAs you can see, Sentinel detected a total of three master and slave nodes, one master and two slave nodes, and a total of three Sentinel nodes. Once the startup is complete, we can simulate the sentinel failover process by actively offline the primary node. First, we connect the primary node with port 6379 and run the following command to check the status of the primary and secondary nodes:
127.0.0.1:6379> Info replication Results are as follows:
Replication role:master Connected_SLAVES :2 Slave0: IP =127.0.0.1,port=6380,state=online,offset=45616,lag=1 Slave1: IP = 127.0.0.1 port = 6381, state = online, offset = 45616, lag = 1 master_repl_offset: 45616 repl_backlog_active: 1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:2 repl_backlog_histlen:45615Copy the codeYou can see that the current master node has two slave nodes with ports 6380 and 6381 respectively. We then execute the following command on the master node:
127.0.0.1:6379> shutdown save then connect to the secondary node whose upper number is 6380 and run the following command:
127.0.0.1:6380> Info replication Results are as follows:
Replication role:master Connected_SLAVES :1 Slave0: IP =127.0.0.1,port=6381,state=online,offset=12344,lag=0 master_repl_offset:12477 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:2 repl_backlog_histlen:12476Copy the codeAs you can see, when the instance of port 6379 goes offline, the instance of port 6380 is reelected as the new primary node, and the instance of port 6381 is set as the secondary node of the instance of 6380. If we re-enable the node with port 6379 and then check the primary/secondary status, the result is as follows:
Replication role:master Connected_SLAVES :2 Slave0: IP =127.0.0.1,port=6381,state=online,offset=59918,lag=0 Slave1: IP = 127.0.0.1 port = 6379, state = online, offset = 59918, lag = 1 master_repl_offset: 60051 repl_backlog_active: 1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:2 repl_backlog_histlen:60050Copy the codeAs you can see, after the redis instance with port 6379 is reconnected, the Sentinel node detects that it is reconnected and sends a command to make it a slave node of the new master node.
4. Redis cluster configuration Redis cluster is a function introduced in Redis 3.0, which effectively solves the requirements of Redis in the distributed aspect. When encountering problems such as single memory, concurrency and flow bottleneck, Cluster solution can be used to achieve the purpose of load balancing. On the other hand, Sentinel in Redis effectively solves the problem of failover, and also solves the problem that the offline client of the primary node cannot identify the new available node. However, sentinel does not failover the offline client of the secondary node. And the client connected to the slave node can not get the new available slave node, and these problems have been effectively solved in the Cluster.
Data in the REDis cluster is linked to slots, which define a total of 16,384 slots. All data will be mapped to a slot in the 16,384 slots according to the consistent hash algorithm. On the other hand, the 16,384 slots are assigned to different Redis nodes according to their Settings.
For example, three instances of Redis are started: Cluster-a, cluster-b, and cluster-c, where slot 0-5460 is assigned to cluster-A, slot 5461-10922 is assigned to cluster-B, and slot 10923-16383 is assigned to cluster-C. But the label is like an array subscript, from 0 to 16383). In other words, data storage is only related to slots, and the number of slots is certain. Because the consistent hash algorithm is certain, the 16384 slots allocated to no matter how many Redis instances, the confirmed data will be allocated to certain slots. Redis clustering is used in this way to achieve the efficiency and high availability of Redis.
One thing to note here is that the consistent hash algorithm has a very strong relationship with the number of nodes used to calculate the mapping position based on the key value of the data. The idea of consistent hash partition is to allocate a token to each node in the system. The token generally ranges from 0 to 2^32. These tokens form a hash ring. The data that needs to be manipulated is stored on this node. Through analysis, it can be found that consistent hash partition has the following problems:
Adding and subtracting nodes will cause some data in the hash ring to fail to be matched, which needs to be manually processed or ignored.
When a small number of nodes are used, node changes will greatly affect the data mapping in the ring, so this method is not suitable for the distributed scheme with a small number of nodes.
Normal consistent hash partitions need to double or subtract half of the nodes when adding or removing nodes to ensure data and load balance.
Because of these problems of consistent hash partition, Redis uses virtual slots to deal with the problem of partition point change, and maps all data to 16384 virtual slots. When the Redis node changes, the slot of data mapping will not change, and this is the basis of node expansion of Redis.
For the configuration of the Redis cluster, first make six copies of the redis. Conf file in the redis installation directory, named as: Conf, cluster-6380.conf, cluster-6381.conf, cluster-6382.conf, cluster-6383.conf, cluster-6384.conf, cluster-6384.conf, cluster-6384.conf, cluster-6384.conf, cluster-6384.conf For a highly available cluster scenario, each node in the cluster will be assigned a slave node to prevent data nodes from going offline due to failure. Here, six redis instances are defined using six configuration files, with three as master nodes and the remaining three as slave nodes. For one of the six configuration files, the following are the parameters that need to be modified:
port 6379
cluster-enabled yes
cluster-node-timeout 15000
cluster-config-file "nodes-6379.conf"
pidfile /var/run/redis_6379.pid
logfile "cluster-6379.log"
dbfilename dump-cluster-6379.rdb
appendfilename "appendonly-cluster-6379.aof"
Copy the codeFor other configuration files, you only need to change the port number of the corresponding item and the file name with the port number to the current port number and file name of the port number. After the configuration file is configured, start each instance in the cluster with the following command:
./src/redis-server cluster-6379.conf
./src/redis-server cluster-6380.conf
./src/redis-server cluster-6381.conf
./src/redis-server cluster-6382.conf
./src/redis-server cluster-6383.conf
./src/redis-server cluster-6384.conf
Copy the codeAfter reading the configuration file, you can find that the primary/secondary relationship of the six instances is not specified or 16384 slots are not allocated during the current configuration and startup. There are two ways to allocate slots and set the master-slave relationship. One is to use Redis – CLI to connect to the cluster node and then use cluster meet command to connect to other nodes. For example, we first run the following command to connect to the node of port 6379:
/ SRC /redis-cli -p 6379 Run the cluster meet command to connect to other nodes.
127.0.0.1:6379>cluster meet 127.0.0.1 6380 127.0.0.1:6379>cluster meet 127.0.0.1 6381 127.0.0.1:6379>cluster meet 127.0.0.1 6379>cluster meet 127.0.0.1 6379 127.0.0.1 6382 127.0.0.1:6379>cluster meet 127.0.0.1 6383 127.0.0.1:6379>cluster meet 127.0.0.1 6384Copy the codeAfter the connection is complete, you can run the cluster Nodes command to view the cluster status:
127.0.0.1:6379 > cluster nodes 4 fa7eac4080f0b667ffeab9b87841da49b84a6e4 127.0.0.1:6384 master - 0, 1468073975551 Connected cfb28ef1deee4e0fa78da86abe5d24566744411e 127.0.0.1:6379 myself, master - 0 0 0 connected Be9485a6a729fc98c5151374bc30277e89a461d8 127.0.0.1:6383 master - 0 1468073978579 4 connected 40622 f9e7adc8ebd77fca0de9edfe691cb8a74fb 127.0.0.1: master 6382-0 1468073980598 3 connected 8 e41673d59c9568aa9d29fb174ce733345b3e8f1 127.0.0.1:6380 master 1468073974541-0 1 connected 40 b8d09d44294d2e23c7c768efc8fcd153446746 127.0.0.1: master 6381-0 1468073979589 2 connectedCopy the codeYou can see that the configured six nodes have been added to the cluster, but they are not available yet because 16,384 slots have not been allocated to the cluster nodes. You can use redis-cli to connect to ports 6379,6380 and 6381 respectively and run the following commands respectively to allocate virtual slots:
127.0.0.1:6379 > cluster addslots {0... 5461} 127.0.0.1:6380 > cluster addslots {5462... 10922} 127.0.0.1:6381 > cluster addslots {10923... 16383}Copy the codeAfter a slot is added, you can run the cluster info command to view the cluster status.
127.0.0.1:6379> cluster info Cluster_state: OK Cluster_SLOts_assigned :16384 Cluster_SLOts_OK :16384 Cluster_SLOts_pfail :0 cluster_slots_fail:0 cluster_known_nodes:6 cluster_size:3 cluster_current_epoch:5 cluster_my_epoch:0 cluster_stats_messages_sent:4874 cluster_stats_messages_received:4726Copy the codeHere, 16,384 virtual slots are allocated to three nodes, and the remaining three nodes are configured as secondary nodes of the three nodes by using the following command to achieve the purpose of high availability:
127.0.0.1:6382 > cluster replicate cfb28ef1deee4e0fa78da86abe5d24566744411e OK 127.0.0.1:6383 > cluster replicate 8 e41673d59c9568aa9d29fb174ce733345b3e8f1 OK 127.0.0.1:6384 > cluster replicate 40 b8d09d44294d2e23c7c768efc8fcd153446746 OKCopy the codeAt this point, all cluster nodes are configured and available. To view the status of the current node, run the cluster Nodes command:
127.0.0.1:6379 > cluster nodes 4 fa7eac4080f0b667ffeab9b87841da49b84a6e4 127.0.0.1:6384 slave 40b8d09d44294d2e23c7c768efc8fcd153446746 0 1468076865939 5 connected cfb28ef1deee4e0fa78da86abe5d24566744411e 127.0.0.1:6379 myself, master - 0 0 0 connected 0-5461 be9485a6a729fc98c5151374bc30277e89a461d8 127.0.0.1:6383 slave 8e41673d59c9568aa9d29fb174ce733345b3e8f1 0 1468076868966 4 connected 40622f9e7adc8ebd77fca0de9edfe691cb8a74fb 127.0.0.1:6382 slave cfb28ef1deee4e0fa78da86abe5d24566744411e 0 1468076869976 3 connected 8 e41673d59c9568aa9d29fb174ce733345b3e8f1 127.0.0.1:6380 master 1468076870987-0 1 connected. 5462-10922 40 b8d09d44294d2e23c7c768efc8fcd153446746 127.0.0.1: master 6381-0 1468076867957 2 connected. 10923-16383Copy the codeWe use redis-cli to connect to the cluster using the following command:
/ SRC /redis-cli -c -p 6380 NOTICE When connecting to the redis instance in cluster mode, you need to add parameter -c to indicate that the redis instance in cluster mode is connected. After connecting, run the get command:
127.0.0.1:6380> get hello
-> Redirected to slot [866] located at 127.0.0.1:6379
(nil)
Copy the codeAs you can see, the get command on port 6380 examples, the first will be for the current key through consistent hashing algorithm calculating the slot, and concludes that the slot is not in the current redis instance, thus is redirected to the target instance to execute the commands, finally found corresponding value without the key, and returns a (nil).
