An overview of the

Redis Sentinel provides high availability for Redis. What this means in practice is that you can use sentry mode to create a Redis deployment that can handle various failures without human intervention. Sentinel mode also provides additional functions such as monitoring, notification, and configuration for the client. Here is a list of sentinel mode’s features at the macro level:

  • Monitor: The sentry constantly checks that the master and slave are working properly.
  • Notification: The API notifies system administrators and other applications when a monitored Redis instance has a problem.
  • Automatic failover: If a master fails, the sentry can start a failover process to upgrade one slave to master, the other slaves are reconfigured to use the new master, and the application uses the new address notified by the Redis server.
  • Configuration provider: Sentry acts as the authoritative source of discovery for the Redis client: the client connects to sentry to request the address of the currently reliable master. If a failure occurs, the sentry will report the new address.

Sentinel’s distributed nature

Redis Sentry is a distributed system: the sentry itself is designed to work with multiple sentry processes. The benefits of having more than one sentry process cooperate include performing fault detection when multiple sentries agree that a master is no longer available. This reduces the chance of bad judgment. Sentinels work even when not all sentinels are working, making the system robust and resistant to failures. After all, a single point of failure makes little sense in a failing system. Redis sentinels, Redis instances (master and slave), and clients are a large distributed system with special features. In this document you will progress from the basic information needed to understand the basic nature of the sentry to the more complex information (which is optional) needed to understand how to use the sentry properly.

1. Machine planning

The cluster architecture diagram is as follows:

 

2. Cluster configuration

2.1. Redis.conf configuration

Master (192.168.50.100) The machine configuration is as follows:

Daemonize yes pidfile"/home/redis/redis/redisRun/redis_6379.pid"  
port 6379  
timeout 0  
tcp-keepalive 0  
loglevel notice  
logfile "/home/redis/redislog/redis.log"  
databases 16  
save 900 1  
save 300 10  
save 60 10000  
stop-writes-on-bgsave-error yes  
rdbcompression yes  
rdbchecksum yes  
dbfilename "dump.rdb"  
dir "/home/redis/redisdb"# If failover is performed, both the master and slave nodes must enter the same password masterauth"123456"
slave-serve-stale-data yes  
slave-read-only yes  
repl-disable-tcp-nodelay no  
slave-priority 98The current redis password requirepass"123456" 
appendonly yes  
# appendfsync always  
appendfsync everysec  
# appendfsync no  
no-appendfsync-on-rewrite no  
auto-aof-rewrite-percentage 100  
auto-aof-rewrite-min-size 64mb  
lua-time-limit 5000  
slowlog-log-slower-than 10000  
slowlog-max-len 128  
notify-keyspace-events ""  
hash-max-ziplist-entries 512  
hash-max-ziplist-value 64  
list-max-ziplist-entries 512  
list-max-ziplist-value 64  
set-max-intset-entries 512  
zset-max-ziplist-entries 128  
zset-max-ziplist-value 64  
activerehashing yes  
client-output-buffer-limit normal 0 0 0  
client-output-buffer-limit slave 256mb 64mb 60  
client-output-buffer-limit pubsub 32mb 8mb 60  
hz 10  
aof-rewrite-incremental-fsync yes  
# Generated by CONFIG REWRITE  
Copy the code

Slave (192.168.50.101-103) The configuration of the Slave is as follows:

daemonize yes  
pidfile "/home/redis/redis/redisRun/redis_6379.pid"  
port 6379  
timeout 0  
tcp-keepalive 0  
loglevel notice  
logfile "/home/redis/redislog/redis.log"  
databases 16  
save 900 1  
save 300 10  
save 60 10000  
stop-writes-on-bgsave-error yes  
rdbcompression yes  
rdbchecksum yes  
dbfilename "dump.rdb"  
dir "/home/redis/redisdb"# Master node password masterauth"123456"
slave-serve-stale-data yes  
slave-read-only yes  
repl-disable-tcp-nodelay no  
slave-priority 98  
requirepass "123456"  
appendonly yes  
# appendfsync always  
appendfsync everysec  
# appendfsync no  
no-appendfsync-on-rewrite no  
auto-aof-rewrite-percentage 100  
auto-aof-rewrite-min-size 64mb  
lua-time-limit 5000  
slowlog-log-slower-than 10000  
slowlog-max-len 128  
notify-keyspace-events ""  
hash-max-ziplist-entries 512  
hash-max-ziplist-value 64  
list-max-ziplist-entries 512  
list-max-ziplist-value 64  
set-max-intset-entries 512  
zset-max-ziplist-entries 128  
zset-max-ziplist-value 64  
activerehashing yes  
client-output-buffer-limit normal 0 0 0  
client-output-buffer-limit slave 256mb 64mb 60  
client-output-buffer-limit pubsub 32mb 8mb 60  
hz 10Aof -rewrite-incremental-fsync yes # Generated by CONFIG rewrite # slaveof192.168. 50100. 6379 
Copy the code

2.2. The sentinel.conf configuration is as follows

Master (192.168.50.100) The machine configuration is as follows:

port 26379
#1Indicates that in sentinel cluster, as long as two nodes detect the failure of the primary node of Redis, the switch will be carried out. Single Sentinel node is invalid (found by our own test). If myMaster does not respond within 3s, myMaster is considered to be down10If mysater does not come back to life after seconds, failover Sentinel Monitor MyMaster is started192.168. 50100. 6379 1
sentinel down-after-milliseconds mymaster 3000
sentinel failover-timeout mymaster 10000Daemonize yes # specify working directory dir"/home/redis/sentinel-work"
protected-mode no
logfile "/home/redis/sentinellog/sentinel.log"#redis primary password sentinel auth-pass mymaster123456
# Generated by CONFIG REWRITE
Copy the code

Slave (192.168.50.100-102) Repeat the preceding configurations

Note: File paths that do not exist in the above configuration need to be created manually. Sentinels can be configured with multiple, preferably at least three nodes, all in the same configuration.

 

3. Start the cluster

Run the following command to start the Redis node of 192.168.50.100-103:

redis-server /home/redis/redis/redis.conf
Copy the code

Start the Redis sentinel node at 192.168.50.100:

redis-sentinel /home/redis/redis/sentinel.conf
Copy the code

4. Screenshot of the effect after startup

4.1 Screenshot of Redis Sentinel

The following figure shows the startup log of a node:

The following figure shows the node startup log of a multi-node cluster:

Mysql > log in to redis for Master (192.168.50.100)

Run the following login commands:

redis-cli -h 192.168. 50100. -p 6379 -a 123456
Copy the code

Listing Master information:

info Replication
Copy the code

The effect is as follows:

4.3 Log in to Redis of Slave (192.168.50.101) to check the status of Slave:

Run the following login commands:

redis-cli -h 192.168. 50101. -p 6379 -a 123456
Copy the code

List Slave information:

info Replication
Copy the code

The effect is as follows:

4.4 Log in to Redis of Slave (192.168.50.102) to check the status of Slave:

Run the following login commands:

redis-cli -h 192.168. 50102. -p 6379 -a 123456
Copy the code

List Slave information:

info Replication
Copy the code

The effect is as follows:

4.5 Log in to Redis of Slave (192.168.50.103) to check the status of Slave:

Run the following login commands:

redis-cli -h 192.168. 50103. -p 6379 -a 123456
Copy the code

List Slave information:

info Replication
Copy the code

The effect is as follows:

5. Scenario testing

5.1 Scenario 1: The Master fails

Master-sentinel serves as the leader of master 1 and selects a slave of Master 1 as the new master. The slave is selected according to a priority for determining DNS conditions. The same priority is obtained through rUNId sorting. However, the priority setting has not been realized yet, so the slave 1 is obtained by directly obtaining RUNID sorting. Then send the slaveofno one command to cancel the slave state of slave 1 and change it to master. When other Sentinels observe that the slave has become master, they know that the error-handling routine is started. Sentinel A is then sent to the other slaves Slaveof new-slave-ip-port, and when all slaves are configured, sentinelA removes the failed master from the monitored masters list and notifies the other Sentinels.

Now 192.168.50.100:6379 is master, and 192.168.50.101:6379, 192.168.50.102:6379, and 192.168.50.103:6379 are Salves.

After the master (192.168.50.100) is disabled, the process of electing a new master is displayed.

Select 192.168.50.103:6379 as master:

5.2 Scenario 2: Master Recovery

Restart the original master and run the command on 192.168.50.100 to start the Redis node

redis-server /home/redis/redis/redis.conf
Copy the code

View the node information. The node becomes Slave

To view Sentinel log information:

The original master is automatically changed to slave but not master.

5.3 Scenario 3: Any Slave Breaks down (192.168.60.102:6379 is used as test)

Now 192.168.50.103:6379 is master and 192.168.50.100:6379, 192.168.50.101:6379 and 192.168.50.102:6379 are Salves.

Disable Slave (192.168.50.102) and view Sentinel log information:

The Slave has sdown to view the Replication information of the Master.

There are only two slaves.

5.4 Scenario 4: Savle Restart

Restart the original Savle and run the command on the machine 192.168.50.102 to start the Redis node

redis-server /home/redis/redis/redis.conf
Copy the code

To view sentinel status:

Sentinel can quickly detect the slave joining the cluster. Query the Replication information of the master: