This is the 26th day of my participation in the August Text Challenge.More challenges in August

【Redis series 】 Redis learning 11, Redis sentinel mode details and actual combat

The master-slave replication we talked about last time was built like this

The host can read and write

Slave machine can only read, not write

Think about it, so can we do the same?

Multiple Redis-Servers are connected end to end

So 6379-6380-6381 when we deploy

At this time, if the host 6379 down, will 6380 become a host?

127.0.0.1:6379 > info replication# ReplicationRole: master connected_slaves: 1 slave0: IP = 127.0.0.1 port = 6380, state = online, offset = 0, lag = 1 master_failover_state:no-failover master_replid:f1e3db9e5e438f5d98e4cad23f684b12d790ae56 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:0 second_repl_offset:-1 repl_backlog_active:1  repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:0Copy the code

We can see that the 6379 has a slave machine and the 6380 itself acts as a slave machine (although the 6380 is the host of the 6381).

When 6379 is down, 6380 is still a slave

127.0.0.1:6379 > shudown not connected >Copy the code

We manually changed the server of REIDS 6380 to the host to see if the master can be taken back after 6379 Redis-server is up

Using Slaveof No One you can set yourself as master

Start the 6379 redis server. –

Redis server/usr/local/redis/redis - 6.2.5/6379. ConfCopy the code

It was discovered that the 6380 was still the main engine and the 6379 became the commander of the light rod

In real projects, we would definitely not deploy in the way mentioned above and in the previous article, their ability to withstand risk is too low

In the actual production environment, the host is down. If the slave machine cannot become the host, it cannot write data until the host replies. This is a serious problem

Let’s take a closer look at how sentry solves the above problems

The guard mode

Mode for automatically electing master

introduce

To actively switch master:

When the host server is down, in the past, we need to manually modify a server from the machine to the host server, which requires manual processing, time-consuming and exhausting, and will cause service unavailable for a period of time, this approach is not desirable

So we have sentinel mode, sentinel mode is the sentinel architecture that redis provides starting with version 2.8 to solve this problem

Sentinel mode, can monitor the background host server failure, if there is a failure, will vote out a slave server to do the host

Sentinel mode is a special mode where Redis provides sentinel commands

Sentry is actually a separate process, and as a process, it runs independently

The idea is that sentry monitors multiple instances of Redis running by sending commands and waiting for the Redis server to respond

The actual practice

In the diagram above, sentinels serve two purposes:

  • By sending commands, the Redis server returns monitoring status information for both primary and secondary servers
  • If the sentinel detects that the master server is down, it automatically switches the slave server to master, notifies other slave servers by publishing subscriptions, and modifs the configuration file to make it the master

However, a sentinel monitoring a redis cluster is much more likely to cause problems. Therefore, our sentinels can also be clustered and each other will monitor each other, as shown in the following figure

Subjective offline

For example, if the master server is down, then one of the sentinels will detect that the system does not immediately perform a failover process, but only the current sentinels determine that the master is not available, which is subjective offline

Objective offline

When the other two sentinels also find that the master server is unavailable, there will be a vote between the sentinels. We will write the specific voting algorithm later. The voting structure is initiated by one sentry and failover is performed. Ask each monitor sentry to switch their monitoring server to the master server, which is objective offline

Let’s configure and enable a sentinel:

Also in our config file directory, which is the same directory as Redis, create a sentinel.conf file and write the configuration, which is also generated by default when redis is installed

As above, the command we are concerned with is

Sentinel Monitor MyMaster 127.0.0.1 6379 1Copy the code

Configure a sentinel to monitor the Redis cluster

Start the Sentry process

Root @ iZuf66y3tuzn4wp3h02t7pZ: / usr/local/redis/redis - 6.2.5 # redis - sentinel sentinel. Conf: 18148 X 26 Aug 2021 22:22:36. 187 X 26 Aug 2021 22:22:36.187 # Redis version=6.2.5, bits=64, commit=00000000, modified=0, pid=18148, Just started 18148:X 26 Aug 2021 22:22:36.187 # Configuration loaded 18148:X 26 Aug 2021 22:22:36.188 * Monotonic clock: POSIX clock_gettime _) _ _) - ` ` __ '- _ _. - ` ` `. ` _.' '- _ Redis 6.2.5 (00000000/0), 64 - bit. - ` `. - ` ` `. ` ` ` \ / _, _' -. _ (  ' , .-` | `, ) Running in sentinel mode |`-._`-... - ` __... -.``-._|'` _.-'| Port: 26379 | `-._ `._ / _.-' | PID: 18148 `-._ `-._ `-./ _.-' _.-' |`-._`-._ `-.__.-' _.-'_.-'| | `-._`-._ _.-'_.-' | https://redis.io `-._ `-._`-.__.-'_.-' _. - '| ` - _ ` - _ ` - __. -' _. - '_. -' | | ` - _ ` - _ _. - '_. -' | ` - _ ` - _ ` - __. - '_. -' _. - '` - _ ` -. __. -' _. - '` -. _ _. -' '-.__.-' 18148:X 26 Aug 2021 22:22:36.189 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. 18148: X 26 Aug 2021 22:22:36. 193 # Sentinel ID is 7 e01f5aa31aadb7fc54ed8ef2579c77120682dc9 18148: X 26 Aug 2021 22:22:36.193 # +monitor master mymaster 127.0.0.1 6379 quorum 1 18148:X 26 Aug 2021 22:22:36.193 * +slave slave 127.0.0.1:6380 127.0.0.1 6380 @myMaster 127.0.0.1 6379 18148:X 26 Aug 2021 22:22:36.196 * +slave slave 127.0.0.1:6381 127.0.0.1 6381 @myMaster 127.0.0.1 6379Copy the code

We can see that after the Sentry process is enabled, the Redis cluster is monitored and the MONITORED REDis IP and port are output

At this point, we check the information of a master Redis server, we can see that there are currently two slaves, now we let the host down

Looking at the sentinel log, we can see that the Sentinel process started voting for the host after 30 seconds

According to the log, we can see that after the host of 6379 went down, sentry elected the slave of 6381 as the new host, and the automatic fault recovery was successful, nice

Check whether the master process on 6381 is a host with one slave

Since the default configuration of the 6379 is the host, when you start the 6379 server again, the 6379 becomes a bare-rod commander, and the slave machine becomes zero

Root @ iZuf66y3tuzn4wp3h02t7pZ: ~ # redis server/usr/local/redis/redis - 6.2.5/6379. The conf root @ iZuf66y3tuzn4wp3h02t7pZ: ~ # Redis -cli -p 6379 127.0.0.1:6379> Info replication# Replicationrole:master connected_slaves:0 master_failover_state:no-failover master_replid:80843f8a6497705983f6463b92d71ebd451ef385 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:0 second_repl_offset:-1 repl_backlog_active:0  repl_backlog_size:1048576 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0Copy the code

The sentinel.conf configuration file contains only a few configuration items.

The sentinel mode is configured as follows:

  • port

The sentinel instance runs on port 26379 by default. If there is a sentinel cluster, we also need to configure each sentinel port

  • dir

Sentinel’s working directory

  • sentinel monitor

IP port of the redis master node monitored by Sentinel sentinel

Master-name: the master node name that can be named by itself consists of only letters a-z, digits 0-9, and periods (. – _).

How many sentinels quorum configures agree that the master master node is out of contact then the master node is objectively out of contact

  • sentine1 auth-pass

When RequirePass Foobared authorization password is enabled in a Redis instance all clients connected to the Redis instance must provide the password

Set the sentinel connection password for the primary and secondary nodes. Notice The primary and secondary nodes must have the same authentication password

  • sentinel down-after-mi 11i seconds <mi 11iseconds>

Sentine1 Specifies the number of milliseconds after which the primary node does not respond

  • sentinel paralle1-syncs

Specifies the maximum number of slaves that can synchronize the new master at the same time during a failover

The smaller the number, the longer it takes to complete the failover

However, a larger number means that more slaves are unavailable due to Replication

Setting this value to 1 ensures that only one slave at a time is unable to process command requests

  • sentinel failover-timeout

Failover -timeout can be used for the following purposes:

1. The interval between two failover operations for the same sentinel and the same master

2. Time is counted when a slave synchronizes data from an incorrect master

Until the slave is corrected to synchronize data to the correct master

3. How long it takes to cancel an ongoing failover

4. When performing failover, set the maximum time for all the slaves to point to the new master

However, even after this timeout, slaves will still be configured correctly to point to the master, but not according to the rules configured for parallel Syncs

The default time is three minutes

  • sentinel notification-script

This script is invoked when sentinel has any warning level events (such as subjective and objective failures of redis instances, etc.)

The script should then notify the system administrator of system malfunctions via email, SMS, etc

When the script is called, two arguments are passed to the script

1. Type of event

2. Description of events

If the script path is configured in the sentinel.conf configuration file, the script must exist in the path and be executable, otherwise sentinel will not start successfully

  • sentinel client-reconfig-script

This configuration is a script for the client to reconfigure the parameters of the master node

This script is called when a master changes due to a failover, notifying the client that the master address has changed

The following parameters will be passed to the script when it is called:

Is one of the Teader or Observer

Generally, failover

The from-ip, from-port, to-ip, to-port parameters are used to communicate with the old master and the new master (i.e., the old s lave)

Port 26379 dir/TMP sentinel monitor myMaster 127.0.0.1 6379 2 Sentine1 auth-pass myMaster MySUPER-- secret-0123password sentine1 down-after-mi 11iseconds mymaster 30000 sentine1 paralle1-syncs mymaster 1 sentine1 fai lover-ti meout mymaster  180000 sentine1 notificati on-script mymaster /var/redis/notify. sh sentine1 client-reconfig-script mymaster /var/redis/reconfig.shCopy the code

References:

redis_doc

Welcome to like, follow and favorites

Friends, your support and encouragement, I insist on sharing, improve the quality of the power

All right, that’s it for this time

Technology is open, our mentality, should be more open. Embrace change, live in the sun, and strive to move forward.

I am Nezha, welcome to like, see you next time ~