introduce

When two or more Redis instances are in active/standby mode, the cluster has high availability. When the master fails, the slave becomes the master to provide read/write services. This operation mechanism is called failover. Who will make failover decisions when master fails?

One way is to keep a daemo process that monitors all master-slave nodes, as shown in the following figure:

A Redis cluster has one master and two slaves, and this daemon monitors these three nodes. However, the availability of a single daemon cannot be guaranteed. Multiple daemons need to be introduced, as shown below:

How do I agree on whether a master is available when multiple Daemons have solved the usability problem and then have a consistency problem? For example, the two daemon1 nodes in the figure above are disconnected from the master, and the connection between daemon1 and master is smooth. Does the mater node need failover?

The Sentinel of Redis provides a set of interaction mechanism among several daemons, which form a cluster and become sentinel cluster. The daemons are also called sentinel nodes. As shown below:

These nodes communicate, elect and negotiate with each other, and show consistency in fault discovery and failover decision of master node.

The Sentinel cluster monitors any number of masters and slaves under the master and automatically upgrades the offline master from a slave under the master to a new master instead of continuing to process command requests.

Start and initialize Sentinel

A Sentinel can be started using the command:

./redis-sentinel .. /sentinel.confCopy the code

Or command:

./redis-server .. /sentinel.conf --sentinelCopy the code

When a Sentinel is started, it needs to perform the following steps:

Initializing the server

Sentinel is essentially a Redis server running in a special mode. It does not perform the same work as a regular Redis server and the initialization process is not exactly the same. For example, regular Redis server initialization loads the RDB or AOF file to recover data, but Sentinel startup does not load because Sentinel does not use a database.

Replace the code used by the normal Redis server with Sentinel specific code

Replace some of the code used by regular Redis servers with Sentinel specific code. Such as ordinary Redis server using the server. The c/redisCommandTable as server command table:

Truct redisCommand redisCommandTable[] = {{"module",moduleCommand,-2,"as",0,NULL,0,0,0}, {" get ", getCommand, 2, rF, 0, NULL, 1,1,1,0,0}, {" set ", the setCommand - 3, "wm", 0, NULL, 1,1,1,0,0}, {" setnx setnxCommand, 3, "wmF," 0, NULL, 1,1,1,0,0}, {" setex ", setexCommand, 4, "wm", 0, NULL, 1,1,1,0,0}, {" psetex ", psetexCommand, 4, "wm", 0, NULL, 1,1,1,0,0}, {" append ", appendCommand, 3, "wm", 0, NULL, 1,1,1,0,0},... {" del ", delCommand, 2, "w", 0, NULL, 1, 1,1,0,0}, {" unlink unlinkCommand, 2, "wF", 0, NULL, 1, 1,1,0,0}, {" exists ", existsCommand, - 2, rF, 0, NULL, 1, 1,1,0,0}, {" setbit ", setbitCommand, 4, "wm", 0, NULL, 1,1,1,0,0}, {" getbit getbitCommand, 3, rF, 0, NULL, 1,1,1,0,0}, {" bitfield bitfieldCommand, 2, "wm", 0, NULL, 1,1,1,0,0}, {" setrange, "setrangeCommand, 4," wm ", 0, NULL, 1,1,1,0,0}, {" getrange ", getrangeCommand, 4, "r", 0, NULL, 1,1,1,0,0}, {" substr, "getrangeCommand, 4," r ", 0, NULL, 1,1,1,0,0}, {" incr ", incrCommand, 2, wmF, 0, NULL, 1,1,1,0,0}, {" decr ", decrCommand, 2, wmF, 0, NULL, 1,1,1,0,0}, {" mget mgetCommand, - 2, rF, 0, NULL, 1, 1,1,0,0}, {" rpush rpushCommand - 3, wmF, 0, NULL, 1,1,1,0,0}, {" lpush, "lpushCommand - 3, wmF, 0, NULL, 1,1,1,0,0}... }Copy the code

Sentinel uses sentinelcmds/sentinel.c as the server list, as shown below:

Struct redisCommand sentinelcmds[] = {"ping",pingCommand,1,"",0,NULL,0,0,0}, {" sentinel ", sentinelCommand, 2, "", 0, NULL, 0,0,0,0,0}, {" subscribe", subscribeCommand, 2, "", 0, NULL, 0,0,0,0,0}, {" unsubscribe ", unsubscribeCommand, 1, "", 0, NULL, 0,0,0,0,0}, {" psubscribe psubscribeCommand, 2," ", 0, NULL, 0,0,0,0,0}, {" punsubscribe ", punsubscribeCommand, 1, "", 0, NULL, 0,0,0,0,0}, {" publish", sentinelPublishCommand, 3, "", 0, NULL, 0,0,0,0,0}, {" info ", sentinelInfoCommand, 1, "", 0, NULL, 0,0,0,0,0}, {" role", sentinelRoleCommand, 1, "l", 0, NULL, 0,0,0,0,0}, {" client, "clientCommand, - 2," rs ", 0, NULL, 0,0,0,0,0}, {" shutdown ", shutdownCommand, 1, "", 0, NULL, 0,0,0,0,0}, {" auth ", authCommand, 2, "sltF," 0, NULL, 0,0,0,0,0}}Copy the code

Example Initialize the Sentinel status

The server initializes a sentinel.c/sentinelState structure that holds all sentinel function-related state in the server.

struct sentinelState { char myid[CONFIG_RUN_ID_SIZE+1]; Uint64_t current_epoch; /* This sentinel ID. Dict *masters is a pointer to the sentinelRedisInstances structure; dict *masters is a pointer to the sentinelRedisInstances structure. /* Dictionary of master sentinelRedisInstances. Key is the instance name, Value is the sentinelRedisInstance structure pointer. */ / /* Are we in TILT mode? */ // The number of scripts currently being executed int running_scripts; /* // The time when you entered tilt mode mstime_t. Mstime_t previous_time; /* When TITL started. /* Last time we ran the time handler. */ / a FIFO queue contains all the user scripts to execute. /* Queue of user scripts to execute. */ char *announce_ip; /* IP addr that is gossiped to other sentinels if not NULL. */ int announce_port; /* Port that is gossiped to other sentinels if non zero. */ unsigned long simfailure_flags; /* Failures simulation. */ int deny_scripts_reconfig; /* Allow SENTINEL SET ... to change script paths at runtime? * /}Copy the code

Initializes the list of Sentinel’s monitoring master servers based on the given configuration file

Initialization of Sentinel status causes initialization of the Masters dictionary, while initialization of the Master dictionary is done according to the loaded Sentinel configuration file.

The key of the dictionary is to monitor the primary server name, the value of the dictionary is being monitored the primary server corresponding sentinel. C/sentinelRedisInstance structure.

Some attributes of the sentinelRedisInstance structure are as follows:

Typedeft struct sentinelRedisInstance {// Int flags; typedeft struct sentinelRedisInstance; /* See SRI_... // The name of the primary server is set by the user in the configuration file. // The name of the secondary server and Sentinel is automatically set by the Sentinel. // The format is IP :port, for example, "127.0.0.1:26379" char *name; /* Master name from the point of view of this sentinel. */ / /* Run ID of this instance, or unique ID if is a Sentinel. /* Configuration epoch. */ // Address to sentinelAddr *addr; /* Master host. */ // the value of the sentinel down-after-milliseconds option // How many milliseconds before an instance does not respond is considered subjective offline (subjectively down) down_after_period; /* Consider it down after that period. */ // Sentinel monitor <master-name> < IP > <redis-port> <quorum // Determine the number of votes required for this instance to be objective Down unsigned int quorum; /* Number of sentinels that need to agree on failure. */ //sentinel parallel-syncs <master-name> <numreplicas> Int PARALLEL_syncs; // The number of secondary servers that can be simultaneously synchronized with the new primary server during a failover operation. /* How many slaves to reconfigure at same time. */ //sentinel failover-timeout <master-name> <milliseconds Mstime_t failover_timeout; /* Max time to refresh failover state. */ }Copy the code

For example, the following configuration files are configured when Sentinel is started:

# sentinel monitor <master-name> < IP > <redis-port> <quorum> Sentinel monitor master1 127.0.0.1 6379 2 # sentinel down-after-milliseconds <master-name> <milliseconds> sentinel down-after-milliseconds master1 30000 # sentinel parallel-syncs <master-name> <numreplicas> sentinel parallel-syncs master1 1 # sentinel failover-timeout <master-name> <milliseconds> sentinel failover-timeout master1 900000Copy the code

Sentinel creates the master1 instance structure as shown in the following figure:

The mechanism of Sentinel status and Masters dictionary is as follows:

Create a network connection to the primary server

Create a network connection to the monitored master server, and Sentinel will become the client of the master server, sending commands to the master server and retrieving information from the command reply.

Sentinel creates two asynchronous network connections to the primary server:

  • Command connection, used to send commands to the master server and receive command replies
  • Subscribe to the connection, subscribe to the main server’s _sentinel_: Hello channel

Sentinel sends and retrieves information

  • Sentinel by default sends INFO commands to monitored masters and slaves via command connections every 10 seconds.

    You can obtain information about the master, including the server running ID recorded in the run_id field and the server role recorded in the role field. In addition, all slave server information, including the IP address and port number of the slave server, will be obtained. Sentinel users do not need to provide the address information of the slave server. The IP address and port number of the slave returned by the master are automatically detected by Sentinel.

    When Sentinel detects a new slave in the master, Sentinel creates a real exception for the new slave. Sentinel also creates command connections and subscription connections to the slave.

    According to the reply from the slave INFO command, Sentinel extracts the following information:

    1. Run ID run_id of slave

    2. Role of slave

    3. Master IP address and port number

    4. Connection status of master and slave master_link_status

    5. Slave_priority of slave

    6. Replication offset of slave slave_REPL_offset

  • Sentinel by default sends a message every two seconds to the _sentinel_: Hello channel of all monitored masters and slaves via a command connection

    Send commands in the following format:

     PUBLISH _sentinel_:hello   "<s_ip>,<s_port>,<s_runid>,<s_epoch>,<m_name>,<m_ip>,<m_port>,<m_epoch>"
Copy the code

Meanings of the preceding command parameters:

parameter meaning
s_ip IP address of Sentinel
s_port The port number of Sentinel
s_runid The run ID of Sentinel
s_runid The run ID of Sentinel
m_name The name of the primary server
m_ip IP address of the primary server
m_port Port number of the primary server
m_epoch Current configuration era of the primary server
  • Once Sentinel establishes a subscription connection with either master or slave, Sentinel sends a subscription to the _sentinel_: Hello channel through the subscription connection. The subscription continues until Sentinel is disconnected from the server

The command is as follows:

SUBSCRIBE sentinel:hello

As shown in the figure above, for each server connected to Sentinel, Sentinel can either send messages to the server channel _sentinel_: Hello via a command connection or receive messages from the server channel _sentinel_: Hello via a subscription connection.

  • Sentinels sense each other, and new sentinels post a message, including their own, to master’s _sentinel_: Hello channel. Other sentinels that are subscribers to that channel find new Sentinels. The new sentinel and other sentinels then create long links.

The interconnected sentinels can exchange information. The Sentinels dictionary in the Sentinel instance structure created for Master holds all Sentinel information that also monitors the master server except Sentinel itself.

Sentinel also creates instances for slave (in the dictionary of Slaves for master instances). We now also know that other sentinels instances have been created (in the Sentinels dictionary of master instances) through the exchange of sentinel information. Let’s sort out the general situation of the instance structure saved in Sentinel, as shown in the figure below:

As you can see from the figure above, the key of the Slave and Sentinel dictionaries consists of their IP address and port port in the format IP :port. The value of the dictionary is its corresponding sentinelRedisInstance instance.

Master fault discovery

Subjective unavailability

By default, Sentinel sends a PING command once per second to all the masters (including master, slave, and other sentinels) with which it has created a command connection, and determines whether the instance is online based on the PING response from the instance.

The PING command can be replied in the following two cases:

  • Valid reply: the instance returns one of the three replies: +PONG, -loading, or -masterDown

  • Invalid reply: any reply other than the valid reply above or no reply within the specified time limit

If the value of the Sentinel configuration file is set to down-after-milliseconds, the Sentinel will return an invalid response for either up-after-milliseconds or milliseconds. Turn on the SRI_S_DOWN flag in the flags attribute of this instance maintained in sentinel, for example, master as shown below:

Objective unavailability

When sentinel finds the subjective unavailability, it will send the “subjective unavailability” to other Sentinels for confirmation. When the number of sentinel nodes confirmed is greater than =quorum, the master is judged to be objectively unavailable, and then the failover process enters.

To send subjective unavailable status to other sentinels, use the following command:

SENTINEL is-master-down-by-addr <ip> <port> <current_epoch> <runid>
Copy the code

The meanings of each parameter are as follows:

  • IP: IP address of the primary server judged to be subjective offline by Sentinel
  • Port: indicates the port address of the primary server judged to be subjectively offline by Sentinel
  • Current_epoch: Configuration epoch for sentinel, used to elect the lead sentinel
  • Runid: the value can be an asterisk (*) or the running ID of Sentinel. An asterisk (*) indicates the objective offline status of the primary server. The running ID of Sentinel is used to elect the lead Sentinel

A sentinel that receives the above command returns a Multi Bulk response with three parameters:

1)

Check result of sentinel on the master: 1: the master is offline; 2: the master is offline

2)

, * means only used to check master offline status, otherwise means running ID of local lead Sentinel (election lead Sentinel)

3) < leader_EPOCH > When leader_runid is present, the leader_EPOCH is always 0. If not, it indicates the configuration epoch of the local lead Sentinel of the target Sentinel (used to elect the lead Sentinel)

Where node quantity limit quorum is configured in the Sentinel configuration file

sentinel monitor <master-name> <ip> <redis-port> <quorum>
Copy the code

The quorum option, which may be different for different Sentinel configurations.

When sentinel considers master to be objectively offline, SRI_O_DOWN flags in the master attribute will be turned on, as shown in the following figure:

The election Sentinel Leader

When a master is down, multiple Sentinel nodes may simultaneously discover and confirm each other’s “subjective unavailability state” through interaction, and simultaneously reach the “objective unavailability state” and intend to launch a failover. However, only one Sentinel node can be used as the failover initiator, so the Sentinel Leader needs to be elected and a sentinel Leader election process needs to be started.

Redis’s Sentinel mechanism implements this election algorithm using a similar Raft protocol:

1. The sentinelState epoch variable is similar to term (election round) in the RAFT protocol.

2. Each sentinel node that has confirmed that master is “objectively unavailable” broadcasts its own entry request to the environment (sentinel IS-master-down-by-addr < IP >

). ,current_epoch for its own configuration epoch, run_id for its own run ID)

3. If the sentinel node that receives the entry request has not received any other entry request, it will put the intention of this round first and reply to it (first come, first served); Leader_runid is the running ID of the source Sentinel that first received the election request. Leader_runid = leader_RUNID = Leader_RUNID = Leader_RUNID = Leader_RUNID = Leader_RUNID Leader_epoch is the configuration epoch of the source Sentinel that first received the request to initiate a campaign.)

4. If each sentinel node initiating the application receives more than half of the intention to agree to a sentinel (possibly itself), the sentinel will be determined as the leader. If the turn lasts long enough without a leader being elected, the next turn starts

After the leader Sentinel is determined, the Leader Sentinel selects a new master from all slaves of the master according to certain rules.

Failover

After the Sentinel Leader is elected, the Sentinel Leader performs failover on the offline master:

  1. The sentinel leader selects a slave with good status and complete data from all the slaves of the offline master and sends the SLAVEOF no one command to convert the slave to the master.

    Let’s see how the new master is selected. Right? The Sentinel Leader saves all slaves that have been taken offline to a list and filters them according to the following rules:

  • The slave with the highest priority is identified by replica-priority in the redis.conf configuration. The default value is 100. 0 indicates a special priority, indicating that it cannot be upgraded to master.

  • If there are multiple slaves with the same priority, the slave with the highest replication offset is selected (data is more complete).

  • If multiple slaves with the same priority and the largest replication offset exist, select the slave with the smallest running ID

After the slave is selected to be upgraded to the new master, the Sentinel Leader sends the SLAVEOF no one command to the slave. The Sentinel sends the INFO to the slave once every second (usually 10 seconds). When the role in the response changes from slave to Master, the Sentinel Leader knows that the role has been upgraded to master.

  1. The sentinel leader sends the SLAVEOF command (SLAVEOF


    ) to the slave of the offline master to copy the new master.

  2. The old master is set to be the slave of the new master, and it continues to be monitored. When it comes online again, Sentinel will execute commands to make it the slave of the new master.

conclusion

Sentinel is a highly available solution for Redis. The number of nodes in a Sentinel cluster needs to be >=3.

By default, Sentinel performs info on master and slave nodes every 10 seconds. Sentinel is used to discover master changes, master/slave relationships, and new slave nodes.

By default, sentinel exchanges information with other Sentinels by sending a message to the _sentinel_: Hello channel of all monitored masters and slaves through command connections every two seconds

By default, the sentinel sends a PING command to the master, slave, and other sentinels every second to determine whether they are offline

Sentinel leaders are elected according to certain rules.

The Sentinel Leader fails over and elects a new master to replace the offline master.

Reference books: Redis Design and Implementation, Deep Into Distributed Caching