Concern public number: XY’s technical circle

As we all know, Redis is an in-memory database. But one big difference between Redis and other in-memory databases such as memcache is that Redis can persist to disk. With persistence, Redis can back up, restore, and copy data.

Redis provides two persistence schemes: RDB and AOF. In Redis 4.0, a new feature is available: hybrid persistence of both. The principles and configuration of Redis’s various persistence schemes are described below.

Use the Info Persistence command to view all current persistence information:

RDB

The principle of

RDB persistence is done by means of snapshots. When the trigger condition is reached, Redis automatically generates a binary copy of all data in memory and stores it on the hard disk.

In the configuration file, you can configure the backup file and directory of the current configuration. You can also run the config command to view and set the backup file and directory:

The trigger condition

RDB is divided into active trigger and passive trigger.

Active triggering means that save and BGSave commands are persisted by the client.

Executing save blocks Redis from responding to any requests from other clients until the RDB snapshot file has been executed and should be used with caution.

Bgsave refers to background Save. When the BGSave command is executed, Redis forks a child process to perform the snapshot operation. Note that Redis blocks during the fork of the child process. When the child process is created, Redis can continue to respond to client requests.

After the child process is created, return to “Background Saving Started”. The child process creates a temporary snapshot file from the memory copy of the main process and replaces the original snapshot file when the snapshot file is complete. After the replacement is complete, the child process sends a signal to the main process to complete the snapshot operation, the main process updates the statistics (info Persistence can be seen), and the child process exits.

There are several cases of passive triggering, which are described below.

Save M N The rule is triggered

Bgsave is automatically triggered if n keys change in Redis within the specified m seconds. This rule is configured in redis.conf by default, and can be used in combination. If one of the rules is met, BGSave is triggered

For example, save 900 1 indicates that Redis triggers the BGSave operation when at least one key changes within 900 seconds.

Flushall trigger

Flushall is used to flush the database. Use it with caution. When flushhall is used, it indicates that we need to flush data.

Shutdown the trigger

Redis saves all data at a secure Angle before shutting down, so that it can be restored by the next startup. You can either use the shutdown command using the client connection or use the script directly to shutdown Redis, executing save before exiting.

The shutdown command can also pass a parameter save/nosave. If the nosave parameter is used, it is not persisted and exits directly.

The primary and secondary replication is triggered

In Redis master-slave replication, the slave node performs a full copy, and the master node executes the BGsave command and sends the RDB file to the slave node.

Data recovery

When Redis crashes or restarts unexpectedly and AOF persistence is not enabled (it is not enabled by default), data will be recovered using the RDB snapshot file. Stop the Redis service and restart it. You will find the following logs:

DB loaded from disk

configuration

AOF

By default, Redis disables AOF persistence, enabling AOF through appendonly for yes, modifying the configuration file or using config set directly on the command line, and then using config rewrite to synchronize to the configuration file. The advantage of client modification is that AOF persistence takes effect directly without restarting Redis.

The principle of

If an RDB is equivalent to a timed backup of a database (cold backup), an AOF is equivalent to a hot backup of a database.

It can be found from the previous introduction that if RDB is used, some data will be lost in unexpected circumstances, such as Redis service down suddenly, before some data in memory is refreshed into disk.

AOF is designed to solve this problem. AOF is short for Append Only File. AOF appends every write command executed by Redis to a disk File. When Redis is started, data is recovered from AOF File first.

Since every write operation needs to be recorded in a file, enabling AOF persistence will have some impact on performance. However, in most cases, the impact is acceptable. We can improve AOF performance by using hard disks with high read/write rates. Compared to RDB persistence, AOF persistence loses less data and consumes less memory (RDB mode bGSVE has memory copies).

AOF implementation is based on Redis communication protocol, the command is written in plain text to the file.

Redis protocol:

First, Redis is divided into lines, each line ending with line \r\n. Each line has a header, which is divided into five types as follows:

+ indicates a correct status information, which is the character following the current line.

– Indicates an error message, which is the character following the current line.

* indicates the total number of lines in the message body, excluding the current line, followed by the specific number of lines.

$indicates the length of the next line, excluding the newline length \r\n, followed by the corresponding length of the data.

: returns a numeric value followed by the corresponding numeric field character.

Redis AOF persistence process

1 Append Write

Redis adds each write command to the buffer aOF_buf using Redis communication protocol. The advantage of this is that in the case of a large number of write requests, some commands are temporarily stored in the buffer and then written to the disk at a time according to the policy. In this way, disk I/O times are reduced and performance is improved.

2 Synchronize commands to hard disks

When a write command writes to the aof_buf buffer, Redis writes the commands in the buffer to a file. Redis provides three synchronization policies determined by the appendfsync configuration parameter.

  • No: do not use the fsync method for synchronization, but hand over the write function of the operating system to perform synchronization. In Linux, the buffer is flushed every 30 seconds.
  • Always: the fsync method is called to force the kernel to write data to the AOF file every time there is a write operation.
  • Everysec: Data will be written to files using a call to the operating system write and flushed from the kernel to disk using fsync once per second. This is a compromise between performance and data security, so Redis recommends using this configuration by default.

3 File Rewriting (Bgrewriteaof)

When AOF is enabled, AOF files will become bigger and bigger as time goes by. Of course, Redis also optimizes AOF files, that is, when the AOF file rewriting condition is triggered, Redis will use BgreWriteAOF to rewrite AOF files. This has the benefit of reducing the AOF file size and facilitating data recovery. Rewrite strategy:

  • Duplicate or invalid commands are not written to the file
  • Expired data is no longer written to the file
  • Multiple commands combined write (when multiple commands can be combined with a single command, they are optimized to be combined as a single command write, for example, “RPUSH list1 a; RPUSH list1 b” merged into “RPUSH list1 a b”)

rewrite

The process of AOF file rewriting is similar to that of RDB snapshot BGSave. The process of AOF file rewriting is similar to that of RDB snapshot BGSave. The process of AOF file rewriting is similar to RDB snapshot BGSAVE.

Process description:

1. Start bgreWriteAOF and check whether the BGSave (RDB persistence)/ bgreWriteAOF command is running. If yes, run the command after it is executed.

2. The main process forks a child process, during which time Redis is blocked.

The main process forks the child process and continues to receive requests. All write commands are still written to the AOF file buffer and synchronized to disk according to the appendfsync policy to ensure that the original AOF file is complete and correct. Since the child process of fork only shares the memory of the main process at fork, Redis uses a rewrite buffer (AOF_rewrite_buf) mechanism to save client write requests after fork, preventing this data from being lost during the generation of new AOF files. At this point, the client’s write request not only writes to the original AOF_buf buffer, but also to the rewrite buffer (aof_rewrite_buf).

4. The child process writes to the new AOF file based on the command rewrite policy through the memory snapshot.

4.1 After the child process writes a new AOF file, it sends a signal to the main process, and the parent process updates the statistics.

4.2 The main process writes data in aOF_rewrite_buf to the new AOF file.

5. Use the new AOF file to overwrite the old AOF file to mark the completion of the AOF rewrite.

Overriding trigger conditions:

AOF file triggering conditions can be divided into manual triggering and automatic triggering.

Manual triggering: The client runs the bgrewriteaof command.

Automatic triggering: The automatic triggering takes effect in collaboration with the following configurations:

  • Auto-aof -rewrite-min-size: indicates the minimum size of an aof file to be rewritten. The aof file can be rewritten only when the aof file size is larger than this value. The default value for 4.0 is 64mb.
  • Auto-aof -rewrite-percentage: The ratio between the current AOF file size and the size since the last aOF rewrite is equal to or equal to a specified percentage increase. For example, 100 indicates that the current AOF file is rewritten only when the current AOF file is twice the size of the last aOF rewrite.

Redis enabled With AOF enabled, the following three variables are maintained

  • The variable aof_current_size that records the current AOF file size.
  • Record the AOF file size variable aof_rewrite_base_size after the last AOF rewrite.
  • Increment percentage variable aof_rewrite_perc.

Each time the serverCron function executes, it checks if all of the following conditions are met, and if all of them are met, it triggers an automatic AOF override:

  • No BGSAVE command (RDB persistence) /AOF persistence executing;
  • No BGREWRITEAOF going on;
  • The current AOF file size is larger than the value of server.aof_rewrite_min_size.
  • The ratio between the current AOF file size and the size since the last rewrite is equal to or greater than the specified percentage of growth (the auto-aof-rewrite-percentage argument)

Data recovery

When AOF is enabled, Redis data recovery preferentially selects AOF for data recovery. Stop the Redis service and restart it. You will find the following logs:

DB loaded from append only file

configuration

RDB versus AOF

Advantages of RDB:

RDB files are small in size and therefore fast in transmission, making them suitable for disaster recovery. RDB enables Redis to perform better, and child processes handle the save work. It is faster than AOF in recovering large data sets.

Advantages of AOF:

Data is more complete and secure, with second data loss (depending on the appendfsync policy). High compatibility, because it is a plaintext file based on Redis communication protocol, so it is easy to read, and compatible with any version of Redis.

Mixed persistence

Mixed persistence is a Redis 4.0 feature.

Mixed persistence is also done through bgreWriteaof. The difference is that when mixed persistence is enabled, the child process forked out first writes the shared memory copy to the AOF file in RDB mode, and then writes the increment command to rewrite the buffer to the FILE in AOF mode. After writing, the main process is notified to update the statistics. The new AOF file replaces the old AOF file.

Simply put: The first half of the new AOF file is full data in RDB format, and the second half is incremental data in AOF format.

Data recovery

With mixed persistence, Redis still loads AOF files first. There are two possible scenarios:

  • The AOF file starts with the RDB format, loading the CONTENT of the RDB first, and then loading the rest of the AOF
  • AOF file starts not in RDB format, loads the entire AOF file directly

configuration

In version 4.0, mixed persistence is disabled by default. You can use the aof-use-rdb-preamble configuration parameter to control mixed persistence. Yes indicates that mixed persistence is enabled, and no indicates that mixed persistence is disabled.

The hybrid mode can combine the advantages of AOF and RDB for fast loading without losing too much data.

Write articles carefully and share with your heart.

Personal website: Yasinshaw.com

Public number: XY’s technical circle