preface

As an in-memory database, Redis is widely used in cache, distributed lock and other scenarios. If the Reids service is down due to power failure or other factors, will the data be lost after the restart?

Redis persistence mechanism

Redis is defined as an in-memory database, but it also supports data persistence. There are two persistence mechanisms in Redis: RDB persistence and AOF persistence.

RDB persistence mechanism

RDB full name:

Redis DataBase is the default persistence scheme in Redis. When a persistent condition is triggered, Redis generates a dump. RDB file by default. When Redis restarts, the dump. RDB file is parsed to restore data.

Trigger condition of RDB mechanism

The RDB persistence mechanism can be triggered in two ways: automatic and manual.

Automatic trigger

Automatic trigger modes can also be divided into three types:

1. When the flushdb command is run in flushhall, data in the dump file is empty (the dump file also stores some headers, so the file itself has content but no data), which has no actual significance.

2. When the shutdown command is run, the dump file is generated.

3. The Redis configuration file is automatically generated. The default configuration of the Redis configuration file is as follows.

Save 900 1 // at least one key is added or updated within 900 seconds save 300 10 // At least 10 keys are added or updated within 300 seconds Save 60 10000 // At least 10000 keys are added or updated within 60 secondsCopy the code

Manual trigger

In addition to automatic firing, Redis also provides two manual firing RDB commands (these two commands cannot be executed at the same time, once one command is executing, the other command will be rejected) :

1.save

This command blocks the Redis server process until the RDB file is successfully created, meaning that the server cannot process any commands sent by the client until the RDB file is generated.

2. Bgsave:

The parent process forks to create a child process. The RDB file is generated by the child process, and the parent process can normally process the commands sent by the client (here is another example of Redis being more than just a single thread).

If you want to know the last time a save or BGSave command was successfully executed, you can run the lastsave command to check, which returns a Unix timestamp.

RDB configuration files

In addition to the configuration parameters mentioned above that trigger the generation of RDB files, the RDB persistence mechanism also has the following commands:

1.dir

RDB file generation directory. The default dump directory is./ (current directory). You can run the config get dir command to check whether the current dump directory is /usr/local/redis-5.0.5/bin.

2.dbfilename

RDB file name. The default is dump.rdb.

3.rdbcompression

Whether the RDB file is an LZF compressed file. The default is yes.

4.rdbchecksum

Whether to enable data verification. The default is yes.

Advantages of RDB mechanism

1.RDB is a compact compressed file that stores files at different points in time and is suitable for DISASTER recovery and data recovery.

2.RDB maximizes Redis performance because the only job the parent needs to do is to spawn a child to do the rest and the parent never performs disk I/O or similar time-consuming operations.

3. Compared with the AOF persistence mechanism introduced later, RDB restores data faster.

Disadvantages of RDB mechanism

1.RDB cannot be backed up in real time, so if Redis stops working abnormally and does not shut down properly, the data from the last backup to the abnormal outage will be lost.

2. The RDB usually requires the parent process to fork to create child threads, so frequent forks with low CPU performance may cause the parent process to become unavailable for a short period of time.

AOF persistence mechanism

AOF, full name: Append Only File, is another persistence mechanism provided in Redis. AOF appends each write operation to a file in the form of a log. When the AOF mechanism is enabled, whenever a command is executed to change Redis data, the command is written to the AOF file. When Redis restarts, commands in the AOF file are executed to recover data according to the log content.

The main difference between AOF and RDB is that AOF records the execution of commands (similar to the statement format of MySQL binlog), while RDB records data (similar to the row format of MySQL binlog).

Note that if both RDB and AOF are enabled, Redis will preferentially select AOF persistence files for data recovery.

How to enable AOF mechanism

The AOF mechanism is disabled by default and can be modified using the following configuration file

Appendonly no// Whether to enable the AOF mechanism. By default, no indicates that the AOF mechanism is disabled. Changing the value to yes indicates that the AOF mechanism is enabledCopy the code

PS: Like the RDB mechanism, the path of the generated file is configured through the dir attribute.

AOF mechanism Whether data is written to disks in real time

The AOF mechanism controls whether data is written to disks in real time. This is similar to the redo log mechanism of MySQL.

When AOF data is written to disk is controlled by the appendfsync parameter:

AOF file overwritten

The AOF mechanism is mainly implemented by recording the execution of commands. As time goes by, the AOF file will inevitably grow larger and there may be many redundant commands. For example, if the set operation on the same key value is executed 10,000 times, the previous 9999 operations are useless to restore the data, and only the last command can be executed to restore the data. It is precisely to avoid this problem that the AOF mechanism provides the function of file rewriting.

Rewriting principle analysis

When AOF is rewritten, Redis does not analyze the original file, because if the original file is too large, analysis will be time-consuming, so Redis chooses to re-read the existing key/value pair in Redis, and then record the value of the key/value pair with a command.

Using only one command also assumes that a set key or list key or hash key cannot contain more than 64 elements, and once that number is exceeded, multiple commands are used to record.

AOF overwrites the buffer

AOF when rewriting will typically have a large number of write operation, so in order not to block the client order request, Redis will rewrite operations into to the child, but in the child process execution may cause a problem, that is rewritten during if performed again at the same time the client sending commands, and how to guarantee the consistency of the data?

To address data inconsistencies, an AOF rewrite buffer was introduced in Redis. When the AOF file is overwritten and a request is received from the client, the command is not only written to the original AOF buffer, but also written to the AOF overwrite buffer at the same time:

Once the child has finished rewriting the AOF file, it signals the parent, which blocks (without executing any commands) and does two things:

1. Refresh the files in the AOF rewrite buffer to the new AOF file.

2. Rename the new AOF file and replace the old AOF file atomically.

After the above two tasks are done, the entire AOF rewrite is complete and the parent process starts receiving commands normally.

AOF mechanism trigger condition

Triggering conditions of AOF mechanism are also divided into automatic triggering and manual triggering.

1. Automatic trigger

Automatic triggering can be set with the following parameters:

Auto-aof -rewrite-percentag // The percentage of files whose size exceeds that of the last AOF rewrite. // Set the minimum AOF file size allowed to be rewritten. The default is 64 MB. This is mainly to avoid cases where the above percentage is met but the file is still small.Copy the code

2. Trigger manually

Run the bgrewriteaof command.

Note: The bgrewriteaof command cannot be executed at the same time as the RDB persistence command bgsave. This is to avoid creating two child processes to do a lot of disk writing at the same time, which would affect Redis performance.

AOF mechanism Mechanism advantages

1. With the AOF mechanism, you can choose between different fsync policies and only lose 1s of data under the default policy.

2. The AOF log is an append only log, so if there is a power outage, there is no search or corruption problem. Even if, for some reason (the disk is full or otherwise), a half-written command in the log ends, the Redis-check-aof tool can easily fix it.

3. Redis can automatically rewrite AOF files in the background when they become too large.

4. Unlike the file format of RDB, AOF is an easy to understand and parse format that contains logs of all operations in turn.

AOF mechanism mechanism disadvantages

1. AOF files are usually larger than equivalent RDB files for the same data set.

2. The AOF mechanism may be slower than the RDB mechanism, depending on the specific strategy of fsync. But in general, fsync Settings are still very high in terms of performance per second, and when disabled, it can be as fast as RDB even under high loads.

3. Because AOF files are in the form of add-on, it may encounter the error of blocking commands such as BRPOP and LPUSH, so that the generated AOF cannot replicate the exact same data set when reloading, while RDB files are created from scratch every time, which is more robust to a certain extent.

conclusion

This paper mainly introduces two kinds of Redis persistence mechanism: RDB and AOF, and respectively introduces the principle of two kinds of persistence mechanism, through the comparison of the two kinds of persistence mechanism analyzed the advantages and disadvantages of the two kinds of persistence mechanism.

The last

I have arranged a: Redis related documents, Java systematic information, (including Java core knowledge points, interview topics and 20 years of the latest Internet real questions, e-books, etc.) need friends can pay attention to the public number can be obtained.