preface

As an in-memory database, Redis is widely used in cache, distributed lock and other scenarios. If the Reids service is down due to power failure or other factors, will the data be lost after the restart?

Redis persistence mechanism

Redis is defined as an in-memory database, but it also supports data persistence. There are two persistence mechanisms in Redis: RDB persistence and AOF persistence.

RDB persistence mechanism

Redis DataBase is the default persistence scheme in Redis. When a persistent condition is triggered, Redis generates a dump. RDB file by default. When Redis restarts, the dump. RDB file is parsed to restore data.

Trigger condition of RDB mechanism

The RDB persistence mechanism can be triggered in two ways: automatic and manual.

Automatic trigger

Automatic trigger modes can also be divided into three types:

  • performflushallCommand (flushdbCommand does not fire), but is generated at this timedumpThe data in the file is empty (dumpThe file also stores some header information, so the file itself has content, just no data), not much practical significance.
  • performshutdownCommands trigger the builddumpFile.
  • Automatically generated through configuration files,RedisThe default configuration of the configuration file is as follows. If any of the three conditions is met, the system triggers the configuration fileRedis 的RDBPersistence mechanism.
save 900 1 // At least one key is added or updated within 900 seconds
save 300 10 // At least 10 keys were added or updated within 300 seconds
save 60 10000 // At least 10000 keys were added or updated within 60 seconds
Copy the code

Manual trigger

In addition to automatic firing, Redis also provides two manual firing RDB commands (these two commands cannot be executed at the same time, once one command is executing, the other command will be rejected) :

  • save: This command blocksRedisServer process until successfully createdRDBFile, that is to say generatingRDBBefore the file, the server cannot process any commands sent by the client.
  • bgsave: The parent process will executeforkAction to create a child process.RDBThe file is generated by the child process, and the parent process normally processes the commands sent by the client (here too)RedisMore than just a single thread).

If you want to know the last time a save or BGSave command was successfully executed, you can run the lastsave command to check, which returns a Unix timestamp.

RDB configuration files

In addition to the configuration parameters mentioned above that trigger the generation of RDB files, the RDB persistence mechanism also has the following commands:

  • dir:rdbFile generation directory. The default is. /(current directory), you can run commandsconfig get dirView the current as illustrated in the figure belowdumpThe generated directory is/ usr/local/redis - 5.0.5 / bin:

  • Dbfilename: indicates the RDB filename. The default is dump.rdb.

  • Rdbcompression: Whether the RDB file is an LZF compressed file. The default is yes.

  • Rdbchecksum: indicates whether data checksum is enabled. The default is yes.

Advantages of RDB mechanism

  • RDBIs a compact compressed file that stores files at different points in time, which is ideal for DISASTER recovery and data recovery.
  • RDBTo the maximumRedisBecause of the performanceRedisThe only job the parent needs to do is to spawn a child to do the rest; the parent never executes the diskI/OOr similar time-consuming operations.
  • With the following introductionAOFComparison of persistence mechanisms,RDBMode restores data faster.

Disadvantages of RDB mechanism

  • RDBThere’s no real time backup, so ifRedisWithout proper shutdown due to an abnormal shutdown, data will be lost between the last backup and the abnormal outage.
  • 2,RDBThe parent process is usually required to executeforkActions create child threads, so if executed frequentlyforkThe operating andCPUIf the performance is not very high, the parent process may become unavailable for a short period of time.

AOF persistence mechanism

AOF, full name: Append Only File, is another persistence mechanism provided in Redis. AOF appends each write operation to a file in the form of a log. When the AOF mechanism is enabled, whenever a command is executed to change Redis data, the command is written to the AOF file. When Redis restarts, commands in the AOF file are executed to recover data according to the log content.

AOF 和 RDBThe biggest differences are:AOFRecords the execution of commands (similar toMySQL 中 binlog 的 statementFormat), whileRDBRecords are data (similar toMySQL 中 binlog 的 rowFormat).

Note that if both RDB and AOF are enabled, Redis will preferentially select AOF persistence files for data recovery.

How to enable AOF mechanism

The AOF mechanism is disabled by default and can be modified using the following configuration file

appendonly no  // Whether to enable the AOF mechanism. The default value is no, and the default value is yes
appendfilename "appendonly.aof"  / / AOF the file name
Copy the code

PS: Like the RDB mechanism, the path of the generated file is configured through the dir attribute.

AOF mechanism Whether data is written to disks in real time

The AOF mechanism controls whether data is written to disks in real time. This is similar to the redo log mechanism of MySQL.

When AOF data is written to disk is controlled by the appendfsync parameter:

appendfsync describe note
always Write to the cache while notifying the operating system to flush (fsync) to disk (although some operating systems may flush as soon as possible rather than in real time) Slow, Safest
everysec Write to cache first, then flush disk every second (default), this mode may lose 1s of data in extreme cases Compromise
no Write only to cache, and the operating system decides when to flush Faster

AOF file overwritten

The AOF mechanism is mainly implemented by recording the execution of commands. As time goes by, the AOF file will inevitably grow larger and there may be many redundant commands. For example, if the set operation on the same key value is executed 10,000 times, the previous 9999 operations are useless to restore the data, and only the last command can be executed to restore the data. It is precisely to avoid this problem that the AOF mechanism provides the function of file rewriting.

Rewriting principle analysis

When AOF is rewritten, Redis does not analyze the original file, because if the original file is too large, analysis will be time-consuming, so Redis chooses to re-read the existing key/value pair in Redis, and then record the value of the key/value pair with a command.

Using only one command also assumes that a set key or list key or hash key cannot contain more than 64 elements, and once that number is exceeded, multiple commands are used to record.

AOF overwrites the buffer

AOF when rewriting will typically have a large number of write operation, so in order not to block the client order request, Redis will rewrite operations into to the child, but in the child process execution may cause a problem, that is rewritten during if performed again at the same time the client sending commands, and how to guarantee the consistency of the data?

To address data inconsistencies, an AOF rewrite buffer was introduced in Redis. When the AOF file is overwritten and a request is received from the client, the command is not only written to the original AOF buffer, but also written to the AOF overwrite buffer at the same time:

Once the child has finished rewriting the AOF file, it signals the parent, which blocks (without executing any commands) and does two things:

  1. willAOFRewrite buffer to refresh files to newAOFWithin the file.
  2. Will the newAOFFiles are renamed and replaced atomically with old onesAOFFile.

After the above two tasks are done, the entire AOF rewrite is complete and the parent process starts receiving commands normally.

AOF mechanism trigger condition

Triggering conditions of AOF mechanism are also divided into automatic triggering and manual triggering.

  • Automatic trigger: Automatic trigger can be set by:
auto-aof-rewrite-percentag // The percentage of files whose size exceeds the last AOF rewrite. The default is 100, which means that the AOF rewrite will be triggered again after it reaches twice the size of the previous AOF rewrite
auto-aof-rewrite-min-size // Sets the minimum AOF file size allowed to be overwritten. The default is 64MB. This is mainly to avoid cases where the above percentage is met but the file is still small.
Copy the code
  • Manual trigger: ExecutebgrewriteaofCommand.

Note: The bgrewriteaof command cannot be executed at the same time as the RDB persistence command bgsave. This is to avoid creating two child processes to do a lot of disk writing at the same time, which would affect Redis performance.

AOF mechanism Mechanism advantages

  • useAOFThe mechanism is free to choose differentfsync(Flush) strategy, and at best is a loss under the default strategy1sThe data.
  • AOFThe log is an append only log, so there are no look-up or corruption issues if a power outage occurs. Even if, for some reason (the disk is full or otherwise), a half-written command in the log ends, the Redis-check-aof tool can easily fix it.
  • whenAOFWhen the file gets too big,RedisIt can be automatically rewritten in the background.
  • Different from theRDBFile format,AOFIs an easy-to-understand and parsed format that contains logs of all operations in sequence.

AOF mechanism mechanism disadvantages

  • For the same data set,AOFFiles are usually more than equivalentRDBThe file is big.
  • According to thefsyncSpecific strategies,AOFMechanism may be moreRDBMechanism is slow. But in general,fsyncSet to still high performance per second, disablefsyncAfter that, even under high load, it can match the speedRDBAs fast.
  • becauseAOFThe file is in append form and may be encounteredBRPOP,LPUSHSuch blocking command errors resulting in generationAOFThe exact same data set cannot be copied on reload, whileRDBFiles are recreated from scratch each time, to some extentRDBFiles are more robust.

conclusion

This paper mainly introduces two kinds of Redis persistence mechanism: RDB and AOF, and respectively introduces the principle of two kinds of persistence mechanism, through the comparison of the two kinds of persistence mechanism analyzed the advantages and disadvantages of the two kinds of persistence mechanism.