Redis is a key-value database server, because Redis is an in-memory database, so there are many characteristics of memory, such as power failure, or process exit, the data in the server will disappear, so you need a method to write data from memory to disk, this process is called data persistence.

There are two methods of persistence. One is RDB, which writes data from memory to disk and generates a compressed RDB file. The other is AOF, which records the commands executed by Redis line by line and appends them to a log-like file.

RDB persistence

There are two commands in Redis that can be used to generate RDB files, one is SAVE and the other is BGSAVE.

SAVE is thread blocked, and when this command is invoked, the server no longer serves until all the data in memory that needs to be stored is persisted as an RDB file.

Unlike Save, BGSave creates a child thread to persist the data, and the master server still serves it.

Either way, the bottom line is to call the same function rdb.c/rdbSave(). Here’s the pseudocode for both:

Def save(): rdbSave() def BGSAVE(): pid = fork() if pid = 0 Signal_parent () if PID > 0 # The parent process continues processing the request and waits for the child's signal through polling Handle_request_and_wait_signal () else: # Fork_error ()Copy the code

Unlike creating an RDB file, the RDB file is automatically loaded when the server starts up, so There is no command for Redis to load the RDB file. It is automatically loaded whenever the Redis server detects the RDB file at startup.

It is also worth noting that since AOF files are usually updated more frequently than RDB files, AOF files are preferred to restore database state if AOF persistence is enabled on the server. The RDB file is used by the server to restore the database state only when AOF persistence is turned off.

Because the BGSAVE command can be executed without blocking the server process, Redis allows the user to have the server automatically execute the BGSAVE command once in a while by setting the save option for the server configuration. You can also save the configuration information to the corresponding. Conf file when starting the Redis server

If the persistent configuration is as follows:

save 900 1

save 300 10

save 60 10000

The BGSAVE command is executed if any of the following three conditions are met:

The server made at least one change to the database within 900 seconds.

The server made at least 10 changes to the database within 300 seconds.

The server made at least 10,000 changes to the database within 60 seconds.

RDB file structure

The value is a string integer that records the RDB file version number. For example, “0006” indicates that the RDB file version is version 6. This chapter describes only the RDB file structure for the sixth edition.

The Databases section contains zero or any number of databases and key-value pair data from each database: If the server’s database state is empty (all databases are empty), this section is also empty and is 0 bytes long. · If the server’s database state is non-empty (at least one database is non-empty), then this section is also non-empty, and the length of this section varies depending on the number, type, and content of key/value pairs held by the database.

The EOF constant is 1 byte long. This constant marks the end of the body of the RDB file. When the reader encounters this value, it knows that all key-value pairs for all databases have been loaded.

Check_sum is an 8-byte unsigned integer that stores a checksum. The checksum is calculated based on REDIS, db_version, Databases, and EOF. When the RDB file is loaded, the server compares the checksum calculated by the loaded data with the checksum recorded by the check_sum to check for errors or damages in the RDB file.

AOF persistence

In addition to RDB persistence, Redis also provides AOF (Append Only File) persistence. Unlike RDB persistence, which records database state by saving key-value pairs in the database, AOF persistence records database state by saving write commands executed by the Redis server, as shown in the figure below:

If we execute the following write command on a blank database, the database will contain three key-value pairs:

redis> SET msg "hello" OK redis> SADD fruits "apple" "banana" "cherry" (integer) 3 redis> RPUSH numbers 128 256 512 (integer) 3
Copy the code

RDB persistence saves the key value pairs of MSG, FRUITS, and numbers in RDB file, while AOF persistence saves the database state in server

The SET, SADD, and RPUSH commands are saved to the AOF file. All commands that are written to AOF files are stored in Redis command request protocol. Because Redis command request protocol is plain text, we can just open an AOF file and see what’s inside. For example, for the three write commands previously executed, the server would produce an AOF file containing the following:

*2\r\n$6\r\nSELECT\r\n$1\r\n0\r\n *3\r\n$3\r\nSET\r\n$3\r\nmsg\r\n$5\r\nhello\r\n *5\r\n$4\r\nSADD\r\n$6\r\nfruits\r\n$5\r\napple\r\n$6\r\nbanana\r\n$6\r\ncherry\r\n *5\r\n$5\r\nRPUSH\r\n$7\r\nnumbers\r\n$3\r\n128\r\n$3\r\n256\r\n$3\r\n512\r\n
Copy the code

In this AOF file, except for the SELECT command to specify the database is automatically added by the server, all the other commands are sent by the client. When the server is started, it can restore the database state before the server is shut down by loading and executing the commands saved in the AOF file. The following is the log printed when the server loads the AOF file and restores the database state:

[8321] 05 Sep 11:58:50.448 # Server started, Redisversion 2.9.11
[8321] 05 Sep 11:58:50.449 * DB loaded from append only file: 0.000 seconds
[8321] 05 Sep 11:58:50.449 * The server is now ready to accept connections on port 6379
Copy the code

AOF persistence process

The implementation of AOF persistence can be divided into three steps: command append (append), file write and file sync.

Command to add

When AOF persistence is enabled, the server appends a write command to the end of the aOF_buf buffer of the server state in a protocol format after executing the write command

Write and synchronize files

The Redis server process is an event loop, in which file events receive and reply to client commands, and time events execute functions that need to be run periodically, such as the serverCron function.

The server calls flushAppendOnlyFile every time it finishes an event loop, because it may write to the aof_buf buffer when processing file events. To consider whether the contents of the AOF_buf buffer need to be written and saved to an AOF file, this process can be represented by the following pseudocode:

def eventLoop(): while True: # handle file events, ProcessFileEvents () # processTimeEvents() # Consider whether to add aOF_buf FlushAppendOnlyFile ()Copy the code

The behavior of the flushAppendOnlyFile function is determined by the value of the appendfsync option configured on the server. The behavior resulting from each value is shown in the following table:

If the user does not actively set a value for the appendfsync option, the default value of the appendfsync option is everysec

Problems with AOF

In order to improve the efficiency of writing files, in modern operating systems, when users call the write function to write some data to a file, the operating system usually stores the written data in a memory buffer temporarily until the buffer space is filled up or the specified time limit is exceeded. To actually write the data in the buffer to disk.

While this improves efficiency, it also poses a security problem for writing data, because if the computer goes down, the written data stored in the memory buffer will be lost.

For this purpose, the system provides two synchronization functions, fsync and fdatasync, which can force the operating system to immediately write the data in the buffer to the hard disk to ensure data writing security.

Efficiency and security of AOF persistence

The value of the appendfsync option in the server configuration directly determines the efficiency and security of AOF persistence.

When appendfsync is always, the server writes all the contents of the aOF_buf buffer to the AOF file and synchronizes the AOF file at each event loop, So always is the slowest of the three appendfsync values, but it is also the safest from a security standpoint, since AOF persistence only loses command data generated in an event loop in the event of a failed outage.

When appendfsync is everysec, the server writes all the contents of the AOF_buf buffer to the AOF file in each event loop and synchronizes the AOF file in child threads every more than one second: In terms of efficiency, Everysec mode is fast enough, and the database loses only a second of command data in the event of a downtime.

When appendfsync is no, the server writes all the contents of the AOF_buf buffer to the AOF file at each event loop, and the operating system controls when the AOF file is synchronized.

FlushAppendOnlyFile in no mode is always the fastest to write to because it does not need to be synchronized. In flushAppendOnlyFile mode, AOF files will accumulate data in the cache for a period of time. Therefore, the single synchronization time in this mode is usually the longest of the three modes: From the perspective of amortized operation, the efficiency of no mode is similar to everysec mode, in that the server using NO mode will lose all write command data since the last time the AOF file was synchronized.

AOF rewrite

AOF persistence is executed by saving write command to record the state of the database, so as the server is running the passage of time, AOF the contents of the file will be more and more, the volume of the file will be more and more big, if not controlled, the volume is too large AOF file is likely to affect Redis server, even the entire host computer, And the larger the size of AOF files, the more time it takes to restore data using AOF files.

To solve the problem of bloated AOF files, Redis provides AOF file rewrite. With this feature, the Redis server can create a new AOF file to replace the existing AOF file. The old and new AOF files hold the same database state, but the new AOF file does not contain any redundant commands that waste space, so the size of the new AOF file is usually much smaller than the size of the old AOF file.

Although Redis names the ability to replace old AOF files with new AOF files as “AOF file rewrite”, in fact, AOF file rewrite does not require any reading, analysis, or writing of existing AOF files. It is done by reading the server’s current database state.

In practice, in order to avoid client input buffer overflows when executing commands, the rewrite program checks the number of elements in the keys before processing lists, hash tables, sets, and ordered sets, which may contain multiple elements. If the number of elements exceeds the value of the redis. H /REDIS_AOF_REWRITE_ITEMS_PER_CMD constant, the rewrite program will use multiple commands to record the value of the key instead of just one command. Overrides can be handed off to child threads.