Redis stores data in memory, and all memory data will be lost due to downtime or restart. The persistence mechanism of Redis ensures that data will not be lost due to failures. Redis provides two persistence modes. One is memory snapshot mode, which generates RDB files. RDB is a full backup of memory data at a certain point in time, and file contents are binary serialized forms with very compact storage structure. The other is AOF log backup. The log is a continuous incremental backup based on data. The log file contains the recorded text of each command executed by the server. There are advantages to both methods, and the following sections describe the implementation principles and usage techniques of both persistence mechanisms.
1. Memory snapshot
When Redis persists snapshot data, it must be able to respond to client requests without blocking online services. Snapshot persistence serializes in-memory data at a point in time and synchronizes it to a disk RDB file. How can backup data be instantly solidified in memory and never change again? How can FILE I/O operations not hinder the normal response of the server to the client? It all starts with Copy On Write.
1.1 Snapshot Principle Copy On Write
Copy On Write (COW) is also called copy-on-write. It is a policy adopted by an operating system to optimize the use of child processes.
The main way to create a process in uniX-like systems is to call the glibc function fork. Linux users know that Linux processes are created through the init process (PID =1) or its subprocess fork(vfork).
Fork () produces a child exactly the same as the parent, returning twice: the pid of the child is returned to the parent, and 0 is returned to the child. If the value is less than 0, the process failed to be created. Here is an example of C:
#include <unistd.h>
#include <stdio.h>
int main() {
pid_t pid;
int count = 0;
pid = fork();
if (pid < 0)
printf("error in fork!");
else if (pid == 0) {
printf("child process, process id is %d/n", getpid());
count++;
} else {
printf("parent process, process id is %d/n", getpid());
count++;
}
printf("count total: %d/n", count);
return 0;
}
Copy the codeThe output is:
parent process, process id is 23049
count total: 1
child process, process id is 23050
count total: 1
Copy the codeWhen the current process calls fork(), it creates a child identical to the current process (except for pid), so the child will also execute the code after fork(). The count variable of the parent process is 1, indicating that the parent process uses its own stack area (the count variable is stored in the stack area).
Let’s take a look at the flow of CPU execution.
When the CPU loads the execution program, it addresses the virtual address first, and then converts the virtual address to a physical address through the MMU(memory management unit). Because only part of the program into memory (in page load), the address is not so there will be looking for memory in the case of missing page exception) (CPU, if in the case of insufficient memory, will by paging algorithm will be came out of the memory page replacement, and then add the page in the peripheral storage into memory, make the program continues to run normally.
Each process in Linux is assigned a virtual address and a physical address. The mapping between the virtual address and physical address is maintained through the MMU. A process is a subject. It has soul and body. Soul is its virtual address space (represented by corresponding data structure), including four parts: body section, data section, heap and stack. Accordingly, the kernel assigns each of the four parts a physical block (the body of the process) : text block, data block, heap block, stack block.
Let’s take a look at the write/copy process for fork:
inforkGenerated when the child process, the operating system for new generation of the child to create a virtual space structure, copy them to the parent structure of virtual space, but not for the segment allocation of physical memory, they share the physical space of the parent, as father and son have a change in the process of the corresponding period of behavior occurs, for the child to the corresponding section of the distribution of physical space, this is the written copy.
1.2 Snapshot Execution Process
Redis calls glibc’s fork during persistence to generate a child process. Snapshot persistence is left entirely to the child process, and the parent process continues to process client requests.
The snapshot save operation can be triggered by typing bgSave on the Redis client. Redis calls bgsaveCommand, which forks a child process that shares the code and data segments in memory with its parent. It is appropriate to compare the parent-child process to a Siamese baby, which is the mechanism of the Linux operating system to share memory resources as much as possible to save memory resources. At the moment of process separation, there is little noticeable change in memory growth. Since there is no data change in the child process, the memory data it can perceive is frozen at the moment the process is generated and will never change. The parent process can continue to handle client requests, and when the child process is pushed out, the parent process calls related functions to handle the cleanup of the child process.
2. AOF persistence
The AOF log stores the sequential instruction sequence of Redis server, and only records the instruction records that modify the memory. With AOF files, it is possible to restore the state of the in-memory data structures of the current instance of Redis by executing all instructions sequentially on an empty instance of Redis, a process called replay.
2.1 AOF log file writing
AOF logs exist in the form of files, which are written to a buffer in the kernel using the write function provided by the operating system. The kernel then asynchronously calls fsync to asynchronously flush the data back to disk. The fsync function is a blocking and slow operation, and if the machine suddenly goes down, the AOF log contents may not arrive and flush completely to disk, and the data will be lost. Redis controls the frequency of fsync through the appendfsync configuration in the following three modes:
- No: Never call fsync and let the operating system decide when to synchronize the disk. This is not safe, but Redis has the best performance.
- Always: Fsync is executed once every write operation, although data security is high and execution is slow.
- Everysec: Fsync is performed every 1s. The 1s interval is configurable. This is a compromise between data security and performance, ensuring high performance while minimizing data loss. It is recommended to be used in the production environment.
2.2 AOF execution process
After Redis receives the client’s instruction, it first performs parameter verification and logical processing. If there is no problem, it determines whether to enable AOF. If yes, it synchronously writes each command into AOF_BUF after execution, which is a global SDS type buffer.
The call function is called for each command execution. Note that the Redis server executes the command before writing it to aof_buf.
2.3 Log slimming – AOF rewrite
When the Redis server is running for a long time, AOF logs become longer and longer. If the instance is down or restarted, it takes a long time to replay the entire AOF logs, and Redis cannot provide services for a long time. Therefore, you need to slim AOF logs, that is, rewrite AOF logs.
Let’s consider the loading process of AOF and RDB files :RDB only needs to load the corresponding data into memory and generate the corresponding data structure. Some structures, such as Intset and Ziplist, are directly saved in accordance with the string when saving, and the loading speed is very fast. According to tests done by Redis authors, RDB can load 1GB files in 10 ~ 20 seconds. AOF is half the speed of RDB (if AOF is overwritten, it will be faster).
The process of slimming AOF logs by using the BgrewriteAOF command on the Redis client is as follows:
The Redis server invoks the bgrewriteaofCommand command to create a pipe. The function of creating a pipe pair is to receive the commands accumulated by the server in batches during AOF rewrite. After the pipe is created,fork the AOF override by calling rewriteAppendOnlyFile. The parent process records some statistics and then continues into the main loop to process client requests. After the child process finishes, it takes care of some cleanup. The slimming work is that the child process generates a corresponding execution command for all the keys in the database, and finally rewrites the command that the parent process continues to execute after starting, and generates a new AOF file.
For example, run the following command:
127.0.0.1:6379> lpush list guo zhao ran
(integer) 3
127.0.0.1:6379> lpop list
"ran"127.0.0.1:6379 > lpop list"zhao"
127.0.0.1:6379> lpush list zhao
(integer2)Copy the codeThe AOF file will hold four commands to operate on the list, but the list now has elements in memory like this:
127.0.0.1:6379> lrange list 0-1 1)"zhao"
2) "guo"
Copy the codeAfter AOF is overwritten, the contents of the log file will be changed directly to lpush list Zhao Guo. Log slimming can reduce file size and improve loading speed.
3. Mixed persistence
RDB and AOF have their own strengths and weaknesses for persistence, so let’s summarize them briefly:
The RDB stores a snapshot of time. If a fault occurs, the data generated between the time of the last RDB execution and the time when the fault occurs is lost. If the Redis data volume is high and the QPS is high, the time required to perform an RDB will increase correspondingly and the data lost in the event of a failure will increase.
AOF stores one command after another. In theory, only one command can be lost when a fault occurs. However, because writing files is expensive in operating systems, Redis provides configuration parameters that allow you to compromise between completeness and performance and set different configuration policies. But replaying AOF logs is still much slower than using RDB.
Redis4.0 addresses this problem with a new persistence option, hybrid persistence. Mixed persistence means that during AOF rewrite, the child process saves the snapshot of the data at the current point in time in RDB file format, and then saves the accumulated commands of the parent process in AOF format. The resulting format is shown below:
Store the CONTENTS of the RDB file with the incremental AOF log file, where the AOF log is no longer the full log, which is usually a small part of the AOF log. Therefore, when Redis restarts, it can load the RDB content first, and then replay the incremental AOF log, which can completely replace the previous full AOF file replay, and the restart efficiency will be greatly improved.
4. Redis persists related configurations
Here’s a summary of the configuration related to persistence in Redis4.0 and what it means.
| Configuration items | An optional value | The default value | role |
|---|---|---|---|
| save | save <seconds> <changes> | save 900 1 save 300 10 save 60 10000 |
Save “”: Disables snapshot backup. This function is disabled by default Save 900 1: A key is changed within 900 seconds and automatically saved to dump. RDB Save 300 10: 10 keys changed within 300 seconds, automatically saved to dump. RDB file Save 60 10000: 10000 keys changed within 60 seconds, automatically saved to the dump. RDB file Saving is triggered when any of the above 3 conditions is met |
| stop-writes-on-bgsave-error | yes/no | yes | After this parameter is enabled, Redis will stop receiving write requests if RDB snapshot is enabled (that is, the save command is configured) and the last snapshot execution fails |
| rdbcompression | yes/no | yes | Whether to compress string data when performing RDB |
| rdbchecksum | yes/no | yes | Whether to enable RDB file verification |
| dbfilename | The file name | dump.rdb | RDB file name |
| dir | The file path | . / | Path for storing RDB and AOF files |
| appendonly | yes/no | no | Whether to enable the AOF function |
| appendfilename | The file name | appendonly.aof | AOF File name |
| appendfsync | always/everysec/no | everysec | Frequency of fsync execution, as described above |
| no-appendfsync-on-rewrite | yes/no | no | When this parameter is enabled, if the background is performing an RDB snapshot or aOF override, the main process does not perform fsync, even if appendsync is set to always or everysec |
| auto-aof-rewrite-percentage | The percentage | 100 | And auto-aof-rewrite-min-size, as explained below |
| auto-aof-rewrite-min-size | The file size | 64M | An AOF rewrite is triggered when the AOF file is larger than 64M and the current size of the AOF file has increased by 100% from the base size. |
| aof-load-truncated | yes/no | yes | AOF is generated in the form of appending logs, and the command may be incomplete when the server fails. When this parameter is turned on, AOF will truncate the incomplete tail of the command to continue loading in this case, and will be prompted in the log. |
| aof-use-rdb-preamble | yes/no | yes | Whether to enable mixed persistence |
| aof-rewrite-incremental-fsync | yes/no | yes | If this parameter is enabled, fsync is performed once every 32MB of data is generated during AOF rewrite |
5. To summarize
Redis is an in-memory database. After the machine fails or restarts, all the in-memory data is lost, so persistence is needed to ensure data security. Redis provides two methods of data persistence: snapshot RDB and AOF log synchronization. The implementation principle of snapshot RDB is Copy On Write. The advantage of snapshot RDB is fast machine loading speed, but the disadvantage is slow execution, and a large amount of data will be lost when QPS is high. AOF stores commands in log files one by one. The advantage is that data is lost as little as possible. The disadvantage is that the replay of log files is slow and the log files are large. Hybrid persistence is recommended in production environments, which combines the best of both RDB and AOF. The article concludes with a summary of Redis persistent configuration items. This article is a summary of the author’s study of Redis persistence, hoping to help readers.