“This is the 12th day of my participation in the Gwen Challenge in November. See details of the event: The Last Gwen Challenge 2021”.
concept
Redis supports two persistence mechanisms, RDB and AOF. The persistence function effectively avoids data loss caused by process exit, and the data can be recovered using the previous persistent file when the next restart
RDB
Redis provides THE RDB persistence function. RDB persistence is the process of generating snapshots of Redis data in memory and saving them to hard disks. RDB persistence can be triggered manually or automatically.
RDB persistence The GENERATED RDB file is a compressed binary file that can be used to restore the complete data of the database when RDB is generated to implement data persistence and avoid data loss.
RDB file creation and loading
Basic concepts of RDB file creation
There are two Redis commands that can be used to generate RDB files, one is SAVE and the other is BGSAVE.
The SAVE command blocks the Redis server process until the RDB file is created, and the server cannot process any command requests while the server process is blocked.
When the SAVE command is being executed, all command requests sent by the client are rejected. The command sent by the client is processed only after the server executes the SAVE command and starts accepting command requests again.
The BGSAVE command does not block the server process. The BGSAVE command gives birth to a child process that creates the RDB file. The server process continues to process command requests.
Creating an RDB file is actually done by the rdbsave function, which is called differently by the save and bgSave commands.
File is loaded
The RDB file load is automatically performed when the server is started, and it is automatically loaded into the RDB file whenever the Redis server detects the RDB file at startup. Let’s take a look at the log file printed when Redis starts
DB loaded from disk
It is important to note that AOF files are usually updated more frequently than RDB files, so if AOF persistence is enabled, the server will use AOF files to restore data preferentially, and only RDB files will be used to restore data when AOF is disabled.
Loading the RDB file is actually done by the rdbload function.
The server blocks while loading the RDB file until the load is complete.
Automatically saved
Because the BGSAVE command does not block the main server process, Redis allows the user to have the server automatically execute the BGSAVE command at regular intervals by setting the save option for the server configuration.
-
The server made at least one change to the database within 900 seconds.
-
The server made at least 10 changes to the database within 300 seconds.
-
The server made at least 10,000 changes to the database within 60 seconds.
As long as any of these three conditions are met, the BGSAVE command is executed.
If the user does not actively set the Save option, the server sets default conditions for the save option, as shown above.
Automatically saved
The server sets the saveParams property of the server state redisServer structure based on the save conditions set by the Save option
The saveParams property is an array, and each element in the array is a SaveParam structure, each of which holds a saveparam condition set by the Save option.
In addition to the SaveParams array, the server state maintains a dirty counter and a LastSave attribute.
The dirty counter is used to record how many changes the server has made to the database state since the last SAVE or BGSAVE command was successfully executed.
The lastSave attribute is a UNIX timeline that records the last time the server successfully executed the SAVE or BGSAVE command.
Each time the server successfully executes a database change command, the dirty counter is updated and the database dirty value increases as the number of changes is made.
Redis’s server periodic operation function serveCron is executed every 100 milliseconds by default, checking whether the save criteria set by the save option have been met, and executing the BGSAVE command if so.
RDB file structure
The RDB file is basically made up of these five parts.
-
REDIS: The RDB file begins with the REDIS section, which holds the five characters “REDIS”. With these 5 characters, the program can quickly check whether the file loaded is an RDB file when loading the file.
-
Db_vsersion is 4 bytes long and its value is an integer represented as a string. Records RDB files to version numbers.
-
Databases: Contains zero or more databases and key-value pairs in each database.
-
The EOF constant is 1 byte long. This constant marks the end of the body of the RDB file. When the program reads this value, it means that all key-value pairs have been loaded.
-
Check_sum is an 8-byte integer that stores a checksum. The checksum is calculated based on REDIS,db_version, Databases, and EOF. When the RDB file is loaded, the server compares the checksum calculated during data loading with the check_sum to check for errors in the RDB file.
The databases section
If databases 0 and 3 are not empty, the server will create an RDB file as shown in the following figure
Each non-empty database consists of SELECTDB, db_number, and key_value_pairs
-
SELECTDB: 1 byte in length. When the program encounters this value, it knows it will read a database number next
-
Db_number: stores a database number. When the program reads this db_number, the server will call the SELECT command to switch the database based on the read database number, so that subsequent key/value pairs can be loaded into the correct database.
-
Key_value_pairs: Stores all key-value pairs of the database. If the key-value pairs have an expiration time, the expiration time is also stored with the key-value pairs.
A key-value pair without expiration time consists of TYPE, key, and value in an RDB file
The key-value pair with expiration time consists of EXPIRETIME_MS,ms,type,key, and value.
Type records the type of value. EXPIRETIME_MS is a constant, 1 byte, that tells the program to read an expiration time in milliseconds next.
Ms is an 8-byte long signed integer that records a timestamp in billionths of a millisecond.
review
AOF
AOF persistence records database state by saving write commands executed by the Redis server.
AOF persistent implementation
Command to add
When AOF persistence is enabled, the server appends a write command to the end of the AOF_buf buffer in Redis protocol format after executing the write command.
Write and synchronize
The Redis server process is an event loop. Before the server terminates an event loop, it calls the flushAppendOnlyFile function to consider whether the contents of the aOF_buf buffer need to be written to and saved to the AOF file.
The behavior of the flushAppendOnlyFile function is determined by the value of the appendfsync option configured by the server
Always: after writing the aof_buf command, the system invoks fsync to synchronize the command to the AOF file. After fsync is complete, the thread returns
Everysec: The aof_buf command is written to the system and the write operation is invoked. After the write operation is complete, the thread returns. The fsync file synchronization operation is invoked once per second by a dedicated thread
No: After the command is written into aof_buf, the system invoks the write operation. Fsync is not performed for AOF files. The operating system is responsible for disk synchronization
When always is configured, AOF files must be synchronized for each write. On SATA disks, Redis can only support several hundred TPS writes. This configuration runs counter to the high performance features of Redis and is not recommended.
If this parameter is set to NO, the interval for each AOF file synchronization is uncontrollable and the amount of data to be synchronized increases. This improves performance, but data security is not guaranteed.
Everysec is the recommended synchronization policy and the default configuration to ensure both performance and data security. In theory only 1 second of data can be lost in the event of a sudden system outage.
rewrite
As commands are continuously written, aOF files will become larger and larger. Oversized AOF files may affect Redis server or even the entire computer. In order to solve this problem, Redis provides aOF file rewriting function. The Redis server creates a new AOF file to replace the existing AOF file. The new AOF file does not contain any redundant commands that waste space, so the new AOF file size is much smaller than the old AOF file size.
Rewrite concrete implementation
AOF file rewrite does not require any reading, parsing, or writing of existing AOF files. This function is implemented by reading the contents of the server’s database. Consider the following scenario
To record the animals key values, the AOF file must hold the four commands listed above.
When the AOF file is rewritten, the server simply reads the current animals key and generates a sadd animals “Dog”, “Panda”, “Tiger”, “Lion” and “Cat” command to replace the four commands. This reduces the number of commands required to save the Animals key from four to one.
AOF background rewrite
AOF rewriting involves a lot of writing operations, so to avoid blocking the server due to rewriting, Redis server uses child processes to execute, while the child processes AOF rewriting, the server process can continue to process command requests.
There is also a problem with using child processes, because during the child process rewrite, the server process continues to process command requests, and new commands modify the existing database contents, resulting in inconsistency between the current database contents and the rewritten AOF file contents.
To solve this problem, the Redis server sets up the AOF rewrite buffer, which is used after the server creates the child process. When the Redis server executes a write command, it sends the write command to both the AOF buffer and the AOF rewrite buffer.
All write commands executed by the server are logged into the AOF rewrite buffer from the time the child process is created for the AOF override.
When the child process completes the AOF rewrite, it sends a signal to the server process, which does two things when it receives the signal
-
Writes all the contents of the AOF rewrite buffer to the new AOF file
-
Rename the new AOF file and atomically overwrite the old AOF file
review
RDB versus AOF
- The loading of RDB files is done automatically when the server starts, and there is no special command. However, because AOF has a higher priority, when AOF is enabled, Redis will load AOF files first to recover data. Only when AOF is off will the RDB file be detected at Redis server startup and automatically loaded. The server blocks while loading the RDB file until the load is complete.
The transaction
Transaction major commands
- multi
At the beginning of transaction command execution, after this command is executed, all subsequent redis commands will be stored in a queue and will not be executed immediately. Only after exec execution, all commands will be executed according to the sequence of commands entered into the queue
- exec
The end of a transaction command, which, when executed, executes all previous commands that have been queued
3.watch
The watch command monitors the key, and if the key sends changes in the current transaction, it will not be executed. Returns the empty
4.DISCARD
Clears all commands previously in the transaction
- unwatch
Clear all previously monitored keys
The sample
Working Principle of the Watch command
A transaction with the watch command manages the client and monitored key in the watched_keys dictionary in the database. When a key is changed, the program turns on the redis_dirty_cas flag for all clients monitoring the modified key.