directory
Redis persistence
How is AOF persistence implemented?
1. Persistent implementation
2. AOF file data loading and restoration
3. Rewrite the AOF file
4. Background rewrite process
RDB persistence implementation principle?
1, manual trigger: Save and BGSave
2, automatic trigger: interval save conditions
What are the differences between AOF and RDB persistence?
1. Advantages and disadvantages of AOF
2. Advantages and disadvantages of RDB
Five, to sum up
Xiao C happened to go for an interview today. At the beginning of the interview, the interviewer asked directly: Since you are familiar with Redis database, what are its persistence methods?
Then, little C did not expect so soon, directly into the interview…… Then come on!
Redis persistence
Redis supports AOF persistence (Append Only File) and RDB persistence (Redis DataBase). The persistence function is very good to avoid the problem of data loss caused by the end of the process, so persistence is very important for data recovery.
Therefore, the following two mechanisms will be explained in detail, easy to understand!
How is AOF persistence implemented?
A: AOF persistence records the state of the database by saving the write command of the Redis database server. It is the mainstream way of Redis persistence to ensure the real-time performance of data persistence. Let me draw you a simple picture:
1. Persistent implementation
AOF persistence is implemented in three steps: command writing, file synchronization, file rewriting, and of course, a restart loading step for data recovery.
Command write: Because Redis’s command request protocol is plain text **, commands written to AOF files are saved in text format.
To see how this works, I have AOF persistence enabled in the redis.windows.conf configuration file, and file synchronization per second is enabled by default.
Then execute a few commands
127.0.0.1:6379> get name
(nil)
127.0.0.1:6379> get age
(nil)
127.0.0.1:6379> set name Charzous
OK
127.0.0.1:6379> set age 21
OK
127.0.0.1:6379> set phone 123
OK
127.0.0.1:6379>
Copy the codeOpen appendone.aof and you can see that each executed command is indeed saved, with the select command specifying that the database was added automatically by the server.
File synchronization:The Redis server’s event loop is responsible for receiving client command requests, the aOF_buf buffer is first written, and the data of the buffer is then synchronized to the hard disk based on the synchronization option value.
Appendfsync controls the AOF buffer synchronization policy. The values are as follows: always: write commands to the buffer and invoke fsync to synchronize the command to the AOF file. When the command is complete, the thread returns the command. After the command is written into the buffer, the system calls the write operation, and the thread returns after completion. The fsync file operation is called once per second by a dedicated thread. 3. No: Call write. The operating system is responsible for aOF synchronization.
Write buffer aOF_buf Is used to improve the efficiency of file writing. When the write function is called, data is temporarily stored in the memory buffer and written to disks when the space is full or the time is up.
2. AOF file data loading and restoration
A: The above demo shows that the AOF file contains all the commands to restore the database state. The server only needs to execute the commands in the file once to restore the database state before the shutdown.
The detailed process looks like this:
- Create a pseudo client that reads write commands from a local AOF file rather than a network connection.
- Parse and read a command from an AOF file.
- Use the pseudo client to execute this command.
- Repeat steps 2 and 3 until all commands saved by AOF are executed.
AOF persistence = AOF persistence = AOF persistence = AOF persistence = AOF
[19992] 10 May 15:18:12.021 * 1 Changes in 900 seconds. Saving... [19992] 10 May 15:18:12.865 * Background saving started by PID 2892 [19992] 10 May 15:18:13.367 # fork operation Complete [19992] 10 May 15:18:13.370 * Background saving terminated with success [19992] 10 May 15:18:13.367 # fork Operation complete [19992] 10 May 15:18:13.370 * Background saving terminated with success [19992] 10 May 16:50:14.547 # User requested shutdown... [19992] 10 May 16:50:14.548 * Calling fsync() on the AOF file. [19992] 10 May 16:50:14.552 * Saving the final RDB Snapshot before Withdraw. [19992] 10 May 16:50:14.605 * DB saved on disk [19992] 10 May 16:50:14.605 # Redis is now ready to exit, bye bye...Copy the codeThen the server and client can be sent to load data without loss.
The server shows loading data from the AOF file:
[11672] 10 May 16:50:33.345 # Server started, Redis version 3.2.100
[11672] 10 May 16:50:33.379 * DB loaded from append only file: 0.031 seconds
[11672] 10 May 16:50:33.379 * The server is now ready to accept connections on port 6379
Copy the codeWhen you restart the client, the data still exists.
Rewrite (AOF)
Rewrite mechanism: The process of converting data in the Redis process into write commands to synchronize to a new AOF file.
(1) Data that has timed out cannot be written into the file. (2) The old AOF file contains invalid commands, and the new AOF file in the reprocess only retains the writing commands of the final data. (3) Multiple write commands can be combined into one. Objective: To reduce the space occupied by files; Smaller AOF files can be loaded faster by the server.
One of the problems with AOF persistence is that as more and more AOF files grow in size, this can have a significant impact on database servers and computers. For example, the client request command contains many Rpushes, and AOF also stores this sequence of commands, so the AOF override function removes redundant commands, making the file size smaller.
Specific implementation principle: First create a new AOF file, for a series of the same type commands, such as the list of keys (RPUSH) and the collection of keys (SADD), first traverse all the keys in the database, then use a command instead of recording before the key values of multiple orders, use the new AOF file instead of the old AOF, save the state of database is the same.
Now for a simple verification, add four list key-value pairs:
127.0.0.1:6379> Rpush list "A" "B"
(integer) 2
127.0.0.1:6379> rpush list "C"
(integer) 3
127.0.0.1:6379> rpush list "D" "E"
(integer) 5
127.0.0.1:6379> rpush list "F"
(integer) 6
Copy the codeManually trigger the rewrite, calling the BGREWRITEAOF function.
127.0.0.1:6379> Bgrewriteaof Background Append only File Started (0.82s)Copy the codeLook at the AOF refile and find it merged into a single command:
4. Background rewrite process
The above example uses a manual trigger to call the BGREWRITEAOF function to let AOF rewrite the file, as well as an automatic trigger, depending on two configuration file parameters.
Manual trigger: Directly invoke the bgrewriteaof command.
Automatic trigger: Automatic trigger is triggered based on the following two parameters. Auto-aof -rewrite-min-size: The minimum size of a file when aof rewrite is run. The default size is 64 MB. Auto-aof -rewrite-percentage: specifies the ratio of the current AOF file space to the size after sprint rewriting. When the value is greater than the minimum file size and the ratio is greater than or equal to the specified value, the alarm is automatically triggered.
It can be seen that the rewriting process takes time, Redis database is a single thread processing command request, so if there is a large number of write operations, will be blocked for a long time, the server can not process the request sent by the client.
So the solution is that Redis puts the reprogram into the child process, which executes it, and the server process can still handle the client’s requests.
The advantage is that the child process has a copy of the parent process’s data, unlike threads, which can ensure data security without locking.
The server is blocked from processing requests from the Client.
During the AOF rewrite, note that the rewrite is an operation on the old AOF file. The server is processing the client request. These commands may update the database, causing the inconsistent state of the database.
To solve this new problem, a new AOF rewrite buffer was introduced! Now, during the child AOF rewrite, the server process needs to do three things:
- Execute client request commands;
- Appends the executed write command to the AOF buffer;
- Appends the executed write command to the AOF rewrite buffer.
Draw a more intuitive picture and show it to the interviewer!
This finally solved the existing problems, can ensure the consistency of the database, the specific process:
- The contents of the AOF buffer are periodically written and synchronized to a file.
- When the child process overrides, the requested commands are saved to the AOF overwrite buffer.
- When the override is complete, the child sends a signal to notify the server process.
- The server writes the contents of the AOF rewrite buffer to the new AOF file, overwriting the old file.
- AOF background rewrite completed.
RDB persistence implementation principle?
A: I have learned that RDB persistence is different from AOF persistence in that it records database state by saving key-value pairs of data, and saves binary snapshot files of current data in hard disk.
1, manual trigger: Save and BGSave
The sava command blocks the server process until the RDB file is saved, during which time the server cannot process command requests.
The BGsave command is similar to the BGWriteAof command in that the server can still handle requests by creating child processes for RDB persistence.
To verify, the client enters two commands and invokes the manual save command.
127.0.0.1:6379> sadd city "A"
(integer) 1
127.0.0.1:6379> sadd city "B"
(integer) 1
127.0.0.1:6379> save
OK
Copy the codeServer side display:
Open the dump. RDB file and see the data in binary format.
Note that when loading the RDB file, the server process blocks until the load is complete.
2, automatic trigger: interval save conditions
Automatically triggers saving RDB files by setting variable parameter values. You can see it in the redies.windows.conf configuration file:
# # In the example below the behaviour will be to save: # after 900 sec (15 min) if at least 1 key changed # after 300 sec (5 min) if at least 10 keys changed # after 60 sec if at least 10000 keys changed # # Note: you can disable saving completely by commenting out all "save" lines. # # It is also possible to remove all the previously configured save # points by adding a save directive with a single empty string argument # like in the following example: # # save "" save 900 1 save 300 10 save 60 10000Copy the codeThis means that RDB persistence is automatically performed as long as one of the three conditions is met:
- The server has made at least one change to the database within 900 seconds.
- The server made at least 10 changes to the database in 300 seconds.
- The server made at least 10,000 changes to the database in 60 seconds.
The server detects one change in the 900s and triggers the RDB save.
[22228] 10 May 21:20:40.053 * 1 changes in 900 seconds. Saving...
[22228] 10 May 21:20:40.426 * Background saving started by pid 13364
[22228] 10 May 21:20:40.629 # fork operation complete
[22228] 10 May 21:20:40.629 * Background saving terminated with success
Copy the codeThe write parameter conditions are stored in the Savaparams array, and the server automatically triggers RDB persistence by maintaining a dirty counter and the lastSave attribute and traversing the set of numbers to determine whether the conditions are met.
Dirty counter: The command increases the count by one if the database is modified once.
Lastsave property: records the timestamp of the lastsave operation.
What are the differences between AOF and RDB persistence?
1. Advantages and disadvantages of AOF
Answer: that say advantage first.
- AOF persistence is real-time and can better protect data from loss. By default, AOF executes fsync operation every second through a background thread and loses data for a maximum of one second.
- AOF files are written in append-only mode, with very high write performance, and files are not easy to damage, even if the tail of the file is damaged, it is easy to repair.
Disadvantages also answer one or two:
- AOF files are usually larger than RDB data snapshot files for the same data, for reasons mentioned earlier.
- AOF is a more vulnerable and bug-prone approach to replaying database state based on command logs than the rDB-based approach of persisting a complete snapshot file at a time.
2. Advantages and disadvantages of RDB
A: Many times there are different ways to deal with the same thing, so the advantages and disadvantages of RDB and AOF should be opposite.
Advantages:
- Compact binary file, representing Redis data snapshot at a point in time, small file size.
- Loading RDB files restores data much faster than AOF.
Disadvantages:
- Real-time persistence is not possible, and each bgSave run forks a child process, which is a heavyweight and costly operation.
- Specific binary format save, each version of the RDB format, the old version of Redis is not compatible with the RDB format of the new version.
Five, to sum up
Time passed so fast that I didn’t expect the interviewer to ask Redis database so detailed. It has been one hour, and XIAO C has explained the principle and practice of Redis AOF and RDB persistence method clearly!
He looked at the time and said, “It’s good. The principle is very detailed, and there are simple examples and practices. The foundation is good, and it will be better if we can apply it in practical projects in the future. Time is about the same, go back and wait for HR notice!
See small C experience, you do not think Redis database AOF and RDB persistence basic knowledge look enough, hurry to collect good, spare!
If you feel good, welcome to “one button three even” oh, like collection attention, comments, questions and suggestions, welcome to exchange learning! Progress together!
