1. What is AOF

AOF AppendOnly File (AppendOnly File) provides a persistent solution to record Redis operations. The operations that occur in Redis are faithfully recorded in files generated by THE AOF. In this way, data status can be restored after the Redis server restarts or goes down. The main role of AOF is to solve the real-time of data persistence, which has been the mainstream way of Redis persistence.

2.AOF persistence principle

AOF persistence principle: Each write operation is recorded in a separate log. Redis initially reads this file to rebuild the data. In other words, Redis logs each write operation. All write commands executed by Redis are recorded (read operations are not recorded). After Redis restarts, the write commands will be executed from front to back according to the contents of log files to complete data recovery and achieve data recovery.

Note: only append log file content but not overwrite the file. Whenever a write command comes in, it is directly saved in our AOF file

2.1AOF Persistence implementation

The implementation of AOF persistence can be divided into command append, file write, file sync, file rewrite and load

The process is as follows:

• All write commands are appended to the AOF buffer.

• The AOF buffer synchronizes data from disks based on the corresponding synchronization policy.

• As AOF files become larger, they need to be rewritten periodically to achieve compression.

• When Redis restarts, AOF files can be loaded for data recovery.

3. Three synchronization strategies for AOF persistence

Before Redis terminates an event loop, it calls the flushAppendOnlyFile function to determine whether the contents of the AOF cache need to be written to and synchronized to the AOF file. There are always, Everysec, and no synchronization policies for AOF persistence

3.1 always

Appendfsync always: Synchronous persistence Every data change is immediately logged to disk with poor performance but better data integrity Redis writes all contents of the AOF buffer to an AOF file in each event loop. The AOF files are synchronized, so always is the least efficient of the appendfsync option’s three values, but it is also the most secure in terms of security. AOF persistence also loses only the number of commands generated in an event loop when downtime occurs.

3.2 Everysec

Appendfsync everysec: asynchronous operation, recording per second If a second goes down and data is lost, is the default synchronization policy, in which Redis writes all the contents of the AOF buffer to the AOF file at each event loop. AOF files are synchronized in child threads every second, but the whole process is asynchronous. In terms of efficiency, this model is fast enough. In the event of a malfunctioning shutdown, only one second of command count is lost

3.3 No

In this policy, Redis writes everything in the AOF buffer to the AOF file at each event loop. The synchronization of AOF files is controlled by the operating system. This mode has the highest speed, but requires a long synchronization interval. Therefore, a large amount of data may be lost when a fault occurs.

3.4 Advantages and disadvantages of the three strategies

1.NO policy: Although it appears to improve performance on the surface, it will block the main process for a relatively long time each time the save command is executed. And data security is not guaranteed, if the Redis server suddenly goes down, then the data not saved from the AOF cache to the hard disk will be lost.

2.ALWAYS policy: the security is guaranteed to the maximum, and the last write operation can be lost at most in theory. However, because every write operation will block the main process, the response speed of the main process of Redis is greatly affected.

3. EVERYSEC policy: The recommended configuration is also the default configuration of Redis, which is relatively balanced between security and performance.

3.5 Comparison of three Synchronization Policies

4. The rewriting mechanism of Redis

4.1 Introduction to rewriting Mechanism

When the size of AOF files exceeds the set threshold, Redis will start the content compression of AOF files and only retain the minimum instruction set that can recover data. You can use the command bgrewriteaof

4.2 Principle of rewriting

If the AOF file continues to grow too large, a new process is forked to rewrite the file (also write temporary files first before rename), traversing the memory of the new process, with a Set statement for each record. The operation of overwriting an AOF file, rather than reading the old AOF file, commands the entire contents of the database in memory to rewrite a new AOF file, similar to a snapshot

4.3 Three trigger mechanisms for rewriting

1. Invoke the bgrewriteaof command manually. If a child process is running, rewrite will be postponed.

2. If the AOF function is enabled manually, rewrite will be triggered to write data from the current database to the rewrite file if there is no RDB child.

In the Redis timer, when there is no RDB or rewrite child running, the rewrite condition triggers once or the AOF file size has reached its configured size.

4.Redis records the size of the AOF file when it was last rewritten. By default, this is triggered when the AOF file is twice the size of the last rewrite and the file is larger than 64M

4.4AOF background rewrite

The AOF override does a lot of writing, and the thread calling it is blocked for a long time, so Redis performs the AOF override in the child process.

1. The Redis process can continue processing client command requests during the AOF rewrite of the child process.

2. The child process has a copy of the parent process’s memory data, which can ensure data security even if locks are not applicable.

During the AOF restart of the child process, Redis receives client commands that modify the existing database state, resulting in inconsistency between the current state of the data and the database state stored in the rewritten AOF file. To do this, Redis sets up an AOF rewrite buffer, which is used after the server creates the child process. When Redis executes a write command, it sends the write command to both the AOF buffer and the AOF rewrite buffer.

When the child completes the AOF rewrite, it sends a signal to the parent. Upon receiving the signal, the parent calls a signal handler and performs the following:

1. Write all contents in the AOF rewrite buffer to the new AOF file to ensure that the database status saved in the new AOF file is the same as the current server status.

2. Rename the new AOF file, atomically overwrite the existing AOF file, and complete the replacement of the old and new files

3. Continue to process client request commands.

During the whole AOF background rewriting process, only the signal processing function will block the main Redis process. At other times, the AOF background rewriting will not block the main process.

5. AOF disadvantage

1. For the data of the same data set, aOF files are much larger than RDB files, and the recovery speed is slower than RDB files

2. The running efficiency of Aof is lower than that of RDB, and the efficiency of the synchronization policy per second is better. The non-synchronization efficiency is the same as that of RDB

6. AOF summary

7. Summary of persistence schemes

1.RDB persistence can take snapshots of your data at specified intervals

2. In the AOF persistent mode, each write operation to the server is recorded. When the server is restarted, these commands are executed again to restore the original data. Redis can also rewrite AOF files in the background so that AOF files are not too large.

3. Cache only: If you only want your data to live while the server is running, you don’t need to use persistence at all

4. Enable both persistence modes

(1) In this case, when Redis restarts, the AOF file will be loaded first to recover the original data, because in general, the data set saved by the AOF file is more complete than that saved by the RDB file.

(2)RDB data is not real-time, and the server will only look for AOF files when both are used. Should we just use AOF? The authors advise against it, as RDB is better for backing up databases (AOF is not easy to back up constantly), restarts quickly, and there are no potential AOF bugs left as a back-up measure.

8. Performance recommendations for both persistence scenarios

1. Since RDB files are only used for backup purposes, it is recommended to persist RDB files only on the Slave and backup them only once every 15 minutes. Only save 9001 is retained.

2. If you Enalbe AOF, the benefit is that in the worst case, you will only lose data for less than two seconds. The cost is constant IO and the inevitable blocking of writing new data to new files in the rewrite process. The frequency of AOF rewrite should be minimized as long as hard drives are permitted; the default base size of AOF rewrite, 64M, is too small and can be set to more than 5G. The default size exceeds 100% of the original size and overrides can be changed to an appropriate value.

3. If AOF is not enabled, only master-slave Replication can be used to implement high availability. It saves a lot of IO and reduces the system volatility that comes with rewriting. The trade-off is that if the Master/Slave is dropped at the same time, the data will be lost for more than ten minutes. The startup script will also compare the RDB files in the two Master/Slave files and load the newer one. Sina Weibo has chosen this structure