Small knowledge, big challenge! This article is participating in the creation activity of “Essential Tips for Programmers”.

This article has participated in the “Digitalstar Project” and won a creative gift package to challenge the creative incentive money.

1, the introduction of

Redis is very fast, a large part of the reason is that Redis data is stored in memory, since in memory, when the server is down or power, all the data will be lost, so Redis provides two mechanisms to ensure that Redis data will not be lost due to failure. This mechanism is called Redis’ persistence mechanism. Redis has two persistence mechanisms:

  • Redis Data Base (RDB) Memory snapshot
  • AOF(Append Only File) Incremental logs

Redis DataBase (RDB) is used to write snapshots of data sets in memory to disks at specified intervals. RDB is a persistent memory snapshot (a binary serialized form of memory data) that generates a snapshot from Redis each time for full data backup. Advantages:

  • Compact storage, saving memory space
  • The recovery rate is very fast
  • Suitable for full backup and full replication scenarios, and often used for disaster recovery (scenarios with low requirements on data integrity and consistency)

Disadvantages:

  • Easy to lose data. Easy to lose data that changes in the Redis server between snapshots.
  • The RDB uses a fork process to back up memory snapshots. This is a heavyweight operation and requires high cost.
  • Fork the child process, which shares memory but can swell up to twice its size if the memory is modified during backup.

Append Only File (AOF) records all memory modification commands (write operations) in an independent log File. After a restart, you can run the Redis command in the AOF File to restore data. AOF can solve the real-time problem of data persistence and is the mainstream persistence scheme in the current Redis persistence mechanism (mixed persistence after 4.0 will be discussed later). Advantages:

  • Data backup is more complete and the probability of data loss is lower, which is suitable for scenarios that have high requirements on data integrity
  • Log files are readable, AOF is more maneuverable and can be repaired through operation log files

Disadvantages:

  • AOF logs grow in size over a long period of time and are time-consuming to recover, requiring regular downsizing of AOF logs (more on that later)
  • The backup restoration speed is slow
  • Frequent synchronous write operations cause performance pressure

The website address

www.redis.io

2, RDB

2.1 introduction

When the RDB persistence scheme is backed up, Redis forks a separate subprocess for persistence, writes data to a temporary file, and replaces the old RDB file after persistence. In the whole persistence process, the main process (the process that provides services for the client) does not participate in IO operations, which ensures the high performance of Redis service. RDB persistence mechanism is suitable for scenarios where data integrity is not high but efficient recovery is required. Here is the RDB persistence flow: \

2.2 the Fork

In RDB persistence, the main process forks a child process to back up the RDB

  • In Linux, fork produces a child process identical to the parent process. The child process has the same data as the parent process, but the child process is a new process and has a parent-child relationship with the original process
  • For efficiency, Copy On Write (COW) is used in The Linux operating system. The fork child process shares a physical memory with its parent process. A Copy of the memory space is copied only when the memory in the process space is modified.

In Redis, RDB persistence takes full advantage of this technique. Redis calls the glibc function to fork a subprocess to handle the persistence, so that the parent process can continue to serve the client. The child of fork initially shares the same block of memory as the parent (the main process of Redis). In the persistence process, when the client requests to modify the data in memory, COW is used to separate the data segment page, that is, copy a block of memory for the main process to modify. \

RDB triggering rules are divided into two categories: manual triggering and automatic triggering. Automatic triggering:

  1. Configuring trigger Rules
  2. Shutdown the trigger
  3. Flushall trigger

Manual trigger:

  1. save
  2. bgsave

2.3 Automatic Triggering

The following describes the configuration trigger rules in the automatic trigger mechanism of Redis RDB persistence mechanism to trigger RDB, involving the configuration of RDB rules, file storage path configuration, file name configuration, file compression configuration, and file integrity verification configuration.

2.3.1 Triggering configuration rules

  • You can search for /snapshot in the Redis. Conf configuration file in the Redis installation directory. The following three lines of data are annotated in the configuration file by default.

The following describes the configuration rules. You can perform proper configuration based on actual requirements

Save 3600 1 -> 3600 seconds 1 key is changed, RDB is triggered save 300 100 -> 300 seconds 100 keys are changed, RDB is triggered save 60 10000 -> 60 seconds 10000 keys are changed, RDB is triggered

  • The path for storing RDB files is specified

You can see the dump. RDB file in the Redis installation directory. If not, connect to the client and run shutdown

  • Set the name of the RDB file

  • Configure RDB file compression

By default, Redis uses the LZF algorithm to compress Redis RDB files, which consumes CPU computing resources, but saves space

  • Example Configure RDB file integrity verification

By default, Redis uses the CRC64 algorithm to verify the integrity of RDB files

2.3.2 shutdown triggered

Shutdown triggers Redis’ RDB persistence mechanism as simple as executing shutdown on the client. \

2.3.3 flushall trigger

To flushhall, it flushes the dump. RDB file. Never, ever flushhall when you are in the office to back up an article and then greet me at that time. Flushhall flusher the dump. RDB file at the same time as flushing Redis data. Otherwise, when flushing Redis again, data will be restored to the data in the last flush. To prove that this file does not retain data, I write a script to test it:

Write a script file for bulk inserts

vi batchKeyInsert.sh
Copy the code
#! /bin/bash for((i=0; i<100000; I++) do echo - en "Hello Redis." | Redis - 192.168.211.108 cli - h - p - c - 6379 x the set name $I > > Redis. The log is doneCopy the code

File empowerment

chmod +x batchKeyInsert.sh
Copy the code

./batchKeyInsert.sh
Copy the code

At this point check dump.rdb\

Execute flushall, then check again, the RBD file is emptied \

2.2 Manual Triggering

RDB persistence can be triggered manually using the save command and the BGsave command. The differences between the two commands are as follows. Save: If the save command is executed, other operations of Redis are blocked. As a result, Redis cannot respond to client requests. This parameter is not recommended. Bgsave: When the BGSave command is executed, the Redis background asynchronously saves snapshots. In this case, Redis still responds to client requests.

2.3 RDB Persistent File Backup

In the actual production environment, we generally do not use the Master node to carry out persistent backup. We will carry out persistent backup of RDB on multiple secondary servers of Redis, in order to back up Redis data for multiple times and prevent the breakdown of network partitions or some nodes or even hardware damage. As operation and maintenance or architect, Leo BA felt that it should be regularly through the script of Redis persistent file transfer backup, so double insurance, more reliable, in case of emergencies, but also more than one hand solution.

3, AOF

3.1 introduction

When the AOF persistence scheme is backed up, all write commands requested by the client are appended to the AOF buffer. The data in the buffer is synchronized to the AOF file on the disk according to the synchronization policy configured in the Redis configuration file. When the AOF file reaches the threshold configured in the rewrite policy, Redis overwrites the AOF log file to make the AOF log file thin. When the Redis service is restarted, data is recovered by loading AOF log files. \

3.2 AOF configuration

3.2.1 Basic Configuration

AOF is not enabled by default. The default value is appendonly no. If it is enabled, change it to appendonly yes\

The default name of the AOF configuration file is appendone.aof \

The address of the configuration file can be obtained by running config get dir on the Redis client. The path is the same as RDB \

3.2.2 Synchronizing frequency Configuration

AOF logs exist in the form of files, and when a program writes to an AOF log file, it actually writes to a memory buffer allocated by the kernel for file descriptors, which is then asynchronously flushed to disk. If the server goes down before the data in the buffer is flushed back to disk, the data will be lost. Therefore, Redis calls fsync(int FID) provided by glibc in Linux to flush the contents of the specified file from the kernel buffer to disk to ensure that the data in the buffer is not lost. However, this is an IO operation and it is very slow compared to Redis performance, so it cannot be performed frequently. There are three configurations for flushing buffers in the Redis configuration file: Appendfsync always Writes to the AOF log every time a Redis write is performed. This configuration cannot be supported by Linux because the concurrency of Redis far exceeds the maximum refresh rate provided by Linux. This configuration is also very performance intensive because it involves IO operations, so it is almost never used

Appendfsync Everysec flusher buffers of data to AOF files once per second, the default policy in the Redis configuration file, which is compatible with a compromise between performance and data integrity. This configuration theoretically loses data in a second or so

Appendfsync no Redis does not actively flush the buffer data to the AOF file, but hands it over to the operating system. This operation is also not recommended and data loss is highly likely. \

Notice To flush buffer data to disk, set the following configuration to no, not yes

no-appendfsync-on-rewrite no
Copy the code

​

3.2.3 AOF repair function

The normal recovery of AOF persistence mechanism is the same as the recovery of RDB persistence mechanism. All you need to do is to put the backup file in the working directory of Redis, and it will be automatically loaded when Redis starts. The AOF persistence mechanism provides the ability to recover when AOF files are abnormal, which is often used in AOF file corruption scenarios.

Test, empty data in Redis service \

Write data \

The AOF log file is refreshed every second when data is written to appendone.aof file \

Opening the file, we can read the contents of the AOF file very clearly and see the instruction sequence \ of Redis

At this time artificial data destruction \

Restart found unable to start (I configured alias start) \

Run redis-check-aof –fix.. Appendone.aof fixes the aof log file \

Some data will be lost during the repair

Connect client to view data \

3.2.4 AOF rewrite

Mentioned AOF shortcomings, said AOF belongs to log to store Redis, in the form of additional written instructions, which can lead to a large number of redundant instruction, making AOF the log file is very large, such as the same key is wrote a 10000 times, the last has been deleted, this situation is not only for memory, can also lead to recovery is very slow, Redis therefore provides rewriting mechanisms to solve this problem. The AOF persistence mechanism of Redis only saves the minimum instruction set of data recovery after rewriting. If we want to manually trigger, we can use the following instruction:

bgrewriteaof
Copy the code

Rewrites after Redis4.0 use RDB snapshot and AOF instruction stitching. The head of AOF file is the binary data of RDB snapshot, and the tail is the instruction of write operation after snapshot generation. Rewriting AOF files will affect Redis performance to some extent, so it cannot be automatically rewritten. Redis provides two parameters for automatic AOF rewriting, which will be rewritten only when both parameters are met: \

Auto-aof-rewrite-percentage 100: specifies the maximum size of a file whose memory size is twice that of the previous one. Auto-aof-rewrite-min-size 64mb: specifies the minimum memory size for rewriting a file

AOF rewrite process is as follows:

  1. Bgrewriteaof triggers rewriting and checks whether BGSave or BgreWriteAOF is being executed. If bgreWriteAOF exists, the bGREWriteAOF is executed after the execution is complete
  2. Fork the main process to prevent the main process from being blocked and unable to provide services, similar to RDB
  3. The child process traverses the Redis memory snapshot and writes the data to the temporary AOF file. At the same time, the child process writes the new write instructions to the two rewrite buffers aOF_buf and aOF_rewrite_buf. The former is to write the old AOF file, and the latter is to flush the data to the temporary AOF file, so as to prevent the loss of new write operations during the snapshot memory traversal
  4. The main process is notified when the child process finishes writing a temporary AOF file
  5. The main process writes the data in the AOF_rewirte_buf buffer in 3 above to the temporary AOF file generated by the child process
  6. The main process replaces the old AOF file with a temporary AOF file to complete the rewrite process

4. Mixed persistence

Most usage scenarios after Redis4.0 do not use RDB or AOF alone for persistence, but instead use a mixture of the best of both. The reason is that although RDB is fast, it will lose a lot of data and cannot guarantee data integrity. AOF can ensure data integrity as much as possible, but performance is a real concern, such as replay recovery data. The log file structure is as follows: \

Mixed persistence is enabled with aof-use-rdb-preamble yes, which is enabled by default for Redis 4.0 + \

To test, we first insert some keys, then execute BGREWRITEAOF to trigger AOF persistence, and then insert some keys \

You will see the following effect, which verifies the mixed persistence method \

5, summary

Finally, to sum up the two, which is better?

  • It is recommended to enable both
  • If you are not sensitive to data, you can choose to use RDB alone
  • AOF alone is not recommended because of the potential for bugs
  • If you’re just doing pure memory caching, you don’t have to do either

The Redis website says: \

If you don’t understand, please read the introduction of Redis Chinese website: \

Redis official website about persistence

Redis. IO/switchable viewer/pers…

Redis Chinese about persistence

Redis. Cn/switchable viewer/pers…