This is the third day of my participation in Gwen Challenge

preface

Redis data is stored in the memory. If you do not configure persistence, the data will be lost after Redis restarts. Therefore, you must enable Redis persistence to save data to disk.

This article, from three aspects to give you a more intuitive Redis persistence

  • The RedisPersistent modeWith what kind of
  • The RedisData recoveryWhat are the steps?
  • Redis in the enterprisePersistence strategyHow to choose?

Let’s start with Redis persistence ！！！！

Redis persistence

Redis provides two ways to persist:

  • One is RDB persistence (RDB persistence where Reids periodically dump database records in memory to disk)
  • The other is AOF persistence (appending the operation logs of Reids to a file).

Snapshot RDB (the snapshot)

By default, Redis stores in-memory database snapshots in a binary file named dump.rdb

Conditions for generating the dump. RDB file

Automatic trigger

Conf file can be set to automatically save once the condition that at least M changes are made within N seconds is met, as shown in the following figure:

For example: Save 60 10000: means redis automatically saves once as long as there are at least 10000 operations within 60 seconds

You can add multiple policies, as long as one of them matches, Redis will automatically save once, to disable RDB just need to comment out all save policies

Performed manually

The dump. RDB file is generated by running the save command or bgsave command on the Redis client. Each time, all redis memory snapshots are taken into a new RDB file and the original RDB snapshot file is overwritten

For example, the existing dump. RDB file was created at 20:19

Execute the save command:

If the creation time of the existing dump. RDB file is the current time, the file has been overwritten

The path of the dump. RDB

The dump. RDB generation path can be set by yourself in the redis. Conf configuration file, as shown in the following figure

Dump. RDB file name

The file name of dump. RDB can be set by yourself in the redis. Conf configuration file, as shown in the following figure

Dump. RDB files are generated synchronously or asynchronously

For example, the save command mentioned above is synchronized during the generation of dump. RDB file, which means that other Redis commands will be blocked during the generation of the file. If the redis memory is larger, the time will be longer. So Redis provides a bgSave copy-on-write mechanism that generates snapshots while still processing commands.

Reids Main thread fork A child process that shares all memory data of the main thread and writes it to a file.

In this case, if the main thread also reads the data, then the main thread and the child process do not affect each other. If the main thread modifies the data, a copy of the data is made. The main thread then modifies the copy directly, and the child process in bgSave writes to the RDB file based on the previously shared memory data.

RDB files are automatically generated using bgSave in the background

Comparison of SAVE and BGSave

AOF (append-only file)

The RDB approach has a serious drawback: if Redis goes down, the server will lose commands that were recently written and not saved to the snapshot. So Redis adds another persistence method: AOF, which records operations and then replays operations, but AOF files only record commands that modify data

Open AOF

You can set appendonly No to appendOnly YES in the redis. Conf file, as shown in the following figure:

Name of the file generated by AOF

The strategy of AOF

AOF supports three strategies:

  1. appendonly always: Fsync is executed every time a new command is appended to the AOF file, which is very slow and very safe
  2. appendfsync everysec: Fsync is performed once per second, which is fast enough and only 1s is lost in the event of a failure
  3. appendfsync no: Never fsync, handing the data to the operating system for processing. The faster and less safe option

The recommended and default measure is appendfSync Everysec, compatible with speed and security

Test AOF

  • Step 1: Enter a few random commands in Redis

  • Step 2: Wait 1s to see if the AOF file is produced

  • Step 3: UsecatCommand to view the AOF fileThe command has been appended to the AOF file

  • Step 4: Analyze the AOF file

The Protocol specification of Redis is Redis Serialization Protocol (Redis Serialization Protocol). AOF files follow this Protocol. Let’s take the set name Miludexiaoguangtou example for preliminary analysis:

  • * 3The: aof file records that each command is executed by*At the beginning, 3 indicates that the command takes several arguments, such as:set name miludexiaoguangtouIt’s three parameters,Set, Name and Miludexiaoguangtou
  • $3:$3That means the bottomsetThere are three characters
  • $4:$4That means the bottomnameThere are four characters
  • $18:$18That means the bottommiludexiaoguangtouThere are 18 characters

AOF the rewrite

AOF files may have too many duplicate and useless instructions. So Redis supports regular production of AOF files based on the latest data in memory, for example, the following instructions are executed

The following 4 incr commands in the AOF file will become:

*3
$3
SET
$2
count
$1
4
Copy the code

The frequency of AOF rewrites can also be configured in the redis.conf configuration file

// aof file size increases by 100% since last rewrite
auto-aof-rewrite-percentage 100
// Aof files will be rewritten automatically until they reach at least 64MB
auto-aof-rewrite-min-size 64mb
Copy the code

AOF files also support manual rewriting. Go to the Redis client and execute bgrewriteAof to rewrite AOF

Overwriting Redis forks a child process to do it (similar to the BGsave command) without much impact on normal command processing

How to choose RDB and AOF

The command RDB AOF
Startup priority low high
volume low big
Recovery rate fast slow
Data security Easy to lose data By strategy

Both production environments can be enabled. If there are both RDB files and AOF files when Redis is started, the AOF file is preferred to recover data, because AOF is generally more secure

Hybrid Persistence (Redis 4.0)

RDB is rarely used to restore memory state when rebooting Redis, because a large amount of data is lost. AOF log replay is usually used, but the performance of AOF log replay is much slower than RDB, which can take a long time to start up if the Redis instance is large

Redis 4.0 addresses this problem by introducing a new persistence approach – hybrid persistence

Enable persistent configuration (AOF must be enabled first)

If mixed persistence is enabled, when AOF is rewritten, it does not simply convert the memory data into RESP commands and write them into AOF files. Instead, it takes RDB snapshots of the memory before the rewriting and writes the RDB snapshots together with the incremental AOF commands into a new AOF file.

The new AOF file is not initially called appendone.aof, but will be renamed after rewriting the new AOF file to override the original AOF file and complete the replacement of the old and new AOF files

So in general, when Redis is started, the memory data in RDB format is loaded first, and then the incremental AOF log is replayed can completely replace the previous FULL AOF file replay, so the restart efficiency is greatly improved

Example: Mixed persistence

If you run the AOF override command manually, it is found that the memory prior to this moment has been RDB snapshot and written to the new AOF file.

After writing two increments to the Redis command, it is found that the append is directly at the end of the AOF file

Hybrid persistent AOF file format

Above is the binary format of RDB (memory compressed data at that moment), followed by the format of AOF commands (incremental commands)

Redis data recovery

So far only talk about Redis persistence, and will generate RDB and AOF two formats of file, but how to restore did not say, the following will start according to the case step by step talk about how to restore Redis data

  • Step 1: Turn off RedisSimulation of downtimeIn the case

  • Step 2: Rename RDB files and AOF filesWhat persistence file doesn't existIn the case

  • Step 3: Start Redis, observe the data inside, and findThere's no key

  • Step 4: BecauseRedis is automatically persisted once it is closedSo we putThe newly produced AOF file was deleted

  • Step 5: Combine the original AOF file with the RDBFile naming restoration

  • Step 6: Restart Redis and observe the data to find the data in the aOF fileIt's already loaded into Redis

Redis persistence strategy

  1. This section describes how to write a crontab scheduling script to copy an RDB or AOF backup file to a directory every hour. Only the backup file generated in the last 48 hours is retained
  2. This section describes how to write the crontab scheduling script to back up one copy of the current day’s data to a directory every day
  3. Each time you copy a backup, you can delete the old backup records
  4. Make a copy of the backup on the current machine to another machine every night to prevent machine damage (multiple backups)

conclusion

This article, from three aspects to give you a more intuitive Redis persistence

  • The RedisPersistent modeWith what kind of
  • The RedisData recoveryWhat are the steps?
  • Redis in the enterprisePersistence strategyHow to choose?

I believe that everyone now has a clear feeling about Redis persistence. Now hurry up and have a chat with your operation and maintenance staff to find out what kind of persistence strategy is used in their enterprises, why they set it like this, and whether there is a better solution?

Phase to recommend

Starting from scratch to learn Redis series (a) | Redis environment set up

Starting from scratch to learn Redis series (2) | how Redis commands commonly used jolt in your business

omg

Finally, if you feel confused about the article, please leave a comment immediately. If you think I have something, please like 👍, follow ❤️ and share 👥, because this will be my motivation to output more high-quality articles, thank you!!

If you want to get books related to Redis, you can follow the wechat official account Java Encyclopedia and enter Redis

Finally, thank you for your support. See you next time!