background

Recently, I need to migrate the data of Redis in my work. After searching online, I feel that the data is very scattered and not very clear. Therefore, I plan to write an article to sort out the knowledge about Redis backup and recovery, hoping to be helpful to you. There are three main methods of Redis data migration:

  • RDB
  • AOF
  • redis-dump

This paper will introduce the operation methods and principles of each method, and analyze and compare the advantages and disadvantages of each method and practical scenarios.

RDB

RDB introduction and principle

RDB persistence is a default redis persistence scheme, the completion of the. RDB file will be generated. The RDB file holds the database’s data in full, but in binary compression. Take a quick look at the beginning of the file:

RDB 0000000 R E D I S 00 9 372 \t R E D I S 0000020 -v E R 005 5.0. 372  \n r e d i 0000040 s - b i t s 300 @ 372 005 c t i m e 007 0000060 \a G 256 ` 372 \b u s e d - m e m 302 P 0000100 260 : 004 372 \f a o f - p r e a m b l 0000120 e 300 \0 376 \0 373 W 330 \0 016" r e c y c 0000140 l e : l i s t : r o u t e : 1 6 0000160 1 6 2 0 6 7 7 6 l W 8 7 i 001 303 C 0000200 { G z 037 z \a \0 \0 R 006 \0 \0 016 \0 \0 @Copy the code

RDB mode is usedSAVEThe command orBGSAVEBoth commands end up calling the rdbSave function, the difference between the twoSAVEIs blocking the main process; whileBGSAVENon-blocking, operating by calling fork. Note, however, that forking out the child process blocks.Since redis is known to be single-threaded, main thread 1 blocks and performance degrades dramatically. soSAVEHardly. The most common one isBGSAVE

RDB backup configuration and practice

configuration

Since RDB is redis persistence, we certainly don’t need to call BGSAVE ourselves to save data. Redis can be configured to automatically execute BGSAVE commands on a regular basis. We can find the redis. Conf file, which contains the following configuration related to RDB backup:

RDB = aOF; RDB = aof
dir /usr/local/var/db/redis/
 
# RDB persistence file name
dbfilename dump.rdb

Whether to compress RDB files
rdbcompression yes

Check whether the RDB file is missing or complete by check_sum
rdbchecksum yes

RDB default backup policy
save 900 1  Bgsave is executed if there is at least one data operation within 900 seconds
save 300 10  Bgsave is executed if there are at least 10 data operations within 300 seconds
save 60 10000  Bgsave is executed if there are at least 10000 data operations within 60 seconds

The backup policy of # RDB will be executed as long as one of them is satisfied. In addition, if you want to cancel automatic backup, you can add save "" at the end.
Copy the code

​

Demonstrates data recovery from an RDB file

  1. Manufacturing data
redis-cli

Clean up the database first127.0.0.1:6379 > FLUSHDB OKThen insert some data
127.0.0.1:6379> SET astr astring
OK
127.0.0.1:6379> RPUSH alist a b c d e f
(integer) 6
127.0.0.1:6379> HSET ahash 1 1
(integer1 127.0.0.1:6379> SADD aset 12 3 4 (integer) 4
127.0.0.1:6379> ZADD azset 1 a
(integer) 1
127.0.0.1:6379> keys *
1) "astr"
2) "aset"
3) "azset"
4) "alist"
5) "ahash"

Use save for immediate backup127.0.0.1:6379 > SAVE OKCopy the code
  1. View the generated RDB file and back it up to another folder
# View the backup folder
ls /usr/local/var/db/redis                                             
dump.rdb

# Save somewhere else
cp dump.rdb ~/back_2021.rdb
Copy the code
  1. Clearing the database
127.0.0.1:6379> FLUSHALL
OK
127.0.0.1:6379> keys *
(empty list or set)
Copy the code
  1. Restore data using backup files
Dump. RDB after clearing redis data, delete the dump
rm dump.rdb

# Stop redis service
redis-cli shutdown

Put the backup file in the dir directory specified in the configuration file
Rename the name specified by dbfilename
cp ~/back_2021.rdb dump.rdb

Start the redis service (note that the configuration file is specified, otherwise dir will change to the startup path)
# my configuration file is in/usr/local/etc/redis. Conf
redis-server /usr/local/etc/redis.conf
Copy the code
  1. View the RDB file loading logs when the service is started
16886:M 28 May 2021 14:19:56.230 # Server initialized
The following line is loaded from the RDB file
16886:M 28 May 2021 14:19:56.231 * DB loaded from disk: 0.001 seconds
16886:M 28 May 2021 14:19:56.231 * Ready to accept connections
Copy the code
  1. Check the data after backup
127.0.0.1:6379 > keys * 1)"astr"
2) "alist"
3) "aset"
4) "azset"
5) "ahash"
Copy the code

RDB backup features

  • Backup data is compressed, so backup files are small
  • Binary file, very poor readability
  • Fast recovery of backup data
  • Because BGSAVE backs up full data each time, the backup frequency is generally configured infrequently, and if something goes wrong, data can be lost for a period of time

AOF

AOF introduction and principle

AOF is short for Append Only File. Unlike RDB, which backs up the entire file at a time, AOF is a command that incrementally adds each operation to a file. Take a quick look at the beginning of an AOF file:

Aof -use-rdb-preamble = no* 2$6
SELECT  # ignore
The $10 * 6$4
SADD  # sadd
$4
aset  # aset
The $1
1     # 1
The $1
2     # 2
The $1
3     # 3
The $1
4     # 4* 8A $5
RPUSH  # rpush
A $5
alist  # alist
The $1
a      # a
The $1
b      # b
The $1
c      # c
The $1
d      # d
The $1
e      # e
The $1
f      # f
Copy the code

Insert key aset, alist… Those orders. After the AOF function is enabled, the operation command will be written into aOF_buf such an AOF memory buffer area after each operation of the database. At the end of each event cycle, the content of AOF_BUF will be written into the file. Then synchronize data periodically to hard disks according to the redis.conf synchronization policy. The appendfsync configuration parameter has three policies:

  • Always At the end of each event loop, it is synchronized to the hard disk file
  • Everysec more than one second after the last synchronization, syncs to disk file again (default)
  • No is never actively synchronized, but is synchronized to a disk file by the operating system

To improve file writing efficiency, when a user uses the write function to write data to a file, the operating system does not directly write the data to the disk. Instead, the operating system provides a file buffer and saves the data to the disk when the buffer is full. However, this operation has the risk of data loss, so the operating system provides the forced drop command, the user can control the flush before the buffer is full. The appendfsync synchronization in Redis refers to calling the flush command.

You can see that in the default configuration, AOF mode only loses one second of data at most.

AOF backup configuration and practice

configuration

The aOF-related configurations are as follows

Whether to enable AOF persistence. After AOF is enabled, the server will read AOF files for recovery during startup
appendonly yes

Aof and RDB are stored in the same directory
dir /usr/local/var/db/redis/

# AOF Specifies the name to save the persistence file
appendfilename "appendonly.aof"

If the aOF file with missing tail is still loaded
aof-load-truncated yes

# aOF rewrites the file immediately if it is twice as large as it was last rewrites
auto-aof-rewrite-percentage 100

Aof files below 64MB will not be rewritten
auto-aof-rewrite-min-size 64mb

When enabled, aof rewrites will be synchronized to hard disk every 32MB
aof-rewrite-incremental-fsync yes

# Whether to enable aOF blend mode
aof-use-rdb-preamble yes
Copy the code

Demonstrate recovery using AOF

The creation and backup of data is similar to the operation of RDB, except that the resulting file is appendone.aof

  1. The aOF file is backed up to another path
cp appendonly.aof ~/back_2021.aof
Copy the code
  1. Clearing the database
127.0.0.1:6379 > FLUSHALL OKCopy the code
  1. Restore using the backup AOF file

# stop service
redis-cli shutdown 

Delete the original aOF file
rm appendonly.aof

Copy the backup AOF file here
cp ~/back_2021.aof ./appendonly.aof

Start redis-server by specifying the configuration file
redis-server /usr/local/etc/redis.conf
Copy the code
  1. View the output log for loading the AOF file when the service is started.
31343:M 28 May 2021 16:24:49.923 # Server initialized
The following log shows the service recovering data from aOF
31343:M 28 May 2021 16:24:49.923 * DB loaded from append only file: 0.001 seconds
31343:M 28 May 2021 16:24:49.923 * Ready to accept connections
Copy the code
  1. View the restored data
127.0.0.1:6379 > keys * 1)"ahash"
2) "aset"
3) "astr"
4) "azset"
5) "alist"
Copy the code

Aof backup features

  • Data integrity, loss of up to one second of data
  • Because original commands are saved, data recovery requires replay commands, which slows the recovery
  • Special format command, readability is good

AOF rewrite

Because the AOF file is incremental, the file gets bigger and bigger as the service runs, which is obviously not reasonable. So Redis implements the function of AOF file rewriting to slim down AOF files.

What is rewriting?

More and more commands are executed, but in fact the data may be deleted and expired, deleted and expired data are no longer exist, and the aOF operation commands that retain them are obviously useless. Therefore, rewriting is actually to write a full copy of AOF according to the data stored in the current database and according to the FORMAT of AOF file, and replace the old AOF after completion.

How does rewriting work?

To create a new aOF for all the current keys, fork the child process instead of the main process. However, there is another problem. When the child process is being rewritten, the main process is still processing services. What about the operation commands generated during this process? The solution is to use a rewrite buffer, starting from the generation of the child process rewrite, subsequent operation commands also write a copy of the rewrite buffer. After the subprocess finishes rewriting, the main process is notified, and the main process writes the remaining commands from the rewrite buffer to the AOF file. The main process blocks when writing the commands of the rewrite buffer, and will not process other commands at this time. After processing, the new AOF file is replaced by the old AOF file. The general process is as follows:

Mixed persistence (introduced by redis4.0)

Aof -use-rdb-preamble Yes (Enabled by default in Redis 5.0) Mixed persistence means that aOF overwrites and eventually generates mixed AOF files in the format of [RDB file][Aof tail]. That is, the full amount of data overwritten by the child process is no longer stored in the AOF format, but in the RDB file format. After the notification is overwritten, the main process adds the remaining buffer commands to the end of the file, and of course the resulting file is still an.aof file. I personally don’t think hybrid persistence is a new way of persistence, it’s just an optimization for the slow recovery of AOF files. A mixed AOF file looks like this:

0000000 R E D I S 0 0 0 9 372 \t r e d i s 0000020 - v e r 005 5 . 0 . 3 372 \n r e d i 0000040 s - b i t s 300 @ 372 005 c t I m e 356 0000060 356 ذ ** '372 \b u s e d - m e m 30p 0000100 \t 020 \0 372 \f a o f - P r e a m b l 0000120  e 300 001 376 \0 373 005 \0 \0 004 a s t r \a a 0000140 s t r i n g \v 004 a s e t 020 002 \0 \0 0000160 \0 004 \0 \0 \0 001 \0 002 \0 003 \0 004 \0 \r 005 a 0000200 h a s h 017 017 \0 \0 \0 \f \0 \0 \0 002 \0 \0 0000220 362 002 362 377 \f 005 a z s e t 020 020 \0 \0 \0 0000240 \r \0 \0 \0 002 \0 \0 001 a 003 362 377 016 005 a l 0000260 i s t 001 035 035 \0 \0 \0 031 \0 \0 \0 006 \0 \0 0000300 001 a 003 001 b 003 001 c 003 001 d 003 001 e 003 001 0000320 f 377 377 237 345 230 % ̫ ** 220 r * 2 \r \n $0000340 6 \r \n S E L E C T \r \n $1 \r \n 0 0000360 \r \n * 3 \r \n S E T \r \n  $ 0000400 2 \r \n k 3 \r \n $ 2 \r \n k 3 \r \n 0000417Copy the code

​

You can see that it starts with R, E, D, I, S the same way that an RDB file starts, and it ends with an Aof file.

redis-dump

introduce

Redis-dump is an open source third-party tool that supports the import and export of Redis data in JSON format. The general principle is to read redis data and convert it to JSON format for storage during export, while import is to convert JSON data to Redis command for execution.

Install and use

Redis-dump is a ruby development tool, you need to install Ruby first

Install Ruby on MAC
brew install ruby
Copy the code

Replace ruby package management source with domestic

# to check the source
gem sources

Delete the old source
gem sources --remove  https://rubygems.org/ 

Add a new source
gem sources --add https://gems.ruby-china.org
Copy the code

Install redis – dump

gem install redis-dump
Copy the code

Export data

Redis -dump -u 127.0.0.1:6379 > data.jsonExport the specified database dataRedis -dump -u 127.0.0.1:6379 -d 15 > data.json# if redis has a passwordRedis-dump -u :[email protected]:6379 > data.jsonExport by key
redis-dump -u :xxxx@localhost:6379 -f adsl:* > data.json
Copy the code

Import data

> data.json redis-load -u 127.0.0.1:6379
# with password
> data.json redis-load -u 127.0.0.1:6379 -a password
Copy the code

Issues for attention

  1. Redis-dump is currently in beta, please test it before using it.
  2. The expiration time of exported data in redis-dump is incorrect. Procedure Here is the data format:
{"db":0."key":"hashkey"."ttl":- 1."type":"hash"."value": {"field_a":"value_a"."field_b":"value_b"."field_c":"value_c"},"size":42}
{"db":0."key":"listkey"."ttl":- 1."type":"list"."value": ["value_0"."value_1"."value_2"."value_0"."value_1"."value_2"]."size":42}
{"db":0."key":"setkey"."ttl":- 1."type":"set"."value": ["value_2"."value_0"."value_1"."value_3"]."size":28}
{"db":0."key":"zsetkey"."ttl":- 1."type":"zset"."value": [["value_0"."100"], ["value_1"."100"], ["value_2"."200"], ["value_3"."300"], ["value_4"."400"]],"size":50}
{"db":0."key":"stringkey"."ttl":79."type":"string"."value":"stringvalue"."size":11}
Copy the code

The expiration time is not a specific timestamp, and will be different from the original expiration time after import

Three ways to compare

Backup file size

rdb < aof < redis-dump

Recovery rate

rdb > aof > redis-dump

File readability

redis-dump > aof > rdb

Easy to operate

aof = rdb > redis-dump

Export by Specified Key

Only redis-dump supports exporting specified data based on certain key rules

conclusion

Data can be backed up and migrated in all three methods. RDB and AOF are mainly used for persistence and recovery, but can also be used for whole library migration. Redis-dump is a tool that can export data according to specified key rules. It can export and restore some keys in batches. However, it is not suitable for persistent data.

If you found this article helpful, please like it at 👍

Ps: Recommend a mall project: Experience, Github, Gitee