What is persistence?

Persistence can be summed up in one sentence as saving data, such as objects in memory, to a storage device that can last forever.

The main application of persistence is to store objects in memory in a database, in disk files, XML data files, and so on.

Persistence can also be understood in two ways:

  • Application layer: If you Close your application and then restart it, the previous data still exists.
  • System layer: If you Shut Down your system and restart it, the previous data still exists.

Why is Redis persistent?

The data types in Redis all support Push/Pop, Add/Remove, intersection union and difference sets, and richer operations, and these operations are atomic.

On this basis, Redis supports sorting in various ways. As with Memcached, the data is cached in memory for efficiency.

Because data is cached in memory, when you restart the system or shut it down, the data in the cache is gone forever.

Therefore, in order to make the data can be stored for a long time, it is necessary to store the data in Redis cache for persistent storage.

How does Redis persist?

Redis had this in mind from the beginning of the design. There are several ways to persist data at different levels:

  • RDB persistence allows you to take snapshots of your data at specified intervals.
  • The AOF persistence mode records each write operation to the server. When the server is restarted, these commands will be executed again to restore the original data. The AOF command saves each write operation to the end of the file using the Redis protocol.

Redis can also rewrite AOF files in the background so that AOF files are not too large.

  • If you only want your data to exist as long as the server is running, you can do so without any persistence.
  • You can also enable both persistence methods. In this case, when Redis restarts, AOF files will be loaded first to recover the original data, since AOF files usually hold more complete data sets than RDB files.

If you don’t know which level of persistence to choose, let’s take a look at the differences between AOF and RDB, and the pros and cons of each, and then decide which level to choose.

Comparison of advantages between RDB mode and AOF mode

Comparison of advantages between RDB and AOF

First of all, let’s take a look at the official description of the advantages of the two ways, and make a comparison, and then take a look at the disadvantages of the two ways.

Advantages of RDB mode:

  • RDB is a very compact file that holds data sets at a point in time, making it ideal for backing up data sets.
  • For example, you can save the last 24 hours of data every hour and the last 30 days of data every day, so that if something goes wrong you can revert to different versions of the data set depending on your needs.
  • RDB is a compact single file that can be easily transported to another remote data center, making it ideal for disaster recovery.
  • When RDB saves RDB files, the only thing the parent process needs to do is Fork out a child process. The child process does all the following work. The parent process does not need to do other IO operations, so THE RDB persistence method can maximize the performance of Redis.
  • Compared to AOF, RDB is faster for recovering large data sets.

When Redis needs to save the dump. RDB file, the server does the following:

  • Redis calls Forks and has both a parent and child process.
  • The child process writes the data set to a temporary RDB file.
  • When the child process finishes writing to the new RDB file, Redis replaces the old RDB file with the new one and deletes the old RDB file.

This way of working allows Redis to benefit from copy-on-write mechanisms.

Advantages of AOF:

  • Using AOF will make your Redis more durable.
  • You can use different Fsync policies: no Fsync, Fsync per second, and the default Fsync per second policy for each write.

Redis still performs very well (Fsync is handled by background threads and the main thread does its best to handle client requests), and you can lose up to a second of data if it fails.

  • The AOF file is an append only log file, so there is no need to write Seek. Even if the full write command is not executed for some reason (disk space is full, the write process is down, etc.), you can use the redis-check-aof tool to fix these problems.
  • Redis can automatically rewrite the AOF in the background when the AOF file becomes too large: the rewritten new AOF file contains the minimum set of commands needed to restore the current data set.

The entire rewrite operation is absolutely safe because Redis continues to append commands to existing AOF files while creating new AOF files, and the existing AOF files will not be lost even if an outage occurs during the rewrite.

Once the new AOF file is created, Redis switches from the old AOF file to the new AOF file and starts appending the new AOF file.

  • The AOF file holds all writes to the database in an orderly fashion, in the Redis protocol format.

As a result, the contents of AOF files are easy to read and parse. Exporting AOF files is also very simple.

For example, if you accidentally execute the FLUSHALL command, but as long as the AOF file hasn’t been overwritten, you can restore the data set to its pre-flushall state by stopping the server, removing the FLUSHALL command at the end of the AOF file, and restarting Redis.

Advantages comparison summary:

  • The RDB mode can save the data in the past period of time and save the result as a single file, which can be backed up to other servers. In addition, the RDB mode can recover a large amount of data faster than the AOF mode.
  • By default, AOF backs up once a second, which is very frequent. It operates by apending logs rather than data, and its rewriting process apends sequentially, so the contents of files are very easy to read.

You can open AOF files to edit them, add or delete records, and then restore them as needed.

The disadvantages of RDB and AOF are compared

Disadvantages of RDB mode:

  • RDB is not for you if you want to minimize data loss in the event that Redis unexpectedly stops working, such as a power outage.

While you can configure different Save times (e.g., every 5 minutes and 100 writes to the dataset), Redis is a bit of a chore to Save the entire dataset in its entirety.

You usually do a full save every 5 minutes or more, and if Redis goes down unexpectedly, you may lose a few minutes of data.

  • The RDB often forks the child process to save the data set to the hard disk. When the data set is large, the Fork process is very time-consuming and may cause Redis to fail to respond to client requests for some milliseconds.

If the data set is large and the CPU performance is not very good, this can last up to a second. AOF also requires forking, but you can adjust the frequency of rewriting log files to improve the durability of the data set.

Disadvantages of the AOF approach:

  • AOF files are usually larger than RDB files for the same data set.
  • Depending on the Fsync strategy used, AOF may be slower than RDB. Fsync per second performance is still very high under normal conditions, and turning off Fsync allows the AOF to be as fast as the RDB, even under high loads.

However, RDB can provide more guaranteed maximum Latency when handling large write loads.

Disadvantages comparison summary:

  • RDB backup frequency is not high, so data may be lost for a short period of time during data recovery, and may affect millisecond requests when the data set is large.
  • AOF file reference is relatively large, and because of the high saving frequency, so the overall speed will be slower than RDB, but the performance is still very high.

RDB and AOF working principle

AOF overrides, like RDB snapshots, make clever use of copy-on-write mechanisms:

  • Redis executes fork() and now has both parent and child processes.
  • The child process starts writing the contents of the new AOF file to the temporary file.
  • For all newly executed write commands, the parent process accumulates them into an in-memory cache and appends the changes to the end of the existing AOF file, so that the existing AOF file is safe even if an outage occurs in the middle of a rewrite.
  • When the child process finishes rewriting, it sends a signal to the parent process, which, upon receiving the signal, appends all the data in the memory cache to the end of the new AOF file.
  • Redis now atomically replacing old files with new ones, after which all commands are appended directly to the end of new AOF files.

Put into practice, RDB and AOF implementation

Enable and configure RDB persistence

Redis’ default persistence is RDB and is turned on by default. RDB can be saved in active and passive ways.

You can enter Save in redis-CLI. Passive saving must meet the trigger conditions set in the configuration file. The default trigger conditions can be seen in the redis.conf file:

save 900 1save 300 10save 60 10000

Its meaning is:

The server made at least one change to the database within 900 seconds. The server made at least 10 changes to the database within 300 seconds. The server made at least 10,000 changes to the database within 60 seconds.

When the trigger condition is met, the data is saved as a snapshot, which is why RDB has less data integrity than AOF.

After the save condition is triggered, a file named dump. RDB is generated in the specified directory. When Redis is started next time, Redis reads the dump. RDB file in this directory and restores the data in the file to Redis.

Where is this catalogue? We can enter the command config get dir in the client to check:

 Copy the code
  1. gannicus@$ src/redis-cli 
  2. 127.0.0.1:6379 > config get dir
  3. 1) “dir” 
  4. 2) “/ home/gannicus/Documents/redis – 5.0.0”
  5. 127.0.0.1:6379 >

Returns the result of “/ home/gannicus/Documents/redis – 5.0.0” is to store the dump. RDB directory.

Before the test, please note the following preconditions: Redis is a compressed package directly downloaded from the official website. After decompression, you can obtain the redis-x.x.x folder.

For example, mine is redis-5.0.0, then go to the folder and install it in the root directory of the redis-5.0.0 project using make command.

RDB passively triggers save tests

As mentioned earlier, it is divided into active saving and passive triggering. Now let’s test the passive triggering. Start redis-server first, then open client redis-cli, add a few records:

127.0.0.1:6379> set LCA 1ok127.0.0.1:6379 > set LCB 1ok127.0.0.1:6379 > set LCC 1OK127.0.0.1:6379> set LCD 1ok127.0.1:6379 > set LCE 1OK127.0.1:6379 > set LCF 1OK127.0.1:6379 > set LCG 1OK127.0.1:6379 > set LCH 1ok127.0.1:6379 > set LCI 1ok127.0.1:6379 > set LCJ 1ok127.0.1:6379 > set LCK 1ok127.0.1:6379 > set LCL 1 ok127. 0.0.1:6379 > set LCM 1 ok

As you can see, 13 records have been added in total:

127.0.0.1:6379 > keys * 1) “lca” 2) “LCD” 3) “the LCG” 4) “lce” 5) “LCB” 6) “LCM” 7) “LCF” 8) “lci” 9) “LCL” 10) “LCC 11)” “LCK 12) LCJ” 13) “LCH” 127.0.0.1:6379 >

Then, the following message appears in the log window of redis-server:

21971:M 21 Oct 2018 16:52:44.062 * 10 Changes in 300 seconds. 21971:M 21 Oct 2018 16:52:44.063 * Background saving started by PID 2255222552:C 21 Oct 2018 16:52:44.066 * DB saved on Disk21971 :M 21 Oct 2018 16:52:44.165 * Background saving terminated with success

It detects that 10 records have been changed within 300 seconds. We have just added 13 data records to redis.conf, which meets the conditions for saving RDB data.

So the data save operation is executed, and the prompt that a 22552 process is opened to perform the save operation, and finally the prompt that the save is successful. The dump. RDB file is generated in the directory.

Now that the Redis process is killed, what data will be saved? Run the kill -9 pid command (pid indicates the process ID) to simulate the abnormal shutdown of Redis, and then start Redis.

Let’s see, are there only 10 records or are there all 13 records?

127.0.0.1:6379 > keys * 1) “LCB” 2) “LCJ” 3) “LCD” 4) “LCH” 5) “lci” 6) “LCC” 7) “LCF” 8) “lce” 9) “lca 10)” “The LCG” 127.0.0.1:6379 >

After the restart, only 10 records out of 13 records will be saved, which confirms the previous statement that data integrity in RDB mode is not reliable unless the moment of breaking is exactly the number of records that meet the trigger condition.

Close the RDB

As mentioned earlier, it is enabled by default. If you don’t need it, you can comment out the 3 configurations in the configuration file and add “save” :

 Copy the code
  1.   save “” 
  2. # save 900 1 
  3. # save 300 10 
  4. # save 60 10000 

After saving the configuration file, you need to restart the Redis service to take effect, and then continue to add a dozen records:

 Copy the code
  1. 127.0.0.1:6379 > keys *
  2.  1) “lcb” 
  3. .
  4. 23) “lca” 
  5. 24) “lcg” 
  6. 127.0.0.1:6379 >

I added 14 more records on the basis of the previous 10. This time, I will also simulate the abnormal shutdown of Redis through kill, and then start the service to see whether the data is still saved:

 Copy the code
  1. 127.0.0.1:6379 > keys *
  2.  1) “lcb” 
  3.  2) “lcj” 
  4.  3) “lcd” 
  5.  4) “lch” 
  6.  5) “lci” 
  7.  6) “lcc” 
  8.  7) “lcf” 
  9.  8) “lce” 
  10.  9) “lca” 
  11. 10) “lcg” 
  12. 127.0.0.1:6379 >

It was found that the later 14 records were not saved, and only the previous 10 records were restored.

And observed the Redis server window log, did not find the same trigger save prompt as before, proving that the RDB mode has been closed.

RDB actively saves tests

Does active shutdown still work with passive triggering through configuration files?

Delete several records on the Redis client (redis-CLI) by using the del command, and then enter the save command to perform the save operation:

 Copy the code
  1. 127.0.0.1:6379 > keys *
  2.  1) “lcc” 
  3.  2) “lch” 
  4.  3) “lcb” 
  5.  4) “lci” 
  6.  5) “lce” 
  7.  6) “lcj” 
  8.  7) “lcg” 
  9.  8) “lca” 
  10.  9) “lcd” 
  11. 10) “lcf” 
  12. 127.0.0.1:6379> del LCA LCB LCC
  13. (integer) 3 
  14. 127.0.0.1:6379 > save
  15. OK 
  16. 127.0.0.1:6379 >

22598:M 21 Oct 2018 17:22:31.365 * DB saved on disk, which tells us that the data has been saved.

So go ahead and simulate an abnormal shutdown, open the service again, and see if these actions are really saved:

 Copy the code
  1. 127.0.0.1:6379 > keys *
  2. 1) “lci” 
  3. 2) “lcj” 
  4. 3) “lcd” 
  5. 4) “lcg” 
  6. 5) “lcf” 
  7. 6) “lce” 
  8. 7) “lch” 
  9. 127.0.0.1:6379 >

Sure enough, the deletions were saved and the three records were not in the recovered data, proving that the active shutdown was not affected by the configuration file. Is there any other way to Save besides Save?

Save and Bgsave Save

Yes, Redis provides two different Save and Bgsave methods, and both methods call the rdbSave function when executed.

But they all call in different ways:

  • Save calls the rdbSave method directly, blocking the main Redis process until the Save is complete. The server cannot process any requests from the client while the main process is blocked.
  • Bgsave forks a child process that calls rdbSave and sends a signal to the main process when the save is complete.

Because rdbSave is invoked in the child process, the Redis server can continue to process client requests during Bgsave execution.

Save is a synchronous operation and Bgsave is an asynchronous operation. The Bgsave command is used in the same way as the Save command:

 Copy the code
  1. 127.0.0.1:6379 > keys *
  2. 1) “lci” 
  3. 2) “lcj” 
  4. 3) “lcd” 
  5. 4) “lcg” 
  6. 5) “lcf” 
  7. 6) “lce” 
  8. 7) “lch” 
  9. 127.0.0.1:6379 > del lci LCJ
  10. (integer) 2 
  11. 127.0.0.1:6379 > bgsave
  12. Background saving started 
  13. 127.0.0.1:6379 > keys *
  14. 1) “lcd” 
  15. 2) “lcg” 
  16. 3) “lcf” 
  17. 4) “lce” 
  18. 5) “lch” 
  19. 127.0.0.1:6379 >

Shutdown to save

In fact, the Shutdown command can save data as well. It saves the data before it shuts down, no surprise?

 Copy the code
  1. 127.0.0.1:6379 > set app. 1
  2. OK 
  3. 127.0.0.1:6379 > 1 set apps
  4. OK 
  5. 127.0.0.1:6379 > keys *
  6. 1) “apps” 
  7. 2) “lcd” 
  8. 3) “lcg” 
  9. 4) “lcf” 
  10. 5) “app” 
  11. 6) “lce” 
  12. 7) “lch” 
  13. 127.0.0.1:6379 > shutdown
  14. not connected> quit 
  15. gannicus@$  

The Redis service was then shut down. We need to restart the Redis service and go to the client to see if it works:

 Copy the code
  1. gannicus@$ src/redis-cli 
  2. 127.0.0.1:6379 > keys *
  3. 1) “lce” 
  4. 2) “lcf” 
  5. 3) “lcd” 
  6. 4) “lch” 
  7. 5) “lcg” 

It didn’t work. Did it sting? Why is that? Shutdown Shutdown Shutdown Shutdown Shutdown Shutdown Shutdown Shutdown Shutdown Shutdown Shutdown

If persistence is enabled, you can use Shutdown to disable it without losing data. If persistence is enabled, you can use Shutdown to disable persistence.

 Copy the code
  1. #   save “”save 900 1 
  2. save 300 10 
  3. save 60 10000 

Then start the Redis service again and try again (process: Add -> shutdown -> Restart service -> View) :

 Copy the code
  1. 127.0.0.1:6379 > set app. 1
  2. OK 
  3. 127.0.0.1:6379 > 1 set apps
  4. OK 
  5. 127.0.0.1:6379 > shutdown
  6. not connected> quit 
  7. gannicus@$ src/redis-cli 
  8. 127.0.0.1:6379 > keys *
  9. 1) “lce” 
  10. 2) “lch” 
  11. 3) “app” 
  12. 4) “lcf” 
  13. 5) “apps” 
  14. 6) “lcd” 
  15. 7) “lcg” 
  16. 127.0.0.1:6379 >

Well, I finally figured it out.

Enable and configure AOF persistence

Open AOF

AOF is disabled by default. If you want to enable AOF, go to the redis.conf file and open the redis.conf file:

 Copy the code
  1. $ vim redis.conf 

Then find appendOnly in the file and change no to yes:

 Copy the code
  1. appendonly yes 

AOF mode persistence is enabled.

Setting the Synchronization Mode

AOF also supports several synchronization modes, which are:

 Copy the code
  1. Appendfsync always # Write to the AOF file every time a data change occurs (safe but time-consuming).
  2. Appendfsync everysec # synchronizes once per second, which is the default policy for AOF.
  3. Appendfsync no # Never synchronize. Efficient but data is not persisted.

The default is Everysec, which you can change to suit your needs. Here I changed the configuration to always:

 Copy the code
  1. appendfsync always 
  2. # appendfsync everysec 
  3. # appendfsync no 

Custom AOF record file name

Redis has a default file name, which appears in the configuration as:

 Copy the code
  1. appendfilename “appendonly.aof” 

You can keep the default name or specify a different filename, such as:

 Copy the code
  1. appendfilename “RNGLetme.aof” 

Appendonly, appendfSync, and appendfilename are set and saved. Restart the Redis service:

 Copy the code
  1. $./redis-server 

Rngletme. aof: rngletme. aof: RNGLetme.

 Copy the code
  1. $cat RNGLetme.aof 

To view the contents inside, because the current data is not changed, so it is blank. Then open the Redis client:

 Copy the code
  1. $./redis-cli 

And add a few data records:

 Copy the code
  1. 127.0.0.1:6379 > set RNG LPL
  2. OK 
  3. 127.0.0.1:6379 > set ig LPL
  4. OK 
  5. 127.0.0.1:6379 > set edg LPL
  6. OK 
  7. 127.0.0.1:6379 > keys *
  8. 1) “edg” 
  9. 2) “rng” 
  10. 3) “ig” 
  11. 127.0.0.1:6379 >

RNGLetme. Aof = RNGLetme. Aof = RNGLetme.

 Copy the code
  1. * 2
  2. $6 
  3. SELECT 
  4. The $1
  5. 0 
  6. * 3
  7. $3 
  8. set 
  9. $3 
  10. rng 
  11. $3 
  12. lpl 
  13. * 3
  14. $3 
  15. set 
  16. $2 
  17. ig 
  18. $3 
  19. lpl 
  20. * 3
  21. $3 
  22. set 
  23. $3 
  24. edg 
  25. $3 
  26. lpl 

Every data addition is recorded. What about delete operations? Are they recorded?

 Copy the code
  1. 127.0.0.1:6379 > del edg
  2. (integer) 1 
  3. 127.0.0.1:6379 > keys *
  4. 1) “rng” 
  5. 2) “ig” 
  6. 127.0.0.1:6379 >

After the deletion, look again at the records in the rngletme. aof file:

Compared with previous records, the operation record of Del EDg is added. This confirms the previous description of AOF: data changes are recorded as a log.

AOF recovery test

The following is also the simulation of Redis abnormal shutdown through the Kill command:

 Copy the code
  1. gannicus@$ kill -9 22645 

Then restart the Redis service:

 Copy the code
  1. $ src/redis-server redis.conf 

Then check with the client to see if all the data is there:

 Copy the code
  1. $ src/redis-cli 
  2. 127.0.0.1:6379 > keys *
  3. 1) “ig” 
  4. 2) “rng” 

As you can see, both RNG and IG are still in place, which means persistence is in effect.

How to switch from RDB mode to AOF mode

In Redis 2.2 or above, it is possible to switch from RDB to AOF without rebooting:

Create a backup of the latest dump. RDB file and put it in a safe place.

Run the following two commands:

 Copy the code
  1. redis-cli config set appendonly yes 
  2. Redis -cli config set save “”

Make sure the write command is correctly appended to the end of the AOF file. The first command executed turns on AOF: Redis blocks until the initial AOF file is created, after which Redis continues processing command requests and begins appending write commands to the end of the AOF file.

The second command is used to disable the RDB function. This step is optional, or you can use both RDB and AOF if you prefer.

Note: Don’t forget to turn on the AOF feature in redis.conf! Otherwise, after the server restarts, the previous configuration SET by using the CONFIG SET command will be forgotten and the program will start the server with the original configuration.

Is RDB preferred or AOF preferred?

After analyzing and comparing the two approaches and testing them, we found that these are two different styles of persistence. So how to choose?

  • For medium to large enterprise applications, if you don’t want to sacrifice data integrity but still want to be efficient, you should use both RDB and AOF.
  • If you don’t want to waste energy on this and just need to ensure data integrity, then using AOF is a priority.
  • RDB is suitable for large-scale data recovery. If services do not have high requirements on data integrity and consistency, RDB is a good choice.

Suggestions for backing up Redis data

Make sure you have a full backup of your data. Disk failures, node failures, and other problems can cause your data to disappear. Not backing up your data can be very dangerous.

Redis is very friendly for data backup because you can copy RDB files while the server is running: once the RDB files are created, no changes are made to them.

When the server creates a new RDB file, it saves the contents of the file in a temporary file. When the temporary file has been written, rename(2) replaces the original RDB file atomically with the temporary file.

This means that copying RDB files is perfectly safe whenever possible:

  • Create a cron job that backs up one RDB file to one folder every hour and one RDB file to another folder every day.
  • Here I recommend an architecture learning exchange circle: 830478757 to help break J bottleneck and improve thinking ability
  • Ensure that snapshots are backed up with date and time information. Use the Find command to delete expired snapshots each time you execute a periodic task script. For example, you can keep snapshots every hour within the last 48 hours and daily snapshots within the last one or two months.
  • At least once a day, back up your RDB outside of your data center, or at least outside of the physical machine on which your Redis server is running.

Redis Password persistence

Data is persisted in Redis, and passwords are persisted. Run the following command on the client:

 Copy the code
  1. config set requirepass zxc9527 

You can set a password for Redis to zxC9527, but when Redis is turned off and restarted, the permission verification function is disabled and the password is no longer required.

Therefore, passwords also need to be persisted in redis.conf. Open redis.conf to find the requirepass configuration item, uncomment it and set the password behind it:

 Copy the code
  1. requirepass zxc9527 

After saving the password, restart the Redis service. The password persistence takes effect.