The following notes and experiments are from the video teaching of The Chinese Stone fir bigman, I followed the experiment, and sorted out the notes in the class.

Read more

  • Learning how to install and use simple commands in Redis

  • Experimental environment construction

  • More highly available documentation for Redis

Implications of Redis persistence for disaster recovery

Redis has a persistence function, there are many videos and materials on the introduction of persistence, so what is the significance of persistence for what scenario?

What happens when a failure occurs

In the real world, there are situations where Redis suddenly hangs up, the process dies, or the machine on which it is working is lost and has a catastrophic failure because redis data is stored in memory

The data in memory is gone, important cache data and so on, and redis will restart, and it will take a lot of work to recover after the restart, and if you just put the data in memory,

There is no way to deal with catastrophic failures, so persistence in Redis is important. Some data can be restored by backing up data periodically.

meaning

In the enterprise redis cluster architecture, the main meaning of persistence is to do disaster recovery, data recovery.

Redis RDB and AOF persistence

RDB persistence mechanism, periodic persistent storage of data in REDis, but also complete data storage. This means that if you store a copy of all the data in redis memory every hour.

The AOF mechanism logs each write command and writes it to a log file in appends-only mode. When Redis restarts, the entire data set can be reconstructed by playing back the write command in the AOF log

The mechanism of AOF only writes to one AOF log file (only appends append-only). In Liunx system, it will pass through OS cache before writing to disk. AOF file stores redis write instructions.

If the maximum memory size is 1 GB, the size of AOF files will be limited to 1 GB. When the memory size exceeds 1 GB, AOF uses a cache flushing algorithm (LRU) to clean up the least-used data, while the rewrite operation of AOF reconstructs a smaller AOF file based on the data in memory redis cleaned up, then deletes the old file.

If Redis fails, the server memory and disk data are lost. You can copy back the previous data from the cloud service, put it in the specified directory, and then restart Redis. Redis will automatically restore the data in memory according to the data in the persistent data file. If both RDB and AOF persistence mechanisms are used, when Redis restarts, AOF will be used to rebuild the data, because the data in AOF is more complete.

Advantages of RDB persistence mechanism

  • RDB will generate multiple data files, and each data file represents the data of REDis at a certain time. This method of multiple data files is very suitable for cold backup. Such complete data files can be sent to some remote secure storage, such as Amazon S3 cloud service. In China, it can be on the ODPS distributed storage of Ali Cloud to regularly back up data in Redis with a predetermined backup strategy

  • RDB has very little impact on the read and write services provided by Redis, so that Redis can maintain high performance, because the main redis process only needs to fork a sub-process and let the sub-process perform disk I/O operations for RDB persistence

  • Compared to AOF persistence, restarting and restoring redis processes directly based on RDB data files is much faster

Disadvantages of the RDB persistence mechanism

  • RDB is not as good as AOF if you want to lose as little data as possible when Redis fails. In general, RDB data snapshot files are generated every 5 minutes or more, at which point you have to accept that if the Redis process goes down, the data in the last 5 minutes will be lost

  • If the RDB forks the RDB snapshot data file every time, if the data file is very large, the service provided to the client may be suspended for milliseconds, or even seconds, so do not persist the operation at long intervals.

Advantages of AOF persistence mechanism

  • AOF can better protect against data loss. Generally, AOF will execute fsync operation every second through a background thread and lose data for a maximum of one second

  • AOF log files are written in appends-only mode, so there is no disk addressing overhead, write performance is very high, and the file is not prone to breakage, and even if the tail of the file is broken, it is easy to repair

  • Even if the AOF log file is too large, the background rewrite operation does not affect the client read and write. When the rewrite log was written, the guidance was compressed to create a minimal log that needed to be retrieved. When a new log file is created, the old log file is written as usual. When the log files after the merge are ready, the old and new log files can be exchanged.

  • Commands for AOF log files are logged in a very readable manner, which is ideal for emergency recovery in the event of catastrophic deletions. Flushhall flushes all data in the flushhall file. Rewrite in the background has not yet happened. Flushhall deletes the last item in the AOF file and then flushes the AOF file back

Disadvantages of AOF persistence mechanism

  • AOF log files are usually larger than RDB data snapshot files for the same data

  • When AOF is enabled, the write QPS supported is lower than the write QPS supported by RDB, because AOF is typically configured to fsync log files once per second, although the performance is still very high

  • In the past, there was a bug in AOF, that is, the same data was not recovered when the logs recorded by AOF were recovered. Therefore, a more complex command log /merge/ playback approach such as AOF is more vulnerable and buggy than the rDB-based approach of persisting a complete data snapshot file at a time. AOF, however, is designed to avoid bugs in the rewrite process, so instead of merging the rewrite log, rewrite it based on the data in memory at the time, which is much more robust.

How to choose BETWEEN RDB and AOF

  • Don’t just use RDB, because that will cause you to lose a lot of data

  • Also don’t just use AOF, because there are two problems with that. First, if you do cold standby through AOF, you can recover faster without RDB. Second, RDB is more robust by simply generating snapshots each time, avoiding the bugs of complex backup and recovery mechanisms such as AOF

  • AOF and RDB persistence mechanisms are used in a comprehensive way. AOF is used to ensure that data is not lost as the first choice for data recovery. RDB is used for varying degrees of cold backup and for quick data recovery when AOF files are lost or corrupted and unavailable

Implement Redis persistence

First, implement both types of persistence in Redis, and do some experiments to test whether they work. AOF persistence is turned off by default, RDB persistence is turned on by default

The configuration file path and all subsequent paths are my own. Their configuration file where they go to match it, all the paths below are according to their own path, do not blindly according to my, unless you according to my environment to build.

RDB persistence

How to configure the RDB persistence mechanism

Conf file, also known as /etc/redis/6379.conf, to configure persistence,

To find a similar configuration item, use /save in vi to find the specified save character :noh unhighlight

save 60 1000

Every 60 seconds, if more than 1000 keys are changed, a new dump. RDB file is generated, which is the complete snapshot of the current redis memory. This operation is also known as snapshotting.

RDB snapshot generation can also be performed synchronously or asynchronously by manually invoking the save or bgsave commands

Save can be set to multiple, i.e. multiple snapshotting checkpoints. Each checkpoint is checked to see if the specified number of keys has changed. If so, a new dump. RDB file is generated

Workflow of RDB persistence mechanism

  • Redis tries to generate the RDB snapshot file itself based on the configuration
  • Fork a child process
  • The child process attempted to dump data into a temporary RDB snapshot file
  • After the RDB snapshot file is generated, the old snapshot file is replaced

Dump. RDB, which overwrites the old snapshot each time a new snapshot is created

Data recovery experiment based on RDB persistence mechanism

  1. In experiment 1, first use the command redis-cli to enter the redis command line, save several pieces of data and restart Redis to see whether the data can still be queried.
$ redis-cli
$ set k1 v1
$ set k2 v2
$ exit
$ redis-cli SHUTDOWN
Go to /etc/init.d and restart Redis using redis_6379
$ cd /etc/init.d
$ ./redis_6379 start
$ redis-cli
$ get k1
"v1"
$ get k2
"v2"
Copy the code

After the above command is executed, it can be seen that if the redis exits normally, the data still exists. This is because stopping Redis through redis- CLI SHUTDOWN is actually a safe exit mode. When Redis exits, the data in the memory will immediately generate a complete RDB snapshot

  1. In experiment 2, store several data in Redis again. This time, we use kill -9 to kill the redis process and delete /var/run/redis_6379.pid file to see whether the data can be queried
$ set k3 v3
$ set k4 v4
$ exit
# by ps - ef | grep redis view redis process
$ ps -ef | grep redis
$ kill62789-9Note that -rf will not prompt you to delete the file
$ rm -rf /var/run/redis_6379.pid 
# Restart Redis again
$ cd /etc/init.d
$ ./redis_6379 start
$ redis-cli
$ get k3
(nil)
$ get k4
(nil)
Copy the code

This time, there is no new data saved, because before the checkpoint of the configuration, there is an accident, at this time, the RDB file cannot be saved, so the file is lost, so this is also a disadvantage of RDB.

  1. In Experiment 3, set savepoints with short intervals, restart the Redis process, and repeat the steps in Experiment 2
$ vi /etc/redis/6379.conf
# Find save 60 1000 after similar Settings add the following content save exit
save 5 1
# indicates that the snapshot file RDB is generated every 5 seconds if only one data is inserted.
$ redis-cli SHUTDOWN
$ /etc/init.d/redis_6379 start
Repeat the steps in Experiment 2 and insert new data
Copy the code

Experiment 3 shows that RDB persistence does save data in memory through manual setting, but it is not recommended to configure checkpoints with such a short interval

AOF persistence

AOF persistence is turned off by default, RDB persistence is turned on by default

How do I configure persistence of AOF

Appendonly No in the configuration file, set it to yes to enable AOF persistence. In a production environment, AOF is normally enabled unless you say it doesn’t matter if you lose data for a few minutes. Data is first written to the OS cache, and then fsync at regular intervals. Even if both AOF and RDB are enabled, data is recovered through AOF first when Redis restarts, because AOF data is relatively complete

AOF’s FSYNC strategy

Appendfsync Everysec in the redis configuration file has three policies available for us to use, default everysec, which we use most often

  • Always: Every time a piece of data is written, the corresponding log fsync is immediately sent to the disk. The performance is very poor and the throughput is low. If you have to make sure that none of the data in Redis is lost, that’s it

  • Everysec: Fsync data from the OS cache to disk every second. This is the most common configuration in production environments, with high performance and QPS of tens of thousands

  • No: Only Redis is responsible for writing data to the OS cache, and then the OS will have its own policy to flush data to disk from time to time

AOF persistent data recovery experiment

  1. /var/run/redis_6379.pid: /var/run/redis_6379.pid: /var/run/redis_6379.pid: /var/run/redis_6379.pid: /var/run/redis_6379.pid: /var/run/redis_6379.pid: /var/run/redis_6379.pid: /var/run/redis_6379.pid: /var/run/redis_6379.pid: /var/run/redis_6379.pid: This will not write command, according to the above RDB experiment can be.

  2. In Experiment 2, turn on the switch of AOF, enable the persistence of AOF, restart the Redis process, write some data, observe the log content in the AOF file, kill -9 kill the Redis process, restart the Redis process, and find that the data is recovered, that is, from the AOF file. /var/redis/6379 /var/redis/6379 /var/redis/6379 /var/redis/6379 /var/redis/6379 /var/redis/6379

$ vi /etc/redis/6379.conf
appendonly yes
$ redis-cli SHUTDOWN
$ /etc/init.d/redis_6379 start
$ redis-cli
$ set mykey1 k1
$ set mykey2 k2
$ exit
$ ps -ef | grep redis
$ kill -9 32523
$ rm -rf /var/run/redis_6379.pid
$ /etc/init.d/redis_6379 start
$ redis-cli
$ get mykey1
"k1"
$ get mykey2
"k2" 
Copy the code

Experiments proved that after we enabled AOF, we could normally save our data in the event of an accident, while RDB did not save our data because it did not reach the checkpoint, so it was necessary to enable both RDB and AOF persistence in the production environment

AOF rewrite operations

Data in Redis is actually limited, many data may automatically expire, may be deleted by the user, may be redis with the cache clearing algorithm to clear, redis data will continue to eliminate the old, some commonly used data will be automatically retained in redis memory

Therefore, it is possible that a lot of data that has been cleaned up before, the corresponding log is still in the AOF, and the AOF log file is only one, which will continue to expand to very large

So AOF automatically does the rewrite operation in the background at regular intervals, such as when 100W data is already in the log; Redis has only 100,000 memory left; Build a set of latest logs based on the current 100,000 data in memory, into the AOF; Overwrite the previous old log; Make sure the AOF log files are not too large and keep the same amount of redis memory data

Before Redis 2.4, you would need to manually write a script (crontab) to rewrite AOF using the BGREWRITEAOF command, but after Redis 2.4, the rewrite will be automated

In the redis configuration file we can configure the rewrite policy

  • Auto-aof-rewrite-percentage 100. This property represents percentage. If the percentage increases to 100%, it doubles the size of the file since the last rewrite
  • Auto-aof -rewrite-min-size 64mb. This attribute indicates that the total size increased is greater than 64mb

For example,

For example the last AOF rewrite, is will then 128 MB to 128 MB and write AOF log, if found that the proportion of growth, more than 100% before, 256 MB, is likely to trigger a rewrite to min – but also the size, 64 MB to compare, 256MB > 64MB will trigger rewrite

Rewrite Workflow

  • Redis fork a child process
  • The child process builds the log based on the data currently in memory and starts writing the log to a new temporary AOF file
  • The main redis process, after receiving the new client write operation, writes to the memory log, and the new log continues to write to the old AOF file
  • After the child process writes the new log file, the main redis process appends the new log in memory to the new AOF file again
  • Replace the old log file with the new one

Repair of damaged files of AOF

If Redis goes down while append data is being sent to AOF files, the AOF files may be damaged

Fix broken AOF files with the redis-check-aof –fix command, which exists in the SRC file of the redis installation package

The repair of this broken file is to delete the last incomplete command in the file, so the repair will also cause some data loss, but it is better than all data loss.

AOF and RDB work simultaneously

  • If RDB does snapshotting, redis will not do AOF rewrite; If Redis were to do AOF rewrite, it would not do RDB snapshotting
  • If the RDB is running snapshotting and the BGREWRITEAOF command is executed, AOF rewrite will be executed only after the RDB snapshot is generated
  • If there are RDB snapshot files and AOF log files at the same time, when Redis restarts, AOF will be used for data recovery first because the logs are more complete
  1. Experiment 3,
  • Appendonly of RDB (dump) and AOF (appendonly of RDB) with some data in RDB and some data in AOF (appendonly of RDB)
  • We simulate breaking the AOF, manually deleting the instruction after the AOF file becomes incomplete, and then fix, one piece of data will be deleted by Fix
  • Rebooting redis with the aOF file repaired by fix, only one data is left

Redis uses aOF files to restore data preferentially, so RDB files will not be restored when aOF files are present

conclusion

In general, Redis is very important as a cache in our project, sharing a lot of pressure for our mysql and other databases, and the data in the cache is the most important, so the persistence in Redis for us is a bit of knowledge we must know

This chapter covers the following points

  • What does persistence mean?

  • How to deal with a catastrophic accident?

Use persistence and periodic backups to prepare for a disaster

  • The pros and cons of two types of persistence

By the above two way of persistence, data recovery depends entirely on the underlying disk persistent, if no data on RDB and aof, or the two files are lost, really not so So in the case of conditional, needs to be suitable for cold standby RDB file periodically backup is to do the last line of defense