Redis persists meaning

Redis is an in-memory database. Data is basically stored in memory. The problem is that once the server process exits, the state of the database in the server will also disappear. Therefore, data needs to be saved to disks or even remote cloud services. When data needs to be restored, data can be directly restored based on the backup file. Redis provides two solutions for persistence to disk: RDB and AOF.

RDB

An RDB file is a snapshot file that periodically generates data. When the Redis process restarts, it automatically reads the RDB file for data restoration. Redis provides a persistent solution for RDB files by default.

RDB file generation and loading

RDB file generated

  1. Save: The save command blocks the Redis server process until the RDB file is created, and the server cannot process any command requests while the server process is blocked
  2. Bgsave: The BGSave command gives birth to a child process, which then creates the RDB file, and the server process (the parent process) continues to process the command request

Generating process

  • Redis tries to generate the RDB snapshot file itself based on the configuration
  • Fork a child process
  • The child process attempted to dump data into a temporary RDB snapshot file
  • After the RDB snapshot file is generated, the old snapshot file is replaced

Each time a new snapshot is created, the previous snapshot is overwritten

RDB file loaded

Redis does not have a specific command for loading RDB files, and the Redis server automatically loads RDB files whenever it detects their presence at startup. And the server will block while loading the RDB file until the load is complete.

configuration

In the redis.config file

// Default Settings save 900 1 Save 300 10 Save 60 10000Copy the code

save n m

The server changes the database at least m times in n seconds

  1. RDB snapshot generation can be performed synchronously or asynchronously by manually invoking the save or bgsave commands.
  2. Save can be set to multiple, i.e. multiple snapshotting checkpoints. Each checkpoint is checked to see if the specified number of keys has changed. If so, a new dump. RDB file is generated.

The principle of

saveparams

When the redis service starts, it will automatically read the redis. Config file information and read multiple configurations into a saveParam array. This array is the value of the saveParams property of the redisServer structure.

Dirty and lastsave

  1. The dirty counter records how many changes (writes, deletes, updates, etc.) the server has made to the database state (all databases on the server) since the last SAVE or BGSAVE command was successfully executed. When the server successfully executes a database modification command, the program updates the dirty counter: the value of the dirty counter increases as many times the command changes the database.
  2. The lastSave attribute is a UNIX timestamp that records the last time the server successfully executed the SAVE or BGSAVE command.

perform

Redis’s server periodic operation function serverCron is executed every 100 milliseconds by default. This function is used to maintain the running server, and one of its jobs is to check whether the save criteria set by the Save option have been met, and if so, to execute the BGSAVE command.

Process:

  1. Calculates the difference between the current time and the last execution time.
  2. Iterate over the saveParams attribute value data and perform an RDB file update when one of the conditions is met.
  3. Update dirty and lastsave values.

RDB pros and cons

advantages

  1. RDB will generate multiple data files, and each data file represents the data of REDis at a certain time. This method of multiple data files is very suitable for cold backup. Such complete data files can be sent to some remote secure storage, such as Amazon S3 cloud service. In China, it can be on the ODPS distributed storage of Ali Cloud to regularly back up data in Redis with a predetermined backup strategy.
  2. RDB has very little impact on the read and write services provided by Redis, so that Redis can maintain high performance, because the main redis process only needs to fork a sub-process and let the sub-process perform disk I/O operations for RDB persistence.
  3. Compared to AOF persistence, restarting and restoring redis processes directly based on RDB data files is much faster.

disadvantages

  1. RDB is not as good as AOF if you want to lose as little data as possible when Redis fails. In general, RDB data snapshot files are generated every 5 minutes or more, at which point you have to accept that if the Redis process goes down, the data in the last 5 minutes will be lost.
  2. Each time the RDB forks a child process to perform the RDB snapshot data file generation, if the data file is very large, the service provided to the client may be suspended for milliseconds, or even seconds.

AOF

In addition to RDB persistence, Redis also provides AOF (Append Only File) persistence. AOF persistence records database state by saving write commands executed by the Redis server.

AOF persistent implementation

Each write command is logged and appended in append-only mode.

process

  1. Redis appends the log file every time it receives a command
  2. Write the log file to the OS cache
  3. Write data from OS cache to AOF file using fsync

configuration

AOF configuration

appendonly yes
Copy the code

AOF persistence is turned off by default, RDB persistence is turned on by default

Appendonly yes, you can turn on AOF persistence. In a production environment, AOF is normally open, unless you say it doesn’t matter if you lose data for a few minutes

Both AOF and RDB are enabled. When Redis restarts, AOF is used for data recovery first, because AOF data is relatively complete

Fsync policy configuration for AOF

# appendfsync always
appendfsync everysec
# appendfsync no
Copy the code
  1. Always: Every time a piece of data is written, the corresponding log fsync is immediately sent to the disk. The performance is very poor and the throughput is low. Make sure that none of the data in Redis is lost and select this policy.
  2. Everysec: Fsync data from OS cache to disk every second. This is the most common configuration in production environments, with high performance and QPS of tens of thousands.
  3. No: Only Redis is responsible for writing data to the OS cache, and then the OS will have its own policy to flush data to disk from time to time.

AOF the rewrite

AOF files are persisted by appending commands. As Redis continues to be used, AOF files will get bigger and bigger. The solution to this problem is the rewrite of AOF files.

Rewrite implementation

process

  1. The AOF file bloats to the configuration size limit, triggering the BGREWEITEAOF command in the AOF file rewrite.
  2. The Redis process forks a child process that builds a new AOF file based on the current redis memory structure.
  3. The construction of the new AOF file has no relationship with the old AOF file, is based on the current REDis memory data structure, and the writing command process will be reconstructed, such as: multiple writes of the same key to form a command.
  4. During rewrite, Redis continues to accept new write commands and cache them in a recache.
  5. When the main redis process receives the rewrite signal, it writes data from the recache pool to a new AOF file and replaces the old one with the new AOF file. Delete old AOF files.

Rewrite configuration

auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
Copy the code
  1. Auto – aof – rewrite – percentage with 100: For example, after the last AOFrewrite, it was 128MB and then it would write 128MB more of AOF, and if it grew to more than 256MB, it might trigger rewrite.
  2. Auto-aof -rewrite-min-size 64MB: rerewrite: 256MB > 64MB: auto-aof-rewrite-min-size 64mb: auto-aof-rewrite-min-size 64mb: auto-aof-rewrite-min-size 64mb: auto-aof-rewrite-min-size 64mb

AOF file damaged repair

The redis-check-aof –fix command is used to fix broken AOF files, usually deleting problematic write commands.

The actual use

AOF and RDB

  1. Don’t just use RDB, because that will cause you to lose a lot of data.
  2. Also don’t just use AOF, because there are two problems with that. First, if you do cold standby through AOF, you can recover faster without RDB. Second, RDB is more robust by simply generating snapshots each time, avoiding the bugs of complex backup and recovery mechanisms such as AOF
  3. AOF and RDB persistence mechanisms are used in a comprehensive way. AOF is used to ensure that data is not lost as the first choice for data recovery. RDB is used for varying degrees of cold backup and for quick data recovery when AOF files are lost or corrupted and unavailable

Matters needing attention

  1. RDB starts by default, AOF needs to be manually enabled, both are enabled, redis restarts, data recovery priority is given to AOF files.
  2. If RDB does snapshotting, redis will not do AOF rewrite; If Redis were to do AOF rewrite, it would not do RDB snapshotting.
  3. If the RDB is running snapshotting and the BGREWRITEAOF command is executed, AOF rewrite will be executed only after the RDB snapshot is generated.

configuration

  1. In the enterprise, the RDB generation strategy is similar to the default. Save 60 10000: If you want to ensure that the RDB loses at most 1 minute of data, try to generate a snapshot every 1 minute, low peak period, data volume is small and unnecessary. 10000-> generate RDB, 1000->RDB, this depends on the actual amount of data in your application and business.

  2. AOF must be turned on, fsync policy everysec.

  3. Auto-aof -rewrite-percentage 100: indicates that the current AOF size expands to more than 100% of the previous aOF size.

    Auto-aof -rewrite-min-size 64MB: 16MB, 32MB depending on how much data you need.

Data Backup Scheme

RDB is great for cold storage, and after each generation, no more changes are made.

  1. Write the crontab scheduling script to back up data
  2. Copy an RDB backup every hour, into a directory, and keep only the last 48 hours of backup
  3. Keep a backup of the RDB every day, go to a directory and keep only the last month’s backup
  4. Every time I copy a backup, I delete the old backup
  5. Every night, back up all the data on your current server and send a copy to a remote cloud service

Script:

/usr/local/redis

Copy a backup file every hour and delete data generated 48 hours agocrontab -e
0 * * * * sh /usr/local/redis/copy/redis_rdb_copy_hourly.sh
 redis_rdb_copy_hourly.sh  \ #! /bin/shcur_date=`date +%Y%m%d%k` rm -rf /usr/local/redis/snapshotting/$cur_date mkdir /usr/local/redis/snapshotting/$cur_date cp /var/redis/6379/dump.rdb /usr/local/redis/snapshotting/$cur_date del_date=`date -d -48hour +%Y%m%d%k` rm -rf /usr/local/redis/snapshotting/$del_date   Copy a backup file every daycrontab -e 0 0 * * * sh /usr/local/redis/copy/redis_rdb_copy_daily.sh  redis_rdb_copy_daily.sh   \ #! /bin/sh cur_date=`date +%Y%m%d` rm -rf /usr/local/redis/snapshotting/$cur_date mkdir /usr/local/redis/snapshotting/$cur_date cp /var/redis/6379/dump.rdb /usr/local/redis/snapshotting/$cur_date del_date=`date -d -1month +%Y%m%d` rm -rf /usr/local/redis/snapshotting/$del_date  Upload all data to a remote cloud server once a dayCopy the code

Data Recovery Solution

  1. If the Redis process is down, restart the Redis process and recover data directly based on the AOF log file. At most, the data will be lost for one second.
  2. If the machine where the Redis process runs is down, restart the machine and try to restart the Redis process to recover data based on the AOF log file. AOF is not damaged and can be directly restored based on AOF. If the AOF file is damaged, use redis-check-aof fix to repair the damaged file.
  3. If the current AOF and RDB files of Redis are lost/corrupted, you can try to restore the data based on one of the current latest copies of RDB data on that machine
  4. The latest AOF and RDB files are lost/damaged to unrecoverable, usually not the machine fault, manually find the latest RDB backup, hour-level backup is ok, hour-level backup must be the latest, copy to redis, you can restore the data of a certain hour.

Reference:

  1. Redis design and implementation
  2. The flow of hundreds of millions of Chinese huperia

This article is formatted using MDNICE