Redis learning – Redis persistence

Whether it’s an interview or a job, persistence is the key.

Generally, when redis occupies more than 20GB of memory, data synchronization and backup must be performed for multiple redis instances to ensure availability.

All files saved by RBD are dump. RDB, which are generated from snapshot configurations in the configuration file. In general business situations, only RDB is required.

Aof is not enabled by default, because it is easy to create large files. Although overwriting is officially available, it can still block files that are too large, so use it with caution

RBD and AOF have different effects on system performance in large data volumes. The specific solution strategy depends on the system resources and actual service conditions.

Data design affects persistence:

Szthanatos. Making. IO/topic/redis…

Why persist?

  1. Reuse data
  2. Back up important data in case of system failure

How to persist

  1. RDB snapshot: All data at a specific time is written to disks
  2. AOF (Append-only file) : all commands are written to this judgment.

Default value: RDB. AOF needs to be manually enabled

Redis. conf indicates the persistence configuration

In the redis.conf file, the following options exist:

RDB configuration in the redis.conf file

RDB compression is enabled by default
rdbcompression yes
# indicates one write within 900 seconds, and the RDB is logged
save 900 1
# RDB backup file name
dbfilename dump.rdb
# indicates the location where backup files are stored
dir . /
Copy the code

Configure AOF in redis.conf

Whether to enable aof, which is disabled by default
appendonly no
#aof file name
appendfilename "appendonly.aof"
# no: don't fsync, just let the OS flush the data when it wants. Faster.
# always: fsync after every write to the append only log. Slow, Safest.
# everysec: fsync only one time every second. Compromise.
appendfsync everysec
If you do not use fsync, write to the disk instead of the buffer, it will block I/O, but it is safest. If you use yes to write to the buffer, it will cause data persistence problems. Up to 30 seconds of data will be lost)
no-appendfsync-on-rewrite no
If the redis content is larger than 64 MB and the capacity is expanded by more than 100%, bgrewrite will be executed
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

Copy the code

RDB

There are several ways to create an RDB snapshot:

  1. The client sends a bgsave command to Redis (note that Windows does not support BGsave). At this point, Reids calls fork to create the child process. The parent process continues processing, the child process writes the snapshot to disk, and the parent process continues processing the request.
  2. The client sends the save command to create a snapshot. Note that this approach blocks the entire parent process. Rarely used, only in special cases.
  3. When redis uses shutdown command to shutdown the server request, at this time redis will stop all work and execute a save, block all clients and no more commands are executed and disk write is performed. After the write is complete, the server is shutdown.
  4. In the Redis cluster, the sync command is sent to perform a replication operation. If the primary server has not executed bgsave or has just executed BGSave, the BGSave will be performed.
  5. Run the flushall command

Some points to note about RDB snapshots:

  1. When RDB is used only, if Redis crashes during snapshot creation, redis will retain the snapshot of the last backup. However, the specific amount of data lost depends on the backup time
  2. Only apply to systems that can tolerate some data loss, otherwise consider Aof persistence
  3. In the case of a large amount of data, especially if the memory is more than 20GB, a synchronization takes 4-6 seconds
    1. One way is to use manual synchronization, in the early hours of the morning suitable for manual blocking synchronization, faster than BGSAVE

One solution:

Log records are used to recover the interrupted logs, and data recovery is carried out

How can I modify the configuration to get the desired persistence?

  1. Modify the save parameter, and try to set save on the development environment simulation line environment. Too frequent will cause resource waste, too rare may lose a large amount of data
  2. Logs are aggregated and calculated based on save to determine the maximum amount of data that will be lost. For example, one hour can be setsave 3600 1

Advantages and disadvantages of RDB:

Advantages:

  1. Suitable for large-scale data recovery
  2. If data is deleted accidentally, it can be restored in time
  3. Recovery speed is generally faster than Aof

Disadvantages:

  1. If Redis goes down unexpectedly, the last modified data is lost. How much data is lost depends on the persistence policy
  2. The fork process occupies some memory space. If the fork memory is too large, the recovery time may be seconds
  3. The data file is compressed by Redis, and the readability is poor

AOF (Append only Fail)

It’s basically a history of our commands, like the History of Linux

This function is disabled by default. You need to manually enable this function. After this function is enabled, restart the system

If the aof file is misplaced, you can use redis-check-aof to fix the file

File synchronization: When a file is written, three things happen:

  1. The file.write() method stores the file into a buffer
  2. File.flush () writes the contents of the buffer to hard disk
  3. Sync Synchronizes files and blocks until data is written to disk

AOC synchronization strategy

options Synchronous frequency
always Each command is written to disk, seriously slowing down Redis
everysec This occurs once per second, showing multiple commands being written to disk
no Operating system decision, Buddha

Analysis:

  1. The first kind of solid state hard disk damage is relatively large, we all know that the life of the solid state erase times is far less than the mechanical hard disk, frequent IO is easy to cause deception to the solid state that a wipe, resulting in a short life of the solid state becomes shorter, basic use, in special cases may be used
  2. The second method is the default method, which is recommended and practical. It can only lose data for one second at most. This method is recommended to ensure the availability of data backup
  3. The third method has the least pressure on the CPU, because it is determined by the system, but it needs to consider whether it can accept unlimited data loss. Another reason is that the buffer in THE I/O is refreshed to the disk at irregular times, so it is not recommended

Rewrite and compress AOF files:

Redis has introduced the bgrewriteaof command to compress files. This is similar to the bgsave command to create snapshots. At the same time, there will be large files in the rewrite of the file system will bring huge pressure to delete the file system, resulting in system congestion.

The command is

bgrewriteaof

The following is an example:

127.0.0.1:16379> BGREWRITEAOF Background append only file outline started

Parameter control:

Auto – aof – rewrite – percentage: 100

Auto – aof – rewrite – min – size: 64 MB

The case configuration here means that the bgrewrite command is triggered when the AOF is greater than 64 and 100% larger

What did Rewrite by Redis Aof do?

  1. Delete some redundant commands
  2. The system detects incorrect commands and clears all the commands under the incorrect commands. Generally, the commands that have not been executed due to the downtime are cleared.

The pros and cons of AOF

Advantages:

  1. Never synchronize, high efficiency
  2. The data is synchronized once per second, and one second of data may be lost
  3. Each change is synchronized and file integrity is good

Disadvantages:

  1. Relative to data files, AOF is much larger than RDB. Repair is slower
  2. There are unknown bugs. For example, there are many strange phenomena if the rewrite of aof file is interrupted suddenly

How to check performance bottlenecks in Redis:

  1. Redis – benchmark’s recommended performance testing tool, is very powerful, specific address is: www.runoob.com/redis/redis…
  2. Redis – call in clislowlog getIs used to return the execution timeMore than redis. ConfNote that this time is only the processing time of the request and does not include the network traffic time.The default value is one second.

Redis.conf: slow log

The following time is expressed in microseconds, so 1000000 is equivalent to one second. Note that a negative number disables the slow log, while a value of zero forces the logging of every command.

The rest of the time is measured in microseconds, so one million is equal to one second. Note that a negative number disables slow logging, while a zero value forces each command to be logged. (in microseconds)

slowlog-log-slower-than 10000

There is no limit to this length. Just be aware that it will consume memory. You can reclaim memory used by the slow log with SLOWLOG RESET.

There is no limit to the length. Note that it consumes memory. You can use SLOWLOG RESET to reclaim memory used by slow logs. (this means that commands after 128 are automatically removed)

slowlog-max-len 128

You can run the SLOWLOG command to RESET the memory occupied by slow logs

127.0.0.1:16379 > SLOWLOG reset OK

== Slow logs are stored in memory, remember ==

Persistence performance recommendations

  • Since RDB files are only used for backup purposes, it is recommended to persist RDB files only on Slave, and only 15 minutes is enough to back up RDB files. Only the rule save 900 1 is retained.
  • If Enalbe AOF, the advantage is that in the worst case you will only lose data for less than two seconds, and it is easy to start the script and just load your own AOF file. The cost is constant IO and almost inevitable blocking as AOF rewrite writes new data to new files. The default base size of 64MB for AOF rewrite is too small and can be set to 5G or more, as long as hard disk permissions are available. By default, overrides can be changed to an appropriate value for more than 100% of the original size.
  • If AOF is not enabled, high availability can be achieved only by master-slave Replication. Eliminating a large chunk of IO also reduces the volatility of the system when rewriting. The cost is that if the Master and Slave fail at the same time, more than ten minutes of data will be lost. The startup script also has to compare the RDB files in the two Master and Slave, and load the newer one. Sina Weibo uses this structure.

Other performance tuning guidelines (highly recommended) :

Szthanatos. Making. IO/topic/redis…

To summarize and compare RDB and Aof:

RDB AOF
Store content data Write operation Log
Performance impact small big
Recovery rate high low
The storage space small big
readability low high
Safety degree Low, low preservation frequency High storage frequency
The default open is no
Storage policy save 900 1: If you modify a file within 900 seconds, the file is savedsave 300 10: Ten changes within 300 seconds will be savedsave 60 10000: 10,000 changes within 60 seconds are saved, allowing customization always: Saves or item by itemeverysec: Saves or every secondno: The system decides when to save

Other expanded knowledge:

About Linux kernel enablementtransparent_hugepageBlocking problems:

Individual for Linux learning art not fine, directly quote the article, infringement please contact delete

Transparent Hugepages

A quick word about THP – remember a database server blocking problem resolved

Official solutions to aof and RDB compromise on performance issues

  1. After redis4.0 there is a parameter called:aof-use-rdb-preamble yes

The parameters are described as follows:

# When rewriting the AOF file, Redis is able to use an RDB preamble in the # AOF file for faster rewrites and recoveries. When this option is turned # on the rewritten AOF file is composed of two different stanzas: # # [RDB file][AOF tail] # # When loading, Redis recognizes that the AOF file starts with the "REDIS" # string and loads the prefixed RDB file, Redis continues loading the AOF # tail. Redis continues loading the AOF # tail. Redis continues loading the AOF # tail. The # on the AOF file overwritten with this option enabled consists of two different sections: # # [RDB file] [AOF tail] # # When loaded, Redis recognizes that the AOF file begins with the # string "Redis" and loads the prefixed RDB file, then proceeds to load the AOF # tail.Copy the code

Redis will convert the earlier part of the data into an RDB file for recovery, and add the more recent data into an AOF file

Restore the RDB file before loading the aof command

How do I clean up memory

After RedDIS4.0, you can enable automatic cleanup by setting ActiveDefRag to yes in the configuration, or by using the Memory Purge command.