Redis persistence trade-offs and choices

The role of persistence

What is persistence

All redis data is stored in memory, and updates to data are asynchronously stored on disk. If data is not persisted to disk, redis data will be lost when the redis server restarts

RDB

What is a RDB

  • RDB persistence is a process in which snapshots of the current process data are generated and saved to disks. RDB persistence can be triggered manually or automatically.

RDB trigger mechanisms – there are three main ones

  • Save (sync)

    The save command: blocks the current Redis server until the RDB process is complete. It will cause a long time of blocking for instances with large memory. Therefore, this command is not recommended in the previous environment

127.0. 01.:6379> save
OK
Copy the code

File policy: If old RDB files exist, the time complexity of new and old replacement: O(n)

  • Bgsave (asynchronous)

    Bgsave: The Redis process forks to create a child process. The RDB persistence process is the responsibility of the child process and ends automatically. Blocking only occurs during the fork phase for a short period of time

File policy: If old RDB files exist, the time complexity of new and old replacement: O(n)

  • Automatic trigger (Redis default RDB persistence mode)

Save: The RDB file is saved in the directory specified by the dir configuration. The filename is specified by the dbfilename configuration. The RDB file can be dynamically executed by running config set dir {newDir} and config set dbfilename {newFileName} during the next run. The RDB file will be saved to a new directory.

Compression: Redis default by LZF algorithm to generate RDB file compression processing, compressed file is far less than the memory size, the default open, can pass parameter config set rdbcompression {yes | no} dynamic modification.

Verify: If Redis refuses to start when loading a corrupted RDB file, and prints the following log:

Short read or OOM loading DB. Unrecoverable error , aborting now.

In this case, you can use the Redis-check-dump tool provided by Redis to detect RDB files and obtain corresponding error reportsCopy the code
127.0. 01.:6379> bgsave
Background saving started
Copy the code

RDB summary

  1. The RDB is a Redis memory-to-disk snapshot for persistence.
  2. Save usually blocks Redis
  3. Bgsave does not block Redis, but forks new frequently
  4. The save autoconfiguration is executed if any criteria are met
advantages
  1. Redis loads RDB recovery data much faster than AOF.
  2. The RDB is a compact binary file that represents a snapshot of Redis data at a point in time. Ideal for backup, full copy, and other scenarios. For example, perform bgSave backups every 6 hours and copy RDB files to a remote machine or file system (such as HDFS) for disaster recovery.

disadvantages

  1. Time consuming and performance consuming, RDB data cannot achieve real-time persistence/second level persistence. Because bgSave forks every time it runs, it is a heavyweight operation that is expensive to execute frequently.
  2. RDB files are saved in a specific binary format. In the process of Redis version evolution, multiple FORMATS of RDB are clumsy, and the old Redis service cannot be compatible with the new RDB format.

How to restore

Automatic recovery: Automatically read dump. RDB file recovery. Move the backup file to the redis installation directory and start the service. To back up the dump. RDB file, run the cp command cp dump. RDB dumo_new. RDB

AOF

What is AOF

AOF(Append only File) persistence: Each write command is recorded in an independent log, and the commands in the AOF file are executed again when the system restarts to recover data. The main role of AOF is to solve the real-time of data persistence, which has been the mainstream way of Redis persistence.

AOF has three strategies

  1. always

  1. everysec

  1. no

The operating system decides

3 the contrast

Rewriting mechanism

As commands write to AOF, files get bigger and bigger. To solve this problem, Redis introduced AOF rewriting to reduce file size. AOF file rewrite is a process in which data in the Redis process is converted into write commands and synchronized to a new AOF file.

role
  • Reduce disk usage
  • Speed up recovery
Two ways to rewrite
  • bgrewriteaof
127.0. 01.:6379> bgrewriteaof
Background append only file rewriting started

Copy the code

  • Aof overrides configuration

Auto-aof -rewrite-min-size and auto-aof-rewrite-percentage parameters determine the automatic trigger time

Auto-aof -rewrite-min-size: indicates the minimum size of a file when aof rewriting is run. The default size is 64MB

Auto-aof-rewrite-percentage: specifies the value of the current AOF file space (aOF_current_size) and aOF_base_size after the last rewrite

Automatic trigger time =aof_current_size>auto-aof-rewrite-min-size && (aof_current_size-aof_base_size)/aof_base_size >= auto-aof-rewrite-percentage

Where aOF_CURRENT_size and aof_base_size can be viewed in the info Persistence statistics.Copy the code
AOF rewrite process

configuration

/ / open aof
appendonly yes

/ / aof log
appendfilename  "appendonly-${port}.aof"

// aOF persistence policy
appendsync eneeyec

/ / directory
dir /bigdiskpath

// Rewrite the content configuration
/ / growth
auto-aof-rewrite-percentage 100

/ / size
auto-aof-rewrite-min-size  64mb

Copy the code

Restore data

1.Set appendOnly Yes;2.Appendone.aof to the directory specified by the dir argument;3.When you start Redis, the file appendone.aof is automatically loaded.Copy the code

The advantages and disadvantages

The advantages of AOF
  1. Using AOF persistence makes Redis much more durable: You can set different fsync policies, such as no fsync, fsync every second, or fsync every time a write command is executed. The default AOF policy is fsync once per second. With this configuration, Redis still performs well and loses at most a second of data in the event of an outage (fsync is performed in background threads, so the main thread can continue to struggle to process command requests).
  2. Redis can automatically rewrite AOF in the background when AOF files become too large
disadvantages
  1. AOF files are usually larger than RDB files for the same data set

RDB and AOF selection

RDB and AOF

RDB or AOF, which one should I use?

In general, if you want to achieve data security comparable to PostgreSQL, you should use both persistence functions, RDB for backup and AOF for recovery

Personal blog: blog.yanxiaolu.cn /