This article is participating in “Java Theme Month – Java Development in Action”, see the activity link for details

This is the first day of my participation in Gwen Challenge

Hi, I’m Koba.

Redis has high performance because all reads and writes are done in memory. When our server is disconnected or restarted, the data will disappear, so how do we solve this problem?

In fact, Redis has provided us with a persistence mechanism, respectively RDB and AOF two ways, next let me see how these two tips ensure data persistence.

persistence

Since Redis is a memory-based database, our data is not secure in the event of a server failure.

At this time, we need to store the data in memory to the disk. When the server restarts, we can recover the data through the disk. This process is called Redis persistence.

Redis persistence

RDB

Introduction to the

The full name of the RDB is Redis Database Backup file, or Redis data snapshot.

  • The RDB file is a compressed binary file (default: dump.rdb);

  • RDB files are saved on hard disk;

  • Record database state by saving key-value pairs in the database.

create

When Redis persists, the program saves the current in-memory database state to disk.

create

There are two main Redis commands to create an RDB file: SAVE and BGSAVE.

SAVE

During command execution, the Redis server process will be blocked and command requests sent by the client will be rejected.

Code examples:

Def SAVE(): # create rdbSave()Copy the code

Here is:

The Save command

BGSAVE

Asynchronous operation, when executing a command, the child process performs the save work, and the server can continue to let the main thread process the command requests sent by the client.

Code examples:

Signal_parent () elif pid > 0: def BGSAVE(): # parent() elif pid > 0: Handle_request_and_wait_signal () else: HANDle_fork_error ()Copy the code

Here is:

BgSave command

load

The load is performed automatically when the server starts.

load

The server blocks while loading the RDB file until the load is complete.

The main Settings

Redis allows users to have the server automatically execute BGSAVE commands at regular intervals by setting the save option for the server configuration.

Set save conditions

The configuration is as follows:

save 900 1
save 300 10

Copy the code

In this case, the BGSAVE command is executed as long as one of the following conditions is met:

  • The server made at least one change to the database within 900 seconds;

  • The server made at least 10 changes to the database within 300 seconds.

saveparams

The server program sets the saveParams property of the server state redisServer structure based on the save conditions set by the Save option.

  • The saveParams property is an array;

  • Each element in the array is a saveParam structure;

  • Each saveParam structure holds a save condition set by the Save option.

Struct saveparam {time_t seconds; Int changes; }Copy the code
dirty

The dirty counter records how many times the server has changed the database state (including writes, deletes, updates, etc.) since the last SAVE or BGSAVE command was successfully executed.

lastsave

Is an UNINX timestamp that records the last time the server successfully executed the SAVE or BGSAVE command.

Check whether the storage conditions are met

The server periodic operation function serverCron (which maintains the running server) executes by default every 100 milliseconds. One of its jobs is to check whether the save criteria set by the save option have been met, and then execute the BGSAVE command.

Code examples:

def serverCron(): # .... Saveparam in server. Saveparams: Save_interval = unixtime_now() -server.lastsave # if the state of the database has been changed more times than the condition set # if the time since the lastsave exceeds the condition set if server.dirty >= saveparam.changes and save_interval > saveparam.seconds: BGSAVE()Copy the code

The default configuration

The default configuration of RDB files is as follows:

################################ SNAPSHOTTING ################################ # # Save the DB on disk: Automatic persistence for a given number of seconds and a given number of write operands to the database. # save <seconds> <changes> # save 900 1 save 300 10 save 60 10000 Yes stop-writes-on-bgsave-error yes # Use LZF to compress string objects for persistence. Rdbcompression yes # Whether to checksum RDB files, usually yes rdbchecksum yes # RDB persistent filename dbfilename dump. RDB # Persistent file storage directory dir./Copy the code

AOF

Introduction to the

AOF stands for Append Only File. A log is a write-after log. A Redis command is executed to write data to the memory before logging.

After writing the log

  • By saving the write command executed by Redis server to record the database state;

  • All commands written to AOF files are saved in Redis command request protocol format.

implementation

AOF persistence process is mainly implemented through the following processes:

AOF process

Command to add

If AOF persistence is enabled, after executing a command, the server appends the executed write command to the end of the aOF_buf buffer of the server state in a protocol format.

File synchronization

The server calls flushAppendOnlyFile every time it terminates an event loop, which considers whether the contents of the aOF_buf buffer need to be written to and saved to an AOF file.

The flushAppendOnlyFile function performs the following process:

  • WRITE: Writes the cache in aOF_buf to the AOF file according to the condition;

  • SAVE: Depending on the condition, call the fsync or fdatasync function to SAVE the AOF file to disk.

This function is influenced by the three appendfsync values configured by the server: always, Everysec, and no, also known as the three policies.

Always

Each command is fsync to the hard disk so redis doesn’t lose its write data.

Always

everysec

The buffer is flushed to hard disk every second (default).

everysec

no

It is not up to us to decide when to flush to hard disk based on the rules of the current operating system.

no

The data load

  1. Create a pseudo client without network connection.

  2. Analyze and read a write command from AOF file;

  3. Use pseudo client to execute read write command;

  4. Continue steps 2 and 3 until all write commands in the AOF file have been processed.

File to rewrite

Why file rewriting is required:

  • To solve the problem of AOF file volume expansion;

  • Replace the existing AOF file by rewriting it to create a new AOF file that does not contain any redundant commands that waste space.

implementation

The implementation principle of file rewriting:

  • There is no need to do anything with existing AOF files;

  • Read the current value of the key directly from the database;

  • Record the key-value pair with a single command instead of multiple commands that record the key-value pair.

The background to rewrite

To avoid blocking the parent process, Redis executes the AOF rewrite in the child process.

During AOF rewrite by the child process, the server process needs to perform three processes:

  1. Execute the command sent by the client.

  2. Appends the executed write command to the AOF buffer;

  3. Appends the executed write command to the AOF rewrite buffer.

Server flow

The default configuration

The default configuration of AOF files is as follows:

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # APPEND ONLY MODE # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # open appendonly AOF persistent way no AOF persistence file name Appendfilename "appendonly. Aof "# fsync buffer data to disk every second appendfsync everysec # appendfsync no # Whether data is not synchronized to aOF file during rewrite No-appendfsync-on-rewrite no # Indicates the rate at which AOF file rewrites are triggered. Auto-off-rewrite-percentage 100 # Indicates the minimum size at which AOF file rewrites are triggered Aof-load-truncated yes # Whether to turn on the mixing switch aof-use-rdb-preamble yesCopy the code

conclusion

Through the above introduction, we must have a general understanding of Redis persistence, so these two ways, how should we choose?

  • For large and medium-sized applications, we should use RDB and AOF in combination to ensure data integrity and high efficiency.

  • AOF is preferred to ensure data integrity and prevent data loss.

  • If you are dealing with large-scale data recovery and pursuing higher and faster efficiency, RDB is preferred.

You can also refer to the following figure for selection:

The main contrast