The author introduces Gu Weitao, who used to work for Baidu and Qihoo 360, and is now a senior DBA of Mingshi Tang Education In Hangzhou. He is good at database monitoring, backup, high availability architecture design and automatic operation and maintenance. He has in-depth research on Redis and MongoDB, and focuses on back-end technologies such as distributed storage, big data storage, message queue and search engine.

preface

There is a problem in the process of loading data using Redis, that is, the Redis service must be restarted. If it is the master slave replication architecture of Redis, it is a very troublesome thing to load data in this way. According to the idea of loading data at the start of Redis, the author improved Redis and realized online loading data. Let’s discuss it with you here.

Design and Implementation

This paper takes Redis 3.0.7 as an example. According to the logic of loading data in the startup process of Redis, two commands are added for Redis, namely LOADAOF and LOADRDB, to LOADAOF and RDB files online respectively.

When executing LOADAOF aofile, call loadAppendOnlyFile to LOADAOF file data. The concrete implementation is as follows:

SRC /aof.c add the following functions:

void loadaofCommand(RedisClient *c) { if (server.rdb_child_pid ! = -1) { addReplyError(c,"Background save already in progress"); } else if (server.aof_child_pid ! = -1) { addReplyError(c,"Can't BGSAVE while AOF log rewriting is in progress"); } else if (c->argc ! = 2){ addReply(c,shared.syntaxerr); return; } else if (loadAppendOnlyFile(c->argv[1]->ptr) == REDIS_OK) { addReplyStatus(c,"online loadaof started,Do not repeat!!!" ); } else { addReply(c,shared.err); }}Copy the code

void loadaofCommand(RedisClient *c) { if (server.rdb_child_pid ! = -1) { addReplyError(c,”Background save already in progress”); } else if (server.aof_child_pid ! = -1) { addReplyError(c,”Can’t BGSAVE while AOF log rewriting is in progress”); } else if (c->argc ! = 2){ addReply(c,shared.syntaxerr); return; } else if (loadAppendOnlyFile(c->argv[1]->ptr) == REDIS_OK) { addReplyStatus(c,”online loadaof started,Do not repeat!!!” ); } else { addReply(c,shared.err); }}

When LOADRDB is executed, the function rdbLoad is called to load the RDB file.

SRC /rdb.c add the following functions:

void loadrdbCommand(RedisClient *c) { if (server.rdb_child_pid ! = -1) { addReplyError(c,"Background save already in progress"); } else if (server.aof_child_pid ! = -1) { addReplyError(c,"Can't BGSAVE while AOF log rewriting is in progress"); } else if (c->argc ! = 2){ addReply(c,shared.syntaxerr); return; } else if (rdbLoad(c->argv[1]->ptr) == REDIS_OK) { addReplyStatus(c,"online loadrdb started,Do not repeat !!!" ); } else { addReply(c,shared.err); }}Copy the code

void loadrdbCommand(RedisClient *c) { if (server.rdb_child_pid ! = -1) { addReplyError(c,”Background save already in progress”); } else if (server.aof_child_pid ! = -1) { addReplyError(c,”Can’t BGSAVE while AOF log rewriting is in progress”); } else if (c->argc ! = 2){ addReply(c,shared.syntaxerr); return; } else if (rdbLoad(c->argv[1]->ptr) == REDIS_OK) { addReplyStatus(c,”online loadrdb started,Do not repeat !!!” ); } else { addReply(c,shared.err); }}

Struct RedisCommand RedisCommandTable[]

Add the following line:

{" loadaof ", loadaofCommand, 2, "ar", 0, NULL, 0,0,0,0,0}, {" loadrdb ", loadrdbCommand, 2, "ar", 0, NULL, 0,0,0,0,0},Copy the code

{” loadaof “, loadaofCommand, 2, “ar”, 0, NULL, 0,0,0,0,0}, {” loadrdb “, loadrdbCommand, 2, “ar”, 0, NULL, 0,0,0,0,0},

SRC/redis. h in void bgsaveCommand(RedisClient *c); Add the following two lines below:

void loadrdbCommand(RedisClient *c);
void loadaofCommand(RedisClient *c);Copy the code

void loadrdbCommand(RedisClient *c); void loadaofCommand(RedisClient *c);

SRC /help.h Add the following to the struct commandHelp function:

"LOADAOF", "aof filename", "load aof file online", 9, "3.0.7"}, {"LOADRDB", "RDB filename", "load rdb file online,which can not repeat !!!" 9, "3.0.7}"Copy the code

“LOADAOF”, “aof filename”, “load aof file online”, 9, “3.0.7”}, {“LOADRDB”, “RDB filename”, “load rdb file online,which can not repeat !!!” 9, “3.0.7}”

Run the following command to compile Redis # CD redis-3.0.7 # make && make PREFIX=/usr/local/Redis30 install

The test results

You can see that without restarting Redis, you can load data.

Application scenarios

1, online data import to test Redis is generally developed by similar requirements for problem analysis and testing.

If there is a misoperation of Redis, online loading of data can be performed, especially in sentinel(S)+Redis master-slave replication architecture. This method is more gradual.

Pay attention to the point

1. If the aofile/rdbfile is copied from Redis, the aofile/rdbfile must also exist on the secondary library when the primary library loads the file online. Otherwise, the secondary library displays an error and exits.

2. When performing online loading operations, do not repeat the execution of the same file content; otherwise, the library will exit abnormally.

If the primary library repeatedly executes loadrdb, the primary library also exits unexpectedly.

There are still some deficiencies in this scheme, so I only do this here, hoping to discuss with you to improve it, so as to facilitate the fast operation of DBA.