The thing is like this, some time ago, the production and transaction of Xiaohei Ge company occasionally reported errors, and the final reason is that the execution of Redis command timed out.
However, the perplexing thing is that the production transaction only uses the simple command Redis set, which is logically impossible to execute so slowly.
So what exactly is causing this problem?
To find out, we looked at Redis’ recent slow logs and found that keys XX* took a lot of time
Seeing the prefix of the key operated by this command, the little black brother found that this was the application he was responsible for. Although the code does not actively use the keys command, but the underlying use of the framework is indirect use, so there is today’s problem.
Question why
The application xiao Hei Ge is responsible for is a management background application. Shiro framework is used for permission management. As there are multiple nodes, distributed Session needs to be used, so Redis is used to store Session information.
Voiceover: If you don’t know about distributed Session, you can read the four distributed consistent Session implementations written by Black brother in a breath
Since Shiro does not directly provide the Redis Session storage component, He has to use Github’s open source component, Shiro-Redis.
Since Shiro framework needs to periodically verify that sessions are valid, Shiro’s underlying layer will call SessionDAO#getActiveSessions to retrieve all Session information.
Shro-redis just inherits the SessionDAO interface. The bottom layer uses the keys command to find all Session keys stored in Redis.
public Set<byte[]> keys(byte[] pattern){
checkAndInit();
Set<byte[]> keys = null;
Jedis jedis = jedisPool.getResource();
try{
keys = jedis.keys(pattern);
}finally{
jedis.close();
}
return keys;
}
Copy the codeFind the cause of the problem, and the solution is relatively simple. Find the solution on Github and upgrade to the latest version of Shro-Redis.
In this release, Shro-redis fixes this problem by replacing keys with the scan command.
public Set<byte[]> keys(byte[] pattern) {
Set<byte[]> keys = null;
Jedis jedis = jedisPool.getResource();
try{
keys = new HashSet<byte[] > (); ScanParams params =new ScanParams();
params.count(count);
params.match(pattern);
byte[] cursor = ScanParams.SCAN_POINTER_START_BINARY;
ScanResult<byte[]> scanResult;
do{
scanResult = jedis.scan(cursor,params);
keys.addAll(scanResult.getResult());
cursor = scanResult.getCursorAsBytes();
}while(scanResult.getStringCursor().compareTo(ScanParams.SCAN_POINTER_START) > 0);
}finally{
jedis.close();
}
return keys;
}
Copy the codeAlthough the problem was successfully solved, little Black brother was still a little confused.
Why does the keys command slow other commands?
Why are Keys queries so slow?
Why is there no problem with the Scan instruction?
How Redis executes commands
Let’s start with the first question: why does the keys command slow down other commands?
To answer this question, let’s first look at the Redis client executing a command:
From the client’s perspective, executing a command consists of three steps:
- Send the command
- Execute the command
- Returns the result
However, this is only the process that the client thinks, but in fact, there may be many clients sending commands to Redis at the same time, and Redis, as we all know, uses a single-threaded model.
In order to process all the client request commands at the same time, Redis adopts the mode of queue execution.
So there are actually four steps for the client to execute a command:
- Send the command
- The command line
- Execute the command
- Returns the result
Because Redis executes commands in a single thread, you can only start executing tasks sequentially from the queue.
As long as 3, the command execution speed of this process is too slow, and other tasks in the queue have to wait. In the eyes of external clients, Redis seems to be blocked, and there is no response.
Therefore, do not use the Redis procedure to execute instructions that need to run for a long time. This may cause Redis to block and affect the execution of other instructions.
Principle of KEYS
Why is the Keys command query so slow?
Before answering this question, think back to the Redis underlying storage structure.
Ali: Are you familiar with HashMap? Well, let’s talk about Redis dictionary. .
Redis uses a dictionary structure that is similar to the Java HashMap structure.
The keys command needs to return all the keys in Redis that match the given pattern. To do this, Redis has to iterate over the underlying array of the HT [0] hash table in the dictionary in O(N) time (N is the number of keys in Redis).
If the number of keys in Redis is small, the execution will still be fast. As the number of Redis keys gradually increases to millions, tens of millions, or even hundreds of millions, the execution speed will be very slow.
Add 10W keys to Redis using lua script, and then query all keys using keys. This query will block for about ten seconds.
Eval "for I =1,100000 do redis. Call ('set', I, I +1) end" 0Copy the codeHere the little black brother uses Docker to deploy Redis, performance may be slightly worse.
SCAN the principle
Finally, let’s look at the third question. Why is the SCAN instruction ok?
This is because the Scan command uses a dark technology – a vernier-based iterator.
Each time the scan command is called, Redis returns a new cursor and a number of keys to the user. The next time you want to retrieve the remaining keys, you need to pass the cursor to the scan command to continue the previous iteration.
Simply put, the scan command uses paging queries for Redis.
Here is an example of an iterative process for the scan command:
The scan command uses the cursor to divide a full query into multiple times, reducing the query complexity.
The time complexity of the scan command is O(N), which is the same as that of keys. However, the scan command only needs to return a small number of keys, so the execution speed is fast.
Finally, while the scan command addresses the keys deficiency, it also introduces some other defects:
- The same element may be returned multiple times, which requires our application to add the ability to handle duplicate elements.
- If an element is added to redis during the iteration, or is removed during the iteration, that element will be returned or not.
The above defects, we need to consider this situation in our development.
In addition to scan, Redis has several other commands for incremental iteration:
sscan: Used to iterate over the database keys in the current database for resolutionsmembersBlocking problems may occurhscanThe hash key command is used to iterate over key-value pairs in a hash key, used to resolvehgetallBlocking problems may occur.zscanThe: command is used to iterate over elements (including element members and element score values) in an ordered collection, used to generatezrangeBlocking problems may occur.
conclusion
Redis uses a single thread to execute commands. All commands sent by clients are now queued and then retrieved from the queue to execute the corresponding commands.
If any task executes too slowly, it will affect the other tasks in the queue, and the delay in getting a response from Redis will appear blocked to external clients.
Therefore, do not execute keys, smembers, hgetall, zrange, and other commands that may cause blocking in production. If you do need to execute them, you can use the corresponding scan command to gradually traverse, which can effectively prevent blocking problems.
Welcome to pay attention to my public account: procedures to get daily dry goods push. If you are interested in my topics, you can also follow my blog: studyidea.cn
