preface

Recently, I have been learning related knowledge of Redis. I have read The Redis development specification of Ali and the book Of Redis Development, Operation and maintenance. Divided into the use of specifications, pits of command, project actual combat operation, operation and maintenance configuration four directions, sorted out the use of Redis 21 attention points, I hope to help you, learn together ha!

1. Usage specifications of Redis

1.1 key points of key specification

When we design the key of Redis, we should pay attention to the following points:

❝

  • The service name is prefixed with key and separated by colons (:) to prevent key conflict overwriting. For example, live: rank: 1
  • The length of a key must be less than 30 characters to ensure that its semantics are clear.
  • Key cannot contain special characters, such as Spaces, newlines, single and double quotation marks, and other escape characters.
  • Try to set TTL for Redis keys to ensure that unused keys can be cleared or eliminated in time.

❞

1.2 key points of value specification

The value of Redis cannot be set arbitrarily.

“Number one” is that storing a large number of bigkeys can be problematic, resulting in slow queries, memory growth, and so on.

  • If the value is a String, the size of a single value cannot exceed 10K.
  • If the hash, list, set, or zset type is used, the number of elements cannot exceed 5000.

Second, choose the right data type. Redis is a String that uses only the set and get types. In fact, Redis provides ** “rich data structure types” **, and some business scenarios are better suited to hash, zset and other data results.

“Counterexample:”

set user:Awesome!:name jay
set user:Awesome!:age 18
Copy the code

“Positive”

hmset user:Awesome! name jay age 18 
Copy the code

1.3. Set the expiration time for keys. Pay attention to the keys of different services and try to disperse the expiration time

  • Because Redis data is stored in memory, and memory resources are precious.
  • We generally use Redis as a cache, not a database, so the life cycle of the key should not be too long.
  • Therefore, it is generally recommended to use ** “expire” for your key **.

If a large number of keys expire at a certain point in time, Redis may have a lag or even a “cache avalanche” at that point in time. Therefore, the expiration time of keys for different services should be scattered. Sometimes, in the same business, you can also add a random value to the time to spread out the expiration time.

1.4. Batch operation is recommended to improve efficiency

When we write SQL everyday, we know that batch operation is more efficient, 50 updates at a time, rather than 50 times, one update at a time. In fact, Redis operation command is the same reason.

The Redis client can execute a command in four processes: 1. Send a command -> 2. Command queuing -> 3. Command execution -> 4. Result is displayed. 1 and 4 are called RRT (command execution round trip time). Redis provides ** batch operation commands, such as mget and mset, to save RRT. However, most commands do not support batch operations, such as Hgetall, and there is no MHGEtall. “Pipeline” ** solves this problem.

❝

What is a Pipeline? It can assemble a group of Redis commands, transmit them to Redis through a RTT, and then return the execution results of this group of Redis commands to the client in sequence.

❞

Let’s take a look at the model with n commands executed without Pipeline:

N times of command execution using Pipeline, the whole process requires 1 RTT, the model is as follows:

2, Redis has pit those commands

2.1. CarefulO(n)Complexity command, for examplehgetall,smember.lrangeEtc.

Because Redis executes commands in a single thread. The time complexity of hgetall, smember and other commands is O(n). When n continues to increase, the Redis CPU continues to soar, blocking the execution of other commands.

❝

Hgetall, smember, lrange and other commands are not necessarily unacceptable. It is necessary to comprehensively evaluate the amount of data and determine the value of n. For example, hgetall, if the hash element n is large, can be preferred to use ** “HScan” **.

❞

2.2 Use the monitor command of Redis with caution

The Redis Monitor command is used to print the commands received by the Redis server in real time. If you want to know what the client is doing to the Redis server, you can use the Monitor command to view the commands. Because “the monitor command can cause redis memory to continuously surge.” **

❝

The monitor model outputs all commands executed on the Redis server. Generally, the QPS of the Redis server is very high, that is, if the monitor command is executed, the Redis server will have a large amount of “inventory” in the output buffer of the monitor client. This takes up a lot of Redis memory.

❞

2.3 Keys instruction cannot be used in production environment

The Redis Keys command is used to search for all Keys that match a given pattern. If you want to check the number of keys of a certain type in Redis, you can use keys as follows:

Keys Key prefix *Copy the code

However, redis keys are traversal matched, complexity is O (n), more data in the database is slower. As we know, Redis is single-threaded. If there is too much data, the keys command will cause the Redis thread to block, and the online service will stop until the command is executed. Therefore, “In general, do not use keys directives in production environments.” The official document also states:

❝

Warning: consider KEYS as a command that should only be used in production environments with extreme care. It may ruin performance when it is executed against large databases. This command is intended for debugging and special operations, such as changing your keyspace layout. Don’t use KEYS in your regular application code. If you’re looking for a way to find keys in a subset of your keyspace, consider using sets.

❞

Instead, you can use the scan directive, which, like the keys command, provides pattern matching. It is also O(n) in complexity, but it is done in steps by cursor and “does not block the Redis thread”; However, there will be a certain “repetition probability”, need to “do a deduplication on the client” **.

❝

Scan supports the incremental iteration command, which also has disadvantages: For example, you can use the SMEMBERS command to return all the elements currently contained in a collection key, but with an incremental iteration command like SCAN, the incremental iteration command provides only limited guarantees about the elements that are returned because the key may be changed during incremental iteration of the key.

❞

2.4 Prohibit the use of flushall and flushdb

❝

  • The Flushall command is used to flush the data of the entire Redis server (delete all keys of all databases).
  • The Flushdb command is used to flush all keys in the current database.

❞

These two commands are atomic and do not terminate execution. Once executed, it will not fail.

2.5 Running the del command

What command do you usually use to delete keys? Is it direct del? If you delete a key, you can use the del command. But have you thought about the time complexity of DEL? Let’s discuss it by case:

  • If a String key is deleted, the time complexity isThe O (1).“You can just del”.
  • If a List/Hash/Set/ZSet type is deleted, its complexity isO(n), n represents the number of elements.

Therefore, if you delete a List/Hash/Set/ZSet key, the more elements, the slower it will be. “Be careful when n is large”, it blocks the main thread. So, how do we delete without del?

❝

  • If it’s a List, you can do itLpop or rpopUntil all elements are deleted.
  • If it’s Hash/Set/ZSet, you can do it firsthscan/sscan/scanQuery, and then executehdel/srem/zremDelete each element in turn.

❞

2.6 Avoid using complex commands such as SORT and SINTER.

Executing complex commands consumes more CPU resources and blocks the main thread. Avoid aggregations such as SORT, SINTER, SINTERSTORE, ZUNIONSTORE, ZINTERSTORE, etc. It is generally recommended to put them on the client.

3. Project actual combat pit avoidance operation

3.1 Points to pay attention to using distributed lock

Distributed lock is the realization of a lock that controls the access to shared resources by different processes in a distributed system. Second kill order, grab red packets and other business scenarios, all need to use distributed lock. We often use Redis as a distributed lock, with these main points to note:

3.1.1 Two commands SETNX + EXPIRE are written separately (typical error implementation example)

if(jedis. Setnx (key_resource_id lock_value) = =1) {/ / lockThe expire (key_resource_id,100);// Set the expiration time
    try {
        do something  // Business request
    }catch(){
  }
  finally {
       jedis.del(key_resource_id); / / releases the lock}}Copy the code

If the process crashes or restarts after executing a setnx lock, the lock will never be available to another thread. This is not the case with distributed locks.

3.1.2 SETNX + value specifies the expiration time.

long expires = System.currentTimeMillis() + expireTime; // System time + set expiration time
String expiresStr = String.valueOf(expires);

// If the current lock does not exist, success is returned
if (jedis.setnx(key_resource_id, expiresStr) == 1) {
        return true;
} 
// If the lock already exists, get the lock expiration time
String currentValueStr = jedis.get(key_resource_id);

// If the obtained expiration time is less than the current system time, it is expired
if(currentValueStr ! = null && Long.parseLong(currentValueStr) < System.currentTimeMillis()) {// The lock has expired. Get the expiration date of the previous lock and set the expiration date of the current lock.
    String oldValueStr = jedis.getSet(key_resource_id, expiresStr);
    
    if(oldValueStr ! = null && oldValueStr.equals(currentValueStr)) {// In the case of multi-threaded concurrency, a thread can be locked only if its value is the same as the current value
         return true; }}// In other cases, lock failure is returned
return false;
}
Copy the code

* Disadvantages of this approach ** :

❝

  • The expiration time is generated by the client. In a distributed environment, the expiration time on each client must be synchronized
  • There is no unique identifier of the save holder and it may be released/unlocked by other clients.
  • When the lock expires, multiple concurrent client requests come over at the same time, all executedjedis.getSet()In the end, only one client can successfully add the lock, but the expiration time of the client lock may be overwritten by other clients.

❞

3.1.3: SET extension command (SET EX PX NX) (note possible problems)

if(jedis. Set (key_resource_id lock_value,"NX"."EX".100s) == 1) {/ / lock
    try {
        do something  // Business processing
    }catch(){
  }
  finally {
       jedis.del(key_resource_id); / / releases the lock}}Copy the code

There could still be problems with this scheme:

  • The lock expired and was released. The service was not finished.
  • The lock was deleted by another thread.

3.1.4 SET EX PX NX + verify the unique random value and then delete it

if(jedis. Set (key_resource_id uni_request_id,"NX"."EX".100s) == 1) {/ / lock
    try {
        do something  // Business processing
    }catch(){
  }
  finally {
       // Check whether the current thread added the lock, if it is released
       if (uni_request_id.equals(jedis.get(key_resource_id))) {
        jedis.del(lockKey); / / releases the lock}}}Copy the code

In this case, determining whether the current thread has added a lock and releasing a lock is not an atomic operation. If jedis.del() is called to release the lock, the lock may no longer belong to the current client.

Lua scripts are usually used instead. The lua script is as follows:

if redis.call('get',KEYS[1]) == ARGV[1] then 
   return redis.call('del',KEYS[1]) 
else
   return 0
end;
Copy the code

3.1.5 Redisson framework + Redlock algorithm to solve the problem of overdue lock release and unfinished service execution + single-machine problem

Redisson uses a Watch Dog to solve the problem of overdue lock release and unfinished business execution. The schematic diagram of Redisson is as follows:

The above distributed locks, there are also single machine problems:

❝

If thread 1 acquires the lock on the master node of Redis, but the lock key has not been synchronized to the slave node. When the master node fails, a slave node becomes the master node. Thread two can acquire the lock with the same key, but thread one has already acquired the lock, so the lock security is lost.

❞

For stand-alone problems, you can use the Redlock algorithm. Interested friends can see my article ha, seven schemes! Discuss the correct use posture of Redis distributed lock

3.2 Attention to Cache consistency

  • If it is a read request, read the cache first, then the database
  • If a write request is made, update the database first and then write to the cache
  • After each update, the cache needs to be cleared
  • Caches generally require some kind of expiration
  • For high consistency requirements, you can use Biglog +MQ.

If you are interested, you can read my article: In a concurrent environment, database operation or cache operation first?

3.3 Reasonably evaluate Redis capacity to avoid invalid expiration time set before due to frequent set overwriting.

As we know, all data structure types of Redis can be set to expire. If you set an expiration time for a string, if you set it again, the expiration time will be invalid.

Redis setKey Redis setKey

void setKey(redisDb *db,robj *key,robj *val) {
    if(lookupKeyWrite(db,key)==NULL) {
       dbAdd(db,key,val);
    }else{
    dbOverwrite(db,key,val);
    }
    incrRefCount(val);
    removeExpire(db,key); // Remove the expiration time
    signalModifiedKey(db,key);
}
Copy the code

In the actual business development, we should also reasonably evaluate the capacity of Redis to avoid frequent set overwriting, resulting in the failure of keys with expiration time set. It’s easy for beginners to make this mistake.

3.4 Cache penetration Problem

Let’s take a look at a common way of using cache: when a read request comes, check the cache first. If a value hits the cache, it is returned directly. When the cache fails, it looks up the database, updates the value of the database to the cache, and returns it.

Cache penetration: Data that does not exist must be queried. If the cache does not hit, the data must be queried from the database. If no data is found, the data will not be written into the cache.

❝

In layman’s terms, when a read request is accessed, neither the cache nor the database has a value, which results in each query request for that value being passed through the database. This is called cache penetration.

❞

Cache penetration typically occurs in one of these situations:

  • “Business bad design”, such as most users do not have daemons on, but you cache every request, query a certain userID to see if there is a daemon.
  • “Business/operations/development errors,” such as cache and database data being deleted by mistake.
  • A “hacker rogue request attack” occurs when a hacker deliberately fabricates a large number of rogue requests to read non-existent business data.

“How do you avoid cache penetration?” There are three ways to do this.

  1. If the request is illegal, we check the parameter in API entry and filter the invalid value.
  2. If the query database is empty, we can set a null value for the cache, or the default value. However, if a write request comes in, the cache needs to be updated to ensure cache consistency, and finally set an appropriate expiration time for the cache. (Commonly used in business, simple and effective)
  3. Use bloom filter to quickly determine whether data exists. That is, when a query request comes in, check whether the value exists through the Bloom filter, and then continue to search.

❝

Bloem filter principle: It consists of an array of bitmaps with an initial value of 0 and N hash functions. One hashes a key N times to obtain N values, hashes the N values in the bit array and sets them to 1. Then, if all of these positions are 1, bloom filter determines that the key exists.

❞

3.5 Cache Snowrush problem

“Cache Snow running:” refers to the large amount of data in the cache to the expiration time, but the query data volume is huge, the request directly access the database, causing the database pressure is too large or even down.

  • Generally, a large amount of data expires at the same time. To solve this problem, you can set the expiration time evenly, that is, the expiration time is relatively discrete. For a large fixed value + a small random value, 5 hours +0 to 1800 seconds.
  • Redis outages can also cause cache snowrush. This requires building a Redis high availability cluster.

3.6 Cache breakdown Problem

Cache breakdown: indicates that a hot key expires at a certain point in time, and a large number of concurrent requests are sent to the key at this point in time, so that a large number of requests are sent to the DB.

Cache breakdown looks a bit like, in fact, it is two differences, cache snow rush refers to the database pressure is too large or even down, cache breakdown is only a large number of concurrent requests to the DB database level. Think of breakdown as a subset of cache Snowrush. Some articles believe that the difference between the two is that the breakdown is targeted at a hot key cache, while Xueben is targeted at many keys.

There are two solutions:

  • 1. Use the mutex scheme. When the cache fails, instead of loading db data immediately, use some atomic operation command with success return, such as setnx of Redis, to operate, and then load DB database data and set the cache. Otherwise, retry to get the cache.
  • 2. Never Expire: When the expiration time is not set but hotspot data is about to expire, the asynchronous thread updates and sets the expiration time.

3.7 Cache hot Key problem

In Redis, we call keys that are accessed frequently hot keys. If a large number of requests are received from a hotspot key, the host resources may be insufficient or even break down. As a result, normal services may be affected.

How do hot keys come about? There are two main reasons:

❝

  • Users consume far more data than they produce, such as seconds kill, hot news and other scenarios that read and write too much.
  • If the performance of a single Redi server is higher than that of a single Redi server, for example, a fixed key is assigned to the same server. As a result, the number of Hash requests exceeds the machine bottleneck, resulting in hot key issues.

❞

So how do you identify hot keys in daily development?

❝

  • Use experience to determine which are hot keys;
  • Client statistics report;
  • Reported by service agent layer

❞

How to solve the hot key problem?

❝

  • Redis cluster expansion: Add fragmented copies to balance read traffic.
  • Hash hot keys, such as backing up a key as key1,key2… KeyN: N backups of the same data are distributed to different fragments. During access, one of the N backups can be randomly accessed to further share read traffic.
  • Use a level 2 cache, or JVM local cache, to reduce Redis read requests.

❞

4. Redis configuration o&M

4.1 Use long Connections instead of short connections and properly configure the client connection pool

  • If a short connection is used, the TCP handshake for three times and wave for four times is required each time, which increases the time required. However, for long connections, the redis command can be used for long connections, which can reduce the time to establish redis connections.
  • Connection pooling enables clients to establish multiple connections without releasing them. Therefore, they do not need to create connections every time when they need to use connections, which saves time. However, parameters must be set properly. If Redis is not operated for a long time, connection resources must be released in a timely manner.

4.2 Use db0 only

The Redis-standalone architecture forbids the use of non-DB0. There are two reasons

  • A connection, Redis execute command select 0 and select 1 switch, will lose new energy.
  • Redis Cluster only supports DB0, which is costly to migrate

4.3 Set maxMemory + appropriate elimination strategy.

To prevent memory overstocking. For example, sometimes, when the business volume increases, the redis key is used a lot, and the memory is directly insufficient, and the operation and maintenance brother also forgets to increase the memory. Did Redis just die like that? Therefore, select maxmemory-policy and set the expiration time based on actual services. There are eight memory flushing strategies:

  • Volatile – LRU: When memory is insufficient to accommodate new writes, the lRU (least recently used) algorithm is used to flush out expired keys;
  • Allkeys-lru: when memory is insufficient to accommodate new writes, the lru (least recently used) algorithm is used to flush out allkeys.
  • Volatile – LFU: New in version 4.0. When the memory is insufficient to accommodate new data, the LFU algorithm is used to delete expired keys.
  • Allkeys-lfu: new in version 4.0. When the memory is insufficient to hold new data, lfu algorithm is used to eliminate allkeys.
  • Volatile -random: Randomly discard data from expired keys when memory is insufficient to accommodate new writes. .
  • Allkeys-random: randomly weed out data from allkeys when memory is insufficient to accommodate new writes.
  • Volatile – TTL: When the memory is insufficient to accommodate new data, the key whose expiration time is set is discarded based on the expiration time. The key whose expiration time is earlier is discarded first.
  • Noeviction: Default policy, new write operations will bug when memory is insufficient to accommodate new write data.

4.4 Enabling the lazy-free mechanism

Redis4.0+ supports lazy-free. If you still have bigKey in your Redis, it is recommended to enable lazy-free. When enabled, Redis removes a bigkey, and the time-consuming operation to free memory is carried out by the background thread, reducing the blocking effect on the main thread.

Reference and thanks

  • Redis should not use the KEYS command, otherwise you will get beaten [1]
  • Aliyun Redis development specification [2]
  • Redis Best Practice Guide: 7 dimensions +43 usage specifications
  • Redis cache penetration and its solution — BloomFilter[3]
  • Redis Cache performance Practice and Summary [4]

Reference

[1]

Redis don’t mess with KEYS command, or you will beat: www.cnblogs.com/tonyY/p/121…

[2]

Ali cloud Redis development specification: developer.aliyun.com/article/531…

[3]

Redis cache to penetrate and solutions – bloom filter BloomFilter:blog.csdn.net/wx152815940…

[4]

Redis cache performance practice and summary: www.shangmayuan.com/a/d2f178b54…

Read more

Implement a simple genetic algorithm from scratch in Python \

5 minutes to master Python random hill-climbing algorithm \

5 minutes to fully understand association rule mining algorithm \

Special recommendation \

\

Click below to read the article and join the community