Cache plays an important role in our daily development. Through cache components, our system can be improved in many ways. Redis is a representative caching component. I’ve been using Redis a lot lately, so I’m going to document some of the problems I’ve encountered in Redis development and some of the specifications. The article has been posted on Github and will be continuously updated. Welcome star and additions

The design of the Key

Easier to manage

That is, you can roughly know the business involved by the name. Typically we name the service: Characteristics, such as pubg_chat:uid:room_id, so as to avoid conflicts as much as possible (of course, it is better to use different Redis for different services).

As far as possible concise

Redis is essentially an in-memory database, and memory is much smaller than hard disk. If the Key is too large, Redis can store less. Therefore, in daily use, recommended keys should be as brief as possible, and abbreviations should be used instead of full words

Avoid special characters

Escape characters such as commas, newlines, Spaces, and quotes should not be used

Setting the life Cycle

  • Redis should not be a permanently stored component, with an expiration date for each Key
  • If you do need Redis to permanently store certain types of content, you should not set up a permanent life cycle at the beginning of the process to avoid maintenance disasters when changes are needed later on because of the asynchronous “renewal” approach

The Value of using

Avoid big keys!

  • Redis is single-threaded. It will execute one command before any other command is executed
  • First of all, when a large Key transmits key-value pairs, it will cause pressure on the network (bandwidth problem), and some proxies will fragment the large content to transmit, thus increasing the network transmission time again
  • Second, the use of O(n) instructions or del commands for structures such as list and hash can cause serious blocking
  • Therefore, in general, the string type should be limited to 10KB, and the number of hash, list, set, and zset elements should not exceed 5000
  • Ps: Large keys can be split using hash

The value of the compression

Protobuf, MessagePack and other methods can be considered for serialization, which can firstly improve the utilization rate of Redis, secondly improve the efficiency of serialization, and thirdly provide the ability of value across languages

Command to use

Avoid frequently appending strings

Consider using a list instead

Set class operation

  • The O (n) instruction should be noted. For set, zset, list, hash, etc., pay attention to the performance impact of the O (n) command. Generally, the direct use of O (n) instructions should be avoided. HSCAN, SSCAN, ZSCAN can be used for progressive operations to prevent command blocking
  • Progressive deletion. Instead of using del directly, you should write your own script to remove it bit by bit

Disabling dangerous Commands

Keys, flushall, flushdb…… This need not say more, a direct Redis meng circle, people are also mute

Make proper use of Pipeline model

  • When MGET has a large amount of data, proxy will unpack and unpack, which will lead to increased pressure on proxy layer, while pipeline mode will directly forward. Therefore, in the case of batch acquisition, pipeline efficiency is generally better than MGET
  • At the same time, two points should be noted:
    • Mget is an atomic operation; Pipeline is not. Therefore, business should not be blindly adopted
    • Pipelines can send different commands, and lua can also do this

Avoid unnecessary instructions

If some Redis clients have probe commands such as TestOnBorrow, such commands should be avoided without special requirements in order to reduce Redis load and network pressure

Special requirements should be made for Lua

  • All KEYS should be passed by the KEYS array
  • All values should be passed by the ARGS array
  • All keys must be in one slot

Performance query instruction

  • Slowlog get: query the slowlog command
  • Info commandStats: Displays the information about the executed commands, including the time and times
  • Client list, which queries commands that cause blocking

Client use

Avoid mixing instances

Instances of different service lines should be split to avoid interlocking problems caused by intermixing, such as key overlap and command blocking

Using connection pools

There are several problems with creating new connections each time you use them:

  • Increased burden on Redis
  • Wasting Network resources
  • Impact on execution efficiency
  • Difficult to maintain

fusing

Redis is also a “service” in nature, so circuit breakers are indispensable

authentication

Avoid abuse of irrelevant services or data errors

Avoid being a message queue

Redis can support message queuing, but its read and write efficiency is not as high as other MQ systems such as Kafka and RabbitMQ. And because its structure is set up to not support some of MQ’s key features, it should be avoided.

other

Elimination strategy

According to the service type, select maxmemory-policy(maximum memory elimination policy) and set the expiration time to volatile- LRU by default. That is, when the maximum memory is exceeded, lRU algorithm is used to delete expired keys to ensure that the data that does not expire will not be deleted, but OOM problems may occur

  • Allkeys-lru: Deletes keys according to the LRU algorithm, regardless of whether the timeout attribute is set, until enough space is made
  • Allkeys-random: randomly remove allkeys until enough space is left.
  • Volatile -random: Deletes expired keys randomly until sufficient space is available
  • Volatile – TTL: Deletes the recently expired data based on the TTL attribute of the key value object. If not, retreat to Noeviction strategy
  • Noeviction: Won’t exclude any data, reject all write operations and return client error message “(error) OOM command not allowed when used memory”, Redis will only respond to read operations
  • Allkey-lfu and volatile- LFU are available after 4.0
    • This can be considered as a further improvement on the LRU algorithm
    • Allkey-lfu: Removes the least frequently used key from all keys
    • Volatile – lFU: Expels the least frequently used key from all keys with expiration time configured

Do not abuse Redis transactions

Redis transactions are not as “safe” as DB transactions and do not support rollback, so they should not be used too much. Because I have never used this part, please refer to the official document

The resources

  • Redis official documentation
  • Aliyun Redis development specification
  • Random notes on improving the Redis LRU algorithm