An overview of the

The redis cluster deployment mode mostly adopts Twemproxy mode to deploy. That is, Twemproxy is used for sharding calculation of Redis key, and the Redis key is sharded and allocated to one of the multiple Redis instances. The architecture diagram of TewmProxy is as follows:



Multiple Redis instances behind Twemproxy have the same memory configuration and CPU configuration. Therefore, once the traffic volume tilt or data volume tilt occurs, one Redis instance may reach the performance bottleneck, and thus the whole cluster may reach the performance bottleneck.

The hot Key appears, causing the cluster traffic tilt

Hot key

, that is, a hot key. In a period of time, the traffic volume of this key is much higher than that of other Redis keys. As a result, most access traffic is concentrated on one Redis instance after proxy sharding. Hot Keys store different hotspot information in different services. Such as

  1. Hot news content in news application;

  2. Activity configuration of an activity that a user participates in crazily in the active system;

  3. Mall second kill system, the most attractive user eye, the most cost-effective commodity information; …

The solution

1. Use local cache

The use of local cache on the client reduces the number of visits to hot keys by redis cluster, but at the same time brings two problems: 1. If all keys that may become hot keys are cached locally, whether the local cache will be too large, thus affecting the cache overhead required by the application itself. 2. How to ensure the consistency of validity period between local cache and Redis cluster data. For these two problems, let’s leave it to the second solution.

2. Use the features of the sharding algorithm to split keys

We know that a Hot key is a hot key because it has only one key that lands on an instance. Therefore, hot keys can be prefixed or suffixed to change the number of hotkeys to a multiple M of the number of Redis instances, so that access to N * M Redis keys can be changed from accessing one Redis key. N*M Redis keys are sharded and distributed to different instances, and the traffic volume is evenly distributed to all instances.

The code is as follows:

// The number of instances of redis const M = 16 // The number of instances of redis multiple (on demand, 2^n, n is usually an integer from 1 to 4) const n = 2 funcmain() {// get the redis instance c, err := redis.dial ("tcp"."127.0.0.1:6379")
    iferr ! = nil { fmt.Println("Connect to redis error", err)
        return
    }
    defer c.Close()

    hotKey := "hotKey:abc"// Random number randNum := GenerateRangeNum(1, N*M) // Get the hotKey tmpHotKey := hotKey +"_"+ strconv.Itoa(randNum) // Hot Key expiration time expireTime := 50 // A random value of expiration time randExpireTime := GenerateRangeNum(0, 5) data, err := redis.String(c.Do("GET", tmpHotKey))
    iferr ! = nil { data, err = redis.String(c.Do("GET", hotKey))
        iferr ! = nil { data = GetDataFromDb() c.Do("SET"."hotKey", data, expireTime)
            c.Do("SET", tmpHotKey, data, expireTime + randExpireTime)
        } else {
            c.Do("SET", tmpHotKey, data, expireTime + randExpireTime)
        }
    }
}

Copy the code

In this code, a TMP key is obtained by a random number greater than or equal to 1 and less than M * N. The program will preferentially access the TMP key. If data is not available, the program will access the original hot key and write the contents of the hot key back to the TMP key. Note that the expiration time of the TMP key is the expiration time of the Hot key plus a small random positive integer, which ensures that when the hot key expires, all TMP keys do not expire at the same time causing a cache avalanche. This is a way to avoid avalanches by slope expiration, and can also use atomic locks to write data more perfectly, reducing the DB pressure.

Another thing worth mentioning is that by default, when generating TMP key, we will use random number as the suffix of hot key, which conforms to the redis namespace, and facilitates key collection and management. However, there is an extreme case, that is, the hot key has a very long length. In this case, random numbers cannot be added as suffixes. The reason is that in the calculation process of Twemproxy sharding algorithm, the weight of the earlier characters is larger, and the weight of the later characters is smaller. In other words, for key names, the greater the difference in the preceding characters, the greater the difference in the calculated fragment values, and the more likely it is to be allocated to different instances (the specific algorithm is not explained here). Therefore, for hot keys with long key names, be careful to place random numbers, such as at the beginning of the last command space (eg: space1:space2:space3_rand instead of space1:space2:rand_space3).

The data volume in the cluster is skewed by the big key

big key

, that is, the data size of a key with a large amount of data is much larger than that of other keys. As a result, after sharding, the memory usage of a specific instance storing the big key is much larger than that of other instances, causing insufficient memory and dragging down the usage of the whole cluster. Big Key is usually represented as different data in different businesses, for example:

  1. Large and lasting building activities in the forum;

  2. A message list of popular chat rooms in a chat system; …

The solution

Split the big key

Value1, value2, value2, value2, value2 valueN,

  1. If the big value is a large JSON, the contents of the key are scattered to each instance in the form of Mset to reduce the influence of the big key on data skew.

// save mset key1, vlaue1, key2, vlaue2... KeyN, valueN // fetch mget key1, key2... keyNCopy the code

  1. If big value is a big list, you can split the list. List_1, list_2, list3, listN

  2. The same applies to other data types.

Both a big key and a hot key

In the development process, some keys not only have a large amount of traffic, but also a large amount of data. At this time, it is necessary to consider the scenario of using this key, whether it is reasonable to store it in redis cluster, and whether it is more appropriate to use other components to store it. If you insist on using Redis for storage, you may consider migrating out of the cluster to a single-master/standby (or 1-master/multi-standby) architecture for storage.

other

How to find the hot key

1. Anticipation

In the business development stage, it is necessary to judge and process the data that may become hot key or big key in advance, which requires an understanding of the product business, a grasp of the operation rhythm and experience in data design.

2. In-process – monitoring and automatic processing

monitoring

  1. On the application side, collect and report the operation of each redis request; It is not recommended, but can be considered in scenarios where O&M resources are insufficient. Development can bypass operation and maintenance);

  2. In the proxy layer, every REDis request is collected and reported. (recommended, operation and maintenance to do natural is the best plan);

  3. Run the monitor command to count hot keys for redis instances (not recommended, because redis memory may burst in high concurrency).

  4. At the machine level, the Redis client uses TCP protocol to interact with the server, and the communication protocol is RESP. If you stand in the perspective of the machine, you can capture all the TCP packets of Redis ports on the machine to complete the statistics of hot key (not recommended, the company has a lot of basic components on each machine, do not add to the mess);

Automatic processing

After monitoring, the program can obtain the Big key and hot key, and then alarm, the program automatically processes the big key and hot key. Or instruct the programmer to use a tool to customize the processing (perform the previously mentioned solution on a particular key in the program)

After 3.

Try not to afterwards bar, are blood and tears of the lessons, do not unfold.

Thank you for reading and welcome to exchange.