preface
Because one line of business was not ideal, the top management decided to pull the business. As far as our technical team is concerned, all server resources and other related resources should be released. Eight application servers were released; One ES server; Delete service tasks related to the distributed scheduled task center. MySQL > delete MySQL; Delete the relevant business cache data in Redis. The CTO named me to lead the charge and docked my performance… Okay, everything else is fine. It won’t take long. Only this delete Redis data, caused me to stay up all night, really frustrated me too!
The difficulties in analysis
Share the Redis service cluster
Since the data of this business line is about 3G in Redis, there is no need to build a separate Redis service cluster. In line with the attitude of saving as much as possible, we decided to share a cluster with other projects at the beginning (the cluster configuration: 16 nodes, 128G memory, still luxurious). The cluster configuration is as follows:
The Key is not properly named
If you want to delete keys, you must first accurately locate which keys need to be deleted. If you do not delete keys, other services will be affected. If the Key itself is set to expire, but some data needs to be persistent. However, the damn project manager kept pushing the project schedule, resulting in the development of many parts of the design is not in place, such as Redis Key scattered in every corner of the project code; For example, the naming is not very standard. I don’t know how to review the code! Oh, there must be no time to review, that damn project manager… I’ll arbitrarily intercept the Key in the payment service and name it:
How’s that？ Don’t laugh at the code we developers write. In practice, it’s even worse! Hope you don’t encounter, or really painful ~
solution
After the above analysis, we can summarize as follows:
- What we really care about are keys that don’t have an expiration date set
- Do not delete the Key by mistake, otherwise the performance of the next month will be lost
- It is difficult to locate keys because of the non-standard naming and use of keys
The scan command does not work for matching keys. It can only be searched by human flesh. Fortunately, Idea is a good search method. In this project, spring-boot-starter-data-redis is used. So I locate the code for all operations on redis by searching RedisTemplate and StringRedisTemplate as follows: 1. Use this code to calculate the prefix of the Key and input it into the text; 2. Use the Python script to load the Key in the text and add the “*” wildcard character after it; 3. Run the Python script to scan the keys using the scan command. 4. For checking convenience, we did not delete the key directly by using del command. Before deleting the key, we first obtained the serialization length of the key by means of debug object key, and then deleted the key and returned the serialization length. In this way, we can count the serialization lengths of all keys to determine how much space we have freed. The key codes are as follows:
def get_key(rdbConn,start):
try:
keys_list = rdbConn.scan(start,count=2000)
return keys_list
except Exception,e:
print e
''' Redis DEBUG OBJECT command got key info '''
def get_key_info(rdbConn,keyName):
try:
rpiple = rdbConn.pipeline()
rpiple.type(keyName)
rpiple.debug_object(keyName)
rpiple.ttl(keyName)
key_info_list = rpiple.execute()
return key_info_list
except Exception,e:
print "INFO : ",e
def redis_key_static(key_info_list):
keyType = key_info_list[0]
keySize = key_info_list[1] ['serializedlength']
keyTtl = key_info_list[2]
key_size_static(keyType,keySize,keyTtl)
Copy the codeIn this way, it is possible to figure out how much memory has been freed. Since the cluster is so close to 70 million keys:
Shame leads to courage
You have never had the experience of clearing resources due to offline services. This time things really let me feel that the subtleties see the truth of kung fu. If we had followed development specifications to use and design Redis Key from the beginning, we wouldn’t have wasted so much time. In order to make the naming and use of keys more standardized, and to avoid such situations in the future, I woke up in the afternoon and added a configuration and custom key serialization to the Redis public component library. The code is as follows:
@ConfigurationProperties(prefix = "spring.redis.prefix")
public class RedisKeyPrefixProperties {
private Boolean enable = Boolean.TRUE;
private String key;
public Boolean getEnable(a) {
return enable;
}
public void setEnable(Boolean enable) {
this.enable = enable;
}
public String getKey(a) {
return key;
}
public void setKey(String key) {
this.key = key; }}Copy the code/ * * *@descSerialization of string adds prefix *@author create by liming sun on 2020-07-21 14:09:51
*/
public class PrefixStringKeySerializer extends StringRedisSerializer {
private Charset charset = StandardCharsets.UTF_8;
private RedisKeyPrefixProperties prefix;
public PrefixStringKeySerializer(RedisKeyPrefixProperties prefix) {
super(a);this.prefix = prefix;
}
@Override
public String deserialize(@Nullable byte[] bytes) {
String saveKey = new String(bytes, charset);
if(prefix.getEnable() ! =null && prefix.getEnable()) {
String prefixKey = spliceKey(prefix.getKey());
int indexOf = saveKey.indexOf(prefixKey);
if (indexOf > 0) { saveKey = saveKey.substring(indexOf); }}return (saveKey.getBytes() == null ? null : saveKey);
}
@Override
public byte[] serialize(@Nullable String key) {
if(prefix.getEnable() ! =null && prefix.getEnable()) {
key = spliceKey(prefix.getKey()) + key;
}
return (key == null ? null : key.getBytes(charset));
}
private String spliceKey(String prefixKey) {
if(StringUtils.isNotBlank(prefixKey) && ! prefixKey.endsWith(":")) {
prefixKey = prefixKey + "... "";
}
returnprefixKey; }}Copy the codeUse effect
In order to avoid this kind of inefficient work and have to do again, we stipulate in the development specification that the use of redis in the new project must set this configuration, the prefix is set to: project number. In addition, keys in a module must be uniformly defined in the RedisKeyConstant class of the binary library. The configuration is as follows:
spring:
redis:
prefix:
enable: true
key: E00P01
Copy the code@Bean
public RedisTemplate<String, Object> redisTemplate(RedisConnectionFactory redisConnectionFactory) {
RedisTemplate<String, Object> redisTemplate = new RedisTemplate<>();
redisTemplate.setConnectionFactory(redisConnectionFactory);
// Support key Serializer for key prefix Settings
redisTemplate.setKeySerializer(new PrefixStringKeySerializer());
redisTemplate.setValueSerializer(new GenericJackson2JsonRedisSerializer());
return redisTemplate;
}
Copy the codeIn this way, we can at least distinguish the key from the project dimension, avoiding the problem of duplicate keys when multiple projects share the same cluster. Keys are divided by project dimension. More convenient management, operation and maintenance. If the management granularity for keys is more fine-grained, we can even refine the specific business dimensions. We pressed in a test environment and adding the key prefix had little impact on Redis performance. Performance acceptable.
conclusion
What I’ve learned from this is that for most developers, the gap isn’t really about intelligence, it’s about attitude. For example, the problem that this incident has exposed is that everyone knows to follow development guidelines, but when it comes to the real battle, few of the developers in charge of the project can consistently do these little things well. In addition, the reviewer’s work is really important, just like that of the discipline inspection Commission. If the Discipline Inspection Commission turns a blind eye, there will be big trouble! After this incident, if god gives another chance, I will say to the project manager: continue to play music, continue to dance ~
What is described in this article is what HAPPENED in my actual project. If you have a better solution, please feel free to comment, leave a message or email: [email protected]. Thank you very much!