Hello, I’m Tiger!
Redis is a popular in-memory database that relies heavily on memory. However, if the memory is improperly used, OOM is generated and services are affected.
Note: This article is based on Redis 6.2
01 What is OOM
Introduced in the Linux operating system early, it is a mechanism to protect the system when the memory is insufficient through the Kill process.
The Redis OOM can be divided into two categories:
1.Redis checks that the memory usage exceeds the upper limit and returns an OOM error. At this time, memory Consuming commands cannot be executed. Such as SET, LPUSH, and so on.
2. The memory of the operating system is insufficient. Select the Kill process with high memory usage. Redis is a heavy user of memory and is at high risk of being killed. You need some redundant memory to avoid this.
This article focuses on the analysis of “Redis itself produces OOM”.
02 Redis OOM Trigger time
In redis.conf, we usually set the memory limit for Redis. To avoid unlimited memory usage being killed by the operating system. The following
Maxmemory 1073741824 // Bytes indicate a maximum memory size of 1 gbCopy the codeIf Redis uses more memory than “maxMemory”, the request will error OOM.
127.0.0.1:6379> Set foo bar (error) OOM command not allowed when used memory > 'maxMemory '.Copy the codeThe OOM error is generated when the command function processCommand is executed. The following code
int processCommand(client *c) {...Is_denyoom_command is the flag indicating whether the command consumes memory
intis_denyoom_command = (c->cmd->flags & CMD_DENYOOM) || (c->cmd->proc == execCommand && (c->mstate.cmd_flags & CMD_DENYOOM)); .if(server.maxmemory && ! server.lua_timedout) {// Check whether OOM is enabled and implement memory reclamation policy. Noeviction is designed to disable recycling policy.
int out_of_memory = (performEvictions() == EVICT_FAIL);
int reject_cmd_on_oom = is_denyoom_command;
// In the context of a transaction, the command queue is added to consume memory by subcommands to be executed
// Therefore, the transaction will check the OOM status even if the command is read
if(c->flags & CLIENT_MULTI && c->cmd->proc ! = execCommand && c->cmd->proc ! = discardCommand && c->cmd->proc ! = resetCommand) { reject_cmd_on_oom =1;
}
// Check whether it is OOM
if (out_of_memory && reject_cmd_on_oom) {
rejectCommand(c, shared.oomerr);
return C_OK;
}
// Lua script special handling. See issue on 5250/6565
if(c->cmd->proc == evalCommand || c->cmd->proc == evalShaCommand) { server.lua_oom = out_of_memory; }}... }Copy the codeWhere the is_denyoom_command variable is retrieved from each redisCommand sflag. Redis sets the flag “use-memory” for commands that consume memory.
The following get command does not consume memory, so there is no use-memory flag. The set command consumes memory, hence the “use-memory” flag.
struct redisCommand redisCommandTable[] ={... {"get",getCommand,2."read-only fast @string".0.NULL.1.1.1.0.0.0},
{"set",setCommand,- 3."write use-memory @string".0.NULL.1.1.1.0.0.0},... }Copy the codeThe performEvictions function internally calls getMaxmemoryState to get the memory status. If more memory is used than maxMemory, C_ERR is returned.
int performEvictions(void) {...// Get the memory usage status
if (getMaxmemoryState(&mem_reported,NULL,&mem_tofree,NULL) == C_OK)
return EVICT_OK;
// If no memory reclamation policy is configured, an OOM error occurs
if (server.maxmemory_policy == MAXMEMORY_NO_EVICTION)
returnEVICT_FAIL; . }Copy the codeYou can see this from the getMaxmemoryState function. Overhead (slave copy buffer and AOF buffer) mem_used compared to server.maxMemory is mem_reported minus overhead (slave copy buffer and AOF buffer).
int getMaxmemoryState(size_t *total, size_t *logical, size_t *tofree, float *level) {
size_t mem_reported, mem_used, mem_tofree;
mem_reported = zmalloc_used_memory();
if (total) *total = mem_reported;
// If maxMemory is not configured or zmalloc allocated memory <= maxMemory, fast return
intreturn_ok_asap = ! server.maxmemory || mem_reported <= server.maxmemory;if(return_ok_asap && ! level)return C_OK;
// mem_used deducts slave buffer and AOF buffer
mem_used = mem_reported;
size_t overhead = freeMemoryGetNotCountedMemory();
mem_used = (mem_used > overhead) ? mem_used-overhead : 0; .if (mem_used <= server.maxmemory) returnC_OK; .return C_ERR;
}
Copy the codeIn summary, when Redis uses more mem_used memory than you configured for maxMemory, it causes the following consequences
- Redis cannot execute “memory consuming commands”, such as SET and LPUSH, with the “use-memory” flag.
- Redis rejects all transaction requests, even if the transaction contains only “read commands”.
It can be seen that Redis itself has a great influence on OOM.
So Redis itself uses memory “mem_used”, which consists of.
03 MEMORY STATS command
The MEMORY STATS command allows you to obtain the current MEMORY cost of Redis. Better analyze and monitor the causes of memory growth. The result is as follows:
127.0. 01.:6379> memory stats
1) "peak.allocated".
2) (integer) 630469680
3) "total.allocated"
4) (integer) 630409184
5) "startup.allocated"
6) (integer) 1028192
7) "replication.backlog"
8) (integer) 0
9) "clients.slaves"
10) (integer) 0
11) "clients.normal"
12) (integer) 17440
13) "aof.buffer"
14) (integer) 0
15) "lua.caches"
16) (integer) 0
17) "db.0"
18) 1) "overhead.hashtable.main"
2) (integer) 10097272
3) "overhead.hashtable.expires"
4) (integer) 32
19) "overhead.total"
20) (integer) 11142936
21) "keys.count"
22) (integer) 200003
23) "keys.bytes-per-key"
24) (integer) 3146
25) "dataset.bytes"
26) (integer) 619266248
27) "dataset.percentage"
28) "98.392906188964844"
29) "peak.percentage"
30) "99.990409851074219"
31) "allocator.allocated"
32) (integer) 630343600
33) "allocator.active"
34) (integer) 634752000
35) "allocator.resident"
36) (integer) 634752000
37) "allocator-fragmentation.ratio"
38) "1.0069936513900757"
39) "allocator-fragmentation.bytes"
40) (integer) 4408400
41) "allocator-rss.ratio"
42) "1"
43) "allocator-rss.bytes"
44) (integer) 0
45) "rss-overhead.ratio"
46) "1.0000597238540649"
47) "rss-overhead.bytes"
48) (integer) 37888
49) "fragmentation"
50) "1.0070537328720093"
51) "fragmentation.bytes"
52) (integer) 4446288
Copy the codeThere are a lot of indicators, so let’s explain them one by one
Peak. allocated: Redis memory usage peak. This value is equivalent to the info command used_memory_peak. Updates are triggered in three places.
- Each event loop in the serverCron
- After each command is executed
- After executing memory-related commands, such as “Memory Usage key”
Total_allocated: Redis is using the allocated total memory(such as libc, Jemalloc or TCmalloc) using zmalloc_used_memory(). Allocated tokens = info total.allocated.
Startup. allocated: The initial memory size obtained in InitServerLast when Redis is started. Data loaded by RDB and AOF is not included because RDB and AOF are loaded after InitServerLast. The value is equivalent to the info startup.allocated command.
int main(a){
InitServerLast(a);loadDataFromDisk(a); }Copy the codeReplication. Backlog: The backlog buffer for master/slave replication. When the master/slave replication is disconnected, the write commands sent by the master to the slave are temporarily stored in the backlog buffer. This buffer can be configured with the repl-backlog-size, which defaults to 1M. Usually we will set it to be larger, such as 256M. Avoid full data synchronization due to buffer overflows.
5. Clients. Slaves: Total cost of all replica nodes. Includes copy read/write buffer, connection context size. ClientsCronTrackClientsMemUsage reference function
6. Clients. normal: Except for all replica nodes, other clients read and write cache and connection context size. It is important to note that reading and writing the big key may result in clients. Normal growth and eventually OOM. Conf “client-output-buffer-limit” can be used to limit the size of the client buffer.
Aof. Buffer: aof and aof rewrite buffer sizes. It may have risen rapidly in the AOF rewrite period.
- AOF buffer: Caches commands executed in an event loop. Write files at the end of each event loop according to the AOF flush policy.
- AOF rewrite buffer: When a fork child is rewriting, the parent receives “write commands” that are temporarily stored in aof_rewrite_buf_blocks. When the child process rewrite ends, it appends the buffer to the AOF file.
8. Lua. caches: Lua scripts and information. Usually not very big.
9. Db. 0: indicates the memory size of DB 0. Usually with a few db, there are a few values. Can be used to help us analyze each DB memory usage. The calculation code is as follows:
for (j = 0; j < server.dbnum; j++) {
redisDb *db = server.db+j;
long long keyscount = dictSize(db->dict);
if (keyscount==0) continue;
mh->total_keys += keyscount;
mh->db = zrealloc(mh->db,sizeof(mh->db[0])*(mh->num_dbs+1));
mh->db[mh->num_dbs].dbid = j;
mem = dictSize(db->dict) * sizeof(dictEntry) +
dictSlots(db->dict) * sizeof(dictEntry*) +
dictSize(db->dict) * sizeof(robj);
mh->db[mh->num_dbs].overhead_ht_main = mem;
mem_total+=mem;
mem = dictSize(db->expires) * sizeof(dictEntry) +
dictSlots(db->expires) * sizeof(dictEntry*);
mh->db[mh->num_dbs].overhead_ht_expires = mem;
mem_total+=mem;
mh->num_dbs++;
}
Copy the codeOverhead. Total: 3-9 Spoils + clients. Normal + Aof.buffer + lua.caches + db.0. Manage the sum of data structures within Redis key space. You can think of the size of the remaining management class memory excluding the data set. Equivalent to info command used_memory_overhead.
Keys. count: indicates the total number of stored keys and the sum of the sizes of all DB hash tables.
12. Keys. bytes-per-key: Average memory size per key, excluding memory at startup.
13. Dataset. Bytes: Redis memory size of all stored data. This value is equal to total_allocate – overhead. Total dataset is what you need to be concerned about, to determine whether the OOM is caused by the client writing a large amount of data to the dataset. Equivalent to the info command used_memory_dateset
14. Dataset. Percentage: Percentage of data storage dataset.
15. Peak. percentage: indicates the maximum percentage of memory usage.
Indicator 16-18 Is sampled by the cronUpdateMemoryStats function to calculate the memory fragmentation rate.
Allocated memory size allocated by Rdis via Zmalloc. Usually the same as usED_memory.
17. Allocator. Active: Redis allocator active page size. Contains memory fragmentation. This value is obtained using the ps or top command and the memory used by Lua is subtracted.
18. Allocator. Resident: Redis resident memory RSS. Contains memory that can be returned to the operating system.
19. Allocator -fragmentation. Ratio: memory fragmentation rate. Allocated tokens equal to Allocator. Active/Allocator
20. Allock-fragmentation. Bytes: specifies the size of the memory fragment. Allocated tokens = allocator. Active-allocator. Allocated Tokens
21. Allocator – rsS.ratio: The allocator may soon release back the page ratio of the operating system. The value is allocator. Resident/Allocator. Active
22. Allocator – rsS.bytes: May soon be released back to the operating system page size. The value is allocator. Resident-allocator. Active
23. Rss-overhead. Ratio: memory overhead ratio independent of the Redis allocator. (Redis 6.2 specifically refers to luA_memory) is equal to process_RSS/allocator_resident
Rss-overhead. Bytes: Memory overhead independent of the Redis allocator (Redis 6.2 specifically refers to lua_memory) is equal to process_RSs-allocator_resident
25. Fragmentation: process OS memory RSS (process_rss)/Redis usage memory (zmalloc_used)
26. Fragmentation. Bytes: process OS memory RSS (process_rss) – Redis usage memory (zmalloc_used)
Why is allocator-fragmentation. Ratio increased when we have fragmentation? This is because fragmentation includes Lua memory usage. Lua allocates memory via malloc. Redis itself allocates memory via Zmalloc. The memory fragmentation rate is too high. Allocator. Active Removes memory occupied by Lua scripts. The memory fragmentation rate will be more accurate. There are many indicators mentioned above. It is recommended that you conduct periodic sampling of the indicators you care about and monitor the alarm.
04 What Operations may Cause Memory Explosion?
As mentioned earlier, Redis determines OOM and actually uses Zmalloc to allocate memory minus slaves Buffer and AOF buffer size and then compares it with MaxMemory.
Some of them need our attention because they can cause sudden memory spikes.
1. Datesets: Pure data size. If at some point, the datesets surge. There are several possibilities
- Write command QPS inflation.
- The client keeps writing the big key.
2. Client. normal: indicates the client buffer. If the client. Normal soared in a certain period of time.
- Read command QPS inflation.
- The client is reading a large number of big keys.
- Lots of new connections added.
Write commands cause memory to grow so much that it makes sense to increase the size of the data set. Why does reading commands increase memory?
Redis uses Zmalloc to allocate memory based on the key size, which is calculated in client.normal. If the QPS for big Key access is particularly high at one time. Redis memory usage will also skyrocket. The following code
void _addReplyProtoToList(client *c, const char *s, size_t len) {
...
if (len) {
size_t size = len < PROTO_REPLY_CHUNK_BYTES? PROTO_REPLY_CHUNK_BYTES: len;
// Allocate memory based on key size
tail = zmalloc(size + sizeof(clientReplyBlock));
/* take over the allocation's internal fragmentation */
tail->size = zmalloc_usable_size(tail) - sizeof(clientReplyBlock);
tail->used = len;
memcpy(tail->buf, s, len);
listAddNodeTail(c->reply, tail);
c->reply_bytes += tail->size;
closeClientOnOutputBufferLimitReached(c, 1); }}Copy the codeThat’s why I always say try to avoid big keys.
3. Aof. Buffer: Indicates the size of the AOF buffer. If the aOF.buffer surges for a certain period of time. This Redis instance is rewriting and may have more big keys than QPS. It is usually recommended to enable AOF on the slave node. Avoid affecting the performance of the primary node.
05 Redis memory reclamation policy
To avoid its own OOM, Redis provides a variety of memory reclamation strategies.
The default is noeviction, which means reclamation will not be performed.
The other 7 are:
- Volatile – LRU: LRU elimination is performed on keys with expiration time.
- Allkeys-lru: performs lru elimination on allkeys.
- Volatile – LFU: LFU elimination is implemented for keys with expiration time.
- Allkeys-lfu: implements the lfu elimination policy among allkeys.
- Volatile -random: The key with expiration time is randomly eliminated.
- Allkeys-random: allkeys are randomly eliminated.
- Volatile – TTL: Eliminates based on expiration time, starting with the shortest TTL.
There are many recycling strategies, but they fall into three categories
- Scope for eliminating keys. All keys are still keys with an expiration date.
- Which algorithm to use. LRU, LFU, random, TTL
- By default, no elimination is performed
I drew a picture for your reference
I have the following suggestions for selecting an elimination strategy.
-
Do not choose noeviction, the default reclamation policy, unless Redis cache data cannot be deprecated. In general, we think that a key without an expiration date cannot be eliminated. Such as caching live goods in Redis. If this parameter is deleted, cache breakdown may occur and service exceptions may occur.
-
Select the one with expiration time for elimination. If most keys in Redis have expiration dates, it is best to weed out keys with expiration dates.
-
Select the LFU policy. Obsolescence is performed based on key usage frequency. LFU is an improved algorithm of LRU. LRU has congenital defects. Even if the key has not been accessed recently, it does not mean that the access frequency is low. It may be accessed in a large number of times suddenly.
-
Different memory elimination algorithms consume different CPU, you need to choose the algorithm based on the actual situation. I recommend that you keep Redis memory in a healthy state. If the CPU usage is high, you are advised to split multiple instances or upgrade the Cluster.
If you come across Redis OOM, please feel free to join us in the comments section
-End-
Finally, welcome to my public account “Tiger”.
I will continue to write better technical articles.
If my article is helpful to you, please like it and follow it