background
In order to reduce memory usage, keys in Redis are usually set to expire by expire.
Setting expiration Time
There are four ways to set the expiration time
Expire <key> < TTL > # expire the key to TTL milliseconds pexpire <key> < TTL > # expire the key to TIMESTAMP Expire <key> <timestamp> # expire <key>Copy the codeThe first three methods are converted to the last method to achieve expiration time
Save expiration time
Let’s look at the structure of redisDb
typedef struct redisDb {
dict *dict; /* The keyspace for this DB */
dict *expires; /* Timeout of keys with a timeout set */
...
}Copy the codeYou can see that the expire dictionary of the redisDb structure holds the expiration times of all keys
The key of an expired dictionary is a pointer to a key object in the key space
The value of the expired dictionary holds the expiration time of the database key to which the key points
Pay attention to
There are duplicates between expired fields and key objects in the key space. In practice, duplicate objects do not exist. Keys in the key space and keys in the expired dictionary point to the same key object
Expiration key judgment
By querying the expired dictionary, check the following conditions to determine whether it is expired
- Checks if the given key is in the expired dictionary and gets the expiration time of the key if it is
- Check whether the current UNIX timestamp is longer than the expiration time of the key
Deletion policy for expired keys
Lazy to delete
A key expiration check is performed when the key is removed, that is, only the current key is deleted and no CPU time is spent on other expired keys
Disadvantages: not memory friendly, if a brother key expires, but will be saved in memory, if the key is not accessed, so long will cause memory waste, even memory leak
How to do that?
The expireIfNeeded method is called before the Redis read/write command is executed
If the key has expired, the expireIfNeeded method removes it
If the key is not expired, the expireIfNeeded method does not process it
Db.c /expireIfNeeded method
Int expireIfNeeded(redisDb *db, robj *key) {return 0 if (! keyIsExpired(db,key)) return 0; If (server.masterhost!) {// If (server.masterhost!) {// If (server.masterhost! = NULL) return 1; Server.stat_expiredkeys ++; server.stat_expiredKeys ++; server.stat_expiredKeys ++; server.stat_expiredKeys ++; // Propagate expiration information to slave nodes and AOF files, propagateExpire(db,key, server.lazyFree_lazy_EXPIRE); NotifyKeyspaceEvent (NOTIFY_EXPIRED, "expired",key,db->id); // notifyKeyspaceEvent(NOTIFY_EXPIRED, "expired",key,db->id); Return server.lazyfree_lazy_EXPIRE? Return server.lazyfree_lazy_expire? dbAsyncDelete(db,key) : dbSyncDelete(db,key); }Copy the codesupplement
We usually say that Redis is single-threaded. In fact, Redis handles network sending and receiving and executing commands in the main thread, but there are other background threads working in Redis. These background threads are usually engaged in IO heavy work, such as disk brushing and other operations.
Lazyfreelazyexpire (introduced in version 4.0) is used to determine whether to implement lazy deletion. The principle is to logically delete expired objects first, and then perform real physical deletion in the background. In this way, the object volume is too large to cause blocking. After the in-depth study of Redis lazyfree principle source location lazyfree. C /dbAsyncDelete method
Periodically delete
The periodic policy deletes expired keys every once in a while and limits the duration and frequency of the deletion operation to reduce the impact on CPU time and memory waste
Redis does expiration scanning 10 times per second by default (as configured in Redis.conf through Hz). Instead of traversing all keys in the expiration dictionary, the following method is used
- Randomly extract 20 keys from an out-of-date dictionary
- Delete expired keys from the 20 keys
- If the percentage of expired keys exceeds 25%, repeat steps 1 and 2
In order to ensure that there will not be excessive circulation in the scan, which will cause the thread to freeze, the upper limit of the scan time is increased. The default is 25 ms (default is in slow mode, if in fast mode, the upper limit of the scan time is 1 ms).
Corresponding source expire. C/activeExpireCycle method
void activeExpireCycle(int type) { ... do { ... If (num > ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP) // The number of expired keys is 20. Num = ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP; while (num--) { dictEntry *de; long long ttl; // Randomly select 20 expiration keys if ((de = dictGetRandomKey(db->expires)) == NULL) break; . / / try to delete expired keys if (activeExpireCycleTryExpire (db, DE, now)) expired++; . }... } while (expired > ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP/4);} while (expired > ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP/4); }... }Copy the codesupplement
If the request comes in and the server is scanning the expired key, it needs to wait for 25 milliseconds. If the timeout time set by the client is less than 25 milliseconds, the link will be closed because of timeout, which will cause an exception. These phenomena cannot be detected from slow query logs, because slow query logs only record logical processing and do not include wait time.
Therefore, when setting the expiration time, we must avoid the phenomenon that a large number of keys expire at the same time. Therefore, if this situation occurs, it is better to add a random range to the expiration time to alleviate the phenomenon that a large number of keys expire at the same time, causing the client to wait for a timeout
Redis Expiration key deletion policy
The Redis server uses a combination of lazy deletion and periodic deletion to balance CPU time and avoid memory waste
Processing of expired keys by AOF, RDB, and replication functions
RDB
Generate the RDB file
When a new RDB file is created using the save command or bgSave command, the program checks the keys in the database and the expired keys are not saved to the newly created RDB file
Load the RDB file
Primary server: When the RDB file is loaded, keys are checked and expired keys are ignored
Slave server: When the RDB file is loaded, all keys are loaded. However, the database on the slave server will be emptied during the master/slave synchronization, so the expired key load will not be affected
AOF
AOF file writes
When an expired key is lazily deleted or periodically deleted, the program appends a del command to the AOF file to show that the key has been deleted
AOF rewrite
The restart process checks the key, and if it expires it will not be saved to the overwritten AOF file
copy
The expiration key deletion action of the secondary server is controlled by the primary server
After removing an expired key, the master server explicitly sends a del command to all slave servers to tell slave servers to remove the expired key
When the secondary server receives a read command sent by the client, it will not delete it even if it encounters an expired key. It will delete it only after receiving the del command from the primary server to ensure data consistency between the primary and secondary servers
Doubt?
- What if the primary and secondary servers are disconnected?
- What if network jitter occurs and the del command sent by the primary server is not transmitted to the secondary server?
In fact, the above two problems Redis developers have taken into account, but the master slave copy involves a lot of knowledge, I will simply say the following ideas to solve, and then share a master slave copy file
Question point 1- What if the primary and secondary servers are disconnected?
Redis uses the PSYNC command to perform the synchronization operation during replication. When the secondary server reconnects to the primary server after disconnection, the primary server sends the write commands executed during the disconnection of the secondary server to the secondary server, and then the secondary server receives and executes these write commands, so that the primary and secondary servers can achieve consistency. How does the master determine which commands are required to disconnect from the slave server? The master server maintains a fixed-length fifO queue, known as the replication backlog buffer, which holds write commands from the master server and their offsets. When the master server propagates commands to the slave server, it also writes commands to the replication backlog buffer. The slave server sends the PSYNC command to the master server with the offset of its latest write command, so that the master server can compare the offset to know where the slave server is disconnected from
Then, let’s look at question point 2- what if network jitter occurs and the del command sent by the primary server is not delivered to the secondary server?
In fact, a heartbeat detection mechanism exists between the primary and secondary servers. The primary and secondary servers send and receive the REPLCONF ACK command to check whether the network connection is normal. From the server to the main server sends REPLCONF ACK command, the primary server will compare their offsets and offset from the server, if the offset is less than its own offset from the server, the server will be found in the copy buffer backlog of missing data from the server, and sends the data to the server, Thus data consistency is achieved
summary
This paper mainly analyzes the expiration strategy of Redis is the use of inert deletion and periodic deletion of two strategies to complete, and then a simple look at the source of the two strategies and how to achieve. Finally, it introduces how to deal with expired keys when Redis performs RDB, AOF and master-slave replication operations, especially how to deal with master-slave replication when the master-slave link is disconnected and network jitter command is lost. Hope you can have a harvest after reading it
The resources
Redis Design and Implementation, 2nd edition. Huang Jianhong
Redis Deep Adventure: Core Principles and Deep Combat. Mr. Qian
Welcome to pay attention to the public number [every day white teeth], get the latest article, we communicate together, common progress!