This document is shared with Huawei cloud community Redis Key Expiration Policy By JavaEdge.

1. Set the key with the expiration time

Expire key seconds setex(String key, int seconds, String value)Copy the code

With the exception of the string’s exclusive expire method, all types rely on the expire method to set the expiration time if:

  • If the time is not set, the cache will never expire

  • Set an expiration time, but then want the cache to never expire, use persist

Set the expiration time of the key. After the timeout, the key is automatically deleted. In Redis terminology, the associated timeout for a key is volatile.

The lifetime can be removed by using the DEL command to delete the entire key, or overwrite by the SET and GETSET commands. This means that if a command only alters the value of a key with a lifetime rather than replacing it with a new key, the lifetime will not be changed. Such operations as incrementing the key with INCR, performing LPUSH to push the new value into the list, or changing the hash field with HSET keep the timeout constant.

  • usePERSISTThe timeout command clears it, making it permanentkey
  • ifkey 被 RENAMECommand modification, the related timeout period will be transferred to the newkey
  • ifkey 被 RENAMECommand modification, such as existingKey_AAnd then callRENAME Key_B Key_ACommand, then regardless of the originalKey_AWhether it’s permanent or set to timeout, will be determinedKey_BOverrides the validity period status

Note that calling EXPIRE/PEXPIRE with a non-positive timeout or EXPIREAT/PEXPIREAT with a past time will cause the key to be deleted rather than expired (therefore, the key event emitted will be del, not expired).

1.1 Refreshing the expiration time

An EXPIRE operation on a key that already has an expiration time will update its expiration time. There are many applications that have this business scenario, such as a session that records sessions.

1.2 Differences of 2.1.3 before Redis

Prior to Redis version 2.1.3, changing a key with an expired set using the command to change its value had the effect of deleting the key completely. This semantics is required because of limitations in the replication layer now being repaired.

EXPIRE returns 0 and does not change the timeout for keys with a timeout set.

1.3 the return value

  • 1If the expiration time is set successfully.
  • 0If thekeyDoes not exist or cannot set expiration time.

1.4 the sample

Suppose you have a Web service that is interested in the last N pages a user visited, such that each adjacent page view is no more than 60 seconds after the previous page. Conceptually, this set of page views can be thought of as a user’s navigation session that might contain interesting information about the products he or she is currently looking for so that you can recommend related products.

This pattern can be easily modeled in Redis using the following strategy: Each time a user executes a page view, you invoke the following command:

MULTI
RPUSH pagewviews.user:<userid> http://.....
EXPIRE pagewviews.user:<userid> 60
EXEC
Copy the code

If the user is idle for more than 60 seconds, the key is removed and only subsequent page views whose differences are less than 60 seconds are logged. This schema is easily modified to use INCR instead of RPUSH lists.

1.5 Key with expiration time

Typically, the Redis key is created with no associated lifetime. The key persists unless the user explicitly removes it (such as the DEL command).

Commands in the EXPIRE family can associate expired items with a given key at the cost of extra memory used by the key. When a key has an expiration set, Redis ensures that the key is deleted after the specified time has passed. Live critical times can be updated or completely removed using EXPIRE and PERSIST commands (or other strict commands).

1.6 Expiration Accuracy

In Redis 2.4, expiration may be inaccurate and may be between 0 and 1 second. Due to Redis 2.6, the expiration error ranges from 0 to 1 ms.

1.7 Expiration and Persistence

The key store for expiration information is an absolute Unix timestamp (milliseconds in Redis 2.6 or later). This means that even when the Redis instance is not active, time is flowing.

For expiration to work well, computer time must be stabilized. If you move an RDB file from two computers with large desync in the clock, interesting things can happen (such as loading all expired keys at load time).

Even run-time instances always check the computer clock, for example, if you set a key to 1000 seconds and then set the computer time to 2000 seconds in the future, the key will expire immediately instead of lasting 1000 seconds.

2. Redis key expiration policy

  • Passive – Lazy delete
  • Active – Delete regularly

In order to ensure the high performance of Redis, there is no single thread dedicated to deleting.

2.1 Lazy Deletion

Do not delete the key when it expires. Each time you obtain the key, check whether the key is expired. If it expires, it is deleted and null is returned.

2.1.1 advantages

The deletion operation only takes place when a key is obtained and only the current key is deleted. Therefore, the deletion operation consumes less CPU time. At this point, you have to delete the key, because if you do not delete it, you will get the expired key.

When the key is queried, Redis lazily checks to see if it is deleted. This is similar to Spring’s lazy initialization.

2.1.2 shortcomings

But this is not enough because there is an expired key that will never be accessed again. If a large number of keys are not obtained for a long time after the TTL is exceeded, memory leaks may occur (garbage occupies a large amount of memory).

These keys are due to expire anyway, so periodically Redis also needs to test a few keys randomly between keys with expiration sets. All expired keys are removed from the key space.

Time to delete

When you set the expiration time of a key, you can create a timer for the key and delete the key when the expiration time is up.

advantages

Ensure that the memory is released as soon as possible.

disadvantages

  • If there are a lot of expired keys, deleting these keys will take up a lot of CPU time. In the case of tight CPU time, the CPU cannot spend all of its time doing important things, but also needs to spend time deleting these keys
  • Timer creation time. If a timer is created for each key whose expiration time is set (a large number of timers are generated), performance is seriously affected

So no one uses it.

2.2 Deleting files Periodically

Delete expired keys at intervals.

advantages

  • Reduce the CPU footprint of delete operations by limiting the length and frequency of delete operations – Deal with the disadvantage of “timed delete”
  • Periodically delete expired keys – Deal with the downside of “lazy deletion”

disadvantages

  • On the memory-friendly side, it’s not as good as timed deletion.
  • In terms of CPU time friendly, it’s better to “lazy delete”

The difficulties in

  • Properly set the execution duration (how long each deletion takes) and the execution frequency (how often each deletion is performed) (this depends on the server running condition)

10 times per second:

  1. Test 20 random keys with expiration dates
  2. Delete all expired keys found
  3. If more than 25% of the keys have expired, start again from Step 1

This is a trivial probability algorithm, assuming the sample is the entire key space, and continues to expire until the percentage of keys that are likely to expire is less than 25%. This means that the maximum number of expired keys using memory at any given moment is equal to the maximum number of write operations per second divided by 4.

Periodic Deletion process

void databasesCron(void) { /* Expire keys by random sampling. Not required for slaves * as master will synthesize DELs for us. */ if (server.active_expire_enabled) { if (iAmMaster()) { activeExpireCycle(ACTIVE_EXPIRE_CYCLE_SLOW); } else { expireSlaveKeys(); }}Copy the code
#define ACTIVE_EXPIRE_CYCLE_KEYS_PER_LOOP 20 /* Keys for each DB loop. */ void activeExpireCycle(int type) { config_keys_per_loop = ACTIVE_EXPIRE_CYCLE_KEYS_PER_LOOP + ACTIVE_EXPIRE_CYCLE_KEYS_PER_LOOP/4*effort, // step1 for (j = 0; j < dbs_per_call && timelimit_exit == 0; J++) {/* if there is nothing to expire, try the next database as soon as possible */ / step2 if ((num = dictSize(db->expires)) == 0) {db->avg_ttl = 0; // step3 break; }}}Copy the code

Delete randomly ≤ a specified number of expiration K for each of the specified N libraries.

  1. Iterate through each database (number of “databases” configured in redis.conf, default 16)
  2. Check for a specified number of keys in the current library (by default, 20 keys are checked per library, which is equivalent to executing the loop 20 times, as described below)
  3. If the current library does not have a K set TTL, the next library is traversed directly
  4. Randomly obtain a TTL set K, check whether it expires, if the expiration, delete
  5. Check whether the periodic deletion has reached the specified period. If so, exit the periodic deletion

The periodic delete program has a global variable current_db that records a library to be traversed. By default, there are 16 libraries, and if the current_db is traversed 10 times, then current_DB will be 11. The next periodic deletion will be traversed from the 11th library, assuming current_DB equals 15, and then from library 0 (where current_db==0).

Expiration strategy adopted by Redis

Lazy delete + delete periodically.

Lazy delete process

When performing operations such as get or setnx, check whether the key is expired:

  • If the key expires, delete it and perform related operations
  • If no, perform the corresponding operation directly

RDB processes expired keys

An expired key has no impact on the RDB:

  • Persist data from an in-memory database to an RDB file

    Before persisting a key, the system checks whether it has expired. Expired keys are not entered into the RDB file

  • Restore data from an RDB file to an in-memory database

    Before data is loaded into the database, K will be expired. If it expires, it will not be imported into the database (main database).

AOF handles expired K

Expired keys have no effect on AOF.

Persist from in-memory database to AOF file

  • When a key expires and has not been deleted, it is persisted (the key does not enter the AOF file because there is no modification command).
  • When a key expires, a del command is appended to the aOF file when a deletion occurs. (The expired key will be deleted when data is recovered from the AOF file in the future.)

AOF rewrite

When overwriting, the system checks whether the key is expired. Expired keys will not be overwritten to aOF files

How expiration is handled in replication links and AOF files

In order to get the correct behavior without sacrificing consistency, when the key expires, the DEL operation synthesizes and acquires all additional slave nodes simultaneously in the AOF file. In this way, the expired process is concentrated in the master node without the possibility of consistency errors.

However, while the slave nodes connected to the master will not expire keys independently (but will wait for dels from the master), they will still use the full state of the existing expired keys in the data set, so when the slave is selected as master, it will be able to expire keys independently and act completely as master.

By default, each Redis server has 16 databases. By default, database 0 is used. All operations are performed on database 0.

Set the number of databases. Mysql > select * from DB 1; mysql > select * from DB 1; Since database 0 is used by default, all cache operations performed on database 0 are stored on database 0. If you want to move a cache from database 0 to database 1, You can use "Move Key 1" databases 16Copy the code
  • Memcached only uses lazy deletion, whereas Redis uses both lazy deletion and periodic deletion. This is another difference (and one that redis has over memcached)
  • For lazy deletes, the expiration of the key is checked not only when the key is retrieved, but also when the key is set (eg.setnx key2 value2: This method is similar to memcached’s add method in that if key2 already exists, it returns false and does nothing; If the set key2 does not exist, this method sets the cache key2-value2. If key2 is already in redis, the setnx method will return false. If key2 is not deleted, the setnx method will return false. Expiration check on KEY2)

However, many expired keys, you did not timely check, regular deletion also missed, a large number of expired keys accumulated memory, Redis memory almost exhausted! Therefore, when the memory is full, there needs to be memory elimination mechanism! This is Redis voluntarily delete data!

Click to follow, the first time to learn about Huawei cloud fresh technology ~