Redis expiration operation and expiration policy

If there is no expiration in Redis, it means that all written keys will remain in Redis as long as they are not deleted actively, and Redis is a memory-based database with very limited memory space.

Overdue operation

Expiration Settings

In Redis, you can set the expiration time in the following four ways:

expire key secondsSet:keyIt expires in n seconds.
pexpire key millisecondsSet:keyExpires after n milliseconds.
expireat key timestampSet:keyExpires after a timestamp (accurate to seconds).
pexpireat key millisecondsTimestampSet:keyExpires after a timestamp (accurate to milliseconds).

You can use the TTL key command (in seconds) or PTTL key (in milliseconds) to see how long the key has to expire.

Redis can use the time command to query the timestamp of the current time (accurate to second).

Several methods in the string that directly manipulate expiration time are listed as follows:

set key value ex seconds: Sets the key-value pair and specifies the expiration time (exact to second).
set key value px milliseconds: Sets the key-value pair and specifies the expiration time (to the millisecond).
setex key seconds valule: Sets the key-value pair and specifies the expiration time (exact to second).

Remove expiration time

Run the persist key command to remove the key value expiration time. -1 indicates that the value will never expire.

Java implements expiration operations

Use Jedis to achieve the operation of Redis, the code:

public class TTLTest {
    public static void main(String[] args) throws InterruptedException {
        // Create a Redis connection
        Jedis jedis = new Jedis("xxx.xxx.xxx.xxx".6379);
        // Set the Redis password (omit this line if there is no password)
        jedis.auth("xxx");
        // Store key-value pairs (never expired by default)
        jedis.set("k"."v");
        // Query TTL (expiration time)
        Long ttl = jedis.ttl("k");
        // Prints the expiration log
        // Expiration time: -1
        System.out.println("Expiration Time:" + ttl);
        // Set the expiration time to 100s
        jedis.expire("k".100);
        // The command is executed after 1s
        Thread.sleep(1000);
        // Prints the expiration log
        TTL=99 after expire
        System.out.println("Execute TTL= after expire" + jedis.ttl("k")); }}Copy the code

For more expiration methods, see the following list:

pexpire(String key, long milliseconds): Set n milliseconds to expire.
expireAt(String key, long unixTime): Set a timestamp to expire (accurate to seconds).
pexpireAt(String key, long millisecondsTimestamp): Set a timestamp to expire in milliseconds.
persist(String key): Removes the expiration time.

public class TTLTest {
    public static void main(String[] args) throws InterruptedException {
        // Create a Redis connection
        Jedis jedis = new Jedis("xxx.xxx.xxx.xxx".6379);
        // Set the Redis password (omit this line if there is no password)
        jedis.auth("xxx");
        // Store key-value pairs (never expired by default)
        jedis.set("k"."v");
        // Query TTL (expiration time)
        Long ttl = jedis.ttl("k");
        // Prints the expiration log
        System.out.println("Expiration Time:" + ttl);
        // Set the expiration time to 100s
        jedis.expire("k".100);
        // The command is executed after 1s
        Thread.sleep(1000);
        // Prints the expiration log
        System.out.println("Execute TTL= after expire" + jedis.ttl("k"));
        // Set n milliseconds to expire
        jedis.pexpire("k".100000);
        // Set a timestamp to expire (accurate to seconds)
        jedis.expireAt("k".1573468990);
        // Set a timestamp to expire (to milliseconds)
        jedis.pexpireAt("k".1573468990000L);
        // Remove the expiration time
        jedis.persist("k"); }}Copy the code

Expiration keys in persistence

Expiration key in RDB

RDB files are divided into two stages, the RDB file generation stage and the loading stage.

1. Generate the RDB file

RDB loading can be divided into the following two situations:

ifRedis 是The mainServer running mode, then loadingRDBThe program will check the keys saved in the file. Expired keys will not be loaded into the database. So the expiration key does not load correctlyRDBThe main server of the file is affected;
ifRedis 是fromServer running mode, then loadingRDBThe file is loaded into the database regardless of whether the key is out of date. butDuring data synchronization between the primary and secondary servers, data on the secondary server will be cleared. So in general, expired key pairs loadRDBThe slave server of the file is also not affected.

Rdb.c: rdbLoad(); rdbLoad(); rdbLoad();

/* Check if the key already expired. This function is used when loading * an RDB file from disk, either at startup, or when an RDB was * received from the master. In the latter case, the master is * responsible for key expiry. If we would expire keys here, The * snapshot taken by the master may not be reflected on the slave. * * If the server is the master, * then the keys are no longer associated with the database when they are expired */
if (server.masterhost == NULL&& expiretime ! =- 1 && expiretime < now) {
    decrRefCount(key);
    decrRefCount(val);
    / / to skip
    continue;
}
Copy the code

Expiration key in AOF

1. Write to the AOF file

When Redis persists in AOF mode, if an expired key has not been removed from the database, the AOF file retains the expired key. When the expired key is removed, Redis appends a DEL command to the AOF file to explicitly remove the key value.

2. AOF rewrite

When AOF overwriting is performed, key value pairs in Redis are checked and expired keys are not saved to the overwritten AOF file, so there is no effect on AOF overwriting.

Expired key for master/slave libraries

When Redis is running in master-slave mode, the slave library does not do expiration scanning, and the slave library is passive in dealing with expiration. That is, even if the key in the slave library has expired, if a client accesses the slave library, the value of the key can still be retrieved and returned as an unexpired key-value pair.

The master library will add a DEL instruction to the AOF file when the key expires and synchronize it to all slave libraries. The slave library will execute this DEL instruction to delete expired keys.

Expiry policies

In Redis, we can set an expiration date for some elements. What does Redis do with the expiration key when it expires?

The expiration key executes the process

Redis knows which key values are out of date because Redis maintains a dictionary of all key values with expiration dates, which we call the expired dictionary.

Expired key source code analysis

The expired key is stored in the redisDb structure, with the source code in the SRC /server.h file (based on Redis 5) :

/* Redis database representation. There are multiple databases identified * by integers from 0 (the default database) up  to the max configured * database. The database number is the 'id' field in the structure. */
typedef struct redisDb {
    dict *dict;                 /* Database key space, where all key pairs */ are stored
    dict *expires;              /* Key expiration time */
    dict *blocking_keys;        /* Keys with clients waiting for data (BLPOP)*/
    dict *ready_keys;           /* Blocked keys that received a PUSH */
    dict *watched_keys;         /* WATCHED keys for MULTI/EXEC CAS */
    int id;                     /* Database ID */
    long long avg_ttl;          /* Average TTL, just for stats */
    list *defrag_later;         /* List of key names to attempt to defrag one by one, gradually. */
} redisDb;
Copy the code

The expired key data structure is shown in the figure below:

Expiry policies

Redis removes expired keys to reduce Redis space footprint, but because Redis is single-line, it is not worth the loss if the deletion operation affects the execution of the main business. Therefore, Redis needs to develop multiple (expired) deletion policies to ensure the performance of the normal execution.

Time to delete

When setting a key value expiration time, a timed event is created, and when the expiration time is reached, the event handler automatically deletes the key.

Advantages: Memory can be freed as quickly as possible.
disadvantagesIn:RedisUnder high load or when a large number of expired keys need to be processed at the same timeRedisThe server is stuck, affecting the execution of main services.

Lazy to delete

Do not delete the expired key. Check whether the key value is expired each time it is obtained from the database. If it is expired, the key value is deleted and NULL is returned.

Advantages: Because the expired key is determined on each access, this policy uses very few system resources.
Disadvantages: The space occupied by the system is not deleted in a timely manner, resulting in reduced space utilization and a certain amount of space waste.

The source code parsing

SRC /db.c file in the expireIfNeeded method, source code is as follows:

int expireIfNeeded(redisDb *db, robj *key) {
    // Determine whether the key is expired
    if(! keyIsExpired(db,key))return 0;
    if(server.masterhost ! =NULL) return 1;
    /* Delete the expiration key */
    // Increase the number of expired keys
    server.stat_expiredkeys++;
    // Propagate the message that the key has expired
    propagateExpire(db,key,server.lazyfree_lazy_expire);
    notifyKeyspaceEvent(NOTIFY_EXPIRED,
        "expired",key,db->id);
    // server.lazyfree_lazy_EXPIRE 1 indicates asynchronous deletion (lazy space release) or synchronous deletion otherwise
    return server.lazyfree_lazy_expire ? dbAsyncDelete(db,key) :
                                         dbSyncDelete(db,key);
}
// Determine whether the key is expired
int keyIsExpired(redisDb *db, robj *key) {
    mstime_t when = getExpire(db,key);
    if (when < 0) return 0; /* No expire for this key */
    /* Don't expire anything while loading. It will be done later. */
    if (server.loading) return 0;
    mstime_t now = server.lua_caller ? server.lua_time_start : mstime();
    return now > when;
}
// Get the expiration time of the key
long long getExpire(redisDb *db, robj *key) {
    dictEntry *de;
    /* No expire? return ASAP */
    if (dictSize(db->expires) == 0 ||
       (de = dictFind(db->expires,key->ptr)) == NULL) return - 1;
    /* The entry was found in the expire dict, this means it should also * be present in the main dict (safety check). */
    serverAssertWithInfo(NULL,key,dictFind(db->dict,key->ptr) ! =NULL);
    return dictGetSignedIntegerVal(de);
}
Copy the code

Before all read/write commands to the database are executed, the expireIfNeeded method is called to determine whether the key value is expired. If the key value expires, it is deleted from the database. Otherwise, no processing is done.

Periodically delete

Check the database every once in a while and randomly remove expired keys.

By default, Redis performs 10 expiration scans per second. This can be configured through the Redis configuration file redis.conf. The configuration key is Hz.

Note: Redis does not iterate over all the keys in the expired dictionary, but rather performs a random selection and deletes the expired keys.

Periodic Deletion process

Randomly extract 20 keys from an out-of-date dictionary.
Delete expired keys from the 20 keys.
If the overduekeyRepeat step 1.

At the same time, the algorithm also increases the upper limit of scanning time, which is no more than 25ms by default, to ensure that expired scanning does not occur excessive circulation and lead to thread deadlock.

advantages: Reduces deletion pairs by limiting the duration and frequency of deletion operationsRedisIn addition, some expired data can be deleted to reduce invalid space occupied by expired keys.
Disadvantages: Memory cleaning is not as good as scheduled deletion, and no lazy deletion uses less system resources.

The source code parsing

The core source code for periodic deletion is in the activeExpireCycle method in the SRC/expirely. c file, the source code is as follows:

void activeExpireCycle(int type) {
    static unsigned int current_db = 0; /* ID of the database that was last periodically deleted */
    static int timelimit_exit = 0;      /* Time limit hit in previous call? * /
    static long long last_fast_cycle = 0; /* The last time a quick periodic deletion was performed */
    int j, iteration = 0;
    int dbs_per_call = CRON_DBS_PER_CALL; // The number of databases iterated over each periodic deletion
    long long start = ustime(), timelimit, elapsed;
    if (clientsArePaused()) return;
    if (type == ACTIVE_EXPIRE_CYCLE_FAST) {
        if(! timelimit_exit)return;
        // ACTIVE_EXPIRE_CYCLE_FAST_DURATION Specifies the execution duration of rapid periodic deletion
        if (start < last_fast_cycle + ACTIVE_EXPIRE_CYCLE_FAST_DURATION*2) return;
        last_fast_cycle = start;
    }
    if (dbs_per_call > server.dbnum || timelimit_exit)
        dbs_per_call = server.dbnum;
    // Slow Indicates the execution duration of periodic deletion
    timelimit = 1000000*ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC/server.hz/100;
    timelimit_exit = 0;
    if (timelimit <= 0) timelimit = 1;
    if (type == ACTIVE_EXPIRE_CYCLE_FAST)
        timelimit = ACTIVE_EXPIRE_CYCLE_FAST_DURATION; /* The execution duration of the delete operation */
    long total_sampled = 0;
    long total_expired = 0;
    for (j = 0; j < dbs_per_call && timelimit_exit == 0; j++) {
        int expired;
        redisDb *db = server.db+(current_db % server.dbnum);
        current_db++;
        do {
            / /...
            expired = 0;
            ttl_sum = 0;
            ttl_samples = 0;
            // The number of keys checked in each database
            if (num > ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP)
                num = ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP;
            // Select num randomly from the database
            while (num--) {
                dictEntry *de;
                long long ttl;
                if ((de = dictGetRandomKey(db->expires)) == NULL) break;
                ttl = dictGetSignedInteger
                // Check the expiration and delete the expiration key
                if (activeExpireCycleTryExpire(db,de,now)) expired++;
                if (ttl > 0) {
                    /* We want the average TTL of keys yet not expired. */
                    ttl_sum += ttl;
                    ttl_samples++;
                }
                total_sampled++;
            }
            total_expired += expired;
            if (ttl_samples) {
                long long avg_ttl = ttl_sum/ttl_samples;
                if (db->avg_ttl == 0) db->avg_ttl = avg_ttl;
                db->avg_ttl = (db->avg_ttl/50) *49 + (avg_ttl/50);
            }
            if ((iteration & 0xf) = =0) { /* check once every 16 iterations. */
                elapsed = ustime()-start;
                if (elapsed > timelimit) {
                    timelimit_exit = 1;
                    server.stat_expired_time_cap_reached_count++;
                    break; }}/* Remove only ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP/4 expiration keys */ per check
        } while (expired > ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP/4);
    }
    / /...
}
Copy the code

The activeExpireCycle method iterates through each database for multiple times at a specified time, checks the expiration time of some expired keys randomly from the expiration dictionary, and deletes expired keys.

This function has two execution modes, one fast and one slow, represented by the timelimit variable in the code, which is used to constrain the running time of this function. In fast mode, the value of timelimit is fixed and equal to the predefined constant ACTIVE_EXPIRE_CYCLE_FAST_DURATION. In slow mode, The value of this variable is calculated with 1000000 * ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC/server.hz/100.

Expiration policy used by Redis

Redis uses an expiration strategy of lazy deletion plus periodic deletion.

reference

Redis Core Principles and Combat

Redis expiration operation and expiration policy

Overdue operation

Expiration Settings

Remove expiration time

Java implements expiration operations

Expiration keys in persistence

Expiration key in RDB

Expiration key in AOF

Expired key for master/slave libraries

Expiry policies

The expiration key executes the process

Expired key source code analysis

Expiry policies

Time to delete

Lazy to delete

Periodically delete

Expiration policy used by Redis

reference

Related Posts

Demo4: Builds a RESTful interface application to access the mysql database on the Springboot framework

Java8 (2) Lambda expressions hand – by – hand learning

The most clear and understandable Go WaitGroup source code analysis