This is article 19 of the Stick to technical Writing Project (including translation). Set a small goal of 999, at least 2 articles per week.

If there are useless keys in redis that do not have TTL set due to historical reasons, waste will be caused. This article mainly describes how to batch modify the TTL of Redis and delete keys using wildcards without blocking redis.

Wildcard deletes the key

redis-cli [-a password] [-h localhost] [-p 6379] --scan --pattern pattern* | xargs redis-cli [-a password] [-h localhost] [-p 6379] del
Copy the code

[] is optional

  • -p port
  • -h indicates the redis host
  • – is a password
  • Pattern * is a wildcard character

The SCAN,SSCAN,HSCAN, and ZSCAN commands all support incremental iteration and only return a small number of elements each time they are executed. Therefore, these commands can be used in production environments. You don’t have the same problems as KEYS or SMEMBERS, which can block the server for several seconds when KEYS is used to process a large database or SMEMBERS is used to process a large collection key.

Resources The redis command SCAN

Batch printing or modifying TTL

use

$ pip install redis
$ python keys.py --help

usage: keys.py [-h] [-p PORT] [-d DB_LIST] [--host HOST] [--password PASSWORD]
               [--expire EXPIRE] [--random_upper RANDOM_UPPER]
               [--max_ttl MAX_TTL] [--prefix PREFIX]

optional arguments:
  -h, --help            show this help message and exit/ help -p PORT PORT of redis/PORT number of redis-d DB_LIST            ex : -d all / -d1,2,3,4 / db index --host host ex: --host 127.0.0.1 / redis host default host --password password ex: --password password/redis Password is empty by default -- Expire expire Unit: SEC,ex 1 days = 86400 SEC: --expire 86400 / Set expiration time, default 0, expire immediately --random_upper random_upper Unit: SEC,ex 1 mins = 60 SEC: -- max_TTL max_TTL unit: SEC,ex 1 mins = 60 SEC: Expire +random(0,random_upper) --max_ttl 60 / maximum expire time allowed. Expire +random(0,random_upper) --prefix user* / Wildcards only apply to certain keysCopy the code

# encoding: utf-8
"" Author: [email protected] time: 2018/4/26 4:34 PM func: Obtain the key in the database that has no TTL set author: [email protected] time: 2019/05/10 8:19am desc: CLI add --host and --password ""
import redis
import argparse
import time
import sys, os
import random

class ShowProcess:
    
    The class that displays the processing progress can display the processing progress by calling the related functions of the class.
    i = 0 # Current processing progress
    max_steps = 0 # Total number of processes required
    max_arrow = 50 The length of the progress bar

    To initialize the function, we need to know the total number of processing times
    def __init__(self, max_steps):
        self.max_steps = max_steps
        self.i = 0

    # display function displays progress according to the current processing progress I
    # effect for [> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >] 100.00%
    def show_process(self, i = None):
        if i is not None:
            self.i = i
        else:
            self.i += 1
        num_arrow = int(self.i * self.max_arrow / self.max_steps) # count how many '>' to display
        num_line = self.max_arrow - num_arrow # calculate how many '-' to display
        percent = self.i * 100.0 / self.max_steps # Calculate the completion progress in xx. Xx % format
        process_bar = '[' + '>' * num_arrow + ' ' * num_line + '] '\
                      + '%.2f' % percent + The '%' + '\r' # string with output, '\r' means return to leftmost without newline
        sys.stdout.write(process_bar) # These two sentences print characters to the terminal
        sys.stdout.flush()

    def close(self, words='done'):
        print(words)
        self.i = 0


def check_ttl(redis_conn, dbindex,max_ttl,random_upper,expire,prefix):
    start_time = time.time()
    changed_ttl_num = 0
    keys_num = redis_conn.dbsize()
    print( "there are {num} keys in db {index} ".format(num=keys_num, index=dbindex))
    process_bar = ShowProcess(keys_num)
    for key in redis_conn.scan_iter(match=prefix,count=1000):
        process_bar.show_process()
        ttl = redis_conn.ttl(key)
        if ttl > max_ttl or ttl == - 1:
            changed_ttl_num += 1
            redis_conn.expire(key, expire + random.randint(0, random_upper))
        else:
            continue

    process_bar.close()
    print("cost time(s):", time.time() - start_time)
    print("changed ttl keys number:", changed_ttl_num)


def main(a):
    parser = argparse.ArgumentParser()
    parser.add_argument('-p', type=int, dest='port', action='store', default=6379,help='port of redis ')
    parser.add_argument('-d', type=str, dest='db_list', action='store', default='0',
                        help='-d all / -d 1,2,3,4')
    parser.add_argument('--host', type=str, dest='host', action='store', default='127.0.0.1',
                        help='ex: --host 127.0.0.1 ')
    parser.add_argument('--password', type=str, dest='password', action='store', default=' ',
                        help='ex : --password password ')
    parser.add_argument('--expire', type=int, dest='expire', action='store', default='0',
                        help='unit: sec ,ex 1 days = 86400 sec: --expire 86400 ')
    parser.add_argument('--random_upper', type=int, dest='random_upper', action='store', default='60',
                        help='unit: sec ,ex 1 mins = 60 sec: --random_upper 60 ')
    parser.add_argument('--max_ttl', type=int, dest='max_ttl', action='store', default='60',
                        help='unit: sec ,ex 1 mins = 60 sec: --max_ttl 60 ')
    parser.add_argument('--prefix', type=str, dest='prefix', action='store', default=The '*',
                        help='prefix of key: --prefix user* ')
    args = parser.parse_args()
    port = args.port
    expire = args.expire
    random_upper = args.random_upper
    max_ttl = args.max_ttl
    prefix = args.prefix
    
    if args.db_list == 'all':
        db_list = [i for i in range(0.16)]
    else:
        db_list = [int(i) for i in args.db_list.split(', ')]
    host = args.host
    password = args.password
    for index in db_list:
        try:
            pool = redis.ConnectionPool(host=host, port=port, db=index,password=password)
            r = redis.StrictRedis(connection_pool=pool)
        except redis.exceptions.ConnectionError as e:
            print(e)
        else:
            check_ttl(r, index,max_ttl,random_upper,expire,prefix)


if __name__ == '__main__':
    main()


Copy the code

Get the key script with no TTL set

Want ads

Friends in Jinan, Shandong, welcome to join us and do things together. Long-term recruitment, Java programmer, big data engineer, operation and maintenance engineer, front-end engineer.