Redis is a memory based efficient key-value non-relational database, access efficiency is very high, and support a variety of storage data structure, is also very simple to use. In this section, we’ll take a look at Python’s Redis operations, focusing on the use of the RedisPy library.

1. Preparation

Before you start, make sure you have the Redis and RedisPy libraries installed. If you want to import/export data, you also need to install RedisDump. If not, refer to Chapter 1.

2. Redis and StrictRedis

The RedisPy library provides two classes Redis and StrictRedis to implement Redis commands.

StrictRedis implements most of the official commands, and the parameters correspond to each other, such as the set() method corresponding to the set method of the Redis command. Redis is a subclass of StrictRedis, and its main function is backward compatibility with several methods in the old library. The lrem() method, for example, swaps the value and num arguments, which are inconsistent with the Redis command line.

StrictRedis is officially recommended, so in this section we will also use the StrictRedis class to demonstrate this.

3. Connect to Redis

Now we have Redis installed locally and running on port 6379 with the password set to Foobared. So, you can connect to Redis and test with the following example:

1
2
3
4
5

from redis import StrictRedis
redis = StrictRedis(host=’localhost’, port=6379, db=0, password=’foobared’)
redis.set(‘name’, ‘Bob’)
print(redis.get(‘name’))

Here we pass in the Redis address, running port, database used, and password information. When not passed by default, the four parameters are localhost, 6379, 0, and None. You declare a StrictRedis object, then call the set() method, set a key-value pair, and get it and print it.

The running results are as follows:

1

b’Bob’

This means we connected successfully and are ready to perform set() and get() operations.

Of course, we can also connect using ConnectionPool as shown in the following example:

1
2
3
4

from redis import StrictRedis, ConnectionPool
pool = ConnectionPool(host=’localhost’, port=6379, db=0, password=’foobared’)
redis = StrictRedis(connection_pool=pool)

The connection is the same. StrictRedis constructs a ConnectionPool with parameters such as host and port, so it is also the same to pass the ConnectionPool directly to StrictRedis.

In addition, ConnectionPool supports building by URL. The URL format supports the following three formats:

1
2
3

redis://[:password]@host:port/db
rediss://[:password]@host:port/db
unix://[:password]@/path/to/socket.sock? db=db

These urls represent Redis TCP connection, Redis TCP+SSL connection, and Redis UNIX Socket connection respectively. We just need to construct one of the above urls, where the password part can be written if there is one, and can be omitted if there is no one. Here is a demonstration of URL connection:

1
2
3

url = ‘redis://:foobared@localhost:6379/0’
pool = ConnectionPool.from_url(url)
redis = StrictRedis(connection_pool=pool)

Here we use the first connection string to connect. First, declare a Redis connection string, then call the from_URL () method to create a ConnectionPool, and then pass it to StrictRedis to complete the connection.

4. Key operation

Table 5-5 summarizes some key judgments and operations.

Table 5-5 Keys and operation methods

methods

role

Parameters that

The sample

example

The sample results

exists(name)

Determine if a key exists

Name: key name

redis.exists('name')

Is there a name key

True

delete(name)

Delete a key

Name: key name

redis.delete('name')

Delete the name key

1

type(name)

Determine key type

Name: key name

redis.type('name')

Determine the key type name

b'string'

keys(pattern)

Gets all the keys that match the rule

Pattern: matching rule

redis.keys('n*')

Gets all keys starting with n

[b'name']

randomkey()

Get a random key

randomkey()

Get a random key

b'name'

rename(src, dst)

Rename key

SRC: the original key name; DST: the new key name

redis.rename('name', 'nickname')

Rename name to nickname

True

dbsize()

Gets the number of keys in the current database

dbsize()

Gets the number of keys in the current database

100

expire(name, time)

Set key expiration time, in seconds

Name: key name; Time: the number of seconds

redis.expire('name', 2)

Set the expiration time for the Name key to 2 seconds

True

ttl(name)

Gets the key expiration time, in seconds. -1 indicates that the key never expires

Name: key name

redis.ttl('name')

Gets the expiration time of the name key

– 1

move(name, db)

Move the key to another database

Name: key name; Db: indicates the database id

move('name', 2)

Move name to database 2

True

flushdb()

Deletes all keys in the currently selected database

flushdb()

Deletes all keys in the currently selected database

True

flushall()

Delete all keys in all databases

flushall()

Delete all keys in all databases

True

5. String operations

Redis supports basic key-value pair storage. Table 5-6 lists the usage of Redis.

Table 5-6 Key-value pair storage format

methods

role

Parameters that

The sample

example

The sample results

set(name, value)

Assign value to string with key name in the database

Name: key name; Value: the value

redis.set('name', 'Bob')

Assign Bob to the value of the name key

True

get(name)

Returns the value of string with key name in the database

Name: key name

redis.get('name')

Return the value of the name key

b'Bob'

getset(name, value)

Assigns a value to a string in the database whose key is name and returns the last value

Name: key name; Value: the new value

redis.getset('name', 'Mike')

Assign name to Mike and get the last value

b'Bob'

mget(keys, *args)

Returns the value of multiple keys

Keys: a list of keys

redis.mget(['name', 'nickname'])

Return the value of name and nickname

[b'Mike', b'Miker']

setnx(name, value)

If the key-value pair does not exist, the value is updated, otherwise unchanged

Name: key name

redis.setnx('newname', 'James')

If the newname key does not exist, set the value to James

The first run is True and the second run is False

setex(name, time, value)

Set the available value to a string value and specify the validity period of this key value

Name: key name; Time: validity period. Value: the value

redis.setex('name', 1, 'James')

Set the name key to James with a validity period of 1 second

True

setrange(name, offset, value)

Sets the substring of the value of the specified key

Name: key name; Offset: the offset; Value: the value

redis.set('name', 'Hello') redis.setrange('name', 6, 'World')

Set name to the Hello string and fill in World at index 6

11. The new string length

mset(mapping)

Batch assignment

Mapping: a dictionary

redis.mset({'name1': 'Durant', 'name2': 'James'})

Set name1 to Durant and name2 to James

True

msetnx(mapping)

Batch assignment is performed when none of the keys exist

Mapping: a dictionary

redis.msetnx({'name3': 'Smith', 'name4': 'Curry'})

Set name3 and name4 only when they do not exist

True

incr(name, amount=1)

A value value operation with a key of name, which defaults to 1, is created and set to amount if no key exists

Name: key name; Amount: Increased value

redis.incr('age', 1)

The value of age increases by 1. If it does not exist, it is created and set to 1

1 is the modified value

decr(name, amount=1)

The value reduction operation with the key name, which defaults to 1, is created if the key does not exist and sets value to -amount

Name: key name; Amount: Reduced value

redis.decr('age', 1)

The value of age is reduced by 1. If it does not exist, it is created and set to -1

-1 is the modified value

append(key, value)

The value of string whose key is name appends value

Key: key name

redis.append('nickname', 'OK')

Append OK to the value of nickname

13 is the modified length of the string

substr(name, start, end=-1)

Returns a substring of string with the key name

Name: key name; Start: indicates the start index. End: terminates the index. The default value is -1, indicating that the index is truncated to the end

redis.substr('name', 1, 4)

Returns the value of name as a string, truncated with indexes 1 to 4

b'ello'

getrange(key, start, end)

Gets the key value substring from start to end

Key: the key name; Start: indicates the start index. End: terminates the index

redis.getrange('name', 1, 4)

Returns the value of name as a string, truncated with indexes 1 to 4

b'ello'

6. List operations

Redis also provides list storage. The elements in a list can be repeated and stored from both ends, as shown in Table 5-7.

Table 5-7 List operations

methods

role

Parameters that

The sample

example

The sample results

rpush(name, *values)

Add an element of value to the end of the list whose key is name

Name: key name; Values: values

redis.rpush('list', 1, 2, 3)

Add 1, 2, 3 to the end of the list whose key is list

3. List size

lpush(name, *values)

Add an element of value to the list header whose key is name

Name: key name; Values: values

redis.lpush('list', 0)

Add 0 to the list header whose key is list

4. List size

llen(name)

Returns the length of the list whose key is name

Name: key name

redis.llen('list')

Returns the length of the list whose key is list

4

lrange(name, start, end)

Returns the elements between start and end in the list with key name

Name: key name; Start: indicates the start index. End: terminates the index

redis.lrange('list', 1, 3)

Returns the list of the index ranges that start with index 1 and end with index 3

[b'3', b'2', b'1']

ltrim(name, start, end)

Intercepts the list with the key name and preserves the contents with the index start to end

Name: key name; Start: indicates the start index. End: terminates the index

ltrim('list', 1, 3)

Reserve elements with indexes 1 through 3 whose key is list

True

lindex(name, index)

Returns the element at index position in the list with key name

Name: key name; Index: an index

redis.lindex('list', 1)

Returns the list index 1 element whose key is list

B ‘2’

lset(name, index, value)

Assign a value to an element at index in a list whose key is name

Name: key name; Index: the position of an index; Value: the value

redis.lset('list', 1, 5)

Assign 5 to the position of index 1 in the list whose key is list

True

lrem(name, count, value)

Delete the element value from the list of count keys

Name: key name; Count: indicates the number of deleted files. Value: the value

redis.lrem('list', 2, 3)

Delete two 3’s from the list with the key list

1 is the number of deletions

lpop(name)

Returns and deletes the first element in the list with the key name

Name: key name

redis.lpop('list')

Returns and deletes the first element in the list named list

b'5'

rpop(name)

Returns and deletes the last element in the list with the key name

Name: key name

redis.rpop('list')

Returns and deletes the last element in the list named list

b'2'

blpop(keys, timeout=0)

Returns and deletes the first element in the list whose name is in keys, or blocks waiting if the list is empty

Keys: list of keys; Timeout: indicates the timeout waiting time. 0 indicates the waiting time

redis.blpop('list')

Returns and deletes the first element in a list whose key is list

[b'5']

brpop(keys, timeout=0)

Returns and deletes the last element in the list with the key name, or blocks if the list is empty

Keys: list of keys; Timeout: indicates the timeout waiting time. 0 indicates the waiting time

redis.brpop('list')

Returns and deletes the last element in the list named list

[b'2']

rpoplpush(src, dst)

Return and remove the last element of the list named SRC and add it to the head of the list named DST

SRC: key of the source list; DST: key of the target list

redis.rpoplpush('list', 'list2')

Remove the end-of-list element with a key of list and add it to the head of the list with a key of list2, then return

b'2'

7. Set operation

Redis also provides collection storage. The elements in a collection are not duplicated. Table 5-8 describes the usage.

Table 5-8 Set operations

methods

role

Parameters that

The sample

example

The sample results

sadd(name, *values)

Adds elements to the collection with key name

Name: key name; Values: the value can be multiple

redis.sadd('tags', 'Book', 'Tea', 'Coffee')

Add Book, Tea, and Coffee to the collection of tags

3 is the number of inserted data

srem(name, *values)

Removes elements from the collection whose key is name

Name: key name; Values: the value can be multiple

redis.srem('tags', 'Book')

Delete Book from the collection with the key tags

1 indicates the number of deleted data

spop(name)

Randomly returns and deletes an element from the collection with key name

Name: key name

redis.spop('tags')

Randomly removes and returns the element from the collection of tags

b'Tea'

smove(src, dst, value)

Remove elements from the SRC corresponding collection and add them to the DST corresponding collection

SRC: source set; DST: target set; Value: indicates the element value

redis.smove('tags', 'tags2', 'Coffee')

Remove the element Coffee from the set with the key tags and add it to the set with the key tags2

True

scard(name)

Returns the number of elements in the collection whose key is name

Name: key name

redis.scard('tags')

Gets the number of elements in the collection whose key is tags

3

sismember(name, value)

Tests whether member is a member of a collection whose key is name

Name: key values

redis.sismember('tags', 'Book')

Check whether Book is a collection element whose key is tags

True

sinter(keys, *args)

Returns the intersection of the set of all given keys

Keys: indicates the list of keys

redis.sinter(['tags', 'tags2'])

Returns the intersection of the set of tags and the set of tags2

{b'Coffee'}

sinterstore(dest, keys, *args)

Find the intersection and save the intersection to the collection of DEST

Dest: Result set; Keys: indicates the list of keys

redis.sinterstore('inttag', ['tags', 'tags2'])

Find the intersection of the set of tags and the set of tags2 and save it as intTag

1

sunion(keys, *args)

Returns the union of all sets of given keys

Keys: indicates the list of keys

redis.sunion(['tags', 'tags2'])

Returns the union of the set of tags and the set of tags2

{b'Coffee', b'Book', b'Pen'}

sunionstore(dest, keys, *args)

Find the union set and save the union to the collection of DEST

Dest: Result set; Keys: indicates the list of keys

redis.sunionstore('inttag', ['tags', 'tags2'])

Find the union of the set of tags and the set of tags2 and save it as intTag

3

sdiff(keys, *args)

Returns the difference set of the set of all given keys

Keys: indicates the list of keys

redis.sdiff(['tags', 'tags2'])

Returns the difference between the set of tags and the set of tags2

{b'Book', b'Pen'}

sdiffstore(dest, keys, *args)

Find the difference set and save the difference set to dest set

Dest: Result set; Keys: indicates the list of keys

redis.sdiffstore('inttag', ['tags', 'tags2'])

Find the difference between the set of tags and the set of tags2 and save it as intTag ‘

3

smembers(name)

Returns all elements of the collection whose key is name

Name: key name

redis.smembers('tags')

Returns all elements of the collection of tags with keys

{b'Pen', b'Book', b'Coffee'}

srandmember(name)

Randomly returns an element in the collection with key name, but does not delete the element

Name: key values

redis.srandmember('tags')

The random return key is an element in the collection of tags

8. Ordered set operation

The ordered set has one more score field than the set, which can be used to sort the data in the set. Its usage is summarized in Table 5-9.

Table 5-9 Ordered set operations

methods

role

Parameters that

The sample

example

The sample results

zadd(name, *args, **kwargs)

Add element member to zset (key = name) and score to sort. If the element exists, its order is updated

Name: key name; Args: variable parameter

redis.zadd('grade', 100, 'Bob', 98, 'Mike')

Add Bob (whose score is 100) and Mike (whose score is 98) to zset with key grade

2, the number of elements added

zrem(name, *values)

Delete elements from zset with key name

Name: key name; Element values:

redis.zrem('grade', 'Mike')

Delete Mike from zset with key grade

1, the number of deleted elements

zincrby(name, value, amount=1)

If the element value already exists in the zset with the key name, the element score is increased by amount. Otherwise, add the element to the collection with the value of score as amount

Name: indicates the key name. Value: element; Amount: Increased score value

redis.zincrby('grade', 'Bob', -2)

Bob’s score in zset with key grade is reduced by 2

98.0, the modified value

zrank(name, value)

Returns the ranking of elements in zset with key name, sorted by score (rank)

Name: key name; Value: indicates the element value

redis.zrank('grade', 'Amy')

Get the rank of Amy in zset with key grade

1

zrevrank(name, value)

Returns the reciprocal ranking (sorted by score) of the elements in the zset with key name

Name: key name; Value: indicates the element value

redis.zrevrank('grade', 'Amy')

Get the reciprocal rank of Amy in zset with key grade

2

zrevrange(name, start, end, withscores=False)

Return all elements in zset with name as index from start to end

Name: key value; Start: starts the index. End: indicates the end index. Withscores: Indicates whether score is included

redis.zrevrange('grade', 0, 3)

Returns the first four elements in zset with key grade

[b'Bob', b'Mike', b'Amy', b'James']

zrangebyscore(name, min, max, start=None, num=None, withscores=False)

Returns the element score in the given interval in zset with key name

Name: key name; Min: lowest score; Max: the highest score; Start: indicates the start index. Num: indicates the number. Withscores: Indicates whether score is included

redis.zrangebyscore('grade', 80, 95)

Returns the element in the zset with key grade between 80 and 95

[b'Bob', b'Mike', b'Amy', b'James']

zcount(name, min, max)

Returns the number of scores in a given interval in zset with key name

Name: key name; Min: lowest score; Max: the highest score

redis.zcount('grade', 80, 95)

Return the number of elements in the zset with grade between 80 and 95

2

zcard(name)

Returns the number of elements of zset whose key is name

Name: key name

redis.zcard('grade')

Get the number of elements in zset with key grade

3

zremrangebyrank(name, min, max)

Removes the elements ranked in the given range from zset with key name

Name: key name; Min: lowest rank; Max: the highest rank

redis.zremrangebyrank('grade', 0, 0)

Delete the first element in the zset with the key grade

1, the number of deleted elements

zremrangebyscore(name, min, max)

Delete the element whose score is in the given interval in zset whose key is name

Name: key name; Min: lowest score; Max: the highest score

redis.zremrangebyscore('grade', 80, 90)

Remove elements with scores between 80 and 90

1, the number of deleted elements

9. Hash

Redis also provides a hash table data structure. We can specify the name of a hash table by using name. The table stores each key and value pair.

Table 5-10 Hash operations

methods

role

Parameters that

The sample

example

The sample results

hset(name, key, value)

Add a mapping to the hash table with key name

Name: key name; Key: mapping key name. Value: indicates the mapping key value

hset('price', 'cake', 5)

Add the mapping to the hash table with the key price, cake with the value 5

1 is the number of mappings to be added

hsetnx(name, key, value)

If the mapping key name does not exist, the mapping is added to the hash table with the key name

Name: key name; Key: mapping key name. Value: indicates the mapping key value

hsetnx('price', 'book', 6)

Add the mapping to the hash table with key price, book with value 6

1 is the number of mappings to be added

hget(name, key)

Returns the value of key in the hash table with key name

Name: key name; Key: mapping key name

redis.hget('price', 'cake')

Gets the value of key named cake in the hash table with key price

5

hmget(name, keys, *args)

Returns the values of each key in the hash table with the key name

Name: key name; Keys: list of mapped key names

redis.hmget('price', ['apple', 'orange'])

Gets the values of apple and orange in the hash table with key price

[b'3', b'7']

hmset(name, mapping)

Batch add mappings to the hash table whose key is name

Name: key name; Mapping: mapping dictionary

redis.hmset('price', {'banana': 2, 'pear': 6})

Batch add mappings to hash table with key price

True

hincrby(name, key, amount=1)

Add amount to the mapped value in the hash table with key name

Name: key name; Key: mapping key name. Amount: an increase

redis.hincrby('price', 'apple', 3)

Increment the value of apple in the hash table with key as price by 3

6. The modified value

hexists(name, key)

Check whether there is a key name mapping in the hash table whose key is name

Name: key name; Key: mapping key name

redis.hexists('price', 'banana')

Whether the value banana exists in the hash table with the key price

True

hdel(name, *keys)

In the hash table with key name, delete the key name key mapping

Name: key name; Keys: mapping key name

redis.hdel('price', 'banana')

Remove the mapping with key name banana from the hash table with key name price

True

hlen(name)

Gets the number of mappings from the hash table with key name

Name: key name

redis.hlen('price')

Gets the number of mappings from the hash table with key price

6

hkeys(name)

Gets all mapped key names from the hash table with key name

Name: key name

redis.hkeys('price')

Gets all mapping key names from the hash table with key price

[b'cake', b'book', b'banana', b'pear']

hvals(name)

Gets all mapped key values from the hash table with key name

Name: key name

redis.hvals('price')

Get all mapped key values from the hash table with key price

[b'5', b'6', b'2', b'6']

hgetall(name)

Gets all mapped key-value pairs from the hash table with key name

Name: key name

redis.hgetall('price')

Get all mapped key-value pairs from the hash table with key price

{b'cake': b'5', b'book': b'6', b'orange': b'7', b'pear': b'6'}

10. RedisDump

RedisDump provides powerful Redis data import and export functions, now let’s see how it is used.

First, make sure RedisDump is installed.

RedisDump provides two executable commands: redis-dump for exporting data and redis-load for importing data.

redis-dump

First, you can enter the following command to view all options:

1

redis-dump -h

The running results are as follows:

Usage: redis-dump [global options] COMMAND [command options]     -u, --uri=S                      Redis URI (e.g. redis://hostname[:port])    -d, --database=S                 Redis database (e.g. -d 15)    -s, --sleep=S                    Sleep for S seconds after dumping (for debugging)    -c, --count=S                    Chunk size (default: 10000)    -f, --filter=S                   Filter selected keys (passed directly to redis' KEYS command) -O, --without_optimizations Disable run time optimizations -V, --version Display version -D, --debug --nosafeCopy the code

-u stands for Redis connection string, -d stands for database code, -s stands for sleep time after export, -c stands for block size, the default value is 10000, -f stands for filter when export, -o stands for disable runtime optimization, -v stands for display version, and -d stands for debugging.

We take the local Redis to do the test, run on port 6379, the password is foobared, and export the command as follows:

1

redis-dump -u :foobared@localhost:6379

If you do not have a password, you can run the following command to omit the password prefix:

1

redis-dump -u localhost:6379

After running, you can output all data from the local database 0 to 15, for example:

1
2
3
4
5
6
7
8

{“db”:0,”key”:”name”,”ttl”:-1,”type”:”string”,”value”:”James”,”size”:5}
{“db”:0,”key”:”name2″,”ttl”:-1,”type”:”string”,”value”:”Durant”,”size”:6}
{“db”:0,”key”:”name3″,”ttl”:-1,”type”:”string”,”value”:”Durant”,”size”:6}
{“db”:0,”key”:”name4″,”ttl”:-1,”type”:”string”,”value”:”HelloWorld”,”size”:10}
{“db”:0,”key”:”name5″,”ttl”:-1,”type”:”string”,”value”:”James”,”size”:5}
{“db”:0,”key”:”name6″,”ttl”:-1,”type”:”string”,”value”:”James”,”size”:5}
{“db”:0,”key”:”age”,”ttl”:-1,”type”:”string”,”value”:”1″,”size”:1}
{“db”:0,”key”:”age2″,”ttl”:-1,”type”:”string”,”value”:”-5″,”size”:2}

Each data contains six fields, including DB (database code), key (key name), TTL (valid time), type (key value), value (content), and size (occupied space).

If you want to output it as a JSON line file, you can use the following command:

1

redis-dump -u :foobared@localhost:6379 > ./redis_data.jl

This will successfully export all data from all Redis databases as JSON row files.

Alternatively, you can specify the export of a database with the -d argument, for example, exporting only the contents of database 1:

1

redis-dump -u :foobared@localhost:6379 -d 1 > ./redis.data.jl

If you only want to export specific content, for example, to export data starting with ADSL, you can add the -f parameter to filter. Run the following command:

1

redis-dump -u :foobared@localhost:6379 -f adsl:* > ./redis.data.jl

The -f parameter is the parameter of the keys command of Redis, which can write some filtering rules.

redis-load

Again, we can start by typing the following command to see all the options available:

1

redis-load -h

The running results are as follows:

redis-load --help  Try: redis-load [global options] COMMAND [command options]     -u, --uri=S                      Redis URI (e.g. redis://hostname[:port])    -d, --database=S                 Redis database (e.g. -d 15)    -s, --sleep=S                    Sleep for S seconds after dumping (for debugging)    -n, --no_check_utf8    -V, --version                    Display version    -D, --debug        --nosafeCopy the code

-u stands for Redis connection string, -d stands for database code, the default is all, -s stands for sleep time after export, -n stands for not detecting UTF-8 encoding, -v stands for displaying version, and -d stands for enabling debugging.

We can import the JSON row file into the Redis database:

1

< redis_data.json redis-load -u :foobared@localhost:6379

This will successfully import the JSON row file into the database.

In addition, the following command can achieve the same effect:

1

cat redis_data.json | redis-load -u :foobared@localhost:6379

In this section, we not only looked at the basic operations of RedisPy on the Redis database, but also demonstrated how RedisDump can import and export data. Due to its convenience and efficiency, we will use Redis to achieve many architectures, such as maintenance agent pool, Cookies pool, ADSL dial-up agent pool, scrapy-Redis distributed architecture, so the operation of Redis needs to be well master.

This resource starting in Cui Qingcai personal blog still find: Python3 tutorial | static find web crawler development practical experience

For more crawler information, please follow my personal wechat official account: Attack Coder

Weixin.qq.com/r/5zsjOyvEZ… (Qr code automatic recognition)