Redis advanced part (2) : the underlying implementation of Redis objects

0. What are the five types of objects

The five types of objects are string, list, set, zset, and hash

1. Why have a relationship

We operate redis mainly through the APIS that operate on objects, rather than through its calls to its underlying data structures (appearance mode). But we also need to understand the underlying layers so that we can write optimal and efficient code.

1. Like Java, objects make development easier, simpler and lower barriers to entry. Developers do not need to understand its complex underlying API, directly call the high-level interface to achieve development.
2. Redis determines whether a command is illegal based on the object type. If you set key value1 value2, an error is reported.
3. Object can contain multiple data structures, making the data store more polymorphic. (Speaker)
4. Reids do garbage collection (reference counting) based on objects.
5. Object with richer properties to help Redis achieve more advanced functionality. (for example, the idle time of the object).

2. Redis object (RedisObject) source analysis

Typedef struct redisObject {// Type unsignedtype: 4; // Unsigned encoding:4; // Pointer to the underlying implementation data structure void * PTR; / /... } robj;Copy the code

The type field

Record the object type.

Type

returns the property of this field.

127.0.0.1:6379 >setHello world OK 127.0.0.1:6379 >typeHello string 127.0.0.1:6379> rpush list 12 3 (integer6379 > 3 127.0.0.1) :type list
list
...
Copy the code

How many types of type are there? Take a look at the chart below:

object	The type field	TYPE Output of the TYPE command
String object	REDIS_STRING	“string”
	A list of objects	REDIS_LIST
	The hash object	REDIS_HASH
	A collection of objects	REDIS_SET
	Ordered set object	REDIS_ZSET

Encoding field

Records the encoding (data structure) used by the object, which is called encoding in Reids.

We can look at encoding in our Redis object like this:

127.0.0.1:6379 > object encoding the hello"embstr"127.0.0.1:6379 > object encoding list"quicklist".Copy the code

Since it specifies the data structure used by the redisObject, there must be a corresponding table:

type	coding	object
REDIS_STRING	REDIS_ENCODING_INT	A string object implemented with an integer value.
REDIS_STRING	REDIS_ENCODING_EMBSTR	A string object implemented using a simple dynamic string encoded by EMbstr.
REDIS_STRING	REDIS_ENCODING_RAW	A string object implemented using a simple dynamic string.
REDIS_LIST	REDIS_ENCODING_ZIPLIST	A list object implemented using a compressed list.
REDIS_LIST	REDIS_ENCODING_LINKEDLIST	A list object implemented using a double-ended linked list.
REDIS_HASH	REDIS_ENCODING_ZIPLIST	A hash object implemented using a compressed list.
REDIS_HASH	REDIS_ENCODING_HT	Hash objects implemented using dictionaries.
REDIS_SET	REDIS_ENCODING_INTSET	A collection object implemented using a collection of integers.
REDIS_SET	REDIS_ENCODING_HT	A collection object implemented using a dictionary.
REDIS_ZSET	REDIS_ENCODING_ZIPLIST	An ordered collection object implemented using a compressed list.
REDIS_ZSET	REDIS_ENCODING_SKIPLIST	An ordered collection object implemented using skip lists and dictionaries.

As you can see, Redis has a very detailed encoding for objects, with three strings and four objects each having two different implementations of the underlying data structure. They have a pattern of using ziplist, IntSet, embstr for small amounts of data, and then upgrading to Skiplist, RAW, LinkedList, and HT for large amounts of data, as I’ll explain later.

3. Analyze the underlying coding implementation of each object (data structure)

3.1 String (String)

There are three string encodings: int, RAW, embstr.

3.1.1 int

When string values are all numbers, int encoding is used.

127.0.0.1:6379 >setNumber 123455 OK 127.0.0.1:6379> Object Encoding number"int"
Copy the code

3.1.2 embstr

If a string or floating point number is less than or equal to 39 bytes, it is stored using the EMBSTR encoding. Embstr memory is generally small, so Redis allocates the memory at one time and continuously (efficient).

127.0.0.1:6379 >set shortStr "suwe suwe suwe"
OK
127.0.0.1:6379> object encoding shortStr
"embstr"
Copy the code

3.1.2 raw

When a string or floating-point number is larger than 39 bytes, it is stored using SDS and encoded as RAW. Since the size of the value is uncertain, the key and value are allocated separately, so memory is allocated twice (and the same is true for recycling twice). Similarly, it must not be contiguous.

127.0.0.1:6379 >set longStr "hello everyone, we dont need to sleep around to go aheard! do you think?"
OK
127.0.0.1:6379> object encoding longStr
"raw"
Copy the code

3.1.3 Coding conversion

As mentioned earlier, Redis automatically transforms the code to accommodate and optimize the storage of the data.

int->raw

Condition: Conversion occurs when a numeric object is append letters.

127.0.0.1:6379 > object encoding number"int"127.0.0.1:6379 > append the number" is a lucky number"
(integer) 24
127.0.0.1:6379> object encoding number
"raw"
Copy the code

embstr->raw

Condition: When modifying embstr, Redis will convert it to RAW before modifying it. So embstr is actually read-only.

127.0.0.1:6379 > object encoding shortStr"embstr"127.0.0.1:6379 > append shortStr"(hhh"
(integer) 18
127.0.0.1:6379> object encoding shortStr
"raw"
Copy the code

3.2 list (list)

List object encoding can be ziplist or LinkedList.

1. Zlbytes Zltail Zllen entry1 Entry2… The entry element contains pre-Length, Encoding, and Content attributes.

2. Linkedlist, similar to a two-way linkedlist, is also covered in the previous chapter.

3.2.1 Code Conversion

ziplist->linkedlist

Condition: All string elements of the list object must be at least 64 bytes long & the list element must be at least 512 bytes long. Conversely, less than 64 and less than 512 use Ziplist instead of linkedList.

This threshold can be modified with either list-max-ziplist-value or list-max-ziplist-entriess

3.3 the hash (hash)

Hash objects are encoded as ziplist and Hashtable

3.3.1 Coding conversion

ziplist->hashtable

Condition: Hash all key and value strings are 64 bytes in length and 512 in number

This threshold can also be modified with the options hash-max-ziplist-value and hash-max-ziplist-entriess

3.4. Set (set)

Collection objects are encoded as intSet and hashtable

3.4.1 track intset

1. All elements of a collection object are integers
2. The number of elements in the collection object does not exceed 512

3.4.2 Coding conversion

intset->hashtable

Condition: Not all elements are integers & the number of elements is greater than or equal to 512

3.5. Ordered Sets (ZSet)

Encoding for ordered collections: Ziplist and Skiplist

You may be curious about ziplist entry. Only the content attribute can store data, and the collection is also in the form of key-value.

The first node holds the key, the second holds the value, and so on…

3.5.1 Why are these two codes used

1. If only ziplist to achieve, can not do the elements of the sort, does not support the scope of the search, can do the elements of the fast search.
2. If you just use Skiplist, you won’t be able to do quick lookups, but you will be able to do element sorting and scoping.

3.5.2 Coding conversion

ziplist->skiplist

Condition: number of ordered set elements >= 128 & length of contained elements >= 64

This threshold can also be modified with the options zset-max-ziplist-value and zset-max-ziplist-entriess

4. Recycling

RedisObject has a field:

typedef struct redisObject { // ... // reference count int refcount; / /... } robj;Copy the code

Redis garbage collection uses reference counting (like the JVM), with a variable at the bottom that counts the usage behavior of an object.

• Initialize to 1
• Object is referenced, +1
• Object reference elimination, -1
• Counter ==0, reclaim object

5. Object sharing

5.1 Object Sharing

1. In Redis, two objects whose values are integer and equal are shared by Redis with a reference count of +1
2. Redis automatically generates integer values from 0 to 9999 and shares them in memory.

5.2 Why Object Sharing

To save memory

5.3 Why Are Strings Not Shared

The cost is too high.

It only takes O(1) time complexity to verify integer equality, while it takes O(n) to verify strings.

6. The idle time of the object

Finally, a redisObject has a field that records when the object was last accessed:

typedef struct redisObject {

    // ...

    unsigned lru:22;

    // ...

} robj;
Copy the code

Because this field records when the object was last accessed, it can be used to see how long the object has not been used, in the current time -lru

127.0.0.1:6379> object idletime hello
(integer) 5110
Copy the code

It is also related to the hot data implementation of Redis. If we choose LR algorithm, when the memory exceeds the threshold, objects with a high idle time will be released and the memory will be reclaimed.

---

Follow my official account, feel free to read all my articles.

To view this article, please click on my GitHub address: github.com/fantj2016/j…

---

---