Redis is indispensable to the Internet in the product development regular weapons, its high performance, rich data structure, easy to use, but also because it’s too easy to use, the development of our students no matter what, no matter how big is this data, data, no matter how much data are all in the last problem is caused by Redis memory usage continues to rise, But we don’t know whether the data inside is useful, whether it can be split and cleaned up.

In order to better use Redis, in addition to make some use of Redis specifications, but also need to have a full understanding of online Redis. So the question is: what is stored in an instance of Redis that uses that much memory? What are the keys? How much space does each key take up?

Snowball currently has dozens of Redis clusters, nearly a thousand Redis instances, and 5T of memory data. We also want to analyze whether there is misuse of business to improve resource utilization. Of course, we have been deeply troubled by this problem, today I will share with you how TO solve this problem, I hope to give you some inspiration.

Is there a safe and efficient way to see detailed reports of Redis memory consumption? There are always more ways than problems. Where there is a need, there is a solution. Snowball SRE team implemented a Redis Data visualization platform RDR (Redis Data Reveal) to solve this problem. RDR can be very convenient to analyze the memory of Reids, to know what keys are in a Redis instance, how much space is occupied by which keys, which keys consume the most memory, and what the proportion is. It is very intuitive.

Design ideas

Let’s see if there’s a way to get all the data from Redis. From my point of view, there are probably the following methods. Let’s analyze the advantages and disadvantages of a word:

1. Use the keys * command to get all the keys, and then get all the content based on the keys.

  • Advantages: Can not use the Redis machine’s hard disk, direct network transmission
  • Disadvantages: If the number of keys is very large, the keys command may cause Redis to freeze and affect the business; Need to request Redis many times, resource consumption; Traversing data is too slow

2. Enable aOF and obtain all data from the AOF file.

  • Advantages: No impact on Redis service, completely offline operation, enough security;
  • Disadvantages: Some Redis instances are frequently written, which is not suitable for aOF. Aof files can be very large, slow to transfer, slow to parse, and inefficient.

3. Use bgSave to obtain the RDB file and parse the data.

  • Advantages: mature mechanism, good reliability; Relatively small files, high transmission and parsing efficiency;
  • Disadvantages: BgSave forks child processes, but it can cause the main process to be stuck for a period of time, which may affect services.

After the evaluation of the above methods, we decided to use the low peak period to do BGSave from nodes to obtain RDB files, which is relatively safe and reliable, and can also cover all business Redis cluster. This means that each instance automatically generates an RDB file every day during the off-peak period, and even a one-day delay in reporting data is acceptable.

The RDB file is equivalent to all the data of the Redis instance, and the next step is to generate the report:

  1. Parse the RDB file to obtain the Key and Value contents.
  2. According to the corresponding data structure and content, estimate the memory consumption, etc.
  3. Make statistics and generate reports;

The logic is very simple, so the design is very clear. The data flow diagram is as follows:

Let’s look at the specific implementation. First of all, the language selection. The components developed by Snowball SRE are basically made of Go language, so the language selection is not complicated, and Go is directly used. And then there are those processes.

1. The analytic RDB

Parse RDB on GitHub provides a number of libraries for parsing RDB files. We use the https://github.com/cupcake/rdb.

2. Estimate the memory consumption

What memory usage does a record have?

We know that the Redis implementation has some basic data structures that are used to implement exposed data types: SDS, dict, Intset, ZipMap, AdList, Ziplist, QuickList, Skiplist, etc. You can estimate how much memory is used by a record by figuring out which data structures are used based on the record’s data type, calculating the memory consumption of these basic data structures, plus the memory usage of the data, plus some additional overhead such as expiration time.

However, because Redis has done a lot of optimization, the same data type, the data structure used in different scenarios may be different. For example, List, the older version of Redis decides which structure to use according to the number of List elements. Adlist is used for short items and ziplist is used for long items. The value can be set through list-max-ziplist-entries.

After version 3.2, all use QuickList. The memory usage of different structures is actually different, and we can’t get the specific configuration when calculating, so we use the default configuration to calculate, and the final value is an estimated value, but it can basically reflect the usage situation. If you are interested in the various data structures used by Redis, and want to know about their design and application scenarios, you can search the relevant materials and read the source code of Redis.

Here’s an example of computational memory usage:

Select * from RDB where key = Hello, value = world, type = string, TTL = 1440

  • For the consumption of a dictEntry, Redis DB is a dict, and each pair of kV is an entry;
  • The cost of a robj, a generic data structure used to store different types of values in the same dict. Its full name is Redis Object.
  • SDS consumption for storing keys, which is the data structure used to store strings in Redis;
  • Storage expiration time consumption;
  • SDS consumption for storing value;

The first four items are basically required to store any key, while the last item varies according to the data structure of the value.

  • One dictEntry has two Pointers, one int64 memory consumption;
  • A robj has a pointer to 1, an int, and several fields using bitfields that consume 4 bytes;
  • The expiration time is also stored as a dictEntry, timestamp int64;
  • SDS requires space to store the header and string length +1. The length of the header varies according to the string length.

We can use this information to figure out how much memory we really need to apply to the operating system. Since Redis supports a variety of Malloc algorithms, we will calculate according to the allocation method of Jemalloc, which is also the point of possible error.

So the final record with a key of Hello would consume 92 bytes on a 64-bit operating system.

Other types of calculations follow the same pattern, but different memory consumption is calculated for different data structures. Remember to consider memory alignment when calculating. In addition, since the algorithm of ZSET involves the random generation of layers, we also use the same algorithm to randomize, but the calculated value is definitely not accurate, which is also an error point.

3. Statistics

Now that we can finally get the memory usage of any key, what are the most meaningful and valuable data?

  • Top N, there is no doubt that the largest N keys must be concerned;
  • Key quantity distribution and memory usage of different data types;
  • Based on prefix classification, a unified prefix generally means that a specific service is using it. Calculate the number of keys and memory usage of each category.

These requirements are also easy to implement:

  1. Maintain a small top heap to store the first N largest, and finally remove the data in the heap;
  2. Count;
  3. Usually have a specific delimiters, such as: |. _ characters, such as according to these characters cut out public prefix to statistics, at the same time all the digital replacement is 0, facilitate classification;

4. Report data

It’s nice to be able to open a web page every day and see the memory usage details of an instance of Redis. Redis memory usage is no longer a black box.

Since the system was launched one year ago, it has provided very important data support for us to optimize the use of Redis resources, improve efficiency and save costs. Moreover, it is fully automated internally, so that the developers themselves can see whether the current use of Redis meets expectations and play a very important role in ensuring business stability. This is also the consistent practice of Snowball’s engineer team. SRE provides efficient tools for development engineers to operate and maintain their own business systems, which can greatly improve production efficiency.

This project refers to the open source project redis-rdb-tool, but the performance is several times more efficient than it. In order to give back to the community, we also hope to have the opportunity to help you, so we decided to open source.

Snowball’s internal system makes automatic RDB file acquisition and backup logic according to its own special scenarios. The open source version removes customization and only retains the analysis logic and pages after obtaining RDB.

The project address is github.com/xueqiu/rdr. If you have any questions, please make an issue or email me directly at [email protected].

The authors introduce

Dong Mingxin, snowball SRE engineer, mainly responsible for ensuring snowball stability, improving resource utilization and improving development efficiency.

Thanks to Guo Lei for correcting this article.