Click “Back-end Development Technology” above and select “Set as star”
Quality articles and resources, timely delivery
Like MySQL, Redis is now a must-have technology for backend development. Interview is more frequently asked, even occupy a large proportion. But many friends have not summed up these knowledge points, how to answer? Or seems to say the answer, but did not answer the main point, or a little deeper knowledge found that they have not read, how can you ask this? Be mediocre, and you’ll be remembered as just another mediocre candidate in a thick pile of resumes.
That’s okay. I’m going to give you the answer as well as the question. Because there are many knowledge points and some complicated knowledge points will be involved, I may write the Redis part in a series with a special topic.
First, what is Redis, and what are the application scenarios
Redis is an open source high performance key-value (key-value) in-memory database developed in C language, it is a NoSQL (not only SQL, generally refers to non-relational database) database.
1. Excellent performance, data in memory, read and write speed is very fast, support concurrent 10W QPS.
2. Data persistence is supported. You can save the data in memory to disk and load it upon restart.
Can master slave copy, sentry, high availability.
Scenario: Can be used as database, cache, message middleware, and so on. The specific manifestations are cache, shared Session, message queue and distributed lock.
Why is Redis so fast?
1. Pure memory operation
2. Single process single thread operation, thread safety, avoid frequent context switch
3. Simple data structure, reasonable and efficient
4. Non-blocking I/O multiplexing mechanism, non-blocking I/O
Multi-channel I/O multiplexing model uses the ability of SELECT, poll and epoll to monitor I/O events of multiple streams at the same time. When idle, the current thread will be blocked. When one or more streams have I/O events, it will wake up from the blocking state. The program then polls all the streams (epoll only polls the streams that actually emitted the event) and only polls the ready streams sequentially, which avoids a lot of useless operations.
“Multiplexing” refers to multiple network connections, and “multiplexing” refers to the reuse of the same thread. The use of multiplex I/O multiplexing technology allows a single thread to efficiently process multiple connection requests (minimizing the time consumption of network IO), and Redis in memory data manipulation speed is very fast, that is to say, in-memory operations will not become a bottleneck affecting Redis performance. The above points contribute to the high throughput of Redis.
What data types does Redis have
Redis has five data types.
1.String is the most basic type of Redis. Each Key corresponds to a Value. Key is a String and Value is not only a String but also a number. Several other data structures are built on top of the string type. The String type is binary safe, meaning that Redis strings can contain any data, such as JPG images or serialized objects. A String value can store a maximum of 512 MB. Commonly used in caching, counting, sharing sessions, rate limiting, and so on.
2. A Hash is a collection of key-values. Redis Hash is a mapping of String keys and values, and Hash is especially good for storing objects. Hashes can be used to store user information, such as implementing a shopping cart.
3.List is used to store multiple ordered strings, sorted by insertion order. You can add an element to the top or bottom of the list. Common commands: lpush, rpush, lPOP, rPOP, lrange (get list fragment) and so on. Application scenario: List application scenario is very many, it is also one of the most important data structure of Redis, such as Twitter follow List, fan List can be implemented with the List structure. Data structures: Lists are linked lists that can be used as message queues. Redis provides Push and Pop operations on a List, as well as an API for manipulating a segment, allowing you to query or delete elements of a segment directly. Implementation: Redis List is a two-way linked List, which can support reverse lookup and traversal, more convenient operation, but brings additional memory overhead.
4. A Set is an unordered collection of type String. Collections are implemented via HashTable. The elements in a Set are non-sequential and non-repetitive. Application scenario: A Redis Set provides the same functions as a List, except that the Set is automatically de-duplicated and provides the function to determine whether a member is in a Set. Using the Set of intersection, union, difference and other operations, you can calculate common preferences, all preferences, preferences and other functions of their own.
5. Just like Set, Zset is a Set of String elements, and no duplicate elements are allowed. Sorted Set has a weight parameter Score, and the elements in the Set can be Sorted according to Score. Common commands include zadd, zrange, zrem, and zcard. Usage scenario: The Sorted Set can be Sorted by the user providing an additional priority (score) parameter, and is inserted in order, that is, automatic sorting. When you need an ordered and non-repetitive list of collections, choose the Sorted Set structure. Compared with Set, Sorted Set is associated with a parameter Score with the weight of Double type, so that the elements in the Set can be Sorted in order according to Score. Redis sorts the members of the Set from smallest to largest by Score. Redis Sorted Set internally uses HashMap and skipList to ensure that data is stored and ordered. HashMap is a mapping of members to Score. The skip list stores all the members and sorts according to the Score stored in HashMap. The structure of skip list can obtain relatively high search efficiency and is relatively simple to implement.
(Zset and skip lists are separate concepts)
4. Redis data expiration strategy
Data expiration policy in Redis adopts periodic deletion + lazy deletion policy
Periodic deletion policy: Redis enables a timer to monitor all keys periodically and determine whether the key expires. If the key expires, it will be deleted. This strategy ensures that expired keys will eventually be deleted, but it has serious drawbacks: it can consume CPU resources by iterating through all the data in memory each time, and when the key has expired but the timer is not aroused, the key can still be used during this period.
Lazy deletion policy: When obtaining a key, check whether the key has expired. If so, the key is deleted. There is a drawback to this approach: if the key is never used, it is always in memory, and it is already out of date, wasting a lot of space.
Combines the complementary, the two strategies of natural, some changes have taken place in deletion policy regularly, is not each scanning all of the key, but a random part of key inspection, thus reducing the loss of CPU resources, inert deletion policy complementary for the check to the key, basically meet the requirement of all. But sometimes it is so coincidental that neither is extracted by timer, nor is used, and how does this data disappear from memory? It doesn’t matter, there’s a memory flush mechanism, which comes into play when you run out of memory. Elimination strategies are divided into:
1. If the memory is insufficient to contain new data, an error message is reported during the new write operation. (Default Redis policy)
2. If the memory is insufficient to hold new data, remove the least recently used Key from the Key space. (RECOMMENDED by LRU)
3. If the memory is insufficient to accommodate new data, remove a Key randomly from the Key space.
4. If the memory is insufficient to accommodate new data, remove the least recently used Key from the Key space whose expiration time is set. This is generally used when Redis is used for both caching and persistent storage.
5. If the memory is insufficient to accommodate new data, remove a Key randomly from the Key space whose expiration time is set.
6. If the memory is insufficient to accommodate new data, the Key with an earlier expiration time is removed from the Key space.
Five, Redis LRU specific implementation:
The traditional LRU is to use the stack form, each time the latest use of the stack to the top of the stack, but using the stack form will cause a large number of non-hot data in the select * header data, so it needs to be improved.
Each time Redis fetches a value by key, it updates the LRU field in the value to the current second-level timestamp. The initial implementation algorithm of Redis is very simple. Five keys are randomly extracted from the dict and the smallest LRU field is eliminated. In 3.0, another version of the algorithm was improved. Firstly, all the randomly selected keys were put into a pool (the size of the pool is 16), and the keys in the pool were sorted according to the lRU size. Each subsequent randomly selected keylRU value must be less than the smallest LRU in the pool before the pool is full. After the pool is full, remove the largest LRU key from the pool every time a new key needs to be added. To flush, select a minimum LRU value from the pool and flush it out.
Cache weeding algorithm: LRU (Least recently used) algorithm weeding data based on the historical access records of data. Its core idea is that “if data has been accessed recently, it has a higher chance of being accessed in the future”.
Use Redis to achieve the best posture of distributed lock
When it comes to locking, it must involve two processes: locking and unlocking. First, the implementation of locking.
public static boolean tryGetDistributedLock(Jedis jedis, String lockKey, String requestId, int expireTime) {
String result = jedis.set(lockKey, requestId, SET_IF_NOT_EXIST, SET_WITH_EXPIRE_TIME, expireTime);
if (LOCK_SUCCESS.equals(result)) {
return true;
}
return false;
}
Copy the codeJedis.set (String key, String value, String NXXX, String expx, int time);
1. The first is key. We use key as the lock because key is unique.
2. The second value is “value”, we pass “requestId”, many children may not understand, there is a key as a lock is not enough, why use “value”? The reason is that when we talked about reliability above, the distributed lock must meet the fourth condition to unlock the bell. By assigning value to requestId, we can know which request added the lock, which can be based on when unlocking. The requestId can be generated using the uuID.randomuuid ().toString() method.
3. The third parameter is NXXX. We fill in NX for this parameter, meaning SET IF NOT EXIST, that is, when key does NOT EXIST, we carry out SET operation; If the key already exists, no operation is performed.
Expx = PX; expx = PX; expx = PX;
5. The fifth parameter is time, which corresponds to the fourth parameter and represents the expiration time of the key. In general, the set() method above results in only two results: 1. There is no lock (key does not exist), then the lock is performed, and the lock is set to an expiration date. 2. No operation is performed on an existing lock.
As the careful child will see, our locking code meets the three conditions described in our reliability. First, set() takes an NX argument to ensure that the function will not be called if a key is already present, meaning that only one client can hold the lock, which is mutually exclusive. Second, because we set an expiration time on the lock, even if the lock owner subsequently crashes and does not unlock it, the lock will automatically unlock when it expires (i.e. the key is deleted) without deadlock. Finally, because we assign value to requestId, which represents the identity of the locked client request, we can verify that the client is the same client when it is unlocked. Since we only consider Redis single-machine deployment scenarios, we will not consider fault tolerance for the moment.
Release the lock:Copy the codepublic static boolean releaseDistributedLock(Jedis jedis, String lockKey, String requestId) {
String script = "if redis.call('get', KEYS[1]) == ARGV[1] then return redis.call('del', KEYS[1]) else return 0 end";
Object result = jedis.eval(script, Collections.singletonList(lockKey), Collections.singletonList(requestId));
if (RELEASE_SUCCESS.equals(result)) {
return true;
}
return false;
}
Copy the codeSince unlocking requires two steps, the first step is to obtain the lock, why not delete it directly? To ensure that the lock is created by the local thread, ensure that the value of the lock is the same to prevent the lock from expired and delete the lock of another thread's set by mistake. The second step is to delete the lock.Copy the codeSince this is a two-step operation, it becomes a non-thread-safe operation on atoms. To ensure atomicity of key operations, REdis officially recommends the use of lua script pairs. Get and del are performed directly inside Redis.
Copy the codeTime is limited and there’s too much to cover, so I’ll continue with Redis in the next post. For example, cache avalanche, breakdown, cache penetration problems, as well as Redis and Mysql dual write consistency scheme, persistence, master-slave replication and other problems.