Introduction of Redis

Redis is an open source (BSD licensed), memory-stored data structure server that can be used as a database, cache, and message queue agent. It supports string, hash, list, collection, ordered collection, and other data types. Built-in replication, Lua scripting, LRU recall, transactions, and different levels of disk persistence, as well as high availability through Redis Sentinel and automatic partitioning through Redis Cluster. In the actual development process, caching is involved in some way, and Redis is generally our best choice for distributed caching. Redis is also known as NoSQL (non-relational database), although it can not completely replace the relational database, but it can be used as a good supplement, small series here also corresponding to collate a Redis actual combat learning notes, which tells about a variety of actual combat solutions and detailed code, to share to you.

Redis usage scenarios

As microservices and distribution are widely used, Redis is used in more and more scenarios. Here I list the main scenarios.

Distributed cache

**

  • Distributed cache: In a distributed system architecture, storing caches in memory is obviously inappropriate because caches need to be shared with other machines. This is where Redis comes in, and caches are the most used scenario for Redis.

**

A distributed lock

**

  • Distributed locking: In the case of high concurrency, we need a lock to prevent the dirty data brought by concurrency, Java lock mechanism is obviously not good for inter-process concurrency, at this time can use Redis single thread feature to achieve our distributed.

**

Session Storage/sharing

**

  • Session storage/sharing: Redis can persist sessions to storage to avoid losing user Session information due to machine downtime.

Publish/subscribe

  • Publish/subscribe: Redis also has a publish/subscribe feature that allows you to publish and subscribe messages to a key value. When a key value is published, all clients that subscribe to it will receive the corresponding message. The most obvious use for this feature is as a real-time messaging system.

Task queue

  • Task queues: The combination of Redis lpush+ BRPOP commands can block queues. Producer clients use LRpush to insert elements from the left side of the list, and multiple consumer clients use BRPOP to block elements from the bottom of the list. Multiple clients ensure load balancing and high availability of consumption.

Limit the interface access frequency

  • Rate limiting and interface frequency limiting: For example, the interface that sends SMS verification codes is usually limited to the frequency at which users can obtain verification codes every minute, for example, no more than five times a minute.

**

Of course, Redis is not only used in so many scenarios, there are many unlisted scenarios, such as counting, leaderboards, etc., which shows the power of Redis. But Redis is essentially a database (non-relational), so it’s worth taking a look at the data structures it supports for storage.

Redis data type

As mentioned earlier, Redis supports the storage of five data types: string, hash table, list, set and ordered set. It is important to understand these five data structures. If you understand these five data structures, you will know about a third of the application knowledge of Redis. Let’s break them down.

String (string)

The string data structure is probably the one we use most often. In Redis, string represents a mutable array of bytes. We initialize the contents of the string, we get the length of the string, we get the substring, we overwrite the substring, we append the substring.

Figure 1. The String data structure of Redis

As shown in the figure above, when we initialize a string in Redis, we preallocate redundant space to reduce the frequent allocation of memory. As shown in Figure 1, the actual allocated space capacity is generally higher than the actual string length len. You’ll be familiar with this pattern if you’ve seen the source code for Java’s ArrayList.

List

In Redis, the storage structure of list list is bidirectional linked list, so it can be seen that its random positioning performance is poor and it is more suitable for first-place insertion and deletion. Like arrays in Java, lists in Redis support access by subscript, except that Redis also provides a negative subscript for lists, -1 for the penultimate element, -2 for the penultimate number, and so on. Because of the excellent performance of adding and deleting at the beginning and end of the comprehensive list, we usually use four instructions rPUSH/RPOP/Lpush/LPOP to use the list as a queue.

Figure 2. List type data structure

As shown in the figure above, a contiguous block of memory is used when a list has few elements. This structure is called ziplist, or compressed list. It stores all the elements right next to each other, allocating a contiguous chunk of memory. Quicklist is used when there is a large amount of data. Because ordinary linked lists require too much additional pointer space, it will be a waste of space. For example, the list contains only int data, and the structure requires two additional Pointers, prev and next. So Redis combines linked lists with Ziplist to make quickList. That is, multiple Ziplists using a bidirectional pointer string to use. This not only meets the fast insert and delete performance, but also does not appear too large space redundancy.

Hash table

Hash is similar to the Java HashMap in that it adopts a two-dimensional structure, with the first dimension being an array and the second dimension being a linked list. Hash keys and values are stored in linked lists, whereas arrays store the headers of each list. During retrieval, the hashcode of the key is computed, then the hashcode is used to locate the table header of the list, and the value is iterated over the list. If you are wondering why a linked list is used to store keys and values, you can store keys and values one by one. If two different keys produce the same Hashcode, we need a linked list to store two key-value pairs. This is called a hash collision.

Set

Those of you familiar with Java should know that the internal implementation of a HashSet uses a HashMap, except that all the values point to the same object. The same is true of Redis’s Set structure, which also uses a Hash structure internally, with all values pointing to the same internal value.

Sorted set (sorted set)

Are sometimes called ZSet, is a special data structure in the Redis, in the ordered set we will be given a weight to each element, its internal elements will be sorted by weights, we can through the command queries within a certain range the weight of elements, this feature can say when we make a list of feature is very practical. The underlying implementation uses two data structures, hash and skip list. The function of hash is to associate the element value with the weight score to ensure the uniqueness of the element value. The corresponding score value can be found through the element value. The purpose of the skip list is to sort the element value and get the list of elements according to the range of score.

Use Redis in Spring Boot projects

The preparatory work

Before we can start using Redis in our Spring Boot project, we need to do some preparatory work.

  1. A machine or virtual machine with Redis installed.
  2. A created Spring Boot project.

Add Redis dependencies

Spring Boot has already provided us with a Starter that integrates Redis. We simply add the following code to the pom.xml file. Spring Boot’s Starter provides a lot of convenience in project dependency management.

Listing 1. Adding a Redis dependency

<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
Copy the code

After adding the dependency, we need to configure the address of Redis and other information to use it. Add the following configuration in application.properties.

Listing 2. Configuring Redis in Spring Boot

Host =192.168.142.132 Spring.redis. Port =6379 # Redis database index (default: 0) Spring.redis Redis server connection password (the default is empty) spring. Redis. Password = # connection pool maximum number of connections (use a negative value indicates no limit) spring. Redis. Jedis. Pool. The Max - active = 8 # Connection pool biggest jam waiting time (use a negative value indicates no limit) spring. Redis. Jedis. Pool. The Max - wait = 1 # connecting the biggest free connection pool spring. Redis. Jedis. Pool. The Max - idle = 8 # Connection pool minimum idle connections in the spring. Redis. Jedis. Pool. Min - idle = 0 # connection timeout time (milliseconds) spring. Redis. Timeout = 0Copy the code

Spring Boot’s spring-boot-starter-data-redis provides a highly encapsulated RedisTemplate class for redis operations and categorizes each type of data structure. Encapsulate the same type of operation as an Operation interface. The RedisTemplate defines operations for each of the five data structures, as follows:

  • String manipulation: redisTemplate opsForValue ()
  • Operation Hash: redistemplate.opsForHash ()
  • List: redistemplate.opsForList ()
  • Set: redistemplate.opsForSet ()
  • ZSet: redistemplate.opsforzset ()

But Spring Boot also provides the StringRedisTemplate class for string data, and it is officially recommended that you use this class to manipulate string data. So how is it different from a RedisTemplate?

  1. RedisTemplate is a generic class, whereas StringRedisTemplate can only operate on data with a String key and value, whereas StringRedisTemplate can operate on any type.
  2. The data between the two is not common. A StringRedisTemplate can only manage data in a StringRedisTemplate, and a RedisTemplate can only manage data in a RedisTemplate.

RedisTemplate configuration

In a Spring Boot project, we only need to maintain a RedisTemplate object and a StringRedisTemplate object. So we need to initialize these two objects through a Configuration class and hand them over to the BeanFactory to manage. We in cn. Itweknow. Sbredis. Config package has built a RedisConfig class below, its content is as follows:

Listing 3. Configuration of RedisTemplate and StringRedisTemplate

@Configuration public class RedisConfig { @Bean @ConditionalOnMissingBean(name = "redisTemplate") public RedisTemplate<String, Object> redisTemplate( RedisConnectionFactory redisConnectionFactory) { Jackson2JsonRedisSerializer<Object> jackson2JsonRedisSerializer = new Jackson2JsonRedisSerializer<Object>(Object.class); ObjectMapper om = new ObjectMapper(); om.setVisibility(PropertyAccessor.ALL, JsonAutoDetect.Visibility.ANY); om.enableDefaultTyping(ObjectMapper.DefaultTyping.NON_FINAL); jackson2JsonRedisSerializer.setObjectMapper(om); RedisTemplate<String, Object> template = new RedisTemplate<String, Object>(); template.setConnectionFactory(redisConnectionFactory); template.setKeySerializer(jackson2JsonRedisSerializer); template.setValueSerializer(jackson2JsonRedisSerializer); template.setHashKeySerializer(jackson2JsonRedisSerializer); template.setHashValueSerializer(jackson2JsonRedisSerializer); template.afterPropertiesSet(); return template; } @Bean @ConditionalOnMissingBean(StringRedisTemplate.class) public StringRedisTemplate stringRedisTemplate( RedisConnectionFactory redisConnectionFactory) { StringRedisTemplate template = new StringRedisTemplate(); template.setConnectionFactory(redisConnectionFactory); return template; }}Copy the code

Operation string

StringRedisTempalte is already initialized above, we just need to inject it with the @AutoWired annotation where we need it.

  1. Void set(K var1, V var2); opsForValue().void set(K var1, V var2);

    @Test public void testSet() { stringRedisTemplate.opsForValue().set(“test-string-value”, “Hello Redis”); }

  2. OpsForValue ().get(Object var1) method to get the value of the specified key.

    @Test public void testGet() { String value = stringRedisTemplate.opsForValue().get(“test-string-value”); System.out.println(value); }

  3. Set the expiration time when setting the value. When setting up the cache, we usually give it an expiration time so that it can refresh periodically. StringRedisTemplate provides a void set(K var1, V var2, Long var3, TimeUnitvar5) to set the expiration time, where var3 is the expiration time. TimeUnit is an enumeration type that we use to set the expiration time in hours, seconds, and so on.

    @Test public void testSetTimeOut() { stringRedisTemplate.opsForValue().set(“test-string-key-time-out”, “Hello Redis”, 3, TimeUnit.HOURS); }

  4. To delete data, we can also delete data with StringRedisTmeplate, which is provided by the Boolean delete(K key) method.

    @Test public void testDeleted() { stringRedisTemplate.delete(“test-string-value”); }

Operating an array

In the Redis data Types section, we mentioned that we often use Redis’s lpush/ Rpush/LPOP/rPOP instructions to implement a queue. These four instructions are also implemented in the RedisTemplate.

  1. LeftPush (K key, V value) inserts an element to the left of a List, as in an array:

    @Test public void testLeftPush() { redisTemplate.opsForList().leftPush(“TestList”, “TestLeftPush”); }

  2. RightPush (K key, V value) to insert an element to the right of the List, such as push elements to the right of the array:

    @Test public void testRightPush() { redisTemplate.opsForList().rightPush(“TestList”, “TestRightPush”); }

  3. After executing the above two tests, we can use the Redis client tool RedisDesktopManager to view the contents of the TestList, as shown below:

When we execute the leftPush method again, the contents of the TestList should look like the following:

You can see that leftPush is actually adding an element to the head of the array, so rightPush is inserting an element to the end of the array.

  1. LeftPop (K key), fetches the first element from the left side of the List and removes, as from the array header:

    @Test public void testLeftPop() { Object leftFirstElement = redisTemplate.opsForList().leftPop(“TestList”); System.out.println(leftFirstElement); }

After executing the above code, you’ll see that the console prints out the TestLeftPush, and then goes to RedisDesktopManager to look at the contents of TestList, as shown below (with an element removed from the top of the array). You will notice that the first element in the array has been removed.

  1. RightPop (K key) to fetch the first element from the right of the List and remove, as from the end of an array:

    @Test public void testRightPop() { Object rightFirstElement = redisTemplate.opsForList().rightPop(“TestList”); System.out.println(rightFirstElement); }

The Hash operation

The Hash data structure in Redis is actually very similar to the HashMap in Java, and the API provided is very similar. Let’s take a look at what apis RedisTemplate provides for hashes.

  1. The Hash element is added.

    @Test public void testPut() { redisTemplate.opsForHash().put(“TestHash”, “FirstElement”, “Hello,Redis hash.”); Assert.assertTrue(redisTemplate.opsForHash().hasKey(“TestHash”, “FirstElement”)); }

  2. Check whether the specified map key exists in the Hash corresponding to the specified key. For the usage, see the preceding code.

  3. Gets the value of the specified key in the Hash corresponding to the specified key.

    @Test public void testGet() { Object element = redisTemplate.opsForHash().get(“TestHash”, “FirstElement”); Assert.assertEquals(“Hello,Redis hash.”, element); }

  4. Delete the key-value pairs of the Hash key corresponding to the Hash key.

    @Test public void testDel() { redisTemplate.opsForHash().delete(“TestHash”, “FirstElement”); Assert.assertFalse(redisTemplate.opsForHash().hasKey(“TestHash”, “FirstElement”)); }

Operating collection

Collections are similar to sets in Java, and RedisTemplate provides a rich API for them.

  1. Adds elements to the collection.

    @Test public void testAdd() { redisTemplate.opsForSet().add(“TestSet”, “e1”, “e2”, “e3”); long size = redisTemplate.opsForSet().size(“TestSet”); Assert.assertEquals(3L, size); }

  2. Gets the element in the collection.

    @Test public void testGet() { Set testSet = redisTemplate.opsForSet().members(“TestSet”); System.out.println(testSet); }

After executing the code above, the console outputs [E1, e3,e2], although of course you may see other results because sets are unordered and not sorted in the order we added them.

  1. Get the length of a collection, shown in the example code for adding elements to a collection.

  2. Removes an element from the collection. ‘

    @Test public void testRemove() { redisTemplate.opsForSet().remove(“TestSet”, “e1”, “e2”); Set testSet = redisTemplate.opsForSet().members(“TestSet”); Assert.assertEquals(“e3”, testSet.toArray()[0]); }

Operatively ordered set

Unlike Set, ZSet maintains a weight value for each element in the collection, so RedisTemplate provides a number of apis related to this weight value.

Add (K key, V value, double score) Adds an element to a variable and specifies the element’s score. Range (K key, long start, long end) gets the elements of the specified range of variables. RangeByLex (K key, rediszsetCommands. Range Range) is used to obtain sorting values that are not score. This sort can only be used if there are the same scores, and if there are different scores the return value is indeterminate. AngeByLex (K key, rediszsetCommands. Range Range, rediszsetCommands. Limit Limit) is used to obtain the length sort value that meets the non-score setting subscript. Add (K key, Set the < ZSetOperations. TypedTuple > tuples) by TypedTuple new data. RangeByScore (K key, double min, double Max) gets the range value based on the set score. RangeByScore (K key, double min, double Max,long offset, long count) gets the final value from the given subscript and length based on the set score. RangeWithScores (K key, long start, long end) gets the interval value of RedisZSetCommands.Tuples.

Implementing distributed locking

The RedisTemplate and StringRedisTemplate classes provide apis for Redis operations, but sometimes these apis do not meet all of our requirements. At this point we can actually interact directly with Redis in the Spring Boot project. For example, when implementing distributed locks, we actually use the RedisTemplate execute method to execute lua scripts to acquire and release locks.

Listing 4. Obtaining the lock

Boolean lockStat = stringRedisTemplate.execute((RedisCallback<Boolean>)connection ->
                    connection.set(key.getBytes(Charset.forName("UTF-8")), value.getBytes(Charset.forName("UTF-8")),
                            Expiration.from(timeout, timeUnit), RedisStringCommands.SetOption.SET_IF_ABSENT));
Copy the code

Listing 5. Releasing the lock

String script = "If redis. Call (' get ', KEYS[1]) == ARGV[1] then return Redis. Call (' del ', KEYS[1]) else return 0 end"; boolean unLockStat = stringRedisTemplate.execute((RedisCallback<Boolean>)connection -> Connection. The eval (script. GetBytes (), ReturnType. BOOLEAN, 1, and the key. The getBytes (Charset. Class.forname (" utf-8 ")), Value. GetBytes (Charset. Class.forname (" utf-8 "))));Copy the code

A few classic questions about Redis

In recent years, Redis has been a hot topic in the interview process. You will be asked about cache and database consistency issues, cache breakdown, cache avalanche, and cache concurrency. So in the last part of the article we will take a look at these questions.

Cache and database consistency problems

For interfaces that have both database operations and cache operations, there are generally two execution orders.

  1. Work with the database first, then the cache. In this case, if the database operation succeeds and the cache operation fails, the cache and database will be inconsistent.
  2. In the second case, the cache operation is performed before the database operation. In this case, if the cache operation succeeds, the database operation failure will also cause the database and cache inconsistency.

In most cases, our caches are theoretically recoverable from the database, so there is basically no problem taking the first order. Caching is generally not recommended for situations where consistency between the database and the cache is essential.

Cache breakdown problem

Cache breakdown indicates that malicious users frequently simulate requests for data that do not exist in the cache. As a result, these requests are directly sent to the data in a short period of time, resulting in a sharp decline in database performance and ultimately affecting the overall performance of the service. This is very easy to encounter in the actual project, such as buying activities, second kill activities of the interface API was a large number of malicious users brush, resulting in a short period of database downtime. There are several solutions to the cache breakdown problem, which are briefly described here.

  1. Use mutex queuing. When fetching data from the cache fails, lock the current interface and release the lock after loading and writing data from the database. If another thread fails to acquire the lock, wait for a while and try again.
  2. Use a Bloom filter. Put all possible data caches into bloom filter, and return quickly when hacker accesses non-existent caches to avoid cache and DB failure.

Cache avalanche problem

A large number of cache failures over a short period of time can also lead to database outages if a large number of requests occur during this period. If the traditional hashing algorithm is used in the data distribution algorithm of Redis cluster, a large number of temporary cache invalidation will occur when adding or removing Redis nodes.

  1. Lock the queue as if it resolves cache penetration.
  2. Set up the backup cache, cache A and cache B, set the timeout period for A, B does not set the timeout period, read cache from A first, A does not read CACHE B, and update cache A and CACHE B.
  3. The consistent hash algorithm is used to calculate the data cache nodes, so that when the number of nodes changes, there will not be a large number of cached data to be migrated.

Cache concurrency issues

This refers to the concurrency problem caused by the simultaneous set value of multiple Redis clients. A more effective solution is to serialize the set operation by placing it in a queue. It must be executed one by one.

The last

Welcome everyone to exchange, like the article remember to pay attention to me like yo, thank you for your support!