No overview

What is NoSQL?

Not Only SQL, a generic term for non-relational databases, as a complement to relational databases.

Why NoSQL?

Disk-based relational databases cannot support high concurrency due to the large number of users in today’s applications. Disk I/O performance is low, data relationship is complex, scalability is poor, and large-scale clustering is not convenient. This led to NoSQL, which uses memory to store data, not relationships.

No characteristic

  • Expansion and scaling (no relationship between data)
  • High performance with large data volume (Redis writes 80,000 times and reads 110,000 times per second)
  • Flexible data model (no need to design database tables beforehand)
  • High availability

There are four types of NoSQL

KV key-value pairs

  • Redis
  • Tair
  • Memecache

Document database

  • MongoDB
  • CouchDB

Column storage database

  • HBase
  • Distributed file systems: HDFS, GFS, and OSS

Graph relational database

  • Neo4j
  • InfoGrid

The current era is 3V + 3 high

The 3V in the era of big data: It mainly describes problems

  1. Mass Volume
  2. Multiple segments
  3. Real-time Velocity

The three high points in the era of big data: mainly the requirements for procedures

  1. High concurrency
  2. High scalability
  3. A high performance

NoSQL and APPLICATION scenarios of SQL

  • Basic product information (name, price, and manufacturer) —- MySQL Relational database
  • Additional product information (description, details, comments) —- MongoDB document database
  • Product Image —- HDFS Distributed file system
  • Search for —- ElasticSearch and Solr
  • Hot information (high frequency, band nature) —- Redis, Tair KV key pair
  • Circle of friends social network —- Neo4j, InfoGrid graph relational database

Summary of Redis

Redis (REmote DIctionary Server) is an open source and high-performance key-value database developed in C language. It supports network, memory and persistent log. Redis periodically writes the updated data to disk or the modified operation to the appended record file, and implements master-slave synchronization on this basis.

Redis features

  • There is no necessary relationship between the data
  • Internal use of single thread
  • High performance (50 concurrent execution of 100,000 requests at 110,000 reads per second and 80,000 writes per second)
  • Multiple data types: String, List, set, Hash, zset, geo, Hyperloglog, bitmap
  • persistence
  • The cluster
  • The transaction

What can Redis do?

  • Accelerate the query of hotspot data (main scenario), such as hotspot products, hotspot news, hotspot information, and promotion information
  • Timeliness information control, such as verification code control, voting control, etc
  • Distributed data sharing, such as session separation in distributed cluster architectures
  • A distributed lock
  • Message queue (publish and subscribe), such as kill, buying, ticket queue, etc
  • Instant information query, such as ranking, all kinds of website visit statistics, bus arrival information, online number of information (chat room, website), equipment signal, etc
  • The cache

List of Redis solutions

  1. Controls the primary key IDS of the database table and provides a generation policy for the primary key of the database table to ensure the uniqueness of the primary key of the database table
  2. Control the life cycle of data, control the business behavior by data failure, applicable to all operations with time-limited control
  3. Used for all kinds of structural and non-structural high temperature data access acceleration
  4. Applied to shopping cart data storage design
  5. It is applied to the data storage design of business such as panic buying, limited purchase, limited coupon, activation code and so on
  6. Applied to data control with sequence of operations
  7. Apply to the latest news display
  8. It is applied to random recommendation information retrieval, such as hot song list recommendation, hot news recommendation, hot tourist routes, APP recommendation, big V recommendation, etc
  9. Applied to the same kind of information association search, two degree association search, deep association search
  10. It is used to merge and get intersection of non-repeating data of the same type
  11. Used for fast deduplication of the same type of data
  12. Apply to service control based on blacklist and whitelist Settings
  13. Apply to the ranking corresponding to the counter combination sorting function
  14. It is used to manage the execution sequence of scheduled tasks or task expiration
  15. Used for timely task/message queue execution management
  16. Service control applied to pay-per-view settlement
  17. Applies to time-sequential data manipulation, regardless of time

Why is Redis single thread so fast?

Redis is based on memory operation, CPU is not the performance bottleneck of Redis, the bottleneck of Redis is based on the memory and network bandwidth of the machine, since it can be achieved with a single thread, then use a single thread. Multithreaded CPU contexts switch, and that’s time consuming! For memory systems, it is most efficient if there is no context switch! Multiple reads and writes are all on the same CPU. In memory, this is the best solution!

Redis download and install

The Windows version

The Windows version has been off for longer

  1. Download the installation package: Github-Redis
  2. Decompress the installation package, which contains:
  • Redis-server. exe Command to start the server
  • Redis -cli.exe command line client
  • Redis.windows. conf core configuration file
  • Redis-benchmark. exe performance test tool
  • Redis-check-off. exe Aof file repair tool
  • Redis-check-dump. exe RDB file check tool (snapshot persistent file)
  1. Double-click redis-server.exe to run the service
  2. Use redis-cli.exe to connect to Redis

Linux version

  1. Download the installation package: wget download. Redis. IO/releases/re…
  2. Decompress the installation package: tar -xvf redis-? .? .? .tar.gz
  3. Access the Redis root directory
  4. Basic environment installation
  • yum install gcc
  • yum install gcc-c++
  1. Compiler: make
  2. Installation: make install. The default installation path is /usr/local/bin
  3. Copy the configuration file redis. Conf from the decompressed folder to the installation path
  4. Change daemonize in redis.conf to yes
  5. Start: redis-server redis.conf

Pressure test

Redis comes with a pressure test tool: Redis-Benchmark.

Redis-benchmark [option] [option value] Redis-benchmark -c 50-n 100000 # 50 concurrent execution of 100000 requestsCopy the code

Redis command

Redis command is case insensitive, Redis command center. Table name: primary Key name: primary Key value [: field name], for example: Order :id:29437

General command

  • Clear: Clears screen content
  • Quit, exit,

    : Exits the cli client
  • Help Command /@group: View the help documents of all commands in a command or group (command format, function description, group, and version).
  • Select index: Switch database. By default, there are 16 databases and database 0 is used by default
  • Dbsize: Check the DB size
  • Flushdb: flushes the current database
  • Flushall: flushes the entire database
  • Shutdown: Shuts down Redis
  • Ping: Test the connection
  • Time: obtains the current system time

Redis-Key

  • Keys * : View all keys (* matches any number of any symbols; ? Matches an arbitrary symbol; [] Matches a specified symbol.
    • h? Llo matches Hello, HallO, and HXlLO
    • H * LLO matches HLLO and Heeeello
    • H [ae] lLO matches Hello and Hallo, but not Hillo
    • H [^e] lLO matching hallo, HBLLO… But it doesn’t match Hello
    • H [A-B] LLO matches HallO and HBLLO
    • If you want to unmatch a character’s special match (regular expression), you can prefix it with \.
  • Exists key: Checks whether the current key exists
  • Del key: deletes the key
  • Move Key 1: Moves the current key to database 1
  • Expire key 10: Sets the expiration time of the key to 10 seconds
  • Pexpire key milliseconds: Sets the expiration time of the key, in milliseconds
  • Expireat key timestamp: sets the key to expireat a certain time
  • Pexpireat key milliseconds-timestamp: Specifies the time at which the key expires
  • TTL key: Indicates the remaining time of the key. The value is in seconds. -1 indicates that the key is always valid. -2 is expired, deleted, or undefined
  • PTTL key: displays the remaining time of the key in milliseconds. -1 indicates that the key is always valid. -2 is expired, deleted, or undefined
  • Persist Key: Converts the key from time-sensitive to permanent
  • Type Key: View the key type
  • Rename key newkey: Changes the key name
  • Renamenx key newkey: if yes, change it

String

You can use incR as a counter (for example, voting, high concurrency is not possible to operate on the database every time, you can write Redis and periodically update to the database; Such as 12306 grab ticket inventory) or distributed ID, you can cache objects, you can use setnx to do distributed locking.

  • Set Key value: Set the value
  • Get key: obtains the value
  • Append Key Value: appends the string. If the current key does not exist, it is equivalent to set key value
  • Strlen key: Gets the length of the string
  • Incr key: increments by 1, requiring that the value can be parsed as a number, but the storage is still a String
  • Decr key: decates by 1
  • Incrby key step: Specifies the increment of the step
  • Decrby key step: specifies the decrement step
  • Getrange key start end: returns a string in the specified range, including both start and end, with subscripts starting at 0 and the penultimate digit if negative (e.g. -1).
  • Setrange key offset value: replaces the string starting at the specified position (offset is also replaced, and only the length of the value string is replaced)
  • Setex key 10 value: Set the value of the key to expire in 10 seconds (set with expire)
  • Psetex key milliseconds Value: Sets the key value, in milliseconds
  • Setnx key value: Creates the key if it does not exist and returns 1, otherwise returns 0 (set if not exist, often used in distributed locks)
  • mset k1 v1 k2 v2 … : Sets multiple values at the same time
  • mget k1 k2 k3 … : Gets multiple values simultaneously
  • msetnx k1 v1 k2 v2 … Atomic operations either succeed together or fail together
  • Getset key value: Get and then set

List

List in Redis is actually a bidirectional linked List, so it is most efficient to insert or modify values on both sides. Can be used as stack (Lpush Lpop), queue (Lpush Rpop), blocking queue. If all values are removed and the list is empty, the key is automatically deleted. You can also save the first page of data in a paging operation to a List, and load the second and more information through the database.

  • Lpush key elem1 elem2 … : Inserts one or more values into the head of the list (left)
  • Lpop Key: Removes the first element of the list and returns
  • Lrange key start end: Get the specific value from the range! Start and end are included, and the subscript of the string starts at 0. If a negative number is entered, it indicates the number of digits to which the string is penultimate (for example, -1).
  • Rpush key elem1 elem2 … : Inserts one or more values to the end of the list (right)
  • Rpop Key: Removes the last element of the list and returns
  • Lindex key index: Obtains a value in the list by subscript
  • Llen key: Returns the length of the list
  • Lrem key Count elem: removes a specified number of elements from the list
  • Ltrim key start end: Truncate a specified range of elements (start, end included), the list is changed, only the truncated portion remains
  • RpopLpush source destination: Removes the last element of the source and puts it in the header of destination
  • Lset key index elem: Replace the index value in the list with another value. Update operation. An error is reported if there is no list (or index) to update
  • Linsert key before/after target elem: Inserts an element elem before or after an element target in a column. If the target does not exist (or the key does not exist), it fails

Hash

Value is a Map set. More suitable for object storage, especially for frequently changing information.

  • Hset key field value [field value …] : Sets the KV key pair
  • Hget Key field: Gets a field value
  • Hmset key field value [field value …] : set KV key pair, same as Hset
  • Hmget key field [field …] : Gets multiple field values
  • Hgetall key: Obtains all field values
  • Hdel key field [field …] : Deletes the specified field
  • Hlen key: obtains the number of fields
  • Hexists key Field: checks whether the specified field exists
  • Hkeys key: Retrieves all fields
  • Hvals key: Obtains all values
  • Hincrby Key field increment: indicates a specified increment
  • Hsetnx Key field Value: Set and return 1 if it does not exist, 0 otherwise

Set

Values in a Set cannot be repeated and are out of order. Srandmember can be randomly selected for recommending hot songs, hot news, etc. Can be used to save friends information, intersection can achieve common friends, other friends recommend “six degrees of space theory”, food recommendation.

  • Sadd key member1 member2 … Add value:
  • Smembers key: Views all values of a specified set
  • Sismember key member: checks whether a value is in the set. Returns 1 or 0
  • Scard key: Retrieves the number of elements in a set
  • Srem key member1 member2 … : Removes the specified element
  • Srandmember key [count] : select a random element, do not replace the element (does not repeat, and if count is greater than the set length, only return all elements in the set)
  • Spop key [count] : deletes elements randomly
  • Smove Source Destination member: Moves the specified element to another set
  • Sdiff k1 [k2 k3 …] Poor: set
  • Sinter k1 [k2 k3 …] : the intersection
  • Sunion k1 [k2 k3 …] And set:
  • Sdiffstore destination k1 [k2 k3 …] : Evaluates the difference set and stores it in the specified set
  • Sinterstore destination k1 [k2 k3 …] : Evaluates the difference set and stores it in the specified set
  • Sunionstore destination k1 [k2 k3 …] : Evaluates the difference set and stores it in the specified set

ZSet

Add a weight on the basis of Set to sort. The commands are the same as Set except that Set commands start with S and ZSet commands start with Z. It can be used as a kind of ranking, for example: Top ranking, intimacy ranking, chat room activity.

  • Zadd key score member [score member …] Add value:
  • Zrange key start stop [withscores] : view elements in a specified range, in positive order
  • Zrevrange key start stop [withscores] : check the range of elements in reverse order
  • Max [withscores] [limit offset count] : -inf = -infinity, Max = + INF = infinity, both min and Max included
  • Max min [withscores] [limit offset count] : Filter by score, display in reverse order (min can be -inf for -infinity, Max can be + INF for + infinity, min and Max included)
  • Zrem key member [member …] : Deletes the specified element
  • Zremrangebyrank key start stop: delete byrank (both start and stop are deleted)
  • Zremrangebyscore key min Max: delete according to score
  • Zcard key: Obtains the number of elements
  • Zcount key min Max: Gets the number of elements in the specified interval (min can be -inf, Max can be + INF, both min and Max are included)
  • Zrank key member: gets the index (starting from 0, also called rank) of the element
  • Zrevrank Key Member: reverse rank
  • Zscore key member: Get the score of an element
  • Zincrby Key Increment Member: Modifies the score of an element

GeoSpatial

You can calculate geographic information, the distance between two places. Can be used to be “nearby”. There are only six special commands, but the underlying implementation is ZSet, so you can use the ZSet command to manipulate Geo.

  • geoadd key longitude latitude member [longitude latitude member …] : Added geographical location (valid longitude from -180 degrees to 180 degrees, valid latitude from -85.05112878 degrees to 85.05112878 degrees)
  • geopos key member [member …] : Gets the target location
  • Geodist Key Member1 Member2 [m/km/ft/mi] : Check the direct distance between two places (m/km/ft/mi, default m)
  • Georadius Key longitude latitude RADIUS M /km/ft/mi [withcoord] [Withdist] [COUNT COUNT] [ASC /desc] : With the specified location, find the data within the radius (withcoord shows the latitude and longitude, withdist shows the distance, COUNT COUNT only shows the specified amount of data)
  • Georadiusbymember Key Member RADIUS M /km/ft/mi [withcoord] [withdist] [COUNT COUNT] [ASC /desc] : finds other elements around the specified element
  • geohash key member [member …] : Returns an 11-bit GeoHash string for the element that converts the two-dimensional latitude and longitude into a one-dimensional string, the closer the two strings are if they are closer together
  • Zrange Key start end: View elements in a specified range in the map
  • Zrem key member [member …] : Deletes the specified element

HyperLogLog

Cardinality estimation algorithm, counting non-repeating elements, has a certain amount of error. The advantage is that the memory footprint is fixed, requiring only 12KB for 2^64 different elements. Such as calculating the UV of a web page (page views: The traditional way is to use a Set to store the user’s ID, and then count the number of elements in the Set. If the number of users is too large, then the memory will be overwhelmed, but with HyperLogLog, Only 12KB of memory is required (PFadd does not allocate 12KB of memory at once, but increases as the cardinality increases; The storage space occupied by the PFmerge command is 12KB, regardless of the amount of data before the merge), with only 0.81% error, which is negligible for UV statistics. So if error is acceptable, then HyperLogLog is definitely acceptable.

  • PFadd key elem [elem …] : Add elements
  • PFcount key [key …] : Indicates the base number of statistics elements
  • PFmerge destkey sourcekey [sourcekey …] : Merges elements and adds destkey

Bitmap

A bitmap, a data structure that manipulates binary bits to record information, has only 0 and 1 states. For example, there are only two cases of clocking and unclocking, 0 means unclocked and 1 means clocked, so it only needs 365bit ≈ 46byte to record the clocking of a user for a year. Other states with only two states can also use Bitmap.

  • Setbit key offset value: Sets the value of key subscript offset (offset starts from 0).
  • Getbit key offset: obtains the value of the target position
  • Bitcount key [start end] : counts the number of 1s in the specified key
  • bitop operation destkey key [key …] : Performs the submit, union, nor, xOR operation on the specified key by bit and saves the result in destkey. (Operation corresponds to and, OR, not, and xOR.)

The transaction

The transactional nature of Redis is a collection of commands. All commands in a transaction are serialized and executed sequentially during the execution of the transaction. Execute a set of commands once, sequentially, and exclusively! Redis transaction has no isolation level (all commands are in the transaction and are not executed directly. They are executed only when the exec command is initiated, and there will be no dirty read, phantom read, etc.); A single command guarantees atomicity, but transactions do not (atomicity: either they succeed together or they fail together).

  • Multi: enables a transaction
  • set… : Command to join
  • Exec: Executes a transaction
  • Discard: cancels a transaction, multi, before exec
  • watch key [key …] Monitoring:
  • Unwatch: Cancels monitoring

Note:

  1. What if the command format is entered incorrectly when defining a transaction?
    • Syntax error: indicates that the written format of the command is incorrect
    • If the commands contained in the defined transaction have syntax errors, none of the commands in the entire transaction will be executed. Include commands that are syntactically correct.
  2. What if an error occurs during command execution while defining a transaction?
    • Run error: a command is formatted correctly but cannot be executed correctly. For example, incr operation on list
    • The commands that run correctly will be executed, but the commands that run incorrectly will not be executed. Note that the data corresponding to the executed commands will not be rolled back automatically. The programmer needs to roll back the data in the code.

Redis lock

Optimistic locking

  • Pessimistic lock: very pessimistic, think that any time will be a problem, no matter what will be locked!
  • Optimistic lock: Very optimistic, think no problem at any time, so will not lock! For example, when updating data, obtain the version first, perform a series of operations, and finally compare the version when updating data.

Specific steps:

  1. Monitoring a Key
  2. Open the transaction
  3. Business operations
  4. Perform transactions
  5. Cancel the monitoring

As soon as the monitored value changes, the transaction fails and all operations are not executed.

127.0.0.1:6379> set money 100 OK 127.0.0.1:6379> set out 0 OK 127.0.0.1:6379> watch money 127.0.0.1:6379> multi # the transaction completes properly and the data does not change during the transaction. OK 127.0.0.1:6379> decrby money 20 QUEUED 127.0.0.1:6379> incrby out 20 QUEUED 127.0.0.1:6379> exec 1) (integer) 80 2) (INTEGER) 20 127.0.0.1:6379> unwatch OKCopy the code
# another thread changed money before executing. 127.0.0.1:6379> watch money # Monitor money OK 127.0.0.1:6379> multi OK 127.0.0.1:6379> DECRBY money 10 QUEUED 127.0.0.1:6379> INCRBY out 10 QUEUED 127.0.0.1:6379> exec # (nil) 127.0.0.1:6379> unwatch # If change fails, unlock first and continue above steps OKCopy the code

A distributed lock

  • Use the setnx command to set the public lock.
  • If the setnx command returns the setting successfully, the user has the right to perform the next service operation.
  • For return Settings that fail, there is no control, queue or wait.
  • Release the lock through DEL.
  • Improved: If an application acquires a lock and goes down during a business operation, the lock will never be released.
    • You can add timeliness to locks via expire or pEXPIRE
    • Since operations are usually microseconds or milliseconds, the lock time should not be too large. The specific time needs to be confirmed after the service test.
      • For example, the maximum execution time of a lock operation is 127ms, and the minimum execution time is 7ms.
      • Max. execution time (million) Indicates the maximum command execution time and the average network delay time (million)
      • Recommended lock time Settings are as follows: Maximum time x 120% + Average network delay x 110%
      • If the maximum service duration is less than or equal to the average network latency, it is usually two orders of magnitude

The Java connection Redis

  • Java connects to the Redis service
    • Jedis is a Java connection development tool recommended by Redis
    • SpringData Redis
    • Lettuce
  • Visual client
    • Redis Desktop Manager
    • Redis Client
    • Redis Studio

Jedis

Import dependence

<! -- https://mvnrepository.com/artifact/redis.clients/jedis --> <dependency> <groupId>redis.clients</groupId> < artifactId > jedis < / artifactId > < version > 3.3.0 < / version > < / dependency >Copy the code

Tests: Connect to the database, perform an operation, disconnect

import redis.clients.jedis.Jedis; Public class TestPing {public static void main(String[] args) {Jedis Jedis = new Jedis("192.168.2.100", 6379); System.out.println(jedis.ping()); jedis.close(); }}Copy the code

The apis are the same as the commands, none of them changing, and transactions are implemented using Jedis.

import redis.clients.jedis.Jedis; import redis.clients.jedis.Transaction; import java.util.List; public class TestTransaction { public static void main(String[] args) { String m1 = "user:1:money"; String m2 = "user:2:money"; New Thread(() -> {Jedis Jedis = new Jedis("127.0.0.1", 6379); jedis.flushDB(); jedis.set(m1, "1000"); jedis.set(m2, "2000"); System.out.println(jedis.watch(m1, m2)); Transaction multi = jedis.multi(); // After the transaction is started, all operations must be performed through the multi object, otherwise an error is reported // JedisDataException: Cannot use Jedis when in Multi. Please use Transaction or reset jedis state. // jedis.incrBy(m1, 200); multi.incrBy(m1, 200); multi.decrBy(m2, 200); // int i = 1 / 0; Thread.sleep(1000); List<Object> exec = multi.exec(); if (exec ! = null) { exec.forEach(o -> System.out.println(o.getClass().getName() + ", " + o)); } else {system.out.println (" execute failed "); } } catch (Exception e) { e.printStackTrace(); Multi.discard (); } finally { System.out.println(jedis.get(m1)); System.out.println(jedis.get(m2)); jedis.unwatch(); jedis.close(); } }).start(); New Thread(() -> {Jedis Jedis = new Jedis("127.0.0.1", 6379); try { Thread.sleep(500); jedis.set(m1, "3000"); } catch (InterruptedException e) { e.printStackTrace(); } }).start(); }}Copy the code

SpringBoot integrate Redis

After SpringBoot2.x, the original Jedis used is replaced with Lettuce.

  • Jedis: Using direct connection, multiple thread operation, is not safe, if you want to avoid unsafe, use Jedis Pool! More like BIO mode
  • Lettuce: With Netty, instances can be shared between multiple threads, no thread insecurity! Can reduce thread data, more like NIO mode

POM depends on:

<? The XML version = "1.0" encoding = "utf-8"? > < project XMLNS = "http://maven.apache.org/POM/4.0.0" XMLNS: xsi = "http://www.w3.org/2001/XMLSchema-instance" Xsi: schemaLocation = "http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" > < modelVersion > 4.0.0 < / modelVersion > < the parent > < groupId > org. Springframework. Boot < / groupId > The < artifactId > spring - the boot - starter - parent < / artifactId > < version > 2.2.6. RELEASE < / version > < relativePath / > <! -- lookup parent from repository --> </parent> <groupId>com.example</groupId> <artifactId>redis</artifactId> 1.0 the SNAPSHOT < version > < / version > < dependencies > < the dependency > < groupId > org. Springframework. Boot < / groupId > <artifactId>spring-boot-starter-data-redis</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> </dependencies> </project>Copy the code

Configuration file: application.properties

Spring. Redis. Host = 127.0.0.1 spring. Redis. Port = 6379Copy the code

Start class: redisApplication.java

package com.example.redis; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; @SpringBootApplication public class RedisApplication { public static void main(String[] args) { SpringApplication.run(RedisApplication.class, args); }}Copy the code

The test class: RedisApplicationTest. Java

package com.example.redis; import org.junit.Test; import org.junit.runner.RunWith; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.boot.test.context.SpringBootTest; import org.springframework.data.redis.core.RedisTemplate; import org.springframework.test.context.junit4.SpringJUnit4ClassRunner; @SpringBootTest @RunWith(SpringJUnit4ClassRunner.class) public class RedisApplicationTest { @Autowired private RedisTemplate redisTemplate; @test public void Test () {/* * redisTemplate; /* * redisTemplate;  * opsForValue ----> String / Bitmap * opsForList ----> List * opsForHash ----> Hash * opsForSet ----> Set * opsForZSet ----> ZSet * opsForGeo ----> Geo * opsForHyperLogLog ----> HyperLogLog */ String k = "rt:test"; redisTemplate.opsForValue().set(k, "hello,world"); System.out.println(redisTemplate.opsForValue().get(k)); }}Copy the code

Run the test and find in redis-cli:

127.0.0.1:6379> keys *
1) "\xac\xed\x00\x05t\x00\art:test"
Copy the code

This is due to the default JDK serialization method (which requires Serializable poJOs), and the source code below shows that RedisTemplate can be overridden.

@bean@conditionalonmissingbean (name = "redisTemplate") // Can be custom to replace this public redisTemplate <Object, Object> redisTemplate(RedisConnectionFactory RedisConnectionFactory) throws UnknownHostException {// Default redisTemplate Without too much setup, redis objects all need to be serialized! RedisTemplate<Object, Object> template = new RedisTemplate<>(); template.setConnectionFactory(redisConnectionFactory); return template; } @conditionalonmissingbean // Since String is the most commonly used, Therefore, a separate public StringRedisTemplate StringRedisTemplate (RedisConnectionFactory RedisConnectionFactory) throws UnknownHostException { StringRedisTemplate template = new StringRedisTemplate(); template.setConnectionFactory(redisConnectionFactory); return template; }Copy the code

Customize the RedisTemplate and add the Jackson serialization. Note that there is no Jackson dependency in the top POM dependency. You can add the Jackson dependency directly by adding the Web initiator as follows:

<! --<dependency>--> <! --<groupId>org.springframework.boot</groupId>--> <! --<artifactId>spring-boot-starter-web</artifactId>--> <! --</dependency>--> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> The < version > 2.10.3 < / version > < / dependency >Copy the code
package com.example.redis.config; import com.fasterxml.jackson.annotation.JsonAutoDetect; import com.fasterxml.jackson.annotation.PropertyAccessor; import com.fasterxml.jackson.databind.ObjectMapper; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.data.redis.connection.RedisConnectionFactory; import org.springframework.data.redis.core.RedisTemplate; import org.springframework.data.redis.serializer.Jackson2JsonRedisSerializer; import org.springframework.data.redis.serializer.StringRedisSerializer; @Configuration public class RedisConfig { @Bean @SuppressWarnings("all") public RedisTemplate<String, Object> redisTemplate(RedisConnectionFactory factory) {// Use <String, Object> redisTemplate <String, Object> template = new RedisTemplate<String, Object>(); template.setConnectionFactory(factory); / / Json serialization configuration Jackson2JsonRedisSerializer Jackson2JsonRedisSerializer = new Jackson2JsonRedisSerializer (Object. The class); ObjectMapper om = new ObjectMapper(); om.setVisibility(PropertyAccessor.ALL, JsonAutoDetect.Visibility.ANY); om.enableDefaultTyping(ObjectMapper.DefaultTyping.NON_FINAL); jackson2JsonRedisSerializer.setObjectMapper(om); StringRedisSerializer StringRedisSerializer = new StringRedisSerializer(); / / key String serialization method template. SetKeySerializer (stringRedisSerializer); / / the hash key also USES String serialization way template. SetHashKeySerializer (stringRedisSerializer); / / value serialization way by Jackson template. SetValueSerializer (jackson2JsonRedisSerializer); / / the hash value of the serialization way by Jackson template. SetHashValueSerializer (jackson2JsonRedisSerializer); template.afterPropertiesSet(); return template; }}Copy the code

Redis configuration

  1. Configuration file unit Unit is case-insensitive

2. INCLUDES INCLUDES, similar to import in Java

# import and load specified configuration file information for redis instance configuration file with more common configuration, easy maintenance include /path/to/other.confCopy the code
  1. Internet NETWORK
Bind 127.0.0.1 # protected-mode yes # protected mode port 6379 # port setting timeout 0 # Close the connection after the client has no action (set to 0 to disable the function)Copy the code
  1. GENERAL: running mode, log, database
Pidfile /var/run/redis_6379.pid = yes pidfile /var/run/redis_6379.pid Specify the server verbosity level. # This can be one of: # debug (a lot of information, useful for development/testing) # verbose (many rarely useful info, but not a mess like the debug level) # notice (moderately verbose, What you want in production probably) Production environment # Warning (only very important/critical messages are logged) loglevel Notice logfile "" # databases 1 # Number of databases 2 # always-show-logo yes # Check whether the logo is displayedCopy the code
  1. Snapshot SNAPSHOTTING: Redis DataBase (RDB) Settings

Aof, Redis is an in-memory database, if there is no persistence, then the data is lost when power is cut off.

Save 900 1 # If at least 10 keys have been changed within 300 seconds, save 300 10 # Save 60 10000 stop-draft-on-bgsave-error yes # If there is an error during the persistence process, Rdbcompression yes # Rdbchecksum yes # Rdbchecksum Dbfilename dump. RDB # RDB filename dir./ # RDB file directoryCopy the code
  1. REPLICATION REPLICATION: primary and secondary REPLICATION
Configure the master address replicaof 127.0.0.1 6379. Configure the master address replicaof 127.0.0.1 6379Copy the code
  1. SECURITY SECURITY: Sets the password
Requirepass 123456 # Set the password for RedisCopy the code
  1. Client: limits the maximum number of connections
Maxclients 10000 # Set the maximum number of clients that can connect to RedisCopy the code
  1. MEMORY MANAGEMENT: Configures the maximum MEMORY and the expulsion algorithm
Maxmemory <bytes> # redis specifies the maximum memory capacity maxmemory-policy noeviction strategy when memory is at its limit. Volatile - lFU: select the least recently used data and discard # 2. Volatile - lFU: select the least recently used data and discard # 3. Volatile - TTL: select the data that will expire and discard # 4. Allkeys-lru: select the least recently used data. Select the least recently used data. Select the least recently used data. Data exclusion # discard data expulsion OOM # LRU means Least Recently Used # LFU means Least Frequently Used maxmemory-samples 5 # Select the number of data to be deleted each timeCopy the code
  1. AOF Configures APPEND ONLY MODE
Appendonly no # do not enable aof mode by default, persist using RDB mode by default, Appendfilename "appendonly. Aof "# appendfsync always # appendfsync everysec # appendfsync everysec # [root@localhost] [root@localhost] [root@localhost] [root@localhost] [root@localhost] [root@localhost] [root@localhost] [root@localhost]Copy the code

More detailed configurations are explained below.

persistence

Redis is an in-memory database. If you do not save the data in memory to disk, the data in the server will be lost once the server process exits, so Redis provides persistence.

RDB (Redis DataBase)

Redis forks a separate subprocess for persistence, writing data to a temporary file that will be replaced by the last persistent file when the persistence process is complete. During the entire process, the main process does not perform any IO operations, which ensures extremely high performance.

If large-scale data recovery is required and the integrity of data recovery is not very sensitive, RDB is more efficient than AOF. The downside of RDB is that data can be lost after the last persistence.

The default is RDB, and you generally do not need to change this configuration.

RDB is saved in dump. RDB, which is configured in the snapshot of the configuration file (dbfilename specifies the name of the configuration file, dir specifies the directory for storing the configuration file). The production environment backs up this file.

Dump. RDB: dump.

  • If the save rule is met, the RDB is triggered
  • Running the flushall command triggers the RDB
  • Shutdown redis triggers RDB (not with kill or other enforcement measures)
  • Run the save command to manually execute the RDB (the save command will block the current Redis server until the current RDB process is complete, which may cause a long time of blocking. It is not recommended for online environment)
  • Run the BGSave command to manually start the background save operation, but not immediately (the BGSave command is optimized for save blocking. Redis uses bgSave for all RDB operations. The save command can be abandoned.

How to restore Redis from dump. RDB? Dump. RDB file in the configured directory. When redis starts, it will automatically check dump. You can also run the following command to view the directory:

127.0.0.1:6379> config get dir
1) "dir"
2) "/usr/local/bin"
Copy the code

You can use redis-check-rdb to repair dump. RDB files.

  1. RDB is a compact binary file with high storage efficiency
  2. RDB stores redis data snapshots at a certain point in time, which is very suitable for data backup, full replication and other scenarios
  3. RDB can recover data much faster than AOF

Disadvantages:

  1. RDB mode can not achieve real-time persistence no matter it is executing instructions or using configuration, and has a greater possibility of data loss
  2. The BGSave directive forks the child process each time it runs, sacrificing some performance
  3. The RDB file format is not unified among many versions of Redis, which may cause data format incompatibility among services of different versions

AOF (Append Only File)

Each write command is recorded in an independent log, and the commands in the AOF file are executed again when the system restarts to recover data. In contrast to RDB, it can simply be described as the process of changing recorded data to record data generation. The main role of AOF is to solve the real-time of data persistence, which has been the mainstream way of Redis persistence at present.



This command is disabled by default and needs to be manually configured. You only need to change appendOnly to yes to enable aof and restart Redis to take effect.

If there is an error in the aOF file, Redis cannot be started. You can use the following command to fix the file

redis-check-aof --fix appendonly.aof
Copy the code

There are three policies for writing data (appendfsync) :

  1. Always (every time) : Every write operation is synchronized to the AOF file with zero data error and low performance. Therefore, this parameter is not recommended.
  2. Everysec (per second) : Synchronizes instructions in the buffer to AOF files every second. It has high data accuracy and high performance. It is recommended and the default configuration
  3. No (system control) : The operating system controls the synchronization period to AOF files. The whole process is uncontrollable

The file saved by RDB is appendone. aof, which is configured in the aof configuration file (appendfilename configuration filename, dir configuration directory).

AOF rewrite

As commands write to AOF, files get bigger and bigger. To solve this problem, Redis introduced AOF rewriting to reduce file size. AOF file rewrite is the process of converting data in the Redis process into write commands to synchronize to a new AOF file. Simply put, it is to convert the execution results of several commands of the same data into the corresponding instructions of the final result data for recording.

Function:

  1. Reduce disk usage and improve disk utilization
  2. Improves the persistence efficiency, reduces the persistent write time, and improves I/O performance
  3. Reduces the data recovery time and improves the data recovery efficiency

Rewrite rule:

  1. Data that has timed out in the process is no longer written to the file
  2. Ignore invalid instructions and use in-process data to generate directly when rewriting, so that the new AOF file only retains the writing commands of the final data, such as Del Key1, hdel Key2, SREM key3, SET KEY4 111, set KEY4 222, etc
  3. Multiple write commands for the same data can be combined into one command, such as lpush list1 a, lpush list1 b, and lpush list1 c. To prevent client buffer overflow caused by excessive data, a maximum of 64 elements can be written into each instruction of the list, set, hash, and zset types

Redis.conf can be configured to override automatically:

no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
Copy the code

You can also override it manually:

bgrewriteaof
Copy the code

RDB and AOF

RDB VS AOF

Persistent mode RDB AOF
Occupying Storage space Small (data level: compressed) Large (instruction level: rewrite)
Storage speed slow fast
Recovery rate fast slow
Data security You lose data By strategy
Resource consumption High/heavyweight Low/lightweight
Startup priority low high

conclusion

  1. RDB persistence allows you to take snapshots of your data at specified intervals
  2. When the server is restarted, it will execute these commands to restore the original data. The AOF command appends each write operation to the end of the file using the Redis protocol. Redis can also rewrite the AOF file in the background. Keep the size of AOF files from becoming too large.
  3. Do caching only, if you only want your data to live while the server is running, you can also do without any persistence
  4. Enable both persistence methods
    1. In this case, when Redis restarts, AOF files will be loaded first to recover the original data, because AOF files usually hold more complete data sets than RDB files.
    2. RDB data is not real-time, and the server will only look for AOF files when both are used, so should only use AOF? The authors advise against it, as RDB is better for backing up databases (AOF is not easy to back up constantly), restarts quickly, and there are no potential AOF bugs left as a back-up measure.
  5. Performance Suggestions
    1. Since RDB files are only used for backup purposes, it is recommended to persist RDB files only on the Slave and backup them only once every 15 minutes, leaving only the save 9001 rule.
    2. If you Enable AOF, you will lose less than two seconds of data in the worst case. The startup script is simpler and you can just load your own AOF files. The second is the end of AOF rewrite. The blocking caused by writing new data to new files in the rewrite process is almost inevitable. The default base size of AOF rewrite (64M) is too small and can be set to more than 5G. The default base size of AOF rewrite (100%) can be changed to an appropriate value.
    3. If AOF is not enabled, high availability can be achieved by master-slave Repllcation alone, saving a lot of I/O and reducing the amount of system volatility that rewrite might cause. The trade-off is that if the Master/Slave goes down at the same time, the data will be lost for more than ten minutes. The startup script will also compare the RDB files in both Master/Slave and load the newer one.

Release subscription

Redis publish subscription (PUB/SUB) is a message communication model: the sender (PUB) sends the message and the subscriber (sub) receives the message.

The Redis client can subscribe to any number of channels. The following figure shows channel Channel1 and the relationship between the three clients that subscribe to this channel — Client2, Client5, and Client1:



When a new message is sent to channel Channel1 via PUBLISH, the message is sent to the three clients that subscribed to it:

  1. First redis- CLI client, subscribe first
Redis 127.0.0.1:6379> SUBSCRIBE runoobChat Reading Messages... (press Ctrl-C to quit) 1) "subscribe" 2) "redisChat" 3) (integer) 1Copy the code
  1. The second Redis – CLI client, publishes messages
Redis 127.0.0.1:6379> PUBLISH runoobChat "redis PUBLISH test" (integer) 1 redis 127.0.0.1:6379> PUBLISH runoobChat "Learn redis by runoob.com" (integer) 1Copy the code
  1. The first Redis-CLI client receives a message
1) "message"
2) "runoobChat"
3) "Redis PUBLISH test"
1) "message"
2) "runoobChat"
3) "Learn redis by runoob.com"
Copy the code

Maintains a channel dictionary, puts subscribers into the corresponding channel linked list, and notifies all subscribers when a message is sent. Observer mode. Message queues are commonly used, such as RabbitMQ (see RabbitMQ in Depth), RocketMQ, ActiveMQ, Kafka.

A master-slave replication

Primary/secondary replication refers to the replication of data from one Redis server to other Redis servers. The former is called the master node (master/leader) and the latter is called the slave node (slave/follower). The replication of data is one-way and can only go from the master node to the slave node.

The characteristics of

  1. The Master can write data, while the Slave can read data
  2. By default, each Redis server is the primary node
  3. A master node can have multiple slave nodes (or none), but a slave node can only have one master node

Function:

  1. Data redundancy: Master/slave replication implements hot backup of data and is a data redundancy method other than persistence.
  2. Fault recovery: When the primary node is faulty, the secondary node provides services for rapid fault recovery. It’s actually redundancy of services.
  3. Load balancing: On the basis of master/slave replication and read/write separation, the master node provides the write service, and the slave node provides the read service (that is, the application connects to the master node when writing Redis data, and the application connects to the slave node when reading Redis data) to share server load. Especially in the scenario of less write and more read, the concurrency of the Redis server can be greatly increased by sharing the read load with multiple slave nodes.
  4. High availability (clustering) cornerstone: In addition to the above, master-slave replication is the foundation upon which sentry and clustering can be implemented, hence master-slave replication is the foundation of High availability in Redis.
  5. Read/write separation: Master writes data and slave reads data, improving the read/write load of the server.

Generally speaking, to apply Redis to engineering projects, only one Redis is not acceptable (down), for the following reasons:

  1. Structurally, a single Redis server can have a single point of failure, and it is stressful for a single server to handle all the request load.
  2. In terms of capacity, the memory capacity of a single Redis server is limited. Even if the memory capacity of a Redis server is 256G, all the memory cannot be used as Redis memory. Generally speaking, the maximum memory used by a single Redis server should not exceed 20G.

Master/slave replication, read/write separation. Read operations are performed 80% of the time. The read load is spread across multiple servers to reduce the load on the servers. Often used in architecture, at least one master and two slave.

The principle of

Phase 1: Establish the connection phase

Establish a connection from the slave to the master so that the master can identify the slave and save the slave port number

  • Step 1: Set the IP address and port of the master and save the master information
  • Step 2: Establish a socket connection
  • Step 3: Send the ping command (timer task)
  • Step 4: Authentication
  • Step 5: Send the slave port information

At this point, the master-slave connection is successful!

Phase 2: Data synchronization workflow

After the slave connects to the master for the first time, all data in the master is copied to the slave (full replication), and the database status of the slave is updated to the current database status of the master.

  • Step 1: Request data synchronization
  • Step 2: Create an RDB to synchronize data
  • Step 3: Restore RDB synchronization data
  • Step 4: Request partial synchronization of data
  • Step 5: Restore some synchronized data

At this point, data synchronization is complete! Master:

  1. If the master has a large amount of data, avoid traffic peak hours during data synchronization to avoid blocking the master and affecting services
  2. An incorrect size of the replication buffer may cause data overflow. If the full replication period is too long and data is already lost during partial replication, a second full replication must be performed, causing the slave to fall into the dead-loop state.
repl-backlog-size 1mb
Copy the code
  1. The memory usage of the master single machine should not be too high. You are advised to use 50%-70% of the memory of the host, and reserve 30%-50% of the memory for executing the BGsave command and creating the replication buffer

Slave:

  1. To avoid server response congestion or data inconsistency during full or partial replication, you are advised to disable external services during this period
slave-serve-stale-data yes|no
Copy the code
  1. During data synchronization, the master sends messages to the slave. You can understand that the master is a client of the slave and actively sends commands to the slave
  2. If multiple slaves synchronize data requests to the master at the same time, the number of RDB files sent by the master increases, causing a huge impact on the bandwidth. If the bandwidth of the master is insufficient, an appropriate amount of peak error is required for data synchronization based on service requirements
  3. If there are too many slaves, you are advised to change the topology structure from one master to multiple slaves to a tree structure. The nodes in the middle are both masters and slaves. Caution When the tree structure is used, the data synchronization delay between the slave and the master at the top layer is larger due to the depth of the hierarchy, and data consistency deteriorates. Therefore, exercise caution

Phase 3: Command transmission phase

If the status of the master database is changed, the status of the master database is inconsistent with that of the master database. In this case, you need to synchronize data between the master database and the slave database to the consistent status. The synchronization action is called command propagation. The master sends the received data change command to the slave. After receiving the command, the slave executes the command.

Disconnection occurred during command propagation:

  • Intermittent network disconnection: Ignored
  • Short network interruption: Partial replication
  • Long time network interruption: Full replication

Three core elements of partial replication

  • Server run ID (RUN ID)

Server runid (runid) is the identification code of each server run. Multiple runs of a server can generate multiple run ids. It consists of 40 characters and is a random hexadecimal character. Used for identification. The RUNID is automatically generated when each Server starts. When the master connects to the slave for the first time, it sends its RUNID to the slave. The slave saves the ID.

  • Replication backlog buffer for the primary server

The replication buffer, also known as the replication backlog buffer, is a first-in, first-out (FIFO) queue that stores commands executed by the server. Each time a command is propagated, the master records the command propagated and stores it in the replication buffer. The default data storage space of the replication buffer is 1 MB. Because the storage space is fixed, when the number of queued elements is greater than the queue length, the first queued element will be ejected and new elements will be queued. When each server is started, if it has AOF enabled or is connected as a master node, the replication buffer is created.

  • Replication offset of the primary and secondary servers

A number that describes the instruction byte position in the copy buffer. Synchronize information and compare the differences between master and Slave. When the slave is disconnected, data is restored. Master copy offset: Records the position (multiple) corresponding to instruction bytes sent to all slaves. Records are sent once. Slave replication offset: Records the location (one) of the command bytes that the slave receives from the master.

heartbeat

When the command is transmitted, the master and slave exchange information and use the heartbeat mechanism for maintenance to keep the connection online.

  1. Master the heart:
    • Instructions: PING
    • Period: determined by rep-ping-slave-period. The default value is 10 seconds
    • Function: Checks whether the slave is online
    • Query: INFO replication, obtain the interval between the last connection to slave. If the lag item remains 0 or 1, it is considered normal
  2. Slave heart:
    • Instruction: REPLCONF ACK {offset}
    • Period: 1 second
    • Function 1: Reports the replication offset of the slave and obtains the latest data change instructions
    • Function 2: Check whether the master is online

Matters needing attention:

  • When most of the slaves are offline or the latency is too high, the master rejects all information synchronization operations to ensure data stability
Min-rabes-to-write 2 Min-rabes-max-lag 8 # When the number of slaves is less than 2 or the delay of all slaves is greater than or equal to 10 seconds, the master write function is forcibly disabled to stop data synchronizationCopy the code
  • The slave sends the REPLCONF ACK command to confirm the slave quantity
  • Slave Delay The slave sends the REPLCONF ACK command to confirm the delay

Q&A

  1. Frequent full replication –master restart
    • Symptom: As the system runs, the data volume of the master increases. Once the master restarts, the RUNID changes, causing full replication of all slaves.
    • Solution:
      1. Create the master_REPLID variable internally, using the same policy as rUNID, and send it to all slaves.
      2. When master is shutdown, run shutdown save for RDB persistence and save runid and offset to RDB file.
        • Repl – id repl – offset
        • You can run the redis-check-rdb command to view the information
      3. After the master restarts, load the RDB file and restore data. After the restart, load the repl-ID and repl-offset saved in the RDB file into memory.
        • Master_repl_id = repl master_REPL_offset = repl-offset
        • You can view the information by running the info command
    • Function: The local machine saves the last runid and restores the value after restart. All slaves consider the runiD as the previous master.
  2. Frequent full replication – Poor network environment
    • Symptom: The network is interrupted and the slave does not provide services.
    • Cause: The replication buffer is too small. After the network is disconnected, the offset of the slave exceeds the threshold, triggering full replication.
    • End result: The slave repeatedly performs full replication.
    • Solution: Change the replication buffer size repl-backlog-size
    • The recommended Settings are as follows:
      1. Calculates the average reconnection duration from the master to slave second
      2. Write_size_per_second getmaster average amount of write command data per second
      3. Optimal copy buffer space = 2 * second * write_size_per_second
  3. Frequent network disconnection – Master disconnects from slave
    • Symptom: The CPU usage of the master is too high or the slave is frequently disconnected.
    • Cause of the problem:
      1. The slave sends the REPLCONF ACK command to the master every second
      2. When a slave receives a slow query (keys *, hgetall, etc.), it takes up a lot of CPU performance
      3. The master invokes replication timing function replicationCron() every second and finds that the slave does not respond for a long time
    • End result: The master resources (output buffer, bandwidth, connection, etc.) are heavily occupied.
    • Solution: Determine whether to release the slave by setting a proper timeout period (repl-timeout defines the timeout threshold. The default value is 60 seconds. If the timeout period exceeds the threshold, the slave is released.
  4. Frequent network disconnection – The slave and master are disconnected
    • Symptom: The connection between the slave and master is disconnected.
    • Cause of the problem:
      1. The frequency of the master ping command is low
      2. The master setting timeout period is short
      3. The ping command lost packets on the network. Procedure
    • Solution: Increase the frequency of sending ping commands repl-ping-slave-period (the timeout period of repl-timeout is at least 5 to 10 times that of ping commands; otherwise, it is easy for slave to determine the timeout).
  5. Data inconsistency
    • Symptom: Data obtained by multiple slaves is not synchronized.
    • Cause: The network information is not synchronized, and the data transmission is delayed.
    • Solution:
      1. Optimize the network environment between the master and the slave, and deploy them in the same machine room. Pay attention to this phenomenon when using cloud servers such as Ali Cloud.
      2. Monitors the delay of the master and slave nodes (judged by offset). If the delay of the slave is too large, the data access of the slave is temporarily blocked.
Slave - serve - stale - data yes | response only after the no # open a few commands such info, slaveof (careful, unless very high requirements for data consistency)Copy the code

Environment configuration

Configure only slave, not master, because the default is master.

127.0.0.1:6379> info replication # Check the current library information # Replication role:master # Role master Connected_SLAVES :0 # No slave master_repl_offset:0 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0Copy the code
  1. Copy the three configuration files (redis_port.conf) and modify the corresponding information:
    • Port to port
    • daemonize yes
    • Pidfile./redis_ Port. Pid
    • Logfile “redis_port.log”
    • Dbfilename Dump_ Port. RDB
    • Appendfilename “appendonly_ port.aof”
  2. Start three Redis
  3. Slave Connects to the master and is configured on the slave end
    • Method 1: The client sends the slaveof


      command. You need to send the command again after the restart

    • Redis-server-slaveof < masterIP >
    • Replicaof < masterIP >
  4. When the master and slave are disconnected, the slave sends commands
Slaveof no one # slave Does not delete existing data, but does not accept data sent by the masterCopy the code
  1. Authorized access (RequirePass set for master) and configured file on slave
masterauth <password>
Copy the code
  1. If the master goes down and the slave is still in the slave role (it will be reconnected if the master recovers), you can use Slaveof no One to make yourself the host and then manually connect the other nodes to the new master.

Sentinal Sentinel mode

If the master fails, the slave cannot be automatically changed to master. Manual intervention is required and services are unavailable for a period of time. This is not a recommended approach; more often, the sentry mode is preferred. Redis has officially provided Sentinel architecture since 2.8 to solve this problem.

introduce

Sentinel is a distributed system used to monitor each server in the master slave structure. When a failure occurs, a new master is selected by voting mechanism and all slaves are connected to the new master.

role

  • monitoring
    • Constantly check whether the master and slave are working properly
    • Master survival detection, master and slave running status detection
  • Notice (reminder)
    • Sends notifications to others (sentinels, clients) when a monitored server has a problem.
  • Automatic failover
    • Disconnect the master from the slave, select one slave as the master, connect the other slaves to the new master, and inform the client of the new server address
  • Pay attention to
    • Sentinel is also a Redis server, but does not provide data services
    • Sentries are usually configured in an odd number

Multiple sentinel mode

A single sentry process monitoring the Redis server can cause problems, so you can use multiple sentries to monitor. Each sentry is also monitored, creating a multi-sentry mode.

If the primary server is down and Sentry 1 detects the result first, the system does not perform a failover immediately, but sentry 1 subjectively considers that the primary server is unavailableSubjective offline. When the sentries behind also detect that the primary server is unavailable and the number reaches a certain value, a vote is held between sentries. The result of the vote is initiated by one sentry and a failover is performed. After the switch succeeds, it passesRelease subscriptionMode, so that each sentinel to monitor their own secondary server implementation switch host, this process is calledObjective offline.

The sentry configuration

Sentinel changes the profile content in real time during its run.

# Example sentinel. Default 26379 port 26379 # Sentinel working directory dir/TMP # Sentinel monitoring redis master node IP port # master-name Master node name can be named by itself only letters A-z, 0-9, and the three characters ".-_". # Quorum configures how many sentinels uniformly consider the master node to be disconnected, # sentinel monitor <master-name> < IP > <redis-port> <quorum> Sentinel monitor myMaster 127.0.0.1 6379 2 # When requirePass Foobared is enabled in the Redis instance, all clients connecting to the Redis instance must provide the password. Sentinel auth-pass mymaster 123456 sentinel auth-pass mymaster 123456 The primary node does not respond to the sentinel, and the sentinel thinks that the primary node is offline subjectively. Default 30 seconds # sentinel down-after-milliseconds <master-name> <milliseconds> sentinel down-after-milliseconds mymaster 30000 # This configuration item specifies the maximum number of slaves that can synchronize the new master at the same time when a failover occurs. The smaller the number, the longer the time required to complete the failover. This means that more slaves are unavailable due to replication. This value can be set to 1 to ensure that only one slave at a time is unable to process command requests. # sentinel parallel-syncs <master-name> < numSlaves > Sentinel parallel-syncs mymaster 1 It can be used in the following ways: The interval between two failover operations for the same sentinel and the same master. # 2. Start time counting when a slave synchronizes data from an incorrect master. Until the slave is corrected to synchronize data to the correct master. # 3. The time required to cancel an ongoing failover. # 4. Maximum time required to configure all Slaves to point to the new Master when performing failover. # however, slaves will still be configured correctly to point to master even after this timeout, [info] [info] [info] [info] [info] [info] [info] [info] [info Mymaster 180000 # SCRIPTS EXECUTION # Configure the SCRIPTS that need to be executed when an event occurs. You can use SCRIPTS to notify administrators, for example, by sending emails when the system is not working properly. If the script returns 1 after execution, the script will be executed again later. The default number of times is 10. If the script returns 2 after execution, or a value higher than 2, the script will not be executed again. If the script is aborted during execution due to a system interrupt signal, it behaves the same as if the value is 1. The maximum execution time of a script is 60 seconds. If this time is exceeded, the script will be terminated by a SIGKILL signal and executed again. # notification script: This script will be invoked when sentinel has any warning level events (such as subjective and objective failures of redis instances, etc.). In this case, this script should notify the system administrator of abnormal system operation via email, SMS, etc. When the script is called, two parameters are passed to the script: the type of the event and the description of the event. If the sentinel.conf configuration file is configured with the script path, then the script must exist in the path and be executable, otherwise sentinel will not start successfully. # sentinel notification-script <master-name> <script-path> sentinel notification-script mymaster /var/redis/notify.sh # When a master changes due to a failover, this script will be called to inform the client that the master address has changed. The following arguments will be passed to the script when it is called: # <master-name> <role> <state> <from-ip> <from-port> <to-ip> <to-port> <role> is one of the "leader" or "observer". # from-ip, from-port, to-ip, to-port are used to communicate with the old master and the new master (i.e., the old slave) # This script should be generic and can be called multiple times, not specific. # sentinel client-reconfig-script <master-name> <script-path> sentinel client-reconfig-script mymaster /var/redis/reconfig.shCopy the code

Sentry start command

Redis-sentinel sentinel_ port.confCopy the code

conclusion

Advantages:

  • Sentinel cluster, based on the master-slave replication mode, has all the advantages of a master-slave configuration
  • Master and slave can be switched, failures can be transferred, and the system is more usable
  • Sentinel mode is an upgrade from master-slave mode, manual to automatic, more robust!

Disadvantages:

  • Redis is not easy to expand online, once the cluster capacity reaches the upper limit, online expansion is very troublesome!
  • Implementing sentry mode configuration is actually quite cumbersome, there are a lot of choices!

Redis cluster

Status quo of the problem

  1. The QPS provided by single Redis can reach 100,000 / SEC, and the current business OPS has reached 200,000 / SEC.
  2. Memory The single-node capacity reaches 256 GB, and the current service requirements are 1 TB.

Solution: Redis Cluster

Cluster architecture

To ensure high availability, at least three active and three slave servers are deployed. One-to-one matching. When the slave goes down, the slave is marked as FAIL by the cluster, which has no impact on the cluster. When the slave goes online again, the FAIL mark of the slave is cancelled by the cluster. When the master fails, the cluster marks this machine as FAIL and its slave automatically becomes the master to provide services. After the old master provides services again, the slave becomes the new master.

The cluster effect

  1. Load balancing is implemented by distributing the access pressure on a single server
  2. The storage pressure of a single server is dispersed to achieve scalability
  3. Reduce service disasters caused by the failure of a single server

Cluster structure design

Redis Chinese cluster specification

Data storage design

  1. Cut all storage space plans into 16,384 pieces (slot slots), each host saves a portion (evenly distributed, and slots are recorded by nodes)
  2. Through the algorithm design, calculate the location of the key to be saved: use CRC16 algorithm to calculate the key, and take the module of 16384 to get the location to be saved
  3. Place the key in the appropriate position
  4. When nodes are added or deleted, slots are recalculated and allocated

Design of cluster internal communication

  1. Each database communicates with each other and stores slot number data in each database
  2. Once hit, return directly
  3. One miss, inform the location

Through the data storage design, ensure that the maximum two hits data, efficient.

The cluster structures,

Single server 3 active server 3 slave server. Redis Chinese Cluster tutorial

  1. Configuration file redis-6379.conf
Bind 127.0.0.1 port 6379 daemonize yes dir /opt/redis-data databases 16 logfile redis-cluster-6379.log pidfile redis-cluster-6379.pid # RDB dbfilename dump-cluster-6379.rdb rdbcompression yes rdbchecksum yes save 10 2 # AOF Appendonly yes appendfsync always appendfilename appendonly-cluster-6379.aof # cluster # Enable cluster cluster-enabled Yes # cluster-config-file node-cluster-6379.conf # Yes # cluster-config-file node-cluster-6379.conf It is used to determine whether the node is offline or switched to secondary node. 30 seconds to 1 minute cluster-node-timeout 10000 # master slave minimum number of connections # cluster-migration-barrier <count>Copy the code
  1. Copy the other five configuration files
sed 's\6379\6380\g' redis-6379.conf > redis-6380.conf
sed 's\6379\6381\g' redis-6379.conf > redis-6381.conf
sed 's\6379\6382\g' redis-6379.conf > redis-6382.conf
sed 's\6379\6383\g' redis-6379.conf > redis-6383.conf
sed 's\6379\6384\g' redis-6379.conf > redis-6384.conf
Copy the code
  1. Open 6 session Windows and start 6 Redis respectively
redis-server /usr/local/bin/redis-cluster-conf/redis-6379.conf
redis-server /usr/local/bin/redis-cluster-conf/redis-6380.conf
redis-server /usr/local/bin/redis-cluster-conf/redis-6381.conf
redis-server /usr/local/bin/redis-cluster-conf/redis-6382.conf
redis-server /usr/local/bin/redis-cluster-conf/redis-6383.conf
redis-server /usr/local/bin/redis-cluster-conf/redis-6384.conf
Copy the code

After the startup is complete, Redis is in the waiting state. The following information is displayed in redis-cluster-6379.log

15844:C 06 Jan 2021 13:16:19.761 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
15844:C 06 Jan 2021 13:16:19.761 # Redis version=5.0.8, bits=64, commit=00000000, modified=0, pid=15844, just started
15844:C 06 Jan 2021 13:16:19.761 # Configuration loaded
15844:M 06 Jan 2021 13:16:19.762 * Increased maximum number of open files to 10032 (it was originally set to 1024).
15844:M 06 Jan 2021 13:16:19.763 * No cluster configuration found, I'm aaaa8ae93c56443127e210f3f54cc1daff01e68b
15844:M 06 Jan 2021 13:16:19.764 * Running mode=cluster, port=6379.
15844:M 06 Jan 2021 13:16:19.765 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
15844:M 06 Jan 2021 13:16:19.765 # Server initialized
15844:M 06 Jan 2021 13:16:19.765 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
15844:M 06 Jan 2021 13:16:19.765 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
15844:M 06 Jan 2021 13:16:19.765 * Ready to accept connections
Copy the code
  1. Open clusters
Redis -cli --cluster create --cluster-replicas 1 127.0.0.1:6379 127.0.0.1:6380\127.0.0.1:6381 127.0.0.1:6382 Rb create --cluster-replicas 1 127.0.0.1:6379 127.0.0.1:6383 127.0.0.1:6384 It can also be started using utils/create-cluster scriptCopy the code

–cluster-replicas 1: indicates that only one slave exists. The redis service will automatically assign the roles of master and slave, as well as the master-slave relationship between master and slave.



At this point, view redis-cluster-6379.log again

15844:M 06 Jan 2021 13:29:11.045 # configEpoch set to 1 via CLUSTER set-config-epoch 15844:M 06 Jan 2021 13:29:11.071 # IP address for this node updated to 127.0.0.1 15844:M 06 Jan 2021 13:29:16.041 # Cluster state changed: Ok 15844:M 06 Jan 2021 13:29:18.128 * Replica 127.0.0.1:6382 asks for synchronization 15844:M 06 Jan 2021 13:29:18.128 *  Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '706b714c3f9f12e032510af413c9fb367ad642bc', my replication IDs are '0054dbc2b6ab7fa0cf25f0f3487a23762de3d10c' and '0000000000000000000000000000000000000000') 15844:M 06 Jan 2021 13:29:18.128 * Starting BGSAVE for SYNC with target: Disk 15844:M 06 Jan 2021 13:29:18.131 * Background Saving started by PID 15883 15883:C 06 Jan 2021 13:29:18.147 * DB Saved on disk 15883:C 06 Jan 2021 13:29:18.147 * RDB: 2 MB of memory used by copy-on-write 15844:M 06 Jan 2021 13:29:18.166 * Background saving terminated with success 15844:M 06 Jan 2021 13:29:18.166 * Synchronization with Replica 127.0.0.1:6382 SucceededCopy the code

You can see that 6379 is the master, and its slave is 6382. After the connection, the slave immediately requests data synchronization.

  1. Client connection test

  • The first time the normal connection is used, the set value is wrong, because the slot 5798 calculated by name is not in the range of 0 to 5460 of 6379.
  • In the second cluster connection mode, the client is automatically redirected to 6380 and the set succeeds.
  • If the cluster connection is adopted for the third time, 6382 GET name will be automatically redirected to 6380, and the GET succeeds.

Unlike Sentinel, slave in cluster mode does not provide read (let alone write) functionality.

  1. View cluster nodes
/ root @ bogon ~ # redis - cli - c - p 6379 127.0.0.1:6379 > cluster nodes c540a50155f5d84fd666addfd41f2587e262e93d 127.0.0.1:6384 @ 16384 slave ae7245c003237ac71baab5d83e7752184cd00735 0 1609915141702 6 connected 8 cbb0a6497be66c0eeb7fa029d8a890372cb6002 127.0.0.1:6380 @ 16380 master - 0 1609915139685 2 connected. 5461-10922 Aaaa8ae93c56443127e210f3f54cc1daff01e68b 127.0.0.1:6379 @ 16379 myself, master 1609915140000 connected 1 0 0-5460 Ae7245c003237ac71baab5d83e7752184cd00735 127.0.0.1:6381 @ 16381 master - 0 1609915138000 3 connected. 10923-16383 B8b8f4bd0492840b8f37a35652cc6078674fa7d7 127.0.0.1:6382 @ 16382 slave aaaa8ae93c56443127e210f3f54cc1daff01e68b 0 1609915140695 4 connected 17695790129 c6e53d974d89df4ad15ee01a7c18e 127.0.0.1:6383 @ 16383 slave 8cbb0a6497be66c0eeb7fa029d8a890372cb6002 0 1609915140000 5 connectedCopy the code

This information is also displayed in node-cluster-6379.conf

[root @ bogon redis - data] # cat node - cluster - 6379. The conf c540a50155f5d84fd666addfd41f2587e262e93d 127.0.0.1:6384 @ 16384 slave ae7245c003237ac71baab5d83e7752184cd00735 0 1609910958000 6 connected 8cbb0a6497be66c0eeb7fa029d8a890372cb6002 127.0.0.1:6380 @ 16380 master - 0 1609910959000 2 connected aaaa8ae93c56443127e210f3f54cc1daff01e68b. 5461-10922 127.0.0.1:6379 @ 16379 myself, master 1609910958000-0 1 connected ae7245c003237ac71baab5d83e7752184cd00735 0-5460 127.0.0.1: master 6381 @ 16381-0 1609910957861 3 connected b8b8f4bd0492840b8f37a35652cc6078674fa7d7. 10923-16383 127.0.0.1:6382 @ 16382 slave aaaa8ae93c56443127e210f3f54cc1daff01e68b 0 1609910959000 4 connected 17695790129 c6e53d974d89df4ad15ee01a7c18e 127.0.0.1:6383 @ 16383 slave 8 cbb0a6497be66c0eeb7fa029d8a890372cb6002 0 1609910959894 5 connected vars currentEpoch 6 lastVoteEpoch 0Copy the code
  1. Slave goes offline. All nodes mark the offline node as FAIL
Log 15844:M 06 Jan 2021 15:10:15.156 # Connection with Replica 127.0.0.1:6382 lost. 15844:M 06 Jan 2021 15:10:25. 343 * FAIL message received from 8 cbb0a6497be66c0eeb7fa029d8a890372cb6002 about B8b8f4bd0492840b8f37a35652cc6078674fa7d7 # node - cluster - 6379. The conf c540a50155f5d84fd666addfd41f2587e262e93d content 127.0.0.1:6384 @ 16384 slave ae7245c003237ac71baab5d83e7752184cd00735 0 1609917023346 6 connected 8 cbb0a6497be66c0eeb7fa029d8a890372cb6002 127.0.0.1:6380 @ 16380 master - 0 1609917024363 2 connected. 5461-10922 Aaaa8ae93c56443127e210f3f54cc1daff01e68b 127.0.0.1:6379 @ 16379 myself, master 1609917023000 connected 1 0 0-5460 Ae7245c003237ac71baab5d83e7752184cd00735 127.0.0.1:6381 @ 16381 master - 0 1609917024057 3 connected. 10923-16383 B8b8f4bd0492840b8f37a35652cc6078674fa7d7 127.0.0.1:6382 @ 16382 slave, fail aaaa8ae93c56443127e210f3f54cc1daff01e68b 17695790129 1609917015221 1609917013191 4 disconnected c6e53d974d89df4ad15ee01a7c18e 127.0.0.1:6383 @ 16383 slave 8cbb0a6497be66c0eeb7fa029d8a890372cb6002 0 1609917022331 5 connected vars currentEpoch 6 lastVoteEpoch 0Copy the code

After 6382 is brought online again, the FAIL flag of 6382 is cleared from all nodes. 6382 requests data synchronization from 6379.

15844: M 06 15:16:28 Jan 2021. 721 * Clear the FAIL state for node b8b8f4bd0492840b8f37a35652cc6078674fa7d7: Replica is reachable again. 15844:M 06 Jan 2021 15:16:29.698 * Replica 127.0.0.1:6382 asks for synchronization 15844:M 06 Jan 2021 15:16:29.698 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '65a1552a9d747d634c8a824ad0b44297ed51012f', my replication IDs are 'f58d385b5df5ee15782872420963857b7e6f8147' and '0000000000000000000000000000000000000000') 15844:M 06 Jan 2021 15:16:29.698 * Starting BGSAVE for SYNC with target: Disk 15844:M 06 Jan 2021 15:16:29.699 * Background saving started by PID 16202 16202:C 06 Jan 2021 15:16:29.701 * DB Saved on disk 16202:C 06 Jan 2021 15:16:29.702 * RDB: 2 MB of memory used by copy-on-write 15844:M 06 Jan 2021 15:16:29.737 * Background saving terminated with success 15844:M 06 Jan 2021 15:16:29.737 * Synchronization with Replica 127.0.0.1:6382 SucceededCopy the code
  1. When the master goes offline, all nodes mark the offline node as FAIL, and its slave automatically changes to master
Log 16198:S 06 Jan 2021 15:18:14.830 # Connection with master lost. 16198:S 06 Jan 2021 1536 :S 06 Jan 2021 15:18:15.281 * Connecting to master 127.0.0.1:6379 16198:S 06 Jan 2021 15:18:15.282 * MASTER <-> REPLICA Sync started 16198:S 06 Jan 2021 15:18:15.282 # Error condition on socket for SYNC: Connection refused 16198:S 06 Jan 2021 15:18:16.297 * Connecting to MASTER 127.0.0.1:6379 16198:S 06 Jan 2021 15:18:16.297 * MASTER <-> REPLICA sync started 16198:S 06 Jan 2021 15:18:16.298 # Error condition on socket for sync: Connection refused 16198:S 06 Jan 2021 15:18:17.316 * Connecting to MASTER 127.0.0.1:6379 16198:S 06 Jan 2021 15:18:17.316 * MASTER <-> REPLICA sync started 16198:S 06 Jan 2021 15:18:17.316 # Error condition on socket for sync: Connection refused 16198:S 06 Jan 2021 15:18:18.338 * Connecting to MASTER 127.0.0.1:6379 16198:S 06 Jan 2021 15:18:18.338 * MASTER <-> REPLICA sync started 16198:S 06 Jan 2021 15:18:18.338 # Error condition on socket for sync: Connection refused 16198:S 06 Jan 2021 15:18:19.352 * Connecting to MASTER 127.0.0.1:6379 16198:S 06 Jan 2021 15:18:19.352 * MASTER <-> REPLICA sync started 16198:S 06 Jan 2021 15:18:19.353 # Error condition on socket for sync: Connection refused 16198:S 06 Jan 2021 15:18:20.371 * Connecting to MASTER 127.0.0.1:6379 16198:S 06 Jan 2021 15:18:20.371 * MASTER <-> REPLICA sync started 16198:S 06 Jan 2021 15:18:20.372 # Error condition on socket for sync: Connection refused 16198:S 06 Jan 2021 15:18:21.387 * Connecting to MASTER 127.0.0.1:6379 16198:S 06 Jan 2021 15:18:21.387 * MASTER <-> REPLICA sync started 16198:S 06 Jan 2021 15:18:21.387 # Error condition on socket for sync: Connection refused 16198:S 06 Jan 2021 15:18:22.402 * Connecting to MASTER 127.0.0.1:6379 16198:S 06 Jan 2021 15:18:22.402 * MASTER <-> REPLICA sync started 16198:S 06 Jan 2021 15:18:22.403 # Error condition on socket for sync: Connection refused 16198:S 06 Jan 2021 15:18:23.419 * Connecting to MASTER 127.0.0.1:6379 16198:S 06 Jan 2021 15:18:23.419 * MASTER <-> REPLICA sync started 16198:S 06 Jan 2021 15:18:23.419 # Error condition on socket for sync: Connection refused 16198:S 06 Jan 2021 15:18:24.439 * Connecting to MASTER 127.0.0.1:6379 16198:S 06 Jan 2021 15:18:24.439 * MASTER <-> REPLICA sync started 16198:S 06 Jan 2021 15:18:24.439 # Error condition on socket for sync: Connection refused 16198:S 06 Jan 2021 15:18:25.457 * Connecting to MASTER 127.0.0.1:6379 16198:S 06 Jan 2021 15:18:25.457 * MASTER <-> REPLICA sync started 16198:S 06 Jan 2021 15:18:25.457 # Error condition on socket for sync: Connection refused 16198:S 06 Jan 2021 15:18:25.662 * FAIL message received from refused 16198:S 06 Jan 2021 15:18:25.662 * FAIL message received from refused 16198:8 cbb0a6497be66c0eeb7fa029d8a890372cb6002 about aaaa8ae93c56443127e210f3f54cc1daff01e68b S 06 15:18:25 Jan 2021. 662  # Cluster state changed: Fail 16198:S 06 Jan 2021 15:18:25.762 # Start of election delayed for 786 milliseconds (rank #0, Offset).16198 :S 06 Jan 2021 15:18:26.472 * Connecting to MASTER 127.0.0.1:6379 16198:S 06 Jan 2021 15:18:26.472 * MASTER <-> REPLICA sync started 16198:S 06 Jan 2021 15:18:26.472 # Error condition on socket for sync: Connection refused 16198:S 06 Jan 2021 15:18:26.573 # Starting a failover election for epoch 7.16198 :S 06 Jan 2021 15:18:26.576 # Failover election won: 16198:S 06 Jan 2021 15:18:26.576 # configEpoch Set to 7 after successful Failover 16198:M 06 Jan 2021 15:18:26. 576 # Setting secondary replication ID to f58d385b5df5ee15782872420963857b7e6f8147, valid up to offset: 5741. A New replication ID is: 98786184 f01dabee0bb9c017e7b15e8d10842885 16198 M 06 15:18:26 Jan 2021. 576 * Discarding Previously cached Master State. 16198:M 06 Jan 2021 15:18:26.576 # Cluster State changed: OKCopy the code

In the log, 10 attempts were made to connect to the master node because cluster-node-timeout is set to 10 seconds in the configuration file (PING the master node once every second). Then, the state of the master node is marked as FAIL and the user becomes the master. If you restore 6379, you’re just a slave.

  1. Cluster Node operation command
    • Cluster nodes: Displays information about cluster nodes
    • Cluster replicate

      : Enters a slave node, Redis, and switches its master node
    • Cluster meet IP :port: Discovers a new node and adds a new primary node
    • Cluster forget < ID > : Ignores a node that has no slot
    • Cluster Failover: Indicates manual failover
  2. For other commands, such as adding new nodes, see Redis Chinese cluster specifications, or the following:
[root@bogon ~]# redis-cli --cluster help Cluster Manager Commands: create host1:port1 ... hostN:portN --cluster-replicas <arg> check host:port --cluster-search-multiple-owners info host:port fix host:port --cluster-search-multiple-owners reshard host:port --cluster-from <arg> --cluster-to <arg> --cluster-slots <arg> --cluster-yes --cluster-timeout <arg> --cluster-pipeline <arg> --cluster-replace rebalance host:port --cluster-weight <node1=w1... nodeN=wN> --cluster-use-empty-masters --cluster-timeout <arg> --cluster-simulate --cluster-pipeline <arg> --cluster-threshold <arg> --cluster-replace add-node new_host:new_port existing_host:existing_port --cluster-slave --cluster-master-id <arg> del-node host:port node_id call host:port command arg arg .. arg set-timeout host:port milliseconds import host:port --cluster-from <arg> --cluster-copy --cluster-replace help For check, fix, reshard, del-node, set-timeout you can specify the host and port of any working node in the cluster.Copy the code

Faqs and Solutions

Using caching can greatly improve application performance and efficiency and avoid frequent access to relational databases, but it also brings some problems. Among them, the most crucial problem, it is the consistent problem of data, tell from strict sense, this problem does not have solution. If the consistency of the data is high, you can’t use caching.

Cache warming

After the system is started, the system breaks down rapidly. The possible cause is that there is no data in the cache and a large number of requests access the database at the same time, which leads to the server breakdown. Load hotspot data into the cache before system startup.

Solution:

  • Cache the data ahead of time

Cache avalanche

A large number of keys fail in a short period of time, resulting in a large number of requests to the database. Redis and database resources may be heavily occupied or even crash.

Solution:

  • Building multi-level caching
    • Nginx cache + Redis cache + EHCache
  • Set up the Redis cluster
    • Prevent too many keys from collective failure due to Redis downtime
  • Traffic limiting and degradation
    • Limit the number of requests, or turn off some functionality
  • Data expiration time is adjusted
    • Super hot data never expires
    • Error peak classification based on service data validity period
    • Expiration time takes the form of fixed time + random value
    • Artificial maintenance

Cache breakdown

Cache breakdown refers to the failure of a key, when a large number of requests flood into the database and breach the cache. In this case, Redis ran smoothly, but the database strain spiked and even crashed.

Solution:

  • Increase the validity period of hotspot data, or even permanent
  • Building multi-level caching
    • Set different expiration times to ensure that they do not expire simultaneously
  • Add a mutex
    • Only one thread with the same key can query the database at a time

The cache to penetrate

Cache penetration means there is no data in Redis, look it up in the database, and there is no data. If a large number of requests are looking for data that does not exist, the Redis server will experience a surge in CPU usage, database stress and even crash.

Solution:

  • Cache is null
    • Cache null data with a short validity period, such as 30 to 60 seconds and a maximum of 5 minutes
  • Whitelist policy
    • Preheat the bitmap corresponding to each category data ID in advance. The ID serves as the offset of the bitmap, which is equivalent to setting the data whitelist. Release normal data when loading, and directly intercept abnormal data when loading (low efficiency)
    • Use bloom filters (hit issues with Bloom filters are negligible for current conditions)
  • The key to encrypt the
    • When the problem occurs, the disaster prevention service key is temporarily started, the encryption service of the key is transmitted at the business layer, and the verification program is set to verify the incoming key
    • For example, 60 encryption strings are randomly allocated every day, two or three are selected and mixed into the page data ID. If the access key does not meet the rules, the data access is rejected