Docker installs a Redis
Docker run --name myredis -d p6379:6379 redis Redis > docker exec it myredis redis-cliCopy the codeIO /commands Retrieves all commands
string
Common command set get setex expire exist del mset mget incr incrby
Less than 1 MB Double the existing space
If the capacity is greater than 1 MB +1 MB
capacity < 512M
list
Common commands rpush lpush rpop lpop llen
Can do queue right in and left out
Can do stack right in and right out
Slow operation lindex Ltrim
Difference between Ltrim and Lrange
If the zipList data is small, change the zipList data to quickList
hash
Common commands hset hlen hget hmset hmget hincrby hincr hgetall
Progressive rehash
Information about the pros and cons of objects stored in hash or string Hash can be used to query certain dictionaries of objects, which saves traffic and avoids the waste of integer search. However, the structure consumes more than string
set and zset
One is an unordered set and one is an ordered set
The zrange books 0-1 command displays the positive order of the query range
Zrevrange Books 0-1 reverse order of the range of queries
zcard count()
Zscore “XXX” query score
Zrank “XXX” query rank
Zrangebyscore 0 9.0 queries elements in a range based on the score value
Zrangebyscore books -inf 8.91 withscores ps: -INF = -∞ withscores
Zrem delete
Ordered collections sort stored set elements by specifying score
Zset implements order through the data structure of a hop table
Generic rules set/hash/list/zset
1. If no create if not exists is created
Drop if no elements
Expire TTL event
Distributed lock redis
Atomic operations are operations that are not interrupted by thread scheduling; Once this operation starts, it runs until it ends without any context switch thread switching. A distributed lock is used to limit concurrent execution of the program
Plan 1
Setnx lock del unlock
Problem: Del cannot execute a program deadlock if the program fails between setnx and DEL
Scheme 2
Setnx + expire adds the lock and sets the expiration time del unlocks
Problem: If a program fails between setnx and EXPIRE, DEL cannot execute and program deadlocks cannot be unlocked through expiration mechanisms
Solution 3 Set XXX EX 5 NX Lock Set expiration time del Unlock
Due to locking the expiration time is set by the Redis author with a command as an atomic operation
Timeout problems
Problem 1: If the program logic takes too long to execute over the expire time, then the lock is released, which may cause data corruption (requiring human intervention).
Problem 2 The del of the previous operation unlocked the lock just captured by the later operation ps: solution stamp each lock to generate a stamp unlock stamp verify that the correct lock is solved Redis also does not provide instructions like delifequals Therefore, lua scripts are needed to handle multiple atomic operations that are performed
Distributed locks in a cluster environment
In the Redis Sentinel cluster, if the master node acquires the lock and the master node hangs during the synchronization process, then the slave node has no lock information. The slave node is promoted to the master node and the lock is released virtually
When Redlock algorithm locks, it will send set(key, value, nx=True, ex= XXX) instruction to the half-node. As long as the half-node set succeeds, the lock is considered successful. To release the lock, the DEL directive is sent to all nodes. However, the Redlock algorithm also needs to consider many details such as error retry, clock drift, and because Redlock needs to read and write to multiple nodes, it means that the performance of Redis is lower than that of single-instance Redis.
Redisson is a Watchdog
When applied in task execution lock will apply additional threads of a guard dog every once in a while (must be shorter than the lock expiration time) to redis renewal lock information expiration time solve tasks unfinished lock expired And because the task performed or accidental cancel disappear Didn’t the guard dog So expired mechanism is still not cause a deadlock
Lock Conflict Handling
Distributed lock, the client in processing the request to add the lock did not add success how to do. There are generally three strategies for handling lock failures:
1 Throw an exception and notify the user to try again later.
2. Sleep for a while and retry.
3 Move the request to a delay queue and try again later
Delay queue
lpush/rpop
Blpop/BRPOP The server automatically disconnects from idle connections to reduce idle resources. At this point blPOP/BRPOP will throw an exception
Implementation of delay queue
The bitmap
A bitmap is actually a small unit bitmap of string
The user’s annual check-in record, signed is 1, not signed is 0, to record 365 days
Setbit Sets the value of the bitmap position
Bitcount Counts the number of 1’s in the specified location.
Bitpos looks for the first 0 or 1 in the specified range (find the day the user first checked in)
Bitfield has three subinstructions, namely GET /set/incrby, which can read and write the specified bit fragment, but can only handle up to 64 consecutive bits. If more than 64 bits are used, multiple subinstructions are used. Bitfield can execute multiple subinstructions at once
www.cnblogs.com/liujiduo/p/… User check-in scenario redis encapsulation
HyperLogLog
HyperLogLog provides two instructions pfadd and pfcount pfmerge
When the count is relatively small, its storage space is stored by sparse matrix, which occupies very small space. Only when the count gradually increases and the space occupied by sparse matrix gradually exceeds the threshold value, will it be converted into dense matrix at one time, occupying 12K space
Provides inaccurate statistics to save space
Bloom filter
Judge heavy scene guava implementation
About the current limit
The system limits a user’s behavior to N times in a specified period of time
Fixed window sliding window leaky bucket token bucket
Blog.csdn.net/weixin_4184…
1. Sliding window scheme
Action_key Specifies the number of times an action can occur within a specified period. Max_count def is_action_allowed(user_id, action_key, period, max_count): Can_reply = is_action_allowed("laoqian", "reply", 60, 5) if can_reply: do_reply() else: raise ActionThresholdOverflow()Copy the code2. Funnel algorithm
About the geo
Scan
1 use high-order carry addition to traverse to avoid slot repetition and omission when considering dictionary expansion and shrinkage. 2 Progressive rehash
Meituan recent Scan of a bug fix mp.weixin.qq.com/s/ufoLJiXE0…
Redis IO thread model
Event rotation (multiplexing) listen for events each client has an instruction queue to ensure that each client responds in the same order and the queue is the same. If the queue is empty, you should cancel the read and write event operator on Selector to avoid event rotation getting a channel and finding no data to read and write. CPU spikes
Redis persistence scheme
RDB snapshot full binary stores current data BGSave
AOF Log Incrementally saves logs BGREWRITEAOF historical modification commands
RBD process fork a sub-process to serialize data write disk dump. RDB Because the parent process needs to operate data in parallel, the parent process needs to use COW to replicate data pages
AOF procedure THE AOF log records all sequence of modifiable instructions since the Redis instance was created. Executing commands before logging this is because Redis is positioned to cache logs as an added feature and writing logs before executing commands affects performance
Since AOF will always swell Redis provides the BgreWriteAOF directive for AOF sliming to open up a sub-process to traverse memory converting it into a series of Redis operation instructions, serializing it into a new AOF log file and merging the Redis instructions during the operation
Fsync is a Linux force-flush command. Redis uses a timer of 1per/s to flush disks, striking a balance between performance and availability
Redis 4.0 Hybrid Persistence will cause more data loss due to RDB snapshot recovery. AOF’s “replay” will make Redis recover very slowly
AOF files are no longer recorded from startup, but since the last RDB file was generated, which allows for minimal recovery of AOF log files and takes full advantage of both
Redis pipline
Ps commands must be independent of each other. The nature of the pipe is that you do not need to wait for data to return before you send the command. Therefore, the return time is determined by the processing time of the slowest command
Redis transactions
The transaction instruction
Multi/exec/discard. Multi indicates the start of a transaction, exec indicates the execution of a transaction, and discard indicates the discarding of a transaction
Redis transactions cannot guarantee atomicity only isolation without rollback
Redis’ optimistic locking mechanism, Watch
PubSub
A queue multicast scheme
CAP
The PRINCIPLE of CAP is the theoretical foundation of distributed storage, just like Newton’s laws of distributed domain. Since CAP’s paper was published, distributed storage middleware has sprung up one after another. Understanding this principle is actually quite simple. In this section, we will first explain it briefly.
Nodes of distributed system are usually distributed on different machines for network isolation, which means that there is A certain risk of network disconnection. A) Consistent B) Availability C) p-partition C) tolerance The technical term for this disconnection scenario is “network partitioning.”
When network partitioning occurs, the two distributed nodes cannot communicate with each other, and the changes we make on one node cannot be synchronized to the other node, so the “consistency” of data cannot be satisfied because the data on the two distributed nodes are no longer consistent. Unless we sacrifice “availability”, that is, the suspension of the distributed node service, the ability to modify data when the network partition occurs, until the network condition is fully restored.
The structure of the redis
Master-slave synchronization
The AP program
Process :Redis synchronizes instruction streams. The master node records the instructions that have modifiable effects on its state in the local buffer and asynchronously synchronizes the instructions in the buffer to the slave node. The slave node executes the synchronized instruction streams to achieve the same state as the master node. One side reports back to the master node where it is synchronized (offset).
Because memory buffers are limited, the Redis main library cannot record all instructions in memory buffers. Redis’s copied buffer is a fixed-length, circular array that overwrites everything from the beginning if the array is full
A long period of disconnection will cause the host node’s buffer to be full. The command is overwritten by the later command, causing the loss of some commands that need to be synchronized. In this case, snapshot synchronization is required. That is, RBD files are passed between master and slave to synchronize the current data of the master node and then restore to the incremental synchronization scheme
The Wait command synchronizes data from the node
Redis. Conf configuration resolution
INCLUDES synchronizes a small number of defined configurations to a large number of configuration files through include if you want to override them at the end
MODULES Starts loading the plug-in module
NETWORK bind 0.0.0.0 open to extranet protected-mode yes Restrict access to only one machine
GENERAL daemonize no Specifies whether to run in the background
SNAPSHOTTING save
REPLICATION
SECURITY
sentinel
Redis Sentinel is a solution where we can automatically switch master and slave to ensure availability.
About the Sentinel master-slave cluster
About redis cluster
About Haproxy Redis primary/secondary cluster
About the Codis cluster solution
About the overdue
Redis creates a set of expired keys and periodically traverses the expired keys to delete them. In addition to this, the idea of lazy deletion is used, where deletion is performed when the user accesses an expired key. One is centralized and one is scattered.
Periodic scan policy.
1. Random 20 keys from the expired dictionary; 2. Delete expired keys from the 20 keys. 3. If more than a quarter of the keys are expired, repeat Step 1.
In addition, the maximum reclamation time is set to 25 Ms. By default. The idea here is very similar to that of garbage collector G1
The cache penetration problem where a large number of keys fail at the same time is avoided by giving expiration time + random time
Slave data is deleted synchronously by the master AOF instruction
LRU
Redis sets maxMemory to a maximum amount of memory that can be used to disable keys.
Six elimination mechanisms:
Noeviction will not continue to service write requests (DEL requests can continue to service), read requests can continue to service. This ensures that data will not be lost, but the online business will not continue. This is the default elimination strategy.
Volatile – LRU attempts to eliminate keys with expiration dates, and the least used keys are preferred. Keys that do not have an expiration date are not obsolete, which ensures that data that needs to be persisted is not suddenly lost.
Volatile – TTL is the same as above, except that the TTL is not the LRU, but the TTL of the remaining life of the key.
Volatile -random is the same as above, except that the eliminated key is a random key from the expired key set.
Allkeys-lru differs from volatile lRU in that the key object to be eliminated is the entire key set, not just the expired key set. This means that keys that are not set to expire will also be eliminated.
Allkeys-random is the same as above, but the elimination strategy is random key.
LRU dictionary plus list arranges lists in the order of put (or otherwise), with elements at the end when accessed. When the space is full, eliminate the header element.
Redis uses an approximate LRU algorithm. To approximate the LRU algorithm, Redis adds an extra small field to each key. This field is 24 bits long, which is the timestamp when it was last accessed
security
To disable certain Redis commands such as rename-command flushall “”
Add password requirepass XXXXX master/slave copy masterauth XXXXX
Use Spiped to proxy SSL for Redis
Lua scripts extend the instruction set
Such as:
del_if_equals.lua
if redis.call("get",KEYS[1]) == ARGV[1] then
return redis.call("del",KEYS[1])
else
return 0
end
Copy the codeSCRIPT LOAD and EVALSHA directives solve long Lua SCRIPT problems. Cache the corresponding SHA1 ID in Redis server. EVALSHA
Useful command line tools
Scan the big key
Redis – cli – bigkeys – I 0.01
Back up RDB files remotely
redis-cli –rdb ./user.rdb