Redis is a typical single-threaded architecture where all read and write operations are done in one main thread. When Redis is used in high-concurrency scenarios, this thread becomes its lifeline. Blocking, even for a short time, is a nightmare for our application. Scenarios leading to blocking problems can be roughly divided into internal and external causes:

  • Inherent causes include inappropriate use of apis or data structures, CPU saturation, persistent blocking, and so on.
  • External causes include CPU contention, memory swapping, and network faults.

I. Internal reasons:

1. Improper use of API or data structure

Screen:

  1. Slow discovery query: executeslowlog get {n}
  2. Discover large objects: Executeredis-cli -h {ip} -p {port} --bigkeys

Solution:

  1. Solve slow query: Adjust the command to low algorithm command
  2. Adjust the large object: Split the large object into multiple small objects to prevent one command from manipulating too much data

CPU saturation

screening

  1. To check the CPU usage, runtopThe command
  2. To query the Redis concurrency, runredis-cli -h {ip} -p {port} --statThe command

Solution:

  1. The clustering level expands and apportionments OPS pressure
  2. Check for excessive memory optimization

Persistent blocking

screening

  1. Check the recent fork operation time: Executeinfo statusLatest_fork_usec indicator is obtained
  2. Check the recent successful flush time of AOF: View logs

Solution:

  1. If the fork operation takes more than one second, avoid using large memory instances and avoid operating systems with slow fork operations
  2. If the AOF flush fsync operation succeeds for more than two seconds, reduce the pressure of other processes on the hard disk

Two, external reasons

1. The CPU competition

Troubleshooting: Run the top or SAR command

Solution:

  • It is not recommended to deploy with other multi-core CPU-intensive services, which can affect Redis throughput
  • Binding the Redis process to the CPU reduces CPU context switching overhead
  • CPU binding is not recommended for primary nodes that enable persistence or participate in replication, as child processes will occupy 90% of the race

2. Swap memory

Screen:

  1. To query the Redis process number, runredis-cli -p 6383 info server | grep process_idThe command
  2. Querying memory swapping information by process ID: Runcat /proc/process_id/smaps | grep Swap

Solution:

  1. Ensure that the machine has sufficient memory available
  2. Ensure that all Redis instances are set to the maximum available memory to prevent uncontrolled Redis growth
  3. Reduce the swap priority of the system

3. Network problems

Screen:

  1. Intermittent network disconnection: Run the SAR -n DEV command to check whether the historical traffic is normal

  2. Redis connection refused: perform Redis – cli – p 6384 info stats | grep rejected_connections view all rejected the number of connections

  3. Connection overflow: a problem with the operating system or Redis client during connection. Reasons can range from process limitations to backlog queue overflows. Run the ulimit -n command to check the resource usage limit of the process. Perform netstat -s | grep overflowed command, check whether there is a growing connection refused

  4. Network latency: Run the redis-cli -h {host} -p {port} –latency command to measure the network latency between machines

Solution:

  1. Avoid cross-room calls between the client and Redis
  2. When clients access Redis, try to use NIO long connection or connection pool
  3. For Redis that supports a large number of connections, increase the Ulimit value and modify the system backlog value
  4. Adjust the network topology by using the same physical server > Same Rack > Cross Rack > Same room > Same city room > Remote room

data

Redis blocking causes and solutions

Redis blocks collating notes

Rational use of their every minute every second of time to learn to improve themselves, do not use “no time” to cover up their ideological laziness! While young, hard to fight, to the future of their own account!

Things to learn