High concurrency

Vertical scaling

Add configuration. Improve the hardware performance of a single machine, increase the memory and CPU, and upgrade the SSD disk.

Horizontal scaling

With the machine. Solve the single machine performance bottleneck.

The cache

  • The local cache

1. Consider the use of strong weak virtual references 2. Consider the impact of JVM garbage collection 3. Consider cache expiration and breakdown issues 4. Ehcache and Guava

  • Distributed cache

1. Consider cache penetration (no data in cache or DB), cache breakdown (no data in cache or DB), and cache avalanche (a large number of caches fail simultaneously). Cache penetration – Cache short-term empty object, cache breakdown – distributed lock, cache avalanche – random expiration time

  • Centralized cache
  • CPU cache
  • Operating system cache
  • HTTP cache
  • Database cache
  • Cache consistency
  • Cache hit ratio
  • Hot key
  • Cache update mode

Cache Aside mode; Read/Write Through mode; Write Behind Caching(Write Back) mode. Reference: Cache update mode

asynchronous

  • Distributed message queue
  • Local message queue
  • Thread asynchronous callback
  • NIO/AIO

A high performance

Increase parallel computing capability

  • Add machine, add configuration
  • The thread pool
  • The HTTP connection pool
  • multiplexing
  • Lock down
  • Parallel I/O

Reduce the response time of a single task

  • The cache
  • Time complexity
  • pooling
  • Lock down
  • Thread preheating
  • A long connection
  • Processing in advance
  • Extended post-processing

High availability

failover

  • Peer node

All nodes perform read and write operations and are stateless. The node fails.

  • Non-peer node

Primary and secondary nodes cluster based on distributed election algorithm; Cluster of nodes based on distributed consistency algorithm for storage. Heartbeat detection mechanism triggers an election operation. (such as Redis sentinel mode or cluster mode, MySQL master-slave switchover, etc.)

Timeout control

Leaving requests waiting indefinitely or tying up resources can squeeze traffic and cause an avalanche of timeouts. Therefore, timeout control is a “home for everyone” strategy, which reflects the wisdom of sacrificing a small number of requests to ensure the overall availability of the system. Common measures: synchronous custom timeout break mechanism, asynchronous wait/notify mechanism, and use of future and countdonwlatch keywords.

Flow control

Timeout control is a truncation measure taken in the request processing stage, while flow control is a measure of interception based on QPS index in the request entering stage. Common algorithms: leaky bucket algorithm and token bucket algorithm.

  • Bucket algorithm

Water droplets flow out of the bucket at a fixed rate, and drop into the bucket at any rate, the capacity of the bucket does not change. Because the capacity in the bucket is fixed, water droplets in the bucket may overflow if the rate of inflow is greater than the rate of outflow. Then the overflow water drop requests are denied access, or directly invoke the service degradation method. Assuming it’s the same time.

  • Token bucket algorithm

Control traffic speed by generating tokens at a specific rate. If no token is obtained from the token bucket, access is denied.

Demotion management

Demotion is a strategy of sacrificing marginal services to ensure the stability of core services.

Circuit breakers

The circuit breaker mechanism is a fast failure implementation scheme, which can reduce the impact on the overall system performance when the downstream service fails.

Retry mechanism

Retry is a fault – tolerant compensation mechanism to ensure service running. Possible values are synchronous limited retry and asynchronous batch retry.

Power etc.

Idempotence is an important means to support retry mechanism. Common measures: Globally unique ID, lock (pessimistic lock optimistic lock)

Gray released

Supports low-volume deployment by machine, observing system logs and service indicators, and pushing the full volume after the system runs smoothly.

Monitoring alarm

Comprehensive monitoring system, including the most basic CPU, memory, disk, network monitoring, as well as Web server, JVM, database, all kinds of middleware monitoring and business indicators monitoring.