Moment For Technology

Failover, service discovery, load balancing, it's all about it!!

Posted on Dec. 2, 2022, 9:36 a.m. by Sumer Bandi
Category: The back-end Tag: The back-end Load balancing

Yes, it's connection pooling, and it's a must to master when playing Internet architecture.

What is connection pooling?

The technology for creating and managing connection bufferpools is essentially a way to reuse resources and improve performance without having to create and destroy connections as often as possible.

Voice: Database connection pool, service connection pool, are connection pools.

Connection pooling plays an extremely important role in microservices layered architecture.

As shown above:

(1) The upper dashed box is the Web cluster;

(2) The lower dotted box is the service cluster;

(3) The green box represents an upstream and downstream connection;

(4) The blue box represents the connection pool;

In this example, a caller will establish two connections to a service node, and the service cluster will have three clusters, so the connection pool will have a total of six connections, from C1 to C6.

What other technical points in the Internet architecture are associated with connection pooling, other than the fact that upper-layer callers can access downstream services by taking incoming and outgoing packets from the pool?

I. Failover and service discovery

As shown above:

(1) Failover. If the old service node S1 fails and the connection of C1 and C2 fails, they will be removed from the connection pool and the subsequent requests will not be sent to the failed node;

(2) Service discovery: if the new service node S4 goes online and the connection between C7 and C8 is established, it will be added to the connection pool, and subsequent requests will be sent to the new node;

Dynamically deleting and adding connections is called dynamic connection pooling.

Service discovery: how does the new node S4 come online?

See "changed configuration, do not want to restart, how to whole?" .

2. Load balancing

Using the polling policy to use the connections in the connection pool one by one can realize the load balancing of downstream services.

Load balancing can also be achieved with a completely random strategy.

As shown above:

Give each connection the same weight, take the connection access downstream, use a random algorithm, which grid to fall on which connection, again in the above example:

n = random() % 6 + 1;


N =[1,2];

N =[3,4], access s2;

If n=[5,6], access s3;

The width of the three intervals is the same, that is, the probability of falling to a service is equal, and the load is balanced.

So, what if the service nodes have different service capabilities, some with strong processing power and some with weak processing power?

3. Static weight load balancing

As shown above:

Each service is configured with a different weight. When the connection pool is initialized, the interval size of different services is different, and the probability of landing on a grid is also different when the connection is accessed downstream:

n = random() % 16 + 1;


N =[1,2];

N =[3,6], access s2;

If n=[7,16], access s3;

The width of the three intervals is proportional to the weight of the service, that is, the probability of falling into a certain service is equal to the weight.

_ Voiceover: _ Nginx supports this, but static weights are too rough.

So what if the service nodes have different service capabilities but are difficult to identify with static weights?

Iv. Dynamic weight load balancing

As shown above: When the connection pool is initialized, a dynamic weight is assigned to the connections.

_ Voiceover: _ Service no longer needs to be configured.

The load is still allocated as before, except:

(1) Connection processing times out, and dynamic weight drops;

(2) The connection processing succeeds, and the dynamic weight rises;

For more details, see "How to Design load Balancing for Heterogeneous Servers". .

Isn't it interesting to be able to distribute the load based on the actual processing power of the service?

About (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.