The source IP address, port number, source MAC address, destination IP address, port number, and destination MAC address are the six elements necessary for communication between computers. Among them, the port number marked the two specific application information in the application layer, namely express specific to send and receive, IP address said two endpoints in the network layer address, namely the delivery address and shipping address, MAC address said between nodes on the data link layer address, or express delivery to the post address.
After understanding how the NAT and FULLNAT models of LVS modify packets and their shortcomings, how can we better optimize the load balancer while standing on the shoulders of the NAT model?
In NAT and FULLNAT modes, both the request packet and the response packet pass through the load balancer. But the response packet is typically much larger than the request packet, which can become a system bottleneck. If the request packet can be forwarded to the real server and the response packet is returned directly from the real server, there is much less pressure on the load balancer. How should this pattern be implemented?
DR model
If the real server can actually respond directly to the client without going through the load balancer. So what does the packet that the load balancer responds to directly back to the client need to meet in order to be received properly by the client?
The packet sent by the real server must match the packet sent by the client when it is received by the client. If there is no match, the client will discard the response packet as soon as it is received.
If the client sends the request packet: CIP ➡️ VIP, the response packet received must be VIP at CIP. For a quick thought, why didn’t you bring the MAC address? The following article has an explanation ~
However, the VIP has been bound to the load balancer, and there are multiple real servers. In the connected network, how can we make the VIP address of multiple real servers and the load balancer be the same? And the real server’s VIP cannot be accessed by other devices. That is to say, the VIP address on each real server can only be known by itself, “I secretly hide the VIP”, and other devices do not know at all.
Another very interesting IP address to mention here is 127.0.0.1/0.0.0. If you think about it, 127.0.0.1 is the same “hidden” IP address on every device, and no other device can access it. This magic IP address is called the Loopback interface. It is a software-only virtual interface. When a request packet reaches the LO interface, the LO interface will “reverse” the packet directly and return it to the device itself. This is the origin of the “loopback” interface. Therefore, if the VIP is bound to the LO interface, is it possible to accomplish “only oneself knows this VIP, other devices do not know this VIP”?
Binding the VIP to the LO interface is only half of the job, which is to bind the same VIP address on multiple real servers. Remember the ARP protocol from the previous article? The ARP protocol collects the mapping relationship between MAC addresses and IP addresses of hosts other than itself in the current LAN. And VIP is an address that cannot be collected by other devices, so we need to make some modifications to the ARP protocol of the real server, so that VIP will not be collected by other devices. Obviously, this involves modifying not only the response of the device to receiving the ARP request (the arp_ignore parameter), but also the request that the device announces to the ARP (the arp_announce parameter).
Arp_ignore: Define the response level when receiving an ARP request 0: Responding to an ARP request for a native IP address received on any network card (including a loop back card), regardless of whether the destination IP address is on the receiving card Only respond to ARP requests where the destination IP address is the address of the receiving network card, and the source IP address of the ARP request must be on the same network segment as the address of the receiving network card
Allow any address on any network card to announce to the network 1: Announce to the target network if possible 2: Announce to the network only if it matches the address on the local interface
As you can see, when arp_ignore is 1 and arp_announce is 1, the VIP that is bound to the LO interface can be hidden on the local server.
(Figure: red indicates outgoing packets; Green represents the data packet returned; Yellow indicates what the load balancer modified; The dashed line indicates that after N next hops, it can not be in the same LAN. The solid line means you can only “jump once”, that is, you must be on the same LAN.)
1. When the computer sends a request packet to the load balancer, the load balancer changes the request packet {destination MAC address} to the real server MAC address, the rest of the information remains the same. The Load Balancer and the real server must be on the same LAN, otherwise the Load Balancer cannot replace the requested packet’s {destination MAC address} with the MAC address of the real server
2. The real server receives the requested data packet and finds that it has a “hidden” VIP, so it receives the data packet and sends it to the corresponding upper application for processing
3. After the processing is completed, the response packet is returned directly to the client. When viewing the TCP connection on the real server, it is: VIP ➡️ CIP
To summarize the characteristics of DR mode: 1. Only modify the “destination MAC address” of the request packet, which is applied to the data link layer. Therefore, the load balancer must be on the same LAN as the real server and cannot forward ports 2. The VIP in the real server can only be “seen” by itself, and other devices are unknowable. Therefore, the VIP must be bound to the LO network card of the real server, and this network card information is not allowed to be announced through the ARP protocol 3. After the requested packet passes through the load balancer, it is directly returned to the client by the real server. The response packet does not need to pass through the load balancer again
Top model
Is there any other way to send the response packet directly back to the client from the real server, other than simply modifying the destination MAC address of the request packet and doing MAC address spoofing? How can VPN support your telecommuting
We have already discussed that if the real server sends the response packets directly back to the client, then the real server must have a “hidden” VIP, which is configured on the LO card and does not allow this VIP to be notified via the ARP protocol.
(Figure: red indicates outgoing packets; Green represents the data packet returned; Yellow indicates what the load balancer modified; The dashed line indicates that after N next hops, it can not be in the same LAN. The solid line means you can only “jump once”, that is, you must be on the same LAN.)
1. When a request packet sent by the computer arrives at the load balancer, the load balancer does not change the source packet, but adds a layer of IP head on the source packet {distribution IP, port number, MAC address} ➡️ {real server IP, port number, MAC address}.
2. After the real server receives the requested packet, it removes the IP header of the outermost package and finds that there is another layer of IP header, and the target IP address can match the address on LO, so it receives the requested packet and gives it to the corresponding upper application for processing
3. After the processing is completed, the response packet is returned directly to the client. When viewing the TCP connection on the real server, it is: VIP ➡️ CIP
Summarize the characteristics of the TUN pattern: 1. Do not change the request packet, but add a layer of IP header information on the request packet. Therefore, the load balancer cannot forward the port, but it may not be in the same LAN as the real server, and the real server needs to support the ability to interpret two layers of IP header information, i.e., the “IP Tunneling” or “IP Encapsulation” protocol 2. The VIP in the real server can only be “seen” by itself, and other devices are unknowable. Therefore, the VIP must be bound to the LO network card of the real server, and this network card information is not allowed to be announced through the ARP protocol 3. After the requested packet passes through the load balancer, it is directly returned to the client by the real server. The response packet does not need to pass through the load balancer again
NAT, FULLNAT, DR, TUN model summary
In DR and TUN modes, only the request packet is forwarded by the Load Balancer, and the response packet is returned directly to the client without passing through the Load Balancer. In general, the response packet is larger than the request packet. If both the request packet and the response packet pass through the load balancer, it may become the system bottleneck under high concurrency.
From our derivation, we can easily derive the characteristics of the various modes and the problems they are trying to solve.
NAT mode, by changing the “destination IP address” and “source IP address” of the packet, will load the request to several real servers, which is the most basic load mode in the four-layer load balancing mode. Because it is a modification at the IP address level, at the network level, ports can be mapped. The response packet returned by the real server must pass through the load balancer, so the default gateway of the real server is required to be the load balancer.
FullNAT pattern is an evolution of NAT pattern. By changing both the “source IP address” and the “target IP address” at the same time, the restriction that the default gateway of the real server in NAT mode must be a load balancer is breached. But by changing both the “source IP address” and the “destination IP address” at the same time, the real connection established by the real server has nothing to do with the client, so the client information will be lost.
DR pattern is another evolution of NAT pattern. Due to the feature that the response packet is much larger than the request packet in the real request, it will become the bottleneck of the system under high concurrency, so the response packet is returned directly to the client from the real server. MAC address spoofing is used to achieve this, acting on the data link layer, so port mapping is not possible. And on the real server, there must be a “hidden” VIP visible only to itself. In the network, real VIPs are tied to load balancers to receive real request packets from clients. The “hidden” VIP is bound to the LO interface of the real server to trick itself into receiving packets whose destination address is VIP. So the default gateway for the real server must also be a load balancer.
The TUN pattern is an evolution of the DR pattern. Break the limitation that the default gateway for a real server in DR mode must be a load balancer. TUN mode does not modify the source packet, but adds an additional IP header information on the source packet, so port mapping cannot be done, and requires the real server to be able to unload two layers of IP header information.
A quick reminder: Why didn’t you include a MAC address when you said it was necessary for a real server to receive packets forwarded by a load balancer?
When ARP protocol and next-hop mechanism were introduced in the section of network communication, we mentioned that packet forwarding in the network is similar to the Courier station in express transmission. Each packet sending will constantly find the MAC address of the “next destination”. Therefore, any change in the physical port involves a change in the source MAC address, which is a feature of IP communication, not a feature of load balancing.
With an understanding of the various modes of load balancing, the next article will take a look at the practice and configuration
About UCloud Load Balancing Service ULB
UCloud load balancing service ULB supports two scenarios of Intranet and external network, and supports two forwarding modes of request broker and packet forwarding. ULB4 is self-developed based on DPDK technology. It adopts a forwarding mode similar to DR. A single server can provide more than 30 million concurrent connections, 10 million PPS, and 10G wire-speed forwarding capacity. Deploy in clusters with at least four servers in a single cluster. Using ECMP+ BGP to achieve high availability. ULB7 is based on HAProxy, which is the FullNat pattern. A single instance can support more than 40W PPS, 2Gbps, and at least 400,000 concurrent connections.
Compared with ULB7, ULB4 has stronger forwarding ability, which is suitable for the scenarios where forwarding performance is pursued. On the other hand, ULB7 can handle seven layers of data, SSL unloading, domain name forwarding, path forwarding, etc., and no additional VIP (virtual IP) configuration is required for back-end nodes.