Linux Virtual Server (LVS) is a Virtual Server cluster system. It works at the transport layer of the OSI model, or four-layer load balancing. LVS itself implements NAT, DR, and TUN models. These models only forward packets and do not establish connections with clients, resulting in low cost and high efficiency. FULLNAT is implemented based on NAT. LVS itself does not support FULLNAT. You need to patch the kernel to use FULLNAT.

This series of data packets are classified according to the processing mode of load balancer, and the realization principle of NAT, FULLNAT, DR and TUN models is discussed from the perspective of communication between computers.

How do two computers communicate over the Internet

Before you can understand LVS load balancing, you need to understand how two computers communicate over the Internet. How can two computers find each other on the Internet?

Let’s take a look at how the delivery is perfectly delivered to you. According to the address you fill in, the express will be sent to your province first, then find your house number in the current province, and finally according to the house number and name, and then personally deliver the express to you.

The same is true for communication between two computers over the Internet. You need to know the IP addresses of both parties, that is, the province and city, and then you need to know the MAC addresses, that is, the house number of both parties. A MAC address marks a unique computer. There may be several different services on the same computer. How do you find the service on the computer as the Courier finds you by name? Yes, by port number.

In this way, each computer in communication needs to provide information is clear, that is, IP address, MAC address, port number. To sum up, the six elements necessary for communication between computers are source IP address, port number, source MAC address, destination IP address, port number, and destination MAC address.

Suppose computers A and B are in the network topology above (not on the same LAN). It can be clearly seen that computer A and computer B need five steps to communicate, among which ①② and ④⑤ have the same principle. Now let’s look at how each of these steps is implemented in the computer world.

First, the IP addresses and port numbers of A and B are known, that is, where A packet is coming from and where it is going. So the question is: How can A know B’s MAC address?

The simplest way is for USER A to save the MAC addresses of all devices on the network and query the MAC addresses of user B before sending them. But with billions of devices on the network and growing, this is clearly not an option. If according to the process of delivery, transfer between each station, so that only need to know the next destination of the delivery, the delivery can be successfully delivered to your hand. The same is true for sending packets on a real network, where the goal now is to determine the MAC address of the “next destination.”

Step 1 and step 2: Gateway A sends data packets to gateway B without knowing the MAC address of gateway A and gateway B on different Lans. In this case, A sends A broadcast packet based on ARP, that is, A packet whose destination MAC address is FF:FF:FF:FF:FF. After receiving the broadcast packet, switch 1 forwards the packet to other ports and records the mapping between A’s MAC address and the switch port. After viewing the MAC address, switch 1 knows which port to forward the packet. After receiving the broadcast packet, Router 1 returns its MAC address. After receiving the return message, USER A knows the MAC address of the next destination. User A resends the packet, sets the destination MAC address to router 1’s MAC address, and records the returned MAC address in the local cache table.

Check the EXISTING MAC addresses in the cache table of the current device. The purpose of ARP-AARP is to find the MAC address of the “next destination”, that is, the mapping between IP addresses and MAC addresses. If the two devices are on the same LAN, Next Destination is the MAC address of the destination device. If the two devices are on different Lans, Next Destination is the MAC address of the gateway.

Step ③ : After steps ① and ②, Router 1 receives packets {A_IP, PORT, A_MAC} {B_IP, PORT, router 1_MAC}. After receiving the packet, Router 1 looks at its routing table, as shown in the following figure.

Run the route-n command to view the routing table of the current device

Router 1 performs a bitwise sum operation between B_IP and the Genmask of each record in the routing table. If the result obtained is the same as the Destination network, the MAC address of the next Destination is the MAC address of the configured Gateway. This method is called the next-hop mechanism. If the IP address of the gateway is 0.0.0.0, direct communication can be performed on the LAN without next hop. At this point, the MAC address of the “next destination” is found again, which is router 2’s MAC address. At this point, router 2 receives packets {A_IP, PORT, router 1_MAC} {B_IP, PORT, Router 2_MAC}. Steps ④ and ⑤ work in the same way as steps ① and ②, which will not be described here.

The purpose of the next hop is to find the next destination, that is, the destination of the next hop. Select the route, obtain the corresponding MAC address, and continue to transmit data packets.

 Sum up: 1. Use ARP to locate the MAC address of the gateway egress or device on the same LAN 2. Locate the MAC address of the gateway based on each rule in the routing table and the destination IP address. 3. The next destination is only one step adjacent to the current address, and the source MAC address and destination MAC address will be replaced after each jump. In data packets, IP address indicates the starting point and end point, MAC address indicates the jumping node, and port number indicates the corresponding application service. Of course, it is not enough to find each other, but also need an agreed communication mode. Various protocols that we are familiar with at ordinary times are all the “agreed communication mode” of computers.

LVS load balancing

As the number of devices using the Internet increases, the number of HTTP requests received by the server increases exponentially. Multiple servers are used to split requests when a single server cannot handle a very large volume of requests. The act of spreading requests across multiple servers is called load balancing.

UCloud Load Balancer (ULB) is a Load balancing service provided by UCloud. It can distribute traffic based on network packets or proxies for multiple hosts or other service instances. In a high concurrency service environment, a service cluster composed of multiple service nodes is constructed by ULB. Service cluster can expand service processing and fault tolerance capabilities, and automatically eliminate the impact of a single service node failure on the whole service, improve service availability. ULB supports HTTP and HTTPS (Nginx or HAPproxy) for Layer 7 protocols. Layer 4 protocols support TCP and UDP (LVS like).

From the point of view of computer communication in the network, rather than using a higher-level application (such as Nginx), the four-tier load balancer is made and the packet transmission link is CIP Trial VIP Trial DIP Trial RIP, i.e., client IP Trial virtual IP Trial distribution IP trial real server IP. For the client, it only needs to know that the requested arrival address is VIP, without considering the load, i.e. CIP VIP is fixed.

Therefore, what the load balancer needs to do is to send the CIP packet to the VIP, forward it to RIP through the DIP, and then return the response packet to CIP after the service responds.

NAT mode

Red indicates the sent data packets, green indicates the returned data packets, yellow indicates the modified content of the load balancer, dotted line indicates that the load balancer has passed N next hops, that is, the load balancer can not reside in the same LAN, and solid line indicates that the load balancer can only jump once, that is, the load balancer must reside in the same LAN.

1. After the computer sends a request packet to the load balancer, the load balancer converts the {destination IP address, port number, and MAC address} of the packet to {IP address of a real server, port number of a real server, and MAC address of a real server}. Other information remains unchanged. Destination network address translation (DNAT) only translates the information about the destination device of a packet without modifying the information about the source device.

2. The real server receives the request packet and returns the response packet as follows: {IP address of a real server, port number of the real server, MAC address of the real server} ➡️ {IP address, port number of the original request, and jump MAC address}. Therefore, you can view the TCP connection as CIP ➡️ RIP on the server.

3. The next destination of the packets returned by the real server must be the load balancer. If the returned data packet is directly returned to the client, the client discards the returned data packet if the source device information is inconsistent with the destination device information. So the default gateway of the real server must be the DIP, ensuring that the “next destination” of the returned packets is the load balancer.

4. After the returned data packet reaches the load balancer, the load balancer translates the IP address, port number, and jumping MAC address of the returned data packet into the destination IP address, port number, and destination MAC address of the original request packet. This method is called SOURCE network address translation (SNAT), which translates the source device information of a packet without modifying the target device information.

5. The load balancer returns the data packet to the client.

 Summarize the features of the NAT mode: 1. Change the Source IP address or Destination IP address of a packet to forward the port. 2. The default gateway of the real server must be a load balancer, so the real server and the load balancer must be on the same LAN 3. All request packets and response packets pass through the load balancer

FULLNAT mode

In NAT mode, load balancers and real servers must reside on the same LAN. However, real servers may reside on different network segments or even in different cities during development. How can THE NAT mode be applied when real servers are distributed on different network segments?

Red indicates the sent data packets, green indicates the returned data packets, yellow indicates the modified content of the load balancer, dotted line indicates that the load balancer has passed N next hops, that is, the load balancer can not reside in the same LAN, and solid line indicates that the load balancer can only jump once, that is, the load balancer must reside in the same LAN.

1. After the computer sends a request packet to the load balancer, the load balancer performs SNAT and DNAT on the request packet and modifies the request packet as follows: {Egress IP address, port number, MAC address of the load balancer} ➡️ {IP address of a real server, port number of a real service, MAC address of a real server}. 2. In this way, the load balancer can transmit data packets independently with the real server. 3. The real server receives the request data packet and returns the response data packet: {IP address of a real server, port number of the real server, MAC address of the real server} ➡️ {IP address, port number, MAC address of the load balancer}. In this case, the TCP connection is displayed as DIP ➡️ RIP on the real server. 4. After the returned data packet reaches the load balancer, the load balancer returns the data packet to perform DNAT and SNAT at the same time. 5. The load balancer returns the data packet to the client.

 Summarize the features of FULL NAT mode: 1. Change the Source IP address and destination IP address of packets to forward packets to the port. 2. A load balancer does not need to exist as a gateway, that is, a load balancer can reside on a different network from a real server. 3. The connection established by the real server is DIP ➡️ RIP, and the IP address of the real client cannot be obtained. All request packets and response packets pass through the load balancer

LVS does not support FULLNAT mode. You need to patch the kernel to use it.

You can see that in NAT and FULLNAT modes, both request packets and response packets pass through the load balancer. But response packets are typically much larger than request packets, which can be a bottleneck for the system. If the request packet can be forwarded to the real server and the response packet can be returned directly by the real server, the load balancer is much less stressful. This pattern should be repeated