On February 23, Beijing time, Xu Zhonghu, a core member of Huawei Cloud Native open Source team, delivered a keynote speech titled Preserve Original Source Address Within Istio at the world’s first community Summit IstioCon 2021.

The speech mainly consists of three parts:

  • Usage scenarios of source addresses

  • Common source address retention mode

  • Istio source address retention scheme

Usage scenarios of source addresses

Source addresses can be used in various scenarios. For example:

1) Simple session persistence. Load balancing can be performed based on the client address and source IP address to forward requests from the same client to the same service instance;

2) Used for security policy control, common blacklist and whitelist Settings;

3) Access logs and monitoring statistics. Access logs and monitoring indicators contain real source addresses, which are helpful for developers to analyze and Debug;

4) Typical technologies of IP telephony (VoIP) and streaming media Services SIP trunk linking is a way to differentiate users based on IP address. Source IP address is very important for SIP.

We can see that different technical scenarios depend on source IP addresses to varying degrees. In some highly dependent scenarios, Istio cannot be applied to those scenarios if source IP addresses cannot be maintained.

Common source IP address retention modes and background information

Before service grid technologies such as Istio, a typical service access model was shown in figure 1. Developers typically deploy application services behind enterprise load balancing devices to meet high concurrency and availability requirements. Enterprises use load balancers to provide unified access to services. Common load balancers include hardware LOAD balancer F5, software load balancer LVS, HaProxy, and Nginx. The modes for maintaining the source IP address provided by different load balancers vary.

The commonly used source IP retention modes from OSI network model L3-L7 are mainly divided into:

1. L3

LVS is the soft load of the kernel layer. In principle, it belongs to a single connection, with the highest performance among soft loads and the most widely used L3 load balancer in the industry. LVS work in NAT mode, DR mode, and tunnel mode. Let’s look at LVS source address retention primarily in NAT mode.

(1) When a user request arrives at the Director Server, the requested data packet will first go to the PREROUTING chain of the kernel space. In this case, the source IP address of the packet is CIP and the destination IP address is VIP.

(2) PREROUTING shows that the destination IP of the packet is the local machine, and the packet is sent to the INPUT chain. LVS mainly work in the INPUT chain.

(3) IPVS compares whether the service requested by the packet is a cluster service. If so, change the destination IP address of the packet to the IP address of the back-end server, and then send the packet to the POSTROUTING chain. In this case, the source IP address of the packet is CIP and the destination IP address is RIP. In this process, the destination IP address is translated (DNAT), and the source IP address remains unchanged.

(4) The POSTROUTING chain sends data packets to Real Server through route selection.

HAProxy transparent transport mode is also a source address preservation mode at L3, but it works in user space and is a typical two-connection model.

1) The user first establishes a connection with HAProxy, and when the user request reaches HAProxy, HAProxy obtains the IP address of the client.

2) HAProxy works in transparent transmission mode, selects a real back-end service according to the load balancing policy, and then binds the IP address of the client to establish a new connection with the back-end service. Transparent proxy depends on the IP_TRANSPARENT of the socket

3) Real server response packets must pass through HAProxy proxy, so transparent transmission still depends on the configuration of personalized routing and TPROXY.

2. L4

TOA: The TOA plug-in obtains the real source IP address based on the Layer 4 protocol (TCP). Essentially, the TOA plug-in inserts the source IP address into the Options field of the TCP protocol. Huawei cloud provides the TOA plug-in. After installing the TOA module in the kernel, users can obtain the source IP address through system call.

Proxy Protocol is another four-tier source address retention method. The Protocol standard was first developed by the HAProxy community. The working principle is very simple. It requires the client to send a Proxy Protocol header after the TCP connection is established and before the application data is sent. The server can obtain the real CLIENT IP address by parsing the Protocol header. The Proxy Protocol standard requires both the client and server to support the Proxy Protocol. Here the Client in the real world can be Envoy, HAProxy, Nginx, and other load balancers that support proxy protocols.

3. L7

This method is the simplest. The most representative one is the HTTP header X-Forwarded-For. X-forwarded-for contains the IP address of the original user.

In addition, developers can customize the protocol type by inserting the client’s source IP address into the protocol frame.

The Istio source address is stuck

Let’s look at the barriers to source address acquisition in the current Istio service grid from the east-west and north-south perspectives of the service grid.

1) When the service grid visits the service from east to west, due to the injection of Sidecar, all traffic in and out of the service is intercepted by an Envoy proxy. When svcA visits svcB, the Sidecar Envoy of svcB establishes a connection with svcB with the source address 127.0.0.1. The destination address is also 127.0.0.1 (PodIP in the future). From the perspective of TCP communication, svcB can only obtain the client address 127.0.0.1.

Intra-cluster service accessCopy the code

Ingress service accessCopy the code

2) In the north-south service access of Istio, user requests are first forwarded to the Ingress Gateway of Istio through the ELB service proxy, and then the Ingress Gateway balances the load of requests to real service instances. As can be seen from the figure, there are more network nodes in the middle, including ELB and Istio. Source addresses are more difficult to maintain.

How do I get source addresses in the Istio service grid

Let’s take a look at some of the basic capabilities associated with source persistence in the Envoy and then try to leverage these basic capabilities to implement an end-to-end source persistence solution in an Istio service grid.

Envoy provides two listeners level filters: source address filter (” Envoy. Filters. The listener. Original_src “) and filter agency agreement (” Envoy. Filters. The listener. Proxy_protocol “).

1) The source address filter acts on the Envoy Inbound listener. It works by retrieving the client address of the downstream connection and establishing a new connection with the upstream service using the client address as the source address. It is IP_TRANSPARENT in nature, which is very important for obtaining TCP source addresses. We mentioned earlier that transparent transport must also address IP packet backhaul, which we will leave behind.

2) The Proxy Protocol filter also acts on the listener and is mainly responsible for parsing the Proxy Protocol. It can obtain the source address from the server, especially suitable for the Ingress service.

Proxy Protocol Transport Socket is an Envoy’s approach to maintain source IP addresses with upstream proxies through Proxy Protocol. It constructs Proxy Protocol headers and proactively sends them to upstream Proxy Protocol supporting services.

HTTP XFF (X-Forwarded-For) automatically adds the IP address of the nearest downstream client to the XFF according to the LDS HTTP connection management configuration.

1. Retain the source IP address of the HTTP protocol

Therefore, it can be seen that the east-West layer 7 HTTP protocol can easily add XFF to facilitate the server to obtain the real IP address of the client. The typical configuration is as follows:

                                                    Ingress HTTP
Copy the code

In the Ingress service access scenario, ELB provides two schemes: TOA and Proxy Protocol. In TOA mode, you must install the TOA kernel module on the node where the Ingress Gateway resides so that the Ingress Gateway can obtain the original client IP address. In the Proxy Protocol scenarios, must through configuration envoy. Filters. The listener. Proxy_protocol filter can make Ingress IP Gateway access to real customers. The detailed configuration is as follows. Of course, this filter must be enabled with great care. Ensure that all traffic entering the Ingress Gateway is the Proxy Protocol.

Two modes, all need to configure the Ingress of gateway to support XFF automatically add, also can pass xff_num_trusted_hops additional configuration Envoy trust agent jump number, detailed reference www.envoyproxy.io/docs/envoy/…

2. The TCP source IP address remains

Istio TCP source address persistence must rely on transparent transport and TPROXY, a feature I didn’t fully support until Istio 1.9. The method is also very easy to use, the user simply sets the Annotation at the Pod sidecars. Istio. IO/interceptionMode: TPROXY.

The TCP source address in the cluster remainsCopy the code

Istio ensures communication between the svcB and Sidecar Envoy via the svcA’s address through the following configuration.

1) Configure a source address filter, Envoy can bind the downstream client’s address (10.244.0.20) svcB to establish connections.

2) Configure TPROXY and personalized routing to ensure that packets sent by svcB to 10.244.0.20 are received Envoy. Packets are marked and non-local packets are routed through local LO based on user-defined routes.

`-A PREROUTING -p tcp -m mark –mark 0x539 -j CONNMARK –save-mark –nfmask 0xffffffff –ctmask 0xffffffff

-A OUTPUT -p tcp -m connmark –mark 0x539 -j CONNMARK –restore-mark –nfmask 0xffffffff –ctmask 0xffffffff

ip -f inet rule add fwmark 1337 lookup 133

ip -f inet route add local default dev lo table 133 `

Ingress TCP source address retentionCopy the code

Ingress TCP source address retention is more complex. In addition to the preceding rules, you need to add the Proxy Protocol. Configure Proxy Protocol on the Ingress Gateway.

Configuration on the svcB envoy. Filters. The listener. Proxy_protocol, guarantee the envoy listener IP access to the original client.

This article summary

Source IP persistence is not a new topic, however we have seen many mature solutions fail in the Istio service grid. In the end, it has a lot to do with the Istio Sidecar agent. The Istio Sidecar auto-injection directly causes the source address retention problem to be solved by personalization. A unified configuration scheme is implemented in combination with Istio configuration management.

This article is based on my sharing in IstioCon 2021. Related case demos can be found at github.com/hzxuzhonghu… Find, hope to inspire you.

Welcome to Huawei cloud native team, we will provide you with:

We update the cloud native technology dynamics, practical progress, application cases and so on every day;

Join the group and the industry technology champion, 10,000 + cloud native lovers link to learn together;

Irregularly invite cloud OG level technology to share technical combat

…………………