By Jimmy Song
IO /blog/ Sideca…
Based on Istio 1.5.1, this article introduces the following:
- What is the Sidecar model and what are its advantages?
- How is sidecar injection done in Istio?
- How does Sidecar Proxy do transparent traffic hijacking?
- How is traffic routed to upstreams?
Previously I wrote about understanding the Envoy Proxy Sidecar injection and traffic hijacking in the Istio Service Mesh based on Istio 1.1, The biggest changes to SIDecar injection and traffic hijacking in Istio 1.5 and Istio 1.1 are:
- Iptables uses command-line tools instead of shell scripts.
- Sidecar inbound and outbound specify ports respectively, whereas the same port (15001) was used before.
Note: Some of the content of this article was included in the Istio Handbook produced by ServiceMesher community.
Sidecars mode
Dividing the functionality of an application into separate processes running in the same minimum scheduling unit (such as Pod in Kubernetes) can be considered sidecar mode. As shown in the figure below, sidecar mode allows you to add more features next to your application without additional third-party component configuration or application code modification.
Like a three-wheeled motorcycle connected to a Sidecar, in a software architecture, a Sidecar connects to a parent application and adds extensions or enhancements to it. Sidecar applications are loosely coupled to the main application. It can shield the differences of different programming languages and realize the functions of observation, monitoring, logging, configuration and circuit breaker of microservices in a unified manner.
Advantages of using the Sidecar pattern
When deploying the service grid using sidECar mode, there is no need to run agents on nodes, but multiple copies of the same Sidecar will be running in the cluster. In sidecar deployment, each application deploys a companion container (such as an Envoy or MOSN) next to its container, called the Sidecar container. Sidecar takes over all traffic going in and out of the application container. In The Pod of Kubernetes, a Sidecar container is injected next to the original application container. The two containers share storage, network and other resources. In a broad sense, this Pod containing Sidecar container can be understood as a host, and the two containers share host resources.
Due to its unique deployment structure, the SIDECAR pattern has the following advantages:
- Abstracting functionality unrelated to application business logic into a common infrastructure reduces the complexity of microservice code.
- Because you no longer need to write the same third-party component configuration files and code, you can reduce code duplication in the microservices architecture.
- Sidecar can be upgraded independently to reduce coupling between application code and the underlying platform.
Sidecar injection in Istio
The following two sidecar injection modes are provided in Istio:
- use
istioctlManual injection. - Automatic Sidecar injection method based on Kubernetes mutating Webhook AddMission Controller.
Regardless of manual or automatic injection, sidecar injection must follow the following steps:
- Kubernetes needs to know the Istio cluster connected to the sidecar to be injected and its configuration.
- Kubernetes needs to know the configuration of the sidecar container to be injected, such as the mirror address and startup parameters.
- Kubernetes fills the sidecar configuration parameters according to the sidecar injection template and the above configuration, and injects the above configuration into the side of the application container.
You can manually inject sidecar using the following command.
istioctl kube-inject -f ${YAML_FILE} | kuebectl apply -f -
Copy the codeThis command is injected using the built-in sidecar configuration of Istio. For details about how to use Istio, see the Istio official website.
After the injection is complete, you will see that Istio has injected the configuration related to initContainer and Sidecar Proxy into the original pod template.
Init container
The Init container is a special container that runs before the application container starts to contain utilities or installation scripts that do not exist in the application image.
Multiple Init containers can be specified in a Pod, and if multiple are specified, the Init containers will be run in sequence. You can run the next Init container only if the current Init container must run successfully. Kubernetes initializes the Pod and runs the application container only after all Init containers have been run.
The Init container uses Linux Namespace, so it has a different view of the file system than the application container. Therefore, they can have access to Secret, whereas the application container cannot.
During Pod startup, the Init container starts sequentially after the network and data volumes are initialized. Each container must exit successfully before the next container starts. If exit due to runtime or failure causes the container to fail to start, it will retry according to the policy specified by the Pod’s restartPolicy. However, if the Pod restartPolicy is set to Always, the Init container will use the restartPolicy when it fails.
The Pod will not become Ready until all Init containers have succeeded. The ports of the Init container will not be clustered in the Service. The Pod being initialized is in a Pending state, but Initializing should be set to true. The Init container terminates automatically when it finishes running.
Please refer to Init Container – Kubernetes Chinese Guide/Cloud Native Application Architecture Practice Manual for more information about Init container.
Sidecar injection sample analysis
Take the YAML of ProductPage in bookinfo provided by Istio as an example. For details about the YAML configuration of bookinfo application, see Bookinfo.yaml.
The following will be explained from the following aspects:
- Injection of the Sidecar container
- Creating iptables rules
- Detailed routing process
apiVersion: apps/v1
kind: Deployment
metadata:
name: productpage-v1
labels:
app: productpage
version: v1
spec:
replicas: 1
selector:
matchLabels:
app: productpage
version: v1
template:
metadata:
labels:
app: productpage
version: v1
spec:
serviceAccountName: bookinfo-productpage
containers:
- name: productpage
image: Docker. IO/istio/examples - bookinfo productpage - v1:1.15.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
volumeMounts:
- name: tmp
mountPath: /tmp
volumes:
- name: tmp
emptyDir: {}
Copy the codeLook again at the ProductPage container Dockerfile.
FROM python:3.7.4-slim
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
COPY test-requirements.txt ./
RUN pip install --no-cache-dir -r test-requirements.txt
COPY productpage.py /opt/microservices/
COPY tests/unit/* /opt/microservices/
COPY templates /opt/microservices/templates
COPY static /opt/microservices/static
COPY requirements.txt /opt/microservices/
ARG flood_factor
ENV FLOOD_FACTOR ${flood_factor:-0}
EXPOSE 9080
WORKDIR /opt/microservices
RUN python -m unittest discover
USER 1
CMD ["python"."productpage.py"."9080"]
Copy the codeWe can see that ENTRYPOINT is not configured in Dockerfile, so the CMD configuration python productPage.py 9080 will be used as the default ENTRYPOINT. With that in mind, let’s look at the configuration after sidecar injection.
$ istioctl kube-inject -f samples/bookinfo/platform/kube/bookinfo.yaml
Copy the codeWe only capture part of the YAML configuration in the Deployment configuration associated with ProductPage.
containers:
- image: Docker. IO/istio/examples - bookinfo productpage - v1:1.15.0 # Application mirroring
name: productpage
ports:
- containerPort: 9080
- args:
- proxy
- sidecar
- --domain
- $(POD_NAMESPACE).svc.cluster.local
- --configPath
- /etc/istio/proxy
- --binaryPath
- /usr/local/bin/envoy
- --serviceCluster
- productpage.$(POD_NAMESPACE)
- --drainDuration
- 45s
- --parentShutdownDuration
- 1m0s
- --discoveryAddress
- istiod.istio-system.svc:15012
- --zipkinAddress
- zipkin.istio-system:9411
- --proxyLogLevel=warning
- --proxyComponentLogLevel=misc:error
- --connectTimeout
- 10s
- --proxyAdminPort
- "15000"
- --concurrency
- "2"
- --controlPlaneAuthPolicy
- NONE
- --dnsRefreshRate
- 300s
- --statusPort
- "15020"
- --trust-domain=cluster.local
- --controlPlaneBootstrap=false
image: Docker. IO/istio proxyv2:1.5.1 # sidecar proxy
name: istio-proxy
ports:
- containerPort: 15090
name: http-envoy-prom
protocol: TCP
initContainers:
- command:
- istio-iptables
- -p
- "15001"
- -z
- "15006"
- -u
- "1337"
- -m
- REDIRECT
- -i
- The '*'
- -x
- ""
- -b
- The '*'
- -d
- 15090, 15020,
image: Docker. IO/istio proxyv2:1.5.1 # init container
name: istio-init
Copy the codeIstio provides the following configurations for application PODS:
- Init container
istio-init: Used to set iptables port forwarding in POD - Sidecars container
istio-proxy: Runs the Sidecar agent, for exampleEnvoy 或 MOSN
Next, we’ll parse the two containers separately.
Init container parsing
The Init container Istio injects into pod is called istio-init. We can see in the YAML file above that Istio injects is Init container.
istio-iptables -p 15001 -z 15006 -u 1337 -m REDIRECT -i The '*' -x "" -b The '*' -d 15090,15020
Copy the codeLet’s examine the container’s Dockerfile again to see how ENTRYPOINT determines the command to execute on startup.
# The preceding content is omitted
# The pilot-agent will bootstrap Envoy.
ENTRYPOINT ["/usr/local/bin/pilot-agent"]
Copy the codeThe istio-init container entry is the /usr/local/bin/istio-iptables command line. The code for this command line tool is located in the tools/istio-iptables directory of the ISTIO source repository.
Note: In Istio 1.1, iptables is still operated using the isito-iptables.sh command line.
Init container startup entry
Init starts with the istio-iptables command line. This command line tool can be used as follows:
$istio-iptables [flags] -p: specifies the sidecar port for redirecting all TCP traffic (default$ENVOY_PORT= 15001) -m: specifies the mode for inbound connections to be redirected to sidecar, "REDIRECT" or "TPROXY" (default$ISTIO_INBOUND_INTERCEPTION_MODE) -b: Comma-separated list of inbound ports whose traffic will be redirected to the Envoy (optional). Use wildcard * to redirect all ports. Null disables all inbound redirection (default$ISTIO_INBOUND_PORTS)-d: Specifies a comma-separated list of optional inbound ports to exclude from redirects to sidecar. Use the wildcard '*' to redirect all inbound traffic (default$ISTIO_LOCAL_EXCLUDE_PORTS) -o: comma-separated list of outbound ports, excluding ports redirected to Envoy. -i: Specifies the IP address range (optional) to be redirected to sidecar. It is a comma-separated LIST in CIDR format. Use the wildcard * to redirect all outbound traffic. An empty list disables all outbound redirection (default$ISTIO_SERVICE_CIDR) -x: Specifies the range of IP addresses to exclude from redirection, a comma-separated list in CIDR format. Use the wildcard '*' to redirect all outbound traffic (default$ISTIO_SERVICE_EXCLUDE_CIDR). -k: indicates the list of virtual interfaces separated by commas. The inbound traffic (from VMS) is regarded as the outbound traffic. -g: Specifies the GID of the user that does not apply redirection. (The default value is the same as that of -u param.) -u: specifies the UID of the user that does not apply redirection. Typically, this is the UID of the proxy container (the default is 1337, which is the UID of isTIO-proxy). -z: The port to which all TCP traffic going to pod/VM should be redirected (default)$INBOUND_CAPTURE_PORT= 15006).Copy the codeAll the above parameters are reassembled into iptables rules. For details on how to use this command, visit tools/istio-iptables/ PKG/CMD /root.go.
This container exists so that the Sidecar agent can intercept all traffic going in and out of the POD, All inbound traffic except ports 15090 (used by Mixer) and 15092 (Ingress Gateway) was redirected to port 15006 (Sidecar). Interception Outbound traffic of the application container is processed by sidecAR (listening through port 15001) and then outbound. For details about the port usage in Istio, see the Istio official documentation.
The command parsing
This startup command does the following:
- Forward all traffic from the application container to port 15006 of sidecar.
- use
istio-proxyRunning as a user, UID 1337, which is the user space sidecar is in, this is alsoistio-proxyThe default user for the container is shown in the YAML configurationrunAsUserField. - Use the default
REDIRECTMode to redirect traffic. - Redirect all outbound traffic to the Sidecar agent (through port 15001).
Because the Init container automatically terminates after it’s initialized, because we can’t log in to the container to see the iptables information, but the Init container initialization results will stay in the application container and sidecar container.
Iptables injection parsing
To see the iptables configuration, we need to log in to the sidecar container as root, because Kubectl cannot use privileged mode to remotely manipulate the Docker container. Therefore, we need to log in to the productPage pod host and use docker command to log in the container to check.
If you are using Kubernetes deployed by Minikube, you can directly log in to the VIRTUAL machine of Minikube and switch to user root. Look at the iptables configuration and list all the rules for the NAT table, because the parameter selected to istio-iptables when Init starts specifies the REDIRECT mode for inbound traffic to sidecar. Therefore, in iptables, only NAT table specifications are configured, and if TPROXY is selected, mangle table specifications are configured. For details about the usage of the iptables command, see the iptables command.
We only look at the iptables rules associated with productPage as follows.
Go to minikube and switch to user root. The default user is docker
$ minikube ssh
$ sudo -i
Productpage Pod istio-proxy container
$ docker top `docker ps|grep "istio-proxy_productpage"|cut -d "" -f1`
UID PID PPID C STIME TTY TIME CMD
1337 10576 10517 0 08:09 ? 00:00:07 /usr/local/bin/pilot-agent proxy sidecar --domain default.svc.cluster.local --configPath /etc/istio/proxy --binaryPath /usr/local/bin/envoy --serviceCluster productpage.default --drainDuration 45s --parentShutdownDuration 1m0s --discoveryAddress istiod.istio-system.svc:15012 --zipkinAddress zipkin.istio-system:9411 --proxyLogLevel=warning --proxyComponentLogLevel=misc:error --connectTimeout 10s --proxyAdminPort 15000 --concurrency 2 --controlPlaneAuthPolicy NONE --dnsRefreshRate 300s --statusPort 15020 --trust-domain=cluster.local --controlPlaneBootstrap=false1337 10660 10576 0 08:09? 00:00:33 /usr/local/bin/envoy -c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s60 --service-cluster productpage.default --service-node Sidecars 172.17.0.16 ~ ~ productpage - v1-7 f44c4d57c - ksf9b. Default to default. SVC. Cluster. The local - Max - obj - name - 189 len --local-address-ip-version v4 --log-format [Envoy (Epoch 0)] [%Y-%m-%d %T.%e][%t][%l][%n] %v-l warning --component-log-level misc:error --concurrency 2
Enter the sidecar namespace (any of the above will do)
$ nsenter -n --target 10660
Copy the codeView the chain of iptables rules in the process’s namespace.
# check NAT table for rule configuration details.
$ iptables -t nat -L -v
# PREROUTING chain: Used for destination address translation (DNAT) to jump all inbound TCP traffic onto the ISTIO_INBOUND chain.
Chain PREROUTING (policy ACCEPT 2701 packets, 162K bytes)
pkts bytes target prot opt in out source destination
2701 162K ISTIO_INBOUND tcp -- any any anywhere anywhere
# INPUT chain: Processing INPUT packets, non-TCP traffic will continue to the OUTPUT chain.
Chain INPUT (policy ACCEPT 2701 packets, 162K bytes)
pkts bytes target prot opt in out source destination
# OUTPUT chain: Jump all outbound packets to ISTIO_OUTPUT chain.
Chain OUTPUT (policy ACCEPT 79 packets, 6761 bytes)
pkts bytes target prot opt in out source destination
15 900 ISTIO_OUTPUT tcp -- any any anywhere anywhere
# POSTROUTING chain: All packets leaving the NETWORK card must enter the POSTROUTING chain first. The kernel determines whether to forward the packets according to their destination. We see that no processing is done here.
Chain POSTROUTING (policy ACCEPT 79 packets, 6761 bytes)
pkts bytes target prot opt in out source destination
# ISTIO_INBOUND chain: Redirect all inbound traffic to the ISTIO_IN_REDIRECT chain, except traffic destined for ports 15090 (used by Mixer) and 15020 (used by Ingress Gateway for Pilot health checks), Traffic sent to the above two ports will return to the call point of the IPtables rule chain, which is POSTROUTING, a successor to the PREROUTING chain.
Chain ISTIO_INBOUND (1 references)
pkts bytes target prot opt in out source destination
0 0 RETURN tcp -- any any anywhere anywhere tcp dpt:ssh
2 120 RETURN tcp -- any any anywhere anywhere tcp dpt:15090
2699 162K RETURN tcp -- any any anywhere anywhere tcp dpt:15020
0 0 ISTIO_IN_REDIRECT tcp -- any any anywhere anywhere
# ISTIO_IN_REDIRECT chain: redirect all inbound traffic to local 15006 port, successfully intercepting traffic to sidecar.
Chain ISTIO_IN_REDIRECT (3 references)
pkts bytes target prot opt in out source destination
0 0 REDIRECT tcp -- any any anywhere anywhere redir ports 15006
# ISTIO_OUTPUT chain: Select outbound traffic that needs to be redirected to the Envoy (i.e. local) and all non-localhost traffic is forwarded to the ISTIO_REDIRECT. To avoid an infinite loop of traffic in this Pod, all traffic to the IStiO-proxy user space is returned to the next rule at its call point, which in this case is the OUTPUT chain, because breaking out of the ISTIO_OUTPUT rule leads to the next chain POSTROUTING. ISTIO_REDIRECT if the destination is not localhost; If the traffic is from the IStio-proxy user space, then it breaks out of the chain and returns to its call chain to proceed to the next rule (the next rule of OUTPUT, without processing the traffic); All non-ISTIO-proxy user space traffic destined to localhost is redirected to ISTIO_REDIRECT.
Chain ISTIO_OUTPUT (1 references)
pkts bytes target prot opt in out sourceDestination 0 0 RETURN all -- any LO 127.0.0.6 anywhere 0 0 ISTIO_IN_REDIRECT all -- any lo anywhere! localhost owner UID match 1337 0 0 RETURN all -- any lo anywhere anywhere ! owner UID match 1337 15 900 RETURN all -- any any anywhere anywhere owner UID match 1337 0 0 ISTIO_IN_REDIRECT all -- any lo anywhere ! localhost owner GID match 1337 0 0 RETURN all -- any lo anywhere anywhere ! owner GID match 1337 0 0 RETURN all -- any any anywhere anywhere owner GID match 1337 0 0 RETURN all -- any any anywhere localhost 0 0 ISTIO_REDIRECT all -- any any anywhere anywhere# ISTIO_REDIRECT chain: Redirect all traffic to Sidecar (local) port 15001.
Chain ISTIO_REDIRECT (1 references)
pkts bytes target prot opt in out source destination
0 0 REDIRECT tcp -- any any anywhere anywhere redir ports 15001
Copy the codeThe figure below shows the productpage service request to http://reviews.default.svc.cluster.local:9080/, when flow into the internal reviews service, How the Sidecar proxy inside the Reviews service does traffic interception and routing.
When the first step starts, the Sidecar in ProductPage Pod has selected a Pod from the Reviews service to request by EDS, knows its IP address, and sends a TCP connection request.
There are three versions of the Reviews service, and each version has an instance. Sidecar works in a similar way in all three versions. The sidecar traffic forwarding steps in one Pod are described below.
Understand the iptables
Iptables is a management tool of the firewall software NetFilter in the Linux kernel. It is located in the user space and is also a part of NetFilter. Located in the kernel space, Netfilter not only has the function of network address translation, but also has the firewall functions such as packet content modification and packet filtering.
Before we look at iptables initialized by the Init container, let’s look at iptables and rule configuration.
The following figure shows the iptables call chain.
Iptables in the table
The iptables version used in the Init container is V1.6.0 and contains 5 tables:
rawUsed to configure packets,rawThe data packets are not tracked by the system.filterIs the default table for all firewall-related operations.natUsed forNetwork address translation(For example, port forwarding).mangleUsed to modify specific packets (seeCorrupt packet).securityUsed forMandatory access controlNetwork rules.
Note: Only the NAT table is used in this example.
The chain types in different tables are as follows:
| Rule name | raw | filter | nat | mangle | security |
|---|---|---|---|---|---|
| PREROUTING | ✓ | ✓ | ✓ | ||
| INPUT | ✓ | ✓ | ✓ | ✓ | |
| OUTPUT | ✓ | ✓ | ✓ | ✓ | |
| POSTROUTING | ✓ | ✓ | |||
| FORWARD | ✓ | ✓ | ✓ | ✓ |
For details about iptables, see Common Iptables Usage Rule Scenarios.
Understand the iptables rules
View the default iptables rules in the IStio-proxy container. By default, the iptables rules in the Filter table are displayed.
$ iptables -L -v
Chain INPUT (policy ACCEPT 350K packets, 63M bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 18M packets, 1916M bytes)
pkts bytes target prot opt in out source destination
Copy the codeWe see three default chains, INPUT, FORWARD, and OUTPUT, with the first line of OUTPUT in each chain representing the chain name (INPUT/FORWARD/OUTPUT in this case) followed by the default policy (ACCEPT).
The following diagram shows the proposed structure of Iptables, where traffic passes through the INPUT chain and then into the upper stack, for example
Multiple rules can be added to each chain, and the rules are executed from front to back. Let’s look at the header definition of a rule.
- PKTS: indicates the number of matched packets processed
- Bytes: Indicates the total size of packets processed.
- Target: If the packet matches the rule, the specified target is executed.
- prot: agreement, for example
tdp,udp,icmp和all. - Opt: rarely used. This column is used to display IP options.
- In: indicates an inbound nic.
- Out: indicates the outbound network adapter.
- source: Indicates the source IP address or subnet of the traffic
anywhere. - destination: Indicates the destination IP address or subnet of the traffic, or is
anywhere.
There is also a column without a table header that is displayed at the end, representing the options for the rule as an extended matching condition for the rule that complements the configuration in the previous columns. Prot, opt, in, OUT, source, and Destination together make up the matching rule, along with a list of extended conditions without a table header displayed after destination. Target is executed when the traffic matches these rules.
For details about iptables rules, see Common Iptables Usage Rule Scenarios.
The type supported by target
The target type can be ACCEPT, REJECT, DROP, LOG, SNAT, MASQUERADE, DNAT, REDIRECT, RETURN, or REDIRECT. The direction of the packet can be determined as long as it is executed in a certain chain and only one rule matches in sequence, except for the RETURN type, which is similar to the RETURN statement in programming languages. It returns to its call point and continues to execute the next rule. For details about the configurations supported by target, see Iptables (1) : Iptables concepts.
You can see from the output that the Init container does not create any rules in the default links of iptables, but instead creates new links.
Description of the traffic routing process
Traffic routing is divided into Inbound and Outbound processes. The following describes the process based on the previous example and sidecar configuration.
Understand the Inbound Handler
The Inbound handler forwards the downstream traffic intercepted by iptables to a localhost and establishes a connection with the APPLICATION container in the Pod. Suppose one of the pods is named review-V1-54B8794DDF-jxksn, Run istioctl proxy-config listener reviews-v1-54b8794DDf-jxksn to check what listeners are in the Pod.
ADDRESS PORT TYPE 172.17.0.15 9080 HTTP <-- Receive all Inbound HTTP traffic, 172.17.0.15 15020 TCP <-- Ingress Gateway, Pilot health check 10.109.20.166 "- Istiod HTTP, TCP 15012 DNS 10.103.34.135 TCP < 14250-14267 TCP + 10.103.34.135 | 10.103.34.135 TCP 14268 | 15020 TCP 10.104.122.175 | 10.104.122.175 TCP 15029 | 15030 TCP 10.104.122.175 | 10.104.122.175 TCP 15031 | 15032 TCP 10.104.122.175 | 10.104.122.175 TCP 15443 | 10.104.122.175 TCP | receiving and 31400 0.0.0.0:15006 listeners pairing of Outbound traffic 10.104.122.175 TCP 443 | 15443 TCP 10.104.62.18 | 10.104.62.18 TCP 443 | 10.106.201.253 TCP 16686 | 443 TCP 10.109.20.166 | 10.96.0.1 TCP 443 | 10.96.0.10 53 TCP | 10.96.0.10 TCP 9153 | 10.98.184.149 15011 TCP | | 10.98.184.149 15012 TCP 10.98.184.149 TCP 443 | 14250 TCP 0.0.0.0 | 0.0.0.0 TCP 15010 | 15014 TCP | 0.0.0.0 0.0.0.0 HTTP 15090 | 20001 TCP 0.0.0.0 | 0.0.0.0 TCP 3000 | 80 TCP 0.0.0.0 | 0.0.0.0 TCP 8080 | 9080 TCP | 0.0.0.0 0.0.0.0 TCP 9090 | 0.0.0.0 TCP < 9411-15001 TCP + 0.0.0.0 < - receive all the iptables interception of Outbound traffic and convert to virtual listener 0.0.0.0 15006 TCP <-- receives all Inbound traffic intercepted by Iptables and forwards it to the virtual listener for processingCopy the codeWhen the traffic from ProductPage arrives at Reviews Pod, the downstream has already explicitly known that the IP address of Pod is 172.17.0.16 so it will visit the Pod, so the request is 172.17.0.15:9080.
virtualInbound Listener
As shown in the Pod Listener list, 0.0.0.0:15006/TCP Listener (whose actual name is virtualInbound) listens for Inbound traffic.
{
"name": "virtualInbound"."address": {
"socketAddress": {
"address": "0.0.0.0"."portValue": 15006}},"filterChains": [{"filters": [/* omit some content */ {"filterChainMatch": {
"destinationPort": 9080."prefixRanges": [{"addressPrefix": "172.17.0.15"."prefixLen": 32}]."applicationProtocols": [
"istio-peer-exchange"."istio"."Istio - HTTP / 1.0"."Istio - HTTP / 1.1"."istio-h2"]},"filters": [{"name": "envoy.filters.network.metadata_exchange"."config": {
"protocol": "istio-peer-exchange"}}, {"name": "envoy.http_connection_manager"."typedConfig": {
"@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager"."statPrefix": "Inbound_172. 17.0.15 _9080"."routeConfig": {
"name": "inbound|9080|http|reviews.default.svc.cluster.local"."virtualHosts": [{"name": "inbound|http|9080"."domains": [
"*"]."routes": [{"name": "default"."match": {
"prefix": "/"
},
"route": {
"cluster": "inbound|9080|http|reviews.default.svc.cluster.local"."timeout": "0s"."maxGrpcTimeout": "0s"
},
"decorator": {
"operation": "reviews.default.svc.cluster.local:9080/*"}}]}],"validateClusters": false} /* omit some content */}Copy the codeThe traffic of the Inbound handler is forwarded to 172.17.0.15_9080 Listener by the virtualInbound Listener. Check the Listener configuration.
Run istioctl PC listener reviews-v1-54b8794ddf-jxksn –address 172.17.0.15 –port 9080 -o json.
[{" name ":" 172.17.0.15 _9080 ", "address" : {" socketAddress ": {" address" : "172.17.0.15", "portValue" : 9080 } }, "filterChains": [ { "filterChainMatch": { "applicationProtocols": [" istio - peer - exchange ", "istio", "istio - HTTP / 1.0", "istio - HTTP / 1.1", "istio - h2]}," filters ": [{" name" : "envoy.http_connection_manager", "config": { ... "routeConfig": { "name": "inbound|9080|http|reviews.default.svc.cluster.local", "virtualHosts": [ { "name": "inbound|http|9080", "domains": [ "*" ], "routes": [ { "name": "default", "match": { "prefix": "/" }, "route": { "cluster": "inbound|9080|http|reviews.default.svc.cluster.local", "timeout": "0s", "maxGrpcTimeout": "0s" }, "decorator": { "operation": "reviews.default.svc.cluster.local:9080/*" } } ] } ], } ... }, { "filterChainMatch": { "transportProtocol": "tls" }, "tlsContext": {... }, "filters": [... ] } ], ... }]Copy the codeIn filterChains. Filters, envoys. Http_connection_manager configuration The configuration said traffic will be passed on to Clusterinbound | | 9080 | HTTP reviews. The default. SVC. Cluster. The local processing.
Cluster inbound|9080|http|reviews.default.svc.cluster.local
Run istioctl proxy – config cluster reviews – v1-54 b8794ddf – JXKSN – FQDN reviews. The default. SVC. Cluster. The local – direction Inbound-o json View the Cluster configuration as follows:
[{"name": "inbound|9080|http|reviews.default.svc.cluster.local"."type": "STATIC"."connectTimeout": "1s"."loadAssignment": {
"clusterName": "inbound|9080|http|reviews.default.svc.cluster.local"."endpoints": [{"lbEndpoints": [{"endpoint": {
"address": {
"socketAddress": {
"address": "127.0.0.1"."portValue": 9080}}}}]}]},"circuitBreakers": {
"thresholds": [{"maxConnections": 4294967295."maxPendingRequests": 4294967295."maxRequests": 4294967295."maxRetries": 4294967295}]}}]Copy the codeYou can see that the Endpoint of the Cluster directly corresponds to localhost, and the traffic is then forwarded by iptables and consumed by the application container.
Understand the Outbound Handler
Because Reviews sends an HTTP request to the Ratings service, the address of the request is: Is the role of http://ratings.default.svc.cluster.local:9080/, Outbound handler iptables intercepted traffic from the local application, Determine how to route to upstream via sidecar.
The request sent by the application container is Outbound traffic, which is hijacked by iptables and forwarded to the Outbound Handler for processing. Then the request passes the virtualOutbound Listener, 0.0.0.0_9080 Listener, Route 9080 is used to locate the cluster of upstream, and EDS is used to locate the Endpoint to perform the routing action.
Route ratings.default.svc.cluster.local:9080
Run istioctl proxy-config routes reviews-v1-54b8794ddf-jxksn –name 9080 -o json to check route configuration. Because sidecars can according to the domains in the HTTP header to match the VirtualHost, so the following only lists the ratings. The default. SVC. Cluster. The local: 9080 a VirtualHost.
[{ { "name": "ratings.default.svc.cluster.local:9080", "domains": [ "ratings.default.svc.cluster.local", "ratings.default.svc.cluster.local:9080", "ratings", "ratings:9080", "ratings.default.svc.cluster", "ratings.default.svc.cluster:9080", "ratings.default.svc", "ratings.default.svc:9080", "Ratings. The default" and "ratings. Default: 9080", "10.98.49.62", "10.98.49.62:9080]," "routes" : [{" name ": "default", "match": { "prefix": "/" }, "route": { "cluster": "outbound|9080||ratings.default.svc.cluster.local", "timeout": "0s", "retryPolicy": { "retryOn": "connect-failure,refused-stream,unavailable,cancelled,resource-exhausted,retriable-status-codes", "numRetries": 2, "retryHostPredicate": [ { "name": "envoy.retry_host_predicates.previous_hosts" } ], "hostSelectionRetryMaxAttempts": "5", "retriableStatusCodes": [ 503 ] }, "maxGrpcTimeout": "0s" }, "decorator": { "operation": "ratings.default.svc.cluster.local:9080/*" } } ] }, ..]Copy the codeCan see from the Virtual Host configuration will flow routing to Clusteroutbound | 9080 | | ratings. The default. SVC. Cluster. The local.
Endpoint outbound|9080||ratings.default.svc.cluster.local
Run istioctl proxy-config endpoint reviews-v1-54b8794ddf-jxksn –port 9080 -o json to check the endpoint configuration. We only choose one of the outbound | 9080 | | ratings. The default. The SVC. Cluster. LocalCluster results are as follows.
{
"clusterName": "outbound|9080||ratings.default.svc.cluster.local"."endpoints": [{"locality": {},"lbEndpoints": [{"endpoint": {
"address": {
"socketAddress": {
"address": "172.33.100.2"."portValue": 9080}}},"metadata": {
"filterMetadata": {
"istio": {
"uid": "kubernetes://ratings-v1-8558d4458d-ns6lk.default"}}}}]}]}Copy the codeThere can be one or more endpoints. Sidecar selects the appropriate Endpoint based on certain rules. The Review service now finds the Endpoint of its Upstream service Rating.
summary
Using the bookInfo example provided by Istio, this article provides a step-by-step guide to sidecar injection, Iptables transparent traffic hijacking, and the implementation details behind traffic routing in Sidecar. Sidecar mode and traffic transparency hijacking are the features and basic functions of Istio Service grid. Understanding the process and implementation details behind this feature will help you understand the principle of Service Mesh and the content in the following chapters of Istio Handbook. The reader is therefore encouraged to try it out from the beginning in his own environment to deepen his understanding.
Traffic hijacking using Iptables is only one of the methods of traffic hijacking in the data plane of the Service Mesh. There are many other traffic hijacking schemes. The following describes traffic hijacking in the official website of cloud native network agent MOSN.
Problems with using iptables for traffic hijacking
Currently, Istio uses Iptables to implement transparent hijacking. There are three main problems:
- It is necessary to realize connection tracking with the help of Conntrack module. In the case of a large number of connections, it will cause great consumption and track table may be full. In order to avoid this problem, the industry has closed Conntrack.
- Iptables is a common module. It takes effect globally and cannot explicitly prohibit associated changes. Therefore, iptables is poorly controllable.
- Iptables redirected traffic essentially exchanges data through loopback. Outbond traffic traverses the protocol stack twice, resulting in loss of forwarding performance in large concurrency scenarios.
These problems do not exist in all scenarios. For example, in some scenarios where the number of connections is not large and NAT tables are not in use, iptables is a simple solution. To fit into a broader scenario, transparency hijacking needs to address these three issues.
Transparent hijack scheme optimization
Use Tproxy to process inbound traffic
Tproxy can be used for inbound traffic redirection without changing destination IP addresses or ports in packets, and does not need to perform connection tracing. Therefore, the ConnTrack module does not create a large number of connections. Due to the kernel version, tProxy application in Outbound has some defects. Currently, Istio supports tproxy to process inbound traffic.
Handle outbound traffic with Hook Connect
In order to adapt to more application scenarios, outbound direction is realized through Hook connect. The implementation principle is as follows:
No matter which transparent hijacking scheme is adopted, the problem of obtaining the real destination IP address/port needs to be solved. The iptables scheme is used to obtain the IP address through getsockopt. Tproxy can directly read the destination address and modify the call interface. Hook Connect scheme is similar to TProxy.
After transparent hijacking is implemented, sockMap can shorten the packet crossing path and improve the forwarding performance in the outbound direction on the premise that the kernel version meets the requirements (above 4.16).
reference
- Debugging Envoy and Istiod – istio.io
- Demystifying the Istio Sidecar injection model – Istio. IO
- Traffic hijacking scheme when MOSN is used as Sidecar – MOSN. IO