Open Source Project Recommendation
Pepper Metrics is an open source tool I developed with my colleagues (github.com/zrbcool/pep…) , by collecting jedis mybatis/httpservlet/dubbo/motan performance statistics, and exposure to Prometheus and other mainstream temporal database compatible data, through grafana show the trend. Its plug-in architecture also makes it easy for users to extend and integrate other open source components. Please give us a STAR, and we welcome you to become developers to submit PR and improve the project together.
Introduction to the
As the concept of Service Mesh is gradually accepted by people and K8S as a container management platform has become a de facto standard, people’s demand for Service governance and traffic management under K8S is increasingly strong. With the promotion of Google, IBM and Lyft, Istio + Envoy Service Mesh and traffic management solutions complement K8S in advanced features such as Service governance, refined traffic management, and grayscale publishing. Due to the advantages of the Go + C++ language stack, K8S is more resource-efficient than Linked scala. It is now almost certainly the de facto standard in the Service Mesh domain. In this article, I use the K8S plus Istio cluster environment created on Ali Cloud as an example to explain how a request flows across the network between K8S and Istio, Envoy and microserver pods.
Basic Knowledge
Many of the concepts and implementation principles of networks and Kubernates have been ignored in this article. If you are having trouble reading them, please refer to the following articles
- The Iptables/Netfilter (www.zsythink.net/archives/11…
- The bridge
- Veth equipment for
- Linux network namespace
- Cni network plug-in mechanism of Kubernetes and Flannel implementation (Ali Cloud uses Flannel implementation of Type Alloc)
- Kubernetes Service (LoadBalancer, NodePort, ClusterIP), Pod, Deployment, etc
- The concept of Service Mesh
- Istio & Envoy
example
Create a cluster named test with two nodes named work001 and work002. (via Kubernetes version of Alicloud container Service application market) deploy istio-ingressGateway and create Service of type LoadBalancer, and deploy POD instance of IStio-ingressGateway with replics 2. After completion, you can see that two POD instances are started on work001 and work002 respectively. After successfully creating an Ali cloud load balancing instance (182.92.251.227), the LoadBalancer Service will automatically create an Ali cloud load balancing instance (182.92.251.227) and use port 31061 of our two nodes as the back-end Service of ali Cloud load balancing instance, as shown in the following figure:
Request from extranet to SLB to work001
Looking at the iptables rules from the machine work001 (192.168.0.17 in the figure above), you can see:
➜ ~ iptables -L -t nat
Chain PREROUTING (policy ACCEPT 244 packets, 14640 bytes)
pkts bytes target prot opt in out sourceDestination 32M 2003M KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */Copy the codeFrom eth0 to CNI0 bridge
Kube-services: kube-services: kube-services: kube-services: kube-services: kube-services: kube-services: kube-services: kube-services: kube-services: kube-services: kube-services: Kube-services: Kube-services: Kube-services: Kube-services: Kube-services: Kube-services:
Chain KUBE-SERVICES (2 references)
pkts bytes target prot opt in out sourceDestination 0 0 kube-fw-hbo2mbmqrzfahkf2 TCP -- * * 0.0.0.0/0 182.92.251.227 /* default/ istio-ingressGateway :http2 Loadbalancer IP */ TCP DPT :80 562 33720 KUBE -nodeports all -- * * 0.0.0.0/0 0.0.0.0/0 / kubernetes service NODEPORTS; NOTE: this must be the last rulein this chain */ ADDRTYPE match dst-type LOCAL
Copy the codeKube-fw-hbo2mbmqrzfahkf2: kube-FW-hBO2mbmqrzfahkf2: kube-FW-hBO2mbmqrzfahkf2: kube-FW-hBO2mbmqrzfahkf2:
Chain KUBE-FW-HBO2MBMQRZFAHKF2 (1 references)
pkts bytes target prot opt in out sourceDestination 0 0 kube-xlb-hBO2mbmqrzfahkf2 all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/ istio-ingressGateway :http2 Loadbalancer IP */ 0 0 kube-mark-drop all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/istio-ingressgateway:http2 loadbalancer IP */ 0 0 kube-mark-drop all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/istio-ingressgateway:http2 loadbalancer IP */ Chain KUBE-MARK-DROP (7 references) pkts bytes target prot optin out sourceDestination 00 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK or 0x8000Copy the codeThe above rule indicates that the request packet is forwarded to the rule chain kube-xlB-hBO2mBMQRzFAHKF2 for processing. If there is no match, it is discarded (kube-mark-drop).
Chain KUBE-XLB-HBO2MBMQRZFAHKF2 (2 references)
pkts bytes target prot opt in out sourceDestination 0 0 kube-svc-hBO2mbmQrzfahkf2 all -- * * 172.20.0.0/16 0.0.0.0/0 /* Redirect Pods Trying to reach External Loadbalancer VIP to clusterIP */ 80 4800 kube-sep-ew7ylhh65nruvifq all -- * * 0.0.0.0/0 0.0.0.0/0 /* Balancing rule 0for default/istio-ingressgateway:http2 */
Copy the codeYou can see that the rule chain has two rules, and since our request came from outside the cluster and entered the cluster via SLB (182.92.251.227:80), it matches the second rule kube-SEP-ew7YLHH65NRuVIfq
Chain KUBE-SEP-EW7YLHH65NRUVIFQ (2 references)
pkts bytes target prot opt in out sourceDestination 00 kube-mark-masq all -- * * 172.20.1.157 0.0.0.0/0 /* default/istio-ingressgateway:http2 */ 80 4800 DNAT TCP -- * * 0.0.0.0/0 0.0.0.0/0 /* default/istio-ingressgateway:http2 */ TCP to:172.20.1.157:80 Chain kube-mark-masq (102 references) pkts bytes target prot optin out sourceDestination 241 14460 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK or 0x4000Copy the codeOur request packet matches the DNAT, which translates our request target address to 172.20.1.157:80 and hands it to the upper-layer protocol stack for processing. The request is then routed to the routing decision.
# work001
➜ ~ ip route
default via 192.168.0.253 dev eth0
169.254.0.0/16 dev eth0 scope link metric 1002
169.254.123.0/24 dev docker0 proto kernel scope link src 169.254.123.1
172.20.1.128/25 dev cni0 proto kernel scope link src 172.20.1.129
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.17
Copy the code172.20.1.157:80 dev cni0 proto kernel scope link SRC 172.20.1.129 The request goes directly to the bridge device CNI0 (IP 172.20.1.129) for processing and the request goes to the CNI0 bridge
From cNI0 bridge to POD container
➜ ~ kubectl get Pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE Caf -sample-mesh- tomcat-77ff76b78f-fhgCL 2/2 Running 0 7d 172.20.1.167cn-beijing. I-2zef83gtte2gddxlpy3q <none> Istio-ingressgateway-84b4868f6b-9xnts 1/1 Running 0 15d 172.20.1.15cn-beijing. I-2zef83gtte2gddxlpy3q <none> Istio-ingressgateway-84b4868f6b-qxz5t 1/1 Running 0 22h 172.20.2.7cn-beijing. I-2zeg4wnagly6jvlcz5kr <none> Zookeeper-854f49888f-4zhhk 2/2 Running 0 8d 172.20.1.165cn-beijing. I-2zef83gtte2gddxlpy3q <none>Copy the code172.20.1.157 is held by istio-ingressgateway-84b4868f6B-9xnts POD.
➜ ~ kubectl exec -it istio-ingressgateway-84b4868f6b-9xnts bash
root@istio-ingressgateway-84b4868f6b-9xnts:/# ip routeDefault via 172.20.1.129 dev eth0 172.20.0.0/16 via 172.20.1.129 dev eth0 172.20.1.128/25 dev eth0 proto kernel scope The link SRC 172.20.1.157Copy the codeWe know that the eth0 network card in K8S is actually one side of the VETH device pair placed in the POD network namespace and renamed eth0, and the other end is plugged into the network bridge CNI0 on the host, which can be viewed from the host
➜ ~ brctl show
bridge name bridge id STP enabled interfaces
cni0 8000.0a58ac140181 no veth0982819b
veth21f3386e
veth295aca54
veth29c64f6d
veth2fc83624
veth5413f391
veth55b5c32f
veth5d630032
veth5f4b9957
veth67b85819
veth6e5e5662
veth6efeda57
veth7413a550
veth831d20f5
veth849fecd0
veth8dbe1d2c
vetha7ec9173
vetha83f9559
vethcc9d3b42
vethd967d407
vethec751a37
vethf69ed27b
Copy the codeIn this case, the bridge CNI0 actually acts as a Layer 2 switch device. All packets sent to CNI0 will be routed to the other side of the corresponding VETH pair, namely the eth0 nic in the network namespace of POD. In our example, it is: POD istio-ingressgateway-84b4868f6B-9xnts (IP address 172.20.1.157)
conclusion
After many hurdles, our request package finally made it inside the IStio-IngressGateway container. Here’s a summary: In this lecture, we mainly introduced how to forward the request packets from the external network (outside the cluster) to the POD (istio-ingressGateway) in our cluster through the four-layer load balancing device SLB of Ali Cloud, which is also referred to as the north-south traffic in our Service Mesh concept. In fact, this network forwarding is all implemented by Ali Cloud LoadBalancer (a Service type of K8s, implemented by various cloud providers) and K8s CNI network plug-in
In the next lecture we will take a look at how this request packet flows through istio-ingressGateway and how ISTIO controls this north-south traffic