Focus on big data and container cloud core technology decryption, if you have any academic exchange, feel free to contact. For more content, please pay attention to the public account of “Data Cloud Technology Community”, or forward the email [email protected].

1 the overall

  • Pilot defines a standard model for services in the grid that is independent of various underlying platforms. Pilot can also extract service information from Mesos, Cloud Foundry, Consul, and develop adapters to integrate other components that provide service discovery into Pilot.
  • Istio and the Envoy Project jointly developed the Envoy V2 API and adopted it as the standard interface for Istio control plane and data plane traffic management.
  • Users can also choose different sidecars and control plane integrations based on their business scenarios, such as high throughput, low latency, high security, and so on. In addition to Istio’s current integration Envoy, you can also interact with Linkerd, Nginmesh, etc

2 Discovery Services

2.1 Control Plane – Pilot-Discovery

  • Obtain Service information from a Service provider, such as Kubernetes or Consul
  • Get traffic rules from K8S API Server (K8S CRD Resource)
  • The service information and traffic rules are converted into a format that can be understood by the data plane and delivered to each SIDecAR in the grid through the standard data plane API
  • The corresponding Docker image is gcr. IO/IStio-release /pilot

2.2 Data Plane

  • There are two processes in the data plane, Pilot agent and envoy

2.2.1 Pilot – agent

  • Generate Envoy configuration files based on configuration information in the K8S API Server
  • Start the Envoy process (most of the Envoy configuration information is dynamically retrieved from Pilot via the xDS interface)
  • The pilot-agent initiates the Envoy process by passing in the Pilot address and zipkin address, and generates an initial configuration file enlist-rev0.json for the Envoy

2.2.1.1 Enlist-rev0. json structure Description

  • Node contains information about nodes Envoy

  • Admin configures the Envoy’s log path and administration port
  • Dynamic_resources configudes dynamic resources, where ADS server is configured.
  • Static_resources configure static resources, including xDS-GRPC and Zipkin clusters. The XDS-grPC cluster corresponds to the ADS configuration in dynamic_resources, specifying the server address Envoy used to fetch dynamic resources.

  • Tracing Configures distributed link Tracing
  • Stats_sinks are configured here with metrics collection sinks directly connected to Envoy and has no relation with Mixer Telemetry. Metrics report in stats format Envoy.

2.2.2 envoy

  • The Admission Webhook mechanism of K8s implements the automatic injection of sidecar. Each microservice in the Mesh is added to an Envoy related container. Here is the Pod content for Productpage microservice. IO/ISTIO-Release /proxy_init and GCR. IO/ISTIo-Release /proxyv2 are also injected into the Pod
[root@hadoop ~]# kubectl exec productpage-v1-99b4c9954-vj6hf -c istio-proxy -- ps -ef UID PID PPID C STIME TTY TIME CMD istio-p+ 1 0 0 03:26 ? 00:00:05 /usr/local/bin/pilot-agent proxy sidecar --domain default.svc.cluster.local --configPath /etc/istio/proxy --binaryPath /usr/local/bin/envoy --serviceCluster productpage.default --drainDuration 45s --parentShutdownDuration 1m0s  --discoveryAddress istiod.istio-system.svc:15012 --zipkinAddress zipkin.istio-system:9411 --proxyLogLevel=warning --proxyComponentLogLevel=misc:error --connectTimeout 10s --proxyAdminPort 15000 --concurrency 2 --controlPlaneAuthPolicy  NONE --dnsRefreshRate 300s --statusPort 15020 --trust-domain=cluster.local --controlPlaneBootstrap=false istio-p+ 20 1 0 03:26 ? 00:00:21 /usr/local/bin/envoy -c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster productpage.default --service-node Sidecars ~ 10.42.0.69 ~ productpage - v1-99 b4c9954 - vj6hf. Default to default. SVC. Cluster. The local - Max - obj - name - 189 len --local-address-ip-version v4 --log-format [Envoy (Epoch 0)] [%Y-%m-%d %T.%e][%t][%l][%n] %v -l warning --component-log-level misc:error --concurrency 2 istio-p+ 36 0 0 06:32 ? 00:00:00 ps -efCopy the code
  • From the Envoy initialization profile, we can see roughly the rationale behind Istio’s approach to service discovery and traffic management via Envoy. That is, the control plane assigns xDS Server information to the Envoy’s initialization file as a static resource, and receives dynamic resource from the xDS Server when the Envoy launches. Includes service information and routing rules in the grid

  • The configuration that actually takes effect in the Envoy is made up of the static configuration in the initialization configuration file and the dynamic configuration retrieved from the Pilot.

Principle 3 envoy

Static and RDS routes can be configured on the Envoy. Static routing configurations are written to the Envoy configuration file. Dynamic routing configuration, also known as Route Discovery Service (RDS), is an Envoy that sends a request to a control surface at run time that delivers a routing configuration.As you can see from the architecture diagram, Envoy’s routing mechanism can be roughly divided into five steps:

  • ① Request route configuration
  • ② Configure the receiving route
  • ③ The HTTP request is received and parsed
  • ④ Route Matching
  • ⑤ Forward HTTP requests

3.0 Envoy launch profile content analysis

  • Defines a network listener named listener_0, listening to the address 10.87:60001.
  • The listener mounted below a filter chain (filter chain), contains only a filter envoy. Httpconnectionmanager, official HTTP connection manager.
  • Dynamic route discovery RDS is configured under the filter, which means that the route configuration is dynamically fetched from a data source at run time.
  • Routeconfigname: uniquely identifies a route configuration so that it can obtain the specified route configuration from the data source.
  • Configsource: indicates the data source of the routing configuration. The value ADS :{} indicates that the data source is the cluster specified by adsconfig.
  • Adsconfig: specifies that all xDS resources (including routing configuration) will be obtained from xdsCluster and the communication protocol is GRPC.
  • Xds_cluster: specifies that the cluster has only one 10.87:61001 hwHost.

Kubectl exec - it productpage - v1-54 b8b9f55 bx2dq - c istio - proxy curl http://127.0.0.1:15000/config_dump > config_dumpCopy the code

3.1 Outbound Cluster

  • Take Details as an example. For Productpage, Details is an external service and therefore contains outbound in its Cluster name.
  • As can be seen from the cluster configuration corresponding to Details service, EDS indicates that the endpoint of the cluster comes from dynamic discovery. In dynamic discovery, EDs_config points to ADS. Finally, it points to the xDS-grPC Cluster (Pilot address) configured in static Resource.

3.2 the Inbound Cluster

  • This Cluster corresponds to a service on an Envoy’s node. If the service receives a request, it is of course an inbound request. For the Productpage Pod Envoy, there is only one corresponding Inbound Cluster, Productpage. The host corresponding to the cluster is 127.0.0.1, that is, the listening port of ProductPage on the loopback address. Since 127.0.0.1 is excluded from the Iptables rules, Inbound requests processed through the Inbound Cluster will be skipped Envoy and sent directly to the Productpage process for processing.

3.3 BlackHoleCluster

  • This is a special Cluster and has no Host configured for the backend to process the request. As the name suggests, the request will be discarded. If a request does not find its target service, it is sent to BlackHoleCluster.

3.4 Listeners

  • Envoy uses listeners to receive and process requests from downstream messengers. The processing logic of listeners is plug-in and can be configured to insert different processing logic by configuring different filters.

3.5 Virtual Listener

  • Envoy creates an inlet listener that listens on port 15001. Iptables intercepts the request and sends it to port 15001. The listener does not process the request but distributes it to other listeners based on the destination of the request.
  • How does Envoy do this on a service basis? You can see that the Listener configuration item use_original_dest is set to true. This configuration requires the Listener to forward the received request to the Listener associated with the original destination address for processing.

4 Route Analysis

  • Look at all listeners corresponding to ProductPage, and the listener at 0.0.0.0:15001 receives all traffic in and out of the Pod, and then passes the request to the virtual listener.
export PILOT_SVC_IP=$(kubectl -n istio-system get svc -l app=pilot -o go-template='{{range .items}}{{.spec.clusterIP}}{{end}}') kubectl get pod -l app=productpage istioctl proxy-config listeners Productpage - v1-99b4C9954-vj6HF ADDRESS PORT TYPE 10.42.0.69 9080 HTTP 10.42.0.69 15020 TCP 10.43.113.103 31400 TCP 10.43.113.103 15443 TCP 10.43.0.1 443 TCP 10.43.86.87 15011 TCP 10.43.39.94 443 TCP 10.43.113.103 443 TCP 10.43.108.37 15012 TCP 10.43.0.10 53 TCP 10.43.86.87 15012 TCP 10.43.86.87 443 TCP 10.43.39.94 15443 TCP 10.43.108.37 443 TCP 10.43.0.10 9153 TCP 10.43.113.103 15020 TCP 10.43.113.103 15031 TCP 10.43.113.103 15032 TCP 10.43.96.17 14250 TCP 0.0.0.0 20001 TCP 0.0.0.0 80 TCP 10.43.113.103 15029 TCP 10.43.96.17 14268 TCP 0.0.0.0 9411 TCP 10.43.113.103 15030 TCP 10.43.96.17 14267 TCP 0.0.0.0 8000 TCP 10.43.34.201 80 TCP 0.0.0.0 15014 TCP 10.43.137.123 443 TCP 0.0.0.0 8080 TCP 0.0.0.0 3000 TCP 10.43.130.148 16686 TCP 0.0.0.0 9090 TCP 0.0.0.0 15010 TCP 0.0.0.0 14250 TCP 0.0.0.0 9080 TCP 0.0.0.0 15001 TCP 0.0.0.0 15006 TCP 0.0.0.0 443 TCP 0.0.0.0 15090 HTTP istioctl proxy-config listeners Productpage-v1-99b4c9954-vj6hf --address 10.43.0.10 --port 53-o json {"name": "envoy. Tcp_proxy ", "typedConfig": envoy envoy { "@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy", "statPrefix": "outbound|53||kube-dns.kube-system.svc.cluster.local", "cluster": "outbound|53||kube-dns.kube-system.svc.cluster.local", "accessLog": [ { "name": "envoy.file_access_log", "typedConfig": { "@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog", "path": "/dev/stdout", "format": "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%DYNAMIC_METADATA(istio.mixer:status)%\" \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n" } } ] } } kubectl exec productpage-v1-99b4c9954-vj6hf -c istio-proxy -- netstat -ln Active Internet connections (only servers) Proto Recv-Q Send -q Local Address Foreign Address State TCP 0 0 0.0.0.0:15090 0.0.0.0:* LISTEN TCP 0 0 0.0.0.0:9080 0.0.0.0:* LISTEN TCP 00 127.0.0.1:15000 0.0.0.0:* LISTEN TCP 00 0.0.0.0:15001 0.0.0.0:* LISTEN TCP 00 0.0.0.0:15006 0.0.0.0:* LISTEN Tcp6 00 :::15020 ::* LISTEN 9080: productPage External service port 15001: Envoy's Virtual Outbound listener, iptable will import Outbound traffic from the ProductPage service into the port to be treated by Envoy 15006: Envoy's Virtual Inbound listener, iptable imports incoming traffic to productPage into this port to be handled by Envoy 15000: Envoy management port, bound to a local loopback address and accessible only within a Pod. 15090: Points to 127.0.0.1:15000 /stats/ Prometheus to provide Envoy performance statistics externallyCopy the code
  • Check the eds
# to check the eds curl http://$PILOT_SVC_IP:8080/debug/edsz | grep "outbound 53 | | | kube - DNS. Kube - system. SVC. The cluster. The local" - A 27-1 B  { "clusterName": "outbound|53||kube-dns.kube-system.svc.cluster.local", "endpoints": [ { "lbEndpoints": [{"endpoint": {"address": {"socketAddress": {"address": "172.30.135.21", "portValue": 53}}}, "metadata": { "filterMetadata": { "istio": { "uid": "kubernetes://coredns-64b597b598-4rstj.kube-system" } } } } ] },Copy the code
  • Each microservice communicates directly with each other through PodIP + Port. Service only makes a logical association to locate Pod, and does not communicate through Service

Envoy implements a traffic forwarding mechanism on its own, and when visiting ClusterIP, envoys forward traffic to specific pods without resorting to kube-Proxy iptables or IPVS rules

curl http://$PILOT_SVC_IP:8080/debug/edsz|grep "outbound|9080||productpage.default.svc.cluster.local" -A 27 -B 1 { "clusterName": "outbound|9080||productpage.default.svc.cluster.local", "endpoints": [ { "locality": { }, "lbEndpoints": [{"endpoint": {"address": {"socketAddress": {"address": "10.42.0.69", "portValue": 9080}}}, "metadata": { "filterMetadata": { "envoy.transport_socket_match": { "tlsMode": "istio" } } }, "loadBalancingWeight": 1 } ], "loadBalancingWeight": 1 }Copy the code
  • Each Sidecar has a listener bound to 0.0.0.0:15001, to which IP Tables routes all inbound and outbound traffic for pods. This listener sets useOriginalDst to true,

This means that it hands the request to the listener that best matches the original target of the request. If no matching virtual listener is found, it sends the request to BlackHoleCluster, which returns 404

istioctl proxy-config listeners productpage-v1-99b4c9954-vj6hf --port 15001 -o json [ { "name": "virtual", "address": {" socketAddress ": {" address" : "0.0.0.0", "portValue" : 15001}}, "filterChains" : [{" filters ": [{" name" : "envoy.tcp_proxy", "config": { "cluster": "BlackHoleCluster", "stat_prefix": "BlackHoleCluster" } } ] } ], "useOriginalDst": true } ]Copy the code
  • Our request is an HTTP outbound request to port 9080, which means it is switched to 0.0.0.0:9080 virtual listener. This listener then looks for the routing configuration in its configured RDS. In this case, it looks for route 9080 in the RDS configured by Pilot
Istioctl proxy-config listeners productPage-v1-99b4c9954-vj6hf --address 0.0.0.0 --port 9080-o json "filters": [ { "name": "envoy.http_connection_manager", "typedConfig": { "@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager", "statPrefix": "Outbound_0. 0.0.0 _9080", "RDS" : {" configSource ": {" ads" : {}}, "routeConfigName" : "9080"},Copy the code
  • The 9080 routing configuration only provides virtual hosts for each service. Our request is heading to the Reviews service, so Envoy will select the virtual host that matches our request to the domain. Once a match is made on the domain, the Envoy looks for the first path that matches the request. In this case, we don’t have any advanced routes, so only one route matches everything. The routing told Envoy will request is sent to the outbound | 9080 | | reviews. The default. SVC. Cluster. The local cluster.
istioctl proxy-config routes productpage-v1-99b4c9954-vj6hf --name 9080 -o json [ { "name": "9080", "virtualHosts": [ { "name": "reviews.default.svc.cluster.local:9080", "domains": [ "reviews.default.svc.cluster.local", "reviews.default.svc.cluster.local:9080", "reviews", "reviews:9080", "reviews.default.svc.cluster", "reviews.default.svc.cluster:9080", "reviews.default.svc", "reviews.default.svc:9080", "Review.default "," review.default :9080", "172.21.152.34", "172.21.152.35:9080"], "routes": [{"match": {"prefix": "/"}, "route" : {" cluster ":" outbound | 9080 | | reviews. The default. The SVC. The cluster. The local ", "timeout" : "0.000 s"},Copy the code
  • This cluster is configured to retrieve associated endpoints from Pilot (via ADS). Therefore, the Envoy will use the serviceName field as the key to look up the list of endpoints and proxy the request to one of them.
istioctl proxy-config cluster productpage-v1-99b4c9954-vj6hf --fqdn reviews.default.svc.cluster.local -o json [ { "name": "outbound|9080||reviews.default.svc.cluster.local", "type": "EDS", "edsClusterConfig": { "edsConfig": { "ads": {} }, "serviceName": "outbound|9080||reviews.default.svc.cluster.local" }, "connectTimeout": "S" 1.000, "circuitBreakers" : {" thresholds ": [] {}}}]Copy the code

5 note

This paper core content from Istio flow depth resolution management implementation mechanism, in order to deepen their understanding, combined with the practice, to write a note, convenient access, content please refer to the original: fuckcloudnative. IO/posts/Istio…

Focus on big data and container cloud core technology decryption, if you have any academic exchange, feel free to contact. For more content, please pay attention to the public account of “Data Cloud Technology Community”, or forward the email [email protected].