This article reprinted from: zhongfox. Making. IO / 2019/01/30 /…

Today’s content mainly includes the following four topics:

• 1 Service Mesh: next-generation microservices

• 2 Istio: second-generation Service Mesh

• 3 Istio data plane

• 4 Istio control plane

First, I’ll take a quick look at the evolution of Service Mesh and see what Istio brings to Service Mesh. Next, I will analyze the main architecture of Istio with you, focusing on the realization of the data side and control side, including sidecar injection, traffic interception, xDS introduction, Istio flow model, distributed tracking, Mixer adapter model and so on. Live demos of ISTIO will also be interspersed.

1. Service Mesh: next-generation microservices

• Application communication mode evolution

• The emergence of the Service Mesh

• Second generation Service Mesh

• Definition of Service Mesh

• Brief Product history of Service Mesh

• Development of Service Mesh in China

1.1 Application Communication Evolution: Network traffic control enters the OPERATING system

In the early stage of computer network development, developers need to deal with network connection problems between servers in their own code, including flow control, cache queue, data encryption and so on. During this time the underlying network logic and business logic are mixed together.

With the development of technology, the emergence of TCP/IP and other network standards to solve the flow control and other problems. Although the network logic code still exists, it has been removed from the application and become part of the operating system network layer, forming the classic network layer pattern.

1.2 Evolution of Application Communication Mode: Emergence of microservice architecture

Microservice architecture is a more complex distributed system, which brings more challenges to operation and maintenance. These challenges mainly include effective management of resources and governance between services, such as:

• Service registration, service discovery

• The service scale

• Health check

• Rapid deployment

• Service fault tolerance: circuit breakers, current limiting, isolation pods, fuse protection, service degradation, etc

• Authentication and Authorization

• Grayscale publishing scheme

• Service invocation observability, metrics collection

• Configuration management

In the implementation of microservice architecture, in order to improve efficiency and reduce the threshold, application developers will implement microservices based on microservice framework. To some extent, the micro-service framework shields users from the complexity of the underlying network and the uncertainty of distributed scenarios. Through API/SDK, it provides service registration discovery, service RPC communication, service configuration management, service load balancing, route limiting, fault tolerance, service monitoring and governance, service publishing and upgrading and other general capabilities. Typical products include:

• Distributed RPC communication framework: COBRA, WebServices, Thrift, GRPC, etc

• Service governance domain-specific libraries and solutions: Hystrix, Zookeeper, Zipkin, Sentinel, etc

• Microservices framework that integrates multiple solutions: SpringCloud, Finagle, Dubbox, etc

The cost of implementing microservices often exceeds the enterprise’s expectation (there are many contents and high threshold), and the time spent on service governance may even be higher than the time spent on product development. In addition, the above scenarios limit the tools, runtimes, and programming languages available. Microservice software libraries are generally focused on a certain platform, which makes heterogeneous systems difficult to be compatible, repetitive work exists, and the system lacks portability.

The popularity of Docker and Kubernetes technology has provided new solutions for the allocation management of Pass resources and the deployment of services, but other service governance problems in the field of microservices still exist.

1.3 The rise of Sidecar mode

Sidecar(sometimes called Agent) adds a proxy between the existing client and server to provide additional functions for applications, such as service discovery, routing proxy, authentication and authorization, link tracing, and so on.

Some precedents for Sidecar use in the industry:

• In 2013, Airbnb developed Synapse and Nerve, an open source implementation of Sidecar

• In 2014, Netflix released Prana, which is also a Sidecar that allows non-JVM applications to plug into their Netflix ECOSYSTEM

1.4 The emergence of Service Mesh

Sidecar to Service Mesh is a scale upgrade, but Service Mesh emphasizes:

• Sidecars (agents) are no longer viewed as separate components, but rather as networks connected by these agents

• Infrastructure, transparent to applications

1.5 Service Mesh Definition

The following is the definition of Service Mesh given by Linkerd CEO William Morgan:

A Service Mesh is a dedicated infrastructure layer for handling service-to-service communication. It’s responsible for the reliable delivery of requests through the complex topology of services that comprise a modern, cloud native application. In practice, the Service Mesh is typically implemented as an array of lightweight network proxies that are deployed alongside application code, without the application needing to be aware.

Service Mesh is an infrastructure layer dedicated to communication between services. It is responsible for reliably delivering requests within the complex service topology of modern cloud native applications. In fact, a Service Mesh is typically implemented through a set of lightweight Sidecar proxies deployed with application code and transparent to the application.

1.6 Second-generation Service Mesh

Istio is the most prominent example of a control panel that knows everything about each broker instance, enabling access control and measurement collection for brokers, improving the observability and governance of the service grid.

1.7 Service Mesh Product History

• On January 15, 2016, Former Twitter infrastructure engineers William Morgan and Oliver Gould launched version 0.0.7 of Linkerd on GitHub, Written in Scala, they also built Buoyant, a small start-up that was recognized as the first Service Mesh in the industry

• In 2016, Matt Klein quietly worked on Envoy development for Lyft. Envoy actually predates Linkerd, but is unknown inside Lyft

• The term “Service Mesh” was used publicly for the first time on September 29, 2016 at SF Microservices. This marked the transition of the term Service Mesh from Buoyant to the community.

• On September 13, 2016, Matt Klein announced Envoy open source on GitHub, releasing version 1.0.0 directly.

• In the second half of 2016, Linkerd released versions 0.8 and 0.9 with support for HTTP/2 and gRPC, with 1.0 on the way. At the same time, thanks to the community’s recognition of Service Mesh, Linkerd began applying to join CNCF at the end of the year

• On January 23, 2017, Linkerd joined CNCF.

• On March 7, 2017, Linkerd announced that it had completed 100 billion product requests

• Linkerd version 1.0 was released on April 25, 2017

• On July 11, 2017, Linkerd released version 1.1.1, announcing integration with the Istio project

• In September 2017, Nginx suddenly announced a Servicemesh, Nginmesh: github.com/nginxinc/ng… , can be used as isTIO data surface, but This project is no longer under active development.

• Conduit 0.1.0 was released on December 5, 2017

Both Envoy and Linkerd are data-surface implementations, competing on the same level. Implemented in C++, they are less efficient in terms of performance and resource consumption than Linkerd implemented in Scala, which is especially important for delay-sensitive and resource-sensitive services.

The Envoy’s major contribution to Istio’s standard data surface implementation is to provide a standard data surface API that delivers service information and traffic rules to the data surface sidecar, as well as support for hot restarts. Istio used the Envoy V1 API in its early years, and the current version uses the V2 API, which has since been deprecated.

By using this standard API, Istio decouples the control and data sides, opening up the possibility of multiple data side SIDECAR implementations. In fact, many Sidecar agents have been integrated with Istio based on the STANDARD API. In addition to Istio’s current integration Envoy, you can also integrate with third-party communication agents such as Linkerd and Nginmesh. You can also write your own Sidecar implementation based on the API.

Decoupling the control side from the data side was a clever move by Istio to outdo Linkerd, the originator of Service Mesh. Istio stands at the level of the control plane, and Linkerd becomes an optional Sidecar implementation.

Conduit’s overall architecture is consistent with Istio, borrowing from Istio’s data plane + control plane design, and the Rust programming language was chosen to implement the data plane to achieve Conduit’s claim of lighter, faster, and ultra-low resource footprint.

1.8 A familiar competitive landscape

Google-led Kubernetes has won the container choreography game, and is doing the same in the service grid game. Istio is also very popular in the community.

Istio CNI is planned as an experimental feature in 1.1, allowing users to customize sidecar’s network through extensions.

1.9 Development of Service Mesh in China

• Ant Financial open source SOFAMesh:

  • Github.com/alipay/sofa…

  • From istio fork

  • Develop a new Sidecar in Golang instead of Envoy

  • In order to avoid the performance bottleneck brought by Mixer, some functions of Mixer were combined into Sidecar

  • The Pilot and Citadel modules have been greatly expanded and enhanced

  • Extended RPC protocol: SOFARPC/HSF/Dubbo

• Huawei:

  • Go – chassis: github.com/go-chassis/… Golang microservices framework, supporting isTIO platform

  • The mesher: github.com/go-mesh/mes… Mesh data surface solution

  • The first cloud vendor in China to provide Service Mesh public Service

  • At present (January 2019) public cloud Istio product online has supported the application of public beta, product form is relatively complete

• Tencent cloud TSF:

  • Retrofit based on Istio, Envoy

  • Supports Kubernetes, virtual machines, and bare-metal services

  • Istio’s capabilities have been expanded and enhanced to fully accommodate Consul

  • Extended support for other binary protocols

• Vipshop

  • OSP (Open Service Platform)

• Sina:

  • Motan is a Java based RPC framework. Weibo Mesh is based on Motan

2. Istio: second-generation Service Mesh

Istio comes from the Greek word for sail, which means “to set sail.”

• 2.1 Istio architecture

• 2.2 Core Functions

• 2.3 Istio demo: BookInfo

2.1 Istio architecture

Istio Architecture (image from Isio official website document)

• The data plane

  • Sidecar

• Control surface

  • Pilot: service discovery and traffic management

  • Mixer: Access control and telemetry

  • Citadel: indicates terminal user authentication and traffic encryption

2.2 Core Functions

• Traffic management

• security

• observability

• Multi-platform support

• Integration and customization

Here is a mind map of my summary of Istio architecture:

2.3 Istio demo: BookInfo

Here is a classic BookInfo Demo of Istio, a heterogeneous microservice system with multiple languages:

Bookinfo Application (image from Isio official website document)

Here I will give you a live demo, starting with the demo installation, and experience the flow control function of ISTIO:

Use the HELM to manage ISTIO

Download istio Release: istio. IO /docs/setup/…

Install istio:

1kubectl apply -f install/kubernetes/helm/istio/templates/crds.yaml2helm install install/kubernetes/helm/istio --name istio --namespace istio-systemCopy the code

To enable sidecar automatic injection, ensure that:

• Ensure kube – opened up ValidatingAdmissionWebhook and MutatingAdmissionWebhook apiserver boot parameters

• Add a label to a namespace: kubectl label namespace default istio-injection=enabled

• Also ensure that kube-Apiserver’s Aggregator layer is enabled: — enable-Aggregator -routing=true and the certificate and API server connectivity is set correctly.

To uninstall ISTIO:

1helm delete --purge istio2kubectl delete -f install/kubernetes/helm/istio/templates/crds.yaml -n istio-systemCopy the code

For more installation options, see: istio. IO /docs/setup/…

Install Bookinfo Demo:

Bookinfo is a multilingual, heterogeneous microservice demo where the ProductPage microservice calls details and Reviews microservices, and Reviews calls Ratings microservices. Reviews microservices come in three versions. For more details about this project please refer to: istio. IO /docs/exampl…

Deploying the application:

1kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yamlCopy the code

This creates the corresponding deployments and service for productPage, Details, ratings, reviews, where reviews has three deployments, representing three different versions.

1 % kubectl get pod2NAME                           READY     STATUS    RESTARTS   AGE3details-v1-6865b9b99d-mnxbt    2/2       Running   0          1m4productpage-v1-f8c8fb8-zjbhh 2/2 Running 0 59s5ratings-v1-77f657f55d-95rcz 2/2 Running 0 1m6reviews-v1-6b7f6db5c5-zqvkn 2/2 Running 0 59s7reviews-v2-7ff5966b99-lw72l 2/2 Running 0 59s8reviews-v3-5df889bcff-w9v7g 2/2 Running 0 59s910 % kubectl get Svc11NAME TYPE cluster-ip external-ip PORT(S) AGE12details ClusterIP 172.18.255.240 < None > 9080/TCP 1m13productPage ClusterIP 172.18.255.137 < None > 9080/TCP 1m14Ratings ClusterIP 172.18.255.41 < None > 9080/TCP 1m15Reviews ClusterIP 172.18.255.140 < none > 9080 / TCP 1 mCopy the code

Configure the inlet traffic:

1kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yamlCopy the code

This creates a gateway for bookinfo-gateway and sends traffic to the ProductPage service

1kubectl get gateway2NAME               AGE3bookinfo-gateway   1mCopy the code

Access/productPage via the LB or Nodeport corresponding to the Bookinfo-gateway, and you can see that the three versions of The Reviews service are switched randomly

Weighted routing

Create 3 Reviews sub-versions with CRD DestinationRule:

1kubectl apply -f samples/bookinfo/networking/destination-rule-reviews.yamlCopy the code

Use CRD VirtualService to adjust the traffic ratio of each Sub-version of reviews service to 50% v1 and 50% V3

1kubectl apply -f samples/bookinfo/networking/virtual-service-reviews-50-v3.yamlCopy the code

When you refresh the page, you can see that you can no longer see the content of Reviews V2, and the page switches between V1 and V3.

Content-based routing

Modified Reviews CRD to route the user version Jason logged in to to v2 and other users to version V3.

1kubectl apply -f samples/bookinfo/networking/virtual-service-reviews-jason-v2-v3.yamlCopy the code

Refresh the page and users logged in using Jason will see the V2 black star version and other users will see the V3 red star version.

For more examples of BookInfo, see: istio. IO /docs/exampl… , if you want to delete the application: execute the script. / samples/bookinfo/platform/kube/cleanup. Sh

3. Istio data plane

• 3.1 Data plane Components

• 3.2 Sidecar Traffic hijacking Principle

• 3.3 Data surface standard API: xDS

• 3.4 Distributed Tracing

3.1 Data plane Components

Istio injection sidecar implementation:

• Automatic injection: use Kubernetes Dynamic Admission Webhooks to inject a new POD: init container + sidecar

• Manual injection: Istioctl kube-inject is used

Inject Pod content:

• Istio-init: Hijacks traffic in pods by configuring iptables

• Istio-proxy: Two processes, pilot-Agent and envoy, initialize and initiate the pilot agent

Sidecar automatic injection implementation

Istio uses Kubernetes Dynamic Admission Webhooks to inject pod sidecar

See istio configuration of these two Webhooks ValidatingWebhookConfiguration and MutatingWebhookConfiguration:

1% kubectl get ValidatingWebhookConfiguration -oyaml2% kubectl get mutatingWebhookConfiguration -oyamlCopy the code

As can be seen:

• IO, rbac.istio. IO, authentication.istio. IO, Resources like networking. Istio. IO in Validating work

• The service IStio-Galley in the namespace IStio-System processes Validating work for other Config.istio. IO resources through routing /admitmixer

• The service ISTIo-Sidecar-Injector in the namespace IStio-system handles the CREATE of other V1 / Pods through route /inject. The injector must have isTIo-injection: Enabled

istio-init

Each Pod on the data plane is injected with an initContainer called istio-init. InitContrainer is a mechanism provided by K8S for performing some initialization tasks in the Pod. After Initialcontainer completes and exits, the other containers in the Pod are started.

1initContainers: 2-image: docker. IO/ISTIO /proxy_init: 1.0.53args :4 - -p5 -"15001"6  - -u7  - "1337"8  - -m9  - REDIRECT10  - -i11  - The '*'12  - -x13  - ""14  - -b15  - "9080"16  - -d17  - ""Copy the code

Istio-init startup command for ENTRYPOINT and arGS:

1/usr/local/bin/istio-iptables.sh -p 15001 -u 1337 -m REDIRECT -i The '*' -x "" -b 9080 -d ""Copy the code

Istio-iptables. sh The source address is github.com/istio/istio…

1$ istio-iptables.sh -p PORT -u UID -g GID [-m mode] [-b ports] [-dPorts] [-i CIDR] [-x CIDR] [-h]2 -p: Specifies the Envoy port to redirect all TCP traffic (default: Envoy)$ENVOY_PORT= 15001) 3 -u: specifies the UID of the user whose redirection is not applied. Typically, this is the UID of the proxy container (default is$ENVOY_USERUid of istio_proxy or 1337) 4-g: specifies the GID of the user for which redirection is not applied. (Same default as -u param) 5-m: Specifies the inbound connection to be redirected to the Envoy's mode, "REDIRECT" or "TPROXY" (default is Envoy.$ISTIO_INBOUND_INTERCEPTION_MODE) 6-b: Comma-separated list of inbound ports whose traffic will be redirected to the Envoy (optional). Use wildcard * to redirect all ports. Null disables all inbound redirection (default$ISTIO_INBOUND_PORTS7)-d: Specifies a comma-separated list of inbound ports to exclude (optional) from the redirection to Envoy. Use the wildcard '*' to redirect all inbound traffic (default$ISTIO_LOCAL_EXCLUDE_PORTS) 8-i: Specifies the range of IP addresses redirected to the (optional) Envoy, comma-separated list in CIDR format. Use the wildcard * to redirect all outbound traffic. An empty list disables all outbound redirection (default$ISTIO_SERVICE_CIDR) 9-x: Specifies the range of IP addresses to exclude from redirection, a comma-separated list in CIDR format. Use the wildcard '*' to redirect all outbound traffic (default$ISTIO_SERVICE_EXCLUDE_CIDR). The 1011 environment variable is located$ISTIO_SIDECAR_CONFIG: (default in/var/lib/istio envoy/sidecars. Env)Copy the code

Istio-init hijacks traffic in Pod by configuring iptable:

• Parameter -P 15001: Data traffic in the Pod is intercepted by iptable and sent to port 15001, which will be listened on by the envoy

• Parameters – u 1337: Used to exclude traffic from user ID 1337, i.e. Envoy itself, to avoid Iptable redirecting data sent by Envoy, UID 1337, i.e. user space Envoy, This is istio – the default user proxy container, see sidecars istio – proxy configuration parameter securityContext. RunAsUser

• Parameter -b 9080 -d “”: Inbound port control that redirects all traffic visiting port 9080 (the port of the application container) to the Envoy proxy

• Parameter -i ‘*’ -x “”: Outbound IP control that redirects all outbound traffic to the Envoy proxy

The Init container is automatically terminated after initialization, but the Init container initialization results (iptables) are retained in the application container and Sidecar container.

istio-proxy

Istio-proxy is injected into the POD where the application container resides in the form of sidecar.

1 - image: docker. IO/istio/proxyv2:1.0.52 name: istio-proxy3 args:4 - proxy5 - sidecar6 - --configPath7 - /etc/istio/proxy8 - --binaryPath9 - /usr/local/bin/envoy10  - --serviceCluster11  - ratings12  - --drainDuration13  - 45s14  - --parentShutdownDuration15  - 1m0s16  - --discoveryAddress17  - istio-pilot.istio-system:1500718  - --discoveryRefreshDelay19  - 1s20  - --zipkinAddress21  - zipkin.istio-system:941122  - --connectTimeout23  - 10s24  - --proxyAdminPort25  - "15000"26 - --controlPlaneAuthPolicy27 - NONE28 env:29 ...... 30 ports:31 - containerPort: 1509032 name: http-envoy-prom33 protocol: TCP34 securityContext:35 runAsUser: 133736 ......Copy the code

There are two processes in the IStio-proxy container, pilot-Agent and envoy:

1~ % kubectl exec productpage-v1-f8c8fb8-wgmzk -c istio-proxy -- ps -ef2UID        PID  PPID  C STIME TTY          TIME CMD3istio-p+     1     0  0 Jan03 ?        00:00:27 /usr/local/bin/pilot-agent proxy sidecar --configPath /etc/istio/proxy --binaryPath /usr/local/bin/envoy --serviceCluster productpage --drainDuration 45s --parentShutdownDuration 1m0s --discoveryAddress istio-pilot.istio-system:15007 --discoveryRefreshDelay 1s --zipkinAddress zipkin.istio-system:9411 --connectTimeout 10s --proxyAdminPort 15000 --controlPlaneAuthPolicy NONE4istio-p+    21     1  0 Jan03 ?        01:26:24 /usr/local/bin/envoy -c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s60 --service-cluster productPage --service-node sidecar~172.18.3.12~ productPage -v1-f8c8fb8-wgmzk.default~default.svc.cluster.local --max-obj-name-len 189 --allow-unknown-fields -l warn --v2-config-onlyCopy the code

You can see:

• /usr/local/bin/pilot-agent is the parent of /usr/local/bin/envoy, The Pilot agent process generates the Envoy’s initial configuration file (/etc/istio/proxy/ enbith-rev0.json) based on the startup parameters and configuration information in the K8S API Server and is responsible for starting the Envoy process

• The launch parameters of the pilot-Agent include: discoveryAddress(Pilot service address), binary location, service cluster name, monitoring indicator reporting address, administration port Envoy, hot restart time, etc

Envoy configuration initialization process:

1. The Pilot agent generates the Envoy’s initial configuration file enbith-rev0.json based on the boot parameters and configuration information in the K8S API Server. This file tells the Envoy to fetch dynamic configuration information from the xDS Server and configures the xDS Server address information. Pilot of the control plane

2. The Pilot-agent launches the envoy process using enbith-rev0.json

3. Obtain Pilot address based on initial configuration, use xDS interface to obtain D dynamic configuration information such as Listener, Cluster, Route from Pilot

4. Envoy initiates the Listener based on the retrieved dynamic configuration and processes the intercepted traffic in combination with the Route and Cluster based on the Listener configuration

View the envoy initial configuration file:

1kubectl exec productpage-v1-f8c8fb8-wgmzk -c istio-proxy -- cat /etc/istio/proxy/envoy-rev0.jsonCopy the code

3.2 Sidecar Traffic hijacking Principle

Sidecar acts as both a forward proxy for the service consumer and a reverse proxy for the service provider. The interception process is as follows:

• In the network namespace where pods reside, iptables rules intercept incoming and outgoing traffic except envoy traffic and redirect it via NAT to port 15001 listened on the envoy.

• Envoy forwards traffic based on XDS rules taken from Pilot.

• The Envoy’s Listener 0.0.0.0:15001 receives all traffic to and from the Pod and passes the request to the corresponding Virtual Listener

• For this POD service, there is an HTTP Listener podIP+ port that receives inbound traffic

• For each Service + non-HTTP port, the listener pairs Outbound non-HTTP traffic

• For each service+ HTTP port, an HTTP listener 0.0.0.0+ port receives outbound traffic

The entire interception and forwarding process is transparent to the Service container. The Service container still uses the Service domain name and port to communicate. The Service domain name is still converted to the Service IP address, but the Service IP address is directly converted to pod IP in sidecar. Compared with the traditional Kubernetes service mechanism, service IP is converted into POD IP on Node, which is implemented by Iptables maintained by Kube-Proxy.

3.3 Data surface standard API: xDS

XDS is a generic name for a class of discovery services, including LDS, RDS, CDS, EDS and SDS. Envoy can dynamically fetch Listener, Route, Cluster, Endpoint and Secret certificate configurations through the xDS API

Concepts covered by the xDS API:

• Host

• Downstream

• Upstream

• Listener

• Cluster

3. Dynamic changes to a configuration without requiring a restart Envoy:

1. Old and new processes communicate using Unix Domain sockets using basic RPC protocol.

2. After the new process has started and done all the initialization, it asks the old process for a copy of the listening socket.

3. When the new process takes over the socket, the old process is notified to close the socket.

4. Notifies the old process to terminate itself.

XDS debugging

The Pilot provides the following debugging interfaces on port 9093:

1# Listeners and routes3usr/usr/usr/usr/usr/usr/usr/usr/usr Clusters9curl $PILOT/debug/cdszCopy the code

Sidecar Envoy also provides a management interface, with a default localhost port of 15000, that can fetch listener, cluster, and complete configuration data

You can run the following command to view the supported debugging interfaces:

1kubectl exec productpage-v1-f8c8fb8-zjbhh-c istio-proxy curl http://127.0.0.1:15000/helpCopy the code

Or forward to the local debugging line

1kubectl port-forward productpage-v1-f8c8fb8-zjbhh 15000Copy the code

Related debugging interfaces:

1 HTTP: / / 127.0.0.1:150002 HTTP: / / 127.0.0.1:15000 /helpHTTP: / / 127.0.0.1:15000 / config_dump4http: / / 127.0.0.1:15000 / listeners5http: / / 127.0.0.1:15000 / clustersCopy the code

Use istioctl to view the proxy configuration:

1istioctl PC {xDS type} {POD_NAME} {Filter condition} {-o json/yaml}23eg:4istioctl PC routes productPage -v1-f8c8fb8-zjbhh --name 9080 -o jsonCopy the code

The xDS types include listener, route, cluster, and endpoint

Analyze xDS: ProductPage visit the Reviews service

View all listeners for product:

1% istioctl pc listener  productpage-v1-f8c8fb8-zjbhH2Address PORT TYPE3172.18.255.178 15011 TCP4172.18.255.194 44134 TCP5172.18.255.110 443 TCP6172.18.255.190 50000 TCP7172.18.255.203 853 TCP8172.18.255.2 443 TCP9172.18.255.239 16686 TCP100.0.0.0 80 TCP11172.18.255.215 3306 TCP12172.18.255.203 31400 TCP13172.18.255.111 443 TCP14172.18.255.203 8060 TCP15172.18.255.203 443 TCP16172.18.255.40 443 TCP17172.18.255.1 443 TCP18172.18.255.53 53 TCP19172.18.255.203 15011 TCP20172.18.255.105 14268 TCP21172.18.255.125 42422 TCP22172.18.255.105 14267 TCP23172.18.255.52 80 TCP240.0.0.0 15010 HTTP250.0.0.0 9411 HTTP260.0.0.0 8060 HTTP270.0.0.0 9080 HTTP280.0.0.0 15004 HTTP290.0.0.0 20001 HTTP300.0.0.0 9093 HTTP310.0.0.0 8080 HTTP320.0.0.0 15030 HTTP330.0.0.0 9091 HTTP340.0.0.0 9090 HTTP350.0.0.0 15031 HTTP360.0.0.0 3000 HTTP370.0.0.0 15001 TCP38172.18.3.50 9080 HTTP This is the service address exposed by the current POD IP. It will be routed to the loopback address. Each pod will be differentCopy the code

Envoy inlet listener:

1% istioctl pc listener  productpage-v1-f8c8fb8-zjbhh --address 0.0.0.0 --port 15001-o json2[3 {4"name": "virtual"May,"address": {6            "socketAddress": {7                "address": "0.0.0.0"8,"portValue": 150019            }10        },11        "filterChains": [12            {13                "filters"14: [{15"name": "envoy.tcp_proxy", 16"config": {17                            "cluster": "BlackHoleCluster"18,"stat_prefix": "BlackHoleCluster"19}20}21]22}23],24"useOriginalDst": true # This means that it passes the request to the listener that best matches the original target of the request. If no matching virtual listener is found, it sends the request to BlackHoleCluster25 that returns 404}26]Copy the code

Below are all the Pod IPS from Reviews

1 % kubectl get ep reviews2NAME      ENDPOINTS                                            AGE3reviews   172.18.2.35:9080,172.18.3.48:9080,172.18.3.49:9080   1dCopy the code

For HTTP access destined for the preceding IP addresses, there is no listener corresponding to these IP addresses. Therefore, port 9080 matches listener 0.0.0.0 9080

View listener 0.0.0.0 9080:

1% istioctl pc listener  productpage-v1-f8c8fb8-zjbhh --address 0.0.0.0 --port 9080-ojson2 {3"name": "0.0.0.0 _9080"4,"address": {5            "socketAddress": {6                "address": "0.0.0.0"7,"portValue": 90808}9},10...... 1112"rds": {13                                "config_source": {14                                    "ads"16:15}, {}"route_config_name": "9080"17}, 18...Copy the code

View route named 9080:

1% istioctl pc routes  productpage-v1-f8c8fb8-zjbhh --name 9080 -o json2​3[4    {5        "name": "9080"6,"virtualHosts": [7            {8                "name": "details.default.svc.cluster.local:9080"9,"domains": [10                    "details.default.svc.cluster.local"11,"details.default.svc.cluster.local:9080", 12"details"13,"details:9080", 14"details.default.svc.cluster", 15"details.default.svc.cluster:9080", 16"details.default.svc", 17"details.default.svc:9080"18,"details.default", 19"details.default:9080", 20"172.18.255.240"21,"172.18.255.240:9080"22, 23"routes": {[24, 25"match": {26                            "prefix": "/"27                        },28                        "route": {29                            "cluster": "outbound|9080||details.default.svc.cluster.local", 30"timeout": "0.000 s"31."maxGrpcTimeout": "0.000 s"32}, 33... 34 {35"name": "productpage.default.svc.cluster.local:9080"And 36"domains": [37                    "productpage.default.svc.cluster.local"38"productpage.default.svc.cluster.local:9080", 39"productpage", 40"productpage:9080", 41"productpage.default.svc.cluster"42,"productpage.default.svc.cluster:9080"43,"productpage.default.svc"44,"productpage.default.svc:9080", 45"productpage.default"46."productpage.default:9080"47."172.18.255.137", 48"172.18.255.137:9080"49                ],50                "routes": [...] 52 {51}, 53"name": "ratings.default.svc.cluster.local:9080"And 54"domains": [55                    "ratings.default.svc.cluster.local"And 56"ratings.default.svc.cluster.local:9080", 57"ratings", 58"ratings:9080"59."ratings.default.svc.cluster", 60"ratings.default.svc.cluster:9080", 61,"ratings.default.svc", 62,"ratings.default.svc:9080", 63,"ratings.default", 64,"ratings.default:9080", 65,"172.18.255.41", 66,"172.18.255.41:9080"67                ],68                "routes": [...] {69}, 70, 71"name": "reviews.default.svc.cluster.local:9080", 72,"domains": [73                    "reviews.default.svc.cluster.local", 74,"reviews.default.svc.cluster.local:9080", 75,"reviews", 76,"reviews:9080", 77,"reviews.default.svc.cluster", 78,"reviews.default.svc.cluster:9080", 79,"reviews.default.svc", 80,"reviews.default.svc:9080", 81,"reviews.default", 82,"reviews.default:9080", 83,"172.18.255.140", 84,"172.18.255.140:9080"85                ],86                "routes": [87                    {88                        "match": {89                            "prefix": "/", 90,"headers": [91                                {92                                    "name": "end-user", 93,"exactMatch": "jason"94                                }95                            ]96                        },97                        "route": {98                            "cluster": "outbound|9080|v2|reviews.default.svc.cluster.local", 99,"timeout": "0.000 s", 100,"maxGrpcTimeout": "0.000 s"101}, 102... {103}, 104, 105"match": {106                            "prefix": "/"107                        },108                        "route": {109                            "cluster": "outbound|9080|v3|reviews.default.svc.cluster.local", 110,"timeout": "0.000 s", 111,"maxGrpcTimeout": "0.000 s"112}, 113... 114}115]116}117],118"validateClusters": false119    }120]Copy the code

As you can see, route 9080 contains all HTTP routing information for this port, which is distributed to each cluster through the virtualHosts list.

Check the virtualHosts reviews. Default. SVC. Cluster. The local: 9080 routes of information, Routed to the cluster can see Jason outbound | | 9080 | v2 reviews. The default. SVC. Cluster. The local

View the cluster:

1% istioctl pc cluster productpage-v1-f8c8fb8-zjbhh --fqdn reviews.default.svc.cluster.local --subset v2 -o json2[3    {4        "name": "outbound|9080|v2|reviews.default.svc.cluster.local"May,"type": "EDS"6,"edsClusterConfig": {7            "edsConfig": {8                "ads"9: {}}, 10"serviceName": "outbound|9080|v2|reviews.default.svc.cluster.local"11        },12        "connectTimeout": "1.000 s"13,"lbPolicy": "RANDOM", 14"circuitBreakers": {15            "thresholds": [16 {}17]18}19}20]Copy the code

View the corresponding endpoint:

1 % istioctl pc endpoint productpage-v1-f8c8fb8-zjbhh --cluster 'outbound|9080|v2|reviews.default.svc.cluster.local'2 the endpoint STATUS CLUSTER3172.18.2.35:9080 HEALTHY outbound | | 9080 | v2 reviews. The default. SVC. Cluster. The localCopy the code

This endpoint is the POD IP corresponding to the Reviews service V2

Ultimate conformance considerations for XDS service interfaces

Follow the make before break model

3.4 Distributed Tracing

The following is a schematic diagram of distributed full-link tracking:

A typical Trace case

Jaeger is an open source full-link tracking system of Uber, which conforms to OpenTracing protocol. Both OpenTracing and Jaeger are member projects of CNCF. The following is the schematic diagram of Jaeger architecture:

Jaeger Architecture Diagram (image from Jaeger official documentation)

A distributed trace system gives developers a visual representation of the call flow. This is critical for troubleshooting and performance optimization of complex microservice systems.

Envoy native SUPPORT for HTTP link tracking:

• Generate the Request ID: Envoy generates the UUID and manipulates the HTTP Header named [x-request-id] as needed. Applications can forward this Header for unified logging and tracing.

• Support for integrating external trace services: Envoy supports pluggable external trace visualization services. Currently supported:

  • LightStep

  • Zipkin or Zipkin-compatible back ends (such as Jaeger)

  • Datadog

• Client-side tracing ID connections: the X-client-trace-id Header can be used to connect untrusted request ids to trusted X-Request-ID headers

Track the propagation of context information

• Regardless of which trace service is used, the X-Request-ID should be propagated so that the correlation record is started in the invoked service

• So if you’re using Zipkin, Envoy propagates B3 headers. (X-B3-Traceid, X-B3-SPANID, X-B3-PARENtSPANID, X-B3-tos, and X-B3-flags.x-B3-tos)

• Context tracing is not zero modification, and upstream applications should propagate tracing related HTTP headers themselves when calling downstream services

4. Istio control surface

• 4.1 Pilot architecture

• 4.2 Traffic Management model

• 4.3 Troubleshooting

• 4.4 Mixer architecture

• 4.5 Mixer adapter model

• 4.6 Mixer Caching Mechanism

4.1 Pilot architecture

Pilot Architecture

• Rules API: encapsulates a unified API externally and is invoked by service developers or O&M personnel for traffic control.

• Envoy API: Encapsulates a unified API for Envoy calls to retrieve registration information, traffic control information, and so on.

• Abstract model layer: Service registration information, traffic control rules, etc., are abstracted to make their description platform-independent.

• Platform adaptation layer: used to adapt various platforms such as Kubernetes, Mesos, Cloud Foundry, etc., to convert platform-specific registration information, resource information, etc., into platform-independent descriptions defined by the abstract model layer. For example, the Kubernetes adapter in Pilot implements the controller necessary to make changes to the POD registration information, ingress resources, and third-party resources used to store traffic management rules in the Watch Kubernetes API Server

4.2 Traffic Management model

• VirtualService

• DestinationRule

• ServiceEntry

• Gateway

VirtualService

VirtualService defines a set of traffic routing rules for a specified service. Each routing rule is a matching rule for a specific protocol. If the traffic matches these characteristics, it is sent to the target service, or a subset or version of the target service, in the service registry based on the rules. The matching rules also contain the definition of the initiator of the traffic, so that the rules can be customized for a specific customer context.

Gateway

The Gateway describes a load balancer that hosts incoming and outgoing connections to grid edges. This specification describes a series of open ports, the protocols used by these ports, the SNI configuration for load balancing, and so on

ServiceEntry

The Istio service grid maintains a platform-independent service registry using a common model. When your service grid needs to access external services, you need to add service registries using ServiceEntry. These services may be specific apis for the network. Or an entry in the registry of services that are inside the grid but not on the platform (such as a set of virtual machine services that need to communicate with Kubernetes services).

EnvoyFilter

EnvoyFilter describes filters for broker services that are used to customize broker configurations generated by Istio Pilot.

Kubernetes Ingress vs Istio Gateway

• Combining l4-6 and L7 specifications, it is not convenient for traditional technology stack users to migrate applications

• Lack of expression:

  • Only limited fields such as Service, port, and HTTP path can be matched for incoming routing traffic

  • The default port number is 80/443

Istio Gateway: ·

• Defines four to six layers of load balancing properties (typically SecOps or NetOps concerns)

  • port

  • Protocol used by the port (HTTP, HTTPS, GRPC, HTTP2, MONGO, TCP, TLS)

  • Hosts

  • TLS SNI Header Routing support

  • TLS configuration support (HTTP automatic 301, certificates, etc.)

  • ip / unix domain socket

Kubernetes, Istio, Envoy xDS model comparison

The following is a loose comparison of the Kubernetes, Istio, and Envoy xDS models

The difference between Kubernetes and Istio service addressing:

Kubernetes:

1. kube-dns: service domain -> service ip

2. kube-proxy(node iptables): service ip -> pod ip

Istio:

1. kube-dns: service domain -> service ip

2. sidecar envoy: service ip -> pod ip

4.3 Troubleshooting

As the separation granularity of microservices increases, service invocations will increase and become more complex. Fan-in and fan-out will increase the risk of invocation failure. The following are common service fault tolerance processing methods:

Istio does not have degrade free processing support: Istio improves the reliability and availability of services in the grid. However, the application still needs to handle failures (errors) and take appropriate fallback actions. For example, an Envoy returns HTTP 503 when all instances in the load balancing pool fail. It is the responsibility of the application to implement the necessary logic to respond appropriately to such HTTP 503 errors from the upstream service.

4.4 Mixer architecture

Mixer Topology (Image from Isio official website document)

The four function points of Istio are connection, security, control and observation. The functions of “control” and “observation” are mainly provided by Mixer components. Mixer plays the following roles in Istio:

• Function: responsible for policy control and telemetry collection

• Architecturally: Provides a plug-in model that can be extended and customized

4.5 Mixer Adapter model

• Attribute

• Template

• Adapter

• Instance

• Handler

• Rule

Attribute

An Attribute is basic data about requests and environments in policy and telemetry. It is a small piece of data used to describe attributes of a particular service request or request environment. For example, properties can specify the size of a particular request, the response code for the operation, the IP address from which the request came, and so on.

• The primary attribute producer in Istio is Envoy, but specialized Mixer adapters can also generate attributes

• For the Attribute Vocabulary, see Attribute Vocabulary

• Data flow: Envoy -> Mixer

Template

Template is an abstraction of the Adapter’s data format and processing interface. Template defines:

• The format of the data sent to the Adapter when the request is processed

• The Adapter must implement the gRPC Service interface

Each Template is defined by template.proto:

• A message named Template

• Name: automatically generated from the package Name of the template

• Template_variety: Optional Check, Report, Quota or AttributeGenerator, which determines the method the adapter must implement. It also determines at what stage in the mixer the instance corresponding to the Template should be generated:

  • Check: An instance is created and sent during Mixer’s Check API call

  • Report: Creates and sends an instance during Mixer’s Report API call

  • Quota: Create and send instance during Mixer’s Check API call (when querying Quota)

  • AttributeGenerator: for both Check, Report Mixer API calls

Templates built into Istio: Istio. IO /docs/refere…

Adapter

Encapsulates the interfaces necessary for Mixer to interact with a specific external infrastructure back end, such as Prometheus or Stackdriver

• Defines templates to work with (configure templates in YAML)

• Defines a GRPC interface to work with a Template data format

• Define the configuration formats required by the Adapter (Params)

• Can process multiple data simultaneously (instance)

IO /docs/refere…

Instance

Represents a concrete implementation of a data format that conforms to a Template definition. The concrete implementation is a user-configured CRD that defines rules for converting Attributes into concrete instances, supporting attribute expressions

• Instance CRD is a data format + attribute converter defined in Template

• The built-in Instance type: Templates

• For the attribute Expression, see: Expression Language

• Data flow: Mixer -> Adapter instance

Handler

The user-configured CRD provides a specific configuration for a specific Adapter and a runnable instance of the Adapter

Rule

A user-configured CRD that configures a set of rules that describe when to invoke a particular adapter (via Handler) and which Instance

conclusion

All problems in computer science can be solved with another layer, except problems with too many layers

Kubernetes itself was already very complex, and Istio added many more concepts for higher level control abstractions. The complexity is kubernetes.

It can be seen that IStio is well designed and has many advantages in dealing with complex scenarios of microservices. However, its disadvantages are still obvious. A high degree of abstraction leads to a lot of performance loss, and the community has many optimization directions. Attempts are made to do a lot of mixer work in the Sidecar to reduce the synchronization request dependence of the Sidecar and mixer, while some other sidecar network schemes are more concerned with optimizing the layer and optimizing the performance overhead of the sidecar layer.

Prior to Istio 1.0, the focus was on functionality, but with the community’s commitment, Istio performance will improve significantly.

The author has been engaged in service governance related work for many years. In the process, I have personally experienced the pain point of micro-service governance, so I pay close attention to the development of Service Mesh. I am also very optimistic about ISTIO. We look forward to some products and technologies in this direction.

References:

• Annual summary of Service Mesh: The war is on
• Why You Should Care About Istio Gateways
• Pattern: Service Mesh
• Mixer Out Of Process Adapter Dev Guide
• Mixer Out of Process Adapter Walkthrough
• Details on xDS REST and gRPC protocols in Envoy
• Delayering Istio with AppSwitch
• Servicemesher Chinese community