The author of this article, Zhao Huabing, will share the practice of service grid technology in 5G Network management platform in Chengdu Ant C space at 1:30 PM tomorrow. Welcome to check the details of the activity.

This section describes Istio Pilot components

In the Istio architecture, the Pilot component is the core component responsible for traffic management in the service grid and configuration delivery between the control plane and the data plane. Pilot internal code structure is more complex, this article will be through the Pilot code in-depth analysis to understand the principle of Pilot implementation.

First, let’s take a look at Pilot’s role in Istio. Pilot transforms service information and configuration data into standard data structures for the xDS interface and delivers them to the data surface Envoy via gRPC. If you think of Pilot as a black box that processes data, it has two inputs and one output:

Current Pilot input consists of two data sources:

  • Service data: comes from Service registries, such as Service registered in Kubernetes, Consul Catalog, etc.
  • Configuration rules: Various configuration rules, including routing rules and traffic management rules, are defined and stored in Kubernetes CRD.

The output of Pilot is the configuration data conforming to the xDS interface and the configuration data is pushed to the data surface Envoy via the gRPC Streaming interface.

Note: Istio code base changes constantly updated, in this paper, based on the code to commit as follows: d539abe00c2599d80c6d64296f78d3bb8ab4b033

Pilot-Discovery code structure

The code for Istio Pilot is divided into Pilot Discovery and Pilot Agent. The Pilot Agent is responsible for the life cycle management on the data side and the Pilot Discovery is the traffic management component on the control side. This article focuses on the code for the control surface section, the Pilot-Discovery.

Here is the main structure of the pilot-Discovery component code:

The entry function of pilot-discovery is the main method in Pilot/CMD /pilot-discovery/main.go. The main method creates a Discovery Server, which contains three parts of logic:

Config Controller

The Config Controller is used to manage various configuration data, including user-created traffic management rules and policies. Istio currently supports three types of Config Controllers:

  • Kubernetes: Uses Kubernetes to store configuration data. This method is directly attached to the powerful CRD mechanism of Kubernetes to store configuration data. It is simple and convenient, and Istio used configuration storage solution at the beginning.
  • Mesh Configuration Protocol (MCP) : The use of Kubernetes to store Configuration data results in the coupling between Istio and Kubernetes, which limits the application of Istio in non-Kubernetes environments. MCP defines a standard protocol for delivering configuration data to the Istio control plane. Istio Pilot serves as an MCP Client. Any Server that implements THE MCP protocol can deliver configuration data to the Pilot through MCP. Thus uncoupling Istio and Kubernetes. If you want to learn more about MCP, see the link below.
  • Memory: an in-memory implementation of Config Controller for testing purposes.

Current CONFIGURATIONS of Istio include:

  • Virtual Service: defines traffic routing rules.
  • Destination Rule: Defines traffic processing rules related to a service or subset, including load balancing policies, connection pool size, circuit breaker Settings, and subset definition.
  • Gateway: defines the exposed services on the Gateway.
  • Service Entry: You can manually add an external Service to the Service grid by defining a Service Entry.
  • Envoy Filter: Add a custom Filter to an Envoy configuration through Pilot.

Service Controller

Service Controller is used to manage various Service Registries and propose Service discovery data. Currently, Istio supports the following Service Registries:

  • Kubernetes: With the Kubernetes Registry, services and instances defined in Kubernetes can be collected in Istio.
  • Consul: Interconnects with Consul Catalog and collects services defined in Consul to Istio.
  • MCP: Similar to the MCP Config Controller, it obtains Service and Service Instance from the MCP Server.
  • Memory: An in-memory implementation of Service Controller, mainly used for testing.

Discovery Service

The Discovery Service contains the following logic:

  • Start the gRPC Server and receive a connection request from the Envoy.
  • Receive the Envoy xDS request, retrieve configuration and Service information from the Config Controller and Service Controller, and generate a response message to send to the Envoy.
  • Listen for configuration change messages from the Config Controller and Service change messages from the Service Controller and push the configuration and Service change content to the Envoy through the xDS interface. (Note: At present, incremental change push is not implemented in Pilot. Each change push is full configuration. Performance problems may occur when there are many services in the grid.)

Pilot-Discovery business process

Pilot-disocvery consists of the following main business processes:

Initializes the major components of Pilot-Discovery

The Pilot/CMD/Pilot-Discovery /main.go command starts with the main method. In this method to create Pilot Server, the Server code in the file Pilot/PKG/bootstrap/Server. Go. The Server does the following initialization:

  • Create and initialize the Config Controller.
  • Create and initialize the Service Controller.
  • Create and initialize the Discovery Server. In Pilot, you create an HTTP Discovery Server based on the Envoy V1 API and a GPRC Discovery Server based on the Envoy V2 API. Since V1 has been abandoned, this paper will mainly analyze gRPC Discovery Server of V2 API.
  • Register the Discovery Server as an Event Handler for Config Controller and Service Controller to listen for configuration and Service change messages.

Create the gRPC Server and receive the connection request from the Envoy

Pilot Server creates a gRPC Server to listen for and receive xDS requests from envoys. Pilot/PKG/proxy/envoy/v2 / ads. Go the DiscoveryServer. StreamAggregatedResources method was registered as gRPC Server service processing method.

When gRPC Server received from Envoy connection, will be called DiscoveryServer. StreamAggregatedResources method, create a XdsConnection object in this method, A Goroutine is enabled to receive and process xDS requests from the client from the connection. If the configuration of the control plane changes, Pilot actively pushes the configuration change to the Envoy through this connection.

Push updates to the Envoy after configuration changes

This is one of the most complex business processes in Pilot, mainly because of the multiple channels and queues used in the code to merge and forward changing messages. The business process is as follows:

  1. The Config Controller or Service Controller notifies the Discovery Server via a callback when the configuration or Service changes, and the Discovery Server puts the change message into the Push Channel.
  2. The Discovery Server receives change messages from the Push Channel through a Goroutine that merges the change messages that have occurred continuously over time. If there are no new change messages after the specified time, the merged messages are added to a Push Queue.
  3. Another Goroutine fetches the change message from the Push Queue and generates XdsEvent, which is sent to the Push Channel of each client connection.
  4. In DiscoveryServer. StreamAggregatedResources method from Push XdsEvent Channel, then according to the context generated in line with the xDS DiscoveryResponse interface specification, Push via gRPC to Envoy. (gRPC will individually assigned to every client connection a goroutine to processing, so the StreamAggregatedResources processing methods of different client connection is in a different goroutine processing)

Respond to an unsolicited xDS request from an Envoy

A two-way Streaming gRPC service call is set up between the Pilot and Envoy, so the Pilot can push to the Envoy when the configuration changes, and the Envoy can initiate an xDS call to request the configuration. An Envoy’s unsolicited xDS request flows as follows:

  1. Envoy sends a DiscoveryRequest by creating a gRPC connection
  2. The Discovery Server receives the Discover Request from the XdsConnection via a Goroutine and sends the request to the ReqChannel
  3. Another Goroutine of the Discovery Server receives the DiscoveryRequest from the ReqChannel, generates the DiscoveryResponse that conforms to the xDS interface specification based on the context, and returns it to the Envoy.

Discovery Server business processes key code snippets

Here are the key snippets of Discovery Server code and the corresponding business logic annotations. For ease of reading, only the logic backbone is kept in the code, leaving out some unimportant details.

Key code for handling xDS requests and pushes

This part of the key code located on the istio. IO/istio/pilot/PKG/proxy/envoy/v2 / ads go file StreamAggregatedResources method. StreamAggregatedResources method are registered as gRPC Server handler, for each client connection, gRPC Server will start a goroutine to processing.

The code mainly contains the following business logic:

  • Receive an xDS request from the gRPC connection and place it in a channel reqChannel.
  • Receive the xDS request from the reqChannel, construct the response based on the type of the xDS request and send it to Envoy.
  • Receive notification of Service or Config changes from the Connection’s pushChannel, construct an xDS response message, and push the changes to the Envoy.
// StreamAggregatedResources implements the ADS interface. func (s *DiscoveryServer) StreamAggregatedResources(stream ads.AggregatedDiscoveryService_StreamAggregatedResourcesServer) error { ...... // Create a Goroutine to receive xDS requests from envoys, Con := newXdsConnection(peerAddr, stream) reqChannel := make(chan * xdsapi.discoveryRequest, 1) go receiveThread(con, reqChannel, &receiveError) ...... For {select{// Receive an active xDS request on reqChannel for Case discReq, ok := <-reqChannel: // Construct an xDS Response according to the type of the request and send it to the Envoy switch discReq.TypeUrl {case ClusterType: err := s.pushCds(con, s.globalPushContext(), versionInfo()) case ListenerType: err := s.pushLds(con, s.globalPushContext(), versionInfo()) case RouteType: err := s.pushRoute(con, s.globalPushContext(), versionInfo()) case EndpointType: err := s.pushEds(s.globalPushContext(), con, versionInfo(), Case pushEv := <-con. PushChannel: // Push changes to err := s.pushConnection(con, pushEv)}}}Copy the code

Key code to handle service and configuration changes

This part of the key code located on the istio. IO/istio/pilot/PKG/proxy/envoy/v2 / discovery go file, listens for service and configuration changes, And will change the message sent by the Channel to the previously mentioned StreamAggregatedResources combined method for processing.

ConfigUpdate is a callback function that handles service and configuration changes and is called by the Service Controller and Config Controller to notify Discovery Server of any changes.

Func (s * DiscoveryServer) ConfigUpdate (the req * model. The PushRequest) {inboundConfigUpdates. Increment () / / service or after configuration changes, Send a PushRequest to a pushChannel. PushChannel < -req}Copy the code

The debounce method merges successive PushRequests and, if no new PushRequest has been received for a period of time, initiates a PushRequest. To avoid heavy pressure on the system due to frequent service and configuration changes.

// The debounce helper function is implemented to enable mocking func debounce(ch chan *model.PushRequest, stopCh <-chan struct{}, pushFn func(req *model.PushRequest)) { ...... pushWorker := func() { eventDelay := time.Since(startDebounce) quietTime := time.Since(lastConfigUpdateTime) // it has PushRequest has been too long or quiet enough // No PushRequest has been received for a while, A further push the if eventDelay > = DebounceMax | | quietTime > = DebounceAfter {if the req. = nil { pushCounter++ adsLog.Infof("Push debounce stable[%d] %d: %v since last change, %v since last push, full=%v", pushCounter, debouncedEvents, quietTime, eventDelay, req.Full) free = false go push(req) req = nil debouncedEvents = 0 } } else { timeChan = time.After(DebounceAfter - quietTime) } } for { select { ...... case r := <-ch: lastConfigUpdateTime = time.Now() if debouncedEvents == 0 { timeChan = time.After(DebounceAfter) startDebounce = PushRequest req = req.Merge(r) case < -timechan: if free { pushWorker() } case <-stopCh: return } } }Copy the code

Complete business process

Refer to the reading

  • Mesh Configuration Protocol (MCP)
  • Pilot Decomposition
  • Istio service registration plug-in mechanism code parsing

About ServiceMesher community

The ServiceMesher community was launched in April 2018 by a group of volunteers who share the same values and ideals.

Community focus areas: container, microservice, Service Mesh, Serverless, embrace open source and cloud native, dedicated to promoting the vigorous development of Service Mesh in China.

Community’s official website: https://www.servicemesher.com